steveloughran commented on PR #1139: URL: https://github.com/apache/parquet-mr/pull/1139#issuecomment-1938465333
Update: I've been looking at overlapping ranges. * The base implementation doesn't care. * the specification and contract tests don't say they MUST be separate * I'm now worried that a bit of the s3a range coalescing code is getting overlaps wrong. Here's what I propose * for hadoop api we say "no overlap" and explicitly raise IOE subclass in the invocation if so. (i.e. always fail fast) * add tests * for parquet, we do the same validation so behaviour is consistent across all impls * If there is an overlap, what to do there? fail or downgrade? The reason i'd like to fail fast in overlaps is because it lines up nicely with ranged HTTP Get requests. S3 doesn't support that, but others might already and if not, could be persuaded. Currently the s3a client will coalesce "close enough" ranges into single GETs, discarding the data in between. This is more efficient for http connection use and latency -even costs less for in-EC2 queries. I do want to add azure abfs vector reads this year. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
