steveloughran commented on issue #1708: HADOOP-16696: Always read ahead config, to use read ahead even for non sequential reads. URL: https://github.com/apache/hadoop/pull/1708#issuecomment-562095185 # setting the readahead I have seen the Parquet readFooter code and agree, readahead matters there. For the S3A readahead (HADOOP-13208? I forget) we worked off ORC traces and optimised for the PositionedReadable reads, which offen surface as a pair of back-to-back reads; Setting a minimum read ahead range handles this codepath. If you're going near read ahead, it would be good if you implement `CanSetReadahead`, as then Applications can configure their read ahead in a filesystem independent manner. I worry a bit about this as with object stores, the effective read ahead range does seem to depend on distance from and hence latency to the store. Presumably your benchmarks were executed in Azure? That is the likely real-world deployment. I'd be curious * what experiments you've done to work out what a good range is with your store/client? * how things behave over long-distance connections? Regarding the patch: when would we ever really want to turn this off? Is a serious question. For ORC, our past studies with S3A say "readahead". For Parquet, the implication is the same as needed. What other uses it do we care about? an HBase trace would be good here. A key thing I'm trying to emphasise is that a small readahead is not expensive compared to the delays caused by having to issue two near-adjacent GET Requests. The main penalty is on a seek outside the range we have to drain the stream, which is ~(bytes-remaining/bandwidth). I would prefer that rather than a boolean value, the stream always does the readahead except when the range == 0. That way setReadahead(0) and the config options are the way to disable it. And as noted: I don't think it should be disabled on random IO. ## S3A reads always show significant speed up with readahead > 0; the only issue is what makes a good value? # Testing Looking at the test, if you add some package-private method to query the readahead state, e.g (a) the range value and (b) bytes remaining in the current buffer. Then you can verify that * read(bytes[n]) decrements the buffer by n * seek(getPos()) doesn't drain the buffer; * seek(getPos() + 1) doesn't it either. a read() should then decrement it by one * seek(0), seek(getPos()) doesn't drain the buffer; a read() should then decrement it by one * setReadahead(0) removes the buffer on the next seek+read outside the buffer. (There's no point discarding the current read, is there?) Yes I am adding extra homework. Sorry. But as others will note: I like my tests, and I like tests in the hadoop- test suites as they catch regressions without needing any external tests.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
