viirya commented on PR #59327: URL: https://github.com/apache/spark/pull/59327#issuecomment-6106325904
@HyukjinKwon Thanks for raising the listing cost. I measured it and it is real. On the end-event reparse path, with 210 MB of uncompressed event lines (built from `spark-events/local-1642039451826` with randomized digits, median of 3 runs, JDK 21): | codec | file size | reparsed tail | `Source.getLines()` (before this PR) | bounded reader (this PR) | |---|---|---|---|---| | lz4 | 56 MB | 155 MB | 0.17 s | 0.90 s | | zstd | 16 MB | 195 MB | 0.17 s | 1.05 s | So this PR would make the listing of compressed completed apps about 5-6x slower on this step. The cause is the per-character `BufferedReader.read()` in `boundedLines`, which full replays have already been paying since SPARK-59407. I opened SPARK-60158 / #59360 to read decoded characters in bulk there while keeping the current semantics. With it, a full replay of the same logs takes 0.20-0.22 s instead of 1.10-1.13 s, about the same as `Source.getLines()`. I'd like to merge #59360 first and this PR after it, so that master never has the slower listing, and to backport the two together. Please hold off on merging this PR until then. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
