andygrove commented on issue #7451: URL: https://github.com/apache/datafusion-comet/issues/7451#issuecomment-6606079738
Found most of the Spark 4.1.3 vs 4.2.0 difference. It is the S3A input stream type, not Spark itself. The default of `fs.s3a.input.stream.type` differs between the Hadoop versions in the two `apache/spark` images (checked in `StreamIntegration.DEFAULT_STREAM_TYPE` of the `hadoop-aws` jars): - Hadoop 3.4.2 (Spark 4.1.3): `classic`. - Hadoop 3.5.0 (Spark 4.2.0): `analytics` (the S3 Analytics Accelerator). I reran Spark 4.2.0 with `spark.hadoop.fs.s3a.input.stream.type=classic`, everything else identical (full run, 103 queries, 2 iterations): | Spark only | Total | |---|---| | Spark 4.2.0, default (`analytics`) | 994 s | | Spark 4.2.0, `classic` | 1,294 s | | Spark 4.1.3, `classic` by default | 1,336 s | - `classic` makes Spark 4.2.0 30% slower (1.37x per query in geomean). That accounts for 300 s of the 342 s gap between 4.1.3 and 4.2.0 (88%). The remaining 42 s (3%) is something else between the two images. - 32 of 103 queries are more than 1.5x slower with `classic`. The biggest are q90 (3.6 s to 12.7 s), q09 (15.1 s to 45.2 s), q88 (18.7 s to 51.3 s), q53 (2.6 s to 7.4 s) and q99 (4.5 s to 11.0 s). Comet 1.2.0 speedup over Spark (total time): | Spark baseline | Speedup | |---|---| | Spark 4.2.0, `analytics` (default) | 2.04x | | Spark 4.2.0, `classic` | 2.65x | | Spark 4.1.3, `classic` (default) | 2.51x | So the smaller speedup on Spark 4.2.0 comes mostly from Spark's own S3 reads getting faster with the accelerator, and Comet's native scan (object_store) does not get that benefit. Questions for follow-up: 1. What does the accelerator do for Spark's reads that Comet's scan doesn't (read coalescing, prefetching of footers and column chunks, request parallelism)? Comet already has footer prefetch and a metadata cache, so the comparison could be per query on the ones where Spark gains most (q90, q09, q88, q53, q99). 2. Where does Comet still read through Hadoop S3A (listing, footers, shuffle and broadcast files), and would those paths benefit from the accelerator? 3. Spark 4.2.0 on `analytics` is the stronger baseline, so I'd use that for benchmark claims. Spark 4.1.3 numbers should mention that they read with the classic stream. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
