comphead opened a new pull request, #7492: URL: https://github.com/apache/datafusion-comet/pull/7492
## Which issue does this PR close? Closes #2216. ## Rationale for this change Comet compresses a broadcast with `spark.io.compression.codec`, and Spark's `TorrentBroadcast` compresses it again with the same codec when `spark.broadcast.compress` is `true`, its default. The measurements in #2216 show the second pass costs driver time before the join can start. That was about 0.2 s per 300 to 460 MiB broadcast with `lz4` and about 0.85 s for a 284 MiB broadcast with `zstd` level 5, and it saved 0 to 10% of the bytes. A Comet-side change would save less than the existing Spark setting and cost executor heap, so this PR documents the setting instead. ## What changes are included in this PR? A "Broadcast compression" subsection under "Broadcast Joins" in the operator tuning guide. It explains that Comet broadcasts are compressed twice, what the second pass costs and saves, and when to set `spark.broadcast.compress=false`, including that the setting also applies to broadcasts that Comet does not create. ## How are these changes tested? Documentation only. `prettier --check` passes on the changed file. The numbers come from the measurements in #2216. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
