andygrove opened a new pull request, #2805: URL: https://github.com/apache/datafusion-ballista/pull/2805
> **Stacked on #2802.** Review only the last commit, `perf: default the shuffle compression codec to zstd`. # Which issue does this PR close? No issue. Split out of #2792. Follow-up for string-heavy shuffles: #2804. # Rationale for this change Shuffles are often bound by how many bytes they move: across the network between executors, or through local disk and the page cache when they spill. For the columns TPC-H q5 shuffles from lineitem (`l_orderkey`, `l_suppkey`, `l_extendedprice`, `l_discount`), Arrow IPC with LZ4 stores about 16 bytes per row, and with zstd about 8.2. Spark + Comet's shuffle stores the same rows at about 14.8. zstd costs a little more CPU to compress, but that is small next to the bytes it saves. Measured with the other changes in #2792 applied, LZ4 against zstd, one iteration each: | TPC-H | LZ4 (s) | zstd (s) | |---|---:|---:| | SF1000, two nodes, 2 executors × 16 cores, 256 partitions | 1702 | 1318 | | SF100, one machine, 2 executors × 8 cores, 16 partitions | 138.8 | 132.3 | At SF1000 the biggest changes are q5 274 to 170 s, q3 113 to 76 s, q9 156 to 117 s, q12 50 to 35 s, q18 183 to 145 s and q21 199 to 161 s. No query got slower beyond noise: q11 went from 5.4 to 5.7 s. zstd is faster even on one machine, where the shuffle never crosses a network, because smaller files mean less disk and page-cache traffic. #2802 is a prerequisite. Without it, the sort shuffle spill manager keeps one writer open per spilled output partition, and with zstd each open writer holds its own compression context outside the memory pool. # What changes are included in this PR? `ballista.shuffle.compression.codec` defaults to `zstd` instead of `lz4`, with a description saying why, and `configs.md` is regenerated. Readers take the codec from the IPC stream metadata, so mixed old and new files read fine, and `lz4` and `none` remain available. The Flight serve path (`ballista.shuffle.remote_read_prefer_flight`, off by default) still re-encodes with LZ4 and is unchanged here. # Are there any user-facing changes? Yes. Shuffle files and block-transfer remote reads use zstd unless `ballista.shuffle.compression.codec` is set. Set it to `lz4` to keep the previous behavior. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
