dwsmith1983 commented on PR #5613: URL: https://github.com/apache/datafusion-comet/pull/5613#issuecomment-6060911076
Spill pressure run on the current head (`155e74060`) against its merge base on main (`f80042f78`): TPC-H SF10, one 8-core executor, `fair_unified` with 2g off-heap so each task gets 256 MB, base, head, head, base at three iterations of the 22 queries in one Spark application per arm, then q18 and q21 at five iterations alternated. | workload | metric | base | head | |---|---|---|---| | 22 queries x3, per arm | disk bytes spilled | 4,215,250,560 | 4,215,250,560 | | 22 queries x3, per arm | spills | 180 | 180 | | q18 x5, per arm | spills / bytes | 80 / 1,405,702,160 | 80 / 1,405,702,160 | | q21 x5, per arm | spills | 0 | 0 | | every arm | failed tasks / executors lost | 0 / 0 | 0 / 0 | Every query result hashes the same across arms and matches the earlier runs. Executor stderr has no lines for `fair limit`, `only got`, `overcommit`, `OutOfMemory`, `NoSuchElementException` or `key not found` on either side. This ran in Docker rather than directly on the host as the earlier runs did, with smaller heaps (driver 1g, executor 1536m) to fit the VM; the off-heap pool and cores are unchanged. Main at `8166586fa`, run the same way, reproduces the earlier numbers exactly (360 spills and 22,361,276,112 bytes for the 22 queries), so the lower spill volume here comes from main itself, most likely the lz4 spill compression in #6698. On wall time, the sum of per-query medians was 34.13 s on head against 34.32 s on base, q18 3.21 s against 3.30 s and q21 5.78 s against 5.90 s. The container swapped during the full-suite arms on both sides, so I read that as no slower rather than faster. The contention gain this PR targets shows in the microbenchmarks in the description, not at this level. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
