andygrove commented on issue #7344: URL: https://github.com/apache/datafusion-comet/issues/7344#issuecomment-6840376052
A follow-up to my comment above, with the full TPC-H suite and the larger sizes for q18. The setup is the same: two 16-core executors, SF1000, Spark 4.1.3. The jemalloc build is 1.2.0 with `--features jemalloc`. **Full suite.** All 22 queries in one application per size, so the executors live through the whole suite. One iteration. Spark used 50% off-heap and Comet + jemalloc 75% off-heap, each engine's best split for q18. | Engine | Pod GiB | Result | Suite time | Peak anon GiB | OOM kills | |---|---:|---|---:|---:|---:| | Spark | 32 | passed | 3,265 s | 26.9 | 0 | | Spark | 16 | passed after task retries | 3,540 s | 15.8 | 4 | | Spark | 12 | failed at q9 | | 11.5 | 5 | | Comet + jemalloc | 32 | passed | 2,235 s | 23.6 | 0 | | Comet + jemalloc | 16 | passed after task retries | 2,368 s | 15.6 | 2 | | Comet + jemalloc | 12 | failed at q5 | | 11.8 | 5 | - Over the whole suite, Comet + jemalloc needs about the same memory as Spark, and is about 1.5x faster. - The q18 result (10 GiB) does not carry over to the suite. Some queries need more memory, or it builds up across queries in long-lived executors. I haven't separated the two. - All failures were cgroup OOM kills, none were Comet pool errors. So even with jemalloc, some native memory is outside what the pool accounts for (question 3). - I didn't run the suite on the glibc release build, so this does not show how much jemalloc helps the suite. Given q18, I'd expect the glibc build to need more. - At 32 GiB Comet is slower than Spark on q7 (131 s vs 120 s), q8 (290 s vs 256 s) and q9 (380 s vs 347 s), and about even on q5. Those join-heavy queries are where both engines failed at 12 GiB. I'll look at them separately. **q18 at larger sizes (question 2).** These are the same sweep from 64 GiB down, comparing the two builds directly: | Pod GiB | 75% off-heap, glibc | 75% off-heap, jemalloc | 50% off-heap, glibc | 50% off-heap, jemalloc | |---:|---:|---:|---:|---:| | 64 | 145.3 s | 149.8 s | 148.5 s | 153.1 s | | 48 | 149.1 s | 164.8 s | 153.7 s | 162.3 s | | 32 | 152.3 s | 162.7 s | 158.0 s | 179.5 s | | 24 | 153.3 s | 183.9 s | 159.8 s | 186.2 s | - With plenty of memory, jemalloc is 3% slower at 64 GiB and 5-20% slower at 24-48 GiB, with more spread between runs. - These are single runs, so I can't yet tell a real cost from noise. - I used jemalloc's default decay settings. The decay time trades memory against speed: a shorter one returns memory sooner, and a longer one may cost less time. It's worth trying both before deciding on a default (question 4). - At 64 GiB, jemalloc's peak anon memory was 16.5 GiB against 24.1 GiB for glibc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
