andygrove commented on PR #7304: URL: https://github.com/apache/datafusion-comet/pull/7304#issuecomment-6528220294
Some data for the timer arm (`-XX:TrimNativeHeapInterval`), from a different workload: the nested TPC-H suite in `benchmarks/nested` at SF1000 on Iceberg. **Setup.** Comet 1.2.0 (system allocator, so glibc) on Spark 4.1.3 and JDK 17.0.20. There are 32 executors with 16 task slots each, a 32 GiB heap, 64 GiB off-heap (`fair_unified`), 10 GiB overhead and a 106 GiB pod limit. Each run is all 22 queries, 3 iterations each, in one application. **Without trimming**, q21 got 11 executors OOMKilled and the query aborted after its stage hit Spark's limit on attempts. Comet's own figures stayed well inside their limits: - The killed executors had 28-33 GiB allocated, against a 64 GiB pool. - The same executors had survived 36 GiB allocated in q21's first iteration. - Between stages they dropped to 0 native plans and about 1 MiB allocated. On fresh executors, q21 alone passes three iterations in a row at the same 35.6 GiB per-executor peak. So the memory that kills the pods builds up over the earlier queries and stages, which matches the description here. **With `-XX:TrimNativeHeapInterval=5000`** (plus `-Xlog:trimnative`), the full suite passed with no failed tasks. | | no trim | trim every 5 s | |---|---|---| | Executors OOMKilled | 11 | 0 | | q21 (3 iterations) | 242 s, then aborted | 204 / 202 / 205 s | | The other 21 queries (sum of best iterations) | 567 s | 588 s (+3.7%) | | Peak process RSS per executor (from the trim log) | | 61-70 GiB, median 66 GiB | | Largest single trim | | 31.8 GiB freed | | Trim duration | | median 424 ms, p99 3.5 s, max 5.3 s | | Time in trims per executor | | about 280 s of a 2,460 s run (11%) | A few observations: - **Retention.** The largest trims (22-32 GiB per executor) came at the same scale as the 28-34.5 GiB you measured after the map wave. Most of what glibc kept was returned. - **Cost.** The timer cost about 4% on the queries that did not need it. That is one run each, so some of it may be noise. Many trims free almost nothing but still take the arena locks, so a longer interval may keep most of the benefit for less; I have not measured that yet. - **Mid-stage trims.** Trims mid-stage did not visibly hurt the heavy stages. q21's sort-merge-join stage, which runs one 512-task wave, took the same time as on fresh executors without trimming. This doesn't decide the timer versus the idle trigger, since I haven't run this PR's build. It does show the JDK flag is enough to keep this workload alive on 1.2.0 today, for about a 4% cost. That's an argument for documenting it, as suggested above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
