andygrove commented on issue #4576: URL: https://github.com/apache/datafusion-comet/issues/4576#issuecomment-5704647453
Correction to my previous comment. I concluded there that the accounting gap is small. That was measured with counters which exclude resident memory, and it is wrong. Measuring jemalloc's `resident` inverts the conclusion. ## The three gaps, TPC-H SF100, per executor | gap | 2g budget p50 | 2g max | 16g budget p50 | 16g max | | --- | --- | --- | --- | --- | | pool reserved to native allocated | 13-14 MB | 405-522 MB | 22-23 MB | 429-504 MB | | native allocated to jemalloc allocated | 19 MB | 302-396 MB | 21 MB | 278-457 MB | | **pool reserved to jemalloc resident** | **1.6-1.7 GB** | **3.7-3.9 GB** | **1.5-1.6 GB** | **3.9-4.2 GB** | `resident/allocated` is 6x to 8x at the median and peaks near 600x. Pool bookkeeping is accurate to within tens of megabytes; the memory that can get an executor killed is pages jemalloc has faulted in and not returned to the OS. ## Resident does not scale with the configured budget This is the part that matters most. Resident is essentially the same at both budgets: | | resident p50 | resident max | | --- | --- | --- | | 2g off-heap | 1895 / 1856 MB | 4263 / 4027 MB | | 16g off-heap | 1992 / 1937 MB | 4584 / 4288 MB | An 8x difference in `spark.memory.offHeap.size` produced no meaningful difference in real resident memory. At 16g the ~4.5 GB peak sits comfortably inside the allotment. At 2g the same ~4.3 GB peak is more than double it. So lowering `spark.memory.offHeap.size`, or lowering `spark.comet.exec.memoryPool.fraction`, does not lower Comet's real footprint. It only lowers what Comet is permitted to reserve. That is a plausible explanation for why tuning the fraction has never been a reliable way to avoid container OOM: it constrains the accounting, not the allocation. ## This is not a jemalloc counter artifact Sampled `/proc/<executor pid>/status` directly during a run: ``` VmRSS: 5219480 kB VmHWM: 5633052 kB VmRSS: 5211048 kB VmHWM: 5615380 kB ``` About 5.2 GB resident per executor, against jemalloc reporting roughly 4.2 GB. The kernel corroborates the finding, and the remaining ~1 GB sits outside jemalloc entirely (JVM heap and non-heap, `CometArrowAllocator`, Spark's own off-heap, mapped files). ## What this implies for the proposed RSS circuit breaker The direction in this issue looks right, with one change to the ground truth. This issue says "Ground truth comes from the allocator's resident stat". I would suggest the kernel's number instead, for two reasons. First, the default build does not use jemalloc. `native/core/src/lib.rs` selects jemalloc only when that feature is requested, then mimalloc, otherwise `std::alloc::System`, and the default feature set is `["hdfs-opendal"]`. A guard reading `stats::resident` would protect only non-default builds. Second, jemalloc's resident covers jemalloc's arenas, while the cgroup kills on the whole container. The ~1 GB difference above is exactly the part a jemalloc-only signal cannot see. Allocator-agnostic alternatives, both a periodic file read rather than a per-allocation cost: - cgroup v2 `memory.current`, which is the number the OOM killer actually compares against - `/proc/self/status` `VmRSS` as a fallback Comet reads neither today. ## Worth trying before building anything A 6x to 8x resident-to-allocated ratio suggests pages are simply not being returned. Before adding a guard it is worth measuring jemalloc decay tuning (`dirty_decay_ms`, `muzzy_decay_ms` via `_RJEM_MALLOC_CONF`). If prompt decay closes most of the gap, the fix is a configuration default rather than a new subsystem. I have not tested this yet. Also relevant: refusing reservations is not the useful response to RSS pressure. As reported in my previous comment, gating reservations refused 30 to 44 times per run and the budget was still exceeded in 3 of 3 runs, because the overshoot is not in reservations. ## Caveats TPC-H SF100 only, one traced run per budget, two executors, and jemalloc as the allocator, which is not the default. Standalone Spark enforces no container limit, so nothing was killed here. Some resident memory is arena capacity that would be reused rather than pure waste. The traces are retained if anyone wants a different cut of the data. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
