andygrove opened a new issue, #6167: URL: https://github.com/apache/datafusion-comet/issues/6167
### What is the problem the feature request solves? #6162 adds a periodic executor log line that reports `allocated` (bytes live through Rust's global allocator) next to `reserved` (bytes in Comet's memory pools), and warns when the estimated native footprint exceeds what the container allows. Neither figure is what gets an executor killed. The kernel and the cluster manager act on dirty resident pages, and allocated bytes differ from those in both directions: - **Allocated but not resident.** Capacity that is reserved and never written counts in full: `Vec::with_capacity`, pre-sized hash tables, and large zeroed allocations that `calloc` serves from fresh `mmap` pages. Pages are only charged once they are touched, so this is memory that might become resident on the next write loop, not memory that is resident now. - **Resident but not allocated.** This includes allocator fragmentation, freed pages the allocator retains rather than returning to the OS, allocations made by C dependencies such as zstd, and Arrow buffers on the JVM side. The second group can dominate. A measurement in #4576 found jemalloc's `resident` at 6x to 8x `allocated` at the median, and about 4.3 GB resident whether `spark.memory.offHeap.size` was 2g or 16g. So the log can show a comfortable gap while the executor is close to its container limit. The container warning in #6162 is an estimate built from allocated bytes plus Spark's off-heap usage, and it can be wrong in either direction. ### Describe the potential solution Add the process's resident anonymous memory to the memory usage log, and base the warning on it. 1. **Report resident memory.** Read `RssAnon` from `/proc/self/status` on the log's timer thread and add it to the line, for example `allocated 2381.9 MiB, reserved 2350.9 MiB, resident 9120.0 MiB (...)`. `RssAnon` counts dirty anonymous pages for the whole process, including the JVM heap. It excludes page cache, which is what made the `memory.current` threshold in #5993 fire on reclaimable memory. 2. **Warn on resident memory against the container size.** Compare against `spark.executor.memory` + `spark.memory.offHeap.size` + the memory overhead (+ `spark.executor.pyspark.memory` when set), using the overhead calculation from #6162, and warn when `RssAnon` passes a high fraction of it, for example 90%. This replaces the estimate in #6162 with the quantity the cluster manager enforces. 3. **Keep `allocated` and `reserved` for attribution.** They remain the only view of how much of Comet's memory the pools do not track. The tuning guide should say that `allocated` is what Rust code has asked for, which can be higher or lower than what is resident, and that `resident` is what counts towards the container. This is observability only. Acting on the number, as the circuit breaker in #4576 proposes, is separate work. ### Additional context - `RssAnon` is Linux-only. On other platforms the line would omit it and the warning would be skipped. With jemalloc, `stats.resident` is a native-only alternative, but it excludes the JVM heap, which also counts towards the container. - On cgroup v2, `memory.stat`'s `anon` field is the container-wide equivalent and includes child processes such as Python workers. It is worth considering if Python UDF workers should count. - Related: #4576 (allocator-level accounting and the RSS circuit breaker), #5997 (bounding the JVM Arrow allocator and native reclaim), #5993 (closed cgroup `memory.current` guard), #6162 (the memory usage log). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
