andygrove commented on issue #4576: URL: https://github.com/apache/datafusion-comet/issues/4576#issuecomment-5709022407
Notes from the container memory guard experiment (#5993, now closed). **What was tried:** sample the container's cgroup `memory.current` at execution checkpoints and fail the current task once usage crosses a configured fraction of `memory.max`, so Spark retries one task instead of the kernel killing the whole executor. **What works:** validated on k3s (Spark-on-K8s, TPC-H SF100 Q9, 1 executor, 9Gi pod limit). cgroup v2 discovery works inside a pod, reads the correct limit, computes the trip point, and the trip propagates as `CometNativeException` and fails the task as intended. The plumbing is sound. **Why it does not work:** `memory.current` counts reclaimable page cache, so it is the wrong signal. | | guard on | guard off | |---|---|---| | result | 24 trips, all tasks of the stage failed, every retry re-tripped, job failed | Q9 completed in 58.62s, 175 rows | | OOMKilled | none | none | Same image, same sizing, same query. Sampling the control run's cgroup: ``` ts cur_MB anon_MB file_MB pct 04:17:13 580 465 106 6 04:17:25 9215 1983 7099 99 04:18:10 9215 5017 4085 99 04:18:13 4732 3070 1532 51 ``` `memory.current` saturates within about 12 seconds and stays pinned at the ceiling, but `anon`, the non-reclaimable part that actually kills a container, peaks at 5044 MB, 55% of the 9216 MB limit. The rest is page cache from reading parquet. As `anon` grew from ~2000 to ~5000 MB, `file` fell from ~7100 to ~4085 MB: the kernel reclaims cache on demand and the total stays at the limit. So a saturated `memory.current` is the normal steady state of any container that reads files, not a sign of distress. The lowest trip observed was 98.9% of the limit, never the configured 90%, because saturation happens before the first sample. Raising the threshold does not fix this, it only delays the false trip. **If picked up again:** threshold on the non-reclaimable portion (`anon` plus unevictable from `memory.stat`) or on PSI (`memory.pressure`), not on `memory.current`, and re-run the same A/B. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
