dwsmith1983 opened a new issue, #7108:
URL: https://github.com/apache/datafusion-comet/issues/7108
### Describe the bug
A TPC-H SF10 run on Iceberg tables with the native Iceberg scan gets the JVM
killed by the cgroup OOM killer
(exit 137). This happens in a 14 GiB container with no swap, a 4g heap and
`spark.memory.offHeap.size=4g`. Every
figure Comet reports stays well inside the limits:
- JVM Native Memory Tracking is flat at about 4.7 GiB.
- Comet's memory log peaks at `allocated 3349 MiB, reserved 3318 MiB`.
- Anonymous memory outside the JVM still reaches about 9.2 GiB.
- The kernel record shows anon-rss of about 14.6 GB, with page cache already
reclaimed.
Most of the difference is memory that has already been freed. Between
queries Comet's live figure drops to about
2 MiB while RSS stays near 12 GiB, and `jcmd <pid> System.trim_native_heap`
gives back 5.1 to 6.0 GiB in one call.
The release build uses the system allocator (glibc). The `jemalloc` and
`mimalloc` features exist, but neither is
in the default build.
On the full 22-query suite (two warm-up rounds and one timed round, 72
queries), changing only the allocator makes
the run pass:
| change | result | peak anon | time per round |
|---|---|---|---|
| none (glibc) | killed | 13.94 GiB | 35.9 s |
| `LD_PRELOAD` of the distro's `libjemalloc.so.2` | passes, all queries |
11.48 GiB | 32.5 to 34.8 s |
| `MALLOC_ARENA_MAX=2` | passes, all queries | 9.33 GiB | 49.3 to 50.0 s |
| `spark.memory.offHeap.size=2g` | RSS unchanged (12.56 GiB at the same
point); a sort then fails to reserve memory | | |
This is not specific to Iceberg. I ran the same Comet build on Parquet and
Delta copies of the same data:
- **Parquet** (`CometNativeScanExec`) finished the full suite at an 11.5 GiB
peak.
- **Delta** (`CometDeltaNativeScanExec`) finished it at 12.0 GiB.
- On both, a trim releases 3.6 to 5.1 GiB while live native memory is a few
MiB.
Iceberg sits 1.5 to 2 GiB higher than the other two and is the one that
crosses the limit; I'll put that part on
#6126.
### Steps to reproduce
This loop runs TPC-H q09 ten times in one JVM. I reproduced it three times
on the setup below.
1. **Image.** Start from `apache/spark:4.1.3-scala2.13-java17-ubuntu` and
add:
- `iceberg-spark-runtime-4.1_2.13-1.12.0.jar` (from Maven Central);
- a Comet jar built from main with `make release PROFILES="-Pspark-4.1"`;
- `tpchgen-cli` 3.0.0;
- Ubuntu's `libjemalloc2`, which is only for the comparison run.
2. **Data.** Generate SF10 with `tpchgen-cli`: part, supplier, partsupp and
nation as one file each, orders and
lineitem in 10 parts each. Load the six tables into an Iceberg Hadoop
catalog with default write settings. Run the
load with `spark.sql.files.maxPartitionBytes=16g` and
`spark.sql.files.openCostInBytes=16g`, so that each generator
file becomes one Iceberg data file (lineitem ends up as 10 files of about
155 MB):
```sql
CREATE NAMESPACE IF NOT EXISTS local.tpch;
CREATE TABLE local.tpch.lineitem USING iceberg AS SELECT * FROM
parquet.`/data/gen/lineitem`;
-- same for orders, part, partsupp, supplier, nation
```
3. **Run.** Run q09 ten times in a single `spark-sql` session, inside
`docker run --rm --memory 14g --memory-swap 14g --cpus 8`:
```
spark-sql --master 'local[8]' --driver-memory 4g \
--conf 'spark.driver.extraJavaOptions=-Xms4g -XX:+AlwaysPreTouch' \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=4g \
--conf spark.driver.extraClassPath=/opt/comet/comet-spark.jar \
--conf spark.plugins=org.apache.spark.CometPlugin \
--conf
spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager
\
--conf spark.comet.shuffle.enabled=true \
--conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.local.type=hadoop \
--conf spark.sql.catalog.local.warehouse=/data/iceberg \
-f loop.sql
```
Results:
| allocator | exit | q09 runs completed | peak anon |
|---|---|---|---|
| glibc (default) | 137, cgroup OOM kill (`CONSTRAINT_MEMCG`,
`anon-rss:14610748kB`) | 8 of 10 (6 in two other runs) | 13.93 GiB |
| `LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2` | 0 | 10 of 10 |
11.59 GiB and 12.98 GiB in two runs |
Two conditions are needed for the kill:
- **The 10-file layout.** With Spark's default read splits (lineitem as 20
files), the same loop peaks at 8.6 GiB and
passes.
- **The pre-touched heap.** Without it the loop peaks about 2 GiB lower and
passes. Pre-touching only makes the heap
resident from the start, which it becomes anyway in a long-running
executor.
The margin under jemalloc is also small in this loop. The clearer signal is
the trim: a single
`System.trim_native_heap` releases 5 to 6 GiB.
The Dockerfile and scripts are small. I can attach them if that helps.
### Expected behavior
The executor stays inside a container sized from the heap plus
`spark.memory.offHeap.size` plus overhead,
or Comet's memory log shows that resident memory is about to cross the
container size.
### Additional context
Environment:
- Comet main at 4a8a9496d119341b6f8ea94959312340c353583c, default cargo
features.
- Spark 4.1.3, Iceberg 1.12.0, Temurin 17.0.19, glibc 2.35 (Ubuntu 22.04).
- Docker Desktop on Apple silicon (linuxkit aarch64, cgroup v2), container
`--memory 14g --memory-swap 14g --cpus 8`,
`local[8]`.
Related:
- #4576 measured the resident versus allocated gap with jemalloc. This shows
what that gap looks like with the
default glibc allocator, and that a large part of it is freed memory glibc
does not return.
- #6167 would make this visible: the log would have shown about 12 GiB of
RssAnon next to 2 MiB allocated. Its
warning would be one way to catch this before the kernel does.
- #6032: the tuning guide could mention the allocator, including
`MALLOC_ARENA_MAX` or a jemalloc build for
container deployments, until the default changes.
Questions for the maintainers:
- Should release builds use jemalloc or mimalloc by default?
- Failing that, should the deployment docs recommend `MALLOC_ARENA_MAX` or
preloading jemalloc?
I have per-run memory.stat samples, NMT summaries, trim results and the
kernel OOM records if they would help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
