dwsmith1983 commented on issue #6126:
URL:
https://github.com/apache/datafusion-comet/issues/6126#issuecomment-6261209204
Here are some figures on the native Iceberg scan.
Setup:
- TPC-H SF10, Comet main (4a8a9496d), Iceberg 1.12.0.
- One 14 GiB container: 4g heap, 4g off-heap, `local[8]`.
- The same data was also loaded as plain Parquet and as Delta.
The OOM kill in #7108 comes mostly from glibc keeping freed memory, which
happens on all three formats. Iceberg holds
more resident memory than the other two, though, and it is the only one that
gets killed.
To compare the scans more closely I set
`spark.sql.autoBroadcastJoinThreshold=-1`. Join strategies then match across
the three formats:
- Parquet and Delta produce identical plans and identical shuffle bytes.
- Iceberg's plans are the same apart from two small differences: q07 reads
nation twice instead of reusing the
exchange, and q22 has one reused subquery fewer.
- Iceberg's shuffle bytes are within 6% on the large queries; q17 is the
widest at 0.97 against 1.03 GB.
Anonymous memory in the container, one run each:
| point | Parquet | Delta | Iceberg |
|---|---|---|---|
| after q05 | 8.21 GiB | 9.09 GiB | 10.68 GiB |
| after q09 | 10.55 GiB | 10.47 GiB | 12.35 GiB |
| peak over the suite | 12.43 GiB, passes | 11.77 GiB, passes | 13.93 GiB,
killed |
| Comet live peak (allocated) | 3.75 GiB | 3.87 GiB | 2.35 GiB |
Parquet and Delta are within run-to-run noise of each other, which is about
0.5 GiB. Iceberg is 1.5 to 2 GiB above
both at every point, even though Comet's live peak is lowest on Iceberg. So
the extra resident memory doesn't show up
in what Comet allocates or reserves. That fits the scan holding buffers
outside the pool, which is what this issue is
about.
The gap is small on scan-only queries. Running q01 11 times on Iceberg adds
about 60 MB per run, while on Parquet the
footprint stays flat. q06 stays flat on both.
I haven't measured where the extra memory goes, and I have no heap profile
yet. Reading iceberg-rust at Comet's
pinned rev (1f3bc340), I found two places that could add to the per-task
working set:
- `ArrowFileReader::get_byte_ranges` merges ranges closer than 1 MiB
(`DEFAULT_RANGE_COALESCE_BYTES`) and fetches up
to 10 merged ranges at once. With 128 MB row groups, one merged range can
cover most of a row group's projected
columns.
- `OpenDalReader::read` ends in `.to_bytes()`, which copies the range into
one buffer when opendal returns it in more
than one chunk.
`dataFileConcurrencyLimit` was left at its default of 1 throughout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
