andygrove opened a new issue, #7688:
URL: https://github.com/apache/datafusion-comet/issues/7688

   ### What is the problem the feature request solves?
   
   The native Parquet scan registers no memory reservation, so its working set 
is invisible to the task memory pool and to Spark behind it. #6125 prototyped 
charging it and was closed after review. This issue records what the prototype 
showed and what review found, so a later attempt does not start from scratch.
   
   **The gap.** With tracing on (`jemalloc,alloc-accounting`), a TPC-H SF100 
run (2 executors x 8 cores, 16g off-heap) showed scan-only queries such as q1, 
q6 and q19 allocating 100 to 150 MiB per executor natively while reserving 
almost nothing from the pool. The gap is a fixed working set per running scan, 
not something that grows over time. Per partition it is the compressed column 
chunks of the row group being decoded (arrow-rs fetches the projected column 
chunks of one row group before decoding it, and DataFusion reads one file per 
partition at a time), plus the decompressed pages and output batch the decoder 
builds. For q1 on SF100 `lineitem` the seven projected columns are 9.1 MiB 
compressed per row group, which matches the roughly 14 MiB per task that was 
unaccounted. The size is set by the file layout (row-group size times projected 
columns), so wide tables with large row groups can hold far more per task.
   
   ### Describe the potential solution
   
   The prototype is on the branch `parquet-scan-memory-accounting` of 
`andygrove/datafusion-comet` (head `3cb14c0f95`). All of it is in 
`EagerPageIndexReaderFactory`, which already owns the scan's `AsyncFileReader`:
   
   - **Data pages, charged exactly.** Each buffer returned by `get_byte_ranges` 
is wrapped in a `Bytes::from_owner` whose owner holds its share of a 
`MemoryReservation`. The reservation is released when arrow-rs drops the last 
reference to the buffer or to any page sliced from it, so the decoder needs no 
change.
   - **Decode buffers, charged as an estimate.** arrow-rs allocates 
decompressed pages and the output batch with no hook to observe them, so each 
open reader held one default page (1 MiB) per projected leaf column plus one 
`batch_size` output batch.
   - **Refusal does not fail the scan.** The prototype used `try_grow` and kept 
reading unaccounted when refused, counting the bytes in a new 
`scan_io_unreserved_bytes` metric.
   
   **Measured results** (TPC-H SF100, same commit with and without the change, 
native allocated bytes minus the pool's reserved total at the same instant, per 
executor):
   
   | | Median gap | p95 gap |
   |---|---|---|
   | Before | 21.9 MiB | 140.2 MiB |
   | After | 7.9 MiB | 94.0 MiB |
   
   For scan-heavy queries the median gap dropped from 71 to 127 MiB to 6 to 19 
MiB. Total run time was unchanged (208.7 s vs 208.8 s) and all 22 result hashes 
matched. At reduced off-heap size (`fair_unified`), no reservation was refused 
even at 2g, so the refusal path was not exercised on this workload. Spark's 
disk spill rose at 2g, mostly in q9 (8.0 to 15.0 GiB) and q10 (5.6 to 6.9 GiB). 
That is consistent with operators seeing less free memory, but it was a single 
run, so it needs repeating.
   
   ### What review found
   
   These need to be resolved by any follow-up. Thanks to @comphead, @viirya, 
@sunchao, @parthchandra, @rich7420 and @dwsmith1983 for the detailed reviews.
   
   **Error handling**
   - `try_grow` can fail for reasons other than refusal. 
`CometFairMemoryPool::try_grow` returns `ResourcesExhausted` only for its own 
local-limit refusal and for a partial grant. Anything else (a 
`SparkOutOfMemoryError` from another consumer's spill, an 
`InterruptedException` from a killed task, a JNI failure) arrives as 
`DataFusionError::External`. The prototype used `is_err()`, which swallowed 
those, so a killed task kept reading. Only `ResourcesExhausted` should mean 
"keep reading unaccounted". Everything else must propagate, with a test using a 
pool whose `try_grow` returns `External`.
   - `grow` instead of `try_grow` is worth reconsidering. Since #6128 the 
unified pools no longer panic on `grow`. They record the bytes and carry what 
Spark declined as overcommit. The fetched pages already exist when charged, so 
this is not admission control. With `grow` the pressure lands on consumers that 
can spill, and both the error-classification question and 
`scan_io_unreserved_bytes` go away. The cost is that `grow` ignores the fair 
limit.
   - If `try_grow` stays, a scan that cannot act on a refusal keeps asking 
Spark for the whole overcommit debt on every call: a JNI round trip per row 
group, and possibly a JVM spill that buys nothing (Spark grants part, the 
unified pool hands the whole grant back, the scan reads anyway). Under 
`greedy_unified` each refusal also logs a warning and dumps all consumers' 
memory usage. The reader should back off after a refusal.
   
   **Consumer registration**
   - One `MemoryConsumer` per reader shrinks the fair pool for the whole task. 
The fair pool's limit is `pool_size / num`, so each registration lowers every 
other consumer's ceiling regardless of how many bytes the scan holds (roughly 
25% in an aggregate + sort + shuffle-writer task). DataFusion also creates a 
second reader per file for bloom-filter pruning and keeps both alive across 
`load_bloom_filters`, so the count churns and the decode estimate is charged 
twice. Register once on the factory, one consumer per scan operator, and hand 
each reader a `new_empty()` handle. This is the same pattern #6247 used for the 
writers. This may also explain part of the q9/q10 spill increase.
   
   **Decode estimate**
   - `leaves * 1 MiB + batch_size * row_width` is fixed at plan time and 
charged before any metadata is read. A 500-column projection charged about 508 
MiB per open reader (532,480,000 bytes in one reproduction) even for a one-row 
or 1 KiB file, and a scan over many small files pays it on every open. Capping 
by file size does not bound it either, since decompressed pages can exceed the 
file.
   - Size it from the file instead. The factory already loads the footer and 
offset index, so per projected leaf it could use the smaller of one page and 
the largest selected column chunk's uncompressed size, plus `min(batch_size, 
num_rows) * row_width`. Charge it at the first `get_byte_ranges` rather than in 
`create_reader`, which keeps it out of the planning step and means a fully 
pruned file never pays it.
   - `row_width` summed top-level fields (a struct counted as a flat 32 bytes) 
while the page term counted leaves, so the batch term under-counts wide nested 
rows. The test covered only `Struct`, not `List<Struct>` or `Map`.
   
   **Blocking and threading**
   - `create_reader` runs in `ParquetMorselizer::prepare_open_file`, which 
DataFusion documents as the CPU-only planning step. A JNI `acquireMemory` there 
can run other consumers' spills and park in `lock.wait()`.
   - Plans with no JVM sources are spawned onto the shared Tokio runtime, so 
grows in `reserve_buffers` can park a runtime worker too. This is most likely 
at the start of a task, which is exactly when a scan makes its first request. A 
task-kill interrupt is not delivered to a worker thread. #6261 since made a 
blocked acquire hand the worker's other tasks to another thread, but that 
thread still waits in Spark.
   
   **Release accounting**
   - Charging per buffer gives N shrinks (one JNI `release_memory` per column 
chunk, each taking the pool mutex). With the page index and a row selection, 
parquet fetches one range per selected page, so it can be one shrink per page. 
One `Arc<MemoryReservation>` shared by every buffer of a fetch gives one grow 
and one shrink, released when the last buffer drops.
   - Coalesced object-store fetches are undercharged. `get_ranges` coalesces 
nearby ranges, so each returned `Bytes` is a slice of one merged allocation and 
any surviving slice pins the gap bytes too. "Charged exactly" holds only for 
the local path. `scan_io_object_store_response_bytes_read` already measures the 
difference.
   - The design depends on an invariant that must be documented in the code: no 
page buffer may end up aliased by an Arrow array that crosses FFI. If it did, 
the final `shrink` would run from the FFI release callback when the JVM closes 
the batch. `CometFairMemoryPool::shrink` and `CometUnifiedMemoryPool::shrink` 
panic when the JNI release fails, and a panic in an `extern "C"` callback 
aborts the executor. Comet requests `Utf8`, not `Utf8View`, so this does not 
fire today. `ByteViewArrayDecoderPlain` can alias the page bytes, so a 
`Utf8View` hint in the file's `ARROW:schema` is safe only because the cast to 
the Spark type copies.
   
   **Scope**
   - `get_bytes` (bloom filters) and `get_metadata` (footer plus page index) 
stay unreserved, and `load_bloom_filters` materializes `BloomFilterStatistics` 
for every row group at once, which is not small on a wide table.
   - The change turned accounting on for every native Parquet scan with no 
switch. A config flag (default `true`) would give users a way out while the 
estimate is refined.
   - `scan_io_unreserved_bytes` was cumulative refused bytes, including the 
one-shot decode estimate, but rendered as a size in the SQL UI so it read as a 
live footprint (or a leak). Any metric of this kind should be named and 
documented as cumulative, or split.
   - `docs/source/contributor-guide/memory_management.md` lists "decompression 
buffers, Parquet metadata structures, `object_store` request buffers" as 
undeclared. It, and its cgroup diagram, must be updated by whichever change 
closes the gap.
   - Spark SQL validation was not done. A change to scan memory behaviour needs 
the `run-*` Spark SQL label or `dev/local-ci.sh spark sql_core-1`, because the 
pull request tier runs only the default profile.
   
   ### Suggested plan
   
   1. Register one `MemoryConsumer` per scan operator on the factory, with a 
`new_empty()` handle per reader.
   2. Share one `Arc<MemoryReservation>` per fetch for the data pages.
   3. Decide `grow` versus `try_grow` for the data pages. If `try_grow`, 
propagate non-`ResourcesExhausted` errors and back off after a refusal.
   4. Size the decode estimate from the file's metadata and charge it at the 
first fetch.
   5. Charge bloom-filter and metadata buffers, or document them as uncharged.
   6. Update `memory_management.md`, add a switch, and run Spark SQL validation.
   
   ### Additional context
   
   Closed attempt: #6125. Related: #6126 (the same gap in the native Iceberg 
scan, which cannot reuse the `AsyncFileReader` approach), #6127 and #6128 (pool 
`grow` behaviour).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to