zhuqi-lucas opened a new pull request, #11532: URL: https://github.com/apache/arrow-rs/pull/11532
# Which issue does this PR close? - Related to apache/datafusion#25896 (surface parquet decompression time in `EXPLAIN ANALYZE`), which needs this collection side first. - Context: apache/arrow-rs#11154 / #11168 (dictionary decompression on the skip path, found only with a CPU profiler). # Rationale for this change Decompression happens inside `decode_page`, under both decoding and skipping, and nothing reports it. A scan that is slow because of its codec cannot be told apart from one that is slow because there is a lot to decode, and the cost of decompressing pages only to walk past them is invisible. On our data servers dictionary decompression on the skip path was 14-18% of CPU, and finding it took a profiler on a production pod. Measured with a local prototype (1M rows, 40k distinct strings, zstd(3)): | read | decompression share | notes | | --- | --- | --- | | full read | 63.9% | dictionary 1.15 ms, data pages 12.76 ms, of 21.78 ms wall | | selective read (1 row in 10,000) | 84.7% | 100 rows out, still 99 data pages / 45 MiB decompressed, all charged to `skip_records` | # What changes are included in this PR? - `parquet::file::metrics::DecompressionMetrics`: time, uncompressed bytes and page count, split by page kind (dictionary / data) and by phase (`Read`: pages decompressed to decode values; `Skip`: pages decompressed only to walk past values). Lives in `file`, not `arrow`, because `decode_page` is in the non-arrow `file` module. - `SerializedPageReader::with_decompression_metrics`; `decode_page` records into it. The column reader marks the phase. - `ArrowReaderMetrics` owns one handle and gives it to every page reader of a scan, on the sync reader and on the push decoder (and so the async reader). New accessors: `decompression()`, `dictionary_decompress_time()`, `data_decompress_time()`, `dictionary_decompressed_bytes()`, `data_decompressed_bytes()`. Decompression is nested inside decoding and skipping, so these numbers are a share of that time, not a stage to add next to it; the docs say so. # Are these changes tested? Yes. Unit tests for the counters, and end-to-end tests on a real zstd file: a full read records everything under `Read`; a selective read records walked pages under `Skip`; disabled metrics report `None`; the push decoder records the same bytes and page counts as the sync reader (this test fails if the push decoder is not wired). Overhead: collection is opt-in through `ArrowReaderMetrics`, off by default. When off, `decode_page` tests one enum discriminant and reads no clock. When on, with 1,000-row pages (1,000 decompress calls per read), 40 interleaved reads each: median 8.05 ms off vs 8.08 ms on (+0.4%, within noise). # Are there any user-facing changes? New public API listed above. No behaviour change when metrics are disabled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
