[
https://issues.apache.org/jira/browse/HIVE-30059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-30059:
--------------------------------
Description:
Add LLAP data-cache integration for Parquet so column chunks and file footers
are read from cache after the first pass, mirroring the ORC EncodedDataReader
path. Today LLAP hosts a Parquet vectorized reader but every read goes to the
filesystem — the LLAP data cache and file-metadata cache are unused for
Parquet, which leaves the hottest table format on the platform paying full IO
cost on every scan.
What this changes
- Route Parquet column-chunk reads through LLAP's DataCache off-heap buffers,
keyed by file id + chunk offset/length, with the layout and buffer-borrowing
logic aligned with the existing ORC encoded path.
- Cache Parquet file footers via LLAP's FileMetadataCache (LlapCacheAwareFs
already registers cache-aware paths for column-chunk fetches; the footer path
is the second half of that story).
- Introduce a ParquetEncodedDataReader on the LLAP daemon side that owns the
cached read + decode loop, and factor the Hive-typed row-group column-reader
construction out of VectorizedParquetRecordReader into a shared
ParquetRowGroupDecoder so both the direct reader and the cache-backed consumer
share the same decode path.
- Add META_HIT / META_MISS counters to the LLAP IO summary for Parquet footer
lookups (data-cache hit counters already come from the shared LLAP IO summary).
For repeat-scan-heavy workloads (BI dashboards, TPC-DS style query mixes), the
ORC-vs-Parquet cache-hit gap dominates second-run latency. Bringing Parquet up
to parity on LLAP caching means the same warm-cache speedup ORC users already
see applies to Parquet tables — measured on TPC-DS Q1 the Map-1 scan's cache
hit ratio jumps from cold-read levels to the same near-100% band ORC vertices
already hit.
> LLAP IO Cache: native parquet data cache
> ----------------------------------------
>
> Key: HIVE-30059
> URL: https://issues.apache.org/jira/browse/HIVE-30059
> Project: Hive
> Issue Type: Sub-task
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
>
> Add LLAP data-cache integration for Parquet so column chunks and file footers
> are read from cache after the first pass, mirroring the ORC EncodedDataReader
> path. Today LLAP hosts a Parquet vectorized reader but every read goes to the
> filesystem — the LLAP data cache and file-metadata cache are unused for
> Parquet, which leaves the hottest table format on the platform paying full IO
> cost on every scan.
> What this changes
> - Route Parquet column-chunk reads through LLAP's DataCache off-heap buffers,
> keyed by file id + chunk offset/length, with the layout and buffer-borrowing
> logic aligned with the existing ORC encoded path.
> - Cache Parquet file footers via LLAP's FileMetadataCache (LlapCacheAwareFs
> already registers cache-aware paths for column-chunk fetches; the footer path
> is the second half of that story).
> - Introduce a ParquetEncodedDataReader on the LLAP daemon side that owns the
> cached read + decode loop, and factor the Hive-typed row-group column-reader
> construction out of VectorizedParquetRecordReader into a shared
> ParquetRowGroupDecoder so both the direct reader and the cache-backed
> consumer share the same decode path.
> - Add META_HIT / META_MISS counters to the LLAP IO summary for Parquet footer
> lookups (data-cache hit counters already come from the shared LLAP IO
> summary).
> For repeat-scan-heavy workloads (BI dashboards, TPC-DS style query mixes),
> the ORC-vs-Parquet cache-hit gap dominates second-run latency. Bringing
> Parquet up to parity on LLAP caching means the same warm-cache speedup ORC
> users already see applies to Parquet tables — measured on TPC-DS Q1 the Map-1
> scan's cache hit ratio jumps from cold-read levels to the same near-100% band
> ORC vertices already hit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
