[
https://issues.apache.org/jira/browse/HIVE-30059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-30059:
--------------------------------
Attachment: HIVE-30059-design.pdf
> LLAP IO Cache: native parquet data cache
> ----------------------------------------
>
> Key: HIVE-30059
> URL: https://issues.apache.org/jira/browse/HIVE-30059
> Project: Hive
> Issue Type: Sub-task
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-30059-design.pdf
>
>
> Add LLAP data-cache integration for Parquet so column chunks and file footers
> are read from cache after the first pass, mirroring the ORC EncodedDataReader
> path. Today LLAP hosts a Parquet vectorized reader but every read goes to the
> filesystem — the LLAP data cache and file-metadata cache are unused for
> Parquet, which leaves the hottest table format on the platform paying full IO
> cost on every scan.
> What this changes
> - Route Parquet column-chunk reads through LLAP's DataCache off-heap buffers,
> keyed by file id + chunk offset/length, with the layout and buffer-borrowing
> logic aligned with the existing ORC encoded path.
> - Cache Parquet file footers via LLAP's FileMetadataCache (LlapCacheAwareFs
> already registers cache-aware paths for column-chunk fetches; the footer path
> is the second half of that story).
> - Introduce a ParquetEncodedDataReader on the LLAP daemon side that owns the
> cached read + decode loop, and factor the Hive-typed row-group column-reader
> construction out of VectorizedParquetRecordReader into a shared
> ParquetRowGroupDecoder so both the direct reader and the cache-backed
> consumer share the same decode path.
> - Add META_HIT / META_MISS counters to the LLAP IO summary for Parquet footer
> lookups (data-cache hit counters already come from the shared LLAP IO
> summary).
> For repeat-scan-heavy workloads (BI dashboards, TPC-DS style query mixes),
> the ORC-vs-Parquet cache-hit gap dominates second-run latency. Bringing
> Parquet up to parity on LLAP caching means the same warm-cache speedup ORC
> users already see applies to Parquet tables — measured on TPC-DS Q1 the Map-1
> scan's cache hit ratio jumps from cold-read levels to the same near-100% band
> ORC vertices already hit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
