deniskuzZ commented on PR #6758:
URL: https://github.com/apache/hive/pull/6758#issuecomment-5660609217
@abstractdog, before you invest more in the Parquet runtime: while working
on the 10 TB Parquet campaign I looked closely at how ProbeDecode behaves
today, and it turns out the feature has never worked end to end — for ORC either
What ProbeDecode is meant to do
At scan time, skip decoding rows whose join key is not in the map-join hash
table. The planner marks the map join; the reader (per format) does the
filtering.
1. The reader side has never filtered a row — ORC included
The only existing runtime is ORC's. Instrumented it on TPC-DS with a live
probe context: 76 reader instances, 232,018 rows decoded, 0 rows filtered.
The cause is a lookup bug: LlapRecordReader.getProbeColIdx() extracts the
column index by matching _col([0-9]+) against the key column name. But that
name is the user's column (ss_item_sk), not _colN, so the match fails and it
returns -1 on every query. With -1, the reader never filters.
So although ORC has shipped this runtime, no ORC query has ever exercised
the filtering path. A Parquet runtime built on the same ProbeDecodeContext
would inherit the same -1.
2. The planner side picks the wrong join
TezCompiler chooses which map join to probe by a ratio that mixes distinct
counts with row counts. Fixed it on a branch; the fix changes the
probeDecodeDetails choice on 81 golden files, so the current selection is
frequently wrong.
3. Measured end to end, it costs rather than gains
10 TB, hive.optimize.scan.probedecode on vs off: 11.45 s vs 11.05 s — 3.7%
slower with it on. That is what you'd expect: the scan carries the probe
plumbing and filters nothing.
Suggestion
Hold #6758 until the whole path is shown to work: pass the key column index
from the plan instead of regexing a name, add a filtered-row counter to the
reader, and show it nonzero on a real query. Otherwise a second runtime only
adds overhead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
