deniskuzZ opened a new pull request, #6867: URL: https://github.com/apache/hive/pull/6867
### What changes were proposed in this pull request? `TezCompiler.selectLowestRatioProbeDecodeMapJoin` picks, for each TableScan, the map join whose hash table the reader can use to skip rows: the candidate with the lowest `getProbeDecodeNDVRatio`, NDV(hash table key) / NDV(scanned column). Both lookups used column names of another namespace: - the big table key was looked up in the TableScan statistics by its post-Select name (`_col1`), which never matches a table column; the ratio fell back to the scan row count - the small table key was looked up in the map join output statistics; it belongs to the namespace of the small table ReduceSink input. With no output column of that name the ratio became join output rows / scan rows; with a colliding name it took the post-join NDV of an unrelated column This PR: - traces each candidate's big table key to its TableScan column first (`OperatorUtils.findTableOriginColExpr`, previously applied only to the winner) and skips a candidate without one or with a type conversion - computes the ratio from the NDV of the small table key in the statistics of the small table ReduceSink input over the NDV of the traced TableScan column - without column statistics, falls back to the small table row count (the hash table size) instead of the join output rows ### Why are the changes needed? The chosen map join and its ratio are wrong whenever the names miss or collide. In master's goldens: - `probedecode_mapjoin_stats.q.out` shows `keyRatio:0.16666666666666666`; the intended ratio is NDV(seller_dim.key4)=2 / NDV(orders_fact.key3)=6 = 0.333 - tpcds30tb `query37`/`query82`: keyRatio x scan rows = exactly 1.0 (`query21`: 4), the NDV of an unrelated post-join column - 61 of 507 `probeDecodeDetails` lines have keyRatio > 1.0 - an inserted GROUP BY that renames a column turns a miss into a false hit: TPC-DS q39's target flips from `inv_date_sk` (0.017) to `inv_item_sk` (7.6e-10 = 1/1,311,525,000) No reader applies probe decode yet (the context reaches only a log line in `OrcEncodedDataConsumer`), so results and performance are unchanged today; once a reader filters by it, this ratio decides which hash table filters the scan. ### Does this PR introduce _any_ user-facing change? EXPLAIN only: `probeDecodeDetails` (cacheKey, bigKeyColName, keyRatio) of the affected scans. ### How was this patch tested? - `probedecode_mapjoin_stats.q` fails against the current golden (keyRatio 0.1667 -> 0.3333); looking the small key up in the map join output statistics again fails the new golden - regenerated the 133 llap and 68 tpcds30tb goldens that print `probeDecodeDetails` (`TestMiniLlapLocalCliDriver`, `TestTezTPCDS30TBPerfCliDriver`): only `probeDecodeDetails` lines change - `murmur_hash_migration.q` and `hybridgrace_hashjoin_2.q` also differ on master without this change (results/runtime statistics); only their `probeDecodeDetails` lines are updated here 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
