pgaref commented on issue #952: HIVE-23006 ProbeDecode compiler support URL: https://github.com/apache/hive/pull/952#issuecomment-609981295 > @pgaref , thanks for changes. Patch looks good. About the name, I think it is fine (probe side for the filtering generated statically in case of expressions or dynamically for joins). > > I think main remaining issue is related to selection in case of multiple MJ operators. I was thinking that for the time being, we could make the policy pluggable via config and have two very simple ones: 1) keep the one with lowest `ndv_JOIN_probe_key_column / ndv_TS_target_column` ratio, or 2) keep those with ratio below a certain threshold specified through a config value. Alternatively, in this patch you could maybe go only with option 1 and tackle other in follow-up. > > You have the information about the stats in the operators themselves (follow `op.getStatistics()` calls to see how it is retrieved per column basis). Thus, I would make that decision in the TezCompiler (once we gather all the context information) rather than GenTezUtils. @jcamachor thanks for the review, I appreciate it! Please check the latest changes with: - More generic probeDecode name for HiveConf - TezCompiler with configurable logic to select a MJ when we have multiple per TS -- currently picking the one with the lowest MJ/TS key Ratio - SharedWork using a TS on multiple branches clears probeContext that would result in wrong results - probeContextDetails as part of the explained plan - ProbeDecode Test cases with and without table stats Thoughts? My last concern is some operators like **VectorPTFEvaluatorDecimalAvg** that do not allow the use of Selected (which is what probedecode is doing) and whether we should take that into account as part of the optimizations.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
