> On Jan. 13, 2017, 12:08 a.m., Jason Dere wrote: > > ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java, line 582 > > <https://reviews.apache.org/r/55156/diff/6/?file=1602202#file1602202line582> > > > > This might be more appropriate on the FilterOperator, as I believe > > there are some instances where the predicates may not be pushed to the > > TableScan, but still exist in the FilterOperator. Or possibly done on both > > - DynamicPartitionPruningOptimization seems to apply it to both. > > Deepak Jaiswal wrote: > Agreed. I will update in the next patch
As of current patch, if the TS does not have the predicate pushed down, the semijoin optimization is not done. Otherwise, TS and the FilterOperator share the filter expression. The code in DynamicPartitionPruningOptimization is actually dead code as it never actually finds any predicate to be removed, not atleast in any of its tests. - Deepak ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55156/#review161398 ----------------------------------------------------------- On Jan. 18, 2017, 8:42 p.m., Deepak Jaiswal wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55156/ > ----------------------------------------------------------- > > (Updated Jan. 18, 2017, 8:42 p.m.) > > > Review request for hive, Gopal V, Gunther Hagleitner, Jason Dere, Prasanth_J, > and Rajesh Balamohan. > > > Bugs: HIVE-15269 > https://issues.apache.org/jira/browse/HIVE-15269 > > > Repository: hive-git > > > Description > ------- > > HIVE-15269 min-max runtime filtering. > The patch also contains the patch for HIVE-15270. > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 96928db > itests/src/test/resources/testconfiguration.properties be5a747 > orc/src/test/org/apache/orc/impl/TestRecordReaderImpl.java cdd62ac > ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java > 69ba4a2 > ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java 940f2dd > ql/src/java/org/apache/hadoop/hive/ql/exec/DynamicValueRegistry.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeColumnEvaluator.java > 24c8281 > > ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantDefaultEvaluator.java > 89a75eb > ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantEvaluator.java > 4fe72a0 > > ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeDynamicValueEvaluator.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluator.java b8d6ab7 > ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorFactory.java > 0d03d8f > ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorHead.java > 42685fb > ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorRef.java > 0a6b66a > ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeFieldEvaluator.java > ff32626 > > ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java > 221abd9 > ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java bd0d28c > ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 6f01da0 > ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java d98cea9 > ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java > ac5331e > ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java 9718c48 > ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCache.java 440e0a1 > ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheWrapper.java 9768efa > ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java 9049ddd > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ObjectCache.java 008f8a4 > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DynamicValueRegistryTez.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectCache.java 0141230 > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java > 955fa80 > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ObjectCache.java 06dca00 > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java > d80f201 > > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java > 0cb6c8a > > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorSMBMapJoinOperator.java > 80b0a14 > ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java > 9d900e4 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java > 26fcc45 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java > 9e9beb0 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/RedundantDynamicPruningConditionsRemoval.java > d9ce017 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java > aa1e509 > ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java e2363eb > ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 35f34da > ql/src/java/org/apache/hadoop/hive/ql/parse/RuntimeValuesInfo.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java e8b003e > ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java cdb9e1b > ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java 1ecbaad > ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 13a0811 > ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicValue.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDynamicValueDesc.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFBloomFilter.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEvaluator.java > 18d5285 > > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInBloomFilter.java > PRE-CREATION > > ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestConvertAstToSearchArg.java > 93b50a6 > ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestSearchArgumentImpl.java > 8cbc26d > ql/src/test/queries/clientpositive/dynamic_semijoin_reduction.q > PRE-CREATION > ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out > d3acbcd > ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction.q.out > PRE-CREATION > storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/LiteralDelegate.java > PRE-CREATION > > storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentFactory.java > 8fda95c > > storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java > 10d8c51 > storage-api/src/java/org/apache/hive/common/util/BloomFilter.java e60690d > > Diff: https://reviews.apache.org/r/55156/diff/ > > > Testing > ------- > > > Thanks, > > Deepak Jaiswal > >
