----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55156/#review161398 -----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java (line 485) <https://reviews.apache.org/r/55156/#comment232656> Add a comment here about how this condition might happen - that this operator has already been removed from the tree. ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java (line 752) <https://reviews.apache.org/r/55156/#comment232634> Maybe explain why the semijoin optimization should be removed in this situation ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java (line 120) <https://reviews.apache.org/r/55156/#comment232696> Can you remove this call from runDynamicPartitionPruning() since it is being called here as well? ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java (line 561) <https://reviews.apache.org/r/55156/#comment232719> OptimizeTezProcContext.pruningOpsRemovedByPriorOpt seems to be used in DynamicPartitionPruning as a way to track which DPP operators have already been removed. Do we need to have something similar to this for the semijoin optimizations? ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java (line 569) <https://reviews.apache.org/r/55156/#comment232716> This might be more appropriate on the FilterOperator, as I believe there are some instances where the predicates may not be pushed to the TableScan, but still exist in the FilterOperator. Or possibly done on both - DynamicPartitionPruningOptimization seems to apply it to both. - Jason Dere On Jan. 11, 2017, 4:20 a.m., Deepak Jaiswal wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55156/ > ----------------------------------------------------------- > > (Updated Jan. 11, 2017, 4:20 a.m.) > > > Review request for hive, Gopal V, Gunther Hagleitner, Jason Dere, Prasanth_J, > and Rajesh Balamohan. > > > Bugs: HIVE-15269 > https://issues.apache.org/jira/browse/HIVE-15269 > > > Repository: hive-git > > > Description > ------- > > HIVE-15269 min-max runtime filtering. > The patch also contains the patch for HIVE-15270. > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16f6c1c > itests/src/test/resources/testconfiguration.properties be5a747 > orc/src/test/org/apache/orc/impl/TestRecordReaderImpl.java cdd62ac > ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java > 69ba4a2 > ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java 940f2dd > ql/src/java/org/apache/hadoop/hive/ql/exec/DynamicValueRegistry.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeColumnEvaluator.java > 24c8281 > > ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantDefaultEvaluator.java > 89a75eb > ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantEvaluator.java > 4fe72a0 > > ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeDynamicValueEvaluator.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluator.java b8d6ab7 > ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorFactory.java > 0d03d8f > ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorHead.java > 42685fb > ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorRef.java > 0a6b66a > ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeFieldEvaluator.java > ff32626 > > ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java > 221abd9 > ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java bd0d28c > ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 46f0ecd > ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java > ac5331e > ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java 9718c48 > ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCache.java 440e0a1 > ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheWrapper.java 9768efa > ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java 9049ddd > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ObjectCache.java 008f8a4 > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DynamicValueRegistryTez.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectCache.java 0141230 > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java > 955fa80 > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ObjectCache.java 06dca00 > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java > d80f201 > > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java > 0cb6c8a > > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorSMBMapJoinOperator.java > 80b0a14 > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java > f6b6447 > > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/DynamicValueVectorExpression.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java > 9d900e4 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java > beed6b8 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java > 26fcc45 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java > 9e9beb0 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/RedundantDynamicPruningConditionsRemoval.java > d9ce017 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java > aa1e509 > ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java e2363eb > ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 35f34da > ql/src/java/org/apache/hadoop/hive/ql/parse/RuntimeValuesInfo.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java e8b003e > ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java cdb9e1b > ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 13a0811 > ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicValue.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDynamicValueDesc.java > PRE-CREATION > > ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestConvertAstToSearchArg.java > 93b50a6 > ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestSearchArgumentImpl.java > 8cbc26d > > ql/src/test/org/apache/hadoop/hive/ql/optimizer/physical/TestVectorizer.java > 3295372 > ql/src/test/queries/clientpositive/dynamic_semijoin_reduction.q > PRE-CREATION > ql/src/test/results/clientpositive/case_sensitivity.q.out 1952c8a > ql/src/test/results/clientpositive/input_testxpath.q.out 422a936 > ql/src/test/results/clientpositive/llap/auto_join1.q.out 3624062 > ql/src/test/results/clientpositive/llap/auto_join21.q.out 91ea004 > ql/src/test/results/clientpositive/llap/auto_join29.q.out 94134c5 > ql/src/test/results/clientpositive/llap/auto_sortmerge_join_1.q.out 9a586fa > ql/src/test/results/clientpositive/llap/auto_sortmerge_join_10.q.out > 4a049c7 > ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out > 5cb3db5 > ql/src/test/results/clientpositive/llap/auto_sortmerge_join_12.q.out > 1bef238 > ql/src/test/results/clientpositive/llap/auto_sortmerge_join_13.q.out > a539e03 > ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 0b1d648 > ql/src/test/results/clientpositive/llap/auto_sortmerge_join_3.q.out d99d425 > ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 069f08d > ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 5e45f9c > ql/src/test/results/clientpositive/llap/auto_sortmerge_join_6.q.out 9e83db1 > ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out 3e4f408 > ql/src/test/results/clientpositive/llap/auto_sortmerge_join_8.q.out dae32a5 > ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 9549778 > ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 20702f9 > ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out f6500eb > ql/src/test/results/clientpositive/llap/bucketmapjoin1.q.out e9bb701 > ql/src/test/results/clientpositive/llap/bucketmapjoin2.q.out 84de3e3 > ql/src/test/results/clientpositive/llap/bucketmapjoin3.q.out 475a9e3 > ql/src/test/results/clientpositive/llap/bucketmapjoin4.q.out 4815a1c > ql/src/test/results/clientpositive/llap/bucketmapjoin7.q.out b515af6 > ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_2.q.out > 8519ff3 > ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_6.q.out > 4b4a95e > ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_7.q.out > 11de932 > ql/src/test/results/clientpositive/llap/column_access_stats.q.out 5a66b0d > ql/src/test/results/clientpositive/llap/constprog_semijoin.q.out 98d2328 > ql/src/test/results/clientpositive/llap/correlationoptimizer1.q.out 5702c6e > ql/src/test/results/clientpositive/llap/correlationoptimizer2.q.out dfb4804 > ql/src/test/results/clientpositive/llap/correlationoptimizer3.q.out 74b3d6c > ql/src/test/results/clientpositive/llap/correlationoptimizer4.q.out 67e636b > ql/src/test/results/clientpositive/llap/correlationoptimizer6.q.out cf66d25 > ql/src/test/results/clientpositive/llap/cross_join.q.out 8578dbf > ql/src/test/results/clientpositive/llap/cross_product_check_1.q.out be303c4 > ql/src/test/results/clientpositive/llap/cross_product_check_2.q.out 3b14b84 > ql/src/test/results/clientpositive/llap/deleteAnalyze.q.out ba14835 > ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out > d3acbcd > ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction.q.out > PRE-CREATION > ql/src/test/results/clientpositive/llap/explainuser_1.q.out bea58fb > ql/src/test/results/clientpositive/llap/explainuser_2.q.out 931f1a2 > ql/src/test/results/clientpositive/llap/explainuser_4.q.out 4084206 > ql/src/test/results/clientpositive/llap/filter_join_breaktask.q.out 284ffb9 > ql/src/test/results/clientpositive/llap/hybridgrace_hashjoin_2.q.out > de81828 > ql/src/test/results/clientpositive/llap/join1.q.out cb0a735 > ql/src/test/results/clientpositive/llap/join32_lessSize.q.out 46d6281 > ql/src/test/results/clientpositive/llap/join_nullsafe.q.out 7e3b67d > ql/src/test/results/clientpositive/llap/llapdecider.q.out c7339c9 > ql/src/test/results/clientpositive/llap/lvj_mapjoin.q.out 366eda0 > ql/src/test/results/clientpositive/llap/mapjoin_decimal.q.out 01f1f80 > ql/src/test/results/clientpositive/llap/mapjoin_mapjoin.q.out e129556 > ql/src/test/results/clientpositive/llap/mrr.q.out 6272d04 > ql/src/test/results/clientpositive/llap/multiMapJoin1.q.out eac269c > ql/src/test/results/clientpositive/llap/multiMapJoin2.q.out ae47be1 > ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out > bd81989 > ql/src/test/results/clientpositive/llap/ptf.q.out 542347d2 > ql/src/test/results/clientpositive/llap/ptf_streaming.q.out d410f5d > ql/src/test/results/clientpositive/llap/reduce_deduplicate_extended.q.out > 99a7119 > ql/src/test/results/clientpositive/llap/skewjoin.q.out 617045c > ql/src/test/results/clientpositive/llap/skewjoinopt15.q.out 2c255ed > ql/src/test/results/clientpositive/llap/smb_cache.q.out 932b211 > ql/src/test/results/clientpositive/llap/smb_mapjoin_14.q.out cafdf5d > ql/src/test/results/clientpositive/llap/smb_mapjoin_15.q.out daf2497 > ql/src/test/results/clientpositive/llap/smb_mapjoin_6.q.out bbf9b46 > ql/src/test/results/clientpositive/llap/stats11.q.out b4643b1 > ql/src/test/results/clientpositive/llap/subquery_exists.q.out 3d8251f > ql/src/test/results/clientpositive/llap/subquery_notin.q.out eb99650 > ql/src/test/results/clientpositive/llap/subquery_views.q.out cf963b2 > ql/src/test/results/clientpositive/llap/tez_bmj_schema_evolution.q.out > c83da8b > ql/src/test/results/clientpositive/llap/tez_dynpart_hashjoin_1.q.out > 25c6f15 > ql/src/test/results/clientpositive/llap/tez_dynpart_hashjoin_2.q.out > b44ab7d > ql/src/test/results/clientpositive/llap/tez_join_result_complex.q.out > fd18117 > ql/src/test/results/clientpositive/llap/tez_self_join.q.out c3a993d > ql/src/test/results/clientpositive/llap/tez_smb_1.q.out 94e519e > ql/src/test/results/clientpositive/llap/tez_smb_empty.q.out 8333037 > ql/src/test/results/clientpositive/llap/tez_vector_dynpart_hashjoin_2.q.out > 189841a > ql/src/test/results/clientpositive/llap/unionDistinct_1.q.out 624d886 > ql/src/test/results/clientpositive/llap/union_top_level.q.out 6ea1f7e > ql/src/test/results/clientpositive/llap/vector_binary_join_groupby.q.out > a510e38 > ql/src/test/results/clientpositive/llap/vector_complex_join.q.out d271f04 > ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out 22d1bbe > ql/src/test/results/clientpositive/llap/windowing_gby.q.out a31a4b2 > ql/src/test/results/clientpositive/perf/query12.q.out 3f25f56 > ql/src/test/results/clientpositive/perf/query13.q.out 2f84cce > ql/src/test/results/clientpositive/perf/query15.q.out 4f4dcc5 > ql/src/test/results/clientpositive/perf/query17.q.out f03aeaa > ql/src/test/results/clientpositive/perf/query18.q.out cf11954 > ql/src/test/results/clientpositive/perf/query19.q.out 0ddcd83 > ql/src/test/results/clientpositive/perf/query20.q.out c525703 > ql/src/test/results/clientpositive/perf/query21.q.out 0d3ea9d > ql/src/test/results/clientpositive/perf/query22.q.out 6ff8896 > ql/src/test/results/clientpositive/perf/query25.q.out 8e5119c > ql/src/test/results/clientpositive/perf/query26.q.out 6f7aaf7 > ql/src/test/results/clientpositive/perf/query27.q.out cbd7d29 > ql/src/test/results/clientpositive/perf/query29.q.out 587c027 > ql/src/test/results/clientpositive/perf/query3.q.out 2845a88 > ql/src/test/results/clientpositive/perf/query31.q.out 433d419 > ql/src/test/results/clientpositive/perf/query32.q.out c80a4cd > ql/src/test/results/clientpositive/perf/query34.q.out 51710bd > ql/src/test/results/clientpositive/perf/query36.q.out b356628 > ql/src/test/results/clientpositive/perf/query39.q.out bde20a6 > ql/src/test/results/clientpositive/perf/query40.q.out a98831d > ql/src/test/results/clientpositive/perf/query42.q.out 7100bb3 > ql/src/test/results/clientpositive/perf/query43.q.out 72712e8 > ql/src/test/results/clientpositive/perf/query46.q.out d0275c6 > ql/src/test/results/clientpositive/perf/query48.q.out 879cb92 > ql/src/test/results/clientpositive/perf/query50.q.out 47a00b0 > ql/src/test/results/clientpositive/perf/query51.q.out 55e08d9 > ql/src/test/results/clientpositive/perf/query52.q.out c61ed30 > ql/src/test/results/clientpositive/perf/query54.q.out 0b59b1b > ql/src/test/results/clientpositive/perf/query55.q.out fb3743d > ql/src/test/results/clientpositive/perf/query58.q.out f7e6a08 > ql/src/test/results/clientpositive/perf/query64.q.out 468c1bc > ql/src/test/results/clientpositive/perf/query65.q.out 060cddf > ql/src/test/results/clientpositive/perf/query66.q.out 5cf60f6 > ql/src/test/results/clientpositive/perf/query67.q.out 1f38027 > ql/src/test/results/clientpositive/perf/query68.q.out b84bdf6 > ql/src/test/results/clientpositive/perf/query7.q.out 65478ab > ql/src/test/results/clientpositive/perf/query70.q.out b4eaee6 > ql/src/test/results/clientpositive/perf/query71.q.out 9bb010c > ql/src/test/results/clientpositive/perf/query72.q.out dff4c3a > ql/src/test/results/clientpositive/perf/query73.q.out 9e1a39a > ql/src/test/results/clientpositive/perf/query75.q.out f78ce8a > ql/src/test/results/clientpositive/perf/query76.q.out c7dbb37 > ql/src/test/results/clientpositive/perf/query79.q.out 6ce9592 > ql/src/test/results/clientpositive/perf/query80.q.out 10ec36a > ql/src/test/results/clientpositive/perf/query82.q.out 6f784c4 > ql/src/test/results/clientpositive/perf/query84.q.out cf68729 > ql/src/test/results/clientpositive/perf/query85.q.out e7f7348 > ql/src/test/results/clientpositive/perf/query86.q.out 6377c43 > ql/src/test/results/clientpositive/perf/query87.q.out 75634d5 > ql/src/test/results/clientpositive/perf/query88.q.out 7b04e9e > ql/src/test/results/clientpositive/perf/query89.q.out 7bc8700 > ql/src/test/results/clientpositive/perf/query90.q.out 92eeba1 > ql/src/test/results/clientpositive/perf/query91.q.out 1538e8b > ql/src/test/results/clientpositive/perf/query92.q.out 28df749 > ql/src/test/results/clientpositive/perf/query93.q.out f28ba41 > ql/src/test/results/clientpositive/perf/query94.q.out 836b16b > ql/src/test/results/clientpositive/perf/query95.q.out 9b0d1b2 > ql/src/test/results/clientpositive/perf/query96.q.out 8ffcbcb > ql/src/test/results/clientpositive/perf/query97.q.out c885e99 > ql/src/test/results/clientpositive/perf/query98.q.out 35cc897 > ql/src/test/results/clientpositive/tez/explainanalyze_1.q.out 05a4f5f > ql/src/test/results/clientpositive/tez/explainanalyze_2.q.out c99acc9 > ql/src/test/results/clientpositive/tez/explainanalyze_3.q.out 8d76ad2 > ql/src/test/results/clientpositive/tez/explainanalyze_4.q.out 27c1bbe > ql/src/test/results/clientpositive/tez/explainanalyze_5.q.out b9d69fd > ql/src/test/results/clientpositive/tez/explainuser_3.q.out 5ba02f3 > ql/src/test/results/clientpositive/tez/hybridgrace_hashjoin_2.q.out 473daca > ql/src/test/results/clientpositive/udf_coalesce.q.out dc3d9df > storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/LiteralDelegate.java > PRE-CREATION > > storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentFactory.java > 8fda95c > > storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java > 10d8c51 > > Diff: https://reviews.apache.org/r/55156/diff/ > > > Testing > ------- > > > Thanks, > > Deepak Jaiswal > >
