[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13640579#comment-13640579 ] Hudson commented on HIVE-3891: -- Integrated in Hive-trunk-hadoop2 #173 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/173/]) HIVE-3891 : physical optimizer changes for auto sort-merge join (Namit Jain via Ashutosh Chauhan) (Revision 1471224) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1471224 Files : * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/conf/hive-default.xml.template * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UnionOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/AbstractJoinTaskDispatcher.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinResolver.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SortMergeJoinResolver.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SortMergeJoinTaskDispatcher.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverCommonJoin.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java * /hive/trunk/ql/src/test/queries/clientnegative/auto_sortmerge_join_1.q * /hive/trunk/ql/src/test/queries/clientpositive/auto_sortmerge_join_1.q * /hive/trunk/ql/src/test/queries/clientpositive/auto_sortmerge_join_10.q * /hive/trunk/ql/src/test/queries/clientpositive/auto_sortmerge_join_13.q * /hive/trunk/ql/src/test/queries/clientpositive/auto_sortmerge_join_14.q * /hive/trunk/ql/src/test/queries/clientpositive/auto_sortmerge_join_2.q * /hive/trunk/ql/src/test/queries/clientpositive/auto_sortmerge_join_3.q * /hive/trunk/ql/src/test/queries/clientpositive/auto_sortmerge_join_4.q * /hive/trunk/ql/src/test/queries/clientpositive/auto_sortmerge_join_5.q * /hive/trunk/ql/src/test/queries/clientpositive/auto_sortmerge_join_6.q * /hive/trunk/ql/src/test/queries/clientpositive/auto_sortmerge_join_7.q * /hive/trunk/ql/src/test/queries/clientpositive/auto_sortmerge_join_8.q * /hive/trunk/ql/src/test/queries/clientpositive/auto_sortmerge_join_9.q * /hive/trunk/ql/src/test/queries/clientpositive/bucketsortoptimize_insert_1.q * /hive/trunk/ql/src/test/queries/clientpositive/bucketsortoptimize_insert_2.q * /hive/trunk/ql/src/test/queries/clientpositive/bucketsortoptimize_insert_3.q * /hive/trunk/ql/src/test/queries/clientpositive/bucketsortoptimize_insert_4.q * /hive/trunk/ql/src/test/queries/clientpositive/bucketsortoptimize_insert_5.q * /hive/trunk/ql/src/test/queries/clientpositive/bucketsortoptimize_insert_6.q * /hive/trunk/ql/src/test/queries/clientpositive/bucketsortoptimize_insert_7.q * /hive/trunk/ql/src/test/queries/clientpositive/bucketsortoptimize_insert_8.q * /hive/trunk/ql/src/test/results/clientnegative/auto_sortmerge_join_1.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_1.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_13.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_14.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_8.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out * /hive/trunk/ql/src/test/results/clientpositive/bucketsortoptimize_insert_2.q.out * /hive/trunk/ql/src/test/results/clientpositive/bucketsortoptimize_insert_4.q.out * /hive/trunk/ql/src/test/results/clientpositive/bucketsortoptimize_insert_5.q.out * /hive/trunk/ql/src/test/results/clientpositive/bucketsortoptimize_insert_6.q.out physical optimizer changes for auto sort-merge join ---
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638973#comment-13638973 ] Namit Jain commented on HIVE-3891: -- comments physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.10.patch, hive.3891.11.patch, hive.3891.12.patch, hive.3891.13.patch, hive.3891.14.patch, hive.3891.15.patch, hive.3891.16.patch, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch, HIVE-3891_8.patch, hive.3891.9.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13640104#comment-13640104 ] Hudson commented on HIVE-3891: -- Integrated in Hive-trunk-h0.21 #2076 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2076/]) HIVE-3891 : physical optimizer changes for auto sort-merge join (Namit Jain via Ashutosh Chauhan) (Revision 1471224) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1471224 Files : * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/conf/hive-default.xml.template * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UnionOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/AbstractJoinTaskDispatcher.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinResolver.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SortMergeJoinResolver.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SortMergeJoinTaskDispatcher.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverCommonJoin.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java * /hive/trunk/ql/src/test/queries/clientnegative/auto_sortmerge_join_1.q * /hive/trunk/ql/src/test/queries/clientpositive/auto_sortmerge_join_1.q * /hive/trunk/ql/src/test/queries/clientpositive/auto_sortmerge_join_10.q * /hive/trunk/ql/src/test/queries/clientpositive/auto_sortmerge_join_13.q * /hive/trunk/ql/src/test/queries/clientpositive/auto_sortmerge_join_14.q * /hive/trunk/ql/src/test/queries/clientpositive/auto_sortmerge_join_2.q * /hive/trunk/ql/src/test/queries/clientpositive/auto_sortmerge_join_3.q * /hive/trunk/ql/src/test/queries/clientpositive/auto_sortmerge_join_4.q * /hive/trunk/ql/src/test/queries/clientpositive/auto_sortmerge_join_5.q * /hive/trunk/ql/src/test/queries/clientpositive/auto_sortmerge_join_6.q * /hive/trunk/ql/src/test/queries/clientpositive/auto_sortmerge_join_7.q * /hive/trunk/ql/src/test/queries/clientpositive/auto_sortmerge_join_8.q * /hive/trunk/ql/src/test/queries/clientpositive/auto_sortmerge_join_9.q * /hive/trunk/ql/src/test/queries/clientpositive/bucketsortoptimize_insert_1.q * /hive/trunk/ql/src/test/queries/clientpositive/bucketsortoptimize_insert_2.q * /hive/trunk/ql/src/test/queries/clientpositive/bucketsortoptimize_insert_3.q * /hive/trunk/ql/src/test/queries/clientpositive/bucketsortoptimize_insert_4.q * /hive/trunk/ql/src/test/queries/clientpositive/bucketsortoptimize_insert_5.q * /hive/trunk/ql/src/test/queries/clientpositive/bucketsortoptimize_insert_6.q * /hive/trunk/ql/src/test/queries/clientpositive/bucketsortoptimize_insert_7.q * /hive/trunk/ql/src/test/queries/clientpositive/bucketsortoptimize_insert_8.q * /hive/trunk/ql/src/test/results/clientnegative/auto_sortmerge_join_1.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_1.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_13.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_14.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_8.q.out * /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out * /hive/trunk/ql/src/test/results/clientpositive/bucketsortoptimize_insert_2.q.out * /hive/trunk/ql/src/test/results/clientpositive/bucketsortoptimize_insert_4.q.out * /hive/trunk/ql/src/test/results/clientpositive/bucketsortoptimize_insert_5.q.out * /hive/trunk/ql/src/test/results/clientpositive/bucketsortoptimize_insert_6.q.out physical optimizer changes for auto sort-merge join ---
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637908#comment-13637908 ] Namit Jain commented on HIVE-3891: -- Refreshed, addressed comments, ran tests. Can you take a look again ? cc [~ashutoshc] physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.10.patch, hive.3891.11.patch, hive.3891.12.patch, hive.3891.13.patch, hive.3891.14.patch, hive.3891.15.patch, hive.3891.16.patch, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch, HIVE-3891_8.patch, hive.3891.9.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635005#comment-13635005 ] Namit Jain commented on HIVE-3891: -- [~ashutoshc], all the tests passed. Since this was accepted sometime back, can you take a look again ? physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.10.patch, hive.3891.11.patch, hive.3891.12.patch, hive.3891.13.patch, hive.3891.14.patch, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch, HIVE-3891_8.patch, hive.3891.9.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635660#comment-13635660 ] Ashutosh Chauhan commented on HIVE-3891: Left some comments on Phabricator. physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.10.patch, hive.3891.11.patch, hive.3891.12.patch, hive.3891.13.patch, hive.3891.14.patch, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch, HIVE-3891_8.patch, hive.3891.9.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13632730#comment-13632730 ] Namit Jain commented on HIVE-3891: -- refreshing again and resolving conflicts physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.10.patch, hive.3891.11.patch, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch, HIVE-3891_8.patch, hive.3891.9.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13628782#comment-13628782 ] Namit Jain commented on HIVE-3891: -- [~ashutoshc], will do physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch, HIVE-3891_8.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13628814#comment-13628814 ] Namit Jain commented on HIVE-3891: -- I am refreshing https://reviews.facebook.net/D8439 with your latest patch, will keep that in sync for latest reviews physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch, HIVE-3891_8.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13628836#comment-13628836 ] Namit Jain commented on HIVE-3891: -- [~ashutoshc], it might be a good idea to get https://issues.apache.org/jira/browse/HIVE-4241 and https://issues.apache.org/jira/browse/HIVE-4328 first. This should be turned off, if HIVE-4241 optimization has triggered. physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch, HIVE-3891_8.patch, hive.3891.9.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13628844#comment-13628844 ] Namit Jain commented on HIVE-3891: -- Otherwise, the refreshed patch looks good. Can you take a look at the above patches (HIVE-4241 and HIVE-4328) first ? As soon as they are in, we can refresh and finish HIVE-3891. physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch, HIVE-3891_8.patch, hive.3891.9.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13607676#comment-13607676 ] Ashutosh Chauhan commented on HIVE-3891: +1 running tests. physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13608334#comment-13608334 ] Ashutosh Chauhan commented on HIVE-3891: Test auto_sortmerge_join_9.q failed, likely because of HIVE-948 where you need to update .q.out file. Also, patch doesn't apply cleanly anymore because of commit of HIVE-4146. I also have few comments which I left on phabricator. physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13593003#comment-13593003 ] Vikram Dixit K commented on HIVE-3891: -- +1 (non-binding) for this patch. Raised HIVE-4106 for the issue discussed here. physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13589373#comment-13589373 ] Namit Jain commented on HIVE-3891: -- Yes, that is true even before my patch physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13589374#comment-13589374 ] Namit Jain commented on HIVE-3891: -- I verified the last query in the attached file in trunk, and got the same results physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13590221#comment-13590221 ] Vikram Dixit K commented on HIVE-3891: -- What I mean is right aliases can be only 1 (position 0; in case of join on different keys) so if the bigTablePosition comes out as 2, the code looks to find something in position 1 causing a failure. physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13590224#comment-13590224 ] Namit Jain commented on HIVE-3891: -- The review request is there : https://reviews.facebook.net/D8439 I will get back to you on the comment physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13590231#comment-13590231 ] Namit Jain commented on HIVE-3891: -- I dont see any change in AbstractSMBJoinProc in this patch physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13590232#comment-13590232 ] Vikram Dixit K commented on HIVE-3891: -- Hey Namit, I thought that this was the patch that changed something in AbstractSMBJoinProcessor. I think this was done as part of HIVE-3403. Just caught this while doing some testing with this patch. Thanks Vikram. physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13590233#comment-13590233 ] Namit Jain commented on HIVE-3891: -- I didn't understand your case. A join B on A.key = B.key join C on A.key = C.key or A join B on A.key = B.key join C on A.value = C.value physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13590234#comment-13590234 ] Namit Jain commented on HIVE-3891: -- Both of them should work. physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[
https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13590250#comment-13590250
]
Vikram Dixit K commented on HIVE-3891:
--
Hi Namit,
The test case I have is very similar to the star join case. I have a 4 way join
with 3 dimension tables and 1 fact table. All the joins are on different keys.
By default, the average partition size is used as the tie breaker for the
choice of big table. However when I run this query, I get the array out of
bounds exception as it chose the table from the 2nd join (T3 from below) as the
big table (index = 2). However, the joinCtx right alias has only one table. So
this resulted in a crash.
{noformat}
T1 \
J1.out \
T2 / J2.out \
T3 / J3.out (Final)
T4 /
{noformat}
Thanks
Vikram.
physical optimizer changes for auto sort-merge join
---
Key: HIVE-3891
URL: https://issues.apache.org/jira/browse/HIVE-3891
Project: Hive
Issue Type: Bug
Reporter: Namit Jain
Assignee: Namit Jain
Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out,
hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch,
hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[
https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13590262#comment-13590262
]
Vikram Dixit K commented on HIVE-3891:
--
I think all this needs is SizeBasedBigTableSelectorForAutoSMJ to have the
following additional snippet:
{noformat}
if (parentOp instanceof TableScanOperator) {
topOps.add(parentOp);
belse if (parentOp instanceof CommonJoinOperator) {
return;
}
/b
else {
...
{noformat}
We shouldn't be taking that operation all the way to the top of the operator
tree which is what is happening. Let me know what you think?
Thanks
Vikram.
physical optimizer changes for auto sort-merge join
---
Key: HIVE-3891
URL: https://issues.apache.org/jira/browse/HIVE-3891
Project: Hive
Issue Type: Bug
Reporter: Namit Jain
Assignee: Namit Jain
Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out,
hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch,
hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[
https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13590294#comment-13590294
]
Namit Jain commented on HIVE-3891:
--
canConvertJoinToBucketMapJoin:
The join should be rejected by:
// This has already been inspected and rejected
if (context.getRejectedJoinOps().contains(joinOp)) {
return false;
}
Can you open a separate jira for this ?
We should never run into this scenario.
physical optimizer changes for auto sort-merge join
---
Key: HIVE-3891
URL: https://issues.apache.org/jira/browse/HIVE-3891
Project: Hive
Issue Type: Bug
Reporter: Namit Jain
Assignee: Namit Jain
Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out,
hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch,
hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13590295#comment-13590295 ] Namit Jain commented on HIVE-3891: -- This is a existing bug, and we need to debug this, but not as part of this patch. physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13589253#comment-13589253 ] Namit Jain commented on HIVE-3891: -- [~vikram.dixit], I am confused. Look at line 965 of auto_sortmerge_join_1.q.out, it is a SMB. Going in more detail: (line 486-492) Stage-5 is a root stage , consists of Stage-6, Stage-7, Stage-1 Stage-6 has a backup stage: Stage-1 Stage-3 depends on stages: Stage-6 Stage-7 has a backup stage: Stage-1 Stage-4 depends on stages: Stage-7 Stage-1 Stage-0 is a root stage Stage6 and 7 are mapjoin jobs, whereas Stage1 is a SMB join. This is the purpose of this jira. If a mapjoin can be performed, that gets priority over SMB join. physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13587961#comment-13587961 ] Vikram Dixit K commented on HIVE-3891: -- Hi Namit, In case of the test auto_sortmerge_join_1.q, I see the output does not seem to be an SMB join on the partitioned table. I am seeing similar behavior when I run on a partitioned table as well. Is there any reason why we are not converting the join to an SMB map-join in the case of partitioned table? Thanks Vikram. physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13581223#comment-13581223 ] Namit Jain commented on HIVE-3891: -- The tests passed physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573343#comment-13573343 ] Namit Jain commented on HIVE-3891: -- https://reviews.facebook.net/D8439 physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.3891.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573240#comment-13573240 ] Namit Jain commented on HIVE-3891: -- After HIVE-3403, an auto sortmerge join can be performed. However, consider the following scenario: T1 - big table: 3 partitions : big partitions T2 - small table: 1 partition both T1 and T2 are sorted/bucketed on the same key. If a join is being performed between T1 and T2, with HIVE-3403, T2 would be picked up as the table on which the mapper would run. Ideally, if T2 is so small that it can fit in memory, a better approach would be map-join. The physical optimizer should change the sort-merge join task into a backup task, with all tables being candidates for map-join (similar to auto convert mapjoin). physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
