----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27265/ -----------------------------------------------------------
Review request for hive. Repository: hive-git Description ------- This change re-uses the SMBJoinOperator for Spark. Background: the logical layer already converts joins to SMB Joins. This changes just introduces a class called "SparkSortMergeJoinFactory" on the Spark-compile path which attaches the data structures (like local work, bucket info) to the MapWork for the SMBJoinOperator to consume. It is largely-based on the MapReduce class "MapJoinFactory". However, in spark-path, it is activated only for SMBJoin and not map-joins, as we have another strategy for map-joins. That is why there's a new optimizer-rule called "TypeRule", so this processor is only run on SMBJoinOperators (which share same name with MapJoinOperators, which is needed for logical-optimizers dealing with hints). One major assumption around the whole SMB concept is that both tables have corresponding buckets. I found during testing of large numbers of buckets (like auto_sortmerge_join_16) that "insert" into a bucketed table wasn't putting the same keys in corresponding buckets. I activated MR-style shuffle (hash-shuffle instead of total-order shuffle), and that seemed to solve the issue. Diffs ----- itests/src/test/resources/testconfiguration.properties 00c9f4d ql/src/java/org/apache/hadoop/hive/ql/lib/TypeRule.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java ae1d1ab ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkSortMergeJoinFactory.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java ed88c60 ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 8e28887 ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java 4f5feca ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 1c663c4 ql/src/test/results/clientpositive/spark/auto_join0.q.out 76ff63d ql/src/test/results/clientpositive/spark/auto_join10.q.out 05a5912 ql/src/test/results/clientpositive/spark/auto_join11.q.out 998c28b ql/src/test/results/clientpositive/spark/auto_join12.q.out d2b7993 ql/src/test/results/clientpositive/spark/auto_join13.q.out 78aa01e ql/src/test/results/clientpositive/spark/auto_join15.q.out 5916070 ql/src/test/results/clientpositive/spark/auto_join16.q.out 0b6807d ql/src/test/results/clientpositive/spark/auto_join18.q.out 6083b38 ql/src/test/results/clientpositive/spark/auto_join18_multi_distinct.q.out 01c8f0a ql/src/test/results/clientpositive/spark/auto_join20.q.out a8f2b9a ql/src/test/results/clientpositive/spark/auto_join21.q.out f9ac35d ql/src/test/results/clientpositive/spark/auto_join22.q.out 516322c ql/src/test/results/clientpositive/spark/auto_join23.q.out ce5a670 ql/src/test/results/clientpositive/spark/auto_join24.q.out 15b8888 ql/src/test/results/clientpositive/spark/auto_join27.q.out 67f5739 ql/src/test/results/clientpositive/spark/auto_join28.q.out b979661 ql/src/test/results/clientpositive/spark/auto_join29.q.out 0951b8d ql/src/test/results/clientpositive/spark/auto_join30.q.out 98b3974 ql/src/test/results/clientpositive/spark/auto_join31.q.out df502c8 ql/src/test/results/clientpositive/spark/auto_join32.q.out 8d83188 ql/src/test/results/clientpositive/spark/auto_smb_mapjoin_14.q.out e64d4fb ql/src/test/results/clientpositive/spark/auto_sortmerge_join_1.q.out 9158d65 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_10.q.out f608cc5 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_11.q.out 3c26363 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_12.q.out 65e496f ql/src/test/results/clientpositive/spark/auto_sortmerge_join_13.q.out a5a281b ql/src/test/results/clientpositive/spark/auto_sortmerge_join_14.q.out 2fc3bb6 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_15.q.out 74cbd7c ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out PRE-CREATION ql/src/test/results/clientpositive/spark/auto_sortmerge_join_2.q.out d1bb7a0 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_3.q.out d57a1d7 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_4.q.out 8244c50 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_5.q.out 2ab1bca ql/src/test/results/clientpositive/spark/auto_sortmerge_join_6.q.out bc4a163 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_7.q.out 16ef3ae ql/src/test/results/clientpositive/spark/auto_sortmerge_join_8.q.out 9fd3e5a ql/src/test/results/clientpositive/spark/auto_sortmerge_join_9.q.out a7f994f ql/src/test/results/clientpositive/spark/bucket2.q.out b1b2997 ql/src/test/results/clientpositive/spark/bucket3.q.out 019c11a ql/src/test/results/clientpositive/spark/bucket4.q.out 2cbab11 ql/src/test/results/clientpositive/spark/bucket_map_join_1.q.out 4ec619e ql/src/test/results/clientpositive/spark/bucket_map_join_2.q.out 1c288c2 ql/src/test/results/clientpositive/spark/bucketmapjoin10.q.out 8be3edd ql/src/test/results/clientpositive/spark/bucketmapjoin11.q.out 9e45843 ql/src/test/results/clientpositive/spark/bucketmapjoin12.q.out 0c1ac4b ql/src/test/results/clientpositive/spark/bucketmapjoin13.q.out dc1b8cf ql/src/test/results/clientpositive/spark/bucketmapjoin8.q.out 6d72fdf ql/src/test/results/clientpositive/spark/bucketmapjoin9.q.out d80bdcf ql/src/test/results/clientpositive/spark/count.q.out c527c1d ql/src/test/results/clientpositive/spark/ctas.q.out 0ded266 ql/src/test/results/clientpositive/spark/disable_merge_for_bucketing.q.out 590b265 ql/src/test/results/clientpositive/spark/escape_clusterby1.q.out 52bdf6a ql/src/test/results/clientpositive/spark/escape_distributeby1.q.out 736db5e ql/src/test/results/clientpositive/spark/escape_orderby1.q.out 6e1c0cf ql/src/test/results/clientpositive/spark/escape_sortby1.q.out 58b663c ql/src/test/results/clientpositive/spark/groupby1.q.out 847f45c ql/src/test/results/clientpositive/spark/groupby10.q.out 2095843 ql/src/test/results/clientpositive/spark/groupby11.q.out 70db5a5 ql/src/test/results/clientpositive/spark/groupby2.q.out 86e2f2a ql/src/test/results/clientpositive/spark/groupby3.q.out 13a5fab ql/src/test/results/clientpositive/spark/groupby3_map.q.out dac2824 ql/src/test/results/clientpositive/spark/groupby3_map_multi_distinct.q.out d2c054a ql/src/test/results/clientpositive/spark/groupby3_map_skew.q.out ec6439a ql/src/test/results/clientpositive/spark/groupby3_noskew.q.out 0c9a7e1 ql/src/test/results/clientpositive/spark/groupby3_noskew_multi_distinct.q.out 42fbb8c ql/src/test/results/clientpositive/spark/groupby4.q.out 318c5a3 ql/src/test/results/clientpositive/spark/groupby7_map.q.out 22a05b5 ql/src/test/results/clientpositive/spark/groupby7_map_multi_single_reducer.q.out bc453c6 ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out 2a07f2a ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out 00a0707 ql/src/test/results/clientpositive/spark/groupby7_noskew_multi_single_reducer.q.out 36640ef ql/src/test/results/clientpositive/spark/groupby8.q.out d8295ce ql/src/test/results/clientpositive/spark/groupby8_map.q.out b9aa597 ql/src/test/results/clientpositive/spark/groupby8_map_skew.q.out b9aa597 ql/src/test/results/clientpositive/spark/groupby8_noskew.q.out b9aa597 ql/src/test/results/clientpositive/spark/groupby9.q.out bec2346 ql/src/test/results/clientpositive/spark/groupby_complex_types.q.out 16fadea ql/src/test/results/clientpositive/spark/groupby_complex_types_multi_single_reducer.q.out 7470843 ql/src/test/results/clientpositive/spark/groupby_cube1.q.out 169c4ac ql/src/test/results/clientpositive/spark/groupby_multi_insert_common_distinct.q.out d3457da ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer.q.out 3abd0e3 ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer2.q.out 7f74c62 ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer3.q.out c4b7419 ql/src/test/results/clientpositive/spark/groupby_position.q.out 9e58189 ql/src/test/results/clientpositive/spark/groupby_ppr.q.out 860aa58 ql/src/test/results/clientpositive/spark/groupby_rollup1.q.out 0aeff6b ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 61dd2be ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 99da734 ql/src/test/results/clientpositive/spark/having.q.out 5e9f20d ql/src/test/results/clientpositive/spark/input14.q.out e7d4db6 ql/src/test/results/clientpositive/spark/input17.q.out 0882a29 ql/src/test/results/clientpositive/spark/input18.q.out 802fb0a ql/src/test/results/clientpositive/spark/input1_limit.q.out 33ecd07 ql/src/test/results/clientpositive/spark/insert_into1.q.out e9be658 ql/src/test/results/clientpositive/spark/insert_into2.q.out 5c8e9c7 ql/src/test/results/clientpositive/spark/insert_into3.q.out 6c0111d ql/src/test/results/clientpositive/spark/join0.q.out 55b725e ql/src/test/results/clientpositive/spark/join15.q.out 1651db1 ql/src/test/results/clientpositive/spark/join18.q.out 7b64fb6 ql/src/test/results/clientpositive/spark/join18_multi_distinct.q.out 57c4516 ql/src/test/results/clientpositive/spark/join20.q.out f06ffac ql/src/test/results/clientpositive/spark/join21.q.out e81ec5a ql/src/test/results/clientpositive/spark/join23.q.out 3982ea7 ql/src/test/results/clientpositive/spark/join29.q.out d5383d5 ql/src/test/results/clientpositive/spark/join30.q.out 5c16622 ql/src/test/results/clientpositive/spark/join31.q.out 9193df9 ql/src/test/results/clientpositive/spark/join35.q.out 1750aec ql/src/test/results/clientpositive/spark/smb_mapjoin_17.q.out 482268c Diff: https://reviews.apache.org/r/27265/diff/ Testing ------- Ran existing auto_sortmerge_* tests. Thanks, Szehon Ho
