----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34576/ -----------------------------------------------------------
(Updated May 29, 2015, 6:15 p.m.) Review request for hive and John Pullokkaran. Repository: hive-git Description ------- Bucketized Table feature fails in some cases. if src & destination is bucketed on same key, and if actual data in the src is not bucketed (because data got loaded using LOAD DATA LOCAL INPATH ) then the data won't be bucketed while writing to destination. Example ---------------------------------------------------------------------- CREATE TABLE P1(key STRING, val STRING) CLUSTERED BY (key) SORTED BY (key) INTO 2 BUCKETS STORED AS TEXTFILE; LOAD DATA LOCAL INPATH '/Users/jp/apache-hive1/data/files/P1.txt' INTO TABLE P1; – perform an insert to make sure there are 2 files INSERT OVERWRITE TABLE P1 select key, val from P1; -------------------------------------------------- This is not a regression. This has never worked. This got only discovered due to Hadoop2 changes. In Hadoop1, in local mode, number of reducers will always be 1, regardless of what is requested by app. Hadoop2 now honors the number of reducer setting in local mode (by spawning threads). Long term solution seems to be to prevent load data for bucketed table. Diffs (updated) ----- ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java e53933e ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 1a9b42b ql/src/test/results/clientnegative/bucket_mapjoin_mismatch1.q.out 623c2e8 ql/src/test/results/clientnegative/bucket_mapjoin_wrong_table_metadata_1.q.out f4522d2 ql/src/test/results/clientnegative/bucket_mapjoin_wrong_table_metadata_2.q.out 9aa9b5d ql/src/test/results/clientnegative/exim_11_nonpart_noncompat_sorting.q.out 9220c8e ql/src/test/results/clientpositive/auto_join32.q.out f862870 ql/src/test/results/clientpositive/auto_join_filters.q.out a6720d9 ql/src/test/results/clientpositive/auto_sortmerge_join_1.q.out 5114038 ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out e6e7ef3 ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out b2e782f ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out 210f1ab ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out a307b13 ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out f4ceee7 ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out 3c2951a ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out e1f3888 ql/src/test/results/clientpositive/auto_sortmerge_join_8.q.out 38ecdbe ql/src/test/results/clientpositive/bucket_map_join_1.q.out 42e6a3f ql/src/test/results/clientpositive/bucket_map_join_2.q.out af73309 ql/src/test/results/clientpositive/bucket_map_join_spark1.q.out 870ecdd ql/src/test/results/clientpositive/bucket_map_join_spark2.q.out 33f5c46 ql/src/test/results/clientpositive/bucket_map_join_spark3.q.out 067d1ff ql/src/test/results/clientpositive/bucketcontext_1.q.out 77bfcf9 ql/src/test/results/clientpositive/bucketcontext_2.q.out a9db13d ql/src/test/results/clientpositive/bucketcontext_3.q.out 9ba3e0c ql/src/test/results/clientpositive/bucketcontext_4.q.out a2b37a8 ql/src/test/results/clientpositive/bucketcontext_5.q.out 3ee1f0e ql/src/test/results/clientpositive/bucketcontext_6.q.out d2304fa ql/src/test/results/clientpositive/bucketcontext_7.q.out 1a105ed ql/src/test/results/clientpositive/bucketcontext_8.q.out 138e415 ql/src/test/results/clientpositive/bucketizedhiveinputformat_auto.q.out 215efdd ql/src/test/results/clientpositive/bucketmapjoin1.q.out 471ff73 ql/src/test/results/clientpositive/bucketmapjoin10.q.out b0e849d ql/src/test/results/clientpositive/bucketmapjoin11.q.out 4263cab ql/src/test/results/clientpositive/bucketmapjoin12.q.out bcd7394 ql/src/test/results/clientpositive/bucketmapjoin2.q.out a8d9e9d ql/src/test/results/clientpositive/bucketmapjoin3.q.out c759f05 ql/src/test/results/clientpositive/bucketmapjoin4.q.out f61500c ql/src/test/results/clientpositive/bucketmapjoin5.q.out 0cb2825 ql/src/test/results/clientpositive/bucketmapjoin7.q.out 667a9db ql/src/test/results/clientpositive/bucketmapjoin8.q.out 252b377 ql/src/test/results/clientpositive/bucketmapjoin9.q.out 5e28dc3 ql/src/test/results/clientpositive/bucketmapjoin_negative.q.out 6ae127d ql/src/test/results/clientpositive/bucketmapjoin_negative2.q.out 4c9f54a ql/src/test/results/clientpositive/bucketmapjoin_negative3.q.out 9a0bfc4 ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 34cd1ff ql/src/test/results/clientpositive/groupby_sort_2.q.out b5e52f1 ql/src/test/results/clientpositive/groupby_sort_3.q.out c16911a ql/src/test/results/clientpositive/groupby_sort_4.q.out a6b1c3d ql/src/test/results/clientpositive/groupby_sort_5.q.out 369e2b5 ql/src/test/results/clientpositive/groupby_sort_7.q.out 7264695 ql/src/test/results/clientpositive/groupby_sort_8.q.out ec16eb0 ql/src/test/results/clientpositive/groupby_sort_9.q.out e49781a ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out 0d631ce ql/src/test/results/clientpositive/groupby_sort_test_1.q.out 8c1765d ql/src/test/results/clientpositive/insert_orig_table.q.out 5eea74d ql/src/test/results/clientpositive/insert_values_orig_table.q.out 684cd1b ql/src/test/results/clientpositive/join_filters.q.out 4f112bd ql/src/test/results/clientpositive/join_nulls.q.out 46e0170 ql/src/test/results/clientpositive/mergejoin.q.out cb96ab3 ql/src/test/results/clientpositive/skewjoin_mapjoin11.q.out 51445a5 ql/src/test/results/clientpositive/skewjoinopt19.q.out 91167db ql/src/test/results/clientpositive/skewjoinopt20.q.out 15e96fd ql/src/test/results/clientpositive/smb_mapjoin_1.q.out 9ab334b ql/src/test/results/clientpositive/smb_mapjoin_10.q.out ea2fa51 ql/src/test/results/clientpositive/smb_mapjoin_2.q.out 379dc0d ql/src/test/results/clientpositive/smb_mapjoin_25.q.out c0a8959 ql/src/test/results/clientpositive/smb_mapjoin_3.q.out 26fa5d4 ql/src/test/results/clientpositive/smb_mapjoin_4.q.out 9fc7f93 ql/src/test/results/clientpositive/smb_mapjoin_5.q.out 6e6882a ql/src/test/results/clientpositive/smb_mapjoin_7.q.out 82f5804 ql/src/test/results/clientpositive/spark/auto_join32.q.out e26e4a2 ql/src/test/results/clientpositive/spark/auto_join_filters.q.out 8934433 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_1.q.out 09d2692 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_12.q.out a70b161 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_2.q.out 2ea0a65 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_3.q.out 6281929 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_4.q.out 31e9d86 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_5.q.out 3eceb0b ql/src/test/results/clientpositive/spark/auto_sortmerge_join_7.q.out ddbca05 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_8.q.out 88d4dcb ql/src/test/results/clientpositive/spark/bucket_map_join_1.q.out 7570ebe ql/src/test/results/clientpositive/spark/bucket_map_join_2.q.out 80b44e9 ql/src/test/results/clientpositive/spark/bucket_map_join_spark1.q.out 6230bef ql/src/test/results/clientpositive/spark/bucket_map_join_spark2.q.out 1a33625 ql/src/test/results/clientpositive/spark/bucket_map_join_spark3.q.out fed923c ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out 65bded2 ql/src/test/results/clientpositive/spark/bucket_map_join_tez2.q.out 33e6d63 ql/src/test/results/clientpositive/spark/bucketmapjoin1.q.out d4a9c98 ql/src/test/results/clientpositive/spark/bucketmapjoin10.q.out 678ad54 ql/src/test/results/clientpositive/spark/bucketmapjoin11.q.out 95606f0 ql/src/test/results/clientpositive/spark/bucketmapjoin12.q.out d6c25e4 ql/src/test/results/clientpositive/spark/bucketmapjoin2.q.out d82480e ql/src/test/results/clientpositive/spark/bucketmapjoin3.q.out 39552c1 ql/src/test/results/clientpositive/spark/bucketmapjoin4.q.out ad2762d ql/src/test/results/clientpositive/spark/bucketmapjoin5.q.out f7c3d4d ql/src/test/results/clientpositive/spark/bucketmapjoin7.q.out 7bfe440 ql/src/test/results/clientpositive/spark/bucketmapjoin8.q.out 4601eb1 ql/src/test/results/clientpositive/spark/bucketmapjoin9.q.out 60bd103 ql/src/test/results/clientpositive/spark/bucketmapjoin_negative.q.out 031c46c ql/src/test/results/clientpositive/spark/bucketmapjoin_negative2.q.out 4a8f46d ql/src/test/results/clientpositive/spark/bucketmapjoin_negative3.q.out a09904e ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 65a8374 ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 9f30e15 ql/src/test/results/clientpositive/spark/skewjoinopt19.q.out f51d805 ql/src/test/results/clientpositive/spark/skewjoinopt20.q.out 338da34 ql/src/test/results/clientpositive/spark/smb_mapjoin_1.q.out 1ff1262 ql/src/test/results/clientpositive/spark/smb_mapjoin_10.q.out cadf08e ql/src/test/results/clientpositive/spark/smb_mapjoin_2.q.out a0d51f3 ql/src/test/results/clientpositive/spark/smb_mapjoin_25.q.out cb811ed ql/src/test/results/clientpositive/spark/smb_mapjoin_3.q.out f46b833 ql/src/test/results/clientpositive/spark/smb_mapjoin_4.q.out a421a42 ql/src/test/results/clientpositive/spark/smb_mapjoin_5.q.out af65010 ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out 622b950 ql/src/test/results/clientpositive/stats11.q.out e51f049 ql/src/test/results/clientpositive/tez/auto_join_filters.q.out 8fde41d ql/src/test/results/clientpositive/tez/auto_sortmerge_join_1.q.out a275d27 ql/src/test/results/clientpositive/tez/auto_sortmerge_join_11.q.out 6ac74ca ql/src/test/results/clientpositive/tez/auto_sortmerge_join_12.q.out e90af15 ql/src/test/results/clientpositive/tez/auto_sortmerge_join_2.q.out 2cb8416 ql/src/test/results/clientpositive/tez/auto_sortmerge_join_3.q.out abeceb8 ql/src/test/results/clientpositive/tez/auto_sortmerge_join_4.q.out 8eb9ce5 ql/src/test/results/clientpositive/tez/auto_sortmerge_join_5.q.out adcc1fa ql/src/test/results/clientpositive/tez/auto_sortmerge_join_7.q.out 2562cb0 ql/src/test/results/clientpositive/tez/auto_sortmerge_join_8.q.out 31b0a97 ql/src/test/results/clientpositive/tez/bucket_map_join_tez1.q.out 61c197f ql/src/test/results/clientpositive/tez/bucket_map_join_tez2.q.out 3f980b6 ql/src/test/results/clientpositive/tez/explainuser_2.q.out 0511819 ql/src/test/results/clientpositive/tez/insert_orig_table.q.out 5eea74d ql/src/test/results/clientpositive/tez/mergejoin.q.out c4be404 ql/src/test/results/clientpositive/tez/tez_fsstat.q.out 3fcf68c ql/src/test/results/clientpositive/tez/tez_smb_1.q.out d970bd9 ql/src/test/results/clientpositive/tez/tez_smb_main.q.out 52e1750 ql/src/test/results/clientpositive/udaf_percentile_approx_23.q.out 14a6874 Diff: https://reviews.apache.org/r/34576/diff/ Testing ------- Thanks, pengcheng xiong
