----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34576/ -----------------------------------------------------------
Review request for hive and John Pullokkaran. Repository: hive-git Description ------- Bucketized Table feature fails in some cases. if src & destination is bucketed on same key, and if actual data in the src is not bucketed (because data got loaded using LOAD DATA LOCAL INPATH ) then the data won't be bucketed while writing to destination. Example ---------------------------------------------------------------------- CREATE TABLE P1(key STRING, val STRING) CLUSTERED BY (key) SORTED BY (key) INTO 2 BUCKETS STORED AS TEXTFILE; LOAD DATA LOCAL INPATH '/Users/jp/apache-hive1/data/files/P1.txt' INTO TABLE P1; – perform an insert to make sure there are 2 files INSERT OVERWRITE TABLE P1 select key, val from P1; -------------------------------------------------- This is not a regression. This has never worked. This got only discovered due to Hadoop2 changes. In Hadoop1, in local mode, number of reducers will always be 1, regardless of what is requested by app. Hadoop2 now honors the number of reducer setting in local mode (by spawning threads). Long term solution seems to be to prevent load data for bucketed table. Diffs ----- ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java e53933e ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 1a9b42b ql/src/test/results/clientnegative/alter_partition_invalidspec.q.out 404115f ql/src/test/results/clientnegative/alter_partition_nodrop.q.out 1c78cff ql/src/test/results/clientnegative/alter_partition_nodrop_table.q.out 3c425da ql/src/test/results/clientnegative/alter_partition_offline.q.out c70fcb4 ql/src/test/results/clientnegative/archive_corrupt.q.out 56e8ec4 ql/src/test/results/clientnegative/bucket_mapjoin_mismatch1.q.out 623c2e8 ql/src/test/results/clientnegative/bucket_mapjoin_wrong_table_metadata_2.q.out 9aa9b5d ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.java1.7.out 4ea70e3 ql/src/test/results/clientnegative/columnstats_partlvl_multiple_part_clause.q.out ce79830 ql/src/test/results/clientnegative/dynamic_partitions_with_whitelist.q.out f069ae8 ql/src/test/results/clientnegative/exim_02_all_part_over_overlap.q.out 3c05600 ql/src/test/results/clientnegative/exim_15_part_nonpart.q.out dfbf025 ql/src/test/results/clientnegative/exim_16_part_noncompat_schema.q.out 4cb6ca7 ql/src/test/results/clientnegative/exim_17_part_spec_underspec.q.out 23caa4a ql/src/test/results/clientnegative/exim_18_part_spec_missing.q.out 23caa4a ql/src/test/results/clientnegative/exim_21_part_managed_external.q.out fd27f29 ql/src/test/results/clientnegative/exim_24_import_part_authfail.q.out 1a9a34d ql/src/test/results/clientnegative/insertover_dynapart_ifnotexists.q.out a40ffab ql/src/test/results/clientnegative/load_exist_part_authfail.q.out 491cfd0 ql/src/test/results/clientnegative/load_part_authfail.q.out 4ea8be9 ql/src/test/results/clientnegative/load_part_nospec.q.out bebaf92 ql/src/test/results/clientnegative/nopart_load.q.out 8815146 ql/src/test/results/clientnegative/protectmode_part2.q.out 16d58c7 ql/src/test/results/clientpositive/alter_concatenate_indexed_table.q.out ffcbcf9 ql/src/test/results/clientpositive/alter_merge.q.out 17d86b8 ql/src/test/results/clientpositive/alter_merge_2.q.out e118c39 ql/src/test/results/clientpositive/alter_merge_stats.q.out fdd2ddc ql/src/test/results/clientpositive/alter_partition_protect_mode.q.out 80990d9 ql/src/test/results/clientpositive/alter_rename_table.q.out 732d8a2 ql/src/test/results/clientpositive/alter_table_cascade.q.out 0139466 ql/src/test/results/clientpositive/auto_join32.q.out bfc8be8 ql/src/test/results/clientpositive/auto_sortmerge_join_1.q.out 383defd ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out e6e7ef3 ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out e9fb705 ql/src/test/results/clientpositive/auto_sortmerge_join_16.q.out d4ecb19 ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out c089419 ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out 6e443fa ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out feaea04 ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out e89f548 ql/src/test/results/clientpositive/auto_sortmerge_join_8.q.out 44c037f ql/src/test/results/clientpositive/bucket_map_join_spark1.q.out 870ecdd ql/src/test/results/clientpositive/bucket_map_join_spark2.q.out 33f5c46 ql/src/test/results/clientpositive/bucket_map_join_spark3.q.out 067d1ff ql/src/test/results/clientpositive/bucketcontext_1.q.out 77bfcf9 ql/src/test/results/clientpositive/bucketcontext_2.q.out a9db13d ql/src/test/results/clientpositive/bucketcontext_3.q.out 9ba3e0c ql/src/test/results/clientpositive/bucketcontext_4.q.out a2b37a8 ql/src/test/results/clientpositive/bucketcontext_6.q.out d2304fa ql/src/test/results/clientpositive/bucketcontext_7.q.out 1a105ed ql/src/test/results/clientpositive/bucketcontext_8.q.out 138e415 ql/src/test/results/clientpositive/bucketizedhiveinputformat_auto.q.out 215efdd ql/src/test/results/clientpositive/bucketmapjoin1.q.out 72f2a07 ql/src/test/results/clientpositive/bucketmapjoin10.q.out b0e849d ql/src/test/results/clientpositive/bucketmapjoin11.q.out 4263cab ql/src/test/results/clientpositive/bucketmapjoin12.q.out bcd7394 ql/src/test/results/clientpositive/bucketmapjoin2.q.out a8d9e9d ql/src/test/results/clientpositive/bucketmapjoin3.q.out c759f05 ql/src/test/results/clientpositive/bucketmapjoin4.q.out f61500c ql/src/test/results/clientpositive/bucketmapjoin5.q.out 0cb2825 ql/src/test/results/clientpositive/bucketmapjoin7.q.out 667a9db ql/src/test/results/clientpositive/bucketmapjoin8.q.out 252b377 ql/src/test/results/clientpositive/bucketmapjoin9.q.out 5e28dc3 ql/src/test/results/clientpositive/bucketmapjoin_negative.q.out 6ae127d ql/src/test/results/clientpositive/bucketmapjoin_negative2.q.out 4c9f54a ql/src/test/results/clientpositive/columnstats_partlvl.q.out 3c22d40 ql/src/test/results/clientpositive/columnstats_partlvl_dp.q.out 18a6909 ql/src/test/results/clientpositive/database.q.out 043d91b ql/src/test/results/clientpositive/database_drop.q.out 225104f ql/src/test/results/clientpositive/drop_partition_with_stats.q.out e27e557 ql/src/test/results/clientpositive/exim_02_part.q.out 6e0988a ql/src/test/results/clientpositive/exim_04_all_part.q.out 862efa3 ql/src/test/results/clientpositive/exim_05_some_part.q.out 1b6a515 ql/src/test/results/clientpositive/exim_06_one_part.q.out 39c83c3 ql/src/test/results/clientpositive/exim_07_all_part_over_nonoverlap.q.out b55a0bd ql/src/test/results/clientpositive/exim_08_nonpart_rename.q.out 740833b ql/src/test/results/clientpositive/exim_09_part_spec_nonoverlap.q.out d71f36f ql/src/test/results/clientpositive/exim_15_external_part.q.out d24f18a ql/src/test/results/clientpositive/exim_16_part_external.q.out af748c9 ql/src/test/results/clientpositive/exim_17_part_managed.q.out a92f95a ql/src/test/results/clientpositive/exim_18_part_external.q.out a082a11 ql/src/test/results/clientpositive/exim_19_00_part_external_location.q.out 5a97e03 ql/src/test/results/clientpositive/exim_19_part_external_location.q.out f9a20f7 ql/src/test/results/clientpositive/exim_20_part_managed_location.q.out b196ba5 ql/src/test/results/clientpositive/exim_23_import_part_authsuccess.q.out 5f78a76 ql/src/test/results/clientpositive/exim_hidden_files.q.out e449e0e ql/src/test/results/clientpositive/global_limit.q.out 7da20d5 ql/src/test/results/clientpositive/groupby_sort_6.q.out c5cb8b9 ql/src/test/results/clientpositive/groupby_sort_7.q.out 7264695 ql/src/test/results/clientpositive/groupby_sort_8.q.out ec16eb0 ql/src/test/results/clientpositive/groupby_sort_9.q.out e49781a ql/src/test/results/clientpositive/infer_bucket_sort_dyn_part.q.out 773a2a8 ql/src/test/results/clientpositive/input40.q.out bb0eabe ql/src/test/results/clientpositive/inputddl6.q.out 5a040e6 ql/src/test/results/clientpositive/inputddl7.q.out 0d64baf ql/src/test/results/clientpositive/insert1_overwrite_partitions.q.out 900babe ql/src/test/results/clientpositive/insert2_overwrite_partitions.q.out 25c438f ql/src/test/results/clientpositive/leftsemijoin.q.out 11f0bb0 ql/src/test/results/clientpositive/load_exist_part_authsuccess.q.out 8ec7e62 ql/src/test/results/clientpositive/load_part_authsuccess.q.out 8249dce ql/src/test/results/clientpositive/loadpart2.q.out 201a957 ql/src/test/results/clientpositive/merge_dynamic_partition.q.out da19b32 ql/src/test/results/clientpositive/merge_dynamic_partition2.q.out 5a2afb0 ql/src/test/results/clientpositive/merge_dynamic_partition3.q.out 86978f3 ql/src/test/results/clientpositive/merge_dynamic_partition4.q.out 86af660 ql/src/test/results/clientpositive/merge_dynamic_partition5.q.out c1468c1 ql/src/test/results/clientpositive/mergejoin.q.out cb96ab3 ql/src/test/results/clientpositive/nullgroup3.q.out 7712d4d ql/src/test/results/clientpositive/nullgroup5.q.out 8a94d62 ql/src/test/results/clientpositive/orc_analyze.q.out a61a2e6 ql/src/test/results/clientpositive/orc_split_elimination.q.out 7134ff5 ql/src/test/results/clientpositive/parquet_serde.q.out e753180 ql/src/test/results/clientpositive/partition_type_check.q.out e25d527 ql/src/test/results/clientpositive/partition_wise_fileformat17.q.out 028a26e ql/src/test/results/clientpositive/partition_wise_fileformat18.q.out 6303d44 ql/src/test/results/clientpositive/repl_1_drop.q.out 9fb65d1 ql/src/test/results/clientpositive/repl_2_exim_basic.q.out 8df0653 ql/src/test/results/clientpositive/repl_3_exim_metadata.q.out 8387c02 ql/src/test/results/clientpositive/smb_mapjoin_10.q.out ea2fa51 ql/src/test/results/clientpositive/spark/auto_join32.q.out 361a968 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_1.q.out 09d2692 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_12.q.out 8102ec1 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out d4ecb19 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_2.q.out 2ea0a65 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_3.q.out 6281929 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_4.q.out 31e9d86 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_7.q.out ddbca05 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_8.q.out 88d4dcb ql/src/test/results/clientpositive/spark/bucket_map_join_spark1.q.out 6230bef ql/src/test/results/clientpositive/spark/bucket_map_join_spark2.q.out 1a33625 ql/src/test/results/clientpositive/spark/bucket_map_join_spark3.q.out fed923c ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out 65bded2 ql/src/test/results/clientpositive/spark/bucket_map_join_tez2.q.out 33e6d63 ql/src/test/results/clientpositive/spark/bucketmapjoin1.q.out 44f4d0c ql/src/test/results/clientpositive/spark/bucketmapjoin10.q.out 678ad54 ql/src/test/results/clientpositive/spark/bucketmapjoin11.q.out 95606f0 ql/src/test/results/clientpositive/spark/bucketmapjoin12.q.out d6c25e4 ql/src/test/results/clientpositive/spark/bucketmapjoin2.q.out d82480e ql/src/test/results/clientpositive/spark/bucketmapjoin3.q.out 39552c1 ql/src/test/results/clientpositive/spark/bucketmapjoin4.q.out ad2762d ql/src/test/results/clientpositive/spark/bucketmapjoin5.q.out f7c3d4d ql/src/test/results/clientpositive/spark/bucketmapjoin7.q.out 7bfe440 ql/src/test/results/clientpositive/spark/bucketmapjoin8.q.out 4601eb1 ql/src/test/results/clientpositive/spark/bucketmapjoin9.q.out 60bd103 ql/src/test/results/clientpositive/spark/bucketmapjoin_negative.q.out 031c46c ql/src/test/results/clientpositive/spark/bucketmapjoin_negative2.q.out 4a8f46d ql/src/test/results/clientpositive/spark/leftsemijoin.q.out 11f0bb0 ql/src/test/results/clientpositive/spark/smb_mapjoin_10.q.out cadf08e ql/src/test/results/clientpositive/spark/stats18.q.out a061846 ql/src/test/results/clientpositive/spark/stats_counter_partitioned.q.out 4b84eca ql/src/test/results/clientpositive/spark/statsfs.q.out b0bca41 ql/src/test/results/clientpositive/stats11.q.out e51f049 ql/src/test/results/clientpositive/stats18.q.out a061846 ql/src/test/results/clientpositive/stats_counter_partitioned.q.out ab1270c ql/src/test/results/clientpositive/statsfs.q.out b0bca41 ql/src/test/results/clientpositive/tez/auto_sortmerge_join_1.q.out a275d27 ql/src/test/results/clientpositive/tez/auto_sortmerge_join_11.q.out 6ac74ca ql/src/test/results/clientpositive/tez/auto_sortmerge_join_12.q.out 8c8a3bf ql/src/test/results/clientpositive/tez/auto_sortmerge_join_16.q.out d4ecb19 ql/src/test/results/clientpositive/tez/auto_sortmerge_join_2.q.out 2cb8416 ql/src/test/results/clientpositive/tez/auto_sortmerge_join_3.q.out abeceb8 ql/src/test/results/clientpositive/tez/auto_sortmerge_join_4.q.out 8eb9ce5 ql/src/test/results/clientpositive/tez/auto_sortmerge_join_7.q.out 2562cb0 ql/src/test/results/clientpositive/tez/auto_sortmerge_join_8.q.out 31b0a97 ql/src/test/results/clientpositive/tez/bucket_map_join_tez1.q.out 61c197f ql/src/test/results/clientpositive/tez/bucket_map_join_tez2.q.out 3f980b6 ql/src/test/results/clientpositive/tez/dynamic_partition_pruning_2.q.out 8b0b81d ql/src/test/results/clientpositive/tez/explainuser_1.q.out b684858 ql/src/test/results/clientpositive/tez/explainuser_2.q.out f84524b ql/src/test/results/clientpositive/tez/leftsemijoin.q.out 11f0bb0 ql/src/test/results/clientpositive/tez/mergejoin.q.out 97df12a ql/src/test/results/clientpositive/tez/orc_analyze.q.out a61a2e6 ql/src/test/results/clientpositive/tez/stats_counter_partitioned.q.out ab1270c ql/src/test/results/clientpositive/tez/tez_fsstat.q.out 3fcf68c ql/src/test/results/clientpositive/tez/tez_smb_1.q.out d970bd9 ql/src/test/results/clientpositive/tez/tez_smb_main.q.out 6183390 ql/src/test/results/clientpositive/truncate_table.q.out 4d8f38c ql/src/test/results/clientpositive/view_cast.q.out 34444ae Diff: https://reviews.apache.org/r/34576/diff/ Testing ------- Thanks, pengcheng xiong
