----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65130/ -----------------------------------------------------------
(Updated Feb. 1, 2018, 7:24 p.m.) Review request for hive, Eugene Koifman, Gopal V, and Jason Dere. Changes ------- Added two new fields in Table metadata. 1. bucketingVersion : Default is 2. 1 for older tables which use JAVA hash. 2 for new tables which will use murmur hash. The plumbing is done. The code to actually change the hashing logic is yet to be done. 2. expertMode : Default false. If a user loads data into bucketed table which does not launch a Tez job(pending work), this is set to true. helps in debugging issues for wrong results in queries with bucketed tables. Load data on bucketed tables can only take names like 000000_0, 000001_0 etc for the file names. It will reject the load otherwise. Fixed CustomPartitionVertex code for SMB and Bucket map joins. The logic to iterate and assign bucket id is replaced by examining the file name and assigning bucket id. For SMB, small table must have less than or equal number of buckets as big table. Bugs: HIVE-18350 https://issues.apache.org/jira/browse/HIVE-18350 Repository: hive-git Description ------- Made changes for both bucketed and non-bucketed tables. Added a positive test for non-bucketed table which renames the loaded file. Added couple of negative tests for bucketed table which reject a load with inconsistent file name. Diffs (updated) ----- hcatalog/core/src/test/java/org/apache/hive/hcatalog/common/TestHCatUtil.java 91aa4fa269 itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java 9614114083 itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestReplChangeManager.java 6ade76d0c2 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java 26afe90faa ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomVertexConfiguration.java ef5e7edcd6 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 9885038588 ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 9b0ffe0e91 ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java dc698c8de8 ql/src/java/org/apache/hadoop/hive/ql/optimizer/metainfo/annotation/OpTraitsRulesProcFactory.java 69d9f3125a ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkMapJoinOptimizer.java bacc44482a ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 5868d4dd56 ql/src/java/org/apache/hadoop/hive/ql/plan/OpTraits.java 9621c3be53 ql/src/test/queries/clientpositive/auto_sortmerge_join_2.q e5fdcb57e4 ql/src/test/queries/clientpositive/auto_sortmerge_join_4.q abf09e5534 ql/src/test/queries/clientpositive/auto_sortmerge_join_5.q b85c4a7aa3 ql/src/test/queries/clientpositive/auto_sortmerge_join_7.q bd780861e3 ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out 5cfc35aa73 ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out 0d586fd26b ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out 45704d1253 ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out 1959075912 ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 054b0d00be ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 95d329862c ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out e711715aa5 ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out 53c685cb11 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_2.q.out 8cfa113794 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_4.q.out fce5e0cfc4 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_5.q.out 8250eca099 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_7.q.out eb813c1734 standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h df646a7d17 standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp 27f8c0f2fc standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java f317b0393f standalone-metastore/src/gen/thrift/gen-php/metastore/Types.php 6878ee1be7 standalone-metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py 25e9a889b2 standalone-metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb 3a11a0582a standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java b3d99a1da5 standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/client/builder/TableBuilder.java 69acf3cfff standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MTable.java 6c40ae8753 standalone-metastore/src/main/thrift/hive_metastore.thrift 93f3e53de2 standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStorePartitionSpecs.java 57e5a4126e standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java 372dee6369 standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestOldSchema.java 6a44833a67 standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java b9a8f61c69 Diff: https://reviews.apache.org/r/65130/diff/5/ Changes: https://reviews.apache.org/r/65130/diff/4-5/ Testing ------- Thanks, Deepak Jaiswal