Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68946/ --- Review request for hive and Ashutosh Chauhan. Bugs: HIVE-20707 https://issues.apache.org/jira/browse/HIVE-20707 Repository: hive-git Description --- HIVE-20707: Automatic MSCK REPAIR for external tables Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d0adc35544cb8ae9d007a1d2ccb9b9565eedca88 data/conf/hive-site.xml 0daf9adc717bc1c4413d2e34691c26a3e2585c77 itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java cffa21af33d5abb2162fa16b6b990a469075f03d ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java e91346228e8724b8253364114145a348a7cbee26 ql/src/java/org/apache/hadoop/hive/ql/metadata/CheckResult.java 0b4240f5665f0b544b2fc5864fc098eb286a281e ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 598bb2ee8b72f1b7f75be7802b4eaae0204c988d ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java ce2b186b4dceda780106776daa022f18388ec76f ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java 7e768dacb0b00a0f1a9e64efbe778f9c2daaa31b ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java a2a0583d4dbdfe9aece1a14ecac24e0e6189cafa ql/src/test/queries/clientpositive/auto_msck_repair_0.q PRE-CREATION ql/src/test/queries/clientpositive/auto_msck_repair_1.q PRE-CREATION ql/src/test/queries/clientpositive/auto_msck_repair_2.q PRE-CREATION ql/src/test/queries/clientpositive/auto_msck_repair_3.q PRE-CREATION ql/src/test/queries/clientpositive/auto_msck_repair_4.q PRE-CREATION ql/src/test/queries/clientpositive/auto_msck_repair_batchsize.q PRE-CREATION ql/src/test/queries/clientpositive/msck_repair_0.q aeb4820af5b6687f7ae4163a94bdd2be25a8b0cd ql/src/test/queries/clientpositive/msck_repair_2.q be745b2d607d8c727b862c71f153f09d5622a8b5 ql/src/test/queries/clientpositive/msck_repair_3.q 140a6904ddc98b165d71a8b24314c56888ccbb9c ql/src/test/queries/clientpositive/msck_repair_batchsize.q 5a7afcca5b86c1887308626c0dc4d99916811bea ql/src/test/queries/clientpositive/msck_repair_drop.q 9923fb50cbdbdf9e8e07276ccaec073c490770e6 ql/src/test/results/clientpositive/auto_msck_repair_0.q.out PRE-CREATION ql/src/test/results/clientpositive/auto_msck_repair_1.q.out PRE-CREATION ql/src/test/results/clientpositive/auto_msck_repair_2.q.out PRE-CREATION ql/src/test/results/clientpositive/auto_msck_repair_3.q.out PRE-CREATION ql/src/test/results/clientpositive/auto_msck_repair_4.q.out PRE-CREATION ql/src/test/results/clientpositive/auto_msck_repair_batchsize.q.out PRE-CREATION ql/src/test/results/clientpositive/msck_repair_0.q.out fa6e4a988273a71b0f9dab64a48ddda6320d5f2f ql/src/test/results/clientpositive/msck_repair_2.q.out 7fbd934e118e81b9c5f028191c7ea6582a34db75 ql/src/test/results/clientpositive/msck_repair_3.q.out 0e153fbe69ba39819fac4629ef1bf5f90c17f37f ql/src/test/results/clientpositive/msck_repair_batchsize.q.out ab4b83137dcf1ce36846ce74e0a546528e81358b ql/src/test/results/clientpositive/msck_repair_drop.q.out 971c1381276fa626bd91d34488a65e3bfb2781ae standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java 294dfb728e12efaa13d239ea7b8949587a50fe1f standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/api/MetastoreException.java PRE-CREATION standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java 7b01678a10f4f0667844fec64ae76695d835bd6e standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java 16f4a50d69f9120d565f61d028b060d7776689fc standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/AutoMsckTask.java PRE-CREATION standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java PRE-CREATION standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckInfo.java PRE-CREATION standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckPartitionExpressionProxy.java PRE-CREATION standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionIterable.java PRE-CREATION standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java f3b38665676391fec9b85eb9a405c14632340dc6 standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/RetryUtilities.java PRE-CREATION Diff: https://reviews.apache.org/r/68946/diff/1/ Testing --- Thanks, Prasanth_J
[jira] [Created] (HIVE-20707) Automatic MSCK REPAIR for external tables
Prasanth Jayachandran created HIVE-20707: Summary: Automatic MSCK REPAIR for external tables Key: HIVE-20707 URL: https://issues.apache.org/jira/browse/HIVE-20707 Project: Hive Issue Type: New Feature Affects Versions: 4.0.0 Reporter: Prasanth Jayachandran Assignee: Prasanth Jayachandran In current scenario, to add partitions for external tables to metastore, MSCK REPAIR command has to be executed manually. To avoid this manual step, external tables can be specified a table property based on which a background metastore thread can add/drop/sync partitions periodically. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 68868: HIVE-17043: Remove non unique columns from group by keys if not referenced later
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68868/ --- (Updated Oct. 7, 2018, 10:01 p.m.) Review request for hive and Jesús Camacho Rodríguez. Changes --- Rebased the patch Bugs: HIVE-17043 https://issues.apache.org/jira/browse/HIVE-17043 Repository: hive-git Description --- This patch implements/test the following optimizations * Removal of group by on primary keys * Reduction of group by keys on primary keys * is NOT NULL filter removal if NOT NULL constraint is defined Diffs (updated) - itests/src/test/resources/testconfiguration.properties 97609cfadd ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveDefaultRelMetadataProvider.java 635d27e723 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 42e60de6a8 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateJoinTransposeRule.java f43ef01293 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelFieldTrimmer.java 5857f730a8 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/EstimateUniqueKeys.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdColumnUniqueness.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java 1ca1937ed9 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdUniqueKeys.java 3bf62c535c ql/src/test/queries/clientpositive/constraints_optimization.q PRE-CREATION ql/src/test/results/clientpositive/llap/constraints_optimization.q.out PRE-CREATION ql/src/test/results/clientpositive/llap/materialized_view_rewrite_1.q.out 735a4db2ee Diff: https://reviews.apache.org/r/68868/diff/7/ Changes: https://reviews.apache.org/r/68868/diff/6-7/ Testing --- Thanks, Vineet Garg
