subject:"Review Request 68946\: HIVE\-20707\: Automatic MSCK REPAIR for external tables"

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

2018-10-29 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68946/
---

(Updated Oct. 29, 2018, 9:17 a.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Changes
---

Uploaded wrong patch before. Moved partition management task to remote 
metastore only task. This fixed druid test failures.


Bugs: HIVE-20707
https://issues.apache.org/jira/browse/HIVE-20707


Repository: hive-git


Description
---

HIVE-20707: Automatic partition management


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e226a1f 
  hbase-handler/src/test/results/positive/external_table_ppd.q.out edcbe7e 
  hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out 
1209c88 
  hbase-handler/src/test/results/positive/hbase_ddl.q.out ccd4148 
  hbase-handler/src/test/results/positive/hbase_queries.q.out eeb97f0 
  hbase-handler/src/test/results/positive/hbasestats.q.out 5a4aea9 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 a9d7468 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 807f159 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 4cc5fa8 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/CheckResult.java 0b4240f 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
598bb2e 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java cff32d3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java bba7d6c 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 27f677e 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
 ce2b186 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java 
9480d38 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java 
a2a0583 
  ql/src/test/queries/clientpositive/msck_repair_acid.q PRE-CREATION 
  ql/src/test/queries/clientpositive/partition_discovery.q PRE-CREATION 
  ql/src/test/results/clientpositive/create_like.q.out f4a5ed5 
  ql/src/test/results/clientpositive/create_like_view.q.out 870f280 
  ql/src/test/results/clientpositive/default_file_format.q.out 0adf5ae 
  ql/src/test/results/clientpositive/druid/druidkafkamini_basic.q.out 883994c 
  ql/src/test/results/clientpositive/druid/druidmini_expressions.q.out 9c9af44 
  ql/src/test/results/clientpositive/druid_topn.q.out 179902a 
  ql/src/test/results/clientpositive/explain_locks.q.out ed7f1e8 
  ql/src/test/results/clientpositive/llap/external_table_purge.q.out 24c778e 
  ql/src/test/results/clientpositive/llap/mm_exim.q.out ee6cf06 
  ql/src/test/results/clientpositive/llap/strict_managed_tables2.q.out f3b6152 
  ql/src/test/results/clientpositive/llap/table_nonprintable.q.out 8221b8c 
  ql/src/test/results/clientpositive/llap/whroot_external1.q.out cac158c 
  ql/src/test/results/clientpositive/msck_repair_acid.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/msck_repair_drop.q.out 2456734 
  ql/src/test/results/clientpositive/partition_discovery.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/rename_external_partition_location.q.out 
02cd814 
  ql/src/test/results/clientpositive/repl_2_exim_basic.q.out b2bcd51 
  ql/src/test/results/clientpositive/show_create_table_alter.q.out 2c75c36 
  ql/src/test/results/clientpositive/show_create_table_partitioned.q.out 
e554a18 
  ql/src/test/results/clientpositive/show_create_table_serde.q.out 8b95c9b 
  ql/src/test/results/clientpositive/spark/stats_noscan_2.q.out 2d713a8 
  ql/src/test/results/clientpositive/stats_noscan_2.q.out 182820f 
  ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out 
2a442b4 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java
 294dfb7 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/api/MetastoreException.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 1d64cce 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 8fb1fa7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckInfo.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckPartitionExpressionProxy.java
 PRE-CREATION

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

2018-10-17 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68946/
---

(Updated Oct. 17, 2018, 7:14 p.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Changes
---

Added sorting to stabilize qfile test output


Bugs: HIVE-20707
https://issues.apache.org/jira/browse/HIVE-20707


Repository: hive-git


Description
---

HIVE-20707: Automatic partition management


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a458ac0 
  hbase-handler/src/test/results/positive/external_table_ppd.q.out edcbe7e 
  hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out 
1209c88 
  hbase-handler/src/test/results/positive/hbase_ddl.q.out ccd4148 
  hbase-handler/src/test/results/positive/hbase_queries.q.out eeb97f0 
  hbase-handler/src/test/results/positive/hbasestats.q.out 5a4aea9 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 a9d7468 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 807f159 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 46bf088 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/CheckResult.java 0b4240f 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
598bb2e 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java cff32d3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java bba7d6c 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 27f677e 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
 ce2b186 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java 
9480d38 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java 
a2a0583 
  ql/src/test/queries/clientpositive/msck_repair_acid.q PRE-CREATION 
  ql/src/test/queries/clientpositive/partition_discovery.q PRE-CREATION 
  ql/src/test/results/clientpositive/create_like.q.out f4a5ed5 
  ql/src/test/results/clientpositive/create_like_view.q.out 870f280 
  ql/src/test/results/clientpositive/default_file_format.q.out 0adf5ae 
  ql/src/test/results/clientpositive/druid/druidkafkamini_basic.q.out 883994c 
  ql/src/test/results/clientpositive/druid/druidmini_expressions.q.out 9c9af44 
  ql/src/test/results/clientpositive/druid_topn.q.out 179902a 
  ql/src/test/results/clientpositive/explain_locks.q.out ed7f1e8 
  ql/src/test/results/clientpositive/llap/external_table_purge.q.out 24c778e 
  ql/src/test/results/clientpositive/llap/mm_exim.q.out ee6cf06 
  ql/src/test/results/clientpositive/llap/strict_managed_tables2.q.out f3b6152 
  ql/src/test/results/clientpositive/llap/whroot_external1.q.out cac158c 
  ql/src/test/results/clientpositive/msck_repair_acid.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/msck_repair_drop.q.out 2456734 
  ql/src/test/results/clientpositive/partition_discovery.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/rename_external_partition_location.q.out 
02cd814 
  ql/src/test/results/clientpositive/repl_2_exim_basic.q.out b2bcd51 
  ql/src/test/results/clientpositive/show_create_table_alter.q.out 2c75c36 
  ql/src/test/results/clientpositive/show_create_table_partitioned.q.out 
e554a18 
  ql/src/test/results/clientpositive/show_create_table_serde.q.out 8b95c9b 
  ql/src/test/results/clientpositive/spark/stats_noscan_2.q.out 2d713a8 
  ql/src/test/results/clientpositive/stats_noscan_2.q.out 182820f 
  ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out 
2a442b4 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java
 294dfb7 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/api/MetastoreException.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 7b01678 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 8fb1fa7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckInfo.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckPartitionExpressionProxy.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 b98b4b4

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

2018-10-16 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68946/
---

(Updated Oct. 16, 2018, 7:02 a.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Changes
---

Addressed review comment and updated druid test golden files.


Bugs: HIVE-20707
https://issues.apache.org/jira/browse/HIVE-20707


Repository: hive-git


Description
---

HIVE-20707: Automatic partition management


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 92a1c31 
  hbase-handler/src/test/results/positive/external_table_ppd.q.out edcbe7e 
  hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out 
1209c88 
  hbase-handler/src/test/results/positive/hbase_ddl.q.out ccd4148 
  hbase-handler/src/test/results/positive/hbase_queries.q.out eeb97f0 
  hbase-handler/src/test/results/positive/hbasestats.q.out 5a4aea9 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 a9d7468 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 807f159 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 46bf088 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/CheckResult.java 0b4240f 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
598bb2e 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java cff32d3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 29f6ecf 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 27f677e 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
 ce2b186 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java 
9480d38 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java 
a2a0583 
  ql/src/test/queries/clientpositive/msck_repair_acid.q PRE-CREATION 
  ql/src/test/queries/clientpositive/partition_discovery.q PRE-CREATION 
  ql/src/test/results/clientpositive/create_like.q.out f4a5ed5 
  ql/src/test/results/clientpositive/create_like_view.q.out 870f280 
  ql/src/test/results/clientpositive/default_file_format.q.out 0adf5ae 
  ql/src/test/results/clientpositive/druid/druidkafkamini_basic.q.out 883994c 
  ql/src/test/results/clientpositive/druid/druidmini_expressions.q.out 9c9af44 
  ql/src/test/results/clientpositive/druid_topn.q.out 179902a 
  ql/src/test/results/clientpositive/explain_locks.q.out ed7f1e8 
  ql/src/test/results/clientpositive/llap/external_table_purge.q.out 24c778e 
  ql/src/test/results/clientpositive/llap/mm_exim.q.out ee6cf06 
  ql/src/test/results/clientpositive/llap/strict_managed_tables2.q.out f3b6152 
  ql/src/test/results/clientpositive/llap/whroot_external1.q.out cac158c 
  ql/src/test/results/clientpositive/msck_repair_acid.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/msck_repair_drop.q.out 2456734 
  ql/src/test/results/clientpositive/partition_discovery.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/rename_external_partition_location.q.out 
02cd814 
  ql/src/test/results/clientpositive/repl_2_exim_basic.q.out b2bcd51 
  ql/src/test/results/clientpositive/show_create_table_alter.q.out 2c75c36 
  ql/src/test/results/clientpositive/show_create_table_partitioned.q.out 
e554a18 
  ql/src/test/results/clientpositive/show_create_table_serde.q.out 8b95c9b 
  ql/src/test/results/clientpositive/spark/stats_noscan_2.q.out 2d713a8 
  ql/src/test/results/clientpositive/stats_noscan_2.q.out 182820f 
  ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out 
2a442b4 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java
 294dfb7 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/api/MetastoreException.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 7b01678 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 16f4a50 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckInfo.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckPartitionExpressionProxy.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 66977d7

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

2018-10-16 Thread j . prasanth . j



> On Oct. 16, 2018, 2:28 a.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
> > Lines 4761 (patched)
> > 
> >
> > Should this be on by default? If there are a lot of external tables 
> > (especially on s3), the metastore could be spending a lot of time doing 
> > auto discover. Could also affect the running of other MetastoreTaskThreads.
> 
> Prasanth_J wrote:
> Yeah. I think this should be default. This will remove manual msck step 
> or periodic msck query (via cron job). This thread kicks in once every 5 
> minutes but if the previous attempt is not done yet it will skip an attempt 
> so as to avoid queue'ing up of background tasks. Also it will use high batch 
> size by default so that in most case there should be 1 MS request per table. 
> MSCK thread also runs in a thread pool. The only place this background thread 
> could be blocked is when exclusive lock is obtained on a table (which gets 
> released after txn timeout of 300s). 
> We could probably restrict this only for EXTERNAL table types (currently 
> it defaults to both EXTERNAL and MANAGED). Since managed is ACID by default 
> we can avoid scanning managed tables. Changing ACID table layout out of band 
> is shooting themselves in the foot anyway.

Actually, thinking about it again. I think I will leave the table types 
(metastore.partition.management.table.types) as such. Required for partition 
retention. If user adds retention period to managed table, they don't have to 
do anything. 

Added a fix in new patch that won't acquire lock unless it is required (change 
detected, adding or dropping partition).


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68946/#review209582
---


On Oct. 16, 2018, 12:21 a.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68946/
> ---
> 
> (Updated Oct. 16, 2018, 12:21 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jason Dere.
> 
> 
> Bugs: HIVE-20707
> https://issues.apache.org/jira/browse/HIVE-20707
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20707: Automatic partition management
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 92a1c31 
>   hbase-handler/src/test/results/positive/external_table_ppd.q.out edcbe7e 
>   hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out 
> 1209c88 
>   hbase-handler/src/test/results/positive/hbase_ddl.q.out ccd4148 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out eeb97f0 
>   hbase-handler/src/test/results/positive/hbasestats.q.out 5a4aea9 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  a9d7468 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 807f159 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 46bf088 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/CheckResult.java 0b4240f 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
> 598bb2e 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java cff32d3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> 29f6ecf 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 27f677e 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
>  ce2b186 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java
>  9480d38 
>   
> ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java 
> a2a0583 
>   ql/src/test/queries/clientpositive/msck_repair_acid.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/partition_discovery.q PRE-CREATION 
>   ql/src/test/results/clientpositive/create_like.q.out f4a5ed5 
>   ql/src/test/results/clientpositive/create_like_view.q.out 870f280 
>   ql/src/test/results/clientpositive/default_file_format.q.out 0adf5ae 
>   ql/src/test/results/clientpositive/druid_topn.q.out 179902a 
>   ql/src/test/results/clientpositive/explain_locks.q.out ed7f1e8 
>   ql/src/test/results/clientpositive/llap/external_table_purge.q.out 24c778e 
>   ql/src/test/results/clientpositive/llap/mm_exim.q.out ee6cf06 
>   ql/src/test/results/clientpositive/llap/strict_managed_tables2.q.out 
> f3b6152 
>   ql/src/test/results/clientpositive/llap/whroot_external1.q.out cac158c 
>   ql/src/test/results/clientpositive/msck_repair_acid.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/msck_repair_drop.q.out 2456734 
>   ql/src/test/results/clientpositive/partition_discovery.q.out PRE-CREATION

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

2018-10-15 Thread j . prasanth . j



> On Oct. 16, 2018, 2:28 a.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
> > Lines 4761 (patched)
> > 
> >
> > Should this be on by default? If there are a lot of external tables 
> > (especially on s3), the metastore could be spending a lot of time doing 
> > auto discover. Could also affect the running of other MetastoreTaskThreads.

Yeah. I think this should be default. This will remove manual msck step or 
periodic msck query (via cron job). This thread kicks in once every 5 minutes 
but if the previous attempt is not done yet it will skip an attempt so as to 
avoid queue'ing up of background tasks. Also it will use high batch size by 
default so that in most case there should be 1 MS request per table. MSCK 
thread also runs in a thread pool. The only place this background thread could 
be blocked is when exclusive lock is obtained on a table (which gets released 
after txn timeout of 300s). 
We could probably restrict this only for EXTERNAL table types (currently it 
defaults to both EXTERNAL and MANAGED). Since managed is ACID by default we can 
avoid scanning managed tables. Changing ACID table layout out of band is 
shooting themselves in the foot anyway.


> On Oct. 16, 2018, 2:28 a.m., Jason Dere wrote:
> > ql/src/test/results/clientpositive/msck_repair_drop.q.out
> > Line 127 (original), 127 (patched)
> > 
> >
> > What is the new ordering of these messages? Looks like it could be 
> > potential issue when diffing golden files?

I think the ordering got changed as part of 
https://issues.apache.org/jira/browse/HIVE-20696 
I will check again to see if we are using hashset anywhere else. The ordering 
is based on partition name so prepending catalog name "hive" might have changed 
the order although not sure (either hashset issue or unstable sort).


> On Oct. 16, 2018, 2:28 a.m., Jason Dere wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java
> > Lines 141 (patched)
> > 
> >
> > Is this variable used? It's logged, but I think retentionSeconds should 
> > be used instead.

good catch. will fix it.


> On Oct. 16, 2018, 2:28 a.m., Jason Dere wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java
> > Lines 142 (patched)
> > 
> >
> > Might want to check for exception from TimeValidator.validate() in 
> > getRententionPeriodInSeconds, or else a bad setting in one table can fail 
> > here and prevent this from running for any tables.
> > But if you do skip that table, make sure the countdown latch is updated 
> > appropriately.

make sense. will fix it.


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68946/#review209582
---


On Oct. 16, 2018, 12:21 a.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68946/
> ---
> 
> (Updated Oct. 16, 2018, 12:21 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jason Dere.
> 
> 
> Bugs: HIVE-20707
> https://issues.apache.org/jira/browse/HIVE-20707
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20707: Automatic partition management
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 92a1c31 
>   hbase-handler/src/test/results/positive/external_table_ppd.q.out edcbe7e 
>   hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out 
> 1209c88 
>   hbase-handler/src/test/results/positive/hbase_ddl.q.out ccd4148 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out eeb97f0 
>   hbase-handler/src/test/results/positive/hbasestats.q.out 5a4aea9 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  a9d7468 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 807f159 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 46bf088 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/CheckResult.java 0b4240f 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
> 598bb2e 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java cff32d3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> 29f6ecf 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 27f677e 
>   
>

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

2018-10-15 Thread Jason Dere


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68946/#review209582
---




ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
Lines 4761 (patched)


Should this be on by default? If there are a lot of external tables 
(especially on s3), the metastore could be spending a lot of time doing auto 
discover. Could also affect the running of other MetastoreTaskThreads.



ql/src/test/results/clientpositive/msck_repair_drop.q.out
Line 127 (original), 127 (patched)


What is the new ordering of these messages? Looks like it could be 
potential issue when diffing golden files?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java
Lines 141 (patched)


Is this variable used? It's logged, but I think retentionSeconds should be 
used instead.



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java
Lines 142 (patched)


Might want to check for exception from TimeValidator.validate() in 
getRententionPeriodInSeconds, or else a bad setting in one table can fail here 
and prevent this from running for any tables.
But if you do skip that table, make sure the countdown latch is updated 
appropriately.


- Jason Dere


On Oct. 16, 2018, 12:21 a.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68946/
> ---
> 
> (Updated Oct. 16, 2018, 12:21 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jason Dere.
> 
> 
> Bugs: HIVE-20707
> https://issues.apache.org/jira/browse/HIVE-20707
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20707: Automatic partition management
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 92a1c31 
>   hbase-handler/src/test/results/positive/external_table_ppd.q.out edcbe7e 
>   hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out 
> 1209c88 
>   hbase-handler/src/test/results/positive/hbase_ddl.q.out ccd4148 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out eeb97f0 
>   hbase-handler/src/test/results/positive/hbasestats.q.out 5a4aea9 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  a9d7468 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 807f159 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 46bf088 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/CheckResult.java 0b4240f 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
> 598bb2e 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java cff32d3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> 29f6ecf 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 27f677e 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
>  ce2b186 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java
>  9480d38 
>   
> ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java 
> a2a0583 
>   ql/src/test/queries/clientpositive/msck_repair_acid.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/partition_discovery.q PRE-CREATION 
>   ql/src/test/results/clientpositive/create_like.q.out f4a5ed5 
>   ql/src/test/results/clientpositive/create_like_view.q.out 870f280 
>   ql/src/test/results/clientpositive/default_file_format.q.out 0adf5ae 
>   ql/src/test/results/clientpositive/druid_topn.q.out 179902a 
>   ql/src/test/results/clientpositive/explain_locks.q.out ed7f1e8 
>   ql/src/test/results/clientpositive/llap/external_table_purge.q.out 24c778e 
>   ql/src/test/results/clientpositive/llap/mm_exim.q.out ee6cf06 
>   ql/src/test/results/clientpositive/llap/strict_managed_tables2.q.out 
> f3b6152 
>   ql/src/test/results/clientpositive/llap/whroot_external1.q.out cac158c 
>   ql/src/test/results/clientpositive/msck_repair_acid.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/msck_repair_drop.q.out 2456734 
>   ql/src/test/results/clientpositive/partition_discovery.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/rename_external_partition_location.q.out 
> 02cd814 
>   ql/src/test/results/clientpositive/repl_2_exim_basic.q.out b2bcd51 
>   ql/src/test/results/clientpositive/show_create_table_alter.q.out 2c75c36 
>   ql/src/test/results/clientpositive/show_create_table_partitioned.q.out 
> e554a18 
>

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

2018-10-15 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68946/
---

(Updated Oct. 16, 2018, 12:21 a.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Changes
---

Some test fixes


Bugs: HIVE-20707
https://issues.apache.org/jira/browse/HIVE-20707


Repository: hive-git


Description
---

HIVE-20707: Automatic partition management


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 92a1c31 
  hbase-handler/src/test/results/positive/external_table_ppd.q.out edcbe7e 
  hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out 
1209c88 
  hbase-handler/src/test/results/positive/hbase_ddl.q.out ccd4148 
  hbase-handler/src/test/results/positive/hbase_queries.q.out eeb97f0 
  hbase-handler/src/test/results/positive/hbasestats.q.out 5a4aea9 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 a9d7468 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 807f159 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 46bf088 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/CheckResult.java 0b4240f 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
598bb2e 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java cff32d3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 29f6ecf 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 27f677e 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
 ce2b186 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java 
9480d38 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java 
a2a0583 
  ql/src/test/queries/clientpositive/msck_repair_acid.q PRE-CREATION 
  ql/src/test/queries/clientpositive/partition_discovery.q PRE-CREATION 
  ql/src/test/results/clientpositive/create_like.q.out f4a5ed5 
  ql/src/test/results/clientpositive/create_like_view.q.out 870f280 
  ql/src/test/results/clientpositive/default_file_format.q.out 0adf5ae 
  ql/src/test/results/clientpositive/druid_topn.q.out 179902a 
  ql/src/test/results/clientpositive/explain_locks.q.out ed7f1e8 
  ql/src/test/results/clientpositive/llap/external_table_purge.q.out 24c778e 
  ql/src/test/results/clientpositive/llap/mm_exim.q.out ee6cf06 
  ql/src/test/results/clientpositive/llap/strict_managed_tables2.q.out f3b6152 
  ql/src/test/results/clientpositive/llap/whroot_external1.q.out cac158c 
  ql/src/test/results/clientpositive/msck_repair_acid.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/msck_repair_drop.q.out 2456734 
  ql/src/test/results/clientpositive/partition_discovery.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/rename_external_partition_location.q.out 
02cd814 
  ql/src/test/results/clientpositive/repl_2_exim_basic.q.out b2bcd51 
  ql/src/test/results/clientpositive/show_create_table_alter.q.out 2c75c36 
  ql/src/test/results/clientpositive/show_create_table_partitioned.q.out 
e554a18 
  ql/src/test/results/clientpositive/show_create_table_serde.q.out 8b95c9b 
  ql/src/test/results/clientpositive/spark/stats_noscan_2.q.out 2d713a8 
  ql/src/test/results/clientpositive/stats_noscan_2.q.out 182820f 
  ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out 
2a442b4 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java
 294dfb7 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/api/MetastoreException.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 7b01678 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 16f4a50 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckInfo.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckPartitionExpressionProxy.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 66977d7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionIterable.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java
 PRE-CREATION

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

2018-10-11 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68946/
---

(Updated Oct. 12, 2018, 1:35 a.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Bugs: HIVE-20707
https://issues.apache.org/jira/browse/HIVE-20707


Repository: hive-git


Description (updated)
---

HIVE-20707: Automatic partition management


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cc6239c 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 cffa21a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java e913462 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 46bf088 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/CheckResult.java 0b4240f 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
598bb2e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 29f6ecf 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 27f677e 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
 ce2b186 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java 
7e768da 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java 
a2a0583 
  ql/src/test/queries/clientpositive/msck_repair_0.q aeb4820 
  ql/src/test/queries/clientpositive/msck_repair_2.q be745b2 
  ql/src/test/queries/clientpositive/msck_repair_3.q 140a690 
  ql/src/test/queries/clientpositive/msck_repair_acid.q PRE-CREATION 
  ql/src/test/queries/clientpositive/msck_repair_batchsize.q 5a7afcc 
  ql/src/test/queries/clientpositive/msck_repair_drop.q 9923fb5 
  ql/src/test/queries/clientpositive/partition_discovery.q PRE-CREATION 
  ql/src/test/results/clientpositive/msck_repair_0.q.out fa6e4a9 
  ql/src/test/results/clientpositive/msck_repair_2.q.out 7fbd934 
  ql/src/test/results/clientpositive/msck_repair_3.q.out 0e153fb 
  ql/src/test/results/clientpositive/msck_repair_acid.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/msck_repair_batchsize.q.out ab4b831 
  ql/src/test/results/clientpositive/msck_repair_drop.q.out 971c138 
  ql/src/test/results/clientpositive/partition_discovery.q.out PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java
 294dfb7 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/api/MetastoreException.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 7b01678 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 16f4a50 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckInfo.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckPartitionExpressionProxy.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 66977d7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionIterable.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
 f3b3866 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/RetryUtilities.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestPartitionManagement.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/68946/diff/2/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

2018-10-11 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68946/
---

(Updated Oct. 12, 2018, 1:35 a.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Changes
---

There is more scope added to this patch

1) Support managed and external tables. Managed tables will get X lock for MSCK 
REPAIR.

2) Retention period for partitions can be specified via table property.

3) MSCK repair can now handle retention periods via config.

4) All external tables will now automatically enable partition discovery.

On top of these there are fixes related to ACID, MSCK was acquiring wrong lock, 
explain locks was not working for msck, fixes related to catalog handling (some 
metastore APIs was not handling catalog names correctly).


Bugs: HIVE-20707
https://issues.apache.org/jira/browse/HIVE-20707


Repository: hive-git


Description
---

HIVE-20707: Automatic MSCK REPAIR for external tables


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cc6239c 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 cffa21a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java e913462 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 46bf088 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/CheckResult.java 0b4240f 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
598bb2e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 29f6ecf 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 27f677e 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
 ce2b186 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java 
7e768da 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java 
a2a0583 
  ql/src/test/queries/clientpositive/msck_repair_0.q aeb4820 
  ql/src/test/queries/clientpositive/msck_repair_2.q be745b2 
  ql/src/test/queries/clientpositive/msck_repair_3.q 140a690 
  ql/src/test/queries/clientpositive/msck_repair_acid.q PRE-CREATION 
  ql/src/test/queries/clientpositive/msck_repair_batchsize.q 5a7afcc 
  ql/src/test/queries/clientpositive/msck_repair_drop.q 9923fb5 
  ql/src/test/queries/clientpositive/partition_discovery.q PRE-CREATION 
  ql/src/test/results/clientpositive/msck_repair_0.q.out fa6e4a9 
  ql/src/test/results/clientpositive/msck_repair_2.q.out 7fbd934 
  ql/src/test/results/clientpositive/msck_repair_3.q.out 0e153fb 
  ql/src/test/results/clientpositive/msck_repair_acid.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/msck_repair_batchsize.q.out ab4b831 
  ql/src/test/results/clientpositive/msck_repair_drop.q.out 971c138 
  ql/src/test/results/clientpositive/partition_discovery.q.out PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java
 294dfb7 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/api/MetastoreException.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 7b01678 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 16f4a50 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckInfo.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckPartitionExpressionProxy.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 66977d7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionIterable.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
 f3b3866 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/RetryUtilities.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestPartitionManagement.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/68946/diff/2/

Changes: https://reviews.apache.org/r/68946/diff/1-2/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

2018-10-08 Thread Jason Dere


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68946/#review209341
---




ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java
Lines 88 (patched)


Can you use FileUtils.HIDDEN_FILES_PATH_FILTER? I believe 
standalone-metastore also has a FileUtils.java


- Jason Dere


On Oct. 8, 2018, 4:16 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68946/
> ---
> 
> (Updated Oct. 8, 2018, 4:16 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jason Dere.
> 
> 
> Bugs: HIVE-20707
> https://issues.apache.org/jira/browse/HIVE-20707
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20707: Automatic MSCK REPAIR for external tables
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> d0adc35544cb8ae9d007a1d2ccb9b9565eedca88 
>   data/conf/hive-site.xml 0daf9adc717bc1c4413d2e34691c26a3e2585c77 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  cffa21af33d5abb2162fa16b6b990a469075f03d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> e91346228e8724b8253364114145a348a7cbee26 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/CheckResult.java 
> 0b4240f5665f0b544b2fc5864fc098eb286a281e 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
> 598bb2ee8b72f1b7f75be7802b4eaae0204c988d 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
>  ce2b186b4dceda780106776daa022f18388ec76f 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java
>  7e768dacb0b00a0f1a9e64efbe778f9c2daaa31b 
>   
> ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java 
> a2a0583d4dbdfe9aece1a14ecac24e0e6189cafa 
>   ql/src/test/queries/clientpositive/auto_msck_repair_0.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/auto_msck_repair_1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/auto_msck_repair_2.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/auto_msck_repair_3.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/auto_msck_repair_4.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/auto_msck_repair_batchsize.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/msck_repair_0.q 
> aeb4820af5b6687f7ae4163a94bdd2be25a8b0cd 
>   ql/src/test/queries/clientpositive/msck_repair_2.q 
> be745b2d607d8c727b862c71f153f09d5622a8b5 
>   ql/src/test/queries/clientpositive/msck_repair_3.q 
> 140a6904ddc98b165d71a8b24314c56888ccbb9c 
>   ql/src/test/queries/clientpositive/msck_repair_batchsize.q 
> 5a7afcca5b86c1887308626c0dc4d99916811bea 
>   ql/src/test/queries/clientpositive/msck_repair_drop.q 
> 9923fb50cbdbdf9e8e07276ccaec073c490770e6 
>   ql/src/test/results/clientpositive/auto_msck_repair_0.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/auto_msck_repair_1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/auto_msck_repair_2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/auto_msck_repair_3.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/auto_msck_repair_4.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/auto_msck_repair_batchsize.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/msck_repair_0.q.out 
> fa6e4a988273a71b0f9dab64a48ddda6320d5f2f 
>   ql/src/test/results/clientpositive/msck_repair_2.q.out 
> 7fbd934e118e81b9c5f028191c7ea6582a34db75 
>   ql/src/test/results/clientpositive/msck_repair_3.q.out 
> 0e153fbe69ba39819fac4629ef1bf5f90c17f37f 
>   ql/src/test/results/clientpositive/msck_repair_batchsize.q.out 
> ab4b83137dcf1ce36846ce74e0a546528e81358b 
>   ql/src/test/results/clientpositive/msck_repair_drop.q.out 
> 971c1381276fa626bd91d34488a65e3bfb2781ae 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java
>  294dfb728e12efaa13d239ea7b8949587a50fe1f 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/api/MetastoreException.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  7b01678a10f4f0667844fec64ae76695d835bd6e 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
>  16f4a50d69f9120d565f61d028b060d7776689fc 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/AutoMsckTask.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
>  PRE-CREATION 
>

Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

2018-10-07 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68946/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-20707
https://issues.apache.org/jira/browse/HIVE-20707


Repository: hive-git


Description
---

HIVE-20707: Automatic MSCK REPAIR for external tables


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
d0adc35544cb8ae9d007a1d2ccb9b9565eedca88 
  data/conf/hive-site.xml 0daf9adc717bc1c4413d2e34691c26a3e2585c77 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 cffa21af33d5abb2162fa16b6b990a469075f03d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
e91346228e8724b8253364114145a348a7cbee26 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/CheckResult.java 
0b4240f5665f0b544b2fc5864fc098eb286a281e 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
598bb2ee8b72f1b7f75be7802b4eaae0204c988d 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
 ce2b186b4dceda780106776daa022f18388ec76f 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java 
7e768dacb0b00a0f1a9e64efbe778f9c2daaa31b 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java 
a2a0583d4dbdfe9aece1a14ecac24e0e6189cafa 
  ql/src/test/queries/clientpositive/auto_msck_repair_0.q PRE-CREATION 
  ql/src/test/queries/clientpositive/auto_msck_repair_1.q PRE-CREATION 
  ql/src/test/queries/clientpositive/auto_msck_repair_2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/auto_msck_repair_3.q PRE-CREATION 
  ql/src/test/queries/clientpositive/auto_msck_repair_4.q PRE-CREATION 
  ql/src/test/queries/clientpositive/auto_msck_repair_batchsize.q PRE-CREATION 
  ql/src/test/queries/clientpositive/msck_repair_0.q 
aeb4820af5b6687f7ae4163a94bdd2be25a8b0cd 
  ql/src/test/queries/clientpositive/msck_repair_2.q 
be745b2d607d8c727b862c71f153f09d5622a8b5 
  ql/src/test/queries/clientpositive/msck_repair_3.q 
140a6904ddc98b165d71a8b24314c56888ccbb9c 
  ql/src/test/queries/clientpositive/msck_repair_batchsize.q 
5a7afcca5b86c1887308626c0dc4d99916811bea 
  ql/src/test/queries/clientpositive/msck_repair_drop.q 
9923fb50cbdbdf9e8e07276ccaec073c490770e6 
  ql/src/test/results/clientpositive/auto_msck_repair_0.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/auto_msck_repair_1.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/auto_msck_repair_2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/auto_msck_repair_3.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/auto_msck_repair_4.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/auto_msck_repair_batchsize.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/msck_repair_0.q.out 
fa6e4a988273a71b0f9dab64a48ddda6320d5f2f 
  ql/src/test/results/clientpositive/msck_repair_2.q.out 
7fbd934e118e81b9c5f028191c7ea6582a34db75 
  ql/src/test/results/clientpositive/msck_repair_3.q.out 
0e153fbe69ba39819fac4629ef1bf5f90c17f37f 
  ql/src/test/results/clientpositive/msck_repair_batchsize.q.out 
ab4b83137dcf1ce36846ce74e0a546528e81358b 
  ql/src/test/results/clientpositive/msck_repair_drop.q.out 
971c1381276fa626bd91d34488a65e3bfb2781ae 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java
 294dfb728e12efaa13d239ea7b8949587a50fe1f 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/api/MetastoreException.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 7b01678a10f4f0667844fec64ae76695d835bd6e 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 16f4a50d69f9120d565f61d028b060d7776689fc 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/AutoMsckTask.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckInfo.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckPartitionExpressionProxy.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionIterable.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
 f3b38665676391fec9b85eb9a405c14632340dc6 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/RetryUtilities.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/68946/diff/1/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

11 matches

Site Navigation

Mail list logo

Footer information