Re: Review Request 69367: Query based compactor for full CRUD Acid tables

2019-01-28 Thread Eugene Koifman 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69367/#review212399
---




itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
Lines 299 (patched)


testMoreBucketsThanReducers/testMoreBucketsThanReducers2 in TestTxnCommands 
force a specific number of reducers



itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
Lines 185 (patched)


nit: since this is filtering for 'base' it's not checking if it 'only' 
contains base...



itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
Lines 195 (patched)


I still don't understand what this comment is conveying.  This is just a 
normal read, so I would assume TezSplitGrouper is not running in compactor mode


- Eugene Koifman


On Jan. 28, 2019, 11:49 a.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69367/
> ---
> 
> (Updated Jan. 28, 2019, 11:49 a.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Bugs: HIVE-20699
> https://issues.apache.org/jira/browse/HIVE-20699
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://jira.apache.org/jira/browse/HIVE-20699
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b3a475478d 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
> d6a41919bf 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java e7aa041c25 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveSplitGenerator.java 
> 15c14c9be5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SplitGrouper.java 7f8bd229a6 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
> fbb931cbcd 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 
> 6d4578e7a0 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 4d55592b63 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> db3b427adc 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> dc05e1990e 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> a0df82cb20 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFValidateAcidSortOrder.java
>  PRE-CREATION 
>   ql/src/test/results/clientpositive/show_functions.q.out c9716e904c 
> 
> 
> Diff: https://reviews.apache.org/r/69367/diff/9/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

Re: Review Request 69704: HIVE-21052

2019-01-28 Thread Jaume Marhuenda 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69704/
---

(Updated Jan. 28, 2019, 10:32 p.m.)


Review request for hive.


Repository: hive-git


Description
---

Make sure transaction get cleaned if they are aborted before addPartitions is 
called


Diffs (updated)
-

  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 dc7b2877bf 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 5dbf634825 
  ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java 3482cfce36 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 06b0209aa0 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CleanerExecutorService.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java a0df82cb20 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/NonReentrantReadWriteLock.java
 PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
5e085f84af 
  
ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestCleanerExecutorService.java
 PRE-CREATION 
  shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
b6f70ebe63 
  shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 
c569b242ae 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionType.java
 7450b27cf3 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 13e287e352 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 8f149d1d6e 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 9e5f0860f2 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 be1f8c7849 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
9576f8775a 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
 ea70503988 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
 8253ccb9c9 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 91a9ab4053 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 898a94dcd6 


Diff: https://reviews.apache.org/r/69704/diff/4/

Changes: https://reviews.apache.org/r/69704/diff/3-4/


Testing
---

Planning to manually test it using the hive-spark connector, which is where 
this bug was discovered.


Thanks,

Jaume Marhuenda

[jira] [Created] (HIVE-21175) Use StandardCharsets Where Possible (Part 2)

2019-01-28 Thread BELUGA BEHR (JIRA) 
BELUGA BEHR created HIVE-21175:
--

 Summary: Use StandardCharsets Where Possible (Part 2)
 Key: HIVE-21175
 URL: https://issues.apache.org/jira/browse/HIVE-21175
 Project: Hive
  Issue Type: Improvement
Affects Versions: 3.2.0
Reporter: BELUGA BEHR
 Fix For: 4.0.0


Additional work not already addressed by [HIVE-21148].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69367: Query based compactor for full CRUD Acid tables

2019-01-28 Thread Vaibhav Gumashta 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69367/
---

(Updated Jan. 28, 2019, 7:49 p.m.)


Review request for hive and Eugene Koifman.


Bugs: HIVE-20699
https://issues.apache.org/jira/browse/HIVE-20699


Repository: hive-git


Description
---

https://jira.apache.org/jira/browse/HIVE-20699


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b3a475478d 
  itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
d6a41919bf 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java e7aa041c25 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveSplitGenerator.java 
15c14c9be5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SplitGrouper.java 7f8bd229a6 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
fbb931cbcd 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 6d4578e7a0 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 4d55592b63 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
db3b427adc 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
dc05e1990e 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java a0df82cb20 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFValidateAcidSortOrder.java
 PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out c9716e904c 


Diff: https://reviews.apache.org/r/69367/diff/9/

Changes: https://reviews.apache.org/r/69367/diff/8-9/


Testing
---


Thanks,

Vaibhav Gumashta

[jira] [Created] (HIVE-21174) hive.stats.ndv.error parameter documentation issue

2019-01-28 Thread Pablo Junge (JIRA) 
Pablo Junge created HIVE-21174:
--

 Summary: hive.stats.ndv.error parameter documentation issue
 Key: HIVE-21174
 URL: https://issues.apache.org/jira/browse/HIVE-21174
 Project: Hive
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 2.3.4, 3.1.1, 3.1.0, 2.3.2, 2.3.1, 3.0.0, 2.3.0, 2.2.0, 
2.1.1, 2.1.0, 2.0.1, 2.0.0, 2.0.2, 2.1.2, 2.4.0, 2.2.1, 2.3.3, 3.0.1, 3.10, 
3.2.0, 3.1.2
Reporter: Pablo Junge
Assignee: Nita Dembla
 Fix For: 2.0.2, 2.1.2, 2.4.0, 2.2.1, 2.3.3, 3.0.1, 3.10, 3.2.0, 
3.1.2, 2.3.4, 3.1.1, 3.1.0, 2.3.2, 2.3.1, 3.0.0, 2.3.0, 2.2.0, 2.1.1, 2.1.0, 
2.0.1, 2.0.0


Hive documentation for hive.stats.ndv.error does not specify that 
hive.stats.ndv.error will only affect FM Sketch and not HLL.

 

https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)