[jira] [Created] (HIVE-23162) Remove swapping logic to merge joins in AST converter

2020-04-08 Thread Jesus Camacho Rodriguez (Jira) 
Jesus Camacho Rodriguez created HIVE-23162:
--

 Summary: Remove swapping logic to merge joins in AST converter
 Key: HIVE-23162
 URL: https://issues.apache.org/jira/browse/HIVE-23162
 Project: Hive
  Issue Type: Bug
  Components: CBO
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


In ASTConverter, there is some logic to invert join inputs so the logic to 
merge joins in SemanticAnalyzer kicks in.
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java#L407

There is a bug because inputs are swapped but the schema is not. However, it 
turns out that logic is not needed now that merging is off by default.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23161) Umbrela issue for Hive on ARM issues

2020-04-08 Thread Zhenyu Zheng (Jira) 
Zhenyu Zheng created HIVE-23161:
---

 Summary: Umbrela issue for Hive on ARM issues
 Key: HIVE-23161
 URL: https://issues.apache.org/jira/browse/HIVE-23161
 Project: Hive
  Issue Type: Bug
Reporter: Zhenyu Zheng






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23160) get_partitions_with_specs fail to close the query

2020-04-08 Thread Vineet Garg (Jira) 
Vineet Garg created HIVE-23160:
--

 Summary: get_partitions_with_specs fail to close the query
 Key: HIVE-23160
 URL: https://issues.apache.org/jira/browse/HIVE-23160
 Project: Hive
  Issue Type: Sub-task
  Components: Metastore, Standalone Metastore
Affects Versions: 4.0.0
Reporter: Vineet Garg


The api relies on try to close the resource (query) but it fails (likely 
because try is calling close but instead closeAll need to be called)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23159) Cleanup ShowCreateTableOperation

2020-04-08 Thread David Mollitor (Jira) 
David Mollitor created HIVE-23159:
-

 Summary: Cleanup ShowCreateTableOperation
 Key: HIVE-23159
 URL: https://issues.apache.org/jira/browse/HIVE-23159
 Project: Hive
  Issue Type: Bug
Reporter: David Mollitor
Assignee: David Mollitor


* Move StringTemplate templates to external files
 * Explore better leveraging StringTemplate capabilities to remove duplicate 
functionality in the class
 * General clean up and formatting



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23158) Optimize S3A recordReader policy for Random IO formats

2020-04-08 Thread Panagiotis Garefalakis (Jira) 
Panagiotis Garefalakis created HIVE-23158:
-

 Summary: Optimize S3A recordReader policy for Random IO formats
 Key: HIVE-23158
 URL: https://issues.apache.org/jira/browse/HIVE-23158
 Project: Hive
  Issue Type: Bug
Reporter: Panagiotis Garefalakis
Assignee: Panagiotis Garefalakis


S3A filesystem client (inherited by Hadoop) supports the notion of input 
policies.
These policies tune the behaviour of HTTP requests that are used for reading 
different filetypes such as TEXT or ORC.

For formats such as ORC and Parquet do a lot of seek operations, thus there is 
an optimized RANDOM mode that reads files only partially instead of fully 
(default).

I am suggesting to add some extra logic as part of HiveInputFormat to make sure 
we optimize for random IO when data is stored on S3A using formats such as ORC 
or Parquet.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72324: HIVE-22750: Consolidate LockType naming

2020-04-08 Thread Marton Bod 


> On April 8, 2020, 3:23 p.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockComponentBuilder.java
> > Line 60 (original), 60 (patched)
> > 
> >
> > add builder for ExclWrite

added


> On April 8, 2020, 3:23 p.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
> > Lines 14 (patched)
> > 
> >
> > why not enum?

That was my initial approach too, but in the end, having a second enum required 
lots of mapping between the two enums, and both having the same value set 
caused quite a bit of duplication too. I think by using this util class, we can 
more cleanly extend the LockType enum with any necessary state and behaviour, 
without introducing duplication.


> On April 8, 2020, 3:23 p.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
> > Lines 34 (patched)
> > 
> >
> > you can move inverse into static and do no calc every time

good spot, fixed


- Marton


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72324/#review220258
---


On April 8, 2020, 3:09 p.m., Marton Bod wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72324/
> ---
> 
> (Updated April 8, 2020, 3:09 p.m.)
> 
> 
> Review request for hive, Denys Kuzmenko, Peter Vary, and Zoltan Chovan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-22750: Consolidate LockType naming
> 
> 
> Diffs
> -
> 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java
>  e249b7775e 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/client/lock/Lock.java
>  52eb6133e7 
>   metastore/scripts/upgrade/hive/hive-schema-4.0.0.hive.sql 03540bba4d 
>   metastore/scripts/upgrade/hive/upgrade-3.1.0-to-4.0.0.hive.sql fa518747de 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 17e6cdf162 
>   ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
> 72f095d264 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 80fb1aff78 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockType.java
>  8ae4351129 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  db4cfb996a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  cf3137928f 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  849970eb56 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockComponentBuilder.java
>  c739d4d196 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 098ddec5dc 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  7d0db0c3a0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
>  da38a6bbd3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStoreTxns.java
>  1dfc105958 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java
>  PRE-CREATION 
>   streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 
> d44065018f 
> 
> 
> Diff: https://reviews.apache.org/r/72324/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Marton Bod
> 
>

Re: Review Request 72324: HIVE-22750: Consolidate LockType naming

2020-04-08 Thread Denys Kuzmenko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72324/#review220258
---



LGTM, just few more comments


standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockComponentBuilder.java
Line 60 (original), 60 (patched)


add builder for ExclWrite



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
Lines 14 (patched)


why not enum?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
Lines 34 (patched)


you can move inverse into static and do no calc every time


- Denys Kuzmenko


On April 8, 2020, 3:09 p.m., Marton Bod wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72324/
> ---
> 
> (Updated April 8, 2020, 3:09 p.m.)
> 
> 
> Review request for hive, Denys Kuzmenko, Peter Vary, and Zoltan Chovan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-22750: Consolidate LockType naming
> 
> 
> Diffs
> -
> 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java
>  e249b7775e 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/client/lock/Lock.java
>  52eb6133e7 
>   metastore/scripts/upgrade/hive/hive-schema-4.0.0.hive.sql 03540bba4d 
>   metastore/scripts/upgrade/hive/upgrade-3.1.0-to-4.0.0.hive.sql fa518747de 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 17e6cdf162 
>   ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
> 72f095d264 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 80fb1aff78 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockType.java
>  8ae4351129 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  db4cfb996a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  cf3137928f 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  849970eb56 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockComponentBuilder.java
>  c739d4d196 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 098ddec5dc 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  7d0db0c3a0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
>  da38a6bbd3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStoreTxns.java
>  1dfc105958 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java
>  PRE-CREATION 
>   streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 
> d44065018f 
> 
> 
> Diff: https://reviews.apache.org/r/72324/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Marton Bod
> 
>

Re: Review Request 72324: HIVE-22750: Consolidate LockType naming

2020-04-08 Thread Marton Bod 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72324/
---

(Updated April 8, 2020, 3:09 p.m.)


Review request for hive, Denys Kuzmenko, Peter Vary, and Zoltan Chovan.


Repository: hive-git


Description
---

HIVE-22750: Consolidate LockType naming


Diffs (updated)
-

  
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java
 e249b7775e 
  
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/client/lock/Lock.java
 52eb6133e7 
  metastore/scripts/upgrade/hive/hive-schema-4.0.0.hive.sql 03540bba4d 
  metastore/scripts/upgrade/hive/upgrade-3.1.0-to-4.0.0.hive.sql fa518747de 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 17e6cdf162 
  ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
72f095d264 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
80fb1aff78 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockType.java
 8ae4351129 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 db4cfb996a 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 cf3137928f 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 849970eb56 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockComponentBuilder.java
 c739d4d196 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
098ddec5dc 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 7d0db0c3a0 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
 da38a6bbd3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStoreTxns.java
 1dfc105958 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java
 PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/TransactionBatch.java d44065018f 


Diff: https://reviews.apache.org/r/72324/diff/3/

Changes: https://reviews.apache.org/r/72324/diff/2-3/


Testing
---


Thanks,

Marton Bod

Re: Review Request 72324: HIVE-22750: Consolidate LockType naming

2020-04-08 Thread Denys Kuzmenko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72324/#review220256
---




metastore/scripts/upgrade/hive/hive-schema-4.0.0.hive.sql
Line 1421 (original), 1421 (patched)


where is excl_write?


- Denys Kuzmenko


On April 6, 2020, 1:58 p.m., Marton Bod wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72324/
> ---
> 
> (Updated April 6, 2020, 1:58 p.m.)
> 
> 
> Review request for hive, Denys Kuzmenko, Peter Vary, and Zoltan Chovan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-22750: Consolidate LockType naming
> 
> 
> Diffs
> -
> 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java
>  e249b7775e 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/client/lock/Lock.java
>  52eb6133e7 
>   metastore/scripts/upgrade/hive/hive-schema-4.0.0.hive.sql 03540bba4d 
>   metastore/scripts/upgrade/hive/upgrade-3.1.0-to-4.0.0.hive.sql fa518747de 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 17e6cdf162 
>   ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
> 72f095d264 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 80fb1aff78 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockType.java
>  8ae4351129 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  db4cfb996a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  cf3137928f 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  849970eb56 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockComponentBuilder.java
>  c739d4d196 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 098ddec5dc 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  7d0db0c3a0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
>  da38a6bbd3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStoreTxns.java
>  1dfc105958 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java
>  PRE-CREATION 
>   streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 
> d44065018f 
> 
> 
> Diff: https://reviews.apache.org/r/72324/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Marton Bod
> 
>

Re: Review Request 72336: HIVE-23114: Insert overwrite with dynamic partitioning is not working correctly with direct insert

2020-04-08 Thread Marta Kuczora via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72336/
---

(Updated April 8, 2020, 1:47 p.m.)


Review request for hive and Peter Vary.


Changes
---

Fixing whitespaces.


Bugs: HIVE-23114
https://issues.apache.org/jira/browse/HIVE-23114


Repository: hive-git


Description
---

The idea behind the patch is the following:
When doing a multi-statement insert overwrite with dynamic partitioning, the 
partition information will be written to the manifest file. With this 
information, each FileSinkOperator can clean-up only the partition directories 
written by the same FileSinkOperator and do not clean-up the partition 
directories written by the other FileSinkOperators.
If a statement from the insert overwrite query, doesn't produce any data, a 
manifest file will still be written, otherwise the missing manifest file would 
result a clean-up on table level which could delete the data written by the 
other FileSinkOperators.


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties e99ce7babb 
  ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
d68d8f9409 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 04166a23ee 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java e25dc54e7d 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 17e6cdf162 
  ql/src/test/queries/clientpositive/acid_direct_insert_insert_overwrite.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/acid_multiinsert_dyn_part.q PRE-CREATION 
  ql/src/test/results/clientpositive/acid_direct_insert_insert_overwrite.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/acid_multiinsert_dyn_part.q.out 
PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/acid_direct_insert_insert_overwrite.q.out
 PRE-CREATION 
  ql/src/test/results/clientpositive/llap/acid_multiinsert_dyn_part.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/72336/diff/2/

Changes: https://reviews.apache.org/r/72336/diff/1-2/


Testing
---

Added specific q tests for different insert overwrite scenarios.


Thanks,

Marta Kuczora

[jira] [Created] (HIVE-23157) Mutual TLS authentication for Metastore

2020-04-08 Thread Thiru Paramasivan (Jira) 
Thiru Paramasivan created HIVE-23157:


 Summary: Mutual TLS authentication for Metastore
 Key: HIVE-23157
 URL: https://issues.apache.org/jira/browse/HIVE-23157
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Affects Versions: 2.3.6, 3.0.0
Reporter: Thiru Paramasivan


Hive Metastore allows various security features. But it does not allow SSL 
client authentication (Mutual TLS or mTLS), even though the underlying 
[ThriftServer|https://github.com/apache/thrift/blob/master/lib/java/src/org/apache/thrift/transport/TSSLTransportFactory.java#L123]
 supports it. This enhancement request is for additional configurations to 
ThriftServer to allow client authentication.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72336: HIVE-23114: Insert overwrite with dynamic partitioning is not working correctly with direct insert

2020-04-08 Thread Peter Vary via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72336/#review220255
---


Fix it, then Ship it!




Single very important comment! :)


ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
Lines 2834 (patched)


nit: extra space


- Peter Vary


On ápr. 8, 2020, 12:20 du, Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72336/
> ---
> 
> (Updated ápr. 8, 2020, 12:20 du)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Bugs: HIVE-23114
> https://issues.apache.org/jira/browse/HIVE-23114
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The idea behind the patch is the following:
> When doing a multi-statement insert overwrite with dynamic partitioning, the 
> partition information will be written to the manifest file. With this 
> information, each FileSinkOperator can clean-up only the partition 
> directories written by the same FileSinkOperator and do not clean-up the 
> partition directories written by the other FileSinkOperators.
> If a statement from the insert overwrite query, doesn't produce any data, a 
> manifest file will still be written, otherwise the missing manifest file 
> would result a clean-up on table level which could delete the data written by 
> the other FileSinkOperators.
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties e99ce7babb 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
> d68d8f9409 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 04166a23ee 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java e25dc54e7d 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 17e6cdf162 
>   ql/src/test/queries/clientpositive/acid_direct_insert_insert_overwrite.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/acid_multiinsert_dyn_part.q PRE-CREATION 
>   
> ql/src/test/results/clientpositive/acid_direct_insert_insert_overwrite.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/acid_multiinsert_dyn_part.q.out 
> PRE-CREATION 
>   
> ql/src/test/results/clientpositive/llap/acid_direct_insert_insert_overwrite.q.out
>  PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/acid_multiinsert_dyn_part.q.out 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72336/diff/1/
> 
> 
> Testing
> ---
> 
> Added specific q tests for different insert overwrite scenarios.
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

[jira] [Created] (HIVE-23156) NPE if -f is used with HCatCLI

2020-04-08 Thread Ivan Suller (Jira) 
Ivan Suller created HIVE-23156:
--

 Summary: NPE if -f is used with HCatCLI
 Key: HIVE-23156
 URL: https://issues.apache.org/jira/browse/HIVE-23156
 Project: Hive
  Issue Type: Improvement
  Components: HCatalog
Reporter: Ivan Suller


After HIVE-22889 if there is no -e cli parameter then a NPE is thrown. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Review Request 72336: HIVE-23114: Insert overwrite with dynamic partitioning is not working correctly with direct insert

2020-04-08 Thread Marta Kuczora via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72336/
---

Review request for hive and Peter Vary.


Bugs: HIVE-23114
https://issues.apache.org/jira/browse/HIVE-23114


Repository: hive-git


Description
---

The idea behind the patch is the following:
When doing a multi-statement insert overwrite with dynamic partitioning, the 
partition information will be written to the manifest file. With this 
information, each FileSinkOperator can clean-up only the partition directories 
written by the same FileSinkOperator and do not clean-up the partition 
directories written by the other FileSinkOperators.
If a statement from the insert overwrite query, doesn't produce any data, a 
manifest file will still be written, otherwise the missing manifest file would 
result a clean-up on table level which could delete the data written by the 
other FileSinkOperators.


Diffs
-

  itests/src/test/resources/testconfiguration.properties e99ce7babb 
  ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
d68d8f9409 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 04166a23ee 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java e25dc54e7d 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 17e6cdf162 
  ql/src/test/queries/clientpositive/acid_direct_insert_insert_overwrite.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/acid_multiinsert_dyn_part.q PRE-CREATION 
  ql/src/test/results/clientpositive/acid_direct_insert_insert_overwrite.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/acid_multiinsert_dyn_part.q.out 
PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/acid_direct_insert_insert_overwrite.q.out
 PRE-CREATION 
  ql/src/test/results/clientpositive/llap/acid_multiinsert_dyn_part.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/72336/diff/1/


Testing
---

Added specific q tests for different insert overwrite scenarios.


Thanks,

Marta Kuczora

[jira] [Created] (HIVE-23155) Hive Replication to support "archival" destinations

2020-04-08 Thread Manish Maheshwari (Jira) 
Manish Maheshwari created HIVE-23155:


 Summary: Hive Replication to support "archival" destinations
 Key: HIVE-23155
 URL: https://issues.apache.org/jira/browse/HIVE-23155
 Project: Hive
  Issue Type: New Feature
Reporter: Manish Maheshwari


Currently in Hive replication any partitions dropped in the source are also 
propagated to the destination. The request here is to support a new destination 
type called as "archival" with the following properties
 * Partitions deleted on source are not deleted in the target
 * Tables deleted on source are not deleted in the target
 ** When new tables with the same name are added in the source, rename the old 
table (s) in the destination with some naming mechanism
 * Overwritten partitions on prem / Acid changes on prem etc are should be 
allowed to be replicated.

Use case - We are using Hive replication from on prem HDFS to S3 where S3 is an 
archive for cold data. We want old  data that is deleted on prem does not get 
deleted on the cloud.

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Review Request 72333: HIVE-23154: Fix race condition in Utilities::mvFileToFinalPath

2020-04-08 Thread Rajesh Balamohan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72333/
---

Review request for hive, Ashutosh Chauhan and Thejas Nair.


Bugs: HIVE-23154
https://issues.apache.org/jira/browse/HIVE-23154


Repository: hive-git


Description
---

With rename(), we could run into a race condition between snapshot of files to 
be moved ("filesKept") and when the fs.rename happens. It is possible that run 
awaay task could have added more files to this. 

1. Patch fixes the problem by relying on local threadpool to move the files 
instead of fs.rename (where S3AFileSystem's rename is inherently parallel).

2. Same race condition issue persists in "insert into" mode as well, which was 
relying on "fs.rename". Patch fixes this issue as well.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java e25dc54e7d 


Diff: https://reviews.apache.org/r/72333/diff/1/


Testing
---


Thanks,

Rajesh Balamohan

[jira] [Created] (HIVE-23154) Fix race condition in Utilities::mvFileToFinalPath

2020-04-08 Thread Rajesh Balamohan (Jira) 
Rajesh Balamohan created HIVE-23154:
---

 Summary: Fix race condition in Utilities::mvFileToFinalPath
 Key: HIVE-23154
 URL: https://issues.apache.org/jira/browse/HIVE-23154
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Reporter: Rajesh Balamohan
 Attachments: HIVE-23154.1.patch

Utilities::mvFileToFinalPath is used for moving files from "/_tmp.-ext to 
"/-ext" folder. Tasks write data to "_tmp" . Before writing to final 
destination, they are moved to "-ext" folder. As part of it, it has checks to 
ensure that run-away task outputs are not copied to "-ext" folder.

Currently, there is a race condition between computing the snapshot of files to 
be copied and the rename operation. Same issue persists in "insert into" case 
as well.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)