Re: Review Request 72336: HIVE-23114: Insert overwrite with dynamic partitioning is not working correctly with direct insert

2020-04-08 Thread Marta Kuczora via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72336/
---

(Updated April 8, 2020, 1:47 p.m.)


Review request for hive and Peter Vary.


Changes
---

Fixing whitespaces.


Bugs: HIVE-23114
https://issues.apache.org/jira/browse/HIVE-23114


Repository: hive-git


Description
---

The idea behind the patch is the following:
When doing a multi-statement insert overwrite with dynamic partitioning, the 
partition information will be written to the manifest file. With this 
information, each FileSinkOperator can clean-up only the partition directories 
written by the same FileSinkOperator and do not clean-up the partition 
directories written by the other FileSinkOperators.
If a statement from the insert overwrite query, doesn't produce any data, a 
manifest file will still be written, otherwise the missing manifest file would 
result a clean-up on table level which could delete the data written by the 
other FileSinkOperators.


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties e99ce7babb 
  ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
d68d8f9409 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 04166a23ee 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java e25dc54e7d 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 17e6cdf162 
  ql/src/test/queries/clientpositive/acid_direct_insert_insert_overwrite.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/acid_multiinsert_dyn_part.q PRE-CREATION 
  ql/src/test/results/clientpositive/acid_direct_insert_insert_overwrite.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/acid_multiinsert_dyn_part.q.out 
PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/acid_direct_insert_insert_overwrite.q.out
 PRE-CREATION 
  ql/src/test/results/clientpositive/llap/acid_multiinsert_dyn_part.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/72336/diff/2/

Changes: https://reviews.apache.org/r/72336/diff/1-2/


Testing
---

Added specific q tests for different insert overwrite scenarios.


Thanks,

Marta Kuczora

Re: Review Request 72336: HIVE-23114: Insert overwrite with dynamic partitioning is not working correctly with direct insert

2020-04-08 Thread Peter Vary via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72336/#review220255
---


Fix it, then Ship it!




Single very important comment! :)


ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
Lines 2834 (patched)


nit: extra space


- Peter Vary


On ápr. 8, 2020, 12:20 du, Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72336/
> ---
> 
> (Updated ápr. 8, 2020, 12:20 du)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Bugs: HIVE-23114
> https://issues.apache.org/jira/browse/HIVE-23114
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The idea behind the patch is the following:
> When doing a multi-statement insert overwrite with dynamic partitioning, the 
> partition information will be written to the manifest file. With this 
> information, each FileSinkOperator can clean-up only the partition 
> directories written by the same FileSinkOperator and do not clean-up the 
> partition directories written by the other FileSinkOperators.
> If a statement from the insert overwrite query, doesn't produce any data, a 
> manifest file will still be written, otherwise the missing manifest file 
> would result a clean-up on table level which could delete the data written by 
> the other FileSinkOperators.
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties e99ce7babb 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
> d68d8f9409 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 04166a23ee 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java e25dc54e7d 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 17e6cdf162 
>   ql/src/test/queries/clientpositive/acid_direct_insert_insert_overwrite.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/acid_multiinsert_dyn_part.q PRE-CREATION 
>   
> ql/src/test/results/clientpositive/acid_direct_insert_insert_overwrite.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/acid_multiinsert_dyn_part.q.out 
> PRE-CREATION 
>   
> ql/src/test/results/clientpositive/llap/acid_direct_insert_insert_overwrite.q.out
>  PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/acid_multiinsert_dyn_part.q.out 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72336/diff/1/
> 
> 
> Testing
> ---
> 
> Added specific q tests for different insert overwrite scenarios.
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

Review Request 72336: HIVE-23114: Insert overwrite with dynamic partitioning is not working correctly with direct insert

2020-04-08 Thread Marta Kuczora via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72336/
---

Review request for hive and Peter Vary.


Bugs: HIVE-23114
https://issues.apache.org/jira/browse/HIVE-23114


Repository: hive-git


Description
---

The idea behind the patch is the following:
When doing a multi-statement insert overwrite with dynamic partitioning, the 
partition information will be written to the manifest file. With this 
information, each FileSinkOperator can clean-up only the partition directories 
written by the same FileSinkOperator and do not clean-up the partition 
directories written by the other FileSinkOperators.
If a statement from the insert overwrite query, doesn't produce any data, a 
manifest file will still be written, otherwise the missing manifest file would 
result a clean-up on table level which could delete the data written by the 
other FileSinkOperators.


Diffs
-

  itests/src/test/resources/testconfiguration.properties e99ce7babb 
  ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
d68d8f9409 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 04166a23ee 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java e25dc54e7d 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 17e6cdf162 
  ql/src/test/queries/clientpositive/acid_direct_insert_insert_overwrite.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/acid_multiinsert_dyn_part.q PRE-CREATION 
  ql/src/test/results/clientpositive/acid_direct_insert_insert_overwrite.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/acid_multiinsert_dyn_part.q.out 
PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/acid_direct_insert_insert_overwrite.q.out
 PRE-CREATION 
  ql/src/test/results/clientpositive/llap/acid_multiinsert_dyn_part.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/72336/diff/1/


Testing
---

Added specific q tests for different insert overwrite scenarios.


Thanks,

Marta Kuczora