subject:"Review Request 71904\: HIVE\-21164\: ACID\: explore how we can avoid a move step during inserts\/compaction"

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

2020-02-18 Thread Marta Kuczora via Review Board



> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
> > Lines 1732-1737 (patched)
> > 
> >
> > What about using lambda here?
> 
> Marta Kuczora wrote:
> Fixed it.

At the end this code part got removed.


- Marta


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71904/#review219487
---


On Feb. 18, 2020, 12:21 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71904/
> ---
> 
> (Updated Feb. 18, 2020, 12:21 p.m.)
> 
> 
> Review request for hive, Gopal V and Peter Vary.
> 
> 
> Bugs: HIVE-21164
> https://issues.apache.org/jira/browse/HIVE-21164
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Extended the original patch with saving the task attempt ids in the file 
> names and also fixed some bugs in the original patch.
> With this fix, inserting into an ACID table would not use move task to place 
> the generated files into the final directory. It will inserts every files to 
> the final directory and then clean up the files which are not needed (like 
> written by failed task attempts).
> Also fixed the replication tests which failed for the original patch as well.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d3cb60b790 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  da677c7977 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
> 056cd27496 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
>  31d15fdef9 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
>  c2aa73b5f1 
>   itests/src/test/resources/testconfiguration.properties 1b1bf1147a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
> 9a3258115b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9ad4e71482 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 06e4ebee82 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6c67bc7dd8 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidInputFormat.java bba3960102 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java 1e8bb223f2 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 2f5ec5270c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 
> 8980a6292a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/RecordUpdater.java 737e6774b7 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 76984abd0a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java 
> c4c56f8477 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
> b8a0f0465c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 
> 398698ec06 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  2543dc6fc4 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1eb9c12cc8 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 73ca658d9c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 33d3beba46 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 
> c102a69f8f 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ecc7bdee4d 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java bed05819b5 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 739f2b654b 
>   ql/src/java/org/apache/hadoop/hive/ql/util/UpgradeTool.java 58e6289583 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnAddPartition.java c9cb6692df 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java 842140815d 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java e56d83158f 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java 908ceb43fc 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnConcatenate.java 8676e0db11 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnExIm.java 66b2b2768b 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java bb55d9fd79 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java ea6b1d9bec 
>   ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 
> af14e628b3 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 83db48e758 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 
> 2c4b69b2fe 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 48e9afc496 
>

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

2020-02-18 Thread Marta Kuczora via Review Board



> On Feb. 4, 2020, 10:16 p.m., Rajesh Balamohan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
> > Line 4382 (original), 4397 (patched)
> > 
> >
> > Is this needed for direct insert?. In objectstores, we could have calls 
> > getting throttled.

That's a really good question, I was thinking about it a lot. I think it is not 
needed. This method does two things: removes the temporarily and duplicated 
files and returns the emptyBuckets list. This list contains elements if the 
number of buckets are bigger than the number of files. In this case, for MM 
tables,  empty files will be created. But this is not the case for ACID tables, 
there won't be any empty files created for ACID tables. I want to revisit this 
topic whether or not we need these empty files, but for now, I would go with 
the same behaviour as for ACID tables. 
About the temp file removal, when the direct insert is finished all files which 
are not committed (meaning not in the manifest files) will be deleted prior to 
this call. So there shouldn't be any unnecessary files left at this point. 
I remove this call, and upload a patch to see the result of the pre-commit 
tests. If everything passes, I think it is safe to remove this call in case of 
direct insert.


- Marta


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71904/#review219494
---


On Feb. 18, 2020, 12:21 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71904/
> ---
> 
> (Updated Feb. 18, 2020, 12:21 p.m.)
> 
> 
> Review request for hive, Gopal V and Peter Vary.
> 
> 
> Bugs: HIVE-21164
> https://issues.apache.org/jira/browse/HIVE-21164
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Extended the original patch with saving the task attempt ids in the file 
> names and also fixed some bugs in the original patch.
> With this fix, inserting into an ACID table would not use move task to place 
> the generated files into the final directory. It will inserts every files to 
> the final directory and then clean up the files which are not needed (like 
> written by failed task attempts).
> Also fixed the replication tests which failed for the original patch as well.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d3cb60b790 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  da677c7977 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
> 056cd27496 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
>  31d15fdef9 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
>  c2aa73b5f1 
>   itests/src/test/resources/testconfiguration.properties 1b1bf1147a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
> 9a3258115b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9ad4e71482 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 06e4ebee82 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6c67bc7dd8 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidInputFormat.java bba3960102 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java 1e8bb223f2 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 2f5ec5270c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 
> 8980a6292a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/RecordUpdater.java 737e6774b7 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 76984abd0a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java 
> c4c56f8477 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
> b8a0f0465c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 
> 398698ec06 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  2543dc6fc4 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1eb9c12cc8 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 73ca658d9c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 33d3beba46 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 
> c102a69f8f 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ecc7bdee4d 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java bed05819b5 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 739f2b654b 
>   ql/src/java/org/apache/hadoop/hive/ql/util/UpgradeTool.java 58e6289583 
>

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

2020-02-18 Thread Marta Kuczora via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71904/
---

(Updated Feb. 18, 2020, 12:21 p.m.)


Review request for hive, Gopal V and Peter Vary.


Bugs: HIVE-21164
https://issues.apache.org/jira/browse/HIVE-21164


Repository: hive-git


Description
---

Extended the original patch with saving the task attempt ids in the file names 
and also fixed some bugs in the original patch.
With this fix, inserting into an ACID table would not use move task to place 
the generated files into the final directory. It will inserts every files to 
the final directory and then clean up the files which are not needed (like 
written by failed task attempts).
Also fixed the replication tests which failed for the original patch as well.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d3cb60b790 
  
hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
 da677c7977 
  itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
056cd27496 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
 31d15fdef9 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
 c2aa73b5f1 
  itests/src/test/resources/testconfiguration.properties 1b1bf1147a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
9a3258115b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9ad4e71482 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 06e4ebee82 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6c67bc7dd8 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidInputFormat.java bba3960102 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java 1e8bb223f2 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 2f5ec5270c 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 8980a6292a 
  ql/src/java/org/apache/hadoop/hive/ql/io/RecordUpdater.java 737e6774b7 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 76984abd0a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java c4c56f8477 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
b8a0f0465c 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 398698ec06 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
 2543dc6fc4 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1eb9c12cc8 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
73ca658d9c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 33d3beba46 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 
c102a69f8f 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ecc7bdee4d 
  ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java bed05819b5 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
739f2b654b 
  ql/src/java/org/apache/hadoop/hive/ql/util/UpgradeTool.java 58e6289583 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnAddPartition.java c9cb6692df 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java 842140815d 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java e56d83158f 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java 908ceb43fc 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnConcatenate.java 8676e0db11 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnExIm.java 66b2b2768b 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java bb55d9fd79 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java ea6b1d9bec 
  ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java af14e628b3 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 83db48e758 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 
2c4b69b2fe 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
48e9afc496 
  ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/CompactorTest.java 
cfd7290762 
  ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestWorker.java 
70ae85c458 
  ql/src/test/queries/clientpositive/tez_acid_union_dynamic_partition.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/tez_acid_union_dynamic_partition_2.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/tez_acid_union_multiinsert.q PRE-CREATION 
  ql/src/test/results/clientpositive/acid_subquery.q.out 1dc1775557 
  ql/src/test/results/clientpositive/create_transactional_full_acid.q.out 
e324d5ec43 
  
ql/src/test/results/clientpositive/encrypted/encryption_insert_partition_dynamic.q.out
 61b0057adb 
  ql/src/test/results/clientpositive/llap/acid_no_buckets.q.out fbf4e481f1 
  ql/src/test/results/clientpositive/llap/insert_overwrite.q.out fbc3326b39

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

2020-02-14 Thread Marta Kuczora via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71904/
---

(Updated Feb. 14, 2020, 4:10 p.m.)


Review request for hive, Gopal V and Peter Vary.


Changes
---

Addressed the review findings


Bugs: HIVE-21164
https://issues.apache.org/jira/browse/HIVE-21164


Repository: hive-git


Description
---

Extended the original patch with saving the task attempt ids in the file names 
and also fixed some bugs in the original patch.
With this fix, inserting into an ACID table would not use move task to place 
the generated files into the final directory. It will inserts every files to 
the final directory and then clean up the files which are not needed (like 
written by failed task attempts).
Also fixed the replication tests which failed for the original patch as well.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2f695d4acc 
  
hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
 da677c7977 
  itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
056cd27496 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
 31d15fdef9 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
 c2aa73b5f1 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
 4c0137 
  ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
9a3258115b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9ad4e71482 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 06e4ebee82 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6c67bc7dd8 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidInputFormat.java bba3960102 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java 1e8bb223f2 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 2f5ec5270c 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 8980a6292a 
  ql/src/java/org/apache/hadoop/hive/ql/io/RecordUpdater.java 737e6774b7 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 76984abd0a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java c4c56f8477 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
b8a0f0465c 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 398698ec06 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
 2543dc6fc4 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 945eafc034 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
73ca658d9c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 33d3beba46 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 
c102a69f8f 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ecc7bdee4d 
  ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java bed05819b5 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
bb70db4524 
  ql/src/java/org/apache/hadoop/hive/ql/util/UpgradeTool.java 58e6289583 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnAddPartition.java c9cb6692df 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java 842140815d 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 88ca683173 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java 908ceb43fc 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnConcatenate.java 8676e0db11 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnExIm.java 66b2b2768b 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java bb55d9fd79 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java ea6b1d9bec 
  ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java af14e628b3 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 83db48e758 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 
2c4b69b2fe 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
48e9afc496 
  ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/CompactorTest.java 
cfd7290762 
  ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestWorker.java 
70ae85c458 
  ql/src/test/queries/clientpositive/tez_acid_union_dynamic_partition.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/tez_acid_union_dynamic_partition_2.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/tez_acid_union_multiinsert.q PRE-CREATION 
  ql/src/test/results/clientpositive/acid_subquery.q.out 1dc1775557 
  ql/src/test/results/clientpositive/create_transactional_full_acid.q.out 
e324d5ec43 
  
ql/src/test/results/clientpositive/encrypted/encryption_insert_partition_dynamic.q.out
 61b0057adb 
  ql/src/test/results/clientpositive/llap/acid_no_buckets.q.out fbf4e481f1

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

2020-02-14 Thread Marta Kuczora via Review Board



> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
> > Lines 1444 (patched)
> > 
> >
> > Why is this null?

It is null, because if the union all optimization is on, the different union 
statements will be translated into different FileSinkOperators and they will 
write to their own separate directories. They are normally writing to the 
staging directory and under folders with specific 'HIVE_UNION_SUBDIR_' prefix. 
Then the move tasks will move these files to the final table directory. In ACID 
tables these FileSinkOperators would write to different delta directories 
anyway, so the tasks could write directly to the final table location instead 
of the 'HIVE_UNION_SUBDIR_' folders. That's why the unionSuffix is null here. 
In other cases, they have the 'HIVE_UNION_SUBDIR_' value.
Btw, I locally modified many union q tests to run with ACID tables and ran them 
with MR and Tez. I found one bug, which I fixed and I also added some union q 
tests to run with ACID table and direct insert.


> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java
> > Lines 77 (patched)
> > 
> >
> > We created this variable - we should use it? Maybe set it even as a 
> > constant?

You're right. I move this as a constant and changed the tests.


- Marta


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71904/#review219487
---


On Jan. 31, 2020, 4:12 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71904/
> ---
> 
> (Updated Jan. 31, 2020, 4:12 p.m.)
> 
> 
> Review request for hive, Gopal V and Peter Vary.
> 
> 
> Bugs: HIVE-21164
> https://issues.apache.org/jira/browse/HIVE-21164
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Extended the original patch with saving the task attempt ids in the file 
> names and also fixed some bugs in the original patch.
> With this fix, inserting into an ACID table would not use move task to place 
> the generated files into the final directory. It will inserts every files to 
> the final directory and then clean up the files which are not needed (like 
> written by failed task attempts).
> Also fixed the replication tests which failed for the original patch as well.
> 
> 
> Diffs
> -
> 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  da677c7977 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
> 056cd27496 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
>  31d15fdef9 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
>  c2aa73b5f1 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
>  4c0137 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
> 9a3258115b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9ad4e71482 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 06e4ebee82 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6c67bc7dd8 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidInputFormat.java bba3960102 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java 1e8bb223f2 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 2f5ec5270c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 
> 8980a6292a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/RecordUpdater.java 737e6774b7 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 76984abd0a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java 
> c4c56f8477 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
> b8a0f0465c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 
> 398698ec06 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  2543dc6fc4 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 7f061d4a6b 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 73ca658d9c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 5fcc367cc9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 
> c102a69f8f 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ecc7bdee4d 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java bed05819b5 
>

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

2020-02-04 Thread Rajesh Balamohan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71904/#review219494
---




ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
Line 4382 (original), 4397 (patched)


Is this needed for direct insert?. In objectstores, we could have calls 
getting throttled.


- Rajesh Balamohan


On Jan. 31, 2020, 4:12 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71904/
> ---
> 
> (Updated Jan. 31, 2020, 4:12 p.m.)
> 
> 
> Review request for hive, Gopal V and Peter Vary.
> 
> 
> Bugs: HIVE-21164
> https://issues.apache.org/jira/browse/HIVE-21164
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Extended the original patch with saving the task attempt ids in the file 
> names and also fixed some bugs in the original patch.
> With this fix, inserting into an ACID table would not use move task to place 
> the generated files into the final directory. It will inserts every files to 
> the final directory and then clean up the files which are not needed (like 
> written by failed task attempts).
> Also fixed the replication tests which failed for the original patch as well.
> 
> 
> Diffs
> -
> 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  da677c7977 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
> 056cd27496 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
>  31d15fdef9 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
>  c2aa73b5f1 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
>  4c0137 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
> 9a3258115b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9ad4e71482 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 06e4ebee82 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6c67bc7dd8 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidInputFormat.java bba3960102 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java 1e8bb223f2 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 2f5ec5270c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 
> 8980a6292a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/RecordUpdater.java 737e6774b7 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 76984abd0a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java 
> c4c56f8477 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
> b8a0f0465c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 
> 398698ec06 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  2543dc6fc4 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 7f061d4a6b 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 73ca658d9c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 5fcc367cc9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 
> c102a69f8f 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ecc7bdee4d 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java bed05819b5 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> bb70db4524 
>   ql/src/java/org/apache/hadoop/hive/ql/util/UpgradeTool.java 58e6289583 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnAddPartition.java c9cb6692df 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java 842140815d 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 88ca683173 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java 908ceb43fc 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnConcatenate.java 8676e0db11 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnExIm.java 66b2b2768b 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java bb55d9fd79 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java ea6b1d9bec 
>   ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 
> af14e628b3 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 83db48e758 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 
> 2c4b69b2fe 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 48e9afc496 
>   ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/CompactorTest.java 
> cfd7290762 
>   ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestWorker.java 
>

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

2020-02-04 Thread Marta Kuczora via Review Board



> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
> > Lines 1732-1737 (patched)
> > 
> >
> > What about using lambda here?

Fixed it.


> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
> > Lines 7442-7443 (original), 7456-7460 (patched)
> > 
> >
> > nit: Maybe if/else

Fixed it.


> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
> > Lines 7562-7563 (original), 7600-7604 (patched)
> > 
> >
> > nit: Maybe if/else?

Fixed it.


- Marta


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71904/#review219487
---


On Jan. 31, 2020, 4:12 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71904/
> ---
> 
> (Updated Jan. 31, 2020, 4:12 p.m.)
> 
> 
> Review request for hive, Gopal V and Peter Vary.
> 
> 
> Bugs: HIVE-21164
> https://issues.apache.org/jira/browse/HIVE-21164
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Extended the original patch with saving the task attempt ids in the file 
> names and also fixed some bugs in the original patch.
> With this fix, inserting into an ACID table would not use move task to place 
> the generated files into the final directory. It will inserts every files to 
> the final directory and then clean up the files which are not needed (like 
> written by failed task attempts).
> Also fixed the replication tests which failed for the original patch as well.
> 
> 
> Diffs
> -
> 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  da677c7977 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
> 056cd27496 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
>  31d15fdef9 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
>  c2aa73b5f1 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
>  4c0137 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
> 9a3258115b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9ad4e71482 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 06e4ebee82 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6c67bc7dd8 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidInputFormat.java bba3960102 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java 1e8bb223f2 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 2f5ec5270c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 
> 8980a6292a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/RecordUpdater.java 737e6774b7 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 76984abd0a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java 
> c4c56f8477 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
> b8a0f0465c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 
> 398698ec06 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  2543dc6fc4 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 7f061d4a6b 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 73ca658d9c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 5fcc367cc9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 
> c102a69f8f 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ecc7bdee4d 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java bed05819b5 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> bb70db4524 
>   ql/src/java/org/apache/hadoop/hive/ql/util/UpgradeTool.java 58e6289583 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnAddPartition.java c9cb6692df 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java 842140815d 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 88ca683173 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java 908ceb43fc 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnConcatenate.java 8676e0db11 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnExIm.java 66b2b2768b 
>

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

2020-02-04 Thread Marta Kuczora via Review Board



> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
> > Lines 7526-7543 (patched)
> > 
> >
> > Is this duplicated code?

Yeah, however I cannot move this whole part to a separate method, because the 
acidOp and the isDirectInsert variables both have to be set. I can create a 
separate method for getting the value of isDirectInsert and a separate method 
for getting the tmp dir.


- Marta


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71904/#review219487
---


On Jan. 31, 2020, 4:12 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71904/
> ---
> 
> (Updated Jan. 31, 2020, 4:12 p.m.)
> 
> 
> Review request for hive, Gopal V and Peter Vary.
> 
> 
> Bugs: HIVE-21164
> https://issues.apache.org/jira/browse/HIVE-21164
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Extended the original patch with saving the task attempt ids in the file 
> names and also fixed some bugs in the original patch.
> With this fix, inserting into an ACID table would not use move task to place 
> the generated files into the final directory. It will inserts every files to 
> the final directory and then clean up the files which are not needed (like 
> written by failed task attempts).
> Also fixed the replication tests which failed for the original patch as well.
> 
> 
> Diffs
> -
> 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  da677c7977 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
> 056cd27496 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
>  31d15fdef9 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
>  c2aa73b5f1 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
>  4c0137 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
> 9a3258115b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9ad4e71482 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 06e4ebee82 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6c67bc7dd8 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidInputFormat.java bba3960102 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java 1e8bb223f2 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 2f5ec5270c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 
> 8980a6292a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/RecordUpdater.java 737e6774b7 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 76984abd0a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java 
> c4c56f8477 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
> b8a0f0465c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 
> 398698ec06 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  2543dc6fc4 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 7f061d4a6b 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 73ca658d9c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 5fcc367cc9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 
> c102a69f8f 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ecc7bdee4d 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java bed05819b5 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> bb70db4524 
>   ql/src/java/org/apache/hadoop/hive/ql/util/UpgradeTool.java 58e6289583 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnAddPartition.java c9cb6692df 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java 842140815d 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 88ca683173 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java 908ceb43fc 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnConcatenate.java 8676e0db11 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnExIm.java 66b2b2768b 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java bb55d9fd79 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java ea6b1d9bec 
>   ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 
> af14e628b3 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 83db48e758 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 
>

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

2020-02-04 Thread Marta Kuczora via Review Board



> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > Thanks for the patch! This will be very-very usefull.
> > Some minor comments, questions...

Thanks a lot for the review!!


> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
> > Lines 55 (patched)
> > 
> >
> > Is this import used?

You're right, it is not used. Removed it.


> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
> > Lines 843 (patched)
> > 
> >
> > Is inheritPerms still a working stuff? I kinda remember that it was 
> > removed from Hive some time ago...

No, I think this log message was just a copy-paste error. Fixed it.


> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
> > Lines 1799 (patched)
> > 
> >
> > Maybe slightly different log message, so we can easily ditinguish 
> > between this and the line below

Fixed it.


> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
> > Lines 7379 (patched)
> > 
> >
> > We might want to make this feature configurable, to turn it on/off in 
> > case we missed some edge cases

You are absolutely right. I introduced a config parameter so we can turn on/off 
this feature.


> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
> > Lines 493-494 (patched)
> > 
> >
> > nit: Formatting? Really not important, just for the completensess shake 
> > :D

Fixed it.


> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
> > Lines 690-691 (patched)
> > 
> >
> > nit: Formatting?

Fixed it.


> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java
> > Lines 1246 (patched)
> > 
> >
> > Is this table always exists? Shall we use "drop table if exists" 
> > instead?

Fixed it.


- Marta


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71904/#review219487
---


On Jan. 31, 2020, 4:12 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71904/
> ---
> 
> (Updated Jan. 31, 2020, 4:12 p.m.)
> 
> 
> Review request for hive, Gopal V and Peter Vary.
> 
> 
> Bugs: HIVE-21164
> https://issues.apache.org/jira/browse/HIVE-21164
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Extended the original patch with saving the task attempt ids in the file 
> names and also fixed some bugs in the original patch.
> With this fix, inserting into an ACID table would not use move task to place 
> the generated files into the final directory. It will inserts every files to 
> the final directory and then clean up the files which are not needed (like 
> written by failed task attempts).
> Also fixed the replication tests which failed for the original patch as well.
> 
> 
> Diffs
> -
> 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  da677c7977 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
> 056cd27496 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
>  31d15fdef9 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
>  c2aa73b5f1 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
>  4c0137 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
> 9a3258115b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9ad4e71482 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 06e4ebee82 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6c67bc7dd8 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidInputFormat.java bba3960102 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java 1e8bb223f2 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

2020-02-04 Thread Peter Vary via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71904/#review219487
---



Thanks for the patch! This will be very-very usefull.
Some minor comments, questions...


itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
Lines 55 (patched)


Is this import used?



ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
Lines 843 (patched)


Is inheritPerms still a working stuff? I kinda remember that it was removed 
from Hive some time ago...



ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
Lines 1444 (patched)


Why is this null?



ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
Lines 1732-1737 (patched)


What about using lambda here?



ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
Lines 1799 (patched)


Maybe slightly different log message, so we can easily ditinguish between 
this and the line below



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
Lines 7379 (patched)


We might want to make this feature configurable, to turn it on/off in case 
we missed some edge cases



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
Lines 7442-7443 (original), 7456-7460 (patched)


nit: Maybe if/else



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
Lines 7526-7543 (patched)


Is this duplicated code?



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
Lines 7562-7563 (original), 7600-7604 (patched)


nit: Maybe if/else?



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 493-494 (patched)


nit: Formatting? Really not important, just for the completensess shake :D



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 690-691 (patched)


nit: Formatting?



ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java
Lines 77 (patched)


We created this variable - we should use it? Maybe set it even as a 
constant?



ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java
Lines 1246 (patched)


Is this table always exists? Shall we use "drop table if exists" instead?


- Peter Vary


On jan. 31, 2020, 4:12 du, Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71904/
> ---
> 
> (Updated jan. 31, 2020, 4:12 du)
> 
> 
> Review request for hive, Gopal V and Peter Vary.
> 
> 
> Bugs: HIVE-21164
> https://issues.apache.org/jira/browse/HIVE-21164
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Extended the original patch with saving the task attempt ids in the file 
> names and also fixed some bugs in the original patch.
> With this fix, inserting into an ACID table would not use move task to place 
> the generated files into the final directory. It will inserts every files to 
> the final directory and then clean up the files which are not needed (like 
> written by failed task attempts).
> Also fixed the replication tests which failed for the original patch as well.
> 
> 
> Diffs
> -
> 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  da677c7977 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
> 056cd27496 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
>  31d15fdef9 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
>  c2aa73b5f1 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
>  4c0137 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
> 9a3258115b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9ad4e71482 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 06e4ebee82 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6c67bc7dd8 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidInputFormat.java bba3960102 
>

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

2020-01-31 Thread Marta Kuczora via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71904/
---

(Updated Jan. 31, 2020, 4:12 p.m.)


Review request for hive, Gopal V and Peter Vary.


Bugs: HIVE-21164
https://issues.apache.org/jira/browse/HIVE-21164


Repository: hive-git


Description
---

Extended the original patch with saving the task attempt ids in the file names 
and also fixed some bugs in the original patch.
With this fix, inserting into an ACID table would not use move task to place 
the generated files into the final directory. It will inserts every files to 
the final directory and then clean up the files which are not needed (like 
written by failed task attempts).
Also fixed the replication tests which failed for the original patch as well.


Diffs (updated)
-

  
hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
 da677c7977 
  itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
056cd27496 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
 31d15fdef9 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
 c2aa73b5f1 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
 4c0137 
  ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
9a3258115b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9ad4e71482 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 06e4ebee82 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6c67bc7dd8 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidInputFormat.java bba3960102 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java 1e8bb223f2 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 2f5ec5270c 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 8980a6292a 
  ql/src/java/org/apache/hadoop/hive/ql/io/RecordUpdater.java 737e6774b7 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 76984abd0a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java c4c56f8477 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
b8a0f0465c 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 398698ec06 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
 2543dc6fc4 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 7f061d4a6b 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
73ca658d9c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 5fcc367cc9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 
c102a69f8f 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ecc7bdee4d 
  ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java bed05819b5 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
bb70db4524 
  ql/src/java/org/apache/hadoop/hive/ql/util/UpgradeTool.java 58e6289583 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnAddPartition.java c9cb6692df 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java 842140815d 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 88ca683173 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java 908ceb43fc 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnConcatenate.java 8676e0db11 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnExIm.java 66b2b2768b 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java bb55d9fd79 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java ea6b1d9bec 
  ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java af14e628b3 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 83db48e758 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 
2c4b69b2fe 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
48e9afc496 
  ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/CompactorTest.java 
cfd7290762 
  ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestWorker.java 
70ae85c458 
  ql/src/test/results/clientpositive/acid_subquery.q.out 1dc1775557 
  ql/src/test/results/clientpositive/create_transactional_full_acid.q.out 
e324d5ec43 
  
ql/src/test/results/clientpositive/encrypted/encryption_insert_partition_dynamic.q.out
 61b0057adb 
  ql/src/test/results/clientpositive/llap/acid_no_buckets.q.out fbf4e481f1 
  ql/src/test/results/clientpositive/llap/insert_overwrite.q.out fbc3326b39 
  ql/src/test/results/clientpositive/llap/mm_all.q.out 226f2a9374 
  ql/src/test/results/clientpositive/mm_all.q.out 143ebd69f9 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java 35a220facd 


Diff: https://reviews.apache.org/r/71904/diff/3/

Changes: https://reviews.apache.org/r/71904/diff/2-3/


Testing

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

2020-01-31 Thread Marta Kuczora via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71904/
---

(Updated Jan. 31, 2020, 10:17 a.m.)


Review request for hive, Gopal V and Peter Vary.


Changes
---

Rebased the patch


Bugs: HIVE-21164
https://issues.apache.org/jira/browse/HIVE-21164


Repository: hive-git


Description
---

Extended the original patch with saving the task attempt ids in the file names 
and also fixed some bugs in the original patch.
With this fix, inserting into an ACID table would not use move task to place 
the generated files into the final directory. It will inserts every files to 
the final directory and then clean up the files which are not needed (like 
written by failed task attempts).
Also fixed the replication tests which failed for the original patch as well.


Diffs (updated)
-

  
hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
 da677c7977 
  itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
056cd27496 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
 31d15fdef9 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
 4c0137 
  ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
9a3258115b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9ad4e71482 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 06e4ebee82 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6c67bc7dd8 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidInputFormat.java bba3960102 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java 1e8bb223f2 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 2f5ec5270c 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 8980a6292a 
  ql/src/java/org/apache/hadoop/hive/ql/io/RecordUpdater.java 737e6774b7 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 76984abd0a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java c4c56f8477 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
b8a0f0465c 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 398698ec06 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
 2543dc6fc4 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 7f061d4a6b 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
73ca658d9c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 5fcc367cc9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 
c102a69f8f 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ecc7bdee4d 
  ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java bed05819b5 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
bb70db4524 
  ql/src/java/org/apache/hadoop/hive/ql/util/UpgradeTool.java 58e6289583 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnAddPartition.java c9cb6692df 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java 842140815d 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 88ca683173 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java 908ceb43fc 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnConcatenate.java 8676e0db11 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnExIm.java 66b2b2768b 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java bb55d9fd79 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java ea6b1d9bec 
  ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java af14e628b3 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 83db48e758 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 
2c4b69b2fe 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
48e9afc496 
  ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/CompactorTest.java 
cfd7290762 
  ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestWorker.java 
70ae85c458 
  ql/src/test/results/clientpositive/acid_subquery.q.out 1dc1775557 
  ql/src/test/results/clientpositive/create_transactional_full_acid.q.out 
e324d5ec43 
  
ql/src/test/results/clientpositive/encrypted/encryption_insert_partition_dynamic.q.out
 61b0057adb 
  ql/src/test/results/clientpositive/llap/acid_no_buckets.q.out fbf4e481f1 
  ql/src/test/results/clientpositive/llap/insert_overwrite.q.out fbc3326b39 
  ql/src/test/results/clientpositive/llap/mm_all.q.out 226f2a9374 
  ql/src/test/results/clientpositive/mm_all.q.out 143ebd69f9 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java 35a220facd 


Diff: https://reviews.apache.org/r/71904/diff/2/

Changes: https://reviews.apache.org/r/71904/diff/1-2/


Testing
---

Had to modify some tests because of the file name changes. Also added

Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

2019-12-12 Thread Marta Kuczora via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71904/
---

Review request for hive, Gopal V and Peter Vary.


Bugs: HIVE-21164
https://issues.apache.org/jira/browse/HIVE-21164


Repository: hive-git


Description
---

Extended the original patch with saving the task attempt ids in the file names 
and also fixed some bugs in the original patch.
With this fix, inserting into an ACID table would not use move task to place 
the generated files into the final directory. It will inserts every files to 
the final directory and then clean up the files which are not needed (like 
written by failed task attempts).
Also fixed the replication tests which failed for the original patch as well.


Diffs
-

  
hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
 da677c7 
  itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
2868427 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
 31d15fd 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 445e39c 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
 b7245e2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
9a32581 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9ad4e71 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 06e4ebe 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 3d30d09 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidInputFormat.java bba3960 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java 1e8bb22 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 3c508ec 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 8980a62 
  ql/src/java/org/apache/hadoop/hive/ql/io/RecordUpdater.java 737e677 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 76984ab 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java c4c56f8 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 2ac6232 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 3fa61d3 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
 2543dc6 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java f4bd0f9 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 73ca658 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 90549f9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java c102a69 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ecc7bde 
  ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java bed0581 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 2b2cc1a 
  ql/src/java/org/apache/hadoop/hive/ql/util/UpgradeTool.java 58e6289 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnAddPartition.java c9cb669 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java 8421408 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 88ca683 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java 908ceb4 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnConcatenate.java 8676e0d 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnExIm.java 66b2b27 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java bb55d9f 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java ea6b1d9 
  ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java af14e62 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java dd70524 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 2c4b69b 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java c033a94 
  ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/CompactorTest.java 
cfd7290 
  ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestWorker.java 70ae85c 
  ql/src/test/results/clientpositive/acid_subquery.q.out 1dc1775 
  ql/src/test/results/clientpositive/create_transactional_full_acid.q.out 
e324d5e 
  
ql/src/test/results/clientpositive/encrypted/encryption_insert_partition_dynamic.q.out
 61b0057 
  ql/src/test/results/clientpositive/llap/acid_no_buckets.q.out 5571c53 
  ql/src/test/results/clientpositive/llap/insert_overwrite.q.out fbc3326 
  ql/src/test/results/clientpositive/llap/mm_all.q.out 7542a6a 
  ql/src/test/results/clientpositive/mm_all.q.out 1377856 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java 58b3ae2 


Diff: https://reviews.apache.org/r/71904/diff/1/


Testing
---

Had to modify some tests because of the file name changes. Also added some 
specific tests.
In the pre-commit run all tests passed successfully.


Thanks,

Marta Kuczora

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

13 matches

Site Navigation

Mail list logo

Footer information