Re: Review Request 72553: HIVE-23555 Cancel compaction jobs when hive.compactor.worker.timeout is reached

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72553/#review220913
---


Ship it!




Ship It!

- Laszlo Pinter


On May 28, 2020, 8:58 a.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72553/
> ---
> 
> (Updated May 28, 2020, 8:58 a.m.)
> 
> 
> Review request for hive, Karen Coppage and Laszlo Pinter.
> 
> 
> Bugs: HIVE-23555
> https://issues.apache.org/jira/browse/HIVE-23555
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Run the actual execution in a new thread, and use Future.get with timeout
> 
> 
> Diffs
> -
> 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  569de706df 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
>  e70d8783bc 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  32fe535b2b 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUpdaterThread.java 
> ecaad509ed 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 5fa3d9ad42 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java 
> b378d40964 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> fa2ede3738 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MetaStoreCompactorThread.java
>  aa258b331f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/RemoteCompactorThread.java
>  4235184fec 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 8180adcd66 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 366282a30f 
>   ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 
> 3ff68a3c7e 
>   ql/src/test/org/apache/hadoop/hive/ql/stats/TestStatsUpdaterThread.java 
> 84827d1604 
>   ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/CompactorTest.java 
> 9a9ab53fcc 
>   ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestWorker.java 
> 443f982d66 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  e20fdaf03d 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreThread.java
>  ea6155200c 
>   streaming/src/test/org/apache/hive/streaming/TestStreaming.java 6101caac66 
> 
> 
> Diff: https://reviews.apache.org/r/72553/diff/2/
> 
> 
> Testing
> ---
> 
> Created unit tests to check the timeout functionality.
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 71812: HIVE-22534: ACID: Improve Compactor thread logging

[Laszlo Pinter via Review Board]

> On March 27, 2020, 2:23 p.m., Karen Coppage wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java
> > Lines 374-375 (original), 360-361 (patched)
> > 
> >
> > Not regression, but it looks like these arguments are switched up:
> > deltaNumThreshold
> > noBase ? "without" : "with"

Good catch, fixed it.


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71812/#review220095
---


On March 26, 2020, 2:43 p.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71812/
> ---
> 
> (Updated March 26, 2020, 2:43 p.m.)
> 
> 
> Review request for hive and Karen Coppage.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-22534: ACID: Improve Compactor thread logging
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 
> 54b616e60c73fa1005c6d679ea76d65e01a0749d 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 543ec0b99124cb38c8508aa2ec2f99cababdbdbd 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 37a58627913c33be37cee7f11d9ca4ee5fd8aff2 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MinorQueryCompactor.java 
> 1bf0beea4022dbabe65a9d2ee0972186b5cb3396 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMinorQueryCompactor.java
>  383891bfadfc1352430e50eac2d60366ad699bf0 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 
> a96cf1e73183232d587b3f4937c0d7e96764e662 
> 
> 
> Diff: https://reviews.apache.org/r/71812/diff/7/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Review Request 72291: HIVE-23107: Remove MIN_HISTORY_LEVEL table

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72291/
---

Review request for hive and Peter Vary.


Repository: hive-git


Description
---

HIVE-23107: Remove MIN_HISTORY_LEVEL table


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 
54b616e60c73fa1005c6d679ea76d65e01a0749d 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
 bf91ae704c83722502acaf445061bf297fde6a6f 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
 19a95b64db64b7f8bf8e82fbdedf5b54cd30aed3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java
 ef88240a791cbc553b6150b54701c7df8daf3b49 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 74ef88545e6fd680b311ca5c5e4d87f053af1026 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java
 41d2e7924b1286f82d212e051714107505fe9661 
  
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
 48ad67623143fcba2bb2507cc36ad46a58c13b75 
  
standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
 7a230bde3a4dcd2504538bf8122db0bb2a59932f 
  
standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
 9ed7f4f8192e1613f8942af3e9496c3ef7f1f04f 
  
standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
 12d24e9a569310c57d86b1229deaa2b1d080e0b8 
  
standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
 bc34b511a978af2a6704e47d8f73c9604c6cebf9 
  
standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
 13f03bce6d579c18ff9a14e09133063fdcdbf7af 
  
standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
 8482b5942c6284eda63ec7d2aa7c1202abb3db49 
  
standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
 cbfdd861fd545ce786f5e267b284196f5fd9af03 
  
standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
 aa35a7a7b392d4d46480c03ed95ffdec12b03b22 
  
standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
 9462328a5f0c6dda8cffb683860153b8eb3aacec 


Diff: https://reviews.apache.org/r/72291/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 71812: HIVE-22534: ACID: Improve Compactor thread logging

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71812/
---

(Updated March 26, 2020, 2:43 p.m.)


Review request for hive and Karen Coppage.


Repository: hive-git


Description (updated)
---

HIVE-22534: Improve Compactor threads logging


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 
54b616e60c73fa1005c6d679ea76d65e01a0749d 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
543ec0b99124cb38c8508aa2ec2f99cababdbdbd 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
37a58627913c33be37cee7f11d9ca4ee5fd8aff2 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MinorQueryCompactor.java 
1bf0beea4022dbabe65a9d2ee0972186b5cb3396 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMinorQueryCompactor.java 
383891bfadfc1352430e50eac2d60366ad699bf0 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 
a96cf1e73183232d587b3f4937c0d7e96764e662 


Diff: https://reviews.apache.org/r/71812/diff/6/

Changes: https://reviews.apache.org/r/71812/diff/5-6/


Testing
---


Thanks,

Laszlo Pinter

Review Request 72193: HIVE-22977: Merge delta files instead of running a query in major/minor compaction

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72193/
---

Review request for hive, Karen Coppage and Peter Vary.


Repository: hive-git


Description
---

HIVE-22977: Merge delta files instead of running a query in major/minor 
compaction


Diffs
-

  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorOnTezTest.java
 78174f345b35709cd654aa81578ab598e0d9ed9c 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
 9659a3f0481dcb2446b197688459f0c1dba867fa 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestMmCompactorOnTez.java
 074430ce7fa0f0617e8fb50c334c14f33cc74d8a 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
c44f2b5026558c9f0a7d6fa03cb6950f24b77da2 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MajorQueryCompactor.java 
93850807137a4cfbd49beb256624b11801bd08d1 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MinorQueryCompactor.java 
01cd2fc93d12002249253added06df70b0c40181 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMajorQueryCompactor.java 
41fdd7e210bfc42c3e41e9f1240d34a51add33a9 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMinorQueryCompactor.java 
feb667cba960c0fdd19c030235eb31ebddfa7ca1 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/OrcFileMerger.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactor.java 
7a9e48ff1eb88f65cc41165b3908f3dee9245f61 


Diff: https://reviews.apache.org/r/72193/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 72043: HIVE-21487: COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes

[Laszlo Pinter via Review Board]

> On Jan. 31, 2020, 9:52 a.m., Peter Vary wrote:
> > Have you tested the sqls on every db?
> > If they run correctly on all supported DB, then +1 from my side
> > 
> > Thanks,
> > Peter

Run tests on derby, mssql, mysql, oracle and postgres. All passed.


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72043/#review219380
---


On Jan. 24, 2020, 10 a.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72043/
> ---
> 
> (Updated Jan. 24, 2020, 10 a.m.)
> 
> 
> Review request for hive, Denys Kuzmenko, Karen Coppage, and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-21487: COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing 
> appropriate indexes
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java
>  7f1c2f8bdcc5a43a816e4cf7461e28e8eacb8511 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
>  482d8aa1630ab854754892f9181cecc3ca12568d 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
>  6fd8ae64cb26ab10ba271afa3ec02d54b0f68a97 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
>  dfd4bd6672459e1431273cd24464a1aa5732803e 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
>  fecfca8c13af42ed871ef632ef7d11cea15ff613 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
>  a9a09307ee53a44ea583990b772dff406cd1d96a 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
>  70b5d9d41ffd91d4729b3518d4bfdb4bf14c3165 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
>  f90d76b4e0c070432a0ac6fe0261f5d72155bb5b 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
>  9f1b98037234b9cbb214181d5ea4987b3a476667 
>   
> standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
>  b992d7a24904d6fbe47e69256916aa69eaddea02 
>   
> standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
>  f9af248e279f519aeb0d3a6d46a678dbd8f431c1 
> 
> 
> Diff: https://reviews.apache.org/r/72043/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Re: Review Request 72145: HIVE-21543: Use FilterHooks for show compactions

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72145/
---

(Updated Feb. 17, 2020, 3:30 p.m.)


Review request for hive, Karen Coppage and Peter Vary.


Changes
---

Fix code review findings.


Repository: hive-git


Description
---

HIVE-21543: Use FilterHooks for show compactions


Diffs (updated)
-

  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 c7e1044589fd1dd970b86259b713d3a44716e7b8 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/FilterUtils.java
 d9da00dd2148b0408548ab7d5c88df014a7f7826 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 b6de1460a565f06217f163ca5f733594a0c8406a 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestFilterHooks.java
 23faa7444a8e9f9c010290539f89b9d8b44f3aa8 


Diff: https://reviews.apache.org/r/72145/diff/2/

Changes: https://reviews.apache.org/r/72145/diff/1-2/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 72109: HIVE-20948: Eliminate file rename in compactor

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72109/
---

(Updated Feb. 17, 2020, 12:53 p.m.)


Review request for hive, Karen Coppage, Marta Kuczora, and Peter Vary.


Changes
---

Fix code review findings


Repository: hive-git


Description
---

HIVE-20948: Eliminate file rename in compactor


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 
9ad4e7148226b91b0c759de54e251893d61725a3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SplitGrouper.java 
076b77877ae748b757a4c9c08532a3ce029fed38 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 
2f5ec5270c0bc7d1a591c9c8c15b1ecb7f9f6ace 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java 
ecc7bdee4dacc03cf59ac5be4bed92a75f8e720b 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
bb70db452402dd690e2136a122e9b3bd11fa7522 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MajorQueryCompactor.java 
f238eb5dd058fc79c5b7ad3b08920c774b1a7f8c 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MinorQueryCompactor.java 
f96a0481b870b04cc97621cd62a43b07ecd5d7fd 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactor.java 
d2349104902c2af3d6020c9599fd3fa20f9a64a5 


Diff: https://reviews.apache.org/r/72109/diff/3/

Changes: https://reviews.apache.org/r/72109/diff/2-3/


Testing
---


Thanks,

Laszlo Pinter

Review Request 72145: HIVE-21543: Use FilterHooks for show compactions

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72145/
---

Review request for hive, Karen Coppage and Peter Vary.


Repository: hive-git


Description
---

HIVE-21543: Use FilterHooks for show compactions


Diffs
-

  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 c7e1044589fd1dd970b86259b713d3a44716e7b8 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/FilterUtils.java
 d9da00dd2148b0408548ab7d5c88df014a7f7826 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 7b7c2d77914ad56d3ae568aa050725b12969895a 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestFilterHooks.java
 23faa7444a8e9f9c010290539f89b9d8b44f3aa8 


Diff: https://reviews.apache.org/r/72145/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 72105: HIVE-22610: Minor compaction for MM (insert-only) tables

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72105/#review219570
---


Ship it!




Ship It!

- Laszlo Pinter


On Feb. 12, 2020, 3:23 p.m., Karen Coppage wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72105/
> ---
> 
> (Updated Feb. 12, 2020, 3:23 p.m.)
> 
> 
> Review request for hive and Laszlo Pinter.
> 
> 
> Bugs: HIVE-22610
> https://issues.apache.org/jira/browse/HIVE-22610
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Minor compaction for MM (insert-only) tables
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorOnTezTest.java
>  PRE-CREATION 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
>  4c01311 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestMmCompactorOnTez.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> bb70db4 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMajorQueryCompactor.java
>  bad5d00 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMinorQueryCompactor.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmQueryCompactorUtils.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactorFactory.java
>  2f2bb21 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 88ca683 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java 
> aabf15c 
> 
> 
> Diff: https://reviews.apache.org/r/72105/diff/2/
> 
> 
> Testing
> ---
> 
> The 3 unit tests removed here are really 2 tests (one is a duplicate). 
> Insert-only minor compaction used to do nothing but remove aborted 
> directories; these 2 tests were written for this functionality specifically. 
> The unit tests added here cover that functionality as well as compaction, and 
> rendered the old unit tests obsolete.
> 
> 
> Thanks,
> 
> Karen Coppage
> 
>

Re: Review Request 72105: HIVE-22610: Minor compaction for MM (insert-only) tables

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72105/#review219569
---


Ship it!




Ship It!

- Laszlo Pinter


On Feb. 12, 2020, 3:23 p.m., Karen Coppage wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72105/
> ---
> 
> (Updated Feb. 12, 2020, 3:23 p.m.)
> 
> 
> Review request for hive and Laszlo Pinter.
> 
> 
> Bugs: HIVE-22610
> https://issues.apache.org/jira/browse/HIVE-22610
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Minor compaction for MM (insert-only) tables
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorOnTezTest.java
>  PRE-CREATION 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
>  4c01311 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestMmCompactorOnTez.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> bb70db4 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMajorQueryCompactor.java
>  bad5d00 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMinorQueryCompactor.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmQueryCompactorUtils.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactorFactory.java
>  2f2bb21 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 88ca683 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java 
> aabf15c 
> 
> 
> Diff: https://reviews.apache.org/r/72105/diff/2/
> 
> 
> Testing
> ---
> 
> The 3 unit tests removed here are really 2 tests (one is a duplicate). 
> Insert-only minor compaction used to do nothing but remove aborted 
> directories; these 2 tests were written for this functionality specifically. 
> The unit tests added here cover that functionality as well as compaction, and 
> rendered the old unit tests obsolete.
> 
> 
> Thanks,
> 
> Karen Coppage
> 
>

Review Request 72109: HIVE-20948: Eliminate file rename in compactor

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72109/
---

Review request for hive, Karen Coppage, Marta Kuczora, and Peter Vary.


Repository: hive-git


Description
---

HIVE-20948: Eliminate file rename in compactor


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 
9ad4e7148226b91b0c759de54e251893d61725a3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SplitGrouper.java 
076b77877ae748b757a4c9c08532a3ce029fed38 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 
2f5ec5270c0bc7d1a591c9c8c15b1ecb7f9f6ace 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java 
ecc7bdee4dacc03cf59ac5be4bed92a75f8e720b 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
bb70db452402dd690e2136a122e9b3bd11fa7522 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MajorQueryCompactor.java 
f238eb5dd058fc79c5b7ad3b08920c774b1a7f8c 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MinorQueryCompactor.java 
f96a0481b870b04cc97621cd62a43b07ecd5d7fd 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactor.java 
d2349104902c2af3d6020c9599fd3fa20f9a64a5 


Diff: https://reviews.apache.org/r/72109/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 72028: HIVE-22729: Provide a failure reason for failed compactions

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72028/
---

(Updated Jan. 30, 2020, 10:23 a.m.)


Review request for hive, Denys Kuzmenko, Karen Coppage, and Peter Vary.


Changes
---

Adding error message column to hive schema.


Repository: hive-git


Description
---

HIVE-22729: Provide a failure reason for failed compactions


Diffs (updated)
-

  metastore/scripts/upgrade/hive/hive-schema-4.0.0.hive.sql 
5421d4d8141becae4e0de6a039bf7c46f0b109bb 
  metastore/scripts/upgrade/hive/upgrade-3.1.0-to-4.0.0.hive.sql 
041190653898a39ef96c6c2bf71c4f4485f6a1a5 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsDesc.java
 9348efc5a12b50f55f5952094882e941158405fd 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsOperation.java
 517d88237cc3b8f0316727bf1eebfc6535152fae 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 
6f642901203ab73699ed694009d48ca77263fb10 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
dedc990d0f1e9123497f0fb7c7b9945c7b29bde2 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 
5aff71e0e981c429f85663300d3e5c21089529a9 
  
ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java 
e5895547e6006f30a37b5ba0b1ce42253129d3b6 
  ql/src/test/results/clientpositive/dbtxnmgr_showlocks.q.out 
03c6724ec2e50ae1f7c642339c1806d0786a9ec5 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java
 4aee45ce5f0e534823194bc84d13b88210ce0b3c 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowCompactResponseElement.java
 8a5682a013b24f8dcf7ad3fdb0b0b606d82cc7c0 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 e8556dcea68f34336df2925c4108e71185d6377f 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 b05e61e84a310273911ef592258bcd3b34e87734 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 868cf69200f69aa89e82b34e22ee0ad792e6d025 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
61a94fee4d82a714c12aeeb27f31e24774592c98 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
 ba45f3945274853fdc84487d93c4c00ff2982541 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
 aded6f5486cc840f397347b39049310009fd3bad 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java
 da5dd61d08e2ca8fe5e80ffdf9fb4a6f4c4d0ba3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 c2c97d96c6cc98f9746069fa725d17d12f6c8642 
  
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
 67102718867233f29ddb2ea8ec3fbcb6560c6c30 
  
standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
 ae0a32541a4bb9179b2bb71ae9f9098d7b35a88e 
  
standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
 221d4f1fffb682aaec3af22a339e7a3077a75f6a 
  
standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
 bc98d5fc4a5637988c97f2e5a0e02d3be16ae0cb 
  
standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
 dd761a66db4826580a67d64879e4c85278b8e20c 
  
standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
 6a040a6a64c2086b5eb68a397697c9e2d2ca4d76 
  
standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
 f5ec1ba1aff89d02b66d6a2cd1da8de1b3b08d06 
  
standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
 c7738be2732b839aa2b460733c092e368909f935 
  
standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
 455f98b72578ff977e29301cd2fc595ae80ee4ca 
  
standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
 5c39b0d9f4d27ab82ef44392818c1810cb7664ce 


Diff: https://reviews.apache.org/r/72028/diff/3/

Changes: https://reviews.apache.org/r/72028/diff/2-3/


Testing
---


Thanks,

Laszlo Pinter

Review Request 72059: HIVE-22793: Update default settings in HMS Benchmarking tool

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72059/
---

Review request for hive, Denys Kuzmenko, Karen Coppage, and Peter Vary.


Repository: hive-git


Description
---

HIVE-22793: Update default settings in HMS Benchmarking tool


Diffs
-

  standalone-metastore/metastore-tools/metastore-benchmarks/README.md 
a8c0a41f559261275aef48eb815c1b14f6cfdaed 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Constants.java
 5a584f6adeccb5213916f23244277609ae8373cc 


Diff: https://reviews.apache.org/r/72059/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 72028: HIVE-22729: Provide a failure reason for failed compactions

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72028/
---

(Updated Jan. 29, 2020, 9:20 a.m.)


Review request for hive, Denys Kuzmenko, Karen Coppage, and Peter Vary.


Repository: hive-git


Description
---

HIVE-22729: Provide a failure reason for failed compactions


Diffs (updated)
-

  
ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsDesc.java
 9348efc5a12b50f55f5952094882e941158405fd 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsOperation.java
 517d88237cc3b8f0316727bf1eebfc6535152fae 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 
6f642901203ab73699ed694009d48ca77263fb10 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
dedc990d0f1e9123497f0fb7c7b9945c7b29bde2 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 
5aff71e0e981c429f85663300d3e5c21089529a9 
  
ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java 
e5895547e6006f30a37b5ba0b1ce42253129d3b6 
  ql/src/test/results/clientpositive/dbtxnmgr_showlocks.q.out 
03c6724ec2e50ae1f7c642339c1806d0786a9ec5 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java
 4aee45ce5f0e534823194bc84d13b88210ce0b3c 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowCompactResponseElement.java
 8a5682a013b24f8dcf7ad3fdb0b0b606d82cc7c0 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 e8556dcea68f34336df2925c4108e71185d6377f 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 b05e61e84a310273911ef592258bcd3b34e87734 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 868cf69200f69aa89e82b34e22ee0ad792e6d025 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
61a94fee4d82a714c12aeeb27f31e24774592c98 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
 ba45f3945274853fdc84487d93c4c00ff2982541 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
 aded6f5486cc840f397347b39049310009fd3bad 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java
 da5dd61d08e2ca8fe5e80ffdf9fb4a6f4c4d0ba3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 c2c97d96c6cc98f9746069fa725d17d12f6c8642 
  
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
 67102718867233f29ddb2ea8ec3fbcb6560c6c30 
  
standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
 ae0a32541a4bb9179b2bb71ae9f9098d7b35a88e 
  
standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
 221d4f1fffb682aaec3af22a339e7a3077a75f6a 
  
standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
 bc98d5fc4a5637988c97f2e5a0e02d3be16ae0cb 
  
standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
 dd761a66db4826580a67d64879e4c85278b8e20c 
  
standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
 6a040a6a64c2086b5eb68a397697c9e2d2ca4d76 
  
standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
 f5ec1ba1aff89d02b66d6a2cd1da8de1b3b08d06 
  
standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
 c7738be2732b839aa2b460733c092e368909f935 
  
standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
 455f98b72578ff977e29301cd2fc595ae80ee4ca 
  
standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
 5c39b0d9f4d27ab82ef44392818c1810cb7664ce 


Diff: https://reviews.apache.org/r/72028/diff/2/

Changes: https://reviews.apache.org/r/72028/diff/1-2/


Testing
---


Thanks,

Laszlo Pinter

Review Request 72043: HIVE-21487: COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72043/
---

Review request for hive, Denys Kuzmenko, Karen Coppage, and Peter Vary.


Repository: hive-git


Description
---

HIVE-21487: COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing 
appropriate indexes


Diffs
-

  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java
 da5dd61d08e2ca8fe5e80ffdf9fb4a6f4c4d0ba3 
  
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
 67102718867233f29ddb2ea8ec3fbcb6560c6c30 
  
standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
 ae0a32541a4bb9179b2bb71ae9f9098d7b35a88e 
  
standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
 221d4f1fffb682aaec3af22a339e7a3077a75f6a 
  
standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
 bc98d5fc4a5637988c97f2e5a0e02d3be16ae0cb 
  
standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
 dd761a66db4826580a67d64879e4c85278b8e20c 
  
standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
 6a040a6a64c2086b5eb68a397697c9e2d2ca4d76 
  
standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
 f5ec1ba1aff89d02b66d6a2cd1da8de1b3b08d06 
  
standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
 c7738be2732b839aa2b460733c092e368909f935 
  
standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
 455f98b72578ff977e29301cd2fc595ae80ee4ca 
  
standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
 5c39b0d9f4d27ab82ef44392818c1810cb7664ce 


Diff: https://reviews.apache.org/r/72043/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Review Request 72028: HIVE-22729: Provide a failure reason for failed compactions

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72028/
---

Review request for hive, Denys Kuzmenko, Karen Coppage, and Peter Vary.


Repository: hive-git


Description
---

HIVE-22729: Provide a failure reason for failed compactions


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsDesc.java
 9348efc5a12b50f55f5952094882e941158405fd 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsOperation.java
 517d88237cc3b8f0316727bf1eebfc6535152fae 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 
6f642901203ab73699ed694009d48ca77263fb10 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
6017fd31b1c180c144c0d37dddc56c44d9d8a05b 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 
5aff71e0e981c429f85663300d3e5c21089529a9 
  
ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java 
e5895547e6006f30a37b5ba0b1ce42253129d3b6 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java
 4aee45ce5f0e534823194bc84d13b88210ce0b3c 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowCompactResponseElement.java
 8a5682a013b24f8dcf7ad3fdb0b0b606d82cc7c0 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 093ad4be273f3544e013c608ba3ca0a66cfaea20 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 0dcca59b6812a892055d5d0615c1c479e8d8d030 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 8d7c32a7658535b2a55695813f5e98a95f79f0a4 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
72ccdd1a0f62ed1c034b17dfbd84a8ab3f49befd 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
 ba45f3945274853fdc84487d93c4c00ff2982541 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
 aded6f5486cc840f397347b39049310009fd3bad 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java
 da5dd61d08e2ca8fe5e80ffdf9fb4a6f4c4d0ba3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 1dc3867929fcdfef53da4859fbe8de2e89435166 
  
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
 67102718867233f29ddb2ea8ec3fbcb6560c6c30 
  
standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
 ae0a32541a4bb9179b2bb71ae9f9098d7b35a88e 
  
standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
 221d4f1fffb682aaec3af22a339e7a3077a75f6a 
  
standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
 bc98d5fc4a5637988c97f2e5a0e02d3be16ae0cb 
  
standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
 dd761a66db4826580a67d64879e4c85278b8e20c 
  
standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
 6a040a6a64c2086b5eb68a397697c9e2d2ca4d76 
  
standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
 f5ec1ba1aff89d02b66d6a2cd1da8de1b3b08d06 
  
standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
 c7738be2732b839aa2b460733c092e368909f935 
  
standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
 455f98b72578ff977e29301cd2fc595ae80ee4ca 
  
standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
 5c39b0d9f4d27ab82ef44392818c1810cb7664ce 


Diff: https://reviews.apache.org/r/72028/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 71988: HIVE-22703: Compaction configuration check when starting HMS/HS2

[Laszlo Pinter via Review Board]

> On Jan. 13, 2020, 2:37 p.m., Denys Kuzmenko wrote:
> > service/src/java/org/apache/hive/service/server/HiveServer2.java
> > Lines 437 (patched)
> > 
> >
> > do we have constant for "hs2" literal?

For "hs2" no, just for "hiveserver2".


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71988/#review219237
---


On Jan. 13, 2020, 2:12 p.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71988/
> ---
> 
> (Updated Jan. 13, 2020, 2:12 p.m.)
> 
> 
> Review request for hive, Denys Kuzmenko, Karen Coppage, and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-22703: Compaction configuration check when starting HMS/HS2
> 
> 
> Diffs
> -
> 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 
> a93cc1b7e1aadf6e2724d667b6e4c9c9ecc38a75 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  94698e6771890a050b00b374ca0ee926f768aa0e 
> 
> 
> Diff: https://reviews.apache.org/r/71988/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Re: Review Request 71888: HIVE-22568: Process compaction candidates in parallel by the Initiator

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71888/#review219239
---


Ship it!




+1 (non-binding)

- Laszlo Pinter


On Dec. 6, 2019, 12:54 p.m., Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71888/
> ---
> 
> (Updated Dec. 6, 2019, 12:54 p.m.)
> 
> 
> Review request for hive, Laszlo Pinter and Peter Vary.
> 
> 
> Bugs: HIVE-22568
> https://issues.apache.org/jira/browse/HIVE-22568
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> `checkForCompaction` includes many file metadata checks and may be expensive. 
> Therefore, make sense using a thread pool here and running 
> `checkForCompactions` in parallel.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 4393a2825e 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 7a0e32463d 
>   ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator.java 
> 564839324f 
> 
> 
> Diff: https://reviews.apache.org/r/71888/diff/1/
> 
> 
> Testing
> ---
> 
> unit test
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Review Request 71988: HIVE-22703: Compaction configuration check when starting HMS/HS2

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71988/
---

Review request for hive, Denys Kuzmenko, Karen Coppage, and Peter Vary.


Repository: hive-git


Description
---

HIVE-22703: Compaction configuration check when starting HMS/HS2


Diffs
-

  service/src/java/org/apache/hive/service/server/HiveServer2.java 
a93cc1b7e1aadf6e2724d667b6e4c9c9ecc38a75 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 94698e6771890a050b00b374ca0ee926f768aa0e 


Diff: https://reviews.apache.org/r/71988/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 71949: HIVE-20934: ACID: Query based compactor for minor compaction

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71949/
---

(Updated Jan. 8, 2020, 10:40 a.m.)


Review request for hive, Denys Kuzmenko, Karen Coppage, and Peter Vary.


Changes
---

Fixed code review findings.


Repository: hive-git


Description
---

HIVE-20934: ACID: Query based compactor for minor compaction


Diffs (updated)
-

  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
 PRE-CREATION 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 445e39c260edc68f511550271a7ac471fae908fe 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
 b7245e2c3570b362a00b65b23f3f84616d0a3d1e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SplitGrouper.java 
33d723a02e28d69a69b88281038f69b5aecfe6a2 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 
dde8878769e10191c5ba61bd1ba44d9b16b172c1 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
2ac6232460fedb8351b5f0cfae2ce2d0f2e2d948 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 
0a96fc30b359043293017b235a36cd044ddb176e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
ad6817c32bbfad1d27023b25912b1204f069a66a 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
2b2cc1a2ba8377aa3681b1a3454a0d64369eef64 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
2253fda6c6ed5e2f70ef7c1166895eb49f600ea9 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MajorQueryCompactor.java 
38689ef86c607a36f8ec961a88578c13bfcd5b01 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MinorQueryCompactor.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMajorQueryCompactor.java 
9b8420902fb688b218fa432d70f71302f9f180e6 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactor.java 
1eab5b888deef2d0fb5c097941a1dafa51c7d46b 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactorFactory.java 
41cb4b64fbc79dcf81919769c567b26a2e18cfe5 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java 
d4c9121c9f17f8d083f1e1af1caf840678a3559d 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForOrcMmTable.java 
d6435342aa1f56ba5495a657b4a43327fdc49645 


Diff: https://reviews.apache.org/r/71949/diff/3/

Changes: https://reviews.apache.org/r/71949/diff/2-3/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 71949: HIVE-20934: ACID: Query based compactor for minor compaction

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71949/#review219172
---




ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SplitGrouper.java
Lines 247 (patched)


I didn't want to make this string globally avalaible for other components 
as well.


- Laszlo Pinter


On Jan. 7, 2020, 2:24 p.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71949/
> ---
> 
> (Updated Jan. 7, 2020, 2:24 p.m.)
> 
> 
> Review request for hive, Denys Kuzmenko, Karen Coppage, and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20934: ACID: Query based compactor for minor compaction
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
>  PRE-CREATION 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  445e39c260edc68f511550271a7ac471fae908fe 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
>  b7245e2c3570b362a00b65b23f3f84616d0a3d1e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SplitGrouper.java 
> 33d723a02e28d69a69b88281038f69b5aecfe6a2 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 
> 3c508ec6cf620aee6a7791c6ab52c331ad5ec6bd 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
> 2ac6232460fedb8351b5f0cfae2ce2d0f2e2d948 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 
> 0a96fc30b359043293017b235a36cd044ddb176e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> ad6817c32bbfad1d27023b25912b1204f069a66a 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 2b2cc1a2ba8377aa3681b1a3454a0d64369eef64 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 7a0e32463d28007cff5526ae037cc1447e50a50b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MajorQueryCompactor.java 
> 38689ef86c607a36f8ec961a88578c13bfcd5b01 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MinorQueryCompactor.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMajorQueryCompactor.java
>  9b8420902fb688b218fa432d70f71302f9f180e6 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactor.java 
> 1eab5b888deef2d0fb5c097941a1dafa51c7d46b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactorFactory.java
>  41cb4b64fbc79dcf81919769c567b26a2e18cfe5 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java 
> d4c9121c9f17f8d083f1e1af1caf840678a3559d 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForOrcMmTable.java 
> d6435342aa1f56ba5495a657b4a43327fdc49645 
> 
> 
> Diff: https://reviews.apache.org/r/71949/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Review Request 71963: HIVE-22700: Compactions may leak memory when unauthorized

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71963/
---

Review request for hive, Denys Kuzmenko, Karen Coppage, and Peter Vary.


Repository: hive-git


Description
---

HIVE-22700: Compactions may leak memory when unauthorized


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
7a0e32463d28007cff5526ae037cc1447e50a50b 


Diff: https://reviews.apache.org/r/71963/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 71949: HIVE-20934: ACID: Query based compactor for minor compaction

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71949/
---

(Updated Jan. 7, 2020, 2:24 p.m.)


Review request for hive, Denys Kuzmenko, Karen Coppage, and Peter Vary.


Changes
---

Fixed code review findings.


Repository: hive-git


Description
---

HIVE-20934: ACID: Query based compactor for minor compaction


Diffs (updated)
-

  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
 PRE-CREATION 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 445e39c260edc68f511550271a7ac471fae908fe 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
 b7245e2c3570b362a00b65b23f3f84616d0a3d1e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SplitGrouper.java 
33d723a02e28d69a69b88281038f69b5aecfe6a2 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 
3c508ec6cf620aee6a7791c6ab52c331ad5ec6bd 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
2ac6232460fedb8351b5f0cfae2ce2d0f2e2d948 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 
0a96fc30b359043293017b235a36cd044ddb176e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
ad6817c32bbfad1d27023b25912b1204f069a66a 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
2b2cc1a2ba8377aa3681b1a3454a0d64369eef64 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
7a0e32463d28007cff5526ae037cc1447e50a50b 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MajorQueryCompactor.java 
38689ef86c607a36f8ec961a88578c13bfcd5b01 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MinorQueryCompactor.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMajorQueryCompactor.java 
9b8420902fb688b218fa432d70f71302f9f180e6 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactor.java 
1eab5b888deef2d0fb5c097941a1dafa51c7d46b 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactorFactory.java 
41cb4b64fbc79dcf81919769c567b26a2e18cfe5 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java 
d4c9121c9f17f8d083f1e1af1caf840678a3559d 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForOrcMmTable.java 
d6435342aa1f56ba5495a657b4a43327fdc49645 


Diff: https://reviews.apache.org/r/71949/diff/2/

Changes: https://reviews.apache.org/r/71949/diff/1-2/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 71949: HIVE-20934: ACID: Query based compactor for minor compaction

[Laszlo Pinter via Review Board]

> On Jan. 7, 2020, 10:21 a.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SplitGrouper.java
> > Lines 282 (patched)
> > 
> >
> > I do not see this in the original code. What is this for?

I made a bit of simplification here. The OrcSplit.parse() call does some 
metadata preparation steps, like fetching the writeID, transactionID and 
bucketID and setting it to the split. Originally, this was done in 
ComparatorCompactor, but my point of view is that the comparator should be used 
only for comparing OrcSplits and not to prepare data. Since, we are already 
iterating over the splits, it made sense to move the call here.


> On Jan. 7, 2020, 10:21 a.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java
> > Lines 1228 (patched)
> > 
> >
> > Do we need this? With delete_delta we do not supposed to have 
> > syntehetic rowIDs...

That is correct, but I had to introduce this condition as well, to avoid 
IllegalStateException. Without this check, the splits in a delete delta 
directory would be considered as original files but without the correct parent.


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71949/#review219140
---


On Jan. 4, 2020, 9:06 a.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71949/
> ---
> 
> (Updated Jan. 4, 2020, 9:06 a.m.)
> 
> 
> Review request for hive, Denys Kuzmenko, Karen Coppage, and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20934: ACID: Query based compactor for minor compaction
> 
> 
> Diffs
> -
> 
>   itests/hive-unit/pom.xml bc20cd6168dd61222c75fb866deada26328986dd 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
>  PRE-CREATION 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  445e39c260edc68f511550271a7ac471fae908fe 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
>  b7245e2c3570b362a00b65b23f3f84616d0a3d1e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SplitGrouper.java 
> 33d723a02e28d69a69b88281038f69b5aecfe6a2 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 
> 3c508ec6cf620aee6a7791c6ab52c331ad5ec6bd 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
> 2ac6232460fedb8351b5f0cfae2ce2d0f2e2d948 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 
> 0a96fc30b359043293017b235a36cd044ddb176e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> 20b0ccd94b5f08aa2c1dace1301a8315bd202bf7 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 2b2cc1a2ba8377aa3681b1a3454a0d64369eef64 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 7a0e32463d28007cff5526ae037cc1447e50a50b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MajorQueryCompactor.java 
> 38689ef86c607a36f8ec961a88578c13bfcd5b01 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MinorQueryCompactor.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMajorQueryCompactor.java
>  9b8420902fb688b218fa432d70f71302f9f180e6 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactor.java 
> 1eab5b888deef2d0fb5c097941a1dafa51c7d46b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactorFactory.java
>  41cb4b64fbc79dcf81919769c567b26a2e18cfe5 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java 
> d4c9121c9f17f8d083f1e1af1caf840678a3559d 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForOrcMmTable.java 
> d6435342aa1f56ba5495a657b4a43327fdc49645 
> 
> 
> Diff: https://reviews.apache.org/r/71949/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Review Request 71949: HIVE-20934: ACID: Query based compactor for minor compaction

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71949/
---

Review request for hive, Denys Kuzmenko, Karen Coppage, and Peter Vary.


Repository: hive-git


Description
---

HIVE-20934: ACID: Query based compactor for minor compaction


Diffs
-

  itests/hive-unit/pom.xml bc20cd6168dd61222c75fb866deada26328986dd 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
 PRE-CREATION 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 445e39c260edc68f511550271a7ac471fae908fe 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
 b7245e2c3570b362a00b65b23f3f84616d0a3d1e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SplitGrouper.java 
33d723a02e28d69a69b88281038f69b5aecfe6a2 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 
3c508ec6cf620aee6a7791c6ab52c331ad5ec6bd 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
2ac6232460fedb8351b5f0cfae2ce2d0f2e2d948 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 
0a96fc30b359043293017b235a36cd044ddb176e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
20b0ccd94b5f08aa2c1dace1301a8315bd202bf7 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
2b2cc1a2ba8377aa3681b1a3454a0d64369eef64 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
7a0e32463d28007cff5526ae037cc1447e50a50b 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MajorQueryCompactor.java 
38689ef86c607a36f8ec961a88578c13bfcd5b01 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MinorQueryCompactor.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMajorQueryCompactor.java 
9b8420902fb688b218fa432d70f71302f9f180e6 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactor.java 
1eab5b888deef2d0fb5c097941a1dafa51c7d46b 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactorFactory.java 
41cb4b64fbc79dcf81919769c567b26a2e18cfe5 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java 
d4c9121c9f17f8d083f1e1af1caf840678a3559d 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForOrcMmTable.java 
d6435342aa1f56ba5495a657b4a43327fdc49645 


Diff: https://reviews.apache.org/r/71949/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Review Request 71948: HIVE-22535: ACID: Check for already running compactions in Worker

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71948/
---

Review request for hive.


Repository: hive-git


Description
---

HIVE-22535: ACID: Check for already running compactions in Worker


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java 
99da86f9102da4df8b3531a3f4dc7f5d0dbb8309 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
7a0e32463d28007cff5526ae037cc1447e50a50b 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 
5aff71e0e981c429f85663300d3e5c21089529a9 
  ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestWorker.java 
70ae85c458eae080a02df965174bf1207d9eaf8b 


Diff: https://reviews.apache.org/r/71948/diff/1/


Testing
---

Created new unit test.


Thanks,

Laszlo Pinter

Re: Review Request 71918: HIVE-22651: Fix TestCrudCompactorOnTez#testCompactionWithSchemaEvolutionNoBucketsMultipleReducers()

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71918/#review219108
---


Ship it!




Ship It!

- Laszlo Pinter


On Dec. 16, 2019, 1:53 p.m., Aron Hamvas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71918/
> ---
> 
> (Updated Dec. 16, 2019, 1:53 p.m.)
> 
> 
> Review request for hive, Denys Kuzmenko, Zoltan Haindrich, Marta Kuczora, and 
> Peter Vary.
> 
> 
> Bugs: HIVE-22651
> https://issues.apache.org/jira/browse/HIVE-22651
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Instead of assuming the exact content of the result set, the test is now 
> expecting the result set to not change and after CRUD major compaction.
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
>  b7245e2c35 
> 
> 
> Diff: https://reviews.apache.org/r/71918/diff/1/
> 
> 
> Testing
> ---
> 
> Executed the changed test.
> 
> 
> Thanks,
> 
> Aron Hamvas
> 
>

Review Request 71844: HIVE-22554: ACID: Wait timeout for blocking compaction should be configurable

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71844/
---

Review request for hive and Peter Vary.


Repository: hive-git


Description
---

HIVE-22554: ACID: Wait timeout for blocking compaction should be configurable


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
4393a2825e1f465781fc07a6678ebaa2bab906bd 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableCompactOperation.java
 fd0ae3a3df731aa690d024dfdbf89f7754ca2a41 


Diff: https://reviews.apache.org/r/71844/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Review Request 71812: HIVE-22534: ACID: Improve Compactor thread logging

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71812/
---

Review request for hive, Denys Kuzmenko and Peter Vary.


Repository: hive-git


Description
---

HIVE-22534: ACID: Improve Compactor thread logging


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
ee2c0f3e23ed716f3de0a2740a96a7ec39251bc2 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MajorQueryCompactor.java 
10681c0202a32c338e58b3e2eede03657a00774f 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMajorQueryCompactor.java 
f7e0a85c1f595bb4f112aa051779db3f00c8e572 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactor.java 
80119de22f602d9e3cb7a1f60b48e05a37c6a047 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactorFactory.java 
41cb4b64fbc79dcf81919769c567b26a2e18cfe5 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 
3270175a80992e0efb1e0bfd1f33ffd8a96fcf87 


Diff: https://reviews.apache.org/r/71812/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 71792: COMPLETED_TXN_COMPONENTS table is never cleaned up unless Compactor runs

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71792/#review218754
---


Ship it!




Lgtm +1

- Laszlo Pinter


On Nov. 21, 2019, 5:35 p.m., Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71792/
> ---
> 
> (Updated Nov. 21, 2019, 5:35 p.m.)
> 
> 
> Review request for hive, Laszlo Pinter and Peter Vary.
> 
> 
> Bugs: HIVE-21917
> https://issues.apache.org/jira/browse/HIVE-21917
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The Initiator thread in the metastore repeatedly loops over entries in the 
> COMPLETED_TXN_COMPONENTS table to determine which partitions / tables might 
> need to be compacted. However, entries are never removed from this table 
> except by a completed Compactor run.
> 
> In a cluster where most tables / partitions are write-once read-many, this 
> results in stale entries in this table never being cleaned up. In a small 
> test cluster, we have observed approximately 45k entries in this table 
> (virtually equal to the number of partitions in the cluster) while < 100 of 
> these tables have delta files at all. Since most of the tables will never get 
> enough writes to trigger a compaction (and in fact have only ever been 
> written to once), the initiator thread keeps trying to evaluate them on every 
> loop.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 610cf05204 
>   
> ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java
>  b28b57779b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
>  8253ccb9c9 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  268038795b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java
>  e840758c9d 
> 
> 
> Diff: https://reviews.apache.org/r/71792/diff/2/
> 
> 
> Testing
> ---
> 
> Unit tests
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Review Request 71775: HIVE-22280: Q tests for partitioned temporary tables

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71775/
---

Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.


Repository: hive-git


Description
---

HIVE-22280: Q tests for partitioned temporary tables


Diffs
-

  itests/src/test/resources/testconfiguration.properties 
2918a6852c6f8448ea44472df0be9d521d5c3b27 
  ql/src/test/queries/clientnegative/temp_table_addpart1.q PRE-CREATION 
  
ql/src/test/queries/clientnegative/temp_table_alter_rename_partition_failure.q 
PRE-CREATION 
  
ql/src/test/queries/clientnegative/temp_table_alter_rename_partition_failure2.q 
PRE-CREATION 
  
ql/src/test/queries/clientnegative/temp_table_alter_rename_partition_failure3.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/temp_table_drop_partition_failure.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/temp_table_drop_partition_filter_failure.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/temp_table_exchange_partitions.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_add_part_exist.q PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_add_part_multiple.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_add_part_with_loc.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_alter_partition_change_col.q 
PRE-CREATION 
  
ql/src/test/queries/clientpositive/temp_table_alter_partition_clusterby_sortby.q
 PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_alter_partition_coltype.q 
PRE-CREATION 
  
ql/src/test/queries/clientpositive/temp_table_alter_partition_onto_nocurrent_db.q
 PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_alter_rename_partition.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_avro_partitioned.q PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_avro_partitioned_native.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_default_partition_name.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_drop_multi_partitions.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_drop_partitions_filter.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_drop_partitions_filter2.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_drop_partitions_filter3.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_drop_partitions_filter4.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_exchange_partition.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_exchange_partition2.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_exchange_partition3.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_exchgpartition2lel.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_insert1_overwrite_partitions.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_insert2_overwrite_partitions.q 
PRE-CREATION 
  
ql/src/test/queries/clientpositive/temp_table_insert_values_dynamic_partitioned.q
 PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_insert_values_partitioned.q 
PRE-CREATION 
  
ql/src/test/queries/clientpositive/temp_table_insert_with_move_files_from_source_dir.q
 PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_llap_partitioned.q PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_load_dyn_part1.q PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_loadpart1.q PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_loadpart2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_merge_dynamic_partition.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_merge_dynamic_partition2.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_merge_dynamic_partition3.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_merge_dynamic_partition4.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_merge_dynamic_partition5.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_multi_insert_partitioned.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_orc_diff_part_cols.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_orc_diff_part_cols2.q 
PRE-CREATION 
  
ql/src/test/queries/clientpositive/temp_table_parquet_mixed_partition_formats.q 
PRE-CREATION 
  
ql/src/test/queries/clientpositive/temp_table_parquet_mixed_partition_formats2.q
 PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_parquet_partitioned.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_parquet_ppd_partition.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_partInit.q PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_partcols1.q PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_partition_boolean.q 
PRE-CREATION 
  

Review Request 71745: HIVE-22473: Avoid reading table as ACID when table name is starting with delta, but table is not transactional

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71745/
---

Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.


Repository: hive-git


Description
---

HIVE-22473: Avoid reading table as ACID when table name is starting with delta, 
but table is not transactional


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 
f72705f7b1f1fbab15ed1d53893dd99b34dfc45e 


Diff: https://reviews.apache.org/r/71745/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Review Request 71744: HIVE-22279: Enable temporary table partitioning

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71744/
---

Review request for hive.


Repository: hive-git


Description
---

HIVE-22279: Enable temporary table partitioning


Testing (updated)
---

Q tests will be covered in HIVE-22280.


Thanks,

Laszlo Pinter

Re: Review Request 71671: HIVE-22401: Refactor CompactorMR

[Laszlo Pinter via Review Board]

> On Oct. 28, 2019, 5:44 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMajorQueryCompactor.java
> > Lines 85-87 (patched)
> > 
> >
> > Maybe this should be checked outside? This is something general? Or am 
> > I mistaken?

In my next change, related to query based minor compaction, I  will refactor 
this part as well.


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71671/#review218420
---


On Oct. 24, 2019, 3:57 p.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71671/
> ---
> 
> (Updated Oct. 24, 2019, 3:57 p.m.)
> 
> 
> Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-22401: Refactor CompactorMR
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 0f1579aa542f83b68f2efc92e08e6c0a32bd113d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MajorQueryCompactor.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMajorQueryCompactor.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactor.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactorFactory.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/71671/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Review Request 71671: HIVE-22401: Refactor CompactorMR

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71671/
---

Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.


Repository: hive-git


Description
---

HIVE-22401: Refactor CompactorMR

At the moment CompactorMR contains all the methods which handles compaction. 
Since compaction can also be triggered as hive queries not only as MR jobs, it 
makes sense to move out these methods to a separate class, to avoid confusion.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
0f1579aa542f83b68f2efc92e08e6c0a32bd113d 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/InsertOnlyMajorQueryCompactor.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MajorQueryCompactor.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactor.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactorFactory.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/71671/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Review Request 71586: HIVE-22230: Add support for filtering partitions on temporary tables.

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71586/
---

Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.


Repository: hive-git


Description
---

HIVE-22230: Add support for filtering partitions on temporary tables.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/metadata/PartitionTree.java 
00b4301f54a670b638e98af75b32a5e5639559d9 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
db3d9dbca899e5563355ee8ea43b4020afcc71e6 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/TempTable.java 
e659b5407d212303ba95b7c2f65baab60989be5e 
  ql/src/test/org/apache/hadoop/hive/metastore/TestMetastoreExpr.java 
c861107ecb1338eafa5c5c84cf36303745676d4e 
  
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientGetPartitionsTempTable.java
 d35621a9c22f2a59653c705b093f19b0c9dc4ff8 
  
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientListPartitionsTempTable.java
 a74b543faa4eb8714be71a9eac6a8f55eaa97dba 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestListPartitions.java
 94d37c220f1ddf510526cb5986df46e5b88e6cc9 


Diff: https://reviews.apache.org/r/71586/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Review Request 71574: HIVE-22212: Implement append partition related methods on temporary tables

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71574/
---

Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.


Repository: hive-git


Description
---

HIVE-22212: Implement append partition related methods on temporary tables


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
2467ee3cfb9af0d64653ad7a012ee6d3e68d6674 
  
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientAppendPartitionTempTable.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestAppendPartitions.java
 e2593fecdce67a2874a14065c9247428ae8852d8 


Diff: https://reviews.apache.org/r/71574/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 71550: HIVE-22137: Implement alter/rename partition related methods on temporary tables

[Laszlo Pinter via Review Board]

> On Sept. 26, 2019, 5:24 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/PartitionTree.java
> > Lines 208-210 (patched)
> > 
> >
> > What happens when one of the alter partition fails only? How does it 
> > handled in the HMS handled tables?

I changed this part of the code. Now if this fails, the whole operation is 
reverted, ensuring transactional like behaviour.


> On Sept. 26, 2019, 5:24 p.m., Peter Vary wrote:
> > ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientAlterPartitionsTempTable.java
> > Lines 195 (patched)
> > 
> >
> > What was the original error on HMS version?
> > Why it is different here?

Changed the exception to InvalidOperationException, and added comment to 
IMetaStoreClient documentation.


> On Sept. 26, 2019, 5:24 p.m., Peter Vary wrote:
> > ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientAlterPartitionsTempTable.java
> > Lines 201 (patched)
> > 
> >
> > What was the original error on HMS version?
> > Why it is different here?

The original error is NullPointerException or TProtocolException, depending on 
what HMS implementation is in place.


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71550/#review217953
---


On Sept. 26, 2019, 4:54 p.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71550/
> ---
> 
> (Updated Sept. 26, 2019, 4:54 p.m.)
> 
> 
> Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-22137: Implement alter/rename partition related methods on temporary 
> tables
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/PartitionTree.java 
> c84c3ef595a5f26232d4f003e46f74bb14a7ec99 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  a5b16d12a514439046929dde031bbe8e80f71a28 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/TempTable.java 
> fa6dddcbadec59579357b9bc3d4ea42e44a1ca6f 
>   
> ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientAlterPartitionsTempTable.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  6c7d80ea576d6b10af191d8ceb158c20b1f70b46 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestAlterPartitions.java
>  4fc3688f2e5c06e254e2177dbf142b424dcfddd8 
> 
> 
> Diff: https://reviews.apache.org/r/71550/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Review Request 71550: HIVE-22137: Implement alter/rename partition related methods on temporary tables

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71550/
---

Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.


Repository: hive-git


Description
---

HIVE-22137: Implement alter/rename partition related methods on temporary tables


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/metadata/PartitionTree.java 
c84c3ef595a5f26232d4f003e46f74bb14a7ec99 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
a5b16d12a514439046929dde031bbe8e80f71a28 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/TempTable.java 
fa6dddcbadec59579357b9bc3d4ea42e44a1ca6f 
  
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientAlterPartitionsTempTable.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestAlterPartitions.java
 4fc3688f2e5c06e254e2177dbf142b424dcfddd8 


Diff: https://reviews.apache.org/r/71550/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 71506: HIVE-22084: Implement exchange partitions related methods on temporary tables.

[Laszlo Pinter via Review Board]

> On Sept. 23, 2019, 11:34 a.m., Peter Vary wrote:
> > Thanks Laszlo!
> > Few quick questions below.
> > We might want to test the cross exchange methods as well (exchange 
> > partition between temp and non-temp tables)
> > 
> > Otherwise LGTM

Thanks Peter for the review, highly appreciate it! I addressed all the review 
findings and added a couple more test to cover the mentioned scenario as well.


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71506/#review217910
---


On Sept. 18, 2019, 2:10 p.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71506/
> ---
> 
> (Updated Sept. 18, 2019, 2:10 p.m.)
> 
> 
> Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-22084: Implement exchange partitions related methods on temporary tables.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  506bf5dfa8404f3645b8c5db7ea19c0b4add33a7 
>   
> ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientExchangePartitionsTempTable.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestExchangePartitions.java
>  1a2b7e4f9f7562e84006e86061b482d1b535197c 
> 
> 
> Diff: https://reviews.apache.org/r/71506/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Review Request 71506: HIVE-22084: Implement exchange partitions related methods on temporary tables.

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71506/
---

Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.


Repository: hive-git


Description
---

HIVE-22084: Implement exchange partitions related methods on temporary tables.


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
506bf5dfa8404f3645b8c5db7ea19c0b4add33a7 
  
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientExchangePartitionsTempTable.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestExchangePartitions.java
 1a2b7e4f9f7562e84006e86061b482d1b535197c 


Diff: https://reviews.apache.org/r/71506/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 71243: HIVE-21875: Implement drop partition related methods on temporary tables.

[Laszlo Pinter via Review Board]

> On Sept. 16, 2019, 11:24 a.m., Gergely Hankó wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
> > Lines 948-952 (patched)
> > 
> >
> > This can be simplified to:
> > `return tableParameters != null && 
> > "true".equals(tableParameters.get(EXTERNAL_PARAM));`
> > 
> > It is more compact and also spares a query on the map.

Did the change. Thanks for the review.


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71243/#review217753
---


On Sept. 16, 2019, 12:39 p.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71243/
> ---
> 
> (Updated Sept. 16, 2019, 12:39 p.m.)
> 
> 
> Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-21875: Implement drop partition related methods on temporary tables.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  a2c84b4620fb1eb90069e294204f604565ffed9b 
>   
> ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientDropPartitionsTempTable.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestDropPartitions.java
>  91c9edac95e9f9688dd2b806aeeb0823af02574f 
> 
> 
> Diff: https://reviews.apache.org/r/71243/diff/2/
> 
> 
> Testing
> ---
> 
> Unit testing is done via 
> TestSessionHiveMetastoreClientDropPartitionsTempTable.java
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Re: Review Request 71243: HIVE-21875: Implement drop partition related methods on temporary tables.

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71243/
---

(Updated Sept. 16, 2019, 12:39 p.m.)


Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.


Repository: hive-git


Description (updated)
---

HIVE-21875: Implement drop partition related methods on temporary tables.


Diffs (updated)
-

  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
a2c84b4620fb1eb90069e294204f604565ffed9b 
  
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientDropPartitionsTempTable.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestDropPartitions.java
 91c9edac95e9f9688dd2b806aeeb0823af02574f 


Diff: https://reviews.apache.org/r/71243/diff/2/

Changes: https://reviews.apache.org/r/71243/diff/1-2/


Testing
---

Unit testing is done via 
TestSessionHiveMetastoreClientDropPartitionsTempTable.java


Thanks,

Laszlo Pinter

Review Request 71243: HIVE-21875: Implement drop partition related methods on temporary tables.

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71243/
---

Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.


Repository: hive-git


Description
---

HIVE-21875: Implement drop partition related methods on temporary tables.

This is one of the subtasks for HIVE-21765, to support partitions on temporary 
tables.


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
a2c84b4620fb1eb90069e294204f604565ffed9b 
  
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientDropPartitionsTempTable.java
 PRE-CREATION 
  
service/src/java/org/apache/hive/service/cli/thrift/ThreadPoolExecutorWithOomHook.java
 129413681045d790ae20bf4d8060f04162224565 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestDropPartitions.java
 91c9edac95e9f9688dd2b806aeeb0823af02574f 


Diff: https://reviews.apache.org/r/71243/diff/1/


Testing
---

Unit testing is done via 
TestSessionHiveMetastoreClientDropPartitionsTempTable.java


Thanks,

Laszlo Pinter

Re: Review Request 71011: HIVE:21957: Create temporary table like should omit transactional properties.

[Laszlo Pinter via Review Board]

> On July 18, 2019, noon, Marta Kuczora wrote:
> > Thanks a lot for the patch!
> > Just one question: could you add a test about the fixed use-case?

I will add test in a separate patch.


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71011/#review216720
---


On July 4, 2019, 1:52 p.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71011/
> ---
> 
> (Updated July 4, 2019, 1:52 p.m.)
> 
> 
> Review request for hive, Marta Kuczora and Thejas Nair.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE:21957: Create temporary table like should omit transactional properties.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> e09fc379f5e0127367e73ed4c4556522de9838a8 
> 
> 
> Diff: https://reviews.apache.org/r/71011/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Review Request 71011: HIVE:21957: Create temporary table like should omit transactional properties.

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71011/
---

Review request for hive, Marta Kuczora and Thejas Nair.


Repository: hive-git


Description
---

HIVE:21957: Create temporary table like should omit transactional properties.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
e09fc379f5e0127367e73ed4c4556522de9838a8 


Diff: https://reviews.apache.org/r/71011/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 70963: HIVE-21874: Implement add partitions related methods on temporary table

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70963/
---

(Updated July 1, 2019, 9:20 a.m.)


Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.


Repository: hive-git


Description
---

HIVE-21874: Implement add partitions related methods on temporary table


Diffs (updated)
-

  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
957ebb12725e9deac7e7644709521a998df4dbb4 
  
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientAddPartitionsFromSpecTempTable.java
 PRE-CREATION 
  
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientAddPartitionsTempTable.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
 a15f5ea0453c7459217d229fa373cc1fec2f4d7a 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitionsFromPartSpec.java
 25643495b53e1ede473c48a90b208b43070ee6aa 


Diff: https://reviews.apache.org/r/70963/diff/2/

Changes: https://reviews.apache.org/r/70963/diff/1-2/


Testing
---

Unit testing is done via TestSessionHiveMetastoreClientAddPartitionsTempTable, 
TestSessionHiveMetastoreClientAddPartitionsFromSpecTempTable.


Thanks,

Laszlo Pinter

Re: Review Request 70963: HIVE-21874: Implement add partitions related methods on temporary table

[Laszlo Pinter via Review Board]

> On June 28, 2019, 2:42 p.m., Marta Kuczora wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
> > Line 1046 (original), 1049-1050 (patched)
> > 
> >
> > Why do you need to make the DB and Table name lower case?

Partition properties like table and db name must be stored in lower case. This 
is the same in HiveMestarore as well. 
Other properties are case sensitive.


> On June 28, 2019, 2:42 p.m., Marta Kuczora wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
> > Lines 1100 (patched)
> > 
> >
> > Why is it needed to get the newly added partition from the "parts" list 
> > as the addPartition method returns the newly added Partition?

Good catch, corrected it.


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70963/#review216227
---


On June 27, 2019, 9:07 a.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70963/
> ---
> 
> (Updated June 27, 2019, 9:07 a.m.)
> 
> 
> Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-21874: Implement add partitions related methods on temporary table
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  957ebb12725e9deac7e7644709521a998df4dbb4 
>   
> ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientAddPartitionsFromSpecTempTable.java
>  PRE-CREATION 
>   
> ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientAddPartitionsTempTable.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
>  a15f5ea0453c7459217d229fa373cc1fec2f4d7a 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitionsFromPartSpec.java
>  25643495b53e1ede473c48a90b208b43070ee6aa 
> 
> 
> Diff: https://reviews.apache.org/r/70963/diff/1/
> 
> 
> Testing
> ---
> 
> Unit testing is done via 
> TestSessionHiveMetastoreClientAddPartitionsTempTable, 
> TestSessionHiveMetastoreClientAddPartitionsFromSpecTempTable.
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Review Request 70963: HIVE-21874: Implement add partitions related methods on temporary table

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70963/
---

Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.


Repository: hive-git


Description
---

HIVE-21874: Implement add partitions related methods on temporary table


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
957ebb12725e9deac7e7644709521a998df4dbb4 
  
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientAddPartitionsFromSpecTempTable.java
 PRE-CREATION 
  
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientAddPartitionsTempTable.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
 a15f5ea0453c7459217d229fa373cc1fec2f4d7a 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitionsFromPartSpec.java
 25643495b53e1ede473c48a90b208b43070ee6aa 


Diff: https://reviews.apache.org/r/70963/diff/1/


Testing
---

Unit testing is done via TestSessionHiveMetastoreClientAddPartitionsTempTable, 
TestSessionHiveMetastoreClientAddPartitionsFromSpecTempTable.


Thanks,

Laszlo Pinter

Re: Review Request 70934: HIVE-18735: Create table like loses transactional attribute.

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70934/
---

(Updated June 25, 2019, 12:47 p.m.)


Review request for hive, Eugene Koifman, Marta Kuczora, Peter Vary, and Adam 
Szita.


Repository: hive-git


Description
---

HIVE-18735: Create table like loses transactional attribute.


Diffs (updated)
-

  hbase-handler/src/test/results/positive/hbase_queries.q.out 
0c21d6d74882788d5748639ea2675579893791af 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
d395db1b59d021789b1bb47c7f09ff337cba2dd0 
  ql/src/test/results/clientpositive/alter_rename_table.q.out 
dd656954a1877f7f808de81f6952d7cf8ebfda2f 
  ql/src/test/results/clientpositive/alter_table_stats_status.q.out 
efa2834e0d6dbd77181473c214b77d09fcc1fe69 
  ql/src/test/results/clientpositive/autoColumnStats_1.q.out 
1f594ddb6816805d22a1152c261dda75490cd5d0 
  ql/src/test/results/clientpositive/autoColumnStats_2.q.out 
121a10384bca03942c297dd0488aceaf0d3bed68 
  ql/src/test/results/clientpositive/autoColumnStats_3.q.out 
777d165dc26fb11a6fd863fe1f375c6ae3d55b2a 
  ql/src/test/results/clientpositive/autoColumnStats_8.q.out 
0e1868bd52d717a6103f1456a1d4e525e85d8622 
  ql/src/test/results/clientpositive/create_alter_list_bucketing_table1.q.out 
593ae8389971449ad0f8704d911f6f7c6bcc 
  ql/src/test/results/clientpositive/create_like.q.out 
f4a5ed55a568b0160a6c87cb2fe8c7cd9b20c7c8 
  ql/src/test/results/clientpositive/create_like2.q.out 
7152f52fcf82d5052a67be6e27bda532f2b521bd 
  ql/src/test/results/clientpositive/create_like_tbl_props.q.out 
4d11fc3c9e39c18dd18fdb585ad1831a0a068768 
  ql/src/test/results/clientpositive/create_table_like_stats.q.out 
4aa1b4f167a99ffc97d97bb62e0f5313fd83314e 
  ql/src/test/results/clientpositive/describe_table.q.out 
8c7a16c4b65d3f3951e6c230c42325056a7eab0b 
  ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
3ceb3d03c2614f3256a822c9f105ed6e9f2bada8 
  ql/src/test/results/clientpositive/explain_ddl.q.out 
c53ffae8003bdcc320d4910f021c821c0777bdeb 
  ql/src/test/results/clientpositive/llap/autoColumnStats_1.q.out 
7272a9c925a4115ee3f1d3a4e6576057d75ac994 
  ql/src/test/results/clientpositive/llap/autoColumnStats_2.q.out 
1a4b164b0925860543dd74215e0820fe84c5f3f1 
  
ql/src/test/results/clientpositive/llap/insert_values_orig_table_use_metadata.q.out
 6c892cc5b87960b086d90c43516526056bdf221f 
  ql/src/test/results/clientpositive/llap/stats_noscan_1.q.out 
af55d23484ddb74a2c5b7f06c4e91a6063ae11dc 
  ql/src/test/results/clientpositive/llap/whroot_external1.q.out 
cac158c92669f1ad532ada3d6620adebeb909eae 
  ql/src/test/results/clientpositive/load_dyn_part8.q.out 
7b1b5c1f862a581af3b2c4cabe21b6d186601652 
  ql/src/test/results/clientpositive/merge3.q.out 
4e670558808894b0dd5f7b8815987e03de1dc6d3 
  ql/src/test/results/clientpositive/mm_default.q.out 
70519b7da8346ddc2de74e46010183d2c9ab11ee 
  ql/src/test/results/clientpositive/partition_discovery.q.out 
cddb6e56ba8db9162c491125e3efd3acd2ed29b2 
  ql/src/test/results/clientpositive/spark/load_dyn_part8.q.out 
aebf4382cd78b02d9b7bab7285254431f04e29c0 
  ql/src/test/results/clientpositive/spark/stats12.q.out 
9db43ef112d0898c08429c839e196b3e48067383 
  ql/src/test/results/clientpositive/spark/stats13.q.out 
4922d717a0074146d6da91aae859f09aa5a2b623 
  ql/src/test/results/clientpositive/spark/stats14.q.out 
eb8a995e298d77098c5d7a01086943dc08307c19 
  ql/src/test/results/clientpositive/spark/stats15.q.out 
3874e6de249428404946f461eec3575d6dcb50a5 
  ql/src/test/results/clientpositive/spark/stats2.q.out 
30339caeb2cff5cc96101d8cbf5f3ed8b5b01667 
  ql/src/test/results/clientpositive/spark/stats6.q.out 
77be16cb13558e6b2af2e772ff0505ea4dba8125 
  ql/src/test/results/clientpositive/spark/stats7.q.out 
fe942ad94b35288b0cc74d1434429378835ce9c2 
  ql/src/test/results/clientpositive/spark/stats8.q.out 
edfbd57f72b55d040233328330562703286627d3 
  ql/src/test/results/clientpositive/spark/stats9.q.out 
ed226b68d21733c7d371472d99f714b759e380e2 
  ql/src/test/results/clientpositive/spark/stats_noscan_1.q.out 
32001f6a6812ea7768933a0b479d2107265ef799 
  ql/src/test/results/clientpositive/stats12.q.out 
50cdace04384f9408298de50a426a22149605563 
  ql/src/test/results/clientpositive/stats13.q.out 
c6364ebf1d8cabe4818c9788d2819e8a4d7a0726 
  ql/src/test/results/clientpositive/stats14.q.out 
faebe8afd83a271a31bb88a51f0b8b6072f69209 
  ql/src/test/results/clientpositive/stats15.q.out 
2bfc6751e9b3300445acda1a967d4d9d39c856ee 
  ql/src/test/results/clientpositive/stats2.q.out 
e1bd37f393310385ed8dbcef88c6fa8bbc93d16c 
  ql/src/test/results/clientpositive/stats4.q.out 
c61fb6f2fcc848d8cd4c16836ad7b1933b3346f8 
  ql/src/test/results/clientpositive/stats6.q.out 
77be16cb13558e6b2af2e772ff0505ea4dba8125 
  ql/src/test/results/clientpositive/stats7.q.out 
bdf6f8602ccfeed6adffdd1f48b8c55cca52c0e3 
  

Review Request 70934: HIVE-18735: Create table like loses transactional attribute.

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70934/
---

Review request for hive, Eugene Koifman, Marta Kuczora, Peter Vary, and Adam 
Szita.


Repository: hive-git


Description
---

HIVE-18735: Create table like loses transactional attribute.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
d395db1b59d021789b1bb47c7f09ff337cba2dd0 


Diff: https://reviews.apache.org/r/70934/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 70867: HIVE-21814: Implement list partitions related methods on temporary tables

[Laszlo Pinter via Review Board]

> On June 19, 2019, 9:39 a.m., Marta Kuczora wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
> > Lines 1250-1255 (patched)
> > 
> >
> > This code piece is used in multiple methods. Maybe it would make sense 
> > to extract it to a separate method.
> > But since you have some more patches to go around the temp table 
> > partition handling, it is ok if you consider fixing this in a next patch.

Thanks for pointing this out. I refactored the code and moved this check into 
another method.


> On June 19, 2019, 9:39 a.m., Marta Kuczora wrote:
> > ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientListPartitionsTempTable.java
> > Lines 133 (patched)
> > 
> >
> > Would it make sense to add test with low max parts number to see if the 
> > method returns the correct number of partitions?

Added two more tests, for checking this use case as well.


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70867/#review215960
---


On June 20, 2019, 9:53 a.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70867/
> ---
> 
> (Updated June 20, 2019, 9:53 a.m.)
> 
> 
> Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-21814: Implement list partitions related methods on temporary tables
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  b71ef5a725d610cda402717f501f6c6a0f653216 
>   
> ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientListPartitionsTempTable.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/ConditionalIgnoreOnSessionHiveMetastoreClient.java
>  99039b08d014cddc9de12e70801267eba7331266 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestListPartitions.java
>  34ceb34de646cc2e501564e9b3a0cb8cc8a034e1 
> 
> 
> Diff: https://reviews.apache.org/r/70867/diff/2/
> 
> 
> Testing
> ---
> 
> Unit testing is done via 
> TestSessionHiveMetastoreClientListPartitionsTempTable.
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Re: Review Request 70867: HIVE-21814: Implement list partitions related methods on temporary tables

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70867/
---

(Updated June 20, 2019, 9:53 a.m.)


Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.


Repository: hive-git


Description (updated)
---

HIVE-21814: Implement list partitions related methods on temporary tables


Diffs (updated)
-

  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
b71ef5a725d610cda402717f501f6c6a0f653216 
  
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientListPartitionsTempTable.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/ConditionalIgnoreOnSessionHiveMetastoreClient.java
 99039b08d014cddc9de12e70801267eba7331266 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestListPartitions.java
 34ceb34de646cc2e501564e9b3a0cb8cc8a034e1 


Diff: https://reviews.apache.org/r/70867/diff/2/

Changes: https://reviews.apache.org/r/70867/diff/1-2/


Testing
---

Unit testing is done via TestSessionHiveMetastoreClientListPartitionsTempTable.


Thanks,

Laszlo Pinter

Review Request 70867: HIVE-21814: Implement list partitions related methods on temporary tables

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70867/
---

Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.


Repository: hive-git


Description
---

HIVE-21814: Implement list partitions related methods on temporary tables

This change is the next step to support partitions on temporary tables. 
HIVE-18739 and HIVE-20661 added partial support for partition columns on 
temporary tables, but it was not complete and it was available only for 
internal usage. This change addresses the missing functionality related to 
listing partitions from temporary tables, although is still remains unexposed 
until all the partition related functionalities (get, list, add, alter etc.) 
are implemented.


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
b71ef5a725d610cda402717f501f6c6a0f653216 
  
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientListPartitionsTempTable.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/ConditionalIgnoreOnSessionHiveMetastoreClient.java
 99039b08d014cddc9de12e70801267eba7331266 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestListPartitions.java
 34ceb34de646cc2e501564e9b3a0cb8cc8a034e1 


Diff: https://reviews.apache.org/r/70867/diff/1/


Testing
---

Unit testing is done via TestSessionHiveMetastoreClientListPartitionsTempTable.


Thanks,

Laszlo Pinter

Review Request 70850: HIVE-21812: Implement get partition related methods on temporary tables

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70850/
---

Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.


Repository: hive-git


Description
---

HIVE-21812: Implement get partition related methods on temporary tables

HIVE-18739 and HIVE-20661 added partial support for partition columns on 
temporary tables, but it was not complete and it was available only for 
internal usage. This change addresses the missing functionality related to 
getting partitions from temporary tables, although is still remains unexposed 
until all the partition related functionalities (get, list, add, alter etc.) 
are implemented.


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
410868cacfe53e8898d4e08572d7a01e05b7eb49 
  
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientGetPartitionsTempTable.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/ConditionalIgnoreOnSessionHiveMetastoreClient.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/CustomIgnoreRule.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/MetaStoreClientTest.java
 dc48fa8308a07f68c5e21a2d95f40127d3ff41df 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestGetPartitions.java
 4d7f7c12203a9a90568f4aae644ff5cabaafa18c 


Diff: https://reviews.apache.org/r/70850/diff/1/


Testing
---

Unit testing is done via TestSessionHiveMetastoreClientGetPartitionsTempTable.


Thanks,

Laszlo Pinter

Re: Review Request 69341: HIVE-20891: Call alter_partition in batch when dynamically loading partitions

[Laszlo Pinter via Review Board]

> On Dec. 4, 2018, 10:08 p.m., Bharathkrishna Guruvayoor Murali wrote:
> > Hi, the patch looks good, I just had a quick question. Does this mean that 
> > all the alters happen in one transaction? Will it prevent concurrent 
> > operations for the whole time a large number of partitions are altered?

No it will not. What was changed in this patch, that the list of partitions are 
passed to the metastore in one call, keeping the opened connections to minimum. 
Under the hood everything stayed the same, so it is possible to perform 
concurrent operations.


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69341/#review211026
---


On Nov. 30, 2018, 11:31 a.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69341/
> ---
> 
> (Updated Nov. 30, 2018, 11:31 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20891: Call alter_partition in batch when dynamically loading partitions
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/metastore/SynchronizedMetaStoreClient.java 
> e8f362357537e73502f743a9df189dec9be2da5d 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
> e185bf49d42da9d1497643c20bbd71edaf071bf1 
> 
> 
> Diff: https://reviews.apache.org/r/69341/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Re: Review Request 69341: HIVE-20891: Call alter_partition in batch when dynamically loading partitions

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69341/
---

(Updated Nov. 30, 2018, 11:31 a.m.)


Review request for hive.


Repository: hive-git


Description
---

HIVE-20891: Call alter_partition in batch when dynamically loading partitions


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/metastore/SynchronizedMetaStoreClient.java 
e8f362357537e73502f743a9df189dec9be2da5d 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
e185bf49d42da9d1497643c20bbd71edaf071bf1 


Diff: https://reviews.apache.org/r/69341/diff/3/

Changes: https://reviews.apache.org/r/69341/diff/2-3/


Testing
---


Thanks,

Laszlo Pinter

Review Request 69341: HIVE-20891: Call alter_partition in batch when dynamically loading partitions

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69341/
---

Review request for hive.


Repository: hive-git


Description
---

HIVE-20891: Call alter_partition in batch when dynamically loading partitions


Diffs
-

  ql/src/java/org/apache/hadoop/hive/metastore/SynchronizedMetaStoreClient.java 
e8f362357537e73502f743a9df189dec9be2da5d 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
e185bf49d42da9d1497643c20bbd71edaf071bf1 


Diff: https://reviews.apache.org/r/69341/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 69167: HIVE-20796: jdbc URL can contain sensitive information that should not be logged

[Laszlo Pinter via Review Board]

> On Oct. 25, 2018, 4:44 p.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
> > Lines 1176 (patched)
> > 
> >
> > Nit: all the trendy kids use StringBuilder now

Removed StringBuffer


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69167/#review210043
---


On Oct. 25, 2018, 1:36 p.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69167/
> ---
> 
> (Updated Oct. 25, 2018, 1:36 p.m.)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20796: jdbc URL can contain sensitive information that should not be 
> logged
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  9c158040497cd3d2762620ce35e2b46bb6d5fffe 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
>  f3b38665676391fec9b85eb9a405c14632340dc6 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetaStoreServerUtils.java
>  f4bdd734dc4e731dda01e6031a4115cde5571baf 
> 
> 
> Diff: https://reviews.apache.org/r/69167/diff/1/
> 
> 
> Testing
> ---
> 
> New unit test created.
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Re: Review Request 69167: HIVE-20796: jdbc URL can contain sensitive information that should not be logged

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69167/
---

(Updated Oct. 26, 2018, 10:34 a.m.)


Review request for hive and Peter Vary.


Repository: hive-git


Description
---

HIVE-20796: jdbc URL can contain sensitive information that should not be logged


Diffs (updated)
-

  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 9c158040497cd3d2762620ce35e2b46bb6d5fffe 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
 f3b38665676391fec9b85eb9a405c14632340dc6 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetaStoreServerUtils.java
 f4bdd734dc4e731dda01e6031a4115cde5571baf 


Diff: https://reviews.apache.org/r/69167/diff/2/

Changes: https://reviews.apache.org/r/69167/diff/1-2/


Testing
---

New unit test created.


Thanks,

Laszlo Pinter

Re: Review Request 69167: HIVE-20796: jdbc URL can contain sensitive information that should not be logged

[Laszlo Pinter via Review Board]

> On Oct. 25, 2018, 4:48 p.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
> > Lines 1163 (patched)
> > 
> >
> > Strings are immutable

done.


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69167/#review210044
---


On Oct. 25, 2018, 1:36 p.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69167/
> ---
> 
> (Updated Oct. 25, 2018, 1:36 p.m.)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20796: jdbc URL can contain sensitive information that should not be 
> logged
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  9c158040497cd3d2762620ce35e2b46bb6d5fffe 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
>  f3b38665676391fec9b85eb9a405c14632340dc6 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetaStoreServerUtils.java
>  f4bdd734dc4e731dda01e6031a4115cde5571baf 
> 
> 
> Diff: https://reviews.apache.org/r/69167/diff/1/
> 
> 
> Testing
> ---
> 
> New unit test created.
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Re: Review Request 69167: HIVE-20796: jdbc URL can contain sensitive information that should not be logged

[Laszlo Pinter via Review Board]

> On Oct. 25, 2018, 10:46 p.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
> > Lines 1172 (patched)
> > 
> >
> > Could be optimized to:
> > 
> > String[] sensitiveData = {"user", "password"};
> > String regex = "([;,\?&]" + String.join("|", sensitiveData) + 
> > ")=.*?([;,&\)]+)";
> > 
> > String result = 
> > Pattern.compile(regex).matcher(connectionURL).replaceAll("$1=***$2");
> 
> Denys Kuzmenko wrote:
> or just connectionURL.replaceAll(regex, "$1=***$2");
> 
> Denys Kuzmenko wrote:
> regex = "([;,?&]" + String.join("|", sensitiveData) + ")=.*?([;,&)]?)";

Thanks for the input. It's much shorter than my implementation.


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69167/#review210059
---


On Oct. 25, 2018, 1:36 p.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69167/
> ---
> 
> (Updated Oct. 25, 2018, 1:36 p.m.)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20796: jdbc URL can contain sensitive information that should not be 
> logged
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  9c158040497cd3d2762620ce35e2b46bb6d5fffe 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
>  f3b38665676391fec9b85eb9a405c14632340dc6 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetaStoreServerUtils.java
>  f4bdd734dc4e731dda01e6031a4115cde5571baf 
> 
> 
> Diff: https://reviews.apache.org/r/69167/diff/1/
> 
> 
> Testing
> ---
> 
> New unit test created.
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Review Request 69167: HIVE-20796: jdbc URL can contain sensitive information that should not be logged

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69167/
---

Review request for hive and Peter Vary.


Repository: hive-git


Description
---

HIVE-20796: jdbc URL can contain sensitive information that should not be logged


Diffs
-

  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 9c158040497cd3d2762620ce35e2b46bb6d5fffe 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
 f3b38665676391fec9b85eb9a405c14632340dc6 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetaStoreServerUtils.java
 f4bdd734dc4e731dda01e6031a4115cde5571baf 


Diff: https://reviews.apache.org/r/69167/diff/1/


Testing
---

New unit test created.


Thanks,

Laszlo Pinter

Re: Review Request 68975: HIVE-20661: Dynamic partitions loading calls add partition for every partition 1-by-1

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68975/
---

(Updated Oct. 19, 2018, 8:30 a.m.)


Review request for hive and Peter Vary.


Repository: hive-git


Description
---

HIVE-20661: Dynamic partitions loading calls add partition for every partition 
1-by-1


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/metastore/SynchronizedMetaStoreClient.java 
0ab77e84c6222d35bcec9ce95ed02014911ef144 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
4de038913a5c9a2c199f71702b8f70ca84d0856b 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
dd23d7db3e70c9540e48c42eb7b9a33ed775cea6 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 aba63f050b5b98a2aeeb0df6ff2de5e6e06761f2 


Diff: https://reviews.apache.org/r/68975/diff/6/

Changes: https://reviews.apache.org/r/68975/diff/5-6/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 68975: HIVE-20661: Dynamic partitions loading calls add partition for every partition 1-by-1

[Laszlo Pinter via Review Board]

> On Oct. 18, 2018, 9:13 a.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Lines 1921 (patched)
> > 
> >
> > nit: one extra space

done.


> On Oct. 18, 2018, 9:13 a.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Line 1876 (original)
> > 
> >
> > This perf logging is removed... Let's think through if we need other 
> > places to add some instead considering the new calling structure

It was not removed, but moved to another place. Now the tracking is started and 
finished in loadPartition().


> On Oct. 18, 2018, 9:13 a.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Lines 2171 (patched)
> > 
> >
> > What happens when only 1 partiton is already exists from multiple ones?
> > 
> > Also, do we need SynchronizedMSC?

Yes, we need this one in every case. The SynchronizedMetaStoreClient is just a 
facede for the SessionHiveMetaStoreClient, which in the other hand, does some 
pre-processing (eg. adding the partition to the temp table), before calling the 
HiveMetaStoreClient.


> On Oct. 18, 2018, 9:13 a.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Lines 2570 (patched)
> > 
> >
> > Do we need to call this every time in the for loop? I kinda remember 
> > that we allow only partitions for a single table only... Or I might be 
> > mistaken, but still might be a good idea to not generate this every time...

I think we do need. The fetching of the snapshot is done inside a callable task 
which is executed on thread pool in parallel. Altough we use only one table, 
the snapshot of it can change from time to time, depending on how many tasks 
were already finished. It's better to be safe, and get a new snapshot every 
time, than have an outdated one.


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68975/#review209735
---


On Oct. 18, 2018, 7:01 a.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68975/
> ---
> 
> (Updated Oct. 18, 2018, 7:01 a.m.)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20661: Dynamic partitions loading calls add partition for every 
> partition 1-by-1
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/metastore/SynchronizedMetaStoreClient.java 
> 0ab77e84c6222d35bcec9ce95ed02014911ef144 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
> 4de038913a5c9a2c199f71702b8f70ca84d0856b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  dd23d7db3e70c9540e48c42eb7b9a33ed775cea6 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  aba63f050b5b98a2aeeb0df6ff2de5e6e06761f2 
> 
> 
> Diff: https://reviews.apache.org/r/68975/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Re: Review Request 68975: HIVE-20661: Dynamic partitions loading calls add partition for every partition 1-by-1

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68975/
---

(Updated Oct. 18, 2018, 7:01 a.m.)


Review request for hive and Peter Vary.


Repository: hive-git


Description
---

HIVE-20661: Dynamic partitions loading calls add partition for every partition 
1-by-1


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/metastore/SynchronizedMetaStoreClient.java 
0ab77e84c6222d35bcec9ce95ed02014911ef144 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
4de038913a5c9a2c199f71702b8f70ca84d0856b 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
dd23d7db3e70c9540e48c42eb7b9a33ed775cea6 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 aba63f050b5b98a2aeeb0df6ff2de5e6e06761f2 


Diff: https://reviews.apache.org/r/68975/diff/5/

Changes: https://reviews.apache.org/r/68975/diff/4-5/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 68975: HIVE-20661: Dynamic partitions loading calls add partition for every partition 1-by-1

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68975/
---

(Updated Oct. 17, 2018, 1:44 p.m.)


Review request for hive and Peter Vary.


Repository: hive-git


Description
---

HIVE-20661: Dynamic partitions loading calls add partition for every partition 
1-by-1


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/metastore/SynchronizedMetaStoreClient.java 
0ab77e84c6222d35bcec9ce95ed02014911ef144 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
4de038913a5c9a2c199f71702b8f70ca84d0856b 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
dd23d7db3e70c9540e48c42eb7b9a33ed775cea6 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 aba63f050b5b98a2aeeb0df6ff2de5e6e06761f2 


Diff: https://reviews.apache.org/r/68975/diff/4/

Changes: https://reviews.apache.org/r/68975/diff/3-4/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 68975: HIVE-20661: Dynamic partitions loading calls add partition for every partition 1-by-1

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68975/
---

(Updated Oct. 13, 2018, 9:55 a.m.)


Review request for hive and Peter Vary.


Repository: hive-git


Description
---

HIVE-20661: Dynamic partitions loading calls add partition for every partition 
1-by-1


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/metastore/SynchronizedMetaStoreClient.java 
0ab77e84c6222d35bcec9ce95ed02014911ef144 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
4de038913a5c9a2c199f71702b8f70ca84d0856b 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
a2b57fb646899c54b63be14a8cde9b8644a973aa 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 a2ec09f7157140fda751ab777649c698b3319a16 


Diff: https://reviews.apache.org/r/68975/diff/3/

Changes: https://reviews.apache.org/r/68975/diff/2-3/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 68975: HIVE-20661: Dynamic partitions loading calls add partition for every partition 1-by-1

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68975/
---

(Updated Oct. 13, 2018, 7:56 a.m.)


Review request for hive and Peter Vary.


Repository: hive-git


Description
---

HIVE-20661: Dynamic partitions loading calls add partition for every partition 
1-by-1


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/metastore/SynchronizedMetaStoreClient.java 
0ab77e84c6222d35bcec9ce95ed02014911ef144 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
4de038913a5c9a2c199f71702b8f70ca84d0856b 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
a2b57fb646899c54b63be14a8cde9b8644a973aa 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 a2ec09f7157140fda751ab777649c698b3319a16 


Diff: https://reviews.apache.org/r/68975/diff/2/

Changes: https://reviews.apache.org/r/68975/diff/1-2/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 68975: HIVE-20661: Dynamic partitions loading calls add partition for every partition 1-by-1

[Laszlo Pinter via Review Board]

> On Oct. 11, 2018, 8:54 a.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Lines 1896 (patched)
> > 
> >
> > What does this method do?
> > Moving files and updating partition data if the partition is already 
> > exists?
> > Javadoc would be good anyway.
> > Follow-up idea: update partition data with alter_partitions (multiple 
> > updates with 1 HMS call)?

Good point, javadoc added.

I noticed that as well, just didn't want to make this change even more 
complicated. There are other points also wich could be be further optimized.  I 
made a note to myself to open a new jira after this is merged in.


> On Oct. 11, 2018, 8:54 a.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Line 1909 (original), 1939 (patched)
> > 
> >
> > nit: maybe do not reformat these lines if they are not needed

done.


> On Oct. 11, 2018, 8:54 a.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Line 1939 (original), 1961 (patched)
> > 
> >
> > nit: maybe do not reformat these lines if they are not needed

done.


> On Oct. 11, 2018, 8:54 a.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Line 1948 (original), 1970 (patched)
> > 
> >
> > nit: maybe do not reformat these lines if they are not needed

done.


> On Oct. 11, 2018, 8:54 a.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Line 1966 (original), 1988 (patched)
> > 
> >
> > nit: maybe do not reformat these lines if they are not needed

done.


> On Oct. 11, 2018, 8:54 a.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Lines 1970-1971 (original), 1992-1993 (patched)
> > 
> >
> > nit: maybe do not reformat these lines if they are not needed

done.


> On Oct. 11, 2018, 8:54 a.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Lines 1978-1979 (original), 2000-2001 (patched)
> > 
> >
> > nit: maybe do not reformat these lines if they are not needed

done.


> On Oct. 11, 2018, 8:54 a.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Lines 2102-2103 (patched)
> > 
> >
> > Concerned about this.
> > When we call Hive.loadPartition we call 2 methods there:
> > - loadPartitionInternal - 1 snapshot
> > - addPartitionToMetastore - 1 snapshot
> > Are we sure that these calls are:
> > - Lightweight - so we can happily call them twice
> > - Return the same value in both ocassions even if some other transacion 
> > is finished during this time?

Yes, I see your point here. 
By checking the implementation of AcidUtils.getTableSnapshot, it is not 
guaranteed that the two calls will have the same result. 
I made a modification to use the same instance of snapshot until the loading of 
partition is done.


> On Oct. 11, 2018, 8:54 a.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Lines 2083-2084 (original), 2129-2130 (patched)
> > 
> >
> > We do not use batching for adding partitions?

What should be the size of the batch? I used BATCH_RETRIEVE_MAX configuration 
parameter for fetching data from metastore. Can I use this one as well, or 
there is another parameter?


> On Oct. 11, 2018, 8:54 a.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Lines 2492-2506 (patched)
> > 
> >
> > Do we need this? Can't we use equals method of maps to compare instead?

Java arrays do not override Object.equals(). Hence if you compare them using 
equals() (which is what the equals methods of all the collection classes do), 
you get "instance equality", instead of "value equality".


> On Oct. 11, 2018, 8:54 a.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Lines 2511 (patched)
> > 
> >
> > Note: We still might fail with OOM if too many new partitions are 
> > there. We store in memory all of the new and old partition specifications. 
> > Was this the same before?

Yes, in the previous 

Review Request 68975: HIVE-20661: Dynamic partitions loading calls add partition for every partition 1-by-1

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68975/
---

Review request for hive and Peter Vary.


Repository: hive-git


Description
---

HIVE-20661: Dynamic partitions loading calls add partition for every partition 
1-by-1


Diffs
-

  ql/src/java/org/apache/hadoop/hive/metastore/SynchronizedMetaStoreClient.java 
0ab77e84c6222d35bcec9ce95ed02014911ef144 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
4de038913a5c9a2c199f71702b8f70ca84d0856b 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
a2b57fb646899c54b63be14a8cde9b8644a973aa 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 a2ec09f7157140fda751ab777649c698b3319a16 


Diff: https://reviews.apache.org/r/68975/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 68767: HIVE-20551: Create PreparedStatement query dynamically when IN clause is used

[Laszlo Pinter via Review Board]

> On Sept. 20, 2018, 10:56 p.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
> > Lines 316 (patched)
> > 
> >
> > 1) Does setObject() work OK on all the jdbc drivers that are supported? 
> > In the oast I have seen cases where it was necessary to dispatch to the 
> > correct method like setString, setInt 
> > 2) can the params over be null? Do we need to call setNull instead of 
> > setObject()? Again we need to consider all the drivers.
> 
> Laszlo Pinter wrote:
> 1) The jdbc driver will do the type checking. A slight disadvantage is 
> the minor overhead, but this is negligible as compared to the better 
> maitainable code you end up with.
> 2) You're correct, I have to make sure that the params[i] is not null or 
> use setNull instead.
> 
> Laszlo Pinter wrote:
> So I did a bit more of a debugging, and my previous comment about the 
> params[i] can be null is not correct. The params can contain partitionIds, 
> storageDescriptorIds, columnDescriptorIds, serdeIds, depeding from where the 
> executeNoResult() is called.  These fields are mandatory and cannot be null. 
> If any of these items are null, means that the metastore db is not consistent 
> and it was corrupted.
> 
> Andrew Sherman wrote:
> I worte this once, but rb ate it, sorry if it duplicates.
> On 1) Did you test with all drivers?
> On 2) I suggest you add some checking to nail down that aprams are 
> non-null. How is the java testing of this class? Do we need negative test 
> cases?

1) Do you know what are the supported jdbc drivers or where could I check them?
2) It doesn't makes sense to have null values in queries like 
```sql
SELECT column_name1 FROM table_name WHERE column_name2 IN (value1, value2 ...);
```
so I filtered them out.


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68767/#review208821
---


On Sept. 24, 2018, 12:16 p.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68767/
> ---
> 
> (Updated Sept. 24, 2018, 12:16 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Peter Vary, and Vihang 
> Karajgaonkar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20551: Create PreparedStatement query dynamically when IN clause is used
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
> 
> 
> Diff: https://reviews.apache.org/r/68767/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Re: Review Request 68767: HIVE-20551: Create PreparedStatement query dynamically when IN clause is used

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68767/
---

(Updated Sept. 24, 2018, 12:16 p.m.)


Review request for hive, Alexander Kolbasov, Peter Vary, and Vihang 
Karajgaonkar.


Repository: hive-git


Description
---

HIVE-20551: Create PreparedStatement query dynamically when IN clause is used


Diffs (updated)
-

  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 571c789eddfd2b1a27c65c48bdc6dccfafaaf676 


Diff: https://reviews.apache.org/r/68767/diff/3/

Changes: https://reviews.apache.org/r/68767/diff/2-3/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 68767: HIVE-20551: Create PreparedStatement query dynamically when IN clause is used

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68767/
---

(Updated Sept. 21, 2018, 3:21 p.m.)


Review request for hive, Alexander Kolbasov, Peter Vary, and Vihang 
Karajgaonkar.


Repository: hive-git


Description
---

HIVE-20551: Create PreparedStatement query dynamically when IN clause is used


Diffs (updated)
-

  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 571c789eddfd2b1a27c65c48bdc6dccfafaaf676 


Diff: https://reviews.apache.org/r/68767/diff/2/

Changes: https://reviews.apache.org/r/68767/diff/1-2/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 68767: HIVE-20551: Create PreparedStatement query dynamically when IN clause is used

[Laszlo Pinter via Review Board]

> On Sept. 20, 2018, 10:56 p.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
> > Lines 316 (patched)
> > 
> >
> > 1) Does setObject() work OK on all the jdbc drivers that are supported? 
> > In the oast I have seen cases where it was necessary to dispatch to the 
> > correct method like setString, setInt 
> > 2) can the params over be null? Do we need to call setNull instead of 
> > setObject()? Again we need to consider all the drivers.
> 
> Laszlo Pinter wrote:
> 1) The jdbc driver will do the type checking. A slight disadvantage is 
> the minor overhead, but this is negligible as compared to the better 
> maitainable code you end up with.
> 2) You're correct, I have to make sure that the params[i] is not null or 
> use setNull instead.

So I did a bit more of a debugging, and my previous comment about the params[i] 
can be null is not correct. The params can contain partitionIds, 
storageDescriptorIds, columnDescriptorIds, serdeIds, depeding from where the 
executeNoResult() is called.  These fields are mandatory and cannot be null. If 
any of these items are null, means that the metastore db is not consistent and 
it was corrupted.


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68767/#review208821
---


On Sept. 19, 2018, 9:46 a.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68767/
> ---
> 
> (Updated Sept. 19, 2018, 9:46 a.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Peter Vary, and Vihang 
> Karajgaonkar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20551: Create PreparedStatement query dynamically when IN clause is used
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
> 
> 
> Diff: https://reviews.apache.org/r/68767/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Re: Review Request 68767: HIVE-20551: Create PreparedStatement query dynamically when IN clause is used

[Laszlo Pinter via Review Board]

> On Sept. 20, 2018, 10:56 p.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
> > Line 316 (original), 315 (patched)
> > 
> >
> > may be neater to use 'for (String param: params)' or whatever the 
> > syntax is

I used this approach, because a counter is required to know to which position 
should the parameter be injected. By default the for-each loop doesn't have an 
internal counter, it is based on the Iterable interface.


> On Sept. 20, 2018, 10:56 p.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
> > Lines 316 (patched)
> > 
> >
> > 1) Does setObject() work OK on all the jdbc drivers that are supported? 
> > In the oast I have seen cases where it was necessary to dispatch to the 
> > correct method like setString, setInt 
> > 2) can the params over be null? Do we need to call setNull instead of 
> > setObject()? Again we need to consider all the drivers.

1) The jdbc driver will do the type checking. A slight disadvantage is the 
minor overhead, but this is negligible as compared to the better maitainable 
code you end up with.
2) You're correct, I have to make sure that the params[i] is not null or use 
setNull instead.


> On Sept. 20, 2018, 10:56 p.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
> > Line 317 (original), 320 (patched)
> > 
> >
> > I note the tracing will be less intersting now. Do we now need to 
> > insert the paramters as well?

I will add the parameters to trace as well.


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68767/#review208821
---


On Sept. 19, 2018, 9:46 a.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68767/
> ---
> 
> (Updated Sept. 19, 2018, 9:46 a.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Peter Vary, and Vihang 
> Karajgaonkar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20551: Create PreparedStatement query dynamically when IN clause is used
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
> 
> 
> Diff: https://reviews.apache.org/r/68767/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Review Request 68767: HIVE-20551: Create PreparedStatement query dynamically when IN clause is used

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68767/
---

Review request for hive, Alexander Kolbasov, Peter Vary, and Vihang 
Karajgaonkar.


Repository: hive-git


Description
---

HIVE-20551: Create PreparedStatement query dynamically when IN clause is used


Diffs
-

  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 571c789eddfd2b1a27c65c48bdc6dccfafaaf676 


Diff: https://reviews.apache.org/r/68767/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Review Request 68765: HIVE-20595: Add findbugs-exclude.xml to metastore-server

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68765/
---

Review request for hive, Alexander Kolbasov, Peter Vary, and Vihang 
Karajgaonkar.


Repository: hive-git


Description
---

HIVE-20595: Add findbugs-exclude.xml to metastore-server


Diffs
-

  standalone-metastore/metastore-server/findbugs/findbugs-exclude.xml 
PRE-CREATION 


Diff: https://reviews.apache.org/r/68765/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 68710: HIVE-20544: TOpenSessionReq logs password and username

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68710/#review208621
---


Ship it!




LGTM +! (non-binding)

- Laszlo Pinter


On Sept. 13, 2018, 10:11 a.m., Karen Coppage wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68710/
> ---
> 
> (Updated Sept. 13, 2018, 10:11 a.m.)
> 
> 
> Review request for hive and Laszlo Pinter.
> 
> 
> Bugs: HIVE-20544
> https://issues.apache.org/jira/browse/HIVE-20544
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TOpenSessionReq, if client protocol is unset, both username and password are 
> logged. Logging a password is a security risk. This patch would hide it with 
> asterisks.
> 
> 
> Diffs
> -
> 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionReq.java
>  3195e704f3 
> 
> 
> Diff: https://reviews.apache.org/r/68710/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Karen Coppage
> 
>

Re: Review Request 68630: HIVE-20420: Provide a fallback authorizer when no other authorizer is in use

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68630/#review208550
---


Ship it!




Thanks for uploading a new patch. LGTM +1

- Laszlo Pinter


On Sept. 11, 2018, 11:19 p.m., Daniel Dai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68630/
> ---
> 
> (Updated Sept. 11, 2018, 11:19 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See HIVE-20420
> 
> 
> Diffs
> -
> 
>   ql/pom.xml a55cbe3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/SettableConfigUpdater.java
>  12be41c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizer.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizerFactory.java
>  PRE-CREATION 
>   ql/src/test/queries/clientnegative/fallbackauth_addjar.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/fallbackauth_compile.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/fallbackauth_create_func1.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/fallbackauth_create_func2.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/fallbackauth_dfs.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/fallbackauth_disallow_transform.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/fallbackauth_load.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/fallbackauth_set_invalidconf.q 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/fallbackauth_addjar.q.out PRE-CREATION 
>   ql/src/test/results/clientnegative/fallbackauth_compile.q.out PRE-CREATION 
>   ql/src/test/results/clientnegative/fallbackauth_create_func1.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/fallbackauth_create_func2.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/fallbackauth_dfs.q.out PRE-CREATION 
>   ql/src/test/results/clientnegative/fallbackauth_disallow_transform.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/fallbackauth_load.q.out PRE-CREATION 
>   ql/src/test/results/clientnegative/fallbackauth_set_invalidconf.q.out 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/68630/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Daniel Dai
> 
>

Review Request 68656: HIVE-20505: upgrade org.openjdk.jmh:jmh-core to 1.21

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68656/
---

Review request for hive.


Repository: hive-git


Description
---

HIVE-20505: upgrade org.openjdk.jmh:jmh-core to 1.21


Diffs
-

  itests/hive-jmh/pom.xml 0abefdf791a04593c547119256a755adcd78bda5 


Diff: https://reviews.apache.org/r/68656/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 68630: HIVE-20420: Provide a fallback authorizer when no other authorizer is in use

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68630/#review208398
---




ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizer.java
Lines 23 (patched)


Unused import.



ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizer.java
Lines 41 (patched)


Unused import.



ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizer.java
Lines 47 (patched)


Unused import.



ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizer.java
Lines 55 (patched)


The conf variable is used only in the constructor. No need to keep it in a 
member variable.



ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizer.java
Lines 58 (patched)


The constructor visibility can be changed to package-private, since it is 
instantiated through FallbackHiveAuthorizerFactory.



ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizer.java
Lines 94 (patched)


The next few methods are declared to throw HiveAccessControlException, 
though it is never thrown in the method. Let's try to keep the code as clean 
and simple as possible.



ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizer.java
Lines 148 (patched)


No need to explicitly declare the type argument. You can use type inference 
in case of generic instance creation.



ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizer.java
Lines 159 (patched)


Unnecessary throws clause declaration.



ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizer.java
Lines 164-171 (patched)


This section can be replaced with a more elegant implementation, using 
stream api.
```java
if (admins != null && Arrays.stream(admins).parallel().anyMatch(n -> 
n.equals(userName)) {
return;
}

```



ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizer.java
Lines 180-187 (patched)


I'm not sure this loop is correct, since it's only checking the first 
element of the hiveObjects list, afterwards breaks the loop. 
If the intention was to check only the first entry, no need to use a loop.
```
boolean needAdmin =
!hiveObjects.isEmpty() && hiveObjects.get(0).getType() == 
HivePrivilegeObject.HivePrivilegeObjectType.LOCAL_URI;
```
If you want to check all the entries, than this implementation is incorrect.



ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizer.java
Lines 188 (patched)


No need to check for the operation type if needAdmin is already true.



ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizer.java
Lines 203-222 (patched)


Som unthrown exception declarations.



ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizerFactory.java
Lines 33 (patched)


The HiveAuthzPluginException is never thrown by the method. Also if you 
remove the exception the ```import 
org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthzPluginException;```
 can be removed.


- Laszlo Pinter


On Sept. 5, 2018, 5:27 p.m., Daniel Dai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68630/
> ---
> 
> (Updated Sept. 5, 2018, 5:27 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See HIVE-20420
> 
> 
> Diffs
> -
> 
>   ql/pom.xml a55cbe3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/SettableConfigUpdater.java
>  12be41c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizer.java
>  PRE-CREATION 
>   
>