[jira] [Created] (HIVE-23316) Add tests to cover database managed location related DDL and fix minor issues

2020-04-28 Thread Miklos Gergely (Jira) 
Miklos Gergely created HIVE-23316:
-

 Summary: Add tests to cover database managed location related DDL 
and fix minor issues
 Key: HIVE-23316
 URL: https://issues.apache.org/jira/browse/HIVE-23316
 Project: Hive
  Issue Type: Bug
Reporter: Miklos Gergely
Assignee: Miklos Gergely


Database managed location was recently introduced, but no tests were added to 
cover it. also the following issues were fixed:
 * ALTER DATABASE ... SET MANAGEDLOCATION ... commands were not handled in a 
separate path as it should, as in DDL each command type have their own 
Analyzer, Desc, and Operation class
 * in case of setting the LOCATION or the MANAGEDLOCATION the location was not 
getting qualified as in the CREATE DATABASE command
 * in case of setting the LOCATION or the MANAGEDLOCATION it was not checked if 
this modification makes the two the same
 * some minor checkstyle issues were fixed as well



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23315) Remove empty line from the end of SHOW EXTENDED TABLES and SHOW MATERIALIZED VIEWS

2020-04-28 Thread Miklos Gergely (Jira) 
Miklos Gergely created HIVE-23315:
-

 Summary: Remove empty line from the end of SHOW EXTENDED TABLES 
and SHOW MATERIALIZED VIEWS
 Key: HIVE-23315
 URL: https://issues.apache.org/jira/browse/HIVE-23315
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Miklos Gergely
Assignee: Miklos Gergely


At the end of each SHOW EXTENDED TABLES; and SHOW MATERIALIZED VIEWS; command 
there is an empty line like this:
{code:java}
+++
|  tab_name  |   table_type   |
+++
| sample_07  | MANAGED_TABLE  |
| sample_08  | MANAGED_TABLE  |
| web_logs   | MANAGED_TABLE  |
|            | NULL           |
+++
{code}
It should be removed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72436: Locks: Implement zero-wait readers

2020-04-28 Thread Denys Kuzmenko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72436/
---

(Updated April 28, 2020, 5:59 p.m.)


Review request for hive, Marton Bod and Peter Vary.


Bugs: HIVE-23293
https://issues.apache.org/jira/browse/HIVE-23293


Repository: hive-git


Description
---

With a new lock type (EXCL_WRITE) for INSERT_OVERWRITE, SHARED_READ does not 
have to wait for any lock - it can fails fast for a pending EXCLUSIVE, because 
even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no semantic 
reason to wait for them.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 8e643fe844 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbLockManager.java fb5a306ac4 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java b4dac4346e 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
f90396b2a3 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockRequest.java
 7402fb30eb 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockResponse.java
 fdaab4b394 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 9fb7ff011a 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 4f317b3453 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 e64ae0ead2 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
 93da0f60ec 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
1e3d6e9b8b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 fe39b0b36e 


Diff: https://reviews.apache.org/r/72436/diff/5/

Changes: https://reviews.apache.org/r/72436/diff/4-5/


Testing
---

DbTxnManager tests


Thanks,

Denys Kuzmenko

[jira] [Created] (HIVE-23314) Upgrade to Kudu 1.12

2020-04-28 Thread Zoltan Haindrich (Jira) 
Zoltan Haindrich created HIVE-23314:
---

 Summary: Upgrade to Kudu 1.12
 Key: HIVE-23314
 URL: https://issues.apache.org/jira/browse/HIVE-23314
 Project: Hive
  Issue Type: Sub-task
Reporter: Zoltan Haindrich


we need KUDU-3044 because it could cause random failures...



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72436: Locks: Implement zero-wait readers

2020-04-28 Thread Peter Vary via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72436/#review220526
---


Ship it!




Ship It!

- Peter Vary


On ápr. 28, 2020, 4:23 du, Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72436/
> ---
> 
> (Updated ápr. 28, 2020, 4:23 du)
> 
> 
> Review request for hive, Marton Bod and Peter Vary.
> 
> 
> Bugs: HIVE-23293
> https://issues.apache.org/jira/browse/HIVE-23293
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> With a new lock type (EXCL_WRITE) for INSERT_OVERWRITE, SHARED_READ does not 
> have to wait for any lock - it can fails fast for a pending EXCLUSIVE, 
> because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no 
> semantic reason to wait for them.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 8e643fe844 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java b4dac4346e 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> f90396b2a3 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockRequest.java
>  7402fb30eb 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockResponse.java
>  fdaab4b394 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  9fb7ff011a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  4f317b3453 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  e64ae0ead2 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
>  93da0f60ec 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 1e3d6e9b8b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  fe39b0b36e 
> 
> 
> Diff: https://reviews.apache.org/r/72436/diff/4/
> 
> 
> Testing
> ---
> 
> DbTxnManager tests
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Re: Review Request 72436: Locks: Implement zero-wait readers

2020-04-28 Thread Denys Kuzmenko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72436/
---

(Updated April 28, 2020, 4:23 p.m.)


Review request for hive, Marton Bod and Peter Vary.


Bugs: HIVE-23293
https://issues.apache.org/jira/browse/HIVE-23293


Repository: hive-git


Description
---

With a new lock type (EXCL_WRITE) for INSERT_OVERWRITE, SHARED_READ does not 
have to wait for any lock - it can fails fast for a pending EXCLUSIVE, because 
even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no semantic 
reason to wait for them.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 8e643fe844 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java b4dac4346e 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
f90396b2a3 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockRequest.java
 7402fb30eb 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockResponse.java
 fdaab4b394 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 9fb7ff011a 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 4f317b3453 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 e64ae0ead2 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
 93da0f60ec 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
1e3d6e9b8b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 fe39b0b36e 


Diff: https://reviews.apache.org/r/72436/diff/4/

Changes: https://reviews.apache.org/r/72436/diff/3-4/


Testing
---

DbTxnManager tests


Thanks,

Denys Kuzmenko

Re: Review Request 72436: Locks: Implement zero-wait readers

2020-04-28 Thread Denys Kuzmenko via Review Board 


> On April 28, 2020, 9:43 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
> > Lines 4353 (patched)
> > 
> >
> > why is this move?

for checking if we have blocked shared read.


> On April 28, 2020, 9:43 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
> > Lines 4389-4391 (patched)
> > 
> >
> > This is strange for me.
> > We do not abort the transaction, but throw an TxnAbortedException, but 
> > remove the Locks?

fixed


- Denys


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72436/#review220517
---


On April 28, 2020, 4:19 p.m., Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72436/
> ---
> 
> (Updated April 28, 2020, 4:19 p.m.)
> 
> 
> Review request for hive, Marton Bod and Peter Vary.
> 
> 
> Bugs: HIVE-23293
> https://issues.apache.org/jira/browse/HIVE-23293
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> With a new lock type (EXCL_WRITE) for INSERT_OVERWRITE, SHARED_READ does not 
> have to wait for any lock - it can fails fast for a pending EXCLUSIVE, 
> because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no 
> semantic reason to wait for them.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 8e643fe844 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java b4dac4346e 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> f90396b2a3 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockRequest.java
>  7402fb30eb 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockResponse.java
>  fdaab4b394 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  9fb7ff011a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  4f317b3453 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  e64ae0ead2 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
>  93da0f60ec 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 1e3d6e9b8b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  fe39b0b36e 
> 
> 
> Diff: https://reviews.apache.org/r/72436/diff/3/
> 
> 
> Testing
> ---
> 
> DbTxnManager tests
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Re: Review Request 72436: Locks: Implement zero-wait readers

2020-04-28 Thread Denys Kuzmenko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72436/
---

(Updated April 28, 2020, 4:19 p.m.)


Review request for hive, Marton Bod and Peter Vary.


Bugs: HIVE-23293
https://issues.apache.org/jira/browse/HIVE-23293


Repository: hive-git


Description
---

With a new lock type (EXCL_WRITE) for INSERT_OVERWRITE, SHARED_READ does not 
have to wait for any lock - it can fails fast for a pending EXCLUSIVE, because 
even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no semantic 
reason to wait for them.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 8e643fe844 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java b4dac4346e 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
f90396b2a3 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockRequest.java
 7402fb30eb 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockResponse.java
 fdaab4b394 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 9fb7ff011a 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 4f317b3453 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 e64ae0ead2 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
 93da0f60ec 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
1e3d6e9b8b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 fe39b0b36e 


Diff: https://reviews.apache.org/r/72436/diff/3/

Changes: https://reviews.apache.org/r/72436/diff/2-3/


Testing
---

DbTxnManager tests


Thanks,

Denys Kuzmenko

Re: Review Request 72388: HIVE-23048 Use sequences for TXN_ID generation

2020-04-28 Thread Peter Varga via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72388/
---

(Updated April 28, 2020, 3:38 p.m.)


Review request for hive and Peter Vary.


Repository: hive-git


Description
---

* Use sequences for TXN_ID generation.
  * Make it possible to open transactions in parallel
  * drop Oracle 11g support


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java b4dac4346e 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 37a5862791 
  
ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java 
7c8903fbae 
  ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
3916e88a9d 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 2c13e8dd03 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java 51b0fa336f 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java 
6525ffc00a 
  ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 1435269ed3 
  
ql/src/test/org/apache/hadoop/hive/ql/lockmgr/DbTxnManagerEndToEndTestBase.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/ITestDbTxnManager.java 
a085e9ff6f 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
f90396b2a3 
  
ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManagerIsolationProperties.java
 PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator.java 
1151466f8c 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 a874121e12 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/SQLGenerator.java
 49b737ecf9 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
 2344c2d5f6 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java
 97a083399a 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 fe39b0b36e 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java
 87130a519d 
  
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
 366b6f02c1 
  
standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
 8a3cd56658 
  
standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
 2e0177723d 
  
standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
 9f3951575b 
  
standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
 0512a45cad 
  
standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
 4b82e36ab4 
  
standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
 db398e5f66 
  
standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
 1be83fc4a9 
  
standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
 e6e30160af 
  
standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
 b90cecb173 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/DbInstallBase.java
 c1a1629548 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/rules/DatabaseRule.java
 a6d22d19ef 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/rules/Oracle.java
 0b070e19ac 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/txn/TestOpenTxn.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/72388/diff/6/

Changes: https://reviews.apache.org/r/72388/diff/5-6/


Testing
---


Thanks,

Peter Varga

Re: Review Request 72388: HIVE-23048 Use sequences for TXN_ID generation

2020-04-28 Thread Peter Varga via Review Board 


> On April 27, 2020, 3:04 p.m., Peter Vary wrote:
> > Fix it and ship it

Thank you for the reviews, fixed the issues and got a green run.


> On April 27, 2020, 3:04 p.m., Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
> > Lines 76 (patched)
> > 
> >
> > TXN_ID should be 10-1

I left the max committed insert and inserted the 10 value this way if 
some administrator want to fix the db manually after the schematool run, it 
will be easier.


> On April 27, 2020, 3:04 p.m., Peter Vary wrote:
> > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/rules/Oracle.java
> > Line 27 (original), 27 (patched)
> > 
> >
> > Is this public?

Yes it is.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72388/#review220500
---


On April 24, 2020, 3:40 p.m., Peter Varga wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72388/
> ---
> 
> (Updated April 24, 2020, 3:40 p.m.)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> * Use sequences for TXN_ID generation.
>   * Make it possible to open transactions in parallel
>   * drop Oracle 11g support
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 37a5862791 
>   
> ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java
>  15fcfc0e35 
>   ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
> f512c1df19 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 2c13e8dd03 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java 51b0fa336f 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java 
> 6525ffc00a 
>   ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 
> 1435269ed3 
>   
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/DbTxnManagerEndToEndTestBase.java
>  PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 497cedd61f 
>   
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManagerIsolationProperties.java
>  PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator.java 
> 1151466f8c 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  a874121e12 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/SQLGenerator.java
>  49b737ecf9 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
>  2344c2d5f6 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java
>  385f9d72cd 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  4a6fa6f620 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java
>  87130a519d 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
>  1ace9d3ef0 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
>  8a3cd56658 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
>  2e0177723d 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
>  9f3951575b 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
>  0512a45cad 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
>  4b82e36ab4 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
>  db398e5f66 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
>  1be83fc4a9 
>   
> standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
>  e6e30160af 
>   
> standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
>  b90cecb173 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/DbInstallBase.java
>  c1a1629548 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/rules/DatabaseRule.java
>  a6d22d19ef 
>   
>

[jira] [Created] (HIVE-23313) Reorganize sketch functions registration logic

2020-04-28 Thread Zoltan Haindrich (Jira) 
Zoltan Haindrich created HIVE-23313:
---

 Summary: Reorganize sketch functions registration logic
 Key: HIVE-23313
 URL: https://issues.apache.org/jira/browse/HIVE-23313
 Project: Hive
  Issue Type: Sub-task
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


* change the procdeure based registration into an enum driven one
* also change the function names into an enum




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23312) Update last event ID more efficiently during incremental run

2020-04-28 Thread PRAVIN KUMAR SINHA (Jira) 
PRAVIN KUMAR SINHA created HIVE-23312:
-

 Summary: Update last event ID more efficiently during incremental 
run
 Key: HIVE-23312
 URL: https://issues.apache.org/jira/browse/HIVE-23312
 Project: Hive
  Issue Type: Improvement
Reporter: PRAVIN KUMAR SINHA
Assignee: PRAVIN KUMAR SINHA


Should be done for a batch of events and/or in asynchronous way.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23311) Fix ValidTxnManager regression

2020-04-28 Thread Peter Varga (Jira) 
Peter Varga created HIVE-23311:
--

 Summary: Fix ValidTxnManager regression
 Key: HIVE-23311
 URL: https://issues.apache.org/jira/browse/HIVE-23311
 Project: Hive
  Issue Type: Bug
  Components: Locking
Reporter: Peter Varga
Assignee: Peter Varga


During query execution if there are only shared lock tables, the txnList in the 
driverContext  should always be considered valid.  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23310) Add .asf.yaml

2020-04-28 Thread Zoltan Haindrich (Jira) 
Zoltan Haindrich created HIVE-23310:
---

 Summary: Add .asf.yaml
 Key: HIVE-23310
 URL: https://issues.apache.org/jira/browse/HIVE-23310
 Project: Hive
  Issue Type: Improvement
  Components: Build Infrastructure
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


Earlier we had github pr-s automatically linked to jiras/etc - that feature 
suddenly stopped working, because some improvements were made by Infra; so that 
we can configure these stuff thru a .asf.yaml file

INFRA-20177



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72423: HIVE-23089

2020-04-28 Thread Krisztian Kasa 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72423/
---

(Updated April 28, 2020, 12:40 p.m.)


Review request for hive, Jesús Camacho Rodríguez and Vineet Garg.


Bugs: HIVE-23089
https://issues.apache.org/jira/browse/HIVE-23089


Repository: hive-git


Description
---

Add constraint checks to CBO plan


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java f94a9f9d70 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 0de3730351 
  ql/src/java/org/apache/hadoop/hive/ql/parse/type/ConstraintExprGenerator.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/type/ExprNodeTypeCheck.java 
3e3d331412 
  ql/src/java/org/apache/hadoop/hive/ql/parse/type/RexNodeTypeCheck.java 
c9131ec018 
  ql/src/test/results/clientnegative/update_notnull_constraint.q.out 32905378e7 
  ql/src/test/results/clientpositive/llap/check_constraint.q.out bc5d361859 
  ql/src/test/results/clientpositive/llap/default_constraint.q.out a04da4e2b4 
  ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
e96363fe29 


Diff: https://reviews.apache.org/r/72423/diff/5/

Changes: https://reviews.apache.org/r/72423/diff/4-5/


Testing
---

mvn test -Dtest.output.overwrite -DskipSparkTests 
-Dtest=TestMiniLlapLocalCliDriver -Dqfile=check_constraint.q,sort_acid.q -pl 
itests/qtest -Pitests
mvn test -Dtest.output.overwrite -DskipSparkTests -Dtest=TestNegativeCliDriver 
-Dqfile=update_notnull_constraint.q -pl itests/qtest -Pitests


Thanks,

Krisztian Kasa

Re: Review Request 72444: HIVE-23280: Trigger compaction with old aborted txns

2020-04-28 Thread Peter Vary via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72444/#review220523
---


Ship it!




Ship It!

- Peter Vary


On ápr. 28, 2020, 10:37 de, Karen Coppage wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72444/
> ---
> 
> (Updated ápr. 28, 2020, 10:37 de)
> 
> 
> Review request for hive, Laszlo Pinter and Peter Vary.
> 
> 
> Bugs: HIVE-23280
> https://issues.apache.org/jira/browse/HIVE-23280
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> When a txn is aborted and the compaction threshold for number of aborted txns 
> is not reached then the aborted transaction can remain forever in the RDBMS 
> database. This could result in several serious performance degradations:
> 
> getOpenTxns has to list this aborted txn forever
> TXN_TO_WRITE_ID table is not cleaned
> We should add a threshold, so after a given time the compaction is started 
> anyway.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e3ddbf197b 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 37a5862791 
>   
> ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java
>  15fcfc0e35 
>   ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator.java 
> 1151466f8c 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java
>  31b6ed450b 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  9fb7ff011a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  4f317b3453 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  e64ae0ead2 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 1e3d6e9b8b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
>  70d63ab18b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
>  2344c2d5f6 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java
>  87130a519d 
> 
> 
> Diff: https://reviews.apache.org/r/72444/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Karen Coppage
> 
>

Re: Review Request 72444: HIVE-23280: Trigger compaction with old aborted txns

2020-04-28 Thread Peter Vary via Review Board 


> On ápr. 28, 2020, 8:54 de, Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
> > Lines 113 (patched)
> > 
> >
> > Do we know for sure, that for null partition that this query is working 
> > for all of the supported databases?
> 
> Karen Coppage wrote:
> Group by partition was already present, I just removed a conditional 
> clause and added 2 columns to the projection without touching partition 
> handling. Do you think it's worth testing in each supported db even for these 
> changes?

Ok. I missed that, and thought that it is a more throughout change.
No need for extra tests then.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72444/#review220514
---


On ápr. 28, 2020, 10:37 de, Karen Coppage wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72444/
> ---
> 
> (Updated ápr. 28, 2020, 10:37 de)
> 
> 
> Review request for hive, Laszlo Pinter and Peter Vary.
> 
> 
> Bugs: HIVE-23280
> https://issues.apache.org/jira/browse/HIVE-23280
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> When a txn is aborted and the compaction threshold for number of aborted txns 
> is not reached then the aborted transaction can remain forever in the RDBMS 
> database. This could result in several serious performance degradations:
> 
> getOpenTxns has to list this aborted txn forever
> TXN_TO_WRITE_ID table is not cleaned
> We should add a threshold, so after a given time the compaction is started 
> anyway.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e3ddbf197b 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 37a5862791 
>   
> ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java
>  15fcfc0e35 
>   ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator.java 
> 1151466f8c 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java
>  31b6ed450b 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  9fb7ff011a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  4f317b3453 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  e64ae0ead2 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 1e3d6e9b8b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
>  70d63ab18b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
>  2344c2d5f6 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java
>  87130a519d 
> 
> 
> Diff: https://reviews.apache.org/r/72444/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Karen Coppage
> 
>

[jira] [Created] (HIVE-23309) Lazy Initialization of Hadoop Shims

2020-04-28 Thread Aasha Medhi (Jira) 
Aasha Medhi created HIVE-23309:
--

 Summary: Lazy Initialization of Hadoop Shims
 Key: HIVE-23309
 URL: https://issues.apache.org/jira/browse/HIVE-23309
 Project: Hive
  Issue Type: Bug
Reporter: Aasha Medhi
Assignee: Aasha Medhi


Initialize hadoop-shims only if CM is enabled



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72444: HIVE-23280: Trigger compaction with old aborted txns

2020-04-28 Thread Karen Coppage via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72444/
---

(Updated April 28, 2020, 10:37 a.m.)


Review request for hive, Laszlo Pinter and Peter Vary.


Changes
---

Patch 4, addressed review comments


Bugs: HIVE-23280
https://issues.apache.org/jira/browse/HIVE-23280


Repository: hive-git


Description
---

When a txn is aborted and the compaction threshold for number of aborted txns 
is not reached then the aborted transaction can remain forever in the RDBMS 
database. This could result in several serious performance degradations:

getOpenTxns has to list this aborted txn forever
TXN_TO_WRITE_ID table is not cleaned
We should add a threshold, so after a given time the compaction is started 
anyway.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e3ddbf197b 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 37a5862791 
  
ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java 
15fcfc0e35 
  ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator.java 
1151466f8c 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java
 31b6ed450b 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 9fb7ff011a 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 4f317b3453 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 e64ae0ead2 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
1e3d6e9b8b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
 70d63ab18b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
 2344c2d5f6 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java
 87130a519d 


Diff: https://reviews.apache.org/r/72444/diff/2/

Changes: https://reviews.apache.org/r/72444/diff/1-2/


Testing
---


Thanks,

Karen Coppage

Re: Review Request 72444: HIVE-23280: Trigger compaction with old aborted txns

2020-04-28 Thread Karen Coppage via Review Board 


> On April 28, 2020, 8:54 a.m., Peter Vary wrote:
> > Thanks for the patch Karen!
> > Few questions below.
> > 
> > Thanks,
> > Peter

Thanks for the review, Peter! Questions are addressed below.


> On April 28, 2020, 8:54 a.m., Peter Vary wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Lines 2852 (patched)
> > 
> >
> > Can we add a comment about valid values, and how to turn this off?

Done


> On April 28, 2020, 8:54 a.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java
> > Lines 92 (patched)
> > 
> >
> > Can we check the valid values, and can we turn this function off?

We can turn it off by setting to a negative number, which used to be checked at 
org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler#isPastAbortedTimeThreshold
 and is now checked before the query to metastore is executed.
Valid values are checked by org.apache.hadoop.hive.conf.Validator.TimeValidator.


> On April 28, 2020, 8:54 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
> > Lines 113 (patched)
> > 
> >
> > Do we know for sure, that for null partition that this query is working 
> > for all of the supported databases?

Group by partition was already present, I just removed a conditional clause and 
added 2 columns to the projection without touching partition handling. Do you 
think it's worth testing in each supported db even for these changes?


> On April 28, 2020, 8:54 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
> > Lines 151 (patched)
> > 
> >
> > Why Instant? I usually use System.currentTimeMillis(). Also it might be 
> > worth to get it only once, and not for every compactionInfo

Ok, will do. (I thought wrongly that System.currentTimeMillis was somehow 
timezone dependendent)


- Karen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72444/#review220514
---


On April 28, 2020, 8:39 a.m., Karen Coppage wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72444/
> ---
> 
> (Updated April 28, 2020, 8:39 a.m.)
> 
> 
> Review request for hive, Laszlo Pinter and Peter Vary.
> 
> 
> Bugs: HIVE-23280
> https://issues.apache.org/jira/browse/HIVE-23280
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> When a txn is aborted and the compaction threshold for number of aborted txns 
> is not reached then the aborted transaction can remain forever in the RDBMS 
> database. This could result in several serious performance degradations:
> 
> getOpenTxns has to list this aborted txn forever
> TXN_TO_WRITE_ID table is not cleaned
> We should add a threshold, so after a given time the compaction is started 
> anyway.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e3ddbf197b 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 37a5862791 
>   
> ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java
>  15fcfc0e35 
>   ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator.java 
> 1151466f8c 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java
>  31b6ed450b 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  9fb7ff011a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  4f317b3453 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  e64ae0ead2 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 1e3d6e9b8b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
>  70d63ab18b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
>  2344c2d5f6 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java
>  87130a519d 
> 
> 
> Diff: https://reviews.apache.org/r/72444/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Karen Coppage
> 
>

Re: Review Request 72436: Locks: Implement zero-wait readers

2020-04-28 Thread Peter Vary via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72436/#review220517
---



Quick quersions.
Thanks for the patch,
Peter


standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Lines 4353 (patched)


why is this move?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Lines 4389-4391 (patched)


This is strange for me.
We do not abort the transaction, but throw an TxnAbortedException, but 
remove the Locks?


- Peter Vary


On ápr. 27, 2020, 11:24 de, Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72436/
> ---
> 
> (Updated ápr. 27, 2020, 11:24 de)
> 
> 
> Review request for hive, Marton Bod and Peter Vary.
> 
> 
> Bugs: HIVE-23293
> https://issues.apache.org/jira/browse/HIVE-23293
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> With a new lock type (EXCL_WRITE) for INSERT_OVERWRITE, SHARED_READ does not 
> have to wait for any lock - it can fails fast for a pending EXCLUSIVE, 
> because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no 
> semantic reason to wait for them.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java b4dac4346e 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 497cedd61f 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockRequest.java
>  7402fb30eb 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  9fb7ff011a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  4f317b3453 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  e64ae0ead2 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
>  93da0f60ec 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 1e3d6e9b8b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  580786832e 
> 
> 
> Diff: https://reviews.apache.org/r/72436/diff/2/
> 
> 
> Testing
> ---
> 
> DbTxnManager tests
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Re: Review Request 72444: HIVE-23280: Trigger compaction with old aborted txns

2020-04-28 Thread Peter Vary via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72444/#review220514
---



Thanks for the patch Karen!
Few questions below.

Thanks,
Peter


common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 2852 (patched)


Can we add a comment about valid values, and how to turn this off?



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java
Lines 92 (patched)


Can we check the valid values, and can we turn this function off?



ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator.java
Lines 262 (patched)


Could we add a check that check that we do not start compaction before the 
threshold?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
Lines 113 (patched)


Do we know for sure, that for null partition that this query is working for 
all of the supported databases?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
Lines 151 (patched)


Why Instant? I usually use System.currentTimeMillis(). Also it might be 
worth to get it only once, and not for every compactionInfo



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
Lines 153 (patched)


Maybe single statement return, like:
return firstAbortedTxnTime + abortedTimeThreshold < 
System.currentTimeMillis()


- Peter Vary


On ápr. 28, 2020, 8:39 de, Karen Coppage wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72444/
> ---
> 
> (Updated ápr. 28, 2020, 8:39 de)
> 
> 
> Review request for hive, Laszlo Pinter and Peter Vary.
> 
> 
> Bugs: HIVE-23280
> https://issues.apache.org/jira/browse/HIVE-23280
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> When a txn is aborted and the compaction threshold for number of aborted txns 
> is not reached then the aborted transaction can remain forever in the RDBMS 
> database. This could result in several serious performance degradations:
> 
> getOpenTxns has to list this aborted txn forever
> TXN_TO_WRITE_ID table is not cleaned
> We should add a threshold, so after a given time the compaction is started 
> anyway.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e3ddbf197b 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 37a5862791 
>   
> ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java
>  15fcfc0e35 
>   ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator.java 
> 1151466f8c 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java
>  31b6ed450b 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  9fb7ff011a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  4f317b3453 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  e64ae0ead2 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 1e3d6e9b8b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
>  70d63ab18b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
>  2344c2d5f6 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java
>  87130a519d 
> 
> 
> Diff: https://reviews.apache.org/r/72444/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Karen Coppage
> 
>

Re: Review Request 72444: HIVE-23280: Trigger compaction with old aborted txns

2020-04-28 Thread Karen Coppage via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72444/
---

(Updated April 28, 2020, 8:39 a.m.)


Review request for hive, Laszlo Pinter and Peter Vary.


Bugs: HIVE-23280
https://issues.apache.org/jira/browse/HIVE-23280


Repository: hive-git


Description
---

When a txn is aborted and the compaction threshold for number of aborted txns 
is not reached then the aborted transaction can remain forever in the RDBMS 
database. This could result in several serious performance degradations:

getOpenTxns has to list this aborted txn forever
TXN_TO_WRITE_ID table is not cleaned
We should add a threshold, so after a given time the compaction is started 
anyway.


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e3ddbf197b 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 37a5862791 
  
ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java 
15fcfc0e35 
  ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator.java 
1151466f8c 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java
 31b6ed450b 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 9fb7ff011a 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 4f317b3453 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 e64ae0ead2 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
1e3d6e9b8b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
 70d63ab18b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
 2344c2d5f6 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java
 87130a519d 


Diff: https://reviews.apache.org/r/72444/diff/1/


Testing
---


Thanks,

Karen Coppage