Re: Review Request 72387: Locks: Add new lock implementations for always zero-wait readers

2020-04-22 Thread Denys Kuzmenko via Review Board 


> On April 22, 2020, 3:22 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/ValidTxnManager.java
> > Line 113 (original), 112 (patched)
> > 
> >
> > Isn't reference level check is risky here?

good catch, need to reorder :)


- Denys


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/#review220426
---


On April 22, 2020, 2:29 p.m., Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72387/
> ---
> 
> (Updated April 22, 2020, 2:29 p.m.)
> 
> 
> Review request for hive, Marton Bod and Peter Vary.
> 
> 
> Bugs: HIVE-19369
> https://issues.apache.org/jira/browse/HIVE-19369
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Hive Locking with Micro-managed and full-ACID tables needs a better locking 
> implementation which allows for no-wait readers always.
> 
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
> 
> Short write-up
> 
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to 
> exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to 
> exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an 
> EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different 
> threads).
> 
> SHARED_READ does not wait for any lock - it fails fast for a pending 
> EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, 
> there's no semantic reason to wait for them to succeed before going ahead 
> with a SHARED_WRITE.
> 
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16bae920df 
>   ql/src/java/org/apache/hadoop/hive/ql/ValidTxnManager.java 4885e437aa 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b 
>   ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
> 1d211857bf 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 73d3b91585 
>   ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f 
>   ql/src/test/results/clientpositive/llap/explain_locks.q.out 3183533478 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
>  22902a9c20 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  d080df417b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
>  f928bf781b 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java
>  3d69a6e7dc 
> 
> 
> Diff: https://reviews.apache.org/r/72387/diff/5/
> 
> 
> Testing
> ---
> 
> Added number of tests under TestDbTxnManager2, TestTxnHandler & extended 
> explain_locks.q test.
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Re: Review Request 72387: Locks: Add new lock implementations for always zero-wait readers

2020-04-22 Thread Denys Kuzmenko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/
---

(Updated April 22, 2020, 3:27 p.m.)


Review request for hive, Marton Bod and Peter Vary.


Bugs: HIVE-19369
https://issues.apache.org/jira/browse/HIVE-19369


Repository: hive-git


Description
---

Hive Locking with Micro-managed and full-ACID tables needs a better locking 
implementation which allows for no-wait readers always.

EXCL_DROP
EXCL_WRITE
SHARED_WRITE
SHARED_READ

Short write-up

EXCL_DROP is a "drop partition" or "drop table" and waits for all others to exit
EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to 
exit.
SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an 
EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different 
threads).

SHARED_READ does not wait for any lock - it fails fast for a pending EXCL_DROP, 
because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no 
semantic reason to wait for them to succeed before going ahead with a 
SHARED_WRITE.

a select * => SHARED_READ
an insert into => SHARED_WRITE
an insert overwrite or MERGE => EXCL_WRITE
a drop table => EXCL_DROP


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16bae920df 
  ql/src/java/org/apache/hadoop/hive/ql/ValidTxnManager.java 4885e437aa 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b 
  ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
1d211857bf 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
73d3b91585 
  ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f 
  ql/src/test/results/clientpositive/llap/explain_locks.q.out 3183533478 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
 22902a9c20 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 d080df417b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
 f928bf781b 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java
 3d69a6e7dc 


Diff: https://reviews.apache.org/r/72387/diff/6/

Changes: https://reviews.apache.org/r/72387/diff/5-6/


Testing
---

Added number of tests under TestDbTxnManager2, TestTxnHandler & extended 
explain_locks.q test.


Thanks,

Denys Kuzmenko

Re: Review Request 72387: Locks: Add new lock implementations for always zero-wait readers

2020-04-22 Thread Peter Vary via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/#review220426
---


Ship it!





ql/src/java/org/apache/hadoop/hive/ql/ValidTxnManager.java
Lines 96 (patched)


nit: new line?



ql/src/java/org/apache/hadoop/hive/ql/ValidTxnManager.java
Line 113 (original), 112 (patched)


Isn't reference level check is risky here?


- Peter Vary


On ápr. 22, 2020, 2:29 du, Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72387/
> ---
> 
> (Updated ápr. 22, 2020, 2:29 du)
> 
> 
> Review request for hive, Marton Bod and Peter Vary.
> 
> 
> Bugs: HIVE-19369
> https://issues.apache.org/jira/browse/HIVE-19369
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Hive Locking with Micro-managed and full-ACID tables needs a better locking 
> implementation which allows for no-wait readers always.
> 
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
> 
> Short write-up
> 
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to 
> exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to 
> exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an 
> EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different 
> threads).
> 
> SHARED_READ does not wait for any lock - it fails fast for a pending 
> EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, 
> there's no semantic reason to wait for them to succeed before going ahead 
> with a SHARED_WRITE.
> 
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16bae920df 
>   ql/src/java/org/apache/hadoop/hive/ql/ValidTxnManager.java 4885e437aa 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b 
>   ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
> 1d211857bf 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 73d3b91585 
>   ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f 
>   ql/src/test/results/clientpositive/llap/explain_locks.q.out 3183533478 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
>  22902a9c20 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  d080df417b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
>  f928bf781b 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java
>  3d69a6e7dc 
> 
> 
> Diff: https://reviews.apache.org/r/72387/diff/5/
> 
> 
> Testing
> ---
> 
> Added number of tests under TestDbTxnManager2, TestTxnHandler & extended 
> explain_locks.q test.
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Re: Review Request 72387: Locks: Add new lock implementations for always zero-wait readers

2020-04-22 Thread Denys Kuzmenko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/
---

(Updated April 22, 2020, 2:29 p.m.)


Review request for hive, Marton Bod and Peter Vary.


Changes
---

reorder checks & fixed isValidTxnListState


Bugs: HIVE-19369
https://issues.apache.org/jira/browse/HIVE-19369


Repository: hive-git


Description
---

Hive Locking with Micro-managed and full-ACID tables needs a better locking 
implementation which allows for no-wait readers always.

EXCL_DROP
EXCL_WRITE
SHARED_WRITE
SHARED_READ

Short write-up

EXCL_DROP is a "drop partition" or "drop table" and waits for all others to exit
EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to 
exit.
SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an 
EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different 
threads).

SHARED_READ does not wait for any lock - it fails fast for a pending EXCL_DROP, 
because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no 
semantic reason to wait for them to succeed before going ahead with a 
SHARED_WRITE.

a select * => SHARED_READ
an insert into => SHARED_WRITE
an insert overwrite or MERGE => EXCL_WRITE
a drop table => EXCL_DROP


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16bae920df 
  ql/src/java/org/apache/hadoop/hive/ql/ValidTxnManager.java 4885e437aa 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b 
  ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
1d211857bf 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
73d3b91585 
  ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f 
  ql/src/test/results/clientpositive/llap/explain_locks.q.out 3183533478 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
 22902a9c20 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 d080df417b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
 f928bf781b 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java
 3d69a6e7dc 


Diff: https://reviews.apache.org/r/72387/diff/5/

Changes: https://reviews.apache.org/r/72387/diff/4-5/


Testing
---

Added number of tests under TestDbTxnManager2, TestTxnHandler & extended 
explain_locks.q test.


Thanks,

Denys Kuzmenko

Re: Review Request 72387: Locks: Add new lock implementations for always zero-wait readers

2020-04-21 Thread Denys Kuzmenko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/
---

(Updated April 21, 2020, 12:14 p.m.)


Review request for hive, Marton Bod and Peter Vary.


Changes
---

added quotes


Bugs: HIVE-19369
https://issues.apache.org/jira/browse/HIVE-19369


Repository: hive-git


Description
---

Hive Locking with Micro-managed and full-ACID tables needs a better locking 
implementation which allows for no-wait readers always.

EXCL_DROP
EXCL_WRITE
SHARED_WRITE
SHARED_READ

Short write-up

EXCL_DROP is a "drop partition" or "drop table" and waits for all others to exit
EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to 
exit.
SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an 
EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different 
threads).

SHARED_READ does not wait for any lock - it fails fast for a pending EXCL_DROP, 
because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no 
semantic reason to wait for them to succeed before going ahead with a 
SHARED_WRITE.

a select * => SHARED_READ
an insert into => SHARED_WRITE
an insert overwrite or MERGE => EXCL_WRITE
a drop table => EXCL_DROP


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16bae920df 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b 
  ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
1d211857bf 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
73d3b91585 
  ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f 
  ql/src/test/results/clientpositive/llap/explain_locks.q.out 3183533478 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
 22902a9c20 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 d080df417b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
 f928bf781b 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java
 3d69a6e7dc 


Diff: https://reviews.apache.org/r/72387/diff/4/

Changes: https://reviews.apache.org/r/72387/diff/3-4/


Testing
---

Added number of tests under TestDbTxnManager2, TestTxnHandler & extended 
explain_locks.q test.


Thanks,

Denys Kuzmenko

Re: Review Request 72387: Locks: Add new lock implementations for always zero-wait readers

2020-04-20 Thread Denys Kuzmenko via Review Board 


> On April 20, 2020, 4:55 p.m., Peter Vary wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Lines 2732 (patched)
> > 
> >
> > Maybe rephrase?
> > Manages concurrency levels for ACID resoruces. Enables users enable 
> > parallel queries by enabling write-write conflict resolution happen only at 
> > commit phase 
> > - If true - no commit phase conflict resolution: 
> >- INSERT OVERWRITE requests EXCLUSIVE locks
> >- UPDATE/DELETE requests EXCL_WRITE lock
> >- INSERT requests SHARED_READ lock
> > - If false - write might fail when committed on conflict check: 
> >- INSERT OVERWRITE requests EXCL_WRITE locks
> >- UPDATE/DELETE/INSERT requests SHARED_READ lock

fixed


- Denys


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/#review220374
---


On April 20, 2020, 8:50 p.m., Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72387/
> ---
> 
> (Updated April 20, 2020, 8:50 p.m.)
> 
> 
> Review request for hive, Marton Bod and Peter Vary.
> 
> 
> Bugs: HIVE-19369
> https://issues.apache.org/jira/browse/HIVE-19369
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Hive Locking with Micro-managed and full-ACID tables needs a better locking 
> implementation which allows for no-wait readers always.
> 
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
> 
> Short write-up
> 
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to 
> exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to 
> exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an 
> EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different 
> threads).
> 
> SHARED_READ does not wait for any lock - it fails fast for a pending 
> EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, 
> there's no semantic reason to wait for them to succeed before going ahead 
> with a SHARED_WRITE.
> 
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16bae920df 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b 
>   ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
> 1d211857bf 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 73d3b91585 
>   ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f 
>   ql/src/test/results/clientpositive/llap/explain_locks.q.out 3183533478 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
>  22902a9c20 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  d080df417b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
>  f928bf781b 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java
>  3d69a6e7dc 
> 
> 
> Diff: https://reviews.apache.org/r/72387/diff/2/
> 
> 
> Testing
> ---
> 
> Added number of tests under TestDbTxnManager2, TestTxnHandler & extended 
> explain_locks.q test.
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Re: Review Request 72387: Locks: Add new lock implementations for always zero-wait readers

2020-04-20 Thread Denys Kuzmenko via Review Board 


> On April 20, 2020, 4:55 p.m., Peter Vary wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Lines 2727-2730 (original), 2727-2730 (patched)
> > 
> >
> > Is this config still needed in this case?

yes, in order to be backward compatible, when user goes with an exclusive 
writes and desides to lower the lock level for the insert overwrite from 
exclusive to exclusive write (insert overwrite can execute in parallel with 
insert - current functionality)


> On April 20, 2020, 4:55 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
> > Line 3027 (original), 3027-3028 (patched)
> > 
> >
> > Maybe get once, and store?

fixed


- Denys


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/#review220374
---


On April 20, 2020, 8:50 p.m., Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72387/
> ---
> 
> (Updated April 20, 2020, 8:50 p.m.)
> 
> 
> Review request for hive, Marton Bod and Peter Vary.
> 
> 
> Bugs: HIVE-19369
> https://issues.apache.org/jira/browse/HIVE-19369
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Hive Locking with Micro-managed and full-ACID tables needs a better locking 
> implementation which allows for no-wait readers always.
> 
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
> 
> Short write-up
> 
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to 
> exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to 
> exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an 
> EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different 
> threads).
> 
> SHARED_READ does not wait for any lock - it fails fast for a pending 
> EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, 
> there's no semantic reason to wait for them to succeed before going ahead 
> with a SHARED_WRITE.
> 
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16bae920df 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b 
>   ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
> 1d211857bf 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 73d3b91585 
>   ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f 
>   ql/src/test/results/clientpositive/llap/explain_locks.q.out 3183533478 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
>  22902a9c20 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  d080df417b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
>  f928bf781b 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java
>  3d69a6e7dc 
> 
> 
> Diff: https://reviews.apache.org/r/72387/diff/2/
> 
> 
> Testing
> ---
> 
> Added number of tests under TestDbTxnManager2, TestTxnHandler & extended 
> explain_locks.q test.
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Re: Review Request 72387: Locks: Add new lock implementations for always zero-wait readers

2020-04-20 Thread Denys Kuzmenko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/
---

(Updated April 20, 2020, 8:50 p.m.)


Review request for hive, Marton Bod and Peter Vary.


Bugs: HIVE-19369
https://issues.apache.org/jira/browse/HIVE-19369


Repository: hive-git


Description
---

Hive Locking with Micro-managed and full-ACID tables needs a better locking 
implementation which allows for no-wait readers always.

EXCL_DROP
EXCL_WRITE
SHARED_WRITE
SHARED_READ

Short write-up

EXCL_DROP is a "drop partition" or "drop table" and waits for all others to exit
EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to 
exit.
SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an 
EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different 
threads).

SHARED_READ does not wait for any lock - it fails fast for a pending EXCL_DROP, 
because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no 
semantic reason to wait for them to succeed before going ahead with a 
SHARED_WRITE.

a select * => SHARED_READ
an insert into => SHARED_WRITE
an insert overwrite or MERGE => EXCL_WRITE
a drop table => EXCL_DROP


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16bae920df 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b 
  ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
1d211857bf 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
73d3b91585 
  ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f 
  ql/src/test/results/clientpositive/llap/explain_locks.q.out 3183533478 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
 22902a9c20 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 d080df417b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
 f928bf781b 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java
 3d69a6e7dc 


Diff: https://reviews.apache.org/r/72387/diff/2/

Changes: https://reviews.apache.org/r/72387/diff/1-2/


Testing
---

Added number of tests under TestDbTxnManager2, TestTxnHandler & extended 
explain_locks.q test.


Thanks,

Denys Kuzmenko

Re: Review Request 72387: Locks: Add new lock implementations for always zero-wait readers

2020-04-20 Thread Peter Vary via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/#review220374
---



Some nits, did not checked all the unit tests changes. Anything I have to check 
particularly?


common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 2727-2730 (original), 2727-2730 (patched)


Is this config still needed in this case?



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 2732 (patched)


Maybe rephrase?
Manages concurrency levels for ACID resoruces. Enables users enable 
parallel queries by enabling write-write conflict resolution happen only at 
commit phase 
- If true - no commit phase conflict resolution: 
   - INSERT OVERWRITE requests EXCLUSIVE locks
   - UPDATE/DELETE requests EXCL_WRITE lock
   - INSERT requests SHARED_READ lock
- If false - write might fail when committed on conflict check: 
   - INSERT OVERWRITE requests EXCL_WRITE locks
   - UPDATE/DELETE/INSERT requests SHARED_READ lock



ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
Line 3027 (original), 3027-3028 (patched)


Maybe get once, and store?



ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java
Lines 758 (patched)


nit: remove spaces



ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java
Line 1142 (original), 1265 (patched)


nit: space too


- Peter Vary


On ápr. 20, 2020, 7:23 de, Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72387/
> ---
> 
> (Updated ápr. 20, 2020, 7:23 de)
> 
> 
> Review request for hive, Marton Bod and Peter Vary.
> 
> 
> Bugs: HIVE-19369
> https://issues.apache.org/jira/browse/HIVE-19369
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Hive Locking with Micro-managed and full-ACID tables needs a better locking 
> implementation which allows for no-wait readers always.
> 
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
> 
> Short write-up
> 
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to 
> exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to 
> exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an 
> EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different 
> threads).
> 
> SHARED_READ does not wait for any lock - it fails fast for a pending 
> EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, 
> there's no semantic reason to wait for them to succeed before going ahead 
> with a SHARED_WRITE.
> 
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7b3acad511 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b 
>   ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
> 1d211857bf 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 73d3b91585 
>   ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f 
>   ql/src/test/results/clientpositive/explain_locks.q.out 3183533478 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
>  22902a9c20 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  331fd4cc8d 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
>  f928bf781b 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java
>  3d69a6e7dc 
> 
> 
> Diff: https://reviews.apache.org/r/72387/diff/1/
> 
> 
> Testing
> ---
> 
> Added number of tests under TestDbTxnManager2, TestTxnHandler & extended 
> explain_locks.q test.
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Review Request 72387: Locks: Add new lock implementations for always zero-wait readers

2020-04-20 Thread Denys Kuzmenko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/
---

Review request for hive, Marton Bod and Peter Vary.


Bugs: HIVE-19369
https://issues.apache.org/jira/browse/HIVE-19369


Repository: hive-git


Description
---

Hive Locking with Micro-managed and full-ACID tables needs a better locking 
implementation which allows for no-wait readers always.

EXCL_DROP
EXCL_WRITE
SHARED_WRITE
SHARED_READ

Short write-up

EXCL_DROP is a "drop partition" or "drop table" and waits for all others to exit
EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to 
exit.
SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an 
EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different 
threads).

SHARED_READ does not wait for any lock - it fails fast for a pending EXCL_DROP, 
because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no 
semantic reason to wait for them to succeed before going ahead with a 
SHARED_WRITE.

a select * => SHARED_READ
an insert into => SHARED_WRITE
an insert overwrite or MERGE => EXCL_WRITE
a drop table => EXCL_DROP


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7b3acad511 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b 
  ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
1d211857bf 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
73d3b91585 
  ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f 
  ql/src/test/results/clientpositive/explain_locks.q.out 3183533478 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
 22902a9c20 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 331fd4cc8d 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
 f928bf781b 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java
 3d69a6e7dc 


Diff: https://reviews.apache.org/r/72387/diff/1/


Testing
---

Added number of tests under TestDbTxnManager2, TestTxnHandler & extended 
explain_locks.q test.


Thanks,

Denys Kuzmenko