[jira] [Created] (HIVE-23262) Remove dependency on activemq

[Ashutosh Chauhan (Jira)]

Ashutosh Chauhan created HIVE-23262:
---

 Summary: Remove dependency on activemq
 Key: HIVE-23262
 URL: https://issues.apache.org/jira/browse/HIVE-23262
 Project: Hive
  Issue Type: Improvement
  Components: HCatalog
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan


Activemq is a test dependency introduced to test message bus feature in 
hcatalog. Even when it was introduced very first, there were concerns of taking 
activemq as a dependency. 
https://issues.apache.org/jira/browse/HCATALOG-3?focusedCommentId=13035113=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13035113
AFAIK no one uses message bus feature of HCatalog. We can remove it altogether. 
As a first step removing tests for it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23261) Check whether encryption is enabled in the cluster before moving files

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-23261:
---

 Summary: Check whether encryption is enabled in the cluster before 
moving files
 Key: HIVE-23261
 URL: https://issues.apache.org/jira/browse/HIVE-23261
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


Similar to HIVE-23212, there is an unwanted check of encryption paths during 
file move operation.

[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L4546]

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23260) Add support for unmodified_metadata capability

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-23260:


 Summary: Add support for unmodified_metadata capability
 Key: HIVE-23260
 URL: https://issues.apache.org/jira/browse/HIVE-23260
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


Currently, the translator removes bucketing info for tables for clients that do 
not possess the HIVEBUCKET2 capability. While this is desirable, some clients 
that have write access to these tables can turn around overwrite the metadata 
thus corrupting original bucketing info.

So adding support for a capability for client that are capable of interpreting 
the original metadata would prevent such corruption.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72387: Locks: Add new lock implementations for always zero-wait readers

[Denys Kuzmenko via Review Board]

> On April 20, 2020, 4:55 p.m., Peter Vary wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Lines 2732 (patched)
> > 
> >
> > Maybe rephrase?
> > Manages concurrency levels for ACID resoruces. Enables users enable 
> > parallel queries by enabling write-write conflict resolution happen only at 
> > commit phase 
> > - If true - no commit phase conflict resolution: 
> >- INSERT OVERWRITE requests EXCLUSIVE locks
> >- UPDATE/DELETE requests EXCL_WRITE lock
> >- INSERT requests SHARED_READ lock
> > - If false - write might fail when committed on conflict check: 
> >- INSERT OVERWRITE requests EXCL_WRITE locks
> >- UPDATE/DELETE/INSERT requests SHARED_READ lock

fixed


- Denys


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/#review220374
---


On April 20, 2020, 8:50 p.m., Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72387/
> ---
> 
> (Updated April 20, 2020, 8:50 p.m.)
> 
> 
> Review request for hive, Marton Bod and Peter Vary.
> 
> 
> Bugs: HIVE-19369
> https://issues.apache.org/jira/browse/HIVE-19369
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Hive Locking with Micro-managed and full-ACID tables needs a better locking 
> implementation which allows for no-wait readers always.
> 
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
> 
> Short write-up
> 
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to 
> exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to 
> exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an 
> EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different 
> threads).
> 
> SHARED_READ does not wait for any lock - it fails fast for a pending 
> EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, 
> there's no semantic reason to wait for them to succeed before going ahead 
> with a SHARED_WRITE.
> 
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16bae920df 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b 
>   ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
> 1d211857bf 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 73d3b91585 
>   ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f 
>   ql/src/test/results/clientpositive/llap/explain_locks.q.out 3183533478 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
>  22902a9c20 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  d080df417b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
>  f928bf781b 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java
>  3d69a6e7dc 
> 
> 
> Diff: https://reviews.apache.org/r/72387/diff/2/
> 
> 
> Testing
> ---
> 
> Added number of tests under TestDbTxnManager2, TestTxnHandler & extended 
> explain_locks.q test.
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Re: Review Request 72387: Locks: Add new lock implementations for always zero-wait readers

[Denys Kuzmenko via Review Board]

> On April 20, 2020, 4:55 p.m., Peter Vary wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Lines 2727-2730 (original), 2727-2730 (patched)
> > 
> >
> > Is this config still needed in this case?

yes, in order to be backward compatible, when user goes with an exclusive 
writes and desides to lower the lock level for the insert overwrite from 
exclusive to exclusive write (insert overwrite can execute in parallel with 
insert - current functionality)


> On April 20, 2020, 4:55 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
> > Line 3027 (original), 3027-3028 (patched)
> > 
> >
> > Maybe get once, and store?

fixed


- Denys


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/#review220374
---


On April 20, 2020, 8:50 p.m., Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72387/
> ---
> 
> (Updated April 20, 2020, 8:50 p.m.)
> 
> 
> Review request for hive, Marton Bod and Peter Vary.
> 
> 
> Bugs: HIVE-19369
> https://issues.apache.org/jira/browse/HIVE-19369
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Hive Locking with Micro-managed and full-ACID tables needs a better locking 
> implementation which allows for no-wait readers always.
> 
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
> 
> Short write-up
> 
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to 
> exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to 
> exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an 
> EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different 
> threads).
> 
> SHARED_READ does not wait for any lock - it fails fast for a pending 
> EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, 
> there's no semantic reason to wait for them to succeed before going ahead 
> with a SHARED_WRITE.
> 
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16bae920df 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b 
>   ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
> 1d211857bf 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 73d3b91585 
>   ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f 
>   ql/src/test/results/clientpositive/llap/explain_locks.q.out 3183533478 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
>  22902a9c20 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  d080df417b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
>  f928bf781b 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java
>  3d69a6e7dc 
> 
> 
> Diff: https://reviews.apache.org/r/72387/diff/2/
> 
> 
> Testing
> ---
> 
> Added number of tests under TestDbTxnManager2, TestTxnHandler & extended 
> explain_locks.q test.
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Re: Review Request 72387: Locks: Add new lock implementations for always zero-wait readers

[Denys Kuzmenko via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/
---

(Updated April 20, 2020, 8:50 p.m.)


Review request for hive, Marton Bod and Peter Vary.


Bugs: HIVE-19369
https://issues.apache.org/jira/browse/HIVE-19369


Repository: hive-git


Description
---

Hive Locking with Micro-managed and full-ACID tables needs a better locking 
implementation which allows for no-wait readers always.

EXCL_DROP
EXCL_WRITE
SHARED_WRITE
SHARED_READ

Short write-up

EXCL_DROP is a "drop partition" or "drop table" and waits for all others to exit
EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to 
exit.
SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an 
EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different 
threads).

SHARED_READ does not wait for any lock - it fails fast for a pending EXCL_DROP, 
because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no 
semantic reason to wait for them to succeed before going ahead with a 
SHARED_WRITE.

a select * => SHARED_READ
an insert into => SHARED_WRITE
an insert overwrite or MERGE => EXCL_WRITE
a drop table => EXCL_DROP


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16bae920df 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b 
  ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
1d211857bf 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
73d3b91585 
  ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f 
  ql/src/test/results/clientpositive/llap/explain_locks.q.out 3183533478 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
 22902a9c20 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 d080df417b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
 f928bf781b 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java
 3d69a6e7dc 


Diff: https://reviews.apache.org/r/72387/diff/2/

Changes: https://reviews.apache.org/r/72387/diff/1-2/


Testing
---

Added number of tests under TestDbTxnManager2, TestTxnHandler & extended 
explain_locks.q test.


Thanks,

Denys Kuzmenko

[jira] [Created] (HIVE-23259) Fix Apache Rat validation

[Jira]

Ismaël Mejía created HIVE-23259:
---

 Summary: Fix Apache Rat validation 
 Key: HIVE-23259
 URL: https://issues.apache.org/jira/browse/HIVE-23259
 Project: Hive
  Issue Type: Bug
  Components: Build Infrastructure
Reporter: Ismaël Mejía
Assignee: Ismaël Mejía


The headers for two files merged as part of HIVE-22750 did not include the 
Apache headers, this issue is to add it and unlock the validation to pass.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72379: "get_splits" udf ignores limit constraint when creating splits

[Adesh Rao]

> On April 20, 2020, 2:19 a.m., Sankar Hariappan wrote:
> > itests/hive-unit/src/test/java/org/apache/hive/jdbc/AbstractTestJdbcGenericUDTFGetSplits.java
> > Lines 181 (patched)
> > 
> >
> > Add test for limit with predicates, groupby queries as well. Also, make 
> > sure, we have partitioned table too.

Added queries with predicates and groupby. Since the existing test suite does 
not have partitioned table, need to go through other tests to figure out the 
standard way to generate a partitioned table.


> On April 20, 2020, 2:19 a.m., Sankar Hariappan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java
> > Line 240 (original), 241 (patched)
> > 
> >
> > Do we need this computation? Can't we just use forceSingleSplit instead 
> > of generateSingleSplit?

Yes, we don't need this for limit query. Removed.


> On April 20, 2020, 2:19 a.m., Sankar Hariappan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java
> > Line 259 (original), 260 (patched)
> > 
> >
> > Error message to be updated for limit query too.

Not required, as not generating single split for limit queries.


> On April 20, 2020, 2:19 a.m., Sankar Hariappan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java
> > Lines 344 (patched)
> > 
> >
> > This issue should be there even for "order by" queries.
> > How it work there?

By table, I meant the original table. The materialized table will have a single 
file only.


> On April 20, 2020, 2:19 a.m., Sankar Hariappan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java
> > Lines 347 (patched)
> > 
> >
> > Since, we always materialize the results for limit query, I think, 
> > forceSingleSplit logic is not needed. Even, if we generate multiple splits 
> > on this temp table, the eventual results will have only expected number of 
> > rows.

Ack. modified the approach to not generate single split.


- Adesh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72379/#review220363
---


On April 20, 2020, 6:10 p.m., Adesh Rao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72379/
> ---
> 
> (Updated April 20, 2020, 6:10 p.m.)
> 
> 
> Review request for hive and Sankar Hariappan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Since limit constraint was ignored while creating splits. If multiple llap 
> daemons execute splits, output contains more rows than specified by limit 
> constraint.
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/AbstractTestJdbcGenericUDTFGetSplits.java
>  8cbca69737 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcGenericUDTFGetSplits.java
>  defbe78802 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcGenericUDTFGetSplits2.java
>  330174513c 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
> 00a6c89b1e 
> 
> 
> Diff: https://reviews.apache.org/r/72379/diff/2/
> 
> 
> Testing
> ---
> 
> Waiting for Hive QA run.
> 
> 
> Thanks,
> 
> Adesh Rao
> 
>

Re: Review Request 72379: "get_splits" udf ignores limit constraint when creating splits

[Adesh Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72379/
---

(Updated April 20, 2020, 6:10 p.m.)


Review request for hive and Sankar Hariappan.


Changes
---

Not forcing single split generation for limit queries


Summary (updated)
-

"get_splits" udf ignores limit constraint when creating splits


Repository: hive-git


Description (updated)
---

Since limit constraint was ignored while creating splits. If multiple llap 
daemons execute splits, output contains more rows than specified by limit 
constraint.


Diffs (updated)
-

  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/AbstractTestJdbcGenericUDTFGetSplits.java
 8cbca69737 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcGenericUDTFGetSplits.java
 defbe78802 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcGenericUDTFGetSplits2.java
 330174513c 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
00a6c89b1e 


Diff: https://reviews.apache.org/r/72379/diff/2/

Changes: https://reviews.apache.org/r/72379/diff/1-2/


Testing
---

Waiting for Hive QA run.


Thanks,

Adesh Rao

Re: Review Request 72387: Locks: Add new lock implementations for always zero-wait readers

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/#review220374
---



Some nits, did not checked all the unit tests changes. Anything I have to check 
particularly?


common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 2727-2730 (original), 2727-2730 (patched)


Is this config still needed in this case?



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 2732 (patched)


Maybe rephrase?
Manages concurrency levels for ACID resoruces. Enables users enable 
parallel queries by enabling write-write conflict resolution happen only at 
commit phase 
- If true - no commit phase conflict resolution: 
   - INSERT OVERWRITE requests EXCLUSIVE locks
   - UPDATE/DELETE requests EXCL_WRITE lock
   - INSERT requests SHARED_READ lock
- If false - write might fail when committed on conflict check: 
   - INSERT OVERWRITE requests EXCL_WRITE locks
   - UPDATE/DELETE/INSERT requests SHARED_READ lock



ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
Line 3027 (original), 3027-3028 (patched)


Maybe get once, and store?



ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java
Lines 758 (patched)


nit: remove spaces



ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java
Line 1142 (original), 1265 (patched)


nit: space too


- Peter Vary


On ápr. 20, 2020, 7:23 de, Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72387/
> ---
> 
> (Updated ápr. 20, 2020, 7:23 de)
> 
> 
> Review request for hive, Marton Bod and Peter Vary.
> 
> 
> Bugs: HIVE-19369
> https://issues.apache.org/jira/browse/HIVE-19369
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Hive Locking with Micro-managed and full-ACID tables needs a better locking 
> implementation which allows for no-wait readers always.
> 
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
> 
> Short write-up
> 
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to 
> exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to 
> exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an 
> EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different 
> threads).
> 
> SHARED_READ does not wait for any lock - it fails fast for a pending 
> EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, 
> there's no semantic reason to wait for them to succeed before going ahead 
> with a SHARED_WRITE.
> 
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7b3acad511 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b 
>   ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
> 1d211857bf 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 73d3b91585 
>   ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f 
>   ql/src/test/results/clientpositive/explain_locks.q.out 3183533478 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
>  22902a9c20 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  331fd4cc8d 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
>  f928bf781b 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java
>  3d69a6e7dc 
> 
> 
> Diff: https://reviews.apache.org/r/72387/diff/1/
> 
> 
> Testing
> ---
> 
> Added number of tests under TestDbTxnManager2, TestTxnHandler & extended 
> explain_locks.q test.
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Review Request 72392: HIVE-23103 Oracle statement batching

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72392/
---

Review request for hive, Denys Kuzmenko and Marton Bod.


Bugs: HIVE-23103
https://issues.apache.org/jira/browse/HIVE-23103


Repository: hive-git


Description
---

Examine how to really get better performance for oracle statement batches.

Oracle JDBC doc describes:

The Oracle implementation of standard update batching does not implement true 
batching for generic statements and callable statements. Even though Oracle 
JDBC supports the use of standard batching for Statement and CallableStatement 
objects, you are unlikely to see performance improvement.

I would look for connection properties to set, so it is handled anyway, or if 
not, then use:

begin
  query1;
  query2;
  query3;
end;
to we will have only a single roundtrip for the db.


Diffs
-

  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java
 bb29410e7d 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 d080df417b 


Diff: https://reviews.apache.org/r/72392/diff/1/


Testing
---

Baseline:
Benchmark(dbProduct)  (txnType)  Mode  Cnt   Score   
Error  Units
TxnHandlerBenchRunner.commitTxn   ORACLEDEFAULTss  100  42.988 ± 
4.569  ms/op
TxnHandlerBenchRunner.commitTxn   ORACLE  READ_ONLYss  100  45.029 ± 
4.686  ms/op

After patch:
Benchmark(dbProduct)  (txnType)  Mode  Cnt   Score   
Error  Units
TxnHandlerBenchRunner.commitTxn   ORACLEDEFAULTss  100  36.208 ± 
3.869  ms/op
TxnHandlerBenchRunner.commitTxn   ORACLE  READ_ONLYss  100  37.038 ± 
3.746  ms/op


Thanks,

Peter Vary

[jira] [Created] (HIVE-23258) Remove BoneCP Connection Pool

[David Mollitor (Jira)]

David Mollitor created HIVE-23258:
-

 Summary: Remove BoneCP Connection Pool
 Key: HIVE-23258
 URL: https://issues.apache.org/jira/browse/HIVE-23258
 Project: Hive
  Issue Type: Improvement
Reporter: David Mollitor
Assignee: David Mollitor


{quote}
BoneCP is a Java JDBC connection pool implementation that is tuned for high 
performance by minimizing lock contention to give greater throughput for your 
application ... but SHOULD NOW BE CONSIDERED DEPRECATED in favour of HikariCP.
{quote}

https://github.com/wwadge/bonecp

The default in Hive 3.x is already HikariCP, so just remove BoneCP in 4.x

https://github.com/apache/hive/blob/branch-3.1/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java#L392



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23257) Fix flaky test TestMiniLlapLocalCliDriver[schema_evol_par_vec_table_non_dictionary_encoding.q]

[Miklos Gergely (Jira)]

Miklos Gergely created HIVE-23257:
-

 Summary: Fix flaky test 
TestMiniLlapLocalCliDriver[schema_evol_par_vec_table_non_dictionary_encoding.q]
 Key: HIVE-23257
 URL: https://issues.apache.org/jira/browse/HIVE-23257
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Miklos Gergely






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23256) Fix flaky test TestMiniLlapLocalCliDriver[schema_evol_par_vec_table_dictionary_encoding.q]

[Miklos Gergely (Jira)]

Miklos Gergely created HIVE-23256:
-

 Summary: Fix flaky test 
TestMiniLlapLocalCliDriver[schema_evol_par_vec_table_dictionary_encoding.q]
 Key: HIVE-23256
 URL: https://issues.apache.org/jira/browse/HIVE-23256
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Miklos Gergely






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23255) Fix flaky test TestMiniLlapLocalCliDriver[temp_table_multi_insert_partitioned.q]

[Miklos Gergely (Jira)]

Miklos Gergely created HIVE-23255:
-

 Summary: Fix flaky test 
TestMiniLlapLocalCliDriver[temp_table_multi_insert_partitioned.q]
 Key: HIVE-23255
 URL: https://issues.apache.org/jira/browse/HIVE-23255
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Miklos Gergely






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72388: HIVE-23048 Use sequences for TXN_ID generation

[Denys Kuzmenko via Review Board]

> On April 20, 2020, 12:19 p.m., Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
> > Lines 780 (patched)
> > 
> >
> > TXN_META_INFO? What is it used for before? Or is it new? Could we use a 
> > specific "state" for example?

My understanding was, it's kinda marker to flag the new txn entries and get 
their ids.


- Denys


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72388/#review220369
---


On April 20, 2020, 7:25 a.m., Peter Varga wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72388/
> ---
> 
> (Updated April 20, 2020, 7:25 a.m.)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> * Use sequences for TXN_ID generation.
>   * Make it possible to open transactions in parallel
>   * drop Oracle 11g support
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 37a5862791 
>   
> ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java
>  15fcfc0e35 
>   ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
> 1d211857bf 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 2c13e8dd03 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java 51b0fa336f 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java 
> 6525ffc00a 
>   ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 
> 1435269ed3 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 73d3b91585 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManagerAcid.java 
> PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator.java 
> 1151466f8c 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  f6097f7520 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/SQLGenerator.java
>  49b737ecf9 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
>  2344c2d5f6 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java
>  bb29410e7d 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  d080df417b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java
>  87130a519d 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
>  1ace9d3ef0 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
>  8a3cd56658 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
>  2e0177723d 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
>  9f3951575b 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
>  0512a45cad 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
>  4b82e36ab4 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
>  db398e5f66 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
>  1be83fc4a9 
>   
> standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
>  e6e30160af 
>   
> standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
>  b90cecb173 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/DbInstallBase.java
>  c1a1629548 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/rules/DatabaseRule.java
>  a6d22d19ef 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/rules/Oracle.java
>  0b070e19ac 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/txn/TestOpenTxn.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72388/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Peter Varga
> 
>

[jira] [Created] (HIVE-23254) Upgrade guava version in hive from 19.0 to 27.0-jre

[Ankur Raj (Jira)]

Ankur Raj created HIVE-23254:


 Summary: Upgrade guava version in hive from 19.0 to 27.0-jre
 Key: HIVE-23254
 URL: https://issues.apache.org/jira/browse/HIVE-23254
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.1.1
Reporter: Ankur Raj


Upgrade guava version in hive from 19.0 to 27.0-jre. 

Hadoop has already upgraded it as part of 
[https://jira.apache.org/jira/browse/HADOOP-16213]

Concern : [https://nvd.nist.gov/vuln/detail/CVE-2018-10237]

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72388: HIVE-23048 Use sequences for TXN_ID generation

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72388/#review220369
---



This is a really big/scary change. I am really interested in the performance 
results! :)
Thanks for all the effort! Some questions below


ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManagerAcid.java
Lines 119 (patched)


nit: space after //



ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManagerAcid.java
Lines 154 (patched)


Could we create a test for multigap case (gap with more then 1?)



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
Lines 563 (patched)


Maybe insert the query instead of getting id and using again. Not very 
important, just asking...



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Lines 424 (patched)


nit: missing line break



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Line 448 (original), 528 (patched)


Just a question: This is so similar to getOpenTxnsInfo... Any way to use 
the same code with different query and different response handling?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Lines 699 (patched)


openTxns(dbConn, stmt, rqst) is used in replTableWriteIdState as well. Do 
we have to check there for timeout as well?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Lines 747 (patched)


nit: typo? "support every"?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Lines 780 (patched)


TXN_META_INFO? What is it used for before? Or is it new? Could we use a 
specific "state" for example?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Line 641 (original), 783 (patched)


Could we use a correct PreparedStatement, with the values set without the 
"hacky" quoting?

Like: pstmt.setXX for state/user/host/type/metainfo?

Or does this have a noticable performance gain?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Lines 891 (patched)


If we implement the stuff in a single query in both cases when it is used, 
we can get rid of this method



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Line 1168 (original), 1418 (patched)


If we do not make such a fuss about the checking, just a simple assert 
instead, we might can inline the method here, since it is not used anywhere 
else? What do you think?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Line 1995 (original), 2245 (patched)


nit: space "MetaException{"



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Lines 2262 (patched)


This is again a single query implemented with 3 SQL query and a java code. 
Am I right?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Line 4539 (original), 4775 (patched)


nit: I do not see what is changed, but if we change please remove double 
spaces too :)



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Line 5078 (original), 5321 (patched)


nit: Missing space: "MetaException{"?



standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
Lines 117 (patched)


nit: extra space? "TXN_COMPONENTS  WITH"



standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/txn/TestOpenTxn.java
Lines 78 (patched)


Please test for multiple gaps too


- Peter Vary


On ápr. 20, 2020, 7:25 de, 

[jira] [Created] (HIVE-23253) Synchronization between external SerDe schemas and Metastore

[Attila Magyar (Jira)]

Attila Magyar created HIVE-23253:


 Summary: Synchronization between external SerDe schemas and 
Metastore
 Key: HIVE-23253
 URL: https://issues.apache.org/jira/browse/HIVE-23253
 Project: Hive
  Issue Type: Bug
  Components: Hive, Metastore
Affects Versions: 3.1.2
Reporter: Attila Magyar
 Fix For: 3.0.0


In HIVE-15995 an ALTER  UPDATE COLUMNS statement was introduce to sync 
external SerDe schema changes with the metastore. This command can only be 
manually invoked.

See it in the documentation.

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable/PartitionUpdatecolumns

 

Maybe it would make sense to run an update columns automatically in certain 
cases to prevent problems coming from cases where the user forgets running the 
update columns manually.

 

One way to reproduce the issue is to change the schema url via an alter table 
statement.
{code:java}
[root@c7401 vagrant]# cat test_schema1.avsc
{
"type":"record",
"name":"test_schema",
"namespace":"gdc_datascience_qa",
"fields":[
{
"name":"name",
"type":[
"null",
"string"
],
"default":null
}
]
}[root@c7401 vagrant]# cat test_schema2.avsc
{
"type":"record",
"name":"test_schema",
"namespace":"gdc_datascience_qa",
"fields":[
{
"name":"name",
"type":[
"null",
"string"
],
"default":null
},
{
"name":"last_name",
"type":[
"null",
"string"
],
"default":null
}
]
}
 {code}
{code:java}
 $ hadoop fs -copyFromLocal *.avsc /tmp/
  [beeline] create external table t1 stored as avro tblproperties 
('avro.schema.url'='/tmp/test_schema1.avsc');
  [beeline] alter table t1 set 
tblproperties('avro.schema.url'='/tmp/test_schema2.avsc'); 
  [beeline] insert into t1 values ('n1', 'l1');
  [beeline] create external table t2 stored as avro tblproperties 
('avro.schema.url'='/tmp/test_schema2.avsc');
  [beeline] insert into t2 values ('n2', 'l2');
  [beeline] insert overwrite table t1 select * from t2; {code}
Error:
{code:java}
 MetaException(message:Column last_name doesn't exist in table t1 in database 
default)
        at 
org.apache.hadoop.hive.metastore.ObjectStore.validateTableCols(ObjectStore.java:8652)
        at 
org.apache.hadoop.hive.metastore.ObjectStore.getMTableColumnStatistics(ObjectStore.java:8602)
        at 
org.apache.hadoop.hive.metastore.ObjectStore.getPartitionColStats(ObjectStore.java:8416)
        at 
org.apache.hadoop.hive.metastore.ObjectStore.updateTableColumnStatistics(ObjectStore.java:8446
 {code}
Running an ALTER UPDATE COLUMNS fixes the problem.

 

cc: [~szita]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23252) Change spark related tests to be optional

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HIVE-23252:
---

 Summary: Change spark related tests to be optional
 Key: HIVE-23252
 URL: https://issues.apache.org/jira/browse/HIVE-23252
 Project: Hive
  Issue Type: Sub-task
  Components: Test
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


HIVE-23137 have disabled the execution of some spark related tests; but they 
would be still considered by a plain maven command - and the spark artifacts 
are (unneccessarily)  still downloaded



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72388: HIVE-23048 Use sequences for TXN_ID generation

[Denys Kuzmenko via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72388/#review220367
---



LGTM, just a few comments


ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManagerAcid.java
Lines 47 (patched)


Looks similar to TestDbTxnManager2. Could we extract common parts (setUp, 
tearDown) into the base class. Also name is misleading, TestDbTxnManager2 also 
covers ACID



standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
Lines 1202 (patched)


I would replace commit to persist, as it's a bit misleading and could be 
confused with txn.commit



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/SQLGenerator.java
Lines 351 (patched)


Could we use Optional here? Returning null is not a good practice.



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
Lines 554 (patched)


Extend commend with commited, as it deletes both aborted & commited



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
Line 552 (original), 560 (patched)


Could we use this query as condition for delete? selete where exists (..)



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Line 172 (original), 225 (patched)


Please rebase, this code is no longer here



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Line 549 (original), 635 (patched)


not needed, could we remove this



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Lines 694 (patched)


Could we change "slower than that" -> "slower than openTxnTimeout" for 
clarity



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Lines 716 (patched)


not needed, could we remove this



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Lines 807 (patched)


could we use stream instead of for loop: 
first = 
txnIds.stream().max(Comparator.comparing(Long::valueOf)).orElse(LONG_MAX)



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Lines 902 (patched)


Can it be null? Could we add constraint on db level instead of multiple 
checks in backend code?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Line 1332 (original), 1579 (patched)


Could we rename to updateWSCommitIdAndCleanUpMetadata for readability



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Lines 5480 (patched)


As mentioned above, I would use optional here.


- Denys Kuzmenko


On April 20, 2020, 7:25 a.m., Peter Varga wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72388/
> ---
> 
> (Updated April 20, 2020, 7:25 a.m.)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> * Use sequences for TXN_ID generation.
>   * Make it possible to open transactions in parallel
>   * drop Oracle 11g support
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 37a5862791 
>   
> ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java
>  15fcfc0e35 
>   ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
> 1d211857bf 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 2c13e8dd03 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java 51b0fa336f 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java 
> 6525ffc00a 
>   ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 
> 1435269ed3 
>   

[jira] [Created] (HIVE-23251) Provide a way to have only a selection of datasets loaded

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HIVE-23251:
---

 Summary: Provide a way to have only a selection of datasets loaded
 Key: HIVE-23251
 URL: https://issues.apache.org/jira/browse/HIVE-23251
 Project: Hive
  Issue Type: Sub-task
  Components: Test
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
 Attachments: HIVE-22942.01.patch

for example sysdb.q is listing all the tables known; which can change depending 
on tests executed prior to this qtest 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72378: HIVE-23201: Improve logging in locking

[Marton Bod]

> On April 17, 2020, 1:46 p.m., Peter Vary wrote:
> > Mostly agree, few comments.
> > I would like to ask you to go through them, and if there are any places 
> > where the problem should not happen use at least info level.
> > 
> > Thanks,
> > Peter

Thanks for the review Peter! I've made changes based on your comments


- Marton


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72378/#review220347
---


On April 17, 2020, 11:30 a.m., Marton Bod wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72378/
> ---
> 
> (Updated April 17, 2020, 11:30 a.m.)
> 
> 
> Review request for hive, Denys Kuzmenko and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-23201: Improve logging in locking
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbLockManager.java 4b6bc3e1e3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  d080df417b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
>  b3a1f826bb 
> 
> 
> Diff: https://reviews.apache.org/r/72378/diff/2/
> 
> 
> Testing
> ---
> 
> Green build: https://builds.apache.org/job/PreCommit-HIVE-Build/21717/
> 
> 
> Thanks,
> 
> Marton Bod
> 
>

[jira] [Created] (HIVE-23250) Scheduled query related qtests may not finish before it's expected

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HIVE-23250:
---

 Summary: Scheduled query related qtests may not finish before it's 
expected
 Key: HIVE-23250
 URL: https://issues.apache.org/jira/browse/HIVE-23250
 Project: Hive
  Issue Type: Sub-task
  Components: Test
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Review Request 72388: HIVE-23048 Use sequences for TXN_ID generation

[Peter Varga via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72388/
---

Review request for hive and Peter Vary.


Repository: hive-git


Description
---

* Use sequences for TXN_ID generation.
  * Make it possible to open transactions in parallel
  * drop Oracle 11g support


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 37a5862791 
  
ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java 
15fcfc0e35 
  ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
1d211857bf 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 2c13e8dd03 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java 51b0fa336f 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java 
6525ffc00a 
  ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 1435269ed3 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
73d3b91585 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManagerAcid.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator.java 
1151466f8c 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 f6097f7520 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/SQLGenerator.java
 49b737ecf9 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
 2344c2d5f6 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java
 bb29410e7d 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 d080df417b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java
 87130a519d 
  
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
 1ace9d3ef0 
  
standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
 8a3cd56658 
  
standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
 2e0177723d 
  
standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
 9f3951575b 
  
standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
 0512a45cad 
  
standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
 4b82e36ab4 
  
standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
 db398e5f66 
  
standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
 1be83fc4a9 
  
standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
 e6e30160af 
  
standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
 b90cecb173 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/DbInstallBase.java
 c1a1629548 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/rules/DatabaseRule.java
 a6d22d19ef 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/rules/Oracle.java
 0b070e19ac 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/txn/TestOpenTxn.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/72388/diff/1/


Testing
---


Thanks,

Peter Varga

Review Request 72387: Locks: Add new lock implementations for always zero-wait readers

[Denys Kuzmenko via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/
---

Review request for hive, Marton Bod and Peter Vary.


Bugs: HIVE-19369
https://issues.apache.org/jira/browse/HIVE-19369


Repository: hive-git


Description
---

Hive Locking with Micro-managed and full-ACID tables needs a better locking 
implementation which allows for no-wait readers always.

EXCL_DROP
EXCL_WRITE
SHARED_WRITE
SHARED_READ

Short write-up

EXCL_DROP is a "drop partition" or "drop table" and waits for all others to exit
EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to 
exit.
SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an 
EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different 
threads).

SHARED_READ does not wait for any lock - it fails fast for a pending EXCL_DROP, 
because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no 
semantic reason to wait for them to succeed before going ahead with a 
SHARED_WRITE.

a select * => SHARED_READ
an insert into => SHARED_WRITE
an insert overwrite or MERGE => EXCL_WRITE
a drop table => EXCL_DROP


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7b3acad511 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b 
  ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
1d211857bf 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
73d3b91585 
  ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f 
  ql/src/test/results/clientpositive/explain_locks.q.out 3183533478 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java
 22902a9c20 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 331fd4cc8d 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java
 f928bf781b 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java
 3d69a6e7dc 


Diff: https://reviews.apache.org/r/72387/diff/1/


Testing
---

Added number of tests under TestDbTxnManager2, TestTxnHandler & extended 
explain_locks.q test.


Thanks,

Denys Kuzmenko

[jira] [Created] (HIVE-23249) Prevent infinite loop in TestJdbcWithMiniLlapArrow

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HIVE-23249:
---

 Summary: Prevent infinite loop in TestJdbcWithMiniLlapArrow
 Key: HIVE-23249
 URL: https://issues.apache.org/jira/browse/HIVE-23249
 Project: Hive
  Issue Type: Sub-task
  Components: Test
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


if we always hit continue 
[here|https://github.com/apache/hive/blob/e65715c6ab12816ba6a79b21e8e9fcbfc6fe41cd/itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniLlapArrow.java#L372];
 the loop may never-ever end...





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72380: HIVE-23207 Create integration tests for TxnManager for different rdbms metastores

[Peter Vary via Review Board]

> On ápr. 17, 2020, 7:03 du, Peter Vary wrote:
> > Thanks Peter for the patch!
> > This fix is long overdue!
> > 
> > I do not understand one thing, see below.
> > 
> > Also I would like to ask Denys to confirm, that running the init sqls again 
> > and again will not cause too much overhead in test runtime as he mentioned 
> > in our discussion. (5 min tests are fine, 1 hour tests are not fine :))
> > 
> > Thanks!
> 
> Peter Varga wrote:
> In this version we do not run the init sql again and again. We just run 
> it once, and we just run the cleanDb between tests. There might be some Test 
> classes that is not true, because there are multiple prepDb calls (I solved 
> in in Qtest, TestDbTxnManager2) but in those places it will be just one call 
> to check if the txns table exists, and then we return

Got it. Thanks!


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72380/#review220352
---


On ápr. 17, 2020, 3:24 du, Peter Varga wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72380/
> ---
> 
> (Updated ápr. 17, 2020, 3:24 du)
> 
> 
> Review request for hive, Denys Kuzmenko and Zoltan Chovan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> In the final version the prepDb creates the transactional tables with the 
> init schema (and ignores the others if they exists). The cleanDb resets the 
> database to the starting point. So between the test cases the cleanDb call is 
> enough. If the prepDb is called unneccessary it will just check if the txns 
> table exist and then return, so it will be fast
> 
> 
> Diffs
> -
> 
>   itests/hive-blobstore/pom.xml 09955c55f3 
>   itests/qtest-accumulo/pom.xml a35d2a8a10 
>   itests/qtest-druid/pom.xml cc0cceff68 
>   itests/qtest-kudu/pom.xml f23399fa37 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMetaStoreHandler.java
>  b86d736a89 
>   pom.xml 90e39702a1 
>   ql/pom.xml d1846c9245 
>   
> ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java
>  15fcfc0e35 
>   ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
> 1d211857bf 
>   
> ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandlerNoConnectionPool.java
>  ebe4880e3a 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/ITestDbTxnManager.java 
> PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 73d3b91585 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DatabaseProduct.java
>  3e56ad513c 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java
>  a66e16973f 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  962a63d418 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
>  1ace9d3ef0 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStoreTxns.java
>  5f3db52c2f 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/rules/Mysql.java
>  c537d95470 
> 
> 
> Diff: https://reviews.apache.org/r/72380/diff/1/
> 
> 
> Testing
> ---
> 
> On my machine the 50 tests in TestDbTxnManager2 on postgres runs under 5 
> minutes.
> 
> 
> Thanks,
> 
> Peter Varga
> 
>

Re: Review Request 72380: HIVE-23207 Create integration tests for TxnManager for different rdbms metastores

[Peter Varga via Review Board]

> On April 17, 2020, 7:03 p.m., Peter Vary wrote:
> > Thanks Peter for the patch!
> > This fix is long overdue!
> > 
> > I do not understand one thing, see below.
> > 
> > Also I would like to ask Denys to confirm, that running the init sqls again 
> > and again will not cause too much overhead in test runtime as he mentioned 
> > in our discussion. (5 min tests are fine, 1 hour tests are not fine :))
> > 
> > Thanks!

In this version we do not run the init sql again and again. We just run it 
once, and we just run the cleanDb between tests. There might be some Test 
classes that is not true, because there are multiple prepDb calls (I solved in 
in Qtest, TestDbTxnManager2) but in those places it will be just one call to 
check if the txns table exists, and then we return


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72380/#review220352
---


On April 17, 2020, 3:24 p.m., Peter Varga wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72380/
> ---
> 
> (Updated April 17, 2020, 3:24 p.m.)
> 
> 
> Review request for hive, Denys Kuzmenko and Zoltan Chovan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> In the final version the prepDb creates the transactional tables with the 
> init schema (and ignores the others if they exists). The cleanDb resets the 
> database to the starting point. So between the test cases the cleanDb call is 
> enough. If the prepDb is called unneccessary it will just check if the txns 
> table exist and then return, so it will be fast
> 
> 
> Diffs
> -
> 
>   itests/hive-blobstore/pom.xml 09955c55f3 
>   itests/qtest-accumulo/pom.xml a35d2a8a10 
>   itests/qtest-druid/pom.xml cc0cceff68 
>   itests/qtest-kudu/pom.xml f23399fa37 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMetaStoreHandler.java
>  b86d736a89 
>   pom.xml 90e39702a1 
>   ql/pom.xml d1846c9245 
>   
> ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java
>  15fcfc0e35 
>   ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
> 1d211857bf 
>   
> ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandlerNoConnectionPool.java
>  ebe4880e3a 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/ITestDbTxnManager.java 
> PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 73d3b91585 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DatabaseProduct.java
>  3e56ad513c 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java
>  a66e16973f 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  962a63d418 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
>  1ace9d3ef0 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStoreTxns.java
>  5f3db52c2f 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/rules/Mysql.java
>  c537d95470 
> 
> 
> Diff: https://reviews.apache.org/r/72380/diff/1/
> 
> 
> Testing
> ---
> 
> On my machine the 50 tests in TestDbTxnManager2 on postgres runs under 5 
> minutes.
> 
> 
> Thanks,
> 
> Peter Varga
> 
>

[jira] [Created] (HIVE-23248) avro-mapred should not pull in org.mortbay.jetty

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HIVE-23248:
---

 Summary: avro-mapred should not pull in org.mortbay.jetty
 Key: HIVE-23248
 URL: https://issues.apache.org/jira/browse/HIVE-23248
 Project: Hive
  Issue Type: Sub-task
  Components: Test
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
 Attachments: HIVE-23248.01.patch

the current avro-mapred dependency pulls in a 
org.mortbay.jetty:servlet-api implementation; which might confuse tests/etc

for example: it could cause TestLlapWebServices to fail



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23247) Increase timeout for some tez tests

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HIVE-23247:
---

 Summary: Increase timeout for some tez tests
 Key: HIVE-23247
 URL: https://issues.apache.org/jira/browse/HIVE-23247
 Project: Hive
  Issue Type: Sub-task
  Components: Test
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


we have a few tez tests which run for ~3.4 seconds with a 5sec timeout

In case these tests run in a more saturated environment they became 
unrelyable...and fail regularily



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23246) Reduce MiniDruidCluster memory requeirements

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HIVE-23246:
---

 Summary: Reduce MiniDruidCluster memory requeirements
 Key: HIVE-23246
 URL: https://issues.apache.org/jira/browse/HIVE-23246
 Project: Hive
  Issue Type: Sub-task
  Components: Test
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


this is needed to enable the druid and kafka related cli tests to be run 
reliably with at most 8G of memory

* reduce processing buffer size to 10M  (from 200M)
* set Xmx512m for spawned peon processes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)