[jira] [Created] (HIVE-22913) HiveReduceExpressionsWithStatsRule should not invoke simplifcation during end of visits

2020-02-19 Thread Zoltan Haindrich (Jira) 
Zoltan Haindrich created HIVE-22913:
---

 Summary: HiveReduceExpressionsWithStatsRule should not invoke 
simplifcation during end of visits
 Key: HIVE-22913
 URL: https://issues.apache.org/jira/browse/HIVE-22913
 Project: Hive
  Issue Type: Improvement
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


* doing a simplification is a full visit of the subtree
* in case the rule makes a structural change; it will invoke simplification 
during exiting the recursion

https://github.com/apache/hive/blob/5012954396c98c94d0ff64fe3bbdb74e6077f190/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsWithStatsRule.java#L275



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22912) Support native submission of Hive queries to a Kubernetes Cluster

2020-02-19 Thread Surbhi Aggarwal (Jira) 
Surbhi Aggarwal created HIVE-22912:
--

 Summary: Support native submission of Hive queries to a Kubernetes 
Cluster
 Key: HIVE-22912
 URL: https://issues.apache.org/jira/browse/HIVE-22912
 Project: Hive
  Issue Type: New Feature
Reporter: Surbhi Aggarwal


So many big data applications are already integrated or trying to natively 
integrate with Kubernetes engine. Should we not work together to support hive 
with this engine?

If efforts are already being spent on this, please point me to it. Thanks !



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72129: HIVE-22850: Optimise lock acquisition in TxnHandler

2020-02-19 Thread Rajesh Balamohan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72129/
---

(Updated Feb. 19, 2020, 2:10 p.m.)


Review request for hive, Gopal V, Peter Vary, and Zoltan Chovan.


Repository: hive-git


Description
---

- Main change is in TxnHandler::checkLock. 
- When all incoming requests are SHARED_READ, we can add a condition in the 
query to retrieve only relevant rows. This avoids significant number of rows 
fetched in the form of "SHARED_READ + ACQUIRED". There is a corner condition of 
"SHARED_WRITE --> SHARED_READ::ACQUIRED", which is misleading in the 
jumpttable. This condition can be optimised later.
- Also, removed the "HL_PARTITION IN" clause which could potentially 
overflow for oracle. Partition details can be filtered out, if the earlier 
query actually returned any rows.
- Rest of the changes, are related to refactoring 
"TxnHandler::enqueueLockWithRetry" to reduce lock scope.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbLockManager.java a8b9653411 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 f53aebe4ad 


Diff: https://reviews.apache.org/r/72129/diff/4/

Changes: https://reviews.apache.org/r/72129/diff/3-4/


Testing
---


File Attachments


HIVE-22850.5.patch
  
https://reviews.apache.org/media/uploaded/files/2020/02/13/74ec6cbd-c552-4d46-b5a6-e2fa6da41bdc__HIVE-22850.5.patch


Thanks,

Rajesh Balamohan

Re: Review Request 72151: HIVE-22376: Cancelled query still prints exception if it was stuck in waiting for lock

2020-02-19 Thread Denys Kuzmenko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72151/#review219612
---




ql/src/java/org/apache/hadoop/hive/ql/Driver.java
Lines 400 (patched)


Could you please elaborate on terminal state. Maybe I am lacking the 
context (is it only when query is cancelled), but it's not obvious for me what 
might lead us to this state.


- Denys Kuzmenko


On Feb. 19, 2020, 9:21 a.m., Aron Hamvas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72151/
> ---
> 
> (Updated Feb. 19, 2020, 9:21 a.m.)
> 
> 
> Review request for hive, Denys Kuzmenko, Marta Kuczora, Laszlo Pinter, and 
> Peter Vary.
> 
> 
> Bugs: HIVE-22376
> https://issues.apache.org/jira/browse/HIVE-22376
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> If the reason for lock acquisition failure is that the query is canceled, the 
> exception should be ignored.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
> 48ebc4f87071bae4cc39309ada8d90dfc5c64f5b 
> 
> 
> Diff: https://reviews.apache.org/r/72151/diff/2/
> 
> 
> Testing
> ---
> 
> 
> File Attachments
> 
> 
> HIVE-22376.patch
>   
> https://reviews.apache.org/media/uploaded/files/2020/02/19/41c20e4c-57f6-46cc-988c-3b76041aa7ee__HIVE-22376.patch
> 
> 
> Thanks,
> 
> Aron Hamvas
> 
>

[jira] [Created] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager

2020-02-19 Thread Oleksiy Sayankin (Jira) 
Oleksiy Sayankin created HIVE-22911:
---

 Summary: Locks entries are left over inside HIVE_LOCKS when using 
DbTxnManager
 Key: HIVE-22911
 URL: https://issues.apache.org/jira/browse/HIVE-22911
 Project: Hive
  Issue Type: Bug
Reporter: Oleksiy Sayankin
Assignee: Oleksiy Sayankin


We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There 
are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the 
top 3 tables which are related to the existing locks:

 
{code}
mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3 
desc limit 10;
+---+--+--+
| HL_DB | HL_TABLE | count(*) |
+---+--+--+
| db1 | table1 | 66984 |
| db1 | table2 | 33208 |
| db1 | table3 | 9315 |
…
{code}

For table “db1. table1”, here are 3 Hive sessions related, and each of the Hive 
session is waiting for 22328 read locks. This is because this table “db1. 
table1” is a huge partition table, and it has more than 200k child partitions. 
I am guessing each of Hive session was trying to do a full table scan on it. I 
group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is the list:

 

{code}
MariaDB [customer]> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as date) 
as dt,count(*) as cnt from HIVE_LOCKS
-> group by 1 order by 1;
+++
| dt | cnt|
+++
| 1969-12-31 |  2 |
| 2019-05-20 | 10 |
| 2019-05-21 |  3 |
| 2019-05-23 |  5 |
| 2019-05-24 |  2 |
| 2019-05-25 |  1 |
| 2019-05-29 |  7 |
| 2019-05-30 |  2 |
| 2019-06-11 | 13 |
| 2019-06-28 |  3 |
| 2019-07-02 |  2 |
| 2019-07-04 |  5 |
| 2019-07-09 |  1 |
| 2019-07-15 |  2 |
| 2019-07-16 |  1 |
| 2019-07-18 |  2 |
| 2019-07-20 |  3 |
| 2019-07-29 |  5 |
| 2019-07-30 |  9 |
| 2019-07-31 |  7 |
| 2019-08-02 |  2 |
| 2019-08-06 |  5 |
| 2019-08-07 | 17 |
| 2019-08-08 |  8 |
| 2019-08-09 |  5 |
| 2019-08-21 |  1 |
| 2019-08-22 | 20 |
| 2019-08-23 |  1 |
| 2019-08-26 |  5 |
| 2019-08-27 | 98 |
| 2019-08-28 |  3 |
| 2019-08-29 |  1 |
| 2019-09-02 |  3 |
| 2019-09-04 |  3 |
| 2019-09-05 |105 |
| 2019-09-06 |  3 |
| 2019-09-07 |  2 |
| 2019-09-09 |  6 |
| 2019-09-12 |  9 |
| 2019-09-13 |  1 |
| 2019-09-17 |  1 |
| 2019-09-24 |  3 |
| 2019-09-26 |  6 |
| 2019-09-27 |  4 |
| 2019-09-30 |  1 |
| 2019-10-01 |  2 |
| 2019-10-03 |  9 |
| 2019-10-04 |  2 |
| 2019-10-06 |  1 |
| 2019-10-08 |  1 |
| 2019-10-09 |  1 |
| 2019-10-10 |  6 |
| 2019-10-11 |  1 |
| 2019-10-16 | 13 |
| 2019-10-17 |  1 |
| 2019-10-18 |  2 |
| 2019-10-19 |  2 |
| 2019-10-21 | 10 |
| 2019-10-22 |  6 |
| 2019-10-28 |  2 |
| 2019-10-29 |  4 |
| 2019-10-30 |  2 |
| 2019-10-31 |  2 |
| 2019-11-05 |  2 |
| 2019-11-06 |  2 |
| 2019-11-11 |  1 |
| 2019-11-13 |  1 |
| 2019-11-14 |  1 |
| 2019-11-21 |  4 |
| 2019-11-26 |  1 |
| 2019-11-27 |  1 |
| 2019-12-05 |  4 |
| 2019-12-06 |  2 |
| 2019-12-12 |  1 |
| 2019-12-14 |  1 |
| 2019-12-15 |  3 |
| 2019-12-16 |  1 |
| 2019-12-17 |  1 |
| 2019-12-18 |  1 |
| 2019-12-19 |  2 |
| 2019-12-20 |  2 |
| 2019-12-23 |  1 |
| 2019-12-27 |  1 |
| 2020-01-07 |  1 |
| 2020-01-08 | 14 |
| 2020-01-09 |  2 |
| 2020-01-12 |372 |
| 2020-01-14 |  2 |
| 2020-01-15 |  1 |
| 2020-01-20 | 11 |
| 2020-01-21 | 119253 |
| 2020-01-23 |113 |
| 2020-01-24 |  4 |
| 2020-01-25 |536 |
| 2020-01-26 |   2132 |
| 2020-01-27 |396 |
| 2020-01-28 |  1 |
| 2020-01-29 |  3 |
| 2020-01-30 | 11 |
| 2020-01-31 | 11 |
| 2020-02-03 |  2 |
| 2020-02-04 |  4 |
| 2020-02-05 |  5 |
| 2020-02-06 |  8 |
| 2020-02-10 | 32 |
| 2020-02-11 | 15 |
| 2020-02-12 | 14 |
| 2020-02-13 |  1 |
| 2020-02-14 | 92 |
+++
109 rows in set (0.16 sec)
{code}

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72151: HIVE-22376: Cancelled query still prints exception if it was stuck in waiting for lock

2020-02-19 Thread Aron Hamvas via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72151/
---

(Updated Feb. 19, 2020, 9:21 a.m.)


Review request for hive, Denys Kuzmenko, Marta Kuczora, Laszlo Pinter, and 
Peter Vary.


Bugs: HIVE-22376
https://issues.apache.org/jira/browse/HIVE-22376


Repository: hive-git


Description
---

If the reason for lock acquisition failure is that the query is canceled, the 
exception should be ignored.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
48ebc4f87071bae4cc39309ada8d90dfc5c64f5b 


Diff: https://reviews.apache.org/r/72151/diff/2/

Changes: https://reviews.apache.org/r/72151/diff/1-2/


Testing
---


File Attachments (updated)


HIVE-22376.patch
  
https://reviews.apache.org/media/uploaded/files/2020/02/19/41c20e4c-57f6-46cc-988c-3b76041aa7ee__HIVE-22376.patch


Thanks,

Aron Hamvas

Review Request 72151: HIVE-22376: Cancelled query still prints exception if it was stuck in waiting for lock

2020-02-19 Thread Aron Hamvas via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72151/
---

Review request for hive, Denys Kuzmenko, Marta Kuczora, Laszlo Pinter, and 
Peter Vary.


Bugs: HIVE-22376
https://issues.apache.org/jira/browse/HIVE-22376


Repository: hive-git


Description
---

If the reason for lock acquisition failure is that the query is canceled, the 
exception should be ignored.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
48ebc4f87071bae4cc39309ada8d90dfc5c64f5b 


Diff: https://reviews.apache.org/r/72151/diff/1/


Testing
---


Thanks,

Aron Hamvas

Review Request 72150: HIVE-22905: Transaction is not aborted when query cancelled, only when session is closed

2020-02-19 Thread Aron Hamvas via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72150/
---

Review request for hive, Denys Kuzmenko, Marta Kuczora, Laszlo Pinter, and 
Peter Vary.


Bugs: HIVE-22905
https://issues.apache.org/jira/browse/HIVE-22905


Repository: hive-git


Description
---

When query is canceled, e.g. while txn is waiting for lock to be acquired, txn 
is not aborted until the session is terminated. Driver.close() should not only 
check whether there are acquired locks, but also whether the txn is still open.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
48ebc4f87071bae4cc39309ada8d90dfc5c64f5b 


Diff: https://reviews.apache.org/r/72150/diff/1/


Testing
---


Thanks,

Aron Hamvas