[jira] [Created] (HIVE-23208) Update guranteed capacity in ZK only when WM is enabled

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-23208:
---

 Summary: Update guranteed capacity in ZK only when WM is enabled
 Key: HIVE-23208
 URL: https://issues.apache.org/jira/browse/HIVE-23208
 Project: Hive
  Issue Type: Improvement
  Components: llap
Reporter: Rajesh Balamohan


[https://github.com/apache/hive/blob/master/llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java#L1091]

 

Though WM is not enabled, it ends up updating ZK for every dag completion 
event. For short running queries with concurrency, this ends up with lots of 
calls to ZK.

It would be good to invoke this only when WM is enabled.

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23207) Create integration tests for TxnManager for different rdbms metstores

[Peter Varga (Jira)]

Peter Varga created HIVE-23207:
--

 Summary: Create integration tests for TxnManager for different 
rdbms metstores
 Key: HIVE-23207
 URL: https://issues.apache.org/jira/browse/HIVE-23207
 Project: Hive
  Issue Type: Improvement
Reporter: Peter Varga
Assignee: Peter Varga


Create an integration test suite that runs tests for TxnManager with the 
metastore configured to use different kind of RDBMS-s. Use the different 
DatabaseRule-s defined in the standalone-metastore for docker environments, and 
use the real init schema for every database type instead of the hardwired 
TxnDbUtil.prepDb.

This test will be useful for easy manual validation of schema changes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23206) Project not defined correctly after reordering a join

[Steve Carlin (Jira)]

Steve Carlin created HIVE-23206:
---

 Summary: Project not defined correctly after reordering a join
 Key: HIVE-23206
 URL: https://issues.apache.org/jira/browse/HIVE-23206
 Project: Hive
  Issue Type: Bug
  Components: CBO
Reporter: Steve Carlin


The following highlighted line seems to be incorrect in the test suite:

[https://github.com/apache/hive/blob/master/ql/src/test/results/clientpositive/perf/tez/cbo_query95.q.out#L89]

Note that the project takes all the columns from the table scan, yet it only 
needs a couple of them.

I did some very small debugging on this.  When I removed the 
applyJoinOrderingTransform here: 
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L1897]
... the problem goes away.  So presumably one of the rules in there is causing 
the problem.

Here is a slightly simplified version of the query which has the same problem 
(using the same tpc-ds database):


explain cbo with ws_wh as

(select ws1.ws_order_number

from web_sales ws1,web_returns wr2 

where ws1.ws_order_number = wr2.wr_order_number)

select 

   ws_order_number

from

   web_sales ws1 

where

ws1.ws_order_number in (select wr_order_number

                            from web_returns,ws_wh

                            where wr_order_number = ws_wh.ws_order_number)

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23205) Do not run q tests with TestCliDriver if they are also set up for any other driver

[Miklos Gergely (Jira)]

Miklos Gergely created HIVE-23205:
-

 Summary: Do not run q tests with TestCliDriver if they are also 
set up for any other driver
 Key: HIVE-23205
 URL: https://issues.apache.org/jira/browse/HIVE-23205
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Reporter: Miklos Gergely
Assignee: Miklos Gergely






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23203) Dummy jira for modifying ptest configuration - before the actual patch

[Miklos Gergely (Jira)]

Miklos Gergely created HIVE-23203:
-

 Summary: Dummy jira for modifying ptest configuration - before the 
actual patch
 Key: HIVE-23203
 URL: https://issues.apache.org/jira/browse/HIVE-23203
 Project: Hive
  Issue Type: Test
  Components: Hive
Reporter: Miklos Gergely


If the ptest configuration is required to be modified, then the server must be 
restarted, which causes the actually processed patch to fail. In order not to 
cause trouble to others, this dummy Jira can be used by queueing this first, 
then the actual modification, finally the pair of this Jira.

When the patch for this one is is processed, the ptest configuration can be 
modified, and the server can be restarted, thus this dummy jira's patch 
processing will fail. Then the actual modification comes, which will result in 
some way or the other. Then the other dummy Jira comes, and the ptest 
modifications can be reverted, and the server can be restarted again.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23204) Dummy jira for modifying ptest configuration - after the actual patch

[Miklos Gergely (Jira)]

Miklos Gergely created HIVE-23204:
-

 Summary: Dummy jira for modifying ptest configuration - after the 
actual patch
 Key: HIVE-23204
 URL: https://issues.apache.org/jira/browse/HIVE-23204
 Project: Hive
  Issue Type: Test
  Components: Hive
Reporter: Miklos Gergely


If the ptest configuration is required to be modified, then the server must be 
restarted, which causes the actually processed patch to fail. In order not to 
cause trouble to others, this dummy Jira can be used by queueing this first, 
then the actual modification, finally the pair of this Jira.

When the patch for this one is is processed, the ptest configuration can be 
modified, and the server can be restarted, thus this dummy jira's patch 
processing will fail. Then the actual modification comes, which will result in 
some way or the other. Then the other dummy Jira comes, and the ptest 
modifications can be reverted, and the server can be restarted again.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Remove REGEX Column Specification

[David Mollitor]

Hey Zoltan,

Thanks for the feedback and for sharing HIVE-16496.

I think HIVE-16496 is a better approach because it allows for the standard
SQL behavior of object identifiers, but the SQL syntax is expanded (instead
of overloaded) to provide this feature.

Also, if a user would like to do some sort of regex, they can query the
information_schema (if/when Hive gets that).

Also, I just re-read my previous email and I do apologize, I provided the
wrong jira.  The correct one for removal is:

https://issues.apache.org/jira/browse/HIVE-23176

Thanks.



David

On Tue, Apr 14, 2020 at 12:16 PM Zoltan Haindrich  wrote:

> Hey,
>
> I don't want to protect this feature - but I think it could be usefull;
> probably it would be ok to remove it but we should provide something else
> instead - I think this is
> the only way to "exclude" some specific columns from the output - without
> listing all the columns.
>
> How much are users actually use this feature?
>
> We had a somewhat related discussion a few years ago:
> https://issues.apache.org/jira/browse/HIVE-16496
>
> cheers,
> Zoltan
>
> On 4/13/20 3:56 PM, David Mollitor wrote:
> > Hello Gang,
> >
> > I've been tracking a lot of issues recently regarding qualified tables
> > names, qualified table names, table names using back ticks, and other
> > similar circumstances.
> >
> > I've looked into trying to address some of these and noted that these
> issue
> > goes way back and are go all the way down to the core of Hive.
> >
> > To start with, I wanted to use the ANTLR grammar to address some of these
> > issues and to standardize behavior across all queries.  For example,
> there
> > is currently a patch that disallows table names from having a 'dot' in
> the
> > name.  I'm not 100% sure it applies to all queries, so  I wanted to
> codify
> > this restriction in the parser grammar.  So it got me looking at the
> > grammar.
> >
> > In parallel, I also tried to build a supplemental parser in Java for
> > parsing table names (HIVE-23150) and I was hitting some weird, and
> > confusing, edge cases bubbling up from the parser.  I eventually traced
> it
> > back to the fact that there are a lot of weird rules around table names
> in
> > the grammar including something called "REGEX Column Specification."
> >
> > This feature is problematic as it blindly labels most table names as
> being
> > a regex.  It really should only apply to column names, but the grammar
> > defines a table name as also possibly being a regex. There is a lot of
> > ambiguity because a table named "a" could be a literal value or a legal
> > regex.  When a table name is defined as a regex, a different code path is
> > taken from when a table name is considered to be a literal value. Where I
> > first saw this issue was in a qtest where a table name `s/c` was
> producing
> > a different result than a table named `s+c`.
> >
> > This regex feature is not something I've seen in MySQL or Postgres.  In
> > MySQL, any table name surrounded with a back tick can be just about any
> > UTF-8 character, so it's not really feasible to tell, without some kind
> of
> > SQL hint, that this table name is a regex or a literal value.
> >
> > This feature adds a lot of ambiguity and complexity, it is not supported
> by
> > other major RDBMS, and it adds only very minor benefit.  I also hope to
> > move Hive in a direction of fully supporting UTF-8.
> >
> > I have put a patch up to remove it:
> > https://issues.apache.org/jira/browse/HIVE-23183
> >
> >
> > References:
> >
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Select#LanguageManualSelect-REGEXColumnSpecification
> >
> >
> > https://dev.mysql.com/doc/refman/8.0/en/identifiers.html
> >
> >
> > Thanks,
> > David
> >
>

Re: Remove REGEX Column Specification

[Zoltan Haindrich]

Hey,

I don't want to protect this feature - but I think it could be usefull; probably it would be ok to remove it but we should provide something else instead - I think this is 
the only way to "exclude" some specific columns from the output - without listing all the columns.


How much are users actually use this feature?

We had a somewhat related discussion a few years ago:
https://issues.apache.org/jira/browse/HIVE-16496

cheers,
Zoltan

On 4/13/20 3:56 PM, David Mollitor wrote:

Hello Gang,

I've been tracking a lot of issues recently regarding qualified tables
names, qualified table names, table names using back ticks, and other
similar circumstances.

I've looked into trying to address some of these and noted that these issue
goes way back and are go all the way down to the core of Hive.

To start with, I wanted to use the ANTLR grammar to address some of these
issues and to standardize behavior across all queries.  For example, there
is currently a patch that disallows table names from having a 'dot' in the
name.  I'm not 100% sure it applies to all queries, so  I wanted to codify
this restriction in the parser grammar.  So it got me looking at the
grammar.

In parallel, I also tried to build a supplemental parser in Java for
parsing table names (HIVE-23150) and I was hitting some weird, and
confusing, edge cases bubbling up from the parser.  I eventually traced it
back to the fact that there are a lot of weird rules around table names in
the grammar including something called "REGEX Column Specification."

This feature is problematic as it blindly labels most table names as being
a regex.  It really should only apply to column names, but the grammar
defines a table name as also possibly being a regex. There is a lot of
ambiguity because a table named "a" could be a literal value or a legal
regex.  When a table name is defined as a regex, a different code path is
taken from when a table name is considered to be a literal value. Where I
first saw this issue was in a qtest where a table name `s/c` was producing
a different result than a table named `s+c`.

This regex feature is not something I've seen in MySQL or Postgres.  In
MySQL, any table name surrounded with a back tick can be just about any
UTF-8 character, so it's not really feasible to tell, without some kind of
SQL hint, that this table name is a regex or a literal value.

This feature adds a lot of ambiguity and complexity, it is not supported by
other major RDBMS, and it adds only very minor benefit.  I also hope to
move Hive in a direction of fully supporting UTF-8.

I have put a patch up to remove it:
https://issues.apache.org/jira/browse/HIVE-23183


References:
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Select#LanguageManualSelect-REGEXColumnSpecification


https://dev.mysql.com/doc/refman/8.0/en/identifiers.html


Thanks,
David

[jira] [Created] (HIVE-23202) Remove unused q.out files, where the q file no longer exists

[Miklos Gergely (Jira)]

Miklos Gergely created HIVE-23202:
-

 Summary: Remove unused q.out files, where the q file no longer 
exists
 Key: HIVE-23202
 URL: https://issues.apache.org/jira/browse/HIVE-23202
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Reporter: Miklos Gergely
Assignee: Miklos Gergely






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23201) Improve logging in locking

[Marton Bod (Jira)]

Marton Bod created HIVE-23201:
-

 Summary: Improve logging in locking
 Key: HIVE-23201
 URL: https://issues.apache.org/jira/browse/HIVE-23201
 Project: Hive
  Issue Type: Improvement
Reporter: Marton Bod
Assignee: Marton Bod


Currently it can be quite difficult to troubleshoot issues related to locking. 
To understand why a particular txn gave up after a while on acquiring a lock, 
you have to connect directly to the backend DB, since we are not logging right 
now which exact locks the txn is waiting for.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Review Request 72360: HIVE-23093: Create new metastore config value for jdbc max batch size

[Marton Bod]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72360/
---

Review request for hive, Denys Kuzmenko and Peter Vary.


Repository: hive-git


Description
---

HIVE-23093: Create new metastore config value for jdbc max batch size


Diffs
-

  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 3bfb0e69cb 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java
 620c77e589 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 7d0db0c3a0 


Diff: https://reviews.apache.org/r/72360/diff/1/


Testing
---


Thanks,

Marton Bod

[jira] [Created] (HIVE-23200) Remove semijoin_reddedup without '.q' from testconfiguration.properties

[Miklos Gergely (Jira)]

Miklos Gergely created HIVE-23200:
-

 Summary: Remove semijoin_reddedup without '.q' from 
testconfiguration.properties
 Key: HIVE-23200
 URL: https://issues.apache.org/jira/browse/HIVE-23200
 Project: Hive
  Issue Type: Improvement
Reporter: Miklos Gergely






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23199) Apache Hive Upgrade 3.1.2 each query is taking 10 seconds additional

[Praneeth Putha (Jira)]

Praneeth Putha created HIVE-23199:
-

 Summary: Apache Hive Upgrade 3.1.2 each query is taking 10 seconds 
additional
 Key: HIVE-23199
 URL: https://issues.apache.org/jira/browse/HIVE-23199
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 3.1.2
Reporter: Praneeth Putha


We upgraded Apache Hive Server2 and Metastore service from 2.3.4 to 3.1.2 and 
hadoop jars from 2.8.5 to 3.2.0.

Use  or show databases is taking 10.2 seconds before upgrade it is 
0.2 seconds.

Observations:

when i run use database. Compiling comand is completed in 0.73 seconds and to 
start execution query it took 10 seconds.

 

 

2020-04-14T12:22:48,173 INFO [58ad961c-26a8-41a0-a628-eb56c18e7ae2 main] 
conf.HiveConf: Using the default value passed in for log id: 
58ad961c-26a8-41a0-a628-eb56c18e7ae2
2020-04-14T12:22:48,246 INFO [58ad961c-26a8-41a0-a628-eb56c18e7ae2 main] 
ql.Driver: Compiling 
command(queryId=root_20200414122248_aa6e88cc-9490-4ec2-aee5-ea0bc076dbb2): use 
test_database
2020-04-14T12:22:48,915 INFO [58ad961c-26a8-41a0-a628-eb56c18e7ae2 main] 
ql.Driver: Concurrency mode is disabled, not creating a lock manager
2020-04-14T12:22:48,942 INFO [58ad961c-26a8-41a0-a628-eb56c18e7ae2 main] 
ql.Driver: Semantic Analysis Completed (retrial = false)
2020-04-14T12:22:48,944 INFO [58ad961c-26a8-41a0-a628-eb56c18e7ae2 main] 
ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null)
2020-04-14T12:22:48,949 INFO [58ad961c-26a8-41a0-a628-eb56c18e7ae2 main] 
metadata.Hive: Dumping metastore api call timing information for : compilation 
phase
2020-04-14T12:22:48,949 INFO [58ad961c-26a8-41a0-a628-eb56c18e7ae2 main] 
metadata.Hive: Total time spent in this metastore function was greater than 
1000ms : getTableObjectsByName_(String, List, )=2884
2020-04-14T12:22:48,949 INFO [58ad961c-26a8-41a0-a628-eb56c18e7ae2 main] 
metadata.Hive: Total time spent in this metastore function was greater than 
1000ms : getTables_(String, String, TableType, )=16779
2020-04-14T12:22:48,951 INFO [58ad961c-26a8-41a0-a628-eb56c18e7ae2 main] 
ql.Driver: Completed compiling 
command(queryId=root_20200414122248_aa6e88cc-9490-4ec2-aee5-ea0bc076dbb2); Time 
taken: 0.73 seconds
2020-04-14T12:22:48,951 INFO [58ad961c-26a8-41a0-a628-eb56c18e7ae2 main] 
reexec.ReExecDriver: Execution #1 of query
2020-04-14T12:22:48,952 INFO [58ad961c-26a8-41a0-a628-eb56c18e7ae2 main] 
ql.Driver: Concurrency mode is disabled, not creating a lock manager
2020-04-14T12:22:48,952 INFO [58ad961c-26a8-41a0-a628-eb56c18e7ae2 main] 
ql.Driver: Executing 
command(queryId=root_20200414122248_aa6e88cc-9490-4ec2-aee5-ea0bc076dbb2): use 
test_database

{color:#FF} *– 10 seconds pause*{color} 

2020-04-14T12:22:58,981 INFO [58ad961c-26a8-41a0-a628-eb56c18e7ae2 main] 
ql.Driver: Starting task [Stage-0:DDL] in serial mode
2020-04-14T12:22:59,003 INFO [58ad961c-26a8-41a0-a628-eb56c18e7ae2 main] 
ql.Driver: Completed executing 
command(queryId=root_20200414122248_aa6e88cc-9490-4ec2-aee5-ea0bc076dbb2); Time 
taken: 10.051 seconds
2020-04-14T12:22:59,004 INFO [58ad961c-26a8-41a0-a628-eb56c18e7ae2 main] 
ql.Driver: OK
2020-04-14T12:22:59,004 INFO [58ad961c-26a8-41a0-a628-eb56c18e7ae2 main] 
ql.Driver: Concurrency mode is disabled, not creating a lock manager
2020-04-14T12:22:59,017 INFO [58ad961c-26a8-41a0-a628-eb56c18e7ae2 main] 
CliDriver: Time taken: 10.787 seconds
2020-04-14T12:22:59,028 INFO [58ad961c-26a8-41a0-a628-eb56c18e7ae2 main] 
conf.HiveConf: Using the default value passed in for log id: 
58ad961c-26a8-41a0-a628-eb56c18e7ae2
2020-04-14T12:22:59,029 INFO [58ad961c-26a8-41a0-a628-eb56c18e7ae2 main] 
session.SessionState: Resetting thread name to main

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72359: HIVE-23104: Minimize critical paths of TxnHandler::commitTxn and abortTxn

[Peter Vary via Review Board]

> On ápr. 14, 2020, 11:22 de, Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
> > Lines 1325 (patched)
> > 
> >
> > Not really important since this is a private method, but why not return 
> > boolean instead?
> 
> Marton Bod wrote:
> Would be nice, but we need the ResultSet to log out the details of what 
> exact conflict has been found.

Valid point :)


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72359/#review220308
---


On ápr. 14, 2020, 8:57 de, Marton Bod wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72359/
> ---
> 
> (Updated ápr. 14, 2020, 8:57 de)
> 
> 
> Review request for hive, Denys Kuzmenko and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-23104: Minimize critical paths of TxnHandler::commitTxn and abortTxn
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  7d0db0c3a0 
> 
> 
> Diff: https://reviews.apache.org/r/72359/diff/2/
> 
> 
> Testing
> ---
> 
> Green build: 
> https://builds.apache.org/job/PreCommit-HIVE-Build/21539/testReport
> Benchmark results attached to ticket: 
> https://issues.apache.org/jira/browse/HIVE-23104
> 
> 
> Thanks,
> 
> Marton Bod
> 
>

[jira] [Created] (HIVE-23198) Add matching logic between CacheTags and proactive eviction requests

[Jira]

Ádám Szita created HIVE-23198:
-

 Summary: Add matching logic between CacheTags and proactive 
eviction requests
 Key: HIVE-23198
 URL: https://issues.apache.org/jira/browse/HIVE-23198
 Project: Hive
  Issue Type: Sub-task
Reporter: Ádám Szita


Implement ProactiveEviction$Request#isTagMatch so that LLAP can evict buffers 
based on their tags matching incoming eviction requests.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23197) Implement proactive eviction in cache policies

[Jira]

Ádám Szita created HIVE-23197:
-

 Summary: Implement proactive eviction in cache policies
 Key: HIVE-23197
 URL: https://issues.apache.org/jira/browse/HIVE-23197
 Project: Hive
  Issue Type: Sub-task
Reporter: Ádám Szita
Assignee: Ádám Szita


Add eviction functionality parts of proactive caching on daemon side. Cache 
policies will have to implement evictEntity method where buffers have to be 
evicted based on the incoming predicate.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72359: HIVE-23104: Minimize critical paths of TxnHandler::commitTxn and abortTxn

[Marton Bod]

> On April 14, 2020, 11:22 a.m., Peter Vary wrote:
> > Thanks for the patch Marton!
> > Some questions, ideas.
> > 
> > Thanks,
> > Peter

Thanks for the review Peter! 
See my answers below and the updated diff


> On April 14, 2020, 11:22 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
> > Line 1248 (original), 1228 (patched)
> > 
> >
> > Is this 'rs' not reused later? Maybe use a local scoped rs here?

No, it's not reused anymore. I've made it locally-scoped.


> On April 14, 2020, 11:22 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
> > Lines 1325 (patched)
> > 
> >
> > Not really important since this is a private method, but why not return 
> > boolean instead?

Would be nice, but we need the ResultSet to log out the details of what exact 
conflict has been found.


> On April 14, 2020, 11:22 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
> > Lines 1351 (patched)
> > 
> >
> > Maybe log the query in debug level?

Sure


> On April 14, 2020, 11:22 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
> > Lines 1375 (patched)
> > 
> >
> > Might worth to initialize with a size

Done


- Marton


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72359/#review220308
---


On April 14, 2020, 8:57 a.m., Marton Bod wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72359/
> ---
> 
> (Updated April 14, 2020, 8:57 a.m.)
> 
> 
> Review request for hive, Denys Kuzmenko and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-23104: Minimize critical paths of TxnHandler::commitTxn and abortTxn
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  7d0db0c3a0 
> 
> 
> Diff: https://reviews.apache.org/r/72359/diff/1/
> 
> 
> Testing
> ---
> 
> Green build: 
> https://builds.apache.org/job/PreCommit-HIVE-Build/21539/testReport
> Benchmark results attached to ticket: 
> https://issues.apache.org/jira/browse/HIVE-23104
> 
> 
> Thanks,
> 
> Marton Bod
> 
>

Re: Review Request 72359: HIVE-23104: Minimize critical paths of TxnHandler::commitTxn and abortTxn

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72359/#review220308
---



Thanks for the patch Marton!
Some questions, ideas.

Thanks,
Peter


standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Line 1248 (original), 1228 (patched)


Is this 'rs' not reused later? Maybe use a local scoped rs here?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Lines 1325 (patched)


Not really important since this is a private method, but why not return 
boolean instead?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Lines 1351 (patched)


Maybe log the query in debug level?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Lines 1375 (patched)


Might worth to initialize with a size


- Peter Vary


On ápr. 14, 2020, 8:57 de, Marton Bod wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72359/
> ---
> 
> (Updated ápr. 14, 2020, 8:57 de)
> 
> 
> Review request for hive, Denys Kuzmenko and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-23104: Minimize critical paths of TxnHandler::commitTxn and abortTxn
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  7d0db0c3a0 
> 
> 
> Diff: https://reviews.apache.org/r/72359/diff/1/
> 
> 
> Testing
> ---
> 
> Green build: 
> https://builds.apache.org/job/PreCommit-HIVE-Build/21539/testReport
> Benchmark results attached to ticket: 
> https://issues.apache.org/jira/browse/HIVE-23104
> 
> 
> Thanks,
> 
> Marton Bod
> 
>

Re: Review Request 72276: HIVE-23084: Implement kill query in multiple HS2 environment

[Adam Szita via Review Board]

> On April 9, 2020, 9:09 a.m., Adam Szita wrote:
> > Ship It!

Committed. This can be closed.


- Adam


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72276/#review220266
---


On April 6, 2020, 10:04 a.m., Peter Varga wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72276/
> ---
> 
> (Updated April 6, 2020, 10:04 a.m.)
> 
> 
> Review request for hive and Adam Szita.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> KILL  command was implemented in:
> 
> https://issues.apache.org/jira/browse/HIVE-17483
> https://issues.apache.org/jira/browse/HIVE-20549
> But it is not working in an environment where service discovery is enabled 
> and more than one HS2 instance is running (except for manually sending the 
> kill query to all HS2 instance).
> 
> Solution:
> 
> If a HS2 instance can't kill a query locally, it should post a kill query 
> request to the Zookeeper
> Every HS2 should watch the Zookeeper for kill query requests and if its 
> running on that instance kill it
> Authorization of kill query should work the same
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 73f185a1f3 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/BaseJdbcWithMiniLlap.java 
> 3973ec9270 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniLlapArrow.java
>  68a515ccbe 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithServiceDiscovery.java
>  PRE-CREATION 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/thrift/TestMiniHS2StateWithNoZookeeper.java
>  99e681e5b2 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/server/TestKillQueryZookeeperManager.java
>  PRE-CREATION 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 
> 1b60a51ebd 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java db965e7a22 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/process/kill/KillQueriesOperation.java
>  afde1a4762 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 
> 8becef1cd3 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 9e497545b5 
>   service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
> 277519cba5 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 181ea5d6d5 
>   service/src/java/org/apache/hive/service/server/KillQueryImpl.java 
> 883e32bd2e 
>   
> service/src/java/org/apache/hive/service/server/KillQueryZookeeperManager.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/common/ZooKeeperHiveHelper.java
>  71d8651712 
> 
> 
> Diff: https://reviews.apache.org/r/72276/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Peter Varga
> 
>

Review Request 72359: HIVE-23104: Minimize critical paths of TxnHandler::commitTxn and abortTxn

[Marton Bod]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72359/
---

Review request for hive, Denys Kuzmenko and Peter Vary.


Repository: hive-git


Description
---

HIVE-23104: Minimize critical paths of TxnHandler::commitTxn and abortTxn


Diffs
-

  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 7d0db0c3a0 


Diff: https://reviews.apache.org/r/72359/diff/1/


Testing
---

Green build: https://builds.apache.org/job/PreCommit-HIVE-Build/21539/testReport
Benchmark results attached to ticket: 
https://issues.apache.org/jira/browse/HIVE-23104


Thanks,

Marton Bod

[jira] [Created] (HIVE-23196) Reduce number of delete calls to NN during Context::clear

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-23196:
---

 Summary: Reduce number of delete calls to NN during Context::clear
 Key: HIVE-23196
 URL: https://issues.apache.org/jira/browse/HIVE-23196
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


{\{Context::clear()}} ends up deleting same directories (or its subdirs) 
multiple times. It would be good to reduce the number of delete calls to NN for 
latency sensitive queries. This also has an impact on concurrent queries.

{noformat}
2020-04-14T04:22:28,703 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 
HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting result dir: 
hdfs://nn1:8020/tmp/hive/rbalamohan/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13/-mr-1
2020-04-14T04:22:28,721 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 
HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting scratch dir: 
hdfs://nn1:8020/tmp/hive/rbalamohan/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13
2020-04-14T04:22:28,737 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 
HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting scratch dir: 
hdfs://nn1:8020/tmp/hive/rbalamohan/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13/-mr-1/.hive-staging_hive_2020-04-14_04-22-24_335_8573832618972595103-13
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72112: HIVE-22869 - Add locking benchmark to metastore-tools/metastore-benchmarks

[Peter Vary via Review Board]

> On ápr. 3, 2020, 9:59 de, Peter Vary wrote:
> > standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
> > Lines 341 (patched)
> > 
> >
> > I put this in the code with HIVE-23042:
> >   boolean openTxn(int numTxns) throws TException {
> > client.open_txns(new OpenTxnRequest(numTxns, "Test", "Host"));
> > return true;
> >   }
> >   
> > Maybe merge those?
> 
> Zoltan Chovan wrote:
> The main difference between our two implementations of openTxn is that 
> mine automatically returns the opened txn's id, in your version there has to 
> be an additional getOpenTxns() call made to get the Id. 
> Not sure if getOpenTxns would return some other ids that belong to an 
> other client when multiple threads are used, o sI might be misunderstanding 
> the getOpenTxns() call.
> What do you think?

I still think it would be worth to keep only one version. In this specific case 
your new version, and use it on the other place.
What do you think?


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220212
---


On ápr. 9, 2020, 2:58 du, Zoltan Chovan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> ---
> 
> (Updated ápr. 9, 2020, 2:58 du)
> 
> 
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter 
> Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Add the possibility to run benchmarks on opening lock in the HMS. Currently 
> this change only introduces single-threaded/single client testing. I'm 
> planning to add multi-client support in a separate change.
> 
> Example parametrisation is as follows:
> hbench -M "lock" -N 10 -d hive_test -W 0 -L 100
> hbench -M ".*Lock.*" -N 10 -d hive_test -W 0 -L 100 -T 8 --params 100
> 
> This will create N number (10) of tables to lock and it'll execute the lock() 
> for L number (100) of times on T (8) threads where each thread will strart an 
> HMS client
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java
>  041cd76234 
>   
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
>  f53f2ef43b 
>   
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
>  7cc1e42a8b 
>   
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
>  101d6759c5 
> 
> 
> Diff: https://reviews.apache.org/r/72112/diff/1/
> 
> 
> Testing
> ---
> 
> 
> File Attachments
> 
> 
> HIVE-22869.2.patch
>   
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
> HIVE-22869.3.patch
>   
> https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
> 
> 
> Thanks,
> 
> Zoltan Chovan
> 
>