[jira] [Created] (HIVE-23099) Improve Logger for Operation Child Classes

[David Mollitor (Jira)]

David Mollitor created HIVE-23099:
-

 Summary: Improve Logger for Operation Child Classes
 Key: HIVE-23099
 URL: https://issues.apache.org/jira/browse/HIVE-23099
 Project: Hive
  Issue Type: Improvement
Reporter: David Mollitor
Assignee: David Mollitor


The {{Operation}} class declares its logger this way:

{code:java|title=Operation.java}
public abstract class Operation {
  public static final Logger LOG = 
LoggerFactory.getLogger(Operation.class.getName());
  ...
}
{code}

Notice that this is an {{abstract}} class, but the {{Logger}} is tied to the 
{{Operation.class.getName()}}.  This means that logging cannot be controlled 
for each subclass of {{Operation}} independently since they all use the same 
static {{Logger}} instance.

Make the LOG a {{protected}} instance variable that inherits the name of the 
child class.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23098) Allow Operation assertState to Accept a Collection

[David Mollitor (Jira)]

David Mollitor created HIVE-23098:
-

 Summary: Allow Operation assertState to Accept a Collection
 Key: HIVE-23098
 URL: https://issues.apache.org/jira/browse/HIVE-23098
 Project: Hive
  Issue Type: Improvement
Reporter: David Mollitor
Assignee: David Mollitor
 Attachments: HIVE-23098.1.patch

{code:java|title=Operation.java}
  protected final void assertState(List states) throws 
HiveSQLException {
if (!states.contains(state)) {
  throw new HiveSQLException("Expected states: " + states.toString() + ", 
but found "
  + this.state);
}
this.lastAccessTime = System.currentTimeMillis();
  }
/*/
public void someMethod() {
assertState(new 
ArrayList(Arrays.asList(OperationState.FINISHED)));
}
{code}

By allowing {{assertState}} to accept a {{Collection}}, one can save an 
allocation and simplify the code:

{code:java}
assertState(Collections.singleton(OperationState.FINISHED));
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23097) LLAP: LLAPServiceDriver is wrongly using fixed llap- prefix for tarball name

[Ramesh Kumar Thangarajan (Jira)]

Ramesh Kumar Thangarajan created HIVE-23097:
---

 Summary: LLAP: LLAPServiceDriver is wrongly using fixed llap- 
prefix for tarball name
 Key: HIVE-23097
 URL: https://issues.apache.org/jira/browse/HIVE-23097
 Project: Hive
  Issue Type: Bug
Reporter: Ramesh Kumar Thangarajan


LLAP: LLAPServiceDriver is wrongly using fixed llap- prefix for tarball name

Missed modifying this change too in the Jira 
https://issues.apache.org/jira/browse/HIVE-22937 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23096) Detect Small Fetch Size on HS2

[David Mollitor (Jira)]

David Mollitor created HIVE-23096:
-

 Summary: Detect Small Fetch Size on HS2
 Key: HIVE-23096
 URL: https://issues.apache.org/jira/browse/HIVE-23096
 Project: Hive
  Issue Type: Sub-task
Reporter: David Mollitor


See if there's an easy and useful way of detecting when user's are using a 
small fetch size inadvertently.  For example, a JDBC fetchSize of 1, for a 
result set of 1,000,000 rows across a high-latency link may crush performance.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23095) NDV might be overestimated for a table with ~70 value

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HIVE-23095:
---

 Summary: NDV might be overestimated for a table with ~70 value
 Key: HIVE-23095
 URL: https://issues.apache.org/jira/browse/HIVE-23095
 Project: Hive
  Issue Type: Bug
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


uncovered during looking into HIVE-23082
https://issues.apache.org/jira/browse/HIVE-23082?focusedCommentId=17067773=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17067773



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72266: HIVE-23075 Add property for manual configuration of SSL version

[Oleksiy Sayankin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72266/
---

(Updated March 27, 2020, 4:43 p.m.)


Review request for hive.


Repository: hive-git


Description
---

Initial commit


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/auth/HiveAuthUtils.java 
c976285f47 
  
common/src/java/org/apache/hadoop/hive/common/auth/HiveServer2TSSLTransportParameters.java
 PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d50912b4e2 
  common/src/java/org/apache/hive/http/HttpServer.java 51a2be2bdf 
  jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java cbf663225e 
  
service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java 
df2d3a7b71 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 205c867db1 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 fc6a2fd43a 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/security/MetastoreTSSLTransportParameters.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/SecurityUtils.java
 bae1ec35b2 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 be4b3794de 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 fc071f9a20 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
 4e75edeae6 


Diff: https://reviews.apache.org/r/72266/diff/2/

Changes: https://reviews.apache.org/r/72266/diff/1-2/


Testing
---


Thanks,

Oleksiy Sayankin

[jira] [Created] (HIVE-23094) Implement Explain CBO of Update and Delete statements

[Krisztian Kasa (Jira)]

Krisztian Kasa created HIVE-23094:
-

 Summary: Implement Explain CBO of Update and Delete statements
 Key: HIVE-23094
 URL: https://issues.apache.org/jira/browse/HIVE-23094
 Project: Hive
  Issue Type: Improvement
  Components: CBO
Reporter: Krisztian Kasa
Assignee: Krisztian Kasa


{code}
set hive.support.concurrency=true;
set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;

create table acidtlb(a int, b int) clustered by (a) into 2 buckets stored as 
orc TBLPROPERTIES ('transactional'='true');

explain cbo
update acidtlb set b=777;
{code}
doesn't print CBO plan.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 71812: HIVE-22534: ACID: Improve Compactor thread logging

[Karen Coppage via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71812/#review220095
---



LGTM, one suggestion below.
Also Zoltan Chovan seemed excited about this change, maybe consider asking him 
if this fulfills his logging wishes :)


ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java
Lines 374-375 (original), 360-361 (patched)


Not regression, but it looks like these arguments are switched up:
deltaNumThreshold
noBase ? "without" : "with"


- Karen Coppage


On March 26, 2020, 2:43 p.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71812/
> ---
> 
> (Updated March 26, 2020, 2:43 p.m.)
> 
> 
> Review request for hive and Karen Coppage.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-22534: ACID: Improve Compactor thread logging
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 
> 54b616e60c73fa1005c6d679ea76d65e01a0749d 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 543ec0b99124cb38c8508aa2ec2f99cababdbdbd 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 37a58627913c33be37cee7f11d9ca4ee5fd8aff2 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MinorQueryCompactor.java 
> 1bf0beea4022dbabe65a9d2ee0972186b5cb3396 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMinorQueryCompactor.java
>  383891bfadfc1352430e50eac2d60366ad699bf0 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 
> a96cf1e73183232d587b3f4937c0d7e96764e662 
> 
> 
> Diff: https://reviews.apache.org/r/71812/diff/7/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

[jira] [Created] (HIVE-23093) Create new metastore config value for jdbc max batch size

[Marton Bod (Jira)]

Marton Bod created HIVE-23093:
-

 Summary: Create new metastore config value for jdbc max batch size
 Key: HIVE-23093
 URL: https://issues.apache.org/jira/browse/HIVE-23093
 Project: Hive
  Issue Type: Improvement
Reporter: Marton Bod
Assignee: Marton Bod


In order to reduce the number of network calls to the metastore DB, there is an 
effort to batch acid/locking DB calls where it makes sense. Existing metastore 
config params do not provide a suitable, generic option to control the jdbc 
statement max batch size. Solution is to create a new config param, and use 
that in the appropriate batching calls in TxnHandler.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23092) Use the SqlSplittableAggFunction in HiveMergeablAggregate functions

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HIVE-23092:
---

 Summary: Use the SqlSplittableAggFunction in HiveMergeablAggregate 
functions
 Key: HIVE-23092
 URL: https://issues.apache.org/jira/browse/HIVE-23092
 Project: Hive
  Issue Type: Sub-task
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


the `SqlSplittableAggFunction` seems to fit our needs; but to use it, some 
further requirements of the interface should be satisfied

https://github.com/apache/hive/pull/960#discussion_r397691314



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23091) SqlFunctionConverter uses both a passed name for the function and the name from UDF's annotation

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HIVE-23091:
---

 Summary: SqlFunctionConverter uses both a passed name for the 
function and the name from UDF's annotation
 Key: HIVE-23091
 URL: https://issues.apache.org/jira/browse/HIVE-23091
 Project: Hive
  Issue Type: Sub-task
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


see if we need both - possibly rename the functions to give some explanation 
whats happening

(background: right now I would like to register some external UDFs which does 
have name set; but I want to register them on a completely different name...)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23090) Introduce a model to describe sub-expression's stat state

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HIVE-23090:
---

 Summary: Introduce a model to describe sub-expression's stat state
 Key: HIVE-23090
 URL: https://issues.apache.org/jira/browse/HIVE-23090
 Project: Hive
  Issue Type: Improvement
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


In 
[StatsRulesProcFactory|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java]
 right now the computation is based on functions which return values of 
numRows; with the extension that the called method may sometimes alter the 
passed Statistics object - this makes the logic a bit trickier here and there - 
having a real model for a subexpression could give natural benefits and may 
help in fixing issues like: HIVE-22238, HIVE-23082



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23089) Add constraint checks to CBO plan

[Krisztian Kasa (Jira)]

Krisztian Kasa created HIVE-23089:
-

 Summary: Add constraint checks to CBO plan
 Key: HIVE-23089
 URL: https://issues.apache.org/jira/browse/HIVE-23089
 Project: Hive
  Issue Type: Improvement
  Components: CBO
Reporter: Krisztian Kasa
Assignee: Krisztian Kasa


{code}
create table acid_uami(i int,
 de decimal(5,2) constraint nn1 not null enforced,
 vc varchar(128) constraint nn2 not null enforced) clustered by 
(i) into 2 buckets stored as orc TBLPROPERTIES ('transactional'='true');
explain
update acid_uami set de=null where i=1;
{code}
Non-CBO path:
{code:java}
Map Operator Tree:
TableScan
alias: acid_uami
filterExpr: ((i = 1) and enforce_constraint(vc is not null)) (type: 
boolean)
Statistics: Num rows: 1 Data size: 216 Basic stats: COMPLETE Column 
stats: NONE
Filter Operator
  predicate: ((i = 1) and enforce_constraint(vc is not null)) 
(type: boolean)
{code}
CBO path:
{code:java}
Map Reduce
  Map Operator Tree:
  TableScan
alias: acid_uami
filterExpr: (i = 1) (type: boolean)
Statistics: Num rows: 1 Data size: 216 Basic stats: COMPLETE Column 
stats: NONE
Filter Operator
  predicate: (i = 1) (type: boolean)
...
  Reduce Operator Tree:
...
 Filter Operator
predicate: enforce_constraint((null is not null and _col3 is not 
null)) (type: boolean)
{code}

In CBO path the enforce_constraint function is added to the plan when CBO plan 
is already generated and optimized.
{code}
HiveSortExchange(distribution=[any], collation=[[0]])
  HiveProject(row__id=[$5], i=[CAST(1):INTEGER], _o__c2=[null:NULL], vc=[$2])
HiveFilter(condition=[=($0, 1)])
  HiveTableScan(table=[[default, acid_uami]], table:alias=[acid_uami])
{code} 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23088) Using Strings from log4j breaks non-log4j users

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created HIVE-23088:
-

 Summary: Using Strings from log4j breaks non-log4j users
 Key: HIVE-23088
 URL: https://issues.apache.org/jira/browse/HIVE-23088
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.1.2
Reporter: Vova Vysotskyi


{{HookUtils}} uses explicitly {{org.apache.logging.log4j.util.Strings}} class 
from log4j, but it may break clients who use other loggers and should exclude 
log4j from the classpath.

{{commons-lang}} has class {{StringUtils}} which may be used as a replacement 
for this one:
 {{Strings.isBlank}} -> {{StringUtils.isBlank}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Review Request 72276: HIVE-23084: Implement kill query in multiple HS2 environment

[Peter Varga via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72276/
---

Review request for hive and Adam Szita.


Repository: hive-git


Description
---

KILL  command was implemented in:

https://issues.apache.org/jira/browse/HIVE-17483
https://issues.apache.org/jira/browse/HIVE-20549
But it is not working in an environment where service discovery is enabled and 
more than one HS2 instance is running (except for manually sending the kill 
query to all HS2 instance).

Solution:

If a HS2 instance can't kill a query locally, it should post a kill query 
request to the Zookeeper
Every HS2 should watch the Zookeeper for kill query requests and if its running 
on that instance kill it
Authorization of kill query should work the same


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 34df01e60e 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/BaseJdbcWithMiniLlap.java 
3973ec9270 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniLlapArrow.java
 68a515ccbe 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithServiceDiscovery.java
 PRE-CREATION 
  
itests/hive-unit/src/test/java/org/apache/hive/service/cli/thrift/TestMiniHS2StateWithNoZookeeper.java
 99e681e5b2 
  
itests/hive-unit/src/test/java/org/apache/hive/service/server/TestKillQueryZookeeperManager.java
 PRE-CREATION 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 
1b60a51ebd 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 
8becef1cd3 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
9e497545b5 
  service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
277519cba5 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 181ea5d6d5 
  service/src/java/org/apache/hive/service/server/KillQueryImpl.java 883e32bd2e 
  
service/src/java/org/apache/hive/service/server/KillQueryZookeeperManager.java 
PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/common/ZooKeeperHiveHelper.java
 71d8651712 


Diff: https://reviews.apache.org/r/72276/diff/1/


Testing
---


Thanks,

Peter Varga

Re: Review request for HIVE-22983

[Zoltan Haindrich]

Hey Zhihua!

I think it's good as is :)
Next time if you would like someone to review your changes; the usual routine 
is to mention someone on the jira.
You could look for "reviewer candidates" by:
* annotating some of the changed files; and take a look at commit date/what 
kind of change was it
* ...or peeking into the commit history of the actual file or package

cheers,
Zoltan

On 3/27/20 8:36 AM, dengzhhu653 wrote:

Hi all
  
 A small fix on comments of ConstantPropagate, Can someone help review this the code changes or give feedback on it?

 https://issues.apache.org/jira/browse/HIVE-22983


 Thanks
 Zhihua

Review request for HIVE-22983

[dengzhhu653]

Hi all
 
A small fix on comments of ConstantPropagate, Can someone help review this 
the code changes or give feedback on it? 
https://issues.apache.org/jira/browse/HIVE-22983


Thanks
Zhihua