RE: [EXTERNAL] Welcome Anishek To Apache Hive PMC

[Sankar Hariappan]

Congrats Anishek! Well deserved... 

Thanks,
Sankar

-Original Message-
From: Ashutosh Chauhan  
Sent: Tuesday, February 11, 2020 9:40 AM
To: dev@hive.apache.org
Subject: [EXTERNAL] Welcome Anishek To Apache Hive PMC

I'm happy to announce Anishek Agarwal as the latest addition to the Apache Hive 
Project Management Committee (PMC).

He has been an important committer to the project and active member of the 
community helping advance Apache Hive.

Congratulations, and thank you for your hard work

Thanks,
Ashutosh

RE: [EXTERNAL] Welcome Mahesh to Hive PMC

[Sankar Hariappan]

Congrats Mahesh! Well deserved.. 

Thanks,
Sankar

-Original Message-
From: Ashutosh Chauhan  
Sent: Tuesday, February 11, 2020 9:37 AM
To: dev@hive.apache.org
Subject: [EXTERNAL] Welcome Mahesh to Hive PMC

Hi all,

It's an honor to announce that Apache Hive PMC has recently voted to invite 
Mahesh Kumar Behera as a new Hive PMC member. Mahesh is a long time Hive 
contributor and committer, and has made significant contribution in Hive.
Please join me in congratulating him and looking forward to a bigger role that 
he will play in Apache Hive project.

Thanks,
Ashutosh

[jira] [Created] (HIVE-22879) Optimise jar file loading in CalcitePlanner

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-22879:
---

 Summary: Optimise jar file loading in CalcitePlanner
 Key: HIVE-22879
 URL: https://issues.apache.org/jira/browse/HIVE-22879
 Project: Hive
  Issue Type: Improvement
  Components: CBO
Reporter: Rajesh Balamohan


{{CalcitePlanner }} internally uses {{org.codehaus.janino.UnitCompiler (calcite 
dependency)}} and this appears to load the jars in every thread. Need to check 
if this can be avoided.

Here is an example.

{noformat}
at java.util.zip.ZipFile.getEntry(Native Method)
at java.util.zip.ZipFile.getEntry(ZipFile.java:310)
- locked <0x0005c1af21c0> (a java.util.jar.JarFile)
at java.util.jar.JarFile.getEntry(JarFile.java:240)
at java.util.jar.JarFile.getJarEntry(JarFile.java:223)
at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1005)
at sun.misc.URLClassPath.getResource(URLClassPath.java:212)
at java.net.URLClassLoader$1.run(URLClassLoader.java:365)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
- locked <0x0005caa3be88> (a java.lang.Object)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at 
org.codehaus.janino.ClassLoaderIClassLoader.findIClass(ClassLoaderIClassLoader.java:89)
at org.codehaus.janino.IClassLoader.loadIClass(IClassLoader.java:312)
- locked <0x000686136868> (a 
org.codehaus.janino.ClassLoaderIClassLoader)
at 
org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java:8556)
at 
org.codehaus.janino.UnitCompiler.reclassifyName(UnitCompiler.java:8478)
at 
org.codehaus.janino.UnitCompiler.reclassifyName(UnitCompiler.java:8471)
at org.codehaus.janino.UnitCompiler.reclassify(UnitCompiler.java:8331)
at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:6855)
at org.codehaus.janino.UnitCompiler.access$14200(UnitCompiler.java:215)
at 
org.codehaus.janino.UnitCompiler$22$2$1.visitAmbiguousName(UnitCompiler.java:6497)
at 
org.codehaus.janino.UnitCompiler$22$2$1.visitAmbiguousName(UnitCompiler.java:6494)
at org.codehaus.janino.Java$AmbiguousName.accept(Java.java:4224)
at 
org.codehaus.janino.UnitCompiler$22$2.visitLvalue(UnitCompiler.java:6494)
at 
org.codehaus.janino.UnitCompiler$22$2.visitLvalue(UnitCompiler.java:6490)
at org.codehaus.janino.Java$Lvalue.accept(Java.java:4148)
at 
org.codehaus.janino.UnitCompiler$22.visitRvalue(UnitCompiler.java:6490)
at 
org.codehaus.janino.UnitCompiler$22.visitRvalue(UnitCompiler.java:6469)
at org.codehaus.janino.Java$Rvalue.accept(Java.java:4116)
at org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:6469)
at org.codehaus.janino.UnitCompiler.findIMethod(UnitCompiler.java:9026)
at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:7106)
at org.codehaus.janino.UnitCompiler.access$15800(UnitCompiler.java:215)
at 
org.codehaus.janino.UnitCompiler$22$2.visitMethodInvocation(UnitCompiler.java:6517)
at 
org.codehaus.janino.UnitCompiler$22$2.visitMethodInvocation(UnitCompiler.java:6490)
at org.codehaus.janino.Java$MethodInvocation.accept(Java.java:5073)
at 
org.codehaus.janino.UnitCompiler$22.visitRvalue(UnitCompiler.java:6490)
at 
org.codehaus.janino.UnitCompiler$22.visitRvalue(UnitCompiler.java:6469)
at org.codehaus.janino.Java$Rvalue.accept(Java.java:4116)
at org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:6469)
at 
org.codehaus.janino.UnitCompiler.findMostSpecificIInvocable(UnitCompiler.java:9237)
at org.codehaus.janino.UnitCompiler.findIMethod(UnitCompiler.java:9123)
at org.codehaus.janino.UnitCompiler.findIMethod(UnitCompiler.java:9025)
at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:5062)
at org.codehaus.janino.UnitCompiler.access$9100(UnitCompiler.java:215)
at 
org.codehaus.janino.UnitCompiler$16.visitMethodInvocation(UnitCompiler.java:4423)
at 
org.codehaus.janino.UnitCompiler$16.visitMethodInvocation(UnitCompiler.java:4396)
at org.codehaus.janino.Java$MethodInvocation.accept(Java.java:5073)
at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:4396)
at 
org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:5662)
at 
org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:5622)
at 

Re: Review Request 72113: DML execution on TEZ always outputs the message 'No rows affected'

[Mustafa Iman]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72113/#review219553
---



+1

- Mustafa Iman


On Feb. 11, 2020, 8:01 a.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72113/
> ---
> 
> (Updated Feb. 11, 2020, 8:01 a.m.)
> 
> 
> Review request for hive, Laszlo Bodor, Mustafa Iman, Panos Garefalakis, and 
> Ramesh Kumar Thangarajan.
> 
> 
> Bugs: HIVE-22870
> https://issues.apache.org/jira/browse/HIVE-22870
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Executing an update or insert statement in beeline doesn't show the actual 
> rows inserted/updated.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java 25dd970a9b1 
> 
> 
> Diff: https://reviews.apache.org/r/72113/diff/1/
> 
> 
> Testing
> ---
> 
> with insert and updates
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Re: [Announce] New committer : Laszlo Pinter

[Chinna Rao Lalam]

Congratulations Laszlo !

On Tue, Feb 11, 2020 at 11:12 AM Anishek Agarwal
 wrote:

> Congratulations Lazlo
>
> On Tue, Feb 11, 2020 at 9:44 AM Ashutosh Chauhan 
> wrote:
>
> > Apache Hive's Project Management Committee (PMC) has invited Laszlo
> Pinter
> > to become a committer, and we are pleased to announce that he has
> accepted.
> >
> > Laszlo welcome, thank you for your contributions, and we look forward
> your
> > further interactions with the community!
> >
> > Thanks,
> > Ashutosh
> >
>


-- 
Hope It Helps,
Chinna

Re: Welcome Anishek To Apache Hive PMC

[Chinna Rao Lalam]

Congratulations Anishek Agarwal !

On Tue, Feb 11, 2020 at 9:40 AM Ashutosh Chauhan 
wrote:

> I'm happy to announce Anishek Agarwal as the latest addition to the Apache
> Hive Project Management Committee (PMC).
>
> He has been an important committer to the project and active member of the
> community helping advance Apache Hive.
>
> Congratulations, and thank you for your hard work
>
> Thanks,
> Ashutosh
>


-- 
Hope It Helps,
Chinna

Re: Welcome Mahesh to Hive PMC

[Chinna Rao Lalam]

Congratulations Mahesh Kumar Behera !

On Tue, Feb 11, 2020 at 9:36 AM Ashutosh Chauhan 
wrote:

> Hi all,
>
> It's an honor to announce that Apache Hive PMC has recently voted to invite
> Mahesh Kumar Behera as a new Hive PMC member. Mahesh is a long time Hive
> contributor and committer, and has made significant contribution in Hive.
> Please join me in congratulating him and looking forward to a bigger role
> that he will play in Apache Hive project.
>
> Thanks,
> Ashutosh
>


-- 
Hope It Helps,
Chinna

[jira] [Created] (HIVE-22878) Add caching of table constraints, foreignKeys in CachedStore

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-22878:
---

 Summary: Add caching of table constraints, foreignKeys in 
CachedStore
 Key: HIVE-22878
 URL: https://issues.apache.org/jira/browse/HIVE-22878
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Reporter: Rajesh Balamohan
 Attachments: Screenshot 2020-02-12 at 9.24.27 AM.jpg, Screenshot 
2020-02-12 at 9.25.33 AM.jpg

All pink bars are misses from cachedstore.

!Screenshot 2020-02-12 at 9.24.27 AM.jpg|width=428,height=314!

 

!Screenshot 2020-02-12 at 9.25.33 AM.jpg|width=648,height=470!

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22877) Wrong decimal boundary for casting to Decimal64

[Mustafa Iman (Jira)]

Mustafa Iman created HIVE-22877:
---

 Summary: Wrong decimal boundary for casting to Decimal64
 Key: HIVE-22877
 URL: https://issues.apache.org/jira/browse/HIVE-22877
 Project: Hive
  Issue Type: Bug
  Components: Vectorization
Affects Versions: 4.0.0
Reporter: Mustafa Iman
Assignee: Mustafa Iman


During vectorization, decimal fields that are obtained via generic udfs are 
cast to Decimal64 in some circumstances. For decimal to decimal64 cast, hive 
compares the source column's `scale + precision` to 18(maximum number of digits 
that can be represented by a long). A decimal can fit in a long as long as its 
`scale` is smaller than or equal to 18. Precision is irrelevant.

Since vectorized generic udf expression takes precision into account, it 
computes wrong output column vector: Decimal instead of Decimal64. This in turn 
causes ClassCastException down the operator chain.

Below query fails with class cast exception:

 
{code:java}
create table mini_store
(
 s_store_sk int,
 s_store_id string
)
row format delimited fields terminated by '\t'
STORED AS ORC;

create table mini_sales
(
 ss_store_sk int,
 ss_quantity int,
 ss_sales_price decimal(7,2)
)
row format delimited fields terminated by '\t'
STORED AS ORC;
insert into mini_store values (1, 'store');
insert into mini_sales values (1, 2, 1.2);
select s_store_id, coalesce(ss_sales_price*ss_quantity,0) sumsales
from mini_sales, mini_store where ss_store_sk = s_store_sk
{code}
 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large.

[Bernard Quizon]

Hi.

We fixed the issue by patching protobuf-java-2.5.0.jar, we
changed CodedInputStream.DEFAULT_SIZE_LIMIT to 1GB.
Uploaded the patched version on our servers and added the location of the
aforementioned jar to the *tez.cluster.additional.classpath.prefix*
(tez-site.xml)
to /path/to/patched/protobuf-java-2.5.0.jar:.
Please note that it should be the first jar on the
*tez.cluster.additional.classpath.prefix*.
Apparently, Tez was using the default 64MB protobuf message limit.

BTW, latest protobuf version was set to Integer.MAX_VALUE.
See
https://github.com/protocolbuffers/protobuf/blob/v3.11.3/java/core/src/main/java/com/google/protobuf/CodedInputStream.java#L62-L65
.

Regards,
Bernard

On Mon, Feb 10, 2020 at 8:23 PM Bernard Quizon <
bernard.qui...@cheetahdigital.com> wrote:

> Hi.
>
> We're using Hive 3.0.1 and we're currently experiencing this issue:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *Error while processing statement: FAILED: Execution Error, return code 2
> from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed,
> vertexName=Map 1, vertexId=vertex_1581309524541_0094_14_00,
> diagnostics=[Vertex vertex_1581309524541_0094_14_00 [Map 1] killed/failed
> due to:INIT_FAILURE, Fail to create InputInitializerManager,
> org.apache.tez.dag.api.TezReflectionException: Unable to instantiate class
> with 1 arguments: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGeneratorat
> org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:71)at
> org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:89)at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:152)at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:148)at
> java.security.AccessController.doPrivileged(Native Method)at
> javax.security.auth.Subject.doAs(Subject.java:422)at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)at
> org.apache.tez.dag.app.dag.RootInputInitializerManager.createInitializer(RootInputInitializerManager.java:148)at
> org.apache.tez.dag.app.dag.RootInputInitializerManager.runInputInitializers(RootInputInitializerManager.java:121)at
> org.apache.tez.dag.app.dag.impl.VertexImpl.setupInputInitializerManager(VertexImpl.java:4122)at
> org.apache.tez.dag.app.dag.impl.VertexImpl.access$3100(VertexImpl.java:207)at
> org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.handleInitEvent(VertexImpl.java:2932)at
> org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2879)at
> org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2861)at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)at
> org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:59)at
> org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1957)at
> org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:206)at
> org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2317)at
> org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2303)at
> org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:180)at
> org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:115)at
> java.lang.Thread.run(Thread.java:745)Caused by:
> java.lang.reflect.InvocationTargetExceptionat
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)at
> org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:68)...
> 25 moreCaused by: com.google.protobuf.InvalidProtocolBufferException:
> Protocol message was too large.  May be malicious.  Use
> CodedInputStream.setSizeLimit() to increase the size limit.at
> com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)at
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)at
> com.google.protobuf.CodedInputStream.isAtEnd(CodedInputStream.java:701)at
> com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:99)at
> org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto.(DAGProtos.java:19294)at
> org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto.(DAGProtos.java:19258)at
> 

[jira] [Created] (HIVE-22876) Do not enforce package-info.java files by checkstyle

[Miklos Gergely (Jira)]

Miklos Gergely created HIVE-22876:
-

 Summary: Do not enforce package-info.java files by checkstyle
 Key: HIVE-22876
 URL: https://issues.apache.org/jira/browse/HIVE-22876
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Reporter: Miklos Gergely
Assignee: Miklos Gergely
 Fix For: 4.0.0


Currently checkstyle enforces every pacakge to have a package-info.java file. 
This is not really followed by anyone, so it can be removed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22875) Refactor query creation in QueryCompactor implementations

[Karen Coppage (Jira)]

Karen Coppage created HIVE-22875:


 Summary: Refactor query creation in QueryCompactor implementations
 Key: HIVE-22875
 URL: https://issues.apache.org/jira/browse/HIVE-22875
 Project: Hive
  Issue Type: Improvement
Reporter: Karen Coppage
Assignee: Karen Coppage


There is a lot of repetition where creation/compaction/drop queries are created 
in MajorQueryCompactor, MinorQueryCompactor, MmMajorQueryCompactor and 
MmMinorQueryCompactor.

Initial idea is to create a CompactionQueryBuilder that all 4 implementations 
would use.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22874) Beeline unable to use credentials from URL.

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-22874:


 Summary: Beeline unable to use credentials from URL.
 Key: HIVE-22874
 URL: https://issues.apache.org/jira/browse/HIVE-22874
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Reporter: Naveen Gangam
Assignee: Naveen Gangam
 Fix For: 4.0.0


Beeline is not using password value from the URL. 
Using LDAP Auth in this case, so the failure is on connect.
bin/beeline -u "jdbc:hive2://localhost:1/default;user=test1;password=test1" 

On the server side in LdapAuthenticator, the principals come out to (via a 
special debug logging)

2020-02-11T11:10:31,613  INFO [HiveServer2-Handler-Pool: Thread-67] 
auth.LdapAuthenticationProviderImpl: Connecting to ldap as 
user/password:test1:anonymous


This bug may have been introduced via
https://github.com/apache/hive/commit/749e831060381a8ae4775630efb72d5cd040652f

pass = "" ( an empty string on this line) 
https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/BeeLine.java#L848

but on this line of code, it checks to see it is null which will not be true 
and hence it never picks up from the jdbc url
https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/BeeLine.java#L900
It has another chance here but pass != null will always be true and never goes 
into the else condition.
https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/BeeLine.java#L909



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22873) Make it possible to identify which hs2 instance executed a scheduled query

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HIVE-22873:
---

 Summary: Make it possible to identify which hs2 instance executed 
a scheduled query
 Key: HIVE-22873
 URL: https://issues.apache.org/jira/browse/HIVE-22873
 Project: Hive
  Issue Type: Improvement
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


right now only the query_id is shown; in case of multiple hs2 instances the 
question...users have to resort to grepping the logs for the given query id



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22872) Support multiple executors for scheduled queries

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HIVE-22872:
---

 Summary: Support multiple executors for scheduled queries
 Key: HIVE-22872
 URL: https://issues.apache.org/jira/browse/HIVE-22872
 Project: Hive
  Issue Type: Improvement
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Review Request 72113: DML execution on TEZ always outputs the message 'No rows affected'

[Attila Magyar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72113/
---

Review request for hive, Laszlo Bodor, Mustafa Iman, Panos Garefalakis, and 
Ramesh Kumar Thangarajan.


Bugs: HIVE-22870
https://issues.apache.org/jira/browse/HIVE-22870


Repository: hive-git


Description
---

Executing an update or insert statement in beeline doesn't show the actual rows 
inserted/updated.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java 25dd970a9b1 


Diff: https://reviews.apache.org/r/72113/diff/1/


Testing
---

with insert and updates


Thanks,

Attila Magyar

[jira] [Created] (HIVE-22871) Show info message in log when hive server return response with success status with info

[Shaohui Liu (Jira)]

Shaohui Liu created HIVE-22871:
--

 Summary: Show info message in log when hive server return response 
with success status with info
 Key: HIVE-22871
 URL: https://issues.apache.org/jira/browse/HIVE-22871
 Project: Hive
  Issue Type: Improvement
Reporter: Shaohui Liu
Assignee: Shaohui Liu


The info messages in TStatus of HiveServer response are very useful to debug 
the SQL executions. Currently, the info messages are ignored when getting 
TStatus: SUCCESS or SUCCESS_WITH_INFO_STATUS. 

It‘s better to show these info messages in LOG when receiving SUCCESS or 
SUCCESS_WITH_INFO_STATUS TStatus.

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22870) DML execution on TEZ always outputs the message 'No rows affected'

[Attila Magyar (Jira)]

Attila Magyar created HIVE-22870:


 Summary: DML execution on TEZ always outputs the message 'No rows 
affected'
 Key: HIVE-22870
 URL: https://issues.apache.org/jira/browse/HIVE-22870
 Project: Hive
  Issue Type: Bug
Reporter: Attila Magyar
Assignee: Attila Magyar


Executing an update or insert statement in beeline doesn't show the actual rows 
inserted/updated.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72108: HIVE-22867

[Attila Magyar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72108/#review219544
---


Ship it!




Ship It!

- Attila Magyar


On Feb. 11, 2020, 9:58 a.m., Krisztian Kasa wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72108/
> ---
> 
> (Updated Feb. 11, 2020, 9:58 a.m.)
> 
> 
> Review request for hive, Attila Magyar and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-22867
> https://issues.apache.org/jira/browse/HIVE-22867
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Add partitioning support to VectorTopNKeyOperator
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java bd8ff6285e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorTopNKeyOperator.java 
> f03d65030d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 27ff0c2484 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/VectorTopNKeyDesc.java 
> 9a266a0c57 
>   ql/src/test/queries/clientpositive/subquery_in.q 96ed1bae41 
>   ql/src/test/queries/clientpositive/subquery_notin.q f25168ab77 
>   ql/src/test/queries/clientpositive/topnkey_windowing.q a5352d2d6c 
>   ql/src/test/queries/clientpositive/vector_windowing_streaming.q 2f7b628db3 
>   ql/src/test/queries/clientpositive/windowing_filter.q 14d0c5a7c8 
>   ql/src/test/results/clientpositive/llap/subquery_in.q.out ea8fe5ea96 
>   ql/src/test/results/clientpositive/llap/subquery_notin.q.out c24b79db86 
>   ql/src/test/results/clientpositive/llap/topnkey_windowing.q.out 52ba490c01 
>   ql/src/test/results/clientpositive/llap/vector_windowing_streaming.q.out 
> b63bcf47f3 
>   ql/src/test/results/clientpositive/llap/windowing_filter.q.out 8ef2261755 
>   ql/src/test/results/clientpositive/topnkey_windowing.q.out c186790bea 
> 
> 
> Diff: https://reviews.apache.org/r/72108/diff/1/
> 
> 
> Testing
> ---
> 
> mvn test -Dtest.output.overwrite -DskipSparkTests 
> -Dtest=TestMiniLlapLocalCliDriver 
> -Dqfile=vector_windowing_streaming.q,subquery_notin.q,subquery_in.q,windowing_filter.q,topnkey_windowing.q
>  -pl itests/qtest -Pitests
> 
> 
> Thanks,
> 
> Krisztian Kasa
> 
>

Review Request 72112: HIVE-22869 - Add locking benchmark to metastore-tools/metastore-benchmarks

[Zoltan Chovan via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/
---

Review request for hive, Denys Kuzmenko and Aron Hamvas.


Repository: hive-git


Description
---

Add the possibility to run benchmarks on opening lock in the HMS. Currently 
this change only introduces single-threaded/single client testing. I'm planning 
to add multi-client support in a separate change.

Example parametrisation is as follows:
hbench -M "lock" -N 10 -d hive_test -W 0 -L 100

This will create N number (10) of tables to lock and it'll execute the lock() 
for L number (100) of times.


Diffs
-

  
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java
 041cd76234 
  
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
 f53f2ef43b 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
 7cc1e42a8b 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
 101d6759c5 


Diff: https://reviews.apache.org/r/72112/diff/1/


Testing
---


Thanks,

Zoltan Chovan

Re: Review Request 72063: HIVE-10362: Support Type check/conversion in dynamic partition column

[Zoltan Haindrich]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72063/#review219540
---



I looked at the diff between 1:4


ql/src/test/results/clientpositive/dynpart_sort_optimization_acid2.q.out
Line 98 (original)


it seems we are missing the stats update task here - is this expected

note: these compute_stats() things started appearing in the 1st patch 
uploaded to the reviewboard



ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark
Line 454 (original), 454 (patched)


with the latest patch(#12)
we got back this null; and also the disagreement between llap/spark...

```
grep val_169 `find . -name auto_sortmerge_join_16.q.out*`|cut -d: -f1|uniq 
-c
 24 
./ql/src/test/results/clientpositive/llap/auto_sortmerge_join_16.q.out
 17 
./ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark
 17 
./ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out
```


- Zoltan Haindrich


On Feb. 7, 2020, 9:40 a.m., Karen Coppage wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72063/
> ---
> 
> (Updated Feb. 7, 2020, 9:40 a.m.)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Bugs: HIVE-10362
> https://issues.apache.org/jira/browse/HIVE-10362
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Example:
> create table dynparttypechecknum (key int, value string) partitioned by (part 
> int);
> insert into dynparttypechecknum partition (part) select key, value, '1' 
> from src limit 1;
> show partitions dynparttypechecknum;
> 
> Partition created will be named:
> part=1
> even though the type of `part` is int.
> 
> Solution is to cast the inserted DP columns in the SelectOperator before 
> FileSinkOperator which creates the partition dir, not after.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 12a022c590 
>   itests/src/test/resources/testconfiguration.properties 99ca9867b1 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> c2514eedb1 
>   ql/src/test/queries/clientpositive/dynpart_cast.q PRE-CREATION 
>   ql/src/test/results/clientpositive/autoColumnStats_6.q.out da3be3e5bb 
>   ql/src/test/results/clientpositive/dynpart_cast.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization_acid2.q.out 
> 43bb789840 
>   ql/src/test/results/clientpositive/infer_bucket_sort_num_buckets.q.out 
> f745b46899 
>   ql/src/test/results/clientpositive/llap/dynpart_cast.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/dynpart_sort_opt_bucketing.q.out 
> 453d2451df 
>   ql/src/test/results/clientpositive/llap/orc_merge1.q.out 9da73e65ac 
>   ql/src/test/results/clientpositive/llap/orc_merge10.q.out a6ea33493f 
>   ql/src/test/results/clientpositive/llap/orc_merge2.q.out 9b0d3b4234 
>   ql/src/test/results/clientpositive/llap/orc_merge_diff_fs.q.out d35f44b10a 
>   ql/src/test/results/clientpositive/llap/rcfile_merge2.q.out fcff20a68e 
>   ql/src/test/results/clientpositive/llap/tez_dml.q.out 4ad78d8582 
>   ql/src/test/results/clientpositive/orc_merge1.q.out 9c07816340 
>   ql/src/test/results/clientpositive/orc_merge10.q.out 4a5f03c82f 
>   ql/src/test/results/clientpositive/orc_merge2.q.out d132d62b18 
>   ql/src/test/results/clientpositive/orc_merge_diff_fs.q.out 7f9a04b09f 
>   ql/src/test/results/clientpositive/smb_join_partition_key.q.out c18d01d26a 
>   
> ql/src/test/results/clientpositive/spark/infer_bucket_sort_num_buckets.q.out 
> 56d5ed945b 
>   ql/src/test/results/clientpositive/spark/orc_merge1.q.out 977c4cbfc1 
>   ql/src/test/results/clientpositive/spark/orc_merge2.q.out 4647b86ea3 
>   ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out b7d3dd725d 
> 
> 
> Diff: https://reviews.apache.org/r/72063/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Karen Coppage
> 
>

Re: HIVE-21508 and Hive 2.3.7 question

[Mass Dosage]

+1.

At Expedia Group  we are big users of Hive and are also experiencing issues
with not being able to use Hive 2.3.x on Java >8 which is starting to
seriously impact some of our applications which require Java 11. We worked
on HIVE-21508 in order to get it merged into the various branches and have
been asking for a Hive 2.3.7 release for months with no replies to our
questions on this mailing list.

Could someone from the Hive community please answer and let us know if
there is the possibility of a Hive 2.3.7 release? I've seen at least two
other requests for this on the list over the past few months.

If not we will be forced to fork the current 2.3 branch and release our own
version of Hive 2.3.7 to Maven Central (with a different group id) so that
we can use it (it sounds like this would be useful to others out there
too). We'd really rather not do this but I don't see any other solutions.

Thanks,

Adrian
-- 
Adrian Woodhead
Principal Engineer
Expedia Group - 407 St John Street, London, EC1V 4EX


On Thu, 30 Jan 2020 at 07:34, Hyukjin Kwon  wrote:

> Hi Hive dev team,
>
> As informed earlier, I, Yuming and many people from spark dev have made
> huge efforts
> to let Spark use official Hive release. Thanks Alan and all Hive dev for
> all the efforts for Hive 2.3.6 to make Spark support JDK 11.
>
> Few months ago, an unexpected problem was found. Spark throws
> ClassCastException when
> initializing HiveMetaStoreClient.
> Please see SPARK-29245 
> for
> more details. This has fixed by HIVE-21508
> .
> We postponed the Hive release request to Spark code freeze schedule to
> avoid multiple requests.
>
> Spark is going to freeze code 31st January (tomorrow), and I currently
> foresee the RC starts around March. So, this will be hopefully the last
> request for Hive release for Spark 3.0.
>
> I was wondering if we could release Hive 2.3.7 soon so Spark can uses it.
>
> Thanks.
>

[jira] [Created] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

[Zoltan Chovan (Jira)]

Zoltan Chovan created HIVE-22869:


 Summary: Add locking benchmark to 
metastore-tools/metastore-benchmarks
 Key: HIVE-22869
 URL: https://issues.apache.org/jira/browse/HIVE-22869
 Project: Hive
  Issue Type: Improvement
Reporter: Zoltan Chovan
Assignee: Zoltan Chovan


Add the possibility to run benchmarks on opening lock in the HMS



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22868) Extract ValidTxnManager

[Miklos Gergely (Jira)]

Miklos Gergely created HIVE-22868:
-

 Summary: Extract ValidTxnManager
 Key: HIVE-22868
 URL: https://issues.apache.org/jira/browse/HIVE-22868
 Project: Hive
  Issue Type: Sub-task
Reporter: Miklos Gergely
Assignee: Miklos Gergely


~250 lines in driver are dedicated for the specific task of validating the 
transaction id list, and writing the valid write ids to the configuration. 
These should be put into a separate class.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Review Request 72109: HIVE-20948: Eliminate file rename in compactor

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72109/
---

Review request for hive, Karen Coppage, Marta Kuczora, and Peter Vary.


Repository: hive-git


Description
---

HIVE-20948: Eliminate file rename in compactor


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 
9ad4e7148226b91b0c759de54e251893d61725a3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SplitGrouper.java 
076b77877ae748b757a4c9c08532a3ce029fed38 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 
2f5ec5270c0bc7d1a591c9c8c15b1ecb7f9f6ace 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java 
ecc7bdee4dacc03cf59ac5be4bed92a75f8e720b 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
bb70db452402dd690e2136a122e9b3bd11fa7522 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MajorQueryCompactor.java 
f238eb5dd058fc79c5b7ad3b08920c774b1a7f8c 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MinorQueryCompactor.java 
f96a0481b870b04cc97621cd62a43b07ecd5d7fd 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactor.java 
d2349104902c2af3d6020c9599fd3fa20f9a64a5 


Diff: https://reviews.apache.org/r/72109/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Re: Is there any way to find Hive query to Datanucleus queries mapping

[Chinna Rao Lalam]

Thanks Zoltan for the prompt reply,

I have checked the code with your insights, Yes with this call we can get
the information like below. Using this data  we can add a log for each
HIVESql overall how much time spent in metadata operations.

metadata.Hive: Time spent in each metastore function (ms):
{getTableColumnStatistics_(String, String, List, String, )=18,
getNotNullConstraints_(NotNullConstraintsRequest, )=3, getTable_(String,
String, boolean, String, )=41, commitTxn_(long, )=53, getValidTxns_(long,
)=5, isCompatibleWith_(Configuration, )=1, openTxn_(String, TxnType, )=17,
flushCache_()=0, getUniqueConstraints_(UniqueConstraintsRequest, )=2,
getPrimaryKeys_(PrimaryKeysRequest, )=2,
getForeignKeys_(ForeignKeysRequest, )=3}

Thanks,
Chinna





On Mon, Feb 10, 2020 at 5:49 PM Zoltan Haindrich  wrote:

> Hey Chinna!
>
> I don't think a mapping like that is easy to get...I would rather try to
> narrow down to a single call which consumes most of the time.
> There is a log message which can help you get to the most relevant
> metastore call:
>
> https://github.com/apache/hive/blob/0d9deba3c15038df4c64ea9b8494d554eb8eea2f/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L5405
>
> cheers,
> Zoltan
>
> On 2/10/20 1:07 PM, Chinna Rao Lalam wrote:
> > Hi All,
> >
> > Is there any way to find Hive query to Datanucleus queries mapping.
> >
> > "select * from table" this hive query will generate multiple Datanucleus
> > queries and execute on configured DB.
> > In our DB some of the queries are running slow, So we want to see
> > hivequery->datanucleus query mapping to find out which hive query of
> > datanucleus query is running slow.
> >
> > If we enable Datanucleus debug log we can see generated queries but not
> > mapping.
> >
> > Thanks
> > Chinna
> >
>


-- 
Hope It Helps,
Chinna

Review Request 72108: HIVE-22867

[Krisztian Kasa]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72108/
---

Review request for hive, Attila Magyar and Jesús Camacho Rodríguez.


Bugs: HIVE-22867
https://issues.apache.org/jira/browse/HIVE-22867


Repository: hive-git


Description
---

Add partitioning support to VectorTopNKeyOperator


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java bd8ff6285e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorTopNKeyOperator.java 
f03d65030d 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
27ff0c2484 
  ql/src/java/org/apache/hadoop/hive/ql/plan/VectorTopNKeyDesc.java 9a266a0c57 
  ql/src/test/queries/clientpositive/subquery_in.q 96ed1bae41 
  ql/src/test/queries/clientpositive/subquery_notin.q f25168ab77 
  ql/src/test/queries/clientpositive/topnkey_windowing.q a5352d2d6c 
  ql/src/test/queries/clientpositive/vector_windowing_streaming.q 2f7b628db3 
  ql/src/test/queries/clientpositive/windowing_filter.q 14d0c5a7c8 
  ql/src/test/results/clientpositive/llap/subquery_in.q.out ea8fe5ea96 
  ql/src/test/results/clientpositive/llap/subquery_notin.q.out c24b79db86 
  ql/src/test/results/clientpositive/llap/topnkey_windowing.q.out 52ba490c01 
  ql/src/test/results/clientpositive/llap/vector_windowing_streaming.q.out 
b63bcf47f3 
  ql/src/test/results/clientpositive/llap/windowing_filter.q.out 8ef2261755 
  ql/src/test/results/clientpositive/topnkey_windowing.q.out c186790bea 


Diff: https://reviews.apache.org/r/72108/diff/1/


Testing
---

mvn test -Dtest.output.overwrite -DskipSparkTests 
-Dtest=TestMiniLlapLocalCliDriver 
-Dqfile=vector_windowing_streaming.q,subquery_notin.q,subquery_in.q,windowing_filter.q,topnkey_windowing.q
 -pl itests/qtest -Pitests


Thanks,

Krisztian Kasa

[jira] [Created] (HIVE-22867) Add partitioning support to VectorTopNKeyOperator

[Krisztian Kasa (Jira)]

Krisztian Kasa created HIVE-22867:
-

 Summary: Add partitioning support to VectorTopNKeyOperator 
 Key: HIVE-22867
 URL: https://issues.apache.org/jira/browse/HIVE-22867
 Project: Hive
  Issue Type: Improvement
  Components: Physical Optimizer
Reporter: Krisztian Kasa






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22866) Add more testcases for scheduled queries

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HIVE-22866:
---

 Summary: Add more testcases for scheduled queries
 Key: HIVE-22866
 URL: https://issues.apache.org/jira/browse/HIVE-22866
 Project: Hive
  Issue Type: Improvement
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


the examples in the wiki should be added as test cases



--
This message was sent by Atlassian Jira
(v8.3.4#803005)