[jira] [Created] (HIVE-22588) Flush the remaining rows for the rest of the grouping sets when switching the vector groupby mode

[Ramesh Kumar Thangarajan (Jira)]

Ramesh Kumar Thangarajan created HIVE-22588:
---

 Summary: Flush the remaining rows for the rest of the grouping 
sets when switching the vector groupby mode
 Key: HIVE-22588
 URL: https://issues.apache.org/jira/browse/HIVE-22588
 Project: Hive
  Issue Type: Bug
Reporter: Ramesh Kumar Thangarajan


Flush the remaining rows for the rest of the grouping sets when switching the 
vector groupby mode



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22587) hive.stats.ndv.error parameter documentation issue in HiveConf.java

[Pablo Junge (Jira)]

Pablo Junge created HIVE-22587:
--

 Summary: hive.stats.ndv.error parameter documentation issue in 
HiveConf.java
 Key: HIVE-22587
 URL: https://issues.apache.org/jira/browse/HIVE-22587
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Reporter: Pablo Junge
Assignee: Pablo Junge


hive.stats.ndv.error parameter documentation should specify that it only 
affects the FM-Sketch algorithm



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 71871: StringIndexOutOfBoundsException when getting sessionId from worker node name

[Panos Garefalakis via Review Board]

> On Dec. 5, 2019, 4:43 p.m., Panos Garefalakis wrote:
> > llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
> > Lines 478 (patched)
> > 
> >
> > Hey Attila,
> > 
> > With Java's short circuiting the the left expression in the && 
> > operarator will always be evaluated which could also throw the error you 
> > are trying to avoid -- to safeguard this operation you would place the 
> > **nodeName.length() > workerNodePrefix.length()** check on the left part of 
> > the expression.
> 
> Attila Magyar wrote:
> Hey Panos,
> 
> That's true, but the error is not originated from the startsWith() but 
> from a subString() expression later on. The startsWith() method doesn't throw 
> any expressions it won't fail regardless the length of nodeName or 
> workerNodePrefix.
> 
> Attila Magyar wrote:
> However there is an off by one error in the substring, this comment made 
> me realize it, fixing it in the 2nd patch..

+1 for the substring


- Panos


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71871/#review218941
---


On Dec. 5, 2019, 6:03 p.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71871/
> ---
> 
> (Updated Dec. 5, 2019, 6:03 p.m.)
> 
> 
> Review request for hive, Laszlo Bodor, prasanthj, and Slim Bouguerra.
> 
> 
> Bugs: HIVE-22577
> https://issues.apache.org/jira/browse/HIVE-22577
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The sequence number from the worker node name might be missing under some 
> circumstances (the root cause is not fully clear it might be a zookeeper bug).
> 
> In this case the following exception occurs:
> 
> Caused by: java.lang.StringIndexOutOfBoundsException: String index out of 
> range: -1Caused by: java.lang.StringIndexOutOfBoundsException: String index 
> out of range: -1 at java.lang.String.substring(String.java:1931) at 
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase.extractSeqNum(ZkRegistryBase.java:781)
>  at 
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase.populateCache(ZkRegistryBase.java:507)
>  at 
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.access$000(LlapZookeeperRegistryImpl.java:65)
>  at
> 
> 
> Diffs
> -
> 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 5751b8ed939 
> 
> 
> Diff: https://reviews.apache.org/r/71871/diff/2/
> 
> 
> Testing
> ---
> 
> qtest
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Re: Review Request 71871: StringIndexOutOfBoundsException when getting sessionId from worker node name

[Attila Magyar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71871/
---

(Updated Dec. 5, 2019, 6:03 p.m.)


Review request for hive, Laszlo Bodor, prasanthj, and Slim Bouguerra.


Changes
---

there is a 2nd bug in the original code an off by one error when using the 
substring.


Bugs: HIVE-22577
https://issues.apache.org/jira/browse/HIVE-22577


Repository: hive-git


Description
---

The sequence number from the worker node name might be missing under some 
circumstances (the root cause is not fully clear it might be a zookeeper bug).

In this case the following exception occurs:

Caused by: java.lang.StringIndexOutOfBoundsException: String index out of 
range: -1Caused by: java.lang.StringIndexOutOfBoundsException: String index out 
of range: -1 at java.lang.String.substring(String.java:1931) at 
org.apache.hadoop.hive.registry.impl.ZkRegistryBase.extractSeqNum(ZkRegistryBase.java:781)
 at 
org.apache.hadoop.hive.registry.impl.ZkRegistryBase.populateCache(ZkRegistryBase.java:507)
 at 
org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.access$000(LlapZookeeperRegistryImpl.java:65)
 at


Diffs (updated)
-

  llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
5751b8ed939 


Diff: https://reviews.apache.org/r/71871/diff/2/

Changes: https://reviews.apache.org/r/71871/diff/1-2/


Testing
---

qtest


Thanks,

Attila Magyar

Re: Review Request 71871: StringIndexOutOfBoundsException when getting sessionId from worker node name

[Attila Magyar]

> On Dec. 5, 2019, 4:43 p.m., Panos Garefalakis wrote:
> > llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
> > Lines 478 (patched)
> > 
> >
> > Hey Attila,
> > 
> > With Java's short circuiting the the left expression in the && 
> > operarator will always be evaluated which could also throw the error you 
> > are trying to avoid -- to safeguard this operation you would place the 
> > **nodeName.length() > workerNodePrefix.length()** check on the left part of 
> > the expression.

Hey Panos,

That's true, but the error is not originated from the startsWith() but from a 
subString() expression later on. The startsWith() method doesn't throw any 
expressions it won't fail regardless the length of nodeName or workerNodePrefix.


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71871/#review218941
---


On Dec. 4, 2019, 11:05 a.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71871/
> ---
> 
> (Updated Dec. 4, 2019, 11:05 a.m.)
> 
> 
> Review request for hive, Laszlo Bodor, prasanthj, and Slim Bouguerra.
> 
> 
> Bugs: HIVE-22577
> https://issues.apache.org/jira/browse/HIVE-22577
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The sequence number from the worker node name might be missing under some 
> circumstances (the root cause is not fully clear it might be a zookeeper bug).
> 
> In this case the following exception occurs:
> 
> Caused by: java.lang.StringIndexOutOfBoundsException: String index out of 
> range: -1Caused by: java.lang.StringIndexOutOfBoundsException: String index 
> out of range: -1 at java.lang.String.substring(String.java:1931) at 
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase.extractSeqNum(ZkRegistryBase.java:781)
>  at 
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase.populateCache(ZkRegistryBase.java:507)
>  at 
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.access$000(LlapZookeeperRegistryImpl.java:65)
>  at
> 
> 
> Diffs
> -
> 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 5751b8ed939 
> 
> 
> Diff: https://reviews.apache.org/r/71871/diff/1/
> 
> 
> Testing
> ---
> 
> qtest
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Re: Review Request 71871: StringIndexOutOfBoundsException when getting sessionId from worker node name

[Panos Garefalakis via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71871/#review218941
---




llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Lines 478 (patched)


Hey Attila,

With Java's short circuiting the the left expression in the && operarator 
will always be evaluated which could also throw the error you are trying to 
avoid -- to safeguard this operation you would place the **nodeName.length() > 
workerNodePrefix.length()** check on the left part of the expression.


- Panos Garefalakis


On Dec. 4, 2019, 11:05 a.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71871/
> ---
> 
> (Updated Dec. 4, 2019, 11:05 a.m.)
> 
> 
> Review request for hive, Laszlo Bodor, prasanthj, and Slim Bouguerra.
> 
> 
> Bugs: HIVE-22577
> https://issues.apache.org/jira/browse/HIVE-22577
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The sequence number from the worker node name might be missing under some 
> circumstances (the root cause is not fully clear it might be a zookeeper bug).
> 
> In this case the following exception occurs:
> 
> Caused by: java.lang.StringIndexOutOfBoundsException: String index out of 
> range: -1Caused by: java.lang.StringIndexOutOfBoundsException: String index 
> out of range: -1 at java.lang.String.substring(String.java:1931) at 
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase.extractSeqNum(ZkRegistryBase.java:781)
>  at 
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase.populateCache(ZkRegistryBase.java:507)
>  at 
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.access$000(LlapZookeeperRegistryImpl.java:65)
>  at
> 
> 
> Diffs
> -
> 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 5751b8ed939 
> 
> 
> Diff: https://reviews.apache.org/r/71871/diff/1/
> 
> 
> Testing
> ---
> 
> qtest
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

[jira] [Created] (HIVE-22586) Temporary table delete fails with "SemanticException No valid privileges"

[Laszlo Pinter (Jira)]

Laszlo Pinter created HIVE-22586:


 Summary: Temporary table delete fails with "SemanticException No 
valid privileges"
 Key: HIVE-22586
 URL: https://issues.apache.org/jira/browse/HIVE-22586
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Reporter: Laszlo Pinter


When using Hive with Sentry, at the moment is not possible to delete a 
temporary tables without granting ALL privileges on database/table level.

Temporary tables are strongly bound to the session, and it is only 
visible/accessible to the user who owns the session, and the table exists until 
the session is not closed. 

Since temporary tables only exists on session level, and they are not connected 
to any database, checking the privileges with Sentry is unnecessary. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22585) Clean up catalog/db/table name usage

[David Lavati (Jira)]

David Lavati created HIVE-22585:
---

 Summary: Clean up catalog/db/table name usage
 Key: HIVE-22585
 URL: https://issues.apache.org/jira/browse/HIVE-22585
 Project: Hive
  Issue Type: Sub-task
Reporter: David Lavati
Assignee: David Lavati


This is a followup to HIVE-21198 to address some additional improvement ideas 
for the TableName object mentioned in [https://github.com/apache/hive/pull/550] 
and attempt to remove all the fishy usages of db/tablenames, as a number of 
places still rely on certain state changes/black magic.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22584) Flakyness in TestTaskExecutorService.testSetCapacity

[Peter Vary (Jira)]

Peter Vary created HIVE-22584:
-

 Summary: Flakyness in TestTaskExecutorService.testSetCapacity
 Key: HIVE-22584
 URL: https://issues.apache.org/jira/browse/HIVE-22584
 Project: Hive
  Issue Type: Test
Reporter: Peter Vary
Assignee: Peter Vary


Very rarely the test fails:
{code}
java.lang.AssertionError: expected:<0> but was:<1>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at 
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testSetCapacity(TestTaskExecutorService.java:515)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}

See: 
https://builds.apache.org/job/PreCommit-HIVE-Build/19739/testReport/org.apache.hadoop.hive.llap.daemon.impl/TestTaskExecutorService/testSetCapacity/



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22583) LLAP cache always misses with non-vectorized serde readers such as OpenCSV

[Jira]

Ádám Szita created HIVE-22583:
-

 Summary: LLAP cache always misses with non-vectorized serde 
readers such as OpenCSV
 Key: HIVE-22583
 URL: https://issues.apache.org/jira/browse/HIVE-22583
 Project: Hive
  Issue Type: Bug
  Components: llap
Reporter: Ádám Szita
Assignee: Ádám Szita


Although after the first read LLAP cache stores data of tables that are not 
using the LazySimple serde, the stored data is then never used in the future 
subsequent queries, causing a full cache miss and re-read each time.

Problem is rooted in SerdeEncodedDataReader#cacheFileData is not taking care of 
creating an entry for the root/struct column of the table. The only cases this 
is taken care of are when a vectorized reader is used _(e.g. LazySimpleSerde's 
LazySimpleDeserializeRead)_, where SerdeEncodedDataReader#processAsyncCacheData 
takes care of this.

This can be reproduced by either using a custom serde, like OpenCSV or using 
LazySimpleSerde, but turning off _hive.llap.io.encode.vector.serde.enabled_.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22582) Avoid reading table as ACID when table name is starting with "delta" , but table is not transactional and BI Split Strategy is used

[Aditya Shah (Jira)]

Aditya Shah created HIVE-22582:
--

 Summary: Avoid reading table as ACID when table name is starting 
with "delta" , but table is not transactional and BI Split Strategy is used
 Key: HIVE-22582
 URL: https://issues.apache.org/jira/browse/HIVE-22582
 Project: Hive
  Issue Type: Bug
Reporter: Aditya Shah


The issue is fixed in HIVE-22473 but missed a check for BI Split Strategy.

Steps to reproduce: 
{code:java}
set hive.exec.orc.split.strategy=BI;
create table delta_result (a int) stored as orc 
tblproperties('transactional'='false');
insert into delta_result select 1;
select * from delta_result;
{code}
Exception Stack Trace:
{code:java}
Caused by: java.lang.RuntimeException: ORC split generation failed with 
exception: String index out of range: -1
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1929)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2016)
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.generateWrappedSplits(FetchOperator.java:461)
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:430)
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:336)
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:576)
... 50 more
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of 
range: -1
at java.lang.String.substring(String.java:1967)
at 
org.apache.hadoop.hive.ql.io.AcidUtils.parsedDelta(AcidUtils.java:1128)
at 
org.apache.hadoop.hive.ql.io.AcidUtils$ParsedDeltaLight.parse(AcidUtils.java:921)
at 
org.apache.hadoop.hive.ql.io.AcidUtils.getLogicalLength(AcidUtils.java:2084)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$BISplitStrategy.getSplits(OrcInputFormat.java:1115)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1905)
... 55 more
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)