[jira] [Commented] (HIVE-15221) Improvement for MapJoin checkMemoryStatus, adding gc before throwing Exception

[Fei Hui (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931075#comment-15931075
 ] 

Fei Hui commented on HIVE-15221:


Writing a sample code, prove that gc happens when we call system.gc.
JDK is openjdk version "1.8.0_91". 
The result is in stat_gc.png, uploaded 
{code:title=TestSystemGC.java|borderStyle=solid}
public class TestSystemGC {
  public static void main(String[] args) throws Exception {
int gcCount = 0;
while(gcCount <= 1) {
  System.gc();
  gcCount += 1;
  System.out.println("gcCount:" + gcCount);
  Thread.sleep(1);
}
  }
}
{code}

> Improvement for MapJoin checkMemoryStatus, adding gc before throwing Exception
> --
>
> Key: HIVE-15221
> URL: https://issues.apache.org/jira/browse/HIVE-15221
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Fei Hui
>Assignee: Fei Hui
> Attachments: HIVE-15221.1.patch, stat_gc.png
>
>
> i see in the current master version
>  percentage = (double) usedMemory / (double) maxHeapSize;
> if  percentage > maxMemoryUsage, then throw MapJoinMemoryExhaustionException
> in my opinion, running is better than fail. after System.gc, ' if percentage 
> > maxMemoryUsage, then throw MapJoinMemoryExhaustionException' maybe better
> And original checking way has a problem: 1) consuming much memory cause gc 
> (e.g young gc), then check after adding row and pass. 2) consuming much 
> memory does not cause gc, then check after adding rows but throw Exception
> sometimes 2) occurs, but it contians less rows than 1).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15221) Improvement for MapJoin checkMemoryStatus, adding gc before throwing Exception

[Fei Hui (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fei Hui updated HIVE-15221:
---
Attachment: stat_gc.png

> Improvement for MapJoin checkMemoryStatus, adding gc before throwing Exception
> --
>
> Key: HIVE-15221
> URL: https://issues.apache.org/jira/browse/HIVE-15221
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Fei Hui
>Assignee: Fei Hui
> Attachments: HIVE-15221.1.patch, stat_gc.png
>
>
> i see in the current master version
>  percentage = (double) usedMemory / (double) maxHeapSize;
> if  percentage > maxMemoryUsage, then throw MapJoinMemoryExhaustionException
> in my opinion, running is better than fail. after System.gc, ' if percentage 
> > maxMemoryUsage, then throw MapJoinMemoryExhaustionException' maybe better
> And original checking way has a problem: 1) consuming much memory cause gc 
> (e.g young gc), then check after adding row and pass. 2) consuming much 
> memory does not cause gc, then check after adding rows but throw Exception
> sometimes 2) occurs, but it contians less rows than 1).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16166) HS2 may still waste up to 15% of memory on duplicate strings

[Misha Dmitriev (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931072#comment-15931072
 ] 

Misha Dmitriev commented on HIVE-16166:
---

[~spena] thank you very much for the logs. I found that the failures occur in 
my code with the stack trace like this:

{code}
java.lang.UnsupportedOperationException
at java.util.AbstractList.set(AbstractList.java:132)
at java.util.AbstractList$ListItr.set(AbstractList.java:426)
at 
org.apache.hadoop.hive.common.StringInternUtils.internStringsInList(StringInternUtils.java:112)
at 
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:320)
at 
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:312)
...
{code}

The piece of StringUtils.java where this is thrown looks like this:

{code}
ListIterator it = list.listIterator();
while (it.hasNext()) {
  it.set(it.next().intern());
}
{code}

This is the standard, official way which one can use to replace elements in any 
List implemented in JDK, e.g. ArrayList or LinkedList. For both of them, 
listIterator() returns an Iterator that correctly implements the set() 
operation. So if the code throws an exception, my guess is that it received 
some List (probably not from JDK) that doesn't provide the proper Iterator 
implementation.

Now, I think there are two alternatives for dealing with this problem.  The 
first is to try to find the problematic List implementation (if it's in Hive) 
and fix it. This is complicated, given that the stack trace doesn't show the 
problematic List subclass upfront, and I for some reason cannot reproduce this 
problem locally. But in any case, even if this is fixed, it doesn't guarantee 
that in the future somebody will not write another incomplete List 
implementation that will cause this problem again. So probably a better 
solution is to just catch the UnsupportedOperationException in my code and 
return as if nothing happened. After all, string interning is a performance 
optimization, it doesn't affect the application semantics, so if it doesn't 
always work as expected, it's not a serious problem. What do you think?

> HS2 may still waste up to 15% of memory on duplicate strings
> 
>
> Key: HIVE-16166
> URL: https://issues.apache.org/jira/browse/HIVE-16166
> Project: Hive
>  Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
> Attachments: ch_2_excerpt.txt, HIVE-16166.01.patch
>
>
> A heap dump obtained from one of our users shows that 15% of memory is wasted 
> on duplicate strings, despite the recent optimizations that I made. The 
> problematic strings just come from different sources this time. See the 
> excerpt from the jxray (www.jxray.com) analysis attached.
> Adding String.intern() calls in the appropriate places reduces the overhead 
> of duplicate strings with this workload to ~6%. The remaining duplicates come 
> mostly from JDK internal and MapReduce data structures, and thus are more 
> difficult to fix.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-14879) integrate MM tables into ACID: replace MM metastore calls and structures with ACID ones

[Wei Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zheng updated HIVE-14879:
-
Attachment: HIVE-14879.2.patch

patch 2 based off latest commit:
{code}
commit ccea0d6ff7aaeaac494d7c7c680a3efad7805e3d
Merge: 8e6719d 2a8d1bf
Author: Sergey Shelukhin 
Date:   Wed Mar 15 16:45:48 2017 -0700

HIVE-14671 : merge master into hive-14535 (Sergey Shelukhin)

Conflicts:
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
{code}

> integrate MM tables into ACID: replace MM metastore calls and structures with 
> ACID ones
> ---
>
> Key: HIVE-14879
> URL: https://issues.apache.org/jira/browse/HIVE-14879
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Wei Zheng
> Attachments: HIVE-14879.1.patch, HIVE-14879.2.patch, HIVE-14879.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15857) Vectorization: Add string conversion case for UDFToInteger, etc

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-15857:

Attachment: HIVE-15857.06.patch

> Vectorization: Add string conversion case for UDFToInteger, etc
> ---
>
> Key: HIVE-15857
> URL: https://issues.apache.org/jira/browse/HIVE-15857
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-15857.01.patch, HIVE-15857.02.patch, 
> HIVE-15857.03.patch, HIVE-15857.04.patch, HIVE-15857.05.patch, 
> HIVE-15857.06.patch
>
>
> Otherwise, VectorUDFAdaptor is used to convert a column from String to Int, 
> etc.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15857) Vectorization: Add string conversion case for UDFToInteger, etc

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-15857:

Status: Patch Available  (was: In Progress)

> Vectorization: Add string conversion case for UDFToInteger, etc
> ---
>
> Key: HIVE-15857
> URL: https://issues.apache.org/jira/browse/HIVE-15857
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-15857.01.patch, HIVE-15857.02.patch, 
> HIVE-15857.03.patch, HIVE-15857.04.patch, HIVE-15857.05.patch, 
> HIVE-15857.06.patch
>
>
> Otherwise, VectorUDFAdaptor is used to convert a column from String to Int, 
> etc.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15857) Vectorization: Add string conversion case for UDFToInteger, etc

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-15857:

Status: In Progress  (was: Patch Available)

> Vectorization: Add string conversion case for UDFToInteger, etc
> ---
>
> Key: HIVE-15857
> URL: https://issues.apache.org/jira/browse/HIVE-15857
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-15857.01.patch, HIVE-15857.02.patch, 
> HIVE-15857.03.patch, HIVE-15857.04.patch, HIVE-15857.05.patch
>
>
> Otherwise, VectorUDFAdaptor is used to convert a column from String to Int, 
> etc.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16180) LLAP: Native memory leak in EncodedReader

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-16180:
---

Assignee: Prasanth Jayachandran  (was: Sergey Shelukhin)

> LLAP: Native memory leak in EncodedReader
> -
>
> Key: HIVE-16180
> URL: https://issues.apache.org/jira/browse/HIVE-16180
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: DirectCleaner.java, FullGC-15GB-cleanup.png, 
> Full-gc-native-mem-cleanup.png, HIVE-16180.03.patch, HIVE-16180.1.patch, 
> HIVE-16180.2.patch, Native-mem-spike.png
>
>
> Observed this in internal test run. There is a native memory leak in Orc 
> EncodedReaderImpl that can cause YARN pmem monitor to kill the container 
> running the daemon. Direct byte buffers are null'ed out which is not 
> guaranteed to be cleaned until next Full GC. To show this issue, attaching a 
> small test program that allocates 3x256MB direct byte buffers. First buffer 
> is null'ed out but still native memory is used. Second buffer user Cleaner to 
> clean up native allocation. Third buffer is also null'ed but this time 
> invoking a System.gc() which cleans up all native memory. Output from the 
> test program is below
> {code}
> Allocating 3x256MB direct memory..
> Native memory used: 786432000
> Native memory used after data1=null: 786432000
> Native memory used after data2.clean(): 524288000
> Native memory used after data3=null: 524288000
> Native memory used without gc: 524288000
> Native memory used after gc: 0
> {code}
> Longer term improvements/solutions:
> 1) Use DirectBufferPool from hadoop or netty's 
> https://netty.io/4.0/api/io/netty/buffer/PooledByteBufAllocator.html as 
> direct byte buffer allocations are expensive (System.gc() + 100ms thread 
> sleep).
> 2) Use HADOOP-12760 for proper cleaner invocation in JDK8 and JDK9



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16180) LLAP: Native memory leak in EncodedReader

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-16180:
---

Assignee: Sergey Shelukhin  (was: Prasanth Jayachandran)

> LLAP: Native memory leak in EncodedReader
> -
>
> Key: HIVE-16180
> URL: https://issues.apache.org/jira/browse/HIVE-16180
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: DirectCleaner.java, FullGC-15GB-cleanup.png, 
> Full-gc-native-mem-cleanup.png, HIVE-16180.03.patch, HIVE-16180.1.patch, 
> HIVE-16180.2.patch, Native-mem-spike.png
>
>
> Observed this in internal test run. There is a native memory leak in Orc 
> EncodedReaderImpl that can cause YARN pmem monitor to kill the container 
> running the daemon. Direct byte buffers are null'ed out which is not 
> guaranteed to be cleaned until next Full GC. To show this issue, attaching a 
> small test program that allocates 3x256MB direct byte buffers. First buffer 
> is null'ed out but still native memory is used. Second buffer user Cleaner to 
> clean up native allocation. Third buffer is also null'ed but this time 
> invoking a System.gc() which cleans up all native memory. Output from the 
> test program is below
> {code}
> Allocating 3x256MB direct memory..
> Native memory used: 786432000
> Native memory used after data1=null: 786432000
> Native memory used after data2.clean(): 524288000
> Native memory used after data3=null: 524288000
> Native memory used without gc: 524288000
> Native memory used after gc: 0
> {code}
> Longer term improvements/solutions:
> 1) Use DirectBufferPool from hadoop or netty's 
> https://netty.io/4.0/api/io/netty/buffer/PooledByteBufAllocator.html as 
> direct byte buffer allocations are expensive (System.gc() + 100ms thread 
> sleep).
> 2) Use HADOOP-12760 for proper cleaner invocation in JDK8 and JDK9



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16180) LLAP: Native memory leak in EncodedReader

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-16180:

Attachment: HIVE-16180.03.patch

this patch goes thru (and improves upon) the existing release-to-zcr mechanism. 
Also useZeroCopy flag is not needed, dataReader tracks that. Need to take care 
of one more spot (marked with TODO#)

> LLAP: Native memory leak in EncodedReader
> -
>
> Key: HIVE-16180
> URL: https://issues.apache.org/jira/browse/HIVE-16180
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: DirectCleaner.java, FullGC-15GB-cleanup.png, 
> Full-gc-native-mem-cleanup.png, HIVE-16180.03.patch, HIVE-16180.1.patch, 
> HIVE-16180.2.patch, Native-mem-spike.png
>
>
> Observed this in internal test run. There is a native memory leak in Orc 
> EncodedReaderImpl that can cause YARN pmem monitor to kill the container 
> running the daemon. Direct byte buffers are null'ed out which is not 
> guaranteed to be cleaned until next Full GC. To show this issue, attaching a 
> small test program that allocates 3x256MB direct byte buffers. First buffer 
> is null'ed out but still native memory is used. Second buffer user Cleaner to 
> clean up native allocation. Third buffer is also null'ed but this time 
> invoking a System.gc() which cleans up all native memory. Output from the 
> test program is below
> {code}
> Allocating 3x256MB direct memory..
> Native memory used: 786432000
> Native memory used after data1=null: 786432000
> Native memory used after data2.clean(): 524288000
> Native memory used after data3=null: 524288000
> Native memory used without gc: 524288000
> Native memory used after gc: 0
> {code}
> Longer term improvements/solutions:
> 1) Use DirectBufferPool from hadoop or netty's 
> https://netty.io/4.0/api/io/netty/buffer/PooledByteBufAllocator.html as 
> direct byte buffer allocations are expensive (System.gc() + 100ms thread 
> sleep).
> 2) Use HADOOP-12760 for proper cleaner invocation in JDK8 and JDK9



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15007) Hive 1.2.2 release planning

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930959#comment-15930959
 ] 

Vaibhav Gumashta commented on HIVE-15007:
-

Also scheduled a QA run on branch-1.1 to see if any of these tests are a 
regression.

> Hive 1.2.2 release planning
> ---
>
> Key: HIVE-15007
> URL: https://issues.apache.org/jira/browse/HIVE-15007
> Project: Hive
>  Issue Type: Task
>Affects Versions: 1.2.1
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-15007-branch-1.1.patch, 
> HIVE-15007-branch-1.2.patch, HIVE-15007-branch-1.2.patch, 
> HIVE-15007-branch-1.2.patch, HIVE-15007-branch-1.2.patch, 
> HIVE-15007-branch-1.2.patch, HIVE-15007-branch-1.2.patch, 
> HIVE-15007.branch-1.2.patch
>
>
> Discussed with [~spena] about triggering unit test runs for 1.2.2 release and 
> creating a patch which will trigger precommits looks like a good way.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15007) Hive 1.2.2 release planning

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-15007:

Attachment: HIVE-15007-branch-1.1.patch

> Hive 1.2.2 release planning
> ---
>
> Key: HIVE-15007
> URL: https://issues.apache.org/jira/browse/HIVE-15007
> Project: Hive
>  Issue Type: Task
>Affects Versions: 1.2.1
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-15007-branch-1.1.patch, 
> HIVE-15007-branch-1.2.patch, HIVE-15007-branch-1.2.patch, 
> HIVE-15007-branch-1.2.patch, HIVE-15007-branch-1.2.patch, 
> HIVE-15007-branch-1.2.patch, HIVE-15007-branch-1.2.patch, 
> HIVE-15007.branch-1.2.patch
>
>
> Discussed with [~spena] about triggering unit test runs for 1.2.2 release and 
> creating a patch which will trigger precommits looks like a good way.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16234) Add support for quarter in trunc udf

[Deepesh Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930931#comment-15930931
 ] 

Deepesh Khandelwal commented on HIVE-16234:
---

Thanks [~ashutoshc] for the review and commit!
[~leftylev] for the doc part, is that something I need to do once 2.2.0 is 
released?

> Add support for quarter in trunc udf
> 
>
> Key: HIVE-16234
> URL: https://issues.apache.org/jira/browse/HIVE-16234
> Project: Hive
>  Issue Type: Improvement
>  Components: UDF
>Reporter: Deepesh Khandelwal
>Assignee: Deepesh Khandelwal
>  Labels: TODOC2.2
> Fix For: 2.2.0
>
> Attachments: HIVE-16234.1.patch
>
>
> Hive has a Date function trunc(string date, string format) that returns date 
> truncated to the unit specified by the format. Supported formats: 
> MONTH/MON/MM, YEAR//YY.
> Goal here is to extend support to QUARTER/Q.
> Example:
> SELECT trunc('2017-03-15', 'Q');
> '2017-01-01'
> SELECT trunc('2017-12-31', 'Q');
> '2017-10-01'



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16250) Update grammar in documentation

[Deborah Hunt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deborah Hunt reassigned HIVE-16250:
---


> Update grammar in documentation
> ---
>
> Key: HIVE-16250
> URL: https://issues.apache.org/jira/browse/HIVE-16250
> Project: Hive
>  Issue Type: Improvement
>Reporter: Deborah Hunt
>Assignee: anybudy
>
> Update text recommended:  Hive is widely applied as a solution to numerous 
> distinct problem types in the domain of big data. Quite clearly it is often 
> used for the ad hoc querying of large datasets.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16245) Vectorization: Does not handle non-column key expressions in MERGEPARTIAL mode

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930897#comment-15930897
 ] 

Jason Dere commented on HIVE-16245:
---

+1 pending test results

> Vectorization: Does not handle non-column key expressions in MERGEPARTIAL mode
> --
>
> Key: HIVE-16245
> URL: https://issues.apache.org/jira/browse/HIVE-16245
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-16245.01.patch
>
>
> When the planner is able to make a column a constant, MERGEPARTIAL mode in 
> VectorGroupByOperator is broken because it doesn't evaluate the key 
> expression.  One result is execution cast exception errors.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16205) Improving type safety in Objectstore

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930893#comment-15930893
 ] 

Vihang Karajgaonkar commented on HIVE-16205:


Thanks for the review [~spena] getTableMeta is called from the 
GetTablesOperations() which is called from the JDBC API 
{{HiveDatabaseMeta.getTables()}} method. Test coverage for this API is already 
present in TestJdbcDriver2 test case.

> Improving type safety in Objectstore
> 
>
> Key: HIVE-16205
> URL: https://issues.apache.org/jira/browse/HIVE-16205
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-16205.01.patch, HIVE-16205.02.patch, 
> HIVE-16205.03.patch
>
>
> Modify the queries in ObjectStore for better type safety



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16230) Enable CBO in presence of hints

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-16230:

Status: Patch Available  (was: Open)

> Enable CBO in presence of hints
> ---
>
> Key: HIVE-16230
> URL: https://issues.apache.org/jira/browse/HIVE-16230
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO, Logical Optimizer
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-16230.1.patch, HIVE-16230.2.patch, 
> HIVE-16230.3.patch, HIVE-16230.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16230) Enable CBO in presence of hints

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-16230:

Attachment: HIVE-16230.3.patch

> Enable CBO in presence of hints
> ---
>
> Key: HIVE-16230
> URL: https://issues.apache.org/jira/browse/HIVE-16230
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO, Logical Optimizer
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-16230.1.patch, HIVE-16230.2.patch, 
> HIVE-16230.3.patch, HIVE-16230.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16230) Enable CBO in presence of hints

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-16230:

Status: Open  (was: Patch Available)

> Enable CBO in presence of hints
> ---
>
> Key: HIVE-16230
> URL: https://issues.apache.org/jira/browse/HIVE-16230
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO, Logical Optimizer
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-16230.1.patch, HIVE-16230.2.patch, 
> HIVE-16230.3.patch, HIVE-16230.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-14879) integrate MM tables into ACID: replace MM metastore calls and structures with ACID ones

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930883#comment-15930883
 ] 

Sergey Shelukhin commented on HIVE-14879:
-

It has a couple of fixes since then that use mmid

> integrate MM tables into ACID: replace MM metastore calls and structures with 
> ACID ones
> ---
>
> Key: HIVE-14879
> URL: https://issues.apache.org/jira/browse/HIVE-14879
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Wei Zheng
> Attachments: HIVE-14879.1.patch, HIVE-14879.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16244) Flaky test : dynamic_semijoin_reduction_3.q

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-16244:

   Resolution: Fixed
Fix Version/s: 2.2.0
   Status: Resolved  (was: Patch Available)

Pushed to master. 

> Flaky test : dynamic_semijoin_reduction_3.q
> ---
>
> Key: HIVE-16244
> URL: https://issues.apache.org/jira/browse/HIVE-16244
> Project: Hive
>  Issue Type: Sub-task
>  Components: Tests
>Affects Versions: 2.2.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 2.2.0
>
> Attachments: HIVE-16244.patch
>
>
> Uses current_date() udf which has non-deterministic value.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16230) Enable CBO in presence of hints

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-16230:

Attachment: HIVE-16230.2.patch

> Enable CBO in presence of hints
> ---
>
> Key: HIVE-16230
> URL: https://issues.apache.org/jira/browse/HIVE-16230
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO, Logical Optimizer
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-16230.1.patch, HIVE-16230.2.patch, HIVE-16230.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16230) Enable CBO in presence of hints

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-16230:

Status: Patch Available  (was: Open)

> Enable CBO in presence of hints
> ---
>
> Key: HIVE-16230
> URL: https://issues.apache.org/jira/browse/HIVE-16230
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO, Logical Optimizer
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-16230.1.patch, HIVE-16230.2.patch, HIVE-16230.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16230) Enable CBO in presence of hints

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-16230:

Status: Open  (was: Patch Available)

> Enable CBO in presence of hints
> ---
>
> Key: HIVE-16230
> URL: https://issues.apache.org/jira/browse/HIVE-16230
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO, Logical Optimizer
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-16230.1.patch, HIVE-16230.2.patch, HIVE-16230.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16249) With column stats, mergejoin.q throws NPE

[Pengcheng Xiong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930870#comment-15930870
 ] 

Pengcheng Xiong commented on HIVE-16249:


test will pass without column stats

> With column stats, mergejoin.q throws NPE
> -
>
> Key: HIVE-16249
> URL: https://issues.apache.org/jira/browse/HIVE-16249
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
>
> stack trace:
> {code}
> 2017-03-17T16:00:26,356 ERROR [3d512d4d-72b5-48fc-92cb-0c72f7c876e5 main] 
> parse.CalcitePlanner: CBO failed, skipping CBO.
> java.lang.NullPointerException
> at 
> org.apache.calcite.rel.metadata.RelMdUtil.estimateFilteredRows(RelMdUtil.java:719)
>  ~[calcite-core-1.10.0.jar:1.10.0]
> at 
> org.apache.calcite.rel.metadata.RelMdRowCount.getRowCount(RelMdRowCount.java:123)
>  ~[calcite-core-1.10.0.jar:1.10.0]
> at GeneratedMetadataHandler_RowCount.getRowCount_$(Unknown Source) 
> ~[?:?]
> at GeneratedMetadataHandler_RowCount.getRowCount(Unknown Source) 
> ~[?:?]
> at GeneratedMetadataHandler_RowCount.getRowCount_$(Unknown Source) 
> ~[?:?]
> at GeneratedMetadataHandler_RowCount.getRowCount(Unknown Source) 
> ~[?:?]
> at 
> org.apache.calcite.rel.metadata.RelMetadataQuery.getRowCount(RelMetadataQuery.java:201)
>  ~[calcite-core-1.10.0.jar:1.10.0]
> at 
> org.apache.calcite.rel.metadata.RelMdRowCount.getRowCount(RelMdRowCount.java:132)
>  ~[calcite-core-1.10.0.jar:1.10.0]
> at GeneratedMetadataHandler_RowCount.getRowCount_$(Unknown Source) 
> ~[?:?]
> at GeneratedMetadataHandler_RowCount.getRowCount(Unknown Source) 
> ~[?:?]
> at GeneratedMetadataHandler_RowCount.getRowCount_$(Unknown Source) 
> ~[?:?]
> at GeneratedMetadataHandler_RowCount.getRowCount(Unknown Source) 
> ~[?:?]
> at 
> org.apache.calcite.rel.metadata.RelMetadataQuery.getRowCount(RelMetadataQuery.java:201)
>  ~[calcite-core-1.10.0.jar:1.10.0]
> at 
> org.apache.calcite.rel.rules.LoptOptimizeJoinRule.swapInputs(LoptOptimizeJoinRule.java:1866)
>  ~[calcite-core-1.10.0.jar:1.10.0]
> at 
> org.apache.calcite.rel.rules.LoptOptimizeJoinRule.createJoinSubtree(LoptOptimizeJoinRule.java:1739)
>  ~[calcite-core-1.10.0.jar:1.10.0]
> at 
> org.apache.calcite.rel.rules.LoptOptimizeJoinRule.addToTop(LoptOptimizeJoinRule.java:1216)
>  ~[calcite-core-1.10.0.jar:1.10.0]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16249) With column stats, mergejoin.q throws NPE

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong reassigned HIVE-16249:
--


> With column stats, mergejoin.q throws NPE
> -
>
> Key: HIVE-16249
> URL: https://issues.apache.org/jira/browse/HIVE-16249
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
>
> stack trace:
> {code}
> 2017-03-17T16:00:26,356 ERROR [3d512d4d-72b5-48fc-92cb-0c72f7c876e5 main] 
> parse.CalcitePlanner: CBO failed, skipping CBO.
> java.lang.NullPointerException
> at 
> org.apache.calcite.rel.metadata.RelMdUtil.estimateFilteredRows(RelMdUtil.java:719)
>  ~[calcite-core-1.10.0.jar:1.10.0]
> at 
> org.apache.calcite.rel.metadata.RelMdRowCount.getRowCount(RelMdRowCount.java:123)
>  ~[calcite-core-1.10.0.jar:1.10.0]
> at GeneratedMetadataHandler_RowCount.getRowCount_$(Unknown Source) 
> ~[?:?]
> at GeneratedMetadataHandler_RowCount.getRowCount(Unknown Source) 
> ~[?:?]
> at GeneratedMetadataHandler_RowCount.getRowCount_$(Unknown Source) 
> ~[?:?]
> at GeneratedMetadataHandler_RowCount.getRowCount(Unknown Source) 
> ~[?:?]
> at 
> org.apache.calcite.rel.metadata.RelMetadataQuery.getRowCount(RelMetadataQuery.java:201)
>  ~[calcite-core-1.10.0.jar:1.10.0]
> at 
> org.apache.calcite.rel.metadata.RelMdRowCount.getRowCount(RelMdRowCount.java:132)
>  ~[calcite-core-1.10.0.jar:1.10.0]
> at GeneratedMetadataHandler_RowCount.getRowCount_$(Unknown Source) 
> ~[?:?]
> at GeneratedMetadataHandler_RowCount.getRowCount(Unknown Source) 
> ~[?:?]
> at GeneratedMetadataHandler_RowCount.getRowCount_$(Unknown Source) 
> ~[?:?]
> at GeneratedMetadataHandler_RowCount.getRowCount(Unknown Source) 
> ~[?:?]
> at 
> org.apache.calcite.rel.metadata.RelMetadataQuery.getRowCount(RelMetadataQuery.java:201)
>  ~[calcite-core-1.10.0.jar:1.10.0]
> at 
> org.apache.calcite.rel.rules.LoptOptimizeJoinRule.swapInputs(LoptOptimizeJoinRule.java:1866)
>  ~[calcite-core-1.10.0.jar:1.10.0]
> at 
> org.apache.calcite.rel.rules.LoptOptimizeJoinRule.createJoinSubtree(LoptOptimizeJoinRule.java:1739)
>  ~[calcite-core-1.10.0.jar:1.10.0]
> at 
> org.apache.calcite.rel.rules.LoptOptimizeJoinRule.addToTop(LoptOptimizeJoinRule.java:1216)
>  ~[calcite-core-1.10.0.jar:1.10.0]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15784) Vectorization: Turn on text vectorization by default

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-15784:

Status: Patch Available  (was: In Progress)

> Vectorization: Turn on text vectorization by default
> 
>
> Key: HIVE-15784
> URL: https://issues.apache.org/jira/browse/HIVE-15784
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-15784.01.patch, HIVE-15784.02.patch, 
> HIVE-15784.03.patch, HIVE-15784.04.patch, HIVE-15784.05.patch
>
>
> *Turn ON text vectorization related variables* 
> hive.vectorized.use.vector.serde.deserialize and 
> hive.vectorized.use.row.serde.deserialize by default.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15784) Vectorization: Turn on text vectorization by default

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-15784:

Attachment: HIVE-15784.05.patch

> Vectorization: Turn on text vectorization by default
> 
>
> Key: HIVE-15784
> URL: https://issues.apache.org/jira/browse/HIVE-15784
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-15784.01.patch, HIVE-15784.02.patch, 
> HIVE-15784.03.patch, HIVE-15784.04.patch, HIVE-15784.05.patch
>
>
> *Turn ON text vectorization related variables* 
> hive.vectorized.use.vector.serde.deserialize and 
> hive.vectorized.use.row.serde.deserialize by default.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15784) Vectorization: Turn on text vectorization by default

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-15784:

Status: In Progress  (was: Patch Available)

Could not find run for #04

> Vectorization: Turn on text vectorization by default
> 
>
> Key: HIVE-15784
> URL: https://issues.apache.org/jira/browse/HIVE-15784
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-15784.01.patch, HIVE-15784.02.patch, 
> HIVE-15784.03.patch, HIVE-15784.04.patch
>
>
> *Turn ON text vectorization related variables* 
> hive.vectorized.use.vector.serde.deserialize and 
> hive.vectorized.use.row.serde.deserialize by default.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-14016) Vectorization: Add support for Grouping Sets

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-14016:

Status: Patch Available  (was: In Progress)

> Vectorization: Add support for Grouping Sets
> 
>
> Key: HIVE-14016
> URL: https://issues.apache.org/jira/browse/HIVE-14016
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-14016.01.patch, HIVE-14016.02.patch, 
> HIVE-14016.03.patch, HIVE-14016.04.patch, HIVE-14016.05.patch, 
> HIVE-14016.06.patch, HIVE-14016.07.patch, HIVE-14016.091.patch, 
> HIVE-14016.092.patch, HIVE-14016.09.patch
>
>
> Rollup and Cube queries are not vectorized today due to the miss of 
> grouping-sets inside vector group by.
> The cube and rollup operators can be shimmed onto the end of the pipeline by 
> converting a single row writer into a multiple row writer.
> The corresponding non-vec loop is as follows
> {code}
>   if (groupingSetsPresent) {
> Object[] newKeysArray = newKeys.getKeyArray();
> Object[] cloneNewKeysArray = new Object[newKeysArray.length];
> for (int keyPos = 0; keyPos < groupingSetsPosition; keyPos++) {
>   cloneNewKeysArray[keyPos] = newKeysArray[keyPos];
> }
> for (int groupingSetPos = 0; groupingSetPos < groupingSets.size(); 
> groupingSetPos++) {
>   for (int keyPos = 0; keyPos < groupingSetsPosition; keyPos++) {
> newKeysArray[keyPos] = null;
>   }
>   FastBitSet bitset = groupingSetsBitSet[groupingSetPos];
>   // Some keys need to be left to null corresponding to that grouping 
> set.
>   for (int keyPos = bitset.nextSetBit(0); keyPos >= 0;
> keyPos = bitset.nextSetBit(keyPos+1)) {
> newKeysArray[keyPos] = cloneNewKeysArray[keyPos];
>   }
>   newKeysArray[groupingSetsPosition] = 
> newKeysGroupingSets[groupingSetPos];
>   processKey(row, rowInspector);
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-14016) Vectorization: Add support for Grouping Sets

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-14016:

Status: In Progress  (was: Patch Available)

Couldn't find run for #091

> Vectorization: Add support for Grouping Sets
> 
>
> Key: HIVE-14016
> URL: https://issues.apache.org/jira/browse/HIVE-14016
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-14016.01.patch, HIVE-14016.02.patch, 
> HIVE-14016.03.patch, HIVE-14016.04.patch, HIVE-14016.05.patch, 
> HIVE-14016.06.patch, HIVE-14016.07.patch, HIVE-14016.091.patch, 
> HIVE-14016.092.patch, HIVE-14016.09.patch
>
>
> Rollup and Cube queries are not vectorized today due to the miss of 
> grouping-sets inside vector group by.
> The cube and rollup operators can be shimmed onto the end of the pipeline by 
> converting a single row writer into a multiple row writer.
> The corresponding non-vec loop is as follows
> {code}
>   if (groupingSetsPresent) {
> Object[] newKeysArray = newKeys.getKeyArray();
> Object[] cloneNewKeysArray = new Object[newKeysArray.length];
> for (int keyPos = 0; keyPos < groupingSetsPosition; keyPos++) {
>   cloneNewKeysArray[keyPos] = newKeysArray[keyPos];
> }
> for (int groupingSetPos = 0; groupingSetPos < groupingSets.size(); 
> groupingSetPos++) {
>   for (int keyPos = 0; keyPos < groupingSetsPosition; keyPos++) {
> newKeysArray[keyPos] = null;
>   }
>   FastBitSet bitset = groupingSetsBitSet[groupingSetPos];
>   // Some keys need to be left to null corresponding to that grouping 
> set.
>   for (int keyPos = bitset.nextSetBit(0); keyPos >= 0;
> keyPos = bitset.nextSetBit(keyPos+1)) {
> newKeysArray[keyPos] = cloneNewKeysArray[keyPos];
>   }
>   newKeysArray[groupingSetsPosition] = 
> newKeysGroupingSets[groupingSetPos];
>   processKey(row, rowInspector);
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-14016) Vectorization: Add support for Grouping Sets

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-14016:

Attachment: HIVE-14016.092.patch

> Vectorization: Add support for Grouping Sets
> 
>
> Key: HIVE-14016
> URL: https://issues.apache.org/jira/browse/HIVE-14016
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-14016.01.patch, HIVE-14016.02.patch, 
> HIVE-14016.03.patch, HIVE-14016.04.patch, HIVE-14016.05.patch, 
> HIVE-14016.06.patch, HIVE-14016.07.patch, HIVE-14016.091.patch, 
> HIVE-14016.092.patch, HIVE-14016.09.patch
>
>
> Rollup and Cube queries are not vectorized today due to the miss of 
> grouping-sets inside vector group by.
> The cube and rollup operators can be shimmed onto the end of the pipeline by 
> converting a single row writer into a multiple row writer.
> The corresponding non-vec loop is as follows
> {code}
>   if (groupingSetsPresent) {
> Object[] newKeysArray = newKeys.getKeyArray();
> Object[] cloneNewKeysArray = new Object[newKeysArray.length];
> for (int keyPos = 0; keyPos < groupingSetsPosition; keyPos++) {
>   cloneNewKeysArray[keyPos] = newKeysArray[keyPos];
> }
> for (int groupingSetPos = 0; groupingSetPos < groupingSets.size(); 
> groupingSetPos++) {
>   for (int keyPos = 0; keyPos < groupingSetsPosition; keyPos++) {
> newKeysArray[keyPos] = null;
>   }
>   FastBitSet bitset = groupingSetsBitSet[groupingSetPos];
>   // Some keys need to be left to null corresponding to that grouping 
> set.
>   for (int keyPos = bitset.nextSetBit(0); keyPos >= 0;
> keyPos = bitset.nextSetBit(keyPos+1)) {
> newKeysArray[keyPos] = cloneNewKeysArray[keyPos];
>   }
>   newKeysArray[groupingSetsPosition] = 
> newKeysGroupingSets[groupingSetPos];
>   processKey(row, rowInspector);
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16248) Branch-1.2: Investigate failure of TestMiniTezCliDriver#bucket_map_join_tez1

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta reassigned HIVE-16248:
---


> Branch-1.2: Investigate failure of TestMiniTezCliDriver#bucket_map_join_tez1
> 
>
> Key: HIVE-16248
> URL: https://issues.apache.org/jira/browse/HIVE-16248
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Blocker
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15134) Branch-1.2: Investigate failure of TestMiniTezCliDriver#vector_auto_smb_mapjoin_14

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-15134:

Summary: Branch-1.2: Investigate failure of 
TestMiniTezCliDriver#vector_auto_smb_mapjoin_14  (was: Branch-1.2: Investigate 
failure of TestMiniTezCliDriver: vector_auto_smb_mapjoin_14, tez_smb_empty, 
bucket_map_join_tez1 )

> Branch-1.2: Investigate failure of 
> TestMiniTezCliDriver#vector_auto_smb_mapjoin_14
> --
>
> Key: HIVE-15134
> URL: https://issues.apache.org/jira/browse/HIVE-15134
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Blocker
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16247) Branch-1.2: Investigate failure of TestMiniTezCliDriver#tez_smb_empty

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta reassigned HIVE-16247:
---


> Branch-1.2: Investigate failure of TestMiniTezCliDriver#tez_smb_empty
> -
>
> Key: HIVE-16247
> URL: https://issues.apache.org/jira/browse/HIVE-16247
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Blocker
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-14879) integrate MM tables into ACID: replace MM metastore calls and structures with ACID ones

[Wei Zheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930854#comment-15930854
 ] 

Wei Zheng commented on HIVE-14879:
--

Sure will do that. Also realized my patch was based off this commit:
{code}
commit 5242f71cb2dfa24368fc8a7b5e9425d28b813f7b
Merge: f1f21e9 45b48d5
Author: Sergey Shelukhin 
Date:   Tue Mar 7 19:28:11 2017 -0800

HIVE-14671 : merge master into hive-14535 (Sergey Shelukhin)

Conflicts:
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java

ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java
{code}
I have to rebase my patch to hive-14535 latest..

> integrate MM tables into ACID: replace MM metastore calls and structures with 
> ACID ones
> ---
>
> Key: HIVE-14879
> URL: https://issues.apache.org/jira/browse/HIVE-14879
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Wei Zheng
> Attachments: HIVE-14879.1.patch, HIVE-14879.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-14879) integrate MM tables into ACID: replace MM metastore calls and structures with ACID ones

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930844#comment-15930844
 ] 

Sergey Shelukhin commented on HIVE-14879:
-

Can you please post an RB? When updating patch, RB supports providing a base 
patch, do the branch patch can be the base patch and this would come on top.
I will review next week.

> integrate MM tables into ACID: replace MM metastore calls and structures with 
> ACID ones
> ---
>
> Key: HIVE-14879
> URL: https://issues.apache.org/jira/browse/HIVE-14879
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Wei Zheng
> Attachments: HIVE-14879.1.patch, HIVE-14879.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16071) HoS RPCServer misuses the timeout in its RPC handshake

[Chaoyu Tang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chaoyu Tang updated HIVE-16071:
---
   Resolution: Fixed
Fix Version/s: 2.2.0
   Status: Resolved  (was: Patch Available)

Committed to 2.2.0. Thanks [~xuefuz], [~lirui] for review.

> HoS RPCServer misuses the timeout in its RPC handshake
> --
>
> Key: HIVE-16071
> URL: https://issues.apache.org/jira/browse/HIVE-16071
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Fix For: 2.2.0
>
> Attachments: HIVE-16071.patch, HIVE-16071.patch, HIVE-16071.patch
>
>
> Based on its property description in HiveConf and the comments in HIVE-12650 
> (https://issues.apache.org/jira/browse/HIVE-12650?focusedCommentId=15128979=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15128979),
>  hive.spark.client.connect.timeout is the timeout when the spark remote 
> driver makes a socket connection (channel) to RPC server. But currently it is 
> also used by the remote driver for RPC client/server handshaking, which is 
> not right. Instead, hive.spark.client.server.connect.timeout should be used 
> and it has already been used by the RPCServer in the handshaking.
> The error like following is usually caused by this issue, since the default 
> hive.spark.client.connect.timeout value (1000ms) used by remote driver for 
> handshaking is a little too short.
> {code}
> 17/02/20 08:46:08 ERROR yarn.ApplicationMaster: User class threw exception: 
> java.util.concurrent.ExecutionException: javax.security.sasl.SaslException: 
> Client closed before SASL negotiation finished.
> java.util.concurrent.ExecutionException: javax.security.sasl.SaslException: 
> Client closed before SASL negotiation finished.
> at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37)
> at 
> org.apache.hive.spark.client.RemoteDriver.(RemoteDriver.java:156)
> at 
> org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:556)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542)
> Caused by: javax.security.sasl.SaslException: Client closed before SASL 
> negotiation finished.
> at 
> org.apache.hive.spark.client.rpc.Rpc$SaslClientHandler.dispose(Rpc.java:453)
> at 
> org.apache.hive.spark.client.rpc.SaslHandler.channelInactive(SaslHandler.java:90)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16071) HoS RPCServer misuses the timeout in its RPC handshake

[Chaoyu Tang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chaoyu Tang updated HIVE-16071:
---
Summary: HoS RPCServer misuses the timeout in its RPC handshake  (was: 
Spark remote driver misuses the timeout in RPC handshake)

> HoS RPCServer misuses the timeout in its RPC handshake
> --
>
> Key: HIVE-16071
> URL: https://issues.apache.org/jira/browse/HIVE-16071
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-16071.patch, HIVE-16071.patch, HIVE-16071.patch
>
>
> Based on its property description in HiveConf and the comments in HIVE-12650 
> (https://issues.apache.org/jira/browse/HIVE-12650?focusedCommentId=15128979=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15128979),
>  hive.spark.client.connect.timeout is the timeout when the spark remote 
> driver makes a socket connection (channel) to RPC server. But currently it is 
> also used by the remote driver for RPC client/server handshaking, which is 
> not right. Instead, hive.spark.client.server.connect.timeout should be used 
> and it has already been used by the RPCServer in the handshaking.
> The error like following is usually caused by this issue, since the default 
> hive.spark.client.connect.timeout value (1000ms) used by remote driver for 
> handshaking is a little too short.
> {code}
> 17/02/20 08:46:08 ERROR yarn.ApplicationMaster: User class threw exception: 
> java.util.concurrent.ExecutionException: javax.security.sasl.SaslException: 
> Client closed before SASL negotiation finished.
> java.util.concurrent.ExecutionException: javax.security.sasl.SaslException: 
> Client closed before SASL negotiation finished.
> at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37)
> at 
> org.apache.hive.spark.client.RemoteDriver.(RemoteDriver.java:156)
> at 
> org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:556)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542)
> Caused by: javax.security.sasl.SaslException: Client closed before SASL 
> negotiation finished.
> at 
> org.apache.hive.spark.client.rpc.Rpc$SaslClientHandler.dispose(Rpc.java:453)
> at 
> org.apache.hive.spark.client.rpc.SaslHandler.channelInactive(SaslHandler.java:90)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-14879) integrate MM tables into ACID: replace MM metastore calls and structures with ACID ones

[Wei Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zheng updated HIVE-14879:
-
Attachment: HIVE-14879.1.patch

Uploading patch 1 for review.

This patch implements the logic for replacing mmWriteId with txnId in Driver, 
and make sure txnId is used everywhere instead of the original mmWriteId. Also 
got rid of ValidWriteIds, and use ValidTxnList for the read path.

Since ACID has limitation regarding INSERT OVERWRITE and CTAS (at least), I 
commented out the corresponding part in mm_all.q test and updated the output.

Right now there's one failing test which is mm_conversions.q. I haven't figured 
out yet.

Once [~sershe] or [~ekoifman] review and agree on this approach, I can go ahead 
and remove the metastore changes, schema changes and corresponding thrift 
changes in next patch. Right now they're not included for simplicity reason for 
the review.

> integrate MM tables into ACID: replace MM metastore calls and structures with 
> ACID ones
> ---
>
> Key: HIVE-14879
> URL: https://issues.apache.org/jira/browse/HIVE-14879
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Wei Zheng
> Attachments: HIVE-14879.1.patch, HIVE-14879.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16071) Spark remote driver misuses the timeout in RPC handshake

[Chaoyu Tang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930841#comment-15930841
 ] 

Chaoyu Tang commented on HIVE-16071:


Two tests failed but none of them are related to this patch (See 
https://builds.apache.org/job/PreCommit-HIVE-Build/4215/testReport/) 

> Spark remote driver misuses the timeout in RPC handshake
> 
>
> Key: HIVE-16071
> URL: https://issues.apache.org/jira/browse/HIVE-16071
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-16071.patch, HIVE-16071.patch, HIVE-16071.patch
>
>
> Based on its property description in HiveConf and the comments in HIVE-12650 
> (https://issues.apache.org/jira/browse/HIVE-12650?focusedCommentId=15128979=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15128979),
>  hive.spark.client.connect.timeout is the timeout when the spark remote 
> driver makes a socket connection (channel) to RPC server. But currently it is 
> also used by the remote driver for RPC client/server handshaking, which is 
> not right. Instead, hive.spark.client.server.connect.timeout should be used 
> and it has already been used by the RPCServer in the handshaking.
> The error like following is usually caused by this issue, since the default 
> hive.spark.client.connect.timeout value (1000ms) used by remote driver for 
> handshaking is a little too short.
> {code}
> 17/02/20 08:46:08 ERROR yarn.ApplicationMaster: User class threw exception: 
> java.util.concurrent.ExecutionException: javax.security.sasl.SaslException: 
> Client closed before SASL negotiation finished.
> java.util.concurrent.ExecutionException: javax.security.sasl.SaslException: 
> Client closed before SASL negotiation finished.
> at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37)
> at 
> org.apache.hive.spark.client.RemoteDriver.(RemoteDriver.java:156)
> at 
> org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:556)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542)
> Caused by: javax.security.sasl.SaslException: Client closed before SASL 
> negotiation finished.
> at 
> org.apache.hive.spark.client.rpc.Rpc$SaslClientHandler.dispose(Rpc.java:453)
> at 
> org.apache.hive.spark.client.rpc.SaslHandler.channelInactive(SaslHandler.java:90)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16176) SchemaTool should exit with non-zero exit code when one or more validator's fail.

[Naveen Gangam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-16176:
-
   Resolution: Fixed
Fix Version/s: 2.2.0
   Status: Resolved  (was: Patch Available)

Fix has been pushed to master. Thank you for the review [~aihuaxu]

> SchemaTool should exit with non-zero exit code when one or more validator's 
> fail.
> -
>
> Key: HIVE-16176
> URL: https://issues.apache.org/jira/browse/HIVE-16176
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 2.2.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-16176.patch, HIVE-16176.patch
>
>
> Currently schematool exits with a code of 0 when one or more schema tool 
> validation fail. Ideally, it should return a non-zero exit code when any of 
> the validators fail.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16245) Vectorization: Does not handle non-column key expressions in MERGEPARTIAL mode

[Matt McCline (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930836#comment-15930836
 ] 

Matt McCline commented on HIVE-16245:
-

For now, disable vectorizing non-column key expressions when MERGEPARTIAL.  Fix 
requires reworking the vector group by key classes to add column projection.

> Vectorization: Does not handle non-column key expressions in MERGEPARTIAL mode
> --
>
> Key: HIVE-16245
> URL: https://issues.apache.org/jira/browse/HIVE-16245
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-16245.01.patch
>
>
> When the planner is able to make a column a constant, MERGEPARTIAL mode in 
> VectorGroupByOperator is broken because it doesn't evaluate the key 
> expression.  One result is execution cast exception errors.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16245) Vectorization: Does not handle non-column key expressions in MERGEPARTIAL mode

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-16245:

Status: Patch Available  (was: Open)

> Vectorization: Does not handle non-column key expressions in MERGEPARTIAL mode
> --
>
> Key: HIVE-16245
> URL: https://issues.apache.org/jira/browse/HIVE-16245
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-16245.01.patch
>
>
> When the planner is able to make a column a constant, MERGEPARTIAL mode in 
> VectorGroupByOperator is broken because it doesn't evaluate the key 
> expression.  One result is execution cast exception errors.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16246) Support auto gather column stats for columns with trailing white spaces

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-16246:
---
Status: Patch Available  (was: Open)

> Support auto gather column stats for columns with trailing white spaces
> ---
>
> Key: HIVE-16246
> URL: https://issues.apache.org/jira/browse/HIVE-16246
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16246.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16246) Support auto gather column stats for columns with trailing white spaces

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-16246:
---
Attachment: HIVE-16246.01.patch

> Support auto gather column stats for columns with trailing white spaces
> ---
>
> Key: HIVE-16246
> URL: https://issues.apache.org/jira/browse/HIVE-16246
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16246.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16205) Improving type safety in Objectstore

[JIRA]

[ 
https://issues.apache.org/jira/browse/HIVE-16205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930818#comment-15930818
 ] 

Sergio Peña commented on HIVE-16205:


The patch looks good [~vihangk1].
+1

I just wonder how getTableMeta is tested. Is {{show tables in DB}} the 
statement that calls it?

> Improving type safety in Objectstore
> 
>
> Key: HIVE-16205
> URL: https://issues.apache.org/jira/browse/HIVE-16205
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-16205.01.patch, HIVE-16205.02.patch, 
> HIVE-16205.03.patch
>
>
> Modify the queries in ObjectStore for better type safety



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15126) Branch-1.2: Fix TestCliDriver.join_merge_multi_expressions.q

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-15126:

Attachment: HIVE-15082.1-branch-1.2.patch

Attaching a patch for branch 1.2

> Branch-1.2: Fix TestCliDriver.join_merge_multi_expressions.q
> 
>
> Key: HIVE-15126
> URL: https://issues.apache.org/jira/browse/HIVE-15126
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test
>Affects Versions: 1.2.1
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Blocker
> Attachments: HIVE-15082.1-branch-1.2.patch, HIVE-15126.1.patch
>
>
> The .out file for join_merge_multi_expressions.q needs to be updated. Current 
> one does not show Select operator in the plan that explain returns.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16206) Make Codahale metrics reporters pluggable

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-16206:
--
Status: Open  (was: Patch Available)

Please address the feedback on RB and then repost the patch. Thanks.

> Make Codahale metrics reporters pluggable
> -
>
> Key: HIVE-16206
> URL: https://issues.apache.org/jira/browse/HIVE-16206
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 2.1.2
>Reporter: Sunitha Beeram
>Assignee: Sunitha Beeram
> Attachments: HIVE-16206.patch
>
>
> Hive metrics code currently allows pluggable metrics handlers - ie, handlers 
> that take care of providing interfaces for metrics collection as well as a 
> reporting; one of the 'handlers' is CodahaleMetrics. Codahale can work with 
> different reporters - currently supported ones are Console, JMX, JSON file 
> and hadoop2 sink. However, adding a new reporter involves changing that 
> class. We would like to make this conf driven just the way MetricsFactory 
> handles configurable Metrics classes.
> Scope of work:
> - Provide a new configuration option, HIVE_CODAHALE_REPORTER_CLASSES that 
> enumerates classes (like HIVE_METRICS_CLASS and unlike HIVE_METRICS_REPORTER).
> - Move JsonFileReporter into its own class.
> - Update CodahaleMetrics.java to read new config option and if the new option 
> is not present, look for the old option and instantiate accordingly) - ie, 
> make the code backward compatible.
> - Update and add new tests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-12905) Issue with mapjoin in tez under certain conditions

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930782#comment-15930782
 ] 

Vaibhav Gumashta commented on HIVE-12905:
-

[~sershe] Removing 1.2.2 from target version as this patch will need few others 
(e.g. HIVE-11262, HIVE-11182 etc) and I'm not sure we should backport all. If 
there is a patch for 1.2 before release, we can always incorporate that; let me 
know if you feel otherwise.

> Issue with mapjoin in tez under certain conditions
> --
>
> Key: HIVE-12905
> URL: https://issues.apache.org/jira/browse/HIVE-12905
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: 1.0.1, 1.2.1, 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HIVE-12905.02.patch, HIVE-12905.1.patch
>
>
> In a specific case where we have an outer join followed by another join on 
> the same key and the non-outer side of the outer join is empty, hive-on-tez 
> produces incorrect results.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-12905) Issue with mapjoin in tez under certain conditions

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-12905:

Target Version/s: 2.0.0, 1.3.0  (was: 1.3.0, 2.0.0, 1.2.2)

> Issue with mapjoin in tez under certain conditions
> --
>
> Key: HIVE-12905
> URL: https://issues.apache.org/jira/browse/HIVE-12905
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: 1.0.1, 1.2.1, 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HIVE-12905.02.patch, HIVE-12905.1.patch
>
>
> In a specific case where we have an outer join followed by another join on 
> the same key and the non-outer side of the outer join is empty, hive-on-tez 
> produces incorrect results.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16206) Make Codahale metrics reporters pluggable

[Carl Steinbach (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930772#comment-15930772
 ] 

Carl Steinbach commented on HIVE-16206:
---

Hi [~sbeeram], I think the pre-commit patch testing job may have skipped this 
ticket (I don't see it listed in the queue 
[here|https://builds.apache.org/job/PreCommit-HIVE-Build/]). Can you please 
attach another copy of the patch following the directions 
[here|https://cwiki.apache.org/confluence/display/Hive/Hive+PreCommit+Patch+Testing]?
 Thanks.

> Make Codahale metrics reporters pluggable
> -
>
> Key: HIVE-16206
> URL: https://issues.apache.org/jira/browse/HIVE-16206
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 2.1.2
>Reporter: Sunitha Beeram
>Assignee: Sunitha Beeram
> Attachments: HIVE-16206.patch
>
>
> Hive metrics code currently allows pluggable metrics handlers - ie, handlers 
> that take care of providing interfaces for metrics collection as well as a 
> reporting; one of the 'handlers' is CodahaleMetrics. Codahale can work with 
> different reporters - currently supported ones are Console, JMX, JSON file 
> and hadoop2 sink. However, adding a new reporter involves changing that 
> class. We would like to make this conf driven just the way MetricsFactory 
> handles configurable Metrics classes.
> Scope of work:
> - Provide a new configuration option, HIVE_CODAHALE_REPORTER_CLASSES that 
> enumerates classes (like HIVE_METRICS_CLASS and unlike HIVE_METRICS_REPORTER).
> - Move JsonFileReporter into its own class.
> - Update CodahaleMetrics.java to read new config option and if the new option 
> is not present, look for the old option and instantiate accordingly) - ie, 
> make the code backward compatible.
> - Update and add new tests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16246) Support auto gather column stats for columns with trailing white spaces

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong reassigned HIVE-16246:
--


> Support auto gather column stats for columns with trailing white spaces
> ---
>
> Key: HIVE-16246
> URL: https://issues.apache.org/jira/browse/HIVE-16246
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16228) Support subqueries in complex expression in SELECT clause

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930753#comment-15930753
 ] 

Ashutosh Chauhan commented on HIVE-16228:
-

Can you also add query 9 from TPCDS suite in TestPerfCliDriver and also 
following variants in tests:
* select max(c1) > ( select count( * )-1 from t1 ) fromt1;
* select c1,  (select count(distinct c2) from  u where u.obid = o.obid 
group by obid) tmp from o right join (select * from  f where d1 > (select 
avg(d1) from f)) t on t.fid = o.fid;
* SELECT DISTINCT t1.c1 , (SELECT COUNT(*) FROM  t2  WHERE t2.c1 = t1.c1 GROUP 
BY t2.c1) , (SELECT COUNT(*) FROM  q1,  t2 WHERE t2.c1 = t1.c1 AND q1.qid = 
t2.qid AND (SELECT SUM(d1) FROM   a1 WHERE a1.qid = q1.qid GROUP BY a1.qid) > 
0) from t1;

> Support subqueries in complex expression in SELECT clause
> -
>
> Key: HIVE-16228
> URL: https://issues.apache.org/jira/browse/HIVE-16228
> Project: Hive
>  Issue Type: Sub-task
>  Components: Logical Optimizer
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-16228.1.patch
>
>
> HIVE-16091 added support for subqueries in SELECT clause but restricted 
> subqueries to top level expressions (more detail is at [LINK | 
> https://cwiki.apache.org/confluence/display/Hive/Subqueries+in+SELECT])
> This restriction will be relaxed to allow subqueries in all kind of 
> expression except UDAF (including UDAs and UDTFs).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16228) Support subqueries in complex expression in SELECT clause

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930732#comment-15930732
 ] 

Ashutosh Chauhan commented on HIVE-16228:
-

Following failures need to be addressed from QA run:

org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_select_aggregate]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_select_udf]


> Support subqueries in complex expression in SELECT clause
> -
>
> Key: HIVE-16228
> URL: https://issues.apache.org/jira/browse/HIVE-16228
> Project: Hive
>  Issue Type: Sub-task
>  Components: Logical Optimizer
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-16228.1.patch
>
>
> HIVE-16091 added support for subqueries in SELECT clause but restricted 
> subqueries to top level expressions (more detail is at [LINK | 
> https://cwiki.apache.org/confluence/display/Hive/Subqueries+in+SELECT])
> This restriction will be relaxed to allow subqueries in all kind of 
> expression except UDAF (including UDAs and UDTFs).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-12943) Use default doesnot work in Hive 1.2.1

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930694#comment-15930694
 ] 

Vaibhav Gumashta commented on HIVE-12943:
-

[~kasjain]  Removing target 1.2 as patch hasn't been committed yet. 

> Use default doesnot work in Hive 1.2.1
> --
>
> Key: HIVE-12943
> URL: https://issues.apache.org/jira/browse/HIVE-12943
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema
>Affects Versions: 1.2.1
>Reporter: Kashish Jain
>Priority: Blocker
> Attachments: HIVE-12943.patch
>
>
> "USE Default" does not work with the latest hive 1.2.1
> The message is 
> "
>Cannot recognize input near 'default' '' '' in switch database 
> statement; line 1 pos 4
>NoViableAltException(81@[])
>at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.identifier(HiveParser_IdentifiersParser.java:11577)
>at 
> org.apache.hadoop.hive.ql.parse.HiveParser.identifier(HiveParser.java:46055)
> "



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (HIVE-12943) Use default doesnot work in Hive 1.2.1

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930694#comment-15930694
 ] 

Vaibhav Gumashta edited comment on HIVE-12943 at 3/17/17 8:45 PM:
--

[~kasjain]  Removing target 1.2 as patch hasn't been committed yet. Adding 
target 1.3


was (Author: vgumashta):
[~kasjain]  Removing target 1.2 as patch hasn't been committed yet. 

> Use default doesnot work in Hive 1.2.1
> --
>
> Key: HIVE-12943
> URL: https://issues.apache.org/jira/browse/HIVE-12943
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema
>Affects Versions: 1.2.1
>Reporter: Kashish Jain
>Priority: Blocker
> Attachments: HIVE-12943.patch
>
>
> "USE Default" does not work with the latest hive 1.2.1
> The message is 
> "
>Cannot recognize input near 'default' '' '' in switch database 
> statement; line 1 pos 4
>NoViableAltException(81@[])
>at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.identifier(HiveParser_IdentifiersParser.java:11577)
>at 
> org.apache.hadoop.hive.ql.parse.HiveParser.identifier(HiveParser.java:46055)
> "



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-12943) Use default doesnot work in Hive 1.2.1

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-12943:

Target Version/s: 1.3.0  (was: 1.2.2)

> Use default doesnot work in Hive 1.2.1
> --
>
> Key: HIVE-12943
> URL: https://issues.apache.org/jira/browse/HIVE-12943
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema
>Affects Versions: 1.2.1
>Reporter: Kashish Jain
>Priority: Blocker
> Attachments: HIVE-12943.patch
>
>
> "USE Default" does not work with the latest hive 1.2.1
> The message is 
> "
>Cannot recognize input near 'default' '' '' in switch database 
> statement; line 1 pos 4
>NoViableAltException(81@[])
>at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.identifier(HiveParser_IdentifiersParser.java:11577)
>at 
> org.apache.hadoop.hive.ql.parse.HiveParser.identifier(HiveParser.java:46055)
> "



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-12469) Bump Commons-Collections dependency from 3.2.1 to 3.2.2. to address vulnerability

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930691#comment-15930691
 ] 

Vaibhav Gumashta commented on HIVE-12469:
-

Committed to 1.2

> Bump Commons-Collections dependency from 3.2.1 to 3.2.2. to address 
> vulnerability
> -
>
> Key: HIVE-12469
> URL: https://issues.apache.org/jira/browse/HIVE-12469
> Project: Hive
>  Issue Type: Bug
>  Components: Build Infrastructure
>Affects Versions: 1.2.1
>Reporter: Reuben Kuhnert
>Assignee: Ashutosh Chauhan
>Priority: Blocker
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-12469.2-branch1.patch, HIVE-12469.2.patch, 
> HIVE-12469.patch
>
>
> Currently the commons-collections (3.2.1) library allows for invocation of 
> arbitrary code through {{InvokerTransformer}}, need to bump the version of 
> commons-collections from 3.2.1 to 3.2.2 to resolve this issue.
> Results of {{mvn dependency:tree}}:
> {code}
> [INFO] 
> 
> [INFO] Building Hive HPL/SQL 2.0.0-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ hive-hplsql ---
> [INFO] org.apache.hive:hive-hplsql:jar:2.0.0-SNAPSHOT
> [INFO] +- com.google.guava:guava:jar:14.0.1:compile
> [INFO] +- commons-collections:commons-collections:jar:3.2.1:compile
> {code}
> {code}
> [INFO] 
> 
> [INFO] Building Hive Packaging 2.0.0-SNAPSHOT
> [INFO] 
> 
> [INFO] +- org.apache.hive:hive-hbase-handler:jar:2.0.0-SNAPSHOT:compile
> [INFO] |  +- org.apache.hbase:hbase-server:jar:1.1.1:compile
> [INFO] |  |  +- commons-collections:commons-collections:jar:3.2.1:compile
> {code}
> {code}
> [INFO] 
> 
> [INFO] Building Hive Common 2.0.0-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ hive-common ---
> [INFO] +- org.apache.hadoop:hadoop-common:jar:2.6.0:compile
> [INFO] |  +- commons-collections:commons-collections:jar:3.2.1:compile
> {code}
> {{Hadoop-Common}} dependency also found in: LLAP, Serde, Storage,  Shims, 
> Shims Common, Shims Scheduler)
> {code}
> [INFO] 
> 
> [INFO] Building Hive Ant Utilities 2.0.0-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ hive-ant ---
> [INFO] |  +- commons-collections:commons-collections:jar:3.1:compile
> {code}
> {code}
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Hive Accumulo Handler 2.0.0-SNAPSHOT
> [INFO] 
> 
> [INFO] +- org.apache.accumulo:accumulo-core:jar:1.6.0:compile
> [INFO] |  +- commons-collections:commons-collections:jar:3.2.1:compile
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13370) Add test for HIVE-11470

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930685#comment-15930685
 ] 

Vaibhav Gumashta commented on HIVE-13370:
-

[~sushanth] Removing target 1.2.2. Feel free to commit to branch 1.2 if you 
need this.

> Add test for HIVE-11470
> ---
>
> Key: HIVE-13370
> URL: https://issues.apache.org/jira/browse/HIVE-13370
> Project: Hive
>  Issue Type: Bug
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
>Priority: Blocker
> Attachments: HIVE-13370.patch
>
>
> HIVE-11470 added capability to handle NULL dynamic partitioning keys 
> properly. However, it did not add a test for the case, we should have one so 
> we don't have future regressions of the same.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-13370) Add test for HIVE-11470

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-13370:

Target Version/s: 1.3.0, 2.2.0  (was: 1.3.0, 1.2.2, 2.2.0)

> Add test for HIVE-11470
> ---
>
> Key: HIVE-13370
> URL: https://issues.apache.org/jira/browse/HIVE-13370
> Project: Hive
>  Issue Type: Bug
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
>Priority: Blocker
> Attachments: HIVE-13370.patch
>
>
> HIVE-11470 added capability to handle NULL dynamic partitioning keys 
> properly. However, it did not add a test for the case, we should have one so 
> we don't have future regressions of the same.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15082) Hive-1.2 cannot read data from complex data types with TIMESTAMP column, stored in Parquet

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930682#comment-15930682
 ] 

Vaibhav Gumashta commented on HIVE-15082:
-

[~osayankin] Sorry for the delay on this. I've started working again on release 
planning for 1.2.2 and have submitted the patch again for QA run.

> Hive-1.2 cannot read data from complex data types with TIMESTAMP column, 
> stored in Parquet
> --
>
> Key: HIVE-15082
> URL: https://issues.apache.org/jira/browse/HIVE-15082
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
> Attachments: HIVE-15082.1-branch-1.2.patch, 
> HIVE-15082.1-branch-1.2.patch, HIVE-15082-branch-1.2.patch, 
> HIVE-15082-branch-1.patch
>
>
> *STEP 1. Create test data*
> {code:sql}
> select * from dual;
> {code}
> *EXPECTED RESULT:*
> {noformat}
> Pretty_UnIQUe_StrinG
> {noformat}
> {code:sql}
> create table test_parquet1(login timestamp) stored as parquet;
> insert overwrite table test_parquet1 select from_unixtime(unix_timestamp()) 
> from dual;
> select * from test_parquet1 limit 1;
> {code}
> *EXPECTED RESULT:*
> No exceptions. Current timestamp as result.
> {noformat}
> 2016-10-27 10:58:19
> {noformat}
> *STEP 2. Store timestamp in array in parquet file*
> {code:sql}
> create table test_parquet2(x array) stored as parquet;
> insert overwrite table test_parquet2 select array(login) from test_parquet1;
> select * from test_parquet2;
> {code}
> *EXPECTED RESULT:*
> No exceptions. Current timestamp in brackets as result.
> {noformat}
> ["2016-10-27 10:58:19"]
> {noformat}
> *ACTUAL RESULT:*
> {noformat}
> ERROR [main]: CliDriver (SessionState.java:printError(963)) - Failed with 
> exception java.io.IOException:parquet.io.ParquetDecodingException: Can not 
> read value at 0 in block -1 in file 
> hdfs:///user/hive/warehouse/test_parquet2/00_0
> java.io.IOException: parquet.io.ParquetDecodingException: Can not read value 
> at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0
> {noformat}
> *ROOT-CAUSE:*
> Incorrect initialization of {{metadata}} {{HashMap}} causes that it has 
> {{null}} value in enumeration 
> {{org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter}} when 
> executing following line:
> {code:java}
>   boolean skipConversion = 
> Boolean.valueOf(metadata.get(HiveConf.ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION.varname));
> {code}
> in element {{ETIMESTAMP_CONVERTER}}.
> JVM throws NPE and parquet library can not read data from file and throws 
> {noformat}
> java.io.IOException:parquet.io.ParquetDecodingException: Can not read value 
> at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0
> {noformat}
> for its turn.
> *SOLUTION:*
> Perform initialization in separate method to skip overriding it with {{null}} 
> value in block of code
> {code:java}
>   if (parent != null) {
>  setMetadata(parent.getMetadata());
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-15082) Hive-1.2 cannot read data from complex data types with TIMESTAMP column, stored in Parquet

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta reassigned HIVE-15082:
---

Assignee: Vaibhav Gumashta  (was: Oleksiy Sayankin)

> Hive-1.2 cannot read data from complex data types with TIMESTAMP column, 
> stored in Parquet
> --
>
> Key: HIVE-15082
> URL: https://issues.apache.org/jira/browse/HIVE-15082
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Oleksiy Sayankin
>Assignee: Vaibhav Gumashta
>Priority: Blocker
> Attachments: HIVE-15082.1-branch-1.2.patch, 
> HIVE-15082.1-branch-1.2.patch, HIVE-15082-branch-1.2.patch, 
> HIVE-15082-branch-1.patch
>
>
> *STEP 1. Create test data*
> {code:sql}
> select * from dual;
> {code}
> *EXPECTED RESULT:*
> {noformat}
> Pretty_UnIQUe_StrinG
> {noformat}
> {code:sql}
> create table test_parquet1(login timestamp) stored as parquet;
> insert overwrite table test_parquet1 select from_unixtime(unix_timestamp()) 
> from dual;
> select * from test_parquet1 limit 1;
> {code}
> *EXPECTED RESULT:*
> No exceptions. Current timestamp as result.
> {noformat}
> 2016-10-27 10:58:19
> {noformat}
> *STEP 2. Store timestamp in array in parquet file*
> {code:sql}
> create table test_parquet2(x array) stored as parquet;
> insert overwrite table test_parquet2 select array(login) from test_parquet1;
> select * from test_parquet2;
> {code}
> *EXPECTED RESULT:*
> No exceptions. Current timestamp in brackets as result.
> {noformat}
> ["2016-10-27 10:58:19"]
> {noformat}
> *ACTUAL RESULT:*
> {noformat}
> ERROR [main]: CliDriver (SessionState.java:printError(963)) - Failed with 
> exception java.io.IOException:parquet.io.ParquetDecodingException: Can not 
> read value at 0 in block -1 in file 
> hdfs:///user/hive/warehouse/test_parquet2/00_0
> java.io.IOException: parquet.io.ParquetDecodingException: Can not read value 
> at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0
> {noformat}
> *ROOT-CAUSE:*
> Incorrect initialization of {{metadata}} {{HashMap}} causes that it has 
> {{null}} value in enumeration 
> {{org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter}} when 
> executing following line:
> {code:java}
>   boolean skipConversion = 
> Boolean.valueOf(metadata.get(HiveConf.ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION.varname));
> {code}
> in element {{ETIMESTAMP_CONVERTER}}.
> JVM throws NPE and parquet library can not read data from file and throws 
> {noformat}
> java.io.IOException:parquet.io.ParquetDecodingException: Can not read value 
> at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0
> {noformat}
> for its turn.
> *SOLUTION:*
> Perform initialization in separate method to skip overriding it with {{null}} 
> value in block of code
> {code:java}
>   if (parent != null) {
>  setMetadata(parent.getMetadata());
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-15082) Hive-1.2 cannot read data from complex data types with TIMESTAMP column, stored in Parquet

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta reassigned HIVE-15082:
---

Assignee: Oleksiy Sayankin  (was: Vaibhav Gumashta)

> Hive-1.2 cannot read data from complex data types with TIMESTAMP column, 
> stored in Parquet
> --
>
> Key: HIVE-15082
> URL: https://issues.apache.org/jira/browse/HIVE-15082
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
> Attachments: HIVE-15082.1-branch-1.2.patch, 
> HIVE-15082.1-branch-1.2.patch, HIVE-15082-branch-1.2.patch, 
> HIVE-15082-branch-1.patch
>
>
> *STEP 1. Create test data*
> {code:sql}
> select * from dual;
> {code}
> *EXPECTED RESULT:*
> {noformat}
> Pretty_UnIQUe_StrinG
> {noformat}
> {code:sql}
> create table test_parquet1(login timestamp) stored as parquet;
> insert overwrite table test_parquet1 select from_unixtime(unix_timestamp()) 
> from dual;
> select * from test_parquet1 limit 1;
> {code}
> *EXPECTED RESULT:*
> No exceptions. Current timestamp as result.
> {noformat}
> 2016-10-27 10:58:19
> {noformat}
> *STEP 2. Store timestamp in array in parquet file*
> {code:sql}
> create table test_parquet2(x array) stored as parquet;
> insert overwrite table test_parquet2 select array(login) from test_parquet1;
> select * from test_parquet2;
> {code}
> *EXPECTED RESULT:*
> No exceptions. Current timestamp in brackets as result.
> {noformat}
> ["2016-10-27 10:58:19"]
> {noformat}
> *ACTUAL RESULT:*
> {noformat}
> ERROR [main]: CliDriver (SessionState.java:printError(963)) - Failed with 
> exception java.io.IOException:parquet.io.ParquetDecodingException: Can not 
> read value at 0 in block -1 in file 
> hdfs:///user/hive/warehouse/test_parquet2/00_0
> java.io.IOException: parquet.io.ParquetDecodingException: Can not read value 
> at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0
> {noformat}
> *ROOT-CAUSE:*
> Incorrect initialization of {{metadata}} {{HashMap}} causes that it has 
> {{null}} value in enumeration 
> {{org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter}} when 
> executing following line:
> {code:java}
>   boolean skipConversion = 
> Boolean.valueOf(metadata.get(HiveConf.ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION.varname));
> {code}
> in element {{ETIMESTAMP_CONVERTER}}.
> JVM throws NPE and parquet library can not read data from file and throws 
> {noformat}
> java.io.IOException:parquet.io.ParquetDecodingException: Can not read value 
> at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0
> {noformat}
> for its turn.
> *SOLUTION:*
> Perform initialization in separate method to skip overriding it with {{null}} 
> value in block of code
> {code:java}
>   if (parent != null) {
>  setMetadata(parent.getMetadata());
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15082) Hive-1.2 cannot read data from complex data types with TIMESTAMP column, stored in Parquet

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-15082:

Attachment: HIVE-15082.1-branch-1.2.patch

Submitting again for QA run

> Hive-1.2 cannot read data from complex data types with TIMESTAMP column, 
> stored in Parquet
> --
>
> Key: HIVE-15082
> URL: https://issues.apache.org/jira/browse/HIVE-15082
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Oleksiy Sayankin
>Assignee: Vaibhav Gumashta
>Priority: Blocker
> Attachments: HIVE-15082.1-branch-1.2.patch, 
> HIVE-15082.1-branch-1.2.patch, HIVE-15082-branch-1.2.patch, 
> HIVE-15082-branch-1.patch
>
>
> *STEP 1. Create test data*
> {code:sql}
> select * from dual;
> {code}
> *EXPECTED RESULT:*
> {noformat}
> Pretty_UnIQUe_StrinG
> {noformat}
> {code:sql}
> create table test_parquet1(login timestamp) stored as parquet;
> insert overwrite table test_parquet1 select from_unixtime(unix_timestamp()) 
> from dual;
> select * from test_parquet1 limit 1;
> {code}
> *EXPECTED RESULT:*
> No exceptions. Current timestamp as result.
> {noformat}
> 2016-10-27 10:58:19
> {noformat}
> *STEP 2. Store timestamp in array in parquet file*
> {code:sql}
> create table test_parquet2(x array) stored as parquet;
> insert overwrite table test_parquet2 select array(login) from test_parquet1;
> select * from test_parquet2;
> {code}
> *EXPECTED RESULT:*
> No exceptions. Current timestamp in brackets as result.
> {noformat}
> ["2016-10-27 10:58:19"]
> {noformat}
> *ACTUAL RESULT:*
> {noformat}
> ERROR [main]: CliDriver (SessionState.java:printError(963)) - Failed with 
> exception java.io.IOException:parquet.io.ParquetDecodingException: Can not 
> read value at 0 in block -1 in file 
> hdfs:///user/hive/warehouse/test_parquet2/00_0
> java.io.IOException: parquet.io.ParquetDecodingException: Can not read value 
> at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0
> {noformat}
> *ROOT-CAUSE:*
> Incorrect initialization of {{metadata}} {{HashMap}} causes that it has 
> {{null}} value in enumeration 
> {{org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter}} when 
> executing following line:
> {code:java}
>   boolean skipConversion = 
> Boolean.valueOf(metadata.get(HiveConf.ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION.varname));
> {code}
> in element {{ETIMESTAMP_CONVERTER}}.
> JVM throws NPE and parquet library can not read data from file and throws 
> {noformat}
> java.io.IOException:parquet.io.ParquetDecodingException: Can not read value 
> at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0
> {noformat}
> for its turn.
> *SOLUTION:*
> Perform initialization in separate method to skip overriding it with {{null}} 
> value in block of code
> {code:java}
>   if (parent != null) {
>  setMetadata(parent.getMetadata());
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-6113) Upgrade DataNucleus [was: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient]

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930671#comment-15930671
 ] 

Vaibhav Gumashta commented on HIVE-6113:


Removing target 1.2.2 as it's a minor upgrade and this change touches lots of 
parts.

> Upgrade DataNucleus [was: Unable to instantiate 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient]
> -
>
> Key: HIVE-6113
> URL: https://issues.apache.org/jira/browse/HIVE-6113
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 1.0.0, 1.2.1
> Environment: hadoop-0.20.2-cdh3u3,hive-0.12.0
>Reporter: William Stone
>Assignee: Oleksiy Sayankin
>Priority: Critical
>  Labels: HiveMetaStoreClient, TODOC2.0, metastore, 
> unable_instantiate
> Fix For: 2.0.0
>
> Attachments: HIVE-6113.10.patch, HIVE-6113-2.patch, 
> HIVE-6113.3.patch, HIVE-6113.4.patch, HIVE-6113.5.patch, HIVE-6113.6.patch, 
> HIVE-6113.7.patch, HIVE-6113.8.patch, HIVE-6113.9.patch, HIVE-6113.patch, 
> HIVE-6113.with.reflection.patch
>
>
> CLEAR LIBRARY CACHE
> When I exccute SQL "use fdm; desc formatted fdm.tableName;"  in python, throw 
> Error as followed.
> but when I tryit again , It will success.
> 2013-12-25 03:01:32,290 ERROR exec.DDLTask (DDLTask.java:execute(435)) - 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
>   at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1143)
>   at 
> org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1128)
>   at 
> org.apache.hadoop.hive.ql.exec.DDLTask.switchDatabase(DDLTask.java:3479)
>   at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:237)
>   at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
>   at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
>   at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414)
>   at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:260)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:217)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:507)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:875)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:769)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:708)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
> Caused by: java.lang.RuntimeException: Unable to instantiate 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
>   at 
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1217)
>   at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:62)
>   at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
>   at 
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372)
>   at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383)
>   at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1139)
>   ... 20 more
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>   at 
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210)
>   ... 25 more
> Caused by: javax.jdo.JDODataStoreException: Exception thrown flushing changes 
> to datastore
> NestedThrowables:
> java.sql.BatchUpdateException: Duplicate entry 'default' for key 
> 'UNIQUE_DATABASE'
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> 

[jira] [Updated] (HIVE-6113) Upgrade DataNucleus [was: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient]

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-6113:
---
Target Version/s: 2.0.0, 1.3.0  (was: 1.3.0, 2.0.0, 1.2.2)

> Upgrade DataNucleus [was: Unable to instantiate 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient]
> -
>
> Key: HIVE-6113
> URL: https://issues.apache.org/jira/browse/HIVE-6113
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 1.0.0, 1.2.1
> Environment: hadoop-0.20.2-cdh3u3,hive-0.12.0
>Reporter: William Stone
>Assignee: Oleksiy Sayankin
>Priority: Critical
>  Labels: HiveMetaStoreClient, TODOC2.0, metastore, 
> unable_instantiate
> Fix For: 2.0.0
>
> Attachments: HIVE-6113.10.patch, HIVE-6113-2.patch, 
> HIVE-6113.3.patch, HIVE-6113.4.patch, HIVE-6113.5.patch, HIVE-6113.6.patch, 
> HIVE-6113.7.patch, HIVE-6113.8.patch, HIVE-6113.9.patch, HIVE-6113.patch, 
> HIVE-6113.with.reflection.patch
>
>
> CLEAR LIBRARY CACHE
> When I exccute SQL "use fdm; desc formatted fdm.tableName;"  in python, throw 
> Error as followed.
> but when I tryit again , It will success.
> 2013-12-25 03:01:32,290 ERROR exec.DDLTask (DDLTask.java:execute(435)) - 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
>   at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1143)
>   at 
> org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1128)
>   at 
> org.apache.hadoop.hive.ql.exec.DDLTask.switchDatabase(DDLTask.java:3479)
>   at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:237)
>   at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
>   at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
>   at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414)
>   at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:260)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:217)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:507)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:875)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:769)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:708)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
> Caused by: java.lang.RuntimeException: Unable to instantiate 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
>   at 
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1217)
>   at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:62)
>   at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
>   at 
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372)
>   at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383)
>   at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1139)
>   ... 20 more
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>   at 
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210)
>   ... 25 more
> Caused by: javax.jdo.JDODataStoreException: Exception thrown flushing changes 
> to datastore
> NestedThrowables:
> java.sql.BatchUpdateException: Duplicate entry 'default' for key 
> 'UNIQUE_DATABASE'
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> 

[jira] [Updated] (HIVE-16186) REPL DUMP shows last event ID of the database even if we use LIMIT option.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-16186:

Status: Patch Available  (was: In Progress)

> REPL DUMP shows last event ID of the database even if we use LIMIT option.
> --
>
> Key: HIVE-16186
> URL: https://issues.apache.org/jira/browse/HIVE-16186
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR
> Attachments: HIVE-16186.01.patch
>
>
> Looks like LIMIT option doesn't work well with REPL DUMP.
> 0: jdbc:hive2://localhost:10001/default> REPL DUMP default FROM 170 LIMIT 1;
> +--+---+
> | dump_dir | last_repl_id  |
> +--+---+
> | /tmp/dump/1489395053411  | 195   |
> +--+---+



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16186) REPL DUMP shows last event ID of the database even if we use LIMIT option.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930665#comment-15930665
 ] 

ASF GitHub Bot commented on HIVE-16186:
---

GitHub user sankarh opened a pull request:

https://github.com/apache/hive/pull/161

HIVE-16186: REPL DUMP shows last event ID of the database even if we use 
LIMIT option.

Returned proper last repl Id if LIMIT option is used.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sankarh/hive HIVE-16186

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/161.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #161


commit de8eb79ef9ec2aac2c31474af04834850947acb6
Author: Sankar Hariappan 
Date:   2017-03-17T20:21:08Z

HIVE-16186: REPL DUMP shows last event ID of the database even if we use 
LIMIT option.




> REPL DUMP shows last event ID of the database even if we use LIMIT option.
> --
>
> Key: HIVE-16186
> URL: https://issues.apache.org/jira/browse/HIVE-16186
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR
> Attachments: HIVE-16186.01.patch
>
>
> Looks like LIMIT option doesn't work well with REPL DUMP.
> 0: jdbc:hive2://localhost:10001/default> REPL DUMP default FROM 170 LIMIT 1;
> +--+---+
> | dump_dir | last_repl_id  |
> +--+---+
> | /tmp/dump/1489395053411  | 195   |
> +--+---+



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16186) REPL DUMP shows last event ID of the database even if we use LIMIT option.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-16186:

Attachment: HIVE-16186.01.patch

Tha last repl ID dumped and displayed in REPL DUMP result are corrected to 
return proper values when use LIMIT option. Also, added an exception if TO 
event ID is less than FROM event ID.
Request [~sushanth], [~thejas], [~anishek] to review the 01.patch!

> REPL DUMP shows last event ID of the database even if we use LIMIT option.
> --
>
> Key: HIVE-16186
> URL: https://issues.apache.org/jira/browse/HIVE-16186
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR
> Attachments: HIVE-16186.01.patch
>
>
> Looks like LIMIT option doesn't work well with REPL DUMP.
> 0: jdbc:hive2://localhost:10001/default> REPL DUMP default FROM 170 LIMIT 1;
> +--+---+
> | dump_dir | last_repl_id  |
> +--+---+
> | /tmp/dump/1489395053411  | 195   |
> +--+---+



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16049) upgrade to jetty 9

[Aihua Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930664#comment-15930664
 ] 

Aihua Xu commented on HIVE-16049:
-

[~thejas] [~busbey] Can you guys help review the patch?

> upgrade to jetty 9
> --
>
> Key: HIVE-16049
> URL: https://issues.apache.org/jira/browse/HIVE-16049
> Project: Hive
>  Issue Type: Improvement
>Reporter: Sean Busbey
>Assignee: Aihua Xu
> Attachments: HIVE-16049.0.patch, HIVE-16049.1.patch, 
> HIVE-16049.2.patch
>
>
> Jetty 7 has been deprecated for a couple of years now. Hadoop and HBase have 
> both updated to Jetty 9 for their next major releases, which will complicate 
> classpath concerns.
> Proactively update to Jetty 9 in the few places we use a web server.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13869) ObjectStore synchronization issue manifested in db notification listener

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930656#comment-15930656
 ] 

Vaibhav Gumashta commented on HIVE-13869:
-

Removing target 1.2.2 (minor upgrade) and adding target 1.3 as there is no 
patch yet.

> ObjectStore synchronization issue manifested in db notification listener
> 
>
> Key: HIVE-13869
> URL: https://issues.apache.org/jira/browse/HIVE-13869
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 1.1.1
>Reporter: Nachiket Vaidya
>Priority: Blocker
>
> I used pyhs2 python client to create tables/partitions in hive. I was working 
> fine until I moved to multithreaded scripts which created 8 connections and 
> ran DDL queries concurrently.
> I got the error as
> {noformat}
> 2016-05-04 17:49:26,226 ERROR 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-4-thread-194]: 
> HMSHandler Fatal error: Invalid state. Transaction has already started
> org.datanucleus.transaction.NucleusTransactionException: Invalid state. 
> Transaction has already started
> at 
> org.datanucleus.transaction.TransactionManager.begin(TransactionManager.java:47)
> at org.datanucleus.TransactionImpl.begin(TransactionImpl.java:131)
> at 
> org.datanucleus.api.jdo.JDOTransaction.internalBegin(JDOTransaction.java:88)
> at 
> org.datanucleus.api.jdo.JDOTransaction.begin(JDOTransaction.java:80)
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.openTransaction(ObjectStore.java:463)
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.addNotificationEvent(ObjectStore.java:7522)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy10.addNotificationEvent(Unknown Source)
> at 
> org.apache.hive.hcatalog.listener.DbNotificationListener.enqueue(DbNotificationListener.java:261)
> at 
> org.apache.hive.hcatalog.listener.DbNotificationListener.onCreateTable(DbNotificationListener.java:123)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1483)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1502)
> at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:138)
> at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at 
> com.sun.proxy.$Proxy14.create_table_with_environment_context(Unknown Source)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:9267)
> {noformat}
> At one point, I got another stack trace as
> {noformat}
> 2016-05-26 12:32:27,904 ERROR 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-4-thread-7]: 
> MetaException(message:java.lang.NullPointerException)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:5535)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.add_partitions_req(HiveMetaStore.java:2308)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:138)
> at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at com.sun.proxy.$Proxy14.add_partitions_req(Unknown Source)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$add_partitions_req.getResult(ThriftHiveMetastore.java:9723)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$add_partitions_req.getResult(ThriftHiveMetastore.java:9707)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at 
> 

[jira] [Updated] (HIVE-13869) ObjectStore synchronization issue manifested in db notification listener

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-13869:

Target Version/s: 1.3.0  (was: 1.2.2)

> ObjectStore synchronization issue manifested in db notification listener
> 
>
> Key: HIVE-13869
> URL: https://issues.apache.org/jira/browse/HIVE-13869
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 1.1.1
>Reporter: Nachiket Vaidya
>Priority: Blocker
>
> I used pyhs2 python client to create tables/partitions in hive. I was working 
> fine until I moved to multithreaded scripts which created 8 connections and 
> ran DDL queries concurrently.
> I got the error as
> {noformat}
> 2016-05-04 17:49:26,226 ERROR 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-4-thread-194]: 
> HMSHandler Fatal error: Invalid state. Transaction has already started
> org.datanucleus.transaction.NucleusTransactionException: Invalid state. 
> Transaction has already started
> at 
> org.datanucleus.transaction.TransactionManager.begin(TransactionManager.java:47)
> at org.datanucleus.TransactionImpl.begin(TransactionImpl.java:131)
> at 
> org.datanucleus.api.jdo.JDOTransaction.internalBegin(JDOTransaction.java:88)
> at 
> org.datanucleus.api.jdo.JDOTransaction.begin(JDOTransaction.java:80)
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.openTransaction(ObjectStore.java:463)
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.addNotificationEvent(ObjectStore.java:7522)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy10.addNotificationEvent(Unknown Source)
> at 
> org.apache.hive.hcatalog.listener.DbNotificationListener.enqueue(DbNotificationListener.java:261)
> at 
> org.apache.hive.hcatalog.listener.DbNotificationListener.onCreateTable(DbNotificationListener.java:123)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1483)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1502)
> at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:138)
> at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at 
> com.sun.proxy.$Proxy14.create_table_with_environment_context(Unknown Source)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:9267)
> {noformat}
> At one point, I got another stack trace as
> {noformat}
> 2016-05-26 12:32:27,904 ERROR 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-4-thread-7]: 
> MetaException(message:java.lang.NullPointerException)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:5535)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.add_partitions_req(HiveMetaStore.java:2308)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:138)
> at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at com.sun.proxy.$Proxy14.add_partitions_req(Unknown Source)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$add_partitions_req.getResult(ThriftHiveMetastore.java:9723)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$add_partitions_req.getResult(ThriftHiveMetastore.java:9707)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at 
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at 
> 

[jira] [Commented] (HIVE-12174) Error: java.lang.RuntimeException: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 112, Size: 11

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930648#comment-15930648
 ] 

Vaibhav Gumashta commented on HIVE-12174:
-

Removing target 1.2.2 (minor upgrade) and adding 1.3.0 since there is no patch 
yet. 

> Error: java.lang.RuntimeException: 
> org.apache.hive.com.esotericsoftware.kryo.KryoException: 
> java.lang.IndexOutOfBoundsException: Index: 112, Size: 11
> -
>
> Key: HIVE-12174
> URL: https://issues.apache.org/jira/browse/HIVE-12174
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Feng Yuan
>Priority: Blocker
>
> kryo serde error:
> insert overwrite table portrait.test1 select iid, feedback_15day, 
> feedback_7day, feedback_5day, feedback_3day, feedback_1day from 
> portrait.rec_feature_feedback a where l_date = '2015-09-09' and bid in 
> ('949722CF_12F7_523A_EE21_E3D591B7E755');
> Query ID = hadoop_20151014180148_6aa9392a-8564-49b4-a538-a7fd1ba18911
> Total jobs = 3
> Launching Job 1 out of 3
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_1442984792179_5081, Tracking URL = 
> http://bjlg-112p13-hadoop03.bfdabc.com:8088/proxy/application_1442984792179_5081/
> Kill Command = /opt/hadoop/hadoop-2.6.0/bin/hadoop job  -kill 
> job_1442984792179_5081
> Hadoop job information for Stage-1: number of mappers: 4; number of reducers: > 0
> 2015-10-14 18:02:16,315 Stage-1 map = 0%,  reduce = 0%
> 2015-10-14 18:02:29,112 Stage-1 map = 50%,  reduce = 0%, Cumulative CPU 18.05 
> sec
> 2015-10-14 18:02:50,403 Stage-1 map = 75%,  reduce = 0%, Cumulative CPU 21.65 
> sec
> 2015-10-14 18:02:55,720 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 
> 25.87 sec
> MapReduce Total cumulative CPU time: 25 seconds 870 msec
> Ended Job = job_1442984792179_5081
> Stage-4 is filtered out by condition resolver.
> Stage-3 is selected by condition resolver.
> Stage-5 is filtered out by condition resolver.
> Launching Job 3 out of 3
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_1442984792179_5082, Tracking URL = 
> http://bjlg-112p13-hadoop03.bfdabc.com:8088/proxy/application_1442984792179_5082/
> Kill Command = /opt/hadoop/hadoop-2.6.0/bin/hadoop job  -kill 
> job_1442984792179_5082
> Hadoop job information for Stage-3: number of mappers: 1; number of reducers: > 0
> 2015-10-14 18:03:20,847 Stage-3 map = 0%,  reduce = 0%
> 2015-10-14 18:03:52,505 Stage-3 map = 100%,  reduce = 0%
> Ended Job = job_1442984792179_5082 with errors
> Error during job, obtaining debugging information...
> Examining task ID: task_1442984792179_5082_m_00 (and more) from job 
> job_1442984792179_5082
> Task with the most failures(4): 
> -
> Task ID:
>   task_1442984792179_5082_m_00
> URL:
>   
> http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1442984792179_5082=task_1442984792179_5082_m_00
> -
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException: 
> org.apache.hive.com.esotericsoftware.kryo.KryoException: 
> java.lang.IndexOutOfBoundsException: Index: 112, Size: 11
> Serialization trace:
> writeType (org.apache.hadoop.hive.ql.plan.FileSinkDesc)
> conf (org.apache.hadoop.hive.ql.exec.FileSinkOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
>   at 
> org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:423)
>   at 
> org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:286)
>   at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:263)
>   at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:478)
>   at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:471)
>   at 
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:648)
>   at 
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:169)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: 
> java.lang.IndexOutOfBoundsException: Index: 112, Size: 11
> 

[jira] [Updated] (HIVE-12174) Error: java.lang.RuntimeException: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 112, Size: 11

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-12174:

Target Version/s: 1.3.0  (was: 1.2.2)

> Error: java.lang.RuntimeException: 
> org.apache.hive.com.esotericsoftware.kryo.KryoException: 
> java.lang.IndexOutOfBoundsException: Index: 112, Size: 11
> -
>
> Key: HIVE-12174
> URL: https://issues.apache.org/jira/browse/HIVE-12174
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Feng Yuan
>Priority: Blocker
>
> kryo serde error:
> insert overwrite table portrait.test1 select iid, feedback_15day, 
> feedback_7day, feedback_5day, feedback_3day, feedback_1day from 
> portrait.rec_feature_feedback a where l_date = '2015-09-09' and bid in 
> ('949722CF_12F7_523A_EE21_E3D591B7E755');
> Query ID = hadoop_20151014180148_6aa9392a-8564-49b4-a538-a7fd1ba18911
> Total jobs = 3
> Launching Job 1 out of 3
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_1442984792179_5081, Tracking URL = 
> http://bjlg-112p13-hadoop03.bfdabc.com:8088/proxy/application_1442984792179_5081/
> Kill Command = /opt/hadoop/hadoop-2.6.0/bin/hadoop job  -kill 
> job_1442984792179_5081
> Hadoop job information for Stage-1: number of mappers: 4; number of reducers: > 0
> 2015-10-14 18:02:16,315 Stage-1 map = 0%,  reduce = 0%
> 2015-10-14 18:02:29,112 Stage-1 map = 50%,  reduce = 0%, Cumulative CPU 18.05 
> sec
> 2015-10-14 18:02:50,403 Stage-1 map = 75%,  reduce = 0%, Cumulative CPU 21.65 
> sec
> 2015-10-14 18:02:55,720 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 
> 25.87 sec
> MapReduce Total cumulative CPU time: 25 seconds 870 msec
> Ended Job = job_1442984792179_5081
> Stage-4 is filtered out by condition resolver.
> Stage-3 is selected by condition resolver.
> Stage-5 is filtered out by condition resolver.
> Launching Job 3 out of 3
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_1442984792179_5082, Tracking URL = 
> http://bjlg-112p13-hadoop03.bfdabc.com:8088/proxy/application_1442984792179_5082/
> Kill Command = /opt/hadoop/hadoop-2.6.0/bin/hadoop job  -kill 
> job_1442984792179_5082
> Hadoop job information for Stage-3: number of mappers: 1; number of reducers: > 0
> 2015-10-14 18:03:20,847 Stage-3 map = 0%,  reduce = 0%
> 2015-10-14 18:03:52,505 Stage-3 map = 100%,  reduce = 0%
> Ended Job = job_1442984792179_5082 with errors
> Error during job, obtaining debugging information...
> Examining task ID: task_1442984792179_5082_m_00 (and more) from job 
> job_1442984792179_5082
> Task with the most failures(4): 
> -
> Task ID:
>   task_1442984792179_5082_m_00
> URL:
>   
> http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1442984792179_5082=task_1442984792179_5082_m_00
> -
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException: 
> org.apache.hive.com.esotericsoftware.kryo.KryoException: 
> java.lang.IndexOutOfBoundsException: Index: 112, Size: 11
> Serialization trace:
> writeType (org.apache.hadoop.hive.ql.plan.FileSinkDesc)
> conf (org.apache.hadoop.hive.ql.exec.FileSinkOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
>   at 
> org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:423)
>   at 
> org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:286)
>   at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:263)
>   at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:478)
>   at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:471)
>   at 
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:648)
>   at 
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:169)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: 
> java.lang.IndexOutOfBoundsException: Index: 112, Size: 11
> Serialization trace:
> writeType (org.apache.hadoop.hive.ql.plan.FileSinkDesc)
> conf 

[jira] [Commented] (HIVE-13282) GroupBy and select operator encounter ArrayIndexOutOfBoundsException

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930647#comment-15930647
 ] 

Vaibhav Gumashta commented on HIVE-13282:
-

[~mmccline] Is it ok to target for 1.3 and remove target 1.2.2 (minor release)?

> GroupBy and select operator encounter ArrayIndexOutOfBoundsException
> 
>
> Key: HIVE-13282
> URL: https://issues.apache.org/jira/browse/HIVE-13282
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 1.2.1, 2.0.0, 2.1.0
>Reporter: Vikram Dixit K
>Assignee: Matt McCline
>Priority: Blocker
> Attachments: HIVE-13282.01.patch, smb_fail_issue.patch, 
> smb_groupby.q, smb_groupby.q.out
>
>
> The group by and select operators run into the ArrayIndexOutOfBoundsException 
> when they incorrectly initialize themselves with tag 0 but the incoming tag 
> id is different.
> {code}
> select count(*) from
> (select rt1.id from
> (select t1.key as id, t1.value as od from tab t1 group by key, value) rt1) vt1
> join
> (select rt2.id from
> (select t2.key as id, t2.value as od from tab_part t2 group by key, value) 
> rt2) vt2
> where vt1.id=vt2.id;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13017) Child process of HiveServer2 fails to get delegation token from non default FileSystem

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930643#comment-15930643
 ] 

Vaibhav Gumashta commented on HIVE-13017:
-

Committed to 1.2

> Child process of HiveServer2 fails to get delegation token from non default 
> FileSystem
> --
>
> Key: HIVE-13017
> URL: https://issues.apache.org/jira/browse/HIVE-13017
> Project: Hive
>  Issue Type: Bug
>  Components: Authentication
>Affects Versions: 1.2.1
> Environment: Secure 
>Reporter: Takahiko Saito
>Assignee: Sushanth Sowmyan
> Fix For: 2.1.0
>
> Attachments: HIVE-13017.2.patch, HIVE-13017.3.patch, HIVE-13017.patch
>
>
> The following query fails, when Azure Filesystem is used as default file 
> system, and HDFS is used for intermediate data.
> {noformat}
> >>>  create temporary table s10k stored as orc as select * from studenttab10k;
> >>>  create temporary table v10k as select * from votertab10k;
> >>>  select registration 
> from s10k s join v10k v 
> on (s.name = v.name) join studentparttab30k p 
> on (p.name = v.name) 
> where s.age < 25 and v.age < 25 and p.age < 25;
> ERROR : Execution failed with exit status: 2
> ERROR : Obtaining error information
> ERROR : 
> Task failed!
> Task ID:
>   Stage-5
> Logs:
> ERROR : /var/log/hive/hiveServer2.log
> Error: Error while processing statement: FAILED: Execution Error, return code 
> 2 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask (state=08S01,code=2)
> Aborting command set because "force" is false and command failed: "select 
> registration 
> from s10k s join v10k v 
> on (s.name = v.name) join studentparttab30k p 
> on (p.name = v.name) 
> where s.age < 25 and v.age < 25 and p.age < 25;"
> Closing: 0: 
> jdbc:hive2://zk2-hs21-h.hdinsight.net:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_h...@hdinsight.net;transportMode=http;httpPath=cliservice
> hiveServer2.log shows:
> 2016-02-02 18:04:34,182 INFO  [HiveServer2-Background-Pool: Thread-517]: 
> log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) -  method=Driver.run from=org.apache.hadoop.hive.ql.Driver>
> 2016-02-02 18:04:34,199 INFO  [HiveServer2-Background-Pool: Thread-517]: 
> log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) -  method=TimeToSubmit from=org.apache.hadoop.hive.ql.Driver>
> 2016-02-02 18:04:34,212 INFO  [HiveServer2-HttpHandler-Pool: Thread-55]: 
> thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(127)) - Could not 
> validate cookie sent, will try to generate a new cookie
> 2016-02-02 18:04:34,213 INFO  [HiveServer2-Background-Pool: Thread-517]: 
> ql.Driver (Driver.java:checkConcurrency(168)) - Concurrency mode is disabled, 
> not creating a lock manager
> 2016-02-02 18:04:34,219 INFO  [HiveServer2-HttpHandler-Pool: Thread-55]: 
> thrift.ThriftHttpServlet (ThriftHttpServlet.java:doKerberosAuth(352)) - 
> Failed to authenticate with http/_HOST kerberos principal, trying with 
> hive/_HOST kerberos principal
> 2016-02-02 18:04:34,219 INFO  [HiveServer2-Background-Pool: Thread-517]: 
> log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) -  method=Driver.execute from=org.apache.hadoop.hive.ql.Driver>
> 2016-02-02 18:04:34,225 INFO  [HiveServer2-Background-Pool: Thread-517]: 
> ql.Driver (Driver.java:execute(1390)) - Setting caller context to query id 
> hive_20160202180429_76ab-64d6-4c89-88b0-6355cc5acbd0
> 2016-02-02 18:04:34,226 INFO  [HiveServer2-Background-Pool: Thread-517]: 
> ql.Driver (Driver.java:execute(1393)) - Starting 
> command(queryId=hive_20160202180429_76ab-64d6-4c89-88b0-6355cc5acbd0): 
> select registration
> from s10k s join v10k v
> on (s.name = v.name) join studentparttab30k p
> on (p.name = v.name)
> where s.age < 25 and v.age < 25 and p.age < 25
> 2016-02-02 18:04:34,228 INFO  [HiveServer2-Background-Pool: Thread-517]: 
> hooks.ATSHook (ATSHook.java:(90)) - Created ATS Hook
> 2016-02-02 18:04:34,229 INFO  [HiveServer2-Background-Pool: Thread-517]: 
> log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) -  method=PreHook.org.apache.hadoop.hive.ql.hooks.ATSHook 
> from=org.apache.hadoop.hive.ql.Driver>
> 2016-02-02 18:04:34,237 INFO  [HiveServer2-HttpHandler-Pool: Thread-55]: 
> thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(169)) - Cookie added 
> for clientUserName hrt_qa
> 2016-02-02 18:04:34,238 INFO  [HiveServer2-Background-Pool: Thread-517]: 
> log.PerfLogger (PerfLogger.java:PerfLogEnd(162)) -  method=PreHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1454436274229 
> end=1454436274238 duration=9 from=org.apache.hadoop.hive.ql.Driver>
> 2016-02-02 18:04:34,239 INFO  [HiveServer2-Background-Pool: Thread-517]: 
> log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) -  

[jira] [Updated] (HIVE-9583) Rolling upgrade of Hive MetaStore Server

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-9583:
---
Fix Version/s: (was: 1.2.2)

> Rolling upgrade of Hive MetaStore Server
> 
>
> Key: HIVE-9583
> URL: https://issues.apache.org/jira/browse/HIVE-9583
> Project: Hive
>  Issue Type: Improvement
>  Components: HCatalog, Metastore
>Affects Versions: 0.14.0
>Reporter: Thiruvel Thirumoolan
>Assignee: Thiruvel Thirumoolan
>  Labels: TODOC1.2, hcatalog, metastore
> Fix For: 1.2.2
>
>
> This is an umbrella JIRA to track all rolling upgrade JIRAs w.r.t MetaStore 
> server. This will be helpful for users deploying Metastore server and 
> connecting to it with HCatalog or Hive CLI interface.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-9583) Rolling upgrade of Hive MetaStore Server

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-9583:
---
Fix Version/s: 1.2.2

> Rolling upgrade of Hive MetaStore Server
> 
>
> Key: HIVE-9583
> URL: https://issues.apache.org/jira/browse/HIVE-9583
> Project: Hive
>  Issue Type: Improvement
>  Components: HCatalog, Metastore
>Affects Versions: 0.14.0
>Reporter: Thiruvel Thirumoolan
>Assignee: Thiruvel Thirumoolan
>  Labels: TODOC1.2, hcatalog, metastore
> Fix For: 1.2.2
>
>
> This is an umbrella JIRA to track all rolling upgrade JIRAs w.r.t MetaStore 
> server. This will be helpful for users deploying Metastore server and 
> connecting to it with HCatalog or Hive CLI interface.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-12088) a simple insert hql throws out NoClassFoundException of MetaException

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930637#comment-15930637
 ] 

Vaibhav Gumashta commented on HIVE-12088:
-

Removing fix version 1.2.2

> a simple insert hql throws out NoClassFoundException of MetaException
> -
>
> Key: HIVE-12088
> URL: https://issues.apache.org/jira/browse/HIVE-12088
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Feng Yuan
> Attachments: hive.log
>
>
> example:
> from portrait.rec_feature_feedback a insert overwrite table portrait.test1 
> select iid, feedback_15day, feedback_7day, feedback_5day, feedback_3day, 
> feedback_1day where l_date = '2015-09-09' and bid in 
> ('949722CF_12F7_523A_EE21_E3D591B7E755');
> log shows:
> Query ID = hadoop_20151012153841_120bee59-56a7-4e53-9c45-76f97c0f50ad
> Total jobs = 3
> Launching Job 1 out of 3
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_1441881651073_95266, Tracking URL = 
> http://bjlg-44p12-rm01:8088/proxy/application_1441881651073_95266/
> Kill Command = /opt/hadoop/hadoop/bin/hadoop job  -kill 
> job_1441881651073_95266
> Hadoop job information for Stage-1: number of mappers: 21; number of 
> reducers: 0
> 2015-10-12 15:39:29,930 Stage-1 map = 0%,  reduce = 0%
> 2015-10-12 15:39:39,597 Stage-1 map = 5%,  reduce = 0%
> 2015-10-12 15:39:40,658 Stage-1 map = 0%,  reduce = 0%
> 2015-10-12 15:39:53,479 Stage-1 map = 5%,  reduce = 0%
> 2015-10-12 15:39:54,535 Stage-1 map = 0%,  reduce = 0%
> 2015-10-12 15:39:55,588 Stage-1 map = 10%,  reduce = 0%
> 2015-10-12 15:39:56,626 Stage-1 map = 5%,  reduce = 0%
> 2015-10-12 15:39:57,687 Stage-1 map = 0%,  reduce = 0%
> 2015-10-12 15:40:06,096 Stage-1 map = 100%,  reduce = 0%
> Ended Job = job_1441881651073_95266 with errors
> Error during job, obtaining debugging information...
> Examining task ID: task_1441881651073_95266_m_00 (and more) from job 
> job_1441881651073_95266
> Examining task ID: task_1441881651073_95266_m_16 (and more) from job 
> job_1441881651073_95266
> Examining task ID: task_1441881651073_95266_m_11 (and more) from job 
> job_1441881651073_95266
> Examining task ID: task_1441881651073_95266_m_18 (and more) from job 
> job_1441881651073_95266
> Examining task ID: task_1441881651073_95266_m_02 (and more) from job 
> job_1441881651073_95266
> Task with the most failures(4): 
> -
> Task ID:
>   task_1441881651073_95266_m_09
> URL:
>   
> http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1441881651073_95266=task_1441881651073_95266_m_09
> -
> Diagnostic Messages for this Task:
> Error: java.lang.ClassNotFoundException: 
> org.apache.hadoop.hive.metastore.api.MetaException
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>   at java.lang.Class.getDeclaredMethods0(Native Method)
>   at java.lang.Class.privateGetDeclaredMethods(Class.java:2570)
>   at java.lang.Class.privateGetPublicMethods(Class.java:2690)
>   at java.lang.Class.getMethods(Class.java:1467)
>   at com.sun.beans.finder.MethodFinder$1.create(MethodFinder.java:54)
>   at com.sun.beans.finder.MethodFinder$1.create(MethodFinder.java:49)
>   at com.sun.beans.util.Cache.get(Cache.java:127)
>   at com.sun.beans.finder.MethodFinder.findMethod(MethodFinder.java:81)
>   at java.beans.Statement.getMethod(Statement.java:357)
>   at java.beans.Statement.invokeInternal(Statement.java:261)
>   at java.beans.Statement.access$000(Statement.java:58)
>   at java.beans.Statement$2.run(Statement.java:185)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.beans.Statement.invoke(Statement.java:182)
>   at java.beans.Expression.getValue(Expression.java:153)
>   at 
> com.sun.beans.decoder.ObjectElementHandler.getValueObject(ObjectElementHandler.java:166)
>   at 
> com.sun.beans.decoder.NewElementHandler.getValueObject(NewElementHandler.java:123)
>   at 
> com.sun.beans.decoder.ElementHandler.getContextBean(ElementHandler.java:113)
>   at 
> com.sun.beans.decoder.NewElementHandler.getContextBean(NewElementHandler.java:109)
>   at 
> com.sun.beans.decoder.ObjectElementHandler.getValueObject(ObjectElementHandler.java:146)
>   at 
> com.sun.beans.decoder.NewElementHandler.getValueObject(NewElementHandler.java:123)
>  

[jira] [Commented] (HIVE-13020) Hive Metastore and HiveServer2 to Zookeeper fails with IBM JDK

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930636#comment-15930636
 ] 

Vaibhav Gumashta commented on HIVE-13020:
-

Committed to 1.2.2

> Hive Metastore and HiveServer2 to Zookeeper fails with IBM JDK
> --
>
> Key: HIVE-13020
> URL: https://issues.apache.org/jira/browse/HIVE-13020
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Metastore, Shims
>Affects Versions: 1.2.0, 1.3.0, 1.2.1
> Environment: Linux X86_64 and IBM JDK 8
>Reporter: Greg Senia
>Assignee: Greg Senia
>  Labels: hdp, ibm, ibm-jdk
> Fix For: 1.3.0, 1.2.2, 2.1.0
>
> Attachments: HIVE-13020.patch, hivemetastore_afterpatch.txt, 
> hivemetastore_beforepatch.txt, hiveserver2_afterpatch.txt, 
> hiveserver2_beforepatch.txt
>
>
> HiveServer2 and Hive Metastore Zookeeper component is hardcoded to only 
> support the Oracle/Open JDK. I was performing testing of Hadoop running on 
> the IBM JDK and discovered this issue and have since drawn up the attached 
> patch. This looks to resolve the issue in a similar manner as how the Hadoop 
> core folks handle the IBM JDK.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-13020) Hive Metastore and HiveServer2 to Zookeeper fails with IBM JDK

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-13020:

Fix Version/s: 1.2.2

> Hive Metastore and HiveServer2 to Zookeeper fails with IBM JDK
> --
>
> Key: HIVE-13020
> URL: https://issues.apache.org/jira/browse/HIVE-13020
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Metastore, Shims
>Affects Versions: 1.2.0, 1.3.0, 1.2.1
> Environment: Linux X86_64 and IBM JDK 8
>Reporter: Greg Senia
>Assignee: Greg Senia
>  Labels: hdp, ibm, ibm-jdk
> Fix For: 1.3.0, 1.2.2, 2.1.0
>
> Attachments: HIVE-13020.patch, hivemetastore_afterpatch.txt, 
> hivemetastore_beforepatch.txt, hiveserver2_afterpatch.txt, 
> hiveserver2_beforepatch.txt
>
>
> HiveServer2 and Hive Metastore Zookeeper component is hardcoded to only 
> support the Oracle/Open JDK. I was performing testing of Hadoop running on 
> the IBM JDK and discovered this issue and have since drawn up the attached 
> patch. This looks to resolve the issue in a similar manner as how the Hadoop 
> core folks handle the IBM JDK.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-13020) Hive Metastore and HiveServer2 to Zookeeper fails with IBM JDK

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-13020:

Target Version/s: 1.3.0, 1.2.2  (was: 1.3.0)

> Hive Metastore and HiveServer2 to Zookeeper fails with IBM JDK
> --
>
> Key: HIVE-13020
> URL: https://issues.apache.org/jira/browse/HIVE-13020
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Metastore, Shims
>Affects Versions: 1.2.0, 1.3.0, 1.2.1
> Environment: Linux X86_64 and IBM JDK 8
>Reporter: Greg Senia
>Assignee: Greg Senia
>  Labels: hdp, ibm, ibm-jdk
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13020.patch, hivemetastore_afterpatch.txt, 
> hivemetastore_beforepatch.txt, hiveserver2_afterpatch.txt, 
> hiveserver2_beforepatch.txt
>
>
> HiveServer2 and Hive Metastore Zookeeper component is hardcoded to only 
> support the Oracle/Open JDK. I was performing testing of Hadoop running on 
> the IBM JDK and discovered this issue and have since drawn up the attached 
> patch. This looks to resolve the issue in a similar manner as how the Hadoop 
> core folks handle the IBM JDK.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-13020) Hive Metastore and HiveServer2 to Zookeeper fails with IBM JDK

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-13020:

Target Version/s: 1.3.0  (was: 1.3.0, 1.2.2)

> Hive Metastore and HiveServer2 to Zookeeper fails with IBM JDK
> --
>
> Key: HIVE-13020
> URL: https://issues.apache.org/jira/browse/HIVE-13020
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Metastore, Shims
>Affects Versions: 1.2.0, 1.3.0, 1.2.1
> Environment: Linux X86_64 and IBM JDK 8
>Reporter: Greg Senia
>Assignee: Greg Senia
>  Labels: hdp, ibm, ibm-jdk
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13020.patch, hivemetastore_afterpatch.txt, 
> hivemetastore_beforepatch.txt, hiveserver2_afterpatch.txt, 
> hiveserver2_beforepatch.txt
>
>
> HiveServer2 and Hive Metastore Zookeeper component is hardcoded to only 
> support the Oracle/Open JDK. I was performing testing of Hadoop running on 
> the IBM JDK and discovered this issue and have since drawn up the attached 
> patch. This looks to resolve the issue in a similar manner as how the Hadoop 
> core folks handle the IBM JDK.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-10308) Vectorization execution throws java.lang.IllegalArgumentException: Unsupported complex type: MAP

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-10308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-10308:

Fix Version/s: (was: 1.2.2)

> Vectorization execution throws java.lang.IllegalArgumentException: 
> Unsupported complex type: MAP
> 
>
> Key: HIVE-10308
> URL: https://issues.apache.org/jira/browse/HIVE-10308
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 0.14.0, 0.13.1, 1.2.0, 1.1.0
>Reporter: Selina Zhang
>Assignee: Matt McCline
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-10308.1.patch
>
>
> Steps to reproduce:
> {code:sql}
> CREATE TABLE test_orc (a INT, b MAP) STORED AS ORC;
> INSERT OVERWRITE TABLE test_orc SELECT 1, MAP(1, "one", 2, "two") FROM src 
> LIMIT 1;
> CREATE TABLE test(key INT) ;
> INSERT OVERWRITE TABLE test SELECT 1 FROM src LIMIT 1;
> set hive.vectorized.execution.enabled=true;
> set hive.auto.convert.join=false;
> select l.key from test l left outer join test_orc r on (l.key= r.a) where r.a 
> is not null;
> {code}
> Stack trace:
> {noformat}
> Caused by: java.lang.IllegalArgumentException: Unsupported complex type: MAP
>   at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory.genVectorExpressionWritable(VectorExpressionWriterFactory.java:456)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory.processVectorInspector(VectorExpressionWriterFactory.java:1191)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.initializeOp(VectorReduceSinkOperator.java:58)
>   at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:362)
>   at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:481)
>   at 
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:438)
>   at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
>   at 
> org.apache.hadoop.hive.ql.exec.MapOperator.initializeMapOperator(MapOperator.java:442)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:198)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-10308) Vectorization execution throws java.lang.IllegalArgumentException: Unsupported complex type: MAP

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-10308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930619#comment-15930619
 ] 

Vaibhav Gumashta commented on HIVE-10308:
-

Removing fix version 1.2.2

> Vectorization execution throws java.lang.IllegalArgumentException: 
> Unsupported complex type: MAP
> 
>
> Key: HIVE-10308
> URL: https://issues.apache.org/jira/browse/HIVE-10308
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 0.14.0, 0.13.1, 1.2.0, 1.1.0
>Reporter: Selina Zhang
>Assignee: Matt McCline
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-10308.1.patch
>
>
> Steps to reproduce:
> {code:sql}
> CREATE TABLE test_orc (a INT, b MAP) STORED AS ORC;
> INSERT OVERWRITE TABLE test_orc SELECT 1, MAP(1, "one", 2, "two") FROM src 
> LIMIT 1;
> CREATE TABLE test(key INT) ;
> INSERT OVERWRITE TABLE test SELECT 1 FROM src LIMIT 1;
> set hive.vectorized.execution.enabled=true;
> set hive.auto.convert.join=false;
> select l.key from test l left outer join test_orc r on (l.key= r.a) where r.a 
> is not null;
> {code}
> Stack trace:
> {noformat}
> Caused by: java.lang.IllegalArgumentException: Unsupported complex type: MAP
>   at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory.genVectorExpressionWritable(VectorExpressionWriterFactory.java:456)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory.processVectorInspector(VectorExpressionWriterFactory.java:1191)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.initializeOp(VectorReduceSinkOperator.java:58)
>   at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:362)
>   at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:481)
>   at 
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:438)
>   at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
>   at 
> org.apache.hadoop.hive.ql.exec.MapOperator.initializeMapOperator(MapOperator.java:442)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:198)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-12495) Lock/unlock table should add database and table information to inputs and outputs of authz hook

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930614#comment-15930614
 ] 

Vaibhav Gumashta commented on HIVE-12495:
-

Removing target 1.2.2 as this is not a blocker.

> Lock/unlock table should add database and table information to inputs and 
> outputs of authz hook
> ---
>
> Key: HIVE-12495
> URL: https://issues.apache.org/jira/browse/HIVE-12495
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization
>Affects Versions: 1.2.1
>Reporter: Dapeng Sun
>Assignee: Dapeng Sun
> Attachments: HIVE-12495.001.patch, HIVE-12495.002.patch
>
>
> According the discussion at HIVE-12367, the jira will target to fix inputs 
> and outputs for lock/unlock table.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-12495) Lock/unlock table should add database and table information to inputs and outputs of authz hook

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-12495:

Target Version/s: 1.3.0, 2.2.0  (was: 1.3.0, 1.2.2, 2.2.0)

> Lock/unlock table should add database and table information to inputs and 
> outputs of authz hook
> ---
>
> Key: HIVE-12495
> URL: https://issues.apache.org/jira/browse/HIVE-12495
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization
>Affects Versions: 1.2.1
>Reporter: Dapeng Sun
>Assignee: Dapeng Sun
> Attachments: HIVE-12495.001.patch, HIVE-12495.002.patch
>
>
> According the discussion at HIVE-12367, the jira will target to fix inputs 
> and outputs for lock/unlock table.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-8678) Pig fails to correctly load DATE fields using HCatalog

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-8678:
---
Fix Version/s: (was: 1.2.2)

> Pig fails to correctly load DATE fields using HCatalog
> --
>
> Key: HIVE-8678
> URL: https://issues.apache.org/jira/browse/HIVE-8678
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.13.1
>Reporter: Michael McLellan
>Assignee: Sushanth Sowmyan
>
> Using:
> Hadoop 2.5.0-cdh5.2.0 
> Pig 0.12.0-cdh5.2.0
> Hive 0.13.1-cdh5.2.0
> When using pig -useHCatalog to load a Hive table that has a DATE field, when 
> trying to DUMP the field, the following error occurs:
> {code}
> 2014-10-30 22:58:05,469 [main] ERROR 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - 
> org.apache.pig.backend.executionengine.ExecException: ERROR 6018: Error 
> converting read value to tuple
> at 
> org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:76)
> at org.apache.hive.hcatalog.pig.HCatLoader.getNext(HCatLoader.java:58)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:211)
> at 
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
> at 
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at 
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to 
> java.sql.Date
> at 
> org.apache.hive.hcatalog.pig.PigHCatUtil.extractPigObject(PigHCatUtil.java:420)
> at 
> org.apache.hive.hcatalog.pig.PigHCatUtil.transformToTuple(PigHCatUtil.java:457)
> at 
> org.apache.hive.hcatalog.pig.PigHCatUtil.transformToTuple(PigHCatUtil.java:375)
> at 
> org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:64)
> 2014-10-30 22:58:05,469 [main] ERROR 
> org.apache.pig.tools.pigstats.SimplePigStats - ERROR 6018: Error converting 
> read value to tuple
> {code}
> It seems to be occuring here: 
> https://github.com/apache/hive/blob/trunk/hcatalog/hcatalog-pig-adapter/src/main/java/org/apache/hive/hcatalog/pig/PigHCatUtil.java#L433
> and that it should be:
> {code}Date d = Date.valueOf(o);{code} 
> instead of 
> {code}Date d = (Date) o;{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-8678) Pig fails to correctly load DATE fields using HCatalog

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930613#comment-15930613
 ] 

Vaibhav Gumashta commented on HIVE-8678:


Removing fix version 1.2.2

> Pig fails to correctly load DATE fields using HCatalog
> --
>
> Key: HIVE-8678
> URL: https://issues.apache.org/jira/browse/HIVE-8678
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.13.1
>Reporter: Michael McLellan
>Assignee: Sushanth Sowmyan
>
> Using:
> Hadoop 2.5.0-cdh5.2.0 
> Pig 0.12.0-cdh5.2.0
> Hive 0.13.1-cdh5.2.0
> When using pig -useHCatalog to load a Hive table that has a DATE field, when 
> trying to DUMP the field, the following error occurs:
> {code}
> 2014-10-30 22:58:05,469 [main] ERROR 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - 
> org.apache.pig.backend.executionengine.ExecException: ERROR 6018: Error 
> converting read value to tuple
> at 
> org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:76)
> at org.apache.hive.hcatalog.pig.HCatLoader.getNext(HCatLoader.java:58)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:211)
> at 
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
> at 
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at 
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to 
> java.sql.Date
> at 
> org.apache.hive.hcatalog.pig.PigHCatUtil.extractPigObject(PigHCatUtil.java:420)
> at 
> org.apache.hive.hcatalog.pig.PigHCatUtil.transformToTuple(PigHCatUtil.java:457)
> at 
> org.apache.hive.hcatalog.pig.PigHCatUtil.transformToTuple(PigHCatUtil.java:375)
> at 
> org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:64)
> 2014-10-30 22:58:05,469 [main] ERROR 
> org.apache.pig.tools.pigstats.SimplePigStats - ERROR 6018: Error converting 
> read value to tuple
> {code}
> It seems to be occuring here: 
> https://github.com/apache/hive/blob/trunk/hcatalog/hcatalog-pig-adapter/src/main/java/org/apache/hive/hcatalog/pig/PigHCatUtil.java#L433
> and that it should be:
> {code}Date d = Date.valueOf(o);{code} 
> instead of 
> {code}Date d = (Date) o;{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-14210) ExecDriver should call jobclient.close() to trigger cleanup

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-14210:

Fix Version/s: 1.2.2

> ExecDriver should call jobclient.close() to trigger cleanup
> ---
>
> Key: HIVE-14210
> URL: https://issues.apache.org/jira/browse/HIVE-14210
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, HiveServer2
>Affects Versions: 1.2.1, 2.0.0, 2.1.0
>Reporter: Thomas Friedrich
>Assignee: Thomas Friedrich
> Fix For: 1.3.0, 1.2.2, 2.2.0, 2.1.1
>
> Attachments: HIVE-14210.1.patch, HIVE-14210.patch
>
>
> We found an issue in a customer environment where the HS2 crashed after a few 
> days and the Java core dump contained several thousands of truststore 
> reloader threads:
> "Truststore reloader thread" #126 daemon prio=5 os_prio=0 
> tid=0x7f680d2e3000 nid=0x98fd waiting on 
> condition [0x7f67e482c000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.security.ssl.ReloadingX509TrustManager.run
> (ReloadingX509TrustManager.java:225)
> at java.lang.Thread.run(Thread.java:745)
> We found the issue to be caused by a bug in Hadoop where the 
> TimelineClientImpl is not destroying the SSLFactory if SSL is enabled in 
> Hadoop and the timeline server is running. I opened YARN-5309 which has more 
> details on the problem, and a patch was submitted a few days back.
> In addition to the changes in Hadoop, there are a couple of Hive changes 
> required:
> - ExecDriver needs to call jobclient.close() to trigger the clean-up of the 
> resources after the submitted job is done/failed
> - Hive needs to pick up a newer release of Hadoop to pick up MAPREDUCE-6618 
> and MAPREDUCE-6621 that fixed issues with calling jobclient.close(). Both 
> fixes are included in Hadoop 2.6.4. 
> However, since we also need to pick up YARN-5309, we need to wait for a new 
> release of Hadoop.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-14210) ExecDriver should call jobclient.close() to trigger cleanup

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-14210:

Target Version/s: 1.2.2, 2.2.0  (was: 1.2.1, 2.2.0)

> ExecDriver should call jobclient.close() to trigger cleanup
> ---
>
> Key: HIVE-14210
> URL: https://issues.apache.org/jira/browse/HIVE-14210
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, HiveServer2
>Affects Versions: 1.2.1, 2.0.0, 2.1.0
>Reporter: Thomas Friedrich
>Assignee: Thomas Friedrich
> Fix For: 1.3.0, 1.2.2, 2.2.0, 2.1.1
>
> Attachments: HIVE-14210.1.patch, HIVE-14210.patch
>
>
> We found an issue in a customer environment where the HS2 crashed after a few 
> days and the Java core dump contained several thousands of truststore 
> reloader threads:
> "Truststore reloader thread" #126 daemon prio=5 os_prio=0 
> tid=0x7f680d2e3000 nid=0x98fd waiting on 
> condition [0x7f67e482c000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.security.ssl.ReloadingX509TrustManager.run
> (ReloadingX509TrustManager.java:225)
> at java.lang.Thread.run(Thread.java:745)
> We found the issue to be caused by a bug in Hadoop where the 
> TimelineClientImpl is not destroying the SSLFactory if SSL is enabled in 
> Hadoop and the timeline server is running. I opened YARN-5309 which has more 
> details on the problem, and a patch was submitted a few days back.
> In addition to the changes in Hadoop, there are a couple of Hive changes 
> required:
> - ExecDriver needs to call jobclient.close() to trigger the clean-up of the 
> resources after the submitted job is done/failed
> - Hive needs to pick up a newer release of Hadoop to pick up MAPREDUCE-6618 
> and MAPREDUCE-6621 that fixed issues with calling jobclient.close(). Both 
> fixes are included in Hadoop 2.6.4. 
> However, since we also need to pick up YARN-5309, we need to wait for a new 
> release of Hadoop.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-14210) ExecDriver should call jobclient.close() to trigger cleanup

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930599#comment-15930599
 ] 

Vaibhav Gumashta commented on HIVE-14210:
-

Committed to 1.2

> ExecDriver should call jobclient.close() to trigger cleanup
> ---
>
> Key: HIVE-14210
> URL: https://issues.apache.org/jira/browse/HIVE-14210
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, HiveServer2
>Affects Versions: 1.2.1, 2.0.0, 2.1.0
>Reporter: Thomas Friedrich
>Assignee: Thomas Friedrich
> Fix For: 1.3.0, 1.2.2, 2.2.0, 2.1.1
>
> Attachments: HIVE-14210.1.patch, HIVE-14210.patch
>
>
> We found an issue in a customer environment where the HS2 crashed after a few 
> days and the Java core dump contained several thousands of truststore 
> reloader threads:
> "Truststore reloader thread" #126 daemon prio=5 os_prio=0 
> tid=0x7f680d2e3000 nid=0x98fd waiting on 
> condition [0x7f67e482c000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.security.ssl.ReloadingX509TrustManager.run
> (ReloadingX509TrustManager.java:225)
> at java.lang.Thread.run(Thread.java:745)
> We found the issue to be caused by a bug in Hadoop where the 
> TimelineClientImpl is not destroying the SSLFactory if SSL is enabled in 
> Hadoop and the timeline server is running. I opened YARN-5309 which has more 
> details on the problem, and a patch was submitted a few days back.
> In addition to the changes in Hadoop, there are a couple of Hive changes 
> required:
> - ExecDriver needs to call jobclient.close() to trigger the clean-up of the 
> resources after the submitted job is done/failed
> - Hive needs to pick up a newer release of Hadoop to pick up MAPREDUCE-6618 
> and MAPREDUCE-6621 that fixed issues with calling jobclient.close(). Both 
> fixes are included in Hadoop 2.6.4. 
> However, since we also need to pick up YARN-5309, we need to wait for a new 
> release of Hadoop.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16245) Vectorization: Does not handle non-column key expressions in MERGEPARTIAL mode

[Matt McCline (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930597#comment-15930597
 ] 

Matt McCline commented on HIVE-16245:
-

Solution has not been verified on the cluster it occurred on yet...

> Vectorization: Does not handle non-column key expressions in MERGEPARTIAL mode
> --
>
> Key: HIVE-16245
> URL: https://issues.apache.org/jira/browse/HIVE-16245
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-16245.01.patch
>
>
> When the planner is able to make a column a constant, MERGEPARTIAL mode in 
> VectorGroupByOperator is broken because it doesn't evaluate the key 
> expression.  One result is execution cast exception errors.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16245) Vectorization: Does not handle non-column key expressions in MERGEPARTIAL mode

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-16245:

Attachment: HIVE-16245.01.patch

> Vectorization: Does not handle non-column key expressions in MERGEPARTIAL mode
> --
>
> Key: HIVE-16245
> URL: https://issues.apache.org/jira/browse/HIVE-16245
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-16245.01.patch
>
>
> When the planner is able to make a column a constant, MERGEPARTIAL mode in 
> VectorGroupByOperator is broken because it doesn't evaluate the key 
> expression.  One result is execution cast exception errors.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16245) Vectorization: Does not handle non-column key expressions in MERGEPARTIAL mode

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline reassigned HIVE-16245:
---


> Vectorization: Does not handle non-column key expressions in MERGEPARTIAL mode
> --
>
> Key: HIVE-16245
> URL: https://issues.apache.org/jira/browse/HIVE-16245
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
>
> When the planner is able to make a column a constant, MERGEPARTIAL mode in 
> VectorGroupByOperator is broken because it doesn't evaluate the key 
> expression.  One result is execution cast exception errors.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15007) Hive 1.2.2 release planning

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930585#comment-15930585
 ] 

Vaibhav Gumashta commented on HIVE-15007:
-

Ran all the tests locally. The ones that need to be looked into (potential 
bugs) are: TestCliDriver#join_merge_multi_expressions, 
TestMinimrCliDriver#exchgpartition2lel, 
TestMiniTezCliDriver#vector_auto_smb_mapjoin_14, 
TestMiniTezCliDriver#tez_smb_empty,  TestMiniTezCliDriver#bucket_map_join_tez1. 
All have a jira for investigation under this umbrella jira. I'll check to see 
if any of these are blockers bugs.

> Hive 1.2.2 release planning
> ---
>
> Key: HIVE-15007
> URL: https://issues.apache.org/jira/browse/HIVE-15007
> Project: Hive
>  Issue Type: Task
>Affects Versions: 1.2.1
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-15007-branch-1.2.patch, 
> HIVE-15007-branch-1.2.patch, HIVE-15007-branch-1.2.patch, 
> HIVE-15007-branch-1.2.patch, HIVE-15007-branch-1.2.patch, 
> HIVE-15007-branch-1.2.patch, HIVE-15007.branch-1.2.patch
>
>
> Discussed with [~spena] about triggering unit test runs for 1.2.2 release and 
> creating a patch which will trigger precommits looks like a good way.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-14210) ExecDriver should call jobclient.close() to trigger cleanup

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-14210:

Target Version/s: 1.2.1, 2.2.0

Needs to be committed to 1.2; I'll commit shortly.

> ExecDriver should call jobclient.close() to trigger cleanup
> ---
>
> Key: HIVE-14210
> URL: https://issues.apache.org/jira/browse/HIVE-14210
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, HiveServer2
>Affects Versions: 1.2.1, 2.0.0, 2.1.0
>Reporter: Thomas Friedrich
>Assignee: Thomas Friedrich
> Fix For: 1.3.0, 2.2.0, 2.1.1
>
> Attachments: HIVE-14210.1.patch, HIVE-14210.patch
>
>
> We found an issue in a customer environment where the HS2 crashed after a few 
> days and the Java core dump contained several thousands of truststore 
> reloader threads:
> "Truststore reloader thread" #126 daemon prio=5 os_prio=0 
> tid=0x7f680d2e3000 nid=0x98fd waiting on 
> condition [0x7f67e482c000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.security.ssl.ReloadingX509TrustManager.run
> (ReloadingX509TrustManager.java:225)
> at java.lang.Thread.run(Thread.java:745)
> We found the issue to be caused by a bug in Hadoop where the 
> TimelineClientImpl is not destroying the SSLFactory if SSL is enabled in 
> Hadoop and the timeline server is running. I opened YARN-5309 which has more 
> details on the problem, and a patch was submitted a few days back.
> In addition to the changes in Hadoop, there are a couple of Hive changes 
> required:
> - ExecDriver needs to call jobclient.close() to trigger the clean-up of the 
> resources after the submitted job is done/failed
> - Hive needs to pick up a newer release of Hadoop to pick up MAPREDUCE-6618 
> and MAPREDUCE-6621 that fixed issues with calling jobclient.close(). Both 
> fixes are included in Hadoop 2.6.4. 
> However, since we also need to pick up YARN-5309, we need to wait for a new 
> release of Hadoop.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-14210) ExecDriver should call jobclient.close() to trigger cleanup

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-14210:

Fix Version/s: (was: 1.2.2)

> ExecDriver should call jobclient.close() to trigger cleanup
> ---
>
> Key: HIVE-14210
> URL: https://issues.apache.org/jira/browse/HIVE-14210
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, HiveServer2
>Affects Versions: 1.2.1, 2.0.0, 2.1.0
>Reporter: Thomas Friedrich
>Assignee: Thomas Friedrich
> Fix For: 1.3.0, 2.2.0, 2.1.1
>
> Attachments: HIVE-14210.1.patch, HIVE-14210.patch
>
>
> We found an issue in a customer environment where the HS2 crashed after a few 
> days and the Java core dump contained several thousands of truststore 
> reloader threads:
> "Truststore reloader thread" #126 daemon prio=5 os_prio=0 
> tid=0x7f680d2e3000 nid=0x98fd waiting on 
> condition [0x7f67e482c000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.security.ssl.ReloadingX509TrustManager.run
> (ReloadingX509TrustManager.java:225)
> at java.lang.Thread.run(Thread.java:745)
> We found the issue to be caused by a bug in Hadoop where the 
> TimelineClientImpl is not destroying the SSLFactory if SSL is enabled in 
> Hadoop and the timeline server is running. I opened YARN-5309 which has more 
> details on the problem, and a patch was submitted a few days back.
> In addition to the changes in Hadoop, there are a couple of Hive changes 
> required:
> - ExecDriver needs to call jobclient.close() to trigger the clean-up of the 
> resources after the submitted job is done/failed
> - Hive needs to pick up a newer release of Hadoop to pick up MAPREDUCE-6618 
> and MAPREDUCE-6621 that fixed issues with calling jobclient.close(). Both 
> fixes are included in Hadoop 2.6.4. 
> However, since we also need to pick up YARN-5309, we need to wait for a new 
> release of Hadoop.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16186) REPL DUMP shows last event ID of the database even if we use LIMIT option.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-16186:

Summary: REPL DUMP shows last event ID of the database even if we use LIMIT 
option.  (was: REPL DUMP always dump all the events in the given range even if 
we use LIMIT option.)

> REPL DUMP shows last event ID of the database even if we use LIMIT option.
> --
>
> Key: HIVE-16186
> URL: https://issues.apache.org/jira/browse/HIVE-16186
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR
>
> Looks like LIMIT option doesn't work well with REPL DUMP.
> 0: jdbc:hive2://localhost:10001/default> REPL DUMP default FROM 170 LIMIT 1;
> +--+---+
> | dump_dir | last_repl_id  |
> +--+---+
> | /tmp/dump/1489395053411  | 195   |
> +--+---+



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)