[jira] [Created] (HIVE-14154) Select from ORC table stored VectorizedOrcInputFormat fails

2016-07-01 Thread Zhu Li (JIRA) 
Zhu Li created HIVE-14154:
-

 Summary: Select from ORC table stored VectorizedOrcInputFormat 
fails
 Key: HIVE-14154
 URL: https://issues.apache.org/jira/browse/HIVE-14154
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Zhu Li


TABLE INFORMATION:
# col_name  data_type   comment 
 
c_1m_l  bigint  
c_10_l  bigint  
c_100_l bigint  
c_1k_l  bigint  
c_10k_l bigint  
c_100k_lbigint  
c_1k_r  bigint  
c_1m_r  bigint  
s_5000_rchar(40)
s_1m_r  char(40)
 
# Detailed Table Information 
Database:   default  
Owner:  hdfs 
CreateTime: Fri Jul 01 17:32:23 PDT 2016 
LastAccessTime: UNKNOWN  
Protect Mode:   None 
Retention:  0
Location:   
hdfs://mynamenode:8020/apps/hive/warehouse/t_1m_2_orc_vec
Table Type: MANAGED_TABLE
Table Parameters:
COLUMN_STATS_ACCURATE   true
numFiles8   
numRows 1000
rawDataSize 312000  
totalSize   105254801   
transient_lastDdlTime   1467419689   
Storage Information  
SerDe Library:  org.apache.hadoop.hive.ql.io.orc.OrcSerde
InputFormat: org.apache.hadoop.hive.ql.io.orc.VectorizedOrcInputFormat  
 
OutputFormat:  org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat  
Compressed: No   
Num Buckets:-1   
Bucket Columns: []   
Sort Columns:   []   
Storage Desc Params: 
serialization.format1 

QUERY: select * from t_1m_2_orc_vec limit 10;

ERROR:
Failed with exception java.io.IOException:java.lang.RuntimeException: 
java.lang.RuntimeException: Failed to load plan: null: 
java.lang.NullPointerException

Additional information:
input format is set as 
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcInputFormat 

When I want to read from this table with HCatalog API, I got the following 
error:
Exception in thread "main" java.lang.RuntimeException: 
java.lang.RuntimeException: Failed to load plan: null: 
java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcInputFormat$VectorizedOrcRecordReader.(VectorizedOrcInputFormat.java:76)
at 
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcInputFormat.getRecordReader(VectorizedOrcInputFormat.java:156)
at 
org.apache.hive.hcatalog.mapreduce.HCatRecordReader.createBaseRecordReader(HCatRecordReader.java:116)
at 
org.apache.hive.hcatalog.mapreduce.HCatRecordReader.initialize(HCatRecordReader.java:91)
at OrcConnector.main(OrcConnector.java:254)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: java.lang.RuntimeException: Failed to load plan: null: 
java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:461)
at 
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:300)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx.init(VectorizedRowBatchCtx.java:171)
at 
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcInputFormat$VectorizedOrcRecordReader.(VectorizedOrcInputFormat.java:74)
... 9 more
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:416)
... 12 more




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14153) Beeline: beeline history doesn't work on Hive2

2016-07-01 Thread Vaibhav Gumashta (JIRA) 
Vaibhav Gumashta created HIVE-14153:
---

 Summary: Beeline: beeline history doesn't work on Hive2
 Key: HIVE-14153
 URL: https://issues.apache.org/jira/browse/HIVE-14153
 Project: Hive
  Issue Type: Bug
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta


The up arrow on console is supposed to display history, which is broken 
currently. Changes in HIVE-6758 broke it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14152) datanucleus.autoStartMechanismMode should set to 'Ignored' to allow rolling downgrade

2016-07-01 Thread Daniel Dai (JIRA) 
Daniel Dai created HIVE-14152:
-

 Summary: datanucleus.autoStartMechanismMode should set to 
'Ignored' to allow rolling downgrade 
 Key: HIVE-14152
 URL: https://issues.apache.org/jira/browse/HIVE-14152
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Reporter: Daniel Dai
Assignee: Daniel Dai


We see the following issue when downgrading metastore:
1. Run some query using new tables
2. Downgrade metastore
3. Restart metastore will complain the new table does not exist

In particular, constaints tables does not exist in branch-1. If we run Hive 2 
and create a constraint, then downgrade metastore to Hive 1, datanucleus will 
complain:
{code}
javax.jdo.JDOFatalUserException: Error starting up DataNucleus : a class 
"org.apache.hadoop.hive.metastore.model.MConstraint" was listed as being 
persisted previously in this datastore, yet the class wasnt found. Perhaps it 
is used by a different DataNucleus-enabled application in this datastore, or 
you have changed your class names.
at 
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:528)
at 
org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:788)
at 
org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333)
at 
org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
at java.security.AccessController.doPrivileged(Native Method)
at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
at 
javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
at 
org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:377)
at 
org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:406)
at 
org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:299)
at 
org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:266)
at 
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
at 
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at 
org.apache.hadoop.hive.metastore.RawStoreProxy.(RawStoreProxy.java:60)
at 
org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:69)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:650)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:628)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:677)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:484)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:77)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:83)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5905)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5900)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6159)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6084)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
{code}

Apparently datanucleus cache some trace about the new table and retry to 
reinstantiate later. This breaks downgrading we shall disable this behavior.

We need to set "datanucleus.autoStartMechanismMode" to "Ignored" to disable the 
check since it becomes a norm in downgrading case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 49498: HIVE-14123: Add beeline configuration option to show database in the prompt

2016-07-01 Thread Vihang Karajgaonkar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49498/#review140391
---



Thanks for the changes. Can you please test your code by setting 
USE_DEPRECATED_CLI=false.


beeline/src/java/org/apache/hive/beeline/BeeLine.java (line 1445)


Is there a reason to remove previous check for 
HiveConf.ConfVars.CLIPRINTCURRENTDB? Looks like this method is getting called 
from getPromptForCli() and if the beeline option for db is false, it will not 
set db name in the cli prompt too irrespective of the value of 
HiveConf.ConfVars.CLIPRINTCURRENTDB. btw, you may want to look at HIVE-14151 if 
you finding trouble while using beeline as cli



beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java (line 69)


nit, add a space after =


- Vihang Karajgaonkar


On July 1, 2016, 3:08 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49498/
> ---
> 
> (Updated July 1, 2016, 3:08 p.m.)
> 
> 
> Review request for hive, Sergio Pena, Szehon Ho, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-14123
> https://issues.apache.org/jira/browse/HIVE-14123
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> There are several jira issues complaining that, the Beeline does not respect 
> hive.cli.print.current.db.
> This is partially true, since in embedded mode, it uses the 
> hive.cli.print.current.db to change the prompt, since HIVE-10511.
> In remote mode, I think this function should use a beeline command line 
> option instead, like for the showHeader option emphasizing, that this is a 
> client side option.
> 
> The patch contains:
> - New configuration option
> - Changing the help text
> - Updating command hooks, to run in remote mode as well
> - Adding new hooks, for connect and go sqllite commands
> - Generalize database connection refresh command
> - Changing prompt
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLine.java 66185f6 
>   beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java 5aaa385 
>   beeline/src/java/org/apache/hive/beeline/ClientCommandHookFactory.java 
> c4d97bc 
>   beeline/src/java/org/apache/hive/beeline/Commands.java 3a204c0 
>   beeline/src/java/org/apache/hive/beeline/ReflectiveCommandHandler.java 
> 3b863ae 
>   beeline/src/main/resources/BeeLine.properties 7500df9 
>   beeline/src/test/org/apache/hive/beeline/TestBeelineArgParsing.java ce1f538 
>   beeline/src/test/org/apache/hive/beeline/TestClientCommandHookFactory.java 
> c86de0a 
> 
> Diff: https://reviews.apache.org/r/49498/diff/
> 
> 
> Testing
> ---
> 
> Unit tests for the hooks, and the configuration option
> 
> Manual test in remote, and embedded mode
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

[jira] [Created] (HIVE-14151) Use of USE_DEPRECATED_CLI environment variable does not work

2016-07-01 Thread Vihang Karajgaonkar (JIRA) 
Vihang Karajgaonkar created HIVE-14151:
--

 Summary: Use of USE_DEPRECATED_CLI environment variable does not 
work
 Key: HIVE-14151
 URL: https://issues.apache.org/jira/browse/HIVE-14151
 Project: Hive
  Issue Type: Bug
  Components: CLI
Reporter: Vihang Karajgaonkar


According to 
https://cwiki.apache.org/confluence/display/Hive/Replacing+the+Implementation+of+Hive+CLI+Using+Beeline
 if we set USE_DEPRECATED_CLI=false it should use beeline for hiveCli. But it 
doesn't seem to work.

In order to reproduce this issue:
{noformat}
$ echo $USE_DEPRECATED_CLI

$ ./hive
Hive-on-MR is deprecated in Hive 2 and may not be available in the future 
versions. Consider using a different execution engine (i.e. tez, spark) or 
using Hive 1.X releases.
hive>
$
$ export USE_DEPRECATED_CLI=false
$ echo $USE_DEPRECATED_CLI
false
$ ./hive
Hive-on-MR is deprecated in Hive 2 and may not be available in the future 
versions. Consider using a different execution engine (i.e. tez, spark) or 
using Hive 1.X releases.
hive>
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14150) Hive does not compile against Hadoop-2.9.0-SNAPSHOT

2016-07-01 Thread Siddharth Seth (JIRA) 
Siddharth Seth created HIVE-14150:
-

 Summary: Hive does not compile against Hadoop-2.9.0-SNAPSHOT
 Key: HIVE-14150
 URL: https://issues.apache.org/jira/browse/HIVE-14150
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.1.0
Reporter: Siddharth Seth
Priority: Critical


JvmPauseMonitor, JvmMetrics used in LLAP are private classes in Hadoop - and 
have changed in 2.9.

Hive has it's own version of JvmPauseMonitor. Need to see what can be done with 
JvmMetrics



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14149) S3A connector throws an AmazonS3Exception on Hadoop3.0 with Java1.8u60 and higher

2016-07-01 Thread Abdullah Yousufi (JIRA) 
Abdullah Yousufi created HIVE-14149:
---

 Summary: S3A connector throws an AmazonS3Exception on Hadoop3.0 
with Java1.8u60 and higher
 Key: HIVE-14149
 URL: https://issues.apache.org/jira/browse/HIVE-14149
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 2.1.0
Reporter: Abdullah Yousufi
Assignee: Abdullah Yousufi


Java1.8u60 and higher cause Joda Time 2.5 to incorrectly format timezones, 
which leads to the aws server rejecting requests with the aws sdk hadoop3.0 
uses. This means any queries involving the s3a connector will return the 
following AmazonS3Exception:
{code}
com.amazonaws.services.s3.model.AmazonS3Exception: AWS authentication requires 
a valid Date or x-amz-date header
{code}

The fix for this is to update Joda Time from 2.5 to 2.8.1. See here for details:
https://github.com/aws/aws-sdk-java/issues/444



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [Announce] New Hive Committer - Mohit Sabharwal

2016-07-01 Thread Sergio Pena 
That's great. Congratulations Mohit !!

On Fri, Jul 1, 2016 at 3:33 PM, Jimmy Xiang  wrote:

> Congrats!!
>
> On Fri, Jul 1, 2016 at 1:04 PM, Lenni Kuff  wrote:
> > Congrats Mohit!
> >
> > On Fri, Jul 1, 2016 at 3:27 PM, Peter Vary  wrote:
> >
> >> Congratulations Mohit!
> >> 2016. júl. 1. 19:10 ezt írta ("Vihang Karajgaonkar" <
> vih...@cloudera.com
> >> >):
> >>
> >> > Congratulations Mohit!
> >> >
> >> > > On Jul 1, 2016, at 10:05 AM, Chao Sun  wrote:
> >> > >
> >> > > Congratulations Mohit! Good job!
> >> > >
> >> > > Best,
> >> > > Chao
> >> > >
> >> > > On Fri, Jul 1, 2016 at 9:57 AM, Szehon Ho  >> > > wrote:
> >> > > On behalf of the Apache Hive PMC, I'm pleased to announce that Mohit
> >> > Sabharwal has been voted a committer on the Apache Hive project.
> >> > >
> >> > > Please join me in congratulating Mohit !
> >> > >
> >> > > Thanks,
> >> > > Szehon
> >> > >
> >> >
> >> >
> >>
>

Re: [Announce] New Hive Committer - Mohit Sabharwal

2016-07-01 Thread Jimmy Xiang 
Congrats!!

On Fri, Jul 1, 2016 at 1:04 PM, Lenni Kuff  wrote:
> Congrats Mohit!
>
> On Fri, Jul 1, 2016 at 3:27 PM, Peter Vary  wrote:
>
>> Congratulations Mohit!
>> 2016. júl. 1. 19:10 ezt írta ("Vihang Karajgaonkar" > >):
>>
>> > Congratulations Mohit!
>> >
>> > > On Jul 1, 2016, at 10:05 AM, Chao Sun  wrote:
>> > >
>> > > Congratulations Mohit! Good job!
>> > >
>> > > Best,
>> > > Chao
>> > >
>> > > On Fri, Jul 1, 2016 at 9:57 AM, Szehon Ho > > > wrote:
>> > > On behalf of the Apache Hive PMC, I'm pleased to announce that Mohit
>> > Sabharwal has been voted a committer on the Apache Hive project.
>> > >
>> > > Please join me in congratulating Mohit !
>> > >
>> > > Thanks,
>> > > Szehon
>> > >
>> >
>> >
>>

Re: [Announce] New Hive Committer - Mohit Sabharwal

2016-07-01 Thread Lenni Kuff 
Congrats Mohit!

On Fri, Jul 1, 2016 at 3:27 PM, Peter Vary  wrote:

> Congratulations Mohit!
> 2016. júl. 1. 19:10 ezt írta ("Vihang Karajgaonkar"  >):
>
> > Congratulations Mohit!
> >
> > > On Jul 1, 2016, at 10:05 AM, Chao Sun  wrote:
> > >
> > > Congratulations Mohit! Good job!
> > >
> > > Best,
> > > Chao
> > >
> > > On Fri, Jul 1, 2016 at 9:57 AM, Szehon Ho  > > wrote:
> > > On behalf of the Apache Hive PMC, I'm pleased to announce that Mohit
> > Sabharwal has been voted a committer on the Apache Hive project.
> > >
> > > Please join me in congratulating Mohit !
> > >
> > > Thanks,
> > > Szehon
> > >
> >
> >
>

Re: Review Request 49498: HIVE-14123: Add beeline configuration option to show database in the prompt

2016-07-01 Thread Peter Vary 


> On júl. 1, 2016, 5:19 du, Szehon Ho wrote:
> > beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java, line 481
> > 
> >
> > It seems a hacky way to detect embedded mode.
> > 
> > What do you think about doing an or, say if the command line opts is 
> > set, or the hive.cli.pring.current.db is in the properties file.  (i guess 
> > now there is beeline.properties)

To tell the truth, I did not know, that there were two modes of running 
beeline, until I found this type of check everywhere in the code, and finally 
found it where this was set. :) See HiveCli. I will think about your 
suggestions too, thanks


> On júl. 1, 2016, 5:19 du, Szehon Ho wrote:
> > beeline/src/java/org/apache/hive/beeline/ClientCommandHookFactory.java, 
> > line 64
> > 
> >
> > Just curious, did we explore an option to parse the database from the 
> > 'use' command on success to avoid the wasteful extra query, or was there 
> > some instances that it did not work as expected?

Actually we could parse the use, and the connect commands. The use parsing was 
already there, and I did the url parsing using the jdbc uri parsing util class, 
but for the go command we do not have anything at hand to use. So for keeping 
the code simple, I decided to use a single hook for every command. My reasoning 
behind it was, that this will run only after issuing one of the use, connrct, 
go commands and it is not prone to any dubious parsing, or other errors.

Thanks for the comments


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49498/#review140385
---


On júl. 1, 2016, 3:08 du, Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49498/
> ---
> 
> (Updated júl. 1, 2016, 3:08 du)
> 
> 
> Review request for hive, Sergio Pena, Szehon Ho, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-14123
> https://issues.apache.org/jira/browse/HIVE-14123
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> There are several jira issues complaining that, the Beeline does not respect 
> hive.cli.print.current.db.
> This is partially true, since in embedded mode, it uses the 
> hive.cli.print.current.db to change the prompt, since HIVE-10511.
> In remote mode, I think this function should use a beeline command line 
> option instead, like for the showHeader option emphasizing, that this is a 
> client side option.
> 
> The patch contains:
> - New configuration option
> - Changing the help text
> - Updating command hooks, to run in remote mode as well
> - Adding new hooks, for connect and go sqllite commands
> - Generalize database connection refresh command
> - Changing prompt
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLine.java 66185f6 
>   beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java 5aaa385 
>   beeline/src/java/org/apache/hive/beeline/ClientCommandHookFactory.java 
> c4d97bc 
>   beeline/src/java/org/apache/hive/beeline/Commands.java 3a204c0 
>   beeline/src/java/org/apache/hive/beeline/ReflectiveCommandHandler.java 
> 3b863ae 
>   beeline/src/main/resources/BeeLine.properties 7500df9 
>   beeline/src/test/org/apache/hive/beeline/TestBeelineArgParsing.java ce1f538 
>   beeline/src/test/org/apache/hive/beeline/TestClientCommandHookFactory.java 
> c86de0a 
> 
> Diff: https://reviews.apache.org/r/49498/diff/
> 
> 
> Testing
> ---
> 
> Unit tests for the hooks, and the configuration option
> 
> Manual test in remote, and embedded mode
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: [Announce] New Hive Committer - Mohit Sabharwal

2016-07-01 Thread Peter Vary 
Congratulations Mohit!
2016. júl. 1. 19:10 ezt írta ("Vihang Karajgaonkar" ):

> Congratulations Mohit!
>
> > On Jul 1, 2016, at 10:05 AM, Chao Sun  wrote:
> >
> > Congratulations Mohit! Good job!
> >
> > Best,
> > Chao
> >
> > On Fri, Jul 1, 2016 at 9:57 AM, Szehon Ho  > wrote:
> > On behalf of the Apache Hive PMC, I'm pleased to announce that Mohit
> Sabharwal has been voted a committer on the Apache Hive project.
> >
> > Please join me in congratulating Mohit !
> >
> > Thanks,
> > Szehon
> >
>
>

Re: Review Request 49498: HIVE-14123: Add beeline configuration option to show database in the prompt

2016-07-01 Thread Szehon Ho 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49498/#review140385
---




beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java (line 481)


It seems a hacky way to detect embedded mode.

What do you think about doing an or, say if the command line opts is set, 
or the hive.cli.pring.current.db is in the properties file.  (i guess now there 
is beeline.properties)



beeline/src/java/org/apache/hive/beeline/ClientCommandHookFactory.java (line 64)


Just curious, did we explore an option to parse the database from the 'use' 
command on success to avoid the wasteful extra query, or was there some 
instances that it did not work as expected?


Thanks for the patch, some comments/questions.

- Szehon Ho


On July 1, 2016, 3:08 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49498/
> ---
> 
> (Updated July 1, 2016, 3:08 p.m.)
> 
> 
> Review request for hive, Sergio Pena, Szehon Ho, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-14123
> https://issues.apache.org/jira/browse/HIVE-14123
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> There are several jira issues complaining that, the Beeline does not respect 
> hive.cli.print.current.db.
> This is partially true, since in embedded mode, it uses the 
> hive.cli.print.current.db to change the prompt, since HIVE-10511.
> In remote mode, I think this function should use a beeline command line 
> option instead, like for the showHeader option emphasizing, that this is a 
> client side option.
> 
> The patch contains:
> - New configuration option
> - Changing the help text
> - Updating command hooks, to run in remote mode as well
> - Adding new hooks, for connect and go sqllite commands
> - Generalize database connection refresh command
> - Changing prompt
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLine.java 66185f6 
>   beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java 5aaa385 
>   beeline/src/java/org/apache/hive/beeline/ClientCommandHookFactory.java 
> c4d97bc 
>   beeline/src/java/org/apache/hive/beeline/Commands.java 3a204c0 
>   beeline/src/java/org/apache/hive/beeline/ReflectiveCommandHandler.java 
> 3b863ae 
>   beeline/src/main/resources/BeeLine.properties 7500df9 
>   beeline/src/test/org/apache/hive/beeline/TestBeelineArgParsing.java ce1f538 
>   beeline/src/test/org/apache/hive/beeline/TestClientCommandHookFactory.java 
> c86de0a 
> 
> Diff: https://reviews.apache.org/r/49498/diff/
> 
> 
> Testing
> ---
> 
> Unit tests for the hooks, and the configuration option
> 
> Manual test in remote, and embedded mode
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: [Announce] New Hive Committer - Mohit Sabharwal

2016-07-01 Thread Vihang Karajgaonkar 
Congratulations Mohit!

> On Jul 1, 2016, at 10:05 AM, Chao Sun  wrote:
> 
> Congratulations Mohit! Good job!
> 
> Best,
> Chao
> 
> On Fri, Jul 1, 2016 at 9:57 AM, Szehon Ho  > wrote:
> On behalf of the Apache Hive PMC, I'm pleased to announce that Mohit 
> Sabharwal has been voted a committer on the Apache Hive project.
> 
> Please join me in congratulating Mohit !
> 
> Thanks,
> Szehon
>

Re: [Announce] New Hive Committer - Mohit Sabharwal

2016-07-01 Thread Chao Sun 
Congratulations Mohit! Good job!

Best,
Chao

On Fri, Jul 1, 2016 at 9:57 AM, Szehon Ho  wrote:

> On behalf of the Apache Hive PMC, I'm pleased to announce that Mohit
> Sabharwal has been voted a committer on the Apache Hive project.
>
> Please join me in congratulating Mohit !
>
> Thanks,
> Szehon
>

[Announce] New Hive Committer - Mohit Sabharwal

2016-07-01 Thread Szehon Ho 
On behalf of the Apache Hive PMC, I'm pleased to announce that Mohit
Sabharwal has been voted a committer on the Apache Hive project.

Please join me in congratulating Mohit !

Thanks,
Szehon

[jira] [Created] (HIVE-14148) Add branch-2.1 branch to pre-commit tests

2016-07-01 Thread JIRA 
Sergio Peña created HIVE-14148:
--

 Summary: Add branch-2.1 branch to pre-commit tests
 Key: HIVE-14148
 URL: https://issues.apache.org/jira/browse/HIVE-14148
 Project: Hive
  Issue Type: Task
  Components: Testing Infrastructure
Reporter: Sergio Peña
Assignee: Sergio Peña






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14147) Hive PPD might remove predicates when they are defined as a simple node e.g. "WHERE pred"

2016-07-01 Thread Jesus Camacho Rodriguez (JIRA) 
Jesus Camacho Rodriguez created HIVE-14147:
--

 Summary: Hive PPD might remove predicates when they are defined as 
a simple node e.g. "WHERE pred"
 Key: HIVE-14147
 URL: https://issues.apache.org/jira/browse/HIVE-14147
 Project: Hive
  Issue Type: Bug
  Components: Logical Optimizer
Affects Versions: 2.2.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez
Priority: Critical






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 49498: HIVE-14123: Add beeline configuration option to show database in the prompt

2016-07-01 Thread Peter Vary 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49498/
---

(Updated July 1, 2016, 3:08 p.m.)


Review request for hive, Sergio Pena, Szehon Ho, and Vihang Karajgaonkar.


Bugs: HIVE-14123
https://issues.apache.org/jira/browse/HIVE-14123


Repository: hive-git


Description
---

There are several jira issues complaining that, the Beeline does not respect 
hive.cli.print.current.db.
This is partially true, since in embedded mode, it uses the 
hive.cli.print.current.db to change the prompt, since HIVE-10511.
In remote mode, I think this function should use a beeline command line option 
instead, like for the showHeader option emphasizing, that this is a client side 
option.

The patch contains:
- New configuration option
- Changing the help text
- Updating command hooks, to run in remote mode as well
- Adding new hooks, for connect and go sqllite commands
- Generalize database connection refresh command
- Changing prompt


Diffs
-

  beeline/src/java/org/apache/hive/beeline/BeeLine.java 66185f6 
  beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java 5aaa385 
  beeline/src/java/org/apache/hive/beeline/ClientCommandHookFactory.java 
c4d97bc 
  beeline/src/java/org/apache/hive/beeline/Commands.java 3a204c0 
  beeline/src/java/org/apache/hive/beeline/ReflectiveCommandHandler.java 
3b863ae 
  beeline/src/main/resources/BeeLine.properties 7500df9 
  beeline/src/test/org/apache/hive/beeline/TestBeelineArgParsing.java ce1f538 
  beeline/src/test/org/apache/hive/beeline/TestClientCommandHookFactory.java 
c86de0a 

Diff: https://reviews.apache.org/r/49498/diff/


Testing
---

Unit tests for the hooks, and the configuration option

Manual test in remote, and embedded mode


Thanks,

Peter Vary

[jira] [Created] (HIVE-14146) Column comments with "\n" character "corrupts" table metadata

2016-07-01 Thread Peter Vary (JIRA) 
Peter Vary created HIVE-14146:
-

 Summary: Column comments with "\n" character "corrupts" table 
metadata
 Key: HIVE-14146
 URL: https://issues.apache.org/jira/browse/HIVE-14146
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Affects Versions: 2.2.0
Reporter: Peter Vary
Assignee: Peter Vary


Create a table with the following(noting the \n in the COMMENT):
{noformat}
CREATE TABLE commtest(first_nm string COMMENT 'Indicates First name\nof an 
individual’);
{noformat}

Describe shows that now the metadata is messed up:
{noformat}
beeline> describe commtest;
+---++---+--+
| col_name  | data_type  |comment|
+---++---+--+
| first_nm | string   | Indicates First name  |
| of an individual  | NULL   | NULL  |
+---++---+--+
{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14145) Too small length of column 'PARAM_VALUE' in table 'SERDE_PARAMS'

2016-07-01 Thread Oleksiy Sayankin (JIRA) 
Oleksiy Sayankin created HIVE-14145:
---

 Summary: Too small length of column 'PARAM_VALUE' in table 
'SERDE_PARAMS'
 Key: HIVE-14145
 URL: https://issues.apache.org/jira/browse/HIVE-14145
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Reporter: Oleksiy Sayankin
Assignee: Oleksiy Sayankin


Customer has following table

{code}
create external table hive_hbase_test(
HBASE_KEY string,
ENTITY_NAME string,
ENTITY_ID string,
CLAIM_HEADER_ID string,
CLAIM_LINE_ID string,
MEDICAL_CLAIM_SOURCE_SYSTEM string,
UNIQUE_MEMBER_ID string,
MEMBER_SOURCE_SYSTEM string,
SUBSCRIBER_ID string,
COVERAGE_CLASS_CODE string,
SERVICING_PROVIDER_ID string,
PROVIDER_SOURCE_SYSTEM string,
SERVICING_PROVIDER_SPECIALTY string,
SERVICING_STANDARD_PROVIDER_SPECIALTY string,
SERVICING_PROVIDER_TYPE_CODE string,
REFERRING_PROVIDER_ID string,
ADMITTING_PROVIDER_ID string,
ATTENDING_PROVIDER_ID string,
OPERATING_PROVIDER_ID string,
BILLING_PROVIDER_ID string,
ORDERING_PROVIDER_ID string,
HEALTH_PLAN_SOURCE_ID string,
HEALTH_PLAN_PAYER_NAME string,
BUSINESS_UNIT string,
OPERATING_UNIT string,
PRODUCT string,
MARKET string,
DEPARTMENT string,
IPA string,
SUPPLEMENTAL_DATA_TYPE string,
PSEUDO_CLAIM_FLAG string,
CLAIM_STATUS string,
CLAIM_LINE_STATUS string,
CLAIM_DENIED_FLAG string,
SERVICE_LINE_DENIED_FLAG string,
DENIED_REASON_CODE string,
SERVICE_LINE_DENIED_REASON_CODE string,
DAYS_DENIED int,
DIAGNOSIS_DATE timestamp,
SERVICE_DATE TIMESTAMP,
SERVICE_FROM_DATE TIMESTAMP,
SERVICE_TO_DATE TIMESTAMP,
ADMIT_DATE TIMESTAMP,
ADMIT_TYPE string,
ADMIT_SOURCE_TYPE string,
DISCHARGE_DATE TIMESTAMP,
DISCHARGE_STATUS_CODE string,
SERVICE_LINE_TYPE_OF_SERVICE string,
TYPE_OF_BILL_CODE string,
INPATIENT_FLAG string,
PLACE_OF_SERVICE_CODE string,
FACILITY_CODE string,
AUTHORIZATION_NUMBER string,
CLAIM_REFERRAL_NUMBER string,
CLAIM_TYPE string,
CLAIM_ADJUSTMENT_TYPE string,
ICD_DIAGNOSIS_CODE_1 string,
PRESENT_ON_ADMISSION_FLAG_1 string,
ICD_DIAGNOSIS_CODE_2 string,
PRESENT_ON_ADMISSION_FLAG_2 string,
ICD_DIAGNOSIS_CODE_3 string,
PRESENT_ON_ADMISSION_FLAG_3 string,
ICD_DIAGNOSIS_CODE_4 string,
PRESENT_ON_ADMISSION_FLAG_4 string,
ICD_DIAGNOSIS_CODE_5 string,
PRESENT_ON_ADMISSION_FLAG_5 string,
ICD_DIAGNOSIS_CODE_6 string,
PRESENT_ON_ADMISSION_FLAG_6 string,
ICD_DIAGNOSIS_CODE_7 string,
PRESENT_ON_ADMISSION_FLAG_7 string,
ICD_DIAGNOSIS_CODE_8 string,
PRESENT_ON_ADMISSION_FLAG_8 string,
ICD_DIAGNOSIS_CODE_9 string,
PRESENT_ON_ADMISSION_FLAG_9 string,
ICD_DIAGNOSIS_CODE_10 string,
PRESENT_ON_ADMISSION_FLAG_10 string,
ICD_DIAGNOSIS_CODE_11 string,
PRESENT_ON_ADMISSION_FLAG_11 string,
ICD_DIAGNOSIS_CODE_12 string,
PRESENT_ON_ADMISSION_FLAG_12 string,
ICD_DIAGNOSIS_CODE_13 string,
PRESENT_ON_ADMISSION_FLAG_13 string,
ICD_DIAGNOSIS_CODE_14 string,
PRESENT_ON_ADMISSION_FLAG_14 string,
ICD_DIAGNOSIS_CODE_15 string,
PRESENT_ON_ADMISSION_FLAG_15 string,
ICD_DIAGNOSIS_CODE_16 string,
PRESENT_ON_ADMISSION_FLAG_16 string,
ICD_DIAGNOSIS_CODE_17 string,
PRESENT_ON_ADMISSION_FLAG_17 string,
ICD_DIAGNOSIS_CODE_18 string,
PRESENT_ON_ADMISSION_FLAG_18 string,
ICD_DIAGNOSIS_CODE_19 string,
PRESENT_ON_ADMISSION_FLAG_19 string,
ICD_DIAGNOSIS_CODE_20 string,
PRESENT_ON_ADMISSION_FLAG_20 string,
ICD_DIAGNOSIS_CODE_21 string,
PRESENT_ON_ADMISSION_FLAG_21 string,
ICD_DIAGNOSIS_CODE_22 string,
PRESENT_ON_ADMISSION_FLAG_22 string,
ICD_DIAGNOSIS_CODE_23 string,
PRESENT_ON_ADMISSION_FLAG_23 string,
ICD_DIAGNOSIS_CODE_24 string,
PRESENT_ON_ADMISSION_FLAG_24 string,
ICD_DIAGNOSIS_CODE_25 string,
PRESENT_ON_ADMISSION_FLAG_25 string,
QUANTITY_OF_SERVICES decimal(10,2),
REVENUE_CODE string,
PROCEDURE_CODE string,
PROCEDURE_CODE_MODIFIER_1 string,
PROCEDURE_CODE_MODIFIER_2 string,
PROCEDURE_CODE_MODIFIER_3 string,
PROCEDURE_CODE_MODIFIER_4 string,
ICD_VERSION_CODE_TYPE string,
ICD_PROCEDURE_CODE_1 string,
ICD_PROCEDURE_CODE_2 string,
ICD_PROCEDURE_CODE_3 string,
ICD_PROCEDURE_CODE_4 string,
ICD_PROCEDURE_CODE_5 string,
ICD_PROCEDURE_CODE_6 string,
ICD_PROCEDURE_CODE_7 string,
ICD_PROCEDURE_CODE_8 string,
ICD_PROCEDURE_CODE_9 string,
ICD_PROCEDURE_CODE_10 string,
ICD_PROCEDURE_CODE_11 string,
ICD_PROCEDURE_CODE_12 string,
ICD_PROCEDURE_CODE_13 string,
ICD_PROCEDURE_CODE_14 string,
ICD_PROCEDURE_CODE_15 string,
ICD_PROCEDURE_CODE_16 string,
ICD_PROCEDURE_CODE_17 string,
ICD_PROCEDURE_CODE_18 string,
ICD_PROCEDURE_CODE_19 string,
ICD_PROCEDURE_CODE_20 string,
ICD_PROCEDURE_CODE_21 string,
ICD_PROCEDURE_CODE_22 string,
ICD_PROCEDURE_CODE_23 string,
ICD_PROCEDURE_CODE_24 string,
ICD_PROCEDURE_CODE_25 string,
E_CODE_1 string,
E_CODE_TYPE_1 string,
E_CODE_2 string,
E_CODE_TYPE_2 string,
E_CODE_3 string,
E_CODE_TYPE_3 string,
EMERGENCY_FLAG string,
HOSPITAL_RELATED_FLAG string,
OUTSIDE_LABS_FLAG string,
PPS_CODE string,
NATIONAL_DRUG_CODE string,
VALUE_AMOUNT decimal(10,2),
CAPITATED_SERVICE_FLAG string,
NETWORK_STATUS_FLAG string,
ADJUDICATED_DATE

Review Request 49494: HIVE-14144: Permanent functions are showing up in show functions, but describe says it doesn't exist

2016-07-01 Thread Rajat Khandelwal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49494/
---

Review request for hive.


Bugs: HIVE-14144
https://issues.apache.org/jira/browse/HIVE-14144


Repository: hive-git


Description
---


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java 
a4584e3eaebc052a6870c543bfbfc9e5d78e771b 
  ql/src/java/org/apache/hadoop/hive/ql/util/ResourceDownloader.java 
14102b5108164616b4cf2e7f7d22c894321f027e 

Diff: https://reviews.apache.org/r/49494/diff/


Testing
---


Thanks,

Rajat Khandelwal

[jira] [Created] (HIVE-14144) Permanent functions are showing up in show functions, but describe says it doesn't exist

2016-07-01 Thread Rajat Khandelwal (JIRA) 
Rajat Khandelwal created HIVE-14144:
---

 Summary: Permanent functions are showing up in show functions, but 
describe says it doesn't exist
 Key: HIVE-14144
 URL: https://issues.apache.org/jira/browse/HIVE-14144
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.1.0
Reporter: Rajat Khandelwal
Assignee: Rajat Khandelwal






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 48886: HIVE-14052: Cleanup of structures required when LLAP access from external clients completes

2016-07-01 Thread Siddharth Seth 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48886/#review140309
---



This is getting rather complicated. Need to think through a simpler approach to 
clean up this data. queryComplete does the following
- ObjectCacheFactory.removeLlapQueryCache(savedQueryId);
- QueryInfo object
- Directories on local disk

One possibility (which is similar but can remove the requirement of the Delayed 
cleanup - assuming this is mainly because of the directory deletion ?)
I think it's possible to have the local-dirs created on a per fragment for 
external requests - since Shuffle is not involved (which is what requires all 
fragments for a query to be under the application dir). This cleanup could be 
delinked from queryComplete.
Cleaning up the in-memory structures could be handled immediately after the 
socket is closed - obtain lock, try cleanup (block submissions or block on new 
submissions).


llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
(line 154)


Nothing ever put into this structure.



llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
(line 158)


There's a window between moving the state to ACTIVE and taking the lock in 
queryCleanup where there's a race between queryComplete and registerFragment 
obtaining the lock - and will lead to the exception.



llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
(line 218)


There's a good chance that this call does nothing - since fragmentComplete 
would already have been invoked by this point.
The Closeable generated earlier holds a reference to QueryFragmentInfo 
which has otherwise been removed.
Think the Closeable generated will get cleaned up when the Socket closes.
Don't think we'll end up accumulating QueryFragmentInfo objects, correct?



llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
(line 289)


Would prefer avoiding IO within a lock. Now this lock is for a single query 
only, and will be held at a time when new fragments are unlikely to show up - 
it is still possible for fragments to show up though.
These fragments would go into a tight loop in this case - waiting to get 
this lock since it's been invalidated by this point.



llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
(line 352)


Why the null check ? This can go into a tight loop.



llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
(line 565)


This belongs inside QueryInfo rather than at the QueryTracker.

Unrelated: Need to move all of these to an interface so that there's good 
sepeartion between methods intended for use by other parts of the system and 
internal methods. Will create a jira and assign to myself.



ql/src/java/org/apache/hadoop/hive/llap/LlapOutputFormatService.java (line 211)


When is the Writer created ? - once the fragment starts executing, or when 
a read request is received from the client.

What I understand from reading the code here is that it's created when a 
read request is received. If that's the case - there's no guarantees that we'll 
actually end up cleaning the structures that were created for this query.



ql/src/java/org/apache/hadoop/hive/llap/LlapOutputFormatService.java (line 289)


Can this be added into the FragmentCompletionHandler interface ?



ql/src/java/org/apache/hadoop/hive/llap/LlapOutputFormatService.java (line 290)


Does it make sense to return a FragmentCompletionHandler rather than a 
Closeable ?



ql/src/test/org/apache/hadoop/hive/llap/TestLlapOutputFormat.java (line 52)


Needs tests.


ObjectCacheFactory.removeLlapQueryCache(savedQueryId);

- Siddharth Seth


On June 17, 2016, 10:31 p.m., Jason Dere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48886/
> ---
> 
> (Updated June 17, 2016, 10:31 p.m.)
> 
> 
> Review request for hive and Siddharth Seth.
> 
> 
> Bugs: HIVE-14052
> https://issues.apache.org/jira/browse/HIVE-14052
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Add a hook to call run QueryTracker.queryComplete if there