[jira] [Created] (HIVE-14733) support in HBaseStore

2016-09-09 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-14733:
---

 Summary: support in HBaseStore
 Key: HIVE-14733
 URL: https://issues.apache.org/jira/browse/HIVE-14733
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin


For expediency, HBaseStore support will be done later once everything works on 
ObjectStore



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14732) support heartbeats for writeIds

2016-09-09 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-14732:
---

 Summary: support heartbeats for writeIds
 Key: HIVE-14732
 URL: https://issues.apache.org/jira/browse/HIVE-14732
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin


Heartbeat and expiration logic is there (or rather on the way), but 
heartbeating code itself isn't. Right now, writeIds only expire after a long 
absolute timeout.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)

2016-09-09 Thread Zhiyuan Yang (JIRA) 
Zhiyuan Yang created HIVE-14731:
---

 Summary: Use Tez cartesian product edge in Hive (unpartitioned 
case only)
 Key: HIVE-14731
 URL: https://issues.apache.org/jira/browse/HIVE-14731
 Project: Hive
  Issue Type: Bug
Reporter: Zhiyuan Yang
Assignee: Zhiyuan Yang


Given cartesian product edge is available in Tez now (see TEZ-3230), let's 
integrate it into Hive on Tez. This allows us to have more than one reducer in 
cross product queries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14730) split up or refactor ObjectStore

2016-09-09 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-14730:
---

 Summary: split up or refactor ObjectStore
 Key: HIVE-14730
 URL: https://issues.apache.org/jira/browse/HIVE-14730
 Project: Hive
  Issue Type: Wish
Reporter: Sergey Shelukhin


This is a wishlist task. 
One of these days, after I have to kill -9 eclipse again because ObjectStore is 
so damn huge, I may do it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14729) analyze table compute stats for columns should collect basic stats as well

2016-09-09 Thread Ashutosh Chauhan (JIRA) 
Ashutosh Chauhan created HIVE-14729:
---

 Summary: analyze table compute stats for columns should collect 
basic stats as well
 Key: HIVE-14729
 URL: https://issues.apache.org/jira/browse/HIVE-14729
 Project: Hive
  Issue Type: Improvement
  Components: Statistics
Reporter: Ashutosh Chauhan


Currently user need to run following two statements to collect stats:
{code}
analyze table T compute statistics;
analyze table T compute statistics for columns;
{code}

Its little confusing for users that they need to collect both stats. Since 
basic stats are cheap to compute, we should collect them even in case of column 
stats, so that by running single statement users can get all stats. This leaves 
with the possibility of fast basic stats collection with first statement which 
will be good to have.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

RE: Remote debugging of HIVE code from Intellij

2016-09-09 Thread Ranjan Banerjee 
Thanks for the pointers Peter. Appreciate them:)

Ranjan

-Original Message-
From: Peter Vary [mailto:pv...@cloudera.com] 
Sent: Friday, September 9, 2016 3:10 AM
To: dev@hive.apache.org
Cc: dev-h...@hive.apache.org
Subject: Re: Remote debugging of HIVE code from Intellij

Hi Ranjan,

You can try the following query in the apache jira 
(https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fissues.apache.org%2fjira%2fbrowse=02%7c01%7crabaner%40microsoft.com%7c08572092bdd6415ff12808d3d899787f%7c72f988bf86f141af91ab2d7cd011db47%7c1%7c0%7c636090126071480480=kwMjmd8QIAr9J1bS8RcK0cMiCkZzTK49vXsD3OYypVs%3d
 
):

project = HIVE AND status in (Open, "In Progress", Reopened) AND labels in 
(newbie, "newbie++", newbiee, newbiew, newdev) AND assignee in (EMPTY)

Not sure that all of these are really easy ones, but you could check some of 
these, so you could test yourself.
I am fairly new too, but happy to help :)

Peter

> On Sep 9, 2016, at 12:07 AM, Ranjan Banerjee  wrote:
> 
> Hi Peter,
>Thank you so much for pointing it out. I ran -Dtest=TestCliDriver and it 
> worked! 
> On a sidenote I would like to contribute to the project. Is there any 
> issue you would suggest that I could look into.
> 
> Thanks
> Ranjan
> 
> -Original Message-
> From: Peter Vary [mailto:pv...@cloudera.com]
> Sent: Thursday, September 8, 2016 6:26 AM
> To: dev@hive.apache.org
> Cc: dev-h...@hive.apache.org
> Subject: Re: Remote debugging of HIVE code from Intellij
> 
> Hi Ranjan,
> 
> I am not sure about what you are about to test, but I think 
> TestPositiveCliDriver is not a valid test name, so not test run in your case.
> 
> Regards,
> Peter
> 
>> On Sep 8, 2016, at 4:08 AM, Ranjan Banerjee  wrote:
>> 
>> Hi,
>>  I am trying to remote debug HIVE code from Intellij. Here are my steps:
>> 1)In HIVE folder: mvn clean install -Dskiptests  2)cd itests/qtest 
>> 3)mvn clean install -Dskiptests  4)mvn -Dmaven.surefire.debug 
>> -Dtest=TestPositiveCliDriver -Dqfile=acid_bucket_pruning.q  I have 
>> setup the remote debugging config in Intellij and have put breakpoint 
>> in sessionstate.java  However I see that the test completes successfully and 
>> does not listen at the 5005 port. Can someone give some pointers as to why 
>> the mvn test is not stopping to listen at this port?
>> 
>> Thanks
>> Ranjan
>

Re: Review Request 51593: HIVE-14063 : Add beeline connection configuration file to automatically connect to HS2 using beeline

2016-09-09 Thread Peter Vary 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51593/#review148335
---


Ship it!




LGTM (non binding)

- Peter Vary


On Sept. 7, 2016, 6:22 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51593/
> ---
> 
> (Updated Sept. 7, 2016, 6:22 p.m.)
> 
> 
> Review request for hive, Mohit Sabharwal, Sergio Pena, and Szehon Ho.
> 
> 
> Bugs: HIVE-14063
> https://issues.apache.org/jira/browse/HIVE-14063
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This change adds a new optional configuration file for Beeline. If this file 
> is present at predefined locations, Beeline will attempt to create the 
> connection url using the hive-site.xml found in classpath and another 
> user-specific configuration file. Beeline then connects automatically using 
> the url generated based on these configuration files. The main objective of 
> the change is to provide user another way to connect to the HiveServer2 
> without providing the connection url everytime. The configuration file uses 
> hadoop xml format so that we can support encryption/obfuscation using hadoop 
> credential manager API in the future.
> 
> Properties in the user-specific configuration file override the properties 
> derived from hive-site.xml.
> 
> Tested using newly added unit tests and itests in various Hiveserver2 
> configurations.
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLine.java 
> 8e65e3987398531cce5c65c383762cf49a52c578 
>   beeline/src/java/org/apache/hive/beeline/Commands.java 
> 2f3ec134098dfa3767bab9545438d1f38f11697c 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/BeelineHS2ConnectionFileParseException.java
>  PRE-CREATION 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/HS2ConnectionFileParser.java
>  PRE-CREATION 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/HS2ConnectionFileUtils.java
>  PRE-CREATION 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/HiveSiteHS2ConnectionFileParser.java
>  PRE-CREATION 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/UserHS2ConnectionFileParser.java
>  PRE-CREATION 
>   
> beeline/src/test/org/apache/hive/beeline/hs2connection/TestUserHS2ConnectionFileParser.java
>  PRE-CREATION 
>   beeline/src/test/resources/hive-site.xml 
> 5f310d68245275ac9dc24df45579784019eea332 
>   beeline/src/test/resources/test-hs2-conn-conf-kerberos-http.xml 
> PRE-CREATION 
>   beeline/src/test/resources/test-hs2-conn-conf-kerberos-nossl.xml 
> PRE-CREATION 
>   beeline/src/test/resources/test-hs2-conn-conf-kerberos-ssl.xml PRE-CREATION 
>   beeline/src/test/resources/test-hs2-connection-conf-list.xml PRE-CREATION 
>   beeline/src/test/resources/test-hs2-connection-config-noauth.xml 
> PRE-CREATION 
>   beeline/src/test/resources/test-hs2-connection-multi-conf-list.xml 
> PRE-CREATION 
>   beeline/src/test/resources/test-hs2-connection-zookeeper-config.xml 
> PRE-CREATION 
>   
> itests/hive-unit/src/test/java/org/apache/hive/beeline/hs2connection/TestBeelineConnectionUsingHiveSite.java
>  PRE-CREATION 
>   
> itests/hive-unit/src/test/java/org/apache/hive/beeline/hs2connection/TestBeelineWithHS2ConnectionFile.java
>  PRE-CREATION 
>   
> itests/hive-unit/src/test/java/org/apache/hive/beeline/hs2connection/TestBeelineWithUserHs2ConnectionFile.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51593/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vihang Karajgaonkar
> 
>

[jira] [Created] (HIVE-14728) Redundant orig files

2016-09-09 Thread Rui Li (JIRA) 
Rui Li created HIVE-14728:
-

 Summary: Redundant orig files
 Key: HIVE-14728
 URL: https://issues.apache.org/jira/browse/HIVE-14728
 Project: Hive
  Issue Type: Bug
Reporter: Rui Li
Priority: Minor


I find some orig files in master, e.g. SemanticAnalyzer.java.orig. Wondering if 
they are added by mistake?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Remote debugging of HIVE code from Intellij

2016-09-09 Thread Peter Vary 
Hi Ranjan,

You can try the following query in the apache jira 
(https://issues.apache.org/jira/browse ):

project = HIVE AND status in (Open, "In Progress", Reopened) AND labels in 
(newbie, "newbie++", newbiee, newbiew, newdev) AND assignee in (EMPTY)

Not sure that all of these are really easy ones, but you could check some of 
these, so you could test yourself.
I am fairly new too, but happy to help :)

Peter

> On Sep 9, 2016, at 12:07 AM, Ranjan Banerjee  wrote:
> 
> Hi Peter,
>Thank you so much for pointing it out. I ran -Dtest=TestCliDriver and it 
> worked! 
> On a sidenote I would like to contribute to the project. Is there any 
> issue you would suggest that I could look into.
> 
> Thanks
> Ranjan
> 
> -Original Message-
> From: Peter Vary [mailto:pv...@cloudera.com] 
> Sent: Thursday, September 8, 2016 6:26 AM
> To: dev@hive.apache.org
> Cc: dev-h...@hive.apache.org
> Subject: Re: Remote debugging of HIVE code from Intellij
> 
> Hi Ranjan,
> 
> I am not sure about what you are about to test, but I think 
> TestPositiveCliDriver is not a valid test name, so not test run in your case.
> 
> Regards,
> Peter
> 
>> On Sep 8, 2016, at 4:08 AM, Ranjan Banerjee  wrote:
>> 
>> Hi,
>>  I am trying to remote debug HIVE code from Intellij. Here are my steps:
>> 1)In HIVE folder: mvn clean install -Dskiptests  2)cd itests/qtest  
>> 3)mvn clean install -Dskiptests
>>  4)mvn -Dmaven.surefire.debug -Dtest=TestPositiveCliDriver 
>> -Dqfile=acid_bucket_pruning.q
>>  I have setup the remote debugging config in Intellij and have put 
>> breakpoint in sessionstate.java
>>  However I see that the test completes successfully and does not listen at 
>> the 5005 port. Can someone give some pointers as to why the mvn test is not 
>> stopping to listen at this port?
>> 
>> Thanks
>> Ranjan
>

Re: Review Request 51468: HIVE-14532 - enable qtests from ide - eclipse

2016-09-09 Thread Peter Vary 


> On Aug. 28, 2016, 3:07 p.m., Peter Vary wrote:
> > itests/util/src/main/java/org/apache/hadoop/hive/cli/control/AbstractCliConfig.java,
> >  lines 80-84
> > 
> >
> > It is mostly just a question, which appeared to me in the morning, 
> > waking up:
> > - There are several other places which are using QTestUtil's query 
> > running framework - do any of them uses these properties? If so, then it 
> > might be good idea to put these defaults to the QTestUtil constructor, so 
> > these initialized there as well, and could be run from browser too.
> > 
> > I could see several valid answers why you did not do this, like:
> > - These are only used by qtest queries
> > - We concentrate on the qtests now, and do not now enough of the other 
> > test to do this change
> > 
> > This morning ideas are sometimes very good, sometimes not so much, just 
> > "stored" here to check when realy aweaken :)
> > 
> > Another one of these ideas:
> > The next develeper, who adds a new variable, he might forget to add it 
> > here, so the test will only work with maven, and not with eclipse, or idea. 
> > I think we should add a comment to the pom.xml, to remaind him to add a 
> > default value here as well. For example after this comment:
> > 
> > 
> > 
> > ${basedir}/${hive.path.to.root}/data/files
> > 
> > What do you think?
> 
> Zoltan Haindrich wrote:
> i think most of these extra properties should be removed later and/or use 
> some convention instead of setting them(warehousedir/tmpdir)
> 
> test.data.files - is a property which has no real alternative 
> value...that directory contains so many specific files that you can't really 
> change it...

My bad again. My comment was not clear enough :(

My second comment was not about the concrete property from the pom.xml, it was 
more about the place where every QTestUtil related property is set. If I have 
provided longer sniplet it might be more clear what I wanted to say:



${basedir}/${hive.path.to.root}/data/files

${basedir}/${hive.path.to.root}/data/files
${test.tmp.dir}
${test.tmp.dir.uri}
${test.dfs.mkdir}

${test.output.overwrite}

${test.warehouse.scheme}${test.warehouse.dir}
true

If we remind the developer in the comment, that the properties should be set in 
AbstractCliConfig as well; if a value is changed, or new value is added, then 
it is less likely that we end up with new tests which could be run with maven, 
but could not be run in IDE. It would be something like this:



Of course the patch is great without this too, just wanted to correct my 
communication error.

Thanks for your patience,
Peter


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51468/#review147100
---


On Sept. 7, 2016, 8:39 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51468/
> ---
> 
> (Updated Sept. 7, 2016, 8:39 p.m.)
> 
> 
> Review request for hive, Balint Molnar, Lefty Leverenz, and Prasanth_J.
> 
> 
> Bugs: HIVE-14532
> https://issues.apache.org/jira/browse/HIVE-14532
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> how to execute qtests for ide wikipage draft:
> v1: http://hastebin.com/paxicutive.vhdl
> 
> the patch itself contains:
> 
> * some automatic property settings to configure qtest related things to be 
> able to execute
> * maven profile to avoid shading plugin invocation during IDE project 
> generation - to void classpath/compile errors i've encountered
> * some test.src.tables related changes - defaulting is now done at a 
> different place
> * eclipse is ok for me
> 
> patch#2 here is patch#3 in the jira
> 
> 
> Diffs
> -
> 
>   itests/hive-jmh/pom.xml f1417fd7e8b70a7a3dd255c25d3909013f02df67 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/AbstractCliConfig.java
>  efbd4657f22e856b9c9ba5f74472ad5fd9f9a5b5 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 69c4974105c6b47cc8e8051cd980274c9c381775 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 4d4a929c159c61f9f4af3238d4b7baff146d346e 
>   jdbc/pom.xml b29739b3f8577c6e363b5c8ee39b63e53a17c907 
>   pom.xml 4c41200ffc8e2c9a1d207f8676f252b94e80e4fd 
>   ql/pom.xml 02ddb805a228ed23694c8a81953dd2400d7308c6 
> 
> Diff: https://reviews.apache.org/r/51468/diff/
> 
> 
> Testing
> ---
> 
> I've tested the draft using eclipse:
> 3.0 -> 3.1 ->

[jira] [Created] (HIVE-14727) llap-server may case file descriptor leak in BuddyAllocator class

2016-09-09 Thread Yechao Chen (JIRA) 
Yechao Chen created HIVE-14727:
--

 Summary: llap-server may case file descriptor leak in 
BuddyAllocator class
 Key: HIVE-14727
 URL: https://issues.apache.org/jira/browse/HIVE-14727
 Project: Hive
  Issue Type: Bug
  Components: llap
Affects Versions: 2.2.0
Reporter: Yechao Chen
Assignee: Yechao Chen


llap-server,the method preallocate(int) of  BuddyAllocator may case file 
descriptor leak when FileChannel map allocate memory error.

the code:
//here if failed
 ByteBuffer rwbuf = rwf.getChannel().map(MapMode.READ_WRITE, 0, 
arenaSize);
// A mapping, once established, is not dependent upon the file channel 
that was used to
// create it. delete file and hold onto the map
   //can not close() and delete file
rwf.close();
rf.delete();




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)