Re: [ANNOUNCE] New PMC Member : John Pullokkaran

[Navis Ryu]

Congratulations!

2015년 11월 25일 수요일, Hari Sivarama Subramaniyan님이
작성한 메시지:

> Congrats, John!
> --
> *From:* Eugene Koifman  >
> *Sent:* Tuesday, November 24, 2015 3:14 PM
> *To:* u...@hive.apache.org
> ;
> dev@hive.apache.org 
> *Subject:* Re: [ANNOUNCE] New PMC Member : John Pullokkaran
>
> Congrats!
>
> From: Sergey Shelukhin  >
> Reply-To: "u...@hive.apache.org
> " <
> u...@hive.apache.org
> >
> Date: Tuesday, November 24, 2015 at 3:13 PM
> To: "u...@hive.apache.org
> " <
> u...@hive.apache.org
> >, "
> dev@hive.apache.org "
> 
> >
> Subject: Re: [ANNOUNCE] New PMC Member : John Pullokkaran
>
> Congrats!
>
> From: Jimmy Xiang  >
> Reply-To: "u...@hive.apache.org
> " <
> u...@hive.apache.org
> >
> Date: Tuesday, November 24, 2015 at 15:07
> To: "dev@hive.apache.org
> " <
> dev@hive.apache.org >
> Cc: "u...@hive.apache.org
> " <
> u...@hive.apache.org
> >
> Subject: Re: [ANNOUNCE] New PMC Member : John Pullokkaran
>
> Congrats!!
>
> On Tue, Nov 24, 2015 at 3:04 PM, Szehon Ho  > wrote:
>
>> Congratulations!
>>
>> On Tue, Nov 24, 2015 at 3:02 PM, Xuefu Zhang > > wrote:
>>
>> > Congratulations, John!
>> >
>> > --Xuefu
>> >
>> > On Tue, Nov 24, 2015 at 3:01 PM, Prasanth J > >
>> > wrote:
>> >
>> >> Congratulations and Welcome John!
>> >>
>> >> Thanks
>> >> Prasanth
>> >>
>> >> On Nov 24, 2015, at 4:59 PM, Ashutosh Chauhan > >
>> >> wrote:
>> >>
>> >> On behalf of the Hive PMC I am delighted to announce John Pullokkaran
>> is
>> >> joining Hive PMC.
>> >> John is a long time contributor in Hive and is focusing on compiler and
>> >> optimizer areas these days.
>> >> Please give John a warm welcome to the project!
>> >>
>> >> Ashutosh
>> >>
>> >>
>> >>
>> >
>>
>
>

Re: [ANNOUNCE] New Hive Committer- Aihua Xu

[Navis Ryu]

Congratulations!

2015-10-22 16:48 GMT+09:00 Chinna Rao Lalam :
> Congrats Aihua!
>
> On Thu, Oct 22, 2015 at 10:45 AM, Vaibhav Gumashta <
> vgumas...@hortonworks.com> wrote:
>
>> Congrats Aihua!
>>
>> ‹Vaibhav
>>
>> On 10/21/15, 4:42 PM, "Pengcheng Xiong"  wrote:
>>
>> >Congrats Aihua!
>> >
>> >On Wed, Oct 21, 2015 at 2:09 PM, Szehon Ho  wrote:
>> >
>> >> The Apache Hive PMC has voted to make Aihua Xu a committer on the Apache
>> >> Hive Project.
>> >>
>> >> Please join me in congratulating Aihua!
>> >>
>> >> Thanks,
>> >> Szehon
>> >>
>>
>>
>
>
> --
> Hope It Helps,
> Chinna

Re: [ANNOUNCE] New Hive Committer - Siddharth Seth

[Navis Ryu]

Congratulations!

2015-10-22 16:49 GMT+09:00 Chinna Rao Lalam :
> Congratulations!
>
> On Thu, Oct 22, 2015 at 12:18 PM, Santlal J Gupta <
> santlal.gu...@bitwiseglobal.com> wrote:
>
>> Congratulations !!
>>
>> -Original Message-
>> From: Chetna C [mailto:chetna@gmail.com]
>> Sent: Thursday, October 22, 2015 8:58 AM
>> To: dev@hive.apache.org
>> Cc: Siddharth Seth
>> Subject: Re: [ANNOUNCE] New Hive Committer - Siddharth Seth
>>
>> Congratulations !!
>> On Oct 22, 2015 5:13 AM, "Pengcheng Xiong"  wrote:
>>
>> > Congrats Sid!
>> >
>> > On Wed, Oct 21, 2015 at 2:14 PM, Sergey Shelukhin
>> > 
>> > wrote:
>> >
>> > > The Apache Hive PMC has voted to make Siddharth Seth a committer on
>> > > the Apache Hive Project.
>> > >
>> > > Please join me in congratulating Sid!
>> > >
>> > > Thanks,
>> > > Sergey.
>> > >
>> > >
>> >
>>
>
>
>
> --
> Hope It Helps,
> Chinna

Re: Review Request 38292: HIVE-11768 java.io.DeleteOnExitHook leaks memory on long running Hive Server2 Instances

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38292/
---

(Updated 10 15, 2015, 12:35 오전)


Review request for hive.


Bugs: HIVE-11768
https://issues.apache.org/jira/browse/HIVE-11768


Repository: hive-git


Description
---

More than 490,000 paths was added to java.io.DeleteOnExitHook on one of our 
long running HiveServer2 instances,taken up more than 100MB on heap.
  Most of the paths contains a suffix of ".pipeout".


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/FileUtils.java 7e4f386 
  common/src/java/org/apache/hive/common/util/ShutdownHookManager.java fd2f20a 
  common/src/test/org/apache/hive/common/util/TestShutdownHookManager.java 
fa30f15 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 92ac209 
  
service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java
 1d1e995 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
175348b 

Diff: https://reviews.apache.org/r/38292/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 38292: HIVE-11768 java.io.DeleteOnExitHook leaks memory on long running Hive Server2 Instances

[Navis Ryu]

> On 10 13, 2015, 9:35 오후, Thejas Nair wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java, line 309
> > <https://reviews.apache.org/r/38292/diff/2/?file=1096900#file1096900line309>
> >
> > I think this method could be re-used in other parts of hive as well. 
> > How about adding this a FileUtils.deleteTmpFile and a new 
> > FileUtils.createTmpFile method ?
> > Can you also add a unit test for those methods ?

Sure


> On 10 13, 2015, 9:35 오후, Thejas Nair wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java, line 871
> > <https://reviews.apache.org/r/38292/diff/2/?file=1096900#file1096900line871>
> >
> > same as above comment . A method in FileUtils for use throughout hive 
> > would be useful.

Consider it done


- Navis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38292/#review102537
-------


On 10 13, 2015, 1:19 오전, Navis Ryu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38292/
> ---
> 
> (Updated 10 13, 2015, 1:19 오전)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> More than 490,000 paths was added to java.io.DeleteOnExitHook on one of our 
> long running HiveServer2 instances,taken up more than 100MB on heap.
>   Most of the paths contains a suffix of ".pipeout".
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hive/common/util/ShutdownHookManager.java 
> fd2f20a 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 41b4bb1 
>   
> service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java
>  1d1e995 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 175348b 
> 
> Diff: https://reviews.apache.org/r/38292/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Navis Ryu
> 
>

Re: Review Request 38292: HIVE-11768 java.io.DeleteOnExitHook leaks memory on long running Hive Server2 Instances

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38292/
---

(Updated 10 14, 2015, 12:49 오전)


Review request for hive.


Changes
---

Addressed comments & fixed test fails


Repository: hive-git


Description
---

More than 490,000 paths was added to java.io.DeleteOnExitHook on one of our 
long running HiveServer2 instances,taken up more than 100MB on heap.
  Most of the paths contains a suffix of ".pipeout".


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/FileUtils.java 7e4f386 
  common/src/java/org/apache/hive/common/util/ShutdownHookManager.java fd2f20a 
  common/src/test/org/apache/hive/common/util/TestShutdownHookManager.java 
fa30f15 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 41b4bb1 
  
service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java
 1d1e995 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
175348b 

Diff: https://reviews.apache.org/r/38292/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 37347: HIVE-11518 Provide interface to adjust required resource for tez tasks

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37347/
---

(Updated 10 13, 2015, 1:48 오전)


Review request for hive.


Changes
---

Rebased to trunk


Bugs: HIVE-11518
https://issues.apache.org/jira/browse/HIVE-11518


Repository: hive-git


Description
---

Resource requirements for each tasks are varied but currently it's fixed to one 
value(via hive.tez.container.size). It would be good to customize resource 
requirements appropriate to expected work.

Suggested interface is quite simple.
{code}
public interface ResourceCalculator {

  Resource adjust(Resource resource, MapWork mapWork);

  Resource adjust(Resource resource, ReduceWork reduceWork);
}
{code}


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0fcd39b 
  common/src/java/org/apache/hive/common/util/ReflectionUtil.java 7059309 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java d250b82 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ResourceCalculator.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/SimpleCalculator.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/37347/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 38292: HIVE-11768 java.io.DeleteOnExitHook leaks memory on long running Hive Server2 Instances

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38292/
---

(Updated 10 13, 2015, 1:19 오전)


Review request for hive.


Repository: hive-git


Description
---

More than 490,000 paths was added to java.io.DeleteOnExitHook on one of our 
long running HiveServer2 instances,taken up more than 100MB on heap.
  Most of the paths contains a suffix of ".pipeout".


Diffs (updated)
-

  common/src/java/org/apache/hive/common/util/ShutdownHookManager.java fd2f20a 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 41b4bb1 
  
service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java
 1d1e995 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
175348b 

Diff: https://reviews.apache.org/r/38292/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 39253: SemanticAnalysis of "a=1" can result in a new Configuration() object

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39253/
---

Review request for hive.


Bugs: HIVE-11679
https://issues.apache.org/jira/browse/HIVE-11679


Repository: hive-git


Description
---

{code}
public static ExprNodeGenericFuncDesc newInstance(GenericUDF genericUDF,
  String funcText,
  List children) throws UDFArgumentException {
...

 if (genericUDF instanceof GenericUDFBaseCompare && children.size() == 2) {

  TypeInfo oiTypeInfo0 = children.get(0).getTypeInfo();
  TypeInfo oiTypeInfo1 = children.get(1).getTypeInfo();

  SessionState ss = SessionState.get();
  Configuration conf = (ss != null) ? ss.getConf() : new Configuration();
{code}

This is both a SessionState.get() which is a threadlocal lookup or worse, a  
new Configuration()  which means XML parsing of multiple files for each 
equality expression in the query.


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/JoinTypeCheckCtx.java
 dccd1d9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java fbe93f9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 61ee2bd 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 2d7d9d4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java e0cd398 
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java f95ee8d 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 4af07ad 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckCtx.java 8ad28be 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java ab5d006 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeColumnListDesc.java 
be78e73 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDesc.java 328bd86 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 
b5d2ddf 
  ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/MatchPath.java cc2b77b 
  ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/TableFunctionResolver.java 
71034d7 

Diff: https://reviews.apache.org/r/39253/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 39253: SemanticAnalysis of "a=1" can result in a new Configuration() object

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39253/
---

(Updated 10 13, 2015, 1:39 오전)


Review request for hive.


Bugs: HIVE-11679
https://issues.apache.org/jira/browse/HIVE-11679


Repository: hive-git


Description
---

{code}
public static ExprNodeGenericFuncDesc newInstance(GenericUDF genericUDF,
  String funcText,
  List children) throws UDFArgumentException {
...

 if (genericUDF instanceof GenericUDFBaseCompare && children.size() == 2) {

  TypeInfo oiTypeInfo0 = children.get(0).getTypeInfo();
  TypeInfo oiTypeInfo1 = children.get(1).getTypeInfo();

  SessionState ss = SessionState.get();
  Configuration conf = (ss != null) ? ss.getConf() : new Configuration();
{code}

This is both a SessionState.get() which is a threadlocal lookup or worse, a  
new Configuration()  which means XML parsing of multiple files for each 
equality expression in the query.


Diffs (updated)
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/JoinTypeCheckCtx.java
 dccd1d9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java fbe93f9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 61ee2bd 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 2d7d9d4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java e0cd398 
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java f95ee8d 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 4af07ad 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckCtx.java 8ad28be 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java ab5d006 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeColumnListDesc.java 
be78e73 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDesc.java 328bd86 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 
b5d2ddf 
  ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/MatchPath.java cc2b77b 
  ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/TableFunctionResolver.java 
71034d7 
  ql/src/test/results/clientnegative/compare_double_bigint.q.out 68a295a 
  ql/src/test/results/clientnegative/compare_string_bigint.q.out 9a604fa 

Diff: https://reviews.apache.org/r/39253/diff/


Testing
---


Thanks,

Navis Ryu

Re: [ANNOUNCE] New Hive PMC Chair - Ashutosh Chauhan

[Navis Ryu]

Congratulations!

2015-09-17 9:35 GMT+09:00 Xu, Cheng A :
> Congratulations, Ashutosh!
>
> -Original Message-
> From: Mohammad Islam [mailto:misla...@yahoo.com.INVALID]
> Sent: Thursday, September 17, 2015 8:23 AM
> To: u...@hive.apache.org; Hive
> Subject: Re: [ANNOUNCE] New Hive PMC Chair - Ashutosh Chauhan
>
> Congratulations Asutosh!
>
>
>  On Wednesday, September 16, 2015 4:51 PM, Bright Ling 
>  wrote:
>
>
>  #yiv7221259285 #yiv7221259285 -- _filtered #yiv7221259285 
> {font-family:SimSun;panose-1:2 1 6 0 3 1 1 1 1 1;} _filtered #yiv7221259285 
> {font-family:PMingLiU;panose-1:2 2 5 0 0 0 0 0 0 0;} _filtered #yiv7221259285 
> {panose-1:2 4 5 3 5 4 6 3 2 4;} _filtered #yiv7221259285 
> {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;} _filtered #yiv7221259285 
> {font-family:Tahoma;panose-1:2 11 6 4 3 5 4 4 2 4;} _filtered #yiv7221259285 
> {panose-1:2 2 5 0 0 0 0 0 0 0;} _filtered #yiv7221259285 {panose-1:2 1 6 0 3 
> 1 1 1 1 1;}#yiv7221259285 #yiv7221259285 p.yiv7221259285MsoNormal, 
> #yiv7221259285 li.yiv7221259285MsoNormal, #yiv7221259285 
> div.yiv7221259285MsoNormal 
> {margin:0cm;margin-bottom:.0001pt;font-size:12.0pt;}#yiv7221259285 a:link, 
> #yiv7221259285 span.yiv7221259285MsoHyperlink 
> {color:blue;text-decoration:underline;}#yiv7221259285 a:visited, 
> #yiv7221259285 span.yiv7221259285MsoHyperlinkFollowed 
> {color:purple;text-decoration:underline;}#yiv7221259285 
> p.yiv7221259285MsoAcetate, #yiv7221259285 li.yiv7221259285MsoAcetate, 
> #yiv7221259285 div.yiv7221259285MsoAcetate 
> {margin:0cm;margin-bottom:.0001pt;font-size:8.0pt;}#yiv7221259285 
> span.yiv7221259285EmailStyle17 {color:#1F497D;}#yiv7221259285 
> span.yiv7221259285BalloonTextChar {}#yiv7221259285 
> .yiv7221259285MsoChpDefault {font-size:10.0pt;} _filtered #yiv7221259285 
> {margin:72.0pt 72.0pt 72.0pt 72.0pt;}#yiv7221259285 
> div.yiv7221259285WordSection1 {}#yiv7221259285 Congratulations Asutosh!   
> From: Sathi Chowdhury [mailto:sathi.chowdh...@lithium.com]
> Sent: Thursday, 17 September 2015 8:04 AM
> To: u...@hive.apache.org
> Subject: Re: [ANNOUNCE] New Hive PMC Chair - Ashutosh ChauhanCongrats 
> Asutosh!From:Sergey Shelukhin
> Reply-To: "u...@hive.apache.org"
> Date: Wednesday, September 16, 2015 at 2:31 PM
> To: "u...@hive.apache.org"
> Subject: Re: [ANNOUNCE] New Hive PMC Chair - Ashutosh ChauhanCongrats!
> From:Alpesh Patel 
> Reply-To: "u...@hive.apache.org" 
> Date: Wednesday, September 16, 2015 at 13:24
> To: "u...@hive.apache.org" 
> Subject: Re: [ANNOUNCE] New Hive PMC Chair - Ashutosh Chauhan
> Congratulations AshutoshOn Wed, Sep 16, 2015 at 1:23 PM, Pengcheng Xiong 
>  wrote: Congratulations Ashutosh!On Wed, Sep 16, 2015 
> at 1:17 PM, John Pullokkaran  wrote: Congrats 
> Ashutosh!From:Vaibhav Gumashta 
> Reply-To: "u...@hive.apache.org" 
> Date: Wednesday, September 16, 2015 at 1:01 PM
> To: "u...@hive.apache.org" , "dev@hive.apache.org" 
> 
> Cc: Ashutosh Chauhan 
> Subject: Re: [ANNOUNCE] New Hive PMC Chair - Ashutosh ChauhanCongrats 
> Ashutosh! —VaibhavFrom:Prasanth Jayachandran 
> 
> Reply-To: "u...@hive.apache.org" 
> Date: Wednesday, September 16, 2015 at 12:50 PM
> To: "dev@hive.apache.org" , "u...@hive.apache.org" 
> 
> Cc: "dev@hive.apache.org" , Ashutosh Chauhan 
> 
> Subject: Re: [ANNOUNCE] New Hive PMC Chair - Ashutosh Chauhan
> Congratulations Ashutosh!
>
>  On Wed, Sep 16, 2015 at 12:48 PM -0700, "Xuefu Zhang"  
> wrote: Congratulations, Ashutosh!. Well-deserved.
>
> Thanks to Carl also for the hard work in the past few years!
>
> --Xuefu
>
> On Wed, Sep 16, 2015 at 12:39 PM, Carl Steinbach  wrote:
>
>> I am very happy to announce that Ashutosh Chauhan is taking over as
>> the new VP of the Apache Hive project. Ashutosh has been a longtime
>> contributor to Hive and has played a pivotal role in many of the major
>> advances that have been made over the past couple of years. Please
>> join me in congratulating Ashutosh on his new role!
>>
>
>

Review Request 38290: Show macro definition for desc function

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38290/
---

Review request for hive.


Bugs: HIVE-11774
https://issues.apache.org/jira/browse/HIVE-11774


Repository: hive-git


Description
---

Currently, desc function shows nothing for macro. It would be helpful if it 
shows the definition of it.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 734742c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java 
b09b706 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java f1fe30d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 7671d29 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java a5d59ae 
  ql/src/java/org/apache/hadoop/hive/ql/parse/MacroSemanticAnalyzer.java 
e3ba201 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateMacroDesc.java 443614c 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMacro.java 
3f505f2 
  ql/src/test/org/apache/hadoop/hive/ql/plan/TestCreateMacroDesc.java 00a24b9 
  ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFMacro.java 
8bbaa6b 
  ql/src/test/queries/clientpositive/macro.q 47b05ff 
  ql/src/test/results/clientpositive/macro.q.out 76ea250 

Diff: https://reviews.apache.org/r/38290/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 38290: HIVE-11774 Show macro definition for desc function

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38290/
---

(Updated 9 11, 2015, 5:53 오전)


Review request for hive.


Summary (updated)
-

HIVE-11774 Show macro definition for desc function 


Bugs: HIVE-11774
https://issues.apache.org/jira/browse/HIVE-11774


Repository: hive-git


Description
---

Currently, desc function shows nothing for macro. It would be helpful if it 
shows the definition of it.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 734742c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java 
b09b706 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java f1fe30d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 7671d29 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java a5d59ae 
  ql/src/java/org/apache/hadoop/hive/ql/parse/MacroSemanticAnalyzer.java 
e3ba201 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateMacroDesc.java 443614c 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMacro.java 
3f505f2 
  ql/src/test/org/apache/hadoop/hive/ql/plan/TestCreateMacroDesc.java 00a24b9 
  ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFMacro.java 
8bbaa6b 
  ql/src/test/queries/clientpositive/macro.q 47b05ff 
  ql/src/test/results/clientpositive/macro.q.out 76ea250 

Diff: https://reviews.apache.org/r/38290/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 38292: HIVE-11768 java.io.DeleteOnExitHook leaks memory on long running Hive Server2 Instances

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38292/
---

Review request for hive.


Repository: hive-git


Description
---

More than 490,000 paths was added to java.io.DeleteOnExitHook on one of our 
long running HiveServer2 instances,taken up more than 100MB on heap.
  Most of the paths contains a suffix of ".pipeout".


Diffs
-

  common/src/java/org/apache/hive/common/util/ShutdownHookManager.java fd2f20a 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 7ed8e5f 
  
service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java
 bcc66cf 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
cc9df76 

Diff: https://reviews.apache.org/r/38292/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 38206: HIVE-11662 Dynamic partitioning cannot be applied to external table which contains part-spec like directory name

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38206/
---

Review request for hive.


Bugs: HIVE-11662
https://issues.apache.org/jira/browse/HIVE-11662


Repository: hive-git


Description
---

Some users want to use part-spec like directory name in their partitioned table 
locations, something like,
{noformat}
/something/warehouse/some_key=some_value
{noformat}

DP calculates additional partitions from full path, and makes exception 
something like,
{noformat}
Failed with exception Partition spec {some_key=some_value, part_key=part_value} 
contains non-partition columns
FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.MoveTask
{noformat}


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java ca86301 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 396c070 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 
3f07ea7 
  ql/src/test/queries/clientpositive/dynamic_partition_insert_external.q 
PRE-CREATION 
  ql/src/test/results/clientpositive/dynamic_partition_insert_external.q.out 
PRE-CREATION 

Diff: https://reviews.apache.org/r/38206/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 37986: Implement "show create database"

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37986/
---

Review request for hive.


Bugs: HIVE-11706
https://issues.apache.org/jira/browse/HIVE-11706


Repository: hive-git


Description
---

HIVE-967 introduced "show create table". How about "show create database"?


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 734742c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 9f8c756 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g cf7ab3a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
a2fbc11 
  ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java 8dbb3c1 
  ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java dee2136 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ShowCreateDatabaseDesc.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/show_create_database.q PRE-CREATION 
  ql/src/test/results/clientpositive/show_create_database.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/tez/show_create_database.q.out 
PRE-CREATION 

Diff: https://reviews.apache.org/r/37986/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 37347: HIVE-11518 Provide interface to adjust required resource for tez tasks

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37347/
---

Review request for hive.


Bugs: HIVE-11518
https://issues.apache.org/jira/browse/HIVE-11518


Repository: hive-git


Description
---

Resource requirements for each tasks are varied but currently it's fixed to one 
value(via hive.tez.container.size). It would be good to customize resource 
requirements appropriate to expected work.

Suggested interface is quite simple.
{code}
public interface ResourceCalculator {

  Resource adjust(Resource resource, MapWork mapWork);

  Resource adjust(Resource resource, ReduceWork reduceWork);
}
{code}


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9cc7987 
  common/src/java/org/apache/hive/common/util/ReflectionUtil.java 05aebc2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java f773cb9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ResourceCalculator.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/SimpleCalculator.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/37347/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 37287: HIVE-11506 Casting varchar/char type to string cannot be vectorized

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37287/
---

(Updated 8 11, 2015, 12:37 오전)


Review request for hive.


Changes
---

Updated golden files


Bugs: HIVE-11506
https://issues.apache.org/jira/browse/HIVE-11506


Repository: hive-git


Description
---

It's not defined in vectorization context.
{code}
explain 
select cast(cast(cstring1 as varchar(10)) as string) x from alltypesorc order 
by x;
{code}

Mapper is not vectorized by exception,
{noformat}
015-08-10 17:02:08,003 INFO  [main]: physical.Vectorizer 
(Vectorizer.java:validateExprNodeDesc(1299)) - Failed to vectorize
org.apache.hadoop.hive.ql.metadata.HiveException: Unhandled cast input type: 
varchar(10)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getCastToString(VectorizationContext.java:1543)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getGenericUDFBridgeVectorExpression(VectorizationContext.java:1379)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getGenericUdfVectorExpression(VectorizationContext.java:1177)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpression(VectorizationContext.java:440)
at 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateExprNodeDesc(Vectorizer.java:1293)
at 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateExprNodeDesc(Vectorizer.java:1284)
at 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateSelectOperator(Vectorizer.java:1116)
at 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateMapWorkOperator(Vectorizer.java:906)
{noformat}


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
5b702bb 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CastStringGroupToString.java
 PRE-CREATION 
  ql/src/test/queries/clientpositive/vectorized_casts.q ade3645 
  ql/src/test/results/clientpositive/tez/vector_char_mapjoin1.q.out f495f95 
  ql/src/test/results/clientpositive/tez/vector_varchar_mapjoin1.q.out ae5be44 
  ql/src/test/results/clientpositive/tez/vectorized_casts.q.out 1dd07b1 
  ql/src/test/results/clientpositive/vector_char_mapjoin1.q.out 744bfb3 
  ql/src/test/results/clientpositive/vector_varchar_mapjoin1.q.out 2994bff 
  ql/src/test/results/clientpositive/vectorized_casts.q.out 1113453 

Diff: https://reviews.apache.org/r/37287/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 37333: HIVE-11515 Still some possible race condition in DynamicPartitionPruner

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37333/
---

Review request for hive.


Bugs: HIVE-11515
https://issues.apache.org/jira/browse/HIVE-11515


Repository: hive-git


Description
---

Even after HIVE-9976, I could see race condition in DPP sometimes. Hard to 
reproduce but it seemed related to the fact that prune() is called by 
thread-pool. With some delay in queue, events from fast tasks are arrived 
before prune() is called.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DynamicPartitionPruner.java 
7abd94d 

Diff: https://reviews.apache.org/r/37333/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 37287: HIVE-11506 Casting varchar/char type to string cannot be vectorized

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37287/
---

Review request for hive.


Bugs: HIVE-11506
https://issues.apache.org/jira/browse/HIVE-11506


Repository: hive-git


Description
---

It's not defined in vectorization context.
{code}
explain 
select cast(cast(cstring1 as varchar(10)) as string) x from alltypesorc order 
by x;
{code}

Mapper is not vectorized by exception,
{noformat}
015-08-10 17:02:08,003 INFO  [main]: physical.Vectorizer 
(Vectorizer.java:validateExprNodeDesc(1299)) - Failed to vectorize
org.apache.hadoop.hive.ql.metadata.HiveException: Unhandled cast input type: 
varchar(10)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getCastToString(VectorizationContext.java:1543)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getGenericUDFBridgeVectorExpression(VectorizationContext.java:1379)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getGenericUdfVectorExpression(VectorizationContext.java:1177)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpression(VectorizationContext.java:440)
at 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateExprNodeDesc(Vectorizer.java:1293)
at 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateExprNodeDesc(Vectorizer.java:1284)
at 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateSelectOperator(Vectorizer.java:1116)
at 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateMapWorkOperator(Vectorizer.java:906)
{noformat}


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
5b702bb 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CastStringGroupToString.java
 PRE-CREATION 
  ql/src/test/queries/clientpositive/vectorized_casts.q ade3645 
  ql/src/test/results/clientpositive/tez/vectorized_casts.q.out 1dd07b1 
  ql/src/test/results/clientpositive/vectorized_casts.q.out 1113453 

Diff: https://reviews.apache.org/r/37287/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 31335: Correlating expression cannot contain unqualified column references

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31335/
---

Review request for hive.


Bugs: HIVE-9734
https://issues.apache.org/jira/browse/HIVE-9734


Repository: hive-git


Description
---

The following valid ISO-SQL constructs will not parse

Unsupported SubQuery Expression 'c1': Correlating expression cannot contain 
unqualified column references

The statement will parse if the predicate is changed to where tjoin1.c1 in ( 
)

select tjoin1.rnum, tjoin1.c1, tjoin1.c2 from tjoin1 where c1 in ( select c1 
from tjoin1) 

select tjoin1.rnum, tjoin1.c1, tjoin1.c2 from tjoin1 where c1 in ( select 
tjoin1.c1 from tjoin1) 

select tjoin1.rnum, tjoin1.c1, tjoin1.c2 from tjoin1 where c1 in ( select x.c1 
from tjoin1 x)


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java 1b6b33b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SubQueryUtils.java 57868b7 
  ql/src/test/queries/clientnegative/subquery_unqual_corr_expr.q 99ff9ca 
  ql/src/test/queries/clientpositive/subquery_unqual_corr_expr.q PRE-CREATION 
  ql/src/test/results/clientnegative/subquery_unqual_corr_expr.q.out f69a538 
  ql/src/test/results/clientpositive/subquery_unqual_corr_expr.q.out 
PRE-CREATION 

Diff: https://reviews.apache.org/r/31335/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 31079: Extend PTFs to provide referenced columns for CP

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31079/
---

Review request for hive.


Bugs: HIVE-9699
https://issues.apache.org/jira/browse/HIVE-9699


Repository: hive-git


Description
---

As described in HIVE-9341, If PTFs can provide referenced column names, column 
pruner can use that.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 
d3e9992 
  ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java 483f55b 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDesc.java 2f31eed 
  
ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PartitionedTableFunctionDef.java 
967caaa 
  ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/MatchPath.java aa48a6c 
  ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/NoopWithMap.java 0b090a9 
  ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/TableFunctionEvaluator.java 
b8b819e 
  ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/TableFunctionResolver.java 
969013c 
  ql/src/test/results/clientpositive/ptf_matchpath.q.out aaa66cf 
  ql/src/test/results/clientpositive/spark/ptf_matchpath.q.out 22c8af3 
  ql/src/test/results/clientpositive/tez/ptf_matchpath.q.out fc2b034 

Diff: https://reviews.apache.org/r/31079/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 29196: Add some explain to PTF operator

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29196/
---

(Updated Feb. 13, 2015, 5:45 a.m.)


Review request for hive.


Changes
---

Addressed comment


Bugs: HIVE-9138
https://issues.apache.org/jira/browse/HIVE-9138


Repository: hive-git


Description
---

PTFOperator does not explain anything in explain statement, making it hard to 
understand the internal works. 


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 78ef0a0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java daf6cb8 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 
f49b7cd 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java fad582a 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SamplePruner.java b0f4b47 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 6c1ab07 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c69ed69 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 5856743 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDesc.java 3ac3245 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java cea9fe2 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/BoundaryDef.java 07590c0 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PTFExpressionDef.java 5d200fb 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PTFInputDef.java 19ed2f2 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PTFQueryInputDef.java 11ef932 
  
ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PartitionedTableFunctionDef.java 
327304c 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowExpressionDef.java 
b96e9d6 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowFrameDef.java 949ed10 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowFunctionDef.java e4ea358 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowTableFunctionDef.java 
083aaf2 
  ql/src/test/queries/clientpositive/ptf_matchpath.q 80dbe29 
  ql/src/test/results/clientpositive/correlationoptimizer12.q.out 1333a8a 
  ql/src/test/results/clientpositive/ctas_colname.q.out 4eb827c 
  ql/src/test/results/clientpositive/groupby_grouping_window.q.out c0e6e5f 
  ql/src/test/results/clientpositive/groupby_resolution.q.out e5d22a3 
  ql/src/test/results/clientpositive/ptf.q.out b82defd 
  ql/src/test/results/clientpositive/ptf_matchpath.q.out e0cea0d 
  ql/src/test/results/clientpositive/ptf_streaming.q.out 6552c3a6 
  ql/src/test/results/clientpositive/quotedid_basic.q.out 6c7a017 
  ql/src/test/results/clientpositive/sample10.q.out 6495fb8 
  ql/src/test/results/clientpositive/sample2.q.out 5b6b0f5 
  ql/src/test/results/clientpositive/sample4.q.out b39cc2c 
  ql/src/test/results/clientpositive/sample6.q.out dd8d288 
  ql/src/test/results/clientpositive/sample9.q.out a8c2e82 
  ql/src/test/results/clientpositive/smb_mapjoin_11.q.out 056c9f1 
  ql/src/test/results/clientpositive/spark/ptf.q.out 2e12c1d 
  ql/src/test/results/clientpositive/spark/ptf_matchpath.q.out e0cea0d 
  ql/src/test/results/clientpositive/spark/ptf_streaming.q.out d759fac 
  ql/src/test/results/clientpositive/spark/sample10.q.out b63718b 
  ql/src/test/results/clientpositive/spark/sample2.q.out eac6fb4 
  ql/src/test/results/clientpositive/spark/sample4.q.out 8030f74 
  ql/src/test/results/clientpositive/spark/sample6.q.out 47da933 
  ql/src/test/results/clientpositive/spark/sample9.q.out ff5d08e 
  ql/src/test/results/clientpositive/spark/smb_mapjoin_11.q.out 6881804 
  ql/src/test/results/clientpositive/spark/subquery_in.q.out 53c28c2 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 5b6242e 
  ql/src/test/results/clientpositive/subquery_in.q.out 3a5e77b 
  ql/src/test/results/clientpositive/subquery_in_having.q.out 1046c9e 
  ql/src/test/results/clientpositive/subquery_notin.q.out 5811394 
  ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out 8c6b202 
  ql/src/test/results/clientpositive/tez/ptf.q.out 3f5f2c6 
  ql/src/test/results/clientpositive/tez/ptf_matchpath.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/tez/ptf_streaming.q.out 79cb92c 
  ql/src/test/results/clientpositive/tez/subquery_in.q.out 69eb568 
  ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out d5d6dc3 
  ql/src/test/results/clientpositive/union_remove_6_subq.q.out be15c1f 
  ql/src/test/results/clientpositive/vectorized_ptf.q.out 6bd98b7 
  ql/src/test/results/clientpositive/windowing_streaming.q.out d45646a 

Diff: https://reviews.apache.org/r/29196/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 26854: HIVE-2573 Create per-session function registry

[Navis Ryu]

 On Feb. 11, 2015, 12:49 a.m., Alexander Pivovarov wrote:
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java, line 341
  https://reviews.apache.org/r/26854/diff/7/?file=858817#file858817line341
 
  What you think about adding 2 static wrapper methods registerUDF and 
  registerGenericUDF and use old code (without system. prefix)

Is it necessary? It's clarifying where the function is added to.


- Navis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review71893
---


On Feb. 9, 2015, 6:56 p.m., Jason Dere wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/26854/
 ---
 
 (Updated Feb. 9, 2015, 6:56 p.m.)
 
 
 Review request for hive, Navis Ryu and Thejas Nair.
 
 
 Bugs: HIVE-2573
 https://issues.apache.org/jira/browse/HIVE-2573
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Small updates to Navis' changes:
 - session registry doesn't lookup metastore for UDFs
 - my feedback from Navis' original patch
 - metastore udfs should not be considered native. This allows them to be 
 added/removed from registry
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8885fd3 
   contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3 
   
 metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
  5f28d73 
   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java a06e858 
   ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0 
   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java bd00bd4 
   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 29fc99b 
   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d72783 
   ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05 
   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 3a2a6ee 
   ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java 
 a29601c 
   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 4aac39a 
   ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java f8007e1 
   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 69a4545 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java 2729ceb 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
  c8c5369 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 
 6a319b7 
   ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 
 1ef6d1b 
   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 20c73cd 
   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 9c7603c 
   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g bbb8eb3 
   ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java 2b239ab 
   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
 9397b7d 
   ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingSpec.java 28afc6b 
   ql/src/java/org/apache/hadoop/hive/ql/plan/FunctionWork.java f968bc1 
   ql/src/java/org/apache/hadoop/hive/ql/plan/ReloadFunctionDesc.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 14bc46a 
   ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/WindowingTableFunction.java 
 903a9b0 
   ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 
 46f8052 
   ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb 
   ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out 
 c7405ed 
   ql/src/test/results/clientnegative/create_function_nonudf_class.q.out 
 d0dd50a 
   ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5 
   ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out ec981ee 
 
 Diff: https://reviews.apache.org/r/26854/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Jason Dere
 

Re: Review Request 26854: HIVE-2573 Create per-session function registry

[Navis Ryu]

 On Feb. 11, 2015, 12:39 a.m., Alexander Pivovarov wrote:
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java, line 265
  https://reviews.apache.org/r/26854/diff/7/?file=858816#file858816line265
 
  remove spaces pls

done


 On Feb. 11, 2015, 12:39 a.m., Alexander Pivovarov wrote:
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java, line 427
  https://reviews.apache.org/r/26854/diff/7/?file=858817#file858817line427
 
  new udf next_day GenericUDFNextDay was added yesterday. Probably should 
  be added here as well

done


- Navis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review71887
---


On Feb. 9, 2015, 6:56 p.m., Jason Dere wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/26854/
 ---
 
 (Updated Feb. 9, 2015, 6:56 p.m.)
 
 
 Review request for hive, Navis Ryu and Thejas Nair.
 
 
 Bugs: HIVE-2573
 https://issues.apache.org/jira/browse/HIVE-2573
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Small updates to Navis' changes:
 - session registry doesn't lookup metastore for UDFs
 - my feedback from Navis' original patch
 - metastore udfs should not be considered native. This allows them to be 
 added/removed from registry
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8885fd3 
   contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3 
   
 metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
  5f28d73 
   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java a06e858 
   ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0 
   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java bd00bd4 
   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 29fc99b 
   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d72783 
   ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05 
   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 3a2a6ee 
   ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java 
 a29601c 
   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 4aac39a 
   ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java f8007e1 
   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 69a4545 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java 2729ceb 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
  c8c5369 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 
 6a319b7 
   ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 
 1ef6d1b 
   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 20c73cd 
   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 9c7603c 
   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g bbb8eb3 
   ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java 2b239ab 
   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
 9397b7d 
   ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingSpec.java 28afc6b 
   ql/src/java/org/apache/hadoop/hive/ql/plan/FunctionWork.java f968bc1 
   ql/src/java/org/apache/hadoop/hive/ql/plan/ReloadFunctionDesc.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 14bc46a 
   ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/WindowingTableFunction.java 
 903a9b0 
   ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 
 46f8052 
   ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb 
   ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out 
 c7405ed 
   ql/src/test/results/clientnegative/create_function_nonudf_class.q.out 
 d0dd50a 
   ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5 
   ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out ec981ee 
 
 Diff: https://reviews.apache.org/r/26854/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Jason Dere
 

Re: Review Request 26854: HIVE-2573 Create per-session function registry

[Navis Ryu]

 On Feb. 9, 2015, 9:46 p.m., Jason Dere wrote:
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java, line 1970
  https://reviews.apache.org/r/26854/diff/7/?file=858817#file858817line1970
 
  Looks like this version of registerTemporaryUDF() is unnecessary, and 
  what caused the stack overflow in TestMacroSemanticAnalyzer during 
  pre-commit tests

Fixed. Thanks.


- Navis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review71702
---


On Feb. 9, 2015, 6:56 p.m., Jason Dere wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/26854/
 ---
 
 (Updated Feb. 9, 2015, 6:56 p.m.)
 
 
 Review request for hive, Navis Ryu and Thejas Nair.
 
 
 Bugs: HIVE-2573
 https://issues.apache.org/jira/browse/HIVE-2573
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Small updates to Navis' changes:
 - session registry doesn't lookup metastore for UDFs
 - my feedback from Navis' original patch
 - metastore udfs should not be considered native. This allows them to be 
 added/removed from registry
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8885fd3 
   contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3 
   
 metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
  5f28d73 
   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java a06e858 
   ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0 
   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java bd00bd4 
   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 29fc99b 
   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d72783 
   ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05 
   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 3a2a6ee 
   ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java 
 a29601c 
   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 4aac39a 
   ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java f8007e1 
   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 69a4545 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java 2729ceb 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
  c8c5369 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 
 6a319b7 
   ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 
 1ef6d1b 
   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 20c73cd 
   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 9c7603c 
   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g bbb8eb3 
   ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java 2b239ab 
   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
 9397b7d 
   ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingSpec.java 28afc6b 
   ql/src/java/org/apache/hadoop/hive/ql/plan/FunctionWork.java f968bc1 
   ql/src/java/org/apache/hadoop/hive/ql/plan/ReloadFunctionDesc.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 14bc46a 
   ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/WindowingTableFunction.java 
 903a9b0 
   ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 
 46f8052 
   ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb 
   ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out 
 c7405ed 
   ql/src/test/results/clientnegative/create_function_nonudf_class.q.out 
 d0dd50a 
   ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5 
   ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out ec981ee 
 
 Diff: https://reviews.apache.org/r/26854/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Jason Dere
 

Review Request 30778: NPE for invalid union all

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30778/
---

Review request for hive.


Bugs: HIVE-9513
https://issues.apache.org/jira/browse/HIVE-9513


Repository: hive-git


Description
---

NPE duting parsing  of :

{noformat}
select * from (
 select * from ( select 1 as id , foo as str_1 from staging.dual ) f
  union all
 select * from ( select 2 as id , bar as str_2 from staging.dual ) g
) e ;
{noformat}


Diffs
-

  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 7c09fcc 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 3a613a2 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 2466d78 
  ql/src/test/queries/clientnegative/union3.q ce65747 
  ql/src/test/queries/clientpositive/union35.q PRE-CREATION 
  ql/src/test/results/clientnegative/union3.q.out de1c62b 
  ql/src/test/results/clientpositive/union35.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/30778/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 30783: JDBC should provide metadata for columns whether a column is a partition column or not

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30783/
---

Review request for hive.


Bugs: HIVE-3050
https://issues.apache.org/jira/browse/HIVE-3050


Repository: hive-git


Description
---

Trivial request from UI developers. 
{code}
DatabaseMetaData databaseMetaData = connection.getMetaData();
ResultSet rs = databaseMetaData.getColumns(null, null, tableName, null);

boolean partitionKey = rs.getBoolean(IS_PARTITION_COLUMN);
{code}
It's not JDBC standard column but seemed to be useful.


Diffs
-

  
service/src/java/org/apache/hive/service/cli/operation/GetColumnsOperation.java 
92ea7b0 

Diff: https://reviews.apache.org/r/30783/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 30779: hive.limit.query.max.table.partition makes queries fail on non-partitioned tables

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30779/
---

Review request for hive.


Repository: hive-git


Description
---

If you use hive.limit.query.max.table.partition to limit the amount of 
partitions that can be queried it makes queries on non-partitioned tables fail.

Example:
{noformat}
CREATE TABLE tmp(test INT);
SELECT COUNT(*) FROM TMP; -- works fine
SET hive.limit.query.max.table.partition=20;
SELECT COUNT(*) FROM TMP; -- generates NPE (FAILED: NullPointerException null)
SET hive.limit.query.max.table.partition=-1;
SELECT COUNT(*) FROM TMP; -- works fine again
{noformat}


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 d18e1a7 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 6c1ab07 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 2466d78 

Diff: https://reviews.apache.org/r/30779/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 30782: Window clause ROW BETWEEN for PRECEDING does not work

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30782/
---

Review request for hive.


Bugs: HIVE-9412
https://issues.apache.org/jira/browse/HIVE-9412


Repository: hive-git


Description
---

When window clause with ROWS is used between two proper PRECEDING boundaries, 
Hive reports syntax errors as following examples.

--For example 1
SELECT name, dept_num, salary,
MAX(salary) OVER (PARTITION BY dept_num ORDER BY 
name ROWS BETWEEN 2 PRECEDING AND 1 PRECEDING) win4_alter
FROM employee_contract;

Error: Error while compiling statement: FAILED: SemanticException Failed to 
breakup Windowing invocations into Groups. At least 1 group must only depend on 
input columns. Also check for circular dependencies.
Underlying error: Window range invalid, start boundary is greater than end 
boundary: window(start=range(2 PRECEDING), end=range(1 PRECEDING)) 
(state=42000,code=4)

--For example 2
SELECT name, dept_num, salary,
MAX(salary) OVER (PARTITION BY dept_num ORDER BY 
name ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING) win1
FROM employee_contract;

Error: Error while compiling statement: FAILED: SemanticException End of a 
WindowFrame cannot be UNBOUNDED PRECEDING (state=42000,code=4)


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/PTFOperator.java e95505c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingSpec.java 28afc6b 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowFunctionDef.java e4ea358 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFAverage.java 
12a327f 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFFirstValue.java 
f679387 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFLastValue.java 
e099154 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMax.java a153818 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMin.java d931d52 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFStreamingEvaluator.java
 d68c085 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java ffb7093 
  ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/WindowingTableFunction.java 
903a9b0 
  ql/src/test/queries/clientpositive/windowing_windowspec.q 6d8ce67 
  ql/src/test/results/clientpositive/windowing_windowspec.q.out 00af6b8 

Diff: https://reviews.apache.org/r/30782/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 26854: HIVE-2573 Create per-session function registry

[Navis Ryu]

 On Nov. 14, 2014, 9:24 p.m., Jason Dere wrote:
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g, line 1626
  https://reviews.apache.org/r/26854/diff/6/?file=762528#file762528line1626
 
  The message here should be reload function statement

Fixed


 On Nov. 14, 2014, 9:24 p.m., Jason Dere wrote:
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionConf.java, line 24
  https://reviews.apache.org/r/26854/diff/6/?file=762534#file762534line24
 
  What about the idea of moving static call to resolveFunctions() to 
  SessionState? I thought that would remove the need for SessionConf, because 
  then Hive class would once again be usable during query runtime. Unless you 
  think it's cleaner to use SessionConf to get HiveConf rather than the Hive 
  object.

Done


 On Nov. 14, 2014, 9:24 p.m., Jason Dere wrote:
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java, line 105
  https://reviews.apache.org/r/26854/diff/6/?file=762515#file762515line105
 
  Ok, so this allows the persistent function list to be reloaded, with an 
  explicit RELOAD command. This should work for now, I suppose it's always 
  possible to add more automatic refreshing of functions later on if folks 
  want that.

Yes, let's do that in following issue.


- Navis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review61331
---


On Nov. 13, 2014, 10:15 p.m., Jason Dere wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/26854/
 ---
 
 (Updated Nov. 13, 2014, 10:15 p.m.)
 
 
 Review request for hive, Navis Ryu and Thejas Nair.
 
 
 Bugs: HIVE-2573
 https://issues.apache.org/jira/browse/HIVE-2573
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Small updates to Navis' changes:
 - session registry doesn't lookup metastore for UDFs
 - my feedback from Navis' original patch
 - metastore udfs should not be considered native. This allows them to be 
 added/removed from registry
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cca57d2 
   contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3 
   
 metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
  88b0791 
   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 292c83c 
   ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0 
   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b 
   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 6323387 
   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 913288f 
   ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05 
   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 18e40b3 
   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java b900627 
   ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 13277a9 
   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 211ab6c 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java e2768ff 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/SqlFunctionConverter.java
  7f52c29 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 
 4b2a81a 
   ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 
 22e5b47 
   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g f412010 
   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g f1365fa 
   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g c960a6b 
   ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java 2b239ab 
   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
 6962ee9 
   ql/src/java/org/apache/hadoop/hive/ql/plan/FunctionWork.java f968bc1 
   ql/src/java/org/apache/hadoop/hive/ql/plan/ReloadFunctionDesc.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/session/SessionConf.java PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 2806bd1 
   ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 
 46f8052 
   ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb 
   ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out 
 c7405ed 
   ql/src/test/results/clientnegative/create_function_nonudf_class.q.out 
 d0dd50a 
   ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5 
   ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out e184787 
 
 Diff

Review Request 30780: Use session classloader instead of application loader

[Navis Ryu]

/ql/io/HiveFileFormatUtils.java e2ae25b 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 9f24190 
  ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanTask.java 
0f7e833 
  
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/truncate/ColumnTruncateTask.java
 73c6dcc 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/AbstractSMBJoinProc.java 
ff34682 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
6bef5f5 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 7ac0551 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDeserializer.java f75bec5 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java b62ffed 
  ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java 
e26031c 
  ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java 
32826e7 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java d3ad515 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect.java 
89496ea 

Diff: https://reviews.apache.org/r/30780/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 30602: HIVE-6099

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30602/#review71156
---



ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java
https://reviews.apache.org/r/30602/#comment116786

if groupKeyIsNotReduceKey is always false, firstRowInGroup will not be 
evaluated. It would be safe to removed it.



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
https://reviews.apache.org/r/30602/#comment116787

not-not-isDistinct.. how about replace it with isDistinct


- Navis Ryu


On Feb. 4, 2015, 9:16 p.m., Ashutosh Chauhan wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/30602/
 ---
 
 (Updated Feb. 4, 2015, 9:16 p.m.)
 
 
 Review request for hive and Navis Ryu.
 
 
 Bugs: HIVE-6099
 https://issues.apache.org/jira/browse/HIVE-6099
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 remove common.distincts optimization
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6d2748e 
   ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 2e40556 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
 96901c9 
   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 2d5e6cf 
   ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 8804258 
   ql/src/test/queries/clientpositive/groupby10.q 8b1fb52 
   ql/src/test/queries/clientpositive/groupby_multi_insert_common_distinct.q 
 b009a8b 
   ql/src/test/results/clientpositive/groupby10.q.out a575089 
   ql/src/test/results/clientpositive/groupby11.q.out 322140e 
   ql/src/test/results/clientpositive/groupby8.q.out 68d6e47 
   ql/src/test/results/clientpositive/groupby8_map.q.out 37301f7 
   ql/src/test/results/clientpositive/groupby8_map_skew.q.out 37301f7 
   ql/src/test/results/clientpositive/groupby8_noskew.q.out 37301f7 
   ql/src/test/results/clientpositive/groupby9.q.out e96a22f 
   
 ql/src/test/results/clientpositive/groupby_multi_insert_common_distinct.q.out 
 c6b5edc 
   ql/src/test/results/clientpositive/spark/groupby10.q.out 2bae6ae 
   ql/src/test/results/clientpositive/spark/groupby11.q.out 378c166 
   ql/src/test/results/clientpositive/spark/groupby8.q.out 53ecd96 
   ql/src/test/results/clientpositive/spark/groupby8_map.q.out 0517b72 
   ql/src/test/results/clientpositive/spark/groupby8_map_skew.q.out 0517b72 
   ql/src/test/results/clientpositive/spark/groupby8_noskew.q.out 0517b72 
   ql/src/test/results/clientpositive/spark/groupby9.q.out 9b5095a 
   
 ql/src/test/results/clientpositive/spark/groupby_multi_insert_common_distinct.q.out
  c2766f8 
   ql/src/test/results/clientpositive/union17.q.out 75d2253 
 
 Diff: https://reviews.apache.org/r/30602/diff/
 
 
 Testing
 ---
 
 Existing tests.
 
 
 Thanks,
 
 Ashutosh Chauhan
 

Re: Review Request 30602: HIVE-6099

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30602/#review70902
---


We can simplify code further

genGroupByPlanGroupByOperator1.distPartAgg=false - 
  GroupByDesc.groupKeyIsNotReduceKey=false - 
  
GroupByOperator.groupKeyIsNotReduceKey/keysCurrentGroup/firstRowInGroup/startGroup/endGroup
 is not needed

- Navis Ryu


On Feb. 4, 2015, 2:02 a.m., Ashutosh Chauhan wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/30602/
 ---
 
 (Updated Feb. 4, 2015, 2:02 a.m.)
 
 
 Review request for hive and Navis Ryu.
 
 
 Bugs: HIVE-6099
 https://issues.apache.org/jira/browse/HIVE-6099
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 remove common.distincts optimization
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6d2748e 
   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 7b645b8 
   ql/src/test/queries/clientpositive/groupby10.q 8b1fb52 
   ql/src/test/queries/clientpositive/groupby_multi_insert_common_distinct.q 
 b009a8b 
   ql/src/test/results/clientpositive/groupby10.q.out a575089 
   ql/src/test/results/clientpositive/groupby11.q.out 322140e 
   ql/src/test/results/clientpositive/groupby8.q.out 68d6e47 
   ql/src/test/results/clientpositive/groupby8_map.q.out 37301f7 
   ql/src/test/results/clientpositive/groupby8_map_skew.q.out 37301f7 
   ql/src/test/results/clientpositive/groupby8_noskew.q.out 37301f7 
   ql/src/test/results/clientpositive/groupby9.q.out e96a22f 
   
 ql/src/test/results/clientpositive/groupby_multi_insert_common_distinct.q.out 
 c6b5edc 
   ql/src/test/results/clientpositive/spark/groupby10.q.out 2bae6ae 
   ql/src/test/results/clientpositive/spark/groupby11.q.out 378c166 
   ql/src/test/results/clientpositive/spark/groupby8.q.out 53ecd96 
   ql/src/test/results/clientpositive/spark/groupby8_map.q.out 0517b72 
   ql/src/test/results/clientpositive/spark/groupby8_map_skew.q.out 0517b72 
   ql/src/test/results/clientpositive/spark/groupby8_noskew.q.out 0517b72 
   ql/src/test/results/clientpositive/spark/groupby9.q.out 9b5095a 
   
 ql/src/test/results/clientpositive/spark/groupby_multi_insert_common_distinct.q.out
  c2766f8 
   ql/src/test/results/clientpositive/union17.q.out 75d2253 
 
 Diff: https://reviews.apache.org/r/30602/diff/
 
 
 Testing
 ---
 
 Existing tests.
 
 
 Thanks,
 
 Ashutosh Chauhan
 

Re: Review Request 30549: SELECT max(bar) FROM foo is broken after ANALYZE ... FOR COLUMNS

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30549/
---

(Updated Feb. 4, 2015, 1:43 a.m.)


Review request for hive.


Changes
---

Updated result  fixed further more(distinct_stats was fall back to normal plan 
by exception making struct OI)


Bugs: HIVE-9397
https://issues.apache.org/jira/browse/HIVE-9397


Repository: hive-git


Description
---

These queries produce an error :

{code:sql}
DROP TABLE IF EXISTS foo;

CREATE TABLE foo (id int) STORED AS ORC;

INSERT INTO TABLE foo VALUES (1);
INSERT INTO TABLE foo VALUES (2);
INSERT INTO TABLE foo VALUES (3);
INSERT INTO TABLE foo VALUES (4);
INSERT INTO TABLE foo VALUES (5);

SELECT max(id) FROM foo;

ANALYZE TABLE foo COMPUTE STATISTICS FOR COLUMNS id;

SELECT max(id) FROM foo;
{code}

The last query throws {{org.apache.hive.service.cli.HiveSQLException}}
{noformat}
0: jdbc:hive2://nc-h04:1/casino SELECT max(id) FROM foo;
+-+--+
| _c0 |
+-+--+
org.apache.hive.service.cli.HiveSQLException: java.lang.ClassCastException
0: jdbc:hive2://nc-h04:1/casino
{noformat}


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/StatsOptimizer.java 6961d7f 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java d1118f1 
  ql/src/test/results/clientpositive/metadata_only_queries.q.out 90c76ed 
  ql/src/test/results/clientpositive/metadata_only_queries_with_filters.q.out 
5be958f 
  ql/src/test/results/clientpositive/spark/metadata_only_queries.q.out 9d6a404 
  
ql/src/test/results/clientpositive/spark/metadata_only_queries_with_filters.q.out
 5be958f 
  ql/src/test/results/clientpositive/tez/metadata_only_queries.q.out 8c0bccb 

Diff: https://reviews.apache.org/r/30549/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 30549: SELECT max(bar) FROM foo is broken after ANALYZE ... FOR COLUMNS

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30549/
---

Review request for hive.


Bugs: HIVE-9397
https://issues.apache.org/jira/browse/HIVE-9397


Repository: hive-git


Description
---

These queries produce an error :

{code:sql}
DROP TABLE IF EXISTS foo;

CREATE TABLE foo (id int) STORED AS ORC;

INSERT INTO TABLE foo VALUES (1);
INSERT INTO TABLE foo VALUES (2);
INSERT INTO TABLE foo VALUES (3);
INSERT INTO TABLE foo VALUES (4);
INSERT INTO TABLE foo VALUES (5);

SELECT max(id) FROM foo;

ANALYZE TABLE foo COMPUTE STATISTICS FOR COLUMNS id;

SELECT max(id) FROM foo;
{code}

The last query throws {{org.apache.hive.service.cli.HiveSQLException}}
{noformat}
0: jdbc:hive2://nc-h04:1/casino SELECT max(id) FROM foo;
+-+--+
| _c0 |
+-+--+
org.apache.hive.service.cli.HiveSQLException: java.lang.ClassCastException
0: jdbc:hive2://nc-h04:1/casino
{noformat}


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/StatsOptimizer.java 6961d7f 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java d1118f1 
  ql/src/test/results/clientpositive/metadata_only_queries.q.out 90c76ed 
  ql/src/test/results/clientpositive/metadata_only_queries_with_filters.q.out 
5be958f 

Diff: https://reviews.apache.org/r/30549/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 30151: Remove Extract Operator its friends from codebase.

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30151/#review70534
---

Ship it!


Ship It!

- Navis Ryu


On Jan. 30, 2015, 7:46 p.m., Ashutosh Chauhan wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/30151/
 ---
 
 (Updated Jan. 30, 2015, 7:46 p.m.)
 
 
 Review request for hive and Navis Ryu.
 
 
 Bugs: HIVE-9416
 https://issues.apache.org/jira/browse/HIVE-9416
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Remove Extract Operator  its friends from codebase.
 
 
 Diffs
 -
 
   ql/src/java/org/apache/hadoop/hive/ql/exec/ExtractOperator.java c299d3a 
   ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java f3c382a 
   ql/src/java/org/apache/hadoop/hive/ql/exec/PTFOperator.java 2e6a880 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorExtractOperator.java 
 7f4bb64 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java
  24ca89f 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java
  f36f843 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java
  137956c 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationUtilities.java
  630a9eb 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplication.java
  3fead79 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/lineage/OpProcFactory.java 
 adca50d 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/BucketingSortingInferenceOptimizer.java
  7954767 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/BucketingSortingOpProcFactory.java
  cf02bec 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
 94b4621 
   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c9a5ce5 
   ql/src/java/org/apache/hadoop/hive/ql/plan/ExtractDesc.java 6762155 
   ql/src/java/org/apache/hadoop/hive/ql/plan/SelectDesc.java fa6b548 
   ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 41862e6 
   ql/src/test/results/clientpositive/bucket1.q.out 13ec735 
   ql/src/test/results/clientpositive/bucket2.q.out 32a77c3 
   ql/src/test/results/clientpositive/bucket3.q.out ff7173e 
   ql/src/test/results/clientpositive/bucket4.q.out b99d12f 
   ql/src/test/results/clientpositive/bucket5.q.out 5992d6d 
   ql/src/test/results/clientpositive/bucket6.q.out 5b23d7d 
   ql/src/test/results/clientpositive/bucketsortoptimize_insert_1.q.out 
 75de953 
   ql/src/test/results/clientpositive/bucketsortoptimize_insert_2.q.out 
 599b8b9 
   ql/src/test/results/clientpositive/bucketsortoptimize_insert_3.q.out 
 7456ab0 
   ql/src/test/results/clientpositive/bucketsortoptimize_insert_4.q.out 
 fd99597 
   ql/src/test/results/clientpositive/bucketsortoptimize_insert_5.q.out 
 8130ab9 
   ql/src/test/results/clientpositive/bucketsortoptimize_insert_6.q.out 
 627aba0 
   ql/src/test/results/clientpositive/disable_merge_for_bucketing.q.out 
 9b058c8 
   ql/src/test/results/clientpositive/dynpart_sort_opt_vectorization.q.out 
 0baa446 
   ql/src/test/results/clientpositive/dynpart_sort_optimization.q.out 494bfa3 
   
 ql/src/test/results/clientpositive/encrypted/encryption_insert_partition_dynamic.q.out
  b6e7b88 
   
 ql/src/test/results/clientpositive/encrypted/encryption_insert_partition_static.q.out
  fc6d2ae 
   ql/src/test/results/clientpositive/load_dyn_part2.q.out 26f318a 
   ql/src/test/results/clientpositive/ptf.q.out 2317347 
   ql/src/test/results/clientpositive/ptf_streaming.q.out 427e635 
   ql/src/test/results/clientpositive/smb_mapjoin_20.q.out 999dabd 
   ql/src/test/results/clientpositive/smb_mapjoin_21.q.out 539b70e 
   ql/src/test/results/clientpositive/spark/bucket2.q.out 5eb28fa 
   ql/src/test/results/clientpositive/spark/bucket3.q.out 1b1010a 
   ql/src/test/results/clientpositive/spark/bucket4.q.out 7dd49ac 
   ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_2.q.out 
 365306e 
   ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_4.q.out 
 3846de7 
   ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_6.q.out 
 5b559c4 
   ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_7.q.out 
 cefc6aa 
   ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_8.q.out 
 ca44d7c 
   ql/src/test/results/clientpositive/spark/disable_merge_for_bucketing.q.out 
 3864c44 
   ql/src/test/results/clientpositive/spark/load_dyn_part2.q.out a8cef34 
   ql/src/test/results/clientpositive/spark/ptf.q.out deebf3a 
   ql/src/test/results/clientpositive/spark/ptf_streaming.q.out cd77c5f 
   ql/src/test/results/clientpositive/spark/smb_mapjoin_20.q.out

Review Request 30345: Problem with subquery using windowing functions

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30345/
---

Review request for hive.


Bugs: HIVE-9228
https://issues.apache.org/jira/browse/HIVE-9228


Repository: hive-git


Description
---

The following query with window functions failed. The internal query works fine.

select col1, col2, col3 from (select col1,col2, col3, count(case when col4=1 
then 1 end ) over (partition by col1, col2) as col5, row_number() over 
(partition by col1, col2 order by col4) as col6 from tab1) t;

HIVE generates an execution plan with 2 jobs. 
1. The first job is to basically calculate window function for col5.  
2. The second job is to calculate window function for col6 and output.

The plan says the first job outputs the columns (col1, col2, col3, col4) to a 
tmp file since only these columns are used in later stage. While, the PTF 
operator for the first job outputs (_wcol0, col1, col2, col3, col4) with _wcol0 
as the result of the window function even it's not used. 

In the second job, the map operator still reads the 4 columns (col1, col2, 
col3, col4) from the temp file using the plan. That causes the exception.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 
57ce849 
  ql/src/java/org/apache/hadoop/hive/ql/parse/RowResolver.java 469dc9f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c05b2aa 
  ql/src/test/queries/clientpositive/windowing_windowspec.q 6d8ce67 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 020fdff 
  ql/src/test/results/clientpositive/subquery_notin.q.out 599a61e 
  ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out 06d5708 
  ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out a814849 
  ql/src/test/results/clientpositive/vectorized_ptf.q.out 1e3c43c 
  ql/src/test/results/clientpositive/windowing_windowspec.q.out 00af6b8 

Diff: https://reviews.apache.org/r/30345/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 29196: Add some explain to PTF operator

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29196/
---

(Updated Jan. 27, 2015, 2:14 a.m.)


Review request for hive.


Bugs: HIVE-9138
https://issues.apache.org/jira/browse/HIVE-9138


Repository: hive-git


Description
---

PTFOperator does not explain anything in explain statement, making it hard to 
understand the internal works. 


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 12fcd6a 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 8e00ee3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java daf6cb8 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 
57ce849 
  ql/src/java/org/apache/hadoop/hive/ql/plan/Explain.java a3408a0 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDesc.java 3ac3245 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java b62ffed 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/BoundaryDef.java 07590c0 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/OrderExpressionDef.java 
e367d13 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PTFExpressionDef.java 5d200fb 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PTFInputDef.java 19ed2f2 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PTFQueryInputDef.java 11ef932 
  
ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PartitionedTableFunctionDef.java 
327304c 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowExpressionDef.java 
b96e9d6 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowFrameDef.java 949ed10 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowFunctionDef.java e4ea358 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowTableFunctionDef.java 
083aaf2 
  ql/src/test/queries/clientpositive/ptf_matchpath.q 80dbe29 
  ql/src/test/results/clientpositive/correlationoptimizer12.q.out c32e41e 
  ql/src/test/results/clientpositive/ctas_colname.q.out 95c7acb 
  ql/src/test/results/clientpositive/groupby_resolution.q.out c611f7d 
  ql/src/test/results/clientpositive/ptf.q.out f678035 
  ql/src/test/results/clientpositive/ptf_matchpath.q.out e0cea0d 
  ql/src/test/results/clientpositive/ptf_streaming.q.out 9cf645d 
  ql/src/test/results/clientpositive/quotedid_basic.q.out b8cd4e9 
  ql/src/test/results/clientpositive/spark/ptf.q.out 8ca5496 
  ql/src/test/results/clientpositive/spark/ptf_matchpath.q.out e0cea0d 
  ql/src/test/results/clientpositive/spark/ptf_streaming.q.out f5ee72d 
  ql/src/test/results/clientpositive/spark/subquery_in.q.out 51b92a3 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 020fdff 
  ql/src/test/results/clientpositive/subquery_in.q.out a2235af 
  ql/src/test/results/clientpositive/subquery_in_having.q.out 03cc2af 
  ql/src/test/results/clientpositive/subquery_notin.q.out 599a61e 
  ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out 06d5708 
  ql/src/test/results/clientpositive/tez/ptf.q.out 6f9dd91 
  ql/src/test/results/clientpositive/tez/ptf_matchpath.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/tez/ptf_streaming.q.out a935ef6 
  ql/src/test/results/clientpositive/tez/subquery_in.q.out 8bc7892 
  ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out a814849 
  ql/src/test/results/clientpositive/vectorized_ptf.q.out 1e3c43c 
  ql/src/test/results/clientpositive/windowing_streaming.q.out ac9e180 

Diff: https://reviews.apache.org/r/29196/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 30151: Remove Extract Operator its friends from codebase.

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30151/#review69732
---


This is huge cleanup. Good work!


ql/src/test/results/clientpositive/bucketsortoptimize_insert_4.q.out
https://reviews.apache.org/r/30151/#comment114490

Any idea why the plan is changed so much?


- Navis Ryu


On Jan. 24, 2015, 6:08 p.m., Ashutosh Chauhan wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/30151/
 ---
 
 (Updated Jan. 24, 2015, 6:08 p.m.)
 
 
 Review request for hive and Navis Ryu.
 
 
 Bugs: HIVE-9416
 https://issues.apache.org/jira/browse/HIVE-9416
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Remove Extract Operator  its friends from codebase.
 
 
 Diffs
 -
 
   ql/src/java/org/apache/hadoop/hive/ql/exec/ExtractOperator.java c299d3a 
   ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java f3c382a 
   ql/src/java/org/apache/hadoop/hive/ql/exec/PTFOperator.java 2e6a880 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 9ed2c61 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorExtractOperator.java 
 7f4bb64 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java
  24ca89f 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java
  e16ba6c 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationUtilities.java
  dc906e8 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplication.java
  3fead79 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/lineage/OpProcFactory.java 
 d6a6ed6 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/BucketingSortingInferenceOptimizer.java
  7954767 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/BucketingSortingOpProcFactory.java
  cf02bec 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
 94b4621 
   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 4364f28 
   ql/src/java/org/apache/hadoop/hive/ql/plan/ExtractDesc.java 6762155 
   ql/src/java/org/apache/hadoop/hive/ql/plan/SelectDesc.java fa6b548 
   ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 41862e6 
   ql/src/test/results/clientpositive/bucket1.q.out 13ec735 
   ql/src/test/results/clientpositive/bucket2.q.out 32a77c3 
   ql/src/test/results/clientpositive/bucket3.q.out ff7173e 
   ql/src/test/results/clientpositive/bucket4.q.out b99d12f 
   ql/src/test/results/clientpositive/bucket5.q.out 5992d6d 
   ql/src/test/results/clientpositive/bucket6.q.out 5b23d7d 
   ql/src/test/results/clientpositive/bucketsortoptimize_insert_1.q.out 
 75de953 
   ql/src/test/results/clientpositive/bucketsortoptimize_insert_2.q.out 
 599b8b9 
   ql/src/test/results/clientpositive/bucketsortoptimize_insert_3.q.out 
 7456ab0 
   ql/src/test/results/clientpositive/bucketsortoptimize_insert_4.q.out 
 fd99597 
   ql/src/test/results/clientpositive/bucketsortoptimize_insert_5.q.out 
 8130ab9 
   ql/src/test/results/clientpositive/bucketsortoptimize_insert_6.q.out 
 627aba0 
   ql/src/test/results/clientpositive/disable_merge_for_bucketing.q.out 
 9b058c8 
   ql/src/test/results/clientpositive/dynpart_sort_opt_vectorization.q.out 
 32e0745 
   ql/src/test/results/clientpositive/dynpart_sort_optimization.q.out 494bfa3 
   
 ql/src/test/results/clientpositive/encrypted/encryption_insert_partition_dynamic.q.out
  b6e7b88 
   
 ql/src/test/results/clientpositive/encrypted/encryption_insert_partition_static.q.out
  fc6d2ae 
   ql/src/test/results/clientpositive/load_dyn_part2.q.out 26f318a 
   ql/src/test/results/clientpositive/ptf.q.out f678035 
   ql/src/test/results/clientpositive/ptf_streaming.q.out 9cf645d 
   ql/src/test/results/clientpositive/smb_mapjoin_20.q.out 999dabd 
   ql/src/test/results/clientpositive/smb_mapjoin_21.q.out 539b70e 
   ql/src/test/results/clientpositive/spark/bucket2.q.out 5eb28fa 
   ql/src/test/results/clientpositive/spark/bucket3.q.out 1b1010a 
   ql/src/test/results/clientpositive/spark/bucket4.q.out 7dd49ac 
   ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_2.q.out 
 365306e 
   ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_4.q.out 
 3846de7 
   ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_6.q.out 
 5b559c4 
   ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_7.q.out 
 cefc6aa 
   ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_8.q.out 
 ca44d7c 
   ql/src/test/results/clientpositive/spark/disable_merge_for_bucketing.q.out 
 3864c44 
   ql/src/test/results/clientpositive/spark/load_dyn_part2.q.out a8cef34 
   ql/src/test/results/clientpositive/spark

Re: Review Request 29800: Apply ColumnPrunning for noop PTFs

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29800/
---

(Updated Jan. 23, 2015, 5:41 a.m.)


Review request for hive.


Changes
---

Updated with full diff


Bugs: HIVE-9341
https://issues.apache.org/jira/browse/HIVE-9341


Repository: hive-git


Description
---

Currently, PTF disables CP optimization, which can make a huge burden. For 
example,
{noformat}
select p_mfgr, p_name, p_size,
rank() over (partition by p_mfgr order by p_name) as r,
dense_rank() over (partition by p_mfgr order by p_name) as dr,
sum(p_retailprice) over (partition by p_mfgr order by p_name rows between 
unbounded preceding and current row) as s1
from noop(on part 
  partition by p_mfgr
  order by p_name
  );

STAGE PLANS:
  Stage: Stage-1
Map Reduce
  Map Operator Tree:
  TableScan
alias: part
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE 
Column stats: NONE
Reduce Output Operator
  key expressions: p_mfgr (type: string), p_name (type: string)
  sort order: ++
  Map-reduce partition columns: p_mfgr (type: string)
  Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE 
Column stats: NONE
  value expressions: p_partkey (type: int), p_name (type: string), 
p_mfgr (type: string), p_brand (type: string), p_type (type: string), p_size 
(type: int), p_container (type: string), p_retailprice (type: double), 
p_comment (type: string), BLOCK__OFFSET__INSIDE__FILE (type: bigint), 
INPUT__FILE__NAME (type: string), ROW__ID (type: 
structtransactionid:bigint,bucketid:int,rowid:bigint)
...
{noformat}

There should be a generic way to discern referenced columns but before that, we 
know CP can be safely applied to noop functions.


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties d08651b 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 479af32 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 
abf32f1 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkSkewJoinProcFactory.java
 fe698ef 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicatePushDown.java ee7328e 
  ql/src/test/queries/clientpositive/ptf.q 56eef0a 
  ql/src/test/queries/clientpositive/ptf_streaming.q 04b674c 
  ql/src/test/results/clientpositive/ptf.q.out 9196b94 
  ql/src/test/results/clientpositive/ptf_streaming.q.out ef7ae88 
  ql/src/test/results/clientpositive/spark/ptf.q.out 9196b94 
  ql/src/test/results/clientpositive/spark/ptf_streaming.q.out ef7ae88 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out f3b61ce 
  ql/src/test/results/clientpositive/tez/ptf.q.out 9196b94 
  ql/src/test/results/clientpositive/tez/ptf_streaming.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out 928c9f0 
  ql/src/test/results/clientpositive/vectorized_ptf.q.out 7fdd1d8 

Diff: https://reviews.apache.org/r/29800/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 29878: Bug with max() together with rank() and grouping sets

[Navis Ryu]

 On Jan. 16, 2015, 6:20 p.m., Ashutosh Chauhan wrote:
  overall looks good. Few minor comments.

Thanks for the review. Fixed BucketingSortingInferenceOptimizer to reflect 
grouping set pruning and changed default value of grouping set id to -1 in 
semijoin cases.


 On Jan. 16, 2015, 6:20 p.m., Ashutosh Chauhan wrote:
  ql/src/test/results/clientpositive/infer_bucket_sort_grouping_operators.q.out,
   line 147
  https://reviews.apache.org/r/29878/diff/1/?file=820775#file820775line147
 
  should this have been only key,value ?

Fixed BucketingSortingInferenceOptimizer to reflect grouping set pruning


 On Jan. 16, 2015, 6:20 p.m., Ashutosh Chauhan wrote:
  ql/src/test/results/clientpositive/infer_bucket_sort_grouping_operators.q.out,
   line 334
  https://reviews.apache.org/r/29878/diff/1/?file=820775#file820775line334
 
  seems like agg column should not be present there? Can you check?

Same with above.


- Navis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29878/#review68438
---


On Jan. 14, 2015, 9:13 a.m., Navis Ryu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/29878/
 ---
 
 (Updated Jan. 14, 2015, 9:13 a.m.)
 
 
 Review request for hive.
 
 
 Bugs: HIVE-9347
 https://issues.apache.org/jira/browse/HIVE-9347
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 It looks like the query below returns incorrect results on Hive 0.13.1, but 
 it was working fine on Hive 0.11. 
 
 I have the following table:
 CREATE  TABLE `t`(
   `category` int, 
   `live` int, 
   `comments` int)
 
 with the following data:
 hive select * from t;
 OK
 3   0   2
 2   0   2
 8   0   2
 
 The query:
 hive select category, max(live) live, max(comments) comments, rank() OVER 
 (PARTITION BY category ORDER BY comments) rank1
 FROM t
 GROUP BY category
 GROUPING SETS ((), (category))
 HAVING max(comments)  0;
 
 return the following results:
 
 NULL1   48  1
 2   1   49  1
 3   1   49  1
 8   1   49  1
 
 When using grouping sets with the rank() function the max() function return 
 incorrect results. Everything works fine if I remove grouping sets clause and 
 split the query into two independent queries or remove the rank() function.
 
 This looks like a bug to me but please review. That said, I'm not sure if 
 it's just Amazon issue or general Hive issue.
 
 
 Diffs
 -
 
   ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 4632f08 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorGroupByOperator.java 
 90b4b12 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 
 afd1738 
   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java cea86df 
   ql/src/test/queries/clientpositive/groupby_grouping_window.q PRE-CREATION 
   ql/src/test/results/clientpositive/annotate_stats_groupby.q.out 89dd1de 
   ql/src/test/results/clientpositive/annotate_stats_groupby2.q.out c3bf0d8 
   ql/src/test/results/clientpositive/groupby_cube1.q.out 4f44d4f 
   ql/src/test/results/clientpositive/groupby_grouping_sets2.q.out 0a21dbe 
   ql/src/test/results/clientpositive/groupby_grouping_sets3.q.out 3597609 
   ql/src/test/results/clientpositive/groupby_grouping_sets4.q.out d1be46d 
   ql/src/test/results/clientpositive/groupby_grouping_sets5.q.out 6d11add 
   ql/src/test/results/clientpositive/groupby_grouping_sets6.q.out d2ff112 
   ql/src/test/results/clientpositive/groupby_rollup1.q.out 0108ce0 
   
 ql/src/test/results/clientpositive/infer_bucket_sort_grouping_operators.q.out 
 301b90c 
 
 Diff: https://reviews.apache.org/r/29878/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Navis Ryu
 

Re: Review Request 29878: Bug with max() together with rank() and grouping sets

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29878/
---

(Updated Jan. 19, 2015, 1:10 a.m.)


Review request for hive.


Changes
---

Addressed comments


Bugs: HIVE-9347
https://issues.apache.org/jira/browse/HIVE-9347


Repository: hive-git


Description
---

It looks like the query below returns incorrect results on Hive 0.13.1, but it 
was working fine on Hive 0.11. 

I have the following table:
CREATE  TABLE `t`(
  `category` int, 
  `live` int, 
  `comments` int)

with the following data:
hive select * from t;
OK
3   0   2
2   0   2
8   0   2

The query:
hive select category, max(live) live, max(comments) comments, rank() OVER 
(PARTITION BY category ORDER BY comments) rank1
FROM t
GROUP BY category
GROUPING SETS ((), (category))
HAVING max(comments)  0;

return the following results:

NULL1   48  1
2   1   49  1
3   1   49  1
8   1   49  1

When using grouping sets with the rank() function the max() function return 
incorrect results. Everything works fine if I remove grouping sets clause and 
split the query into two independent queries or remove the rank() function.

This looks like a bug to me but please review. That said, I'm not sure if it's 
just Amazon issue or general Hive issue.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 4632f08 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorGroupByOperator.java 
90b4b12 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 
afd1738 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/BucketingSortingInferenceOptimizer.java
 87fba2d 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/BucketingSortingOpProcFactory.java
 82f4243 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java b93a293 
  ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 7a0b0da 

Diff: https://reviews.apache.org/r/29878/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 30018: ReduceSinkOperator of PTFOperator can have redundant key columns

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30018/
---

Review request for hive.


Bugs: HIVE-4809
https://issues.apache.org/jira/browse/HIVE-4809


Repository: hive-git


Description
---

For example, we have a simple query like this ...
{code:sql}
SELECT x.a, x.b, count(x.b) OVER (PARTITION BY x.a) FROM src x;
{\code}

The plan of it is ...
{code}
STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 is a root stage

STAGE PLANS:
  Stage: Stage-1
Map Reduce
  Alias - Map Operator Tree:
x 
  TableScan
alias: x
Reduce Output Operator
  key expressions:
expr: a
type: int
expr: a
type: int
  sort order: ++
  Map-reduce partition columns:
expr: a
type: int
  tag: -1
  value expressions:
expr: a
type: int
expr: b
type: string
  Reduce Operator Tree:
Extract
  PTF Operator
Select Operator
  expressions:
expr: _col0
type: int
expr: _col1
type: string
expr: _wcol0
type: bigint
  outputColumnNames: _col0, _col1, _col2
  File Output Operator
compressed: false
GlobalTableId: 0
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: 
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat

  Stage: Stage-0
Fetch Operator
  limit: -1
{\code}

The ReduceSinkOperator has two a in its key columns. This redundancy can 
increase the size of map output.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java d1d866f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java cea86df 
  ql/src/test/results/clientpositive/correlationoptimizer12.q.out 015e021 
  ql/src/test/results/clientpositive/ctas_colname.q.out 079b981 
  ql/src/test/results/clientpositive/groupby_resolution.q.out 491593f 
  ql/src/test/results/clientpositive/quotedid_basic.q.out 4384faa 
  ql/src/test/results/clientpositive/spark/subquery_in.q.out d2269f3 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 4f95e3f 
  ql/src/test/results/clientpositive/subquery_in.q.out 35c8c0b 
  ql/src/test/results/clientpositive/subquery_in_having.q.out 28087dd 
  ql/src/test/results/clientpositive/subquery_notin.q.out 49fce1c 
  ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out 24be8ec 
  ql/src/test/results/clientpositive/tez/subquery_in.q.out 59a846d 
  ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out 2363b02 
  ql/src/test/results/clientpositive/union_remove_6_subq.q.out 63598f9 
  ql/src/test/results/clientpositive/vectorized_ptf.q.out 3d60d7f 
  ql/src/test/results/clientpositive/windowing_streaming.q.out 6a55352 

Diff: https://reviews.apache.org/r/30018/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 29878: Bug with max() together with rank() and grouping sets

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29878/
---

Review request for hive.


Bugs: HIVE-9347
https://issues.apache.org/jira/browse/HIVE-9347


Repository: hive-git


Description
---

It looks like the query below returns incorrect results on Hive 0.13.1, but it 
was working fine on Hive 0.11. 

I have the following table:
CREATE  TABLE `t`(
  `category` int, 
  `live` int, 
  `comments` int)

with the following data:
hive select * from t;
OK
3   0   2
2   0   2
8   0   2

The query:
hive select category, max(live) live, max(comments) comments, rank() OVER 
(PARTITION BY category ORDER BY comments) rank1
FROM t
GROUP BY category
GROUPING SETS ((), (category))
HAVING max(comments)  0;

return the following results:

NULL1   48  1
2   1   49  1
3   1   49  1
8   1   49  1

When using grouping sets with the rank() function the max() function return 
incorrect results. Everything works fine if I remove grouping sets clause and 
split the query into two independent queries or remove the rank() function.

This looks like a bug to me but please review. That said, I'm not sure if it's 
just Amazon issue or general Hive issue.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 4632f08 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorGroupByOperator.java 
90b4b12 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 
afd1738 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java cea86df 
  ql/src/test/queries/clientpositive/groupby_grouping_window.q PRE-CREATION 
  ql/src/test/results/clientpositive/annotate_stats_groupby.q.out 89dd1de 
  ql/src/test/results/clientpositive/annotate_stats_groupby2.q.out c3bf0d8 
  ql/src/test/results/clientpositive/groupby_cube1.q.out 4f44d4f 
  ql/src/test/results/clientpositive/groupby_grouping_sets2.q.out 0a21dbe 
  ql/src/test/results/clientpositive/groupby_grouping_sets3.q.out 3597609 
  ql/src/test/results/clientpositive/groupby_grouping_sets4.q.out d1be46d 
  ql/src/test/results/clientpositive/groupby_grouping_sets5.q.out 6d11add 
  ql/src/test/results/clientpositive/groupby_grouping_sets6.q.out d2ff112 
  ql/src/test/results/clientpositive/groupby_rollup1.q.out 0108ce0 
  ql/src/test/results/clientpositive/infer_bucket_sort_grouping_operators.q.out 
301b90c 

Diff: https://reviews.apache.org/r/29878/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 29800: Apply ColumnPrunning for noop PTFs

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29800/
---

Review request for hive.


Bugs: HIVE-9341
https://issues.apache.org/jira/browse/HIVE-9341


Repository: hive-git


Description
---

Currently, PTF disables CP optimization, which can make a huge burden. For 
example,
{noformat}
select p_mfgr, p_name, p_size,
rank() over (partition by p_mfgr order by p_name) as r,
dense_rank() over (partition by p_mfgr order by p_name) as dr,
sum(p_retailprice) over (partition by p_mfgr order by p_name rows between 
unbounded preceding and current row) as s1
from noop(on part 
  partition by p_mfgr
  order by p_name
  );

STAGE PLANS:
  Stage: Stage-1
Map Reduce
  Map Operator Tree:
  TableScan
alias: part
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE 
Column stats: NONE
Reduce Output Operator
  key expressions: p_mfgr (type: string), p_name (type: string)
  sort order: ++
  Map-reduce partition columns: p_mfgr (type: string)
  Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE 
Column stats: NONE
  value expressions: p_partkey (type: int), p_name (type: string), 
p_mfgr (type: string), p_brand (type: string), p_type (type: string), p_size 
(type: int), p_container (type: string), p_retailprice (type: double), 
p_comment (type: string), BLOCK__OFFSET__INSIDE__FILE (type: bigint), 
INPUT__FILE__NAME (type: string), ROW__ID (type: 
structtransactionid:bigint,bucketid:int,rowid:bigint)
...
{noformat}

There should be a generic way to discern referenced columns but before that, we 
know CP can be safely applied to noop functions.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 
afd1738 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicatePushDown.java ee7328e 

Diff: https://reviews.apache.org/r/29800/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 29753: Support split sampling for parallel order by

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29753/
---

Review request for hive.


Bugs: HIVE-9210
https://issues.apache.org/jira/browse/HIVE-9210


Repository: hive-git


Description
---

Random sampling cannot sure it covers whole range of input data. See HIVE-7661


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8264b16 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 0ccab02 
  ql/src/java/org/apache/hadoop/hive/ql/exec/PartitionKeySampler.java 96f4530 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 2227e6f 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 9f8c091 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java ac2cc86 
  ql/src/test/queries/clientpositive/parallel_orderby.q 73c3940 
  ql/src/test/results/clientpositive/parallel_orderby.q.out 2f4ac8f 

Diff: https://reviews.apache.org/r/29753/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 29498: Upgrade JavaEWAH version to allow for unsorted bitset creation

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29498/
---

Review request for hive.


Bugs: HIVE-8181
https://issues.apache.org/jira/browse/HIVE-8181


Repository: hive-git


Description
---

JavaEWAH has removed the restriction that bitsets can only be set in order in 
the latest release. 

Currently the use of {{ewah_bitmap}} UDAF requires a {{SORT BY}}.

{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
java.lang.RuntimeException: Can't set bits out of order with 
EWAHCompressedBitmap
at 
org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:824)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at 
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at 
org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at 
org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:249)
... 7 more
Caused by: java.lang.RuntimeException: Can't set bits out of order with 
EWAHCompressedBitmap
at 
{code}


Diffs
-

  pom.xml 0e30078 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFEWAHBitmapBop.java
 58ea3ba 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEWAHBitmap.java 
e4b412e 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java e3fb558 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEWAHBitmapAnd.java 
7838b54 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEWAHBitmapEmpty.java
 4a14a65 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEWAHBitmapOr.java 
d438f82 
  ql/src/test/queries/clientpositive/index_bitmap2.q 89fbe76 
  ql/src/test/queries/clientpositive/udf_bitmap_empty.q 142b248 
  ql/src/test/results/clientpositive/index_bitmap2.q.out 73c5b90 
  ql/src/test/results/clientpositive/index_bitmap3.q.out 599bf3a 
  ql/src/test/results/clientpositive/index_bitmap_auto.q.out 81c1795 
  ql/src/test/results/clientpositive/udf_bitmap_and.q.out 8c93398 
  ql/src/test/results/clientpositive/udf_bitmap_empty.q.out ca96e78 
  ql/src/test/results/clientpositive/udf_bitmap_or.q.out 43521da 

Diff: https://reviews.apache.org/r/29498/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 23351: Support direct fetch for lateral views, sub queries, etc.

[Navis Ryu]

 On Nov. 3, 2014, 10:28 p.m., John Pullokkaran wrote:
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java, 
  line 162
  https://reviews.apache.org/r/23351/diff/1/?file=626500#file626500line162
 
  Can't we use ParseContext.topToTable to get to Table given a TS 
  object instead of walking the QB tree?

Good point! Sorry for missing this comment. I'll update the patch shortly after.


- Navis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23351/#review59662
---


On July 9, 2014, 6:55 a.m., Navis Ryu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/23351/
 ---
 
 (Updated July 9, 2014, 6:55 a.m.)
 
 
 Review request for hive.
 
 
 Bugs: HIVE-5718
 https://issues.apache.org/jira/browse/HIVE-5718
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Extend HIVE-2925 with LV and SubQ.
 
 
 Diffs
 -
 
   ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 5d41fa1 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java 
 7413d2b 
   ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java 908db1e 
   ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 911ac8a 
   ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java 32d84ea 
   ql/src/test/queries/clientpositive/nonmr_fetch.q 2a92d17 
   ql/src/test/queries/clientpositive/nonmr_fetch_threshold.q e6343e2 
   ql/src/test/results/clientpositive/explain_logical.q.out bb26e8c 
   ql/src/test/results/clientpositive/lateral_view_noalias.q.out d51b2de 
   ql/src/test/results/clientpositive/nonmr_fetch.q.out 5a13e84 
   ql/src/test/results/clientpositive/nonmr_fetch_threshold.q.out 39cdfa6 
   ql/src/test/results/clientpositive/select_dummy_source.q.out 2742d56 
   ql/src/test/results/clientpositive/subquery_alias.q.out 37bc3a4 
   ql/src/test/results/clientpositive/udf_explode.q.out 4eeedeb 
   ql/src/test/results/clientpositive/udf_inline.q.out e065bed 
   ql/src/test/results/clientpositive/udf_reflect2.q.out 6b19277 
   ql/src/test/results/clientpositive/udf_to_unix_timestamp.q.out 447ef87 
   ql/src/test/results/clientpositive/udtf_explode.q.out ae95907 
 
 Diff: https://reviews.apache.org/r/23351/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Navis Ryu
 

Review Request 29390: CBO changes constant to column type

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29390/
---

Review request for hive.


Bugs: HIVE-9195
https://issues.apache.org/jira/browse/HIVE-9195


Repository: hive-git


Description
---

Making testcase for HIVE-8613, I've found CBO changes constant expr to column 
expr. For example (only in test mode).
{code}
CREATE TABLE bucket (key double, value string) CLUSTERED BY (key) SORTED BY 
(key DESC)  INTO 4 BUCKETS STORED AS TEXTFILE;
load data local inpath '../../data/files/srcsortbucket1outof4.txt' INTO TABLE 
bucket;
load data local inpath '../../data/files/srcsortbucket2outof4.txt' INTO TABLE 
bucket;
load data local inpath '../../data/files/srcsortbucket3outof4.txt' INTO TABLE 
bucket;
load data local inpath '../../data/files/srcsortbucket4outof4.txt' INTO TABLE 
bucket;

select percentile_approx(case when key  100 then cast('NaN' as double) else 
key end, 0.5) from bucket;
{code}

It works in shell but in TestCliDriver, that induces argument type exception 
creating udaf evaluator, which expects constant OI for second argument.
{noformat}
2014-12-22 17:03:31,433 ERROR parse.SemanticAnalyzer 
(SemanticAnalyzer.java:analyzeInternal(10102)) - CBO failed, skipping CBO.
org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: The second argument 
must be a constant, but double was passed instead.
at 
org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileApprox.getEvaluator(GenericUDAFPercentileApprox.java:146)
at 
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getGenericUDAFEvaluator(FunctionRegistry.java:1160)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getGenericUDAFEvaluator(SemanticAnalyzer.java:3794)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapGroupByOperator(SemanticAnalyzer.java:4467)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapAggrNoSkew(SemanticAnalyzer.java:5536)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8884)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9745)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9638)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10086)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:224)
at 
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:224)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:419)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1107)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1155)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1044)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1034)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:206)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:158)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:369)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:304)
at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:877)
at 
org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:136)
at 
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23(TestCliDriver.java:120)
{noformat}


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
 99b2950 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java bd610d9 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeColumnListDesc.java 
f8738cd 
  ql/src/test/queries/clientpositive/udaf_percentile_approx_23.q fa1dd6e 
  ql/src/test/results/clientpositive/udaf_percentile_approx_23.q.out 32ae74b 
  
serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyVoidObjectInspector.java
 d63d75a 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
 e844979 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableVoidObjectInspector.java
 02b672a 

Diff: https://reviews.apache.org/r/29390/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 29300: Merging HIVE-5871 into LazySimpleSerDe

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29300/
---

Review request for hive.


Bugs: HIVE-9172
https://issues.apache.org/jira/browse/HIVE-9172


Repository: hive-git


Description
---

Merging multi character support for field delimiter to LazySimpleSerDe


Diffs
-

  
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/AccumuloRowSerializer.java
 46c3c1a 
  
accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/TestLazyAccumuloMap.java
 2479fb4 
  
accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/mr/TestHiveAccumuloTableOutputFormat.java
 706b26e 
  
accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/serde/DelimitedAccumuloRowIdFactory.java
 4bb5419 
  
accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/serde/FirstCharAccumuloCompositeRowId.java
 f835a96 
  
accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/serde/TestAccumuloRowSerializer.java
 f613a58 
  
accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/serde/TestAccumuloSerDe.java
 18b84e4 
  
accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/serde/TestDefaultAccumuloRowIdFactory.java
 d464740 
  
common/src/java/org/apache/hadoop/hive/common/io/NonSyncByteArrayOutputStream.java
 a9f5ed6 
  contrib/src/java/org/apache/hadoop/hive/contrib/serde2/MultiDelimitSerDe.java 
72f4234 
  data/files/loc2.txt PRE-CREATION 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java 
3bbab20 
  hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java 
b2bdd19 
  
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/DelimitedInputWriter.java
 36834b1 
  ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexResult.java 332ced8 
  ql/src/test/queries/clientpositive/delimiter.q 14d508c 
  ql/src/test/results/clientpositive/delimiter.q.out 064d5df 
  
serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroLazyObjectInspector.java 
2fb1c28 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java e3968a9 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 
95e30db 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java 9a246af 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 3943508 
  
serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyObjectInspectorFactory.java
 1abd8a5 
  
serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazySimpleStructObjectInspector.java
 9611e9f 
  serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyArrayMapStruct.java 
48f3b02 

Diff: https://reviews.apache.org/r/29300/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 29037: Avoid creating serde for partitions if possible in FetchTask

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29037/
---

(Updated Dec. 18, 2014, 7:04 a.m.)


Review request for hive.


Bugs: HIVE-7977
https://issues.apache.org/jira/browse/HIVE-7977


Repository: hive-git


Description
---

Currently, FetchTask creates SerDe instance thrice for each partition, which 
can be avoided if it's same with table SerDe.


Diffs (updated)
-

  data/files/datatypes.txt 0228a27 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1355230 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 6338c3c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/PartitionKeySampler.java 5d126a5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapper.java ca65a8e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 29be691 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java 32d84ea 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java 374e8b6 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedListObjectInspector.java
 6a9215b 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedMapObjectInspector.java
 975d5cd 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedObjectInspectorFactory.java
 2db3819 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedStructObjectInspector.java
 ef66e97 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedUnionObjectInspector.java
 521fdd6 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java
 8a42577 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java
 49884b8 

Diff: https://reviews.apache.org/r/29037/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 29195: incompatFileSet in AbstractFileMergeOperator should be marked to skip task id check

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29195/
---

Review request for hive.


Bugs: HIVE-9076
https://issues.apache.org/jira/browse/HIVE-9076


Repository: hive-git


Description
---

In some file composition, AbstractFileMergeOperator removes incompatible files. 
For example,
{noformat}
00_0 (v12)
00_0_copy_1 (v12)
00_1 (v11)
00_1_copy_1 (v11)
00_1_copy_2 (v11)
00_2 (v12)
{noformat}

00_1 (v11) will be removed because 00 is assigned to new merged file.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
6c80a14 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 4170659 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java 
297ce44 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestOperators.java 9359a7a 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestUtilities.java 69f8889 
  ql/src/test/queries/clientpositive/orc_merge_incompat1.q 0348948 
  ql/src/test/results/clientpositive/orc_merge_incompat1.q.out 7a9c772 
  ql/src/test/results/clientpositive/tez/orc_merge_incompat1.q.out ea2dd5d 

Diff: https://reviews.apache.org/r/29195/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 29196: Add some explain to PTF operator

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29196/
---

Review request for hive.


Bugs: HIVE-9138
https://issues.apache.org/jira/browse/HIVE-9138


Repository: hive-git


Description
---

PTFOperator does not explain anything in explain statement, making it hard to 
understand the internal works. 


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 60d7e43 
  ql/src/java/org/apache/hadoop/hive/ql/plan/Explain.java a3408a0 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDesc.java 3ac3245 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java ae99bd3 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/BoundaryDef.java 07590c0 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PTFExpressionDef.java 5d200fb 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PTFInputDef.java 19ed2f2 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PTFQueryInputDef.java 11ef932 
  
ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PartitionedTableFunctionDef.java 
327304c 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowExpressionDef.java 
b96e9d6 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowFrameDef.java 949ed10 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowFunctionDef.java e4ea358 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowTableFunctionDef.java 
083aaf2 
  ql/src/test/results/clientpositive/correlationoptimizer12.q.out 015e021 
  ql/src/test/results/clientpositive/ctas_colname.q.out 079b981 
  ql/src/test/results/clientpositive/groupby_resolution.q.out 491593f 
  ql/src/test/results/clientpositive/quotedid_basic.q.out 4384faa 
  ql/src/test/results/clientpositive/subquery_in.q.out 35c8c0b 
  ql/src/test/results/clientpositive/subquery_in_having.q.out 28087dd 
  ql/src/test/results/clientpositive/subquery_notin.q.out 49fce1c 
  ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out 24be8ec 
  ql/src/test/results/clientpositive/tez/subquery_in.q.out 59a846d 
  ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out b140d19 
  ql/src/test/results/clientpositive/vectorized_ptf.q.out 53afcc7 
  ql/src/test/results/clientpositive/windowing_streaming.q.out 6a55352 

Diff: https://reviews.apache.org/r/29196/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 29037: Avoid creating serde for partitions if possible in FetchTask

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29037/
---

Review request for hive.


Bugs: HIVE-7977
https://issues.apache.org/jira/browse/HIVE-7977


Repository: hive-git


Description
---

Currently, FetchTask creates SerDe instance thrice for each partition, which 
can be avoided if it's same with table SerDe.


Diffs
-

  data/files/datatypes.txt 0228a27 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 395a5f5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 6338c3c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java d2b5c05 
  ql/src/java/org/apache/hadoop/hive/ql/exec/PartitionKeySampler.java 5d126a5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapper.java ca65a8e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 6c8624f 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java 32d84ea 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java 374e8b6 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedListObjectInspector.java
 6a9215b 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedMapObjectInspector.java
 975d5cd 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedObjectInspectorFactory.java
 2db3819 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedStructObjectInspector.java
 ef66e97 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedUnionObjectInspector.java
 521fdd6 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java
 8a42577 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java
 49884b8 

Diff: https://reviews.apache.org/r/29037/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 28990: ANALYZE TABLE statement fails on postgres metastore

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28990/
---

Review request for hive.


Bugs: HIVE-7351
https://issues.apache.org/jira/browse/HIVE-7351


Repository: hive-git


Description
---

Metastore code use method {{PreparedStatement.setQueryTimeout(int)}} of JDBC 
Driver :

Current JDBC driver doesn't implements this method.

{noformat}
2014-07-07 17:52:38,239 ERROR 
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher: Error during JDBC 
connection to 
jdbc:postgresql://nc-h04:5432/metastore?user=hiveuserpassword=mvsmt4521. 
org.postgresql.util.PSQLException: Method 
org.postgresql.jdbc4.Jdbc4PreparedStatement.setQueryTimeout(int) is not yet 
implemented.
at org.postgresql.Driver.notImplemented(Driver.java:753)
at 
org.postgresql.jdbc2.AbstractJdbc2Statement.setQueryTimeout(AbstractJdbc2Statement.java:666)
at 
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$1.run(JDBCStatsPublisher.java:80)
at 
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$1.run(JDBCStatsPublisher.java:77)
at 
org.apache.hadoop.hive.ql.exec.Utilities.executeWithRetry(Utilities.java:2637)
at 
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher.connect(JDBCStatsPublisher.java:96)
at 
org.apache.hadoop.hive.ql.exec.TableScanOperator.publishStats(TableScanOperator.java:280)
at 
org.apache.hadoop.hive.ql.exec.TableScanOperator.closeOp(TableScanOperator.java:226)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:583)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:595)
at 
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:227)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
{noformat}


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 4170659 
  ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java 
d6067e7 
  ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java 
c1621e0 

Diff: https://reviews.apache.org/r/28990/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 28982: Data loading from text files or text file processing doesn't handle nulls correctly

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28982/
---

Review request for hive.


Bugs: HIVE-8848
https://issues.apache.org/jira/browse/HIVE-8848


Repository: hive-git


Description
---

I am not sure how nulls are supposed to be stored in text tables, but after 
loading some data with null or NULL strings, or x00 characters, we get 
bunch of annoying logging from LazyPrimitive that data is not in INT format and 
was converted to null, with data being null (string saying null, I assume 
from the code).
Either load should load them as nulls, or there should be some defined way to 
load nulls.


Diffs
-

  
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/LazyAccumuloMap.java 
effdc4b 
  
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/LazyAccumuloRow.java 
3dc2d3c 
  
accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/serde/FirstCharAccumuloCompositeRowId.java
 8c1ec8d 
  data/files/cbo_t1.txt e8034a4 
  data/files/cbo_t2.txt 34633d3 
  data/files/cbo_t3.txt a9c995e 
  data/files/cbo_t4.txt 56e0794 
  data/files/cbo_t5.txt 245b1b9 
  data/files/cbo_t6.txt dd72edd 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java 
2727b36 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java 8a3eac7 
  
hbase-handler/src/test/org/apache/hadoop/hive/hbase/SampleHBaseKeyFactory.java 
2d647e2 
  
hbase-handler/src/test/org/apache/hadoop/hive/hbase/SampleHBaseKeyFactory2.java 
7086d57 
  ql/src/test/results/clientpositive/cbo_gby.q.out 7344c11 
  ql/src/test/results/clientpositive/cbo_join.q.out e4e4e25 
  ql/src/test/results/clientpositive/cbo_limit.q.out f4385fa 
  ql/src/test/results/clientpositive/cbo_semijoin.q.out b7f6be4 
  ql/src/test/results/clientpositive/cbo_simple_select.q.out 87b85cf 
  ql/src/test/results/clientpositive/cbo_udf_udaf.q.out ed3a3e7 
  ql/src/test/results/clientpositive/cbo_union.q.out eb02b03 
  ql/src/test/results/clientpositive/cbo_views.q.out 10fa8c8 
  ql/src/test/results/clientpositive/cbo_windowing.q.out a96def8 
  ql/src/test/results/clientpositive/tez/cbo_gby.q.out 7344c11 
  ql/src/test/results/clientpositive/tez/cbo_join.q.out e4e4e25 
  ql/src/test/results/clientpositive/tez/cbo_limit.q.out f4385fa 
  ql/src/test/results/clientpositive/tez/cbo_semijoin.q.out b7f6be4 
  ql/src/test/results/clientpositive/tez/cbo_simple_select.q.out 87b85cf 
  ql/src/test/results/clientpositive/tez/cbo_udf_udaf.q.out ed3a3e7 
  ql/src/test/results/clientpositive/tez/cbo_union.q.out eb02b03 
  ql/src/test/results/clientpositive/tez/cbo_views.q.out 10fa8c8 
  ql/src/test/results/clientpositive/tez/cbo_windowing.q.out a96def8 
  
serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroLazyObjectInspector.java 
a6e8c2c 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyArray.java ce0cfb3 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyBinary.java c25a1b8 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyMap.java e6932d9 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyNonPrimitive.java 
e4cffc9 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyObject.java 9b5ccbe 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyObjectBase.java 7e42b3f 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyPrimitive.java 7d23c46 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyString.java 75b9556 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java 588cc8c 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUnion.java 22fc638 
  serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryObject.java 
b3625b3 

Diff: https://reviews.apache.org/r/28982/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 28988: hive.default.fileformat should accept all formats described by StorageFormatDescriptor

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28988/
---

Review request for hive.


Bugs: HIVE-8591
https://issues.apache.org/jira/browse/HIVE-8591


Repository: hive-git


Description
---

FileFormats are described by StorageFormatDescriptor, which is added in 
HIVE-5976. Validator for FileFormats should reflect that also.


Diffs
-

  common/pom.xml ede8aea 
  common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2e2bf5a 
  ql/pom.xml 515d235 
  ql/src/java/org/apache/hadoop/hive/ql/io/StorageFormatFactory.java e0bf153 

Diff: https://reviews.apache.org/r/28988/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 26966: Path type entities should use qualified path rather than string

[Navis Ryu]

 On Oct. 21, 2014, 5:04 p.m., Thejas Nair wrote:
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java, line 
  236
  https://reviews.apache.org/r/26966/diff/1/?file=727028#file727028line236
 
  if the user has specified using syntax that this is a local file path, 
  should we use that information instead of trying to automatically 
  determine/guess that ?
 
 Navis Ryu wrote:
 I've missed this comment. Imho, we should throw exception when user 
 specified non-local path with local keyword. Should we?
 
 Thejas Nair wrote:
 Yes, that would be a good solution.

Ah, sorry. It's already in LoadSemanticAnalyzer.applyConstraints().

```java
if (isLocal  !fromURI.getScheme().equals(file)) {
  throw new SemanticException(ErrorMsg.ILLEGAL_PATH.getMsg(ast,
Source file system should be \file\ if \local\ is specified));
}
```


- Navis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26966/#review57607
---


On Oct. 21, 2014, 6:55 a.m., Navis Ryu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/26966/
 ---
 
 (Updated Oct. 21, 2014, 6:55 a.m.)
 
 
 Review request for hive.
 
 
 Bugs: HIVE-8357
 https://issues.apache.org/jira/browse/HIVE-8357
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Path type entities should use qualified path rather than string
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/common/FileUtils.java 95e8d7c 
   
 itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestCreateUdfEntities.java
  c60f856 
   ql/src/java/org/apache/hadoop/hive/ql/Driver.java e254505 
   ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java 76b1f01 
   ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java 8b6a923 
   ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java c5be822 
   ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
 37cbf7f 
   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
 c060d74 
   ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java 
 f96209c 
   ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 
 22e5b47 
   ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
 4891518 
   ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
 8bd24d3 
   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1b7a41d 
   ql/src/test/results/clientnegative/udf_local_resource.q.out 2f2227b 
   ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out e184787 
   ql/src/test/results/clientpositive/add_part_multiple.q.out 975baa5 
   ql/src/test/results/clientpositive/alter2.q.out 3356ab9 
   ql/src/test/results/clientpositive/alter5.q.out a83b68d 
   ql/src/test/results/clientpositive/exim_17_part_managed.q.out 9036a28 
 
 Diff: https://reviews.apache.org/r/26966/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Navis Ryu
 

Re: Review Request 26966: Path type entities should use qualified path rather than string

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26966/
---

(Updated Dec. 12, 2014, 7:31 a.m.)


Review request for hive.


Bugs: HIVE-8357
https://issues.apache.org/jira/browse/HIVE-8357


Repository: hive-git


Description
---

Path type entities should use qualified path rather than string


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/FileUtils.java 3c34d7a 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestCreateUdfEntities.java
 c60f856 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 395a5f5 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java 76b1f01 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java 8b6a923 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java fc8afc0 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java f33abb4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java d2c4dc0 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java 
f96209c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 
01b97c8 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
eda7984 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 1ae5b71 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 6f44f9c 
  ql/src/test/results/clientnegative/udf_local_resource.q.out 2f2227b 
  ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out e184787 
  ql/src/test/results/clientpositive/add_part_multiple.q.out 975baa5 
  ql/src/test/results/clientpositive/alter2.q.out cce6e0a 
  ql/src/test/results/clientpositive/alter5.q.out 583be83 
  ql/src/test/results/clientpositive/exim_17_part_managed.q.out 9036a28 

Diff: https://reviews.apache.org/r/26966/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 26966: Path type entities should use qualified path rather than string

[Navis Ryu]

 On Dec. 9, 2014, 8:41 p.m., Mohit Sabharwal wrote:
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java, line 
  236
  https://reviews.apache.org/r/26966/diff/1/?file=727028#file727028line236
 
  Looks like FileUtils.isLocalFile is a 'best effort' check based on URI 
  format. So probably better to just trust the user ?

As commented above, path should have 'file' prefix for 'local' path. I think 
changing behaviour seemed need more debate in other issue.


- Navis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26966/#review64432
---


On Dec. 12, 2014, 7:31 a.m., Navis Ryu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/26966/
 ---
 
 (Updated Dec. 12, 2014, 7:31 a.m.)
 
 
 Review request for hive.
 
 
 Bugs: HIVE-8357
 https://issues.apache.org/jira/browse/HIVE-8357
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Path type entities should use qualified path rather than string
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/common/FileUtils.java 3c34d7a 
   
 itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestCreateUdfEntities.java
  c60f856 
   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 395a5f5 
   ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java 76b1f01 
   ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java 8b6a923 
   ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java fc8afc0 
   ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
 f33abb4 
   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
 d2c4dc0 
   ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java 
 f96209c 
   ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 
 01b97c8 
   ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
 eda7984 
   ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
 1ae5b71 
   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 6f44f9c 
   ql/src/test/results/clientnegative/udf_local_resource.q.out 2f2227b 
   ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out e184787 
   ql/src/test/results/clientpositive/add_part_multiple.q.out 975baa5 
   ql/src/test/results/clientpositive/alter2.q.out cce6e0a 
   ql/src/test/results/clientpositive/alter5.q.out 583be83 
   ql/src/test/results/clientpositive/exim_17_part_managed.q.out 9036a28 
 
 Diff: https://reviews.apache.org/r/26966/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Navis Ryu
 

Re: Review Request 26966: Path type entities should use qualified path rather than string

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26966/
---

(Updated Dec. 12, 2014, 7:32 a.m.)


Review request for hive.


Bugs: HIVE-8357
https://issues.apache.org/jira/browse/HIVE-8357


Repository: hive-git


Description
---

Path type entities should use qualified path rather than string


Diffs
-

  common/src/java/org/apache/hadoop/hive/common/FileUtils.java 3c34d7a 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestCreateUdfEntities.java
 c60f856 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 395a5f5 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java 76b1f01 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java 8b6a923 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java fc8afc0 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java f33abb4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java d2c4dc0 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java 
f96209c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 
01b97c8 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
eda7984 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 1ae5b71 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 6f44f9c 
  ql/src/test/results/clientnegative/udf_local_resource.q.out 2f2227b 
  ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out e184787 
  ql/src/test/results/clientpositive/add_part_multiple.q.out 975baa5 
  ql/src/test/results/clientpositive/alter2.q.out cce6e0a 
  ql/src/test/results/clientpositive/alter5.q.out 583be83 
  ql/src/test/results/clientpositive/exim_17_part_managed.q.out 9036a28 

Diff: https://reviews.apache.org/r/26966/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 26966: Path type entities should use qualified path rather than string

[Navis Ryu]

 On Dec. 9, 2014, 8:31 p.m., Mohit Sabharwal wrote:
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java, line 556
  https://reviews.apache.org/r/26966/diff/1/?file=727019#file727019line556
 
  nit: while we're here, this code will be cleaner if we replace the many 
  continues with bunch of else if {} blocks or, better, a switch {}

Introduces Entity.isPathType(), which seemed cleaner enough.


- Navis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26966/#review64424
---


On Dec. 12, 2014, 7:31 a.m., Navis Ryu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/26966/
 ---
 
 (Updated Dec. 12, 2014, 7:31 a.m.)
 
 
 Review request for hive.
 
 
 Bugs: HIVE-8357
 https://issues.apache.org/jira/browse/HIVE-8357
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Path type entities should use qualified path rather than string
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/common/FileUtils.java 3c34d7a 
   
 itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestCreateUdfEntities.java
  c60f856 
   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 395a5f5 
   ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java 76b1f01 
   ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java 8b6a923 
   ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java fc8afc0 
   ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
 f33abb4 
   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
 d2c4dc0 
   ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java 
 f96209c 
   ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 
 01b97c8 
   ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
 eda7984 
   ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
 1ae5b71 
   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 6f44f9c 
   ql/src/test/results/clientnegative/udf_local_resource.q.out 2f2227b 
   ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out e184787 
   ql/src/test/results/clientpositive/add_part_multiple.q.out 975baa5 
   ql/src/test/results/clientpositive/alter2.q.out cce6e0a 
   ql/src/test/results/clientpositive/alter5.q.out 583be83 
   ql/src/test/results/clientpositive/exim_17_part_managed.q.out 9036a28 
 
 Diff: https://reviews.apache.org/r/26966/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Navis Ryu
 

Re: Review Request 25628: alias of compound aggregation functions fails in having clause

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25628/
---

(Updated Dec. 12, 2014, 7:57 a.m.)


Review request for hive.


Bugs: HIVE-7981
https://issues.apache.org/jira/browse/HIVE-7981


Repository: hive-git


Description
---

hive select max(time)-min(time) as span from mytable group by name having 
span0;

FAILED: SemanticException [Error 10025]: Line 1:92 Expression not in GROUP BY 
key '0'


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 02c4be9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 6f44f9c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckCtx.java b19e2bf 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java bd610d9 
  ql/src/test/queries/clientpositive/having3.q PRE-CREATION 
  ql/src/test/results/clientpositive/having3.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/25628/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 28936: Set completer in CliDriver is not working

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28936/
---

Review request for hive.


Bugs: HIVE-9077
https://issues.apache.org/jira/browse/HIVE-9077


Repository: hive-git


Description
---

NO PRECOMMIT TESTS

Seemed broken in HIVE-8609
{noformat}
hive (default) set Exception in thread main java.lang.NullPointerException
at 
jline.console.completer.ArgumentCompleter$AbstractArgumentDelimiter.delimit(ArgumentCompleter.java:283)
at 
jline.console.completer.ArgumentCompleter.complete(ArgumentCompleter.java:116)
at 
jline.console.completer.ArgumentCompleter.complete(ArgumentCompleter.java:152)
at org.apache.hadoop.hive.cli.CliDriver$6.complete(CliDriver.java:567)
at jline.console.ConsoleReader.complete(ConsoleReader.java:3261)
at jline.console.ConsoleReader.readLine(ConsoleReader.java:2621)
at jline.console.ConsoleReader.readLine(ConsoleReader.java:2269)
at 
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:714)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:639)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:578)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
{noformat}


Diffs
-

  cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java aec5018 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2e2bf5a 
  common/src/java/org/apache/hadoop/hive/conf/Validator.java bb0f836 

Diff: https://reviews.apache.org/r/28936/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 28884: Provide compile + run method for HiveServer2

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28884/
---

Review request for hive.


Bugs: HIVE-4923
https://issues.apache.org/jira/browse/HIVE-4923


Repository: hive-git


Description
---

Make additional thrift method for asynchronous execution.

execute=compile+run

This addresses same thing with HIVE-4569 but with simpler implementation. It's 
just for reference (We are using this for months).


Diffs
-

  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
23a1b97 
  jdbc/pom.xml 6385093 
  jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java d8e33d3 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 395a5f5 
  ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java 8e1e6e2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 4170659 
  ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 4e2b130 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1b5864e 
  service/if/TCLIService.thrift 4024bb3 
  service/src/gen/thrift/gen-cpp/TCLIService.h 030475b 
  service/src/gen/thrift/gen-cpp/TCLIService.cpp 209ce63 
  service/src/gen/thrift/gen-cpp/TCLIService_server.skeleton.cpp 988bb4c 
  service/src/gen/thrift/gen-cpp/TCLIService_types.h f32dc3c 
  service/src/gen/thrift/gen-cpp/TCLIService_types.cpp 326d25b 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCLIService.java
 54851b8 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCompileRes.java
 PRE-CREATION 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRunReq.java
 PRE-CREATION 
  
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TTypeQualifiers.java
 393 
  service/src/gen/thrift/gen-php/TCLIService.php d246296 
  service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote f6ff43f 
  service/src/gen/thrift/gen-py/TCLIService/TCLIService.py ebc6574 
  service/src/gen/thrift/gen-py/TCLIService/ttypes.py 6cd64d0 
  service/src/gen/thrift/gen-rb/t_c_l_i_service.rb fd1ca9a 
  service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb c731544 
  service/src/java/org/apache/hive/service/CompileResult.java PRE-CREATION 
  service/src/java/org/apache/hive/service/cli/CLIService.java 98cf6f8 
  service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 
9cad5be 
  service/src/java/org/apache/hive/service/cli/ICLIService.java c9cc1f4 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
74d7d53 
  service/src/java/org/apache/hive/service/cli/session/HiveSession.java 6359a5b 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
cabf32a 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
3345b5f 
  
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java 
1af4539 

Diff: https://reviews.apache.org/r/28884/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 28837: SessionState.getUserFromAuthenticator() should be used instead of SessionState.getName()

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28837/
---

Review request for hive.


Bugs: HIVE-8832
https://issues.apache.org/jira/browse/HIVE-8832


Repository: hive-git


Description
---

It's not valid sometimes.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 395a5f5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125 
  
ql/src/java/org/apache/hadoop/hive/ql/security/HadoopDefaultAuthenticator.java 
18e4e00 
  
ql/src/java/org/apache/hadoop/hive/ql/security/SessionStateUserAuthenticator.java
 a77e93f 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveV1Authorizer.java
 ac1cc47 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1b5864e 

Diff: https://reviews.apache.org/r/28837/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 26966: Path type entities should use qualified path rather than string

[Navis Ryu]

 On Oct. 21, 2014, 5:04 p.m., Thejas Nair wrote:
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java, line 
  236
  https://reviews.apache.org/r/26966/diff/1/?file=727028#file727028line236
 
  if the user has specified using syntax that this is a local file path, 
  should we use that information instead of trying to automatically 
  determine/guess that ?

I've missed this comment. Imho, we should throw exception when user specified 
non-local path with local keyword. Should we?


- Navis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26966/#review57607
---


On Oct. 21, 2014, 6:55 a.m., Navis Ryu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/26966/
 ---
 
 (Updated Oct. 21, 2014, 6:55 a.m.)
 
 
 Review request for hive.
 
 
 Bugs: HIVE-8357
 https://issues.apache.org/jira/browse/HIVE-8357
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Path type entities should use qualified path rather than string
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/common/FileUtils.java 95e8d7c 
   
 itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestCreateUdfEntities.java
  c60f856 
   ql/src/java/org/apache/hadoop/hive/ql/Driver.java e254505 
   ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java 76b1f01 
   ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java 8b6a923 
   ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java c5be822 
   ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
 37cbf7f 
   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
 c060d74 
   ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java 
 f96209c 
   ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 
 22e5b47 
   ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
 4891518 
   ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
 8bd24d3 
   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1b7a41d 
   ql/src/test/results/clientnegative/udf_local_resource.q.out 2f2227b 
   ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out e184787 
   ql/src/test/results/clientpositive/add_part_multiple.q.out 975baa5 
   ql/src/test/results/clientpositive/alter2.q.out 3356ab9 
   ql/src/test/results/clientpositive/alter5.q.out a83b68d 
   ql/src/test/results/clientpositive/exim_17_part_managed.q.out 9036a28 
 
 Diff: https://reviews.apache.org/r/26966/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Navis Ryu
 

Review Request 28839: Support custom virtual columns from serde implementation

[Navis Ryu]

/NullStructSerDe.java 83f34ce 
  serde/src/java/org/apache/hadoop/hive/serde2/OpenCSVSerde.java 44b5ae7 
  serde/src/java/org/apache/hadoop/hive/serde2/RecordIdentifier.java 
PRE-CREATION 
  serde/src/java/org/apache/hadoop/hive/serde2/RegexSerDe.java 38e8b77 
  serde/src/java/org/apache/hadoop/hive/serde2/TypedSerDe.java f868b22 
  serde/src/java/org/apache/hadoop/hive/serde2/VirtualColumn.java PRE-CREATION 
  serde/src/java/org/apache/hadoop/hive/serde2/VirtualColumnProvider.java 
PRE-CREATION 
  serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 7639a2b 
  
serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java
 2b7fba6 
  serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDeBase.java 
77c1e2f 
  serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 
21cbd90 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 
95e30db 
  serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 
62cba01 
  serde/src/java/org/apache/hadoop/hive/serde2/thrift/ThriftDeserializer.java 
b78ec69 

Diff: https://reviews.apache.org/r/28839/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 15855: Check DATABASES/TABLE privilege for metastore authorizer

[Navis Ryu]

 On May 24, 2014, 1:17 a.m., Ashutosh Chauhan wrote:
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, 
  line 830
  https://reviews.apache.org/r/15855/diff/1/?file=391130#file391130line830
 
  Technically, its a post event since event = ObjectStore call already 
  happened on previous line.

Yes, technically it is post event. But if the result is not delivered to 
client, can we regard that as pre-event? It whould be meaningless to authorize 
without databases/tables to be used. Should we introduce post-event listeners?


 On May 24, 2014, 1:17 a.m., Ashutosh Chauhan wrote:
  ql/src/java/org/apache/hadoop/hive/ql/security/authorization/AuthorizationPreEventListener.java,
   line 246
  https://reviews.apache.org/r/15855/diff/1/?file=391134#file391134line246
 
  If user is not authorized for show databases, than authorization 
  exception should be thrown instead of not showing that database.

If it's some real access(read or write) to the database or table, it's 
definitely right to throw exception (in that action). But this is just for 
showing/listing legitimated databases/tables for the user. Isn't it?


- Navis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15855/#review43888
---


On Nov. 26, 2013, 8:49 a.m., Navis Ryu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/15855/
 ---
 
 (Updated Nov. 26, 2013, 8:49 a.m.)
 
 
 Review request for hive.
 
 
 Bugs: HIVE-5890
 https://issues.apache.org/jira/browse/HIVE-5890
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Commands like show database are not checked in metastore event listener.
 
 
 Diffs
 -
 
   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
 01c2626 
   
 metastore/src/java/org/apache/hadoop/hive/metastore/events/PreEventContext.java
  5021a73 
   
 metastore/src/java/org/apache/hadoop/hive/metastore/events/PreGetDatabasesEvent.java
  PRE-CREATION 
   
 metastore/src/java/org/apache/hadoop/hive/metastore/events/PreGetTablesEvent.java
  PRE-CREATION 
   
 ql/src/java/org/apache/hadoop/hive/ql/security/authorization/AuthorizationPreEventListener.java
  9a90549 
 
 Diff: https://reviews.apache.org/r/15855/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Navis Ryu
 

Review Request 28274: Refactoring of PassThroughOutputFormat

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28274/
---

Review request for hive.


Bugs: HIVE-8910
https://issues.apache.org/jira/browse/HIVE-8910


Repository: hive-git


Description
---

It's overly complicated just for doing simple wrapping of output format. Before 
things get more worse, we should refactor this codes.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 4fb30bc 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 913288f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/RowContainer.java 
5271e91 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 2ac60c0 
  ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughOutputFormat.java 
04eff93 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 13277a9 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 211ab6c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
4891518 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 016a6d8 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java deba198 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java 10c38d9 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java fdc1f62 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java 408ddf5 

Diff: https://reviews.apache.org/r/28274/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 27367: support ISO-2012 timestamp literals

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27367/
---

(Updated Oct. 31, 2014, 5:03 a.m.)


Review request for hive.


Changes
---

Added more tests (shamelessly copied from partition_date.q)
Currently, direct SQL cannot be applied (tried but casting to timestamp is not 
supported in some database(mysql, etc.))


Bugs: HIVE-3187
https://issues.apache.org/jira/browse/HIVE-3187


Repository: hive-git


Description
---

Enable the JDBC driver/Hive SQL engine to accept JDBC canonical or ISO-SQL 20xx 
Timestamp literals

ie.
select 1 from cert.tversion tversion where timestamp '1989-01-01 
10:20:30.0'  timestamp '2000-12-31 12:15:30.12300'

instead of
unix_timestamp('.)


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g c903e8f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 13d5255 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java e065983 
  ql/src/test/queries/clientnegative/timestamp_literal.q PRE-CREATION 
  ql/src/test/queries/clientpositive/partition_timestamp.q PRE-CREATION 
  ql/src/test/queries/clientpositive/partition_timestamp2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/timestamp_literal.q PRE-CREATION 
  ql/src/test/results/clientnegative/date_literal2.q.out 82f6425 
  ql/src/test/results/clientnegative/date_literal3.q.out 82f6425 
  ql/src/test/results/clientnegative/illegal_partition_type4.q.out e388086 
  ql/src/test/results/clientnegative/timestamp_literal.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/partition_timestamp.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/partition_timestamp2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/timestamp_literal.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/27367/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 26854: HIVE-2573 Create per-session function registry

[Navis Ryu]

 On Oct. 23, 2014, 9:50 p.m., Jason Dere wrote:
  ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java, line 465
  https://reviews.apache.org/r/26854/diff/1-3/?file=723909#file723909line465
 
  There is no longer a way to query the metastore for UDFs apart from the 
  static initialization. So if one CLI user creates a permanent UDF, another 
  user on CLI, or HS2, will not be able to use that new UDF if the 2nd CLI or 
  HS2 was initialized before this UDF was created.
 
 Navis Ryu wrote:
 Permanent functions (persistent function seemed better name, imho) are 
 registered to system registry, which is shared to all clients. So if one user 
 creates new permanent function, it's shared to all clients. The time a user 
 accesses the function, the class is loaded with required resources and 
 registered to session registry as a temporary function.
 
 Jason Dere wrote:
 So this would work if all clients are using hiveserver2, because all 
 clients in this scenario would share the same system registry.
 But if one or more clients are using the Hive CLI, any persistent UDFs 
 created/dropped by this CLI client would not be reflected in the other 
 clients (or HS2), since it's a different process/system registry.

I've missed this message. Yes, it will act like you've commented. But is it a 
common case to expect things done in separate VM refleceted to others? Should 
dropping presistent function by a jdbc client be reflected to other CLI clients?


- Navis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review57952
---


On Oct. 30, 2014, 11:41 p.m., Jason Dere wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/26854/
 ---
 
 (Updated Oct. 30, 2014, 11:41 p.m.)
 
 
 Review request for hive, Navis Ryu and Thejas Nair.
 
 
 Bugs: HIVE-2573
 https://issues.apache.org/jira/browse/HIVE-2573
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Small updates to Navis' changes:
 - session registry doesn't lookup metastore for UDFs
 - my feedback from Navis' original patch
 - metastore udfs should not be considered native. This allows them to be 
 added/removed from registry
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7d8e5bc 
   contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3 
   
 itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java
  6647ce5 
   
 metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
  88b0791 
   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 9ac540e 
   ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0 
   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b 
   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java e43a328 
   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 5bdeb92 
   ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05 
   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 4e3df75 
   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java b900627 
   ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 13277a9 
   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 211ab6c 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java e2768ff 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/SqlFunctionConverter.java
  793f117 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 
 1796b7b 
   ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 
 22e5b47 
   ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java 2b239ab 
   ql/src/java/org/apache/hadoop/hive/ql/session/SessionConf.java PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java af633cb 
   ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 
 46f8052 
   ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb 
   ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out 
 c7405ed 
   ql/src/test/results/clientnegative/create_function_nonudf_class.q.out 
 d0dd50a 
   ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5 
   ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out e184787 
   service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java 
 fd38907 
 
 Diff: https://reviews.apache.org/r/26854/diff/
 
 
 Testing

Re: Review Request 26854: HIVE-2573 Create per-session function registry

[Navis Ryu]

 On Oct. 31, 2014, 1:32 a.m., Jason Dere wrote:
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java, line 1594
  https://reviews.apache.org/r/26854/diff/3-5/?file=729496#file729496line1594
 
  Are these changes meant for this Jira?
  
  Hmm, I think I see why - have the persistent function lookups done 
  during static initialization now caused the Hive class to not be 
  instantiable during runtime? Could you try moving the persistent function 
  lookups out of static initialization, and into a method, which gets called 
  (but only initialized once) during SessionState.start()? Would that take 
  care of the issue?

Hive.class is an access point to metastore, which should not be referenced from 
runtime classes like Table, Task, etc. Theses changes are for that. Decoupling 
persistent function registration code parts from static initializer seemed good 
idea. I'll try that.


 On Oct. 31, 2014, 1:32 a.m., Jason Dere wrote:
  service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java,
   line 134
  https://reviews.apache.org/r/26854/diff/3-5/?file=729506#file729506line134
 
  What's the issue here?

I tried but couldn't make run MR in hadoop-2. Any idea?


 On Oct. 31, 2014, 1:32 a.m., Jason Dere wrote:
  itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java,
   line 176
  https://reviews.apache.org/r/26854/diff/5/?file=745070#file745070line176
 
  Is this change relevant to this Jira? Or is this just a general fix to 
  TestJdbcWithMiniKdc.testNegativeTokenAuth, which I have noticed to be 
  failing consistently in the precommit tests?

This is my bad from HIVE-8186. Chainging error message will make this test 
pass, but HIVE-8481 would be better place to to. I'll removed this part.


 On Oct. 31, 2014, 1:32 a.m., Jason Dere wrote:
  metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java,
   line 144
  https://reviews.apache.org/r/26854/diff/5/?file=745071#file745071line144
 
  Are the changes in this file meant for this Jira?

This is called from static initializer in Hive(class). ObjectStore always 
returns collection but it's not true in this class. Can be fixed by checking 
null but seemed better to fix this for consistency.


 On Oct. 31, 2014, 1:32 a.m., Jason Dere wrote:
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java, line 399
  https://reviews.apache.org/r/26854/diff/5/?file=745080#file745080line399
 
  Was this line meant to be in here?

Yes, my bad. The remnant of futile attempts to run TestHiveServerSessions in 
hadoop-2. Will be removed.


- Navis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review59281
---


On Oct. 30, 2014, 11:41 p.m., Jason Dere wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/26854/
 ---
 
 (Updated Oct. 30, 2014, 11:41 p.m.)
 
 
 Review request for hive, Navis Ryu and Thejas Nair.
 
 
 Bugs: HIVE-2573
 https://issues.apache.org/jira/browse/HIVE-2573
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Small updates to Navis' changes:
 - session registry doesn't lookup metastore for UDFs
 - my feedback from Navis' original patch
 - metastore udfs should not be considered native. This allows them to be 
 added/removed from registry
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7d8e5bc 
   contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3 
   
 itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java
  6647ce5 
   
 metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
  88b0791 
   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 9ac540e 
   ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0 
   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b 
   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java e43a328 
   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 5bdeb92 
   ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05 
   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 4e3df75 
   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java b900627 
   ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 13277a9 
   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 211ab6c 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java e2768ff

Review Request 27367: support ISO-2012 timestamp literals

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27367/
---

Review request for hive.


Bugs: HIVE-3187
https://issues.apache.org/jira/browse/HIVE-3187


Repository: hive-git


Description
---

Enable the JDBC driver/Hive SQL engine to accept JDBC canonical or ISO-SQL 20xx 
Timestamp literals

ie.
select 1 from cert.tversion tversion where timestamp '1989-01-01 
10:20:30.0'  timestamp '2000-12-31 12:15:30.12300'

instead of
unix_timestamp('.)


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g c903e8f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 13d5255 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java e065983 
  ql/src/test/queries/clientnegative/timestamp_literal.q PRE-CREATION 
  ql/src/test/queries/clientpositive/timestamp_literal.q PRE-CREATION 
  ql/src/test/results/clientnegative/date_literal2.q.out 82f6425 
  ql/src/test/results/clientnegative/date_literal3.q.out 82f6425 
  ql/src/test/results/clientnegative/illegal_partition_type4.q.out e388086 
  ql/src/test/results/clientnegative/timestamp_literal.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/timestamp_literal.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/27367/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 22903: Extend join transitivity PPD to non-column expressions

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22903/
---

(Updated Oct. 29, 2014, 1:24 a.m.)


Review request for hive.


Changes
---

Fixed test case (auto_join33)


Bugs: HIVE-7111
https://issues.apache.org/jira/browse/HIVE-7111


Repository: hive-git


Description
---

Join transitive in PPD only supports column expressions, but it's possible to 
extend this to generic expressions.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java 686befd 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/ExprWalkerInfo.java f7a3f1c 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java ffaf5cc 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicatePushDown.java bf3a5d7 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java 
1476e1a 
  ql/src/test/queries/clientpositive/auto_join33.q PRE-CREATION 
  ql/src/test/results/clientpositive/auto_join33.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/vector_mapjoin_reduce.q.out e48775e 

Diff: https://reviews.apache.org/r/22903/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 27225: HCatPartition needs getPartCols method

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27225/
---

Review request for hive.


Bugs: HIVE-7408
https://issues.apache.org/jira/browse/HIVE-7408


Repository: hive-git


Description
---

org.apache.hive.hcatalog.api.HCatPartition has getColumns method. However, it 
is not partition column. HCatPartition needs getPartCols method.


Diffs
-

  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/data/schema/HCatSchemaUtils.java
 16c1604 
  
hcatalog/webhcat/java-client/src/main/java/org/apache/hive/hcatalog/api/HCatClientHMSImpl.java
 b3afa72 
  
hcatalog/webhcat/java-client/src/main/java/org/apache/hive/hcatalog/api/HCatPartition.java
 52586a0 
  
hcatalog/webhcat/java-client/src/test/java/org/apache/hive/hcatalog/api/TestHCatClient.java
 f9f7b04 

Diff: https://reviews.apache.org/r/27225/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 26854: HIVE-2573 Create per-session function registry

[Navis Ryu]

 On Oct. 23, 2014, 9:50 p.m., Jason Dere wrote:
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java, line 42
  https://reviews.apache.org/r/26854/diff/1-3/?file=723906#file723906line42
 
  Can we replace isNative/isPersistent with an enum that has BUILTIN, 
  PERMANENT, TEMPORARY (or equivalent terms)?

Sure.


 On Oct. 23, 2014, 9:50 p.m., Jason Dere wrote:
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java, line 128
  https://reviews.apache.org/r/26854/diff/1-3/?file=723908#file723908line128
 
  Why was this check removed? If you are permanent UDFs with Hive CLI, 
  you would have to make sure the UDF resources are available from the 
  cluster as opposed to just on the local filesystem of the client that 
  created the UDF.

Seemed removed by mistake. I'll revert that.


 On Oct. 23, 2014, 9:50 p.m., Jason Dere wrote:
  ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java, line 433
  https://reviews.apache.org/r/26854/diff/1-3/?file=723909#file723909line433
 
  I thought builtin functions aren't allowed to be removed?
  Does this mean that we could create a function using the same class as 
  a built-in function (create a synonym), and deleting this new function will 
  cause this class to be removed from the builtin set?

This should be removed. Thanks.


 On Oct. 23, 2014, 9:50 p.m., Jason Dere wrote:
  ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java, line 465
  https://reviews.apache.org/r/26854/diff/1-3/?file=723909#file723909line465
 
  There is no longer a way to query the metastore for UDFs apart from the 
  static initialization. So if one CLI user creates a permanent UDF, another 
  user on CLI, or HS2, will not be able to use that new UDF if the 2nd CLI or 
  HS2 was initialized before this UDF was created.

Permanent functions (persistent function seemed better name, imho) are 
registered to system registry, which is shared to all clients. So if one user 
creates new permanent function, it's shared to all clients. The time a user 
accesses the function, the class is loaded with required resources and 
registered to session registry as a temporary function.


 On Oct. 23, 2014, 9:50 p.m., Jason Dere wrote:
  ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java, line 511
  https://reviews.apache.org/r/26854/diff/1-3/?file=723909#file723909line511
 
  I think I see what you're trying do here, trying to add a mechanism so 
  that if a function is deleted in one session, the other sessions will also 
  see it as discarded if they try to look it up. But I don't actually see 
  discarded being set to true.

My mistake. Fixed.


 On Oct. 23, 2014, 9:50 p.m., Jason Dere wrote:
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java, line 152
  https://reviews.apache.org/r/26854/diff/3/?file=729496#file729496line152
 
  I don't think it's necessary to pre-emptively query the metastore for 
  permanent UDFs during initialization. If we have user on Hive CLI, we will 
  automatically lookup metastore/download UDF resources, when they may not 
  even be using any of these UDFs during their session. How about we keep the 
  existing behavior that we only look them up when they are used during a 
  query?
  
  Also, if we are doing this during static initialization, is Hive be in 
  a state that it can query the metastore? Not sure if there is any other 
  initialization that may need to take place beforehand.

bq. we only look them up when they are used during a query?
This method just stores meta information (classname, resources, etc.) for the 
function without loading any resources/classes, lessening redundant metastore 
accesses from all clients. When user accesses the function it's registered to 
session registry with the information as described above. 

bq. Not sure if there is any other initialization that may need to take place 
beforehand.
Afaik, metastore has static lock for initialization which is checked before 
client accesses internal of it. If it does not act like that, we should fix 
metastore.


- Navis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review57952
---


On Oct. 23, 2014, 12:20 a.m., Jason Dere wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/26854/
 ---
 
 (Updated Oct. 23, 2014, 12:20 a.m.)
 
 
 Review request for hive, Navis Ryu and Thejas Nair.
 
 
 Bugs: HIVE-2573
 https://issues.apache.org/jira/browse/HIVE-2573
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Small updates to Navis' changes:
 - session registry doesn't lookup metastore for UDFs
 - my feedback from Navis' original patch
 - metastore udfs should not be considered native. This allows

Review Request 26966: Path type entities should use qualified path rather than string

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26966/
---

Review request for hive.


Bugs: HIVE-8357
https://issues.apache.org/jira/browse/HIVE-8357


Repository: hive-git


Description
---

Path type entities should use qualified path rather than string


Diffs
-

  common/src/java/org/apache/hadoop/hive/common/FileUtils.java 95e8d7c 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestCreateUdfEntities.java
 c60f856 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java e254505 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java 76b1f01 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java 8b6a923 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java c5be822 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 37cbf7f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java c060d74 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java 
f96209c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 
22e5b47 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
4891518 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 8bd24d3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1b7a41d 
  ql/src/test/results/clientnegative/udf_local_resource.q.out 2f2227b 
  ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out e184787 
  ql/src/test/results/clientpositive/add_part_multiple.q.out 975baa5 
  ql/src/test/results/clientpositive/alter2.q.out 3356ab9 
  ql/src/test/results/clientpositive/alter5.q.out a83b68d 
  ql/src/test/results/clientpositive/exim_17_part_managed.q.out 9036a28 

Diff: https://reviews.apache.org/r/26966/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 25628: alias of compound aggregation functions fails in having clause

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25628/
---

(Updated Oct. 20, 2014, 9:14 a.m.)


Review request for hive.


Changes
---

Rebased to trunk


Bugs: HIVE-7981
https://issues.apache.org/jira/browse/HIVE-7981


Repository: hive-git


Description
---

hive select max(time)-min(time) as span from mytable group by name having 
span0;

FAILED: SemanticException [Error 10025]: Line 1:92 Expression not in GROUP BY 
key '0'


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 02c4be9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1b7a41d 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckCtx.java 3b6178f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java e065983 
  ql/src/test/queries/clientpositive/having3.q PRE-CREATION 
  ql/src/test/results/clientpositive/having3.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/25628/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 26917: Make OrcNewInputFormat return row number as a key

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26917/
---

Review request for hive.


Bugs: trunk
https://issues.apache.org/jira/browse/trunk


Repository: hive-git


Description
---

Key is null in map when OrcNewInputFormat is used as Input Format Class

When using OrcNewInputFormat as input format class for my map reduce job, I 
find its key is always null in my map method. This gives me no way to get row 
number in my map method.  If you compare RCFileInputFormat (for RC file), its 
key in map method returns the row number so I know which row I am processing. 

Is there any workaround for me to get the row number from my map method?  Of 
course, I can count the row number by myself.  But that has two problems: #1 I 
have to assume the row is coming in the order; #2 I will get duplicated (and 
wrong) row numbers if a big input file causes multiple file splits (which will 
trigger my map method multiple times in different data nodes).   At this point, 
I am really seeking a better way to get row number for each processed row in 
map method.

Here is what I have in my map logs:

[2014-08-06 09:39:25 DEBUG com..hadoop.orcfile.OrcFileMap]: Mapper 
Input Key: (null)
[2014-08-06 09:39:25 DEBUG com..hadoop.orcfile.OrcFileMap]: Mapper 
Input Value: {Q8151, T9976, 69976, 8156756, 966798161, 
97898989898, Laura, laura...@gmail.com}

My map method is:

protected void map(Object key, Writable value, Context context)
throws IOException, InterruptedException {
logger.debug(Mapper Input Key:  + key);
logger.debug(Mapper Input Value:  + value.toString());
.
}

The fix should be: add  following statement in nextKeyValue() method and pass 
the result all the way up to the map() method as its key:

  reader.getRowNumber(); 


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcNewInputFormat.java b6ad0dc 

Diff: https://reviews.apache.org/r/26917/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 26918: Failed to locate the winutils binary when loading JDBC driver on Windows

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26918/
---

Review request for hive.


Bugs: HIVE-7845
https://issues.apache.org/jira/browse/HIVE-7845


Repository: hive-git


Description
---

This ERROR is thrown on Windows platforms when loading the JDBC driver, 
subsequent attempts will succeed.  The Hadoop binaries are indeed not 
available, but shouldn't be required when just using JDBC.

13:20:00 [ERROR pool-2-thread-4 Shell.getWinUtilsPath] Failed to locate the 
winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the 
Hadoop binaries.
   at 
org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:324)
   at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:339)
   at org.apache.hadoop.util.Shell.clinit(Shell.java:332)
   at 
org.apache.hadoop.hive.conf.HiveConf$ConfVars.findHadoopBinary(HiveConf.java:918)
   at 
org.apache.hadoop.hive.conf.HiveConf$ConfVars.clinit(HiveConf.java:228)
   at 
org.apache.hive.jdbc.HiveConnection.isHttpTransportMode(HiveConnection.java:304)
   at 
org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:181)
   at 
org.apache.hive.jdbc.HiveConnection.init(HiveConnection.java:164)
   at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown 
Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at com.onseven.dbvis.d.B.D.?(Z:1548)
   at com.onseven.dbvis.d.B.F$A.call(Z:278)
   at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown 
Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown 
Source)
   at java.lang.Thread.run(Unknown Source)


Diffs
-

  common/src/java/org/apache/hadoop/hive/common/FileUtils.java 95e8d7c 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0683921 
  common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java fce46de 
  contrib/src/test/org/apache/hadoop/hive/contrib/mr/TestGenericMR.java 5fedb3d 
  
hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatPartitionPublish.java
 5f9379b 
  
hcatalog/webhcat/java-client/src/test/java/org/apache/hive/hcatalog/api/TestHCatClient.java
 f9f7b04 
  
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/ExecServiceImpl.java
 5b4fd9b 
  
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/LaunchMapper.java
 fb9d767 
  
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonUtils.java
 d2dec54 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java a035ff1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ScriptOperator.java 9d17165 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 155002a 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 93e284e 
  ql/src/test/org/apache/hadoop/hive/ql/WindowsPathUtil.java 294a3dd 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 0eb7c5a 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java 
5b8ec60 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpCLIService.java 
cfa7284 
  shims/common/src/main/java/org/apache/hadoop/fs/ProxyLocalFileSystem.java 
228a972 
  shims/common/src/main/java/org/apache/hadoop/hive/shims/Environments.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/26918/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 25059: Ambiguous column reference error on query

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25059/
---

(Updated Oct. 14, 2014, 6:33 a.m.)


Review request for hive.


Changes
---

Addressed comments


Bugs: HIVE-7733
https://issues.apache.org/jira/browse/HIVE-7733


Repository: hive-git


Description
---

{noformat}
CREATE TABLE agg1 
  ( 
 col0 INT, 
 col1 STRING, 
 col2 DOUBLE 
  ); 

explain SELECT single_use_subq11.a1 AS a1, 
   single_use_subq11.a2 AS a2 
FROM   (SELECT Sum(agg1.col2) AS a1 
FROM   agg1 
GROUP  BY agg1.col0) single_use_subq12 
   JOIN (SELECT alias.a2 AS a0, 
alias.a1 AS a1, 
alias.a1 AS a2 
 FROM   (SELECT agg1.col1 AS a0, 
'42'  AS a1, 
agg1.col0 AS a2 
 FROM   agg1 
 UNION ALL 
 SELECT agg1.col1 AS a0, 
'41'  AS a1, 
agg1.col0 AS a2 
 FROM   agg1) alias 
 GROUP  BY alias.a2, 
   alias.a1) single_use_subq11 
 ON ( single_use_subq11.a0 = single_use_subq11.a0 );
{noformat}

Gets the following error:
FAILED: SemanticException [Error 10007]: Ambiguous column reference a2
Looks like this query had been working in 0.12 but starting failing with this 
error in 0.13


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java b3c4b47 
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 86e4602 
  ql/src/java/org/apache/hadoop/hive/ql/parse/RowResolver.java 9c55379 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java e1eca13 
  ql/src/test/queries/clientnegative/ambiguous_col0.q 46349c6 
  ql/src/test/queries/clientnegative/ambiguous_col1.q 9e8bcbd 
  ql/src/test/queries/clientnegative/ambiguous_col2.q 33d4aed 
  ql/src/test/queries/clientpositive/complex_alias.q PRE-CREATION 
  ql/src/test/results/clientnegative/ambiguous_col.q.out 237c21f 
  ql/src/test/results/clientnegative/ambiguous_col0.q.out 237c21f 
  ql/src/test/results/clientnegative/ambiguous_col1.q.out 237c21f 
  ql/src/test/results/clientnegative/ambiguous_col2.q.out 237c21f 
  ql/src/test/results/clientpositive/ambiguous_col.q.out e8760f1 
  ql/src/test/results/clientpositive/complex_alias.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/25059/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 25238: make timestamp accessible in the hbase KeyValue

[Navis Ryu]

 On Oct. 14, 2014, 1:50 p.m., Swarnim Kulkarni wrote:
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java, line 
  147
  https://reviews.apache.org/r/25238/diff/1/?file=673709#file673709line147
 
  I think timestamp and long make sense. I am a little confused on why we 
  have string as a valid type for timestamp. I understand the flexibility it 
  provides but I don't see a place where we do a validation that the string 
  actually corresponds to a valid timestamp.

Ok


- Navis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25238/#review56517
---


On Sept. 2, 2014, 6:15 a.m., Navis Ryu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/25238/
 ---
 
 (Updated Sept. 2, 2014, 6:15 a.m.)
 
 
 Review request for hive.
 
 
 Bugs: HIVE-2828
 https://issues.apache.org/jira/browse/HIVE-2828
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Originated from HIVE-2781 and not accepted, but I think this could be helpful 
 to someone.
 
 By using special column notation ':timestamp' in HBASE_COLUMNS_MAPPING, user 
 might access timestamp value in hbase KeyValue.
 {code}
 CREATE TABLE hbase_table (key int, value string, time timestamp)
   STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
   WITH SERDEPROPERTIES (hbase.columns.mapping = :key,cf:string,:timestamp)
 {code}
 
 
 Diffs
 -
 
   hbase-handler/src/java/org/apache/hadoop/hive/hbase/ColumnMappings.java 
 8735fbc 
   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java 
 fe6081e 
   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 6c1ce5c 
   
 hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeParameters.java 
 8ea4408 
   
 hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
 feb3cd1 
   
 hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseInputFormatUtil.java
  5aa1d796 
   
 hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
  4ac0803 
   hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java 
 3e8b8fd 
   
 hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java 
 9a31f0f 
   hbase-handler/src/test/queries/positive/hbase_timestamp.q PRE-CREATION 
   hbase-handler/src/test/results/positive/hbase_timestamp.q.out PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java 
 683618f 
   serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 
 0ab27ff 
 
 Diff: https://reviews.apache.org/r/25238/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Navis Ryu
 

Re: Review Request 25238: make timestamp accessible in the hbase KeyValue

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25238/
---

(Updated Oct. 15, 2014, 12:45 a.m.)


Review request for hive.


Changes
---

Removed String mapping


Bugs: HIVE-2828
https://issues.apache.org/jira/browse/HIVE-2828


Repository: hive-git


Description
---

Originated from HIVE-2781 and not accepted, but I think this could be helpful 
to someone.

By using special column notation ':timestamp' in HBASE_COLUMNS_MAPPING, user 
might access timestamp value in hbase KeyValue.
{code}
CREATE TABLE hbase_table (key int, value string, time timestamp)
  STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
  WITH SERDEPROPERTIES (hbase.columns.mapping = :key,cf:string,:timestamp)
{code}


Diffs (updated)
-

  hbase-handler/src/java/org/apache/hadoop/hive/hbase/ColumnMappings.java 
fbd1308 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/DefaultHBaseKeyFactory.java 
12c5377 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java 
c6c42b4 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java aedd843 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeParameters.java 
9efa494 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
feb3cd1 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseInputFormatUtil.java
 5aa1d796 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
 4ac0803 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java 6ac8423 
  hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java 
9a31f0f 
  hbase-handler/src/test/queries/positive/hbase_timestamp.q PRE-CREATION 
  hbase-handler/src/test/results/positive/hbase_timestamp.q.out PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java 
960fc1d 
  serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 
0ab27ff 

Diff: https://reviews.apache.org/r/25238/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 26435: Self join may fail if one side has VCs and other doesn't

[Navis Ryu]

 On Oct. 8, 2014, 6:44 p.m., Sergey Shelukhin wrote:
  ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java, line 85
  https://reviews.apache.org/r/26435/diff/1/?file=715105#file715105line85
 
  is it possible to document those with a small comment; as well as 
  MapOpMeta and MapOpCtx classes?

Sure.


 On Oct. 8, 2014, 6:44 p.m., Sergey Shelukhin wrote:
  data/conf/hive-log4j.properties, line 84
  https://reviews.apache.org/r/26435/diff/1/?file=715104#file715104line84
 
  are ObjectStore and Operator lines intended? they'd affect logging that 
  might be useful for other tests

ObjectStore is a little noisy, IMHO. But ok, I'll revert that.


 On Oct. 8, 2014, 6:44 p.m., Sergey Shelukhin wrote:
  ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java, line 368
  https://reviews.apache.org/r/26435/diff/1/?file=715105#file715105line368
 
  why are the calles to SerDeUtils gone? Just asking

It's same with pd.getDeserializer(hconf);


 On Oct. 8, 2014, 6:44 p.m., Sergey Shelukhin wrote:
  ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java, line 467
  https://reviews.apache.org/r/26435/diff/1/?file=715105#file715105line467
 
  one child operator can be present for several paths or in several 
  MapOpCtx?

Can be. But rowObjectInspector is handed over once and cannot be changed. So 
all OIs of MapOpCtxs for the operator should be all the same.


 On Oct. 8, 2014, 6:44 p.m., Sergey Shelukhin wrote:
  ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java, line 445
  https://reviews.apache.org/r/26435/diff/1/?file=715105#file715105line445
 
  is children.size() check no longer necessary?

I'm not sure on this a little. Previously, setChildren() made 
childrenOpToOpCtxMap only for operators handling first input path. Now it makes 
contexts for all operators in conf.getAliasToWork(). It's why children.size() 
== 0 check and extraChildrenToClose is removed.


 On Oct. 8, 2014, 6:44 p.m., Sergey Shelukhin wrote:
  ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java, line 425
  https://reviews.apache.org/r/26435/diff/1/?file=715105#file715105line425
 
  this creates a context and checks for its presence, but then puts the 
  result of some call taking the context to map. Would it make sense to put 
  context into map first and then do processing on it (that I assume the call 
  does)? That would be less confusing.

Line 372 adds onefile--ListMapOpCtx mapping and context is added to 
ListMapOpCtx (which is 'contexts' in source code).


 On Oct. 8, 2014, 6:44 p.m., Sergey Shelukhin wrote:
  ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java, line 202
  https://reviews.apache.org/r/26435/diff/1/?file=715105#file715105line202
 
  nit: hash codes could be combined similar to how Java does it ( * prime 
  + next)

hashCode and equals seemed not needed. I'll remove those two.


- Navis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26435/#review55851
---


On Oct. 8, 2014, 4:06 a.m., Navis Ryu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/26435/
 ---
 
 (Updated Oct. 8, 2014, 4:06 a.m.)
 
 
 Review request for hive.
 
 
 Bugs: HIVE-8186
 https://issues.apache.org/jira/browse/HIVE-8186
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 {noformat}select t1.BLOCK__OFFSET__INSIDE__FILE,t2.BLOCK__OFFSET__INSIDE__FILE
 from src t1 join src t2 on t1.key = t2.key;{noformat}
 Passes
 {noformat}select t2.BLOCK__OFFSET__INSIDE__FILE
 from src t1 join src t2 on t1.key = t2.key;{noformat}
 Fails.
 
 The issue is that LazyBinarySerDe OI receives data intended for 
 UnionStructObjectInspector.
 Judging by the above it has something to do with scanning table once for two 
 aliases.
 
 I'll look tomorrow
 
 
 Diffs
 -
 
   data/conf/hive-log4j.properties 7f5dfc4 
   ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java f624bf4 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 3dc7c76 
   ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java d8698da 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 155002a 
   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapOperator.java 
 311f6d6 
   ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java 78d4d1f 
   ql/src/test/org/apache/hadoop/hive/ql/exec/TestOperators.java 90e4cad 
   ql/src/test/queries/clientpositive/join_vc.q 63b3da7 
   ql/src/test/results/clientpositive/join_vc.q.out 12004ca 
 
 Diff: https://reviews.apache.org/r/26435/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Navis Ryu
 

Re: Review Request 26435: Self join may fail if one side has VCs and other doesn't

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26435/
---

(Updated Oct. 14, 2014, 2:36 a.m.)


Review request for hive.


Changes
---

Addressed comments and modified the way handing over the table name to script 
operator


Bugs: HIVE-8186
https://issues.apache.org/jira/browse/HIVE-8186


Repository: hive-git


Description
---

{noformat}select t1.BLOCK__OFFSET__INSIDE__FILE,t2.BLOCK__OFFSET__INSIDE__FILE
from src t1 join src t2 on t1.key = t2.key;{noformat}
Passes
{noformat}select t2.BLOCK__OFFSET__INSIDE__FILE
from src t1 join src t2 on t1.key = t2.key;{noformat}
Fails.

The issue is that LazyBinarySerDe OI receives data intended for 
UnionStructObjectInspector.
Judging by the above it has something to do with scanning table once for two 
aliases.

I'll look tomorrow


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6a1a5f0 
  data/conf/hive-log4j.properties 7f5dfc4 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java f624bf4 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 3dc7c76 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java d8698da 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ScriptOperator.java 8228e09 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 155002a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapOperator.java 
311f6d6 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java 78d4d1f 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestOperators.java 90e4cad 
  ql/src/test/queries/clientpositive/join_vc.q 63b3da7 
  ql/src/test/results/clientpositive/join_vc.q.out 12004ca 

Diff: https://reviews.apache.org/r/26435/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 26406: Implement support for BETWEEN in SELECT list

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26406/
---

Review request for hive.


Bugs: HIVE-5744
https://issues.apache.org/jira/browse/HIVE-5744


Repository: hive-git


Description
---

Queries like 

SELECT col1 BETWEEN 0 and 10 from T;

fail in vectorized mode. Support needs to be implemented for a BETWEEN 
expression in the SELECT list, comparable to how it was added for comparison 
operators (, , ...). These were done by adding new, templates that return a 
value for a comparison instead of applying a filter. See 
ColumnCompareScalar.txt under ql/src/gen for an example.


Diffs
-

  ant/src/org/apache/hadoop/hive/ant/GenVectorCode.java 731b686 
  ql/src/gen/vectorization/ExpressionTemplates/ColumnBetween.txt PRE-CREATION 
  ql/src/gen/vectorization/ExpressionTemplates/FilterColumnBetween.txt e9aaaf2 
  ql/src/gen/vectorization/ExpressionTemplates/FilterDecimalColumnBetween.txt 
829b9ca 
  ql/src/gen/vectorization/ExpressionTemplates/TruncStringColumnBetween.txt 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java a105eca 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
34f5823 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/DecimalColumnBetween.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/StringColumnBetween.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorExpression.java
 d7ace6d 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 13d5255 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java af633cb 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBetween.java 
9d9ee57 
  ql/src/test/org/apache/hadoop/hive/metastore/TestMetastoreExpr.java 93981fa 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 0eb7c5a 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorFilterOperator.java 
24a52a6 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorizationContext.java 
9e3628f 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorFilterExpressions.java
 92e5a06 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorLogicalExpressions.java
 60e20a3 
  ql/src/test/queries/clientpositive/vector_between_in.q 1bc6611 
  ql/src/test/results/clientpositive/udf_between.q.out cb63d1c 
  ql/src/test/results/clientpositive/vector_between_in.q.out 631ac19 

Diff: https://reviews.apache.org/r/26406/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 26435: Self join may fail if one side has VCs and other doesn't

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26435/
---

Review request for hive.


Bugs: HIVE-8186
https://issues.apache.org/jira/browse/HIVE-8186


Repository: hive-git


Description
---

{noformat}select t1.BLOCK__OFFSET__INSIDE__FILE,t2.BLOCK__OFFSET__INSIDE__FILE
from src t1 join src t2 on t1.key = t2.key;{noformat}
Passes
{noformat}select t2.BLOCK__OFFSET__INSIDE__FILE
from src t1 join src t2 on t1.key = t2.key;{noformat}
Fails.

The issue is that LazyBinarySerDe OI receives data intended for 
UnionStructObjectInspector.
Judging by the above it has something to do with scanning table once for two 
aliases.

I'll look tomorrow


Diffs
-

  data/conf/hive-log4j.properties 7f5dfc4 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java f624bf4 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 3dc7c76 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java d8698da 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 155002a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapOperator.java 
311f6d6 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java 78d4d1f 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestOperators.java 90e4cad 
  ql/src/test/queries/clientpositive/join_vc.q 63b3da7 
  ql/src/test/results/clientpositive/join_vc.q.out 12004ca 

Diff: https://reviews.apache.org/r/26435/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 26218: Add configuration for custom services in hiveserver2

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26218/
---

Review request for hive.


Bugs: HIVE-8319
https://issues.apache.org/jira/browse/HIVE-8319


Repository: hive-git


Description
---

NO PRECOMMIT TESTS

Register services to hiveserver2, for example, 
{noformat}
property
  namehive.server2.service.classesname
  
valuecom.nexr.hive.service.HiveStatus,com.nexr.hive.service.AzkabanServicevalue
/property
property
  nameazkaban.ssl.portname
  name...name
/property
{noformat}


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b0cb98b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 155002a 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/HookUtils.java 390ffd9 
  service/src/java/org/apache/hive/service/server/HiveServer2.java c667533 

Diff: https://reviews.apache.org/r/26218/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 26218: Add configuration for custom services in hiveserver2

[Navis Ryu]

 On Oct. 1, 2014, 7:38 a.m., Lefty Leverenz wrote:
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, lines 1492-1494
  https://reviews.apache.org/r/26218/diff/1/?file=710029#file710029line1492
 
  Maybe I'm just being dense, but does same lifecycle mean the same 
  lifecycle as HS2?

Yes. I should append of it to description.


- Navis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26218/#review55057
---


On Oct. 1, 2014, 7:24 a.m., Navis Ryu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/26218/
 ---
 
 (Updated Oct. 1, 2014, 7:24 a.m.)
 
 
 Review request for hive.
 
 
 Bugs: HIVE-8319
 https://issues.apache.org/jira/browse/HIVE-8319
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 NO PRECOMMIT TESTS
 
 Register services to hiveserver2, for example, 
 {noformat}
 property
   namehive.server2.service.classesname
   
 valuecom.nexr.hive.service.HiveStatus,com.nexr.hive.service.AzkabanServicevalue
 /property
 property
   nameazkaban.ssl.portname
   name...name
 /property
 {noformat}
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b0cb98b 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 155002a 
   ql/src/java/org/apache/hadoop/hive/ql/hooks/HookUtils.java 390ffd9 
   service/src/java/org/apache/hive/service/server/HiveServer2.java c667533 
 
 Diff: https://reviews.apache.org/r/26218/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Navis Ryu
 

Review Request 26257: Index related events should be delivered to metastore event listener

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26257/
---

Review request for hive and Thejas Nair.


Bugs: HIVE-3781
https://issues.apache.org/jira/browse/HIVE-3781


Repository: hive-git


Description
---

An event listener must be called for any DDL activity. For example, 
create_index, drop_index today does not call metaevent listener. 


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b0cb98b 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java
 3b5f65f 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java f5e35b8 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
47eca29 
  
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java 
ec1dca2 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 637a39a 
  metastore/src/java/org/apache/hadoop/hive/metastore/events/AddIndexEvent.java 
PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/events/AlterIndexEvent.java 
PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/events/DropIndexEvent.java 
PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/events/PreAddIndexEvent.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/events/PreAlterIndexEvent.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/events/PreDropIndexEvent.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/events/PreEventContext.java 
dbc3247 
  metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 
37daf52 
  metastore/src/test/org/apache/hadoop/hive/metastore/DummyPreListener.java 
9a943b2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java d5374bc 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 0f7af9a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 4e58ad8 
  ql/src/java/org/apache/hadoop/hive/ql/plan/DropIndexDesc.java 5ef14ef 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java 153908c 
  ql/src/test/queries/clientpositive/drop_index.q 54ef823 
  ql/src/test/results/clientnegative/alter_concatenate_indexed_table.q.out 
19fbf51 
  ql/src/test/results/clientpositive/alter_concatenate_indexed_table.q.out 
ffcbcf9 
  ql/src/test/results/clientpositive/index_auth.q.out 385b639 
  ql/src/test/results/clientpositive/index_auto.q.out d1f5fa2 
  ql/src/test/results/clientpositive/index_auto_file_format.q.out eca656f 
  ql/src/test/results/clientpositive/index_auto_mult_tables.q.out ab0f06f 
  ql/src/test/results/clientpositive/index_auto_mult_tables_compact.q.out 
20fa238 
  ql/src/test/results/clientpositive/index_auto_multiple.q.out 2408d48 
  ql/src/test/results/clientpositive/index_auto_partitioned.q.out da57bea 
  ql/src/test/results/clientpositive/index_auto_self_join.q.out 94ded26 
  ql/src/test/results/clientpositive/index_auto_update.q.out 85f26ac 
  ql/src/test/results/clientpositive/index_bitmap_auto_partitioned.q.out 
473df0a 
  ql/src/test/results/clientpositive/index_bitmap_compression.q.out b5a172f 
  ql/src/test/results/clientpositive/index_compression.q.out 241c2e9 
  ql/src/test/results/clientpositive/index_serde.q.out 5b4cbfc 
  ql/src/test/results/clientpositive/ql_rewrite_gbtoidx.q.out f7da06c 
  ql/src/test/results/clientpositive/show_indexes_edge_cases.q.out cc6a405 
  ql/src/test/results/clientpositive/show_indexes_syntax.q.out bc96359 
  ql/src/test/results/clientpositive/union_view.q.out bbbf5e7 

Diff: https://reviews.apache.org/r/26257/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 25688: hive jdbc can not used by jmeter, because of unsupported auto commit feature

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25688/
---

Review request for hive.


Bugs: HIVE-6705
https://issues.apache.org/jira/browse/HIVE-6705


Repository: hive-git


Description
---

In apache jmeter ,the autocommit property is required.
but in the hive jdbc the auto commit is unsupported method.

in 
/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java

{quote}
 public void setAutoCommit(boolean autoCommit) throws SQLException {
// TODO Auto-generated method stub

  throw new {color:red}  SQLException(Method not supported);
{color}
  }
{quote}

so ,should  we make a mock to support  the auto commit property == false ?

{quote}
public void setAutoCommit(boolean autoCommit) throws SQLException {
  // TODO Auto-generated method stub
 {color:red}if(autoCommit) {color}
  throw new SQLException(Method not supported);
 else
  return;
 }
{quote}


Diffs
-

  jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java 59ce692 
  jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java cbcfec7 

Diff: https://reviews.apache.org/r/25688/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 25625: Support non-constant expressions for MAP type indices.

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25625/
---

Review request for hive.


Repository: hive-git


Description
---

Here is my sample:
{code}
CREATE TABLE RECORD(RecordID string, BatchDate string, Country string) 
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' 
WITH SERDEPROPERTIES (hbase.columns.mapping = :key,D:BatchDate,D:Country) 
TBLPROPERTIES (hbase.table.name = RECORD); 


CREATE TABLE KEY_RECORD(KeyValue String, RecordId mapstring,string) 
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' 
WITH SERDEPROPERTIES (hbase.columns.mapping = :key, K:) 
TBLPROPERTIES (hbase.table.name = KEY_RECORD); 
{code}
The following join statement doesn't work. 
{code}
SELECT a.*, b.* from KEY_RECORD a join RECORD b 
WHERE a.RecordId[b.RecordID] is not null;
{code}
FAILED: SemanticException 2:16 Non-constant expression for map indexes not 
supported. Error encountered near token 'RecordID' 


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java b3389cc 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java c503bbb 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java e44f5ae 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIndex.java 
5911f2c 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestFunctionRegistry.java d7d2a34 
  ql/src/test/queries/clientpositive/array_map_access_nonconstant.q 
PRE-CREATION 
  ql/src/test/queries/negative/invalid_list_index.q c40f079 
  ql/src/test/queries/negative/invalid_list_index2.q 99d0b3d 
  ql/src/test/queries/negative/invalid_map_index.q c2b9eab 
  ql/src/test/queries/negative/invalid_map_index2.q 5828f07 
  ql/src/test/results/clientpositive/array_map_access_nonconstant.q.out 
PRE-CREATION 
  ql/src/test/results/compiler/errors/invalid_list_index.q.out a4179cd 
  ql/src/test/results/compiler/errors/invalid_list_index2.q.out aaa9455 
  ql/src/test/results/compiler/errors/invalid_map_index.q.out 4eb7afa 
  ql/src/test/results/compiler/errors/invalid_map_index2.q.out edc9bda 

Diff: https://reviews.apache.org/r/25625/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 25625: Support non-constant expressions for MAP type indices.

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25625/
---

(Updated Sept. 15, 2014, 1:33 a.m.)


Review request for hive.


Bugs: HIVE-7325
https://issues.apache.org/jira/browse/HIVE-7325


Repository: hive-git


Description
---

Here is my sample:
{code}
CREATE TABLE RECORD(RecordID string, BatchDate string, Country string) 
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' 
WITH SERDEPROPERTIES (hbase.columns.mapping = :key,D:BatchDate,D:Country) 
TBLPROPERTIES (hbase.table.name = RECORD); 


CREATE TABLE KEY_RECORD(KeyValue String, RecordId mapstring,string) 
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' 
WITH SERDEPROPERTIES (hbase.columns.mapping = :key, K:) 
TBLPROPERTIES (hbase.table.name = KEY_RECORD); 
{code}
The following join statement doesn't work. 
{code}
SELECT a.*, b.* from KEY_RECORD a join RECORD b 
WHERE a.RecordId[b.RecordID] is not null;
{code}
FAILED: SemanticException 2:16 Non-constant expression for map indexes not 
supported. Error encountered near token 'RecordID' 


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java b3389cc 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java c503bbb 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java e44f5ae 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIndex.java 
5911f2c 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestFunctionRegistry.java d7d2a34 
  ql/src/test/queries/clientpositive/array_map_access_nonconstant.q 
PRE-CREATION 
  ql/src/test/queries/negative/invalid_list_index.q c40f079 
  ql/src/test/queries/negative/invalid_list_index2.q 99d0b3d 
  ql/src/test/queries/negative/invalid_map_index.q c2b9eab 
  ql/src/test/queries/negative/invalid_map_index2.q 5828f07 
  ql/src/test/results/clientpositive/array_map_access_nonconstant.q.out 
PRE-CREATION 
  ql/src/test/results/compiler/errors/invalid_list_index.q.out a4179cd 
  ql/src/test/results/compiler/errors/invalid_list_index2.q.out aaa9455 
  ql/src/test/results/compiler/errors/invalid_map_index.q.out 4eb7afa 
  ql/src/test/results/compiler/errors/invalid_map_index2.q.out edc9bda 

Diff: https://reviews.apache.org/r/25625/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 25628: alias of compound aggregation functions fails in having clause

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25628/
---

Review request for hive.


Bugs: HIVE-7981
https://issues.apache.org/jira/browse/HIVE-7981


Repository: hive-git


Description
---

hive select max(time)-min(time) as span from mytable group by name having 
span0;

FAILED: SemanticException [Error 10025]: Line 1:92 Expression not in GROUP BY 
key '0'


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 86e4602 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 392f7ce 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckCtx.java a95ae20 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java e44f5ae 
  ql/src/test/queries/clientpositive/having3.q PRE-CREATION 
  ql/src/test/results/clientpositive/having3.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/25628/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 25059: Ambiguous column reference error on query

[Navis Ryu]

 On Sept. 10, 2014, 11:58 p.m., Ashutosh Chauhan wrote:
  ql/src/java/org/apache/hadoop/hive/ql/parse/RowResolver.java, line 213
  https://reviews.apache.org/r/25059/diff/1/?file=669167#file669167line213
 
  I am bit confused here. Shouldn't this be if (tableAlias == null || 
  tableAlias.equalsIgnoreCase(prev.getTabAlias()))
  
  Note missing !
  We want to throw an exception if tabAlias = null or if non-null than 
  equal to prev. Isnt it?
  
  But, I see existing code is doing the same. So, confused here.

!tableAlias.equalsIgnoreCase(prev.getTabAlias()) means the column is from 
different table alias, which makes it an ambiguous alias. This can be done at 
last stage of processing the subquery. I'll update the patch.


- Navis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25059/#review52982
---


On Aug. 26, 2014, 9:06 a.m., Navis Ryu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/25059/
 ---
 
 (Updated Aug. 26, 2014, 9:06 a.m.)
 
 
 Review request for hive.
 
 
 Bugs: HIVE-7733
 https://issues.apache.org/jira/browse/HIVE-7733
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 {noformat}
 CREATE TABLE agg1 
   ( 
  col0 INT, 
  col1 STRING, 
  col2 DOUBLE 
   ); 
 
 explain SELECT single_use_subq11.a1 AS a1, 
single_use_subq11.a2 AS a2 
 FROM   (SELECT Sum(agg1.col2) AS a1 
 FROM   agg1 
 GROUP  BY agg1.col0) single_use_subq12 
JOIN (SELECT alias.a2 AS a0, 
 alias.a1 AS a1, 
 alias.a1 AS a2 
  FROM   (SELECT agg1.col1 AS a0, 
 '42'  AS a1, 
 agg1.col0 AS a2 
  FROM   agg1 
  UNION ALL 
  SELECT agg1.col1 AS a0, 
 '41'  AS a1, 
 agg1.col0 AS a2 
  FROM   agg1) alias 
  GROUP  BY alias.a2, 
alias.a1) single_use_subq11 
  ON ( single_use_subq11.a0 = single_use_subq11.a0 );
 {noformat}
 
 Gets the following error:
 FAILED: SemanticException [Error 10007]: Ambiguous column reference a2
 Looks like this query had been working in 0.12 but starting failing with this 
 error in 0.13
 
 
 Diffs
 -
 
   ql/src/java/org/apache/hadoop/hive/ql/parse/RowResolver.java 33b8a21 
   ql/src/test/queries/clientpositive/complex_alias.q PRE-CREATION 
   ql/src/test/results/clientpositive/complex_alias.q.out PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/25059/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Navis Ryu
 

Re: Review Request 25059: Ambiguous column reference error on query

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25059/
---

(Updated Sept. 12, 2014, 1:26 a.m.)


Review request for hive.


Bugs: HIVE-7733
https://issues.apache.org/jira/browse/HIVE-7733


Repository: hive-git


Description
---

{noformat}
CREATE TABLE agg1 
  ( 
 col0 INT, 
 col1 STRING, 
 col2 DOUBLE 
  ); 

explain SELECT single_use_subq11.a1 AS a1, 
   single_use_subq11.a2 AS a2 
FROM   (SELECT Sum(agg1.col2) AS a1 
FROM   agg1 
GROUP  BY agg1.col0) single_use_subq12 
   JOIN (SELECT alias.a2 AS a0, 
alias.a1 AS a1, 
alias.a1 AS a2 
 FROM   (SELECT agg1.col1 AS a0, 
'42'  AS a1, 
agg1.col0 AS a2 
 FROM   agg1 
 UNION ALL 
 SELECT agg1.col1 AS a0, 
'41'  AS a1, 
agg1.col0 AS a2 
 FROM   agg1) alias 
 GROUP  BY alias.a2, 
   alias.a1) single_use_subq11 
 ON ( single_use_subq11.a0 = single_use_subq11.a0 );
{noformat}

Gets the following error:
FAILED: SemanticException [Error 10007]: Ambiguous column reference a2
Looks like this query had been working in 0.12 but starting failing with this 
error in 0.13


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/RowResolver.java 2e58b80 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 496f6a6 
  ql/src/test/queries/clientpositive/complex_alias.q PRE-CREATION 
  ql/src/test/results/clientnegative/ambiguous_col.q.out 237c21f 
  ql/src/test/results/clientnegative/ambiguous_col0.q.out 237c21f 
  ql/src/test/results/clientnegative/ambiguous_col1.q.out 237c21f 
  ql/src/test/results/clientnegative/ambiguous_col2.q.out 237c21f 
  ql/src/test/results/clientpositive/ambiguous_col.q.out e8760f1 
  ql/src/test/results/clientpositive/complex_alias.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/25059/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 23352: Support non-constant expressions for MAP type indices.

[Navis Ryu]

 On Sept. 9, 2014, 6:23 p.m., Jason Dere wrote:
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java, line 
  768
  https://reviews.apache.org/r/23352/diff/1/?file=626518#file626518line768
 
  could we also use implicitConvertable() here?

Will introduce more work to be done in index udf. I'll update that.


- Navis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23352/#review52751
---


On July 9, 2014, 6:57 a.m., Navis Ryu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/23352/
 ---
 
 (Updated July 9, 2014, 6:57 a.m.)
 
 
 Review request for hive.
 
 
 Bugs: HIVE-7325
 https://issues.apache.org/jira/browse/HIVE-7325
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Here is my sample:
 {code}
 CREATE TABLE RECORD(RecordID string, BatchDate string, Country string) 
 STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' 
 WITH SERDEPROPERTIES (hbase.columns.mapping = :key,D:BatchDate,D:Country) 
 TBLPROPERTIES (hbase.table.name = RECORD); 
 
 
 CREATE TABLE KEY_RECORD(KeyValue String, RecordId mapstring,string) 
 STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' 
 WITH SERDEPROPERTIES (hbase.columns.mapping = :key, K:) 
 TBLPROPERTIES (hbase.table.name = KEY_RECORD); 
 {code}
 The following join statement doesn't work. 
 {code}
 SELECT a.*, b.* from KEY_RECORD a join RECORD b 
 WHERE a.RecordId[b.RecordID] is not null;
 {code}
 FAILED: SemanticException 2:16 Non-constant expression for map indexes not 
 supported. Error encountered near token 'RecordID' 
 
 
 Diffs
 -
 
   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 9889cfe 
   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
 e44f5ae 
   ql/src/test/queries/clientpositive/array_map_access_nonconstant.q 
 PRE-CREATION 
   ql/src/test/queries/negative/invalid_list_index.q c40f079 
   ql/src/test/queries/negative/invalid_list_index2.q 99d0b3d 
   ql/src/test/queries/negative/invalid_map_index2.q 5828f07 
   ql/src/test/results/clientpositive/array_map_access_nonconstant.q.out 
 PRE-CREATION 
   ql/src/test/results/compiler/errors/invalid_list_index.q.out a4179cd 
   ql/src/test/results/compiler/errors/invalid_list_index2.q.out aaa9455 
   ql/src/test/results/compiler/errors/invalid_map_index2.q.out edc9bda 
   
 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java
  5ccacf1 
 
 Diff: https://reviews.apache.org/r/23352/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Navis Ryu
 

Review Request 25238: make timestamp accessible in the hbase KeyValue

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25238/
---

Review request for hive.


Bugs: HIVE-2828
https://issues.apache.org/jira/browse/HIVE-2828


Repository: hive-git


Description
---

Originated from HIVE-2781 and not accepted, but I think this could be helpful 
to someone.

By using special column notation ':timestamp' in HBASE_COLUMNS_MAPPING, user 
might access timestamp value in hbase KeyValue.
{code}
CREATE TABLE hbase_table (key int, value string, time timestamp)
  STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
  WITH SERDEPROPERTIES (hbase.columns.mapping = :key,cf:string,:timestamp)
{code}


Diffs
-

  hbase-handler/src/java/org/apache/hadoop/hive/hbase/ColumnMappings.java 
8735fbc 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java 
fe6081e 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 6c1ce5c 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeParameters.java 
8ea4408 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
feb3cd1 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseInputFormatUtil.java
 5aa1d796 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
 4ac0803 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java 3e8b8fd 
  hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java 
9a31f0f 
  hbase-handler/src/test/queries/positive/hbase_timestamp.q PRE-CREATION 
  hbase-handler/src/test/results/positive/hbase_timestamp.q.out PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java 
683618f 
  serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 
0ab27ff 

Diff: https://reviews.apache.org/r/25238/diff/


Testing
---


Thanks,

Navis Ryu