[jira] [Created] (HIVE-16006) Incremental REPL LOAD doesn't operate on the target database if name differs from source database.

[Sankar Hariappan (JIRA)]

Sankar Hariappan created HIVE-16006:
---

 Summary: Incremental REPL LOAD doesn't operate on the target 
database if name differs from source database.
 Key: HIVE-16006
 URL: https://issues.apache.org/jira/browse/HIVE-16006
 Project: Hive
  Issue Type: Bug
  Components: repl
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan


During "Incremental Load", it is not considering the database name input in the 
command line. Hence load doesn't happen. At the same time, database with 
original name is getting modified.
Steps:
1. REPL DUMP default FROM 52;
2. REPL LOAD replDb FROM '/tmp/dump/1487588522621';
– This step modifies the default Db instead of replDb.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16005) miscellaneous small fixes to help with debuggability

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-16005:
-

 Summary: miscellaneous small fixes to help with debuggability
 Key: HIVE-16005
 URL: https://issues.apache.org/jira/browse/HIVE-16005
 Project: Hive
  Issue Type: Task
Reporter: Siddharth Seth


- Include proc_ in cli, beeline, metastore, hs2 process args
- LLAP history logger - log QueryId instead of dagName (dag name is free 
flowing text)
- LLAP JXM ExecutorStatus - Log QueryId instead of dagName. Sort by running / 
queued
- Include thread name in TaskRunnerCallable so that it shows up in stack traces 
(will cause extra output in logs)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16004) OutOfMemory in SparkReduceRecordHandler with vectorization mode

[Colin Ma (JIRA)]

Colin Ma created HIVE-16004:
---

 Summary: OutOfMemory in SparkReduceRecordHandler with 
vectorization mode
 Key: HIVE-16004
 URL: https://issues.apache.org/jira/browse/HIVE-16004
 Project: Hive
  Issue Type: Bug
Reporter: Colin Ma
Assignee: Colin Ma


For the query 28 of TPCs-BB with 1T data, the executor memory is set as 30G. 
Get the following exception:
java.lang.OutOfMemoryError
at 
java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)
at 
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil.setVector(VectorizedBatchUtil.java:467)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil.addRowToBatchFrom(VectorizedBatchUtil.java:238)
at 
org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.processVectors(SparkReduceRecordHandler.java:367)
at 
org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.processRow(SparkReduceRecordHandler.java:286)
at 
org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.processRow(SparkReduceRecordHandler.java:220)
at 
org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunctionResultList.processNextRecord(HiveReduceFunctionResultList.java:49)
at 
org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunctionResultList.processNextRecord(HiveReduceFunctionResultList.java:28)
at 
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList.hasNext(HiveBaseFunctionResultList.java:85)
at 
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at 
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$12.apply(AsyncRDDActions.scala:127)
at 
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$12.apply(AsyncRDDActions.scala:127)
at 
org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1974)
at 
org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1974)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:85)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) 

I think DataOutputBuffer isn't cleared on time cause this problem.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [ANNOUNCE] New committer: Zoltan Haindrich

[Lefty Leverenz]

Congratulations Zoltan, and thanks for all your wiki contributions!

-- Lefty


On Tue, Feb 21, 2017 at 10:00 PM, Zoltan Haindrich  wrote:

> Thank you everyone!
>
> It's really an honor to be associated with this project!
>
> About me: I've a pretty broad spectrum of interest in general, I always
> want to clearly understand the big picture. I tend to focus on creating
> flexible solutions.
> I usually collect all kind of (sometimes wierd) ideas while I work on
> things - but I only employ any of them when it aids the introduction of
> some new feature...
>
> I'm looking forward to continue working with you on Hive!
>
> cheers,
> Zoltan
>
>
> On 02/22/2017 01:47 AM, Sergey Shelukhin wrote:
>
>> Congratulations!
>>
>> On 17/2/21, 16:43, "Prasanth Jayachandran" > >
>> wrote:
>>
>> Congratulations Zoltan!!
>>>
>>> Thanks
>>> Prasanth
>>>
>>>
>>>
>>>
>>> On Tue, Feb 21, 2017 at 4:35 PM -0800, "Eugene Koifman"
>>> mailto:ekoif...@hortonworks.com>> wrote:
>>>
>>>
>>> Congratulations!
>>>
>>> On 2/21/17, 4:17 PM, "Vihang Karajgaonkar"  wrote:
>>>
>>>Congrats Zoltan!
>>>
>>>On Tue, Feb 21, 2017 at 4:16 PM, Vaibhav Gumashta  wrote:
>>>
>>>> Congrats Zoltan!
>>>>
>>>> On 2/21/17, 4:16 PM, "Jimmy Xiang"  wrote:
>>>>
>>>> >Congrats, Zoltan!!
>>>> >
>>>> >On Tue, Feb 21, 2017 at 4:15 PM, Sushanth Sowmyan
>>>> >wrote:
>>>> >> Congrats, Zoltan!
>>>> >>
>>>> >> Welcome aboard. :)
>>>> >>
>>>> >> On Feb 21, 2017 15:42, "Rajesh Balamohan"
>>>> wrote:
>>>> >>
>>>> >>> Congrats Zoltan. :)
>>>> >>>
>>>> >>> ~Rajesh.B
>>>> >>>
>>>> >>> On Wed, Feb 22, 2017 at 4:43 AM, Wei Zheng
>>>> >>>wrote:
>>>> >>>
>>>> >>> > Congrats Zoltan!
>>>> >>> >
>>>> >>> > Thanks,
>>>> >>> > Wei
>>>> >>> >
>>>> >>> > On 2/21/17, 13:09, "Alan Gates"  wrote:
>>>> >>> >
>>>> >>> > On behalf of the Hive PMC I am happy to announce Zoltan
>>>> >>>Haindrich is
>>>> >>> > our newest committer.  He has been contributing to Hive for
>>> several
>>>> >>> months
>>>> >>> > across a number of areas, including the parser, HiveServer2,
>>> and
>>>> >>>cleaning
>>>> >>> > up unit tests and documentation.  Please join me in welcoming
>>> Zoltan
>>>> >>>to
>>>> >>> > Hive.
>>>> >>> >
>>>> >>> > Zoltan, feel free to say a few words introducing yourself
>>> if you
>>>> >>> would
>>>> >>> > like to.
>>>> >>> >
>>>> >>> > Alan.
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>>
>>>> >
>>>>
>>>>
>>>
>>>
>>>
>>>
>>

[jira] [Created] (HIVE-16003) Blobstores should use fs.listFiles(path, recursive=true) rather than FileUtils.listStatusRecursively

[Sahil Takiar (JIRA)]

Sahil Takiar created HIVE-16003:
---

 Summary: Blobstores should use fs.listFiles(path, recursive=true) 
rather than FileUtils.listStatusRecursively
 Key: HIVE-16003
 URL: https://issues.apache.org/jira/browse/HIVE-16003
 Project: Hive
  Issue Type: Sub-task
Reporter: Sahil Takiar
Assignee: Sahil Takiar


{{FileUtils.listStatusRecursively}} can be slow on blobstores because 
{{listStatus}} calls are applied recursively to a given directory. This can be 
especially bad on tables with multiple levels of partitioning.

The {{FileSystem}} API provides an optimized API called {{listFiles(path, 
recursive)}} that can be used to invoke an optimized recursive directory 
listing.

The problem is that the {{listFiles(path, recursive)}} API doesn't provide a 
option to pass in a {{PathFilter}}, while {{FileUtils.listStatusRecursively}} 
uses a custom HIDDEN_FILES_PATH_FILTER.

To fix this we could either:

1: Modify the FileSystem API to provide a {{listFiles(path, recursive, 
PathFilter)}} method
2: Add conditional logic so that blobstores invoke {{listFiles(path, 
recursive)}} and the rest of the code uses the current implementation of 
{{FileUtils.listStatusRecursively}}
3: Replace the implementation of {{FileUtils.listStatusRecursively}} with 
{{listFiles(path, recursive)}} and apply the {{PathFilter}} on the results



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [ANNOUNCE] New committer: Zoltan Haindrich

[Zoltan Haindrich]

Thank you everyone!

It's really an honor to be associated with this project!

About me: I've a pretty broad spectrum of interest in general, I always 
want to clearly understand the big picture. I tend to focus on creating 
flexible solutions.
I usually collect all kind of (sometimes wierd) ideas while I work on 
things - but I only employ any of them when it aids the introduction of 
some new feature...


I'm looking forward to continue working with you on Hive!

cheers,
Zoltan

On 02/22/2017 01:47 AM, Sergey Shelukhin wrote:

Congratulations!

On 17/2/21, 16:43, "Prasanth Jayachandran" 
wrote:


Congratulations Zoltan!!

Thanks
Prasanth




On Tue, Feb 21, 2017 at 4:35 PM -0800, "Eugene Koifman"
mailto:ekoif...@hortonworks.com>> wrote:


Congratulations!

On 2/21/17, 4:17 PM, "Vihang Karajgaonkar"  wrote:

   Congrats Zoltan!

   On Tue, Feb 21, 2017 at 4:16 PM, Vaibhav Gumashta  wrote:

   > Congrats Zoltan!
   >
   > On 2/21/17, 4:16 PM, "Jimmy Xiang"  wrote:
   >
   > >Congrats, Zoltan!!
   > >
   > >On Tue, Feb 21, 2017 at 4:15 PM, Sushanth Sowmyan
   > >wrote:
   > >> Congrats, Zoltan!
   > >>
   > >> Welcome aboard. :)
   > >>
   > >> On Feb 21, 2017 15:42, "Rajesh Balamohan"
   > wrote:
   > >>
   > >>> Congrats Zoltan. :)
   > >>>
   > >>> ~Rajesh.B
   > >>>
   > >>> On Wed, Feb 22, 2017 at 4:43 AM, Wei Zheng
   > >>>wrote:
   > >>>
   > >>> > Congrats Zoltan!
   > >>> >
   > >>> > Thanks,
   > >>> > Wei
   > >>> >
   > >>> > On 2/21/17, 13:09, "Alan Gates"  wrote:
   > >>> >
   > >>> > On behalf of the Hive PMC I am happy to announce Zoltan
   > >>>Haindrich is
   > >>> > our newest committer.  He has been contributing to Hive for
several
   > >>> months
   > >>> > across a number of areas, including the parser, HiveServer2,
and
   > >>>cleaning
   > >>> > up unit tests and documentation.  Please join me in welcoming
Zoltan
   > >>>to
   > >>> > Hive.
   > >>> >
   > >>> > Zoltan, feel free to say a few words introducing yourself
if you
   > >>> would
   > >>> > like to.
   > >>> >
   > >>> > Alan.
   > >>> >
   > >>> >
   > >>> >
   > >>>
   > >
   >
   >

Re: [ANNOUNCE] New committer: Zoltan Haindrich

[Rui Li]

Congratulations!

On Wed, Feb 22, 2017 at 8:47 AM, Sergey Shelukhin 
wrote:

> Congratulations!
>
> On 17/2/21, 16:43, "Prasanth Jayachandran" 
> wrote:
>
> >Congratulations Zoltan!!
> >
> >Thanks
> >Prasanth
> >
> >
> >
> >
> >On Tue, Feb 21, 2017 at 4:35 PM -0800, "Eugene Koifman"
> >mailto:ekoif...@hortonworks.com>> wrote:
> >
> >
> >Congratulations!
> >
> >On 2/21/17, 4:17 PM, "Vihang Karajgaonkar"  wrote:
> >
> >Congrats Zoltan!
> >
> >On Tue, Feb 21, 2017 at 4:16 PM, Vaibhav Gumashta  wrote:
> >
> >> Congrats Zoltan!
> >>
> >> On 2/21/17, 4:16 PM, "Jimmy Xiang"  wrote:
> >>
> >> >Congrats, Zoltan!!
> >> >
> >> >On Tue, Feb 21, 2017 at 4:15 PM, Sushanth Sowmyan
> >> >wrote:
> >> >> Congrats, Zoltan!
> >> >>
> >> >> Welcome aboard. :)
> >> >>
> >> >> On Feb 21, 2017 15:42, "Rajesh Balamohan"
> >> wrote:
> >> >>
> >> >>> Congrats Zoltan. :)
> >> >>>
> >> >>> ~Rajesh.B
> >> >>>
> >> >>> On Wed, Feb 22, 2017 at 4:43 AM, Wei Zheng
> >> >>>wrote:
> >> >>>
> >> >>> > Congrats Zoltan!
> >> >>> >
> >> >>> > Thanks,
> >> >>> > Wei
> >> >>> >
> >> >>> > On 2/21/17, 13:09, "Alan Gates"  wrote:
> >> >>> >
> >> >>> > On behalf of the Hive PMC I am happy to announce Zoltan
> >> >>>Haindrich is
> >> >>> > our newest committer.  He has been contributing to Hive for
> >several
> >> >>> months
> >> >>> > across a number of areas, including the parser, HiveServer2,
> >and
> >> >>>cleaning
> >> >>> > up unit tests and documentation.  Please join me in welcoming
> >Zoltan
> >> >>>to
> >> >>> > Hive.
> >> >>> >
> >> >>> > Zoltan, feel free to say a few words introducing yourself
> >if you
> >> >>> would
> >> >>> > like to.
> >> >>> >
> >> >>> > Alan.
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>>
> >> >
> >>
> >>
> >
> >
> >
>
>


-- 
Best regards!
Rui Li
Cell: (+86) 13564950210

Re: Review Request 56902: make explain formatted to include opId and etc

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56902/
---

(Updated Feb. 22, 2017, 4:24 a.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-15955


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 74cec3e 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/AnnotateReduceSinkOutputOperator.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java c6287e4 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java b8c2d42 
  ql/src/test/queries/clientpositive/explain_formatted_oid.q PRE-CREATION 
  ql/src/test/results/clientpositive/explain_formatted_oid.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/input4.q.out 83912f6 
  ql/src/test/results/clientpositive/join0.q.out c02319e 
  ql/src/test/results/clientpositive/parallel_join0.q.out c02319e 
  ql/src/test/results/clientpositive/plan_json.q.out 75d5b73 
  ql/src/test/results/clientpositive/vector_outer_join3.q.out 49c658b 
  ql/src/test/results/clientpositive/vector_outer_join4.q.out fce35a1 
  ql/src/test/results/clientpositive/vector_outer_join6.q.out a910ed2 

Diff: https://reviews.apache.org/r/56902/diff/


Testing
---


Thanks,

pengcheng xiong

Re: Review Request 56687: Intern strings in various critical places to reduce memory consumption.

[Sahil Takiar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56687/#review166298
---



Overal, looks good, just a few minor comments.


common/src/java/org/apache/hadoop/hive/common/StringInternUtils.java (line 29)


How will this class relate to the intern utils provided in 
`HiveStringUtils`?



common/src/java/org/apache/hadoop/hive/common/StringInternUtils.java (line 87)


Add check to see if path is null



ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenMRSkewJoinProcessor.java
 (line 394)


Did the changes to the skew join come up when running the `count()` 
queries? Or did you notice that this code could benefit from interning and 
decide to update it.


- Sahil Takiar


On Feb. 14, 2017, 11:03 p.m., Misha Dmitriev wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56687/
> ---
> 
> (Updated Feb. 14, 2017, 11:03 p.m.)
> 
> 
> Review request for hive, Chaoyu Tang, Mohit Sabharwal, and Sergio Pena.
> 
> 
> Bugs: https://issues.apache.org/jira/browse/HIVE-15882
> 
> https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/HIVE-15882
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See the description of the problem in 
> https://issues.apache.org/jira/browse/HIVE-15882 Interning strings per this 
> review removes most of the overhead due to duplicate strings.
> 
> Also, where maps in several places are created from other maps, use the 
> original map's size for the new map. This is to avoid the situation when a 
> map with default capacity (typically 16) is created to hold just 2-3 entries, 
> and the rest of the internal 16-entry array is wasted.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/StringInternUtils.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
> e81cbce3e333d44a4088c10491f399e92a505293 
>   ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java 
> 08420664d59f28f75872c25c9f8ee42577b23451 
>   ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java 
> e91064b9c75e8adb2b36f21ff19ec0c1539b03b9 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 
> 51530ac16c92cc75d501bfcb573557754ba0c964 
>   ql/src/java/org/apache/hadoop/hive/ql/io/SymbolicInputFormat.java 
> 55b3b551a1dac92583b6e03b10beb8172ca93d45 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java 
> 82dc89803be9cf9e0018720eeceb90ff450bfdc8 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 
> c0edde9e92314d86482b5c46178987e79fae57fe 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 
> c6ae6f290857cfd10f1023058ede99bf4a10f057 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 24d16812515bdfa90b4be7a295c0388fcdfe95ef 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenMRSkewJoinProcessor.java
>  ede4fcbe342052ad86dadebcc49da2c0f515ea98 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/NullScanTaskDispatcher.java
>  0882ae2c6205b1636cbc92e76ef66bb70faadc76 
>   
> ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 
> 68b0ad9ea63f051f16fec3652d8525f7ab07eb3f 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 
> d4bdd96eaf8d179bed43b8a8c3be0d338940154a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MsckDesc.java 
> b7a7e4b7a5f8941b080c7805d224d3885885f444 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java 
> 73981e826870139a42ad881103fdb0a2ef8433a2 
> 
> Diff: https://reviews.apache.org/r/56687/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Misha Dmitriev
> 
>

[jira] [Created] (HIVE-16002) Correlated IN subquery with aggregate asserts in sq_count_check UDF

[Vineet Garg (JIRA)]

Vineet Garg created HIVE-16002:
--

 Summary: Correlated IN subquery with aggregate asserts in 
sq_count_check UDF
 Key: HIVE-16002
 URL: https://issues.apache.org/jira/browse/HIVE-16002
 Project: Hive
  Issue Type: Bug
Reporter: Vineet Garg
Assignee: Vineet Garg


==Reproducer==

{code:SQL}
create table t(i int, j int);
insert into t values(0,1), (0,2);

create table tt(i int, j int);
insert into tt values(0,3);

select * from t where i IN (select count(i) from tt where tt.j = t.j);
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 56902: make explain formatted to include opId and etc

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56902/#review166282
---




ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java (line 86)


Please mark this as transient.


- Ashutosh Chauhan


On Feb. 21, 2017, 11:13 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56902/
> ---
> 
> (Updated Feb. 21, 2017, 11:13 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15955
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 74cec3e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/AnnotateReduceSinkOutputOperator.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java c6287e4 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java b8c2d42 
>   ql/src/test/queries/clientpositive/explain_formatted_oid.q PRE-CREATION 
>   ql/src/test/results/clientpositive/explain_formatted_oid.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/56902/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 56902: make explain formatted to include opId and etc

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56902/#review166276
---




ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java (line 799)


Declare string constants as class variable.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/AnnotateReduceSinkOutputOperator.java
 (lines 44 - 48)


Copy-paste?



ql/src/test/queries/clientpositive/explain_formatted_oid.q (line 1)


This should not be necessary. May as well.


- Ashutosh Chauhan


On Feb. 21, 2017, 11:13 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56902/
> ---
> 
> (Updated Feb. 21, 2017, 11:13 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15955
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 74cec3e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/AnnotateReduceSinkOutputOperator.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java c6287e4 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java b8c2d42 
>   ql/src/test/queries/clientpositive/explain_formatted_oid.q PRE-CREATION 
>   ql/src/test/results/clientpositive/explain_formatted_oid.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/56902/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: [ANNOUNCE] New committer: Zoltan Haindrich

[Sergey Shelukhin]

Congratulations!

On 17/2/21, 16:43, "Prasanth Jayachandran" 
wrote:

>Congratulations Zoltan!!
>
>Thanks
>Prasanth
>
>
>
>
>On Tue, Feb 21, 2017 at 4:35 PM -0800, "Eugene Koifman"
>mailto:ekoif...@hortonworks.com>> wrote:
>
>
>Congratulations!
>
>On 2/21/17, 4:17 PM, "Vihang Karajgaonkar"  wrote:
>
>Congrats Zoltan!
>
>On Tue, Feb 21, 2017 at 4:16 PM, Vaibhav Gumashta  wrote:
>
>> Congrats Zoltan!
>>
>> On 2/21/17, 4:16 PM, "Jimmy Xiang"  wrote:
>>
>> >Congrats, Zoltan!!
>> >
>> >On Tue, Feb 21, 2017 at 4:15 PM, Sushanth Sowmyan
>> >wrote:
>> >> Congrats, Zoltan!
>> >>
>> >> Welcome aboard. :)
>> >>
>> >> On Feb 21, 2017 15:42, "Rajesh Balamohan"
>> wrote:
>> >>
>> >>> Congrats Zoltan. :)
>> >>>
>> >>> ~Rajesh.B
>> >>>
>> >>> On Wed, Feb 22, 2017 at 4:43 AM, Wei Zheng
>> >>>wrote:
>> >>>
>> >>> > Congrats Zoltan!
>> >>> >
>> >>> > Thanks,
>> >>> > Wei
>> >>> >
>> >>> > On 2/21/17, 13:09, "Alan Gates"  wrote:
>> >>> >
>> >>> > On behalf of the Hive PMC I am happy to announce Zoltan
>> >>>Haindrich is
>> >>> > our newest committer.  He has been contributing to Hive for
>several
>> >>> months
>> >>> > across a number of areas, including the parser, HiveServer2,
>and
>> >>>cleaning
>> >>> > up unit tests and documentation.  Please join me in welcoming
>Zoltan
>> >>>to
>> >>> > Hive.
>> >>> >
>> >>> > Zoltan, feel free to say a few words introducing yourself
>if you
>> >>> would
>> >>> > like to.
>> >>> >
>> >>> > Alan.
>> >>> >
>> >>> >
>> >>> >
>> >>>
>> >
>>
>>
>
>
>

Re: [ANNOUNCE] New committer: Zoltan Haindrich

[Prasanth Jayachandran]

Congratulations Zoltan!!

Thanks
Prasanth




On Tue, Feb 21, 2017 at 4:35 PM -0800, "Eugene Koifman" 
mailto:ekoif...@hortonworks.com>> wrote:


Congratulations!

On 2/21/17, 4:17 PM, "Vihang Karajgaonkar"  wrote:

Congrats Zoltan!

On Tue, Feb 21, 2017 at 4:16 PM, Vaibhav Gumashta  wrote:

> Congrats Zoltan!
>
> On 2/21/17, 4:16 PM, "Jimmy Xiang"  wrote:
>
> >Congrats, Zoltan!!
> >
> >On Tue, Feb 21, 2017 at 4:15 PM, Sushanth Sowmyan
> >wrote:
> >> Congrats, Zoltan!
> >>
> >> Welcome aboard. :)
> >>
> >> On Feb 21, 2017 15:42, "Rajesh Balamohan"
> wrote:
> >>
> >>> Congrats Zoltan. :)
> >>>
> >>> ~Rajesh.B
> >>>
> >>> On Wed, Feb 22, 2017 at 4:43 AM, Wei Zheng
> >>>wrote:
> >>>
> >>> > Congrats Zoltan!
> >>> >
> >>> > Thanks,
> >>> > Wei
> >>> >
> >>> > On 2/21/17, 13:09, "Alan Gates"  wrote:
> >>> >
> >>> > On behalf of the Hive PMC I am happy to announce Zoltan
> >>>Haindrich is
> >>> > our newest committer.  He has been contributing to Hive for several
> >>> months
> >>> > across a number of areas, including the parser, HiveServer2, and
> >>>cleaning
> >>> > up unit tests and documentation.  Please join me in welcoming Zoltan
> >>>to
> >>> > Hive.
> >>> >
> >>> > Zoltan, feel free to say a few words introducing yourself if you
> >>> would
> >>> > like to.
> >>> >
> >>> > Alan.
> >>> >
> >>> >
> >>> >
> >>>
> >
>
>

Re: [ANNOUNCE] New committer: Zoltan Haindrich

[Eugene Koifman]

Congratulations!

On 2/21/17, 4:17 PM, "Vihang Karajgaonkar"  wrote:

Congrats Zoltan!

On Tue, Feb 21, 2017 at 4:16 PM, Vaibhav Gumashta  wrote:

> Congrats Zoltan!
>
> On 2/21/17, 4:16 PM, "Jimmy Xiang"  wrote:
>
> >Congrats, Zoltan!!
> >
> >On Tue, Feb 21, 2017 at 4:15 PM, Sushanth Sowmyan 
> >wrote:
> >> Congrats, Zoltan!
> >>
> >> Welcome aboard. :)
> >>
> >> On Feb 21, 2017 15:42, "Rajesh Balamohan" 
> wrote:
> >>
> >>> Congrats Zoltan. :)
> >>>
> >>> ~Rajesh.B
> >>>
> >>> On Wed, Feb 22, 2017 at 4:43 AM, Wei Zheng 
> >>>wrote:
> >>>
> >>> > Congrats Zoltan!
> >>> >
> >>> > Thanks,
> >>> > Wei
> >>> >
> >>> > On 2/21/17, 13:09, "Alan Gates"  wrote:
> >>> >
> >>> > On behalf of the Hive PMC I am happy to announce Zoltan
> >>>Haindrich is
> >>> > our newest committer.  He has been contributing to Hive for several
> >>> months
> >>> > across a number of areas, including the parser, HiveServer2, and
> >>>cleaning
> >>> > up unit tests and documentation.  Please join me in welcoming Zoltan
> >>>to
> >>> > Hive.
> >>> >
> >>> > Zoltan, feel free to say a few words introducing yourself if you
> >>> would
> >>> > like to.
> >>> >
> >>> > Alan.
> >>> >
> >>> >
> >>> >
> >>>
> >
>
>

[jira] [Created] (HIVE-16001) add test for merge + runtime filtering

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-16001:
-

 Summary: add test for merge + runtime filtering
 Key: HIVE-16001
 URL: https://issues.apache.org/jira/browse/HIVE-16001
 Project: Hive
  Issue Type: Sub-task
  Components: Transactions
Affects Versions: 2.2.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


make sure merge works with HIVE-15802 and HIVE-15269



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 56688: HIVE-15881: Use new thread count variable name instead of mapred.dfsclient.parallelism.max

[Sergey Shelukhin]

> On Feb. 21, 2017, 10:41 p.m., Thomas Poepping wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java, line 2085
> > 
> >
> > The default is already defined in HiveConf, right? Do we need to 
> > specify another default here?
> 
> Sergio Pena wrote:
> Yes. This is the only method that is available on Hiveconf
> - public int getInt(String name, int defaultValue)

HiveConf.getInt(conf, var) can be used to avoid this. It takes any 
Configuration object, not just HiveConf


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56688/#review166250
---


On Feb. 21, 2017, 9:01 p.m., Sergio Pena wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56688/
> ---
> 
> (Updated Feb. 21, 2017, 9:01 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-15881
> https://issues.apache.org/jira/browse/HIVE-15881
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Deprecates the use of mapred.dfsclient.parallelism.max, and instead use 
> hive.exec.input.listing.max.threads
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 1af59ba5879d3ad24148fa86f8ce725acef2bb63 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
> e81cbce3e333d44a4088c10491f399e92a505293 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestUtilities.java 
> 5a9d83ca4890a68a63f8f6c145d30906a58c7e73 
> 
> Diff: https://reviews.apache.org/r/56688/diff/
> 
> 
> Testing
> ---
> 
> Added tests to TestUtilities.
> Waiting for HiveQA
> 
> 
> Thanks,
> 
> Sergio Pena
> 
>

Re: Review Request 56688: HIVE-15881: Use new thread count variable name instead of mapred.dfsclient.parallelism.max

[Vihang Karajgaonkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56688/#review166274
---



LGTM. Thanks for the patch Sergio!

- Vihang Karajgaonkar


On Feb. 21, 2017, 9:01 p.m., Sergio Pena wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56688/
> ---
> 
> (Updated Feb. 21, 2017, 9:01 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-15881
> https://issues.apache.org/jira/browse/HIVE-15881
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Deprecates the use of mapred.dfsclient.parallelism.max, and instead use 
> hive.exec.input.listing.max.threads
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 1af59ba5879d3ad24148fa86f8ce725acef2bb63 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
> e81cbce3e333d44a4088c10491f399e92a505293 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestUtilities.java 
> 5a9d83ca4890a68a63f8f6c145d30906a58c7e73 
> 
> Diff: https://reviews.apache.org/r/56688/diff/
> 
> 
> Testing
> ---
> 
> Added tests to TestUtilities.
> Waiting for HiveQA
> 
> 
> Thanks,
> 
> Sergio Pena
> 
>

Re: [ANNOUNCE] New committer: Zoltan Haindrich

[Vihang Karajgaonkar]

Congrats Zoltan!

On Tue, Feb 21, 2017 at 4:16 PM, Vaibhav Gumashta  wrote:

> Congrats Zoltan!
>
> On 2/21/17, 4:16 PM, "Jimmy Xiang"  wrote:
>
> >Congrats, Zoltan!!
> >
> >On Tue, Feb 21, 2017 at 4:15 PM, Sushanth Sowmyan 
> >wrote:
> >> Congrats, Zoltan!
> >>
> >> Welcome aboard. :)
> >>
> >> On Feb 21, 2017 15:42, "Rajesh Balamohan" 
> wrote:
> >>
> >>> Congrats Zoltan. :)
> >>>
> >>> ~Rajesh.B
> >>>
> >>> On Wed, Feb 22, 2017 at 4:43 AM, Wei Zheng 
> >>>wrote:
> >>>
> >>> > Congrats Zoltan!
> >>> >
> >>> > Thanks,
> >>> > Wei
> >>> >
> >>> > On 2/21/17, 13:09, "Alan Gates"  wrote:
> >>> >
> >>> > On behalf of the Hive PMC I am happy to announce Zoltan
> >>>Haindrich is
> >>> > our newest committer.  He has been contributing to Hive for several
> >>> months
> >>> > across a number of areas, including the parser, HiveServer2, and
> >>>cleaning
> >>> > up unit tests and documentation.  Please join me in welcoming Zoltan
> >>>to
> >>> > Hive.
> >>> >
> >>> > Zoltan, feel free to say a few words introducing yourself if you
> >>> would
> >>> > like to.
> >>> >
> >>> > Alan.
> >>> >
> >>> >
> >>> >
> >>>
> >
>
>

Re: [ANNOUNCE] New committer: Zoltan Haindrich

[Vaibhav Gumashta]

Congrats Zoltan!

On 2/21/17, 4:16 PM, "Jimmy Xiang"  wrote:

>Congrats, Zoltan!!
>
>On Tue, Feb 21, 2017 at 4:15 PM, Sushanth Sowmyan 
>wrote:
>> Congrats, Zoltan!
>>
>> Welcome aboard. :)
>>
>> On Feb 21, 2017 15:42, "Rajesh Balamohan"  wrote:
>>
>>> Congrats Zoltan. :)
>>>
>>> ~Rajesh.B
>>>
>>> On Wed, Feb 22, 2017 at 4:43 AM, Wei Zheng 
>>>wrote:
>>>
>>> > Congrats Zoltan!
>>> >
>>> > Thanks,
>>> > Wei
>>> >
>>> > On 2/21/17, 13:09, "Alan Gates"  wrote:
>>> >
>>> > On behalf of the Hive PMC I am happy to announce Zoltan
>>>Haindrich is
>>> > our newest committer.  He has been contributing to Hive for several
>>> months
>>> > across a number of areas, including the parser, HiveServer2, and
>>>cleaning
>>> > up unit tests and documentation.  Please join me in welcoming Zoltan
>>>to
>>> > Hive.
>>> >
>>> > Zoltan, feel free to say a few words introducing yourself if you
>>> would
>>> > like to.
>>> >
>>> > Alan.
>>> >
>>> >
>>> >
>>>
>

Re: [ANNOUNCE] New committer: Zoltan Haindrich

[Jimmy Xiang]

Congrats, Zoltan!!

On Tue, Feb 21, 2017 at 4:15 PM, Sushanth Sowmyan  wrote:
> Congrats, Zoltan!
>
> Welcome aboard. :)
>
> On Feb 21, 2017 15:42, "Rajesh Balamohan"  wrote:
>
>> Congrats Zoltan. :)
>>
>> ~Rajesh.B
>>
>> On Wed, Feb 22, 2017 at 4:43 AM, Wei Zheng  wrote:
>>
>> > Congrats Zoltan!
>> >
>> > Thanks,
>> > Wei
>> >
>> > On 2/21/17, 13:09, "Alan Gates"  wrote:
>> >
>> > On behalf of the Hive PMC I am happy to announce Zoltan Haindrich is
>> > our newest committer.  He has been contributing to Hive for several
>> months
>> > across a number of areas, including the parser, HiveServer2, and cleaning
>> > up unit tests and documentation.  Please join me in welcoming Zoltan to
>> > Hive.
>> >
>> > Zoltan, feel free to say a few words introducing yourself if you
>> would
>> > like to.
>> >
>> > Alan.
>> >
>> >
>> >
>>

Re: [ANNOUNCE] New committer: Zoltan Haindrich

[Sushanth Sowmyan]

Congrats, Zoltan!

Welcome aboard. :)

On Feb 21, 2017 15:42, "Rajesh Balamohan"  wrote:

> Congrats Zoltan. :)
>
> ~Rajesh.B
>
> On Wed, Feb 22, 2017 at 4:43 AM, Wei Zheng  wrote:
>
> > Congrats Zoltan!
> >
> > Thanks,
> > Wei
> >
> > On 2/21/17, 13:09, "Alan Gates"  wrote:
> >
> > On behalf of the Hive PMC I am happy to announce Zoltan Haindrich is
> > our newest committer.  He has been contributing to Hive for several
> months
> > across a number of areas, including the parser, HiveServer2, and cleaning
> > up unit tests and documentation.  Please join me in welcoming Zoltan to
> > Hive.
> >
> > Zoltan, feel free to say a few words introducing yourself if you
> would
> > like to.
> >
> > Alan.
> >
> >
> >
>

Re: [VOTE] Should we release hive-storage-2.2.1rc0?

[Owen O'Malley]

With three +1's and no -1's the vote passes.

Thanks for voting Alan and Ashutosh!
   Owen

On Tue, Feb 21, 2017 at 3:47 PM, Ashutosh Chauhan 
wrote:

> +1 built from source and checked signatures. Looks good.
>
> On Fri, Feb 17, 2017 at 12:22 PM, Alan Gates  wrote:
>
> > +1.  Did a build with a clean maven repo, ran rat, checked the
> signatures,
> > checked for any binary files in the tarball.
> >
> > Alan.
> >
> > > On Feb 16, 2017, at 10:56 AM, Owen O'Malley 
> wrote:
> > >
> > > All,
> > >   We've made some fixes to the storage-api since 2.2.0 and I'd like to
> > > make a new release.
> > >
> > > Issues closed:
> > > * HIVE-15700: BytesColumnVector can get stuck trying to resize byte
> > buffer
> > > * HIVE-15143 : add logging for HIVE-15024
> > > * HIVE-15929. Fix HiveDecimalWritable compatibility.
> > >
> > > Artifacts:
> > > tag: https://github.com/apache/hive/releases/tag/storage-
> > release-2.2.1rc0
> > > tar ball: http://home.apache.org/~omalley/hive-storage-2.2.1/
> > >
> > > Thanks,
> > >   Owen
> >
> >
>

Re: [VOTE] Should we release hive-storage-2.2.1rc0?

[Ashutosh Chauhan]

+1 built from source and checked signatures. Looks good.

On Fri, Feb 17, 2017 at 12:22 PM, Alan Gates  wrote:

> +1.  Did a build with a clean maven repo, ran rat, checked the signatures,
> checked for any binary files in the tarball.
>
> Alan.
>
> > On Feb 16, 2017, at 10:56 AM, Owen O'Malley  wrote:
> >
> > All,
> >   We've made some fixes to the storage-api since 2.2.0 and I'd like to
> > make a new release.
> >
> > Issues closed:
> > * HIVE-15700: BytesColumnVector can get stuck trying to resize byte
> buffer
> > * HIVE-15143 : add logging for HIVE-15024
> > * HIVE-15929. Fix HiveDecimalWritable compatibility.
> >
> > Artifacts:
> > tag: https://github.com/apache/hive/releases/tag/storage-
> release-2.2.1rc0
> > tar ball: http://home.apache.org/~omalley/hive-storage-2.2.1/
> >
> > Thanks,
> >   Owen
>
>

Re: [ANNOUNCE] New committer: Zoltan Haindrich

[Rajesh Balamohan]

Congrats Zoltan. :)

~Rajesh.B

On Wed, Feb 22, 2017 at 4:43 AM, Wei Zheng  wrote:

> Congrats Zoltan!
>
> Thanks,
> Wei
>
> On 2/21/17, 13:09, "Alan Gates"  wrote:
>
> On behalf of the Hive PMC I am happy to announce Zoltan Haindrich is
> our newest committer.  He has been contributing to Hive for several months
> across a number of areas, including the parser, HiveServer2, and cleaning
> up unit tests and documentation.  Please join me in welcoming Zoltan to
> Hive.
>
> Zoltan, feel free to say a few words introducing yourself if you would
> like to.
>
> Alan.
>
>
>

[GitHub] hive pull request #152: HIVE-15934 : Downgrade Maven surefire plugin from 2....

[weiatwork]

GitHub user weiatwork opened a pull request:

https://github.com/apache/hive/pull/152

HIVE-15934 : Downgrade Maven surefire plugin from 2.19.1 to 2.18.1 (W…

…ei Zheng, reviewed by Zoltan Haindrich)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/hive HIVE-15934

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/152.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #152


commit cc9085617b8749b8eb0a69fb893133ac04915eb8
Author: Wei Zheng 
Date:   2017-02-21T23:31:51Z

HIVE-15934 : Downgrade Maven surefire plugin from 2.19.1 to 2.18.1 (Wei 
Zheng, reviewed by Zoltan Haindrich)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Review Request 56902: make explain formatted to include opId and etc

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56902/
---

Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-15955


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 74cec3e 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/AnnotateReduceSinkOutputOperator.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java c6287e4 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java b8c2d42 
  ql/src/test/queries/clientpositive/explain_formatted_oid.q PRE-CREATION 
  ql/src/test/results/clientpositive/explain_formatted_oid.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/56902/diff/


Testing
---


Thanks,

pengcheng xiong

Re: [ANNOUNCE] New committer: Zoltan Haindrich

[Wei Zheng]

Congrats Zoltan!

Thanks,
Wei

On 2/21/17, 13:09, "Alan Gates"  wrote:

On behalf of the Hive PMC I am happy to announce Zoltan Haindrich is our 
newest committer.  He has been contributing to Hive for several months across a 
number of areas, including the parser, HiveServer2, and cleaning up unit tests 
and documentation.  Please join me in welcoming Zoltan to Hive.

Zoltan, feel free to say a few words introducing yourself if you would like 
to.

Alan.

Re: Review Request 56688: HIVE-15881: Use new thread count variable name instead of mapred.dfsclient.parallelism.max

[Sergio Pena]

> On Feb. 21, 2017, 10:41 p.m., Thomas Poepping wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java, line 2085
> > 
> >
> > The default is already defined in HiveConf, right? Do we need to 
> > specify another default here?

Yes. This is the only method that is available on Hiveconf
- public int getInt(String name, int defaultValue)


- Sergio


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56688/#review166250
---


On Feb. 21, 2017, 9:01 p.m., Sergio Pena wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56688/
> ---
> 
> (Updated Feb. 21, 2017, 9:01 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-15881
> https://issues.apache.org/jira/browse/HIVE-15881
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Deprecates the use of mapred.dfsclient.parallelism.max, and instead use 
> hive.exec.input.listing.max.threads
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 1af59ba5879d3ad24148fa86f8ce725acef2bb63 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
> e81cbce3e333d44a4088c10491f399e92a505293 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestUtilities.java 
> 5a9d83ca4890a68a63f8f6c145d30906a58c7e73 
> 
> Diff: https://reviews.apache.org/r/56688/diff/
> 
> 
> Testing
> ---
> 
> Added tests to TestUtilities.
> Waiting for HiveQA
> 
> 
> Thanks,
> 
> Sergio Pena
> 
>

Re: Review Request 56688: HIVE-15881: Use new thread count variable name instead of mapred.dfsclient.parallelism.max

[Thomas Poepping]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56688/#review166250
---



Otherwise LGTM


ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java (line 2084)


The default is already defined in HiveConf, right? Do we need to specify 
another default here?


- Thomas Poepping


On Feb. 21, 2017, 9:01 p.m., Sergio Pena wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56688/
> ---
> 
> (Updated Feb. 21, 2017, 9:01 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-15881
> https://issues.apache.org/jira/browse/HIVE-15881
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Deprecates the use of mapred.dfsclient.parallelism.max, and instead use 
> hive.exec.input.listing.max.threads
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 1af59ba5879d3ad24148fa86f8ce725acef2bb63 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
> e81cbce3e333d44a4088c10491f399e92a505293 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestUtilities.java 
> 5a9d83ca4890a68a63f8f6c145d30906a58c7e73 
> 
> Diff: https://reviews.apache.org/r/56688/diff/
> 
> 
> Testing
> ---
> 
> Added tests to TestUtilities.
> Waiting for HiveQA
> 
> 
> Thanks,
> 
> Sergio Pena
> 
>

Re: Review Request 56546: Allow user to update AVRO table schema via command even if table has external schema

[Aihua Xu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56546/#review166239
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (line 1207)


Should the command like 'alter table add column' always alter the avro 
schema? Just wondering if it's necessary to add this configuration.

So what we will do is to update the schema file?



ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java (line 3561)


I guess it's for performance reason that we are not getting sd all the time 
since for some following cases, probably sd is not used. Can you double check?



ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java (line 3568)


This is avro specific call. I'm wondering if we can achieve this by 
something like, giving SerDe a function handleAddColumn(). Not sure if it makes 
sense. Can you take a look?



ql/src/test/queries/clientpositive/avro_add_column_extschema.q (line 27)


Can you also print out show create table? I'm wondering if the schema 
literal gets updated? And also if we use avro.schema.url instead of literal, 
will it work as well?


- Aihua Xu


On Feb. 21, 2017, 9:52 p.m., Adam Szita wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56546/
> ---
> 
> (Updated Feb. 21, 2017, 9:52 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Peter Vary, and Sergio Pena.
> 
> 
> Bugs: HIVE-13780
> https://issues.apache.org/jira/browse/HIVE-13780
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Allow user to update AVRO table schema via command even if table's definition 
> was defined through schema file / literal in tblproperties
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> b27b663b94f41a8250b79139ed9f7275b10cf9a3 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> adabe70fa8f0fe1b990c6ac578a14ff5af06fc93 
>   ql/src/test/queries/clientnegative/avro_add_column_extschema.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/avro_add_column_extschema.q PRE-CREATION 
>   ql/src/test/results/clientnegative/avro_add_column_extschema.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/avro_add_column_extschema.q.out 
> PRE-CREATION 
>   serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java 
> f18585da1d108abdd500437362eb388b21030ec7 
> 
> Diff: https://reviews.apache.org/r/56546/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Adam Szita
> 
>

[jira] [Created] (HIVE-16000) LLAP: LLAP log urls improvements

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-16000:


 Summary: LLAP: LLAP log urls improvements
 Key: HIVE-16000
 URL: https://issues.apache.org/jira/browse/HIVE-16000
 Project: Hive
  Issue Type: Bug
  Components: llap
Affects Versions: 2.2.0
Reporter: Prasanth Jayachandran


Follow up for HIVE-15971
1) Make NodeManager web address port available via ServiceInstance or something 
better (other than reading from configuration)
2) When llap node goes down log URL cannot be constructed since we rely on 
information from service registry. Instead YARN NodeId can be extended to 
provided necessary information (container id) for constructing the log url. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-15999) Fix flakiness in TestDbTxnManager2

[Wei Zheng (JIRA)]

Wei Zheng created HIVE-15999:


 Summary: Fix flakiness in TestDbTxnManager2
 Key: HIVE-15999
 URL: https://issues.apache.org/jira/browse/HIVE-15999
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 2.2.0
Reporter: Wei Zheng
Assignee: Wei Zheng


Right now there is test flakiness wrt. TestDbTxnManager2. The error is like 
this:
{code}
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.checkExpectedLocks
 Error Details
Table/View 'TXNS' already exists in Schema 'APP'.
{code}
The failure is due to HiveConf used in the test being polluted by some test, 
e.g. in testDummyTxnManagerOnAcidTable(), conf entry HIVE_TXN_MANAGER is set to 
"org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager" but not switched back.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[ANNOUNCE] New committer: Zoltan Haindrich

[Alan Gates]

On behalf of the Hive PMC I am happy to announce Zoltan Haindrich is our newest 
committer.  He has been contributing to Hive for several months across a number 
of areas, including the parser, HiveServer2, and cleaning up unit tests and 
documentation.  Please join me in welcoming Zoltan to Hive.

Zoltan, feel free to say a few words introducing yourself if you would like to.

Alan.

Re: Review Request 56688: HIVE-15881: Use new thread count variable name instead of mapred.dfsclient.parallelism.max

[Sergio Pena]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56688/
---

(Updated Feb. 21, 2017, 9:01 p.m.)


Review request for hive.


Changes
---

Addressed comments from Thomas.


Bugs: HIVE-15881
https://issues.apache.org/jira/browse/HIVE-15881


Repository: hive-git


Description
---

Deprecates the use of mapred.dfsclient.parallelism.max, and instead use 
hive.exec.input.listing.max.threads


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
1af59ba5879d3ad24148fa86f8ce725acef2bb63 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
e81cbce3e333d44a4088c10491f399e92a505293 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestUtilities.java 
5a9d83ca4890a68a63f8f6c145d30906a58c7e73 

Diff: https://reviews.apache.org/r/56688/diff/


Testing
---

Added tests to TestUtilities.
Waiting for HiveQA


Thanks,

Sergio Pena

Re: Review Request 56763: HIVE-15931: JDBC: Improve logging when using ZooKeeper

[Vaibhav Gumashta]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56763/
---

(Updated Feb. 21, 2017, 9:01 p.m.)


Review request for hive, Peter Vary and Thejas Nair.


Bugs: HIVE-15931
https://issues.apache.org/jira/browse/HIVE-15931


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-15931


Diffs (updated)
-

  jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 535ad3d 
  jdbc/src/java/org/apache/hive/jdbc/HiveDriver.java a349f8b 
  jdbc/src/java/org/apache/hive/jdbc/Utils.java bfae8b9 
  jdbc/src/java/org/apache/hive/jdbc/ZooKeeperHiveClientHelper.java 8d6003a 

Diff: https://reviews.apache.org/r/56763/diff/


Testing
---


Thanks,

Vaibhav Gumashta

[jira] [Created] (HIVE-15998) Flaky test: TestCliDriver index_auto_mult_tables_compact

[Thejas M Nair (JIRA)]

Thejas M Nair created HIVE-15998:


 Summary: Flaky test: TestCliDriver index_auto_mult_tables_compact
 Key: HIVE-15998
 URL: https://issues.apache.org/jira/browse/HIVE-15998
 Project: Hive
  Issue Type: Sub-task
Reporter: Thejas M Nair


This was seen in 
https://builds.apache.org/job/PreCommit-HIVE-Build/3666/testReport/ and 
https://builds.apache.org/job/PreCommit-HIVE-Build/3571/testReport

However, the tests run fine locally.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-15997) Resource leaks when query is cancelled

[Yongzhi Chen (JIRA)]

Yongzhi Chen created HIVE-15997:
---

 Summary: Resource leaks when query is cancelled 
 Key: HIVE-15997
 URL: https://issues.apache.org/jira/browse/HIVE-15997
 Project: Hive
  Issue Type: Bug
Reporter: Yongzhi Chen
Assignee: Yongzhi Chen


There may some resource leaks when query is cancelled.
We see following stacks in the log:
Possible files and folder leak:
{noformat}
2017-02-02 06:23:25,410 WARN  hive.ql.Context: [HiveServer2-Background-Pool: 
Thread-61]: Error Removing Scratch: java.io.IOException: Failed on local 
exception: java.nio.channels.ClosedByInterruptException; Host Details : local 
host is: "ychencdh511t-1.vpc.cloudera.com/172.26.11.50"; destination host is: 
"ychencdh511t-1.vpc.cloudera.com":8020; 
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1476)
at org.apache.hadoop.ipc.Client.call(Client.java:1409)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy25.delete(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:535)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy26.delete(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2059)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:671)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:671)
at org.apache.hadoop.hive.ql.Context.removeScratchDir(Context.java:405)
at org.apache.hadoop.hive.ql.Context.clear(Context.java:541)
at org.apache.hadoop.hive.ql.Driver.releaseContext(Driver.java:2109)
at org.apache.hadoop.hive.ql.Driver.closeInProcess(Driver.java:2150)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1472)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1212)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1207)
at 
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
at 
org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
at 
org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:293)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
at 
org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:306)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedByInterruptException
at 
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:681)
at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
at 
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:615)
at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:714)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:376)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1525)
at org.apache.hadoop.ipc.Client.call(Client.java:1448)
... 35 more

2017-02-02 12:26:52,706 INFO  
org.apache.hive.service.cli.operation.OperationManager: 
[HiveServer2-Background-Pool: Thread-23]: Operation is timed 
out,operation=OperationHandle [opType=EXECUTE_STATEMENT, 
getHandleIdentifier()=2af82100-94cf-4f

[jira] [Created] (HIVE-15996) Implement multiargument GROUPING function

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-15996:
--

 Summary: Implement multiargument GROUPING function
 Key: HIVE-15996
 URL: https://issues.apache.org/jira/browse/HIVE-15996
 Project: Hive
  Issue Type: New Feature
Affects Versions: 2.2.0
Reporter: Carter Shanklin
Assignee: Jesus Camacho Rodriguez


Per the SQL standard section 6.9:
GROUPING ( CR1, ..., CRN-1, CRN )
is equivalent to:
CAST ( ( 2 * GROUPING ( CR1, ..., CRN-1 ) + GROUPING ( CRN ) ) AS IDT )
So for example:
select c1, c2, c3, grouping(c1, c2, c3) from e011_02 group by rollup(c1, c2, 
c3);
Should be allowed and equivalent to:
select c1, c2, c3, 4*grouping(c1) + 2*grouping(c2) + grouping(c3) from e011_02 
group by rollup(c1, c2, c3);



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-15995) Syncing metastore table with serde schema

[Michal Ferlinski (JIRA)]

Michal Ferlinski created HIVE-15995:
---

 Summary: Syncing metastore table with serde schema
 Key: HIVE-15995
 URL: https://issues.apache.org/jira/browse/HIVE-15995
 Project: Hive
  Issue Type: Bug
Reporter: Michal Ferlinski


Hive enables table schema evolution via properties. For avro e.g. we could 
alter the 'avro.schema.url' property to update table schema to the next 
version. Updating properties however doesn't affect column list stored in 
metastore DB so the table is not in the newest version when returned from 
metastore API. This is problem for tools working with metastore (e.g. Presto).

To solve this issue I suggest to introduce new DDL statement syncing metastore 
columns with those from serde:
{code}
ALTER TABLE user_test1 UPDATE COLUMNS
{code}

This will be serde format independent solution. 
Let me know what you think. Patch is in progress.

To reproduce, follow the instructions below:
- Create table based on avro schema version 1 (cxv1.avsc)
{code}
CREATE EXTERNAL TABLE user_test1
  PARTITIONED BY (dt string)
  ROW FORMAT SERDE
  'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
  STORED AS INPUTFORMAT
  'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
  OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
  LOCATION
  '/tmp/schema-evolution/user_test1'
  TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
{code}
- Update schema to version 2 (cx2.avsc)
{code}
ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
'/tmp/schema-evolution/cx2.avsc');
{code}
- Print serde columns (top info) and metastore columns (Detailed Table 
Information):
{code}
DESCRIBE EXTENDED user_test1
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-15994) Grouping function error when grouping sets are not specified

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-15994:
--

 Summary: Grouping function error when grouping sets are not 
specified
 Key: HIVE-15994
 URL: https://issues.apache.org/jira/browse/HIVE-15994
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Carter Shanklin
Assignee: Jesus Camacho Rodriguez


hive> select grouping(c1), lead(c2) over(partition by grouping(c1)) from 
e011_01 group by c1, c2;
FAILED: SemanticException Failed to breakup Windowing invocations into Groups. 
At least 1 group must only depend on input columns. Also check for circular 
dependencies.
Underlying error: org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:48 
Arguments length mismatch 'c1': grouping() requires 2 argument, got 1



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)