Review Request 63533: HIVE-17906 use kill query mechanics to kill queries in WM

2017-11-02 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63533/
---

Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPool.java 9c2b3ea15c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WmTezSession.java 0dd1433a6c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
b0c6d583f0 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
94f42ddae0 


Diff: https://reviews.apache.org/r/63533/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-17972) Implement Parquet vectorization reader for Map type

2017-11-02 Thread Colin Ma (JIRA) 
Colin Ma created HIVE-17972:
---

 Summary: Implement Parquet vectorization reader for Map type
 Key: HIVE-17972
 URL: https://issues.apache.org/jira/browse/HIVE-17972
 Project: Hive
  Issue Type: Sub-task
Reporter: Colin Ma
Assignee: Colin Ma
Priority: Major


Parquet vectorized reader can't support map type, it should be supported to 
improve the performance when the query with map type.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 63528: HIVE-17969: Metastore to alter table in batches of partitions when renaming table

2017-11-02 Thread Peter Vary via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63528/#review189979
---



Looks good. One question only.
Thanks, Peter


standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
Lines 265 (patched)


Shouldn't we use 
parts = msdb.getPartitions(dbname, name, partitionBatchSize);
instead
parts = msdb.getPartitions(dbname, name, -1);

This way we will load only the required amount of partitions, instead of 
all.


- Peter Vary


On Nov. 2, 2017, 10:02 p.m., Adam Szita wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63528/
> ---
> 
> (Updated Nov. 2, 2017, 10:02 p.m.)
> 
> 
> Review request for hive, Peter Vary and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-17969
> https://issues.apache.org/jira/browse/HIVE-17969
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Refactoring alter table code to use batching of partitions when calling the 
> heavy removeUnusedColumnDescriptor method
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
>  ccadac1ada6aaae884ab39f5d99e91b8c542404e 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  62801c53853dbafb7c425cff943ec819dcee4800 
> 
> 
> Diff: https://reviews.apache.org/r/63528/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Adam Szita
> 
>

Review Request 63528: HIVE-17969: Metastore to alter table in batches of partitions when renaming table

2017-11-02 Thread Adam Szita via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63528/
---

Review request for hive, Peter Vary and Barna Zsombor Klara.


Bugs: HIVE-17969
https://issues.apache.org/jira/browse/HIVE-17969


Repository: hive-git


Description
---

Refactoring alter table code to use batching of partitions when calling the 
heavy removeUnusedColumnDescriptor method


Diffs
-

  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 ccadac1ada6aaae884ab39f5d99e91b8c542404e 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 62801c53853dbafb7c425cff943ec819dcee4800 


Diff: https://reviews.apache.org/r/63528/diff/1/


Testing
---


Thanks,

Adam Szita

Re: Review Request 63427: HIVE-17936

2017-11-02 Thread Lefty Leverenz 


> On Nov. 1, 2017, 3:48 a.m., Lefty Leverenz wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Lines 3017-3018 (patched)
> > 
> >
> > Please spell out the acronyms TS, DPP, and nDVs in the parameter 
> > description.
> > 
> > Also, the description isn't clear to me -- "to control if ..." what?  
> > Something about reduction, or feeding?  (Forgive my ignorance, I'm just 
> > looking at the grammatical structure of the sentence along with the 
> > parameter name.)
> 
> Deepak Jaiswal wrote:
> Thanks for the feedback. How about this below?
> 
> The factor to decide if semijoin branch feeding into a TableScan which 
> has an outgoing Dynamic Partition pruning(DPP) branch based on number of 
> distinct values.
> 
> Lefty Leverenz wrote:
> Is "feeding" what gets decided?  (In other words, should it be "feeds"?)  
> If not, what's the verb that "decide if" refers to?
> 
> Thanks for spelling out the acronyms.  Keeping DPP in parentheses is 
> good, although you need a space before the opening parenthesis.  And maybe 
> "pruning" should be "Pruning" just for consistency.
> 
> Deepak Jaiswal wrote:
> Thanks. Let me update the patch.

Looks good, thanks again.

+1 (just for the parameter description)


- Lefty


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63427/#review189778
---


On Nov. 2, 2017, 8:45 p.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63427/
> ---
> 
> (Updated Nov. 2, 2017, 8:45 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jason Dere.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Dynamic Semijoin Reduction : markSemiJoinForDPP marks unwanted semijoin 
> branches
> 
> In method markSemiJoinForDPP (HIVE-17399), the nDVs comparison should not 
> have equality as there is a chance that the values are same on both sides and 
> the branch is still marked as good when it shouldn't be.
> Add a configurable factor to see how useful this is if nDVs on smaller side 
> are only slightly less than that on TS side.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6631a6e45d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java da30c3b642 
>   ql/src/test/queries/clientpositive/dynamic_semijoin_reduction.q 6cc0a7f7a9 
>   ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction.q.out 
> 1a1a4d9b2d 
> 
> 
> Diff: https://reviews.apache.org/r/63427/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: Review Request 63427: HIVE-17936

2017-11-02 Thread Deepak Jaiswal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63427/
---

(Updated Nov. 2, 2017, 8:45 p.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Repository: hive-git


Description
---

Dynamic Semijoin Reduction : markSemiJoinForDPP marks unwanted semijoin branches

In method markSemiJoinForDPP (HIVE-17399), the nDVs comparison should not have 
equality as there is a chance that the values are same on both sides and the 
branch is still marked as good when it shouldn't be.
Add a configurable factor to see how useful this is if nDVs on smaller side are 
only slightly less than that on TS side.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6631a6e45d 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java da30c3b642 
  ql/src/test/queries/clientpositive/dynamic_semijoin_reduction.q 6cc0a7f7a9 
  ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction.q.out 
1a1a4d9b2d 


Diff: https://reviews.apache.org/r/63427/diff/3/

Changes: https://reviews.apache.org/r/63427/diff/2-3/


Testing
---


Thanks,

Deepak Jaiswal

Re: Review Request 63346: HIVE-17902 add a notions of default pool and unmanaged mapping

2017-11-02 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63346/
---

(Updated Nov. 2, 2017, 8:43 p.m.)


Review request for hive, Harish Jaiprakash and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestTriggersWorkloadManager.java
 0ec7e85f5c 
  metastore/scripts/upgrade/derby/046-HIVE-17566.derby.sql 16190dc0a1 
  metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql 4c35380af4 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql c1578fcda9 
  metastore/scripts/upgrade/mssql/031-HIVE-17566.mssql.sql 0ee0f758f1 
  metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql 86f3e5c4a9 
  metastore/scripts/upgrade/mysql/046-HIVE-17566.mysql.sql 1affa7a119 
  metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql 2453bb92c6 
  metastore/scripts/upgrade/oracle/046-HIVE-17566.oracle.sql 0ee0f758f1 
  metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql 47bfc5ae08 
  metastore/scripts/upgrade/postgres/045-HIVE-17566.postgres.sql aa27a6486b 
  metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql 5c770e2659 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 591be49af1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java a1b7cfb158 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 
96dc7d3c98 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
b0c6d583f0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManagerFederation.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java
 f1258bab17 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/TextMetaDataFormatter.java
 9df1b82d62 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
6a2ff75c84 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 0263df0d04 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 306559c722 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 69a1d63f93 
  ql/src/java/org/apache/hadoop/hive/ql/plan/AlterResourcePlanDesc.java 
f0658a665e 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
94f42ddae0 
  ql/src/test/queries/clientpositive/resourceplan.q f1be695cb8 
  ql/src/test/results/clientpositive/llap/resourceplan.q.out 5cdfc9de3e 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 2c4fe7fc2b 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 62801c5385 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MWMResourcePlan.java
 27a1bd8f25 
  standalone-metastore/src/main/resources/package.jdo 296e273ec8 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 60531457b7 


Diff: https://reviews.apache.org/r/63346/diff/7/

Changes: https://reviews.apache.org/r/63346/diff/6-7/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 63427: HIVE-17936

2017-11-02 Thread Deepak Jaiswal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63427/
---

(Updated Nov. 2, 2017, 8:40 p.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Summary (updated)
-

HIVE-17936


Repository: hive-git


Description
---

Dynamic Semijoin Reduction : markSemiJoinForDPP marks unwanted semijoin branches

In method markSemiJoinForDPP (HIVE-17399), the nDVs comparison should not have 
equality as there is a chance that the values are same on both sides and the 
branch is still marked as good when it shouldn't be.
Add a configurable factor to see how useful this is if nDVs on smaller side are 
only slightly less than that on TS side.


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6631a6e45d 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java da30c3b642 
  ql/src/test/queries/clientpositive/dynamic_semijoin_reduction.q 6cc0a7f7a9 
  ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction.q.out 
1a1a4d9b2d 


Diff: https://reviews.apache.org/r/63427/diff/2/


Testing
---


Thanks,

Deepak Jaiswal

Re: Review Request 63427: HIVE-17396

2017-11-02 Thread Deepak Jaiswal 


> On Nov. 1, 2017, 3:48 a.m., Lefty Leverenz wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Lines 3017-3018 (patched)
> > 
> >
> > Please spell out the acronyms TS, DPP, and nDVs in the parameter 
> > description.
> > 
> > Also, the description isn't clear to me -- "to control if ..." what?  
> > Something about reduction, or feeding?  (Forgive my ignorance, I'm just 
> > looking at the grammatical structure of the sentence along with the 
> > parameter name.)
> 
> Deepak Jaiswal wrote:
> Thanks for the feedback. How about this below?
> 
> The factor to decide if semijoin branch feeding into a TableScan which 
> has an outgoing Dynamic Partition pruning(DPP) branch based on number of 
> distinct values.
> 
> Lefty Leverenz wrote:
> Is "feeding" what gets decided?  (In other words, should it be "feeds"?)  
> If not, what's the verb that "decide if" refers to?
> 
> Thanks for spelling out the acronyms.  Keeping DPP in parentheses is 
> good, although you need a space before the opening parenthesis.  And maybe 
> "pruning" should be "Pruning" just for consistency.

Thanks. Let me update the patch.


- Deepak


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63427/#review189778
---


On Oct. 31, 2017, 1:19 a.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63427/
> ---
> 
> (Updated Oct. 31, 2017, 1:19 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jason Dere.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Dynamic Semijoin Reduction : markSemiJoinForDPP marks unwanted semijoin 
> branches
> 
> In method markSemiJoinForDPP (HIVE-17399), the nDVs comparison should not 
> have equality as there is a chance that the values are same on both sides and 
> the branch is still marked as good when it shouldn't be.
> Add a configurable factor to see how useful this is if nDVs on smaller side 
> are only slightly less than that on TS side.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6631a6e45d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java da30c3b642 
>   ql/src/test/queries/clientpositive/dynamic_semijoin_reduction.q 6cc0a7f7a9 
>   ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction.q.out 
> 1a1a4d9b2d 
> 
> 
> Diff: https://reviews.apache.org/r/63427/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: Review Request 63427: HIVE-17396

2017-11-02 Thread Lefty Leverenz 


> On Nov. 1, 2017, 3:48 a.m., Lefty Leverenz wrote:
> >

Nit:  This RB summary has the wrong JIRA number -- it should be 17936, not 
17396.

But the first line of the description has the correct JIRA text, so it's easy 
enough to find with a search.


- Lefty


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63427/#review189778
---


On Oct. 31, 2017, 1:19 a.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63427/
> ---
> 
> (Updated Oct. 31, 2017, 1:19 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jason Dere.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Dynamic Semijoin Reduction : markSemiJoinForDPP marks unwanted semijoin 
> branches
> 
> In method markSemiJoinForDPP (HIVE-17399), the nDVs comparison should not 
> have equality as there is a chance that the values are same on both sides and 
> the branch is still marked as good when it shouldn't be.
> Add a configurable factor to see how useful this is if nDVs on smaller side 
> are only slightly less than that on TS side.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6631a6e45d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java da30c3b642 
>   ql/src/test/queries/clientpositive/dynamic_semijoin_reduction.q 6cc0a7f7a9 
>   ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction.q.out 
> 1a1a4d9b2d 
> 
> 
> Diff: https://reviews.apache.org/r/63427/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: Review Request 63427: HIVE-17396

2017-11-02 Thread Lefty Leverenz 


> On Nov. 1, 2017, 3:48 a.m., Lefty Leverenz wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Lines 3017-3018 (patched)
> > 
> >
> > Please spell out the acronyms TS, DPP, and nDVs in the parameter 
> > description.
> > 
> > Also, the description isn't clear to me -- "to control if ..." what?  
> > Something about reduction, or feeding?  (Forgive my ignorance, I'm just 
> > looking at the grammatical structure of the sentence along with the 
> > parameter name.)
> 
> Deepak Jaiswal wrote:
> Thanks for the feedback. How about this below?
> 
> The factor to decide if semijoin branch feeding into a TableScan which 
> has an outgoing Dynamic Partition pruning(DPP) branch based on number of 
> distinct values.

Is "feeding" what gets decided?  (In other words, should it be "feeds"?)  If 
not, what's the verb that "decide if" refers to?

Thanks for spelling out the acronyms.  Keeping DPP in parentheses is good, 
although you need a space before the opening parenthesis.  And maybe "pruning" 
should be "Pruning" just for consistency.


- Lefty


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63427/#review189778
---


On Oct. 31, 2017, 1:19 a.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63427/
> ---
> 
> (Updated Oct. 31, 2017, 1:19 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jason Dere.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Dynamic Semijoin Reduction : markSemiJoinForDPP marks unwanted semijoin 
> branches
> 
> In method markSemiJoinForDPP (HIVE-17399), the nDVs comparison should not 
> have equality as there is a chance that the values are same on both sides and 
> the branch is still marked as good when it shouldn't be.
> Add a configurable factor to see how useful this is if nDVs on smaller side 
> are only slightly less than that on TS side.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6631a6e45d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java da30c3b642 
>   ql/src/test/queries/clientpositive/dynamic_semijoin_reduction.q 6cc0a7f7a9 
>   ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction.q.out 
> 1a1a4d9b2d 
> 
> 
> Diff: https://reviews.apache.org/r/63427/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: ptest not picking patches

2017-11-02 Thread Deepak Jaiswal 
In my case, I just kept reuploading the patches by updating the patch version. 
Retrying again.

On 11/2/17, 12:00 PM, "Prasanth Jayachandran"  
wrote:

So this essentially means a patch can no longer go from resolved -> 
reopened -> patch available state anymore?
I also uploaded the same patch (from HIVE-17834) to a different jira 
HIVE-17885 but that was also not picked up.

Thanks
Prasanth

> On Nov 2, 2017, at 11:53 AM, Vihang Karajgaonkar  
wrote:
> 
> I see this on the latest run on the pre-commit job's console output
> https://builds.apache.org/job/PreCommit-Admin/319187/console
> 
> HIVE-17834,12895143 has been processed, ignoring
> 
> 
> 
> 
> 
> On Thu, Nov 2, 2017 at 11:49 AM, Prasanth Jayachandran <
> pjayachand...@hortonworks.com> wrote:
> 
>> I don’t see my jira picked up too HIVE-17834, HIVE-17885. Some patches
>> gets picked but some of them are not.
>> 
>> Thanks
>> Prasanth
>> 
>>> On Nov 2, 2017, at 11:47 AM, Vihang Karajgaonkar 
>> wrote:
>>> 
>>> I don't see any problems currently on the queue.
>>> https://builds.apache.org/job/PreCommit-HIVE-Build/
>>> 
>>> What was your JIRA number?
>>> 
>>> On Thu, Nov 2, 2017 at 11:44 AM, Deepak Jaiswal <
>> djais...@hortonworks.com>
>>> wrote:
>>> 
 Is there anyone looking at it?
 My third attempt failed. All the tests which were in queue last night
>> are
 not there.
 Was Jenkins rebooted again?
 
 On 11/1/17, 9:35 PM, "Vineet Garg"  wrote:
 
   Same here. I have submitted/re-submitted patches for two jiras and
 they haven’t been picked up for few days now. I saw one of them in 
queue
 today but I no longer see it.
 
   Vineet
 
> On Nov 1, 2017, at 9:27 PM, Prasanth Jayachandran <
 pjayachand...@hortonworks.com> wrote:
> 
> There is definitely something going on. I submitted multiple patches
 for precommit today and all of them are gone from the queue.
> 
> Thanks
> Prasanth
> 
> 
>> On Nov 1, 2017, at 11:28 AM, Vihang Karajgaonkar <
 vih...@cloudera.com> wrote:
>> 
>> Is there a way to find out if Jenkins was restarted since we
 submitted the
>> patch? Not sure if we can fix this in our precommit job. There have
 been
>> far too many instances recently when Jenkins was either restarted
 or was
>> non-responsive for long durations which delayed testing for our
 patches.
>> 
>> On Wed, Nov 1, 2017 at 11:22 AM, Ashutosh Chauhan <
 hashut...@apache.org>
>> wrote:
>> 
>>> There was an apache infra issue which was making jenkins to not
 submit
>>> patches to ptest server. https://issues.apache.org/
 jira/browse/INFRA-15358
>>> Also jenkins was restarted overnight.
>>> 
>>> On Wed, Nov 1, 2017 at 11:15 AM, Chris Drome
 
>>> wrote:
>>> 
 Me as well for HIVE-17853.
 
 Attempting to cancel/resubmit patches.
 
 On Wed, Nov 1, 2017 at 10:50 AM, Prasanth Jayachandran <
 pjayachand...@hortonworks.com> wrote:
 
> Me too. Seen this yesterday with HIVE-17834.
> 
> Thanks
> Prasanth
> 
> 
> 
> On Wed, Nov 1, 2017 at 10:26 AM -0700, "Deepak Jaiswal" <
> djais...@hortonworks.com>
 wrote:
> 
> 
> Hi,
> 
> I uploaded couple of patches but they don't appear in the
 Pre-commit
 tests
> queue.
> Is anyone else facing this?
> 
> Regards,
> Deepak
> 
> 
 
>>> 
> 
 
 
 
 
>> 
>>

Re: ptest not picking patches

2017-11-02 Thread Prasanth Jayachandran 
So this essentially means a patch can no longer go from resolved -> reopened -> 
patch available state anymore?
I also uploaded the same patch (from HIVE-17834) to a different jira HIVE-17885 
but that was also not picked up.

Thanks
Prasanth

> On Nov 2, 2017, at 11:53 AM, Vihang Karajgaonkar  wrote:
> 
> I see this on the latest run on the pre-commit job's console output
> https://builds.apache.org/job/PreCommit-Admin/319187/console
> 
> HIVE-17834,12895143 has been processed, ignoring
> 
> 
> 
> 
> 
> On Thu, Nov 2, 2017 at 11:49 AM, Prasanth Jayachandran <
> pjayachand...@hortonworks.com> wrote:
> 
>> I don’t see my jira picked up too HIVE-17834, HIVE-17885. Some patches
>> gets picked but some of them are not.
>> 
>> Thanks
>> Prasanth
>> 
>>> On Nov 2, 2017, at 11:47 AM, Vihang Karajgaonkar 
>> wrote:
>>> 
>>> I don't see any problems currently on the queue.
>>> https://builds.apache.org/job/PreCommit-HIVE-Build/
>>> 
>>> What was your JIRA number?
>>> 
>>> On Thu, Nov 2, 2017 at 11:44 AM, Deepak Jaiswal <
>> djais...@hortonworks.com>
>>> wrote:
>>> 
 Is there anyone looking at it?
 My third attempt failed. All the tests which were in queue last night
>> are
 not there.
 Was Jenkins rebooted again?
 
 On 11/1/17, 9:35 PM, "Vineet Garg"  wrote:
 
   Same here. I have submitted/re-submitted patches for two jiras and
 they haven’t been picked up for few days now. I saw one of them in queue
 today but I no longer see it.
 
   Vineet
 
> On Nov 1, 2017, at 9:27 PM, Prasanth Jayachandran <
 pjayachand...@hortonworks.com> wrote:
> 
> There is definitely something going on. I submitted multiple patches
 for precommit today and all of them are gone from the queue.
> 
> Thanks
> Prasanth
> 
> 
>> On Nov 1, 2017, at 11:28 AM, Vihang Karajgaonkar <
 vih...@cloudera.com> wrote:
>> 
>> Is there a way to find out if Jenkins was restarted since we
 submitted the
>> patch? Not sure if we can fix this in our precommit job. There have
 been
>> far too many instances recently when Jenkins was either restarted
 or was
>> non-responsive for long durations which delayed testing for our
 patches.
>> 
>> On Wed, Nov 1, 2017 at 11:22 AM, Ashutosh Chauhan <
 hashut...@apache.org>
>> wrote:
>> 
>>> There was an apache infra issue which was making jenkins to not
 submit
>>> patches to ptest server. https://issues.apache.org/
 jira/browse/INFRA-15358
>>> Also jenkins was restarted overnight.
>>> 
>>> On Wed, Nov 1, 2017 at 11:15 AM, Chris Drome
 
>>> wrote:
>>> 
 Me as well for HIVE-17853.
 
 Attempting to cancel/resubmit patches.
 
 On Wed, Nov 1, 2017 at 10:50 AM, Prasanth Jayachandran <
 pjayachand...@hortonworks.com> wrote:
 
> Me too. Seen this yesterday with HIVE-17834.
> 
> Thanks
> Prasanth
> 
> 
> 
> On Wed, Nov 1, 2017 at 10:26 AM -0700, "Deepak Jaiswal" <
> djais...@hortonworks.com>
 wrote:
> 
> 
> Hi,
> 
> I uploaded couple of patches but they don't appear in the
 Pre-commit
 tests
> queue.
> Is anyone else facing this?
> 
> Regards,
> Deepak
> 
> 
 
>>> 
> 
 
 
 
 
>> 
>>

Re: ptest not picking patches

2017-11-02 Thread Vihang Karajgaonkar 
I see this on the latest run on the pre-commit job's console output
https://builds.apache.org/job/PreCommit-Admin/319187/console

HIVE-17834,12895143 has been processed, ignoring





On Thu, Nov 2, 2017 at 11:49 AM, Prasanth Jayachandran <
pjayachand...@hortonworks.com> wrote:

> I don’t see my jira picked up too HIVE-17834, HIVE-17885. Some patches
> gets picked but some of them are not.
>
> Thanks
> Prasanth
>
> > On Nov 2, 2017, at 11:47 AM, Vihang Karajgaonkar 
> wrote:
> >
> > I don't see any problems currently on the queue.
> > https://builds.apache.org/job/PreCommit-HIVE-Build/
> >
> > What was your JIRA number?
> >
> > On Thu, Nov 2, 2017 at 11:44 AM, Deepak Jaiswal <
> djais...@hortonworks.com>
> > wrote:
> >
> >> Is there anyone looking at it?
> >> My third attempt failed. All the tests which were in queue last night
> are
> >> not there.
> >> Was Jenkins rebooted again?
> >>
> >> On 11/1/17, 9:35 PM, "Vineet Garg"  wrote:
> >>
> >>Same here. I have submitted/re-submitted patches for two jiras and
> >> they haven’t been picked up for few days now. I saw one of them in queue
> >> today but I no longer see it.
> >>
> >>Vineet
> >>
> >>> On Nov 1, 2017, at 9:27 PM, Prasanth Jayachandran <
> >> pjayachand...@hortonworks.com> wrote:
> >>>
> >>> There is definitely something going on. I submitted multiple patches
> >> for precommit today and all of them are gone from the queue.
> >>>
> >>> Thanks
> >>> Prasanth
> >>>
> >>>
>  On Nov 1, 2017, at 11:28 AM, Vihang Karajgaonkar <
> >> vih...@cloudera.com> wrote:
> 
>  Is there a way to find out if Jenkins was restarted since we
> >> submitted the
>  patch? Not sure if we can fix this in our precommit job. There have
> >> been
>  far too many instances recently when Jenkins was either restarted
> >> or was
>  non-responsive for long durations which delayed testing for our
> >> patches.
> 
>  On Wed, Nov 1, 2017 at 11:22 AM, Ashutosh Chauhan <
> >> hashut...@apache.org>
>  wrote:
> 
> > There was an apache infra issue which was making jenkins to not
> >> submit
> > patches to ptest server. https://issues.apache.org/
> >> jira/browse/INFRA-15358
> > Also jenkins was restarted overnight.
> >
> > On Wed, Nov 1, 2017 at 11:15 AM, Chris Drome
> >> 
> > wrote:
> >
> >> Me as well for HIVE-17853.
> >>
> >> Attempting to cancel/resubmit patches.
> >>
> >> On Wed, Nov 1, 2017 at 10:50 AM, Prasanth Jayachandran <
> >> pjayachand...@hortonworks.com> wrote:
> >>
> >>> Me too. Seen this yesterday with HIVE-17834.
> >>>
> >>> Thanks
> >>> Prasanth
> >>>
> >>>
> >>>
> >>> On Wed, Nov 1, 2017 at 10:26 AM -0700, "Deepak Jaiswal" <
> >>> djais...@hortonworks.com>
> >> wrote:
> >>>
> >>>
> >>> Hi,
> >>>
> >>> I uploaded couple of patches but they don't appear in the
> >> Pre-commit
> >> tests
> >>> queue.
> >>> Is anyone else facing this?
> >>>
> >>> Regards,
> >>> Deepak
> >>>
> >>>
> >>
> >
> >>>
> >>
> >>
> >>
> >>
>
>

[GitHub] hive pull request #270: Hive17967

2017-11-02 Thread alanfgates 
GitHub user alanfgates opened a pull request:

https://github.com/apache/hive/pull/270

Hive17967

See 
https://issues.apache.org/jira/browse/HIVE-17967?focusedCommentId=16236387=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16236387
 for a few important comments on this patch.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alanfgates/hive hive17967

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/270.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #270


commit 49e3ba4c13a9532fc872541536ed79ddfb744fab
Author: Alan Gates 
Date:   2017-11-02T16:21:25Z

WIP.  Have moved HMS, but tests don't all pass yet.

commit 3ff8517247286d1efcb0de5321a29859263c6639
Author: Alan Gates 
Date:   2017-10-17T16:19:22Z

Fixes after rebase.  Moved EventCleaner and DumpDirCleaner threads to 
ThreadPool.

commit e1086463c2ae9e6df14150796f5e79ae37632b12
Author: Alan Gates 
Date:   2017-10-17T18:03:43Z

Moved TestObjectStore2 back into TestObjectStore now that HiveMetaStore has 
moved.

commit d476dd585976203bac77945ead708ba58492eac1
Author: Alan Gates 
Date:   2017-11-02T18:33:40Z

Fixes after rebase.

commit 54af42ad35501c8cd0ad3bb5bae256e6191fbe14
Author: Alan Gates 
Date:   2017-11-02T18:44:43Z

Fixed warnings in pom.  Added back deprecated method in ListenerEvent.




---

Re: ptest not picking patches

2017-11-02 Thread Prasanth Jayachandran 
I don’t see my jira picked up too HIVE-17834, HIVE-17885. Some patches gets 
picked but some of them are not.

Thanks
Prasanth

> On Nov 2, 2017, at 11:47 AM, Vihang Karajgaonkar  wrote:
> 
> I don't see any problems currently on the queue.
> https://builds.apache.org/job/PreCommit-HIVE-Build/
> 
> What was your JIRA number?
> 
> On Thu, Nov 2, 2017 at 11:44 AM, Deepak Jaiswal 
> wrote:
> 
>> Is there anyone looking at it?
>> My third attempt failed. All the tests which were in queue last night are
>> not there.
>> Was Jenkins rebooted again?
>> 
>> On 11/1/17, 9:35 PM, "Vineet Garg"  wrote:
>> 
>>Same here. I have submitted/re-submitted patches for two jiras and
>> they haven’t been picked up for few days now. I saw one of them in queue
>> today but I no longer see it.
>> 
>>Vineet
>> 
>>> On Nov 1, 2017, at 9:27 PM, Prasanth Jayachandran <
>> pjayachand...@hortonworks.com> wrote:
>>> 
>>> There is definitely something going on. I submitted multiple patches
>> for precommit today and all of them are gone from the queue.
>>> 
>>> Thanks
>>> Prasanth
>>> 
>>> 
 On Nov 1, 2017, at 11:28 AM, Vihang Karajgaonkar <
>> vih...@cloudera.com> wrote:
 
 Is there a way to find out if Jenkins was restarted since we
>> submitted the
 patch? Not sure if we can fix this in our precommit job. There have
>> been
 far too many instances recently when Jenkins was either restarted
>> or was
 non-responsive for long durations which delayed testing for our
>> patches.
 
 On Wed, Nov 1, 2017 at 11:22 AM, Ashutosh Chauhan <
>> hashut...@apache.org>
 wrote:
 
> There was an apache infra issue which was making jenkins to not
>> submit
> patches to ptest server. https://issues.apache.org/
>> jira/browse/INFRA-15358
> Also jenkins was restarted overnight.
> 
> On Wed, Nov 1, 2017 at 11:15 AM, Chris Drome
>> 
> wrote:
> 
>> Me as well for HIVE-17853.
>> 
>> Attempting to cancel/resubmit patches.
>> 
>> On Wed, Nov 1, 2017 at 10:50 AM, Prasanth Jayachandran <
>> pjayachand...@hortonworks.com> wrote:
>> 
>>> Me too. Seen this yesterday with HIVE-17834.
>>> 
>>> Thanks
>>> Prasanth
>>> 
>>> 
>>> 
>>> On Wed, Nov 1, 2017 at 10:26 AM -0700, "Deepak Jaiswal" <
>>> djais...@hortonworks.com>
>> wrote:
>>> 
>>> 
>>> Hi,
>>> 
>>> I uploaded couple of patches but they don't appear in the
>> Pre-commit
>> tests
>>> queue.
>>> Is anyone else facing this?
>>> 
>>> Regards,
>>> Deepak
>>> 
>>> 
>> 
> 
>>> 
>> 
>> 
>> 
>>

Re: ptest not picking patches

2017-11-02 Thread Vihang Karajgaonkar 
I don't see any problems currently on the queue.
https://builds.apache.org/job/PreCommit-HIVE-Build/

What was your JIRA number?

On Thu, Nov 2, 2017 at 11:44 AM, Deepak Jaiswal 
wrote:

> Is there anyone looking at it?
> My third attempt failed. All the tests which were in queue last night are
> not there.
> Was Jenkins rebooted again?
>
> On 11/1/17, 9:35 PM, "Vineet Garg"  wrote:
>
> Same here. I have submitted/re-submitted patches for two jiras and
> they haven’t been picked up for few days now. I saw one of them in queue
> today but I no longer see it.
>
> Vineet
>
> > On Nov 1, 2017, at 9:27 PM, Prasanth Jayachandran <
> pjayachand...@hortonworks.com> wrote:
> >
> > There is definitely something going on. I submitted multiple patches
> for precommit today and all of them are gone from the queue.
> >
> > Thanks
> > Prasanth
> >
> >
> >> On Nov 1, 2017, at 11:28 AM, Vihang Karajgaonkar <
> vih...@cloudera.com> wrote:
> >>
> >> Is there a way to find out if Jenkins was restarted since we
> submitted the
> >> patch? Not sure if we can fix this in our precommit job. There have
> been
> >> far too many instances recently when Jenkins was either restarted
> or was
> >> non-responsive for long durations which delayed testing for our
> patches.
> >>
> >> On Wed, Nov 1, 2017 at 11:22 AM, Ashutosh Chauhan <
> hashut...@apache.org>
> >> wrote:
> >>
> >>> There was an apache infra issue which was making jenkins to not
> submit
> >>> patches to ptest server. https://issues.apache.org/
> jira/browse/INFRA-15358
> >>> Also jenkins was restarted overnight.
> >>>
> >>> On Wed, Nov 1, 2017 at 11:15 AM, Chris Drome
> 
> >>> wrote:
> >>>
>  Me as well for HIVE-17853.
> 
>  Attempting to cancel/resubmit patches.
> 
>  On Wed, Nov 1, 2017 at 10:50 AM, Prasanth Jayachandran <
>  pjayachand...@hortonworks.com> wrote:
> 
> > Me too. Seen this yesterday with HIVE-17834.
> >
> > Thanks
> > Prasanth
> >
> >
> >
> > On Wed, Nov 1, 2017 at 10:26 AM -0700, "Deepak Jaiswal" <
> > djais...@hortonworks.com>
> wrote:
> >
> >
> > Hi,
> >
> > I uploaded couple of patches but they don't appear in the
> Pre-commit
>  tests
> > queue.
> > Is anyone else facing this?
> >
> > Regards,
> > Deepak
> >
> >
> 
> >>>
> >
>
>
>
>

Re: Review Request 63458: HIVE-16917: HiveServer2 guard rails - Limit concurrent connections from user

2017-11-02 Thread Thejas Nair 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63458/#review189945
---


Ship it!




Ship It!

- Thejas Nair


On Nov. 2, 2017, 6:25 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63458/
> ---
> 
> (Updated Nov. 2, 2017, 6:25 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-16917
> https://issues.apache.org/jira/browse/HIVE-16917
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16917: HiveServer2 guard rails - Limit concurrent connections from user
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a3c853a 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/session/TestHiveSessionImpl.java
>  ebcf4a8 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionBase.java 
> 9436a25 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 0206fe3 
>   
> service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java
>  8975aee 
>   service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
> 9b2ae57 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 6354c8c 
>   
> service/src/test/org/apache/hive/service/cli/TestCLIServiceConnectionLimits.java
>  PRE-CREATION 
>   
> service/src/test/org/apache/hive/service/cli/session/TestPluggableHiveSessionImpl.java
>  47f95c5 
> 
> 
> Diff: https://reviews.apache.org/r/63458/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

Re: ptest not picking patches

2017-11-02 Thread Deepak Jaiswal 
Is there anyone looking at it?
My third attempt failed. All the tests which were in queue last night are not 
there.
Was Jenkins rebooted again?

On 11/1/17, 9:35 PM, "Vineet Garg"  wrote:

Same here. I have submitted/re-submitted patches for two jiras and they 
haven’t been picked up for few days now. I saw one of them in queue today but I 
no longer see it.

Vineet

> On Nov 1, 2017, at 9:27 PM, Prasanth Jayachandran 
 wrote:
> 
> There is definitely something going on. I submitted multiple patches for 
precommit today and all of them are gone from the queue. 
> 
> Thanks
> Prasanth
> 
> 
>> On Nov 1, 2017, at 11:28 AM, Vihang Karajgaonkar  
wrote:
>> 
>> Is there a way to find out if Jenkins was restarted since we submitted 
the
>> patch? Not sure if we can fix this in our precommit job. There have been
>> far too many instances recently when Jenkins was either restarted or was
>> non-responsive for long durations which delayed testing for our patches.
>> 
>> On Wed, Nov 1, 2017 at 11:22 AM, Ashutosh Chauhan 
>> wrote:
>> 
>>> There was an apache infra issue which was making jenkins to not submit
>>> patches to ptest server. 
https://issues.apache.org/jira/browse/INFRA-15358
>>> Also jenkins was restarted overnight.
>>> 
>>> On Wed, Nov 1, 2017 at 11:15 AM, Chris Drome 
>>> wrote:
>>> 
 Me as well for HIVE-17853.
 
 Attempting to cancel/resubmit patches.
 
 On Wed, Nov 1, 2017 at 10:50 AM, Prasanth Jayachandran <
 pjayachand...@hortonworks.com> wrote:
 
> Me too. Seen this yesterday with HIVE-17834.
> 
> Thanks
> Prasanth
> 
> 
> 
> On Wed, Nov 1, 2017 at 10:26 AM -0700, "Deepak Jaiswal" <
> djais...@hortonworks.com> wrote:
> 
> 
> Hi,
> 
> I uploaded couple of patches but they don't appear in the Pre-commit
 tests
> queue.
> Is anyone else facing this?
> 
> Regards,
> Deepak
> 
> 
 
>>> 
>

[jira] [Created] (HIVE-17971) Possible misuse of getDataSizeFromColumnStats

2017-11-02 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-17971:
---

 Summary: Possible misuse of getDataSizeFromColumnStats
 Key: HIVE-17971
 URL: https://issues.apache.org/jira/browse/HIVE-17971
 Project: Hive
  Issue Type: Bug
  Components: Statistics
Reporter: Zoltan Haindrich
Priority: Major


the same method's return value is used with 

[used as 
{{stats.setDataSize()}}|https://github.com/apache/hive/blob/10aa33072316a23ab7e21dd8d9d78a3ae1664b9a/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L512]

[used as 
{{stats.addToDataSize()}}|https://github.com/apache/hive/blob/10aa33072316a23ab7e21dd8d9d78a3ae1664b9a/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L498]

seems odd...




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 63458: HIVE-16917: HiveServer2 guard rails - Limit concurrent connections from user

2017-11-02 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63458/
---

(Updated Nov. 2, 2017, 6:25 p.m.)


Review request for hive, Sergey Shelukhin and Thejas Nair.


Changes
---

Addressed review comments. Handled forwarded addresses case as well.


Bugs: HIVE-16917
https://issues.apache.org/jira/browse/HIVE-16917


Repository: hive-git


Description
---

HIVE-16917: HiveServer2 guard rails - Limit concurrent connections from user


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a3c853a 
  
itests/hive-unit/src/test/java/org/apache/hive/service/cli/session/TestHiveSessionImpl.java
 ebcf4a8 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionBase.java 
9436a25 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
0206fe3 
  
service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java
 8975aee 
  service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
9b2ae57 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
6354c8c 
  
service/src/test/org/apache/hive/service/cli/TestCLIServiceConnectionLimits.java
 PRE-CREATION 
  
service/src/test/org/apache/hive/service/cli/session/TestPluggableHiveSessionImpl.java
 47f95c5 


Diff: https://reviews.apache.org/r/63458/diff/3/

Changes: https://reviews.apache.org/r/63458/diff/2-3/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 63458: HIVE-16917: HiveServer2 guard rails - Limit concurrent connections from user

2017-11-02 Thread j . prasanth . j 


> On Nov. 2, 2017, 5:01 p.m., Thejas Nair wrote:
> > service/src/java/org/apache/hive/service/cli/session/SessionManager.java
> > Lines 477 (patched)
> > 
> >
> > Can you also log the key on which limit has been reached ? (username 
> > and/or ipaddress).
> > It would help with troubleshooting from the server logs.

Done.


> On Nov. 2, 2017, 5:01 p.m., Thejas Nair wrote:
> > service/src/java/org/apache/hive/service/cli/session/SessionManager.java
> > Lines 508 (patched)
> > 
> >
> > safer to check - if( connectionCount >= limit )

Done.


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63458/#review189929
---


On Nov. 2, 2017, 8:08 a.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63458/
> ---
> 
> (Updated Nov. 2, 2017, 8:08 a.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-16917
> https://issues.apache.org/jira/browse/HIVE-16917
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16917: HiveServer2 guard rails - Limit concurrent connections from user
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a3c853a 
>   service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
> 9b2ae57 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 6354c8c 
>   
> service/src/test/org/apache/hive/service/cli/TestCLIServiceConnectionLimits.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/63458/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

[jira] [Created] (HIVE-17970) LOAD DATA with OVERWRITE doesn't use base_n directory concept

2017-11-02 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-17970:
-

 Summary: LOAD DATA with OVERWRITE doesn't use base_n directory 
concept
 Key: HIVE-17970
 URL: https://issues.apache.org/jira/browse/HIVE-17970
 Project: Hive
  Issue Type: Sub-task
  Components: Transactions
Affects Versions: 3.0.0
Reporter: Eugene Koifman
Priority: Major


Judging by 

{code:java}
Hive.loadTable(Path loadPath, String tableName, LoadFileType loadFileType, 
boolean isSrcLocal,
  boolean isSkewedStoreAsSubdir, boolean isAcid, boolean 
hasFollowingStatsTask,
  Long txnId, int stmtId, boolean isMmTable)
{code}

LOAD DATA with OVERWRITE will delete all existing data then write new data into 
the table.  This logic makes sense for non-acid tables but for Acid/MM it 
should work like INSERT OVERWRITE statement and write new data to base_n/. This 
way the lock manager can be used to either get an X lock for IOW and thus block 
all readers or let it run with SemiShared and let readers continue and make the 
system more concurrent.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 63458: HIVE-16917: HiveServer2 guard rails - Limit concurrent connections from user

2017-11-02 Thread Thejas Nair 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63458/#review189929
---




service/src/java/org/apache/hive/service/cli/session/SessionManager.java
Lines 477 (patched)


Can you also log the key on which limit has been reached ? (username and/or 
ipaddress).
It would help with troubleshooting from the server logs.



service/src/java/org/apache/hive/service/cli/session/SessionManager.java
Lines 508 (patched)


safer to check - if( connectionCount >= limit )


- Thejas Nair


On Nov. 2, 2017, 8:08 a.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63458/
> ---
> 
> (Updated Nov. 2, 2017, 8:08 a.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-16917
> https://issues.apache.org/jira/browse/HIVE-16917
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16917: HiveServer2 guard rails - Limit concurrent connections from user
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a3c853a 
>   service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
> 9b2ae57 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 6354c8c 
>   
> service/src/test/org/apache/hive/service/cli/TestCLIServiceConnectionLimits.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/63458/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

[jira] [Created] (HIVE-17969) Metastore to alter table in batches of partitions when renaming table

2017-11-02 Thread Adam Szita (JIRA) 
Adam Szita created HIVE-17969:
-

 Summary: Metastore to alter table in batches of partitions when 
renaming table
 Key: HIVE-17969
 URL: https://issues.apache.org/jira/browse/HIVE-17969
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Reporter: Adam Szita
Assignee: Adam Szita
Priority: Major






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 63470: HIVE-17767 Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN

2017-11-02 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63470/#review189924
---




ql/src/test/results/clientpositive/llap/subquery_in.q.out
Lines 685 (patched)


Wasteful Gby.



ql/src/test/results/clientpositive/llap/subquery_multi.q.out
Lines 2947-2950 (patched)


Extra scan  and Gby.



ql/src/test/results/clientpositive/llap/subquery_multi.q.out
Lines 3406-3409 (patched)


Extra Gby.



ql/src/test/results/clientpositive/llap/subquery_multi.q.out
Lines 3869-3871 (patched)


Extra Gby.



ql/src/test/results/clientpositive/llap/subquery_select.q.out
Line 2052 (original), 2051 (patched)


Is this change expected?



ql/src/test/results/clientpositive/subquery_in_having.q.out
Lines 1799-1808 (original), 1805-1815 (patched)


Lets move this to test to llaplocal only.



ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out
Lines 209-212 (original), 220-223 (patched)


This Gby is wasteful.


- Ashutosh Chauhan


On Nov. 2, 2017, 4:51 a.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63470/
> ---
> 
> (Updated Nov. 2, 2017, 4:51 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-17767
> https://issues.apache.org/jira/browse/HIVE-17767
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch does the following:
> * Added back earlier patch to generate value generator
> * Added logic to rewrite EXISTS/IN correlated subqueries into LEFT SEMI JOIN
> * Remove SemiJoinTransposeRule (This rule pushes semi join underneath its 
> left join which might not be semantically correct thing to do)
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 462f332e99 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveSubQRemoveRelBuilder.java
>  3a1897f4aa 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
>  62125f0fb7 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java
>  2dca6a25ac 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/PlanModifierForASTConv.java
>  5e8a994873 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 80351bef87 
>   ql/src/test/queries/clientpositive/subquery_exists.q 19c42f0c29 
>   ql/src/test/queries/clientpositive/subquery_in.q 4ba170a706 
>   ql/src/test/results/clientpositive/constprog_partitioner.q.out 87618df902 
>   ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_2.q.out 
> 87e08fbcde 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 6e55acf0d8 
>   ql/src/test/results/clientpositive/llap/subquery_exists.q.out e206f0851e 
>   ql/src/test/results/clientpositive/llap/subquery_in.q.out af42131bc2 
>   ql/src/test/results/clientpositive/llap/subquery_multi.q.out 96fe17a05a 
>   ql/src/test/results/clientpositive/llap/subquery_notin.q.out 8e2ca937af 
>   ql/src/test/results/clientpositive/llap/subquery_scalar.q.out c89d053b4a 
>   ql/src/test/results/clientpositive/llap/subquery_select.q.out 118f6ebccf 
>   ql/src/test/results/clientpositive/llap/subquery_views.q.out a9a81133b5 
>   ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out 
> 4e6f00f6b7 
>   ql/src/test/results/clientpositive/masking_12.q.out 540c53e825 
>   ql/src/test/results/clientpositive/masking_3.q.out 1114c80676 
>   ql/src/test/results/clientpositive/masking_4.q.out 527da21610 
>   ql/src/test/results/clientpositive/perf/spark/query10.q.out eb3a2f6699 
>   ql/src/test/results/clientpositive/perf/spark/query16.q.out b74d721d41 
>   ql/src/test/results/clientpositive/perf/spark/query35.q.out 8759b71b8c 
>   ql/src/test/results/clientpositive/perf/spark/query69.q.out e4430beaac 
>   ql/src/test/results/clientpositive/perf/spark/query94.q.out 43b8c77bdc 
>   ql/src/test/results/clientpositive/perf/tez/query10.q.out cf3651b35b 
>   ql/src/test/results/clientpositive/perf/tez/query14.q.out b2a45f155a 
>   ql/src/test/results/clientpositive/perf/tez/query16.q.out a7b710d6e1 
>   ql/src/test/results/clientpositive/perf/tez/query23.q.out 7112de61d9 
>   ql/src/test/results/clientpositive/perf/tez/query35.q.out a72f57816e 
>   ql/src/test/results/clientpositive/perf/tez/query69.q.out 591f3fcdb0 
>

[GitHub] hive pull request #261: HIVE-17812

2017-11-02 Thread alanfgates 
Github user alanfgates closed the pull request at:

https://github.com/apache/hive/pull/261


---

[jira] [Created] (HIVE-17968) TPCDS query70 generates NPE in vectorization, works fine with vectorization disabled.

2017-11-02 Thread Nita Dembla (JIRA) 
Nita Dembla created HIVE-17968:
--

 Summary: TPCDS query70 generates NPE in vectorization, works fine 
with vectorization disabled.
 Key: HIVE-17968
 URL: https://issues.apache.org/jira/browse/HIVE-17968
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Nita Dembla
Priority: Major


Running into the following NPE while running query 70


ERROR : FAILED: Execution Error, return code 2 from 
org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Reducer 
2, vertexId=vertex_1509557139747_0023_2_07, diagnostics=[Task failed, 
taskId=task_1509557139747_0023_2_07_00, diagnostics=[TaskAttempt 0 failed, 
info=[Error: Error while running task ( failure ) : 
attempt_1509557139747_0023_2_07_00_0:java.lang.RuntimeException: 
java.lang.RuntimeException: Hive Runtime Error while closing operators: null
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:283)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:237)
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at 
org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Hive Runtime Error while closing 
operators: null
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:407)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:271)
Caused by: java.lang.NullPointerException
at java.lang.System.arraycopy(Native Method)
at org.apache.hadoop.io.Text.set(Text.java:225)
at 
org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$8.writeValue(VectorExpressionWriterFactory.java:1061)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapperBatch.getWritableKeyValue(VectorHashKeyWrapperBatch.java:1005)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.writeSingleRow(VectorGroupByOperator.java:1082)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.access$800(VectorGroupByOperator.java:67)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeReduceMergePartial.close(VectorGroupByOperator.java:856)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.closeOp(VectorGroupByOperator.java:1123)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:709)
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:383)
... 16 more
], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) : 
attempt_1509557139747_0023_2_07_00_1:java.lang.RuntimeException: 
java.lang.RuntimeException: Hive Runtime Error while closing operators: null
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:283)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:237)
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at

[jira] [Created] (HIVE-17967) Move HiveMetaStore class

2017-11-02 Thread Alan Gates (JIRA) 
Alan Gates created HIVE-17967:
-

 Summary: Move HiveMetaStore class
 Key: HIVE-17967
 URL: https://issues.apache.org/jira/browse/HIVE-17967
 Project: Hive
  Issue Type: Sub-task
  Components: Standalone Metastore
Reporter: Alan Gates
Assignee: Alan Gates
Priority: Major


We need to move HiveMetaStore and a few tightly integrated classes into the 
standalone metastore.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17966) org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveArrayInspector - Review

2017-11-02 Thread BELUGA BEHR (JIRA) 
BELUGA BEHR created HIVE-17966:
--

 Summary: 
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveArrayInspector - Review
 Key: HIVE-17966
 URL: https://issues.apache.org/jira/browse/HIVE-17966
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2, Serializers/Deserializers
Affects Versions: 3.0.0
Reporter: BELUGA BEHR
 Fix For: 3.0.0, 2.3.2


* Simplify
* Make faster - perform bulk operations instead of iterating



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17965) Remove HIVELIMITTABLESCANPARTITION support

2017-11-02 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-17965:
---

 Summary: Remove HIVELIMITTABLESCANPARTITION support
 Key: HIVE-17965
 URL: https://issues.apache.org/jira/browse/HIVE-17965
 Project: Hive
  Issue Type: Improvement
Reporter: Zoltan Haindrich


HIVE-13884 marked it as deprecated



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17964) HoS: some spark configs doesn't require re-creating a session

2017-11-02 Thread Rui Li (JIRA) 
Rui Li created HIVE-17964:
-

 Summary: HoS: some spark configs doesn't require re-creating a 
session
 Key: HIVE-17964
 URL: https://issues.apache.org/jira/browse/HIVE-17964
 Project: Hive
  Issue Type: Improvement
Reporter: Rui Li
Priority: Minor


I guess the {{hive.spark.}} configs were initially intended for the RSC. 
Therefore when they're changed, we'll re-create the session for them to take 
effect. There're some configs not related to RSC that also start with 
{{hive.spark.}}. We'd better rename them so that we don't unnecessarily 
re-create sessions, which is usually time consuming.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 63458: HIVE-16917: HiveServer2 guard rails - Limit concurrent connections from user

2017-11-02 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63458/
---

(Updated Nov. 2, 2017, 8:08 a.m.)


Review request for hive, Sergey Shelukhin and Thejas Nair.


Changes
---

Addressed review comments.


Bugs: HIVE-16917
https://issues.apache.org/jira/browse/HIVE-16917


Repository: hive-git


Description
---

HIVE-16917: HiveServer2 guard rails - Limit concurrent connections from user


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a3c853a 
  service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
9b2ae57 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
6354c8c 
  
service/src/test/org/apache/hive/service/cli/TestCLIServiceConnectionLimits.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/63458/diff/2/

Changes: https://reviews.apache.org/r/63458/diff/1-2/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 63486: HIVE-17953 Metrics should move to destination atomically

2017-11-02 Thread Barna Zsombor Klara 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63486/#review189889
---


Ship it!




Ship It!

- Barna Zsombor Klara


On nov. 1, 2017, 11:25 du, Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63486/
> ---
> 
> (Updated nov. 1, 2017, 11:25 du)
> 
> 
> Review request for hive, Sahil Takiar and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-17953
> https://issues.apache.org/jira/browse/HIVE-17953
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17953 Metrics should move to destination atomically
> 
> 
> Diffs
> -
> 
>   
> common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
>  b9be8bd18285df42a82385f7ce73d9e1924fee4c 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/metrics/JsonReporter.java
>  04a5f02a91b76ed3775f7c1c8e0122a135564c70 
> 
> 
> Diff: https://reviews.apache.org/r/63486/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>