Re: Review Request 67731: HIVE-19902: Provide Metastore micro-benchmarks

2018-08-09 Thread Alexander Kolbasov 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67731/
---

(Updated Aug. 10, 2018, 12:22 a.m.)


Review request for hive, Andrew Sherman, Alan Gates, Janaki Lahorani, Marta 
Kuczora, Peter Vary, Sahil Takiar, and Vihang Karajgaonkar.


Changes
---

re-merged with master


Bugs: HIVE-19902
https://issues.apache.org/jira/browse/HIVE-19902


Repository: hive-git


Description
---

HIVE-19902: Provide Metastore micro-benchmarks


Diffs (updated)
-

  standalone-metastore/metastore-tools/metastore-benchmarks/README.md 
PRE-CREATION 
  standalone-metastore/metastore-tools/metastore-benchmarks/pom.xml 
PRE-CREATION 
  
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchData.java
 PRE-CREATION 
  
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java
 PRE-CREATION 
  
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
 PRE-CREATION 
  
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/resources/log4j.properties
 PRE-CREATION 
  
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/resources/log4j2.xml
 PRE-CREATION 
  standalone-metastore/metastore-tools/pom.xml PRE-CREATION 
  standalone-metastore/metastore-tools/tools-common/pom.xml PRE-CREATION 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java
 PRE-CREATION 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Constants.java
 PRE-CREATION 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
 PRE-CREATION 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/MicroBenchmark.java
 PRE-CREATION 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
 PRE-CREATION 
  
standalone-metastore/metastore-tools/tools-common/src/test/java/org/apache/hadoop/hive/metastore/tools/HMSClientTest.java
 PRE-CREATION 
  
standalone-metastore/metastore-tools/tools-common/src/test/java/org/apache/hadoop/hive/metastore/tools/UtilTest.java
 PRE-CREATION 
  standalone-metastore/pom.xml 99bf437e4c54721f76ecfc147e839498453ad60d 


Diff: https://reviews.apache.org/r/67731/diff/5/

Changes: https://reviews.apache.org/r/67731/diff/4-5/


Testing
---


Thanks,

Alexander Kolbasov

Re: Review Request 67981: HIVE-20196: Remove MetastoreConf dependency on server-specific classes

2018-08-09 Thread Alexander Kolbasov 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67981/
---

(Updated Aug. 9, 2018, 11:50 p.m.)


Review request for hive, Alan Gates, Peter Vary, Sahil Takiar, and Vihang 
Karajgaonkar.


Changes
---

Just convert classes to strings.


Summary (updated)
-

HIVE-20196: Remove MetastoreConf dependency on server-specific classes


Bugs: HIVE-20196
https://issues.apache.org/jira/browse/HIVE-20196


Repository: hive-git


Description (updated)
---

HIVE-20196: Remove MetastoreConf dependency on server-specific classes


Diffs (updated)
-

  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 cf5fbbefa6a5b0b7b3443071cce0a76d715e5163 


Diff: https://reviews.apache.org/r/67981/diff/2/

Changes: https://reviews.apache.org/r/67981/diff/1-2/


Testing
---


Thanks,

Alexander Kolbasov

[jira] [Created] (HIVE-20358) Allow setting variable value from Hive metastore table properties

2018-08-09 Thread Zheng Shao (JIRA) 
Zheng Shao created HIVE-20358:
-

 Summary: Allow setting variable value from Hive metastore table 
properties
 Key: HIVE-20358
 URL: https://issues.apache.org/jira/browse/HIVE-20358
 Project: Hive
  Issue Type: New Feature
  Components: Query Processor
Reporter: Zheng Shao


Hive already supports set command as well as variable substitution:


{{set start_ds=2018-08-01;}}

{{SELECT COUNT(*) FROM t WHERE ds >= '${hiveconf:start_ds}';}}

 

Or:

{{set start_ds='2018-08-01';}}

{{SELECT COUNT(*) FROM t WHERE ds >= ${hiveconf:start_ds};}}

 

This issue propose to extend the set syntax to allow running UDF and a UDF that 
queries metastore:

{{SET  ;}}

 

For example:

{{set start_ds GET_TABLE_PROPERTY('mydb', 'mytable', 'last_modified_time');}}

{{set start_ds GET_PARTITION_PROPERTY('mydb', 'mytable', 'ds=2018-01-01/hr=12', 
'last_modified_time');}}

 

This will allow query workflows like the following:

{{set last_run_time GET_TABLE_PROPERTY('mydb', 'mytable', 
'last_modified_time');}}

{{INSERT INTO TABLE mytable SELECT * FROM src WHERE src.time > last_run_time;}}

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67954: HIVE-20194: HiveMetastoreClient should use reflection to instantiate embedded HMS instance

2018-08-09 Thread Alexander Kolbasov 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67954/
---

(Updated Aug. 9, 2018, 10:35 p.m.)


Review request for hive, Alan Gates, Peter Vary, Sahil Takiar, and Vihang 
Karajgaonkar.


Changes
---

Fixed HiveMetaStoreClientPreCatalog to use the new signature for 
newRetryingHMSHandler


Bugs: HIVE-20194
https://issues.apache.org/jira/browse/HIVE-20194


Repository: hive-git


Description
---

HIVE-20194: HiveMetastoreClient should use reflection to instantiate embedded 
HMS instance


Diffs (updated)
-

  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 a53d4be03d695bf2176436967026757391531bc9 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 91c86a749c7afb06737c850e57f60820710c51f5 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 34055d2d4d39dc63d505a5ef95d190aa80a49d14 


Diff: https://reviews.apache.org/r/67954/diff/5/

Changes: https://reviews.apache.org/r/67954/diff/4-5/


Testing
---


Thanks,

Alexander Kolbasov

Re: Review Request 68281: HIVE-20354

2018-08-09 Thread Eugene Koifman 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68281/#review207051
---




ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java
Lines 972 (patched)


what if some table is named "select_table"


- Eugene Koifman


On Aug. 9, 2018, 12:19 p.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68281/
> ---
> 
> (Updated Aug. 9, 2018, 12:19 p.m.)
> 
> 
> Review request for hive, Eugene Koifman and Jason Dere.
> 
> 
> Bugs: HIVE-20354
> https://issues.apache.org/jira/browse/HIVE-20354
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Semijoin hints dont work with merge statements.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g f4d12ae564 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 463880587e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java 
> 8df290435d 
>   ql/src/test/queries/clientpositive/semijoin_hint.q de176affd3 
>   ql/src/test/results/clientpositive/llap/semijoin_hint.q.out 679916de07 
> 
> 
> Diff: https://reviews.apache.org/r/68281/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: Review Request 68281: HIVE-20354

2018-08-09 Thread Deepak Jaiswal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68281/
---

(Updated Aug. 9, 2018, 7:19 p.m.)


Review request for hive, Eugene Koifman and Jason Dere.


Changes
---

Implemented review comments.


Bugs: HIVE-20354
https://issues.apache.org/jira/browse/HIVE-20354


Repository: hive-git


Description
---

Semijoin hints dont work with merge statements.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g f4d12ae564 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 463880587e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java 
8df290435d 
  ql/src/test/queries/clientpositive/semijoin_hint.q de176affd3 
  ql/src/test/results/clientpositive/llap/semijoin_hint.q.out 679916de07 


Diff: https://reviews.apache.org/r/68281/diff/2/

Changes: https://reviews.apache.org/r/68281/diff/1-2/


Testing
---


Thanks,

Deepak Jaiswal

Re: Review Request 68281: HIVE-20354

2018-08-09 Thread Deepak Jaiswal 


> On Aug. 9, 2018, 6:33 p.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java
> > Lines 1000 (patched)
> > 
> >
> > why not save it directly into setHintList()?

It has to be first processed before it can be set. Anyway I am going to abandon 
this approach in favor of what Eugene suggested.


- Deepak


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68281/#review207047
---


On Aug. 9, 2018, 5:44 p.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68281/
> ---
> 
> (Updated Aug. 9, 2018, 5:44 p.m.)
> 
> 
> Review request for hive, Eugene Koifman and Jason Dere.
> 
> 
> Bugs: HIVE-20354
> https://issues.apache.org/jira/browse/HIVE-20354
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Semijoin hints dont work with merge statements.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g f4d12ae564 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 463880587e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java 
> 8df290435d 
>   ql/src/test/queries/clientpositive/semijoin_hint.q de176affd3 
>   ql/src/test/results/clientpositive/llap/semijoin_hint.q.out 679916de07 
> 
> 
> Diff: https://reviews.apache.org/r/68281/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: Review Request 68281: HIVE-20354

2018-08-09 Thread Gopal V 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68281/#review207047
---




ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java
Lines 1000 (patched)


why not save it directly into setHintList()?


- Gopal V


On Aug. 9, 2018, 5:44 p.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68281/
> ---
> 
> (Updated Aug. 9, 2018, 5:44 p.m.)
> 
> 
> Review request for hive, Eugene Koifman and Jason Dere.
> 
> 
> Bugs: HIVE-20354
> https://issues.apache.org/jira/browse/HIVE-20354
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Semijoin hints dont work with merge statements.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g f4d12ae564 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 463880587e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java 
> 8df290435d 
>   ql/src/test/queries/clientpositive/semijoin_hint.q de176affd3 
>   ql/src/test/results/clientpositive/llap/semijoin_hint.q.out 679916de07 
> 
> 
> Diff: https://reviews.apache.org/r/68281/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

[jira] [Created] (HIVE-20357) Introduce initOrUpgradeSchema option to schema tool

2018-08-09 Thread Daniel Dai (JIRA) 
Daniel Dai created HIVE-20357:
-

 Summary: Introduce initOrUpgradeSchema option to schema tool
 Key: HIVE-20357
 URL: https://issues.apache.org/jira/browse/HIVE-20357
 Project: Hive
  Issue Type: Improvement
  Components: Standalone Metastore
Reporter: Daniel Dai
Assignee: Daniel Dai


Currently, schematool has two option: initSchema/upgradeSchema. User needs to 
use different command line for different action. However, from the schema 
version stored in db, we shall able to figure out if there's a need to 
init/upgrade, and choose the right action automatically.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20356) Follow up of HIVE-19613: GenericUDTFGetSplits should handle fetch task with temp table rewrite

2018-08-09 Thread Jaume M (JIRA) 
Jaume M created HIVE-20356:
--

 Summary: Follow up of HIVE-19613: GenericUDTFGetSplits should 
handle fetch task with temp table rewrite
 Key: HIVE-20356
 URL: https://issues.apache.org/jira/browse/HIVE-20356
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.1.0
Reporter: Jaume M


As mentioned in HIVE-19631, we should fix this 
[here|https://github.com/apache/hive/blob/873d31f33a061cd38be7de91b208987871fb612e/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java#L321]
 as well. cc [~prasanth_j] [~ewohlstadter]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20355) Clean up parameter of HiveConnection.setSchema

2018-08-09 Thread Daniel Dai (JIRA) 
Daniel Dai created HIVE-20355:
-

 Summary: Clean up parameter of HiveConnection.setSchema
 Key: HIVE-20355
 URL: https://issues.apache.org/jira/browse/HIVE-20355
 Project: Hive
  Issue Type: Bug
  Components: JDBC
Reporter: Daniel Dai
Assignee: Daniel Dai


Not immediately exploitable, as HS2 only allow one statement a time. But in 
future, we may support multiple statement in HiveStatement, so better to clean 
up the database parameter to avoid potential sql injection.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68281: HIVE-20354

2018-08-09 Thread Eugene Koifman 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68281/#review207043
---




ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java
Lines 1000 (patched)


Modifying parse tree directly is not a good idea - it messes up internal 
ANTLR strucutres and may cause issues downstream.  You should inject the hint 
into 'rewrittenQueryStr' so that a complete new statement is parsed - that is 
the model for all other parts of Merge reparsing.



ql/src/test/queries/clientpositive/semijoin_hint.q
Lines 116 (patched)


it may be useful to one statment with hint and another w/o hint - to see 
clearly the difference in the plan.


- Eugene Koifman


On Aug. 9, 2018, 10:44 a.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68281/
> ---
> 
> (Updated Aug. 9, 2018, 10:44 a.m.)
> 
> 
> Review request for hive, Eugene Koifman and Jason Dere.
> 
> 
> Bugs: HIVE-20354
> https://issues.apache.org/jira/browse/HIVE-20354
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Semijoin hints dont work with merge statements.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g f4d12ae564 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 463880587e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java 
> 8df290435d 
>   ql/src/test/queries/clientpositive/semijoin_hint.q de176affd3 
>   ql/src/test/results/clientpositive/llap/semijoin_hint.q.out 679916de07 
> 
> 
> Diff: https://reviews.apache.org/r/68281/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Review Request 68281: HIVE-20354

2018-08-09 Thread Deepak Jaiswal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68281/
---

Review request for hive and Jason Dere.


Bugs: HIVE-20354
https://issues.apache.org/jira/browse/HIVE-20354


Repository: hive-git


Description
---

Semijoin hints dont work with merge statements.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g f4d12ae564 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 463880587e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java 
8df290435d 
  ql/src/test/queries/clientpositive/semijoin_hint.q de176affd3 
  ql/src/test/results/clientpositive/llap/semijoin_hint.q.out 679916de07 


Diff: https://reviews.apache.org/r/68281/diff/1/


Testing
---


Thanks,

Deepak Jaiswal

[jira] [Created] (HIVE-20354) Semijoin hints dont work with merge statements

2018-08-09 Thread Deepak Jaiswal (JIRA) 
Deepak Jaiswal created HIVE-20354:
-

 Summary: Semijoin hints dont work with merge statements
 Key: HIVE-20354
 URL: https://issues.apache.org/jira/browse/HIVE-20354
 Project: Hive
  Issue Type: Bug
Reporter: Deepak Jaiswal
Assignee: Deepak Jaiswal


When merge statement is rewritten, it ignores any comment in the query which 
may include hints like semijoin.
If it is, it should not be ignored.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20353) Follow redirects when hive connects to a passive druid overlord/coordinator

2018-08-09 Thread Nishant Bangarwa (JIRA) 
Nishant Bangarwa created HIVE-20353:
---

 Summary: Follow redirects when hive connects to a passive druid 
overlord/coordinator
 Key: HIVE-20353
 URL: https://issues.apache.org/jira/browse/HIVE-20353
 Project: Hive
  Issue Type: Bug
  Components: Druid integration
Reporter: Nishant Bangarwa
Assignee: Nishant Bangarwa


When we have multiple druid coordinators/overlords and hive tries to connect to 
a passive one, it will get a redirect. Currently the http client in druid 
storage handler does not follow redirects. We need to check if there is a 
redirect and follow that for druid overlord/coordinator



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20352) Vectorization: Support grouping function

2018-08-09 Thread Matt McCline (JIRA) 
Matt McCline created HIVE-20352:
---

 Summary: Vectorization: Support grouping function
 Key: HIVE-20352
 URL: https://issues.apache.org/jira/browse/HIVE-20352
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Matt McCline
Assignee: Matt McCline


Support native vectorization for grouping function (part of Grouping Sets) so 
we don't need to use VectorUDFAdaptor.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20351) GenericUDFNamedStruct should constant fold at compile time

2018-08-09 Thread Mykhailo Kysliuk (JIRA) 
Mykhailo Kysliuk created HIVE-20351:
---

 Summary: GenericUDFNamedStruct should constant fold at compile time
 Key: HIVE-20351
 URL: https://issues.apache.org/jira/browse/HIVE-20351
 Project: Hive
  Issue Type: Bug
Reporter: Mykhailo Kysliuk
Assignee: Mykhailo Kysliuk


Reproduced at hive-3.0.

When we run hive query:

{code:java}
select named_struct('Total','Total') from test;
{code}

We could see the ERROR at hiveserver logs:
{code:java}
2018-05-25T15:18:13,182 ERROR [main] optimizer.ConstantPropagateProcFactory: 
Unable to evaluate 
org.apache.hadoop.hive.ql.udf.generic.GenericUDFNamedStruct@a0bf272. Return 
value unrecoginizable.
{code}

This error is not harmless because all results are correct. But named_struct 
constant values should be processed correctly.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] hive pull request #411: HIVE-20150: TopNKey pushdown (Teddy Choi)

2018-08-09 Thread pudidic 
GitHub user pudidic opened a pull request:

https://github.com/apache/hive/pull/411

HIVE-20150: TopNKey pushdown (Teddy Choi)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pudidic/hive HIVE-20150

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/411.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #411


commit abb2ad51b6802752b78ee37c0b58e13c74730ad1
Author: Teddy Choi 
Date:   2018-08-09T15:06:11Z

HIVE-20150: TopNKey pushdown (Teddy Choi)




---

Re: Update protobuf version in pom.xml

2018-08-09 Thread Alan Gates 
For info on submitting the change, see
https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-CreatingaPatch
Note that once you have created the JIRA ticket you can also reference that
JIRA ticket in a PR and github will annotate the JIRA ticket with the PR
link.  But we still need a patch since that's how the current CI system
works.

I'll let others reply to the protoc part of the question, as I don't know
the answer.

Alan.



On Wed, Aug 8, 2018 at 11:42 PM Naresh Bhat  wrote:

> Hi,
>
> I was trying to compile the Hive master branch on AArch64 hardware.  I am
> facing the protoc issue because in pom.xml file use the version 2.5.0 which
> does not have AArch64 binaries available
> https://repo.maven.apache.org/maven2/com/google/protobuf/protoc/ .  After
> updating it to latest version 3.6.1, I could able to compile Hive master
> branch without any issues on ARM64 machine. I have created a patch and
> which is available -
>
> https://git.linaro.org/people/naresh.bhat/apache/hive.git/commit/?id=14410fbd6a3203a39f2503368c5e51dc6d11b432
>
> My questions are
>
> 1. Why still old protoc version is being used in Hive pom.xml files ?  Can
> we update it to latest available version i.e. 3.6.1 ?
> 2. How should I give the patch pull request i.e. through git-mail-send or
> via using github ?
>
> Thanks and Regards
> -Naresh Bhat
>

[jira] [Created] (HIVE-20350) Unnecessary value assignment

2018-08-09 Thread Laszlo Pinter (JIRA) 
Laszlo Pinter created HIVE-20350:


 Summary: Unnecessary value assignment
 Key: HIVE-20350
 URL: https://issues.apache.org/jira/browse/HIVE-20350
 Project: Hive
  Issue Type: Improvement
Reporter: Laszlo Pinter
Assignee: Laszlo Pinter


There is an unnecessary value assignment in Hive.java 
{code:java}
newPartPath = oldPartPath == null ? newPartPath = genPartPathFromTable(tbl, 
partSpec,
tblDataLocationPath) : oldPartPath;{code}
This can be changed to
{code:java}
newPartPath = oldPartPath == null ? genPartPathFromTable(tbl, partSpec,
tblDataLocationPath) : oldPartPath;{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20349) Implement Retry Logic in HiveDruidSplit for Scan Queries

2018-08-09 Thread Nishant Bangarwa (JIRA) 
Nishant Bangarwa created HIVE-20349:
---

 Summary: Implement Retry Logic in HiveDruidSplit for Scan Queries
 Key: HIVE-20349
 URL: https://issues.apache.org/jira/browse/HIVE-20349
 Project: Hive
  Issue Type: Bug
Reporter: Nishant Bangarwa
Assignee: Nishant Bangarwa


while distributing druid scan query we check where the segments are loaded and 
then each HiveDruidSplit directly queries the historical node. 
There are few cases when we need to retry and refetch the segments. 

# The segment is loaded on multiple historical nodes and one of them went down. 
in this case when we do not get response from one segment, we query the next 
replica. 
# The segment was loaded onto a realtime task and was handed over, when we 
query the realtime task has already finished. In this case there is no replica. 
The Split needs to query the broker again for the location of the segment and 
then send the query to correct historical node. 

This is also the root cause of failure of druidkafkamini_basic.q test, where 
the segment handover happens before the scan query is executed.

Note: This is not a problem when we are directly querying Druid brokers as the 
broker handles the retry logic. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68275: HIVE-20348 : Hive HCat does not create a proper 'client' on kerberos cluster without hive metastore

2018-08-09 Thread Oleksiy Sayankin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68275/
---

(Updated Aug. 9, 2018, 2:38 p.m.)


Review request for hive and Lefty Leverenz.


Summary (updated)
-

HIVE-20348 : Hive HCat does not create a proper 'client' on kerberos cluster 
without hive metastore


Repository: hive-git


Description (updated)
---

Initial commit


Diffs
-

  hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/Security.java 
50b746e6fd 


Diff: https://reviews.apache.org/r/68275/diff/1/


Testing
---


Thanks,

Oleksiy Sayankin

Review Request 68275: Initial commit

2018-08-09 Thread Oleksiy Sayankin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68275/
---

Review request for hive.


Repository: hive-git


Description
---

Hive HCat does not create a proper 'client' on kerberos cluster without hive 
metastore


Diffs
-

  hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/Security.java 
50b746e6fd 


Diff: https://reviews.apache.org/r/68275/diff/1/


Testing
---


Thanks,

Oleksiy Sayankin

[jira] [Created] (HIVE-20348) Hive HCat does not create a proper "client" on kerberos cluster without hive metastore

2018-08-09 Thread Oleksiy Sayankin (JIRA) 
Oleksiy Sayankin created HIVE-20348:
---

 Summary: Hive HCat does not create a proper "client" on kerberos 
cluster without hive metastore
 Key: HIVE-20348
 URL: https://issues.apache.org/jira/browse/HIVE-20348
 Project: Hive
  Issue Type: Bug
Reporter: Oleksiy Sayankin
Assignee: Oleksiy Sayankin


*STEPS TO REPRODUCE:*

1. Configure Hive to use embedded Metastore (do not specify 
{{hive.metastore.uris}} in {{hive-site.xml}});

2. Create a database and a table in MySQL:
{code:java}
mysql -uroot -p123456 -e "CREATE DATABASE test;CREATE TABLE test.test (id 
INT);INSERT INTO test.test VALUES (1),(2),(3)"
{code}

3. Create a table in Hive:

{code:java}
hive -e "CREATE TABLE default.test (id INT)"
{code}

4. Run Sqoop import command:

{code:java}
sqoop import --connect 'jdbc:mysql://localhost:3306/test' --username root 
--password 123456 --table test  --hcatalog-database "default" --hcatalog-table 
"test" --verbose -m 1
{code}


*ACTUAL RESULT:*

Sqoop import command fails with an exception:

{code:java}
18/08/08 01:07:09 ERROR tool.ImportTool: Encountered IOException running import 
job: org.apache.hive.hcatalog.common.HCatException : 2001 : Error setting 
output information. Cause : java.lang.NullPointerException
at 
org.apache.hive.hcatalog.mapreduce.HCatOutputFormat.setOutput(HCatOutputFormat.java:220)
at 
org.apache.hive.hcatalog.mapreduce.HCatOutputFormat.setOutput(HCatOutputFormat.java:70)
at 
org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.configureHCat(SqoopHCatUtilities.java:361)
at 
org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.configureImportOutputFormat(SqoopHCatUtilities.java:783)
at 
org.apache.sqoop.mapreduce.ImportJobBase.configureOutputFormat(ImportJobBase.java:98)
at 
org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:259)
at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:689)
at org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:118)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:498)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:606)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.security.token.Token.decodeWritable(Token.java:256)
at 
org.apache.hadoop.security.token.Token.decodeFromUrlString(Token.java:275)
at 
org.apache.hive.hcatalog.common.HCatUtil.extractThriftToken(HCatUtil.java:351)
at 
org.apache.hive.hcatalog.mapreduce.Security.handleSecurity(Security.java:139)
at 
org.apache.hive.hcatalog.mapreduce.HCatOutputFormat.setOutput(HCatOutputFormat.java:214)
... 15 more
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Update protobuf version in pom.xml

2018-08-09 Thread Naresh Bhat 
Hi,

I was trying to compile the Hive master branch on AArch64 hardware.  I am
facing the protoc issue because in pom.xml file use the version 2.5.0 which
does not have AArch64 binaries available
https://repo.maven.apache.org/maven2/com/google/protobuf/protoc/ .  After
updating it to latest version 3.6.1, I could able to compile Hive master
branch without any issues on ARM64 machine. I have created a patch and
which is available -
https://git.linaro.org/people/naresh.bhat/apache/hive.git/commit/?id=14410fbd6a3203a39f2503368c5e51dc6d11b432

My questions are

1. Why still old protoc version is being used in Hive pom.xml files ?  Can
we update it to latest available version i.e. 3.6.1 ?
2. How should I give the patch pull request i.e. through git-mail-send or
via using github ?

Thanks and Regards
-Naresh Bhat

Facing an issue with the ReplDumpTaskTest.java in Hive Query Language.

2018-08-09 Thread Alisha Prabhu 
[INFO] Running org.apache.hadoop.hive.ql.exec.repl.ReplDumpTaskTest
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 5.536 s 
<<< FAILURE! - in org.apache.hadoop.hive.ql.exec.repl.ReplDumpTaskTest
[ERROR] 
removeDBPropertyToPreventRenameWhenBootstrapDumpOfTableFails(org.apache.hadoop.hive.ql.exec.repl.ReplDumpTaskTest)
  Time elapsed: 2.942 s  <<< ERROR!
java.lang.Exception: Unexpected exception, 
expected 
but was
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.handleException(PowerMockJUnit44RunnerDelegateImpl.java:370)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.handleException(PowerMockJUnit47RunnerDelegateImpl.java:126)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.handleInvocationTargetException(PowerMockJUnit44RunnerDelegateImpl.java:343)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:321)
at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:88)
at 
org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:96)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.executeTest(PowerMockJUnit44RunnerDelegateImpl.java:300)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTestInSuper(PowerMockJUnit47RunnerDelegateImpl.java:131)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.access$100(PowerMockJUnit47RunnerDelegateImpl.java:59)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner$TestExecutorStatement.evaluate(PowerMockJUnit47RunnerDelegateImpl.java:147)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.evaluateStatement(PowerMockJUnit47RunnerDelegateImpl.java:107)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTest(PowerMockJUnit47RunnerDelegateImpl.java:82)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runBeforesThenTestThenAfters(PowerMockJUnit44RunnerDelegateImpl.java:288)
at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:86)
at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:49)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.invokeTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:208)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.runMethods(PowerMockJUnit44RunnerDelegateImpl.java:147)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$1.run(PowerMockJUnit44RunnerDelegateImpl.java:121)
at 
org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:33)
at 
org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:45)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.run(PowerMockJUnit44RunnerDelegateImpl.java:123)
at 
org.powermock.modules.junit4.common.internal.impl.JUnit4TestSuiteChunkerImpl.run(JUnit4TestSuiteChunkerImpl.java:121)
at 
org.powermock.modules.junit4.common.internal.impl.AbstractCommonPowerMockRunner.run(AbstractCommonPowerMockRunner.java:53)
at 
org.powermock.modules.junit4.PowerMockRunner.run(PowerMockRunner.java:59)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
at 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
Caused by: Wanted but not invoked 
org.apache.hadoop.hive.ql.parse.repl.dump.Utils.resetDbBootstrapDumpState(
null,
null,
null
);
Actually, there were zero interactions with this mock.
at org.powermock.core.MockGateway.doMethodCall(MockGateway.java:173)
at