date:20171117

Re: Review Request 63927: HIVE-17631 upgrade orc to 1.4.0

2017-11-17 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63927/#review191431
---




ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
Line 2124 (original), 2126 (patched)


Not all versions of orc has hasNull flag in proto. So hasHasNull has to be 
verified before .hasNull()? If it doesn't have hasNull flag it will just return 
false which is wrong as it may or may not have nulls.


- Prasanth_J


On Nov. 18, 2017, 1:24 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63927/
> ---
> 
> (Updated Nov. 18, 2017, 1:24 a.m.)
> 
> 
> Review request for hive, Jason Dere and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  c32f79fe95 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
>  599b5191c6 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java
>  1ec702011a 
>   pom.xml 04fb7c3c55 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 1e5b841f4b 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java c150b6 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> 80b7be8e5a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java d2bb6417a2 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java 
> a916d58f3e 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicValue.java a20328cb69 
> 
> 
> Diff: https://reviews.apache.org/r/63927/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-18100) Some tests time out

2017-11-17 Thread Ashutosh Chauhan (JIRA)

Ashutosh Chauhan created HIVE-18100:
---

 Summary: Some tests time out
 Key: HIVE-18100
 URL: https://issues.apache.org/jira/browse/HIVE-18100
 Project: Hive
  Issue Type: Test
  Components: Test
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan


Some tests had 100s of queries in a single query which times out resulting in 
Hive QA failures.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 63927: HIVE-17631 upgrade orc to 1.4.0

2017-11-17 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63927/
---

(Updated Nov. 18, 2017, 1:24 a.m.)


Review request for hive, Jason Dere and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 c32f79fe95 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
 599b5191c6 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java
 1ec702011a 
  pom.xml 04fb7c3c55 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 1e5b841f4b 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java c150b6 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
80b7be8e5a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java d2bb6417a2 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java 
a916d58f3e 
  ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicValue.java a20328cb69 


Diff: https://reviews.apache.org/r/63927/diff/3/

Changes: https://reviews.apache.org/r/63927/diff/2-3/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 63927: HIVE-17631 upgrade orc to 1.4.0

2017-11-17 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63927/#review191430
---


Ship it!




Ship It!

- Prasanth_J


On Nov. 18, 2017, 1:24 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63927/
> ---
> 
> (Updated Nov. 18, 2017, 1:24 a.m.)
> 
> 
> Review request for hive, Jason Dere and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  c32f79fe95 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
>  599b5191c6 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java
>  1ec702011a 
>   pom.xml 04fb7c3c55 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 1e5b841f4b 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java c150b6 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> 80b7be8e5a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java d2bb6417a2 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java 
> a916d58f3e 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicValue.java a20328cb69 
> 
> 
> Diff: https://reviews.apache.org/r/63927/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 63893: HIVE-14487

2017-11-17 Thread Ashutosh Chauhan



> On Nov. 17, 2017, 11:22 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
> > Line 1488 (original), 1488 (patched)
> > 
> >
> > This table object is never read from. Any reason for its existence.
> 
> Jesús Camacho Rodríguez wrote:
> It is passed to StatsWork in L1533. I think we do not need the full 
> table, as StatsWork will only use some of the information. I can tackle in a 
> follow-up?

Ok, lets take this as a follow-up.


> On Nov. 17, 2017, 11:22 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java
> > Lines 341 (patched)
> > 
> >
> > We don't store comments for regular tables in property? Is this special 
> > for (materialized) views?
> 
> Jesús Camacho Rodríguez wrote:
> In fact, apparently we do: L779 in CreateTableDesc.java. The code was 
> originally based on that one, thus they should be quite similar.

Right. I got confused with column comments which are stored in a seperate field 
in metastore table.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63893/#review191406
---


On Nov. 18, 2017, 12:33 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63893/
> ---
> 
> (Updated Nov. 18, 2017, 12:33 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-14487
> https://issues.apache.org/jira/browse/HIVE-14487
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14487
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 
> 3b1005f85c419b1fa5646bc5a4b96788250e7c4b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> e7af5e004fb560b574b82f6d1b60517511802f37 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 67739a1db9fc52a67f4f5ea7dba80fe0e95750c8 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
> e9ae590f4c291ad00e385899822ef944d50cdcde 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 55b1da90300bdc864e9258d1c49c3c21791f3b64 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 
> f3f1a7e8da52332d93aa57865617befd6c07a2fc 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 1de3dd7230c5c3708fd7dd6baf9c6913e91cab3c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
> d139155a1d1433ca9186994cef2616f007a66fe9 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/BasicStatsWork.java 
> 0621bd413ec578f30a236de0be5774fc83457aa5 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java 
> 9425f6e89113afd82d1cdca30c75da6101b63b4a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ImportTableDesc.java 
> 1770046869170c30ebb27a84db7566d477bd9877 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/LoadFileDesc.java 
> c09589c5f23825ea39b85737e04cc0e365e61fea 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 
> 3c1e92af12185bb95e27fbf0af3038ca27566197 
>   
> ql/src/test/queries/clientnegative/materialized_view_authorization_rebuild_no_grant.q
>  PRE-CREATION 
>   
> ql/src/test/queries/clientnegative/materialized_view_authorization_rebuild_other.q
>  PRE-CREATION 
>   ql/src/test/queries/clientpositive/materialized_view_create_rewrite.q 
> 1749cb023ae2e4476ac386b667a708f50f8db462 
>   ql/src/test/queries/clientpositive/materialized_view_create_rewrite_3.q 
> PRE-CREATION 
>   
> ql/src/test/results/clientnegative/materialized_view_authorization_rebuild_no_grant.q.out
>  PRE-CREATION 
>   
> ql/src/test/results/clientnegative/materialized_view_authorization_rebuild_other.q.out
>  PRE-CREATION 
>   
> ql/src/test/results/clientpositive/beeline/materialized_view_create_rewrite.q.out
>  81a7950773983e28a0f36ce806b6cda394993ac0 
>   ql/src/test/results/clientpositive/create_view.q.out 
> 8763fdb75fc910f75c53346c9541703e800c128c 
>   ql/src/test/results/clientpositive/create_view_translate.q.out 
> e5748976bdb271bef386e6820be5236a36d4c941 
>   ql/src/test/results/clientpositive/explain_ddl.q.out 
> f0e54c515377e7e6e002b7cc9ae986340a93acce 
>   ql/src/test/results/clientpositive/llap/materialized_view_create.q.out 
> PRE-CREATION 
>   
> ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite.q.out
>  PRE-CREATION 
>   
> ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_2.q.out
>  PRE-CREATION 
>   
> ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_3.q.out
>  PRE-CREATION 
>   
>

Re: Review Request 63890: HIVE-18025: Push resource plan changes to tez/unmanaged sessions

2017-11-17 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63890/
---

(Updated Nov. 18, 2017, 12:58 a.m.)


Review request for hive and Sergey Shelukhin.


Changes
---

Addressed review comments.


Bugs: HIVE-18025
https://issues.apache.org/jira/browse/HIVE-18025


Repository: hive-git


Description
---

HIVE-18025: Push resource plan changes to tez/unmanaged sessions


Diffs (updated)
-

  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestTriggersNoTezSessionPool.java
 bcce3dc 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestTriggersTezSessionPoolManager.java
 b377275 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java e7af5e0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
4e48f15 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolSession.java 
769b24a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 3990f95 
  ql/src/java/org/apache/hadoop/hive/ql/wm/MetastoreGlobalTriggersFetcher.java 
87c007f 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 5a6d0c8 


Diff: https://reviews.apache.org/r/63890/diff/3/

Changes: https://reviews.apache.org/r/63890/diff/2-3/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 63893: HIVE-14487

2017-11-17 Thread Jesús Camacho Rodríguez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63893/
---

(Updated Nov. 18, 2017, 12:33 a.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-14487
https://issues.apache.org/jira/browse/HIVE-14487


Repository: hive-git


Description (updated)
---

HIVE-14487


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 
3b1005f85c419b1fa5646bc5a4b96788250e7c4b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
e7af5e004fb560b574b82f6d1b60517511802f37 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
67739a1db9fc52a67f4f5ea7dba80fe0e95750c8 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
e9ae590f4c291ad00e385899822ef944d50cdcde 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
55b1da90300bdc864e9258d1c49c3c21791f3b64 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 
f3f1a7e8da52332d93aa57865617befd6c07a2fc 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
1de3dd7230c5c3708fd7dd6baf9c6913e91cab3c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
d139155a1d1433ca9186994cef2616f007a66fe9 
  ql/src/java/org/apache/hadoop/hive/ql/plan/BasicStatsWork.java 
0621bd413ec578f30a236de0be5774fc83457aa5 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java 
9425f6e89113afd82d1cdca30c75da6101b63b4a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ImportTableDesc.java 
1770046869170c30ebb27a84db7566d477bd9877 
  ql/src/java/org/apache/hadoop/hive/ql/plan/LoadFileDesc.java 
c09589c5f23825ea39b85737e04cc0e365e61fea 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 
3c1e92af12185bb95e27fbf0af3038ca27566197 
  
ql/src/test/queries/clientnegative/materialized_view_authorization_rebuild_no_grant.q
 PRE-CREATION 
  
ql/src/test/queries/clientnegative/materialized_view_authorization_rebuild_other.q
 PRE-CREATION 
  ql/src/test/queries/clientpositive/materialized_view_create_rewrite.q 
1749cb023ae2e4476ac386b667a708f50f8db462 
  ql/src/test/queries/clientpositive/materialized_view_create_rewrite_3.q 
PRE-CREATION 
  
ql/src/test/results/clientnegative/materialized_view_authorization_rebuild_no_grant.q.out
 PRE-CREATION 
  
ql/src/test/results/clientnegative/materialized_view_authorization_rebuild_other.q.out
 PRE-CREATION 
  
ql/src/test/results/clientpositive/beeline/materialized_view_create_rewrite.q.out
 81a7950773983e28a0f36ce806b6cda394993ac0 
  ql/src/test/results/clientpositive/create_view.q.out 
8763fdb75fc910f75c53346c9541703e800c128c 
  ql/src/test/results/clientpositive/create_view_translate.q.out 
e5748976bdb271bef386e6820be5236a36d4c941 
  ql/src/test/results/clientpositive/explain_ddl.q.out 
f0e54c515377e7e6e002b7cc9ae986340a93acce 
  ql/src/test/results/clientpositive/llap/materialized_view_create.q.out 
PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite.q.out 
PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_2.q.out
 PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_3.q.out
 PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_multi_db.q.out
 PRE-CREATION 
  ql/src/test/results/clientpositive/llap/materialized_view_describe.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/materialized_view_drop.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/materialized_view_rewrite_ssb.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/materialized_view_rewrite_ssb_2.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/selectDistinctStar.q.out 
befb6e79929ca7fffc9ee56544c25a27c2a84cbd 
  ql/src/test/results/clientpositive/llap/union_top_level.q.out 
a0102321711de82100a5b117b853a364109a2622 
  ql/src/test/results/clientpositive/llap/vector_windowing.q.out 
776ae3e085840dcaa298c143966ff1f12e508c3e 
  ql/src/test/results/clientpositive/materialized_view_create.q.out 
0fca69f4fa94a24482d09a1b30a72611eec6c643 
  ql/src/test/results/clientpositive/materialized_view_create_rewrite.q.out 
81a7950773983e28a0f36ce806b6cda394993ac0 
  ql/src/test/results/clientpositive/materialized_view_create_rewrite_3.q.out 
PRE-CREATION 
  
ql/src/test/results/clientpositive/materialized_view_create_rewrite_multi_db.q.out
 f715803438d100599fc669cb9b1086523cd44099 
  ql/src/test/results/clientpositive/materialized_view_describe.q.out 
8b7691279c0b24c599656c372de22ef9cf83f8f7 


Diff: https://reviews.apache.org/r/63893/diff/4/

Changes: https://reviews.apache.org/r/63893/diff/3-4/


Testing
---


Thanks,

Jesús Camacho Rodríguez

Re: Review Request 63893: HIVE-14487

2017-11-17 Thread Jesús Camacho Rodríguez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63893/
---

(Updated Nov. 18, 2017, 12:31 a.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-14487
https://issues.apache.org/jira/browse/HIVE-14487


Repository: hive-git


Description (updated)
---

wip


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 
3b1005f85c419b1fa5646bc5a4b96788250e7c4b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
e7af5e004fb560b574b82f6d1b60517511802f37 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
 51b6ef58fc196be716c4b07287fbc83503d1df50 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
67739a1db9fc52a67f4f5ea7dba80fe0e95750c8 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
e9ae590f4c291ad00e385899822ef944d50cdcde 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
55b1da90300bdc864e9258d1c49c3c21791f3b64 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 
f3f1a7e8da52332d93aa57865617befd6c07a2fc 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
1de3dd7230c5c3708fd7dd6baf9c6913e91cab3c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
d139155a1d1433ca9186994cef2616f007a66fe9 
  ql/src/java/org/apache/hadoop/hive/ql/plan/BasicStatsWork.java 
0621bd413ec578f30a236de0be5774fc83457aa5 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java 
9425f6e89113afd82d1cdca30c75da6101b63b4a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ImportTableDesc.java 
1770046869170c30ebb27a84db7566d477bd9877 
  ql/src/java/org/apache/hadoop/hive/ql/plan/LoadFileDesc.java 
c09589c5f23825ea39b85737e04cc0e365e61fea 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 
3c1e92af12185bb95e27fbf0af3038ca27566197 
  
ql/src/test/queries/clientnegative/materialized_view_authorization_rebuild_no_grant.q
 PRE-CREATION 
  
ql/src/test/queries/clientnegative/materialized_view_authorization_rebuild_other.q
 PRE-CREATION 
  ql/src/test/queries/clientpositive/materialized_view_create_rewrite.q 
1749cb023ae2e4476ac386b667a708f50f8db462 
  ql/src/test/queries/clientpositive/materialized_view_create_rewrite_3.q 
PRE-CREATION 
  
ql/src/test/results/clientnegative/materialized_view_authorization_rebuild_no_grant.q.out
 PRE-CREATION 
  
ql/src/test/results/clientnegative/materialized_view_authorization_rebuild_other.q.out
 PRE-CREATION 
  
ql/src/test/results/clientpositive/beeline/materialized_view_create_rewrite.q.out
 81a7950773983e28a0f36ce806b6cda394993ac0 
  ql/src/test/results/clientpositive/create_view.q.out 
8763fdb75fc910f75c53346c9541703e800c128c 
  ql/src/test/results/clientpositive/create_view_translate.q.out 
e5748976bdb271bef386e6820be5236a36d4c941 
  ql/src/test/results/clientpositive/explain_ddl.q.out 
f0e54c515377e7e6e002b7cc9ae986340a93acce 
  ql/src/test/results/clientpositive/llap/materialized_view_create.q.out 
PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite.q.out 
PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_2.q.out
 PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_3.q.out
 PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_multi_db.q.out
 PRE-CREATION 
  ql/src/test/results/clientpositive/llap/materialized_view_describe.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/materialized_view_drop.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/materialized_view_rewrite_ssb.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/materialized_view_rewrite_ssb_2.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/selectDistinctStar.q.out 
befb6e79929ca7fffc9ee56544c25a27c2a84cbd 
  ql/src/test/results/clientpositive/llap/union_top_level.q.out 
a0102321711de82100a5b117b853a364109a2622 
  ql/src/test/results/clientpositive/llap/vector_windowing.q.out 
776ae3e085840dcaa298c143966ff1f12e508c3e 
  ql/src/test/results/clientpositive/materialized_view_create.q.out 
0fca69f4fa94a24482d09a1b30a72611eec6c643 
  ql/src/test/results/clientpositive/materialized_view_create_rewrite.q.out 
81a7950773983e28a0f36ce806b6cda394993ac0 
  ql/src/test/results/clientpositive/materialized_view_create_rewrite_3.q.out 
PRE-CREATION 
  
ql/src/test/results/clientpositive/materialized_view_create_rewrite_multi_db.q.out
 f715803438d100599fc669cb9b1086523cd44099 
  ql/src/test/results/clientpositive/materialized_view_describe.q.out 
8b7691279c0b24c599656c372de22ef9cf83f8f7 


Diff: https://reviews.apache.org/r/63893/diff/3/

Changes: https://reviews.apache.org/r/63893/diff/2-3/


Testing
---


Thanks,

Jesús Camacho Rodríguez

Re: Review Request 63927: HIVE-17631 upgrade orc to 1.4.0

2017-11-17 Thread j . prasanth . j



> On Nov. 18, 2017, 12:16 a.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
> > Lines 2123 (patched)
> > 
> >
> > Can you set truthValues[pred] to YES_NO or YES_NO_NULL instead, as some 
> > AND filters can still prune stripes.

YES_NO_NULL if nulls present else YES_NO


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63927/#review191419
---


On Nov. 17, 2017, 11:33 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63927/
> ---
> 
> (Updated Nov. 17, 2017, 11:33 p.m.)
> 
> 
> Review request for hive, Jason Dere and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  c32f79fe95 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
>  599b5191c6 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java
>  1ec702011a 
>   pom.xml 04fb7c3c55 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 1e5b841f4b 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java c150b6 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> 80b7be8e5a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java d2bb6417a2 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java 
> a916d58f3e 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicValue.java a20328cb69 
> 
> 
> Diff: https://reviews.apache.org/r/63927/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 63927: HIVE-17631 upgrade orc to 1.4.0

2017-11-17 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63927/#review191419
---




ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
Lines 2123 (patched)


Can you set truthValues[pred] to YES_NO or YES_NO_NULL instead, as some AND 
filters can still prune stripes.


- Prasanth_J


On Nov. 17, 2017, 11:33 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63927/
> ---
> 
> (Updated Nov. 17, 2017, 11:33 p.m.)
> 
> 
> Review request for hive, Jason Dere and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  c32f79fe95 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
>  599b5191c6 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java
>  1ec702011a 
>   pom.xml 04fb7c3c55 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 1e5b841f4b 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java c150b6 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> 80b7be8e5a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java d2bb6417a2 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java 
> a916d58f3e 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicValue.java a20328cb69 
> 
> 
> Diff: https://reviews.apache.org/r/63927/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 63893: HIVE-14487

2017-11-17 Thread Jesús Camacho Rodríguez



> On Nov. 17, 2017, 11:22 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
> > Line 1488 (original), 1488 (patched)
> > 
> >
> > This table object is never read from. Any reason for its existence.

It is passed to StatsWork in L1533. I think we do not need the full table, as 
StatsWork will only use some of the information. I can tackle in a follow-up?


> On Nov. 17, 2017, 11:22 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
> > Lines 1515 (patched)
> > 
> >
> > Is it necessary to make metastore call to get Table? Can this be 
> > avoided.

Observe this is the same as L1495. As mentioned above, I think we do not need 
the complete table for StatsWork, just a few properties. However, to make sure 
we have everything needed, I took it from metastore as it is already an 
existing one. I can take care of this in the follow-up to check whether we need 
to pass the complete table.


> On Nov. 17, 2017, 11:22 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java
> > Lines 341 (patched)
> > 
> >
> > We don't store comments for regular tables in property? Is this special 
> > for (materialized) views?

In fact, apparently we do: L779 in CreateTableDesc.java. The code was 
originally based on that one, thus they should be quite similar.


- Jesús


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63893/#review191406
---


On Nov. 17, 2017, 7:26 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63893/
> ---
> 
> (Updated Nov. 17, 2017, 7:26 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-14487
> https://issues.apache.org/jira/browse/HIVE-14487
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14487
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 
> 3b1005f85c419b1fa5646bc5a4b96788250e7c4b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> e7af5e004fb560b574b82f6d1b60517511802f37 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 67739a1db9fc52a67f4f5ea7dba80fe0e95750c8 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
> e9ae590f4c291ad00e385899822ef944d50cdcde 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 55b1da90300bdc864e9258d1c49c3c21791f3b64 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 
> f3f1a7e8da52332d93aa57865617befd6c07a2fc 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 1de3dd7230c5c3708fd7dd6baf9c6913e91cab3c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
> d139155a1d1433ca9186994cef2616f007a66fe9 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/BasicStatsWork.java 
> 0621bd413ec578f30a236de0be5774fc83457aa5 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java 
> 9425f6e89113afd82d1cdca30c75da6101b63b4a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ImportTableDesc.java 
> 1770046869170c30ebb27a84db7566d477bd9877 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/LoadFileDesc.java 
> c09589c5f23825ea39b85737e04cc0e365e61fea 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 
> 3c1e92af12185bb95e27fbf0af3038ca27566197 
>   
> ql/src/test/queries/clientnegative/materialized_view_authorization_rebuild_no_grant.q
>  PRE-CREATION 
>   
> ql/src/test/queries/clientnegative/materialized_view_authorization_rebuild_other.q
>  PRE-CREATION 
>   ql/src/test/queries/clientpositive/materialized_view_create_rewrite.q 
> 1749cb023ae2e4476ac386b667a708f50f8db462 
>   ql/src/test/queries/clientpositive/materialized_view_create_rewrite_3.q 
> PRE-CREATION 
>   
> ql/src/test/results/clientnegative/materialized_view_authorization_rebuild_no_grant.q.out
>  PRE-CREATION 
>   
> ql/src/test/results/clientnegative/materialized_view_authorization_rebuild_other.q.out
>  PRE-CREATION 
>   
> ql/src/test/results/clientpositive/beeline/materialized_view_create_rewrite.q.out
>  81a7950773983e28a0f36ce806b6cda394993ac0 
>   ql/src/test/results/clientpositive/create_view.q.out 
> 8763fdb75fc910f75c53346c9541703e800c128c 
>   ql/src/test/results/clientpositive/create_view_translate.q.out 
> e5748976bdb271bef386e6820be5236a36d4c941 
>   ql/src/test/results/clientpositive/explain_ddl.q.out 
>

Re: Review Request 63927: HIVE-17631 upgrade orc to 1.4.0

2017-11-17 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63927/
---

(Updated Nov. 17, 2017, 11:33 p.m.)


Review request for hive, Jason Dere and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 c32f79fe95 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
 599b5191c6 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java
 1ec702011a 
  pom.xml 04fb7c3c55 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 1e5b841f4b 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java c150b6 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
80b7be8e5a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java d2bb6417a2 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java 
a916d58f3e 
  ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicValue.java a20328cb69 


Diff: https://reviews.apache.org/r/63927/diff/2/

Changes: https://reviews.apache.org/r/63927/diff/1-2/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 63893: HIVE-14487

2017-11-17 Thread Ashutosh Chauhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63893/#review191406
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
Line 1488 (original), 1488 (patched)


This table object is never read from. Any reason for its existence.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
Lines 1515 (patched)


Is it necessary to make metastore call to get Table? Can this be avoided.



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
Lines 12708 (patched)


May use getTableObjectByName() to cache table object.



ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java
Lines 341 (patched)


We don't store comments for regular tables in property? Is this special for 
(materialized) views?


- Ashutosh Chauhan


On Nov. 17, 2017, 7:26 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63893/
> ---
> 
> (Updated Nov. 17, 2017, 7:26 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-14487
> https://issues.apache.org/jira/browse/HIVE-14487
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14487
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 
> 3b1005f85c419b1fa5646bc5a4b96788250e7c4b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> e7af5e004fb560b574b82f6d1b60517511802f37 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 67739a1db9fc52a67f4f5ea7dba80fe0e95750c8 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
> e9ae590f4c291ad00e385899822ef944d50cdcde 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 55b1da90300bdc864e9258d1c49c3c21791f3b64 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 
> f3f1a7e8da52332d93aa57865617befd6c07a2fc 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 1de3dd7230c5c3708fd7dd6baf9c6913e91cab3c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
> d139155a1d1433ca9186994cef2616f007a66fe9 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/BasicStatsWork.java 
> 0621bd413ec578f30a236de0be5774fc83457aa5 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java 
> 9425f6e89113afd82d1cdca30c75da6101b63b4a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ImportTableDesc.java 
> 1770046869170c30ebb27a84db7566d477bd9877 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/LoadFileDesc.java 
> c09589c5f23825ea39b85737e04cc0e365e61fea 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 
> 3c1e92af12185bb95e27fbf0af3038ca27566197 
>   
> ql/src/test/queries/clientnegative/materialized_view_authorization_rebuild_no_grant.q
>  PRE-CREATION 
>   
> ql/src/test/queries/clientnegative/materialized_view_authorization_rebuild_other.q
>  PRE-CREATION 
>   ql/src/test/queries/clientpositive/materialized_view_create_rewrite.q 
> 1749cb023ae2e4476ac386b667a708f50f8db462 
>   ql/src/test/queries/clientpositive/materialized_view_create_rewrite_3.q 
> PRE-CREATION 
>   
> ql/src/test/results/clientnegative/materialized_view_authorization_rebuild_no_grant.q.out
>  PRE-CREATION 
>   
> ql/src/test/results/clientnegative/materialized_view_authorization_rebuild_other.q.out
>  PRE-CREATION 
>   
> ql/src/test/results/clientpositive/beeline/materialized_view_create_rewrite.q.out
>  81a7950773983e28a0f36ce806b6cda394993ac0 
>   ql/src/test/results/clientpositive/create_view.q.out 
> 8763fdb75fc910f75c53346c9541703e800c128c 
>   ql/src/test/results/clientpositive/create_view_translate.q.out 
> e5748976bdb271bef386e6820be5236a36d4c941 
>   ql/src/test/results/clientpositive/explain_ddl.q.out 
> f0e54c515377e7e6e002b7cc9ae986340a93acce 
>   ql/src/test/results/clientpositive/llap/materialized_view_create.q.out 
> PRE-CREATION 
>   
> ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite.q.out
>  PRE-CREATION 
>   
> ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_2.q.out
>  PRE-CREATION 
>   
> ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_3.q.out
>  PRE-CREATION 
>   
> ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_multi_db.q.out
>  PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/materialized_view_describe.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/materialized_view_drop.q.out 
> PRE-CREATION 
>

[jira] [Created] (HIVE-18099) Hive shouldn't pickup mapreduce conf for Tez

2017-11-17 Thread Zhiyuan Yang (JIRA)

Zhiyuan Yang created HIVE-18099:
---

 Summary: Hive shouldn't pickup mapreduce conf for Tez
 Key: HIVE-18099
 URL: https://issues.apache.org/jira/browse/HIVE-18099
 Project: Hive
  Issue Type: Bug
Reporter: Zhiyuan Yang
Assignee: Zhiyuan Yang


Right now Hive is reading some mapreduce conf for Tez engine. 
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java?utf8=%E2%9C%93#L720
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java?#L796
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java?#L860



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 63927: HIVE-17631 upgrade orc to 1.4.0

2017-11-17 Thread Sergey Shelukhin



> On Nov. 17, 2017, 10:56 p.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
> > Lines 2124 (patched)
> > 
> >
> > This is handled by ORC-148 already. Any exception will not prune 
> > RG/Stripe.

Doesn't work, tests fail


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63927/#review191405
---


On Nov. 17, 2017, 10:42 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63927/
> ---
> 
> (Updated Nov. 17, 2017, 10:42 p.m.)
> 
> 
> Review request for hive, Jason Dere and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  c32f79fe95 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
>  599b5191c6 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java
>  1ec702011a 
>   pom.xml 04fb7c3c55 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 1e5b841f4b 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java c150b6 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> 80b7be8e5a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java d2bb6417a2 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java 
> a916d58f3e 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicValue.java a20328cb69 
> 
> 
> Diff: https://reviews.apache.org/r/63927/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 63890: HIVE-18025: Push resource plan changes to tez/unmanaged sessions

2017-11-17 Thread j . prasanth . j



> On Nov. 17, 2017, 8:18 p.m., Sergey Shelukhin wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java
> > Lines 517 (patched)
> > 
> >
> > this seems to get all triggers for the pools... I can file a follow-up 
> > jira, I think we need to make addition of triggers to Tez more explicit

Until we have explicit GLOBAL flag, this will get all triggers that is tied to 
default pool path.


> On Nov. 17, 2017, 8:18 p.m., Sergey Shelukhin wrote:
> > standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/WMFullResourcePlan.java
> > Line 329 (original), 329 (patched)
> > 
> >
> > this looks like a change to generated code without a thrift change. The 
> > null check (or isSet check?) should be in the caller

Oops. Didn't realize this is generated file. Moved it to caller.


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63890/#review191387
---


On Nov. 17, 2017, 2:20 a.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63890/
> ---
> 
> (Updated Nov. 17, 2017, 2:20 a.m.)
> 
> 
> Review request for hive and Sergey Shelukhin.
> 
> 
> Bugs: HIVE-18025
> https://issues.apache.org/jira/browse/HIVE-18025
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18025: Push resource plan changes to tez/unmanaged sessions
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestTriggersNoTezSessionPool.java
>  bcce3dc 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestTriggersTezSessionPoolManager.java
>  b377275 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java e7af5e0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 4e48f15 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolSession.java 
> 769b24a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 3990f95 
>   
> ql/src/java/org/apache/hadoop/hive/ql/wm/MetastoreGlobalTriggersFetcher.java 
> 87c007f 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 5a6d0c8 
>   
> standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/WMFullResourcePlan.java
>  1d21908 
> 
> 
> Diff: https://reviews.apache.org/r/63890/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

Re: Review Request 63927: HIVE-17631 upgrade orc to 1.4.0

2017-11-17 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63927/#review191405
---




pom.xml
Line 185 (original), 185 (patched)


Use 1.4.1?



ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
Lines 2124 (patched)


This is handled by ORC-148 already. Any exception will not prune RG/Stripe.



ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java
Lines 147 (patched)


looks like its already returned in close()? plz remove the TODO if so


- Prasanth_J


On Nov. 17, 2017, 10:42 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63927/
> ---
> 
> (Updated Nov. 17, 2017, 10:42 p.m.)
> 
> 
> Review request for hive, Jason Dere and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  c32f79fe95 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
>  599b5191c6 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java
>  1ec702011a 
>   pom.xml 04fb7c3c55 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 1e5b841f4b 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java c150b6 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> 80b7be8e5a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java d2bb6417a2 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java 
> a916d58f3e 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicValue.java a20328cb69 
> 
> 
> Diff: https://reviews.apache.org/r/63927/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-18098) Add support for Export/Import for Acid tables

2017-11-17 Thread Eugene Koifman (JIRA)

Eugene Koifman created HIVE-18098:
-

 Summary: Add support for Export/Import for Acid tables
 Key: HIVE-18098
 URL: https://issues.apache.org/jira/browse/HIVE-18098
 Project: Hive
  Issue Type: New Feature
  Components: Transactions
Reporter: Eugene Koifman


How should this work?
For regular tables export just copies the files under table root to a specified 
directory.
This doesn't make sense for Acid tables:
* Some data may belong to aborted transactons
* Transaction IDs are imbedded into data/files names.  You'd have export delta/ 
and base/ each of which may have files with the same names, e.g. bucket_0.  
 
* On import these IDs won't make sense in a different cluster or even a 
different table (which may have delta_x_x for example for the same x (but 
different data of course).
* Export creates a _metadata column types, storage format, etc.  Perhaps it can 
include info about aborted IDs (if the whole file can't be skipped).
* Even importing into the same table on the same cluster may be a problem.  For 
example delta_5_5/ existed at the time of export and was included in the 
export.  But 2 days later it may not exist because it was compacted and cleaned.
* If importing back into the same table on the same cluster, the data could be 
imported into a different transaction (assuming per table writeIDs) w/o having 
to remap the IDs in the rows themselves.
* support Import Overwrite?
* Support Import as a new txn with remapping of ROW_IDs?  The new writeID can 
be stored in a delta_x_x/_meta_data and ROW__IDs can be remapped at read time 
(like isOriginal) and made permanent by compaction.
* It doesn't seem reasonable to import acid data into non-acid table





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 63927: HIVE-17631 upgrade orc to 1.4.0

2017-11-17 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63927/
---

Review request for hive, Jason Dere and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs
-

  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 c32f79fe95 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
 599b5191c6 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java
 1ec702011a 
  pom.xml 04fb7c3c55 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 1e5b841f4b 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java c150b6 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
80b7be8e5a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java d2bb6417a2 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java 
a916d58f3e 
  ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicValue.java a20328cb69 


Diff: https://reviews.apache.org/r/63927/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-18097) query move phase 2 - oversubscribing the destination pool

2017-11-17 Thread Sergey Shelukhin (JIRA)

Sergey Shelukhin created HIVE-18097:
---

 Summary: query move phase 2 - oversubscribing the destination pool
 Key: HIVE-18097
 URL: https://issues.apache.org/jira/browse/HIVE-18097
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin


Final design TBD.
We may add an option to the trigger: "on full kill/ignore/delay kill".
The former will kill the query if the dest is full. 
The ignore one would not perform the move... The thinking is that admin would 
add another trigger to kill queries after some time (e.g. - move after 10s, 
kill after 30s), and thus give slow/large queries a grace period in case dest 
is full so they cannot gracefully release the capacity.
It's possible to add a third option - "delay kill", where we would ignore the 
move, but kill the query if someone requests capacity and the pool is full.

Note that none of these requires oversubscription which is a pain to track.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18096) add a user-friendly show plan command

2017-11-17 Thread Sergey Shelukhin (JIRA)

Sergey Shelukhin created HIVE-18096:
---

 Summary: add a user-friendly show plan command
 Key: HIVE-18096
 URL: https://issues.apache.org/jira/browse/HIVE-18096
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin


For admin to be able to get an overview of a resource plan.
We need to try to do this using sysdb. 
If that is not possible to do in a nice way, we'd do a text-based one like 
query explain, or desc extended table.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18095) add a global flag to triggers (applies to all WM pools & container based sessions)

2017-11-17 Thread Sergey Shelukhin (JIRA)

Sergey Shelukhin created HIVE-18095:
---

 Summary: add a global flag to triggers (applies to all WM pools & 
container based sessions)
 Key: HIVE-18095
 URL: https://issues.apache.org/jira/browse/HIVE-18095
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin


cc [~prasanth_j]

It should be impossible to attach global triggers for pools. Setting global 
flag should probably automatically remove attachments to pools.
Global triggers would only support actions that Tez supports (for simplicity; 
also, for now, move doesn't make a lot of sense because the trigger would apply 
again after the move).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 63890: HIVE-18025: Push resource plan changes to tez/unmanaged sessions

2017-11-17 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63890/#review191387
---




itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestTriggersNoTezSessionPool.java
Line 59 (original), 58 (patched)


doesn't look like pool is used



ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
Line 716 (original), 717 (patched)


nit: WorkloadManager.getInstance is enough to determine if it's enabled. 
WM is initialized based on queue name only



ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
Lines 743 (patched)


hmm... why is this an either or case? it's possible to have both in the 
same HS2. Should it call the else in both cases?



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java
Line 511 (original), 511 (patched)


openSessions appears to be synchronized elsewhere, accessing it here may 
not be thread safe. Might make sense to call this under the same sync block 
that makes the changes to the list.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java
Lines 517 (patched)


this seems to get all triggers for the pools... I can file a follow-up 
jira, I think we need to make addition of triggers to Tez more explicit



service/src/java/org/apache/hive/service/server/HiveServer2.java
Line 218 (original), 220 (patched)


same about else



standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/WMFullResourcePlan.java
Line 329 (original), 329 (patched)


this looks like a change to generated code without a thrift change. The 
null check (or isSet check?) should be in the caller


- Sergey Shelukhin


On Nov. 17, 2017, 2:20 a.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63890/
> ---
> 
> (Updated Nov. 17, 2017, 2:20 a.m.)
> 
> 
> Review request for hive and Sergey Shelukhin.
> 
> 
> Bugs: HIVE-18025
> https://issues.apache.org/jira/browse/HIVE-18025
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18025: Push resource plan changes to tez/unmanaged sessions
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestTriggersNoTezSessionPool.java
>  bcce3dc 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestTriggersTezSessionPoolManager.java
>  b377275 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java e7af5e0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 4e48f15 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolSession.java 
> 769b24a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 3990f95 
>   
> ql/src/java/org/apache/hadoop/hive/ql/wm/MetastoreGlobalTriggersFetcher.java 
> 87c007f 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 5a6d0c8 
>   
> standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/WMFullResourcePlan.java
>  1d21908 
> 
> 
> Diff: https://reviews.apache.org/r/63890/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

Re: Review Request 63864: HIVE-18072 WM - fix various bugs based on cluster testing - part 2

2017-11-17 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63864/
---

(Updated Nov. 17, 2017, 7:55 p.m.)


Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 a02a414a76 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
 9dc521e39e 
  
llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskSchedulerService.java
 51d2e08393 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 
851245c154 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
3990f95334 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
5ba6639e0c 


Diff: https://reviews.apache.org/r/63864/diff/4/

Changes: https://reviews.apache.org/r/63864/diff/3-4/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 63893: HIVE-14487

2017-11-17 Thread Jesús Camacho Rodríguez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63893/
---

(Updated Nov. 17, 2017, 7:26 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

Updating q files


Bugs: HIVE-14487
https://issues.apache.org/jira/browse/HIVE-14487


Repository: hive-git


Description
---

HIVE-14487


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 
3b1005f85c419b1fa5646bc5a4b96788250e7c4b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
e7af5e004fb560b574b82f6d1b60517511802f37 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
67739a1db9fc52a67f4f5ea7dba80fe0e95750c8 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
e9ae590f4c291ad00e385899822ef944d50cdcde 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
55b1da90300bdc864e9258d1c49c3c21791f3b64 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 
f3f1a7e8da52332d93aa57865617befd6c07a2fc 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
1de3dd7230c5c3708fd7dd6baf9c6913e91cab3c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
d139155a1d1433ca9186994cef2616f007a66fe9 
  ql/src/java/org/apache/hadoop/hive/ql/plan/BasicStatsWork.java 
0621bd413ec578f30a236de0be5774fc83457aa5 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java 
9425f6e89113afd82d1cdca30c75da6101b63b4a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ImportTableDesc.java 
1770046869170c30ebb27a84db7566d477bd9877 
  ql/src/java/org/apache/hadoop/hive/ql/plan/LoadFileDesc.java 
c09589c5f23825ea39b85737e04cc0e365e61fea 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 
3c1e92af12185bb95e27fbf0af3038ca27566197 
  
ql/src/test/queries/clientnegative/materialized_view_authorization_rebuild_no_grant.q
 PRE-CREATION 
  
ql/src/test/queries/clientnegative/materialized_view_authorization_rebuild_other.q
 PRE-CREATION 
  ql/src/test/queries/clientpositive/materialized_view_create_rewrite.q 
1749cb023ae2e4476ac386b667a708f50f8db462 
  ql/src/test/queries/clientpositive/materialized_view_create_rewrite_3.q 
PRE-CREATION 
  
ql/src/test/results/clientnegative/materialized_view_authorization_rebuild_no_grant.q.out
 PRE-CREATION 
  
ql/src/test/results/clientnegative/materialized_view_authorization_rebuild_other.q.out
 PRE-CREATION 
  
ql/src/test/results/clientpositive/beeline/materialized_view_create_rewrite.q.out
 81a7950773983e28a0f36ce806b6cda394993ac0 
  ql/src/test/results/clientpositive/create_view.q.out 
8763fdb75fc910f75c53346c9541703e800c128c 
  ql/src/test/results/clientpositive/create_view_translate.q.out 
e5748976bdb271bef386e6820be5236a36d4c941 
  ql/src/test/results/clientpositive/explain_ddl.q.out 
f0e54c515377e7e6e002b7cc9ae986340a93acce 
  ql/src/test/results/clientpositive/llap/materialized_view_create.q.out 
PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite.q.out 
PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_2.q.out
 PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_3.q.out
 PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_multi_db.q.out
 PRE-CREATION 
  ql/src/test/results/clientpositive/llap/materialized_view_describe.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/materialized_view_drop.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/materialized_view_rewrite_ssb.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/materialized_view_rewrite_ssb_2.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/selectDistinctStar.q.out 
befb6e79929ca7fffc9ee56544c25a27c2a84cbd 
  ql/src/test/results/clientpositive/llap/union_top_level.q.out 
a0102321711de82100a5b117b853a364109a2622 
  ql/src/test/results/clientpositive/llap/vector_windowing.q.out 
776ae3e085840dcaa298c143966ff1f12e508c3e 
  ql/src/test/results/clientpositive/materialized_view_create.q.out 
0fca69f4fa94a24482d09a1b30a72611eec6c643 
  ql/src/test/results/clientpositive/materialized_view_create_rewrite.q.out 
81a7950773983e28a0f36ce806b6cda394993ac0 
  ql/src/test/results/clientpositive/materialized_view_create_rewrite_3.q.out 
PRE-CREATION 
  
ql/src/test/results/clientpositive/materialized_view_create_rewrite_multi_db.q.out
 f715803438d100599fc669cb9b1086523cd44099 
  ql/src/test/results/clientpositive/materialized_view_describe.q.out 
8b7691279c0b24c599656c372de22ef9cf83f8f7 


Diff: https://reviews.apache.org/r/63893/diff/2/

Changes: https://reviews.apache.org/r/63893/diff/1-2/


Testing
---


Thanks,

Jesús Camacho Rodríguez

[jira] [Created] (HIVE-18094) exceptions in sysdb test logs

2017-11-17 Thread Sergey Shelukhin (JIRA)

Sergey Shelukhin created HIVE-18094:
---

 Summary: exceptions in sysdb test logs
 Key: HIVE-18094
 URL: https://issues.apache.org/jira/browse/HIVE-18094
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


I was looking at some other test logs and found lots of errors like this in the 
logs (repeated for many sysdb queries)
{noformat}
2017-11-17T00:46:13,222 DEBUG [abb838ff-8a13-4170-b2ec-3755c3959dbb main] 
conf.VariableSubstitution: Substitution is on: 

select * from TABLES order by TABLE_SCHEMA, TABLE_NAME


2017-11-17T00:46:13,576 DEBUG [TezTR-327432_1_74_1_0_0] 
dao.GenericJdbcDatabaseAccessor: Query to execute is [SELECT
  "TBL_ID",
  "CREATE_TIME",
  "DB_ID",
  "LAST_ACCESS_TIME",
  "OWNER",
  "RETENTION",
  "SD_ID",
  "TBL_NAME",
  "TBL_TYPE",
  "VIEW_EXPANDED_TEXT",
  "VIEW_ORIGINAL_TEXT",
  "IS_REWRITE_ENABLED"
FROM TBLS {LIMIT 1}]
2017-11-17T00:46:13,576 DEBUG [TezTR-327432_1_74_1_0_0] exec.MapOperator: 
Adding alias t to work list for file 
file:/home/hiveptest/35.188.147.28-hiveptest-1/apache-github-source-source/itests/qtest/target/localfs/warehouse/sys.db/tbls
2017-11-17T00:46:13,576 DEBUG [TezTR-327432_1_74_1_0_0] jdbc.JdbcSerDe: 
Initializing the SerDe
2017-11-17T00:46:13,576 DEBUG [TezTR-327432_1_74_1_0_0] 
conf.JdbcStorageConfigManager: Resolving db type: DERBY
2017-11-17T00:46:13,578 ERROR [TezTR-327432_1_74_0_0_0] 
jdbc.QueryConditionBuilder: Error during condition build
java.lang.ArrayIndexOutOfBoundsException: 0
at java.beans.XMLDecoder.readObject(XMLDecoder.java:250) ~[?:1.8.0_102]
at 
org.apache.hive.storage.jdbc.QueryConditionBuilder.createConditionString(QueryConditionBuilder.java:125)
 [hive-jdbc-handler-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hive.storage.jdbc.QueryConditionBuilder.buildCondition(QueryConditionBuilder.java:74)
 [hive-jdbc-handler-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hive.storage.jdbc.conf.JdbcStorageConfigManager.getQueryToExecute(JdbcStorageConfigManager.java:123)
 [hive-jdbc-handler-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getRecordIterator(GenericJdbcDatabaseAccessor.java:141)
 [hive-jdbc-handler-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:58) 
[hive-jdbc-handler-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:35) 
[hive-jdbc-handler-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:360)
 [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79) 
[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33) 
[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
 [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
 [tez-mapreduce-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT]
at 
org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116) 
[tez-mapreduce-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
 [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)
 [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:254)
 [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:237) 
[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
 [tez-runtime-internals-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT]
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
 [tez-runtime-internals-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT]
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
 [tez-runtime-internals-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method) 
~[?:1.8.0_102]
at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_102]
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
 [hadoop-common-3.0.0-beta1.jar:?]
at

[jira] [Created] (HIVE-18093) Improve logging when HoS application is killed

2017-11-17 Thread Sahil Takiar (JIRA)

Sahil Takiar created HIVE-18093:
---

 Summary: Improve logging when HoS application is killed
 Key: HIVE-18093
 URL: https://issues.apache.org/jira/browse/HIVE-18093
 Project: Hive
  Issue Type: Sub-task
  Components: Spark
Reporter: Sahil Takiar


When a HoS jobs is explicitly killed via a user (via a yarn command), the logs 
just say "RPC channel closed"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18092) ColumnStats autogather may only run if BasicStats are being collected

2017-11-17 Thread Zoltan Haindrich (JIRA)

Zoltan Haindrich created HIVE-18092:
---

 Summary: ColumnStats autogather may only run if BasicStats are 
being collected
 Key: HIVE-18092
 URL: https://issues.apache.org/jira/browse/HIVE-18092
 Project: Hive
  Issue Type: Sub-task
  Components: Statistics
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


currently; if column.stats.autogather is enabled; the hbase tests are run into 
an exception which arises from the fact that basic stats are not collected on 
that table...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18091) Failing tests of itests/qtest-spark and itests/hive-unit on branch-1

2017-11-17 Thread Daniel Voros (JIRA)

Daniel Voros created HIVE-18091:
---

 Summary: Failing tests of itests/qtest-spark and itests/hive-unit 
on branch-1
 Key: HIVE-18091
 URL: https://issues.apache.org/jira/browse/HIVE-18091
 Project: Hive
  Issue Type: Bug
  Components: Test, Testing Infrastructure
Reporter: Daniel Voros
Assignee: Daniel Voros


Seen this while looking at ptest results for HIVE-17947 but is probably an 
older issue.

Tests under itests/qtest-spark and itests/hive-unit fail when trying to execute 
the download-spark plugin with:

{code}
[INFO]
[INFO] 
[INFO] Building Hive Integration - Unit Tests 1.3.0-SNAPSHOT
[INFO] 
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @ 
hive-it-unit ---
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (download-spark) @ hive-it-unit ---
[INFO] Executing tasks

main:
 [exec] + /bin/pwd
 [exec] + BASE_DIR=./target
 [exec] + HIVE_ROOT=./target/../../../
 [exec] + DOWNLOAD_DIR=./../thirdparty
 [exec] + mkdir -p ./../thirdparty
 [exec] 
/home/hiveptest/35.192.99.254-hiveptest-0/apache-github-branch-1-source/itests/hive-unit
 [exec] + download 
http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.5.0-bin-hadoop2-without-hive.tgz
 spark
 [exec] + 
url=http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.5.0-bin-hadoop2-without-hive.tgz
 [exec] + finalName=spark
 [exec] ++ basename 
http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.5.0-bin-hadoop2-without-hive.tgz
 [exec] + tarName=spark-1.5.0-bin-hadoop2-without-hive.tgz
 [exec] + rm -rf ./target/spark
 [exec] + [[ ! -f ./../thirdparty/spark-1.5.0-bin-hadoop2-without-hive.tgz 
]]
 [exec] + tar -zxf ./../thirdparty/spark-1.5.0-bin-hadoop2-without-hive.tgz 
-C ./target
 [exec] + mv ./target/spark-1.5.0-bin-hadoop2-without-hive ./target/spark
 [exec] + cp -f ./target/../../..//data/conf/spark/log4j.properties 
./target/spark/conf/
 [exec] + sed '/package /d' 
/data/hiveptest/working/apache-github-branch-1-source/itests/../contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleAdd.java
 [exec] sed: can't read 
/data/hiveptest/working/apache-github-branch-1-source/itests/../contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleAdd.java:
 No such file or directory
 [exec] + javac -cp 
/data/hiveptest/working/maven/org/apache/hive/hive-exec/1.3.0-SNAPSHOT/hive-exec-1.3.0-SNAPSHOT.jar
 /tmp/UDFExampleAdd.java -d /tmp
 [exec] + jar -cf /tmp/udfexampleadd-1.0.jar -C /tmp UDFExampleAdd.class
 [exec] /tmp/UDFExampleAdd.class : no such file or directory
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 8.376s
[INFO] Finished at: Mon Nov 06 22:29:39 UTC 2017
[INFO] Final Memory: 18M/241M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-antrun-plugin:1.7:run (download-spark) on 
project hive-it-unit: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part .. @ 4:141 in 
/home/hiveptest/35.192.99.254-hiveptest-0/apache-github-branch-1-source/itests/hive-unit/target/antrun/build-main.xml
{code}

{{mvn antrun:run@download-spark}} passes when run locally, so I guess it might 
be an issue with the way wer're executing ptests.

Full list of classes with the same error:

{code}
TestAcidOnTez
TestAdminUser
TestAuthorizationPreEventListener
TestAuthzApiEmbedAuthorizerInEmbed
TestAuthzApiEmbedAuthorizerInRemote
TestBeeLineWithArgs
TestCLIAuthzSessionContext
TestClearDanglingScratchDir
TestClientSideAuthorizationProvider
TestCompactor
TestCreateUdfEntities
TestCustomAuthentication
TestDBTokenStore
TestDDLWithRemoteMetastoreSecondNamenode
TestDynamicSerDe
TestEmbeddedHiveMetaStore
TestEmbeddedThriftBinaryCLIService
TestFilterHooks
TestFolderPermissions
TestHS2AuthzContext
TestHS2AuthzSessionContext
TestHS2ClearDanglingScratchDir
TestHS2ImpersonationWithRemoteMS
TestHiveAuthorizerCheckInvocation
TestHiveAuthorizerShowFilters
TestHiveHistory
TestHiveMetaStoreTxns
TestHiveMetaStoreWithEnvironmentContext
TestHiveMetaTool
TestHiveServer2
TestHiveServer2SessionTimeout
TestHiveSessionImpl
TestHs2Hooks
TestJdbcDriver2
TestJdbcMetadataApiAuth
TestJdbcWithLocalClusterSpark
TestJdbcWithMiniHS2
TestJdbcWithMiniMr
TestJdbcWithSQLAuthUDFBlacklist
TestJdbcWithSQLAuthorization
TestLocationQueries
TestMTQueries
TestMarkPartition
TestMarkPartitionRemote
TestMetaStoreAuthorization
TestMetaStoreConnectionUrlHook
TestMetaStoreEndFunctionListener
TestMetaStoreEventListener

[jira] [Created] (HIVE-18090) acid heartbeat fails when metastore is connected via hadoop credential

2017-11-17 Thread anishek (JIRA)

anishek created HIVE-18090:
--

 Summary: acid heartbeat fails when metastore is connected via 
hadoop credential
 Key: HIVE-18090
 URL: https://issues.apache.org/jira/browse/HIVE-18090
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 3.0.0
Reporter: anishek
Assignee: anishek
 Fix For: 3.0.0


steps to recreate the issue. assuming two users 
* test
* another 

create two jceks files for each user and place them on hdfs with access to that 
file only allowed to the user. hdfs locations with permissions 
{code}
-rwx--   1 another another492 2017-11-16 13:06 
/user/another/another.jceks
-rwx--   1 test test489 2017-11-16 13:05 /user/test/test.jceks
{code}

password used to create 
* /user/another/another.jceks -- another
* /user/test/test.jceks -- test

enable ACID on HS2. started hiveserver2 with 
{code}
hive.metastore.warehouse.dir=file:///tmp/hive/test-warehouse
{code}

connect to the server using beeline using any user:
{code}
create table a (i int, b string);
insert into a values (0 , '0'), (1 , '1'), (2 , '2'), (3 , '3'), (4 , '4'), (5 
, '5'), (6 , '6'), (7 , '7'), (8 , '8'), (9 , '9'), (10 , '10'), (11 , '11'), 
(12 , '12'), (13 , '13'), (14 , '14'), (15 , '15'), (16 , '16'), (17 , '17'), 
(18 , '18'), (19 , '19'), (20 , '20'), (21 , '21'), (22 , '22'), (23 , '23'), 
(24 , '24'), (25 , '25'), (26 , '26'), (27 , '27'), (28 , '28'), (29 , '29'), 
(30 , '30'), (31 , '31'), (32 , '32'), (33 , '33'), (34 , '34'), (35 , '35'), 
(36 , '36'), (37 , '37'), (38 , '38'), (39 , '39'), (40 , '40'), (41 , '41'), 
(42 , '42'), (43 , '43'), (44 , '44'), (45 , '45'), (46 , '46'), (47 , '47'), 
(48 , '48'), (49 , '49'), (50 , '50'), (51 , '51'), (52 , '52'), (53 , '53'), 
(54 , '54'), (55 , '55'), (56 , '56'), (57 , '57'), (58 , '58'), (59 , '59'), 
(60 , '60'), (61 , '61'), (62 , '62'), (63 , '63'), (64 , '64'), (65 , '65'), 
(66 , '66'), (67 , '67'), (68 , '68'), (69 , '69'), (70 , '70'), (71 , '71'), 
(72 , '72'), (73 , '73'), (74 , '74'), (75 , '75'), (76 , '76'), (77 , '77'), 
(78 , '78'), (79 , '79'), (80 , '80'), (81 , '81'), (82 , '82'), (83 , '83'), 
(84 , '84'), (85 , '85'), (86 , '86'), (87 , '87'), (88 , '88'), (89 , '89'), 
(90 , '90'), (91 , '91'), (92 , '92'), (93 , '93'), (94 , '94'), (95 , '95'), 
(96 , '96'), (97 , '97'), (98 , '98'), (99 , '99');
{code}

exit beeline and connect with user test
{code}
./beeline -u 
"jdbc:hive2://localhost:1/default?hive.strict.checks.cartesian.product=false;hive.txn.timeout=4s;hive.txn.heartbeat.threadpool.size=1;hadoop.security.credential.provider.path=jceks://hdfs/user/test/test.jceks;ssl.server.keystore.keypassword=test"
 -n test

create table a_acid (i int, b string) clustered by (i) into 8 buckets stored as 
orc tblproperties('transactional'='tur');

insert overwrite table a_acid select a2.i, a3.b from a a1 join a a2 join a a3 
on 1=1;
{code}

open another beeline session with user another:
{code}
./beeline -u 
"jdbc:hive2://localhost:1/default?hive.strict.checks.cartesian.product=false;hive.txn.timeout=4s;hive.txn.heartbeat.threadpool.size=1;hadoop.security.credential.provider.path=jceks://hdfs/user/another/another.jceks;ssl.server.keystore.keypassword=another"
 -n another

create table another_a_acid (i int, b string) clustered by (i) into 8 buckets 
stored as orc tblproperties('transactional'='tur');

insert overwrite table another_a_acid select a2.i, a3.b from a a1 join a a2 
join a a3 on 1=1;

{code}

fails with exception 
{code}
2017-11-17T12:15:52,664 DEBUG [Heartbeater-1] retry.RetryInvocationHandler: 
Exception while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over 
null. Not retrying because try once and fail.
org.apache.hadoop.ipc.RemoteException: Permission denied: user=test, 
access=EXECUTE, inode="/user/another/another.jceks":another:another:drwx--
at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:259)
at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:205)
at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
at 
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1955)
at 
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:109)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:4111)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1137)
at

Re: Review Request 63927: HIVE-17631 upgrade orc to 1.4.0

[jira] [Created] (HIVE-18100) Some tests time out

Re: Review Request 63927: HIVE-17631 upgrade orc to 1.4.0

Re: Review Request 63927: HIVE-17631 upgrade orc to 1.4.0

Re: Review Request 63893: HIVE-14487

Re: Review Request 63890: HIVE-18025: Push resource plan changes to tez/unmanaged sessions

Re: Review Request 63893: HIVE-14487

Re: Review Request 63893: HIVE-14487

Re: Review Request 63927: HIVE-17631 upgrade orc to 1.4.0

Re: Review Request 63927: HIVE-17631 upgrade orc to 1.4.0

Re: Review Request 63893: HIVE-14487

Re: Review Request 63927: HIVE-17631 upgrade orc to 1.4.0

Re: Review Request 63893: HIVE-14487

[jira] [Created] (HIVE-18099) Hive shouldn't pickup mapreduce conf for Tez

Re: Review Request 63927: HIVE-17631 upgrade orc to 1.4.0

Re: Review Request 63890: HIVE-18025: Push resource plan changes to tez/unmanaged sessions

Re: Review Request 63927: HIVE-17631 upgrade orc to 1.4.0

[jira] [Created] (HIVE-18098) Add support for Export/Import for Acid tables

Review Request 63927: HIVE-17631 upgrade orc to 1.4.0

[jira] [Created] (HIVE-18097) query move phase 2 - oversubscribing the destination pool

[jira] [Created] (HIVE-18096) add a user-friendly show plan command

[jira] [Created] (HIVE-18095) add a global flag to triggers (applies to all WM pools & container based sessions)

Re: Review Request 63890: HIVE-18025: Push resource plan changes to tez/unmanaged sessions

Re: Review Request 63864: HIVE-18072 WM - fix various bugs based on cluster testing - part 2

Re: Review Request 63893: HIVE-14487

[jira] [Created] (HIVE-18094) exceptions in sysdb test logs

[jira] [Created] (HIVE-18093) Improve logging when HoS application is killed

[jira] [Created] (HIVE-18092) ColumnStats autogather may only run if BasicStats are being collected

[jira] [Created] (HIVE-18091) Failing tests of itests/qtest-spark and itests/hive-unit on branch-1

[jira] [Created] (HIVE-18090) acid heartbeat fails when metastore is connected via hadoop credential

30 matches

Site Navigation

Mail list logo

Footer information