date:20200108

[jira] [Commented] (HIVE-22007) Do not push unsupported types to specific JDBC sources from Calcite

2020-01-08 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011506#comment-17011506
 ] 

Hive QA commented on HIVE-22007:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
6s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} | {color:green} ql: The patch generated 0 new + 0 unchanged - 4 
fixed = 0 total (was 4) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 25s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20124/dev-support/hive-personality.sh
 |
| git revision | master / 706c1d4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-20124/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Do not push unsupported types to specific JDBC sources from Calcite
> ---
>
> Key: HIVE-22007
> URL: https://issues.apache.org/jira/browse/HIVE-22007
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-22007.01.patch, HIVE-22007.01.patch, 
> HIVE-22007.patch
>
>
> We should not push a project expression if it uses a type that a specific 
> dialect does not support, e.g., boolean in Oracle.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22710) OOM Exception when enable CBO

2020-01-08 Thread Chinna Rao Lalam (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinna Rao Lalam updated HIVE-22710:

Description: 
When set hive.cbo.enable=true, and execute a SQL contains union all and join to 
insert the result into three tables,  OOM will appear in the compilation phase. 
Attached error logs.

With attached SQL file OOM will come. 

Same problem observed in HIVE master branch also.

{noformat}

 Caused by: java.lang.OutOfMemoryError: Java heap space
 at java.util.Arrays.copyOf(Arrays.java:3332) ~[?:1.8.0_201]
 at 
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
 ~[?:1.8.0_201]
 at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) 
~[?:1.8.0_201]
 at java.lang.StringBuilder.append(StringBuilder.java:136) ~[?:1.8.0_201]
 at org.apache.calcite.rex.RexCall.computeDigest(RexCall.java:167) 
~[calcite-core-1.19.0.jar:1.19.0]
 at org.apache.calcite.rex.RexCall.toString(RexCall.java:183) 
~[calcite-core-1.19.0.jar:1.19.0]
 at java.lang.String.valueOf(String.java:2994) ~[?:1.8.0_201]
 at java.lang.StringBuilder.append(StringBuilder.java:131) ~[?:1.8.0_201]
 at org.apache.calcite.rex.RexCall.appendOperands(RexCall.java:109) 
~[calcite-core-1.19.0.jar:1.19.0]
 at org.apache.calcite.rex.RexCall.computeDigest(RexCall.java:166) 
~[calcite-core-1.19.0.jar:1.19.0]
 at org.apache.calcite.rex.RexCall.toString(RexCall.java:183) 
~[calcite-core-1.19.0.jar:1.19.0]
 at 
org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates$JoinConditionBasedPredicateInference.(HiveRelMdPredicates.java:349)
 ~[hive-exec-3.1.0.jar:3.1.0]
 at 
org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates$JoinConditionBasedPredicateInference.(HiveRelMdPredicates.java:319)
 ~[hive-exec-3.1.0.jar:3.1.0]
 at 
org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates.getPredicates(HiveRelMdPredicates.java:183)
 ~[hive-exec-3.1.0.jar:3.1.0]

{noformat}



  was:When set hive.cbo.enable=true, and execute a SQL contains union all and 
join to insert the result into three tables. Outofmemoryerror will appear in 
the compilation phase. For detailed SQL statements and error logs, please refer 
to the attachment. The same problem exists in hive 4.0.0


> OOM Exception when enable CBO
> -
>
> Key: HIVE-22710
> URL: https://issues.apache.org/jira/browse/HIVE-22710
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.1.0, 3.1.2
>Reporter: LuGuangMing
>Priority: Major
> Attachments: error_stack.log, sample.sql
>
>
> When set hive.cbo.enable=true, and execute a SQL contains union all and join 
> to insert the result into three tables,  OOM will appear in the compilation 
> phase. Attached error logs.
> With attached SQL file OOM will come. 
> Same problem observed in HIVE master branch also.
> {noformat}
>  Caused by: java.lang.OutOfMemoryError: Java heap space
>  at java.util.Arrays.copyOf(Arrays.java:3332) ~[?:1.8.0_201]
>  at 
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
>  ~[?:1.8.0_201]
>  at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) 
> ~[?:1.8.0_201]
>  at java.lang.StringBuilder.append(StringBuilder.java:136) ~[?:1.8.0_201]
>  at org.apache.calcite.rex.RexCall.computeDigest(RexCall.java:167) 
> ~[calcite-core-1.19.0.jar:1.19.0]
>  at org.apache.calcite.rex.RexCall.toString(RexCall.java:183) 
> ~[calcite-core-1.19.0.jar:1.19.0]
>  at java.lang.String.valueOf(String.java:2994) ~[?:1.8.0_201]
>  at java.lang.StringBuilder.append(StringBuilder.java:131) ~[?:1.8.0_201]
>  at org.apache.calcite.rex.RexCall.appendOperands(RexCall.java:109) 
> ~[calcite-core-1.19.0.jar:1.19.0]
>  at org.apache.calcite.rex.RexCall.computeDigest(RexCall.java:166) 
> ~[calcite-core-1.19.0.jar:1.19.0]
>  at org.apache.calcite.rex.RexCall.toString(RexCall.java:183) 
> ~[calcite-core-1.19.0.jar:1.19.0]
>  at 
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates$JoinConditionBasedPredicateInference.(HiveRelMdPredicates.java:349)
>  ~[hive-exec-3.1.0.jar:3.1.0]
>  at 
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates$JoinConditionBasedPredicateInference.(HiveRelMdPredicates.java:319)
>  ~[hive-exec-3.1.0.jar:3.1.0]
>  at 
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates.getPredicates(HiveRelMdPredicates.java:183)
>  ~[hive-exec-3.1.0.jar:3.1.0]
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22710) OOM Exception when enable CBO

2020-01-08 Thread Chinna Rao Lalam (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinna Rao Lalam updated HIVE-22710:

Summary: OOM Exception when enable CBO  (was: When set CBO on, 
CalcitePlanner genOPTree will be Out Of Memory)

> OOM Exception when enable CBO
> -
>
> Key: HIVE-22710
> URL: https://issues.apache.org/jira/browse/HIVE-22710
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.1.0, 3.1.2
>Reporter: LuGuangMing
>Priority: Major
> Attachments: error_stack.log, sample.sql
>
>
> When set hive.cbo.enable=true, and execute a SQL contains union all and join 
> to insert the result into three tables. Outofmemoryerror will appear in the 
> compilation phase. For detailed SQL statements and error logs, please refer 
> to the attachment. The same problem exists in hive 4.0.0



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22709) NullPointerException during query compilation after HIVE-22578

2020-01-08 Thread Aron Hamvas (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011500#comment-17011500
 ] 

Aron Hamvas commented on HIVE-22709:


LGTM.

> NullPointerException during query compilation after HIVE-22578
> --
>
> Key: HIVE-22709
> URL: https://issues.apache.org/jira/browse/HIVE-22709
> Project: Hive
>  Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-22709.1.patch, results_cache_with_auth.q
>
>
> Getting a NPE during query compilation, when query results cache and Ranger 
> auth is enabled. This seems to have been caused by HIVE-22578.
> {noformat}
>  java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getQueryStringFromAst(SemanticAnalyzer.java:14987)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getQueryStringForCache(SemanticAnalyzer.java:15036)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.createLookupInfoForQuery(SemanticAnalyzer.java:15077)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12513)
>   at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:358)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:283)
>   at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:171)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:283)
>   at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:219)
>   at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:103)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:215)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:828)
>   at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:774)
>   at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:768)
>   at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125)
>   at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:249)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:193)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:415)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:346)
>   at 
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:708)
>   at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:678)
>   at 
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:169)
>   at 
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
>   at 
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:59)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22710) When set CBO on, CalcitePlanner genOPTree will be Out Of Memory

2020-01-08 Thread Chinna Rao Lalam (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011498#comment-17011498
 ] 

Chinna Rao Lalam commented on HIVE-22710:
-

Hi [~jcamachorodriguez],

We encountered this issue and when analyzing we come across these issues 
CALCITE-2883 And HIVE-21365. Do you think still OOM problem is there in 
HepPlanner of CALCITE.  Please throw some insights about this issue.

 

 

> When set CBO on, CalcitePlanner genOPTree will be Out Of Memory
> ---
>
> Key: HIVE-22710
> URL: https://issues.apache.org/jira/browse/HIVE-22710
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.1.0, 3.1.2
>Reporter: LuGuangMing
>Priority: Major
> Attachments: error_stack.log, sample.sql
>
>
> When set hive.cbo.enable=true, and execute a SQL contains union all and join 
> to insert the result into three tables. Outofmemoryerror will appear in the 
> compilation phase. For detailed SQL statements and error logs, please refer 
> to the attachment. The same problem exists in hive 4.0.0



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22007) Do not push unsupported types to specific JDBC sources from Calcite

2020-01-08 Thread Jesus Camacho Rodriguez (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-22007:
---
Attachment: HIVE-22007.01.patch

> Do not push unsupported types to specific JDBC sources from Calcite
> ---
>
> Key: HIVE-22007
> URL: https://issues.apache.org/jira/browse/HIVE-22007
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-22007.01.patch, HIVE-22007.01.patch, 
> HIVE-22007.patch
>
>
> We should not push a project expression if it uses a type that a specific 
> dialect does not support, e.g., boolean in Oracle.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22573) HMS filter metadata using Ranger plugin

2020-01-08 Thread Ramesh Mani (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011436#comment-17011436
 ] 

Ramesh Mani commented on HIVE-22573:


[~samuelan] [~thejas] Can this merged?

> HMS filter metadata using Ranger plugin
> ---
>
> Key: HIVE-22573
> URL: https://issues.apache.org/jira/browse/HIVE-22573
> Project: Hive
>  Issue Type: New Feature
>  Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Sam An
>Assignee: Ramesh Mani
>Priority: Major
> Attachments: HIVE-22573.2.patch, HIVE-22573.3.patch, 
> HIVE-22573.4.patch, HIVE-22573.5.patch, HIVE-22573.patch
>
>
> HMS needs to filter out rows based on Ranger plugin. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (HIVE-22573) HMS filter metadata using Ranger plugin

2020-01-08 Thread Ramesh Mani (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011436#comment-17011436
 ] 

Ramesh Mani edited comment on HIVE-22573 at 1/9/20 5:53 AM:


[~samuelan] [~thejas] Can this be merged?


was (Author: rmani):
[~samuelan] [~thejas] Can this merged?

> HMS filter metadata using Ranger plugin
> ---
>
> Key: HIVE-22573
> URL: https://issues.apache.org/jira/browse/HIVE-22573
> Project: Hive
>  Issue Type: New Feature
>  Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Sam An
>Assignee: Ramesh Mani
>Priority: Major
> Attachments: HIVE-22573.2.patch, HIVE-22573.3.patch, 
> HIVE-22573.4.patch, HIVE-22573.5.patch, HIVE-22573.patch
>
>
> HMS needs to filter out rows based on Ranger plugin. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22708) Test fix for http transport

2020-01-08 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011410#comment-17011410
 ] 

Hive QA commented on HIVE-22708:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12990364/HIVE-22708.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17858 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamptz_2] 
(batchId=90)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/20123/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20123/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20123/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12990364 - PreCommit-HIVE-Build

> Test fix for http transport
> ---
>
> Key: HIVE-22708
> URL: https://issues.apache.org/jira/browse/HIVE-22708
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-22708.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22704) Distribution package incorrectly ships the upgrade.order files from the metastore module

2020-01-08 Thread Naveen Gangam (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011389#comment-17011389
 ] 

Naveen Gangam commented on HIVE-22704:
--

[~kgyrtkirk] The change makes sense to me. So +1 for me pending tests.
Also should we take this chance to also relocate the 
"https://github.com/apache/hive/tree/master/metastore/scripts/upgrade/hive/*4.0.0.sql;
 files to the be under standalone-metastore directory. It is inconsistent with 
the other DBs.

> Distribution package incorrectly ships the upgrade.order files from the 
> metastore module
> 
>
> Key: HIVE-22704
> URL: https://issues.apache.org/jira/browse/HIVE-22704
> Project: Hive
>  Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22704.01.patch
>
>
> Expected would be to take the upgrade.order from the standalon-metastore but 
> HIVE-19938 have broke that when it [introduced a 
> regex|https://github.infra.cloudera.com/CDH/hive/commit/bae4c08408720d56159847603dfb8d9dab9fe1b7#diff-44c56fadf216c807d6828b60825e43baR219]
> {code}
> $ echo asd > ./metastore/scripts/upgrade/postgres/upgrade.order.postgres
> $ mvn install -pl packaging/ -Pdist -am -DskipTests -q
> $ cat 
> ./packaging/target/apache-hive-4.0.0-SNAPSHOT-bin/apache-hive-4.0.0-SNAPSHOT-bin/scripts/metastore/upgrade/postgres/upgrade.order.postgres
> asd
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22355) Beeline should not prompt for hive user and password when authentication is NONE

2020-01-08 Thread Naveen Gangam (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011371#comment-17011371
 ] 

Naveen Gangam commented on HIVE-22355:
--

[~matijhs] Looks like the pre-commits did not run with the latest patch. Could 
you please re-submit the patch after reviewing the following comments (just 
nits)?

Could you please merge these 2 logs messages on consecutive lines of code into 
single log message? We don't want too much noise on the beeline shell
beeLine.info("Authentication type:" + auth);
beeLine.info("Connecting to " + url);

Also perhaps we could make this log message a debug message? Currently this 
should only be printed when auth=NONE but wondering if this is useful at info 
level?
beeLine.info("username:" + username);

Otherwise, the patch looks good to me. +1 pending test results. Thank you


> Beeline should not prompt for hive user and password when authentication is 
> NONE
> 
>
> Key: HIVE-22355
> URL: https://issues.apache.org/jira/browse/HIVE-22355
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: HIVE-22355.1.patch, HIVE-22355.2.patch, 
> HIVE-22355.3.patch, HIVE-22355.4.patch
>
>
> Beeline - without adding the jdbc url - prompts for username and password in 
> case hive.server2.authentication=NONE, which is possibly pointless and can be 
> misleading for users as any input is accepted.
> In addition, Sqoop has dropped hive cli recently in favor of beeline and if 
> there is no authentication set in Hive, Sqoop fails to connect as the process 
> stops waiting for the user/password input. 
> I think it would be nice to check the auth type "NONE" before reading unused 
> inputs from the console before this point:
> https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/Commands.java#L1641



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22708) Test fix for http transport

2020-01-08 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011370#comment-17011370
 ] 

Hive QA commented on HIVE-22708:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
42s{color} | {color:blue} service in master has 49 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
50s{color} | {color:red} service generated 1 new + 49 unchanged - 0 fixed = 50 
total (was 49) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 47s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:service |
|  |  Found reliance on default encoding in 
org.apache.hive.service.CookieSigner.verifyAndExtract(String):in 
org.apache.hive.service.CookieSigner.verifyAndExtract(String): 
String.getBytes()  At CookieSigner.java:[line 84] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20123/dev-support/hive-personality.sh
 |
| git revision | master / 706c1d4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-20123/yetus/new-findbugs-service.html
 |
| modules | C: service U: service |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-20123/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Test fix for http transport
> ---
>
> Key: HIVE-22708
> URL: https://issues.apache.org/jira/browse/HIVE-22708
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-22708.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22708) Test fix for http transport

2020-01-08 Thread Naveen Gangam (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-22708:
-
Attachment: HIVE-22708.patch

> Test fix for http transport
> ---
>
> Key: HIVE-22708
> URL: https://issues.apache.org/jira/browse/HIVE-22708
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-22708.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22708) Test fix for http transport

2020-01-08 Thread Naveen Gangam (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-22708:
-
Status: Patch Available  (was: Open)

> Test fix for http transport
> ---
>
> Key: HIVE-22708
> URL: https://issues.apache.org/jira/browse/HIVE-22708
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-22708.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22708) Test fix for http transport

2020-01-08 Thread Naveen Gangam (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-22708:
-
Summary: Test fix for http transport  (was: To be updated later)

> Test fix for http transport
> ---
>
> Key: HIVE-22708
> URL: https://issues.apache.org/jira/browse/HIVE-22708
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22707) MergeJoinWork should be considered while collecting DAG credentials

2020-01-08 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011342#comment-17011342
 ] 

Hive QA commented on HIVE-22707:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12990251/HIVE-22707.02.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 17858 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/20122/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20122/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20122/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12990251 - PreCommit-HIVE-Build

> MergeJoinWork should be considered while collecting DAG credentials
> ---
>
> Key: HIVE-22707
> URL: https://issues.apache.org/jira/browse/HIVE-22707
> Project: Hive
>  Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-22707.01.patch, HIVE-22707.02.patch
>
>
> Given a scenario, when there are 2 different buckets, and the output is 
> written to another bucket than the source. Under specific circumstances, 
> FileSinkOperator is only used in Reducer stages, and if a root work in that 
> stage is a merge join work, it's not scanned for output uris/paths, therefore 
> needed delegation tokens are not fetched for e.g. the output s3 bucket.
> https://github.com/apache/hive/blob/0df4f6c61010b64246d4790f9ce14e966ef34dcb/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L1507-L1514
> {code}
>   public void addCredentials(BaseWork work, DAG dag) throws IOException {
> 
> dag.getCredentials().mergeAll(UserGroupInformation.getCurrentUser().getCredentials());
> if (work instanceof MapWork) {
>   addCredentials((MapWork) work, dag);
> } else if (work instanceof ReduceWork) {
>   addCredentials((ReduceWork) work, dag);
> }
>   }
> {code}
> sample plan, note Merge Join Operator [MERGEJOIN_35]
> {code}
> ++
> |  Explain   |
> ++
> | Plan optimized by CBO. |
> ||
> | Vertex dependency in root stage|
> | Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE) |
> | Reducer 3 <- Reducer 2 (CUSTOM_SIMPLE_EDGE)|
> ||
> | Stage-3|
> |   Stats Work{} |
> | Stage-4|
> |   Create 
> Table{"name:":"tpcds_bin_partitioned_orc_1000.catalog_sales_out"} |
> | Stage-0|
> |   Move Operator|
> | Stage-1|
> |   Reducer 3|
> |   File Output Operator [FS_20] |
> | Group By Operator [GBY_18] (rows=1 width=440) |
> |   
> Output:["_col0"],aggregations:["compute_stats(VALUE._col0)"] |
> | <-Reducer 2 [CUSTOM_SIMPLE_EDGE]   |
> |   File Output Operator [FS_10] |
> | 
> table:{"name:":"tpcds_bin_partitioned_orc_1000.catalog_sales_out"} |
> | Select Operator [SEL_9] (rows=8400 width=7) |
> |   Output:["_col0"] |
> |   Merge Join Operator [MERGEJOIN_35] (rows=8400 
> width=7) |
> | 
> Conds:RS_38._col1=RS_41._col0(Inner),Output:["_col1"] |
> |   <-Map 1 [SIMPLE_EDGE] vectorized |
> | SHUFFLE [RS_38]|
> |   PartitionCols:_col1  |
> |   Select Operator [SEL_37] (rows=16799 width=15) |
> | Output:["_col1"]   |
> | Filter Operator [FIL_36] (rows=16799 width=15) |
> |   predicate:((cs_sold_time_sk = 74858L) and 
> cs_call_center_sk is not null) |
> |   TableScan [TS_0] (rows=1439980416 width=15) |
> | 
> tpcds_bin_partitioned_orc_1000@catalog_sales,cs, ACID 
>

[jira] [Commented] (HIVE-22707) MergeJoinWork should be considered while collecting DAG credentials

2020-01-08 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011303#comment-17011303
 ] 

Hive QA commented on HIVE-22707:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
50s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
45s{color} | {color:red} ql: The patch generated 1 new + 39 unchanged - 5 fixed 
= 40 total (was 44) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 56s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20122/dev-support/hive-personality.sh
 |
| git revision | master / 706c1d4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-20122/yetus/diff-checkstyle-ql.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-20122/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> MergeJoinWork should be considered while collecting DAG credentials
> ---
>
> Key: HIVE-22707
> URL: https://issues.apache.org/jira/browse/HIVE-22707
> Project: Hive
>  Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-22707.01.patch, HIVE-22707.02.patch
>
>
> Given a scenario, when there are 2 different buckets, and the output is 
> written to another bucket than the source. Under specific circumstances, 
> FileSinkOperator is only used in Reducer stages, and if a root work in that 
> stage is a merge join work, it's not scanned for output uris/paths, therefore 
> needed delegation tokens are not fetched for e.g. the output s3 bucket.
> https://github.com/apache/hive/blob/0df4f6c61010b64246d4790f9ce14e966ef34dcb/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L1507-L1514
> {code}
>   public void addCredentials(BaseWork work, DAG dag) throws IOException {
> 
> dag.getCredentials().mergeAll(UserGroupInformation.getCurrentUser().getCredentials());
> if (work instanceof MapWork) {
>   addCredentials((MapWork) work, dag);
> } else if (work instanceof ReduceWork) {
>   addCredentials((ReduceWork) work, dag);
> }
>   }
> {code}
> sample plan, note Merge Join Operator [MERGEJOIN_35]
> {code}
> ++
> |  Explain   |
>

[jira] [Commented] (HIVE-22007) Do not push unsupported types to specific JDBC sources from Calcite

2020-01-08 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011287#comment-17011287
 ] 

Hive QA commented on HIVE-22007:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12990235/HIVE-22007.01.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17858 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.schq.TestScheduledQueryService.testScheduledQueryExecution
 (batchId=355)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/20121/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20121/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20121/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12990235 - PreCommit-HIVE-Build

> Do not push unsupported types to specific JDBC sources from Calcite
> ---
>
> Key: HIVE-22007
> URL: https://issues.apache.org/jira/browse/HIVE-22007
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-22007.01.patch, HIVE-22007.patch
>
>
> We should not push a project expression if it uses a type that a specific 
> dialect does not support, e.g., boolean in Oracle.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22007) Do not push unsupported types to specific JDBC sources from Calcite

2020-01-08 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011096#comment-17011096
 ] 

Hive QA commented on HIVE-22007:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m 
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
43s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
39s{color} | {color:green} ql: The patch generated 0 new + 0 unchanged - 4 
fixed = 0 total (was 4) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m  4s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20121/dev-support/hive-personality.sh
 |
| git revision | master / 706c1d4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-20121/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Do not push unsupported types to specific JDBC sources from Calcite
> ---
>
> Key: HIVE-22007
> URL: https://issues.apache.org/jira/browse/HIVE-22007
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-22007.01.patch, HIVE-22007.patch
>
>
> We should not push a project expression if it uses a type that a specific 
> dialect does not support, e.g., boolean in Oracle.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22256) Rewriting fails when `IN` clause has items in different order in MV and query.

2020-01-08 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011082#comment-17011082
 ] 

Hive QA commented on HIVE-22256:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12990232/HIVE-22256.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 166 failed/errored test(s), 17859 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_part] 
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer8] 
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_join_preds] 
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join34] (batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join35] (batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join45] (batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join46] (batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join47] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_emit_interval] 
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin46] (batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin47] (batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pcs] (batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pointlookup2] 
(batchId=93)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pointlookup3] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pointlookup5] 
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_46] 
(batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_47] 
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_offcbo] 
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_between_columns] 
(batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_join46_mr] 
(batchId=72)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_16]
 (batchId=180)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_7]
 (batchId=180)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cross_prod_1]
 (batchId=185)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[estimate_pkfk_push]
 (batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[external_jdbc_table_perf]
 (batchId=189)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join46] 
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_emit_interval]
 (batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mapjoin46] 
(batchId=181)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mapjoin_emit_interval]
 (batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_ssb]
 (batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_ssb_2]
 (batchId=180)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multi_in_clause]
 (batchId=185)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sharedwork] 
(batchId=185)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_between_columns]
 (batchId=184)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_join46]
 (batchId=177)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketsortoptimize_insert_7]
 (batchId=140)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join34] 
(batchId=148)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join35] 
(batchId=146)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query12] 
(batchId=305)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query13] 
(batchId=305)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query15] 
(batchId=305)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query20] 
(batchId=305)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query21] 
(batchId=305)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query22] 
(batchId=305)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query34] 
(batchId=305)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query37] 
(batchId=305)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query40] 
(batchId=305)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query48] 
(batchId=305)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query58] 
(batchId=305)

[jira] [Commented] (HIVE-22256) Rewriting fails when `IN` clause has items in different order in MV and query.

2020-01-08 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011056#comment-17011056
 ] 

Hive QA commented on HIVE-22256:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  2m  
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
12s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
16s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 32s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20120/dev-support/hive-personality.sh
 |
| git revision | master / 706c1d4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql itests U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-20120/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Rewriting fails when `IN` clause has items in different order in MV and query.
> --
>
> Key: HIVE-22256
> URL: https://issues.apache.org/jira/browse/HIVE-22256
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO, Materialized views
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-22256.patch, expr2.sql
>
>
> Rewriting fails on following materialized view and query (script is also 
> attached):
> create materialized view view2 stored as orc as (select prod_id, cust_id, 
> store_id, sale_date, qty, amt, descr from sales where cust_id in (1,2,3,4,5));
> explain extended select prod_id, cust_id  from sales where cust_id in 
> (5,1,2,3,4);



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22583) LLAP cache always misses with non-vectorized serde readers such as OpenCSV

2020-01-08 Thread Slim Bouguerra (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011048#comment-17011048
 ] 

Slim Bouguerra commented on HIVE-22583:
---

[~szita] I think that might be the same thing, in fact the tez counters depends 
on HDFS counters and that is related to the file format that can change and 
thus the bytes count can change.
Think of it that the byte read or miss by the cache are relative the ORC file 
formats.
As i said i think for now we can avoid this test case that can be flaky and 
work on a query that can run against the cache only, that's more robust IMO. 

> LLAP cache always misses with non-vectorized serde readers such as OpenCSV
> --
>
> Key: HIVE-22583
> URL: https://issues.apache.org/jira/browse/HIVE-22583
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22583.0.patch, HIVE-22583.1.patch, 
> HIVE-22583.2.patch
>
>
> Although after the first read LLAP cache stores data of tables that are not 
> using the LazySimple serde, the stored data is then never used in the future 
> subsequent queries, causing a full cache miss and re-read each time.
> Problem is rooted in SerdeEncodedDataReader#cacheFileData is not taking care 
> of creating an entry for the root/struct column of the table. The only cases 
> this is taken care of are when a vectorized reader is used _(e.g. 
> LazySimpleSerde's LazySimpleDeserializeRead)_, where 
> SerdeEncodedDataReader#processAsyncCacheData takes care of this.
> This can be reproduced by either using a custom serde, like OpenCSV or using 
> LazySimpleSerde, but turning off _hive.llap.io.encode.vector.serde.enabled_.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22707) MergeJoinWork should be considered while collecting DAG credentials

2020-01-08 Thread Ashutosh Chauhan (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011039#comment-17011039
 ] 

Ashutosh Chauhan commented on HIVE-22707:
-

+1 pending tests

> MergeJoinWork should be considered while collecting DAG credentials
> ---
>
> Key: HIVE-22707
> URL: https://issues.apache.org/jira/browse/HIVE-22707
> Project: Hive
>  Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-22707.01.patch, HIVE-22707.02.patch
>
>
> Given a scenario, when there are 2 different buckets, and the output is 
> written to another bucket than the source. Under specific circumstances, 
> FileSinkOperator is only used in Reducer stages, and if a root work in that 
> stage is a merge join work, it's not scanned for output uris/paths, therefore 
> needed delegation tokens are not fetched for e.g. the output s3 bucket.
> https://github.com/apache/hive/blob/0df4f6c61010b64246d4790f9ce14e966ef34dcb/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L1507-L1514
> {code}
>   public void addCredentials(BaseWork work, DAG dag) throws IOException {
> 
> dag.getCredentials().mergeAll(UserGroupInformation.getCurrentUser().getCredentials());
> if (work instanceof MapWork) {
>   addCredentials((MapWork) work, dag);
> } else if (work instanceof ReduceWork) {
>   addCredentials((ReduceWork) work, dag);
> }
>   }
> {code}
> sample plan, note Merge Join Operator [MERGEJOIN_35]
> {code}
> ++
> |  Explain   |
> ++
> | Plan optimized by CBO. |
> ||
> | Vertex dependency in root stage|
> | Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE) |
> | Reducer 3 <- Reducer 2 (CUSTOM_SIMPLE_EDGE)|
> ||
> | Stage-3|
> |   Stats Work{} |
> | Stage-4|
> |   Create 
> Table{"name:":"tpcds_bin_partitioned_orc_1000.catalog_sales_out"} |
> | Stage-0|
> |   Move Operator|
> | Stage-1|
> |   Reducer 3|
> |   File Output Operator [FS_20] |
> | Group By Operator [GBY_18] (rows=1 width=440) |
> |   
> Output:["_col0"],aggregations:["compute_stats(VALUE._col0)"] |
> | <-Reducer 2 [CUSTOM_SIMPLE_EDGE]   |
> |   File Output Operator [FS_10] |
> | 
> table:{"name:":"tpcds_bin_partitioned_orc_1000.catalog_sales_out"} |
> | Select Operator [SEL_9] (rows=8400 width=7) |
> |   Output:["_col0"] |
> |   Merge Join Operator [MERGEJOIN_35] (rows=8400 
> width=7) |
> | 
> Conds:RS_38._col1=RS_41._col0(Inner),Output:["_col1"] |
> |   <-Map 1 [SIMPLE_EDGE] vectorized |
> | SHUFFLE [RS_38]|
> |   PartitionCols:_col1  |
> |   Select Operator [SEL_37] (rows=16799 width=15) |
> | Output:["_col1"]   |
> | Filter Operator [FIL_36] (rows=16799 width=15) |
> |   predicate:((cs_sold_time_sk = 74858L) and 
> cs_call_center_sk is not null) |
> |   TableScan [TS_0] (rows=1439980416 width=15) |
> | 
> tpcds_bin_partitioned_orc_1000@catalog_sales,cs, ACID 
> table,Tbl:COMPLETE,Col:PARTIAL,Output:["cs_sold_time_sk","cs_call_center_sk"] 
> |
> |   <-Map 4 [SIMPLE_EDGE] vectorized |
> | SHUFFLE [RS_41]|
> |   PartitionCols:_col0  |
> |   Select Operator [SEL_40] (rows=21 width=107) |
> | Output:["_col0"]   |
> | Filter Operator [FIL_39] (rows=21 width=107) |
> |   predicate:((CAST( cc_county AS STRING) = 
> 'Williamson County') and cc_call_center_sk is not null) |
> |   TableScan [TS_3] (rows=42 width=107) |
> | 
> tpcds_bin_partitioned_orc_1000@call_center,cc, ACID 
> table,Tbl:COMPLETE,Col:COMPLETE,Output:["cc_call_center_sk","cc_county"] |
> |

[jira] [Commented] (HIVE-22707) MergeJoinWork should be considered while collecting DAG credentials

2020-01-08 Thread Jira



[ 
https://issues.apache.org/jira/browse/HIVE-22707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011034#comment-17011034
 ] 

László Bodor commented on HIVE-22707:
-

thanks [~ashutoshc]
handled in  [^HIVE-22707.02.patch] 
UnionWork.getAllRootOperators returns an empty hashset now, it won't have an 
effect, but at least it will work properly if it changes this behavior

> MergeJoinWork should be considered while collecting DAG credentials
> ---
>
> Key: HIVE-22707
> URL: https://issues.apache.org/jira/browse/HIVE-22707
> Project: Hive
>  Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-22707.01.patch, HIVE-22707.02.patch
>
>
> Given a scenario, when there are 2 different buckets, and the output is 
> written to another bucket than the source. Under specific circumstances, 
> FileSinkOperator is only used in Reducer stages, and if a root work in that 
> stage is a merge join work, it's not scanned for output uris/paths, therefore 
> needed delegation tokens are not fetched for e.g. the output s3 bucket.
> https://github.com/apache/hive/blob/0df4f6c61010b64246d4790f9ce14e966ef34dcb/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L1507-L1514
> {code}
>   public void addCredentials(BaseWork work, DAG dag) throws IOException {
> 
> dag.getCredentials().mergeAll(UserGroupInformation.getCurrentUser().getCredentials());
> if (work instanceof MapWork) {
>   addCredentials((MapWork) work, dag);
> } else if (work instanceof ReduceWork) {
>   addCredentials((ReduceWork) work, dag);
> }
>   }
> {code}
> sample plan, note Merge Join Operator [MERGEJOIN_35]
> {code}
> ++
> |  Explain   |
> ++
> | Plan optimized by CBO. |
> ||
> | Vertex dependency in root stage|
> | Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE) |
> | Reducer 3 <- Reducer 2 (CUSTOM_SIMPLE_EDGE)|
> ||
> | Stage-3|
> |   Stats Work{} |
> | Stage-4|
> |   Create 
> Table{"name:":"tpcds_bin_partitioned_orc_1000.catalog_sales_out"} |
> | Stage-0|
> |   Move Operator|
> | Stage-1|
> |   Reducer 3|
> |   File Output Operator [FS_20] |
> | Group By Operator [GBY_18] (rows=1 width=440) |
> |   
> Output:["_col0"],aggregations:["compute_stats(VALUE._col0)"] |
> | <-Reducer 2 [CUSTOM_SIMPLE_EDGE]   |
> |   File Output Operator [FS_10] |
> | 
> table:{"name:":"tpcds_bin_partitioned_orc_1000.catalog_sales_out"} |
> | Select Operator [SEL_9] (rows=8400 width=7) |
> |   Output:["_col0"] |
> |   Merge Join Operator [MERGEJOIN_35] (rows=8400 
> width=7) |
> | 
> Conds:RS_38._col1=RS_41._col0(Inner),Output:["_col1"] |
> |   <-Map 1 [SIMPLE_EDGE] vectorized |
> | SHUFFLE [RS_38]|
> |   PartitionCols:_col1  |
> |   Select Operator [SEL_37] (rows=16799 width=15) |
> | Output:["_col1"]   |
> | Filter Operator [FIL_36] (rows=16799 width=15) |
> |   predicate:((cs_sold_time_sk = 74858L) and 
> cs_call_center_sk is not null) |
> |   TableScan [TS_0] (rows=1439980416 width=15) |
> | 
> tpcds_bin_partitioned_orc_1000@catalog_sales,cs, ACID 
> table,Tbl:COMPLETE,Col:PARTIAL,Output:["cs_sold_time_sk","cs_call_center_sk"] 
> |
> |   <-Map 4 [SIMPLE_EDGE] vectorized |
> | SHUFFLE [RS_41]|
> |   PartitionCols:_col0  |
> |   Select Operator [SEL_40] (rows=21 width=107) |
> | Output:["_col0"]   |
> | Filter Operator [FIL_39] (rows=21 width=107) |
> |   predicate:((CAST( cc_county AS STRING) = 
> 'Williamson County') and cc_call_center_sk is not null) |
> |   TableScan [TS_3] (rows=42 width=107) |
> |

[jira] [Updated] (HIVE-22707) MergeJoinWork should be considered while collecting DAG credentials

2020-01-08 Thread Jira



 [ 
https://issues.apache.org/jira/browse/HIVE-22707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

László Bodor updated HIVE-22707:

Attachment: HIVE-22707.02.patch

> MergeJoinWork should be considered while collecting DAG credentials
> ---
>
> Key: HIVE-22707
> URL: https://issues.apache.org/jira/browse/HIVE-22707
> Project: Hive
>  Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-22707.01.patch, HIVE-22707.02.patch
>
>
> Given a scenario, when there are 2 different buckets, and the output is 
> written to another bucket than the source. Under specific circumstances, 
> FileSinkOperator is only used in Reducer stages, and if a root work in that 
> stage is a merge join work, it's not scanned for output uris/paths, therefore 
> needed delegation tokens are not fetched for e.g. the output s3 bucket.
> https://github.com/apache/hive/blob/0df4f6c61010b64246d4790f9ce14e966ef34dcb/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L1507-L1514
> {code}
>   public void addCredentials(BaseWork work, DAG dag) throws IOException {
> 
> dag.getCredentials().mergeAll(UserGroupInformation.getCurrentUser().getCredentials());
> if (work instanceof MapWork) {
>   addCredentials((MapWork) work, dag);
> } else if (work instanceof ReduceWork) {
>   addCredentials((ReduceWork) work, dag);
> }
>   }
> {code}
> sample plan, note Merge Join Operator [MERGEJOIN_35]
> {code}
> ++
> |  Explain   |
> ++
> | Plan optimized by CBO. |
> ||
> | Vertex dependency in root stage|
> | Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE) |
> | Reducer 3 <- Reducer 2 (CUSTOM_SIMPLE_EDGE)|
> ||
> | Stage-3|
> |   Stats Work{} |
> | Stage-4|
> |   Create 
> Table{"name:":"tpcds_bin_partitioned_orc_1000.catalog_sales_out"} |
> | Stage-0|
> |   Move Operator|
> | Stage-1|
> |   Reducer 3|
> |   File Output Operator [FS_20] |
> | Group By Operator [GBY_18] (rows=1 width=440) |
> |   
> Output:["_col0"],aggregations:["compute_stats(VALUE._col0)"] |
> | <-Reducer 2 [CUSTOM_SIMPLE_EDGE]   |
> |   File Output Operator [FS_10] |
> | 
> table:{"name:":"tpcds_bin_partitioned_orc_1000.catalog_sales_out"} |
> | Select Operator [SEL_9] (rows=8400 width=7) |
> |   Output:["_col0"] |
> |   Merge Join Operator [MERGEJOIN_35] (rows=8400 
> width=7) |
> | 
> Conds:RS_38._col1=RS_41._col0(Inner),Output:["_col1"] |
> |   <-Map 1 [SIMPLE_EDGE] vectorized |
> | SHUFFLE [RS_38]|
> |   PartitionCols:_col1  |
> |   Select Operator [SEL_37] (rows=16799 width=15) |
> | Output:["_col1"]   |
> | Filter Operator [FIL_36] (rows=16799 width=15) |
> |   predicate:((cs_sold_time_sk = 74858L) and 
> cs_call_center_sk is not null) |
> |   TableScan [TS_0] (rows=1439980416 width=15) |
> | 
> tpcds_bin_partitioned_orc_1000@catalog_sales,cs, ACID 
> table,Tbl:COMPLETE,Col:PARTIAL,Output:["cs_sold_time_sk","cs_call_center_sk"] 
> |
> |   <-Map 4 [SIMPLE_EDGE] vectorized |
> | SHUFFLE [RS_41]|
> |   PartitionCols:_col0  |
> |   Select Operator [SEL_40] (rows=21 width=107) |
> | Output:["_col0"]   |
> | Filter Operator [FIL_39] (rows=21 width=107) |
> |   predicate:((CAST( cc_county AS STRING) = 
> 'Williamson County') and cc_call_center_sk is not null) |
> |   TableScan [TS_3] (rows=42 width=107) |
> | 
> tpcds_bin_partitioned_orc_1000@call_center,cc, ACID 
> table,Tbl:COMPLETE,Col:COMPLETE,Output:["cc_call_center_sk","cc_county"] |
> |   PARTITION_ONLY_SHUFFLE [RS_17]   |

[jira] [Comment Edited] (HIVE-22653) Remove commons-lang leftovers

2020-01-08 Thread Jesus Camacho Rodriguez (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011033#comment-17011033
 ] 

Jesus Camacho Rodriguez edited comment on HIVE-22653 at 1/8/20 10:09 PM:
-

It seems this commit broke the build.

https://builds.apache.org/job/PreCommit-HIVE-Build/20118/consoleText

I am reverting and reopening this issue.


was (Author: jcamachorodriguez):
It seems this commit broke the build.

https://builds.apache.org/job/PreCommit-HIVE-Build/20118/consoleText

I am going to revert, please feel free to recommit if I am wrong.

> Remove commons-lang leftovers
> -
>
> Key: HIVE-22653
> URL: https://issues.apache.org/jira/browse/HIVE-22653
> Project: Hive
>  Issue Type: Bug
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22653.01.patch, HIVE-22653.01.patch, 
> HIVE-22653.02.patch, HIVE-22653.03.patch
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> HIVE-7145 removed commons-lang - in favor of commons-lang3 - as a direct 
> dependency, however a high number of files still refer to commons-lang, which 
> is transitively brought in either way.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (HIVE-22653) Remove commons-lang leftovers

2020-01-08 Thread Jesus Camacho Rodriguez (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez reopened HIVE-22653:


> Remove commons-lang leftovers
> -
>
> Key: HIVE-22653
> URL: https://issues.apache.org/jira/browse/HIVE-22653
> Project: Hive
>  Issue Type: Bug
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22653.01.patch, HIVE-22653.01.patch, 
> HIVE-22653.02.patch, HIVE-22653.03.patch
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> HIVE-7145 removed commons-lang - in favor of commons-lang3 - as a direct 
> dependency, however a high number of files still refer to commons-lang, which 
> is transitively brought in either way.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22653) Remove commons-lang leftovers

2020-01-08 Thread Jesus Camacho Rodriguez (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011033#comment-17011033
 ] 

Jesus Camacho Rodriguez commented on HIVE-22653:


It seems this commit broke the build.

https://builds.apache.org/job/PreCommit-HIVE-Build/20118/consoleText

I am going to revert, please feel free to recommit if I am wrong.

> Remove commons-lang leftovers
> -
>
> Key: HIVE-22653
> URL: https://issues.apache.org/jira/browse/HIVE-22653
> Project: Hive
>  Issue Type: Bug
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22653.01.patch, HIVE-22653.01.patch, 
> HIVE-22653.02.patch, HIVE-22653.03.patch
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> HIVE-7145 removed commons-lang - in favor of commons-lang3 - as a direct 
> dependency, however a high number of files still refer to commons-lang, which 
> is transitively brought in either way.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22709) NullPointerException during query compilation after HIVE-22578

2020-01-08 Thread Jason Dere (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-22709:
--
Attachment: HIVE-22709.1.patch

> NullPointerException during query compilation after HIVE-22578
> --
>
> Key: HIVE-22709
> URL: https://issues.apache.org/jira/browse/HIVE-22709
> Project: Hive
>  Issue Type: Bug
>Reporter: Jason Dere
>Priority: Major
> Attachments: HIVE-22709.1.patch, results_cache_with_auth.q
>
>
> Getting a NPE during query compilation, when query results cache and Ranger 
> auth is enabled. This seems to have been caused by HIVE-22578.
> {noformat}
>  java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getQueryStringFromAst(SemanticAnalyzer.java:14987)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getQueryStringForCache(SemanticAnalyzer.java:15036)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.createLookupInfoForQuery(SemanticAnalyzer.java:15077)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12513)
>   at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:358)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:283)
>   at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:171)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:283)
>   at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:219)
>   at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:103)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:215)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:828)
>   at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:774)
>   at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:768)
>   at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125)
>   at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:249)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:193)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:415)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:346)
>   at 
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:708)
>   at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:678)
>   at 
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:169)
>   at 
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
>   at 
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:59)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22709) NullPointerException during query compilation after HIVE-22578

2020-01-08 Thread Jason Dere (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-22709:
--
Assignee: Jason Dere
  Status: Patch Available  (was: Open)

Attaching patch to fix this - changing "ast" to "astForAnalyze" within 
SemanticAnalyzer.analyzeInternal(). Test case also added.

[~hamvas.aron], [~kgyrtkirk] can you review?

> NullPointerException during query compilation after HIVE-22578
> --
>
> Key: HIVE-22709
> URL: https://issues.apache.org/jira/browse/HIVE-22709
> Project: Hive
>  Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-22709.1.patch, results_cache_with_auth.q
>
>
> Getting a NPE during query compilation, when query results cache and Ranger 
> auth is enabled. This seems to have been caused by HIVE-22578.
> {noformat}
>  java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getQueryStringFromAst(SemanticAnalyzer.java:14987)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getQueryStringForCache(SemanticAnalyzer.java:15036)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.createLookupInfoForQuery(SemanticAnalyzer.java:15077)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12513)
>   at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:358)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:283)
>   at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:171)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:283)
>   at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:219)
>   at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:103)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:215)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:828)
>   at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:774)
>   at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:768)
>   at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125)
>   at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:249)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:193)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:415)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:346)
>   at 
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:708)
>   at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:678)
>   at 
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:169)
>   at 
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
>   at 
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:59)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22707) MergeJoinWork should be considered while collecting DAG credentials

2020-01-08 Thread Ashutosh Chauhan (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011016#comment-17011016
 ] 

Ashutosh Chauhan commented on HIVE-22707:
-

There is also {{UnionWork}} Seems like thats not handled either. That also 
needs to be visited and collected uris for.

> MergeJoinWork should be considered while collecting DAG credentials
> ---
>
> Key: HIVE-22707
> URL: https://issues.apache.org/jira/browse/HIVE-22707
> Project: Hive
>  Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-22707.01.patch
>
>
> Given a scenario, when there are 2 different buckets, and the output is 
> written to another bucket than the source. Under specific circumstances, 
> FileSinkOperator is only used in Reducer stages, and if a root work in that 
> stage is a merge join work, it's not scanned for output uris/paths, therefore 
> needed delegation tokens are not fetched for e.g. the output s3 bucket.
> https://github.com/apache/hive/blob/0df4f6c61010b64246d4790f9ce14e966ef34dcb/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L1507-L1514
> {code}
>   public void addCredentials(BaseWork work, DAG dag) throws IOException {
> 
> dag.getCredentials().mergeAll(UserGroupInformation.getCurrentUser().getCredentials());
> if (work instanceof MapWork) {
>   addCredentials((MapWork) work, dag);
> } else if (work instanceof ReduceWork) {
>   addCredentials((ReduceWork) work, dag);
> }
>   }
> {code}
> sample plan, note Merge Join Operator [MERGEJOIN_35]
> {code}
> ++
> |  Explain   |
> ++
> | Plan optimized by CBO. |
> ||
> | Vertex dependency in root stage|
> | Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE) |
> | Reducer 3 <- Reducer 2 (CUSTOM_SIMPLE_EDGE)|
> ||
> | Stage-3|
> |   Stats Work{} |
> | Stage-4|
> |   Create 
> Table{"name:":"tpcds_bin_partitioned_orc_1000.catalog_sales_out"} |
> | Stage-0|
> |   Move Operator|
> | Stage-1|
> |   Reducer 3|
> |   File Output Operator [FS_20] |
> | Group By Operator [GBY_18] (rows=1 width=440) |
> |   
> Output:["_col0"],aggregations:["compute_stats(VALUE._col0)"] |
> | <-Reducer 2 [CUSTOM_SIMPLE_EDGE]   |
> |   File Output Operator [FS_10] |
> | 
> table:{"name:":"tpcds_bin_partitioned_orc_1000.catalog_sales_out"} |
> | Select Operator [SEL_9] (rows=8400 width=7) |
> |   Output:["_col0"] |
> |   Merge Join Operator [MERGEJOIN_35] (rows=8400 
> width=7) |
> | 
> Conds:RS_38._col1=RS_41._col0(Inner),Output:["_col1"] |
> |   <-Map 1 [SIMPLE_EDGE] vectorized |
> | SHUFFLE [RS_38]|
> |   PartitionCols:_col1  |
> |   Select Operator [SEL_37] (rows=16799 width=15) |
> | Output:["_col1"]   |
> | Filter Operator [FIL_36] (rows=16799 width=15) |
> |   predicate:((cs_sold_time_sk = 74858L) and 
> cs_call_center_sk is not null) |
> |   TableScan [TS_0] (rows=1439980416 width=15) |
> | 
> tpcds_bin_partitioned_orc_1000@catalog_sales,cs, ACID 
> table,Tbl:COMPLETE,Col:PARTIAL,Output:["cs_sold_time_sk","cs_call_center_sk"] 
> |
> |   <-Map 4 [SIMPLE_EDGE] vectorized |
> | SHUFFLE [RS_41]|
> |   PartitionCols:_col0  |
> |   Select Operator [SEL_40] (rows=21 width=107) |
> | Output:["_col0"]   |
> | Filter Operator [FIL_39] (rows=21 width=107) |
> |   predicate:((CAST( cc_county AS STRING) = 
> 'Williamson County') and cc_call_center_sk is not null) |
> |   TableScan [TS_3] (rows=42 width=107) |
> | 
> tpcds_bin_partitioned_orc_1000@call_center,cc, ACID 
>

[jira] [Commented] (HIVE-22709) NullPointerException during query compilation after HIVE-22578

2020-01-08 Thread Jason Dere (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011011#comment-17011011
 ] 

Jason Dere commented on HIVE-22709:
---

I think I see what happened - when the method signature of 
SemanticAnalyzer.analyzeInternal was changed from 
{code}
  void analyzeInternal(ASTNode ast, Supplier pcf) throws 
SemanticException {
{code}
to
{code}
 void analyzeInternal(ASTNode astToAnalyze, Supplier pcf) 
throws SemanticException {
{code}

then in this line:
{code}
lookupInfo = createLookupInfoForQuery(ast);
{code}
"ast" changed from being a method parameter to the SemanticAnalyzer.ast field. 
We just need to change that line to lookupInfo = 
createLookupInfoForQuery(astToAnalyze);
In fact all references to "ast" in that method will need to be changed to 
astToAnalyze since the parameter was renamed, it is unfortunate that 
SemanticAnalyzer had an ast field which hid this error.

> NullPointerException during query compilation after HIVE-22578
> --
>
> Key: HIVE-22709
> URL: https://issues.apache.org/jira/browse/HIVE-22709
> Project: Hive
>  Issue Type: Bug
>Reporter: Jason Dere
>Priority: Major
> Attachments: results_cache_with_auth.q
>
>
> Getting a NPE during query compilation, when query results cache and Ranger 
> auth is enabled. This seems to have been caused by HIVE-22578.
> {noformat}
>  java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getQueryStringFromAst(SemanticAnalyzer.java:14987)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getQueryStringForCache(SemanticAnalyzer.java:15036)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.createLookupInfoForQuery(SemanticAnalyzer.java:15077)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12513)
>   at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:358)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:283)
>   at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:171)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:283)
>   at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:219)
>   at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:103)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:215)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:828)
>   at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:774)
>   at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:768)
>   at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125)
>   at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:249)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:193)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:415)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:346)
>   at 
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:708)
>   at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:678)
>   at 
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:169)
>   at 
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
>   at 
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:59)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22709) NullPointerException during query compilation after HIVE-22578

2020-01-08 Thread Jason Dere (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010964#comment-17010964
 ] 

Jason Dere commented on HIVE-22709:
---

Attaching a qfile test which reproduces the issue. It passes if I revert the 
changes from HIVE-22578.

> NullPointerException during query compilation after HIVE-22578
> --
>
> Key: HIVE-22709
> URL: https://issues.apache.org/jira/browse/HIVE-22709
> Project: Hive
>  Issue Type: Bug
>Reporter: Jason Dere
>Priority: Major
> Attachments: results_cache_with_auth.q
>
>
> Getting a NPE during query compilation, when query results cache and Ranger 
> auth is enabled. This seems to have been caused by HIVE-22578.
> {noformat}
>  java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getQueryStringFromAst(SemanticAnalyzer.java:14987)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getQueryStringForCache(SemanticAnalyzer.java:15036)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.createLookupInfoForQuery(SemanticAnalyzer.java:15077)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12513)
>   at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:358)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:283)
>   at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:171)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:283)
>   at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:219)
>   at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:103)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:215)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:828)
>   at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:774)
>   at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:768)
>   at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125)
>   at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:249)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:193)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:415)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:346)
>   at 
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:708)
>   at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:678)
>   at 
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:169)
>   at 
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
>   at 
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:59)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22709) NullPointerException during query compilation after HIVE-22578

2020-01-08 Thread Jason Dere (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-22709:
--
Attachment: results_cache_with_auth.q

> NullPointerException during query compilation after HIVE-22578
> --
>
> Key: HIVE-22709
> URL: https://issues.apache.org/jira/browse/HIVE-22709
> Project: Hive
>  Issue Type: Bug
>Reporter: Jason Dere
>Priority: Major
> Attachments: results_cache_with_auth.q
>
>
> Getting a NPE during query compilation, when query results cache and Ranger 
> auth is enabled. This seems to have been caused by HIVE-22578.
> {noformat}
>  java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getQueryStringFromAst(SemanticAnalyzer.java:14987)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getQueryStringForCache(SemanticAnalyzer.java:15036)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.createLookupInfoForQuery(SemanticAnalyzer.java:15077)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12513)
>   at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:358)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:283)
>   at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:171)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:283)
>   at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:219)
>   at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:103)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:215)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:828)
>   at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:774)
>   at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:768)
>   at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125)
>   at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:249)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:193)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:415)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:346)
>   at 
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:708)
>   at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:678)
>   at 
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:169)
>   at 
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
>   at 
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:59)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22707) MergeJoinWork should be considered while collecting DAG credentials

2020-01-08 Thread Jira



 [ 
https://issues.apache.org/jira/browse/HIVE-22707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

László Bodor updated HIVE-22707:

Attachment: HIVE-22707.01.patch

> MergeJoinWork should be considered while collecting DAG credentials
> ---
>
> Key: HIVE-22707
> URL: https://issues.apache.org/jira/browse/HIVE-22707
> Project: Hive
>  Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-22707.01.patch
>
>
> Given a scenario, when there are 2 different buckets, and the output is 
> written to another bucket than the source. Under specific circumstances, 
> FileSinkOperator is only used in Reducer stages, and if a root work in that 
> stage is a merge join work, it's not scanned for output uris/paths, therefore 
> needed delegation tokens are not fetched for e.g. the output s3 bucket.
> https://github.com/apache/hive/blob/0df4f6c61010b64246d4790f9ce14e966ef34dcb/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L1507-L1514
> {code}
>   public void addCredentials(BaseWork work, DAG dag) throws IOException {
> 
> dag.getCredentials().mergeAll(UserGroupInformation.getCurrentUser().getCredentials());
> if (work instanceof MapWork) {
>   addCredentials((MapWork) work, dag);
> } else if (work instanceof ReduceWork) {
>   addCredentials((ReduceWork) work, dag);
> }
>   }
> {code}
> sample plan, not Merge Join Operator [MERGEJOIN_35]
> {code}
> ++
> |  Explain   |
> ++
> | Plan optimized by CBO. |
> ||
> | Vertex dependency in root stage|
> | Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE) |
> | Reducer 3 <- Reducer 2 (CUSTOM_SIMPLE_EDGE)|
> ||
> | Stage-3|
> |   Stats Work{} |
> | Stage-4|
> |   Create 
> Table{"name:":"tpcds_bin_partitioned_orc_1000.catalog_sales_out"} |
> | Stage-0|
> |   Move Operator|
> | Stage-1|
> |   Reducer 3|
> |   File Output Operator [FS_20] |
> | Group By Operator [GBY_18] (rows=1 width=440) |
> |   
> Output:["_col0"],aggregations:["compute_stats(VALUE._col0)"] |
> | <-Reducer 2 [CUSTOM_SIMPLE_EDGE]   |
> |   File Output Operator [FS_10] |
> | 
> table:{"name:":"tpcds_bin_partitioned_orc_1000.catalog_sales_out"} |
> | Select Operator [SEL_9] (rows=8400 width=7) |
> |   Output:["_col0"] |
> |   Merge Join Operator [MERGEJOIN_35] (rows=8400 
> width=7) |
> | 
> Conds:RS_38._col1=RS_41._col0(Inner),Output:["_col1"] |
> |   <-Map 1 [SIMPLE_EDGE] vectorized |
> | SHUFFLE [RS_38]|
> |   PartitionCols:_col1  |
> |   Select Operator [SEL_37] (rows=16799 width=15) |
> | Output:["_col1"]   |
> | Filter Operator [FIL_36] (rows=16799 width=15) |
> |   predicate:((cs_sold_time_sk = 74858L) and 
> cs_call_center_sk is not null) |
> |   TableScan [TS_0] (rows=1439980416 width=15) |
> | 
> tpcds_bin_partitioned_orc_1000@catalog_sales,cs, ACID 
> table,Tbl:COMPLETE,Col:PARTIAL,Output:["cs_sold_time_sk","cs_call_center_sk"] 
> |
> |   <-Map 4 [SIMPLE_EDGE] vectorized |
> | SHUFFLE [RS_41]|
> |   PartitionCols:_col0  |
> |   Select Operator [SEL_40] (rows=21 width=107) |
> | Output:["_col0"]   |
> | Filter Operator [FIL_39] (rows=21 width=107) |
> |   predicate:((CAST( cc_county AS STRING) = 
> 'Williamson County') and cc_call_center_sk is not null) |
> |   TableScan [TS_3] (rows=42 width=107) |
> | 
> tpcds_bin_partitioned_orc_1000@call_center,cc, ACID 
> table,Tbl:COMPLETE,Col:COMPLETE,Output:["cc_call_center_sk","cc_county"] |
> |   PARTITION_ONLY_SHUFFLE [RS_17]   |
> |

[jira] [Updated] (HIVE-22707) MergeJoinWork should be considered while collecting DAG credentials

2020-01-08 Thread Jira



 [ 
https://issues.apache.org/jira/browse/HIVE-22707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

László Bodor updated HIVE-22707:

Description: 
Given a scenario, when there are 2 different buckets, and the output is written 
to another bucket than the source. Under specific circumstances, 
FileSinkOperator is only used in Reducer stages, and if a root work in that 
stage is a merge join work, it's not scanned for output uris/paths, therefore 
needed delegation tokens are not fetched for e.g. the output s3 bucket.

https://github.com/apache/hive/blob/0df4f6c61010b64246d4790f9ce14e966ef34dcb/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L1507-L1514
{code}
  public void addCredentials(BaseWork work, DAG dag) throws IOException {

dag.getCredentials().mergeAll(UserGroupInformation.getCurrentUser().getCredentials());
if (work instanceof MapWork) {
  addCredentials((MapWork) work, dag);
} else if (work instanceof ReduceWork) {
  addCredentials((ReduceWork) work, dag);
}
  }
{code}

sample plan, note Merge Join Operator [MERGEJOIN_35]
{code}
++
|  Explain   |
++
| Plan optimized by CBO. |
||
| Vertex dependency in root stage|
| Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE) |
| Reducer 3 <- Reducer 2 (CUSTOM_SIMPLE_EDGE)|
||
| Stage-3|
|   Stats Work{} |
| Stage-4|
|   Create 
Table{"name:":"tpcds_bin_partitioned_orc_1000.catalog_sales_out"} |
| Stage-0|
|   Move Operator|
| Stage-1|
|   Reducer 3|
|   File Output Operator [FS_20] |
| Group By Operator [GBY_18] (rows=1 width=440) |
|   
Output:["_col0"],aggregations:["compute_stats(VALUE._col0)"] |
| <-Reducer 2 [CUSTOM_SIMPLE_EDGE]   |
|   File Output Operator [FS_10] |
| 
table:{"name:":"tpcds_bin_partitioned_orc_1000.catalog_sales_out"} |
| Select Operator [SEL_9] (rows=8400 width=7) |
|   Output:["_col0"] |
|   Merge Join Operator [MERGEJOIN_35] (rows=8400 width=7) |
| Conds:RS_38._col1=RS_41._col0(Inner),Output:["_col1"] 
|
|   <-Map 1 [SIMPLE_EDGE] vectorized |
| SHUFFLE [RS_38]|
|   PartitionCols:_col1  |
|   Select Operator [SEL_37] (rows=16799 width=15) |
| Output:["_col1"]   |
| Filter Operator [FIL_36] (rows=16799 width=15) |
|   predicate:((cs_sold_time_sk = 74858L) and 
cs_call_center_sk is not null) |
|   TableScan [TS_0] (rows=1439980416 width=15) |
| 
tpcds_bin_partitioned_orc_1000@catalog_sales,cs, ACID 
table,Tbl:COMPLETE,Col:PARTIAL,Output:["cs_sold_time_sk","cs_call_center_sk"] |
|   <-Map 4 [SIMPLE_EDGE] vectorized |
| SHUFFLE [RS_41]|
|   PartitionCols:_col0  |
|   Select Operator [SEL_40] (rows=21 width=107) |
| Output:["_col0"]   |
| Filter Operator [FIL_39] (rows=21 width=107) |
|   predicate:((CAST( cc_county AS STRING) = 
'Williamson County') and cc_call_center_sk is not null) |
|   TableScan [TS_3] (rows=42 width=107) |
| 
tpcds_bin_partitioned_orc_1000@call_center,cc, ACID 
table,Tbl:COMPLETE,Col:COMPLETE,Output:["cc_call_center_sk","cc_county"] |
|   PARTITION_ONLY_SHUFFLE [RS_17]   |
| Group By Operator [GBY_16] (rows=1 width=424) |
|   Output:["_col0"],aggregations:["compute_stats(col1, 
'hll')"] |
|   Select Operator [SEL_15] (rows=8400 width=7) |
| Output:["col1"]|
|  Please refer to the previous Select Operator [SEL_9] 
|
| Stage-2|
|   Dependency Collection{}  |
|  Please refer to the previous Stage-1  |
||
++

[jira] [Updated] (HIVE-22707) MergeJoinWork should be considered while collecting DAG credentials

2020-01-08 Thread Jira



 [ 
https://issues.apache.org/jira/browse/HIVE-22707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

László Bodor updated HIVE-22707:

Status: Patch Available  (was: Open)

> MergeJoinWork should be considered while collecting DAG credentials
> ---
>
> Key: HIVE-22707
> URL: https://issues.apache.org/jira/browse/HIVE-22707
> Project: Hive
>  Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-22707.01.patch
>
>
> Given a scenario, when there are 2 different buckets, and the output is 
> written to another bucket than the source. Under specific circumstances, 
> FileSinkOperator is only used in Reducer stages, and if a root work in that 
> stage is a merge join work, it's not scanned for output uris/paths, therefore 
> needed delegation tokens are not fetched for e.g. the output s3 bucket.
> https://github.com/apache/hive/blob/0df4f6c61010b64246d4790f9ce14e966ef34dcb/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L1507-L1514
> {code}
>   public void addCredentials(BaseWork work, DAG dag) throws IOException {
> 
> dag.getCredentials().mergeAll(UserGroupInformation.getCurrentUser().getCredentials());
> if (work instanceof MapWork) {
>   addCredentials((MapWork) work, dag);
> } else if (work instanceof ReduceWork) {
>   addCredentials((ReduceWork) work, dag);
> }
>   }
> {code}
> sample plan, not Merge Join Operator [MERGEJOIN_35]
> {code}
> ++
> |  Explain   |
> ++
> | Plan optimized by CBO. |
> ||
> | Vertex dependency in root stage|
> | Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE) |
> | Reducer 3 <- Reducer 2 (CUSTOM_SIMPLE_EDGE)|
> ||
> | Stage-3|
> |   Stats Work{} |
> | Stage-4|
> |   Create 
> Table{"name:":"tpcds_bin_partitioned_orc_1000.catalog_sales_out"} |
> | Stage-0|
> |   Move Operator|
> | Stage-1|
> |   Reducer 3|
> |   File Output Operator [FS_20] |
> | Group By Operator [GBY_18] (rows=1 width=440) |
> |   
> Output:["_col0"],aggregations:["compute_stats(VALUE._col0)"] |
> | <-Reducer 2 [CUSTOM_SIMPLE_EDGE]   |
> |   File Output Operator [FS_10] |
> | 
> table:{"name:":"tpcds_bin_partitioned_orc_1000.catalog_sales_out"} |
> | Select Operator [SEL_9] (rows=8400 width=7) |
> |   Output:["_col0"] |
> |   Merge Join Operator [MERGEJOIN_35] (rows=8400 
> width=7) |
> | 
> Conds:RS_38._col1=RS_41._col0(Inner),Output:["_col1"] |
> |   <-Map 1 [SIMPLE_EDGE] vectorized |
> | SHUFFLE [RS_38]|
> |   PartitionCols:_col1  |
> |   Select Operator [SEL_37] (rows=16799 width=15) |
> | Output:["_col1"]   |
> | Filter Operator [FIL_36] (rows=16799 width=15) |
> |   predicate:((cs_sold_time_sk = 74858L) and 
> cs_call_center_sk is not null) |
> |   TableScan [TS_0] (rows=1439980416 width=15) |
> | 
> tpcds_bin_partitioned_orc_1000@catalog_sales,cs, ACID 
> table,Tbl:COMPLETE,Col:PARTIAL,Output:["cs_sold_time_sk","cs_call_center_sk"] 
> |
> |   <-Map 4 [SIMPLE_EDGE] vectorized |
> | SHUFFLE [RS_41]|
> |   PartitionCols:_col0  |
> |   Select Operator [SEL_40] (rows=21 width=107) |
> | Output:["_col0"]   |
> | Filter Operator [FIL_39] (rows=21 width=107) |
> |   predicate:((CAST( cc_county AS STRING) = 
> 'Williamson County') and cc_call_center_sk is not null) |
> |   TableScan [TS_3] (rows=42 width=107) |
> | 
> tpcds_bin_partitioned_orc_1000@call_center,cc, ACID 
> table,Tbl:COMPLETE,Col:COMPLETE,Output:["cc_call_center_sk","cc_county"] |
> |   PARTITION_ONLY_SHUFFLE [RS_17]   |
> |

[jira] [Commented] (HIVE-22707) MergeJoinWork should be considered while collecting DAG credentials

2020-01-08 Thread Jira



[ 
https://issues.apache.org/jira/browse/HIVE-22707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010922#comment-17010922
 ] 

László Bodor commented on HIVE-22707:
-

cc: [~ashutoshc], if you can take a look, thanks

> MergeJoinWork should be considered while collecting DAG credentials
> ---
>
> Key: HIVE-22707
> URL: https://issues.apache.org/jira/browse/HIVE-22707
> Project: Hive
>  Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-22707.01.patch
>
>
> Given a scenario, when there are 2 different buckets, and the output is 
> written to another bucket than the source. Under specific circumstances, 
> FileSinkOperator is only used in Reducer stages, and if a root work in that 
> stage is a merge join work, it's not scanned for output uris/paths, therefore 
> needed delegation tokens are not fetched for e.g. the output s3 bucket.
> https://github.com/apache/hive/blob/0df4f6c61010b64246d4790f9ce14e966ef34dcb/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L1507-L1514
> {code}
>   public void addCredentials(BaseWork work, DAG dag) throws IOException {
> 
> dag.getCredentials().mergeAll(UserGroupInformation.getCurrentUser().getCredentials());
> if (work instanceof MapWork) {
>   addCredentials((MapWork) work, dag);
> } else if (work instanceof ReduceWork) {
>   addCredentials((ReduceWork) work, dag);
> }
>   }
> {code}
> sample plan, not Merge Join Operator [MERGEJOIN_35]
> {code}
> ++
> |  Explain   |
> ++
> | Plan optimized by CBO. |
> ||
> | Vertex dependency in root stage|
> | Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE) |
> | Reducer 3 <- Reducer 2 (CUSTOM_SIMPLE_EDGE)|
> ||
> | Stage-3|
> |   Stats Work{} |
> | Stage-4|
> |   Create 
> Table{"name:":"tpcds_bin_partitioned_orc_1000.catalog_sales_out"} |
> | Stage-0|
> |   Move Operator|
> | Stage-1|
> |   Reducer 3|
> |   File Output Operator [FS_20] |
> | Group By Operator [GBY_18] (rows=1 width=440) |
> |   
> Output:["_col0"],aggregations:["compute_stats(VALUE._col0)"] |
> | <-Reducer 2 [CUSTOM_SIMPLE_EDGE]   |
> |   File Output Operator [FS_10] |
> | 
> table:{"name:":"tpcds_bin_partitioned_orc_1000.catalog_sales_out"} |
> | Select Operator [SEL_9] (rows=8400 width=7) |
> |   Output:["_col0"] |
> |   Merge Join Operator [MERGEJOIN_35] (rows=8400 
> width=7) |
> | 
> Conds:RS_38._col1=RS_41._col0(Inner),Output:["_col1"] |
> |   <-Map 1 [SIMPLE_EDGE] vectorized |
> | SHUFFLE [RS_38]|
> |   PartitionCols:_col1  |
> |   Select Operator [SEL_37] (rows=16799 width=15) |
> | Output:["_col1"]   |
> | Filter Operator [FIL_36] (rows=16799 width=15) |
> |   predicate:((cs_sold_time_sk = 74858L) and 
> cs_call_center_sk is not null) |
> |   TableScan [TS_0] (rows=1439980416 width=15) |
> | 
> tpcds_bin_partitioned_orc_1000@catalog_sales,cs, ACID 
> table,Tbl:COMPLETE,Col:PARTIAL,Output:["cs_sold_time_sk","cs_call_center_sk"] 
> |
> |   <-Map 4 [SIMPLE_EDGE] vectorized |
> | SHUFFLE [RS_41]|
> |   PartitionCols:_col0  |
> |   Select Operator [SEL_40] (rows=21 width=107) |
> | Output:["_col0"]   |
> | Filter Operator [FIL_39] (rows=21 width=107) |
> |   predicate:((CAST( cc_county AS STRING) = 
> 'Williamson County') and cc_call_center_sk is not null) |
> |   TableScan [TS_3] (rows=42 width=107) |
> | 
> tpcds_bin_partitioned_orc_1000@call_center,cc, ACID 
> table,Tbl:COMPLETE,Col:COMPLETE,Output:["cc_call_center_sk","cc_county"] |
> |

[jira] [Assigned] (HIVE-22708) To be updated later

2020-01-08 Thread Naveen Gangam (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam reassigned HIVE-22708:



> To be updated later
> ---
>
> Key: HIVE-22708
> URL: https://issues.apache.org/jira/browse/HIVE-22708
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22007) Do not push unsupported types to specific JDBC sources from Calcite

2020-01-08 Thread Jesus Camacho Rodriguez (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-22007:
---
Attachment: (was: HIVE-22007.01.patch)

> Do not push unsupported types to specific JDBC sources from Calcite
> ---
>
> Key: HIVE-22007
> URL: https://issues.apache.org/jira/browse/HIVE-22007
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-22007.01.patch, HIVE-22007.patch
>
>
> We should not push a project expression if it uses a type that a specific 
> dialect does not support, e.g., boolean in Oracle.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22007) Do not push unsupported types to specific JDBC sources from Calcite

2020-01-08 Thread Jesus Camacho Rodriguez (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-22007:
---
Attachment: HIVE-22007.01.patch

> Do not push unsupported types to specific JDBC sources from Calcite
> ---
>
> Key: HIVE-22007
> URL: https://issues.apache.org/jira/browse/HIVE-22007
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-22007.01.patch, HIVE-22007.patch
>
>
> We should not push a project expression if it uses a type that a specific 
> dialect does not support, e.g., boolean in Oracle.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22007) Do not push unsupported types to specific JDBC sources from Calcite

2020-01-08 Thread Jesus Camacho Rodriguez (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-22007:
---
Attachment: HIVE-22007.01.patch

> Do not push unsupported types to specific JDBC sources from Calcite
> ---
>
> Key: HIVE-22007
> URL: https://issues.apache.org/jira/browse/HIVE-22007
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-22007.01.patch, HIVE-22007.patch
>
>
> We should not push a project expression if it uses a type that a specific 
> dialect does not support, e.g., boolean in Oracle.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22007) Do not push unsupported types to specific JDBC sources from Calcite

2020-01-08 Thread Jesus Camacho Rodriguez (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-22007:
---
Status: Patch Available  (was: In Progress)

> Do not push unsupported types to specific JDBC sources from Calcite
> ---
>
> Key: HIVE-22007
> URL: https://issues.apache.org/jira/browse/HIVE-22007
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-22007.01.patch, HIVE-22007.patch
>
>
> We should not push a project expression if it uses a type that a specific 
> dialect does not support, e.g., boolean in Oracle.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22256) Rewriting fails when `IN` clause has items in different order in MV and query.

2020-01-08 Thread Jesus Camacho Rodriguez (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-22256:
---
Attachment: HIVE-22256.patch

> Rewriting fails when `IN` clause has items in different order in MV and query.
> --
>
> Key: HIVE-22256
> URL: https://issues.apache.org/jira/browse/HIVE-22256
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO, Materialized views
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-22256.patch, expr2.sql
>
>
> Rewriting fails on following materialized view and query (script is also 
> attached):
> create materialized view view2 stored as orc as (select prod_id, cust_id, 
> store_id, sale_date, qty, amt, descr from sales where cust_id in (1,2,3,4,5));
> explain extended select prod_id, cust_id  from sales where cust_id in 
> (5,1,2,3,4);



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22256) Rewriting fails when `IN` clause has items in different order in MV and query.

2020-01-08 Thread Jesus Camacho Rodriguez (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-22256:
---
Status: Patch Available  (was: In Progress)

> Rewriting fails when `IN` clause has items in different order in MV and query.
> --
>
> Key: HIVE-22256
> URL: https://issues.apache.org/jira/browse/HIVE-22256
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO, Materialized views
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: expr2.sql
>
>
> Rewriting fails on following materialized view and query (script is also 
> attached):
> create materialized view view2 stored as orc as (select prod_id, cust_id, 
> store_id, sale_date, qty, amt, descr from sales where cust_id in (1,2,3,4,5));
> explain extended select prod_id, cust_id  from sales where cust_id in 
> (5,1,2,3,4);



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HIVE-22256) Rewriting fails when `IN` clause has items in different order in MV and query.

2020-01-08 Thread Jesus Camacho Rodriguez (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez reassigned HIVE-22256:
--

Assignee: Jesus Camacho Rodriguez

> Rewriting fails when `IN` clause has items in different order in MV and query.
> --
>
> Key: HIVE-22256
> URL: https://issues.apache.org/jira/browse/HIVE-22256
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO, Materialized views
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: expr2.sql
>
>
> Rewriting fails on following materialized view and query (script is also 
> attached):
> create materialized view view2 stored as orc as (select prod_id, cust_id, 
> store_id, sale_date, qty, amt, descr from sales where cust_id in (1,2,3,4,5));
> explain extended select prod_id, cust_id  from sales where cust_id in 
> (5,1,2,3,4);



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work started] (HIVE-22256) Rewriting fails when `IN` clause has items in different order in MV and query.

2020-01-08 Thread Jesus Camacho Rodriguez (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-22256 started by Jesus Camacho Rodriguez.
--
> Rewriting fails when `IN` clause has items in different order in MV and query.
> --
>
> Key: HIVE-22256
> URL: https://issues.apache.org/jira/browse/HIVE-22256
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO, Materialized views
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: expr2.sql
>
>
> Rewriting fails on following materialized view and query (script is also 
> attached):
> create materialized view view2 stored as orc as (select prod_id, cust_id, 
> store_id, sale_date, qty, amt, descr from sales where cust_id in (1,2,3,4,5));
> explain extended select prod_id, cust_id  from sales where cust_id in 
> (5,1,2,3,4);



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HIVE-22259) Rewriting fails for `BETWEEN` clauses with different ranges in MV and query

2020-01-08 Thread Jesus Camacho Rodriguez (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez reassigned HIVE-22259:
--

Assignee: Jesus Camacho Rodriguez

> Rewriting fails for `BETWEEN` clauses with different ranges in MV and query
> ---
>
> Key: HIVE-22259
> URL: https://issues.apache.org/jira/browse/HIVE-22259
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO, Materialized views
>Reporter: Steve Carlin
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: expr5.sql
>
>
> Script attached.
> The following query does not rewrite:
> create materialized view view9 stored as orc as (select prod_id, cust_id, 
> store_id, sale_date, qty, amt, descr from sales where cust_id >= 1 and 
> prod_id < 31);
>  
> -- this is not ok
> explain extended select  * from sales where cust_id between 1 and 20 and 
> prod_id < 31;



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HIVE-22258) Rewriting fails for `IN` clauses in MV and query when we use equals or subset in the query

2020-01-08 Thread Jesus Camacho Rodriguez (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez reassigned HIVE-22258:
--

Assignee: Jesus Camacho Rodriguez

> Rewriting fails for `IN` clauses in MV and query when we use equals or subset 
> in the query
> --
>
> Key: HIVE-22258
> URL: https://issues.apache.org/jira/browse/HIVE-22258
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO, Materialized views
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: in-pred.sql
>
>
> ...as stated in title.  Script is attached.  The issue can be seen with these 
> queries:
>  
> create materialized view av1 stored as orc as select state, year, 
> sum(population) from census_pop where year IN (2010, 2018) group by state, 
> year;
> -- this is ok
> explain extended select state, year, sum(population) from census_pop where 
> year IN (2010, 2018) group by state, year;
> -- this is not ok
> explain extended select state, year, sum(population) from census_pop where 
> year = 2010 group by state, year;
> -- this is not ok
> explain extended select state, year, sum(population) from census_pop where 
> year in (2010) group by state, year;



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work started] (HIVE-22258) Rewriting fails for `IN` clauses in MV and query when we use equals or subset in the query

2020-01-08 Thread Jesus Camacho Rodriguez (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-22258 started by Jesus Camacho Rodriguez.
--
> Rewriting fails for `IN` clauses in MV and query when we use equals or subset 
> in the query
> --
>
> Key: HIVE-22258
> URL: https://issues.apache.org/jira/browse/HIVE-22258
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO, Materialized views
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: in-pred.sql
>
>
> ...as stated in title.  Script is attached.  The issue can be seen with these 
> queries:
>  
> create materialized view av1 stored as orc as select state, year, 
> sum(population) from census_pop where year IN (2010, 2018) group by state, 
> year;
> -- this is ok
> explain extended select state, year, sum(population) from census_pop where 
> year IN (2010, 2018) group by state, year;
> -- this is not ok
> explain extended select state, year, sum(population) from census_pop where 
> year = 2010 group by state, year;
> -- this is not ok
> explain extended select state, year, sum(population) from census_pop where 
> year in (2010) group by state, year;



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work started] (HIVE-22259) Rewriting fails for `BETWEEN` clauses with different ranges in MV and query

2020-01-08 Thread Jesus Camacho Rodriguez (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-22259 started by Jesus Camacho Rodriguez.
--
> Rewriting fails for `BETWEEN` clauses with different ranges in MV and query
> ---
>
> Key: HIVE-22259
> URL: https://issues.apache.org/jira/browse/HIVE-22259
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO, Materialized views
>Reporter: Steve Carlin
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: expr5.sql
>
>
> Script attached.
> The following query does not rewrite:
> create materialized view view9 stored as orc as (select prod_id, cust_id, 
> store_id, sale_date, qty, amt, descr from sales where cust_id >= 1 and 
> prod_id < 31);
>  
> -- this is not ok
> explain extended select  * from sales where cust_id between 1 and 20 and 
> prod_id < 31;



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22707) MergeJoinWork should be considered while collecting DAG credentials

2020-01-08 Thread Jira



 [ 
https://issues.apache.org/jira/browse/HIVE-22707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

László Bodor updated HIVE-22707:

Description: 
Given a scenario, when there are 2 different buckets, and the output is written 
to another bucket than the source. Under specific circumstances, 
FileSinkOperator is only used in Reducer stages, and if a root work in that 
stage is a merge join work, it's not scanned for output uris/paths, therefore 
needed delegation tokens are not fetched for e.g. the output s3 bucket.

https://github.com/apache/hive/blob/0df4f6c61010b64246d4790f9ce14e966ef34dcb/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L1507-L1514
{code}
  public void addCredentials(BaseWork work, DAG dag) throws IOException {

dag.getCredentials().mergeAll(UserGroupInformation.getCurrentUser().getCredentials());
if (work instanceof MapWork) {
  addCredentials((MapWork) work, dag);
} else if (work instanceof ReduceWork) {
  addCredentials((ReduceWork) work, dag);
}
  }
{code}

sample plan, not Merge Join Operator [MERGEJOIN_35]
{code}
++
|  Explain   |
++
| Plan optimized by CBO. |
||
| Vertex dependency in root stage|
| Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE) |
| Reducer 3 <- Reducer 2 (CUSTOM_SIMPLE_EDGE)|
||
| Stage-3|
|   Stats Work{} |
| Stage-4|
|   Create 
Table{"name:":"tpcds_bin_partitioned_orc_1000.catalog_sales_out"} |
| Stage-0|
|   Move Operator|
| Stage-1|
|   Reducer 3|
|   File Output Operator [FS_20] |
| Group By Operator [GBY_18] (rows=1 width=440) |
|   
Output:["_col0"],aggregations:["compute_stats(VALUE._col0)"] |
| <-Reducer 2 [CUSTOM_SIMPLE_EDGE]   |
|   File Output Operator [FS_10] |
| 
table:{"name:":"tpcds_bin_partitioned_orc_1000.catalog_sales_out"} |
| Select Operator [SEL_9] (rows=8400 width=7) |
|   Output:["_col0"] |
|   Merge Join Operator [MERGEJOIN_35] (rows=8400 width=7) |
| Conds:RS_38._col1=RS_41._col0(Inner),Output:["_col1"] 
|
|   <-Map 1 [SIMPLE_EDGE] vectorized |
| SHUFFLE [RS_38]|
|   PartitionCols:_col1  |
|   Select Operator [SEL_37] (rows=16799 width=15) |
| Output:["_col1"]   |
| Filter Operator [FIL_36] (rows=16799 width=15) |
|   predicate:((cs_sold_time_sk = 74858L) and 
cs_call_center_sk is not null) |
|   TableScan [TS_0] (rows=1439980416 width=15) |
| 
tpcds_bin_partitioned_orc_1000@catalog_sales,cs, ACID 
table,Tbl:COMPLETE,Col:PARTIAL,Output:["cs_sold_time_sk","cs_call_center_sk"] |
|   <-Map 4 [SIMPLE_EDGE] vectorized |
| SHUFFLE [RS_41]|
|   PartitionCols:_col0  |
|   Select Operator [SEL_40] (rows=21 width=107) |
| Output:["_col0"]   |
| Filter Operator [FIL_39] (rows=21 width=107) |
|   predicate:((CAST( cc_county AS STRING) = 
'Williamson County') and cc_call_center_sk is not null) |
|   TableScan [TS_3] (rows=42 width=107) |
| 
tpcds_bin_partitioned_orc_1000@call_center,cc, ACID 
table,Tbl:COMPLETE,Col:COMPLETE,Output:["cc_call_center_sk","cc_county"] |
|   PARTITION_ONLY_SHUFFLE [RS_17]   |
| Group By Operator [GBY_16] (rows=1 width=424) |
|   Output:["_col0"],aggregations:["compute_stats(col1, 
'hll')"] |
|   Select Operator [SEL_15] (rows=8400 width=7) |
| Output:["col1"]|
|  Please refer to the previous Select Operator [SEL_9] 
|
| Stage-2|
|   Dependency Collection{}  |
|  Please refer to the previous Stage-1  |
||
++

[jira] [Updated] (HIVE-22707) MergeJoinWork should be considered while collecting DAG credentials

2020-01-08 Thread Jira



 [ 
https://issues.apache.org/jira/browse/HIVE-22707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

László Bodor updated HIVE-22707:

Description: Given a scenario, when there are 2 different buckets, and the 
output is written to another bucket than the source. Under specific 
circumstances, FileSinkOperator is only used in Reducer stages, and if a root 
work is a merge join work, it's not scanned for output uris/paths, therefore 
needed delegation tokens are not fetched for e.g. the output s3 bucket.

> MergeJoinWork should be considered while collecting DAG credentials
> ---
>
> Key: HIVE-22707
> URL: https://issues.apache.org/jira/browse/HIVE-22707
> Project: Hive
>  Issue Type: Bug
>Reporter: László Bodor
>Priority: Major
>
> Given a scenario, when there are 2 different buckets, and the output is 
> written to another bucket than the source. Under specific circumstances, 
> FileSinkOperator is only used in Reducer stages, and if a root work is a 
> merge join work, it's not scanned for output uris/paths, therefore needed 
> delegation tokens are not fetched for e.g. the output s3 bucket.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HIVE-22707) MergeJoinWork should be considered while collecting DAG credentials

2020-01-08 Thread Jira



 [ 
https://issues.apache.org/jira/browse/HIVE-22707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

László Bodor reassigned HIVE-22707:
---

Assignee: László Bodor

> MergeJoinWork should be considered while collecting DAG credentials
> ---
>
> Key: HIVE-22707
> URL: https://issues.apache.org/jira/browse/HIVE-22707
> Project: Hive
>  Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
>
> Given a scenario, when there are 2 different buckets, and the output is 
> written to another bucket than the source. Under specific circumstances, 
> FileSinkOperator is only used in Reducer stages, and if a root work is a 
> merge join work, it's not scanned for output uris/paths, therefore needed 
> delegation tokens are not fetched for e.g. the output s3 bucket.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction

2020-01-08 Thread Laszlo Pinter (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Pinter updated HIVE-20934:
-
Attachment: HIVE-20934.10.patch

> ACID: Query based compactor for minor compaction
> 
>
> Key: HIVE-20934
> URL: https://issues.apache.org/jira/browse/HIVE-20934
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.1.1
>Reporter: Vaibhav Gumashta
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, 
> HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, 
> HIVE-20934.06.patch, HIVE-20934.07.patch, HIVE-20934.08.patch, 
> HIVE-20934.09.patch, HIVE-20934.10.patch
>
>
> Follow up of HIVE-20699. This is to enable running minor compactions as a 
> HiveQL query 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22653) Remove commons-lang leftovers

2020-01-08 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010836#comment-17010836
 ] 

Hive QA commented on HIVE-22653:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 10m 
27s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
20s{color} | {color:blue} storage-api in master has 58 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
17s{color} | {color:blue} shims/common in master has 6 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
22s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  2m 
25s{color} | {color:blue} standalone-metastore/metastore-common in master has 
37 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
31s{color} | {color:blue} common in master has 63 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
37s{color} | {color:blue} serde in master has 198 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
31s{color} | {color:blue} llap-common in master has 90 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
26s{color} | {color:blue} llap-tez in master has 18 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
25s{color} | {color:blue} spark-client in master has 10 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  1m 
13s{color} | {color:blue} standalone-metastore/metastore-server in master has 
181 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
1s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
42s{color} | {color:blue} llap-server in master has 90 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
39s{color} | {color:blue} service in master has 49 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
31s{color} | {color:blue} accumulo-handler in master has 20 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
27s{color} | {color:blue} jdbc in master has 16 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
30s{color} | {color:blue} beeline in master has 48 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
27s{color} | {color:blue} cli in master has 9 extant Findbugs warnings. {color} 
|
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
28s{color} | {color:blue} druid-handler in master has 3 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
32s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
35s{color} | {color:blue} hcatalog/core in master has 36 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
24s{color} | {color:blue} hcatalog/hcatalog-pig-adapter in master has 2 extant 
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
26s{color} | {color:blue} hcatalog/server-extensions in master has 3 extant 
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
26s{color} | {color:blue} hcatalog/webhcat/java-client in master has 3 extant

[jira] [Commented] (HIVE-22706) Jdbc storage handler incorrectly interprets boolean column value in derby

2020-01-08 Thread Syed Shameerur Rahman (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010832#comment-17010832
 ] 

Syed Shameerur Rahman commented on HIVE-22706:
--

[~kgyrtkirk] Is it similar to https://issues.apache.org/jira/browse/HIVE-22433  
?

> Jdbc storage handler incorrectly interprets boolean column value in derby
> -
>
> Key: HIVE-22706
> URL: https://issues.apache.org/jira/browse/HIVE-22706
> Project: Hive
>  Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>
> in case the column value is false ; the storage handler interprets it as true



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-20934) ACID: Query based compactor for minor compaction

2020-01-08 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010829#comment-17010829
 ] 

Hive QA commented on HIVE-20934:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12990213/HIVE-20934.09.patch

{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 17869 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/20111/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20111/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20111/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12990213 - PreCommit-HIVE-Build

> ACID: Query based compactor for minor compaction
> 
>
> Key: HIVE-20934
> URL: https://issues.apache.org/jira/browse/HIVE-20934
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.1.1
>Reporter: Vaibhav Gumashta
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, 
> HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, 
> HIVE-20934.06.patch, HIVE-20934.07.patch, HIVE-20934.08.patch, 
> HIVE-20934.09.patch
>
>
> Follow up of HIVE-20699. This is to enable running minor compactions as a 
> HiveQL query 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HIVE-22706) Jdbc storage handler incorrectly interprets boolean column value in derby

2020-01-08 Thread Zoltan Haindrich (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich reassigned HIVE-22706:
---


> Jdbc storage handler incorrectly interprets boolean column value in derby
> -
>
> Key: HIVE-22706
> URL: https://issues.apache.org/jira/browse/HIVE-22706
> Project: Hive
>  Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>
> in case the column value is false ; the storage handler interprets it as true



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-20934) ACID: Query based compactor for minor compaction

2020-01-08 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010808#comment-17010808
 ] 

Hive QA commented on HIVE-20934:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
34s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  6m 
28s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  1m  
5s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
44s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
50s{color} | {color:green} ql: The patch generated 0 new + 408 unchanged - 4 
fixed = 408 total (was 412) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
22s{color} | {color:red} itests/hive-unit: The patch generated 3 new + 138 
unchanged - 25 fixed = 141 total (was 163) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  7m  
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 26s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20111/dev-support/hive-personality.sh
 |
| git revision | master / 948144a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-20111/yetus/diff-checkstyle-itests_hive-unit.txt
 |
| modules | C: ql itests/hive-unit U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-20111/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> ACID: Query based compactor for minor compaction
> 
>
> Key: HIVE-20934
> URL: https://issues.apache.org/jira/browse/HIVE-20934
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.1.1
>Reporter: Vaibhav Gumashta
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, 
> HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, 
> HIVE-20934.06.patch, HIVE-20934.07.patch, HIVE-20934.08.patch, 
> HIVE-20934.09.patch
>
>
> Follow up of HIVE-20699. This is to enable running minor compactions as a 
> HiveQL query 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-22653) Remove commons-lang leftovers

2020-01-08 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22653?focusedWorklogId=368248=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368248
 ]

ASF GitHub Bot logged work on HIVE-22653:
-

Author: ASF GitHub Bot
Created on: 08/Jan/20 16:24
Start Date: 08/Jan/20 16:24
Worklog Time Spent: 10m 
  Work Description: kgyrtkirk commented on pull request #870: HIVE-22653: 
Remove commons-lang leftovers
URL: https://github.com/apache/hive/pull/870
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 368248)
Time Spent: 20m  (was: 10m)

> Remove commons-lang leftovers
> -
>
> Key: HIVE-22653
> URL: https://issues.apache.org/jira/browse/HIVE-22653
> Project: Hive
>  Issue Type: Bug
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22653.01.patch, HIVE-22653.01.patch, 
> HIVE-22653.02.patch, HIVE-22653.03.patch
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> HIVE-7145 removed commons-lang - in favor of commons-lang3 - as a direct 
> dependency, however a high number of files still refer to commons-lang, which 
> is transitively brought in either way.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22653) Remove commons-lang leftovers

2020-01-08 Thread Zoltan Haindrich (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich updated HIVE-22653:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

pushed to master. Thank you David for fixing this!

> Remove commons-lang leftovers
> -
>
> Key: HIVE-22653
> URL: https://issues.apache.org/jira/browse/HIVE-22653
> Project: Hive
>  Issue Type: Bug
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22653.01.patch, HIVE-22653.01.patch, 
> HIVE-22653.02.patch, HIVE-22653.03.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> HIVE-7145 removed commons-lang - in favor of commons-lang3 - as a direct 
> dependency, however a high number of files still refer to commons-lang, which 
> is transitively brought in either way.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22261) Add tests for materialized view rewriting with window functions

2020-01-08 Thread Jesus Camacho Rodriguez (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-22261:
---
Summary: Add tests for materialized view rewriting with window functions  
(was: Materialized view rewriting does not support window functions)

> Add tests for materialized view rewriting with window functions
> ---
>
> Key: HIVE-22261
> URL: https://issues.apache.org/jira/browse/HIVE-22261
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO, Materialized views, Tests
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-22261.patch, af2.sql
>
>
> Materialized views don't support window functions.  At a minimum, we should 
> print a friendlier message when the rewrite fails (it can still be created 
> with a "disable rewrite")
> Script is attached
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22261) Materialized view rewriting does not support window functions

2020-01-08 Thread Jesus Camacho Rodriguez (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-22261:
---
Component/s: Tests

> Materialized view rewriting does not support window functions
> -
>
> Key: HIVE-22261
> URL: https://issues.apache.org/jira/browse/HIVE-22261
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO, Materialized views, Tests
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-22261.patch, af2.sql
>
>
> Materialized views don't support window functions.  At a minimum, we should 
> print a friendlier message when the rewrite fails (it can still be created 
> with a "disable rewrite")
> Script is attached
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22261) Materialized view rewriting does not support window functions

2020-01-08 Thread Jesus Camacho Rodriguez (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010789#comment-17010789
 ] 

Jesus Camacho Rodriguez commented on HIVE-22261:


[~vgarg], [~kgyrtkirk], can you review this patch? There is no code changes, it 
only adds tests. Thanks

> Materialized view rewriting does not support window functions
> -
>
> Key: HIVE-22261
> URL: https://issues.apache.org/jira/browse/HIVE-22261
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO, Materialized views
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-22261.patch, af2.sql
>
>
> Materialized views don't support window functions.  At a minimum, we should 
> print a friendlier message when the rewrite fails (it can still be created 
> with a "disable rewrite")
> Script is attached
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22653) Remove commons-lang leftovers

2020-01-08 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010776#comment-17010776
 ] 

Hive QA commented on HIVE-22653:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12990205/HIVE-22653.03.patch

{color:green}SUCCESS:{color} +1 due to 24 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 17858 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/20110/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20110/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20110/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12990205 - PreCommit-HIVE-Build

> Remove commons-lang leftovers
> -
>
> Key: HIVE-22653
> URL: https://issues.apache.org/jira/browse/HIVE-22653
> Project: Hive
>  Issue Type: Bug
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22653.01.patch, HIVE-22653.01.patch, 
> HIVE-22653.02.patch, HIVE-22653.03.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> HIVE-7145 removed commons-lang - in favor of commons-lang3 - as a direct 
> dependency, however a high number of files still refer to commons-lang, which 
> is transitively brought in either way.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22704) Distribution package incorrectly ships the upgrade.order files from the metastore package

2020-01-08 Thread Zoltan Haindrich (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich updated HIVE-22704:

Summary: Distribution package incorrectly ships the upgrade.order files 
from the metastore package  (was: Distribution package ships the upgrade.order 
files from the metastore package)

> Distribution package incorrectly ships the upgrade.order files from the 
> metastore package
> -
>
> Key: HIVE-22704
> URL: https://issues.apache.org/jira/browse/HIVE-22704
> Project: Hive
>  Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22704.01.patch
>
>
> Expected would be to take the upgrade.order from the standalon-metastore but 
> HIVE-19938 have broke that when it [introduced a 
> regex|https://github.infra.cloudera.com/CDH/hive/commit/bae4c08408720d56159847603dfb8d9dab9fe1b7#diff-44c56fadf216c807d6828b60825e43baR219]
> {code}
> $ echo asd > ./metastore/scripts/upgrade/postgres/upgrade.order.postgres
> $ mvn install -pl packaging/ -Pdist -am -DskipTests -q
> $ cat 
> ./packaging/target/apache-hive-4.0.0-SNAPSHOT-bin/apache-hive-4.0.0-SNAPSHOT-bin/scripts/metastore/upgrade/postgres/upgrade.order.postgres
> asd
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22704) Distribution package incorrectly ships the upgrade.order files from the metastore module

2020-01-08 Thread Zoltan Haindrich (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich updated HIVE-22704:

Summary: Distribution package incorrectly ships the upgrade.order files 
from the metastore module  (was: Distribution package incorrectly ships the 
upgrade.order files from the metastore package)

> Distribution package incorrectly ships the upgrade.order files from the 
> metastore module
> 
>
> Key: HIVE-22704
> URL: https://issues.apache.org/jira/browse/HIVE-22704
> Project: Hive
>  Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22704.01.patch
>
>
> Expected would be to take the upgrade.order from the standalon-metastore but 
> HIVE-19938 have broke that when it [introduced a 
> regex|https://github.infra.cloudera.com/CDH/hive/commit/bae4c08408720d56159847603dfb8d9dab9fe1b7#diff-44c56fadf216c807d6828b60825e43baR219]
> {code}
> $ echo asd > ./metastore/scripts/upgrade/postgres/upgrade.order.postgres
> $ mvn install -pl packaging/ -Pdist -am -DskipTests -q
> $ cat 
> ./packaging/target/apache-hive-4.0.0-SNAPSHOT-bin/apache-hive-4.0.0-SNAPSHOT-bin/scripts/metastore/upgrade/postgres/upgrade.order.postgres
> asd
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work started] (HIVE-22705) LLAP cache is polluted by query-based compactor

2020-01-08 Thread Jira



 [ 
https://issues.apache.org/jira/browse/HIVE-22705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-22705 started by Ádám Szita.
-
> LLAP cache is polluted by query-based compactor
> ---
>
> Key: HIVE-22705
> URL: https://issues.apache.org/jira/browse/HIVE-22705
> Project: Hive
>  Issue Type: Improvement
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
>
> One of the steps that query-based compaction does is the verification of ACID 
> sort order by using the _validate_acid_sort_order_ UDF. This is a 
> prerequisite before the actual compaction can happen, and is done by a [query 
> that reads the whole table 
> content|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MajorQueryCompactor.java#L161-L167].
> This results in the whole table content being populated into the cache. The 
> problem is that this content is not useful and will rather pollute the cache 
> space, as it can never be used again: cache content binds to files (file IDs) 
> that obviously will be changed in this case by compaction.
> I propose we disable LLAP caching in the session of query-based compaction's 
> queries.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HIVE-22705) LLAP cache is polluted by query-based compactor

2020-01-08 Thread Jira



 [ 
https://issues.apache.org/jira/browse/HIVE-22705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ádám Szita reassigned HIVE-22705:
-


> LLAP cache is polluted by query-based compactor
> ---
>
> Key: HIVE-22705
> URL: https://issues.apache.org/jira/browse/HIVE-22705
> Project: Hive
>  Issue Type: Improvement
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
>
> One of the steps that query-based compaction does is the verification of ACID 
> sort order by using the _validate_acid_sort_order_ UDF. This is a 
> prerequisite before the actual compaction can happen, and is done by a [query 
> that reads the whole table 
> content|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MajorQueryCompactor.java#L161-L167].
> This results in the whole table content being populated into the cache. The 
> problem is that this content is not useful and will rather pollute the cache 
> space, as it can never be used again: cache content binds to files (file IDs) 
> that obviously will be changed in this case by compaction.
> I propose we disable LLAP caching in the session of query-based compaction's 
> queries.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22704) Distribution package ships the upgrade.order files from the metastore package

2020-01-08 Thread Zoltan Haindrich (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010750#comment-17010750
 ] 

Zoltan Haindrich commented on HIVE-22704:
-

patch#1) proposed solution: 

* remove regex filter
* remove duplicate order files which are not part of the distribution

> Distribution package ships the upgrade.order files from the metastore package
> -
>
> Key: HIVE-22704
> URL: https://issues.apache.org/jira/browse/HIVE-22704
> Project: Hive
>  Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22704.01.patch
>
>
> Expected would be to take the upgrade.order from the standalon-metastore but 
> HIVE-19938 have broke that when it [introduced a 
> regex|https://github.infra.cloudera.com/CDH/hive/commit/bae4c08408720d56159847603dfb8d9dab9fe1b7#diff-44c56fadf216c807d6828b60825e43baR219]
> {code}
> $ echo asd > ./metastore/scripts/upgrade/postgres/upgrade.order.postgres
> $ mvn install -pl packaging/ -Pdist -am -DskipTests -q
> $ cat 
> ./packaging/target/apache-hive-4.0.0-SNAPSHOT-bin/apache-hive-4.0.0-SNAPSHOT-bin/scripts/metastore/upgrade/postgres/upgrade.order.postgres
> asd
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22704) Distribution package ships the upgrade.order files from the metastore package

2020-01-08 Thread Zoltan Haindrich (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich updated HIVE-22704:

Status: Patch Available  (was: Open)

> Distribution package ships the upgrade.order files from the metastore package
> -
>
> Key: HIVE-22704
> URL: https://issues.apache.org/jira/browse/HIVE-22704
> Project: Hive
>  Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22704.01.patch
>
>
> Expected would be to take the upgrade.order from the standalon-metastore but 
> HIVE-19938 have broke that when it [introduced a 
> regex|https://github.infra.cloudera.com/CDH/hive/commit/bae4c08408720d56159847603dfb8d9dab9fe1b7#diff-44c56fadf216c807d6828b60825e43baR219]
> {code}
> $ echo asd > ./metastore/scripts/upgrade/postgres/upgrade.order.postgres
> $ mvn install -pl packaging/ -Pdist -am -DskipTests -q
> $ cat 
> ./packaging/target/apache-hive-4.0.0-SNAPSHOT-bin/apache-hive-4.0.0-SNAPSHOT-bin/scripts/metastore/upgrade/postgres/upgrade.order.postgres
> asd
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22704) Distribution package ships the upgrade.order files from the metastore package

2020-01-08 Thread Zoltan Haindrich (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich updated HIVE-22704:

Attachment: HIVE-22704.01.patch

> Distribution package ships the upgrade.order files from the metastore package
> -
>
> Key: HIVE-22704
> URL: https://issues.apache.org/jira/browse/HIVE-22704
> Project: Hive
>  Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22704.01.patch
>
>
> Expected would be to take the upgrade.order from the standalon-metastore but 
> HIVE-19938 have broke that when it [introduced a 
> regex|https://github.infra.cloudera.com/CDH/hive/commit/bae4c08408720d56159847603dfb8d9dab9fe1b7#diff-44c56fadf216c807d6828b60825e43baR219]
> {code}
> $ echo asd > ./metastore/scripts/upgrade/postgres/upgrade.order.postgres
> $ mvn install -pl packaging/ -Pdist -am -DskipTests -q
> $ cat 
> ./packaging/target/apache-hive-4.0.0-SNAPSHOT-bin/apache-hive-4.0.0-SNAPSHOT-bin/scripts/metastore/upgrade/postgres/upgrade.order.postgres
> asd
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22704) Distribution package ships the upgrade.order files from the metastore package

2020-01-08 Thread Zoltan Haindrich (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich updated HIVE-22704:

Description: 
Expected would be to take the upgrade.order from the standalon-metastore but 
HIVE-19938 have broke that when it [introduced a 
regex|https://github.infra.cloudera.com/CDH/hive/commit/bae4c08408720d56159847603dfb8d9dab9fe1b7#diff-44c56fadf216c807d6828b60825e43baR219]

{code}
$ echo asd > ./metastore/scripts/upgrade/postgres/upgrade.order.postgres
$ mvn install -pl packaging/ -Pdist -am -DskipTests -q
$ cat 
./packaging/target/apache-hive-4.0.0-SNAPSHOT-bin/apache-hive-4.0.0-SNAPSHOT-bin/scripts/metastore/upgrade/postgres/upgrade.order.postgres
asd
{code}

  was:
Expected would be to take the upgrade.order from the standalon-metastore but 
HIVE-19938 have broke that when it [introduced a 
regex|https://github.infra.cloudera.com/CDH/hive/commit/bae4c08408720d56159847603dfb8d9dab9fe1b7#diff-44c56fadf216c807d6828b60825e43baR219]

{code}
echo asd > ./metastore/scripts/upgrade/postgres/upgrade.order.postgres
mvn install -pl packaging/ -Pdist -am -DskipTests -q
cat 
./packaging/target/apache-hive-4.0.0-SNAPSHOT-bin/apache-hive-4.0.0-SNAPSHOT-bin/scripts/metastore/upgrade/postgres/upgrade.order.postgres
asd
{code}


> Distribution package ships the upgrade.order files from the metastore package
> -
>
> Key: HIVE-22704
> URL: https://issues.apache.org/jira/browse/HIVE-22704
> Project: Hive
>  Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>
> Expected would be to take the upgrade.order from the standalon-metastore but 
> HIVE-19938 have broke that when it [introduced a 
> regex|https://github.infra.cloudera.com/CDH/hive/commit/bae4c08408720d56159847603dfb8d9dab9fe1b7#diff-44c56fadf216c807d6828b60825e43baR219]
> {code}
> $ echo asd > ./metastore/scripts/upgrade/postgres/upgrade.order.postgres
> $ mvn install -pl packaging/ -Pdist -am -DskipTests -q
> $ cat 
> ./packaging/target/apache-hive-4.0.0-SNAPSHOT-bin/apache-hive-4.0.0-SNAPSHOT-bin/scripts/metastore/upgrade/postgres/upgrade.order.postgres
> asd
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HIVE-22704) Distribution package ships the upgrade.order files from the metastore package

2020-01-08 Thread Zoltan Haindrich (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich reassigned HIVE-22704:
---


> Distribution package ships the upgrade.order files from the metastore package
> -
>
> Key: HIVE-22704
> URL: https://issues.apache.org/jira/browse/HIVE-22704
> Project: Hive
>  Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>
> Expected would be to take the upgrade.order from the standalon-metastore but 
> HIVE-19938 have broke that when it [introduced a 
> regex|https://github.infra.cloudera.com/CDH/hive/commit/bae4c08408720d56159847603dfb8d9dab9fe1b7#diff-44c56fadf216c807d6828b60825e43baR219]
> {code}
> echo asd > ./metastore/scripts/upgrade/postgres/upgrade.order.postgres
> mvn install -pl packaging/ -Pdist -am -DskipTests -q
> cat 
> ./packaging/target/apache-hive-4.0.0-SNAPSHOT-bin/apache-hive-4.0.0-SNAPSHOT-bin/scripts/metastore/upgrade/postgres/upgrade.order.postgres
> asd
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction

2020-01-08 Thread Laszlo Pinter (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Pinter updated HIVE-20934:
-
Attachment: HIVE-20934.09.patch

> ACID: Query based compactor for minor compaction
> 
>
> Key: HIVE-20934
> URL: https://issues.apache.org/jira/browse/HIVE-20934
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.1.1
>Reporter: Vaibhav Gumashta
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, 
> HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, 
> HIVE-20934.06.patch, HIVE-20934.07.patch, HIVE-20934.08.patch, 
> HIVE-20934.09.patch
>
>
> Follow up of HIVE-20699. This is to enable running minor compactions as a 
> HiveQL query 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HIVE-22703) Compaction configuration check when starting HMS/HS2

2020-01-08 Thread Laszlo Pinter (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Pinter reassigned HIVE-22703:



> Compaction configuration check when starting HMS/HS2
> 
>
> Key: HIVE-22703
> URL: https://issues.apache.org/jira/browse/HIVE-22703
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
>
> Currently when starting HMS we can have bugous configuration which prevents 
> compatction to work. We should find a way to inform the admin about the 
> configuration error, or even prevent HMS to start in this case.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22395) Add ability to read Druid metastore password from jceks

2020-01-08 Thread Nishant Bangarwa (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010682#comment-17010682
 ] 

Nishant Bangarwa commented on HIVE-22395:
-

committed via 
https://github.com/apache/hive/commit/948144a49753d3955505f428d427fb7b2fb9642a

> Add ability to read Druid metastore password from jceks
> ---
>
> Key: HIVE-22395
> URL: https://issues.apache.org/jira/browse/HIVE-22395
> Project: Hive
>  Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-22395.1.patch, HIVE-22395.2.patch, 
> HIVE-22395.2.patch, HIVE-22395.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22395) Add ability to read Druid metastore password from jceks

2020-01-08 Thread Nishant Bangarwa (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nishant Bangarwa updated HIVE-22395:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Add ability to read Druid metastore password from jceks
> ---
>
> Key: HIVE-22395
> URL: https://issues.apache.org/jira/browse/HIVE-22395
> Project: Hive
>  Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-22395.1.patch, HIVE-22395.2.patch, 
> HIVE-22395.2.patch, HIVE-22395.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22653) Remove commons-lang leftovers

2020-01-08 Thread David Lavati (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Lavati updated HIVE-22653:

Attachment: HIVE-22653.03.patch
Status: Patch Available  (was: In Progress)

commons-lang is a banned import now, handled by an enforcer rule

> Remove commons-lang leftovers
> -
>
> Key: HIVE-22653
> URL: https://issues.apache.org/jira/browse/HIVE-22653
> Project: Hive
>  Issue Type: Bug
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22653.01.patch, HIVE-22653.01.patch, 
> HIVE-22653.02.patch, HIVE-22653.03.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> HIVE-7145 removed commons-lang - in favor of commons-lang3 - as a direct 
> dependency, however a high number of files still refer to commons-lang, which 
> is transitively brought in either way.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22653) Remove commons-lang leftovers

2020-01-08 Thread David Lavati (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Lavati updated HIVE-22653:

Status: In Progress  (was: Patch Available)

> Remove commons-lang leftovers
> -
>
> Key: HIVE-22653
> URL: https://issues.apache.org/jira/browse/HIVE-22653
> Project: Hive
>  Issue Type: Bug
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22653.01.patch, HIVE-22653.01.patch, 
> HIVE-22653.02.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> HIVE-7145 removed commons-lang - in favor of commons-lang3 - as a direct 
> dependency, however a high number of files still refer to commons-lang, which 
> is transitively brought in either way.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction

2020-01-08 Thread Laszlo Pinter (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Pinter updated HIVE-20934:
-
Attachment: HIVE-20934.08.patch

> ACID: Query based compactor for minor compaction
> 
>
> Key: HIVE-20934
> URL: https://issues.apache.org/jira/browse/HIVE-20934
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.1.1
>Reporter: Vaibhav Gumashta
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, 
> HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, 
> HIVE-20934.06.patch, HIVE-20934.07.patch, HIVE-20934.08.patch
>
>
> Follow up of HIVE-20699. This is to enable running minor compactions as a 
> HiveQL query 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-20934) ACID: Query based compactor for minor compaction

2020-01-08 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010652#comment-17010652
 ] 

Hive QA commented on HIVE-20934:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12990191/HIVE-20934.07.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/20109/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20109/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20109/

Messages:
{noformat}
 This message was trimmed, see log for full details 
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-runner/9.3.27.v20190418/jetty-runner-9.3.27.v20190418.jar(javax/servlet/FilterConfig.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-runner/9.3.27.v20190418/jetty-runner-9.3.27.v20190418.jar(javax/servlet/ServletException.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-runner/9.3.27.v20190418/jetty-runner-9.3.27.v20190418.jar(javax/servlet/ServletRequest.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-runner/9.3.27.v20190418/jetty-runner-9.3.27.v20190418.jar(javax/servlet/ServletResponse.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-runner/9.3.27.v20190418/jetty-runner-9.3.27.v20190418.jar(javax/servlet/annotation/WebFilter.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-runner/9.3.27.v20190418/jetty-runner-9.3.27.v20190418.jar(javax/servlet/http/HttpServletRequest.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-runner/9.3.27.v20190418/jetty-runner-9.3.27.v20190418.jar(javax/servlet/http/HttpServletResponse.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/classification/target/hive-classification-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/common/classification/InterfaceAudience$LimitedPrivate.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/classification/target/hive-classification-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/common/classification/InterfaceStability$Unstable.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/ByteArrayOutputStream.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/OutputStream.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/Closeable.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/AutoCloseable.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/Flushable.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(javax/xml/bind/annotation/XmlRootElement.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/commons/commons-exec/1.1/commons-exec-1.1.jar(org/apache/commons/exec/ExecuteException.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/security/PrivilegedExceptionAction.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/ExecutionException.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/TimeoutException.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/fs/FileSystem.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/HadoopShimsSecure.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/ShimLoader.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/HadoopShims.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/HadoopShims$WebHCatJTShim.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/util/ToolRunner.class)]]
[loading

[jira] [Commented] (HIVE-22593) Dynamically partitioned MM (insert-only ACID) tables don't compact automatically

2020-01-08 Thread Peter Vary (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010638#comment-17010638
 ] 

Peter Vary commented on HIVE-22593:
---

+1

> Dynamically partitioned MM (insert-only ACID) tables don't compact 
> automatically
> 
>
> Key: HIVE-22593
> URL: https://issues.apache.org/jira/browse/HIVE-22593
> Project: Hive
>  Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-22593.01.patch, HIVE-22593.01.patch, 
> HIVE-22593.02.patch, HIVE-22593.03.patch, HIVE-22593.04.patch, 
> HIVE-22593.04.patch
>
>
> Dynamic partitions of MM tables aren't entered into the HMS table 
> TXN_COMPONENTS. On inserting into such tables we see this line in the HMS log:
> {code:java}
> Expected to move at least one record from txn_components to 
> completed_txn_components when committing txn!{code}
> (This is not the case for non-partitioned MM tables.)
> Since the partitions aren't entered into COMPLETED_TXN_COMPONENTS, they 
> aren't considered for automatic compaction.
> Probably the culprit is 
> org.apache.hadoop.hive.ql.metadata.Hive#loadDynamicPartitions which has an 
> isAcid parameter that is always false regarding MM tables, and also because 
> MM tables' "write type" is AcidUtils.Operation.NOT_ACID and not INSERT.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction

2020-01-08 Thread Laszlo Pinter (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Pinter updated HIVE-20934:
-
Attachment: HIVE-20934.07.patch

> ACID: Query based compactor for minor compaction
> 
>
> Key: HIVE-20934
> URL: https://issues.apache.org/jira/browse/HIVE-20934
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.1.1
>Reporter: Vaibhav Gumashta
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, 
> HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, 
> HIVE-20934.06.patch, HIVE-20934.07.patch
>
>
> Follow up of HIVE-20699. This is to enable running minor compactions as a 
> HiveQL query 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22653) Remove commons-lang leftovers

2020-01-08 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010610#comment-17010610
 ] 

Hive QA commented on HIVE-22653:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12990187/HIVE-22653.02.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/20108/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20108/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20108/

Messages:
{noformat}
 This message was trimmed, see log for full details 
error: a/jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java: does not exist 
in index
error: a/jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java: does not exist 
in index
error: 
a/llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java:
 does not exist in index
error: a/llap-common/src/test/org/apache/hadoop/hive/llap/TestRow.java: does 
not exist in index
error: 
a/llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java:
 does not exist in index
error: 
a/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTaskReporter.java:
 does not exist in index
error: 
a/llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java:
 does not exist in index
error: 
a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveClientCache.java: 
does not exist in index
error: 
a/metastore/src/java/org/apache/hadoop/hive/metastore/SerDeStorageSchemaReader.java:
 does not exist in index
error: a/pom.xml: does not exist in index
error: a/ql/pom.xml: does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/Context.java: does not exist in 
index
error: a/ql/src/java/org/apache/hadoop/hive/ql/ddl/DDLUtils.java: does not 
exist in index
error: 
a/ql/src/java/org/apache/hadoop/hive/ql/ddl/database/alter/location/AlterDatabaseSetLocationOperation.java:
 does not exist in index
error: 
a/ql/src/java/org/apache/hadoop/hive/ql/ddl/function/desc/DescFunctionOperation.java:
 does not exist in index
error: 
a/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/AbstractAlterTableOperation.java:
 does not exist in index
error: 
a/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/create/show/ShowCreateTableOperation.java:
 does not exist in index
error: 
a/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/DescTableOperation.java: 
does not exist in index
error: 
a/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/AlterTableSetPropertiesOperation.java:
 does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java: does not exist 
in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java: does 
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java: does 
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/OrcFileMergeOperator.java: 
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java: does not 
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java: does 
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java: does 
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java: 
does not exist in index
error: 
a/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java:
 does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java: 
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadWork.java: 
does not exist in index
error: 
a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/events/filesystem/BootstrapEventsIterator.java:
 does not exist in index
error: 
a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/events/filesystem/DatabaseEventsIterator.java:
 does not exist in index
error: 
a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/events/filesystem/FSDatabaseEvent.java:
 does not exist in index
error: 
a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/events/filesystem/FSTableEvent.java:
 does not exist in index
error: 
a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/LoadFunction.java:
 does not exist in index
error: 
a/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java: 
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlan.java: does 
not exist in index
error: 
a/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/LocalSparkJobMonitor.java:
 does not exist in index
error: 
a/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RemoteSparkJobMonitor.java:
 does not exist in index
error:

[jira] [Updated] (HIVE-22653) Remove commons-lang leftovers

2020-01-08 Thread David Lavati (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Lavati updated HIVE-22653:

Attachment: HIVE-22653.02.patch
Status: Patch Available  (was: In Progress)

> Remove commons-lang leftovers
> -
>
> Key: HIVE-22653
> URL: https://issues.apache.org/jira/browse/HIVE-22653
> Project: Hive
>  Issue Type: Bug
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22653.01.patch, HIVE-22653.01.patch, 
> HIVE-22653.02.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> HIVE-7145 removed commons-lang - in favor of commons-lang3 - as a direct 
> dependency, however a high number of files still refer to commons-lang, which 
> is transitively brought in either way.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22653) Remove commons-lang leftovers

2020-01-08 Thread David Lavati (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010606#comment-17010606
 ] 

David Lavati commented on HIVE-22653:
-

TestGenericUDFInitCap#testInitCap was failing indeed.

commons-lang3.WordUtils became deprecated and got moved to commons-text 
instead. However a regression was introduced there as mentioned in TEXT-175.

As a workaround I've introduced all the whitespace chars as delimiters - which 
matches the original implementation through Character.isWhitespace (except for 
also enabling non-breaking spaces, see 
[https://docs.oracle.com/javase/8/docs/api/java/lang/Character.html#isWhitespace-char-])

> Remove commons-lang leftovers
> -
>
> Key: HIVE-22653
> URL: https://issues.apache.org/jira/browse/HIVE-22653
> Project: Hive
>  Issue Type: Bug
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22653.01.patch, HIVE-22653.01.patch, 
> HIVE-22653.02.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> HIVE-7145 removed commons-lang - in favor of commons-lang3 - as a direct 
> dependency, however a high number of files still refer to commons-lang, which 
> is transitively brought in either way.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22653) Remove commons-lang leftovers

2020-01-08 Thread David Lavati (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Lavati updated HIVE-22653:

Status: In Progress  (was: Patch Available)

> Remove commons-lang leftovers
> -
>
> Key: HIVE-22653
> URL: https://issues.apache.org/jira/browse/HIVE-22653
> Project: Hive
>  Issue Type: Bug
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22653.01.patch, HIVE-22653.01.patch, 
> HIVE-22653.02.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> HIVE-7145 removed commons-lang - in favor of commons-lang3 - as a direct 
> dependency, however a high number of files still refer to commons-lang, which 
> is transitively brought in either way.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-21213) Acid table bootstrap replication needs to handle directory created by compaction with txn id

2020-01-08 Thread Ashutosh Bapat (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-21213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010595#comment-17010595
 ] 

Ashutosh Bapat commented on HIVE-21213:
---

Reviewed the changes. They look fine to me.

> Acid table bootstrap replication needs to handle directory created by 
> compaction with txn id
> 
>
> Key: HIVE-21213
> URL: https://issues.apache.org/jira/browse/HIVE-21213
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, HiveServer2, repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-21213.01.patch, HIVE-21213.02.patch, 
> HIVE-21213.03.patch, HIVE-21213.04.patch, HIVE-21213.05.patch
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> The current implementation of compaction uses the txn id in the directory 
> name. This is used to isolate the queries from reading the directory until 
> compaction has finished and to avoid the compactor marking used earlier. In 
> case of replication, during bootstrap , directory is copied as it is with the 
> same name from source to destination cluster. But the directory created by 
> compaction with txn id can not be copied as the txn list at target may be 
> different from source. The txn id which is valid at source may be an aborted 
> txn at target. So conversion logic is required to create a new directory with 
> valid txn at target and dump the data to the newly created directory.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-20934) ACID: Query based compactor for minor compaction

2020-01-08 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010592#comment-17010592
 ] 

Hive QA commented on HIVE-20934:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12990182/HIVE-20934.06.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/20107/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20107/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20107/

Messages:
{noformat}
 This message was trimmed, see log for full details 
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-runner/9.3.27.v20190418/jetty-runner-9.3.27.v20190418.jar(javax/servlet/FilterConfig.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-runner/9.3.27.v20190418/jetty-runner-9.3.27.v20190418.jar(javax/servlet/ServletException.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-runner/9.3.27.v20190418/jetty-runner-9.3.27.v20190418.jar(javax/servlet/ServletRequest.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-runner/9.3.27.v20190418/jetty-runner-9.3.27.v20190418.jar(javax/servlet/ServletResponse.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-runner/9.3.27.v20190418/jetty-runner-9.3.27.v20190418.jar(javax/servlet/annotation/WebFilter.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-runner/9.3.27.v20190418/jetty-runner-9.3.27.v20190418.jar(javax/servlet/http/HttpServletRequest.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-runner/9.3.27.v20190418/jetty-runner-9.3.27.v20190418.jar(javax/servlet/http/HttpServletResponse.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/classification/target/hive-classification-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/common/classification/InterfaceAudience$LimitedPrivate.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/classification/target/hive-classification-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/common/classification/InterfaceStability$Unstable.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/ByteArrayOutputStream.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/OutputStream.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/Closeable.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/AutoCloseable.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/Flushable.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(javax/xml/bind/annotation/XmlRootElement.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/commons/commons-exec/1.1/commons-exec-1.1.jar(org/apache/commons/exec/ExecuteException.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/security/PrivilegedExceptionAction.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/ExecutionException.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/TimeoutException.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/fs/FileSystem.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/HadoopShimsSecure.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/ShimLoader.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/HadoopShims.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/HadoopShims$WebHCatJTShim.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/util/ToolRunner.class)]]
[loading

[jira] [Commented] (HIVE-20901) running compactor when there is nothing to do produces duplicate data

2020-01-08 Thread Abhishek Somani (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010587#comment-17010587
 ] 

Abhishek Somani commented on HIVE-20901:


[~pvary] Sure!

> running compactor when there is nothing to do produces duplicate data
> -
>
> Key: HIVE-20901
> URL: https://issues.apache.org/jira/browse/HIVE-20901
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Abhishek Somani
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20901.1.patch, HIVE-20901.2.patch
>
>
> suppose we run minor compaction 2 times, via alter table
> The 2nd request to compaction should have nothing to do but I don't think 
> there is a check for that.  It's visible in the context of HIVE-20823, where 
> each compactor run produces a delta with new visibility suffix so we end up 
> with something like
> {noformat}
> target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands3-1541810844849/warehouse/t/
> ├── delete_delta_001_002_v019
> │   ├── _orc_acid_version
> │   └── bucket_0
> ├── delete_delta_001_002_v021
> │   ├── _orc_acid_version
> │   └── bucket_0
> ├── delta_001_001_
> │   ├── _orc_acid_version
> │   └── bucket_0
> ├── delta_001_002_v019
> │   ├── _orc_acid_version
> │   └── bucket_0
> ├── delta_001_002_v021
> │   ├── _orc_acid_version
> │   └── bucket_0
> └── delta_002_002_
>     ├── _orc_acid_version
>     └── bucket_0{noformat}
> i.e. 2 deltas with the same write ID range
> this is bad.  Probably happens today as well but new run produces a delta 
> with the same name and clobbers the previous one, which may interfere with 
> writers
>  
> need to investigate
>  
> -The issue (I think) is that {{AcidUtils.getAcidState()}} then returns both 
> deltas as if they were distinct and it effectively duplicates data.-  There 
> is no data duplication - {{getAcidState()}} will not use 2 deltas with the 
> same {{writeid}} range
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22593) Dynamically partitioned MM (insert-only ACID) tables don't compact automatically

2020-01-08 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010584#comment-17010584
 ] 

Hive QA commented on HIVE-22593:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12990176/HIVE-22593.04.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 17858 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/20106/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20106/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20106/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12990176 - PreCommit-HIVE-Build

> Dynamically partitioned MM (insert-only ACID) tables don't compact 
> automatically
> 
>
> Key: HIVE-22593
> URL: https://issues.apache.org/jira/browse/HIVE-22593
> Project: Hive
>  Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-22593.01.patch, HIVE-22593.01.patch, 
> HIVE-22593.02.patch, HIVE-22593.03.patch, HIVE-22593.04.patch, 
> HIVE-22593.04.patch
>
>
> Dynamic partitions of MM tables aren't entered into the HMS table 
> TXN_COMPONENTS. On inserting into such tables we see this line in the HMS log:
> {code:java}
> Expected to move at least one record from txn_components to 
> completed_txn_components when committing txn!{code}
> (This is not the case for non-partitioned MM tables.)
> Since the partitions aren't entered into COMPLETED_TXN_COMPONENTS, they 
> aren't considered for automatic compaction.
> Probably the culprit is 
> org.apache.hadoop.hive.ql.metadata.Hive#loadDynamicPartitions which has an 
> isAcid parameter that is always false regarding MM tables, and also because 
> MM tables' "write type" is AcidUtils.Operation.NOT_ACID and not INSERT.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction

2020-01-08 Thread Laszlo Pinter (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Pinter updated HIVE-20934:
-
Attachment: HIVE-20934.06.patch

> ACID: Query based compactor for minor compaction
> 
>
> Key: HIVE-20934
> URL: https://issues.apache.org/jira/browse/HIVE-20934
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.1.1
>Reporter: Vaibhav Gumashta
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, 
> HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, 
> HIVE-20934.06.patch
>
>
> Follow up of HIVE-20699. This is to enable running minor compactions as a 
> HiveQL query 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (HIVE-20948) Eliminate file rename in compactor

2020-01-08 Thread Peter Vary (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-20948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010561#comment-17010561
 ] 

Peter Vary edited comment on HIVE-20948 at 1/8/20 10:43 AM:


[~asomani]: Even after fixing HIVE-21164 the CRUD compactor has an extra move 
step which should be avoided if possible. CC: [~lpinter]


was (Author: pvary):
[~asomani]: Even after fixing HIVE-21164 the CRUD compactor has an extra move 
step which could be avoided if possible. 

> Eliminate file rename in compactor
> --
>
> Key: HIVE-20948
> URL: https://issues.apache.org/jira/browse/HIVE-20948
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Priority: Major
>
> Once HIVE-20823 is committed, we should investigate if it's possible to have 
> compactor write directly to base_x_cZ or delta_x_y_cZ.  
> For query based compaction: can we control location of temp table dir?  We 
> support external temp tables so this may work but we'd need to have non-acid 
> insert create files with {{bucket_x}} names.
>  
> For MR/Tez/LLAP based (should this be done at all?), need to figure out how 
> retries of tasks will work.  Just like we currently generate an MR job to 
> compact, we should be able to generate a Tez job.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-20948) Eliminate file rename in compactor

2020-01-08 Thread Peter Vary (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-20948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010561#comment-17010561
 ] 

Peter Vary commented on HIVE-20948:
---

[~asomani]: Even after fixing HIVE-21164 the CRUD compactor has an extra move 
step which could be avoided if possible. 

> Eliminate file rename in compactor
> --
>
> Key: HIVE-20948
> URL: https://issues.apache.org/jira/browse/HIVE-20948
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Priority: Major
>
> Once HIVE-20823 is committed, we should investigate if it's possible to have 
> compactor write directly to base_x_cZ or delta_x_y_cZ.  
> For query based compaction: can we control location of temp table dir?  We 
> support external temp tables so this may work but we'd need to have non-acid 
> insert create files with {{bucket_x}} names.
>  
> For MR/Tez/LLAP based (should this be done at all?), need to figure out how 
> retries of tasks will work.  Just like we currently generate an MR job to 
> compact, we should be able to generate a Tez job.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-20901) running compactor when there is nothing to do produces duplicate data

2020-01-08 Thread Peter Vary (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary updated HIVE-20901:
--
Fix Version/s: 4.0.0
   Resolution: Duplicate
   Status: Resolved  (was: Patch Available)

[~asomani]: If you do not mind I close this jira as it was fixed by HIVE-9995. 
Sorry for the confusion, I have found this jira only now :(

> running compactor when there is nothing to do produces duplicate data
> -
>
> Key: HIVE-20901
> URL: https://issues.apache.org/jira/browse/HIVE-20901
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Abhishek Somani
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20901.1.patch, HIVE-20901.2.patch
>
>
> suppose we run minor compaction 2 times, via alter table
> The 2nd request to compaction should have nothing to do but I don't think 
> there is a check for that.  It's visible in the context of HIVE-20823, where 
> each compactor run produces a delta with new visibility suffix so we end up 
> with something like
> {noformat}
> target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands3-1541810844849/warehouse/t/
> ├── delete_delta_001_002_v019
> │   ├── _orc_acid_version
> │   └── bucket_0
> ├── delete_delta_001_002_v021
> │   ├── _orc_acid_version
> │   └── bucket_0
> ├── delta_001_001_
> │   ├── _orc_acid_version
> │   └── bucket_0
> ├── delta_001_002_v019
> │   ├── _orc_acid_version
> │   └── bucket_0
> ├── delta_001_002_v021
> │   ├── _orc_acid_version
> │   └── bucket_0
> └── delta_002_002_
>     ├── _orc_acid_version
>     └── bucket_0{noformat}
> i.e. 2 deltas with the same write ID range
> this is bad.  Probably happens today as well but new run produces a delta 
> with the same name and clobbers the previous one, which may interfere with 
> writers
>  
> need to investigate
>  
> -The issue (I think) is that {{AcidUtils.getAcidState()}} then returns both 
> deltas as if they were distinct and it effectively duplicates data.-  There 
> is no data duplication - {{getAcidState()}} will not use 2 deltas with the 
> same {{writeid}} range
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22663) Quote all table and column names or do not quote any

2020-01-08 Thread Zoltan Chovan (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010552#comment-17010552
 ] 

Zoltan Chovan commented on HIVE-22663:
--

Sure, I'll create a PR. Unfortunately the failing tests seem to be related, I'm 
currently working on fixing those.

> Quote all table and column names or do not quote any
> 
>
> Key: HIVE-22663
> URL: https://issues.apache.org/jira/browse/HIVE-22663
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Zoltan Chovan
>Priority: Major
> Attachments: HIVE-22663.patch
>
>
> The change in HIVE-22546 is causing following stack trace when I run Hive 
> with PostgreSQL as backend db for the metastore.
> 0: jdbc:hive2://localhost:1> create database dumpdb with 
> ('repl.source.for'='1,2,3');0: jdbc:hive2://localhost:1> create database 
> dumpdb with ('repl.source.for'='1,2,3');Error: Error while compiling 
> statement: FAILED: ParseException line 1:28 missing KW_DBPROPERTIES at '(' 
> near '' (state=42000,code=4)0: jdbc:hive2://localhost:1> create 
> database dumpdb with dbproperties ('repl.source.for'='1,2,3');ERROR : FAILED: 
> Hive Internal Error: org.apache.hadoop.hive.ql.lockmgr.LockException(Error 
> communicating with the 
> metastore)org.apache.hadoop.hive.ql.lockmgr.LockException: Error 
> communicating with the metastore at 
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.commitTxn(DbTxnManager.java:541)
>  at 
> org.apache.hadoop.hive.ql.Driver.releaseLocksAndCommitOrRollback(Driver.java:687)
>  at 
> org.apache.hadoop.hive.ql.Driver.releaseLocksAndCommitOrRollback(Driver.java:653)
>  at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:969)
> ... stack trace clipped
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  at java.lang.Thread.run(Thread.java:748)Caused by: 
> MetaException(message:Unable to update transaction database 
> org.postgresql.util.PSQLException: ERROR: relation 
> "materialization_rebuild_locks" does not exist  Position: 13 at 
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440)
>  at 
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183)
>  at 
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308) 
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441) at 
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365) at 
> This happens because the table names in all the queries in TxnHandler.java 
> (including the one at 1312, which causes this stack trace) are not quoting 
> the table names. All the tablenames and column names should be quoted there. 
> Just the change in HIVE-22546 won't suffice.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22661) Compaction fails on non bucketed table with data loaded inpath

2020-01-08 Thread Jira



 [ 
https://issues.apache.org/jira/browse/HIVE-22661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ádám Szita updated HIVE-22661:
--
Fix Version/s: 4.0.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

Committed to master. Thanks for reviewing Laszlo and Peter.

> Compaction fails on non bucketed table with data loaded inpath
> --
>
> Key: HIVE-22661
> URL: https://issues.apache.org/jira/browse/HIVE-22661
> Project: Hive
>  Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22661.0.patch, HIVE-22661.1.patch, 
> HIVE-22661.2.patch
>
>
> Compaction cannot handle situations where:
>  * data was ingested with {{LOAD DATA INPATH}}
>  * this ingest method is run multiple times, and
>  ** with different number of files getting created in the delta directories
> Therefore, for file/dir structures such as:
> {code:java}
> /warehouse/tablespace/managed/hive/comp3/delta_001_001_
> /warehouse/tablespace/managed/hive/comp3/delta_001_001_/00_0
> /warehouse/tablespace/managed/hive/comp3/delta_001_001_/01_0
> /warehouse/tablespace/managed/hive/comp3/delta_002_002_
> /warehouse/tablespace/managed/hive/comp3/delta_002_002_/00_0
> /warehouse/tablespace/managed/hive/comp3/delta_002_002_/01_0
> /warehouse/tablespace/managed/hive/comp3/delta_002_002_/02_0 
> {code}
> Although the table is not bucketed, bucket is calculated from the (raw) 
> files' names. Compaction in the above case will fail on delta1-1 not having 
> data for 'bucket' 2.
> Steps to repro using small dataset:
> {code:java}
> set tez.grouping.min-size=8;
> set tez.grouping.max-size=8;
> set mapreduce.input.fileinputformat.split.minsize=8;
> set mapreduce.input.fileinputformat.split.minsize=8;
> create external table comp0 (a string);
> insert into comp0 values ("qwertyuiopasdfghjklzxcvbnm");
> insert into comp0 values ("qwertyuiopasdfghjklzxcvbnm");
> create external table comp1 stored as orc as select * from comp0;
> insert into comp0 values ("qwertyuiopasdfghjklzxcvbnm");
> create external table comp2 stored as orc as select * from comp0;
> create table comp3 (a string);
> load data inpath '/warehouse/tablespace/external/hive/comp1' into table comp3;
> load data inpath '/warehouse/tablespace/external/hive/comp2' into table 
> comp3;{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22663) Quote all table and column names or do not quote any

2020-01-08 Thread Ashutosh Bapat (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010541#comment-17010541
 ] 

Ashutosh Bapat commented on HIVE-22663:
---

I looked at the patch. The patch is huge so didn't go into the details. Can you 
please create a PR so that it's easy to review and provide comments on 
particular changes if necessary?

I have only one comment right now. Please handle the table names in a fashion 
similar to MetaStoreDirectSql.java. In this case, we might want to go a step 
further and handle column names in the same fashion.

> Quote all table and column names or do not quote any
> 
>
> Key: HIVE-22663
> URL: https://issues.apache.org/jira/browse/HIVE-22663
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Zoltan Chovan
>Priority: Major
> Attachments: HIVE-22663.patch
>
>
> The change in HIVE-22546 is causing following stack trace when I run Hive 
> with PostgreSQL as backend db for the metastore.
> 0: jdbc:hive2://localhost:1> create database dumpdb with 
> ('repl.source.for'='1,2,3');0: jdbc:hive2://localhost:1> create database 
> dumpdb with ('repl.source.for'='1,2,3');Error: Error while compiling 
> statement: FAILED: ParseException line 1:28 missing KW_DBPROPERTIES at '(' 
> near '' (state=42000,code=4)0: jdbc:hive2://localhost:1> create 
> database dumpdb with dbproperties ('repl.source.for'='1,2,3');ERROR : FAILED: 
> Hive Internal Error: org.apache.hadoop.hive.ql.lockmgr.LockException(Error 
> communicating with the 
> metastore)org.apache.hadoop.hive.ql.lockmgr.LockException: Error 
> communicating with the metastore at 
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.commitTxn(DbTxnManager.java:541)
>  at 
> org.apache.hadoop.hive.ql.Driver.releaseLocksAndCommitOrRollback(Driver.java:687)
>  at 
> org.apache.hadoop.hive.ql.Driver.releaseLocksAndCommitOrRollback(Driver.java:653)
>  at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:969)
> ... stack trace clipped
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  at java.lang.Thread.run(Thread.java:748)Caused by: 
> MetaException(message:Unable to update transaction database 
> org.postgresql.util.PSQLException: ERROR: relation 
> "materialization_rebuild_locks" does not exist  Position: 13 at 
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440)
>  at 
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183)
>  at 
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308) 
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441) at 
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365) at 
> This happens because the table names in all the queries in TxnHandler.java 
> (including the one at 1312, which causes this stack trace) are not quoting 
> the table names. All the tablenames and column names should be quoted there. 
> Just the change in HIVE-22546 won't suffice.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

1 2 >

1 - 100 of 108 matches

Mail list logo