[jira] [Commented] (HIVE-19488) enable CM root based on db parameter, identifying a db as source of replication.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486743#comment-16486743
 ] 

Hive QA commented on HIVE-19488:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
23s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
35s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
45s{color} | {color:blue} ql in master has 2322 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  2m 
45s{color} | {color:blue} standalone-metastore in master has 216 extant 
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
34s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
45s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
40s{color} | {color:red} ql: The patch generated 6 new + 465 unchanged - 1 
fixed = 471 total (was 466) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
21s{color} | {color:red} standalone-metastore: The patch generated 1 new + 462 
unchanged - 1 fixed = 463 total (was 463) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  7m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
11s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 24s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11149/dev-support/hive-personality.sh
 |
| git revision | master / ed95ee9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-11149/yetus/patch-mvninstall-itests_hive-unit.txt
 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-11149/yetus/patch-mvninstall-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-11149/yetus/diff-checkstyle-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-11149/yetus/diff-checkstyle-standalone-metastore.txt
 |
| modules | C: itests/hive-unit ql standalone-metastore U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-11149/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> enable CM root based on db parameter, identifying a db as source of 
> replication.
> 
>
> Key: 

[jira] [Updated] (HIVE-19488) enable CM root based on db parameter, identifying a db as source of replication.

[mahesh kumar behera (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera updated HIVE-19488:
---
Attachment: HIVE-19488.07.patch

> enable CM root based on db parameter, identifying a db as source of 
> replication.
> 
>
> Key: HIVE-19488
> URL: https://issues.apache.org/jira/browse/HIVE-19488
> Project: Hive
>  Issue Type: Task
>  Components: Hive, HiveServer2, repl
>Affects Versions: 3.1.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.1.0
>
> Attachments: HIVE-19488.01.patch, HIVE-19488.02.patch, 
> HIVE-19488.03.patch, HIVE-19488.04.patch, HIVE-19488.05.patch, 
> HIVE-19488.06.patch, HIVE-19488.07.patch
>
>
> * add a parameter at db level to identify if its a source of replication. 
> beacon will set this.
>  * Enable CM root only for databases that are a source of a replication 
> policy, for other db's skip the CM root functionality.
>  * prevent database drop if the parameter indicating its source of a 
> replication, is set.
>  * as an upgrade to this version, beacon should set the property on all 
> existing database policies, in affect.
>  * the parameter should be of the form . –  repl.source.for : List < policy 
> ids >



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19670) Correlated subquery in select produces result instead of throwing an error

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg reassigned HIVE-19670:
--


> Correlated subquery in select produces result instead of throwing an error 
> ---
>
> Key: HIVE-19670
> URL: https://issues.apache.org/jira/browse/HIVE-19670
> Project: Hive
>  Issue Type: Bug
>  Components: Logical Optimizer
>Affects Versions: 2.3.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
>
> {code:sql}
> create table friends (
>   id int not null,
>   friend_id int,
>   primary key (id),
>   foreign key (friend_id) references friends (id)
> );
> insert into friends values
> (210, null),
> (209, 210),
> (202, 209),
> (208, 202),
> (207, 209),
> (203, 207),
> (201, null),
> (204, null),
> (205, null),
> (206, 209);
> SELECT id,
>  (SELECT id
>   FROM friends as f2
>   WHERE f2.friend_id = f1.friend_id) AS friend
> FROM friends as f1
> WHERE id = '202'
> {code}
> Select query should throw an error since subquery will result in more than 
> one row. But hive produces following result:
> {noformat}
> 202   202
> 202   207
> 202   206
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19613) GenericUDTFGetSplits should handle fetch task with temp table rewrite

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486706#comment-16486706
 ] 

Hive QA commented on HIVE-19613:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12924493/HIVE-19613.3.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 14395 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/11148/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11148/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11148/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12924493 - PreCommit-HIVE-Build

> GenericUDTFGetSplits should handle fetch task with temp table rewrite
> -
>
> Key: HIVE-19613
> URL: https://issues.apache.org/jira/browse/HIVE-19613
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Eric Wohlstadter
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19613.1.patch, HIVE-19613.2.patch, 
> HIVE-19613.3.patch
>
>
> GenericUDTFGetSplits fails for fetch task only queries. Fetch task only 
> queries can be handled same way as >1 task queries using temp tables. 
> {code:java}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Was expecting a 
> single TezTask.
> at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:262)
> at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:201)
> at org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:984)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:930)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:917)
> at 
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:984)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:930)
> at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at 
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:492)
> at 
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:484)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
> ... 16 more{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19440) Make StorageBasedAuthorizer work with information schema

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-19440:
--
Attachment: HIVE-19440.7.patch

> Make StorageBasedAuthorizer work with information schema
> 
>
> Key: HIVE-19440
> URL: https://issues.apache.org/jira/browse/HIVE-19440
> Project: Hive
>  Issue Type: Improvement
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Blocker
> Fix For: 3.1.0
>
> Attachments: HIVE-19440.1.patch, HIVE-19440.2.patch, 
> HIVE-19440.3.patch, HIVE-19440.4.patch, HIVE-19440.5.patch, 
> HIVE-19440.6.patch, HIVE-19440.7.patch
>
>
> With HIVE-19161, Hive information schema works with external authorizer (such 
> as ranger). However, we also need to make StorageBasedAuthorizer 
> synchronization work as it is also widely use.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19440) Make StorageBasedAuthorizer work with information schema

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486690#comment-16486690
 ] 

Daniel Dai commented on HIVE-19440:
---

HIVE-19440.7.patch to fix ptest failures.

> Make StorageBasedAuthorizer work with information schema
> 
>
> Key: HIVE-19440
> URL: https://issues.apache.org/jira/browse/HIVE-19440
> Project: Hive
>  Issue Type: Improvement
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Blocker
> Fix For: 3.1.0
>
> Attachments: HIVE-19440.1.patch, HIVE-19440.2.patch, 
> HIVE-19440.3.patch, HIVE-19440.4.patch, HIVE-19440.5.patch, 
> HIVE-19440.6.patch, HIVE-19440.7.patch
>
>
> With HIVE-19161, Hive information schema works with external authorizer (such 
> as ranger). However, we also need to make StorageBasedAuthorizer 
> synchronization work as it is also widely use.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19613) GenericUDTFGetSplits should handle fetch task with temp table rewrite

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486685#comment-16486685
 ] 

Hive QA commented on HIVE-19613:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
44s{color} | {color:blue} ql in master has 2322 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 14s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11148/dev-support/hive-personality.sh
 |
| git revision | master / ed95ee9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-11148/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> GenericUDTFGetSplits should handle fetch task with temp table rewrite
> -
>
> Key: HIVE-19613
> URL: https://issues.apache.org/jira/browse/HIVE-19613
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Eric Wohlstadter
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19613.1.patch, HIVE-19613.2.patch, 
> HIVE-19613.3.patch
>
>
> GenericUDTFGetSplits fails for fetch task only queries. Fetch task only 
> queries can be handled same way as >1 task queries using temp tables. 
> {code:java}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Was expecting a 
> single TezTask.
> at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:262)
> at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:201)
> at org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:984)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:930)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:917)
> at 
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:984)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:930)
> at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at 
> 

[jira] [Commented] (HIVE-19588) Several invocation of file listing when creating VectorizedOrcAcidRowBatchReader

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486662#comment-16486662
 ] 

Hive QA commented on HIVE-19588:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12924492/HIVE-19588.3.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14396 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[parallel_orderby] 
(batchId=94)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/11147/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11147/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11147/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12924492 - PreCommit-HIVE-Build

> Several invocation of file listing when creating 
> VectorizedOrcAcidRowBatchReader
> 
>
> Key: HIVE-19588
> URL: https://issues.apache.org/jira/browse/HIVE-19588
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Nita Dembla
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19588.1.patch, HIVE-19588.2.patch, 
> HIVE-19588.3.patch, Screen Shot 2018-05-16 at 2.23.25 PM.png
>
>
> Looks like we are doing file listing several times when creating one instance 
> of VectorizedOrcAcidRowBatchReader
>  AcidUtils.parseBaseOrDeltaBucketFilename() does full file listing (when 
> there are files with bucket_* prefix) just to get a single file out of a path 
> to figure out if it has ACID schema (as part of HIVE-18190).
>  There is full file listing where we populate
>  1) ColumnizedDeleteEventRegistry
>  2) SortMergedDeleteEventRegistry
>  3) Twice in computeOffsetAndBucket()
>  
> Attaching profiles which [~gopalv] took while debugging. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19588) Several invocation of file listing when creating VectorizedOrcAcidRowBatchReader

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486647#comment-16486647
 ] 

Hive QA commented on HIVE-19588:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
40s{color} | {color:blue} llap-server in master has 86 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
47s{color} | {color:blue} ql in master has 2322 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
19s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} llap-server: The patch generated 0 new + 28 
unchanged - 1 fixed = 28 total (was 29) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
44s{color} | {color:red} ql: The patch generated 18 new + 585 unchanged - 5 
fixed = 603 total (was 590) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 26s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11147/dev-support/hive-personality.sh
 |
| git revision | master / ed95ee9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-11147/yetus/patch-mvninstall-llap-server.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-11147/yetus/diff-checkstyle-ql.txt
 |
| modules | C: llap-server ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-11147/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Several invocation of file listing when creating 
> VectorizedOrcAcidRowBatchReader
> 
>
> Key: HIVE-19588
> URL: https://issues.apache.org/jira/browse/HIVE-19588
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Nita Dembla
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19588.1.patch, HIVE-19588.2.patch, 
> HIVE-19588.3.patch, Screen Shot 2018-05-16 at 2.23.25 PM.png
>
>
> Looks like we are doing file listing several times when creating one instance 
> of VectorizedOrcAcidRowBatchReader
>  

[jira] [Commented] (HIVE-19454) Test failure : org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion1 fails with java.lang.AssertionError

[Vineet Garg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486639#comment-16486639
 ] 

Vineet Garg commented on HIVE-19454:


[~djaiswal] This shouldn't be committed to master (or branch-3) since it 
doesn't have green run.

> Test failure : 
> org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion1 fails 
> with java.lang.AssertionError
> --
>
> Key: HIVE-19454
> URL: https://issues.apache.org/jira/browse/HIVE-19454
> Project: Hive
>  Issue Type: Bug
> Environment: uname -a
> Linux pts00433-vm40 3.10.0-862.el7.ppc64le #1 SMP Wed Mar 21 22:20:19 UTC 
> 2018 ppc64le ppc64le ppc64le GNU/Linux
>Reporter: Alisha Prabhu
>Priority: Major
> Attachments: HIVE-19454.1.patch, HIVE-19454.2.patch
>
>
> org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion1
>  refers to the same test case.
> Maven command used is : mvn -Dtest=TestTxnCommands2 test
> Error:
> {code:java}
> [INFO] Running org.apache.hadoop.hive.ql.TestTxnCommands2
> [ERROR] Tests run: 44, Failures: 1, Errors: 0, Skipped: 3, Time elapsed: 
> 618.215 s <<< FAILURE! - in org.apache.hadoop.hive.ql.TestTxnCommands2
> [ERROR] 
> testNonAcidToAcidConversion1(org.apache.hadoop.hive.ql.TestTxnCommands2)  
> Time elapsed: 17.557 s  <<< FAILURE!
> java.lang.AssertionError
>     at org.junit.Assert.fail(Assert.java:86)
>     at org.junit.Assert.assertTrue(Assert.java:41)
>     at org.junit.Assert.assertTrue(Assert.java:52)
>     at 
> org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion1(TestTxnCommands2.java:499)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>     at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>     at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>     at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>     at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>     at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>     at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
>     at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
>     at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>     at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>     at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>     at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>     at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>     at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>     at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>     at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>     at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>     at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>     at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
>     at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
>     at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
>     at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
>     at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
>     at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
>     at 
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
>     at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Failures:
> [ERROR]   TestTxnCommands2.testNonAcidToAcidConversion1:499
> [INFO]
> [ERROR] Tests run: 44, Failures: 1, Errors: 0, Skipped: 3
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HIVE-19665) Update version to 3.0.1-SNAPSHOT in branch-3.0

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg resolved HIVE-19665.

Resolution: Fixed

> Update version to 3.0.1-SNAPSHOT in branch-3.0
> --
>
> Key: HIVE-19665
> URL: https://issues.apache.org/jira/browse/HIVE-19665
> Project: Hive
>  Issue Type: Task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19665.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19578) HLL merges tempList on every add

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486623#comment-16486623
 ] 

Hive QA commented on HIVE-19578:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12924495/HIVE-19578.3.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 28 failed/errored test(s), 14395 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_1]
 (batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[limit_pushdown3]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[limit_pushdown]
 (batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_vector_nohybridgrace]
 (batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mapjoin_decimal]
 (batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[offset_limit_ppd_optimizer]
 (batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[semijoin_hint]
 (batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_dynpart_hashjoin_2]
 (batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_vector_dynpart_hashjoin_2]
 (batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_left_outer_join]
 (batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_limit]
 (batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_short_regress]
 (batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_context]
 (batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_distinct_gby]
 (batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_mapjoin]
 (batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_nested_mapjoin]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_shufflejoin]
 (batchId=173)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[hybridgrace_hashjoin_1]
 (batchId=106)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2 
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitions
 (batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert
 (batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
 (batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles 
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent 
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomReadOps 
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead 
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes 
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerVertexRawInputSplitsNoKill
 (batchId=241)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/11146/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11146/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11146/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 28 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12924495 - PreCommit-HIVE-Build

> HLL merges tempList on every add
> 
>
> Key: HIVE-19578
> URL: https://issues.apache.org/jira/browse/HIVE-19578
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19578.1.patch, HIVE-19578.2.patch, 
> HIVE-19578.3.patch, Screen Shot 2018-05-16 at 15.29.12 .png
>
>
>  See comments on HIVE-18866; this has significant perf overhead after the 
> even bigger overhead from hashing is removed.  !Screen Shot 2018-05-16 at 
> 15.29.12 .png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19667) Remove distribution management tag from pom.xml

[Vineet Garg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486622#comment-16486622
 ] 

Vineet Garg commented on HIVE-19667:


[~ashutoshc] can you take a look?

> Remove distribution management tag from pom.xml
> ---
>
> Key: HIVE-19667
> URL: https://issues.apache.org/jira/browse/HIVE-19667
> Project: Hive
>  Issue Type: Task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19667.1.patch
>
>
> This tag overrides apaches configuration in maven settings file and makes it 
> impossible to publish maven artifacts. There is no way around to it either.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19450) OOM due to map join and backup task not invoked

[zhuwei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhuwei updated HIVE-19450:
--
Status: Patch Available  (was: Open)

add null pointer check

> OOM due to map join and backup task not invoked
> ---
>
> Key: HIVE-19450
> URL: https://issues.apache.org/jira/browse/HIVE-19450
> Project: Hive
>  Issue Type: Bug
>Reporter: zhuwei
>Assignee: zhuwei
>Priority: Major
> Attachments: HIVE-19450.1.patch, HIVE-19450.2.patch
>
>
> Map join task may cause OOM due to orc compression , in most cases , a backup 
> task will be invoked. However , if the size of hash table is close to memory 
> limit, the task which load the hash table will NOT fail . OOM will happen in 
> next task witch do local join. The load task has a backup but next task not. 
> So in this case , the whole query will fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19450) OOM due to map join and backup task not invoked

[zhuwei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhuwei updated HIVE-19450:
--
Attachment: HIVE-19450.2.patch

> OOM due to map join and backup task not invoked
> ---
>
> Key: HIVE-19450
> URL: https://issues.apache.org/jira/browse/HIVE-19450
> Project: Hive
>  Issue Type: Bug
>Reporter: zhuwei
>Assignee: zhuwei
>Priority: Major
> Attachments: HIVE-19450.1.patch, HIVE-19450.2.patch
>
>
> Map join task may cause OOM due to orc compression , in most cases , a backup 
> task will be invoked. However , if the size of hash table is close to memory 
> limit, the task which load the hash table will NOT fail . OOM will happen in 
> next task witch do local join. The load task has a backup but next task not. 
> So in this case , the whole query will fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19450) OOM due to map join and backup task not invoked

[zhuwei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhuwei updated HIVE-19450:
--
Status: Open  (was: Patch Available)

> OOM due to map join and backup task not invoked
> ---
>
> Key: HIVE-19450
> URL: https://issues.apache.org/jira/browse/HIVE-19450
> Project: Hive
>  Issue Type: Bug
>Reporter: zhuwei
>Assignee: zhuwei
>Priority: Major
> Attachments: HIVE-19450.1.patch
>
>
> Map join task may cause OOM due to orc compression , in most cases , a backup 
> task will be invoked. However , if the size of hash table is close to memory 
> limit, the task which load the hash table will NOT fail . OOM will happen in 
> next task witch do local join. The load task has a backup but next task not. 
> So in this case , the whole query will fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19577) CREATE TEMPORARY TABLE LIKE and INSERT generate output format mismatch errors

[Steve Yeom (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486605#comment-16486605
 ] 

Steve Yeom commented on HIVE-19577:
---

One test failure with age 1 of the previous run is clear with my environment. 

> CREATE TEMPORARY TABLE LIKE  and INSERT generate output format mismatch errors
> --
>
> Key: HIVE-19577
> URL: https://issues.apache.org/jira/browse/HIVE-19577
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19577.01.patch, HIVE-19577.02.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19577) CREATE TEMPORARY TABLE LIKE and INSERT generate output format mismatch errors

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486600#comment-16486600
 ] 

Jason Dere commented on HIVE-19577:
---

+1

> CREATE TEMPORARY TABLE LIKE  and INSERT generate output format mismatch errors
> --
>
> Key: HIVE-19577
> URL: https://issues.apache.org/jira/browse/HIVE-19577
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19577.01.patch, HIVE-19577.02.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19577) CREATE TEMPORARY TABLE LIKE and INSERT generate output format mismatch errors

[Steve Yeom (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Yeom updated HIVE-19577:
--
Attachment: HIVE-19577.02.patch

> CREATE TEMPORARY TABLE LIKE  and INSERT generate output format mismatch errors
> --
>
> Key: HIVE-19577
> URL: https://issues.apache.org/jira/browse/HIVE-19577
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19577.01.patch, HIVE-19577.02.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19578) HLL merges tempList on every add

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486592#comment-16486592
 ] 

Hive QA commented on HIVE-19578:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  2m 
55s{color} | {color:blue} standalone-metastore in master has 216 extant 
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
17s{color} | {color:red} standalone-metastore: The patch generated 2 new + 19 
unchanged - 2 fixed = 21 total (was 21) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m  
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 23s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11146/dev-support/hive-personality.sh
 |
| git revision | master / ed95ee9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-11146/yetus/diff-checkstyle-standalone-metastore.txt
 |
| modules | C: standalone-metastore U: standalone-metastore |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-11146/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> HLL merges tempList on every add
> 
>
> Key: HIVE-19578
> URL: https://issues.apache.org/jira/browse/HIVE-19578
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19578.1.patch, HIVE-19578.2.patch, 
> HIVE-19578.3.patch, Screen Shot 2018-05-16 at 15.29.12 .png
>
>
>  See comments on HIVE-18866; this has significant perf overhead after the 
> even bigger overhead from hashing is removed.  !Screen Shot 2018-05-16 at 
> 15.29.12 .png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19666) SQL standard auth for create fn may make an impossible privilege check (branch-2

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19666:

Summary: SQL standard auth for create fn may make an impossible privilege 
check (branch-2  (was: SQL standard auth for create fn may make an impossible 
privilege check)

> SQL standard auth for create fn may make an impossible privilege check 
> (branch-2
> 
>
> Key: HIVE-19666
> URL: https://issues.apache.org/jira/browse/HIVE-19666
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19666.01-branch-2.patch, HIVE-19666.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19666) SQL standard auth for create fn may make an impossible privilege check

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486570#comment-16486570
 ] 

Sergey Shelukhin commented on HIVE-19666:
-

added a test based on external discussion; also targeting this to branch-2

> SQL standard auth for create fn may make an impossible privilege check
> --
>
> Key: HIVE-19666
> URL: https://issues.apache.org/jira/browse/HIVE-19666
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19666.01-branch-2.patch, HIVE-19666.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19666) SQL standard auth for create fn may make an impossible privilege check (branch-2)

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19666:

Summary: SQL standard auth for create fn may make an impossible privilege 
check (branch-2)  (was: SQL standard auth for create fn may make an impossible 
privilege check (branch-2)

> SQL standard auth for create fn may make an impossible privilege check 
> (branch-2)
> -
>
> Key: HIVE-19666
> URL: https://issues.apache.org/jira/browse/HIVE-19666
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19666.01-branch-2.patch, HIVE-19666.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19666) SQL standard auth for create fn may make an impossible privilege check

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19666:

Attachment: HIVE-19666.01-branch-2.patch

> SQL standard auth for create fn may make an impossible privilege check
> --
>
> Key: HIVE-19666
> URL: https://issues.apache.org/jira/browse/HIVE-19666
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19666.01-branch-2.patch, HIVE-19666.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19465) Upgrade ORC to 1.5.0

[Jesus Camacho Rodriguez (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486568#comment-16486568
 ] 

Jesus Camacho Rodriguez commented on HIVE-19465:


[~prasanth_j], I have uploaded a new patch that disables the failing tests due 
to ORC-367 (left a note to reenable in HIVE-19669) and changes the version in 
standalone-metastore.

> Upgrade ORC to 1.5.0
> 
>
> Key: HIVE-19465
> URL: https://issues.apache.org/jira/browse/HIVE-19465
> Project: Hive
>  Issue Type: Improvement
>  Components: File Formats
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Blocker
> Attachments: HIVE-19465.01.patch, HIVE-19465.02.patch, 
> HIVE-19465.03.patch, HIVE-19465.04.patch, HIVE-19465.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19614) GenericUDTFGetSplits does not honor ORDER BY

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486566#comment-16486566
 ] 

Hive QA commented on HIVE-19614:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12924494/HIVE-19614.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 14395 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/11145/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11145/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11145/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12924494 - PreCommit-HIVE-Build

> GenericUDTFGetSplits does not honor ORDER BY
> 
>
> Key: HIVE-19614
> URL: https://issues.apache.org/jira/browse/HIVE-19614
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Eric Wohlstadter
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19614.1.patch, HIVE-19614.2.patch
>
>
> GenericUDTFGetSplits handles ORDER BY by writing the results to temp table. 
> However running select * on that temp table may create >1 splits which will 
> lose the original ordering. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19465) Upgrade ORC to 1.5.0

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-19465:
---
Attachment: HIVE-19465.04.patch

> Upgrade ORC to 1.5.0
> 
>
> Key: HIVE-19465
> URL: https://issues.apache.org/jira/browse/HIVE-19465
> Project: Hive
>  Issue Type: Improvement
>  Components: File Formats
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Blocker
> Attachments: HIVE-19465.01.patch, HIVE-19465.02.patch, 
> HIVE-19465.03.patch, HIVE-19465.04.patch, HIVE-19465.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18973) Make transaction system work with catalogs

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486559#comment-16486559
 ] 

Alan Gates commented on HIVE-18973:
---

I screwed up the automatic linking to github, but the link to the pull request 
is above under External issue URL.  [~ekoifman] if you can provide any feedback 
on this it would be much appreciated.

> Make transaction system work with catalogs
> --
>
> Key: HIVE-18973
> URL: https://issues.apache.org/jira/browse/HIVE-18973
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
>
> The transaction tables need to understand catalogs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18973) Make transaction system work with catalogs

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-18973:
--
External issue URL: https://github.com/apache/hive/pull/353
Labels: pull-request-available  (was: )

> Make transaction system work with catalogs
> --
>
> Key: HIVE-18973
> URL: https://issues.apache.org/jira/browse/HIVE-18973
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
>
> The transaction tables need to understand catalogs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19465) Upgrade ORC to 1.5.0

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486548#comment-16486548
 ] 

Prasanth Jayachandran commented on HIVE-19465:
--

Should standalone-metastore/pom.xml also be updated to reflect the new version?

> Upgrade ORC to 1.5.0
> 
>
> Key: HIVE-19465
> URL: https://issues.apache.org/jira/browse/HIVE-19465
> Project: Hive
>  Issue Type: Improvement
>  Components: File Formats
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Blocker
> Attachments: HIVE-19465.01.patch, HIVE-19465.02.patch, 
> HIVE-19465.03.patch, HIVE-19465.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19577) CREATE TEMPORARY TABLE LIKE and INSERT generate output format mismatch errors

[Jesus Camacho Rodriguez (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486547#comment-16486547
 ] 

Jesus Camacho Rodriguez commented on HIVE-19577:


[~steveyeom2017], to avoid any flakiness, I would avoid writing to {{/tmp}} and 
instead I would write to a custom folder for this test.

> CREATE TEMPORARY TABLE LIKE  and INSERT generate output format mismatch errors
> --
>
> Key: HIVE-19577
> URL: https://issues.apache.org/jira/browse/HIVE-19577
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19577.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18685) Add catalogs to metastore

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-18685:
--
Labels: pull-request-available  (was: )

> Add catalogs to metastore
> -
>
> Key: HIVE-18685
> URL: https://issues.apache.org/jira/browse/HIVE-18685
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore, Parser, Security, SQL
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Attachments: HMS Catalog Design Doc.pdf
>
>
> SQL supports two levels of namespaces, called in the spec catalogs and 
> schemas (with schema being equivalent to Hive's database).  I propose to add 
> the upper level of catalog.  The attached design doc covers the use cases, 
> requirements, and brief discussion of how it will be implemented in a 
> backwards compatible way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18685) Add catalogs to metastore

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486543#comment-16486543
 ] 

ASF GitHub Bot commented on HIVE-18685:
---

GitHub user alanfgates opened a pull request:

https://github.com/apache/hive/pull/353

HIVE-18685



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alanfgates/hive hive18973

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/353.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #353


commit bfec5838be14192fee6af60c21916b75e869db2a
Author: Alan Gates 
Date:   2018-04-26T21:10:14Z

HIVE-19323 Create metastore SQL install and upgrade scripts for 3.1

commit cfa777cc7d428074da96591b1a1b18cc537c519e
Author: Alan Gates 
Date:   2018-04-26T23:27:18Z

HIVE-18685 Make transaction system work with catalogs




> Add catalogs to metastore
> -
>
> Key: HIVE-18685
> URL: https://issues.apache.org/jira/browse/HIVE-18685
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore, Parser, Security, SQL
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Attachments: HMS Catalog Design Doc.pdf
>
>
> SQL supports two levels of namespaces, called in the spec catalogs and 
> schemas (with schema being equivalent to Hive's database).  I propose to add 
> the upper level of catalog.  The attached design doc covers the use cases, 
> requirements, and brief discussion of how it will be implemented in a 
> backwards compatible way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18973) Make transaction system work with catalogs

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486541#comment-16486541
 ] 

Alan Gates commented on HIVE-18973:
---

As a side comment, as part of this I fixed issues where the short version of 
Hive name hadn't been updated to 3.1.  Also discovered that the wrong version 
of upgrade.order files were being used in tests and in the binary distribution 
and fixed that as well.  This doesn't affect 3.0 as the two sets were identical 
in that release.

> Make transaction system work with catalogs
> --
>
> Key: HIVE-18973
> URL: https://issues.apache.org/jira/browse/HIVE-18973
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>
> The transaction tables need to understand catalogs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19631) reduce epic locking in AbstractService

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486536#comment-16486536
 ] 

Sergey Shelukhin commented on HIVE-19631:
-

Updated hiveConf related methods. I think the other ones can be made 
non-synchronized separately to reduce the scope of the fix, I'm not sure about 
correctness in their case, i.e. about getting tokens from metastore in parallel.

> reduce epic locking in AbstractService
> --
>
> Key: HIVE-19631
> URL: https://issues.apache.org/jira/browse/HIVE-19631
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19631.01.patch, HIVE-19631.patch
>
>
> Some services have synchronized method that do lengthy remote calls, and 
> these block everyone trying to e.g. getHiveConf. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19669) Upgrade ORC to 1.5.1

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez reassigned HIVE-19669:
--


> Upgrade ORC to 1.5.1
> 
>
> Key: HIVE-19669
> URL: https://issues.apache.org/jira/browse/HIVE-19669
> Project: Hive
>  Issue Type: Improvement
>  Components: File Formats
>Affects Versions: 3.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19631) reduce epic locking in AbstractService

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19631:

Attachment: HIVE-19631.01.patch

> reduce epic locking in AbstractService
> --
>
> Key: HIVE-19631
> URL: https://issues.apache.org/jira/browse/HIVE-19631
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19631.01.patch, HIVE-19631.patch
>
>
> Some services have synchronized method that do lengthy remote calls, and 
> these block everyone trying to e.g. getHiveConf. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19418) add background stats updater similar to compactor

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19418:

Attachment: (was: HIVE-19418.02.patch)

> add background stats updater similar to compactor
> -
>
> Key: HIVE-19418
> URL: https://issues.apache.org/jira/browse/HIVE-19418
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19418.01.patch, HIVE-19418.02.patch, 
> HIVE-19418.patch
>
>
> There's a JIRA HIVE-19416 to add snapshot version to stats for MM/ACID tables 
> to make them usable in a transaction without breaking ACID (for metadata-only 
> optimization). However, stats for ACID tables can still become unusable if 
> e.g. two parallel inserts run - neither sees the data written by the other, 
> so after both finish, the snapshots on either set of stats won't match the 
> current snapshot and the stats will be unusable.
> Additionally, for ACID and non-ACID tables alike, a lot of the stats, with 
> some exceptions like numRows, cannot be aggregated (i.e. you cannot combine 
> ndvs from two inserts), and for ACID even less can be aggregated (you cannot 
> derive min/max if some rows are deleted but you don't scan the rest of the 
> dataset).
> Therefore we will add background logic to metastore (similar to, and 
> partially inside, the ACID compactor) to update stats.
> It will have 3 modes of operation.
> 1) Off.
> 2) Update only the stats that exist but are out of date (generating stats can 
> be expensive, so if the user is only analyzing a subset of tables it should 
> be able to only update that subset). We can simply look at existing stats and 
> only analyze for the relevant partitions and columns.
> 3) On: 2 + create stats for all tables and columns missing stats.
> There will also be a table parameter to skip stats update. 
> In phase 1, the process will operate outside of compactor, and run analyze 
> command on the table. The analyze command will automatically save the stats 
> with ACID snapshot information if needed, based on HIVE-19416, so we don't 
> need to do any special state management and this will work for all table 
> types. However it's also more expensive.
> In phase 2, we can explore adding stats collection during MM compaction that 
> uses a temp table. If we don't have open writers during major compaction (so 
> we overwrite all of the data), the temp table stats can simply be copied over 
> to the main table with correct snapshot information, saving us a table scan.
> In phase 3, we can add custom stats collection logic to full ACID compactor 
> that is not query based, the same way as we'd do for (2). Alternatively we 
> can wait for ACID compactor to become query based and just reuse (2).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19418) add background stats updater similar to compactor

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486532#comment-16486532
 ] 

Sergey Shelukhin commented on HIVE-19418:
-

Rebased the patch (no conflicts, just some offset changes) to run HiveQA again

> add background stats updater similar to compactor
> -
>
> Key: HIVE-19418
> URL: https://issues.apache.org/jira/browse/HIVE-19418
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19418.01.patch, HIVE-19418.02.patch, 
> HIVE-19418.patch
>
>
> There's a JIRA HIVE-19416 to add snapshot version to stats for MM/ACID tables 
> to make them usable in a transaction without breaking ACID (for metadata-only 
> optimization). However, stats for ACID tables can still become unusable if 
> e.g. two parallel inserts run - neither sees the data written by the other, 
> so after both finish, the snapshots on either set of stats won't match the 
> current snapshot and the stats will be unusable.
> Additionally, for ACID and non-ACID tables alike, a lot of the stats, with 
> some exceptions like numRows, cannot be aggregated (i.e. you cannot combine 
> ndvs from two inserts), and for ACID even less can be aggregated (you cannot 
> derive min/max if some rows are deleted but you don't scan the rest of the 
> dataset).
> Therefore we will add background logic to metastore (similar to, and 
> partially inside, the ACID compactor) to update stats.
> It will have 3 modes of operation.
> 1) Off.
> 2) Update only the stats that exist but are out of date (generating stats can 
> be expensive, so if the user is only analyzing a subset of tables it should 
> be able to only update that subset). We can simply look at existing stats and 
> only analyze for the relevant partitions and columns.
> 3) On: 2 + create stats for all tables and columns missing stats.
> There will also be a table parameter to skip stats update. 
> In phase 1, the process will operate outside of compactor, and run analyze 
> command on the table. The analyze command will automatically save the stats 
> with ACID snapshot information if needed, based on HIVE-19416, so we don't 
> need to do any special state management and this will work for all table 
> types. However it's also more expensive.
> In phase 2, we can explore adding stats collection during MM compaction that 
> uses a temp table. If we don't have open writers during major compaction (so 
> we overwrite all of the data), the temp table stats can simply be copied over 
> to the main table with correct snapshot information, saving us a table scan.
> In phase 3, we can add custom stats collection logic to full ACID compactor 
> that is not query based, the same way as we'd do for (2). Alternatively we 
> can wait for ACID compactor to become query based and just reuse (2).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19418) add background stats updater similar to compactor

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19418:

Attachment: HIVE-19418.02.patch

> add background stats updater similar to compactor
> -
>
> Key: HIVE-19418
> URL: https://issues.apache.org/jira/browse/HIVE-19418
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19418.01.patch, HIVE-19418.02.patch, 
> HIVE-19418.02.patch, HIVE-19418.patch
>
>
> There's a JIRA HIVE-19416 to add snapshot version to stats for MM/ACID tables 
> to make them usable in a transaction without breaking ACID (for metadata-only 
> optimization). However, stats for ACID tables can still become unusable if 
> e.g. two parallel inserts run - neither sees the data written by the other, 
> so after both finish, the snapshots on either set of stats won't match the 
> current snapshot and the stats will be unusable.
> Additionally, for ACID and non-ACID tables alike, a lot of the stats, with 
> some exceptions like numRows, cannot be aggregated (i.e. you cannot combine 
> ndvs from two inserts), and for ACID even less can be aggregated (you cannot 
> derive min/max if some rows are deleted but you don't scan the rest of the 
> dataset).
> Therefore we will add background logic to metastore (similar to, and 
> partially inside, the ACID compactor) to update stats.
> It will have 3 modes of operation.
> 1) Off.
> 2) Update only the stats that exist but are out of date (generating stats can 
> be expensive, so if the user is only analyzing a subset of tables it should 
> be able to only update that subset). We can simply look at existing stats and 
> only analyze for the relevant partitions and columns.
> 3) On: 2 + create stats for all tables and columns missing stats.
> There will also be a table parameter to skip stats update. 
> In phase 1, the process will operate outside of compactor, and run analyze 
> command on the table. The analyze command will automatically save the stats 
> with ACID snapshot information if needed, based on HIVE-19416, so we don't 
> need to do any special state management and this will work for all table 
> types. However it's also more expensive.
> In phase 2, we can explore adding stats collection during MM compaction that 
> uses a temp table. If we don't have open writers during major compaction (so 
> we overwrite all of the data), the temp table stats can simply be copied over 
> to the main table with correct snapshot information, saving us a table scan.
> In phase 3, we can add custom stats collection logic to full ACID compactor 
> that is not query based, the same way as we'd do for (2). Alternatively we 
> can wait for ACID compactor to become query based and just reuse (2).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19660) update branch-3 to be version 3.1 and fix storage-api mismatch

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19660:

Description: 
Currently we have branch-3.0 presumably for 3.0.X releases, but branch-3 is 
marked as 3.0 and master as 3.1-SNAPSHOT.
Also storage-api features break the build on branch-3 because storage-api 
version does not match.

  was:
Currently we have branch-3.0 for presumably branch-3.0.X, but branch-3 is 
marked as 3.0 and master as 3.1-SNAPSHOT.
Also storage-api features break the build on branch-3 because storage-api 
version does not match.


> update branch-3 to be version 3.1 and fix storage-api mismatch
> --
>
> Key: HIVE-19660
> URL: https://issues.apache.org/jira/browse/HIVE-19660
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19660-branch-3.patch
>
>
> Currently we have branch-3.0 presumably for 3.0.X releases, but branch-3 is 
> marked as 3.0 and master as 3.1-SNAPSHOT.
> Also storage-api features break the build on branch-3 because storage-api 
> version does not match.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19614) GenericUDTFGetSplits does not honor ORDER BY

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486531#comment-16486531
 ] 

Hive QA commented on HIVE-19614:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
49s{color} | {color:blue} ql in master has 2322 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 20s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11145/dev-support/hive-personality.sh
 |
| git revision | master / ed95ee9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-11145/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> GenericUDTFGetSplits does not honor ORDER BY
> 
>
> Key: HIVE-19614
> URL: https://issues.apache.org/jira/browse/HIVE-19614
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Eric Wohlstadter
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19614.1.patch, HIVE-19614.2.patch
>
>
> GenericUDTFGetSplits handles ORDER BY by writing the results to temp table. 
> However running select * on that temp table may create >1 splits which will 
> lose the original ordering. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19643) MM table conversion doesn't need full ACID structure checks

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19643:

Attachment: HIVE-19643.01.patch

> MM table conversion doesn't need full ACID structure checks
> ---
>
> Key: HIVE-19643
> URL: https://issues.apache.org/jira/browse/HIVE-19643
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Jason Dere
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19643.01.patch, HIVE-19643.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19643) MM table conversion doesn't need full ACID structure checks

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486529#comment-16486529
 ] 

Sergey Shelukhin commented on HIVE-19643:
-

Rebased the patch

> MM table conversion doesn't need full ACID structure checks
> ---
>
> Key: HIVE-19643
> URL: https://issues.apache.org/jira/browse/HIVE-19643
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Jason Dere
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19643.01.patch, HIVE-19643.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19668) 11.8% of the heap wasted due to duplicate org.antlr.runtime.CommonToken's

[Misha Dmitriev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Misha Dmitriev reassigned HIVE-19668:
-


> 11.8% of the heap wasted due to duplicate org.antlr.runtime.CommonToken's
> -
>
> Key: HIVE-19668
> URL: https://issues.apache.org/jira/browse/HIVE-19668
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: image-2018-05-22-17-41-39-572.png
>
>
> I've recently analyzed a HS2 heap dump, obtained when there was a huge memory 
> spike during compilation of some big query. The analysis was done with jxray 
> ([www.jxray.com).|http://www.jxray.com)./] It turns out that more than 90% of 
> the 20G heap was used by data structures associated with query parsing 
> ({{org.apache.hadoop.hive.ql.parse.QBExpr}}). There are probably multiple 
> opportunities for optimizations here. One of them is to stop the code from 
> creating duplicate instances of {{org.antlr.runtime.CommonToken}} class. See 
> a sample of these objects in the attached image:
> !image-2018-05-22-17-41-39-572.png|width=879,height=399!
> Looks like these particular {{CommonToken}} objects are constants, that don't 
> change once created. I see some code, e.g. in 
> {{org.apache.hadoop.hive.ql.parse.CalcitePlanner}}, where such objects are 
> apparently repeatedly created with e.g. {{new 
> CommonToken(HiveParser.TOK_INSERT, "TOK_INSERT")}} If these 33 token kinds 
> are instead created once and reused, we will save more than 1/10th of the 
> heap in this scenario. Plus, since these objects are small but very numerous, 
> getting rid of them will remove a gread deal of pressure from the GC.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19586) Optimize Count(distinct X) pushdown based on the storage capabilities

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486521#comment-16486521
 ] 

Ashutosh Chauhan commented on HIVE-19586:
-

[~bslim] Can you please reupload the patch for Hive QA run.

> Optimize Count(distinct X) pushdown based on the storage capabilities 
> --
>
> Key: HIVE-19586
> URL: https://issues.apache.org/jira/browse/HIVE-19586
> Project: Hive
>  Issue Type: Improvement
>  Components: Druid integration, Logical Optimizer
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19586.2.patch, HIVE-19586.3.patch, HIVE-19586.patch
>
>
> h1. Goal
> Provide a way to rewrite queries with combination of COUNT(Distinct) and 
> Aggregates like SUM as a series of Group By.
> This can be useful to push down to Druid queries like 
> {code}
>  select count(DISTINCT interval_marker), count (distinct dim), sum(num_l) 
> FROM druid_test_table GROUP  BY `__time`, `zone` ;
> {code}
> In general this can be useful to be used in cases where storage handlers can 
> not perform count (distinct column)
> h1. How to do it.
> Use the Calcite rule {code} 
> org.apache.calcite.rel.rules.AggregateExpandDistinctAggregatesRule{code} that 
> breaks down Count distinct to a single Group by with Grouping sets or 
> multiple series of Group by that might be linked with Joins if multiple 
> counts are present.
> FYI today Hive does have a similar rule {code} 
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveExpandDistinctAggregatesRule{code},
>  but it only provides a rewrite to Grouping sets based plan.
> I am planing to use the actual Calcite rule, [~ashutoshc] any concerns or 
> caveats to be aware of?
> h2. Concerns/questions
> Need to have a way to switch between Grouping sets or Simple chained group by 
> based on the plan cost. For instance for Druid based scan it makes always 
> sense (at least today) to push down a series of Group by and stitch result 
> sets in Hive later (as oppose to scan everything). 
> But this might be not true for other storage handler that can handle Grouping 
> sets it is better to push down the Grouping sets as one table scan.
> Am still unsure how i can lean on the cost optimizer to select the best plan, 
> [~ashutoshc]/[~jcamachorodriguez] any inputs?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19660) update branch-3 to be version 3.1 and fix storage-api mismatch

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486520#comment-16486520
 ] 

Sergey Shelukhin commented on HIVE-19660:
-

Should I just push this? not sure if branch-3 is set up for tests and whether 
they are reliable

> update branch-3 to be version 3.1 and fix storage-api mismatch
> --
>
> Key: HIVE-19660
> URL: https://issues.apache.org/jira/browse/HIVE-19660
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19660-branch-3.patch
>
>
> Currently we have branch-3.0 for presumably branch-3.0.X, but branch-3 is 
> marked as 3.0 and master as 3.1-SNAPSHOT.
> Also storage-api features break the build on branch-3 because storage-api 
> version does not match.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19666) SQL standard auth for create fn may make an impossible privilege check

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486516#comment-16486516
 ] 

Sergey Shelukhin commented on HIVE-19666:
-

[~thejas] can you take a look?

> SQL standard auth for create fn may make an impossible privilege check
> --
>
> Key: HIVE-19666
> URL: https://issues.apache.org/jira/browse/HIVE-19666
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19666.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19666) SQL standard auth for create fn may make an impossible privilege check

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-19666:
---

Assignee: Sergey Shelukhin

> SQL standard auth for create fn may make an impossible privilege check
> --
>
> Key: HIVE-19666
> URL: https://issues.apache.org/jira/browse/HIVE-19666
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19666.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19643) MM table conversion doesn't need full ACID structure checks

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486512#comment-16486512
 ] 

Hive QA commented on HIVE-19643:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12924461/HIVE-19643.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/11144/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11144/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11144/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-05-23 00:24:38.764
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-11144/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-05-23 00:24:38.767
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at ed95ee9 HIVE-19659: update master to version 4.0 (Vineet Garg, 
reviewed by Ashutosh Chauhan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at ed95ee9 HIVE-19659: update master to version 4.0 (Vineet Garg, 
reviewed by Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-05-23 00:24:40.003
+ rm -rf ../yetus_PreCommit-HIVE-Build-11144
+ mkdir ../yetus_PreCommit-HIVE-Build-11144
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-11144
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-11144/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: patch failed: ql/src/test/queries/clientpositive/mm_conversions.q:15
Falling back to three-way merge...
Applied patch to 'ql/src/test/queries/clientpositive/mm_conversions.q' with 
conflicts.
error: patch failed: 
ql/src/test/results/clientpositive/llap/mm_conversions.q.out:299
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/llap/mm_conversions.q.out' 
with conflicts.
Going to apply patch with: git apply -p0
error: patch failed: ql/src/test/queries/clientpositive/mm_conversions.q:15
Falling back to three-way merge...
Applied patch to 'ql/src/test/queries/clientpositive/mm_conversions.q' with 
conflicts.
error: patch failed: 
ql/src/test/results/clientpositive/llap/mm_conversions.q.out:299
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/llap/mm_conversions.q.out' 
with conflicts.
U ql/src/test/queries/clientpositive/mm_conversions.q
U ql/src/test/results/clientpositive/llap/mm_conversions.q.out
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12924461 - PreCommit-HIVE-Build

> MM table conversion doesn't need full ACID structure checks
> ---
>
> Key: HIVE-19643
> URL: https://issues.apache.org/jira/browse/HIVE-19643
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Jason Dere
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19643.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19667) Remove distribution management tag from pom.xml

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-19667:
---
Attachment: HIVE-19667.1.patch

> Remove distribution management tag from pom.xml
> ---
>
> Key: HIVE-19667
> URL: https://issues.apache.org/jira/browse/HIVE-19667
> Project: Hive
>  Issue Type: Task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19667.1.patch
>
>
> This tag overrides apaches configuration in maven settings file and makes it 
> impossible to publish maven artifacts. There is no way around to it either.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19667) Remove distribution management tag from pom.xml

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-19667:
---
Status: Patch Available  (was: Open)

> Remove distribution management tag from pom.xml
> ---
>
> Key: HIVE-19667
> URL: https://issues.apache.org/jira/browse/HIVE-19667
> Project: Hive
>  Issue Type: Task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19667.1.patch
>
>
> This tag overrides apaches configuration in maven settings file and makes it 
> impossible to publish maven artifacts. There is no way around to it either.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19667) Remove distribution management tag from pom.xml

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-19667:
---
Description: This tag overrides apaches configuration in maven settings 
file and makes it impossible to publish maven artifacts. There is no way around 
to it either.  (was: This tag overrides apaches configuration in maven settings 
file and there is no way to get around this. I am not sure why this was added.)

> Remove distribution management tag from pom.xml
> ---
>
> Key: HIVE-19667
> URL: https://issues.apache.org/jira/browse/HIVE-19667
> Project: Hive
>  Issue Type: Task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
>
> This tag overrides apaches configuration in maven settings file and makes it 
> impossible to publish maven artifacts. There is no way around to it either.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19465) Upgrade ORC to 1.5.0

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486509#comment-16486509
 ] 

Hive QA commented on HIVE-19465:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  3m 
23s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
33s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
42s{color} | {color:blue} llap-server in master has 86 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
52s{color} | {color:blue} ql in master has 2322 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
22s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  7m 
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m  
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m  
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m 
59s{color} | {color:red} root: The patch generated 6 new + 1647 unchanged - 11 
fixed = 1653 total (was 1658) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
17s{color} | {color:red} itests/hive-unit: The patch generated 4 new + 184 
unchanged - 4 fixed = 188 total (was 188) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} llap-server: The patch generated 0 new + 108 
unchanged - 2 fixed = 108 total (was 110) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
45s{color} | {color:red} ql: The patch generated 2 new + 840 unchanged - 5 
fixed = 842 total (was 845) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
15s{color} | {color:green} The patch streaming passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  6m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  7m 
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
11s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m 55s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  
xml  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11143/dev-support/hive-personality.sh
 |
| git revision | master / ed95ee9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-11143/yetus/diff-checkstyle-root.txt
 |
| checkstyle | 

[jira] [Commented] (HIVE-19465) Upgrade ORC to 1.5.0

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486506#comment-16486506
 ] 

Hive QA commented on HIVE-19465:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12924462/HIVE-19465.03.patch

{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 14395 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schemeAuthority] 
(batchId=152)
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testSeek[0] (batchId=278)
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testSeek[1] (batchId=278)
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testZeroCopySeek[0] (batchId=278)
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testZeroCopySeek[1] (batchId=278)
org.apache.hive.jdbc.TestActivePassiveHA.testActivePassiveHA (batchId=241)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/11143/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11143/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11143/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12924462 - PreCommit-HIVE-Build

> Upgrade ORC to 1.5.0
> 
>
> Key: HIVE-19465
> URL: https://issues.apache.org/jira/browse/HIVE-19465
> Project: Hive
>  Issue Type: Improvement
>  Components: File Formats
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Blocker
> Attachments: HIVE-19465.01.patch, HIVE-19465.02.patch, 
> HIVE-19465.03.patch, HIVE-19465.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19667) Remove distribution management tag from pom.xml

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg reassigned HIVE-19667:
--


> Remove distribution management tag from pom.xml
> ---
>
> Key: HIVE-19667
> URL: https://issues.apache.org/jira/browse/HIVE-19667
> Project: Hive
>  Issue Type: Task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
>
> This tag overrides apaches configuration in maven settings file and there is 
> no way to get around this. I am not sure why this was added.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19666) SQL standard auth for create fn may make an impossible privilege check

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19666:

Attachment: HIVE-19666.patch

> SQL standard auth for create fn may make an impossible privilege check
> --
>
> Key: HIVE-19666
> URL: https://issues.apache.org/jira/browse/HIVE-19666
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19666.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19666) SQL standard auth for create fn may make an impossible privilege check

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19666:

Status: Patch Available  (was: Open)

> SQL standard auth for create fn may make an impossible privilege check
> --
>
> Key: HIVE-19666
> URL: https://issues.apache.org/jira/browse/HIVE-19666
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19666.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18117) Create TestCliDriver for HDFS EC

[Andrew Sherman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16485744#comment-16485744
 ] 

Andrew Sherman commented on HIVE-18117:
---

The 3 asf license warnings in 
[http://104.198.109.242/logs//PreCommit-HIVE-Build-3/yetus/patch-asflicense-problems.txt]
 are not caused by this change.

The checkstyle warning TestErasureCodingHDFSCliDriver.java:48:  public static 
TestRule cliClassRule = adapter.buildClassRule();:26: warning: Variable 
'cliClassRule' must be private and have accessor methodsI
Is a false positive as Junit requires that fields annotated with ClassRule must 
be public. 

The single most  recent failure 
[https://builds.apache.org/job/PreCommit-HIVE-Build/3/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_sample6_/]
  is spurious.

So I think this change is good to go, please take a look [~stakiar]

> Create TestCliDriver for HDFS EC
> 
>
> Key: HIVE-18117
> URL: https://issues.apache.org/jira/browse/HIVE-18117
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Andrew Sherman
>Priority: Major
> Attachments: HIVE-18117.1.patch, HIVE-18117.2.patch, 
> HIVE-18117.3.patch, HIVE-18117.4.patch, HIVE-18117.5.patch, 
> HIVE-18117.6.patch, HIVE-18117.7.patch, HIVE-18117.8.patch, HIVE-18117.9.patch
>
>
> Should be able to do something similar to what we do for HDFS encryption.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19577) CREATE TEMPORARY TABLE LIKE and INSERT generate output format mismatch errors

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484732#comment-16484732
 ] 

Hive QA commented on HIVE-19577:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12924442/HIVE-19577.01.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14396 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=69)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/11142/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11142/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11142/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12924442 - PreCommit-HIVE-Build

> CREATE TEMPORARY TABLE LIKE  and INSERT generate output format mismatch errors
> --
>
> Key: HIVE-19577
> URL: https://issues.apache.org/jira/browse/HIVE-19577
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19577.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19665) Update version to 3.0.1-SNAPSHOT in branch-3.0

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484727#comment-16484727
 ] 

Sergey Shelukhin commented on HIVE-19665:
-

+1

> Update version to 3.0.1-SNAPSHOT in branch-3.0
> --
>
> Key: HIVE-19665
> URL: https://issues.apache.org/jira/browse/HIVE-19665
> Project: Hive
>  Issue Type: Task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19665.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HIVE-19664) LLAP reader changes for decimal 64

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran resolved HIVE-19664.
--
Resolution: Duplicate

Dup of HIVE-19629

> LLAP reader changes for decimal 64
> --
>
> Key: HIVE-19664
> URL: https://issues.apache.org/jira/browse/HIVE-19664
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
>
> With ORC 1.5.0, LLAP readers has to be updated to support decimal 64 readers. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19641) sync up hadoop version used by storage-api with hive

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-19641:
-
Fix Version/s: (was: 3.0.1)

> sync up hadoop version used by storage-api with hive
> 
>
> Key: HIVE-19641
> URL: https://issues.apache.org/jira/browse/HIVE-19641
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19641.1.patch
>
>
> There is hadoop version mismatch between hive and storage-api and hence 
> different transitive dependency versions gets pulled.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19577) CREATE TEMPORARY TABLE LIKE and INSERT generate output format mismatch errors

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484718#comment-16484718
 ] 

Hive QA commented on HIVE-19577:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
30s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
45s{color} | {color:blue} ql in master has 2322 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
6s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  5m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch 199 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 52s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11142/dev-support/hive-personality.sh
 |
| git revision | master / ed95ee9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-11142/yetus/whitespace-eol.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-11142/yetus/whitespace-tabs.txt
 |
| modules | C: . itests ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-11142/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> CREATE TEMPORARY TABLE LIKE  and INSERT generate output format mismatch errors
> --
>
> Key: HIVE-19577
> URL: https://issues.apache.org/jira/browse/HIVE-19577
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19577.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19665) Update version to 3.0.1-SNAPSHOT in branch-3.0

[Vineet Garg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484717#comment-16484717
 ] 

Vineet Garg commented on HIVE-19665:


[~sershe] Can you take a quick look?

> Update version to 3.0.1-SNAPSHOT in branch-3.0
> --
>
> Key: HIVE-19665
> URL: https://issues.apache.org/jira/browse/HIVE-19665
> Project: Hive
>  Issue Type: Task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19665.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19665) Update version to 3.0.1-SNAPSHOT in branch-3.0

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-19665:
---
Attachment: HIVE-19665.1.patch

> Update version to 3.0.1-SNAPSHOT in branch-3.0
> --
>
> Key: HIVE-19665
> URL: https://issues.apache.org/jira/browse/HIVE-19665
> Project: Hive
>  Issue Type: Task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19665.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19665) Update version to 3.0.1-SNAPSHOT in branch-3.0

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg reassigned HIVE-19665:
--


> Update version to 3.0.1-SNAPSHOT in branch-3.0
> --
>
> Key: HIVE-19665
> URL: https://issues.apache.org/jira/browse/HIVE-19665
> Project: Hive
>  Issue Type: Task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19664) LLAP reader changes for decimal 64

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran reassigned HIVE-19664:



> LLAP reader changes for decimal 64
> --
>
> Key: HIVE-19664
> URL: https://issues.apache.org/jira/browse/HIVE-19664
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
>
> With ORC 1.5.0, LLAP readers has to be updated to support decimal 64 readers. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19663) refactor LLAP IO report generation

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19663:

Description: 
Follow-up from HIVE-19642.
Instead of each component calling some other component in a chain, all the 
parts of the state dump should be called in one place to avoid weird 
dependencies/sequences that need to be accounted for to generate the report.

  was:
Follow-up from HIVE-19642.
Instead of each component calling some other component in a chain, all the 
parts of the dump should be called in one place to avoid weird 
dependencies/sequences that need to be accounted for to generate the report.


> refactor LLAP IO report generation
> --
>
> Key: HIVE-19663
> URL: https://issues.apache.org/jira/browse/HIVE-19663
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Major
>
> Follow-up from HIVE-19642.
> Instead of each component calling some other component in a chain, all the 
> parts of the state dump should be called in one place to avoid weird 
> dependencies/sequences that need to be accounted for to generate the report.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19642) add cache dump when LLAP cache is full on branch-2 (and fix retry issues)

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19642:

   Resolution: Fixed
Fix Version/s: 2.4.0
   Status: Resolved  (was: Patch Available)

Committed to branch-2. Thanks for the review! Filed a follow-up jira

> add cache dump when LLAP cache is full on branch-2 (and fix retry issues)
> -
>
> Key: HIVE-19642
> URL: https://issues.apache.org/jira/browse/HIVE-19642
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 2.4.0
>
> Attachments: HIVE-19642-branch-2.patch
>
>
> LLAP cache doesn't have defragmentation on branch-2 so it retries when 
> failing to evict.
> There are a couple of issues with retries (e.g. sleep is not applied properly 
> and time can go negative), also it would be helpful to make use of existing 
> cache dump logic used in LLAP IO view to dump log state into the log in such 
> cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19605) TAB_COL_STATS table has no index on db/table name

[Todd Lipcon (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484687#comment-16484687
 ] 

Todd Lipcon commented on HIVE-19605:


Seems the slow query on get_table is due to the "initialization" queries in 
ensureDbInit(). This is worked around by HIVE-19310.

However, the API calls that actually are meant to fetch column stats are still 
slow and this should be fixed for their sake.

> TAB_COL_STATS table has no index on db/table name
> -
>
> Key: HIVE-19605
> URL: https://issues.apache.org/jira/browse/HIVE-19605
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Todd Lipcon
>Assignee: Vihang Karajgaonkar
>Priority: Major
>
> The TAB_COL_STATS table is missing an index on (CAT_NAME, DB_NAME, 
> TABLE_NAME). The getTableColumnStatistics call queries based on this tuple. 
> This makes those queries take a significant amount of time in large 
> metastores since they do a full table scan.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19641) sync up hadoop version used by storage-api with hive

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-19641:
-
Fix Version/s: 3.0.1

> sync up hadoop version used by storage-api with hive
> 
>
> Key: HIVE-19641
> URL: https://issues.apache.org/jira/browse/HIVE-19641
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Fix For: 3.0.1, 4.0.0
>
> Attachments: HIVE-19641.1.patch
>
>
> There is hadoop version mismatch between hive and storage-api and hence 
> different transitive dependency versions gets pulled.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19465) Upgrade ORC to 1.5.0

[Owen O'Malley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484662#comment-16484662
 ] 

Owen O'Malley commented on HIVE-19465:
--

[~jcamachorodriguez], you should also change the version of ORC (orc.version) 
in the standalone metastore to 1.5.1 (once it is released).

> Upgrade ORC to 1.5.0
> 
>
> Key: HIVE-19465
> URL: https://issues.apache.org/jira/browse/HIVE-19465
> Project: Hive
>  Issue Type: Improvement
>  Components: File Formats
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Blocker
> Attachments: HIVE-19465.01.patch, HIVE-19465.02.patch, 
> HIVE-19465.03.patch, HIVE-19465.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19605) TAB_COL_STATS table has no index on db/table name

[Todd Lipcon (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484648#comment-16484648
 ] 

Todd Lipcon commented on HIVE-19605:


Upon further inspection it seems the above query is likely generated during the 
initialization of ObjectStore, not directly within the get_table call. So, any 
call can end up making this query and generating a big outlier.

> TAB_COL_STATS table has no index on db/table name
> -
>
> Key: HIVE-19605
> URL: https://issues.apache.org/jira/browse/HIVE-19605
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Todd Lipcon
>Assignee: Vihang Karajgaonkar
>Priority: Major
>
> The TAB_COL_STATS table is missing an index on (CAT_NAME, DB_NAME, 
> TABLE_NAME). The getTableColumnStatistics call queries based on this tuple. 
> This makes those queries take a significant amount of time in large 
> metastores since they do a full table scan.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19619) Allow comparisons between doubles and bigints

[Vineet Garg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484639#comment-16484639
 ] 

Vineet Garg commented on HIVE-19619:


Pushed to branch-3

> Allow comparisons between doubles and bigints
> -
>
> Key: HIVE-19619
> URL: https://issues.apache.org/jira/browse/HIVE-19619
> Project: Hive
>  Issue Type: Bug
>  Components: Types
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
>Priority: Major
> Fix For: 3.1.0, 4.0.0
>
> Attachments: HIVE-19619.01-branch-3.patch, HIVE-19619.patch, 
> HIVE-19619_1.patch, HIVE-19619_branch-3.patch
>
>
> Hive doesn't allow binary comparison between doubles and bigints. That is 
> overly restrictive since all other DBs (mysql, postgres, oracle and sql 
> server) allows that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19619) Allow comparisons between doubles and bigints

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-19619:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Allow comparisons between doubles and bigints
> -
>
> Key: HIVE-19619
> URL: https://issues.apache.org/jira/browse/HIVE-19619
> Project: Hive
>  Issue Type: Bug
>  Components: Types
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
>Priority: Major
> Fix For: 3.1.0, 4.0.0
>
> Attachments: HIVE-19619.01-branch-3.patch, HIVE-19619.patch, 
> HIVE-19619_1.patch, HIVE-19619_branch-3.patch
>
>
> Hive doesn't allow binary comparison between doubles and bigints. That is 
> overly restrictive since all other DBs (mysql, postgres, oracle and sql 
> server) allows that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19619) Allow comparisons between doubles and bigints

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-19619:
---
Fix Version/s: 4.0.0

> Allow comparisons between doubles and bigints
> -
>
> Key: HIVE-19619
> URL: https://issues.apache.org/jira/browse/HIVE-19619
> Project: Hive
>  Issue Type: Bug
>  Components: Types
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
>Priority: Major
> Fix For: 3.1.0, 4.0.0
>
> Attachments: HIVE-19619.01-branch-3.patch, HIVE-19619.patch, 
> HIVE-19619_1.patch, HIVE-19619_branch-3.patch
>
>
> Hive doesn't allow binary comparison between doubles and bigints. That is 
> overly restrictive since all other DBs (mysql, postgres, oracle and sql 
> server) allows that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (HIVE-19661) switch Hive UDFs to use Re2J regex engine

[Rajkumar Singh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-19661 started by Rajkumar Singh.
-
> switch Hive UDFs to use Re2J regex engine
> -
>
> Key: HIVE-19661
> URL: https://issues.apache.org/jira/browse/HIVE-19661
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
>
> Java regex engine can be very slow in some cases e.g. 
> https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8203458



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19661) switch Hive UDFs to use Re2J regex engine

[Rajkumar Singh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajkumar Singh reassigned HIVE-19661:
-

Assignee: Rajkumar Singh

> switch Hive UDFs to use Re2J regex engine
> -
>
> Key: HIVE-19661
> URL: https://issues.apache.org/jira/browse/HIVE-19661
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
>
> Java regex engine can be very slow in some cases e.g. 
> https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8203458



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18098) Add support for Export/Import for Acid tables

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18098:
--
Fix Version/s: (was: 4.0.0)
   3.0.0

> Add support for Export/Import for Acid tables
> -
>
> Key: HIVE-18098
> URL: https://issues.apache.org/jira/browse/HIVE-18098
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Fix For: 3.0.0
>
>
> How should this work?
> For regular tables export just copies the files under table root to a 
> specified directory.
> This doesn't make sense for Acid tables:
> * Some data may belong to aborted transactons
> * Transaction IDs are imbedded into data/files names.  You'd have export 
> delta/ and base/ each of which may have files with the same names, e.g. 
> bucket_0.   
> * On import these IDs won't make sense in a different cluster or even a 
> different table (which may have delta_x_x for example for the same x (but 
> different data of course).
> * Export creates a _metadata column types, storage format, etc.  Perhaps it 
> can include info about aborted IDs (if the whole file can't be skipped).
> * Even importing into the same table on the same cluster may be a problem.  
> For example delta_5_5/ existed at the time of export and was included in the 
> export.  But 2 days later it may not exist because it was compacted and 
> cleaned.
> * If importing back into the same table on the same cluster, the data could 
> be imported into a different transaction (assuming per table writeIDs) w/o 
> having to remap the IDs in the rows themselves.
> * support Import Overwrite?
> * Support Import as a new txn with remapping of ROW_IDs?  The new writeID can 
> be stored in a delta_x_x/_meta_data and ROW__IDs can be remapped at read time 
> (like isOriginal) and made permanent by compaction.
> * It doesn't seem reasonable to import acid data into non-acid table
> Perhaps import can work similar to Load Data: look at the file imported, if 
> it has Acid columns, leave a note in the delta_x_x/_meta_data to indicate 
> that these columns should be skipped a new ROW_IDs assigned at read time.
> h3. Case I
> Table has delta_7_7 and delta_8_8.  Sine both may have bucket_, we could 
> export to export_dir and rename files as bucket_ and bucket__copy_1.  
> Load Data supports input dir with copy_N files.
> h3. Case II
> what if we have delete_delta_9_9 in the source.  Now you can't just ignore 
> ROW_IDs after import.
> * -Only export the latest base_N?  Or more generally up to the smallest 
> deleted ROW_ID (which may be hard to find w/o scanning all deletes.  The 
> export then would have to be done under X lock to prevent new concurrent 
> deletes)-
> * Stash all deletes in some additional file file which on import gets added 
> into the target delta/ so that Acid reader can apply them to the data in this 
> delta/ but so that they don't clash with 'normal' deletes that exist in the 
> table.
> ** here we may also have multiple delete_delta/ with identical file names.  
> Does delete delta reader handle copy_N files?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18741) Add support for Import into Acid table

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18741:
--
Fix Version/s: (was: 4.0.0)
   3.0.0

> Add support for Import into Acid table
> --
>
> Key: HIVE-18741
> URL: https://issues.apache.org/jira/browse/HIVE-18741
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18741.01.patch
>
>
> This should follow Load Data approach (or use load data directly)
> Note that import supports partition spec
> Does import support loading files not created by Export?  If so, similarly to 
> HIVE-19029 - should check for Acid meta columns and reject



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19605) TAB_COL_STATS table has no index on db/table name

[Vihang Karajgaonkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vihang Karajgaonkar reassigned HIVE-19605:
--

Assignee: Vihang Karajgaonkar

> TAB_COL_STATS table has no index on db/table name
> -
>
> Key: HIVE-19605
> URL: https://issues.apache.org/jira/browse/HIVE-19605
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Todd Lipcon
>Assignee: Vihang Karajgaonkar
>Priority: Major
>
> The TAB_COL_STATS table is missing an index on (CAT_NAME, DB_NAME, 
> TABLE_NAME). The getTableColumnStatistics call queries based on this tuple. 
> This makes those queries take a significant amount of time in large 
> metastores since they do a full table scan.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19639) a transactional Hive table cannot be imported as an external table

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484622#comment-16484622
 ] 

Sergey Shelukhin commented on HIVE-19639:
-

Committed to branch-3

> a transactional Hive table cannot be imported as an external table
> --
>
> Key: HIVE-19639
> URL: https://issues.apache.org/jira/browse/HIVE-19639
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.1.0, 4.0.0
>
> Attachments: HIVE-19639.patch
>
>
> When transactional table is imported to a external table, the table should be 
> imported as a non transactional table, as external tables cannot be 
> transactional.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19605) TAB_COL_STATS table has no index on db/table name

[Todd Lipcon (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484623#comment-16484623
 ] 

Todd Lipcon commented on HIVE-19605:


It seems like this table can also be called from a get_table call. Oddly, the 
query being generated is:

SELECT 'org.apache.hadoop.hive.metastore.model.MTableColumnStatistics' AS 
NUCLEUS_TYPE,`A0`.`AVG_COL_LEN`,`A0`.`COLUMN_NAME`,`A0`.`COLUMN_TYPE`,`A0`.`DB_NAME`,`A0`.`BIG_DECIMAL_HIGH_VALUE`,`A0`.`BIG_DECIMAL_LOW_VALUE`,`A0`.`DOUBLE_HIGH_VALUE`,`A0`.`DOUBLE_LOW_VALUE`,`A0`.`LAST_ANALYZED`,`A0`.`LONG_HIGH_VALUE`,`A0`.`LONG_LOW_VALUE`,`A0`.`MAX_COL_LEN`,`A0`.`NUM_DISTINCTS`,`A0`.`NUM_FALSES`,`A0`.`NUM_NULLS`,`A0`.`NUM_TRUES`,`A0`.`TABLE_NAME`,`A0`.`CS_ID`
 FROM `TAB_COL_STATS` `A0` WHERE `A0`.`DB_NAME` = '';

(note the empty db_name).

Given the lack of index, this takes 450ms on the HMS instance I am testing (if 
the mysql query cache is disabled)

> TAB_COL_STATS table has no index on db/table name
> -
>
> Key: HIVE-19605
> URL: https://issues.apache.org/jira/browse/HIVE-19605
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Todd Lipcon
>Priority: Major
>
> The TAB_COL_STATS table is missing an index on (CAT_NAME, DB_NAME, 
> TABLE_NAME). The getTableColumnStatistics call queries based on this tuple. 
> This makes those queries take a significant amount of time in large 
> metastores since they do a full table scan.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19642) add cache dump when LLAP cache is full on branch-2 (and fix retry issues)

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484620#comment-16484620
 ] 

Hive QA commented on HIVE-19642:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12924437/HIVE-19642-branch-2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 10668 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
 (batchId=227)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_tableproperty_optimize]
 (batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explaindenpendencydiffengs]
 (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] 
(batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic] 
(batchId=139)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[table_nonprintable]
 (batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
 (batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
 (batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_parquet_types]
 (batchId=155)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[merge_negative_5]
 (batchId=88)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
 (batchId=98)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[explaindenpendencydiffengs]
 (batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_input_format_excludes]
 (batchId=117)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_ptf] 
(batchId=125)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=176)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/11141/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11141/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11141/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12924437 - PreCommit-HIVE-Build

> add cache dump when LLAP cache is full on branch-2 (and fix retry issues)
> -
>
> Key: HIVE-19642
> URL: https://issues.apache.org/jira/browse/HIVE-19642
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19642-branch-2.patch
>
>
> LLAP cache doesn't have defragmentation on branch-2 so it retries when 
> failing to evict.
> There are a couple of issues with retries (e.g. sleep is not applied properly 
> and time can go negative), also it would be helpful to make use of existing 
> cache dump logic used in LLAP IO view to dump log state into the log in such 
> cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19639) a transactional Hive table cannot be imported as an external table

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19639:

Fix Version/s: 3.1.0

> a transactional Hive table cannot be imported as an external table
> --
>
> Key: HIVE-19639
> URL: https://issues.apache.org/jira/browse/HIVE-19639
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.1.0, 4.0.0
>
> Attachments: HIVE-19639.patch
>
>
> When transactional table is imported to a external table, the table should be 
> imported as a non transactional table, as external tables cannot be 
> transactional.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19531) TransactionalValidationListener is getting catalog name from conf instead of table object.

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-19531:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> TransactionalValidationListener is getting catalog name from conf instead of 
> table object.
> --
>
> Key: HIVE-19531
> URL: https://issues.apache.org/jira/browse/HIVE-19531
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 3.0.1
>
> Attachments: HIVE-19531.1again.patch, HIVE-19531.1take3.patch, 
> HIVE-19531.1take4.patch, HIVE-19531.patch
>
>
> TransactionalValidationListener.validateTableStructure get the catalog from 
> the conf file rather than taking it from the passed in table structure.  This 
> causes createTable operations to fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19531) TransactionalValidationListener is getting catalog name from conf instead of table object.

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484614#comment-16484614
 ] 

Alan Gates commented on HIVE-19531:
---

Patch pushed to branch-3 and branch-3.0.

> TransactionalValidationListener is getting catalog name from conf instead of 
> table object.
> --
>
> Key: HIVE-19531
> URL: https://issues.apache.org/jira/browse/HIVE-19531
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 3.0.1
>
> Attachments: HIVE-19531.1again.patch, HIVE-19531.1take3.patch, 
> HIVE-19531.1take4.patch, HIVE-19531.patch
>
>
> TransactionalValidationListener.validateTableStructure get the catalog from 
> the conf file rather than taking it from the passed in table structure.  This 
> causes createTable operations to fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19639) a transactional Hive table cannot be imported as an external table

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-19639:
---
Fix Version/s: (was: 3.1.0)

> a transactional Hive table cannot be imported as an external table
> --
>
> Key: HIVE-19639
> URL: https://issues.apache.org/jira/browse/HIVE-19639
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19639.patch
>
>
> When transactional table is imported to a external table, the table should be 
> imported as a non transactional table, as external tables cannot be 
> transactional.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19496) Check untar folder

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-19496:
---
Fix Version/s: (was: 4.0.0)
   3.1.0

> Check untar folder
> --
>
> Key: HIVE-19496
> URL: https://issues.apache.org/jira/browse/HIVE-19496
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19496.1.patch, HIVE-19496.2.patch
>
>
> We need to check if the file is under untar folder.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19317) Handle schema evolution from int like types to decimal

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-19317:
---
Fix Version/s: (was: 4.0.0)
   3.1.0

> Handle schema evolution from int like types to decimal
> --
>
> Key: HIVE-19317
> URL: https://issues.apache.org/jira/browse/HIVE-19317
> Project: Hive
>  Issue Type: Bug
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19317.1.patch, HIVE-19317.2.patch, 
> HIVE-19317.3.patch, HIVE-19317.4.patch, HIVE-19317.5.patch
>
>
> If int like type is changed to decimal on parquet data, select results in 
> errors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19041) Thrift deserialization of Partition objects should intern fields

[Vineet Garg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484597#comment-16484597
 ] 

Vineet Garg commented on HIVE-19041:


Replaced fix version 3.1.0 with 4.0.0 since master is tracking 4.0.0

> Thrift deserialization of Partition objects should intern fields
> 
>
> Key: HIVE-19041
> URL: https://issues.apache.org/jira/browse/HIVE-19041
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Fix For: 2.4.0, 4.0.0
>
> Attachments: HIVE-19041.01.patch, HIVE-19041.02.patch, 
> HIVE-19041.03.patch, HIVE-19041.04.patch, HIVE-19041.05-branch-2.patch
>
>
> When a client is creating large number of partitions, the thrift objects are 
> deserialized into Partition objects. The read method of these objects does 
> not intern the inputformat, location, outputformat which cause large number 
> of duplicate Strings in the HMS memory. We should intern these objects while 
> deserialization to reduce memory pressure. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19041) Thrift deserialization of Partition objects should intern fields

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-19041:
---
Fix Version/s: (was: 3.1.0)
   4.0.0

> Thrift deserialization of Partition objects should intern fields
> 
>
> Key: HIVE-19041
> URL: https://issues.apache.org/jira/browse/HIVE-19041
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Fix For: 2.4.0, 4.0.0
>
> Attachments: HIVE-19041.01.patch, HIVE-19041.02.patch, 
> HIVE-19041.03.patch, HIVE-19041.04.patch, HIVE-19041.05-branch-2.patch
>
>
> When a client is creating large number of partitions, the thrift objects are 
> deserialized into Partition objects. The read method of these objects does 
> not intern the inputformat, location, outputformat which cause large number 
> of duplicate Strings in the HMS memory. We should intern these objects while 
> deserialization to reduce memory pressure. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19388) ClassCastException during VectorMapJoinCommonOperator initialization

[Vineet Garg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484594#comment-16484594
 ] 

Vineet Garg commented on HIVE-19388:


[~vihangk1] I am updating the fix version to 4.0.0 and removing 3.1.0 since 
this was pushed to master (4.0.0) and not branch-3 (3.1.0)

> ClassCastException during VectorMapJoinCommonOperator initialization
> 
>
> Key: HIVE-19388
> URL: https://issues.apache.org/jira/browse/HIVE-19388
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2, 3.1.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Fix For: 2.4.0, 4.0.0
>
> Attachments: HIVE-19388.01.patch, HIVE-19388.02.patch
>
>
> I see the following exceptions when I a mapjoin operator is being initialized 
> on Hive-on-Spark and when vectorization is turned on.
> This happens when the hashTable is empty. The code in 
> {{MapJoinTableContainerSerDe#getDefaultEmptyContainer}} method returns a 
> HashMapWrapper while the VectorMapJoinOperator expects a 
> {{MapJoinBytesTableContainer}} when {{hive.mapjoin.optimized.hashtable}} is 
> set to true.
> {noformat}
> Caused by: java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper cannot be cast to 
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinTableContainerDirectAccess
>  at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.optimized.VectorMapJoinOptimizedHashTable.(VectorMapJoinOptimizedHashTable.java:92)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.optimized.VectorMapJoinOptimizedHashMap.(VectorMapJoinOptimizedHashMap.java:127)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.optimized.VectorMapJoinOptimizedStringHashMap.(VectorMapJoinOptimizedStringHashMap.java:60)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.optimized.VectorMapJoinOptimizedCreateHashTable.createHashTable(VectorMapJoinOptimizedCreateHashTable.java:80)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator.setUpHashTable(VectorMapJoinCommonOperator.java:485)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator.completeInitializationOp(VectorMapJoinCommonOperator.java:461)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:471)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:401) 
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:574) 
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:526) 
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:387) 
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.init(SparkMapRecordHandler.java:109)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>  ... 16 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19388) ClassCastException during VectorMapJoinCommonOperator initialization

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-19388:
---
Fix Version/s: (was: 3.1.0)
   4.0.0

> ClassCastException during VectorMapJoinCommonOperator initialization
> 
>
> Key: HIVE-19388
> URL: https://issues.apache.org/jira/browse/HIVE-19388
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2, 3.1.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Fix For: 2.4.0, 4.0.0
>
> Attachments: HIVE-19388.01.patch, HIVE-19388.02.patch
>
>
> I see the following exceptions when I a mapjoin operator is being initialized 
> on Hive-on-Spark and when vectorization is turned on.
> This happens when the hashTable is empty. The code in 
> {{MapJoinTableContainerSerDe#getDefaultEmptyContainer}} method returns a 
> HashMapWrapper while the VectorMapJoinOperator expects a 
> {{MapJoinBytesTableContainer}} when {{hive.mapjoin.optimized.hashtable}} is 
> set to true.
> {noformat}
> Caused by: java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper cannot be cast to 
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinTableContainerDirectAccess
>  at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.optimized.VectorMapJoinOptimizedHashTable.(VectorMapJoinOptimizedHashTable.java:92)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.optimized.VectorMapJoinOptimizedHashMap.(VectorMapJoinOptimizedHashMap.java:127)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.optimized.VectorMapJoinOptimizedStringHashMap.(VectorMapJoinOptimizedStringHashMap.java:60)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.optimized.VectorMapJoinOptimizedCreateHashTable.createHashTable(VectorMapJoinOptimizedCreateHashTable.java:80)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator.setUpHashTable(VectorMapJoinCommonOperator.java:485)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator.completeInitializationOp(VectorMapJoinCommonOperator.java:461)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:471)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:401) 
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:574) 
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:526) 
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:387) 
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.init(SparkMapRecordHandler.java:109)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>  ... 16 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19639) a transactional Hive table cannot be imported as an external table

[Vineet Garg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484590#comment-16484590
 ] 

Vineet Garg commented on HIVE-19639:


I don't see this commit in branch-3. Should the fix version be only 4.0.0?

> a transactional Hive table cannot be imported as an external table
> --
>
> Key: HIVE-19639
> URL: https://issues.apache.org/jira/browse/HIVE-19639
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.1.0, 4.0.0
>
> Attachments: HIVE-19639.patch
>
>
> When transactional table is imported to a external table, the table should be 
> imported as a non transactional table, as external tables cannot be 
> transactional.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18098) Add support for Export/Import for Acid tables

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-18098:
---
Fix Version/s: (was: 3.1.0)
   4.0.0

> Add support for Export/Import for Acid tables
> -
>
> Key: HIVE-18098
> URL: https://issues.apache.org/jira/browse/HIVE-18098
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Fix For: 4.0.0
>
>
> How should this work?
> For regular tables export just copies the files under table root to a 
> specified directory.
> This doesn't make sense for Acid tables:
> * Some data may belong to aborted transactons
> * Transaction IDs are imbedded into data/files names.  You'd have export 
> delta/ and base/ each of which may have files with the same names, e.g. 
> bucket_0.   
> * On import these IDs won't make sense in a different cluster or even a 
> different table (which may have delta_x_x for example for the same x (but 
> different data of course).
> * Export creates a _metadata column types, storage format, etc.  Perhaps it 
> can include info about aborted IDs (if the whole file can't be skipped).
> * Even importing into the same table on the same cluster may be a problem.  
> For example delta_5_5/ existed at the time of export and was included in the 
> export.  But 2 days later it may not exist because it was compacted and 
> cleaned.
> * If importing back into the same table on the same cluster, the data could 
> be imported into a different transaction (assuming per table writeIDs) w/o 
> having to remap the IDs in the rows themselves.
> * support Import Overwrite?
> * Support Import as a new txn with remapping of ROW_IDs?  The new writeID can 
> be stored in a delta_x_x/_meta_data and ROW__IDs can be remapped at read time 
> (like isOriginal) and made permanent by compaction.
> * It doesn't seem reasonable to import acid data into non-acid table
> Perhaps import can work similar to Load Data: look at the file imported, if 
> it has Acid columns, leave a note in the delta_x_x/_meta_data to indicate 
> that these columns should be skipped a new ROW_IDs assigned at read time.
> h3. Case I
> Table has delta_7_7 and delta_8_8.  Sine both may have bucket_, we could 
> export to export_dir and rename files as bucket_ and bucket__copy_1.  
> Load Data supports input dir with copy_N files.
> h3. Case II
> what if we have delete_delta_9_9 in the source.  Now you can't just ignore 
> ROW_IDs after import.
> * -Only export the latest base_N?  Or more generally up to the smallest 
> deleted ROW_ID (which may be hard to find w/o scanning all deletes.  The 
> export then would have to be done under X lock to prevent new concurrent 
> deletes)-
> * Stash all deletes in some additional file file which on import gets added 
> into the target delta/ so that Acid reader can apply them to the data in this 
> delta/ but so that they don't clash with 'normal' deletes that exist in the 
> table.
> ** here we may also have multiple delete_delta/ with identical file names.  
> Does delete delta reader handle copy_N files?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19555) Enable TestMiniLlapLocalCliDriver#tez_dynpart_hashjoin_1.q and TestMiniLlapLocalCliDriver#tez_vector_dynpart_hashjoin_1.q

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-19555:
---
Fix Version/s: (was: 3.1.0)
   4.0.0

> Enable TestMiniLlapLocalCliDriver#tez_dynpart_hashjoin_1.q and 
> TestMiniLlapLocalCliDriver#tez_vector_dynpart_hashjoin_1.q
> -
>
> Key: HIVE-19555
> URL: https://issues.apache.org/jira/browse/HIVE-19555
> Project: Hive
>  Issue Type: Test
>  Components: Test
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jason Dere
>Priority: Critical
> Fix For: 4.0.0
>
> Attachments: HIVE-19555.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19468) Add Apache license to TestTxnConcatenate

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-19468:
---
Fix Version/s: (was: 3.1.0)
   4.0.0

> Add Apache license to TestTxnConcatenate
> 
>
> Key: HIVE-19468
> URL: https://issues.apache.org/jira/browse/HIVE-19468
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Igor Kryvenko
>Assignee: Igor Kryvenko
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19468.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19304) Update templates.py based on config changes in YARN-7142 and YARN-8122

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-19304:
---
Fix Version/s: (was: 3.1.0)
   4.0.0

> Update templates.py based on config changes in YARN-7142 and YARN-8122
> --
>
> Key: HIVE-19304
> URL: https://issues.apache.org/jira/browse/HIVE-19304
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Gour Saha
>Assignee: Gour Saha
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19304.001.patch, HIVE-19304.01.patch
>
>
> Now that YARN-7142 is committed and YARN-8122 will be committed soon, we need 
> to update templates.py based on config changes for placement policy and 
> health threshold monitor.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)