[jira] [Commented] (HIVE-20563) Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type are different

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616612#comment-16616612
 ] 

Hive QA commented on HIVE-20563:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
54s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
43s{color} | {color:red} ql: The patch generated 1 new + 374 unchanged - 0 
fixed = 375 total (was 374) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 12s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13823/dev-support/hive-personality.sh
 |
| git revision | master / 37120b8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13823/yetus/diff-checkstyle-ql.txt
 |
| modules | C: itests ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13823/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type 
> are different
> ---
>
> Key: HIVE-20563
> URL: https://issues.apache.org/jira/browse/HIVE-20563
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Matt McCline
>Priority: Major
> Attachments: HIVE-20563.01.patch
>
>
> With the following stacktrace:
> {code}
> java.lang.Exception: java.lang.RuntimeException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row
> at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492) 
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552) 
> [hadoop-mapreduce-client-common-3.1.0.jar:?]
> Caused by: java.lang.RuntimeException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163) 
> ~[hive-exec-4.0.0-

[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

[Misha Dmitriev (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616608#comment-16616608
 ] 

Misha Dmitriev commented on HIVE-17684:
---

[~stakiar] there seems to be some unrelated compilation failure. What do we do 
now?

> HoS memory issues with MapJoinMemoryExhaustionHandler
> -
>
> Key: HIVE-17684
> URL: https://issues.apache.org/jira/browse/HIVE-17684
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HIVE-17684.01.patch, HIVE-17684.02.patch, 
> HIVE-17684.03.patch, HIVE-17684.04.patch, HIVE-17684.05.patch, 
> HIVE-17684.06.patch, HIVE-17684.07.patch, HIVE-17684.08.patch, 
> HIVE-17684.09.patch
>
>
> We have seen a number of memory issues due the {{HashSinkOperator}} use of 
> the {{MapJoinMemoryExhaustionHandler}}. This handler is meant to detect 
> scenarios where the small table is taking too much space in memory, in which 
> case a {{MapJoinMemoryExhaustionError}} is thrown.
> The configs to control this logic are:
> {{hive.mapjoin.localtask.max.memory.usage}} (default 0.90)
> {{hive.mapjoin.followby.gby.localtask.max.memory.usage}} (default 0.55)
> The handler works by using the {{MemoryMXBean}} and uses the following logic 
> to estimate how much memory the {{HashMap}} is consuming: 
> {{MemoryMXBean#getHeapMemoryUsage().getUsed() / 
> MemoryMXBean#getHeapMemoryUsage().getMax()}}
> The issue is that {{MemoryMXBean#getHeapMemoryUsage().getUsed()}} can be 
> inaccurate. The value returned by this method returns all reachable and 
> unreachable memory on the heap, so there may be a bunch of garbage data, and 
> the JVM just hasn't taken the time to reclaim it all. This can lead to 
> intermittent failures of this check even though a simple GC would have 
> reclaimed enough space for the process to continue working.
> We should re-think the usage of {{MapJoinMemoryExhaustionHandler}} for HoS. 
> In Hive-on-MR this probably made sense to use because every Hive task was run 
> in a dedicated container, so a Hive Task could assume it created most of the 
> data on the heap. However, in Hive-on-Spark there can be multiple Hive Tasks 
> running in a single executor, each doing different things.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616603#comment-16616603
 ] 

Hive QA commented on HIVE-17684:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939814/HIVE-17684.09.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 112 failed/errored test(s), 14941 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parallel_orderby] 
(batchId=57)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketmapjoin7]
 (batchId=187)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning]
 (batchId=187)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_2]
 (batchId=189)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_3]
 (batchId=189)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_6]
 (batchId=188)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_recursive_mapjoin]
 (batchId=188)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
 (batchId=188)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_use_ts_stats_for_mapjoin]
 (batchId=188)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
 (batchId=187)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_inner_join]
 (batchId=189)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join0]
 (batchId=189)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join1]
 (batchId=188)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join2]
 (batchId=187)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join3]
 (batchId=188)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join4]
 (batchId=190)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join5]
 (batchId=190)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join0] 
(batchId=149)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join10] 
(batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join11] 
(batchId=113)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join12] 
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join13] 
(batchId=146)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join14] 
(batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join15] 
(batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join17] 
(batchId=147)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join19] 
(batchId=138)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join1] 
(batchId=145)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join20] 
(batchId=150)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join21] 
(batchId=147)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join22] 
(batchId=134)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join23] 
(batchId=117)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join24] 
(batchId=143)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join26] 
(batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join29] 
(batchId=134)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join2] 
(batchId=138)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join30] 
(batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join31] 
(batchId=130)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join3] 
(batchId=147)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join4] 
(batchId=141)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join5] 
(batchId=143)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join8] 
(batchId=148)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join9] 
(batchId=144)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join_filters] 
(batchId=136)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join_nulls] 
(batchId=140)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join_stats2] 
(batchId=149)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join_stats] 

[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616602#comment-16616602
 ] 

Hive QA commented on HIVE-17684:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
34s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  3m 
33s{color} | {color:red} root in master failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
20s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
33s{color} | {color:blue} common in master has 64 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
0s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  7m 
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  3m 
35s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  3m 35s{color} 
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
16s{color} | {color:red} common: The patch generated 3 new + 425 unchanged - 0 
fixed = 428 total (was 425) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
26s{color} | {color:red} root: The patch generated 3 new + 425 unchanged - 0 
fixed = 428 total (was 425) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
39s{color} | {color:red} ql: The patch generated 5 new + 6 unchanged - 0 fixed 
= 11 total (was 6) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  4m 
10s{color} | {color:red} ql generated 1 new + 2310 unchanged - 1 fixed = 2311 
total (was 2311) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  7m 
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 22s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  Class org.apache.hadoop.hive.ql.exec.HashTableSinkOperator defines 
non-transient non-serializable instance field memoryExhaustionChecker  In 
HashTableSinkOperator.java:instance field memoryExhaustionChecker  In 
HashTableSinkOperator.java |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  
xml  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13822/dev-support/hive-personality.sh
 |
| git revision | master / 37120b8 |
| Default Java | 1.8.0_111 |
| compile | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13822/yetus/branch-compile-root.txt
 |
| findbugs | v3.0.0 |
| compile | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13822/yetus/patch-compile-root.txt
 |
| javac | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13822/yetus/patch-compile-root.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13822/yetus/diff-checkstyle-common.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13822/yetus/diff-checkstyle-root.txt
 |
| checkstyle | 
http

[jira] [Updated] (HIVE-20556) Expose an API to retrieve the TBL_ID from TBLS in the metastore tables

[Jaume M (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-20556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jaume M updated HIVE-20556:
---
Attachment: HIVE-20556.4.patch
Status: Patch Available  (was: Open)

> Expose an API to retrieve the TBL_ID from TBLS in the metastore tables
> --
>
> Key: HIVE-20556
> URL: https://issues.apache.org/jira/browse/HIVE-20556
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore, Standalone Metastore
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20556.1.patch, HIVE-20556.2.patch, 
> HIVE-20556.3.patch, HIVE-20556.4.patch
>
>
> We have two options to do this
> 1) Use the current MTable and add a field for this value
> 2) Add an independent API call to the metastore that would return the TBL_ID.
> Option 1 is preferable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20556) Expose an API to retrieve the TBL_ID from TBLS in the metastore tables

[Jaume M (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-20556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jaume M updated HIVE-20556:
---
Status: Open  (was: Patch Available)

> Expose an API to retrieve the TBL_ID from TBLS in the metastore tables
> --
>
> Key: HIVE-20556
> URL: https://issues.apache.org/jira/browse/HIVE-20556
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore, Standalone Metastore
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20556.1.patch, HIVE-20556.2.patch, 
> HIVE-20556.3.patch, HIVE-20556.4.patch
>
>
> We have two options to do this
> 1) Use the current MTable and add a field for this value
> 2) Add an independent API call to the metastore that would return the TBL_ID.
> Option 1 is preferable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20556) Expose an API to retrieve the TBL_ID from TBLS in the metastore tables

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616594#comment-16616594
 ] 

Hive QA commented on HIVE-20556:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939845/HIVE-20556.3.patch

{color:green}SUCCESS:{color} +1 due to 9 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 147 failed/errored test(s), 14443 tests 
executed
*Failed tests:*
{noformat}
TestAcidTableSerializer - did not produce a TEST-*.xml file (likely timed out) 
(batchId=215)
TestBucketIdResolverImpl - did not produce a TEST-*.xml file (likely timed out) 
(batchId=215)
TestCompactor - did not produce a TEST-*.xml file (likely timed out) 
(batchId=240)
TestContext - did not produce a TEST-*.xml file (likely timed out) (batchId=299)
TestDbTxnManager - did not produce a TEST-*.xml file (likely timed out) 
(batchId=315)
TestDbTxnManager2 - did not produce a TEST-*.xml file (likely timed out) 
(batchId=314)
TestDelimitedInputWriter - did not produce a TEST-*.xml file (likely timed out) 
(batchId=215)
TestDummyTxnManager - did not produce a TEST-*.xml file (likely timed out) 
(batchId=315)
TestEmbeddedLockManager - did not produce a TEST-*.xml file (likely timed out) 
(batchId=315)
TestGroupingValidator - did not produce a TEST-*.xml file (likely timed out) 
(batchId=215)
TestHeartbeatTimerTask - did not produce a TEST-*.xml file (likely timed out) 
(batchId=215)
TestHiveCredentialProviders - did not produce a TEST-*.xml file (likely timed 
out) (batchId=299)
TestHiveInputSplitComparator - did not produce a TEST-*.xml file (likely timed 
out) (batchId=315)
TestHiveLockObject - did not produce a TEST-*.xml file (likely timed out) 
(batchId=315)
TestKeyWrapperFactory - did not produce a TEST-*.xml file (likely timed out) 
(batchId=299)
TestLock - did not produce a TEST-*.xml file (likely timed out) (batchId=215)
TestLog4j2Appenders - did not produce a TEST-*.xml file (likely timed out) 
(batchId=299)
TestMetaStorePartitionHelper - did not produce a TEST-*.xml file (likely timed 
out) (batchId=215)
TestMsckDropPartitionsInBatches - did not produce a TEST-*.xml file (likely 
timed out) (batchId=299)
TestMutations - did not produce a TEST-*.xml file (likely timed out) 
(batchId=215)
TestMutatorClient - did not produce a TEST-*.xml file (likely timed out) 
(batchId=215)
TestMutatorCoordinator - did not produce a TEST-*.xml file (likely timed out) 
(batchId=215)
TestMutatorImpl - did not produce a TEST-*.xml file (likely timed out) 
(batchId=215)
TestOperators - did not produce a TEST-*.xml file (likely timed out) 
(batchId=299)
TestRecordInspectorImpl - did not produce a TEST-*.xml file (likely timed out) 
(batchId=215)
TestResetProcessor - did not produce a TEST-*.xml file (likely timed out) 
(batchId=315)
TestSequenceValidator - did not produce a TEST-*.xml file (likely timed out) 
(batchId=215)
TestSlidingFilenameRolloverStrategy - did not produce a TEST-*.xml file (likely 
timed out) (batchId=299)
TestStreaming - did not produce a TEST-*.xml file (likely timed out) 
(batchId=215)
TestStreaming - did not produce a TEST-*.xml file (likely timed out) 
(batchId=322)
TestStreamingDynamicPartitioning - did not produce a TEST-*.xml file (likely 
timed out) (batchId=322)
TestTransaction - did not produce a TEST-*.xml file (likely timed out) 
(batchId=215)
TestTxnCommands - did not produce a TEST-*.xml file (likely timed out) 
(batchId=315)
TestTxnCommands2 - did not produce a TEST-*.xml file (likely timed out) 
(batchId=297)
TestTxnCommands2WithSplitUpdateAndVectorization - did not produce a TEST-*.xml 
file (likely timed out) (batchId=310)
TestTxnCommandsWithSplitUpdateAndVectorization - did not produce a TEST-*.xml 
file (likely timed out) (batchId=299)
TestTxnNoBucketsVectorized - did not produce a TEST-*.xml file (likely timed 
out) (batchId=299)
TestWarehousePartitionHelper - did not produce a TEST-*.xml file (likely timed 
out) (batchId=215)
TestZookeeperLockManager - did not produce a TEST-*.xml file (likely timed out) 
(batchId=315)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[import_exported_table]
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[import_exported_table]
 (batchId=189)
org.apache.hadoop.hive.metastore.tools.TestSchemaToolCatalogOps.moveDatabase 
(batchId=237)
org.apache.hadoop.hive.metastore.tools.TestSchemaToolCatalogOps.moveTable 
(batchId=237)
org.apache.hadoop.hive.metastore.tools.TestSchemaToolCatalogOps.moveTableToNonExistentDb
 (batchId=237)
org.apache.hadoop.hive.metastore.tools.TestSchemaToolCatalogOps.moveTableWithExistingTableOfSameNameAlreadyInTargetDatabase
 (batchId=237)
org.apache.hadoop.hive.metastore.tools.TestSchemaToolCatalogOps.moveTableWithinCatalog
 (batchId=237)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion 
(batchId=239)
or

[jira] [Commented] (HIVE-20556) Expose an API to retrieve the TBL_ID from TBLS in the metastore tables

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616583#comment-16616583
 ] 

Hive QA commented on HIVE-20556:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  2m 
22s{color} | {color:blue} standalone-metastore/metastore-common in master has 
28 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
36s{color} | {color:blue} hcatalog/core in master has 31 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs 
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
13s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
41s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
43s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 43s{color} 
| {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
11s{color} | {color:red} itests/hcatalog-unit: The patch generated 1 new + 16 
unchanged - 0 fixed = 17 total (was 16) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
1s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
40s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
14s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 35s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13821/dev-support/hive-personality.sh
 |
| git revision | master / 37120b8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13821/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13821/yetus/patch-mvninstall-itests_hive-unit.txt
 |
| compile | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13821/yetus/patch-compile-itests_hive-unit.txt
 |
| javac | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13821/yetus/patch-compile-itests_hive-unit.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13821/yetus/diff-checkstyle-itests_hcatalog-unit.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13821/yetus/whitespace-eol.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13821/yetus/patch-findbugs-itests_

[jira] [Commented] (HIVE-20545) Exclude large-sized parameters from serialization of Table and Partition thrift objects in HMS notifications

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616577#comment-16616577
 ] 

Hive QA commented on HIVE-20545:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939807/HIVE-20545.1.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13820/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13820/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13820/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-09-16 03:36:02.443
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-13820/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-09-16 03:36:02.447
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 37120b8 Missed files: HIVE-20524: Schema Evolution checking is 
broken in going from Hive version 2 to version 3 for ALTER TABLE VARCHAR to 
DECIMAL
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 37120b8 Missed files: HIVE-20524: Schema Evolution checking is 
broken in going from Hive version 2 to version 3 for ALTER TABLE VARCHAR to 
DECIMAL
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-09-16 03:36:03.992
+ rm -rf ../yetus_PreCommit-HIVE-Build-13820
+ mkdir ../yetus_PreCommit-HIVE-Build-13820
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-13820
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-13820/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
 does not exist in index
error: 
metastore-server/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
 does not exist in index
error: 
metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/json/JSONMessageFactory.java:
 does not exist in index
error: src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java: 
does not exist in index
error: 
src/main/java/org/apache/hadoop/hive/metastore/messaging/json/JSONMessageFactory.java:
 does not exist in index
The patch does not appear to apply with p0, p1, or p2
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-13820
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12939807 - PreCommit-HIVE-Build

> Exclude large-sized parameters from serialization of Table and Partition 
> thrift objects in HMS notifications
> 
>
> Key: HIVE-20545
> URL: https://issues.apache.org/jira/browse/HIVE-20545
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20545.1.patch
>
>
> Clients can add large-sized parameters in Table/Partition objects. So we need 
> to enable adding regex patterns through HiveConf to match parameters to be 
> filtered from table and partition objects before serialization in HMS 
> notifications.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20538) Allow to store a key value together with a transaction.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616576#comment-16616576
 ] 

Hive QA commented on HIVE-20538:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939846/HIVE-20538.3.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 14943 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13819/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13819/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13819/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12939846 - PreCommit-HIVE-Build

> Allow to store a key value together with a transaction.
> ---
>
> Key: HIVE-20538
> URL: https://issues.apache.org/jira/browse/HIVE-20538
> Project: Hive
>  Issue Type: New Feature
>  Components: Standalone Metastore, Transactions
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20538.1.patch, HIVE-20538.1.patch, 
> HIVE-20538.2.patch, HIVE-20538.3.patch
>
>
> This can be useful for example to know if a transaction has already happened.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20538) Allow to store a key value together with a transaction.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616570#comment-16616570
 ] 

Hive QA commented on HIVE-20538:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  2m 
23s{color} | {color:blue} standalone-metastore/metastore-common in master has 
28 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
14s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
13s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 52s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13819/dev-support/hive-personality.sh
 |
| git revision | master / 37120b8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13819/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13819/yetus/whitespace-eol.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13819/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
 |
| modules | C: standalone-metastore/metastore-common 
standalone-metastore/metastore-server U: standalone-metastore |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13819/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Allow to store a key value together with a transaction.
> ---
>
> Key: HIVE-20538
> URL: https://issues.apache.org/jira/browse/HIVE-20538
> Project: Hive
>  Issue Type: New Feature
>  Components: Standalone Metastore, Transactions
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20538.1.patch, HIVE-20538.1.patch, 
> HIVE-20538.2.patch, HIVE-20538.3.patch
>
>
> This can be useful for example to know if a transaction has already happened.



--
This message was sent by Atlassian JIRA
(v7.6.3#7600

[jira] [Commented] (HIVE-17040) Join elimination in the presence of FK relationship

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616568#comment-16616568
 ] 

Hive QA commented on HIVE-17040:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939794/HIVE-17040.06.patch

{color:green}SUCCESS:{color} +1 due to 10 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 14944 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13818/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13818/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13818/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12939794 - PreCommit-HIVE-Build

> Join elimination in the presence of FK relationship
> ---
>
> Key: HIVE-17040
> URL: https://issues.apache.org/jira/browse/HIVE-17040
> Project: Hive
>  Issue Type: Sub-task
>  Components: Logical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-17040.01.patch, HIVE-17040.02.patch, 
> HIVE-17040.04.patch, HIVE-17040.05.patch, HIVE-17040.06.patch, 
> HIVE-17040.patch
>
>
> If the PK/UK table is not filtered, we can safely remove the join.
> A simple example:
> {code:sql}
> SELECT c_current_cdemo_sk
> FROM customer, customer_address
> ON c_current_addr_sk = ca_address_sk;
> {code}
> As a Calcite rule, we could implement this rewriting by 1) matching a Project 
> on top of a Join operator, 2) checking that only columns from the FK are used 
> in the Project, 3) checking that the join condition matches the FK - PK/UK 
> relationship, 4) pulling all the predicates from the PK/UK side and checking 
> that the input is not filtered, and 5) removing the join, possibly adding a 
> IS NOT NULL condition on the join column from the FK side.
> If the PK/UK table is filtered, we should still transform the Join into a 
> SemiJoin operator.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17040) Join elimination in the presence of FK relationship

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616566#comment-16616566
 ] 

Hive QA commented on HIVE-17040:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
38s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  3m 
39s{color} | {color:red} root in master failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
19s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
29s{color} | {color:blue} common in master has 64 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
52s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  7m 
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  3m 
48s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  3m 48s{color} 
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
43s{color} | {color:red} ql: The patch generated 13 new + 173 unchanged - 7 
fixed = 186 total (was 180) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 28 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  4m 
17s{color} | {color:red} ql generated 1 new + 2311 unchanged - 0 fixed = 2312 
total (was 2311) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  8m  
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 17s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  Dead store to p1 in 
org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveJoinConstraintsRule$EquivalenceClasses.addEquivalenceClass(RexTableInputRef,
 RexTableInputRef)  At 
HiveJoinConstraintsRule.java:org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveJoinConstraintsRule$EquivalenceClasses.addEquivalenceClass(RexTableInputRef,
 RexTableInputRef)  At HiveJoinConstraintsRule.java:[line 453] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13818/dev-support/hive-personality.sh
 |
| git revision | master / 37120b8 |
| Default Java | 1.8.0_111 |
| compile | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13818/yetus/branch-compile-root.txt
 |
| findbugs | v3.0.0 |
| compile | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13818/yetus/patch-compile-root.txt
 |
| javac | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13818/yetus/patch-compile-root.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13818/yetus/diff-checkstyle-ql.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13818/yetus/whitespace-eol.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13818/yetus/new-findbugs-ql.html
 |
| modules | C: common . itests ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13818/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically gen

[jira] [Commented] (HIVE-20568) There is no need to convert the dbname to pattern while pulling tablemeta

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616555#comment-16616555
 ] 

Hive QA commented on HIVE-20568:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939798/HIVE-20568.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 14941 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13817/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13817/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13817/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12939798 - PreCommit-HIVE-Build

> There is no need to convert the dbname to pattern while pulling tablemeta
> -
>
> Key: HIVE-20568
> URL: https://issues.apache.org/jira/browse/HIVE-20568
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Affects Versions: 0.4.0
> Environment: Hive-4,Java-8
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Minor
> Attachments: HIVE-20568.patch
>
>
> there is no need to convert the dbname to pattern, dbNamePattern is just a 
> dbName which we are passing to getTableMeta
> https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/GetTablesOperation.java#L117



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20516) alter table drop partition should be compatible with old metastore, as partition pruner does

[jinzheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-20516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jinzheng updated HIVE-20516:

Attachment: (was: temp.diff)

> alter table drop partition should be compatible with old metastore, as 
> partition pruner does
> 
>
> Key: HIVE-20516
> URL: https://issues.apache.org/jira/browse/HIVE-20516
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 3.0.0
> Environment: all
>Reporter: jinzheng
>Assignee: jinzheng
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: temp.diff
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
>     After the change of  HIVE-4914, we always push the partition expression 
> to metastore, to avoid filtering partition by partition names.
>     And HIVE-4914 added some protection in Partition pruner, in case that 
> metastore may not have api of get_partitions_by_expr.
>     Therefore, we should also add similar protection to another calling 
> point, when dealing with "alter table drop partition".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20516) alter table drop partition should be compatible with old metastore, as partition pruner does

[jinzheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-20516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jinzheng updated HIVE-20516:

Attachment: temp.diff

> alter table drop partition should be compatible with old metastore, as 
> partition pruner does
> 
>
> Key: HIVE-20516
> URL: https://issues.apache.org/jira/browse/HIVE-20516
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 3.0.0
> Environment: all
>Reporter: jinzheng
>Assignee: jinzheng
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: temp.diff
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
>     After the change of  HIVE-4914, we always push the partition expression 
> to metastore, to avoid filtering partition by partition names.
>     And HIVE-4914 added some protection in Partition pruner, in case that 
> metastore may not have api of get_partitions_by_expr.
>     Therefore, we should also add similar protection to another calling 
> point, when dealing with "alter table drop partition".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18908) FULL OUTER JOIN to MapJoin

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-18908:

Attachment: HIVE-18908.09998.patch

> FULL OUTER JOIN to MapJoin
> --
>
> Key: HIVE-18908
> URL: https://issues.apache.org/jira/browse/HIVE-18908
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: FULL OUTER MapJoin Code Changes.docx, 
> HIVE-18908.01.patch, HIVE-18908.02.patch, HIVE-18908.03.patch, 
> HIVE-18908.04.patch, HIVE-18908.05.patch, HIVE-18908.06.patch, 
> HIVE-18908.08.patch, HIVE-18908.09.patch, HIVE-18908.091.patch, 
> HIVE-18908.092.patch, HIVE-18908.093.patch, HIVE-18908.096.patch, 
> HIVE-18908.097.patch, HIVE-18908.098.patch, HIVE-18908.099.patch, 
> HIVE-18908.0991.patch, HIVE-18908.0992.patch, HIVE-18908.0993.patch, 
> HIVE-18908.0994.patch, HIVE-18908.0995.patch, HIVE-18908.0996.patch, 
> HIVE-18908.0997.patch, HIVE-18908.0998.patch, HIVE-18908.0999.patch, 
> HIVE-18908.09991.patch, HIVE-18908.09992.patch, HIVE-18908.09993.patch, 
> HIVE-18908.09994.patch, HIVE-18908.09995.patch, HIVE-18908.09996.patch, 
> HIVE-18908.09997.patch, HIVE-18908.09998.patch, JOIN to MAPJOIN 
> Transformation.pdf, SHARED-MEMORY FULL OUTER MapJoin.pdf
>
>
> Currently, we do not support FULL OUTER JOIN in MapJoin.
> Rough TPC-DS timings run on laptop:
> (NOTE: Query 51 has PTF as a bigger serial portion -- Amdahl's law at play)
> FULL OUTER MapJoin OFF =  MergeJoin
> Query 51:
> o Vectorization OFF
> • FULL OUTER MapJoin OFF: 4:30 minutes
> • FULL OUTER MapJoin ON: 4:37 minutes
> o Vectorization ON
> • FULL OUTER MapJoin OFF: 2:35 minutes
> • FULL OUTER MapJoin ON: 1:47 minutes
> Query 97:
> o Vectorization OFF
> • FULL OUTER MapJoin OFF: 2:37 minutes
> • FULL OUTER MapJoin ON: 2:42 minutes
> o Vectorization ON
> • FULL OUTER MapJoin OFF: 1:17 minutes
> • FULL OUTER MapJoin ON: 0:06 minutes
> FULL OUTER Join 10,000,000 rows against 323,910 small table keys
> o Vectorization ON
> • FULL OUTER MapJoin OFF: 14:56 minutes
> • FULL OUTER MapJoin ON: 1:45 minutes
> FULL OUTER Join 10,000,000 rows against 1,000 small table keys
> o Vectorization ON
> • FULL OUTER MapJoin OFF: 12:37 minutes
> • FULL OUTER MapJoin ON: 1:38 minutes
> Hopefully, someone will do large scale cluster testing.  
> [DynamicPartitionedHashJoin] MapJoin should scale dramatically better than 
> [Sort] MergeJoin reduce-shuffle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18908) FULL OUTER JOIN to MapJoin

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-18908:

Status: Patch Available  (was: In Progress)

> FULL OUTER JOIN to MapJoin
> --
>
> Key: HIVE-18908
> URL: https://issues.apache.org/jira/browse/HIVE-18908
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: FULL OUTER MapJoin Code Changes.docx, 
> HIVE-18908.01.patch, HIVE-18908.02.patch, HIVE-18908.03.patch, 
> HIVE-18908.04.patch, HIVE-18908.05.patch, HIVE-18908.06.patch, 
> HIVE-18908.08.patch, HIVE-18908.09.patch, HIVE-18908.091.patch, 
> HIVE-18908.092.patch, HIVE-18908.093.patch, HIVE-18908.096.patch, 
> HIVE-18908.097.patch, HIVE-18908.098.patch, HIVE-18908.099.patch, 
> HIVE-18908.0991.patch, HIVE-18908.0992.patch, HIVE-18908.0993.patch, 
> HIVE-18908.0994.patch, HIVE-18908.0995.patch, HIVE-18908.0996.patch, 
> HIVE-18908.0997.patch, HIVE-18908.0998.patch, HIVE-18908.0999.patch, 
> HIVE-18908.09991.patch, HIVE-18908.09992.patch, HIVE-18908.09993.patch, 
> HIVE-18908.09994.patch, HIVE-18908.09995.patch, HIVE-18908.09996.patch, 
> HIVE-18908.09997.patch, HIVE-18908.09998.patch, JOIN to MAPJOIN 
> Transformation.pdf, SHARED-MEMORY FULL OUTER MapJoin.pdf
>
>
> Currently, we do not support FULL OUTER JOIN in MapJoin.
> Rough TPC-DS timings run on laptop:
> (NOTE: Query 51 has PTF as a bigger serial portion -- Amdahl's law at play)
> FULL OUTER MapJoin OFF =  MergeJoin
> Query 51:
> o Vectorization OFF
> • FULL OUTER MapJoin OFF: 4:30 minutes
> • FULL OUTER MapJoin ON: 4:37 minutes
> o Vectorization ON
> • FULL OUTER MapJoin OFF: 2:35 minutes
> • FULL OUTER MapJoin ON: 1:47 minutes
> Query 97:
> o Vectorization OFF
> • FULL OUTER MapJoin OFF: 2:37 minutes
> • FULL OUTER MapJoin ON: 2:42 minutes
> o Vectorization ON
> • FULL OUTER MapJoin OFF: 1:17 minutes
> • FULL OUTER MapJoin ON: 0:06 minutes
> FULL OUTER Join 10,000,000 rows against 323,910 small table keys
> o Vectorization ON
> • FULL OUTER MapJoin OFF: 14:56 minutes
> • FULL OUTER MapJoin ON: 1:45 minutes
> FULL OUTER Join 10,000,000 rows against 1,000 small table keys
> o Vectorization ON
> • FULL OUTER MapJoin OFF: 12:37 minutes
> • FULL OUTER MapJoin ON: 1:38 minutes
> Hopefully, someone will do large scale cluster testing.  
> [DynamicPartitionedHashJoin] MapJoin should scale dramatically better than 
> [Sort] MergeJoin reduce-shuffle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18908) FULL OUTER JOIN to MapJoin

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-18908:

Status: In Progress  (was: Patch Available)

> FULL OUTER JOIN to MapJoin
> --
>
> Key: HIVE-18908
> URL: https://issues.apache.org/jira/browse/HIVE-18908
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: FULL OUTER MapJoin Code Changes.docx, 
> HIVE-18908.01.patch, HIVE-18908.02.patch, HIVE-18908.03.patch, 
> HIVE-18908.04.patch, HIVE-18908.05.patch, HIVE-18908.06.patch, 
> HIVE-18908.08.patch, HIVE-18908.09.patch, HIVE-18908.091.patch, 
> HIVE-18908.092.patch, HIVE-18908.093.patch, HIVE-18908.096.patch, 
> HIVE-18908.097.patch, HIVE-18908.098.patch, HIVE-18908.099.patch, 
> HIVE-18908.0991.patch, HIVE-18908.0992.patch, HIVE-18908.0993.patch, 
> HIVE-18908.0994.patch, HIVE-18908.0995.patch, HIVE-18908.0996.patch, 
> HIVE-18908.0997.patch, HIVE-18908.0998.patch, HIVE-18908.0999.patch, 
> HIVE-18908.09991.patch, HIVE-18908.09992.patch, HIVE-18908.09993.patch, 
> HIVE-18908.09994.patch, HIVE-18908.09995.patch, HIVE-18908.09996.patch, 
> HIVE-18908.09997.patch, JOIN to MAPJOIN Transformation.pdf, SHARED-MEMORY 
> FULL OUTER MapJoin.pdf
>
>
> Currently, we do not support FULL OUTER JOIN in MapJoin.
> Rough TPC-DS timings run on laptop:
> (NOTE: Query 51 has PTF as a bigger serial portion -- Amdahl's law at play)
> FULL OUTER MapJoin OFF =  MergeJoin
> Query 51:
> o Vectorization OFF
> • FULL OUTER MapJoin OFF: 4:30 minutes
> • FULL OUTER MapJoin ON: 4:37 minutes
> o Vectorization ON
> • FULL OUTER MapJoin OFF: 2:35 minutes
> • FULL OUTER MapJoin ON: 1:47 minutes
> Query 97:
> o Vectorization OFF
> • FULL OUTER MapJoin OFF: 2:37 minutes
> • FULL OUTER MapJoin ON: 2:42 minutes
> o Vectorization ON
> • FULL OUTER MapJoin OFF: 1:17 minutes
> • FULL OUTER MapJoin ON: 0:06 minutes
> FULL OUTER Join 10,000,000 rows against 323,910 small table keys
> o Vectorization ON
> • FULL OUTER MapJoin OFF: 14:56 minutes
> • FULL OUTER MapJoin ON: 1:45 minutes
> FULL OUTER Join 10,000,000 rows against 1,000 small table keys
> o Vectorization ON
> • FULL OUTER MapJoin OFF: 12:37 minutes
> • FULL OUTER MapJoin ON: 1:38 minutes
> Hopefully, someone will do large scale cluster testing.  
> [DynamicPartitionedHashJoin] MapJoin should scale dramatically better than 
> [Sort] MergeJoin reduce-shuffle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20568) There is no need to convert the dbname to pattern while pulling tablemeta

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616548#comment-16616548
 ] 

Hive QA commented on HIVE-20568:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
37s{color} | {color:blue} service in master has 48 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 42s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13817/dev-support/hive-personality.sh
 |
| git revision | master / 37120b8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: service U: service |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13817/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> There is no need to convert the dbname to pattern while pulling tablemeta
> -
>
> Key: HIVE-20568
> URL: https://issues.apache.org/jira/browse/HIVE-20568
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Affects Versions: 0.4.0
> Environment: Hive-4,Java-8
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Minor
> Attachments: HIVE-20568.patch
>
>
> there is no need to convert the dbname to pattern, dbNamePattern is just a 
> dbName which we are passing to getTableMeta
> https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/GetTablesOperation.java#L117



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20291) Allow HiveStreamingConnection to receive a WriteId

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616544#comment-16616544
 ] 

Hive QA commented on HIVE-20291:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939796/HIVE-20291.6.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14945 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[schema_evol_undecorated] 
(batchId=58)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery (batchId=251)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13816/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13816/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13816/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12939796 - PreCommit-HIVE-Build

> Allow HiveStreamingConnection to receive a WriteId
> --
>
> Key: HIVE-20291
> URL: https://issues.apache.org/jira/browse/HIVE-20291
> Project: Hive
>  Issue Type: Improvement
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-20291.1.patch, HIVE-20291.2.patch, 
> HIVE-20291.3.patch, HIVE-20291.4.patch, HIVE-20291.5.patch, HIVE-20291.6.patch
>
>
> If the writeId is received externally it won't need to open connections to 
> the metastore. It won't be able to the commit in this case as well so it must 
> be done by the entity passing the writeId.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20291) Allow HiveStreamingConnection to receive a WriteId

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616539#comment-16616539
 ] 

Hive QA commented on HIVE-20291:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
28s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
15s{color} | {color:red} streaming: The patch generated 3 new + 488 unchanged - 
86 fixed = 491 total (was 574) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
16s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 12s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13816/dev-support/hive-personality.sh
 |
| git revision | master / e691294 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13816/yetus/diff-checkstyle-streaming.txt
 |
| modules | C: streaming U: streaming |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13816/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Allow HiveStreamingConnection to receive a WriteId
> --
>
> Key: HIVE-20291
> URL: https://issues.apache.org/jira/browse/HIVE-20291
> Project: Hive
>  Issue Type: Improvement
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-20291.1.patch, HIVE-20291.2.patch, 
> HIVE-20291.3.patch, HIVE-20291.4.patch, HIVE-20291.5.patch, HIVE-20291.6.patch
>
>
> If the writeId is received externally it won't need to open connections to 
> the metastore. It won't be able to the commit in this case as well so it must 
> be done by the entity passing the writeId.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20462) "CREATE VIEW IF NOT EXISTS" fails if view already exists

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-20462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-20462:
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 4.0.0
   Status: Resolved  (was: Patch Available)

Patch pushed to master. Thanks Jesus for review!

> "CREATE VIEW IF NOT EXISTS" fails if view already exists
> 
>
> Key: HIVE-20462
> URL: https://issues.apache.org/jira/browse/HIVE-20462
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.1.0
> Environment: HDP 3.0
> Hive 3.1.0.3.0.0.0-1634
>Reporter: static-max
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20462.1.patch
>
>
> After upgrading to Hive 3.1 from Hive 1.2, I can't run my queries anymore. It 
> seems that "if not exists" is ignored in "create view".
> *Example:*
>  # create view if not exists test as select 1;
>  # create view if not exists test as select 1;
> The second call fails with "Table already exists: default.test". I changed 
> this to "create or replace view" as a workaround.
> Also documented [link 
> here|https://community.hortonworks.com/questions/214772/hdp-3-hive-create-view-if-not-exists-fails.html|]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-14431) Recognize COALESCE as CASE

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616534#comment-16616534
 ] 

Hive QA commented on HIVE-14431:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939788/HIVE-14431.04.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 14941 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[deleteAnalyze] 
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby13] (batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[innerjoin1] (batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_cond_pushdown_unqual5]
 (batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[schema_evol_undecorated] 
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[semijoin4] (batchId=92)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[semijoin5] (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_coalesce] 
(batchId=11)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[semijoin6] 
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_coalesce]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_coalesce_2]
 (batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_coalesce_3]
 (batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_coalesce_4]
 (batchId=158)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query40] 
(batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query49] 
(batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query67] 
(batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query75] 
(batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query77] 
(batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query78] 
(batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query80] 
(batchId=266)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query78] 
(batchId=264)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13815/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13815/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13815/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 21 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12939788 - PreCommit-HIVE-Build

> Recognize COALESCE as CASE
> --
>
> Key: HIVE-14431
> URL: https://issues.apache.org/jira/browse/HIVE-14431
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-14431.01.patch, HIVE-14431.03.patch, 
> HIVE-14431.04.patch, HIVE-14431.2.patch, HIVE-14431.patch
>
>
> Transform:
> {code}
> (COALESCE(a, '')  = '') OR
>(a = 'A' AND b = c)  OR
>(a = 'B' AND b = d) OR
>(a = 'C' AND b = e) OR
>(a = 'D' AND b = f) OR
>(a = 'E' AND b = g) OR
>(a = 'F' AND b = h)
> {code}
> into:
> {code}
> (a='') OR
>(a is null) OR
>(a = 'A' AND b = c)  OR
>(a = 'B' AND b = d) OR
>(a = 'C' AND b = e) OR
>(a = 'D' AND b = f) OR
>(a = 'E' AND b = g) OR
>(a = 'F' AND b = h)
> {code}
> With complex queries, this will lead us to factor more predicates that could 
> be pushed to the TS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20462) "CREATE VIEW IF NOT EXISTS" fails if view already exists

[Jesus Camacho Rodriguez (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616530#comment-16616530
 ] 

Jesus Camacho Rodriguez commented on HIVE-20462:


+1

> "CREATE VIEW IF NOT EXISTS" fails if view already exists
> 
>
> Key: HIVE-20462
> URL: https://issues.apache.org/jira/browse/HIVE-20462
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.1.0
> Environment: HDP 3.0
> Hive 3.1.0.3.0.0.0-1634
>Reporter: static-max
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20462.1.patch
>
>
> After upgrading to Hive 3.1 from Hive 1.2, I can't run my queries anymore. It 
> seems that "if not exists" is ignored in "create view".
> *Example:*
>  # create view if not exists test as select 1;
>  # create view if not exists test as select 1;
> The second call fails with "Table already exists: default.test". I changed 
> this to "create or replace view" as a workaround.
> Also documented [link 
> here|https://community.hortonworks.com/questions/214772/hdp-3-hive-create-view-if-not-exists-fails.html|]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-14431) Recognize COALESCE as CASE

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616529#comment-16616529
 ] 

Hive QA commented on HIVE-14431:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m 
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
11s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
40s{color} | {color:red} ql: The patch generated 2 new + 80 unchanged - 0 fixed 
= 82 total (was 80) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 19s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13815/dev-support/hive-personality.sh
 |
| git revision | master / e691294 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13815/yetus/diff-checkstyle-ql.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13815/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Recognize COALESCE as CASE
> --
>
> Key: HIVE-14431
> URL: https://issues.apache.org/jira/browse/HIVE-14431
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-14431.01.patch, HIVE-14431.03.patch, 
> HIVE-14431.04.patch, HIVE-14431.2.patch, HIVE-14431.patch
>
>
> Transform:
> {code}
> (COALESCE(a, '')  = '') OR
>(a = 'A' AND b = c)  OR
>(a = 'B' AND b = d) OR
>(a = 'C' AND b = e) OR
>(a = 'D' AND b = f) OR
>(a = 'E' AND b = g) OR
>(a = 'F' AND b = h)
> {code}
> into:
> {code}
> (a='') OR
>(a is null) OR
>(a = 'A' AND b = c)  OR
>(a = 'B' AND b = d) OR
>(a = 'C' AND b = e) OR
>(a = 'D' AND b = f) OR
>(a = 'E' AND b = g) OR
>(a = 'F' AND b = h)
> {code}
> With complex queries, this will lead us to factor more predicates that could 
> be pushed to the TS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18908) FULL OUTER JOIN to MapJoin

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616528#comment-16616528
 ] 

Hive QA commented on HIVE-18908:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 9s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  3m 
42s{color} | {color:red} root in master failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  3m 
 3s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
24s{color} | {color:blue} storage-api in master has 48 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
32s{color} | {color:blue} common in master has 64 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
42s{color} | {color:blue} serde in master has 195 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 11m 
51s{color} | {color:red} branch/itests/hive-jmh cannot run convertXmlToText 
from findbugs {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
51s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  8m 
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  3m 
34s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  3m 34s{color} 
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
11s{color} | {color:green} The patch storage-api passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
18s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
21s{color} | {color:green} The patch serde passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
34s{color} | {color:green} The patch . passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} itests/hive-jmh: The patch generated 0 new + 11 
unchanged - 6 fixed = 11 total (was 17) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m 
33s{color} | {color:red} ql: The patch generated 312 new + 3561 unchanged - 174 
fixed = 3873 total (was 3735) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 13 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
20s{color} | {color:red} patch/storage-api cannot run setBugDatabaseInfo from 
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
31s{color} | {color:red} patch/common cannot run setBugDatabaseInfo from 
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
41s{color} | {color:red} patch/serde cannot run setBugDatabaseInfo from 
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  3m 
34s{color} | {color:red} patch/itests/hive-jmh cannot run setBugDatabaseInfo 
from findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  7m  
4s{color} | {color:red} patch/ql cannot run setBugDatabaseInfo from findbugs 
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  9m 
15s{color} | {color:red} root generated 2 new + 388 unchanged - 0 fixed = 390 
total (was 388) {color} |
| {color:red}-1{color} | {color:red} j

[jira] [Commented] (HIVE-18908) FULL OUTER JOIN to MapJoin

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616522#comment-16616522
 ] 

Hive QA commented on HIVE-18908:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939785/HIVE-18908.09997.patch

{color:green}SUCCESS:{color} +1 due to 64 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 14966 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[schema_evol_undecorated] 
(batchId=58)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_fullouter_mapjoin_1_fast]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_fullouter_mapjoin_1_optimized]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_fullouter_mapjoin_1_optimized_passthru]
 (batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing]
 (batchId=181)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13814/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13814/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13814/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12939785 - PreCommit-HIVE-Build

> FULL OUTER JOIN to MapJoin
> --
>
> Key: HIVE-18908
> URL: https://issues.apache.org/jira/browse/HIVE-18908
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: FULL OUTER MapJoin Code Changes.docx, 
> HIVE-18908.01.patch, HIVE-18908.02.patch, HIVE-18908.03.patch, 
> HIVE-18908.04.patch, HIVE-18908.05.patch, HIVE-18908.06.patch, 
> HIVE-18908.08.patch, HIVE-18908.09.patch, HIVE-18908.091.patch, 
> HIVE-18908.092.patch, HIVE-18908.093.patch, HIVE-18908.096.patch, 
> HIVE-18908.097.patch, HIVE-18908.098.patch, HIVE-18908.099.patch, 
> HIVE-18908.0991.patch, HIVE-18908.0992.patch, HIVE-18908.0993.patch, 
> HIVE-18908.0994.patch, HIVE-18908.0995.patch, HIVE-18908.0996.patch, 
> HIVE-18908.0997.patch, HIVE-18908.0998.patch, HIVE-18908.0999.patch, 
> HIVE-18908.09991.patch, HIVE-18908.09992.patch, HIVE-18908.09993.patch, 
> HIVE-18908.09994.patch, HIVE-18908.09995.patch, HIVE-18908.09996.patch, 
> HIVE-18908.09997.patch, JOIN to MAPJOIN Transformation.pdf, SHARED-MEMORY 
> FULL OUTER MapJoin.pdf
>
>
> Currently, we do not support FULL OUTER JOIN in MapJoin.
> Rough TPC-DS timings run on laptop:
> (NOTE: Query 51 has PTF as a bigger serial portion -- Amdahl's law at play)
> FULL OUTER MapJoin OFF =  MergeJoin
> Query 51:
> o Vectorization OFF
> • FULL OUTER MapJoin OFF: 4:30 minutes
> • FULL OUTER MapJoin ON: 4:37 minutes
> o Vectorization ON
> • FULL OUTER MapJoin OFF: 2:35 minutes
> • FULL OUTER MapJoin ON: 1:47 minutes
> Query 97:
> o Vectorization OFF
> • FULL OUTER MapJoin OFF: 2:37 minutes
> • FULL OUTER MapJoin ON: 2:42 minutes
> o Vectorization ON
> • FULL OUTER MapJoin OFF: 1:17 minutes
> • FULL OUTER MapJoin ON: 0:06 minutes
> FULL OUTER Join 10,000,000 rows against 323,910 small table keys
> o Vectorization ON
> • FULL OUTER MapJoin OFF: 14:56 minutes
> • FULL OUTER MapJoin ON: 1:45 minutes
> FULL OUTER Join 10,000,000 rows against 1,000 small table keys
> o Vectorization ON
> • FULL OUTER MapJoin OFF: 12:37 minutes
> • FULL OUTER MapJoin ON: 1:38 minutes
> Hopefully, someone will do large scale cluster testing.  
> [DynamicPartitionedHashJoin] MapJoin should scale dramatically better than 
> [Sort] MergeJoin reduce-shuffle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20325) FlakyTest: TestMiniDruidCliDriver

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-20325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-20325:

   Resolution: Fixed
Fix Version/s: 4.0.0
   Status: Resolved  (was: Patch Available)

Pushed to master. Thanks, Slim!

> FlakyTest: TestMiniDruidCliDriver
> -
>
> Key: HIVE-20325
> URL: https://issues.apache.org/jira/browse/HIVE-20325
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: slim bouguerra
>Priority: Blocker
> Fix For: 4.0.0
>
> Attachments: HIVE-20325.patch
>
>
> TestMiniDruidCliDriver is failing intermittently a significant percentage of 
> the time.
> druid_timestamptz
> druidmini_joins
> druidmini_masking
> druidmini_test1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20325) FlakyTest: TestMiniDruidCliDriver

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616504#comment-16616504
 ] 

Ashutosh Chauhan commented on HIVE-20325:
-

+1

> FlakyTest: TestMiniDruidCliDriver
> -
>
> Key: HIVE-20325
> URL: https://issues.apache.org/jira/browse/HIVE-20325
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: slim bouguerra
>Priority: Blocker
> Attachments: HIVE-20325.patch
>
>
> TestMiniDruidCliDriver is failing intermittently a significant percentage of 
> the time.
> druid_timestamptz
> druidmini_joins
> druidmini_masking
> druidmini_test1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616495#comment-16616495
 ] 

Hive QA commented on HIVE-20306:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939786/HIVE-20306.16.patch

{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 14973 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_basic]
 (batchId=264)
org.apache.hive.jdbc.miniHS2.TestHs2ConnectionMetricsBinary.testOpenConnectionMetrics
 (batchId=255)
org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthBinary.testAuthorization1 
(batchId=265)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13813/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13813/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13813/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12939786 - PreCommit-HIVE-Build

> Implement projection spec for fetching only requested fields from partitions
> 
>
> Key: HIVE-20306
> URL: https://issues.apache.org/jira/browse/HIVE-20306
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch, 
> HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch, 
> HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.09.patch, 
> HIVE-20306.10.patch, HIVE-20306.11.patch, HIVE-20306.12.patch, 
> HIVE-20306.13.patch, HIVE-20306.14.patch, HIVE-20306.15.patch, 
> HIVE-20306.16.patch, HIVE-20306.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20524) Schema Evolution checking is broken in going from Hive version 2 to version 3 for ALTER TABLE VARCHAR to DECIMAL

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-20524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-20524:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Schema Evolution checking is broken in going from Hive version 2 to version 3 
> for ALTER TABLE VARCHAR to DECIMAL
> 
>
> Key: HIVE-20524
> URL: https://issues.apache.org/jira/browse/HIVE-20524
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 4.0.0
>
> Attachments: HIVE-20524.01.patch, HIVE-20524.02.patch, 
> HIVE-20524.03.patch, HIVE-20524.04.patch
>
>
> Issue that started this JIRA:
> {code}
> create external table varchar_decimal (c1 varchar(25));
> alter table varchar_decimal change c1 c1 decimal(31,0);
> ERROR : FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. The following 
> columns have types incompatible with the existing columns in their respective 
> positions :
> c1
> {code}
> There appear to be 2 issues here:
> 1) When hive.metastore.disallow.incompatible.col.type.changes is true (the 
> default) we only allow StringFamily (STRING, CHAR, VARCHAR) conversion to a 
> number that can hold the largest numbers.  The theory being we don't want 
> data loss you would get by converting the StringFamily field into integers, 
> etc.  In Hive version 2 the hierarchy of numbers had DECIMAL at the top.  At 
> some point during Hive version 2 we realized this was incorrect and put 
> DOUBLE the top.
> However, the Hive2 Hive version 2 TypeInfoUtils.implicitConversion method 
> allows StringFamily to either DOUBLE or DECIMAL conversion.
> The new org.apache.hadoop.hive.metastore.ColumnType class under Hive version 
> 3 hive-standalone-metadata-server method checkColTypeChangeCompatible only 
> allows DOUBLE.
> This JIRA fixes that problem.
> 2) Also, the checkColTypeChangeCompatible method lost a version 2 series bug 
> fix that drops CHAR/VARCHAR (and DECIMAL I think) type decorations when 
> checking for Schema Evolution compatibility.  So, when that code is checking 
> if a data type "varchar(25)" is StringFamily it fails because the "(25)" 
> didn't get removed properly.
> This JIRA fixes issue #2 also.
> NOTE: Hive1 version 2 did undecoratedTypeName(oldType) and Hive2 version 
> performed the logic in TypeInfoUtils.implicitConvertible on the 
> PrimitiveCategory not the raw type string.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20524) Schema Evolution checking is broken in going from Hive version 2 to version 3 for ALTER TABLE VARCHAR to DECIMAL

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-20524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-20524:

Fix Version/s: 4.0.0

> Schema Evolution checking is broken in going from Hive version 2 to version 3 
> for ALTER TABLE VARCHAR to DECIMAL
> 
>
> Key: HIVE-20524
> URL: https://issues.apache.org/jira/browse/HIVE-20524
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 4.0.0
>
> Attachments: HIVE-20524.01.patch, HIVE-20524.02.patch, 
> HIVE-20524.03.patch, HIVE-20524.04.patch
>
>
> Issue that started this JIRA:
> {code}
> create external table varchar_decimal (c1 varchar(25));
> alter table varchar_decimal change c1 c1 decimal(31,0);
> ERROR : FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. The following 
> columns have types incompatible with the existing columns in their respective 
> positions :
> c1
> {code}
> There appear to be 2 issues here:
> 1) When hive.metastore.disallow.incompatible.col.type.changes is true (the 
> default) we only allow StringFamily (STRING, CHAR, VARCHAR) conversion to a 
> number that can hold the largest numbers.  The theory being we don't want 
> data loss you would get by converting the StringFamily field into integers, 
> etc.  In Hive version 2 the hierarchy of numbers had DECIMAL at the top.  At 
> some point during Hive version 2 we realized this was incorrect and put 
> DOUBLE the top.
> However, the Hive2 Hive version 2 TypeInfoUtils.implicitConversion method 
> allows StringFamily to either DOUBLE or DECIMAL conversion.
> The new org.apache.hadoop.hive.metastore.ColumnType class under Hive version 
> 3 hive-standalone-metadata-server method checkColTypeChangeCompatible only 
> allows DOUBLE.
> This JIRA fixes that problem.
> 2) Also, the checkColTypeChangeCompatible method lost a version 2 series bug 
> fix that drops CHAR/VARCHAR (and DECIMAL I think) type decorations when 
> checking for Schema Evolution compatibility.  So, when that code is checking 
> if a data type "varchar(25)" is StringFamily it fails because the "(25)" 
> didn't get removed properly.
> This JIRA fixes issue #2 also.
> NOTE: Hive1 version 2 did undecoratedTypeName(oldType) and Hive2 version 
> performed the logic in TypeInfoUtils.implicitConvertible on the 
> PrimitiveCategory not the raw type string.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20524) Schema Evolution checking is broken in going from Hive version 2 to version 3 for ALTER TABLE VARCHAR to DECIMAL

[Matt McCline (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616492#comment-16616492
 ] 

Matt McCline commented on HIVE-20524:
-

Committed to master. [~jdere] thank you for your code review!

> Schema Evolution checking is broken in going from Hive version 2 to version 3 
> for ALTER TABLE VARCHAR to DECIMAL
> 
>
> Key: HIVE-20524
> URL: https://issues.apache.org/jira/browse/HIVE-20524
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 4.0.0
>
> Attachments: HIVE-20524.01.patch, HIVE-20524.02.patch, 
> HIVE-20524.03.patch, HIVE-20524.04.patch
>
>
> Issue that started this JIRA:
> {code}
> create external table varchar_decimal (c1 varchar(25));
> alter table varchar_decimal change c1 c1 decimal(31,0);
> ERROR : FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. The following 
> columns have types incompatible with the existing columns in their respective 
> positions :
> c1
> {code}
> There appear to be 2 issues here:
> 1) When hive.metastore.disallow.incompatible.col.type.changes is true (the 
> default) we only allow StringFamily (STRING, CHAR, VARCHAR) conversion to a 
> number that can hold the largest numbers.  The theory being we don't want 
> data loss you would get by converting the StringFamily field into integers, 
> etc.  In Hive version 2 the hierarchy of numbers had DECIMAL at the top.  At 
> some point during Hive version 2 we realized this was incorrect and put 
> DOUBLE the top.
> However, the Hive2 Hive version 2 TypeInfoUtils.implicitConversion method 
> allows StringFamily to either DOUBLE or DECIMAL conversion.
> The new org.apache.hadoop.hive.metastore.ColumnType class under Hive version 
> 3 hive-standalone-metadata-server method checkColTypeChangeCompatible only 
> allows DOUBLE.
> This JIRA fixes that problem.
> 2) Also, the checkColTypeChangeCompatible method lost a version 2 series bug 
> fix that drops CHAR/VARCHAR (and DECIMAL I think) type decorations when 
> checking for Schema Evolution compatibility.  So, when that code is checking 
> if a data type "varchar(25)" is StringFamily it fails because the "(25)" 
> didn't get removed properly.
> This JIRA fixes issue #2 also.
> NOTE: Hive1 version 2 did undecoratedTypeName(oldType) and Hive2 version 
> performed the logic in TypeInfoUtils.implicitConvertible on the 
> PrimitiveCategory not the raw type string.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18778) Needs to capture input/output entities in explain

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-18778:
--
Attachment: HIVE-18778.4.patch

> Needs to capture input/output entities in explain
> -
>
> Key: HIVE-18778
> URL: https://issues.apache.org/jira/browse/HIVE-18778
> Project: Hive
>  Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-18778-SparkPositive.patch, HIVE-18778.1.patch, 
> HIVE-18778.2.patch, HIVE-18778.3.patch, HIVE-18778.4.patch, 
> HIVE-18778_TestCliDriver.patch, HIVE-18788_SparkNegative.patch, 
> HIVE-18788_SparkPerf.patch
>
>
> With Sentry enabled, commands like explain drop table foo fail with {{explain 
> drop table foo;}}
> {code}
> Error: Error while compiling statement: FAILED: SemanticException No valid 
> privileges
>  Required privilege( Table) not available in input privileges
>  The required privileges: (state=42000,code=4)
> {code}
> Sentry fails to authorize because the ExplainSemanticAnalyzer uses an 
> instance of DDLSemanticAnalyzer to analyze the explain query.
> {code}
> BaseSemanticAnalyzer sem = SemanticAnalyzerFactory.get(conf, input);
> sem.analyze(input, ctx);
> sem.validate()
> {code}
> The inputs/outputs entities for this query are set in the above code. 
> However, these are never set on the instance of ExplainSemanticAnalyzer 
> itself and thus is not propagated into the HookContext in the calling Driver 
> code.
> {code}
> sem.analyze(tree, ctx); --> this results in calling the above code that uses 
> DDLSA
> hookCtx.update(sem); --> sem is an instance of ExplainSemanticAnalyzer, this 
> code attempts to update the HookContext with the input/output info from ESA 
> which is never set.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616485#comment-16616485
 ] 

Hive QA commented on HIVE-20306:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
 5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
27s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  2m 
22s{color} | {color:blue} standalone-metastore/metastore-common in master has 
28 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
14s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
20s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 53 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
1s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
13s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 17s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13813/dev-support/hive-personality.sh
 |
| git revision | master / 2c8e679 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13813/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13813/yetus/patch-mvninstall-itests_hcatalog-unit.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13813/yetus/whitespace-eol.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13813/yetus/whitespace-tabs.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13813/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
 |
| modules | C: standalone-metastore/metastore-common itests/hcatalog-unit 
standalone-metastore/metastore-server U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13813/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Implement projection spec for fetching only requested fields from partitions
> 
>
> Key: HIVE-20306
> URL: https://issues.apache.org/jira/browse/HIVE-20306
> Project: Hive
>  

[jira] [Commented] (HIVE-20524) Schema Evolution checking is broken in going from Hive version 2 to version 3 for ALTER TABLE VARCHAR to DECIMAL

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616463#comment-16616463
 ] 

Hive QA commented on HIVE-20524:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939780/HIVE-20524.04.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 14941 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13812/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13812/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13812/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12939780 - PreCommit-HIVE-Build

> Schema Evolution checking is broken in going from Hive version 2 to version 3 
> for ALTER TABLE VARCHAR to DECIMAL
> 
>
> Key: HIVE-20524
> URL: https://issues.apache.org/jira/browse/HIVE-20524
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20524.01.patch, HIVE-20524.02.patch, 
> HIVE-20524.03.patch, HIVE-20524.04.patch
>
>
> Issue that started this JIRA:
> {code}
> create external table varchar_decimal (c1 varchar(25));
> alter table varchar_decimal change c1 c1 decimal(31,0);
> ERROR : FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. The following 
> columns have types incompatible with the existing columns in their respective 
> positions :
> c1
> {code}
> There appear to be 2 issues here:
> 1) When hive.metastore.disallow.incompatible.col.type.changes is true (the 
> default) we only allow StringFamily (STRING, CHAR, VARCHAR) conversion to a 
> number that can hold the largest numbers.  The theory being we don't want 
> data loss you would get by converting the StringFamily field into integers, 
> etc.  In Hive version 2 the hierarchy of numbers had DECIMAL at the top.  At 
> some point during Hive version 2 we realized this was incorrect and put 
> DOUBLE the top.
> However, the Hive2 Hive version 2 TypeInfoUtils.implicitConversion method 
> allows StringFamily to either DOUBLE or DECIMAL conversion.
> The new org.apache.hadoop.hive.metastore.ColumnType class under Hive version 
> 3 hive-standalone-metadata-server method checkColTypeChangeCompatible only 
> allows DOUBLE.
> This JIRA fixes that problem.
> 2) Also, the checkColTypeChangeCompatible method lost a version 2 series bug 
> fix that drops CHAR/VARCHAR (and DECIMAL I think) type decorations when 
> checking for Schema Evolution compatibility.  So, when that code is checking 
> if a data type "varchar(25)" is StringFamily it fails because the "(25)" 
> didn't get removed properly.
> This JIRA fixes issue #2 also.
> NOTE: Hive1 version 2 did undecoratedTypeName(oldType) and Hive2 version 
> performed the logic in TypeInfoUtils.implicitConvertible on the 
> PrimitiveCategory not the raw type string.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20462) "CREATE VIEW IF NOT EXISTS" fails if view already exists

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616456#comment-16616456
 ] 

Daniel Dai commented on HIVE-20462:
---

[~jcamachorodriguez], can you review?

> "CREATE VIEW IF NOT EXISTS" fails if view already exists
> 
>
> Key: HIVE-20462
> URL: https://issues.apache.org/jira/browse/HIVE-20462
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.1.0
> Environment: HDP 3.0
> Hive 3.1.0.3.0.0.0-1634
>Reporter: static-max
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20462.1.patch
>
>
> After upgrading to Hive 3.1 from Hive 1.2, I can't run my queries anymore. It 
> seems that "if not exists" is ignored in "create view".
> *Example:*
>  # create view if not exists test as select 1;
>  # create view if not exists test as select 1;
> The second call fails with "Table already exists: default.test". I changed 
> this to "create or replace view" as a workaround.
> Also documented [link 
> here|https://community.hortonworks.com/questions/214772/hdp-3-hive-create-view-if-not-exists-fails.html|]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20355) Clean up parameter of HiveConnection.setSchema

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-20355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-20355:
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 4.0.0
   Status: Resolved  (was: Patch Available)

Patch pushed to master. Thanks Sankar for review!

> Clean up parameter of HiveConnection.setSchema
> --
>
> Key: HIVE-20355
> URL: https://issues.apache.org/jira/browse/HIVE-20355
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20355.1.patch
>
>
> Not immediately exploitable, as HS2 only allow one statement a time. But in 
> future, we may support multiple statement in HiveStatement, so better to 
> clean up the database parameter to avoid potential sql injection.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20424) schematool shall not pollute beeline history

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-20424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-20424:
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 4.0.0
   Status: Resolved  (was: Patch Available)

Patch pushed to master. Thanks Sankar for review!

> schematool shall not pollute beeline history
> 
>
> Key: HIVE-20424
> URL: https://issues.apache.org/jira/browse/HIVE-20424
> Project: Hive
>  Issue Type: Bug
>  Components: Standalone Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20424.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20524) Schema Evolution checking is broken in going from Hive version 2 to version 3 for ALTER TABLE VARCHAR to DECIMAL

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616443#comment-16616443
 ] 

Hive QA commented on HIVE-20524:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  2m 
29s{color} | {color:blue} standalone-metastore/metastore-common in master has 
28 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
59s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  6m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 57s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13812/dev-support/hive-personality.sh
 |
| git revision | master / 08d9083 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common itests ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13812/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Schema Evolution checking is broken in going from Hive version 2 to version 3 
> for ALTER TABLE VARCHAR to DECIMAL
> 
>
> Key: HIVE-20524
> URL: https://issues.apache.org/jira/browse/HIVE-20524
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20524.01.patch, HIVE-20524.02.patch, 
> HIVE-20524.03.patch, HIVE-20524.04.patch
>
>
> Issue that started this JIRA:
> {code}
> create external table varchar_decimal (c1 varchar(25));
> alter table varchar_decimal change c1 c1 decimal(31,0);
> ERROR : FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. The following 
> columns have types incompatible with the existing columns in their respective 
> positions :
> c1
> {code}
> There appear to be 2 issues here:
> 1) When hive.metastore.disallow.incompatible.col.type.changes is true (the 
> default) we only allow

[jira] [Commented] (HIVE-20325) FlakyTest: TestMiniDruidCliDriver

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616428#comment-16616428
 ] 

Hive QA commented on HIVE-20325:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939779/HIVE-20325.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 14940 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13811/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13811/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13811/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12939779 - PreCommit-HIVE-Build

> FlakyTest: TestMiniDruidCliDriver
> -
>
> Key: HIVE-20325
> URL: https://issues.apache.org/jira/browse/HIVE-20325
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: slim bouguerra
>Priority: Blocker
> Attachments: HIVE-20325.patch
>
>
> TestMiniDruidCliDriver is failing intermittently a significant percentage of 
> the time.
> druid_timestamptz
> druidmini_joins
> druidmini_masking
> druidmini_test1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20538) Allow to store a key value together with a transaction.

[Jaume M (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-20538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jaume M updated HIVE-20538:
---
Status: Open  (was: Patch Available)

> Allow to store a key value together with a transaction.
> ---
>
> Key: HIVE-20538
> URL: https://issues.apache.org/jira/browse/HIVE-20538
> Project: Hive
>  Issue Type: New Feature
>  Components: Standalone Metastore, Transactions
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20538.1.patch, HIVE-20538.1.patch, 
> HIVE-20538.2.patch, HIVE-20538.3.patch
>
>
> This can be useful for example to know if a transaction has already happened.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20538) Allow to store a key value together with a transaction.

[Jaume M (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-20538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jaume M updated HIVE-20538:
---
Attachment: HIVE-20538.3.patch
Status: Patch Available  (was: Open)

> Allow to store a key value together with a transaction.
> ---
>
> Key: HIVE-20538
> URL: https://issues.apache.org/jira/browse/HIVE-20538
> Project: Hive
>  Issue Type: New Feature
>  Components: Standalone Metastore, Transactions
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20538.1.patch, HIVE-20538.1.patch, 
> HIVE-20538.2.patch, HIVE-20538.3.patch
>
>
> This can be useful for example to know if a transaction has already happened.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20556) Expose an API to retrieve the TBL_ID from TBLS in the metastore tables

[Jaume M (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-20556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jaume M updated HIVE-20556:
---
Attachment: HIVE-20556.3.patch
Status: Patch Available  (was: Open)

> Expose an API to retrieve the TBL_ID from TBLS in the metastore tables
> --
>
> Key: HIVE-20556
> URL: https://issues.apache.org/jira/browse/HIVE-20556
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore, Standalone Metastore
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20556.1.patch, HIVE-20556.2.patch, 
> HIVE-20556.3.patch
>
>
> We have two options to do this
> 1) Use the current MTable and add a field for this value
> 2) Add an independent API call to the metastore that would return the TBL_ID.
> Option 1 is preferable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20556) Expose an API to retrieve the TBL_ID from TBLS in the metastore tables

[Jaume M (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-20556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jaume M updated HIVE-20556:
---
Status: Open  (was: Patch Available)

> Expose an API to retrieve the TBL_ID from TBLS in the metastore tables
> --
>
> Key: HIVE-20556
> URL: https://issues.apache.org/jira/browse/HIVE-20556
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore, Standalone Metastore
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20556.1.patch, HIVE-20556.2.patch
>
>
> We have two options to do this
> 1) Use the current MTable and add a field for this value
> 2) Add an independent API call to the metastore that would return the TBL_ID.
> Option 1 is preferable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20325) FlakyTest: TestMiniDruidCliDriver

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616404#comment-16616404
 ] 

Hive QA commented on HIVE-20325:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
45s{color} | {color:blue} itests/util in master has 52 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
51s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
23s{color} | {color:blue} testutils/ptest2 in master has 24 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
14s{color} | {color:green} itests/qtest: The patch generated 0 new + 0 
unchanged - 3 fixed = 0 total (was 3) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
14s{color} | {color:red} itests/util: The patch generated 1 new + 87 unchanged 
- 2 fixed = 88 total (was 89) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
40s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 7s{color} | {color:green} The patch ptest2 passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
14s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 29s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13811/dev-support/hive-personality.sh
 |
| git revision | master / 08d9083 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13811/yetus/diff-checkstyle-itests_util.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13811/yetus/patch-asflicense-problems.txt
 |
| modules | C: itests itests/qtest itests/util ql testutils/ptest2 U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13811/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> FlakyTest: TestMiniDruidCliDriver
> -
>
> Key: HIVE-20325
> URL: https://issues.apache.org/jira/browse/HIVE-20325
> Project: Hive
>  Issue Type: Bug
>  Components: Hi

[jira] [Updated] (HIVE-20527) Intern table descriptors from spark task

[Janaki Lahorani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-20527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Janaki Lahorani updated HIVE-20527:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Intern table descriptors from spark task
> 
>
> Key: HIVE-20527
> URL: https://issues.apache.org/jira/browse/HIVE-20527
> Project: Hive
>  Issue Type: Bug
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Attachments: HIVE-20527.1.patch, HIVE-20527.1.patch, 
> HIVE-20527.1.patch
>
>
> Table descriptors from MR tasks and Tez tasks are interned.  This fix is to 
> intern table desc from spark tasks as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20527) Intern table descriptors from spark task

[Janaki Lahorani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-20527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Janaki Lahorani updated HIVE-20527:
---
Fix Version/s: 4.0.0

> Intern table descriptors from spark task
> 
>
> Key: HIVE-20527
> URL: https://issues.apache.org/jira/browse/HIVE-20527
> Project: Hive
>  Issue Type: Bug
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20527.1.patch, HIVE-20527.1.patch, 
> HIVE-20527.1.patch
>
>
> Table descriptors from MR tasks and Tez tasks are interned.  This fix is to 
> intern table desc from spark tasks as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20526) Add test case for HIVE-20489

[Janaki Lahorani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-20526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Janaki Lahorani updated HIVE-20526:
---
  Resolution: Fixed
Target Version/s: 4.0.0
  Status: Resolved  (was: Patch Available)

> Add test case for HIVE-20489
> 
>
> Key: HIVE-20526
> URL: https://issues.apache.org/jira/browse/HIVE-20526
> Project: Hive
>  Issue Type: Bug
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Attachments: HIVE-20526.1.patch, HIVE-20526.1.patch
>
>
> Add a test case for the issue discussed in HIVE-20489.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20526) Add test case for HIVE-20489

[Janaki Lahorani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-20526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Janaki Lahorani updated HIVE-20526:
---
Fix Version/s: 4.0.0

> Add test case for HIVE-20489
> 
>
> Key: HIVE-20526
> URL: https://issues.apache.org/jira/browse/HIVE-20526
> Project: Hive
>  Issue Type: Bug
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20526.1.patch, HIVE-20526.1.patch
>
>
> Add a test case for the issue discussed in HIVE-20489.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20535) Add new configuration to set the size of the global compile lock

[denys kuzmenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-20535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

denys kuzmenko updated HIVE-20535:
--
Attachment: HIVE-20535.3.patch

> Add new configuration to set the size of the global compile lock
> 
>
> Key: HIVE-20535
> URL: https://issues.apache.org/jira/browse/HIVE-20535
> Project: Hive
>  Issue Type: Task
>  Components: HiveServer2
>Reporter: denys kuzmenko
>Assignee: denys kuzmenko
>Priority: Major
> Attachments: HIVE-20535.1.patch, HIVE-20535.2.patch, 
> HIVE-20535.3.patch
>
>
> When removing the compile lock, it is quite risky to remove it entirely.
> It would be good to provide a pool size for the concurrent compilation, so 
> the administrator can limit the load



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20538) Allow to store a key value together with a transaction.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616391#comment-16616391
 ] 

Hive QA commented on HIVE-20538:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939801/HIVE-20538.2.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13810/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13810/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13810/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-09-15 16:38:34.308
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-13810/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-09-15 16:38:34.311
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 08d9083 HIVE-18583 : Enable DateRangeRules (Nishant Bangarwa via 
Ashutosh Chauhan)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 08d9083 HIVE-18583 : Enable DateRangeRules (Nishant Bangarwa via 
Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-09-15 16:38:34.925
+ rm -rf ../yetus_PreCommit-HIVE-Build-13810
+ mkdir ../yetus_PreCommit-HIVE-Build-13810
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-13810
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-13810/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: 
a/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CommitTxnRequest.java:
 does not exist in index
error: 
a/standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php:
 does not exist in index
error: 
a/standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py:
 does not exist in index
error: 
a/standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb:
 does not exist in index
error: 
a/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift: 
does not exist in index
error: 
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
 does not exist in index
error: 
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java:
 does not exist in index
error: 
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
 does not exist in index
error: 
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java:
 does not exist in index
error: 
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStoreTxns.java:
 does not exist in index
error: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
 does not exist in index
error: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java:
 does not exist in index
error: 
metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CommitTxnRequest.java:
 does not exist in index
error: metastore-common/src/gen/thrift/gen-php/metastore/Types.php: does not 
exist in index
error: metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py: does 
not exist in index
er

[jira] [Commented] (HIVE-20556) Expose an API to retrieve the TBL_ID from TBLS in the metastore tables

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616387#comment-16616387
 ] 

Hive QA commented on HIVE-20556:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939813/HIVE-20556.2.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13809/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13809/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13809/

Messages:
{noformat}
 This message was trimmed, see log for full details 
+ mvn -B clean install -DskipTests -T 4 -q 
-Dmaven.repo.local=/data/hiveptest/working/maven
[ERROR] COMPILATION ERROR : 
[ERROR] 
/data/hiveptest/working/apache-github-source-source/itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java:[403,9]
 no suitable constructor found for 
Table(java.lang.String,java.lang.String,java.lang.String,int,int,int,org.apache.hadoop.hive.metastore.api.StorageDescriptor,,java.util.Map,,,java.lang.String)
constructor org.apache.hadoop.hive.metastore.api.Table.Table() is not 
applicable
  (actual and formal argument lists differ in length)
constructor 
org.apache.hadoop.hive.metastore.api.Table.Table(long,java.lang.String,java.lang.String,java.lang.String,int,int,int,org.apache.hadoop.hive.metastore.api.StorageDescriptor,java.util.List,java.util.Map,java.lang.String,java.lang.String,java.lang.String)
 is not applicable
  (actual and formal argument lists differ in length)
constructor 
org.apache.hadoop.hive.metastore.api.Table.Table(org.apache.hadoop.hive.metastore.api.Table)
 is not applicable
  (actual and formal argument lists differ in length)
[ERROR] 
/data/hiveptest/working/apache-github-source-source/itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java:[430,9]
 no suitable constructor found for 
Table(java.lang.String,java.lang.String,java.lang.String,int,int,int,org.apache.hadoop.hive.metastore.api.StorageDescriptor,,java.util.Map,,,)
constructor org.apache.hadoop.hive.metastore.api.Table.Table() is not 
applicable
  (actual and formal argument lists differ in length)
constructor 
org.apache.hadoop.hive.metastore.api.Table.Table(long,java.lang.String,java.lang.String,java.lang.String,int,int,int,org.apache.hadoop.hive.metastore.api.StorageDescriptor,java.util.List,java.util.Map,java.lang.String,java.lang.String,java.lang.String)
 is not applicable
  (actual and formal argument lists differ in length)
constructor 
org.apache.hadoop.hive.metastore.api.Table.Table(org.apache.hadoop.hive.metastore.api.Table)
 is not applicable
  (actual and formal argument lists differ in length)
[ERROR] 
/data/hiveptest/working/apache-github-source-source/itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java:[458,9]
 no suitable constructor found for 
Table(java.lang.String,java.lang.String,java.lang.String,int,int,int,org.apache.hadoop.hive.metastore.api.StorageDescriptor,java.util.ArrayList,java.util.Map,,,)
constructor org.apache.hadoop.hive.metastore.api.Table.Table() is not 
applicable
  (actual and formal argument lists differ in length)
constructor 
org.apache.hadoop.hive.metastore.api.Table.Table(long,java.lang.String,java.lang.String,java.lang.String,int,int,int,org.apache.hadoop.hive.metastore.api.StorageDescriptor,java.util.List,java.util.Map,java.lang.String,java.lang.String,java.lang.String)
 is not applicable
  (actual and formal argument lists differ in length)
constructor 
org.apache.hadoop.hive.metastore.api.Table.Table(org.apache.hadoop.hive.metastore.api.Table)
 is not applicable
  (actual and formal argument lists differ in length)
[ERROR] 
/data/hiveptest/working/apache-github-source-source/itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java:[465,9]
 no suitable constructor found for 
Table(java.lang.String,java.lang.String,java.lang.String,int,int,int,org.apache.hadoop.hive.metastore.api.StorageDescriptor,java.util.ArrayList,java.util.Map,,,)
constructor org.apache.hadoop.hive.metastore.api.Table.Table() is not 
applicable
  (actual and formal argument lists differ in length)
constructor 
org.apache.hadoop.hive.metastore.api.Table.Table(long,java.lang.String,java.lang.String,java.lang.String,int,int,int,org.apache.hadoop.hive.metastore.api.StorageDescriptor,java.util.List,java.util.Map,java.lang.String,java.lang.String,java.lang.String)
 is not applicable
  (actual and formal argument lists differ in length)
constructor 
org.apache.hadoop.hive.metastore.api.Table.Table(org.apache.hadoop.hive.metastore.api.Table)
 is not applicable
  (actual and

[jira] [Commented] (HIVE-20561) Use the position of the Kafka Consumer to track progress instead of Consumer Records offsets

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616385#comment-16616385
 ] 

Hive QA commented on HIVE-20561:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939753/HIVE-20561.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14940 tests 
executed
*Failed tests:*
{noformat}
org.apache.hive.service.auth.TestCustomAuthentication.org.apache.hive.service.auth.TestCustomAuthentication
 (batchId=247)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13808/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13808/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13808/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12939753 - PreCommit-HIVE-Build

> Use the position of the Kafka Consumer to track progress instead of Consumer 
> Records offsets
> 
>
> Key: HIVE-20561
> URL: https://issues.apache.org/jira/browse/HIVE-20561
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20561.patch
>
>
> Kafka Partitions with transactional messages (post 0.11) will include commit 
> or abort markers which indicate the result of a transaction. The markers are 
> not returned to applications, yet have an offset in the log. Therefore the 
> end of Stream position can be the offset of a control message. 
> This Patch change the way how we keep track of the consumer position by using 
>  {code} consumer.position(topicP) {code} as oppose to using the offset of the 
> consumed messages.
> Also I have done some refactoring to help code readability  hopefully.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20561) Use the position of the Kafka Consumer to track progress instead of Consumer Records offsets

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616363#comment-16616363
 ] 

Hive QA commented on HIVE-20561:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
23s{color} | {color:blue} kafka-handler in master has 1 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m  
9s{color} | {color:red} kafka-handler: The patch generated 5 new + 2 unchanged 
- 0 fixed = 7 total (was 2) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 29s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13808/dev-support/hive-personality.sh
 |
| git revision | master / 08d9083 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13808/yetus/diff-checkstyle-kafka-handler.txt
 |
| modules | C: kafka-handler U: kafka-handler |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13808/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Use the position of the Kafka Consumer to track progress instead of Consumer 
> Records offsets
> 
>
> Key: HIVE-20561
> URL: https://issues.apache.org/jira/browse/HIVE-20561
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20561.patch
>
>
> Kafka Partitions with transactional messages (post 0.11) will include commit 
> or abort markers which indicate the result of a transaction. The markers are 
> not returned to applications, yet have an offset in the log. Therefore the 
> end of Stream position can be the offset of a control message. 
> This Patch change the way how we keep track of the consumer position by using 
>  {code} consumer.position(topicP) {code} as oppose to using the offset of the 
> consumed messages.
> Also I have done some refactoring to help code readability  hopefully.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17084) Turn on hive.stats.fetch.column.stats configuration flag

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616358#comment-16616358
 ] 

Hive QA commented on HIVE-17084:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939719/HIVE-17084.18.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13807/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13807/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13807/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL 
https://issues.apache.org/jira/secure/attachment/12939719/HIVE-17084.18.patch 
was found in seen patch url's cache and a test was probably run already on it. 
Aborting...
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12939719 - PreCommit-HIVE-Build

> Turn on hive.stats.fetch.column.stats configuration flag
> 
>
> Key: HIVE-17084
> URL: https://issues.apache.org/jira/browse/HIVE-17084
> Project: Hive
>  Issue Type: Task
>  Components: Statistics
>Reporter: Vineet Garg
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-17084.08.patch, HIVE-17084.09.patch, 
> HIVE-17084.1.patch, HIVE-17084.10.patch, HIVE-17084.11.patch, 
> HIVE-17084.12.patch, HIVE-17084.13.patch, HIVE-17084.14.patch, 
> HIVE-17084.15.patch, HIVE-17084.16.patch, HIVE-17084.17.patch, 
> HIVE-17084.18.patch, HIVE-170884.4.patch, HIVE-170884.5.patch, 
> HIVE-170884.7.patch
>
>
> This flag is off by default and could result in bad plans due to missing 
> column statistics.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20560) Set hive.llap.memory.oversubscription.max.executors.per.query to be 1/3rd of num of llap daemons

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616357#comment-16616357
 ] 

Hive QA commented on HIVE-20560:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939715/HIVE-20560.01.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 14941 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez2]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join32_lessSize]
 (batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_llap] 
(batchId=169)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13806/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13806/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13806/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12939715 - PreCommit-HIVE-Build

> Set hive.llap.memory.oversubscription.max.executors.per.query to be 1/3rd of 
> num of llap daemons
> 
>
> Key: HIVE-20560
> URL: https://issues.apache.org/jira/browse/HIVE-20560
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20560.01.patch
>
>
> Unless hive.llap.memory.oversubscription.max.executors.per.query is not 
> specified by the user,
> it should be calculated like this:
> {noformat}
> hive.llap.memory.oversubscription.max.executors.per.query = min(max(1, 
> int(hive.llap.daemon.num.executors / 3)),8)
> {noformat}
> In container mode it remains 3 (unless specified otherwise).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20560) Set hive.llap.memory.oversubscription.max.executors.per.query to be 1/3rd of num of llap daemons

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616329#comment-16616329
 ] 

Hive QA commented on HIVE-20560:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
 8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
30s{color} | {color:blue} common in master has 64 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
16s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 23s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13806/dev-support/hive-personality.sh
 |
| git revision | master / 08d9083 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13806/yetus/whitespace-tabs.txt
 |
| modules | C: common ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13806/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Set hive.llap.memory.oversubscription.max.executors.per.query to be 1/3rd of 
> num of llap daemons
> 
>
> Key: HIVE-20560
> URL: https://issues.apache.org/jira/browse/HIVE-20560
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20560.01.patch
>
>
> Unless hive.llap.memory.oversubscription.max.executors.per.query is not 
> specified by the user,
> it should be calculated like this:
> {noformat}
> hive.llap.memory.oversubscription.max.executors.per.query = min(max(1, 
> int(hive.llap.daemon.num.executors / 3)),8)
> {noformat}
> In container mode it remains 3 (unless specified otherwise).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20535) Add new configuration to set the size of the global compile lock

[denys kuzmenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-20535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

denys kuzmenko updated HIVE-20535:
--
Attachment: HIVE-20535.2.patch

> Add new configuration to set the size of the global compile lock
> 
>
> Key: HIVE-20535
> URL: https://issues.apache.org/jira/browse/HIVE-20535
> Project: Hive
>  Issue Type: Task
>  Components: HiveServer2
>Reporter: denys kuzmenko
>Assignee: denys kuzmenko
>Priority: Major
> Attachments: HIVE-20535.1.patch, HIVE-20535.2.patch
>
>
> When removing the compile lock, it is quite risky to remove it entirely.
> It would be good to provide a pool size for the concurrent compilation, so 
> the administrator can limit the load



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20559) Turn on hive.transpose.aggr.join by default

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616306#comment-16616306
 ] 

Hive QA commented on HIVE-20559:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939701/HIVE-20559.01.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 218 failed/errored test(s), 14806 tests 
executed
*Failed tests:*
{noformat}
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=154)

[orc_ppd_schema_evol_3a.q,import_exported_table.q,tez_union_dynamic_partition.q,except_distinct.q,orc_merge10.q,mapreduce1.q,mm_all.q,orc_merge4.q,dynamic_semijoin_user_level.q,temp_table_external.q,llap_udf.q,schemeAuthority.q,external_table_with_space_in_location_path.q,external_table_purge.q,remote_script.q]
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=164)

[list_bucket_dml_10.q,mapjoin_emit_interval.q,acid_globallimit.q,deleteAnalyze.q,delete_where_no_match.q,semijoin_hint.q,vector_groupby_grouping_sets4.q,materialized_view_create_rewrite_rebuild_dummy.q,cbo_views.q,vectorization_part.q,auto_join30.q,dynamic_partition_pruning.q,cluster.q,vector_order_null.q,schema_evol_text_nonvec_part_llap_io.q,bucketmapjoin7.q,cbo_rp_unionDistinct_2.q,subquery_nested_subquery.q,limit_pushdown3.q,vector_outer_join2.q,smb_mapjoin_18.q,vector_varchar_4.q,ppd_union_view.q,union6.q,vector_decimal_4.q,schema_evol_orc_acid_part.q,cbo_subq_in.q,cross_join.q,tez_smb_main.q,vectorized_timestamp_funcs.q]
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=165)

[vector_auto_smb_mapjoin_14.q,tez_dml.q,database.q,cross_prod_4.q,vector_bround.q,orc_ppd_schema_evol_1b.q,vector_join30.q,smb_mapjoin_6.q,cte_3.q,vector_reduce_groupby_decimal.q,vectorized_dynamic_partition_pruning.q,vector_decimal_cast.q,schema_evol_text_vecrow_part_all_primitive_llap_io.q,orc_merge6.q,cte_mat_1.q,vector_char_mapjoin1.q,cte_5.q,subquery_shared_alias.q,vector_decimal_2.q,vector_outer_reference_windowed.q,columnStatsUpdateForStatsOptimizer_1.q,vector_outer_join3.q,vector_string_concat.q,vector_annotate_stats_select.q,vector_windowing_windowspec.q,sharedworkext.q,dynamic_semijoin_reduction_4.q,vectorized_context.q,metadata_only_queries.q,auto_sortmerge_join_12.q]
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=167)

[tez_dynpart_hashjoin_3.q,dpp.q,topnkey.q,tez_join.q,vectorized_rcfile_columnar.q,groupby_resolution.q,vector_reuse_scratchcols.q,delete_where_non_partitioned.q,orc_merge11.q,schema_evol_orc_acid_table.q,cbo_semijoin.q,orc_merge_incompat_schema.q,subquery_scalar.q,vectorization_11.q,orc_llap_nonvector.q,kryo.q,vector_reduce2.q,vector_interval_mapjoin.q,schema_evol_orc_acidvec_table_update_llap_io.q,tez_joins_explain.q,explainuser_4.q,vector_llap_io_data_conversion.q,vector_aggregate_9.q,insert_after_drop_partition.q,alter_merge_stats_orc.q,default_constraint.q,vectorization_part_varchar.q,llap_acid_fast.q,subquery_select.q,delete_where_partitioned.q]
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=169)

[dynamic_semijoin_reduction.q,authorization_view_8.q,orc_llap.q,vectorization_pushdown.q,bucket_num_reducers_acid2.q,auto_sortmerge_join_1.q,results_cache_empty_result.q,lineage3.q,cross_product_check_1.q,results_cache_quoted_identifiers.q,q93_with_constraints.q,unionDistinct_3.q,cbo_join.q,vector_struct_in.q,bucketmapjoin3.q,vector_groupby_cube1.q,current_date_timestamp.q,groupby2.q,schema_evol_text_vec_table.q,vectorized_join46.q,mm_loaddata.q,retry_failure.q,multiMapJoin1.q,vector_char_simple.q,llap_decimal64_reader.q,vectorization_input_format_excludes.q,leftsemijoin.q,special_character_in_tabnames_1.q,cte_mat_2.q,vectorization_8.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin] 
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join26] (batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_const] (batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_cross_product_check_2]
 (batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[convert_decimal64_to_decimal]
 (batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer10] 
(batchId=82)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer11] 
(batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer15] 
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer7] 
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cross_product_check_1] 
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cross_produc

[jira] [Commented] (HIVE-18767) Some alterPartitions invocations throw 'NumberFormatException: null'

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616307#comment-16616307
 ] 

Hive QA commented on HIVE-18767:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939275/HIVE-18767.2-branch-3.1.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13805/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13805/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13805/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL 
https://issues.apache.org/jira/secure/attachment/12939275/HIVE-18767.2-branch-3.1.patch
 was found in seen patch url's cache and a test was probably run already on it. 
Aborting...
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12939275 - PreCommit-HIVE-Build

> Some alterPartitions invocations throw 'NumberFormatException: null'
> 
>
> Key: HIVE-18767
> URL: https://issues.apache.org/jira/browse/HIVE-18767
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.3.3, 3.1.0, 4.0.0, 3.2.0
>Reporter: Yuming Wang
>Assignee: Mass Dosage
>Priority: Major
> Fix For: 2.3.3, 4.0.0
>
> Attachments: HIVE-18767-branch-2.3.patch, HIVE-18767-branch-2.patch, 
> HIVE-18767-branch-3.1.patch, HIVE-18767-branch-3.patch, HIVE-18767.1.patch, 
> HIVE-18767.2-branch-2.3.patch, HIVE-18767.2-branch-2.patch, 
> HIVE-18767.2-branch-3.1.patch, HIVE-18767.2.patch, HIVE-18767.3.patch, 
> HIVE-18767.4.patch, HIVE-18767.5.patch, HIVE-18767.6.patch
>
>
> Error messages:
> {noformat}
> [info] Cause: java.lang.NumberFormatException: null
> [info] at java.lang.Long.parseLong(Long.java:552)
> [info] at java.lang.Long.parseLong(Long.java:631)
> [info] at 
> org.apache.hadoop.hive.metastore.MetaStoreUtils.isFastStatsSame(MetaStoreUtils.java:315)
> [info] at 
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:605)
> [info] at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:3837)
> [info] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [info] at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [info] at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [info] at java.lang.reflect.Method.invoke(Method.java:498)
> [info] at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> [info] at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> [info] at 
> com.sun.proxy.$Proxy23.alter_partitions_with_environment_context(Unknown 
> Source)
> [info] at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_partitions(HiveMetaStoreClient.java:1527)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20559) Turn on hive.transpose.aggr.join by default

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616291#comment-16616291
 ] 

Hive QA commented on HIVE-20559:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
17s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
35s{color} | {color:blue} common in master has 64 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 36s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13804/dev-support/hive-personality.sh
 |
| git revision | master / 08d9083 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common U: common |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13804/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Turn on hive.transpose.aggr.join by default
> ---
>
> Key: HIVE-20559
> URL: https://issues.apache.org/jira/browse/HIVE-20559
> Project: Hive
>  Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20559.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17084) Turn on hive.stats.fetch.column.stats configuration flag

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616278#comment-16616278
 ] 

Hive QA commented on HIVE-17084:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939719/HIVE-17084.18.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 14940 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_merge_multi_expressions]
 (batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[manyViewJoin] 
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nonmr_fetch] (batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_part_project]
 (batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pcr] (batchId=64)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_recursive_mapjoin]
 (batchId=188)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join1]
 (batchId=188)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join2]
 (batchId=187)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[pcr] (batchId=136)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query23] 
(batchId=266)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13803/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13803/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13803/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12939719 - PreCommit-HIVE-Build

> Turn on hive.stats.fetch.column.stats configuration flag
> 
>
> Key: HIVE-17084
> URL: https://issues.apache.org/jira/browse/HIVE-17084
> Project: Hive
>  Issue Type: Task
>  Components: Statistics
>Reporter: Vineet Garg
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-17084.08.patch, HIVE-17084.09.patch, 
> HIVE-17084.1.patch, HIVE-17084.10.patch, HIVE-17084.11.patch, 
> HIVE-17084.12.patch, HIVE-17084.13.patch, HIVE-17084.14.patch, 
> HIVE-17084.15.patch, HIVE-17084.16.patch, HIVE-17084.17.patch, 
> HIVE-17084.18.patch, HIVE-170884.4.patch, HIVE-170884.5.patch, 
> HIVE-170884.7.patch
>
>
> This flag is off by default and could result in bad plans due to missing 
> column statistics.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17084) Turn on hive.stats.fetch.column.stats configuration flag

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616273#comment-16616273
 ] 

Hive QA commented on HIVE-17084:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 3s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
31s{color} | {color:blue} common in master has 64 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
30s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
23s{color} | {color:blue} contrib in master has 13 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
32s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
7s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  6m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 44s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13803/dev-support/hive-personality.sh
 |
| git revision | master / 08d9083 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common accumulo-handler contrib hbase-handler 
itests/hive-blobstore ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13803/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Turn on hive.stats.fetch.column.stats configuration flag
> 
>
> Key: HIVE-17084
> URL: https://issues.apache.org/jira/browse/HIVE-17084
> Project: Hive
>  Issue Type: Task
>  Components: Statistics
>Reporter: Vineet Garg
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-17084.08.patch, HIVE-17084.09.patch, 
> HIVE-17084.1.patch, HIVE-17084.10.patch, HIVE-17084.11.patch, 
> HIVE-17084.12.patch, HIVE-17084.13.patch, HIVE-17084.14.patch, 
> HIVE-17084.15.patch, HIVE-17084.1

[jira] [Commented] (HIVE-20551) Create PreparedStatement query dynamically when IN clause is used

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616266#comment-16616266
 ] 

Hive QA commented on HIVE-20551:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939694/HIVE-20551.02.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 14940 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13802/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13802/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13802/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12939694 - PreCommit-HIVE-Build

> Create PreparedStatement query dynamically when IN clause is used
> -
>
> Key: HIVE-20551
> URL: https://issues.apache.org/jira/browse/HIVE-20551
> Project: Hive
>  Issue Type: Bug
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-20551.01.patch, HIVE-20551.02.patch
>
>
> In the MetaStoreDirectSql class when IN clause is used, the query statement 
> is created via string concatenation.
> Since JDBC API allows only one literal for one “?” parameter, 
> PreparedStatement doesn’t work for IN clause queries. To create the 
> PreparedStatement query dynamically based on the size of the elements in IN 
> clause, the makeParams() should be used instead of concatenation. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20551) Create PreparedStatement query dynamically when IN clause is used

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616261#comment-16616261
 ] 

Hive QA commented on HIVE-20551:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 7s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
14s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
14s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 45s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13802/dev-support/hive-personality.sh
 |
| git revision | master / 08d9083 |
| Default Java | 1.8.0_111 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13802/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13802/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
 |
| modules | C: standalone-metastore/metastore-server U: 
standalone-metastore/metastore-server |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13802/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Create PreparedStatement query dynamically when IN clause is used
> -
>
> Key: HIVE-20551
> URL: https://issues.apache.org/jira/browse/HIVE-20551
> Project: Hive
>  Issue Type: Bug
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-20551.01.patch, HIVE-20551.02.patch
>
>
> In the MetaStoreDirectSql class when IN clause is used, the query statement 
> is created via string concatenation.
> Since JDBC API allows only one literal for one “?” parameter, 
> PreparedStatement doesn’t work for IN clause queries. To create the 
> PreparedStatement query dynamically based on the size of the elements in IN 
> clause, the makeParams() should be used instead of concatenation. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20536) Add Surrogate Keys function to Hive

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616255#comment-16616255
 ] 

Hive QA commented on HIVE-20536:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939682/HIVE-20536.03.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14942 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[show_functions] 
(batchId=78)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13801/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13801/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13801/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12939682 - PreCommit-HIVE-Build

> Add Surrogate Keys function to Hive
> ---
>
> Key: HIVE-20536
> URL: https://issues.apache.org/jira/browse/HIVE-20536
> Project: Hive
>  Issue Type: Task
>  Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-20536.01.patch, HIVE-20536.02.patch, 
> HIVE-20536.03.patch
>
>
> Surrogate keys is an ability to generate and use unique integers for each row 
> in a table. If we have that ability then in conjunction with default clause 
> we can get surrogate keys functionality. Consider following ddl:
> create table t1 (a string, b bigint default unique_long());
> We already have default clause wherein you can specify a function to provide 
> values. So, what we need is udf which can generate unique longs for each row 
> across queries for a table. 
> Idea is to use write_id . This is a column in metastore table TXN_COMPONENTS 
> whose value is determined at compile time to be used during query execution. 
> Each query execution generates a new write_id. So, we can seed udf with this 
> value during compilation.
> Then we statically allocate ranges for each task from which it can draw next 
> long. So, lets say 64-bit write_id we divy up such that last 24 bits belong 
> to original usage of it that is txns. Next 16 bits are used for task_attempts 
> and last 24 bits to generate new long for each row. This implies we can allow 
> 17M txns, 65K tasks and 17M rows in a task. If you hit any of those limits we 
> can fail the query.
> Implementation wise: serialize write_id in initialize() of udf. Then during 
> execute() we find out what task_attempt current task is and use it along with 
> write_id() to get starting long and give a new value on each invocation of 
> execute().
> Here we are assuming write_id can be determined at compile time, which should 
> be the case but we need to figure out how to get handle to it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20355) Clean up parameter of HiveConnection.setSchema

[Sankar Hariappan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616251#comment-16616251
 ] 

Sankar Hariappan commented on HIVE-20355:
-

+1

> Clean up parameter of HiveConnection.setSchema
> --
>
> Key: HIVE-20355
> URL: https://issues.apache.org/jira/browse/HIVE-20355
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20355.1.patch
>
>
> Not immediately exploitable, as HS2 only allow one statement a time. But in 
> future, we may support multiple statement in HiveStatement, so better to 
> clean up the database parameter to avoid potential sql injection.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20536) Add Surrogate Keys function to Hive

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616250#comment-16616250
 ] 

Hive QA commented on HIVE-20536:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
14s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
48s{color} | {color:red} ql: The patch generated 2 new + 639 unchanged - 0 
fixed = 641 total (was 639) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  4m 
21s{color} | {color:red} ql generated 1 new + 2311 unchanged - 0 fixed = 2312 
total (was 2311) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 41s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  Unchecked/unconfirmed cast from 
org.apache.hadoop.hive.ql.exec.MapredContext to 
org.apache.hadoop.hive.ql.exec.tez.TezContext in 
org.apache.hadoop.hive.ql.udf.generic.GenericUDFSurrogateKey.configure(MapredContext)
  At GenericUDFSurrogateKey.java:org.apache.hadoop.hive.ql.exec.tez.TezContext 
in 
org.apache.hadoop.hive.ql.udf.generic.GenericUDFSurrogateKey.configure(MapredContext)
  At GenericUDFSurrogateKey.java:[line 101] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13801/dev-support/hive-personality.sh
 |
| git revision | master / 08d9083 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13801/yetus/diff-checkstyle-ql.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13801/yetus/whitespace-eol.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13801/yetus/new-findbugs-ql.html
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13801/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Add Surrogate Keys function to Hive
> ---
>
> Key: HIVE-20536
> URL: https://issues.apache.org/jira/browse/HIVE-20536
> Project: Hive
>  Issue Type: Task
>  Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-20536.01.patch, HIVE-20536.02.patch, 
> HIVE-20536.03.patch
>
>
> Surrogate keys is an ability to generate and use unique integers for each row 
> in a table. If we have that ability then in conjunction with default clause 
> we can get surrogate keys functionality. Consider following ddl:
> create table t1 (a string, b bigint d

[jira] [Commented] (HIVE-20424) schematool shall not pollute beeline history

[Sankar Hariappan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616245#comment-16616245
 ] 

Sankar Hariappan commented on HIVE-20424:
-

+1

> schematool shall not pollute beeline history
> 
>
> Key: HIVE-20424
> URL: https://issues.apache.org/jira/browse/HIVE-20424
> Project: Hive
>  Issue Type: Bug
>  Components: Standalone Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20424.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20445) Add randomized tests to TestArrowColumnarBatchSerDe

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616242#comment-16616242
 ] 

Hive QA commented on HIVE-20445:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939679/HIVE-20445.3.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 14941 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_dynamic_partition]
 (batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_expressions]
 (batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1]
 (batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_alter]
 (batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_insert]
 (batchId=193)
org.apache.hive.jdbc.miniHS2.TestHs2ConnectionMetricsHttp.testOpenConnectionMetrics
 (batchId=255)
org.apache.hive.service.auth.TestCustomAuthentication.org.apache.hive.service.auth.TestCustomAuthentication
 (batchId=247)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13800/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13800/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13800/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12939679 - PreCommit-HIVE-Build

> Add randomized tests to TestArrowColumnarBatchSerDe
> ---
>
> Key: HIVE-20445
> URL: https://issues.apache.org/jira/browse/HIVE-20445
> Project: Hive
>  Issue Type: Test
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Minor
>  Labels: pull-request-available
> Attachments: HIVE-20445.1.patch, HIVE-20445.2.patch, 
> HIVE-20445.2.patch, HIVE-20445.3.patch
>
>
> Use random schemas/data for TestArrowColumnarBatchSerDe to improve coverage 
> of arrow conversion/serialization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20445) Add randomized tests to TestArrowColumnarBatchSerDe

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616224#comment-16616224
 ] 

Hive QA commented on HIVE-20445:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
50s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
58s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
53s{color} | {color:red} ql: The patch generated 64 new + 1143 unchanged - 94 
fixed = 1207 total (was 1237) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 56s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13800/dev-support/hive-personality.sh
 |
| git revision | master / 08d9083 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13800/yetus/diff-checkstyle-ql.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13800/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Add randomized tests to TestArrowColumnarBatchSerDe
> ---
>
> Key: HIVE-20445
> URL: https://issues.apache.org/jira/browse/HIVE-20445
> Project: Hive
>  Issue Type: Test
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Minor
>  Labels: pull-request-available
> Attachments: HIVE-20445.1.patch, HIVE-20445.2.patch, 
> HIVE-20445.2.patch, HIVE-20445.3.patch
>
>
> Use random schemas/data for TestArrowColumnarBatchSerDe to improve coverage 
> of arrow conversion/serialization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20423) Set NULLS LAST as the default null ordering

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616216#comment-16616216
 ] 

Hive QA commented on HIVE-20423:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939670/HIVE-20423.8-branch-3.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 120 failed/errored test(s), 14446 tests 
executed
*Failed tests:*
{noformat}
TestAddPartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=228)
TestAddPartitionsFromPartSpec - did not produce a TEST-*.xml file (likely timed 
out) (batchId=230)
TestAdminUser - did not produce a TEST-*.xml file (likely timed out) 
(batchId=236)
TestAggregateStatsCache - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestAlterPartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestAppendPartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestAutoPurgeTables - did not produce a TEST-*.xml file (likely timed out) 
(batchId=252)
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=273)
TestCachedStore - did not produce a TEST-*.xml file (likely timed out) 
(batchId=236)
TestCatalogCaching - did not produce a TEST-*.xml file (likely timed out) 
(batchId=236)
TestCatalogNonDefaultClient - did not produce a TEST-*.xml file (likely timed 
out) (batchId=228)
TestCatalogNonDefaultSvr - did not produce a TEST-*.xml file (likely timed out) 
(batchId=236)
TestCatalogOldClient - did not produce a TEST-*.xml file (likely timed out) 
(batchId=228)
TestCatalogs - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestCheckConstraint - did not produce a TEST-*.xml file (likely timed out) 
(batchId=228)
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed 
out) (batchId=238)
TestDatabases - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestDeadline - did not produce a TEST-*.xml file (likely timed out) 
(batchId=236)
TestDefaultConstraint - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestDropPartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=228)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=273)
TestEmbeddedHiveMetaStore - did not produce a TEST-*.xml file (likely timed 
out) (batchId=231)
TestExchangePartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestFMSketchSerialization - did not produce a TEST-*.xml file (likely timed 
out) (batchId=238)
TestFilterHooks - did not produce a TEST-*.xml file (likely timed out) 
(batchId=228)
TestForeignKey - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestFunctions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=228)
TestGetPartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=228)
TestGetTableMeta - did not produce a TEST-*.xml file (likely timed out) 
(batchId=228)
TestHLLNoBias - did not produce a TEST-*.xml file (likely timed out) 
(batchId=238)
TestHLLSerialization - did not produce a TEST-*.xml file (likely timed out) 
(batchId=238)
TestHdfsUtils - did not produce a TEST-*.xml file (likely timed out) 
(batchId=236)
TestHiveAlterHandler - did not produce a TEST-*.xml file (likely timed out) 
(batchId=228)
TestHiveMetaStoreGetMetaConf - did not produce a TEST-*.xml file (likely timed 
out) (batchId=236)
TestHiveMetaStorePartitionSpecs - did not produce a TEST-*.xml file (likely 
timed out) (batchId=230)
TestHiveMetaStoreSchemaMethods - did not produce a TEST-*.xml file (likely 
timed out) (batchId=236)
TestHiveMetaStoreTimeout - did not produce a TEST-*.xml file (likely timed out) 
(batchId=238)
TestHiveMetaStoreTxns - did not produce a TEST-*.xml file (likely timed out) 
(batchId=238)
TestHiveMetaStoreWithEnvironmentContext - did not produce a TEST-*.xml file 
(likely timed out) (batchId=233)
TestHiveMetastoreCli - did not produce a TEST-*.xml file (likely timed out) 
(batchId=228)
TestHyperLogLog - did not produce a TEST-*.xml file (likely timed out) 
(batchId=238)
TestHyperLogLogDense - did not produce a TEST-*.xml file (likely timed out) 
(batchId=238)
TestHyperLogLogMerge - did not produce a TEST-*.xml file (likely timed out) 
(batchId=238)
TestHyperLogLogSparse - did not produce a TEST-*.xml file (likely timed out) 
(batchId=238)
TestJSONMessageDeserializer - did not produce a TEST-*.xml file (likely timed 
out) (batchId=236)
TestListPartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=228)
TestLocationQueries - did not produce a TEST-*.xml file (likely timed out) 
(batchId=252)
TestLockRequestBuilder - did not produce a TEST-*.xml file (likely timed out) 
(batchId=228)
TestMarkPartition - did not produce a TEST-*.xml file (like

[jira] [Updated] (HIVE-20517) Creation of staging directory and Move operation is taking time in S3

[mahesh kumar behera (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-20517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera updated HIVE-20517:
---
Status: Patch Available  (was: Open)

fixed review comments

> Creation of staging directory and Move operation is taking time in S3
> -
>
> Key: HIVE-20517
> URL: https://issues.apache.org/jira/browse/HIVE-20517
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20517.01.patch, HIVE-20517.02.patch, 
> HIVE-20517.03.patch
>
>
> Operations like insert and add partition creates a staging directory to 
> generate the files and then move the files created to actual location. In 
> replication flow, the files are first copied to the staging directory and 
> then moved (rename) to the actual table location. In case of S3, move is not 
> an atomic operation. It internally does a copy and delete. So it can not 
> guarantee the consistency required. So it is better to copy the files 
> directly to the actual location. This will help in avoiding the staging 
> directory creation (which takes 1-2 seconds in s3) and move (which takes time 
> proportional to file size).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20517) Creation of staging directory and Move operation is taking time in S3

[mahesh kumar behera (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-20517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera updated HIVE-20517:
---
Attachment: HIVE-20517.03.patch

> Creation of staging directory and Move operation is taking time in S3
> -
>
> Key: HIVE-20517
> URL: https://issues.apache.org/jira/browse/HIVE-20517
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20517.01.patch, HIVE-20517.02.patch, 
> HIVE-20517.03.patch
>
>
> Operations like insert and add partition creates a staging directory to 
> generate the files and then move the files created to actual location. In 
> replication flow, the files are first copied to the staging directory and 
> then moved (rename) to the actual table location. In case of S3, move is not 
> an atomic operation. It internally does a copy and delete. So it can not 
> guarantee the consistency required. So it is better to copy the files 
> directly to the actual location. This will help in avoiding the staging 
> directory creation (which takes 1-2 seconds in s3) and move (which takes time 
> proportional to file size).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20517) Creation of staging directory and Move operation is taking time in S3

[mahesh kumar behera (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-20517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera updated HIVE-20517:
---
Status: Open  (was: Patch Available)

> Creation of staging directory and Move operation is taking time in S3
> -
>
> Key: HIVE-20517
> URL: https://issues.apache.org/jira/browse/HIVE-20517
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20517.01.patch, HIVE-20517.02.patch, 
> HIVE-20517.03.patch
>
>
> Operations like insert and add partition creates a staging directory to 
> generate the files and then move the files created to actual location. In 
> replication flow, the files are first copied to the staging directory and 
> then moved (rename) to the actual table location. In case of S3, move is not 
> an atomic operation. It internally does a copy and delete. So it can not 
> guarantee the consistency required. So it is better to copy the files 
> directly to the actual location. This will help in avoiding the staging 
> directory creation (which takes 1-2 seconds in s3) and move (which takes time 
> proportional to file size).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)