[jira] [Commented] (HIVE-14615) Temp table leaves behind insert command

2017-12-29 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306551#comment-16306551
 ] 

Hive QA commented on HIVE-14615:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904041/HIVE-14615.1.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 367 failed/errored test(s), 11543 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_index] 
(batchId=237)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[mapjoin2] 
(batchId=245)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
 (batchId=245)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_dynamic_partitions]
 (batchId=248)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_table]
 (batchId=248)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_directory]
 (batchId=248)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions]
 (batchId=248)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions_merge_move]
 (batchId=248)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions_merge_only]
 (batchId=248)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions_move_only]
 (batchId=248)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_table]
 (batchId=248)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_insert_overwrite] 
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_join] (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin] 
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_subquery] 
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_vectorization] 
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_view_delete] 
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alterColumnStatsPart] 
(batchId=85)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alterColumnStats] 
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_add_partition]
 (batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_partition_drop]
 (batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_update_status]
 (batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_update_status_disable_bitvector]
 (batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_view_col_type] 
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[analyze_tbl_date] 
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[array_size_estimation] 
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_10] 
(batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_5] 
(batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_5a] 
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] 
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avrotblsjoin] 
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[basicstat_partval] 
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_insert] 
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cmdwithcomments] 
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnStatsUpdateForStatsOptimizer_2]
 (batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[column_names_with_leading_and_trailing_spaces]
 (batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comma_in_column_name] 
(batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[concat_op] (batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constantPropWhen] 
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constantfolding] 
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_5] (batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_7] (batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_4] (batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_5] (batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] 
(batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_precision] 
(batchId=52)

[jira] [Commented] (HIVE-14615) Temp table leaves behind insert command

2017-12-29 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306536#comment-16306536
 ] 

Hive QA commented on HIVE-14615:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  5m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
11s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 13s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 035eca3 |
| Default Java | 1.8.0_111 |
| modules | C: ql hbase-handler . U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8390/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Temp table leaves behind insert command
> ---
>
> Key: HIVE-14615
> URL: https://issues.apache.org/jira/browse/HIVE-14615
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Chaoyu Tang
>Assignee: Andrew Sherman
> Attachments: HIVE-14615.1.patch
>
>
> {code}
> create table test (key int, value string);
> insert into test values (1, 'val1');
> show tables;
> test
> values__tmp__table__1
> {code}
> the temp table values__tmp__table__1 was resulted from insert into ...values
> and exists until logout the session.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-14615) Temp table leaves behind insert command

2017-12-29 Thread Andrew Sherman (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-14615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Sherman updated HIVE-14615:
--
Attachment: HIVE-14615.1.patch

> Temp table leaves behind insert command
> ---
>
> Key: HIVE-14615
> URL: https://issues.apache.org/jira/browse/HIVE-14615
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Chaoyu Tang
>Assignee: Andrew Sherman
> Attachments: HIVE-14615.1.patch
>
>
> {code}
> create table test (key int, value string);
> insert into test values (1, 'val1');
> show tables;
> test
> values__tmp__table__1
> {code}
> the temp table values__tmp__table__1 was resulted from insert into ...values
> and exists until logout the session.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-14615) Temp table leaves behind insert command

2017-12-29 Thread Andrew Sherman (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-14615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Sherman updated HIVE-14615:
--
Status: Patch Available  (was: Open)

> Temp table leaves behind insert command
> ---
>
> Key: HIVE-14615
> URL: https://issues.apache.org/jira/browse/HIVE-14615
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Chaoyu Tang
>Assignee: Andrew Sherman
> Attachments: HIVE-14615.1.patch
>
>
> {code}
> create table test (key int, value string);
> insert into test values (1, 'val1');
> show tables;
> test
> values__tmp__table__1
> {code}
> the temp table values__tmp__table__1 was resulted from insert into ...values
> and exists until logout the session.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18347) Allow dynamic lookup of Hive Metastores via Consul

2017-12-29 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306497#comment-16306497
 ] 

Hive QA commented on HIVE-18347:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904039/HIVE-18347.3.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 11542 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] 
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] 
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_lineage2]
 (batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage2] 
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage3] 
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols]
 (batchId=158)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
 (batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
 (batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
 (batchId=213)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=222)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8389/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8389/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8389/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12904039 - PreCommit-HIVE-Build

> Allow dynamic lookup of Hive Metastores via Consul
> --
>
> Key: HIVE-18347
> URL: https://issues.apache.org/jira/browse/HIVE-18347
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-18347.1.patch, HIVE-18347.2.patch, 
> HIVE-18347.3.patch
>
>
> In our organization, we have deployed HiveMetastore and HiveServer2 on Mesos 
> as dynamic services for scalability and flexibility.
> In this architecture, we would like to allow HiveServer2 to dynamically load 
> balance between Metastores (which may be scaled up and down or to different 
> nodes) for different requests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18347) Allow dynamic lookup of Hive Metastores via Consul

2017-12-29 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306465#comment-16306465
 ] 

Hive QA commented on HIVE-18347:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
21s{color} | {color:red} standalone-metastore: The patch generated 7 new + 750 
unchanged - 2 fixed = 757 total (was 752) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
13s{color} | {color:red} contrib: The patch generated 463 new + 0 unchanged - 0 
fixed = 463 total (was 0) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch 114 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
56s{color} | {color:red} standalone-metastore generated 1 new + 59 unchanged - 
0 fixed = 60 total (was 59) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 15s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  
xml  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 035eca3 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8389/yetus/diff-checkstyle-standalone-metastore.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8389/yetus/diff-checkstyle-contrib.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8389/yetus/whitespace-tabs.txt
 |
| javadoc | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8389/yetus/diff-javadoc-javadoc-standalone-metastore.txt
 |
| modules | C: common standalone-metastore contrib U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8389/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Allow dynamic lookup of Hive Metastores via Consul
> --
>
> Key: HIVE-18347
> URL: https://issues.apache.org/jira/browse/HIVE-18347
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-18347.1.patch, HIVE-18347.2.patch, 
> HIVE-18347.3.patch
>
>
> In our organization, we have deployed HiveMetastore and HiveServer2 on Mesos 
> as dynamic services for scalability and flexibility.
> In this architecture, we would like to allow HiveServer2 to dynamically load

[jira] [Updated] (HIVE-18347) Allow dynamic lookup of Hive Metastores via Consul

2017-12-29 Thread Szehon Ho (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-18347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szehon Ho updated HIVE-18347:
-
Attachment: HIVE-18347.3.patch

Fix compiling of just the 'contrib' module by adding explicit dependency to 
metastore.

> Allow dynamic lookup of Hive Metastores via Consul
> --
>
> Key: HIVE-18347
> URL: https://issues.apache.org/jira/browse/HIVE-18347
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-18347.1.patch, HIVE-18347.2.patch, 
> HIVE-18347.3.patch
>
>
> In our organization, we have deployed HiveMetastore and HiveServer2 on Mesos 
> as dynamic services for scalability and flexibility.
> In this architecture, we would like to allow HiveServer2 to dynamically load 
> balance between Metastores (which may be scaled up and down or to different 
> nodes) for different requests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18347) Allow dynamic lookup of Hive Metastores via Consul

2017-12-29 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306443#comment-16306443
 ] 

Hive QA commented on HIVE-18347:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904024/HIVE-18347.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 11542 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] 
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_lineage2]
 (batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage2] 
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage3] 
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols]
 (batchId=158)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
 (batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
 (batchId=213)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=222)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8388/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8388/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8388/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12904024 - PreCommit-HIVE-Build

> Allow dynamic lookup of Hive Metastores via Consul
> --
>
> Key: HIVE-18347
> URL: https://issues.apache.org/jira/browse/HIVE-18347
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-18347.1.patch, HIVE-18347.2.patch
>
>
> In our organization, we have deployed HiveMetastore and HiveServer2 on Mesos 
> as dynamic services for scalability and flexibility.
> In this architecture, we would like to allow HiveServer2 to dynamically load 
> balance between Metastores (which may be scaled up and down or to different 
> nodes) for different requests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18347) Allow dynamic lookup of Hive Metastores via Consul

2017-12-29 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306408#comment-16306408
 ] 

Hive QA commented on HIVE-18347:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
13s{color} | {color:red} contrib in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
13s{color} | {color:red} contrib in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 13s{color} 
| {color:red} contrib in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
21s{color} | {color:red} standalone-metastore: The patch generated 7 new + 750 
unchanged - 2 fixed = 757 total (was 752) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
12s{color} | {color:red} contrib: The patch generated 463 new + 0 unchanged - 0 
fixed = 463 total (was 0) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch 114 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  1m  
4s{color} | {color:red} standalone-metastore generated 1 new + 59 unchanged - 0 
fixed = 60 total (was 59) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
10s{color} | {color:red} contrib generated 2 new + 1 unchanged - 0 fixed = 3 
total (was 1) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 10s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  
xml  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 035eca3 |
| Default Java | 1.8.0_111 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8388/yetus/patch-mvninstall-contrib.txt
 |
| compile | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8388/yetus/patch-compile-contrib.txt
 |
| javac | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8388/yetus/patch-compile-contrib.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8388/yetus/diff-checkstyle-standalone-metastore.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8388/yetus/diff-checkstyle-contrib.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8388/yetus/whitespace-tabs.txt
 |
| javadoc | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8388/yetus/diff-javadoc-javadoc-standalone-metastore.txt
 |
| javadoc | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8388/yetus/diff-javadoc-javadoc-contrib.txt
 |
| modules | C: common standalone-metastore contrib U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8388/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Allow dynamic lookup of Hive Metastores via Consul
>

[jira] [Updated] (HIVE-18347) Allow dynamic lookup of Hive Metastores via Consul

2017-12-29 Thread Szehon Ho (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-18347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szehon Ho updated HIVE-18347:
-
Component/s: Metastore

> Allow dynamic lookup of Hive Metastores via Consul
> --
>
> Key: HIVE-18347
> URL: https://issues.apache.org/jira/browse/HIVE-18347
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-18347.1.patch, HIVE-18347.2.patch
>
>
> In our organization, we have deployed HiveMetastore and HiveServer2 on Mesos 
> as dynamic services for scalability and flexibility.
> In this architecture, we would like to allow HiveServer2 to dynamically load 
> balance between Metastores (which may be scaled up and down or to different 
> nodes) for different requests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18347) Allow dynamic lookup of Hive Metastores via Consul

2017-12-29 Thread Szehon Ho (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-18347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szehon Ho updated HIVE-18347:
-
Attachment: HIVE-18347.2.patch

> Allow dynamic lookup of Hive Metastores via Consul
> --
>
> Key: HIVE-18347
> URL: https://issues.apache.org/jira/browse/HIVE-18347
> Project: Hive
>  Issue Type: New Feature
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-18347.1.patch, HIVE-18347.2.patch
>
>
> In our organization, we have deployed HiveMetastore and HiveServer2 on Mesos 
> as dynamic services for scalability and flexibility.
> In this architecture, we would like to allow HiveServer2 to dynamically load 
> balance between Metastores (which may be scaled up and down or to different 
> nodes) for different requests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18347) Allow dynamic lookup of Hive Metastores via Consul

2017-12-29 Thread Szehon Ho (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-18347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szehon Ho updated HIVE-18347:
-
Status: Patch Available  (was: Open)

This is my first patch in awhile, I hope I did it correctly :)

> Allow dynamic lookup of Hive Metastores via Consul
> --
>
> Key: HIVE-18347
> URL: https://issues.apache.org/jira/browse/HIVE-18347
> Project: Hive
>  Issue Type: New Feature
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-18347.1.patch, HIVE-18347.2.patch
>
>
> In our organization, we have deployed HiveMetastore and HiveServer2 on Mesos 
> as dynamic services for scalability and flexibility.
> In this architecture, we would like to allow HiveServer2 to dynamically load 
> balance between Metastores (which may be scaled up and down or to different 
> nodes) for different requests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18347) Allow dynamic lookup of Hive Metastores via Consul

2017-12-29 Thread Szehon Ho (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-18347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szehon Ho updated HIVE-18347:
-
Attachment: HIVE-18347.1.patch

This patch allows a plugable Hive Metastore URI Resolver hook that is called to 
resolve Metastore URI's from HiveMetastoreClient, and implements one that we 
use in production.  This connects to our Marathon-based Consul service for 
lookup of a particular Consul-registered service, and will read a consul-based 
scheme for hive.metastore.uris.

One can imagine other schemes, like for example Zookeeper-based Metastore 
registration, those could also be implemented via this plugin.

> Allow dynamic lookup of Hive Metastores via Consul
> --
>
> Key: HIVE-18347
> URL: https://issues.apache.org/jira/browse/HIVE-18347
> Project: Hive
>  Issue Type: New Feature
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-18347.1.patch
>
>
> In our organization, we have deployed HiveMetastore and HiveServer2 on Mesos 
> as dynamic services for scalability and flexibility.
> In this architecture, we would like to allow HiveServer2 to dynamically load 
> balance between Metastores (which may be scaled up and down or to different 
> nodes) for different requests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-18347) Allow dynamic lookup of Hive Metastores via Consul

2017-12-29 Thread Szehon Ho (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-18347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szehon Ho reassigned HIVE-18347:


Assignee: Szehon Ho

> Allow dynamic lookup of Hive Metastores via Consul
> --
>
> Key: HIVE-18347
> URL: https://issues.apache.org/jira/browse/HIVE-18347
> Project: Hive
>  Issue Type: New Feature
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>
> In our organization, we have deployed HiveMetastore and HiveServer2 on Mesos 
> as dynamic services for scalability and flexibility.
> In this architecture, we would like to allow HiveServer2 to dynamically load 
> balance between Metastores (which may be scaled up and down or to different 
> nodes) for different requests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18301) Investigate to enable MapInput cache in Hive on Spark

2017-12-29 Thread Rui Li (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-18301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306218#comment-16306218
 ] 

Rui Li commented on HIVE-18301:
---

I think SMB map join is one case where the "input file change" event is 
necessary -- whenever the big table input file changes, SMBMapJoinOperator 
needs to find corresponding input files for small tables in order to performa 
bucketed join. Maybe we can identify all such cases and make sure MapInput 
cache is disabled. For other cases, we can cache MapInput and just fix the NPE.
[~xuefuz], could you share your thoughts on this? Thanks.

> Investigate to enable MapInput cache in Hive on Spark
> -
>
> Key: HIVE-18301
> URL: https://issues.apache.org/jira/browse/HIVE-18301
> Project: Hive
>  Issue Type: Bug
>Reporter: liyunzhang
>Assignee: liyunzhang
>
> Before IOContext problem is found in MapTran when spark rdd cache is enabled 
> in HIVE-8920.
> so we disabled rdd cache in MapTran at 
> [SparkPlanGenerator|https://github.com/kellyzly/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java#L202].
>   The problem is IOContext seems not initialized correctly in the spark yarn 
> client/cluster mode and caused the exception like 
> {code}
> Job aborted due to stage failure: Task 93 in stage 0.0 failed 4 times, most 
> recent failure: Lost task 93.3 in stage 0.0 (TID 616, bdpe48): 
> java.lang.RuntimeException: Error processing row: 
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:165)
>   at 
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:48)
>   at 
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
>   at 
> org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList.hasNext(HiveBaseFunctionResultList.java:85)
>   at 
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42)
>   at 
> org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
>   at org.apache.spark.scheduler.Task.run(Task.scala:85)
>   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.exec.AbstractMapOperator.getNominalPath(AbstractMapOperator.java:101)
>   at 
> org.apache.hadoop.hive.ql.exec.MapOperator.cleanUpInputFileChangedOp(MapOperator.java:516)
>   at 
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1187)
>   at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)
>   at 
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:152)
>   ... 12 more
> Driver stacktrace:
> {code}
> in yarn client/cluster mode, sometimes 
> [ExecMapperContext#currentInputPath|https://github.com/kellyzly/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapperContext.java#L109]
>  is null when rdd cach is enabled.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)