[jira] [Commented] (HIVE-18299) DbNotificationListener fail on mysql with "select for update"

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296405#comment-16296405
 ] 

Hive QA commented on HIVE-18299:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
19s{color} | {color:red} standalone-metastore: The patch generated 2 new + 486 
unchanged - 0 fixed = 488 total (was 486) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 42s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 9efed65 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8313/yetus/diff-checkstyle-standalone-metastore.txt
 |
| modules | C: standalone-metastore U: standalone-metastore |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8313/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> DbNotificationListener fail on mysql with "select for update"
> -
>
> Key: HIVE-18299
> URL: https://issues.apache.org/jira/browse/HIVE-18299
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-18299.1.patch
>
>
> This is a continuation of HIVE-17830, which haven't solved the issue. We need 
> to run "SET \@\@session.sql_mode=ANSI_QUOTES" statement before we run 
> select \"NEXT_EVENT_ID\" from \"NOTIFICATION_SEQUENCE\"". We shall keep table 
> name quoted to be in consistent with rest of ObjectStore code. This approach 
> is the same as what MetaStoreDirectSql take (set session variable before 
> every query).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17982) Move metastore specific itests

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296375#comment-16296375
 ] 

Hive QA commented on HIVE-17982:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12902732/HIVE-17982.patch

{color:green}SUCCESS:{color} +1 due to 28 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 11528 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] 
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
 (batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin] 
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
 (batchId=178)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
 (batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_10]
 (batchId=138)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketsortoptimize_insert_7]
 (batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi] 
(batchId=113)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut 
(batchId=219)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=226)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8312/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8312/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8312/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12902732 - PreCommit-HIVE-Build

> Move metastore specific itests
> --
>
> Key: HIVE-17982
> URL: https://issues.apache.org/jira/browse/HIVE-17982
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Attachments: HIVE-17982.patch
>
>
> There are a number of tests in itests/hive-unit/.../metastore that are 
> metastore specific.  I suspect they were initially placed in itests only 
> because the metastore pulling in a few plugins from ql.
> Given that we need to be able to release the metastore separately, we need to 
> be able to test it completely as a standalone entity.  So I propose to move a 
> number of the itests over into standalone-metastore.  I will only move tests 
> that are isolated to the metastore.  Anything that tests wider functionality 
> I plan to leave in itests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18159) Vectorization: Support Map type in MapWork

[Ferdinand Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296363#comment-16296363
 ] 

Ferdinand Xu commented on HIVE-18159:
-

LGTM +1 pending on the test

> Vectorization: Support Map type in MapWork
> --
>
> Key: HIVE-18159
> URL: https://issues.apache.org/jira/browse/HIVE-18159
> Project: Hive
>  Issue Type: Improvement
>Reporter: Colin Ma
>Assignee: Colin Ma
> Attachments: HIVE-18159.001.patch, HIVE-18159.002.patch
>
>
> Support Complex Types in vectorization is finished in HIVE-16589, but Map 
> type is still not support in MapWork. This ticket is target to support it for 
> MapWork when vectorization is enable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18159) Vectorization: Support Map type in MapWork

[Colin Ma (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296338#comment-16296338
 ] 

Colin Ma commented on HIVE-18159:
-

[~Ferd], thanks for your review, the patch is updated according to your 
comments on RB and fix the problems from Yetus.

> Vectorization: Support Map type in MapWork
> --
>
> Key: HIVE-18159
> URL: https://issues.apache.org/jira/browse/HIVE-18159
> Project: Hive
>  Issue Type: Improvement
>Reporter: Colin Ma
>Assignee: Colin Ma
> Attachments: HIVE-18159.001.patch, HIVE-18159.002.patch
>
>
> Support Complex Types in vectorization is finished in HIVE-16589, but Map 
> type is still not support in MapWork. This ticket is target to support it for 
> MapWork when vectorization is enable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18159) Vectorization: Support Map type in MapWork

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated HIVE-18159:

Attachment: HIVE-18159.002.patch

> Vectorization: Support Map type in MapWork
> --
>
> Key: HIVE-18159
> URL: https://issues.apache.org/jira/browse/HIVE-18159
> Project: Hive
>  Issue Type: Improvement
>Reporter: Colin Ma
>Assignee: Colin Ma
> Attachments: HIVE-18159.001.patch, HIVE-18159.002.patch
>
>
> Support Complex Types in vectorization is finished in HIVE-16589, but Map 
> type is still not support in MapWork. This ticket is target to support it for 
> MapWork when vectorization is enable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18297) Add builder for metastore Thrift classes missed in the first pass

[Peter Vary (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296310#comment-16296310
 ] 

Peter Vary commented on HIVE-18297:
---

Hi [~alangates]
I have a working one for functions.
That will need a rebase and such, so I can post it only in the beginning of 
next year, if you think that is enough.
Thanks for your work on HMS.
Peter

> Add builder for metastore Thrift classes missed in the first pass
> -
>
> Key: HIVE-18297
> URL: https://issues.apache.org/jira/browse/HIVE-18297
> Project: Hive
>  Issue Type: Task
>  Components: Standalone Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>
> The first pass of adding builders for the metastore Thrift classes missed 
> Function, statistics, and WM* objects.  Builders for these should be added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17982) Move metastore specific itests

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296300#comment-16296300
 ] 

Hive QA commented on HIVE-17982:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
15s{color} | {color:green} standalone-metastore: The patch generated 0 new + 
195 unchanged - 1 fixed = 195 total (was 196) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 59s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  
xml  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 9efed65 |
| Default Java | 1.8.0_111 |
| modules | C: standalone-metastore itests/hive-unit U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8312/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Move metastore specific itests
> --
>
> Key: HIVE-17982
> URL: https://issues.apache.org/jira/browse/HIVE-17982
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Attachments: HIVE-17982.patch
>
>
> There are a number of tests in itests/hive-unit/.../metastore that are 
> metastore specific.  I suspect they were initially placed in itests only 
> because the metastore pulling in a few plugins from ql.
> Given that we need to be able to release the metastore separately, we need to 
> be able to test it completely as a standalone entity.  So I propose to move a 
> number of the itests over into standalone-metastore.  I will only move tests 
> that are isolated to the metastore.  Anything that tests wider functionality 
> I plan to leave in itests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18298) Fix TestReplicationScenarios.testConstraints

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296271#comment-16296271
 ] 

Hive QA commented on HIVE-18298:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12902729/HIVE-18298.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 11528 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] 
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
 (batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
 (batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin] 
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=160)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[materialized_view_authorization_create_no_grant]
 (batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
 (batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_1]
 (batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_10]
 (batchId=138)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketsortoptimize_insert_7]
 (batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi] 
(batchId=113)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut 
(batchId=209)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8311/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8311/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8311/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 19 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12902729 - PreCommit-HIVE-Build

> Fix TestReplicationScenarios.testConstraints
> 
>
> Key: HIVE-18298
> URL: https://issues.apache.org/jira/browse/HIVE-18298
> Project: Hive
>  Issue Type: Bug
>  Components: repl
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-18298.1.patch
>
>
> The test if broken by HIVE-16603. Currently on constraints are created 
> without order on replication destination cluster during bootstrap, after 
> HIVE-16603, it is no longer possible. We need to create foreign keys at last 
> after all primary keys are created.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18268) Hive Prepared Statement when split with double quoted in query fails

[Choi JaeHwan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Choi JaeHwan updated HIVE-18268:

Attachment: HIVE-18268.4.patch

> Hive Prepared Statement when split with double quoted in query fails
> 
>
> Key: HIVE-18268
> URL: https://issues.apache.org/jira/browse/HIVE-18268
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 2.3.2
>Reporter: Choi JaeHwan
>Assignee: Choi JaeHwan
> Fix For: 3.0.0, 2.4.0, 2.3.3
>
> Attachments: HIVE-18268.1.patch, HIVE-18268.2.patch, 
> HIVE-18268.3.patch, HIVE-18268.4.patch, HIVE-18268.patch
>
>
> HIVE-13625, Change sql statement split when odd number of escape characters, 
> and add parameter counter validation, above 
> {code:java}
> // prev code
> StringBuilder newSql = new StringBuilder(parts.get(0));
> for(int i=1;i   if(!parameters.containsKey(i)){
> throw new SQLException("Parameter #"+i+" is unset");
>   }
>   newSql.append(parameters.get(i));
>   newSql.append(parts.get(i));
> }
> // change from HIVE-13625
> int paramLoc = 1;
> while (getCharIndexFromSqlByParamLocation(sql, '?', paramLoc) > 0) {
>   // check the user has set the needs parameters
>   if (parameters.containsKey(paramLoc)) {
> int tt = getCharIndexFromSqlByParamLocation(newSql.toString(), '?', 
> 1);
> newSql.deleteCharAt(tt);
> newSql.insert(tt, parameters.get(paramLoc));
>   }
>   paramLoc++;
> }
> {code}
> If the number of split SQL and the number of parameters are not matched, an 
> SQLException is thrown
> Currently, when splitting SQL, there is no processing for double quoted, and 
> when the token ('?' ) is between double quote, SQL is split.
> i think when the token between double quoted is literal, it is correct to not 
> split.
> for example, above the query;
> {code:java}
> // Some comments here
> 1:  String query =  " select 1 from x where qa="?" "
> 2:  String query = " SELECT 1 FROM `x` WHERE (trecord LIKE "ALA[d_?]%")
> {code}
> ? is literal, then query do not split. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18268) Hive Prepared Statement when split with double quoted in query fails

[Choi JaeHwan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Choi JaeHwan updated HIVE-18268:

Status: Patch Available  (was: Open)

> Hive Prepared Statement when split with double quoted in query fails
> 
>
> Key: HIVE-18268
> URL: https://issues.apache.org/jira/browse/HIVE-18268
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 2.3.2
>Reporter: Choi JaeHwan
>Assignee: Choi JaeHwan
> Fix For: 3.0.0, 2.4.0, 2.3.3
>
> Attachments: HIVE-18268.1.patch, HIVE-18268.2.patch, 
> HIVE-18268.3.patch, HIVE-18268.4.patch, HIVE-18268.patch
>
>
> HIVE-13625, Change sql statement split when odd number of escape characters, 
> and add parameter counter validation, above 
> {code:java}
> // prev code
> StringBuilder newSql = new StringBuilder(parts.get(0));
> for(int i=1;i   if(!parameters.containsKey(i)){
> throw new SQLException("Parameter #"+i+" is unset");
>   }
>   newSql.append(parameters.get(i));
>   newSql.append(parts.get(i));
> }
> // change from HIVE-13625
> int paramLoc = 1;
> while (getCharIndexFromSqlByParamLocation(sql, '?', paramLoc) > 0) {
>   // check the user has set the needs parameters
>   if (parameters.containsKey(paramLoc)) {
> int tt = getCharIndexFromSqlByParamLocation(newSql.toString(), '?', 
> 1);
> newSql.deleteCharAt(tt);
> newSql.insert(tt, parameters.get(paramLoc));
>   }
>   paramLoc++;
> }
> {code}
> If the number of split SQL and the number of parameters are not matched, an 
> SQLException is thrown
> Currently, when splitting SQL, there is no processing for double quoted, and 
> when the token ('?' ) is between double quote, SQL is split.
> i think when the token between double quoted is literal, it is correct to not 
> split.
> for example, above the query;
> {code:java}
> // Some comments here
> 1:  String query =  " select 1 from x where qa="?" "
> 2:  String query = " SELECT 1 FROM `x` WHERE (trecord LIKE "ALA[d_?]%")
> {code}
> ? is literal, then query do not split. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18268) Hive Prepared Statement when split with double quoted in query fails

[Choi JaeHwan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Choi JaeHwan updated HIVE-18268:

Status: Open  (was: Patch Available)

> Hive Prepared Statement when split with double quoted in query fails
> 
>
> Key: HIVE-18268
> URL: https://issues.apache.org/jira/browse/HIVE-18268
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 2.3.2
>Reporter: Choi JaeHwan
>Assignee: Choi JaeHwan
> Fix For: 3.0.0, 2.4.0, 2.3.3
>
> Attachments: HIVE-18268.1.patch, HIVE-18268.2.patch, 
> HIVE-18268.3.patch, HIVE-18268.patch
>
>
> HIVE-13625, Change sql statement split when odd number of escape characters, 
> and add parameter counter validation, above 
> {code:java}
> // prev code
> StringBuilder newSql = new StringBuilder(parts.get(0));
> for(int i=1;i   if(!parameters.containsKey(i)){
> throw new SQLException("Parameter #"+i+" is unset");
>   }
>   newSql.append(parameters.get(i));
>   newSql.append(parts.get(i));
> }
> // change from HIVE-13625
> int paramLoc = 1;
> while (getCharIndexFromSqlByParamLocation(sql, '?', paramLoc) > 0) {
>   // check the user has set the needs parameters
>   if (parameters.containsKey(paramLoc)) {
> int tt = getCharIndexFromSqlByParamLocation(newSql.toString(), '?', 
> 1);
> newSql.deleteCharAt(tt);
> newSql.insert(tt, parameters.get(paramLoc));
>   }
>   paramLoc++;
> }
> {code}
> If the number of split SQL and the number of parameters are not matched, an 
> SQLException is thrown
> Currently, when splitting SQL, there is no processing for double quoted, and 
> when the token ('?' ) is between double quote, SQL is split.
> i think when the token between double quoted is literal, it is correct to not 
> split.
> for example, above the query;
> {code:java}
> // Some comments here
> 1:  String query =  " select 1 from x where qa="?" "
> 2:  String query = " SELECT 1 FROM `x` WHERE (trecord LIKE "ALA[d_?]%")
> {code}
> ? is literal, then query do not split. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18301) Investigate to enable MapInput cache in Hive on Spark

[liyunzhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyunzhang updated HIVE-18301:
--
Description: 
Before IOContext problem is found in MapTran when spark rdd cache is enabled in 
HIVE-8920.
so we disabled rdd cache in MapTran at 
[SparkPlanGenerator|https://github.com/kellyzly/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java#L202].
  The problem is IOContext seems not initialized correctly in the spark yarn 
client/cluster mode and caused the exception like 
{code}
Job aborted due to stage failure: Task 93 in stage 0.0 failed 4 times, most 
recent failure: Lost task 93.3 in stage 0.0 (TID 616, bdpe48): 
java.lang.RuntimeException: Error processing row: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:165)
at 
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:48)
at 
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
at 
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList.hasNext(HiveBaseFunctionResultList.java:85)
at 
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42)
at 
org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
at org.apache.spark.scheduler.Task.run(Task.scala:85)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.AbstractMapOperator.getNominalPath(AbstractMapOperator.java:101)
at 
org.apache.hadoop.hive.ql.exec.MapOperator.cleanUpInputFileChangedOp(MapOperator.java:516)
at 
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1187)
at 
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)
at 
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:152)
... 12 more

Driver stacktrace:
{code}
in yarn client/cluster mode, sometimes 
[currenthttps://github.com/kellyzly/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapperContext.java#L109

  was:
Before IOContext problem is found in MapTran when spark rdd cache is enabled in 
HIVE-8920.
so we disabled rdd cache in MapTran at 
[SparkPlanGenerator|https://github.com/kellyzly/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java#L202].
  The problem is IOContext seems not initialized well in the spark yarn 
client/cluster mode and caused the exception like
{code}
Job aborted due to stage failure: Task 93 in stage 0.0 failed 4 times, most 
recent failure: Lost task 93.3 in stage 0.0 (TID 616, bdpe48): 
java.lang.RuntimeException: Error processing row: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:165)
at 
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:48)
at 
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
at 
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList.hasNext(HiveBaseFunctionResultList.java:85)
at 
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42)
at 
org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
at org.apache.spark.scheduler.Task.run(Task.scala:85)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.AbstractMapOperator.getNominalPath(AbstractMapOperator.java:101)
at 
org.apache.hadoop.hive.ql.exec.MapOperator.cleanUpInputFileChangedOp(MapOperator.java:516)
at 

[jira] [Updated] (HIVE-18301) Investigate to enable MapInput cache in Hive on Spark

[liyunzhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyunzhang updated HIVE-18301:
--
Description: 
Before IOContext problem is found in MapTran when spark rdd cache is enabled in 
HIVE-8920.
so we disabled rdd cache in MapTran at 
[SparkPlanGenerator|https://github.com/kellyzly/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java#L202].
  The problem is IOContext seems not initialized correctly in the spark yarn 
client/cluster mode and caused the exception like 
{code}
Job aborted due to stage failure: Task 93 in stage 0.0 failed 4 times, most 
recent failure: Lost task 93.3 in stage 0.0 (TID 616, bdpe48): 
java.lang.RuntimeException: Error processing row: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:165)
at 
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:48)
at 
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
at 
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList.hasNext(HiveBaseFunctionResultList.java:85)
at 
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42)
at 
org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
at org.apache.spark.scheduler.Task.run(Task.scala:85)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.AbstractMapOperator.getNominalPath(AbstractMapOperator.java:101)
at 
org.apache.hadoop.hive.ql.exec.MapOperator.cleanUpInputFileChangedOp(MapOperator.java:516)
at 
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1187)
at 
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)
at 
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:152)
... 12 more

Driver stacktrace:
{code}
in yarn client/cluster mode, sometimes 
[ExecMapperContext#currentInputPath|https://github.com/kellyzly/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapperContext.java#L109]
 is null when rdd cach is enabled.

  was:
Before IOContext problem is found in MapTran when spark rdd cache is enabled in 
HIVE-8920.
so we disabled rdd cache in MapTran at 
[SparkPlanGenerator|https://github.com/kellyzly/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java#L202].
  The problem is IOContext seems not initialized correctly in the spark yarn 
client/cluster mode and caused the exception like 
{code}
Job aborted due to stage failure: Task 93 in stage 0.0 failed 4 times, most 
recent failure: Lost task 93.3 in stage 0.0 (TID 616, bdpe48): 
java.lang.RuntimeException: Error processing row: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:165)
at 
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:48)
at 
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
at 
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList.hasNext(HiveBaseFunctionResultList.java:85)
at 
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42)
at 
org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
at org.apache.spark.scheduler.Task.run(Task.scala:85)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.AbstractMapOperator.getNominalPath(AbstractMapOperator.java:101)
at 

[jira] [Assigned] (HIVE-18301) Investigate to enable MapInput cache in Hive on Spark

[liyunzhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyunzhang reassigned HIVE-18301:
-


> Investigate to enable MapInput cache in Hive on Spark
> -
>
> Key: HIVE-18301
> URL: https://issues.apache.org/jira/browse/HIVE-18301
> Project: Hive
>  Issue Type: Bug
>Reporter: liyunzhang
>Assignee: liyunzhang
>
> Before IOContext problem is found in MapTran when spark rdd cache is enabled 
> in HIVE-8920.
> so we disabled rdd cache in MapTran at 
> [SparkPlanGenerator|https://github.com/kellyzly/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java#L202].
>   The problem is IOContext seems not initialized well in the spark yarn 
> client/cluster mode and caused the exception like
> {code}
> Job aborted due to stage failure: Task 93 in stage 0.0 failed 4 times, most 
> recent failure: Lost task 93.3 in stage 0.0 (TID 616, bdpe48): 
> java.lang.RuntimeException: Error processing row: 
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:165)
>   at 
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:48)
>   at 
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
>   at 
> org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList.hasNext(HiveBaseFunctionResultList.java:85)
>   at 
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42)
>   at 
> org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
>   at org.apache.spark.scheduler.Task.run(Task.scala:85)
>   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.exec.AbstractMapOperator.getNominalPath(AbstractMapOperator.java:101)
>   at 
> org.apache.hadoop.hive.ql.exec.MapOperator.cleanUpInputFileChangedOp(MapOperator.java:516)
>   at 
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1187)
>   at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)
>   at 
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:152)
>   ... 12 more
> Driver stacktrace:
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18298) Fix TestReplicationScenarios.testConstraints

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296226#comment-16296226
 ] 

Hive QA commented on HIVE-18298:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
37s{color} | {color:red} ql: The patch generated 9 new + 14 unchanged - 5 fixed 
= 23 total (was 19) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 37s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 9efed65 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8311/yetus/diff-checkstyle-ql.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8311/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Fix TestReplicationScenarios.testConstraints
> 
>
> Key: HIVE-18298
> URL: https://issues.apache.org/jira/browse/HIVE-18298
> Project: Hive
>  Issue Type: Bug
>  Components: repl
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-18298.1.patch
>
>
> The test if broken by HIVE-16603. Currently on constraints are created 
> without order on replication destination cluster during bootstrap, after 
> HIVE-16603, it is no longer possible. We need to create foreign keys at last 
> after all primary keys are created.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18270) count(distinct) using join and group by produce incorrect output when hive.auto.convert.join=false and hive.auto.convert.join.noconditionaltask=false

[Zac Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zac Zhou updated HIVE-18270:

Attachment: HIVE-18270.3.patch

As hive 3.0 has refactored the code, and the ReducesinkDeduplicationUtils class 
was added in HIVE-17037.  It looks like that the bug dose not exist in the 
master.  

> count(distinct) using join and group by produce incorrect output when 
> hive.auto.convert.join=false and 
> hive.auto.convert.join.noconditionaltask=false
> -
>
> Key: HIVE-18270
> URL: https://issues.apache.org/jira/browse/HIVE-18270
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.1, 2.1.1, 2.2.0, 2.3.0
>Reporter: Zac Zhou
>Assignee: Zac Zhou
> Attachments: HIVE-18270.1.patch, HIVE-18270.2.patch, 
> HIVE-18270.3.patch
>
>
> When I run the following query:
> explain 
> SELECT foo.id, count(distinct foo.line_id) as factor from 
>  foo JOIN bar ON (foo.id = bar.id)
>  WHERE foo.orders != 'blah'  
>  group by foo.id; 
> The following error is got:
> java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
>   at java.util.ArrayList.rangeCheck(ArrayList.java:635)
>   at java.util.ArrayList.get(ArrayList.java:411)
>   at 
> org.apache.hadoop.hive.ql.optimizer.correlation.ReduceSinkDeDuplication$AbsctractReducerReducerProc.merge(ReduceSinkDeDuplication.java:216)
>   at 
> org.apache.hadoop.hive.ql.optimizer.correlation.ReduceSinkDeDuplication$JoinReducerProc.process(ReduceSinkDeDuplication.java:557)
>   at 
> org.apache.hadoop.hive.ql.optimizer.correlation.ReduceSinkDeDuplication$AbsctractReducerReducerProc.process(ReduceSinkDeDuplication.java:166)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:95)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:79)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:133)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:110)
>   at 
> org.apache.hadoop.hive.ql.optimizer.correlation.ReduceSinkDeDuplication.transform(ReduceSinkDeDuplication.java:108)
>   at 
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:192)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10201)
>   at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:209)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
>   at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> It looks like it is a bug of ReduceSinkDeDuplication optimizer. 
> Since the columns of count distinct need to be added into reduce key for 
> sorting, the reducesink of group can't be replaced with the ones of join. 
> In the case of count distinct query, reducesink of group should not be merged 
>  
>   



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-14792) AvroSerde reads the remote schema-file at least once per mapper, per table reference.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296208#comment-16296208
 ] 

Hive QA commented on HIVE-14792:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12902717/HIVE-14792.3.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 87 failed/errored test(s), 11528 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_subquery] 
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_column_stats]
 (batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_stats_status]
 (batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[analyze_tbl_date] 
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_1] 
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_6] 
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_view_3] 
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_view_4] 
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_1] 
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_2] 
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_3] 
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnStatsUpdateForStatsOptimizer_2]
 (batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnstats_infinity] 
(batchId=76)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_like_tbl_props] 
(batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_table_like_stats] 
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[deleteAnalyze] 
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_all_partitioned] 
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_where_partitioned]
 (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_whole_partition] 
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_acid] (batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_acid_fast] 
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] 
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[outer_reference_windowed]
 (batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[quotedid_stats] 
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats14] (batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats15] (batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats9] (batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_invalidation] 
(batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[temp_table_display_colstats_tbllvl]
 (batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_stats] (batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_all_partitioned] 
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_where_partitioned]
 (batchId=62)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] 
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets]
 (batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[autoColumnStats_1]
 (batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[autoColumnStats_2]
 (batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
 (batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnStatsUpdateForStatsOptimizer_1]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[column_table_stats]
 (batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[column_table_stats_orc]
 (batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[deleteAnalyze]
 (batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[delete_all_partitioned]
 (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[delete_where_partitioned]
 (batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[delete_whole_partition]
 (batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
 (batchId=161)

[jira] [Commented] (HIVE-17829) ArrayIndexOutOfBoundsException - HBASE-backed tables with Avro schema in Hive2

[anishek (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296205#comment-16296205
 ] 

anishek commented on HIVE-17829:


[~thejas] will look at the tests too add one. Yeh looks like if there are 
comments then users should be able to get around this issue.


> ArrayIndexOutOfBoundsException - HBASE-backed tables with Avro schema in Hive2
> --
>
> Key: HIVE-17829
> URL: https://issues.apache.org/jira/browse/HIVE-17829
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Affects Versions: 2.1.0
>Reporter: Chiran Ravani
>Assignee: anishek
>Priority: Critical
> Attachments: HIVE-17829.0.patch
>
>
> Stack
> {code}
> 2017-10-09T09:39:54,804 ERROR [HiveServer2-Background-Pool: Thread-95]: 
> metadata.Table (Table.java:getColsInternal(642)) - Unable to get field from 
> serde: org.apache.hadoop.hive.hbase.HBaseSerDe
> java.lang.ArrayIndexOutOfBoundsException: 1
> at java.util.Arrays$ArrayList.get(Arrays.java:3841) ~[?:1.8.0_77]
> at 
> org.apache.hadoop.hive.serde2.BaseStructObjectInspector.init(BaseStructObjectInspector.java:104)
>  ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.init(LazySimpleStructObjectInspector.java:97)
>  ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.(LazySimpleStructObjectInspector.java:77)
>  ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyObjectInspectorFactory.getLazySimpleStructObjectInspector(LazyObjectInspectorFactory.java:115)
>  ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.hbase.HBaseLazyObjectFactory.createLazyHBaseStructInspector(HBaseLazyObjectFactory.java:79)
>  ~[hive-hbase-handler-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.hbase.HBaseSerDe.initialize(HBaseSerDe.java:127) 
> ~[hive-hbase-handler-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54) 
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:531) 
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:424)
>  ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:411)
>  ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:279)
>  ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:261) 
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:639) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:622) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:833) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:869) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4228) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:347) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1905) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1607) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1354) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1123) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1116) 
> 

[jira] [Updated] (HIVE-18283) Better error message and error code for HoS exceptions

[Chao Sun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao Sun updated HIVE-18283:

Attachment: HIVE-18283.3.patch

Thanks [~xuefuz] and [~asherman] for the comments! They are very helpful. 
Attaching patch v3 to address the comments.

> Better error message and error code for HoS exceptions
> --
>
> Key: HIVE-18283
> URL: https://issues.apache.org/jira/browse/HIVE-18283
> Project: Hive
>  Issue Type: Improvement
>  Components: Spark
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-18283.0.patch, HIVE-18283.1.patch, 
> HIVE-18283.2.patch, HIVE-18283.3.patch
>
>
> Right now HoS only use a few error codes. For the majority of the errors, 
> user will see an error code 1 followed by a lengthy stacktrace. This is not 
> ideal since:
> 1. It is often hard to find the root cause - sometimes it is hidden deeply 
> inside the stacktrace.
> 2. After identifying the root cause, it is not easy to find a fix. Often user 
> have to copy & paste the error message and google them. 
> 3. It is not clear whether the error is transient or not, depending on which 
> user may want to retry the query. 
> To improve the above, this JIRA propose to assign error code & canonical 
> error messages for different HoS errors. We can take advantage of the 
> existing {{ErrorMsg}} class.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-14792) AvroSerde reads the remote schema-file at least once per mapper, per table reference.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296170#comment-16296170
 ] 

Hive QA commented on HIVE-14792:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
16s{color} | {color:red} common: The patch generated 1 new + 930 unchanged - 1 
fixed = 931 total (was 931) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
29s{color} | {color:red} ql: The patch generated 1 new + 10 unchanged - 0 fixed 
= 11 total (was 10) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
11s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m  1s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 9efed65 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8310/yetus/diff-checkstyle-common.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8310/yetus/diff-checkstyle-ql.txt
 |
| modules | C: common ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8310/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> AvroSerde reads the remote schema-file at least once per mapper, per table 
> reference.
> -
>
> Key: HIVE-14792
> URL: https://issues.apache.org/jira/browse/HIVE-14792
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
>  Labels: TODOC2.2, TODOC2.4
> Fix For: 3.0.0, 2.4.0, 2.2.1
>
> Attachments: HIVE-14792.1.patch, HIVE-14792.3.patch
>
>
> Avro tables that use "external" schema files stored on HDFS can cause 
> excessive calls to {{FileSystem::open()}}, especially for queries that spawn 
> large numbers of mappers.
> This is because of the following code in {{AvroSerDe::initialize()}}:
> {code:title=AvroSerDe.java|borderStyle=solid}
> public void initialize(Configuration configuration, Properties properties) 
> throws SerDeException {
> // ...
> if (hasExternalSchema(properties)
> || columnNameProperty == null || columnNameProperty.isEmpty()
> || 

[jira] [Commented] (HIVE-18294) add switch to make acid table the default

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296148#comment-16296148
 ] 

Hive QA commented on HIVE-18294:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12902716/HIVE-18294.03.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 50 failed/errored test(s), 11528 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats] 
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization_acid2]
 (batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_orig_table] 
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_conversions] 
(batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ppd_decimal] 
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
 (batchId=22)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all] 
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
 (batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
 (batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_orig_table]
 (batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table]
 (batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mm_conversions]
 (batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez]
 (batchId=103)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3] 
(batchId=102)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] 
(batchId=102)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[vector_join_part_col_char]
 (batchId=102)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
 (batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
 (batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_10]
 (batchId=138)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketsortoptimize_insert_7]
 (batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi] 
(batchId=113)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query39] 
(batchId=248)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut 
(batchId=209)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadData (batchId=257)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataNonAcid2AcidConversion 
(batchId=257)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataNonAcid2AcidConversionVectorized
 (batchId=257)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataPartitioned (batchId=257)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataUpdate (batchId=257)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataUpdateVectorized (batchId=257)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataVectorized (batchId=257)
org.apache.hadoop.hive.ql.TestTxnLoadData.testAbort (batchId=257)
org.apache.hadoop.hive.ql.TestTxnLoadData.testMultiStatement (batchId=257)
org.apache.hadoop.hive.ql.TestTxnLoadData.testMultiStatementVectorized 
(batchId=257)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testNoBuckets (batchId=278)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testNonAcidToAcidVectorzied 
(batchId=278)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testToAcidConversion02 (batchId=278)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testNoBuckets (batchId=278)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testNonAcidToAcidVectorzied
 (batchId=278)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testToAcidConversion02 
(batchId=278)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.checkExpectedLocks 
(batchId=291)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=226)

[jira] [Updated] (HIVE-17434) Using "add jar " from viewFs always occurred hdfs mismatch error

[Bang Xiao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bang Xiao updated HIVE-17434:
-
Status: Patch Available  (was: In Progress)

> Using "add jar " from viewFs always occurred hdfs mismatch error
> 
>
> Key: HIVE-17434
> URL: https://issues.apache.org/jira/browse/HIVE-17434
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: shenxianqiang
>Assignee: Bang Xiao
>Priority: Minor
> Fix For: 1.2.1
>
> Attachments: HIVE-17434-1.patch, HIVE-17434.patch
>
>
> add jar viewfs://nsX//lib/common.jar
> always occure mismatch error



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18294) add switch to make acid table the default

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296081#comment-16296081
 ] 

Hive QA commented on HIVE-18294:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
 6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
16s{color} | {color:red} standalone-metastore: The patch generated 1 new + 209 
unchanged - 0 fixed = 210 total (was 209) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
39s{color} | {color:red} ql: The patch generated 2 new + 1089 unchanged - 0 
fixed = 1091 total (was 1089) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 19s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 9efed65 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8309/yetus/diff-checkstyle-standalone-metastore.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8309/yetus/diff-checkstyle-ql.txt
 |
| modules | C: common standalone-metastore ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8309/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> add switch to make acid table the default
> -
>
> Key: HIVE-18294
> URL: https://issues.apache.org/jira/browse/HIVE-18294
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18294.01.patch, HIVE-18294.03.patch
>
>
> it would be convenient for testing to have a switch that enables the behavior 
> where all suitable table tables (currently ORC + not sorted) are 
> automatically reacted with transactional=true.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (HIVE-18265) desc formatted/extended or show create table can not fully display the result when field or table comment contains tab character

[Hui Huang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296062#comment-16296062
 ] 

Hui Huang edited comment on HIVE-18265 at 12/19/17 2:45 AM:


[~asherman] could you take a look when you have time? I have ran the test cases 
without added codes, those failed test cases are still appeared, so I don't 
think these failed testcase is related. Thanks!


was (Author: bigrey):
https://issues.apache.org/jira/secure/ViewProfile.jspa?name=asherman could you 
take a look when you have time? I have ran the test cases without added codes, 
those failed test cases are still appeared, so I don't think these failed 
testcase is related. Thanks!

> desc formatted/extended or show create table can not fully display the result 
> when field or table comment contains tab character
> 
>
> Key: HIVE-18265
> URL: https://issues.apache.org/jira/browse/HIVE-18265
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 3.0.0
>Reporter: Hui Huang
>Assignee: Hui Huang
> Fix For: 3.0.0
>
> Attachments: HIVE-18265.1.patch, HIVE-18265.patch
>
>
> Here are some examples:
> create table test_comment (id1 string comment 'full_\tname1', id2 string 
> comment 'full_\tname2', id3 string comment 'full_\tname3') stored as textfile;
> When execute `show create table test_comment`, we can see the following 
> content in the console,
> {quote}
> createtab_stmt
> CREATE TABLE `test_comment`(
>   `id1` string COMMENT 'full_
>   `id2` string COMMENT 'full_
>   `id3` string COMMENT 'full_
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> STORED AS INPUTFORMAT
>   'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
>   'hdfs://xxx/user/huanghui/warehouse/huanghuitest.db/test_comment'
> TBLPROPERTIES (
>   'transient_lastDdlTime'='1513095570')
> {quote}
> And the output of `desc formatted table ` is a little similar,
> {quote}
> col_name  data_type   comment
> \# col_name   data_type   comment
> id1   string  full_
> id2   string  full_
> id3   string  full_
> \# Detailed Table Information
> (ignore)...
> {quote}
> When execute `desc extended test_comment`, the problem is more obvious,
> {quote}
> col_name  data_type   comment
> id1   string  full_
> id2   string  full_
> id3   string  full_
> Detailed Table InformationTable(tableName:test_comment, 
> dbName:huanghuitest, owner:huanghui, createTime:1513095570, lastAccessTime:0, 
> retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:id1, type:string, 
> comment:full_name1), FieldSchema(name:id2, type:string, comment:full_
> {quote}
> *the rest of the content is lost*.
> The content is not really lost, it's just can not display normal. Because 
> hive store the result in LazyStruct, and LazyStruct use '\t' as field 
> separator:
> {code:java}
> // LazyStruct.java#parse()
> // Go through all bytes in the byte[]
> while (fieldByteEnd <= structByteEnd) {
>   if (fieldByteEnd == structByteEnd || bytes[fieldByteEnd] == separator) {
> // Reached the end of a field?
> if (lastColumnTakesRest && fieldId == fields.length - 1) {
>   fieldByteEnd = structByteEnd;
> }
> startPosition[fieldId] = fieldByteBegin;
> fieldId++;
> if (fieldId == fields.length || fieldByteEnd == structByteEnd) {
>   // All fields have been parsed, or bytes have been parsed.
>   // We need to set the startPosition of fields.length to ensure we
>   // can use the same formula to calculate the length of each field.
>   // For missing fields, their starting positions will all be the 
> same,
>   // which will make their lengths to be -1 and uncheckedGetField will
>   // return these fields as NULLs.
>   for (int i = fieldId; i <= fields.length; i++) {
> startPosition[i] = fieldByteEnd + 1;
>   }
>   break;
> }
> fieldByteBegin = fieldByteEnd + 1;
> fieldByteEnd++;
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18265) desc formatted/extended or show create table can not fully display the result when field or table comment contains tab character

[Hui Huang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296062#comment-16296062
 ] 

Hui Huang commented on HIVE-18265:
--

https://issues.apache.org/jira/secure/ViewProfile.jspa?name=asherman could you 
take a look when you have time? I have ran the test cases without added codes, 
those failed test cases are still appeared, so I don't think these failed 
testcase is related. Thanks!

> desc formatted/extended or show create table can not fully display the result 
> when field or table comment contains tab character
> 
>
> Key: HIVE-18265
> URL: https://issues.apache.org/jira/browse/HIVE-18265
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 3.0.0
>Reporter: Hui Huang
>Assignee: Hui Huang
> Fix For: 3.0.0
>
> Attachments: HIVE-18265.1.patch, HIVE-18265.patch
>
>
> Here are some examples:
> create table test_comment (id1 string comment 'full_\tname1', id2 string 
> comment 'full_\tname2', id3 string comment 'full_\tname3') stored as textfile;
> When execute `show create table test_comment`, we can see the following 
> content in the console,
> {quote}
> createtab_stmt
> CREATE TABLE `test_comment`(
>   `id1` string COMMENT 'full_
>   `id2` string COMMENT 'full_
>   `id3` string COMMENT 'full_
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> STORED AS INPUTFORMAT
>   'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
>   'hdfs://xxx/user/huanghui/warehouse/huanghuitest.db/test_comment'
> TBLPROPERTIES (
>   'transient_lastDdlTime'='1513095570')
> {quote}
> And the output of `desc formatted table ` is a little similar,
> {quote}
> col_name  data_type   comment
> \# col_name   data_type   comment
> id1   string  full_
> id2   string  full_
> id3   string  full_
> \# Detailed Table Information
> (ignore)...
> {quote}
> When execute `desc extended test_comment`, the problem is more obvious,
> {quote}
> col_name  data_type   comment
> id1   string  full_
> id2   string  full_
> id3   string  full_
> Detailed Table InformationTable(tableName:test_comment, 
> dbName:huanghuitest, owner:huanghui, createTime:1513095570, lastAccessTime:0, 
> retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:id1, type:string, 
> comment:full_name1), FieldSchema(name:id2, type:string, comment:full_
> {quote}
> *the rest of the content is lost*.
> The content is not really lost, it's just can not display normal. Because 
> hive store the result in LazyStruct, and LazyStruct use '\t' as field 
> separator:
> {code:java}
> // LazyStruct.java#parse()
> // Go through all bytes in the byte[]
> while (fieldByteEnd <= structByteEnd) {
>   if (fieldByteEnd == structByteEnd || bytes[fieldByteEnd] == separator) {
> // Reached the end of a field?
> if (lastColumnTakesRest && fieldId == fields.length - 1) {
>   fieldByteEnd = structByteEnd;
> }
> startPosition[fieldId] = fieldByteBegin;
> fieldId++;
> if (fieldId == fields.length || fieldByteEnd == structByteEnd) {
>   // All fields have been parsed, or bytes have been parsed.
>   // We need to set the startPosition of fields.length to ensure we
>   // can use the same formula to calculate the length of each field.
>   // For missing fields, their starting positions will all be the 
> same,
>   // which will make their lengths to be -1 and uncheckedGetField will
>   // return these fields as NULLs.
>   for (int i = fieldId; i <= fields.length; i++) {
> startPosition[i] = fieldByteEnd + 1;
>   }
>   break;
> }
> fieldByteBegin = fieldByteEnd + 1;
> fieldByteEnd++;
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18283) Better error message and error code for HoS exceptions

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296059#comment-16296059
 ] 

Hive QA commented on HIVE-18283:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12902712/HIVE-18283.2.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 11133 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=93)

[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

[Sahil Takiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296053#comment-16296053
 ] 

Sahil Takiar commented on HIVE-17684:
-

I'm not positive, but I think if the Hadoop 3.0.0 artifacts were just uploaded 
it takes a while for them to be replicated across all mirrors, so maybe that is 
what we are seeing.

Sometime there can be issues if you don't upload a patch that has been rebased 
on master.

So I would re-generate the patch after rebasing on master and then re-attach v2 
of the patch tomorrow morning. Hopefully that works.

> HoS memory issues with MapJoinMemoryExhaustionHandler
> -
>
> Key: HIVE-17684
> URL: https://issues.apache.org/jira/browse/HIVE-17684
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Misha Dmitriev
> Attachments: HIVE-17684.01.patch
>
>
> We have seen a number of memory issues due the {{HashSinkOperator}} use of 
> the {{MapJoinMemoryExhaustionHandler}}. This handler is meant to detect 
> scenarios where the small table is taking too much space in memory, in which 
> case a {{MapJoinMemoryExhaustionError}} is thrown.
> The configs to control this logic are:
> {{hive.mapjoin.localtask.max.memory.usage}} (default 0.90)
> {{hive.mapjoin.followby.gby.localtask.max.memory.usage}} (default 0.55)
> The handler works by using the {{MemoryMXBean}} and uses the following logic 
> to estimate how much memory the {{HashMap}} is consuming: 
> {{MemoryMXBean#getHeapMemoryUsage().getUsed() / 
> MemoryMXBean#getHeapMemoryUsage().getMax()}}
> The issue is that {{MemoryMXBean#getHeapMemoryUsage().getUsed()}} can be 
> inaccurate. The value returned by this method returns all reachable and 
> unreachable memory on the heap, so there may be a bunch of garbage data, and 
> the JVM just hasn't taken the time to reclaim it all. This can lead to 
> intermittent failures of this check even though a simple GC would have 
> reclaimed enough space for the process to continue working.
> We should re-think the usage of {{MapJoinMemoryExhaustionHandler}} for HoS. 
> In Hive-on-MR this probably made sense to use because every Hive task was run 
> in a dedicated container, so a Hive Task could assume it created most of the 
> data on the heap. However, in Hive-on-Spark there can be multiple Hive Tasks 
> running in a single executor, each doing different things.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18248) Clean up parameters

[Sahil Takiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296047#comment-16296047
 ] 

Sahil Takiar commented on HIVE-18248:
-

[~janulatha] are the test failures related?

Otherwise +1 LGTM

> Clean up parameters
> ---
>
> Key: HIVE-18248
> URL: https://issues.apache.org/jira/browse/HIVE-18248
> Project: Hive
>  Issue Type: Bug
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
> Fix For: 3.0.0
>
> Attachments: HIVE-18248.1.patch, HIVE-18248.2.patch
>
>
> Clean up of parameters that need not change at run time.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

[Misha Dmitriev (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296040#comment-16296040
 ] 

Misha Dmitriev commented on HIVE-17684:
---

Thank you for taking a look, [~stakiar]. Yes, naturally this code builds for me 
locally:

{code}
$ mvn clean install -DskipTests
...
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 03:58 min
[INFO] Finished at: 2017-12-18T13:12:43-08:00
[INFO] Final Memory: 369M/2219M
[INFO] 
{code}

The error in this build looks somewhat strange in that it mentions datanucleus. 
Another strange thing that I see in the console log is a few lines above:

{code}
error: a/pom.xml: does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java: 
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java: does not 
exist in index
Going to apply patch with: git apply -p1
{code}

I had a suspicion that maybe my local code base is too far behind, so I've just 
run 'git fetch; git rebase' - this reapplied my change without problems. So I 
am not sure what's going  on here.


> HoS memory issues with MapJoinMemoryExhaustionHandler
> -
>
> Key: HIVE-17684
> URL: https://issues.apache.org/jira/browse/HIVE-17684
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Misha Dmitriev
> Attachments: HIVE-17684.01.patch
>
>
> We have seen a number of memory issues due the {{HashSinkOperator}} use of 
> the {{MapJoinMemoryExhaustionHandler}}. This handler is meant to detect 
> scenarios where the small table is taking too much space in memory, in which 
> case a {{MapJoinMemoryExhaustionError}} is thrown.
> The configs to control this logic are:
> {{hive.mapjoin.localtask.max.memory.usage}} (default 0.90)
> {{hive.mapjoin.followby.gby.localtask.max.memory.usage}} (default 0.55)
> The handler works by using the {{MemoryMXBean}} and uses the following logic 
> to estimate how much memory the {{HashMap}} is consuming: 
> {{MemoryMXBean#getHeapMemoryUsage().getUsed() / 
> MemoryMXBean#getHeapMemoryUsage().getMax()}}
> The issue is that {{MemoryMXBean#getHeapMemoryUsage().getUsed()}} can be 
> inaccurate. The value returned by this method returns all reachable and 
> unreachable memory on the heap, so there may be a bunch of garbage data, and 
> the JVM just hasn't taken the time to reclaim it all. This can lead to 
> intermittent failures of this check even though a simple GC would have 
> reclaimed enough space for the process to continue working.
> We should re-think the usage of {{MapJoinMemoryExhaustionHandler}} for HoS. 
> In Hive-on-MR this probably made sense to use because every Hive task was run 
> in a dedicated container, so a Hive Task could assume it created most of the 
> data on the heap. However, in Hive-on-Spark there can be multiple Hive Tasks 
> running in a single executor, each doing different things.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

[Sahil Takiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296028#comment-16296028
 ] 

Sahil Takiar commented on HIVE-17684:
-

Thanks for taking this on [~mi...@cloudera.com]. Looks like Hive QA is failing 
due to:

{code}
[ERROR] Failed to execute goal on project hive-shims-common: Could not resolve 
dependencies for project 
org.apache.hive.shims:hive-shims-common:jar:3.0.0-SNAPSHOT: The following 
artifacts could not be resolved: org.apache.hadoop:hadoop-common:jar:3.0.0, 
org.apache.hadoop:hadoop-auth:jar:3.0.0, 
org.apache.hadoop:hadoop-hdfs-client:jar:3.0.0, 
org.apache.hadoop:hadoop-yarn-api:jar:3.0.0, 
org.apache.hadoop:hadoop-yarn-client:jar:3.0.0, 
org.apache.hadoop:hadoop-mapreduce-client-core:jar:3.0.0, 
org.apache.hadoop:hadoop-yarn-common:jar:3.0.0, 
org.apache.hadoop:hadoop-annotations:jar:3.0.0: Could not find artifact 
org.apache.hadoop:hadoop-common:jar:3.0.0 in datanucleus
{code}

Maybe the Hadoop 3.0.0 artifacts haven't propagated to datanucleus yet. Can you 
build your patch locally?

> HoS memory issues with MapJoinMemoryExhaustionHandler
> -
>
> Key: HIVE-17684
> URL: https://issues.apache.org/jira/browse/HIVE-17684
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Misha Dmitriev
> Attachments: HIVE-17684.01.patch
>
>
> We have seen a number of memory issues due the {{HashSinkOperator}} use of 
> the {{MapJoinMemoryExhaustionHandler}}. This handler is meant to detect 
> scenarios where the small table is taking too much space in memory, in which 
> case a {{MapJoinMemoryExhaustionError}} is thrown.
> The configs to control this logic are:
> {{hive.mapjoin.localtask.max.memory.usage}} (default 0.90)
> {{hive.mapjoin.followby.gby.localtask.max.memory.usage}} (default 0.55)
> The handler works by using the {{MemoryMXBean}} and uses the following logic 
> to estimate how much memory the {{HashMap}} is consuming: 
> {{MemoryMXBean#getHeapMemoryUsage().getUsed() / 
> MemoryMXBean#getHeapMemoryUsage().getMax()}}
> The issue is that {{MemoryMXBean#getHeapMemoryUsage().getUsed()}} can be 
> inaccurate. The value returned by this method returns all reachable and 
> unreachable memory on the heap, so there may be a bunch of garbage data, and 
> the JVM just hasn't taken the time to reclaim it all. This can lead to 
> intermittent failures of this check even though a simple GC would have 
> reclaimed enough space for the process to continue working.
> We should re-think the usage of {{MapJoinMemoryExhaustionHandler}} for HoS. 
> In Hive-on-MR this probably made sense to use because every Hive task was run 
> in a dedicated container, so a Hive Task could assume it created most of the 
> data on the heap. However, in Hive-on-Spark there can be multiple Hive Tasks 
> running in a single executor, each doing different things.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18285) StatsTask uses a cached ql.metadata.Table object

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296026#comment-16296026
 ] 

Ashutosh Chauhan commented on HIVE-18285:
-

[~kgyrtkirk] Can you please review this?

> StatsTask uses a cached ql.metadata.Table object
> 
>
> Key: HIVE-18285
> URL: https://issues.apache.org/jira/browse/HIVE-18285
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Statistics
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18285.01.patch
>
>
> this then causes BasicStatsTask.aggregateStats(Hive) to call 
> Hive.alterTable() with a stale Table object.  (It misses any changes made by 
> any MetaStorePreEventListener)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18224) Introduce interface above driver

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296024#comment-16296024
 ] 

Ashutosh Chauhan commented on HIVE-18224:
-

Can you please create a RB for this?

> Introduce interface above driver
> 
>
> Key: HIVE-18224
> URL: https://issues.apache.org/jira/browse/HIVE-18224
> Project: Hive
>  Issue Type: Sub-task
>  Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18224.01.patch, HIVE-18224.02.patch, 
> HIVE-18224.03.patch
>
>
> Add an interface above driver; and use it outside of ql.
> The goal is to enable the overlaying of the Driver with some strategy.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18283) Better error message and error code for HoS exceptions

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296005#comment-16296005
 ] 

Hive QA commented on HIVE-18283:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
29s{color} | {color:red} ql: The patch generated 2 new + 72 unchanged - 2 fixed 
= 74 total (was 74) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m  6s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 9efed65 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8308/yetus/diff-checkstyle-ql.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8308/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Better error message and error code for HoS exceptions
> --
>
> Key: HIVE-18283
> URL: https://issues.apache.org/jira/browse/HIVE-18283
> Project: Hive
>  Issue Type: Improvement
>  Components: Spark
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-18283.0.patch, HIVE-18283.1.patch, 
> HIVE-18283.2.patch
>
>
> Right now HoS only use a few error codes. For the majority of the errors, 
> user will see an error code 1 followed by a lengthy stacktrace. This is not 
> ideal since:
> 1. It is often hard to find the root cause - sometimes it is hidden deeply 
> inside the stacktrace.
> 2. After identifying the root cause, it is not easy to find a fix. Often user 
> have to copy & paste the error message and google them. 
> 3. It is not clear whether the error is transient or not, depending on which 
> user may want to retry the query. 
> To improve the above, this JIRA propose to assign error code & canonical 
> error messages for different HoS errors. We can take advantage of the 
> existing {{ErrorMsg}} class.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18300) Use LAST_INSERT_ID for Sequence Tables for MySQL

[BELUGA BEHR (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

BELUGA BEHR updated HIVE-18300:
---
Description: 
Instead of using SELECT FOR UPDATE, for MYSQL, use the recommended approach 
which only requires once client interaction instead of two:

{code:sql}
UPDATE child_codes SET counter_field = LAST_INSERT_ID(counter_field + 1);
SELECT LAST_INSERT_ID();
{code}

https://dev.mysql.com/doc/refman/5.7/en/innodb-locking-reads.html

  was:
Instead of using SELECT FOR UPDATE, for MYSQL, use the recommended approach 
which only requires once client interaction instead of two:

{code:sql}
UPDATE child_codes SET counter_field = LAST_INSERT_ID(counter_field + 1);
SELECT LAST_INSERT_ID();
{code}


> Use LAST_INSERT_ID for Sequence Tables for MySQL
> 
>
> Key: HIVE-18300
> URL: https://issues.apache.org/jira/browse/HIVE-18300
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Priority: Trivial
>
> Instead of using SELECT FOR UPDATE, for MYSQL, use the recommended approach 
> which only requires once client interaction instead of two:
> {code:sql}
> UPDATE child_codes SET counter_field = LAST_INSERT_ID(counter_field + 1);
> SELECT LAST_INSERT_ID();
> {code}
> https://dev.mysql.com/doc/refman/5.7/en/innodb-locking-reads.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18283) Better error message and error code for HoS exceptions

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295991#comment-16295991
 ] 

Hive QA commented on HIVE-18283:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12902712/HIVE-18283.2.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 11529 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_udf2] 
(batchId=73)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
 (batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
 (batchId=178)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
 (batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
 (batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_10]
 (batchId=138)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketsortoptimize_insert_7]
 (batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi] 
(batchId=113)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut 
(batchId=209)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=226)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8307/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8307/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8307/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12902712 - PreCommit-HIVE-Build

> Better error message and error code for HoS exceptions
> --
>
> Key: HIVE-18283
> URL: https://issues.apache.org/jira/browse/HIVE-18283
> Project: Hive
>  Issue Type: Improvement
>  Components: Spark
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-18283.0.patch, HIVE-18283.1.patch, 
> HIVE-18283.2.patch
>
>
> Right now HoS only use a few error codes. For the majority of the errors, 
> user will see an error code 1 followed by a lengthy stacktrace. This is not 
> ideal since:
> 1. It is often hard to find the root cause - sometimes it is hidden deeply 
> inside the stacktrace.
> 2. After identifying the root cause, it is not easy to find a fix. Often user 
> have to copy & paste the error message and google them. 
> 3. It is not clear whether the error is transient or not, depending on which 
> user may want to retry the query. 
> To improve the above, this JIRA propose to assign error code & canonical 
> error messages for different HoS errors. We can take advantage of the 
> existing {{ErrorMsg}} class.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HIVE-17671) TableScanDesc.isAcidTable is restricted to FullAcid tables

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman resolved HIVE-17671.
---
   Resolution: Resolved
Fix Version/s: 3.0.0

superseded by HIVE-18124

> TableScanDesc.isAcidTable is restricted to FullAcid tables
> --
>
> Key: HIVE-17671
> URL: https://issues.apache.org/jira/browse/HIVE-17671
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 3.0.0
>
>
> _isAcidTable = AcidUtils.isAcidTable(this.tableMetadata);_
> is changed to 
> _isAcidTable = AcidUtils.isFullAcidTable(this.tableMetadata);_
> This property is then checked all over the place - why?
> This then affects TableScanDesc.isAcidTable() so FetchTask, HiveInputFormat 
> etc assume that they are handling Acid read only if it's full acid... this 
> doesn't look right



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17671) TableScanDesc.isAcidTable is restricted to FullAcid tables

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman reassigned HIVE-17671:
-

Assignee: Eugene Koifman

> TableScanDesc.isAcidTable is restricted to FullAcid tables
> --
>
> Key: HIVE-17671
> URL: https://issues.apache.org/jira/browse/HIVE-17671
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> _isAcidTable = AcidUtils.isAcidTable(this.tableMetadata);_
> is changed to 
> _isAcidTable = AcidUtils.isFullAcidTable(this.tableMetadata);_
> This property is then checked all over the place - why?
> This then affects TableScanDesc.isAcidTable() so FetchTask, HiveInputFormat 
> etc assume that they are handling Acid read only if it's full acid... this 
> doesn't look right



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18283) Better error message and error code for HoS exceptions

[Andrew Sherman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295967#comment-16295967
 ] 

Andrew Sherman commented on HIVE-18283:
---

Patch looks good to me.

If you do rework it again you might
1) change the order of arguments to assertEquals as the expected value is 
supposed to be first in junit
2) refactor out some of testGetHiveException() so that you can call
{noformat}
checkExpectedException("\t diagnostics: Application 
application_1508358311878_3322732 "
+ "failed 1 times due to ApplicationMaster for attempt "
+ "appattempt_1508358311878_3322732_01 timed out. Failing 
the application.", 
 ErrorMsg.SPARK_CREATE_CLIENT_TIMEOUT);
checkExpectedException("\t diagnostics: Application 
application_1508358311878_333 "
+ "submitted by user hive to unknown queue: foo", 
 ErrorMsg.SPARK_CREATE_CLIENT_INVALID_QUEUE);
{noformat}
etc. as this would make it more readable and extendable.

But these are nits, the basic code is good and will be useful

> Better error message and error code for HoS exceptions
> --
>
> Key: HIVE-18283
> URL: https://issues.apache.org/jira/browse/HIVE-18283
> Project: Hive
>  Issue Type: Improvement
>  Components: Spark
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-18283.0.patch, HIVE-18283.1.patch, 
> HIVE-18283.2.patch
>
>
> Right now HoS only use a few error codes. For the majority of the errors, 
> user will see an error code 1 followed by a lengthy stacktrace. This is not 
> ideal since:
> 1. It is often hard to find the root cause - sometimes it is hidden deeply 
> inside the stacktrace.
> 2. After identifying the root cause, it is not easy to find a fix. Often user 
> have to copy & paste the error message and google them. 
> 3. It is not clear whether the error is transient or not, depending on which 
> user may want to retry the query. 
> To improve the above, this JIRA propose to assign error code & canonical 
> error messages for different HoS errors. We can take advantage of the 
> existing {{ErrorMsg}} class.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18283) Better error message and error code for HoS exceptions

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295922#comment-16295922
 ] 

Hive QA commented on HIVE-18283:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
31s{color} | {color:red} ql: The patch generated 2 new + 72 unchanged - 2 fixed 
= 74 total (was 74) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 56s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 9efed65 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8307/yetus/diff-checkstyle-ql.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8307/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Better error message and error code for HoS exceptions
> --
>
> Key: HIVE-18283
> URL: https://issues.apache.org/jira/browse/HIVE-18283
> Project: Hive
>  Issue Type: Improvement
>  Components: Spark
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-18283.0.patch, HIVE-18283.1.patch, 
> HIVE-18283.2.patch
>
>
> Right now HoS only use a few error codes. For the majority of the errors, 
> user will see an error code 1 followed by a lengthy stacktrace. This is not 
> ideal since:
> 1. It is often hard to find the root cause - sometimes it is hidden deeply 
> inside the stacktrace.
> 2. After identifying the root cause, it is not easy to find a fix. Often user 
> have to copy & paste the error message and google them. 
> 3. It is not clear whether the error is transient or not, depending on which 
> user may want to retry the query. 
> To improve the above, this JIRA propose to assign error code & canonical 
> error messages for different HoS errors. We can take advantage of the 
> existing {{ErrorMsg}} class.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295914#comment-16295914
 ] 

Hive QA commented on HIVE-17684:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12902709/HIVE-17684.01.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8306/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8306/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8306/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-12-19 00:09:55.007
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-8306/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-12-19 00:09:55.009
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
   12a33fd..9efed65  master -> origin/master
 + bd21225...b3cb852 standalone-metastore -> origin/standalone-metastore  
(forced update)
+ git reset --hard HEAD
HEAD is now at 12a33fd HIVE-17981 Create a set of builders for Thrift classes.  
This closes #274.  (Alan Gates, reviewed by Peter Vary)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
  (use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 9efed65 HIVE-18124 clean up isAcidTable() API vs 
isInsertOnlyTable() (Eugene Koifman, reviewed by Alan Gates)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-12-19 00:10:00.836
+ rm -rf ../yetus
+ mkdir ../yetus
+ cp -R . ../yetus
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-8306/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: a/pom.xml: does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java: 
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java: does not 
exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q 
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: protoc version: 250, detected platform: linux/amd64
protoc-jar: executing: [/tmp/protoc8478854674645480907.exe, 
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore,
 
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources,
 
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator  Version 3.5.2
Output file 
/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
 does not exist: must build 
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
[ERROR] Failed to execute goal on project hive-shims-common: Could not resolve 
dependencies for project 
org.apache.hive.shims:hive-shims-common:jar:3.0.0-SNAPSHOT: The following 
artifacts could not be resolved: org.apache.hadoop:hadoop-common:jar:3.0.0, 
org.apache.hadoop:hadoop-auth:jar:3.0.0, 
org.apache.hadoop:hadoop-hdfs-client:jar:3.0.0, 
org.apache.hadoop:hadoop-yarn-api:jar:3.0.0, 
org.apache.hadoop:hadoop-yarn-client:jar:3.0.0, 

[jira] [Commented] (HIVE-18218) SMB Join : Handle buckets with no splits.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295910#comment-16295910
 ] 

Hive QA commented on HIVE-18218:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12902705/HIVE-18218.1.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 33 failed/errored test(s), 11135 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=93)

[jira] [Commented] (HIVE-18283) Better error message and error code for HoS exceptions

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295898#comment-16295898
 ] 

Xuefu Zhang commented on HIVE-18283:


Patch looks good to me except for a minor improvement on a faster match() with 
precompiled patterns.

> Better error message and error code for HoS exceptions
> --
>
> Key: HIVE-18283
> URL: https://issues.apache.org/jira/browse/HIVE-18283
> Project: Hive
>  Issue Type: Improvement
>  Components: Spark
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-18283.0.patch, HIVE-18283.1.patch, 
> HIVE-18283.2.patch
>
>
> Right now HoS only use a few error codes. For the majority of the errors, 
> user will see an error code 1 followed by a lengthy stacktrace. This is not 
> ideal since:
> 1. It is often hard to find the root cause - sometimes it is hidden deeply 
> inside the stacktrace.
> 2. After identifying the root cause, it is not easy to find a fix. Often user 
> have to copy & paste the error message and google them. 
> 3. It is not clear whether the error is transient or not, depending on which 
> user may want to retry the query. 
> To improve the above, this JIRA propose to assign error code & canonical 
> error messages for different HoS errors. We can take advantage of the 
> existing {{ErrorMsg}} class.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17830) dbnotification fails to work with rdbms other than postgres

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295893#comment-16295893
 ] 

Daniel Dai commented on HIVE-17830:
---

Created HIVE-18299 for continuation work.

> dbnotification fails to work with rdbms other than postgres
> ---
>
> Key: HIVE-17830
> URL: https://issues.apache.org/jira/browse/HIVE-17830
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: anishek
>Assignee: Daniel Dai
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-17830.0.patch, HIVE-17830.1.patch
>
>
> as part of HIVE-17721 we had changed the direct sql to acquire the lock for 
> postgres as
> {code}
> select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update;
> {code}
> however this breaks other databases and we have to use different sql 
> statements for different databases 
> for postgres use
> {code}
> select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update;
> {code}
> for SQLServer 
> {code}
> select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" with (updlock);
> {code}
> for other databases 
> {code}
> select NEXT_EVENT_ID from NOTIFICATION_SEQUENCE for update;
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18299) DbNotificationListener fail on mysql with "select for update"

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-18299:
--
Status: Patch Available  (was: Open)

> DbNotificationListener fail on mysql with "select for update"
> -
>
> Key: HIVE-18299
> URL: https://issues.apache.org/jira/browse/HIVE-18299
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-18299.1.patch
>
>
> This is a continuation of HIVE-17830, which haven't solved the issue. We need 
> to run "SET \@\@session.sql_mode=ANSI_QUOTES" statement before we run 
> select \"NEXT_EVENT_ID\" from \"NOTIFICATION_SEQUENCE\"". We shall keep table 
> name quoted to be in consistent with rest of ObjectStore code. This approach 
> is the same as what MetaStoreDirectSql take (set session variable before 
> every query).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17981) Create a set of builders for Thrift classes

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295895#comment-16295895
 ] 

Alan Gates commented on HIVE-17981:
---

Looks like I messed up and took the yetus build for the tests.  My mistake.  
But looking at the test failures they all either have age > 1 or are clearly 
not related, as they are in LLAP and HCat encryption, which this code has 
nothing to do with.

> Create a set of builders for Thrift classes
> ---
>
> Key: HIVE-17981
> URL: https://issues.apache.org/jira/browse/HIVE-17981
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-17981.2.patch, HIVE-17981.3.patch, 
> HIVE-17981.copy.patch, HIVE-17981.patch
>
>
> Instantiating some of the Thrift classes is painful.  Consider building a 
> {{Table}} object, which requires a {{StorageDescriptor}}, which requires a 
> {{SerDeInfo}} and a list of {{FieldInfo}}.  All that is really necessary for 
> a Table in the most simple case is a name, a database, and some columns.  But 
> currently creating even a simple Table requires 20+ lines of code.  This is 
> particularly painful in tests.  
> I propose to add a set of builders.  These will come with reasonable defaults 
> to minimize the boilerplate code.  They will also include simple methods for 
> common operations (like adding columns, or a parameter) without requiring the 
> user to create all the sub-objects (like {{StorageDescriptor}}).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-18299) DbNotificationListener fail on mysql with "select for update"

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai reassigned HIVE-18299:
-


> DbNotificationListener fail on mysql with "select for update"
> -
>
> Key: HIVE-18299
> URL: https://issues.apache.org/jira/browse/HIVE-18299
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>
> This is a continuation of HIVE-17830, which haven't solved the issue. We need 
> to run "SET \@\@session.sql_mode=ANSI_QUOTES" statement before we run 
> select \"NEXT_EVENT_ID\" from \"NOTIFICATION_SEQUENCE\"". We shall keep table 
> name quoted to be in consistent with rest of ObjectStore code. This approach 
> is the same as what MetaStoreDirectSql take (set session variable before 
> every query).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18299) DbNotificationListener fail on mysql with "select for update"

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-18299:
--
Attachment: HIVE-18299.1.patch

> DbNotificationListener fail on mysql with "select for update"
> -
>
> Key: HIVE-18299
> URL: https://issues.apache.org/jira/browse/HIVE-18299
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-18299.1.patch
>
>
> This is a continuation of HIVE-17830, which haven't solved the issue. We need 
> to run "SET \@\@session.sql_mode=ANSI_QUOTES" statement before we run 
> select \"NEXT_EVENT_ID\" from \"NOTIFICATION_SEQUENCE\"". We shall keep table 
> name quoted to be in consistent with rest of ObjectStore code. This approach 
> is the same as what MetaStoreDirectSql take (set session variable before 
> every query).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17982) Move metastore specific itests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295890#comment-16295890
 ] 

ASF GitHub Bot commented on HIVE-17982:
---

GitHub user alanfgates opened a pull request:

https://github.com/apache/hive/pull/279

HIVE-17982 Move metastore specific itests



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alanfgates/hive hive17982

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/279.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #279


commit 002233b90126469edd61658c1a32988438dfae85
Author: Alan Gates 
Date:   2017-10-26T16:49:19Z

HIVE-17982 Move metastore specific itests




> Move metastore specific itests
> --
>
> Key: HIVE-17982
> URL: https://issues.apache.org/jira/browse/HIVE-17982
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Attachments: HIVE-17982.patch
>
>
> There are a number of tests in itests/hive-unit/.../metastore that are 
> metastore specific.  I suspect they were initially placed in itests only 
> because the metastore pulling in a few plugins from ql.
> Given that we need to be able to release the metastore separately, we need to 
> be able to test it completely as a standalone entity.  So I propose to move a 
> number of the itests over into standalone-metastore.  I will only move tests 
> that are isolated to the metastore.  Anything that tests wider functionality 
> I plan to leave in itests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17982) Move metastore specific itests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-17982:
--
Labels: pull-request-available  (was: )

> Move metastore specific itests
> --
>
> Key: HIVE-17982
> URL: https://issues.apache.org/jira/browse/HIVE-17982
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Attachments: HIVE-17982.patch
>
>
> There are a number of tests in itests/hive-unit/.../metastore that are 
> metastore specific.  I suspect they were initially placed in itests only 
> because the metastore pulling in a few plugins from ql.
> Given that we need to be able to release the metastore separately, we need to 
> be able to test it completely as a standalone entity.  So I propose to move a 
> number of the itests over into standalone-metastore.  I will only move tests 
> that are isolated to the metastore.  Anything that tests wider functionality 
> I plan to leave in itests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17982) Move metastore specific itests

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-17982:
--
Attachment: HIVE-17982.patch

This patch moves tests that focus solely on the metastore.  This did mean 
changing a lot of code to use the new Thrift builders and calls rather than 
SQL.  I tried to make sure the Thrift operations mirrored what the SQL 
operations were doing.

> Move metastore specific itests
> --
>
> Key: HIVE-17982
> URL: https://issues.apache.org/jira/browse/HIVE-17982
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-17982.patch
>
>
> There are a number of tests in itests/hive-unit/.../metastore that are 
> metastore specific.  I suspect they were initially placed in itests only 
> because the metastore pulling in a few plugins from ql.
> Given that we need to be able to release the metastore separately, we need to 
> be able to test it completely as a standalone entity.  So I propose to move a 
> number of the itests over into standalone-metastore.  I will only move tests 
> that are isolated to the metastore.  Anything that tests wider functionality 
> I plan to leave in itests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17982) Move metastore specific itests

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-17982:
--
Status: Patch Available  (was: Open)

> Move metastore specific itests
> --
>
> Key: HIVE-17982
> URL: https://issues.apache.org/jira/browse/HIVE-17982
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>
> There are a number of tests in itests/hive-unit/.../metastore that are 
> metastore specific.  I suspect they were initially placed in itests only 
> because the metastore pulling in a few plugins from ql.
> Given that we need to be able to release the metastore separately, we need to 
> be able to test it completely as a standalone entity.  So I propose to move a 
> number of the itests over into standalone-metastore.  I will only move tests 
> that are isolated to the metastore.  Anything that tests wider functionality 
> I plan to leave in itests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18283) Better error message and error code for HoS exceptions

[Chao Sun (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295876#comment-16295876
 ] 

Chao Sun commented on HIVE-18283:
-

I don't think the test failures are related. [~xuefuz], [~asherman]: could you 
take a look when you have time? Thanks!

> Better error message and error code for HoS exceptions
> --
>
> Key: HIVE-18283
> URL: https://issues.apache.org/jira/browse/HIVE-18283
> Project: Hive
>  Issue Type: Improvement
>  Components: Spark
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-18283.0.patch, HIVE-18283.1.patch, 
> HIVE-18283.2.patch
>
>
> Right now HoS only use a few error codes. For the majority of the errors, 
> user will see an error code 1 followed by a lengthy stacktrace. This is not 
> ideal since:
> 1. It is often hard to find the root cause - sometimes it is hidden deeply 
> inside the stacktrace.
> 2. After identifying the root cause, it is not easy to find a fix. Often user 
> have to copy & paste the error message and google them. 
> 3. It is not clear whether the error is transient or not, depending on which 
> user may want to retry the query. 
> To improve the above, this JIRA propose to assign error code & canonical 
> error messages for different HoS errors. We can take advantage of the 
> existing {{ErrorMsg}} class.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HIVE-18124) clean up isAcidTable() API vs isInsertOnlyTable()

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman resolved HIVE-18124.
---
   Resolution: Fixed
Fix Version/s: 3.0.0

patch 7 attached for completeness - includes additional doc comments in 
AcidUtils.AcidOperationalProperties
committed to master
thanks Alan for the review

>  clean up isAcidTable() API vs isInsertOnlyTable()
> --
>
> Key: HIVE-18124
> URL: https://issues.apache.org/jira/browse/HIVE-18124
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 3.0.0
>
> Attachments: HIVE-18124.01.patch, HIVE-18124.02.patch, 
> HIVE-18124.03.patch, HIVE-18124.04.patch, HIVE-18124.05.patch, 
> HIVE-18124.06.patch, HIVE-18124.07.patch
>
>
> With the addition of MM tables (_AcidUtils.isInsertOnlyTable(table)_) the 
> methods in AcidUtils and dependent places are very muddled.  There are now a 
> number of places where we have something like _isAcidTable = 
> AcidUtils.isFullAcidTable(table)_  and a later getter
> _boolean isAcidTable() \{ return isAcidTable;\}_
> Need to clean it up so that there is a isTransactional(Table) that checks 
> transactional=true setting and isAcid(Table) to mean full ACID and 
> isInsertOnly(Table) to mean MM tables.
> This would accurately describe the semantics of the tables.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18218) SMB Join : Handle buckets with no splits.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295825#comment-16295825
 ] 

Hive QA commented on HIVE-18218:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 14s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 12a33fd |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8305/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> SMB Join : Handle buckets with no splits.
> -
>
> Key: HIVE-18218
> URL: https://issues.apache.org/jira/browse/HIVE-18218
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Attachments: HIVE-18218.1.patch
>
>
> While working on HIVE-18208, it was found that with SMB, the results are 
> incorrect. This most likely is a product issue.
> auto_sortmerge_join_16 fails with wrong results due to this.
> cc [~hagleitn]
> The current logic in CustomPartitionVertex assumes that there is a split for 
> each bucket whereas in  Tez, we can have no splits for empty buckets.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18298) Fix TestReplicationScenarios.testConstraints

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-18298:
--
Status: Patch Available  (was: Open)

> Fix TestReplicationScenarios.testConstraints
> 
>
> Key: HIVE-18298
> URL: https://issues.apache.org/jira/browse/HIVE-18298
> Project: Hive
>  Issue Type: Bug
>  Components: repl
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-18298.1.patch
>
>
> The test if broken by HIVE-16603. Currently on constraints are created 
> without order on replication destination cluster during bootstrap, after 
> HIVE-16603, it is no longer possible. We need to create foreign keys at last 
> after all primary keys are created.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18298) Fix TestReplicationScenarios.testConstraints

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-18298:
--
Attachment: HIVE-18298.1.patch

In the patch, repl dump will split constraint dump file into two: one for 
foreign key and one for other. While doing repl load, Hive will create all 
other constraints first, then create foreign keys.

> Fix TestReplicationScenarios.testConstraints
> 
>
> Key: HIVE-18298
> URL: https://issues.apache.org/jira/browse/HIVE-18298
> Project: Hive
>  Issue Type: Bug
>  Components: repl
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-18298.1.patch
>
>
> The test if broken by HIVE-16603. Currently on constraints are created 
> without order on replication destination cluster during bootstrap, after 
> HIVE-16603, it is no longer possible. We need to create foreign keys at last 
> after all primary keys are created.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-18298) Fix TestReplicationScenarios.testConstraints

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai reassigned HIVE-18298:
-


> Fix TestReplicationScenarios.testConstraints
> 
>
> Key: HIVE-18298
> URL: https://issues.apache.org/jira/browse/HIVE-18298
> Project: Hive
>  Issue Type: Bug
>  Components: repl
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>
> The test if broken by HIVE-16603. Currently on constraints are created 
> without order on replication destination cluster during bootstrap, after 
> HIVE-16603, it is no longer possible. We need to create foreign keys at last 
> after all primary keys are created.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17981) Create a set of builders for Thrift classes

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295802#comment-16295802
 ] 

Hive QA commented on HIVE-17981:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12902703/HIVE-17981.3.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 11135 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=93)

[jira] [Commented] (HIVE-17981) Create a set of builders for Thrift classes

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295794#comment-16295794
 ] 

Alan Gates commented on HIVE-17981:
---

Created HIVE-18297 for builders for objects not covered in this JIRA.

> Create a set of builders for Thrift classes
> ---
>
> Key: HIVE-17981
> URL: https://issues.apache.org/jira/browse/HIVE-17981
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-17981.2.patch, HIVE-17981.3.patch, 
> HIVE-17981.copy.patch, HIVE-17981.patch
>
>
> Instantiating some of the Thrift classes is painful.  Consider building a 
> {{Table}} object, which requires a {{StorageDescriptor}}, which requires a 
> {{SerDeInfo}} and a list of {{FieldInfo}}.  All that is really necessary for 
> a Table in the most simple case is a name, a database, and some columns.  But 
> currently creating even a simple Table requires 20+ lines of code.  This is 
> particularly painful in tests.  
> I propose to add a set of builders.  These will come with reasonable defaults 
> to minimize the boilerplate code.  They will also include simple methods for 
> common operations (like adding columns, or a parameter) without requiring the 
> user to create all the sub-objects (like {{StorageDescriptor}}).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-18297) Add builder for metastore Thrift classes missed in the first pass

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates reassigned HIVE-18297:
-


> Add builder for metastore Thrift classes missed in the first pass
> -
>
> Key: HIVE-18297
> URL: https://issues.apache.org/jira/browse/HIVE-18297
> Project: Hive
>  Issue Type: Task
>  Components: Standalone Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>
> The first pass of adding builders for the metastore Thrift classes missed 
> Function, statistics, and WM* objects.  Builders for these should be added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18124) clean up isAcidTable() API vs isInsertOnlyTable()

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18124:
--
Attachment: HIVE-18124.07.patch

>  clean up isAcidTable() API vs isInsertOnlyTable()
> --
>
> Key: HIVE-18124
> URL: https://issues.apache.org/jira/browse/HIVE-18124
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18124.01.patch, HIVE-18124.02.patch, 
> HIVE-18124.03.patch, HIVE-18124.04.patch, HIVE-18124.05.patch, 
> HIVE-18124.06.patch, HIVE-18124.07.patch
>
>
> With the addition of MM tables (_AcidUtils.isInsertOnlyTable(table)_) the 
> methods in AcidUtils and dependent places are very muddled.  There are now a 
> number of places where we have something like _isAcidTable = 
> AcidUtils.isFullAcidTable(table)_  and a later getter
> _boolean isAcidTable() \{ return isAcidTable;\}_
> Need to clean it up so that there is a isTransactional(Table) that checks 
> transactional=true setting and isAcid(Table) to mean full ACID and 
> isInsertOnly(Table) to mean MM tables.
> This would accurately describe the semantics of the tables.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18124) clean up isAcidTable() API vs isInsertOnlyTable()

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18124:
--
Status: Open  (was: Patch Available)

>  clean up isAcidTable() API vs isInsertOnlyTable()
> --
>
> Key: HIVE-18124
> URL: https://issues.apache.org/jira/browse/HIVE-18124
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18124.01.patch, HIVE-18124.02.patch, 
> HIVE-18124.03.patch, HIVE-18124.04.patch, HIVE-18124.05.patch, 
> HIVE-18124.06.patch
>
>
> With the addition of MM tables (_AcidUtils.isInsertOnlyTable(table)_) the 
> methods in AcidUtils and dependent places are very muddled.  There are now a 
> number of places where we have something like _isAcidTable = 
> AcidUtils.isFullAcidTable(table)_  and a later getter
> _boolean isAcidTable() \{ return isAcidTable;\}_
> Need to clean it up so that there is a isTransactional(Table) that checks 
> transactional=true setting and isAcid(Table) to mean full ACID and 
> isInsertOnly(Table) to mean MM tables.
> This would accurately describe the semantics of the tables.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18124) clean up isAcidTable() API vs isInsertOnlyTable()

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18124:
--
Attachment: HIVE-18124.06.patch

>  clean up isAcidTable() API vs isInsertOnlyTable()
> --
>
> Key: HIVE-18124
> URL: https://issues.apache.org/jira/browse/HIVE-18124
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18124.01.patch, HIVE-18124.02.patch, 
> HIVE-18124.03.patch, HIVE-18124.04.patch, HIVE-18124.05.patch, 
> HIVE-18124.06.patch
>
>
> With the addition of MM tables (_AcidUtils.isInsertOnlyTable(table)_) the 
> methods in AcidUtils and dependent places are very muddled.  There are now a 
> number of places where we have something like _isAcidTable = 
> AcidUtils.isFullAcidTable(table)_  and a later getter
> _boolean isAcidTable() \{ return isAcidTable;\}_
> Need to clean it up so that there is a isTransactional(Table) that checks 
> transactional=true setting and isAcid(Table) to mean full ACID and 
> isInsertOnly(Table) to mean MM tables.
> This would accurately describe the semantics of the tables.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17981) Create a set of builders for Thrift classes

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-17981:
--
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Patch committed to master.  Thanks Peter for the review.  Also, I'll file JIRAs 
for the other raised issues.

> Create a set of builders for Thrift classes
> ---
>
> Key: HIVE-17981
> URL: https://issues.apache.org/jira/browse/HIVE-17981
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-17981.2.patch, HIVE-17981.3.patch, 
> HIVE-17981.copy.patch, HIVE-17981.patch
>
>
> Instantiating some of the Thrift classes is painful.  Consider building a 
> {{Table}} object, which requires a {{StorageDescriptor}}, which requires a 
> {{SerDeInfo}} and a list of {{FieldInfo}}.  All that is really necessary for 
> a Table in the most simple case is a name, a database, and some columns.  But 
> currently creating even a simple Table requires 20+ lines of code.  This is 
> particularly painful in tests.  
> I propose to add a set of builders.  These will come with reasonable defaults 
> to minimize the boilerplate code.  They will also include simple methods for 
> common operations (like adding columns, or a parameter) without requiring the 
> user to create all the sub-objects (like {{StorageDescriptor}}).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17981) Create a set of builders for Thrift classes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295764#comment-16295764
 ] 

ASF GitHub Bot commented on HIVE-17981:
---

Github user asfgit closed the pull request at:

https://github.com/apache/hive/pull/274


> Create a set of builders for Thrift classes
> ---
>
> Key: HIVE-17981
> URL: https://issues.apache.org/jira/browse/HIVE-17981
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Attachments: HIVE-17981.2.patch, HIVE-17981.3.patch, 
> HIVE-17981.copy.patch, HIVE-17981.patch
>
>
> Instantiating some of the Thrift classes is painful.  Consider building a 
> {{Table}} object, which requires a {{StorageDescriptor}}, which requires a 
> {{SerDeInfo}} and a list of {{FieldInfo}}.  All that is really necessary for 
> a Table in the most simple case is a name, a database, and some columns.  But 
> currently creating even a simple Table requires 20+ lines of code.  This is 
> particularly painful in tests.  
> I propose to add a set of builders.  These will come with reasonable defaults 
> to minimize the boilerplate code.  They will also include simple methods for 
> common operations (like adding columns, or a parameter) without requiring the 
> user to create all the sub-objects (like {{StorageDescriptor}}).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-14792) AvroSerde reads the remote schema-file at least once per mapper, per table reference.

[Mithun Radhakrishnan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mithun Radhakrishnan updated HIVE-14792:

Status: Patch Available  (was: Reopened)

> AvroSerde reads the remote schema-file at least once per mapper, per table 
> reference.
> -
>
> Key: HIVE-14792
> URL: https://issues.apache.org/jira/browse/HIVE-14792
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.0, 1.2.1
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
>  Labels: TODOC2.2, TODOC2.4
> Fix For: 3.0.0, 2.4.0, 2.2.1
>
> Attachments: HIVE-14792.1.patch, HIVE-14792.3.patch
>
>
> Avro tables that use "external" schema files stored on HDFS can cause 
> excessive calls to {{FileSystem::open()}}, especially for queries that spawn 
> large numbers of mappers.
> This is because of the following code in {{AvroSerDe::initialize()}}:
> {code:title=AvroSerDe.java|borderStyle=solid}
> public void initialize(Configuration configuration, Properties properties) 
> throws SerDeException {
> // ...
> if (hasExternalSchema(properties)
> || columnNameProperty == null || columnNameProperty.isEmpty()
> || columnTypeProperty == null || columnTypeProperty.isEmpty()) {
>   schema = determineSchemaOrReturnErrorSchema(configuration, properties);
> } else {
>   // Get column names and sort order
>   columnNames = Arrays.asList(columnNameProperty.split(","));
>   columnTypes = 
> TypeInfoUtils.getTypeInfosFromTypeString(columnTypeProperty);
>   schema = getSchemaFromCols(properties, columnNames, columnTypes, 
> columnCommentProperty);
>  
> properties.setProperty(AvroSerdeUtils.AvroTableProperties.SCHEMA_LITERAL.getPropName(),
>  schema.toString());
> }
> // ...
> }
> {code}
> For tables using {{avro.schema.url}}, every time the SerDe is initialized 
> (i.e. at least once per mapper), the schema file is read remotely. For 
> queries with thousands of mappers, this leads to a stampede to the handful 
> (3?) datanodes that host the schema-file. In the best case, this causes 
> slowdowns.
> It would be preferable to distribute the Avro-schema to all mappers as part 
> of the job-conf. The alternatives aren't exactly appealing:
> # One can't rely solely on the {{column.list.types}} stored in the Hive 
> metastore. (HIVE-14789).
> # {{avro.schema.literal}} might not always be usable, because of the 
> size-limit on table-parameters. The typical size of the Avro-schema file is 
> between 0.5-3MB, in my limited experience. Bumping the max table-parameter 
> size isn't a great solution.
> If the {{avro.schema.file}} were read during query-planning, and made 
> available as part of table-properties (but not serialized into the 
> metastore), the downstream logic will remain largely intact. I have a patch 
> that does this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (HIVE-14792) AvroSerde reads the remote schema-file at least once per mapper, per table reference.

[Mithun Radhakrishnan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mithun Radhakrishnan reopened HIVE-14792:
-

> AvroSerde reads the remote schema-file at least once per mapper, per table 
> reference.
> -
>
> Key: HIVE-14792
> URL: https://issues.apache.org/jira/browse/HIVE-14792
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
>  Labels: TODOC2.2, TODOC2.4
> Fix For: 3.0.0, 2.4.0, 2.2.1
>
> Attachments: HIVE-14792.1.patch, HIVE-14792.3.patch
>
>
> Avro tables that use "external" schema files stored on HDFS can cause 
> excessive calls to {{FileSystem::open()}}, especially for queries that spawn 
> large numbers of mappers.
> This is because of the following code in {{AvroSerDe::initialize()}}:
> {code:title=AvroSerDe.java|borderStyle=solid}
> public void initialize(Configuration configuration, Properties properties) 
> throws SerDeException {
> // ...
> if (hasExternalSchema(properties)
> || columnNameProperty == null || columnNameProperty.isEmpty()
> || columnTypeProperty == null || columnTypeProperty.isEmpty()) {
>   schema = determineSchemaOrReturnErrorSchema(configuration, properties);
> } else {
>   // Get column names and sort order
>   columnNames = Arrays.asList(columnNameProperty.split(","));
>   columnTypes = 
> TypeInfoUtils.getTypeInfosFromTypeString(columnTypeProperty);
>   schema = getSchemaFromCols(properties, columnNames, columnTypes, 
> columnCommentProperty);
>  
> properties.setProperty(AvroSerdeUtils.AvroTableProperties.SCHEMA_LITERAL.getPropName(),
>  schema.toString());
> }
> // ...
> }
> {code}
> For tables using {{avro.schema.url}}, every time the SerDe is initialized 
> (i.e. at least once per mapper), the schema file is read remotely. For 
> queries with thousands of mappers, this leads to a stampede to the handful 
> (3?) datanodes that host the schema-file. In the best case, this causes 
> slowdowns.
> It would be preferable to distribute the Avro-schema to all mappers as part 
> of the job-conf. The alternatives aren't exactly appealing:
> # One can't rely solely on the {{column.list.types}} stored in the Hive 
> metastore. (HIVE-14789).
> # {{avro.schema.literal}} might not always be usable, because of the 
> size-limit on table-parameters. The typical size of the Avro-schema file is 
> between 0.5-3MB, in my limited experience. Bumping the max table-parameter 
> size isn't a great solution.
> If the {{avro.schema.file}} were read during query-planning, and made 
> available as part of table-properties (but not serialized into the 
> metastore), the downstream logic will remain largely intact. I have a patch 
> that does this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-14792) AvroSerde reads the remote schema-file at least once per mapper, per table reference.

[Mithun Radhakrishnan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mithun Radhakrishnan updated HIVE-14792:

Attachment: HIVE-14792.3.patch

Addendum. (This adds the {{avro.schema.literal}} value to the 
{{TBLPROPERTIES}}, instead of {{SERDEPROPERTIES}}.)

> AvroSerde reads the remote schema-file at least once per mapper, per table 
> reference.
> -
>
> Key: HIVE-14792
> URL: https://issues.apache.org/jira/browse/HIVE-14792
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
>  Labels: TODOC2.2, TODOC2.4
> Fix For: 3.0.0, 2.4.0, 2.2.1
>
> Attachments: HIVE-14792.1.patch, HIVE-14792.3.patch
>
>
> Avro tables that use "external" schema files stored on HDFS can cause 
> excessive calls to {{FileSystem::open()}}, especially for queries that spawn 
> large numbers of mappers.
> This is because of the following code in {{AvroSerDe::initialize()}}:
> {code:title=AvroSerDe.java|borderStyle=solid}
> public void initialize(Configuration configuration, Properties properties) 
> throws SerDeException {
> // ...
> if (hasExternalSchema(properties)
> || columnNameProperty == null || columnNameProperty.isEmpty()
> || columnTypeProperty == null || columnTypeProperty.isEmpty()) {
>   schema = determineSchemaOrReturnErrorSchema(configuration, properties);
> } else {
>   // Get column names and sort order
>   columnNames = Arrays.asList(columnNameProperty.split(","));
>   columnTypes = 
> TypeInfoUtils.getTypeInfosFromTypeString(columnTypeProperty);
>   schema = getSchemaFromCols(properties, columnNames, columnTypes, 
> columnCommentProperty);
>  
> properties.setProperty(AvroSerdeUtils.AvroTableProperties.SCHEMA_LITERAL.getPropName(),
>  schema.toString());
> }
> // ...
> }
> {code}
> For tables using {{avro.schema.url}}, every time the SerDe is initialized 
> (i.e. at least once per mapper), the schema file is read remotely. For 
> queries with thousands of mappers, this leads to a stampede to the handful 
> (3?) datanodes that host the schema-file. In the best case, this causes 
> slowdowns.
> It would be preferable to distribute the Avro-schema to all mappers as part 
> of the job-conf. The alternatives aren't exactly appealing:
> # One can't rely solely on the {{column.list.types}} stored in the Hive 
> metastore. (HIVE-14789).
> # {{avro.schema.literal}} might not always be usable, because of the 
> size-limit on table-parameters. The typical size of the Avro-schema file is 
> between 0.5-3MB, in my limited experience. Bumping the max table-parameter 
> size isn't a great solution.
> If the {{avro.schema.file}} were read during query-planning, and made 
> available as part of table-properties (but not serialized into the 
> metastore), the downstream logic will remain largely intact. I have a patch 
> that does this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18286) java.lang.ClassCastException: org.apache.hadoop.hive.ql.exec.vector.DoubleColumnVector cannot be cast to org.apache.hadoop.hive.ql.exec.vector.LongColumnVector

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18286:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

committed to master
thanks Alan for the review

> java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.exec.vector.DoubleColumnVector cannot be cast to 
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
> ---
>
> Key: HIVE-18286
> URL: https://issues.apache.org/jira/browse/HIVE-18286
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 3.0.0
>
> Attachments: HIVE-18286.01.patch
>
>
> {noformat}
> mvn test -Dtest=TestCliDriver -Dqfile=vector_outer_join3.q
> create table small_alltypesorc1a as 
> select * from alltypesorc 
> where cint is not null and cstring1 is not null 
> order by ctinyint, csmallint, cint, cbigint, cfloat, cdouble, cstring1, 
> cstring2, ctimestamp1, ctimestamp2, cboolean1, cboolean2
>  limit 5;
> {noformat}
> {noformat}
> 2017-12-14T14:33:28,633  WARN [Thread-2754] mapred.LocalJobRunner: 
> job_local113844877_0036
> java.lang.Exception: java.io.IOException: java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.exec.vector.DoubleColumnVector cannot be cast to 
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
> at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492) 
> ~[hadoop-mapreduce-client-common-3.0.0-beta1.jar:?]
> at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552) 
> [hadoop-mapreduce-client-common-3.0.0-beta1.jar:?]
> Caused by: java.io.IOException: java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.exec.vector.DoubleColumnVector cannot be cast to 
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
> at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
>  ~[hive-shims-common-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
>  ~[hive-shims-common-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
>  ~[hadoop-mapreduce-client-core-3.0.0-beta1.jar:?]
> at 
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185) 
> ~[hadoop-mapreduce-client-core-3.0.0-beta1.jar:?]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52) 
> ~[hadoop-mapreduce-client-core-3.0.0-beta1.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459) 
> ~[hadoop-mapreduce-client-core-3.0.0-beta1.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) 
> ~[hadoop-mapreduce-client-core-3.0.0-beta1.jar:?]
> at 
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
>  ~[hadoop-mapreduce-client-common-3.0.0-beta1.jar:?]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[?:1.8.0_25]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> ~[?:1.8.0_25]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  ~[?:1.8.0_25]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  ~[?:1.8.0_25]
> at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_25]
> Caused by: java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.exec.vector.DoubleColumnVector cannot be cast to 
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
> at 
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.findRecordsWithInvalidTransactionIds(VectorizedOrcAcidRowBatchReader.java:531)
>  ~[hive-exec-3.0.0\
> -SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> 

[jira] [Updated] (HIVE-18294) add switch to make acid table the default

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18294:
--
Status: Patch Available  (was: Open)

> add switch to make acid table the default
> -
>
> Key: HIVE-18294
> URL: https://issues.apache.org/jira/browse/HIVE-18294
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18294.01.patch, HIVE-18294.03.patch
>
>
> it would be convenient for testing to have a switch that enables the behavior 
> where all suitable table tables (currently ORC + not sorted) are 
> automatically reacted with transactional=true.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18294) add switch to make acid table the default

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18294:
--
Attachment: HIVE-18294.03.patch

> add switch to make acid table the default
> -
>
> Key: HIVE-18294
> URL: https://issues.apache.org/jira/browse/HIVE-18294
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18294.01.patch, HIVE-18294.03.patch
>
>
> it would be convenient for testing to have a switch that enables the behavior 
> where all suitable table tables (currently ORC + not sorted) are 
> automatically reacted with transactional=true.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-14792) AvroSerde reads the remote schema-file at least once per mapper, per table reference.

[Mithun Radhakrishnan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295744#comment-16295744
 ] 

Mithun Radhakrishnan commented on HIVE-14792:
-

[~aihuaxu], sorry for the bother, but it looks like my fix here is not 
complete. 
On enabling {{hive.optimize.update.table.properties.from.serde}}, one sees 
errors when prefetching Avro schemas, such as the following:
{noformat}
Caused by: java.lang.RuntimeException: Map operator initialization failed
at 
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:137) 
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) ~[?:?]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_144]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
at 
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:110) 
~[hadoop-common-3.0.0-beta1.jar:?]
at 
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:79) 
~[hadoop-common-3.0.0-beta1.jar:?]
at 
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137) 
~[hadoop-common-3.0.0-beta1.jar:?]
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) 
~[hadoop-mapreduce-client-core-3.0.0-beta1.jar:?]
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) ~[?:?]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_144]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
at 
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:110) 
~[hadoop-common-3.0.0-beta1.jar:?]
at 
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:79) 
~[hadoop-common-3.0.0-beta1.jar:?]
at 
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137) 
~[hadoop-common-3.0.0-beta1.jar:?]
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:456) 
~[hadoop-mapreduce-client-core-3.0.0-beta1.jar:?]
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) 
~[hadoop-mapreduce-client-core-3.0.0-beta1.jar:?]
at 
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
 ~[hadoop-mapreduce-client-common-3.0.0-beta1.jar:?]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[?:1.8.0_144]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
~[?:1.8.0_144]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
~[?:1.8.0_144]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
~[?:1.8.0_144]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_144]
Caused by: java.lang.RuntimeException: cannot find field number from 
[0:error_error_error_error_error_error_error, 1:cannot_determine_schema, 
2:check, 3:schema, 4:url, 5:and, 6:literal]
at 
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:530)
 ~[hive-serde-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
 ~[hive-serde-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:56)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:1096) 
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:1122)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:75)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:367) 
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:557) 
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:509) 
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:377) 
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.exec.MapOperator.initializeMapOperator(MapOperator.java:504)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:116) 
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
{noformat}

The reason we're not seeing this failure in regular builds is that 

[jira] [Commented] (HIVE-17981) Create a set of builders for Thrift classes

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295742#comment-16295742
 ] 

Hive QA commented on HIVE-17981:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
15s{color} | {color:red} standalone-metastore: The patch generated 50 new + 0 
unchanged - 0 fixed = 50 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 10s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 8259022 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8304/yetus/diff-checkstyle-standalone-metastore.txt
 |
| modules | C: standalone-metastore U: standalone-metastore |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8304/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Create a set of builders for Thrift classes
> ---
>
> Key: HIVE-17981
> URL: https://issues.apache.org/jira/browse/HIVE-17981
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Attachments: HIVE-17981.2.patch, HIVE-17981.3.patch, 
> HIVE-17981.copy.patch, HIVE-17981.patch
>
>
> Instantiating some of the Thrift classes is painful.  Consider building a 
> {{Table}} object, which requires a {{StorageDescriptor}}, which requires a 
> {{SerDeInfo}} and a list of {{FieldInfo}}.  All that is really necessary for 
> a Table in the most simple case is a name, a database, and some columns.  But 
> currently creating even a simple Table requires 20+ lines of code.  This is 
> particularly painful in tests.  
> I propose to add a set of builders.  These will come with reasonable defaults 
> to minimize the boilerplate code.  They will also include simple methods for 
> common operations (like adding columns, or a parameter) without requiring the 
> user to create all the sub-objects (like {{StorageDescriptor}}).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18295) Add ability to ignore invalid values in JSON SerDe

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295717#comment-16295717
 ] 

ASF GitHub Bot commented on HIVE-18295:
---

GitHub user mattk42 opened a pull request:

https://github.com/apache/hive/pull/278

HIVE-18295 - Add ability to ignore invalid values in JSON SerDe 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattk42/hive master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/278.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #278


commit 7d1a1aa42c4661ba9373c21acacc059bf894d1ed
Author: Matt Knox 
Date:   2017-12-15T19:21:25Z

Add ability to ignore values that cannot be parsed into the final type.




> Add ability to ignore invalid values in JSON SerDe
> --
>
> Key: HIVE-18295
> URL: https://issues.apache.org/jira/browse/HIVE-18295
> Project: Hive
>  Issue Type: Improvement
>  Components: HCatalog
>Reporter: Matthew Knox
>Assignee: Matthew Knox
>  Labels: pull-request-available
>
> It would be nice to be able to configure the JSON SerDe to ignore invalid 
> values while parsing JSON. 
> In our case our raw JSON data is ingested from multiple sources, some of 
> which unreliably sanitize the data. Our current practice is to cleanse the 
> data after ingestion, but that can lead to other issues as well. Having the 
> ability to simply default to NULL if a value can not be parsed would be 
> immensely helpful to us.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18283) Better error message and error code for HoS exceptions

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295718#comment-16295718
 ] 

Hive QA commented on HIVE-18283:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12902702/HIVE-18283.1.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 11532 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions_move_only]
 (batchId=249)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] 
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
 (batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=160)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
 (batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_10]
 (batchId=138)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketsortoptimize_insert_7]
 (batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi] 
(batchId=113)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut 
(batchId=209)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=226)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8303/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8303/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8303/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12902702 - PreCommit-HIVE-Build

> Better error message and error code for HoS exceptions
> --
>
> Key: HIVE-18283
> URL: https://issues.apache.org/jira/browse/HIVE-18283
> Project: Hive
>  Issue Type: Improvement
>  Components: Spark
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-18283.0.patch, HIVE-18283.1.patch, 
> HIVE-18283.2.patch
>
>
> Right now HoS only use a few error codes. For the majority of the errors, 
> user will see an error code 1 followed by a lengthy stacktrace. This is not 
> ideal since:
> 1. It is often hard to find the root cause - sometimes it is hidden deeply 
> inside the stacktrace.
> 2. After identifying the root cause, it is not easy to find a fix. Often user 
> have to copy & paste the error message and google them. 
> 3. It is not clear whether the error is transient or not, depending on which 
> user may want to retry the query. 
> To improve the above, this JIRA propose to assign error code & canonical 
> error messages for different HoS errors. We can take advantage of the 
> existing {{ErrorMsg}} class.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18295) Add ability to ignore invalid values in JSON SerDe

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-18295:
--
Labels: pull-request-available  (was: )

> Add ability to ignore invalid values in JSON SerDe
> --
>
> Key: HIVE-18295
> URL: https://issues.apache.org/jira/browse/HIVE-18295
> Project: Hive
>  Issue Type: Improvement
>  Components: HCatalog
>Reporter: Matthew Knox
>Assignee: Matthew Knox
>  Labels: pull-request-available
>
> It would be nice to be able to configure the JSON SerDe to ignore invalid 
> values while parsing JSON. 
> In our case our raw JSON data is ingested from multiple sources, some of 
> which unreliably sanitize the data. Our current practice is to cleanse the 
> data after ingestion, but that can lead to other issues as well. Having the 
> ability to simply default to NULL if a value can not be parsed would be 
> immensely helpful to us.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-18295) Add ability to ignore invalid values in JSON SerDe

[Matthew Knox (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthew Knox reassigned HIVE-18295:
---


> Add ability to ignore invalid values in JSON SerDe
> --
>
> Key: HIVE-18295
> URL: https://issues.apache.org/jira/browse/HIVE-18295
> Project: Hive
>  Issue Type: Improvement
>  Components: HCatalog
>Reporter: Matthew Knox
>Assignee: Matthew Knox
>
> It would be nice to be able to configure the JSON SerDe to ignore invalid 
> values while parsing JSON. 
> In our case our raw JSON data is ingested from multiple sources, some of 
> which unreliably sanitize the data. Our current practice is to cleanse the 
> data after ingestion, but that can lead to other issues as well. Having the 
> ability to simply default to NULL if a value can not be parsed would be 
> immensely helpful to us.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18294) add switch to make acid table the default

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18294:
--
Attachment: HIVE-18294.01.patch

> add switch to make acid table the default
> -
>
> Key: HIVE-18294
> URL: https://issues.apache.org/jira/browse/HIVE-18294
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18294.01.patch
>
>
> it would be convenient for testing to have a switch that enables the behavior 
> where all suitable table tables (currently ORC + not sorted) are 
> automatically reacted with transactional=true.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18283) Better error message and error code for HoS exceptions

[Chao Sun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao Sun updated HIVE-18283:

Attachment: HIVE-18283.2.patch

Patch v2 to fix check style issues.

> Better error message and error code for HoS exceptions
> --
>
> Key: HIVE-18283
> URL: https://issues.apache.org/jira/browse/HIVE-18283
> Project: Hive
>  Issue Type: Improvement
>  Components: Spark
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-18283.0.patch, HIVE-18283.1.patch, 
> HIVE-18283.2.patch
>
>
> Right now HoS only use a few error codes. For the majority of the errors, 
> user will see an error code 1 followed by a lengthy stacktrace. This is not 
> ideal since:
> 1. It is often hard to find the root cause - sometimes it is hidden deeply 
> inside the stacktrace.
> 2. After identifying the root cause, it is not easy to find a fix. Often user 
> have to copy & paste the error message and google them. 
> 3. It is not clear whether the error is transient or not, depending on which 
> user may want to retry the query. 
> To improve the above, this JIRA propose to assign error code & canonical 
> error messages for different HoS errors. We can take advantage of the 
> existing {{ErrorMsg}} class.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18052) Run p-tests on mm tables

[Steve Yeom (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295693#comment-16295693
 ] 

Steve Yeom commented on HIVE-18052:
---

The results show test failures by TestMiniLlapLocalCliDriver is decreased from 
290 to 217. 
In days, the number may decrease by another 120 or so.

> Run p-tests on mm tables
> 
>
> Key: HIVE-18052
> URL: https://issues.apache.org/jira/browse/HIVE-18052
> Project: Hive
>  Issue Type: Task
>Reporter: Steve Yeom
>Assignee: Steve Yeom
> Attachments: HIVE-18052.1.patch, HIVE-18052.10.patch, 
> HIVE-18052.2.patch, HIVE-18052.3.patch, HIVE-18052.4.patch, 
> HIVE-18052.5.patch, HIVE-18052.6.patch, HIVE-18052.7.patch, 
> HIVE-18052.8.patch, HIVE-18052.9.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18283) Better error message and error code for HoS exceptions

[Chao Sun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao Sun updated HIVE-18283:

Attachment: HIVE-18283.1.patch

> Better error message and error code for HoS exceptions
> --
>
> Key: HIVE-18283
> URL: https://issues.apache.org/jira/browse/HIVE-18283
> Project: Hive
>  Issue Type: Improvement
>  Components: Spark
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-18283.0.patch, HIVE-18283.1.patch
>
>
> Right now HoS only use a few error codes. For the majority of the errors, 
> user will see an error code 1 followed by a lengthy stacktrace. This is not 
> ideal since:
> 1. It is often hard to find the root cause - sometimes it is hidden deeply 
> inside the stacktrace.
> 2. After identifying the root cause, it is not easy to find a fix. Often user 
> have to copy & paste the error message and google them. 
> 3. It is not clear whether the error is transient or not, depending on which 
> user may want to retry the query. 
> To improve the above, this JIRA propose to assign error code & canonical 
> error messages for different HoS errors. We can take advantage of the 
> existing {{ErrorMsg}} class.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18283) Better error message and error code for HoS exceptions

[Chao Sun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao Sun updated HIVE-18283:

Attachment: (was: HIVE-18283.1.patch)

> Better error message and error code for HoS exceptions
> --
>
> Key: HIVE-18283
> URL: https://issues.apache.org/jira/browse/HIVE-18283
> Project: Hive
>  Issue Type: Improvement
>  Components: Spark
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-18283.0.patch
>
>
> Right now HoS only use a few error codes. For the majority of the errors, 
> user will see an error code 1 followed by a lengthy stacktrace. This is not 
> ideal since:
> 1. It is often hard to find the root cause - sometimes it is hidden deeply 
> inside the stacktrace.
> 2. After identifying the root cause, it is not easy to find a fix. Often user 
> have to copy & paste the error message and google them. 
> 3. It is not clear whether the error is transient or not, depending on which 
> user may want to retry the query. 
> To improve the above, this JIRA propose to assign error code & canonical 
> error messages for different HoS errors. We can take advantage of the 
> existing {{ErrorMsg}} class.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (HIVE-17829) ArrayIndexOutOfBoundsException - HBASE-backed tables with Avro schema in Hive2

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295677#comment-16295677
 ] 

Thejas M Nair edited comment on HIVE-17829 at 12/18/17 9:21 PM:


The change looks good. Can you also add a UT ? (a .q one similar to other hbase 
tests in hive?)
For users without a fix, can they workaround it by adding comments ?



was (Author: thejas):
The change looks good. Can you also add a UT ?
For users without a fix, can they workaround it by adding comments ?


> ArrayIndexOutOfBoundsException - HBASE-backed tables with Avro schema in Hive2
> --
>
> Key: HIVE-17829
> URL: https://issues.apache.org/jira/browse/HIVE-17829
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Affects Versions: 2.1.0
>Reporter: Chiran Ravani
>Assignee: anishek
>Priority: Critical
> Attachments: HIVE-17829.0.patch
>
>
> Stack
> {code}
> 2017-10-09T09:39:54,804 ERROR [HiveServer2-Background-Pool: Thread-95]: 
> metadata.Table (Table.java:getColsInternal(642)) - Unable to get field from 
> serde: org.apache.hadoop.hive.hbase.HBaseSerDe
> java.lang.ArrayIndexOutOfBoundsException: 1
> at java.util.Arrays$ArrayList.get(Arrays.java:3841) ~[?:1.8.0_77]
> at 
> org.apache.hadoop.hive.serde2.BaseStructObjectInspector.init(BaseStructObjectInspector.java:104)
>  ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.init(LazySimpleStructObjectInspector.java:97)
>  ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.(LazySimpleStructObjectInspector.java:77)
>  ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyObjectInspectorFactory.getLazySimpleStructObjectInspector(LazyObjectInspectorFactory.java:115)
>  ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.hbase.HBaseLazyObjectFactory.createLazyHBaseStructInspector(HBaseLazyObjectFactory.java:79)
>  ~[hive-hbase-handler-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.hbase.HBaseSerDe.initialize(HBaseSerDe.java:127) 
> ~[hive-hbase-handler-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54) 
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:531) 
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:424)
>  ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:411)
>  ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:279)
>  ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:261) 
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:639) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:622) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:833) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:869) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4228) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:347) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1905) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1607) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1354) 
> 

[jira] [Commented] (HIVE-17829) ArrayIndexOutOfBoundsException - HBASE-backed tables with Avro schema in Hive2

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295677#comment-16295677
 ] 

Thejas M Nair commented on HIVE-17829:
--

The change looks good. Can you also add a UT ?
For users without a fix, can they workaround it by adding comments ?


> ArrayIndexOutOfBoundsException - HBASE-backed tables with Avro schema in Hive2
> --
>
> Key: HIVE-17829
> URL: https://issues.apache.org/jira/browse/HIVE-17829
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Affects Versions: 2.1.0
>Reporter: Chiran Ravani
>Assignee: anishek
>Priority: Critical
> Attachments: HIVE-17829.0.patch
>
>
> Stack
> {code}
> 2017-10-09T09:39:54,804 ERROR [HiveServer2-Background-Pool: Thread-95]: 
> metadata.Table (Table.java:getColsInternal(642)) - Unable to get field from 
> serde: org.apache.hadoop.hive.hbase.HBaseSerDe
> java.lang.ArrayIndexOutOfBoundsException: 1
> at java.util.Arrays$ArrayList.get(Arrays.java:3841) ~[?:1.8.0_77]
> at 
> org.apache.hadoop.hive.serde2.BaseStructObjectInspector.init(BaseStructObjectInspector.java:104)
>  ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.init(LazySimpleStructObjectInspector.java:97)
>  ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.(LazySimpleStructObjectInspector.java:77)
>  ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyObjectInspectorFactory.getLazySimpleStructObjectInspector(LazyObjectInspectorFactory.java:115)
>  ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.hbase.HBaseLazyObjectFactory.createLazyHBaseStructInspector(HBaseLazyObjectFactory.java:79)
>  ~[hive-hbase-handler-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.hbase.HBaseSerDe.initialize(HBaseSerDe.java:127) 
> ~[hive-hbase-handler-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54) 
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:531) 
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:424)
>  ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:411)
>  ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:279)
>  ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:261) 
> ~[hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:639) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:622) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:833) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:869) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4228) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:347) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1905) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1607) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1354) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1123) 
> [hive-exec-2.1.0.2.6.2.0-205.jar:2.1.0.2.6.2.0-205]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1116) 
> 

[jira] [Updated] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

[Misha Dmitriev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Misha Dmitriev updated HIVE-17684:
--
Attachment: HIVE-17684.01.patch

> HoS memory issues with MapJoinMemoryExhaustionHandler
> -
>
> Key: HIVE-17684
> URL: https://issues.apache.org/jira/browse/HIVE-17684
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Misha Dmitriev
> Attachments: HIVE-17684.01.patch
>
>
> We have seen a number of memory issues due the {{HashSinkOperator}} use of 
> the {{MapJoinMemoryExhaustionHandler}}. This handler is meant to detect 
> scenarios where the small table is taking too much space in memory, in which 
> case a {{MapJoinMemoryExhaustionError}} is thrown.
> The configs to control this logic are:
> {{hive.mapjoin.localtask.max.memory.usage}} (default 0.90)
> {{hive.mapjoin.followby.gby.localtask.max.memory.usage}} (default 0.55)
> The handler works by using the {{MemoryMXBean}} and uses the following logic 
> to estimate how much memory the {{HashMap}} is consuming: 
> {{MemoryMXBean#getHeapMemoryUsage().getUsed() / 
> MemoryMXBean#getHeapMemoryUsage().getMax()}}
> The issue is that {{MemoryMXBean#getHeapMemoryUsage().getUsed()}} can be 
> inaccurate. The value returned by this method returns all reachable and 
> unreachable memory on the heap, so there may be a bunch of garbage data, and 
> the JVM just hasn't taken the time to reclaim it all. This can lead to 
> intermittent failures of this check even though a simple GC would have 
> reclaimed enough space for the process to continue working.
> We should re-think the usage of {{MapJoinMemoryExhaustionHandler}} for HoS. 
> In Hive-on-MR this probably made sense to use because every Hive task was run 
> in a dedicated container, so a Hive Task could assume it created most of the 
> data on the heap. However, in Hive-on-Spark there can be multiple Hive Tasks 
> running in a single executor, each doing different things.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

[Misha Dmitriev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Misha Dmitriev updated HIVE-17684:
--
Status: Patch Available  (was: In Progress)

> HoS memory issues with MapJoinMemoryExhaustionHandler
> -
>
> Key: HIVE-17684
> URL: https://issues.apache.org/jira/browse/HIVE-17684
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Misha Dmitriev
> Attachments: HIVE-17684.01.patch
>
>
> We have seen a number of memory issues due the {{HashSinkOperator}} use of 
> the {{MapJoinMemoryExhaustionHandler}}. This handler is meant to detect 
> scenarios where the small table is taking too much space in memory, in which 
> case a {{MapJoinMemoryExhaustionError}} is thrown.
> The configs to control this logic are:
> {{hive.mapjoin.localtask.max.memory.usage}} (default 0.90)
> {{hive.mapjoin.followby.gby.localtask.max.memory.usage}} (default 0.55)
> The handler works by using the {{MemoryMXBean}} and uses the following logic 
> to estimate how much memory the {{HashMap}} is consuming: 
> {{MemoryMXBean#getHeapMemoryUsage().getUsed() / 
> MemoryMXBean#getHeapMemoryUsage().getMax()}}
> The issue is that {{MemoryMXBean#getHeapMemoryUsage().getUsed()}} can be 
> inaccurate. The value returned by this method returns all reachable and 
> unreachable memory on the heap, so there may be a bunch of garbage data, and 
> the JVM just hasn't taken the time to reclaim it all. This can lead to 
> intermittent failures of this check even though a simple GC would have 
> reclaimed enough space for the process to continue working.
> We should re-think the usage of {{MapJoinMemoryExhaustionHandler}} for HoS. 
> In Hive-on-MR this probably made sense to use because every Hive task was run 
> in a dedicated container, so a Hive Task could assume it created most of the 
> data on the heap. However, in Hive-on-Spark there can be multiple Hive Tasks 
> running in a single executor, each doing different things.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

[Misha Dmitriev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Misha Dmitriev reassigned HIVE-17684:
-

Assignee: Misha Dmitriev  (was: Sahil Takiar)

> HoS memory issues with MapJoinMemoryExhaustionHandler
> -
>
> Key: HIVE-17684
> URL: https://issues.apache.org/jira/browse/HIVE-17684
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Misha Dmitriev
>
> We have seen a number of memory issues due the {{HashSinkOperator}} use of 
> the {{MapJoinMemoryExhaustionHandler}}. This handler is meant to detect 
> scenarios where the small table is taking too much space in memory, in which 
> case a {{MapJoinMemoryExhaustionError}} is thrown.
> The configs to control this logic are:
> {{hive.mapjoin.localtask.max.memory.usage}} (default 0.90)
> {{hive.mapjoin.followby.gby.localtask.max.memory.usage}} (default 0.55)
> The handler works by using the {{MemoryMXBean}} and uses the following logic 
> to estimate how much memory the {{HashMap}} is consuming: 
> {{MemoryMXBean#getHeapMemoryUsage().getUsed() / 
> MemoryMXBean#getHeapMemoryUsage().getMax()}}
> The issue is that {{MemoryMXBean#getHeapMemoryUsage().getUsed()}} can be 
> inaccurate. The value returned by this method returns all reachable and 
> unreachable memory on the heap, so there may be a bunch of garbage data, and 
> the JVM just hasn't taken the time to reclaim it all. This can lead to 
> intermittent failures of this check even though a simple GC would have 
> reclaimed enough space for the process to continue working.
> We should re-think the usage of {{MapJoinMemoryExhaustionHandler}} for HoS. 
> In Hive-on-MR this probably made sense to use because every Hive task was run 
> in a dedicated container, so a Hive Task could assume it created most of the 
> data on the heap. However, in Hive-on-Spark there can be multiple Hive Tasks 
> running in a single executor, each doing different things.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Work started] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler

[Misha Dmitriev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-17684 started by Misha Dmitriev.
-
> HoS memory issues with MapJoinMemoryExhaustionHandler
> -
>
> Key: HIVE-17684
> URL: https://issues.apache.org/jira/browse/HIVE-17684
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Misha Dmitriev
>
> We have seen a number of memory issues due the {{HashSinkOperator}} use of 
> the {{MapJoinMemoryExhaustionHandler}}. This handler is meant to detect 
> scenarios where the small table is taking too much space in memory, in which 
> case a {{MapJoinMemoryExhaustionError}} is thrown.
> The configs to control this logic are:
> {{hive.mapjoin.localtask.max.memory.usage}} (default 0.90)
> {{hive.mapjoin.followby.gby.localtask.max.memory.usage}} (default 0.55)
> The handler works by using the {{MemoryMXBean}} and uses the following logic 
> to estimate how much memory the {{HashMap}} is consuming: 
> {{MemoryMXBean#getHeapMemoryUsage().getUsed() / 
> MemoryMXBean#getHeapMemoryUsage().getMax()}}
> The issue is that {{MemoryMXBean#getHeapMemoryUsage().getUsed()}} can be 
> inaccurate. The value returned by this method returns all reachable and 
> unreachable memory on the heap, so there may be a bunch of garbage data, and 
> the JVM just hasn't taken the time to reclaim it all. This can lead to 
> intermittent failures of this check even though a simple GC would have 
> reclaimed enough space for the process to continue working.
> We should re-think the usage of {{MapJoinMemoryExhaustionHandler}} for HoS. 
> In Hive-on-MR this probably made sense to use because every Hive task was run 
> in a dedicated container, so a Hive Task could assume it created most of the 
> data on the heap. However, in Hive-on-Spark there can be multiple Hive Tasks 
> running in a single executor, each doing different things.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18286) java.lang.ClassCastException: org.apache.hadoop.hive.ql.exec.vector.DoubleColumnVector cannot be cast to org.apache.hadoop.hive.ql.exec.vector.LongColumnVector

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295668#comment-16295668
 ] 

Alan Gates commented on HIVE-18286:
---

+1

> java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.exec.vector.DoubleColumnVector cannot be cast to 
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
> ---
>
> Key: HIVE-18286
> URL: https://issues.apache.org/jira/browse/HIVE-18286
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 3.0.0
>
> Attachments: HIVE-18286.01.patch
>
>
> {noformat}
> mvn test -Dtest=TestCliDriver -Dqfile=vector_outer_join3.q
> create table small_alltypesorc1a as 
> select * from alltypesorc 
> where cint is not null and cstring1 is not null 
> order by ctinyint, csmallint, cint, cbigint, cfloat, cdouble, cstring1, 
> cstring2, ctimestamp1, ctimestamp2, cboolean1, cboolean2
>  limit 5;
> {noformat}
> {noformat}
> 2017-12-14T14:33:28,633  WARN [Thread-2754] mapred.LocalJobRunner: 
> job_local113844877_0036
> java.lang.Exception: java.io.IOException: java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.exec.vector.DoubleColumnVector cannot be cast to 
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
> at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492) 
> ~[hadoop-mapreduce-client-common-3.0.0-beta1.jar:?]
> at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552) 
> [hadoop-mapreduce-client-common-3.0.0-beta1.jar:?]
> Caused by: java.io.IOException: java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.exec.vector.DoubleColumnVector cannot be cast to 
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
> at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
>  ~[hive-shims-common-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
>  ~[hive-shims-common-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
>  ~[hadoop-mapreduce-client-core-3.0.0-beta1.jar:?]
> at 
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185) 
> ~[hadoop-mapreduce-client-core-3.0.0-beta1.jar:?]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52) 
> ~[hadoop-mapreduce-client-core-3.0.0-beta1.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459) 
> ~[hadoop-mapreduce-client-core-3.0.0-beta1.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) 
> ~[hadoop-mapreduce-client-core-3.0.0-beta1.jar:?]
> at 
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
>  ~[hadoop-mapreduce-client-common-3.0.0-beta1.jar:?]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[?:1.8.0_25]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> ~[?:1.8.0_25]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  ~[?:1.8.0_25]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  ~[?:1.8.0_25]
> at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_25]
> Caused by: java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.exec.vector.DoubleColumnVector cannot be cast to 
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
> at 
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.findRecordsWithInvalidTransactionIds(VectorizedOrcAcidRowBatchReader.java:531)
>  ~[hive-exec-3.0.0\
> -SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.next(VectorizedOrcAcidRowBatchReader.java:462)
>  

[jira] [Commented] (HIVE-18124) clean up isAcidTable() API vs isInsertOnlyTable()

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295662#comment-16295662
 ] 

Alan Gates commented on HIVE-18124:
---

Checking that I have this straight:  acid = original acid, support 
update/delete/merge, isInertOnly = micromanaged (aka 1/4 acid), transactional = 
acid || isInsertOnly.  Is that right?

Assuming I have that right, the changes seem fine.  We should just write that 
down somewhere, preferably in the AcidUtils file, so future maintainers can 
keep up with what's going on.  If it's already written somewhere else just a 
pointer to it in AcidUtils is fine.

With that caveat, +1.

>  clean up isAcidTable() API vs isInsertOnlyTable()
> --
>
> Key: HIVE-18124
> URL: https://issues.apache.org/jira/browse/HIVE-18124
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18124.01.patch, HIVE-18124.02.patch, 
> HIVE-18124.03.patch, HIVE-18124.04.patch, HIVE-18124.05.patch
>
>
> With the addition of MM tables (_AcidUtils.isInsertOnlyTable(table)_) the 
> methods in AcidUtils and dependent places are very muddled.  There are now a 
> number of places where we have something like _isAcidTable = 
> AcidUtils.isFullAcidTable(table)_  and a later getter
> _boolean isAcidTable() \{ return isAcidTable;\}_
> Need to clean it up so that there is a isTransactional(Table) that checks 
> transactional=true setting and isAcid(Table) to mean full ACID and 
> isInsertOnly(Table) to mean MM tables.
> This would accurately describe the semantics of the tables.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18218) SMB Join : Handle buckets with no splits.

[Deepak Jaiswal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepak Jaiswal updated HIVE-18218:
--
Status: Patch Available  (was: In Progress)

> SMB Join : Handle buckets with no splits.
> -
>
> Key: HIVE-18218
> URL: https://issues.apache.org/jira/browse/HIVE-18218
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Attachments: HIVE-18218.1.patch
>
>
> While working on HIVE-18208, it was found that with SMB, the results are 
> incorrect. This most likely is a product issue.
> auto_sortmerge_join_16 fails with wrong results due to this.
> cc [~hagleitn]
> The current logic in CustomPartitionVertex assumes that there is a split for 
> each bucket whereas in  Tez, we can have no splits for empty buckets.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18218) SMB Join : Handle buckets with no splits.

[Deepak Jaiswal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepak Jaiswal updated HIVE-18218:
--
Attachment: HIVE-18218.1.patch

> SMB Join : Handle buckets with no splits.
> -
>
> Key: HIVE-18218
> URL: https://issues.apache.org/jira/browse/HIVE-18218
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Attachments: HIVE-18218.1.patch
>
>
> While working on HIVE-18208, it was found that with SMB, the results are 
> incorrect. This most likely is a product issue.
> auto_sortmerge_join_16 fails with wrong results due to this.
> cc [~hagleitn]
> The current logic in CustomPartitionVertex assumes that there is a split for 
> each bucket whereas in  Tez, we can have no splits for empty buckets.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (HIVE-18218) SMB Join : Handle buckets with no splits.

[Deepak Jaiswal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295645#comment-16295645
 ] 

Deepak Jaiswal edited comment on HIVE-18218 at 12/18/17 8:55 PM:
-

Initial patch for review.
Added provisions to fall back to reduceside join if small table has more 
buckets than big table.


was (Author: djaiswal):
Initial patch for review.


> SMB Join : Handle buckets with no splits.
> -
>
> Key: HIVE-18218
> URL: https://issues.apache.org/jira/browse/HIVE-18218
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>
> While working on HIVE-18208, it was found that with SMB, the results are 
> incorrect. This most likely is a product issue.
> auto_sortmerge_join_16 fails with wrong results due to this.
> cc [~hagleitn]
> The current logic in CustomPartitionVertex assumes that there is a split for 
> each bucket whereas in  Tez, we can have no splits for empty buckets.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18218) SMB Join : Handle buckets with no splits.

[Deepak Jaiswal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepak Jaiswal updated HIVE-18218:
--
Attachment: (was: HIVE-18218.1.patch)

> SMB Join : Handle buckets with no splits.
> -
>
> Key: HIVE-18218
> URL: https://issues.apache.org/jira/browse/HIVE-18218
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>
> While working on HIVE-18208, it was found that with SMB, the results are 
> incorrect. This most likely is a product issue.
> auto_sortmerge_join_16 fails with wrong results due to this.
> cc [~hagleitn]
> The current logic in CustomPartitionVertex assumes that there is a split for 
> each bucket whereas in  Tez, we can have no splits for empty buckets.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18218) SMB Join : Handle buckets with no splits.

[Deepak Jaiswal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepak Jaiswal updated HIVE-18218:
--
Attachment: HIVE-18218.1.patch

Initial patch for review.


> SMB Join : Handle buckets with no splits.
> -
>
> Key: HIVE-18218
> URL: https://issues.apache.org/jira/browse/HIVE-18218
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Attachments: HIVE-18218.1.patch
>
>
> While working on HIVE-18208, it was found that with SMB, the results are 
> incorrect. This most likely is a product issue.
> auto_sortmerge_join_16 fails with wrong results due to this.
> cc [~hagleitn]
> The current logic in CustomPartitionVertex assumes that there is a split for 
> each bucket whereas in  Tez, we can have no splits for empty buckets.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Work started] (HIVE-18218) SMB Join : Handle buckets with no splits.

[Deepak Jaiswal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-18218 started by Deepak Jaiswal.
-
> SMB Join : Handle buckets with no splits.
> -
>
> Key: HIVE-18218
> URL: https://issues.apache.org/jira/browse/HIVE-18218
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>
> While working on HIVE-18208, it was found that with SMB, the results are 
> incorrect. This most likely is a product issue.
> auto_sortmerge_join_16 fails with wrong results due to this.
> cc [~hagleitn]
> The current logic in CustomPartitionVertex assumes that there is a split for 
> each bucket whereas in  Tez, we can have no splits for empty buckets.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18283) Better error message and error code for HoS exceptions

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295642#comment-16295642
 ] 

Hive QA commented on HIVE-18283:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
31s{color} | {color:red} ql: The patch generated 18 new + 72 unchanged - 2 
fixed = 90 total (was 74) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
23s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 45s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 8259022 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8303/yetus/diff-checkstyle-ql.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8303/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Better error message and error code for HoS exceptions
> --
>
> Key: HIVE-18283
> URL: https://issues.apache.org/jira/browse/HIVE-18283
> Project: Hive
>  Issue Type: Improvement
>  Components: Spark
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-18283.0.patch, HIVE-18283.1.patch
>
>
> Right now HoS only use a few error codes. For the majority of the errors, 
> user will see an error code 1 followed by a lengthy stacktrace. This is not 
> ideal since:
> 1. It is often hard to find the root cause - sometimes it is hidden deeply 
> inside the stacktrace.
> 2. After identifying the root cause, it is not easy to find a fix. Often user 
> have to copy & paste the error message and google them. 
> 3. It is not clear whether the error is transient or not, depending on which 
> user may want to retry the query. 
> To improve the above, this JIRA propose to assign error code & canonical 
> error messages for different HoS errors. We can take advantage of the 
> existing {{ErrorMsg}} class.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17981) Create a set of builders for Thrift classes

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-17981:
--
Attachment: HIVE-17981.3.patch

> Create a set of builders for Thrift classes
> ---
>
> Key: HIVE-17981
> URL: https://issues.apache.org/jira/browse/HIVE-17981
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Attachments: HIVE-17981.2.patch, HIVE-17981.3.patch, 
> HIVE-17981.copy.patch, HIVE-17981.patch
>
>
> Instantiating some of the Thrift classes is painful.  Consider building a 
> {{Table}} object, which requires a {{StorageDescriptor}}, which requires a 
> {{SerDeInfo}} and a list of {{FieldInfo}}.  All that is really necessary for 
> a Table in the most simple case is a name, a database, and some columns.  But 
> currently creating even a simple Table requires 20+ lines of code.  This is 
> particularly painful in tests.  
> I propose to add a set of builders.  These will come with reasonable defaults 
> to minimize the boilerplate code.  They will also include simple methods for 
> common operations (like adding columns, or a parameter) without requiring the 
> user to create all the sub-objects (like {{StorageDescriptor}}).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-18294) add switch to make acid table the default

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman reassigned HIVE-18294:
-


> add switch to make acid table the default
> -
>
> Key: HIVE-18294
> URL: https://issues.apache.org/jira/browse/HIVE-18294
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> it would be convenient for testing to have a switch that enables the behavior 
> where all suitable table tables (currently ORC + not sorted) are 
> automatically reacted with transactional=true.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18052) Run p-tests on mm tables

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295631#comment-16295631
 ] 

Hive QA commented on HIVE-18052:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  9m 
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  4m 
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  9m 
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m 
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
45s{color} | {color:red} ql: The patch generated 6 new + 1638 unchanged - 2 
fixed = 1644 total (was 1640) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m 
49s{color} | {color:red} root: The patch generated 6 new + 2757 unchanged - 2 
fixed = 2763 total (was 2759) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  4m 
59s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m  
1s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m  
2s{color} | {color:red} hive-minikdc in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m  
1s{color} | {color:red} hive-unit in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:blue}0{color} | {color:blue} asflicense {color} | {color:blue}  0m  
4s{color} | {color:blue} ASF License check generated no output? {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 22s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  
xml  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 8259022 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8301/yetus/diff-checkstyle-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8301/yetus/diff-checkstyle-root.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8301/yetus/whitespace-eol.txt 
|
| javadoc | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8301/yetus/patch-javadoc-root.txt
 |
| javadoc | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8301/yetus/patch-javadoc-itests_hcatalog-unit.txt
 |
| javadoc | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8301/yetus/patch-javadoc-itests_hive-minikdc.txt
 |
| javadoc | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8301/yetus/patch-javadoc-itests_hive-unit.txt
 |
| modules | C: common standalone-metastore ql service hcatalog/core 
hcatalog/hcatalog-pig-adapter hcatalog/server-extensions 
hcatalog/webhcat/java-client hcatalog/streaming . itests/hcatalog-unit