[jira] [Commented] (HIVE-18976) Add ability to setup Druid Kafka Ingestion from Hive

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403283#comment-16403283
 ] 

Hive QA commented on HIVE-18976:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914906/HIVE-18976.patch

{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 33 failed/errored test(s), 13223 tests 
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=251)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=251)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=251)
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=251)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=94)

[jira] [Commented] (HIVE-18976) Add ability to setup Druid Kafka Ingestion from Hive

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403271#comment-16403271
 ] 

Hive QA commented on HIVE-18976:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  3m 
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m 
41s{color} | {color:red} root: The patch generated 19 new + 657 unchanged - 2 
fixed = 676 total (was 659) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
10s{color} | {color:red} druid-handler: The patch generated 9 new + 106 
unchanged - 0 fixed = 115 total (was 106) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
13s{color} | {color:red} itests/qtest: The patch generated 3 new + 0 unchanged 
- 0 fixed = 3 total (was 0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m  
9s{color} | {color:red} itests/qtest-druid: The patch generated 3 new + 4 
unchanged - 1 fixed = 7 total (was 5) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
15s{color} | {color:red} itests/util: The patch generated 4 new + 119 unchanged 
- 1 fixed = 123 total (was 120) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 17s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  
xml  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9677/dev-support/hive-personality.sh
 |
| git revision | master / d2d50e6 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9677/yetus/diff-checkstyle-root.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9677/yetus/diff-checkstyle-druid-handler.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9677/yetus/diff-checkstyle-itests_qtest.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9677/yetus/diff-checkstyle-itests_qtest-druid.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9677/yetus/diff-checkstyle-itests_util.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9677/yetus/patch-asflicense-problems.txt
 |
| modules | C: common . druid-handler itests/qtest itests/qtest-druid 
itests/util ql U: . |
| Console output | 

[jira] [Commented] (HIVE-18930) Fix ppd_join5 test

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403256#comment-16403256
 ] 

Hive QA commented on HIVE-18930:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914907/HIVE-18930.01.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 13017 tests 
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=92)

[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=94)

[jira] [Commented] (HIVE-18930) Fix ppd_join5 test

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403231#comment-16403231
 ] 

Hive QA commented on HIVE-18930:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
41s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}  1m  2s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9676/dev-support/hive-personality.sh
 |
| git revision | master / d2d50e6 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9676/yetus/patch-asflicense-problems.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9676/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Fix ppd_join5 test
> --
>
> Key: HIVE-18930
> URL: https://issues.apache.org/jira/browse/HIVE-18930
> Project: Hive
>  Issue Type: Bug
>  Components: Test
>Reporter: Kryvenko Igor
>Assignee: Kryvenko Igor
>Priority: Major
> Attachments: HIVE-18930.01.patch
>
>
> It seems that HIVE-18068 have broken this test.
> Log from Jenkins
> {code:java}
>  type="java.lang.AssertionError">
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18264) CachedStore: Store cached partitions/col stats within the table cache and make prewarm non-blocking

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403230#comment-16403230
 ] 

Hive QA commented on HIVE-18264:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914898/HIVE-18264.8.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 31 failed/errored test(s), 13414 tests 
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=92)

[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)

[jira] [Commented] (HIVE-18982) Provide a CLI option to manually trigger failover

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403222#comment-16403222
 ] 

Prasanth Jayachandran commented on HIVE-18982:
--

Initial patch up for review.  Will update the patch with more comments and will 
add test cases in next patch. 

> Provide a CLI option to manually trigger failover
> -
>
> Key: HIVE-18982
> URL: https://issues.apache.org/jira/browse/HIVE-18982
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18982.1.patch
>
>
> HIVE-18281 added active-passive HA. There might be a administrative need to 
> trigger a manual failover of HS2 Active server. Add command line tool to view 
> list of all HS2 instances and trigger manual failover (only under force 
> mode). The clients currently connected to active HS2 will be closed. In 
> future, more options to existing clients connections can be handled via 
> configs/options (like wait until timeout, wait until current sessions are 
> closed etc.)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18982) Provide a CLI option to manually trigger failover

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-18982:
-
Attachment: HIVE-18982.1.patch

> Provide a CLI option to manually trigger failover
> -
>
> Key: HIVE-18982
> URL: https://issues.apache.org/jira/browse/HIVE-18982
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18982.1.patch
>
>
> HIVE-18281 added active-passive HA. There might be a administrative need to 
> trigger a manual failover of HS2 Active server. Add command line tool to view 
> list of all HS2 instances and trigger manual failover (only under force 
> mode). The clients currently connected to active HS2 will be closed. In 
> future, more options to existing clients connections can be handled via 
> configs/options (like wait until timeout, wait until current sessions are 
> closed etc.)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18982) Provide a CLI option to manually trigger failover

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-18982:
-
Status: Patch Available  (was: Open)

> Provide a CLI option to manually trigger failover
> -
>
> Key: HIVE-18982
> URL: https://issues.apache.org/jira/browse/HIVE-18982
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18982.1.patch
>
>
> HIVE-18281 added active-passive HA. There might be a administrative need to 
> trigger a manual failover of HS2 Active server. Add command line tool to view 
> list of all HS2 instances and trigger manual failover (only under force 
> mode). The clients currently connected to active HS2 will be closed. In 
> future, more options to existing clients connections can be handled via 
> configs/options (like wait until timeout, wait until current sessions are 
> closed etc.)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18264) CachedStore: Store cached partitions/col stats within the table cache and make prewarm non-blocking

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403197#comment-16403197
 ] 

Hive QA commented on HIVE-18264:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
16s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m  
9s{color} | {color:red} itests/hcatalog-unit: The patch generated 1 new + 22 
unchanged - 0 fixed = 23 total (was 22) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
22s{color} | {color:red} standalone-metastore: The patch generated 28 new + 
1033 unchanged - 26 fixed = 1061 total (was 1059) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
46s{color} | {color:red} standalone-metastore generated 4 new + 57 unchanged - 
2 fixed = 61 total (was 59) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 30s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9675/dev-support/hive-personality.sh
 |
| git revision | master / d2d50e6 |
| Default Java | 1.8.0_111 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9675/yetus/patch-mvninstall-itests_hcatalog-unit.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9675/yetus/diff-checkstyle-itests_hcatalog-unit.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9675/yetus/diff-checkstyle-standalone-metastore.txt
 |
| javadoc | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9675/yetus/diff-javadoc-javadoc-standalone-metastore.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9675/yetus/patch-asflicense-problems.txt
 |
| modules | C: itests/hcatalog-unit service standalone-metastore U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9675/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> CachedStore: Store cached partitions/col stats within the table cache and 
> make prewarm non-blocking
> ---
>
> Key: HIVE-18264
> URL: https://issues.apache.org/jira/browse/HIVE-18264
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-18264.1.patch, HIVE-18264.2.patch, 
> HIVE-18264.3.patch, HIVE-18264.4.patch, HIVE-18264.5.patch, 
> 

[jira] [Commented] (HIVE-18957) Upgrade Calcite version to 1.16.0

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403189#comment-16403189
 ] 

Hive QA commented on HIVE-18957:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914896/HIVE-18957.02.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9674/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9674/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9674/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-03-17 01:24:13.188
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-9674/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-03-17 01:24:13.211
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
   e7480d7..d2d50e6  master -> origin/master
+ git reset --hard HEAD
HEAD is now at e7480d7 HIVE-18633: Service discovery for Active/Passive HA mode 
(Prasanth Jayachandran reviewed by Sergey Shelukhin)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
  (use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at d2d50e6 HIVE-18886: ACID: NPE on unexplained mysql exceptions 
(Gopal V, reviewed by Eugene Koifman)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-03-17 01:24:20.744
+ rm -rf ../yetus_PreCommit-HIVE-Build-9674
+ mkdir ../yetus_PreCommit-HIVE-Build-9674
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-9674
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-9674/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: a/pom.xml: does not exist in index
error: 
a/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java:
 does not exist in index
error: 
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HiveMaterializedViewRule.java:
 does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java: does 
not exist in index
error: 
a/ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/TestCBORuleFiredOnlyOnce.java:
 does not exist in index
error: a/ql/src/test/queries/clientpositive/druidmini_mv.q: does not exist in 
index
error: 
a/ql/src/test/results/clientpositive/beeline/materialized_view_create_rewrite.q.out:
 does not exist in index
error: a/ql/src/test/results/clientpositive/druid/druidmini_mv.q.out: does not 
exist in index
error: a/ql/src/test/results/clientpositive/druid_basic2.q.out: does not exist 
in index
error: a/ql/src/test/results/clientpositive/druid_basic3.q.out: does not exist 
in index
error: a/ql/src/test/results/clientpositive/druid_intervals.q.out: does not 
exist in index
error: a/ql/src/test/results/clientpositive/druid_topn.q.out: does not exist in 
index
error: 
a/ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite.q.out:
 does not exist in index
error: 
a/ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_dummy.q.out:
 does not exist in index
error: 
a/ql/src/test/results/clientpositive/llap/vectorization_short_regress.q.out: 
does not exist in index
error: 
a/ql/src/test/results/clientpositive/materialized_view_create_rewrite.q.out: 
does not exist in index
error: 
a/ql/src/test/results/clientpositive/spark/vectorization_short_regress.q.out: 
does not exist in index
error: patch failed: 
ql/src/test/results/clientpositive/druid/druidmini_mv.q.out:330
Falling back to 

[jira] [Commented] (HIVE-18986) Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns

[Aihua Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403188#comment-16403188
 ] 

Aihua Xu commented on HIVE-18986:
-

patch-1: use directsql to get the column stats which can use in statement and 
can run in batch. Tested with large table and it works. I will not add such 
unit test for this case.

> Table rename will run java.lang.StackOverflowError in dataNucleus if the 
> table contains large number of columns
> ---
>
> Key: HIVE-18986
> URL: https://issues.apache.org/jira/browse/HIVE-18986
> Project: Hive
>  Issue Type: Improvement
>  Components: Standalone Metastore
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18986.1.patch
>
>
> If the table contains a lot of columns e.g, 5k, simple table rename would 
> fail with the following stack trace. The issue is datanucleus can't handle 
> the query with lots of colName='c1' && colName='c2'.
>  
> 2018-03-13 17:19:52,770 INFO 
> org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-5-thread-200]: 
> ugi=anonymous ip=10.17.100.135 cmd=source:10.17.100.135 alter_table: 
> db=default tbl=fgv_full_var_pivoted02 newtbl=fgv_full_var_pivoted 2018-03-13 
> 17:20:00,495 ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: 
> [pool-5-thread-200]: java.lang.StackOverflowError at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:330) at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18986) Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-18986:

Status: Patch Available  (was: Open)

> Table rename will run java.lang.StackOverflowError in dataNucleus if the 
> table contains large number of columns
> ---
>
> Key: HIVE-18986
> URL: https://issues.apache.org/jira/browse/HIVE-18986
> Project: Hive
>  Issue Type: Improvement
>  Components: Standalone Metastore
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18986.1.patch
>
>
> If the table contains a lot of columns e.g, 5k, simple table rename would 
> fail with the following stack trace. The issue is datanucleus can't handle 
> the query with lots of colName='c1' && colName='c2'.
>  
> 2018-03-13 17:19:52,770 INFO 
> org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-5-thread-200]: 
> ugi=anonymous ip=10.17.100.135 cmd=source:10.17.100.135 alter_table: 
> db=default tbl=fgv_full_var_pivoted02 newtbl=fgv_full_var_pivoted 2018-03-13 
> 17:20:00,495 ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: 
> [pool-5-thread-200]: java.lang.StackOverflowError at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:330) at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18781) Create/Replicate Open, Commit (without writes) and Abort Txn events

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403187#comment-16403187
 ] 

Hive QA commented on HIVE-18781:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914887/HIVE-18781.04.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 418 failed/errored test(s), 12930 tests 
executed
*Failed tests:*
{noformat}
TestCopyUtils - did not produce a TEST-*.xml file (likely timed out) 
(batchId=229)
TestExportImport - did not produce a TEST-*.xml file (likely timed out) 
(batchId=229)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=94)

[jira] [Updated] (HIVE-18986) Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-18986:

Attachment: HIVE-18986.1.patch

> Table rename will run java.lang.StackOverflowError in dataNucleus if the 
> table contains large number of columns
> ---
>
> Key: HIVE-18986
> URL: https://issues.apache.org/jira/browse/HIVE-18986
> Project: Hive
>  Issue Type: Improvement
>  Components: Standalone Metastore
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18986.1.patch
>
>
> If the table contains a lot of columns e.g, 5k, simple table rename would 
> fail with the following stack trace. The issue is datanucleus can't handle 
> the query with lots of colName='c1' && colName='c2'.
>  
> 2018-03-13 17:19:52,770 INFO 
> org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-5-thread-200]: 
> ugi=anonymous ip=10.17.100.135 cmd=source:10.17.100.135 alter_table: 
> db=default tbl=fgv_full_var_pivoted02 newtbl=fgv_full_var_pivoted 2018-03-13 
> 17:20:00,495 ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: 
> [pool-5-thread-200]: java.lang.StackOverflowError at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:330) at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-18986) Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu reassigned HIVE-18986:
---


> Table rename will run java.lang.StackOverflowError in dataNucleus if the 
> table contains large number of columns
> ---
>
> Key: HIVE-18986
> URL: https://issues.apache.org/jira/browse/HIVE-18986
> Project: Hive
>  Issue Type: Improvement
>  Components: Standalone Metastore
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Fix For: 3.0.0
>
>
> If the table contains a lot of columns e.g, 5k, simple table rename would 
> fail with the following stack trace. The issue is datanucleus can't handle 
> the query with lots of colName='c1' && colName='c2'.
>  
> 2018-03-13 17:19:52,770 INFO 
> org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-5-thread-200]: 
> ugi=anonymous ip=10.17.100.135 cmd=source:10.17.100.135 alter_table: 
> db=default tbl=fgv_full_var_pivoted02 newtbl=fgv_full_var_pivoted 2018-03-13 
> 17:20:00,495 ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: 
> [pool-5-thread-200]: java.lang.StackOverflowError at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:330) at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-17843) UINT32 Parquet columns are handled as signed INT32-s, silently reading incorrect data

[Janaki Lahorani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Janaki Lahorani updated HIVE-17843:
---
Attachment: HIVE-17843.1.patch

> UINT32 Parquet columns are handled as signed INT32-s, silently reading 
> incorrect data
> -
>
> Key: HIVE-17843
> URL: https://issues.apache.org/jira/browse/HIVE-17843
> Project: Hive
>  Issue Type: Bug
>Reporter: Zoltan Ivanfi
>Assignee: Janaki Lahorani
>Priority: Major
> Attachments: HIVE-17843.1.patch
>
>
> An unsigned 32 bit Parquet column, such as
> {noformat}
> optional int32 uint_32_col (UINT_32)
> {noformat}
> is read by Hive as if it were signed, leading to incorrect results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18781) Create/Replicate Open, Commit (without writes) and Abort Txn events

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403150#comment-16403150
 ] 

Hive QA commented on HIVE-18781:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
15s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
17s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
29s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
14s{color} | {color:red} itests/hive-unit: The patch generated 19 new + 3 
unchanged - 0 fixed = 22 total (was 3) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
37s{color} | {color:red} ql: The patch generated 25 new + 287 unchanged - 0 
fixed = 312 total (was 287) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
27s{color} | {color:red} standalone-metastore: The patch generated 29 new + 
1240 unchanged - 0 fixed = 1269 total (was 1240) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 9 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
14s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 37s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9673/dev-support/hive-personality.sh
 |
| git revision | master / d2d50e6 |
| Default Java | 1.8.0_111 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9673/yetus/patch-mvninstall-hcatalog_server-extensions.txt
 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9673/yetus/patch-mvninstall-itests_hcatalog-unit.txt
 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9673/yetus/patch-mvninstall-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9673/yetus/diff-checkstyle-itests_hive-unit.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9673/yetus/diff-checkstyle-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9673/yetus/diff-checkstyle-standalone-metastore.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9673/yetus/whitespace-eol.txt 
|
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9673/yetus/patch-asflicense-problems.txt
 |
| modules | C: hcatalog/server-extensions itests/hcatalog-unit itests/hive-unit 

[jira] [Updated] (HIVE-18862) qfiles: prepare .q files for using datasets

[Laszlo Bodor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Bodor updated HIVE-18862:

Attachment: HIVE-18862.03.patch

> qfiles: prepare .q files for using datasets
> ---
>
> Key: HIVE-18862
> URL: https://issues.apache.org/jira/browse/HIVE-18862
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18862.01.patch, HIVE-18862.02.patch, 
> HIVE-18862.03.patch
>
>
> # Parse .q files for source table usage
>  # Add needed dataset annotations
>  # Remove setInitScript("q_test_init.sql"); from CliConfig



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18862) qfiles: prepare .q files for using datasets

[Laszlo Bodor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Bodor updated HIVE-18862:

Attachment: (was: HIVE-18862.03.patch)

> qfiles: prepare .q files for using datasets
> ---
>
> Key: HIVE-18862
> URL: https://issues.apache.org/jira/browse/HIVE-18862
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18862.01.patch, HIVE-18862.02.patch
>
>
> # Parse .q files for source table usage
>  # Add needed dataset annotations
>  # Remove setInitScript("q_test_init.sql"); from CliConfig



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18862) qfiles: prepare .q files for using datasets

[Laszlo Bodor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Bodor updated HIVE-18862:

Attachment: HIVE-18862.03.patch

> qfiles: prepare .q files for using datasets
> ---
>
> Key: HIVE-18862
> URL: https://issues.apache.org/jira/browse/HIVE-18862
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18862.01.patch, HIVE-18862.02.patch, 
> HIVE-18862.03.patch
>
>
> # Parse .q files for source table usage
>  # Add needed dataset annotations
>  # Remove setInitScript("q_test_init.sql"); from CliConfig



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18984) Make time window configurable per materialized view

[Jesus Camacho Rodriguez (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403133#comment-16403133
 ] 

Jesus Camacho Rodriguez commented on HIVE-18984:


[~ashutoshc], could you take a look? Thanks
https://reviews.apache.org/r/66125/


> Make time window configurable per materialized view
> ---
>
> Key: HIVE-18984
> URL: https://issues.apache.org/jira/browse/HIVE-18984
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18984.patch
>
>
> Currently, {{hive.materializedview.rewriting.time.window}} can be used to 
> specify a time window after which outdated materialized views become invalid 
> for automatic query rewriting (default value is 0). We would like to be able 
> to specify this property for each individual materialized view too via 
> tblproperties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18976) Add ability to setup Druid Kafka Ingestion from Hive

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403131#comment-16403131
 ] 

Hive QA commented on HIVE-18976:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914906/HIVE-18976.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9672/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9672/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9672/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-03-16 23:51:01.270
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-9672/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-03-16 23:51:01.274
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at e7480d7 HIVE-18633: Service discovery for Active/Passive HA mode 
(Prasanth Jayachandran reviewed by Sergey Shelukhin)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at e7480d7 HIVE-18633: Service discovery for Active/Passive HA mode 
(Prasanth Jayachandran reviewed by Sergey Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-03-16 23:51:04.500
+ rm -rf ../yetus_PreCommit-HIVE-Build-9672
+ mkdir ../yetus_PreCommit-HIVE-Build-9672
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-9672
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-9672/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: a/common/src/java/org/apache/hadoop/hive/conf/Constants.java: does not 
exist in index
error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: does not 
exist in index
error: a/druid-handler/pom.xml: does not exist in index
error: 
a/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java:
 does not exist in index
error: a/itests/qtest-druid/pom.xml: does not exist in index
error: 
a/itests/qtest-druid/src/main/java/org/apache/hive/druid/MiniDruidCluster.java: 
does not exist in index
error: 
a/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java: 
does not exist in index
error: a/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java: 
does not exist in index
error: a/pom.xml: does not exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q 
-Dmaven.repo.local=/data/hiveptest/working/maven
[ERROR] Failed to execute goal on project hive-shims-common: Could not resolve 
dependencies for project 
org.apache.hive.shims:hive-shims-common:jar:3.0.0-SNAPSHOT: Could not find 
artifact commons-codec:commons-codec:jar:1.7 in datanucleus 
(http://www.datanucleus.org/downloads/maven2) -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hive-shims-common
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12914906 - PreCommit-HIVE-Build

> Add ability to setup Druid Kafka Ingestion from Hive
> 

[jira] [Updated] (HIVE-18984) Make time window configurable per materialized view

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-18984:
---
Status: Patch Available  (was: In Progress)

> Make time window configurable per materialized view
> ---
>
> Key: HIVE-18984
> URL: https://issues.apache.org/jira/browse/HIVE-18984
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18984.patch
>
>
> Currently, {{hive.materializedview.rewriting.time.window}} can be used to 
> specify a time window after which outdated materialized views become invalid 
> for automatic query rewriting (default value is 0). We would like to be able 
> to specify this property for each individual materialized view too via 
> tblproperties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18984) Make time window configurable per materialized view

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-18984:
---
Attachment: HIVE-18984.patch

> Make time window configurable per materialized view
> ---
>
> Key: HIVE-18984
> URL: https://issues.apache.org/jira/browse/HIVE-18984
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18984.patch
>
>
> Currently, {{hive.materializedview.rewriting.time.window}} can be used to 
> specify a time window after which outdated materialized views become invalid 
> for automatic query rewriting (default value is 0). We would like to be able 
> to specify this property for each individual materialized view too via 
> tblproperties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18747) Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403128#comment-16403128
 ] 

Hive QA commented on HIVE-18747:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914878/HIVE-18747.01.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 13019 tests 
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=92)

[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=94)

[jira] [Updated] (HIVE-18978) ConditionalTask.addDependentTask(Task t) adds t in the wrong place

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18978:
--
Description: 
{\{ConditionalTask.addDependentTask(Task t) }} is implemented like this:
{noformat}
/**
* Add a dependent task on the current conditional task. The task will not be a 
direct child of
* conditional task. Actually it will be added as child task of associated tasks.
*
* @return true if the task got added false if it already existed
*/
@Override
public boolean addDependentTask(Task dependent) {
  boolean ret = false;
  if (getListTasks() != null) {
ret = true;
for (Task tsk : getListTasks()) {
  ret = ret & tsk.addDependentTask(dependent);
}
  }
  return ret;
}
{noformat}
So let’s say, the tasks in the ConditionalTask are A,B,C, but they have 
children.
{noformat}
CondTask
  |--A
 |--A1
|-A2
  |--B
 |--B1
  |--C
|--C1
{noformat}
The way ConditionalTask.addDependent() is implemented, MyTask becomes a sibling 
of A1,
 B1 and C1. So even if only 1 branch of ConditionalTask is executed (and 
parallel task
 execution is enabled), there is no guarantee (as I see) that MyTask runs after 
A2 or
 B1 or C1, which is really what is needed.

 

Once this is done add a .q file test that records a plan for Export from Acid: 
HIVE-18739

  was:
{\{ConditionalTask.addDependentTask(Task t) }} is implemented like this:
{noformat}
/**
* Add a dependent task on the current conditional task. The task will not be a 
direct child of
* conditional task. Actually it will be added as child task of associated tasks.
*
* @return true if the task got added false if it already existed
*/
@Override
public boolean addDependentTask(Task dependent) {
  boolean ret = false;
  if (getListTasks() != null) {
ret = true;
for (Task tsk : getListTasks()) {
  ret = ret & tsk.addDependentTask(dependent);
}
  }
  return ret;
}
{noformat}
So let’s say, the tasks in the ConditionalTask are A,B,C, but they have 
children.
{noformat}
CondTask
  |--A
 |--A1
|-A2
  |--B
 |--B1
  |--C
|--C1
{noformat}
The way ConditionalTask.addDependent() is implemented, MyTask becomes a sibling 
of A1,
 B1 and C1. So even if only 1 branch of ConditionalTask is executed (and 
parallel task
 execution is enabled), there is no guarantee (as I see) that MyTask runs after 
A2 or
 B1 or C1, which is really what is needed.

 

Once this is done add a .q file test that records a plan for Export from Acid: 
HIVE-18978


> ConditionalTask.addDependentTask(Task t) adds t in the wrong place
> --
>
> Key: HIVE-18978
> URL: https://issues.apache.org/jira/browse/HIVE-18978
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> {\{ConditionalTask.addDependentTask(Task t) }} is implemented like this:
> {noformat}
> /**
> * Add a dependent task on the current conditional task. The task will not be 
> a direct child of
> * conditional task. Actually it will be added as child task of associated 
> tasks.
> *
> * @return true if the task got added false if it already existed
> */
> @Override
> public boolean addDependentTask(Task dependent) {
>   boolean ret = false;
>   if (getListTasks() != null) {
> ret = true;
> for (Task tsk : getListTasks()) {
>   ret = ret & tsk.addDependentTask(dependent);
> }
>   }
>   return ret;
> }
> {noformat}
> So let’s say, the tasks in the ConditionalTask are A,B,C, but they have 
> children.
> {noformat}
> CondTask
>   |--A
>  |--A1
> |-A2
>   |--B
>  |--B1
>   |--C
> |--C1
> {noformat}
> The way ConditionalTask.addDependent() is implemented, MyTask becomes a 
> sibling of A1,
>  B1 and C1. So even if only 1 branch of ConditionalTask is executed (and 
> parallel task
>  execution is enabled), there is no guarantee (as I see) that MyTask runs 
> after A2 or
>  B1 or C1, which is really what is needed.
>  
> Once this is done add a .q file test that records a plan for Export from 
> Acid: HIVE-18739



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18739) Add support for Export from unpartitioned Acid table

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18739:
--
Attachment: HIVE-18739.09.patch

> Add support for Export from unpartitioned Acid table
> 
>
> Key: HIVE-18739
> URL: https://issues.apache.org/jira/browse/HIVE-18739
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18739.01.patch, HIVE-18739.04.patch, 
> HIVE-18739.04.patch, HIVE-18739.06.patch, HIVE-18739.08.patch, 
> HIVE-18739.09.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18985) CLI tool to list Tez AMs that are registered with Zookeeper

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403103#comment-16403103
 ] 

Prasanth Jayachandran commented on HIVE-18985:
--

cc/ [~sershe]


> CLI tool to list Tez AMs that are registered with Zookeeper
> ---
>
> Key: HIVE-18985
> URL: https://issues.apache.org/jira/browse/HIVE-18985
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Priority: Major
>
> For easier debugging it will be easier to have a tool that lists all Tez 
> Sessions that are registered with zookeeper.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18747) Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403070#comment-16403070
 ] 

Hive QA commented on HIVE-18747:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
41s{color} | {color:red} ql: The patch generated 10 new + 529 unchanged - 0 
fixed = 539 total (was 529) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
24s{color} | {color:red} standalone-metastore: The patch generated 4 new + 1172 
unchanged - 1 fixed = 1176 total (was 1173) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 31 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 18s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9671/dev-support/hive-personality.sh
 |
| git revision | master / e7480d7 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9671/yetus/diff-checkstyle-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9671/yetus/diff-checkstyle-standalone-metastore.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9671/yetus/whitespace-eol.txt 
|
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9671/yetus/patch-asflicense-problems.txt
 |
| modules | C: metastore ql standalone-metastore U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9671/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.
> --
>
> Key: HIVE-18747
> URL: https://issues.apache.org/jira/browse/HIVE-18747
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Minor
>  Labels: ACID, pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18747.01.patch
>
>
> Per table write ID implementation (HIVE-18192) maintains a map between txn ID 
> and table write ID in TXN_TO_WRITE_ID meta table. 
> The entries in this table is used to generate 

[jira] [Work started] (HIVE-18984) Make time window configurable per materialized view

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-18984 started by Jesus Camacho Rodriguez.
--
> Make time window configurable per materialized view
> ---
>
> Key: HIVE-18984
> URL: https://issues.apache.org/jira/browse/HIVE-18984
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Currently, {{hive.materializedview.rewriting.time.window}} can be used to 
> specify a time window after which outdated materialized views become invalid 
> for automatic query rewriting (default value is 0). We would like to be able 
> to specify this property for each individual materialized view too via 
> tblproperties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-18984) Make time window configurable per materialized view

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez reassigned HIVE-18984:
--


> Make time window configurable per materialized view
> ---
>
> Key: HIVE-18984
> URL: https://issues.apache.org/jira/browse/HIVE-18984
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Currently, {{hive.materializedview.rewriting.time.window}} can be used to 
> specify a time window after which outdated materialized views become invalid 
> for automatic query rewriting (default value is 0). We would like to be able 
> to specify this property for each individual materialized view too via 
> tblproperties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18979) Enable AggregateReduceFunctionsRule from Calcite

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-18979:
---
Issue Type: Improvement  (was: Task)

> Enable AggregateReduceFunctionsRule from Calcite
> 
>
> Key: HIVE-18979
> URL: https://issues.apache.org/jira/browse/HIVE-18979
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18979.patch
>
>
> The rule rewrites aggregate functions into simpler ones. It should help us 
> gathering more common sub-expressions for a given plan, and it should have a 
> positive effect on materialized view rewriting coverage too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18975) NPE when inserting NULL value in structure and array with HBase table

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402648#comment-16402648
 ] 

Hive QA commented on HIVE-18975:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914854/HIVE-18975.1.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 129 failed/errored test(s), 13819 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats] 
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] 
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_1] 
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_with_masking]
 (batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[source] (batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_10_trims] 
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_from_unixtime] 
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_instr] (batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union25] (batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_remove_17] 
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_script] 
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_between_columns] 
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_char_2] 
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[view_cbo] (batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_streaming] 
(batchId=68)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_mv] 
(batchId=251)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] 
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez_empty]
 (batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby_groupingset_bug]
 (batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin] 
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
 (batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[update_access_time_non_current_db]
 (batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_div0]
 (batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_semijoin_reduction]
 (batchId=153)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[mm_convert] 
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[mm_truncate_cols]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[set_hiveconf_validation0]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[set_hiveconf_validation1]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[set_hiveconf_validation2]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[show_create_table_does_not_exist]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[show_tables_bad1]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[show_tables_bad2]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[show_tables_bad_db1]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[show_tablestatus]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_bucketmapjoin]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_mapjoin_14] 
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[sortmerge_mapjoin_mismatch_1]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[special_character_in_tabnames_1]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[split_sample_out_of_range]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[split_sample_wrong_format]
 (batchId=95)

[jira] [Updated] (HIVE-18979) Enable AggregateReduceFunctionsRule from Calcite

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-18979:
---
Attachment: HIVE-18979.patch

> Enable AggregateReduceFunctionsRule from Calcite
> 
>
> Key: HIVE-18979
> URL: https://issues.apache.org/jira/browse/HIVE-18979
> Project: Hive
>  Issue Type: Task
>  Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18979.patch
>
>
> The rule rewrites aggregate functions into simpler ones. It should help us 
> gathering more common sub-expressions for a given plan, and it should have a 
> positive effect on materialized view rewriting coverage too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18979) Enable AggregateReduceFunctionsRule from Calcite

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-18979:
---
Status: Patch Available  (was: In Progress)

> Enable AggregateReduceFunctionsRule from Calcite
> 
>
> Key: HIVE-18979
> URL: https://issues.apache.org/jira/browse/HIVE-18979
> Project: Hive
>  Issue Type: Task
>  Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18979.patch
>
>
> The rule rewrites aggregate functions into simpler ones. It should help us 
> gathering more common sub-expressions for a given plan, and it should have a 
> positive effect on materialized view rewriting coverage too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18633) Service discovery for Active/Passive HA mode

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-18633:
-
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

No related test failures.
Committed to master. Thanks for the reviews.

> Service discovery for Active/Passive HA mode
> 
>
> Key: HIVE-18633
> URL: https://issues.apache.org/jira/browse/HIVE-18633
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18633.1.patch, HIVE-18633.2.patch
>
>
> HIVE-18281 adds Active/Passive HA mode for HS2. This jira is to extend the 
> feature so that clients will connect only to leader instance via JDBC + 
> service discovery. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18633) Service discovery for Active/Passive HA mode

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-18633:
-
Attachment: HIVE-18633.2.patch

> Service discovery for Active/Passive HA mode
> 
>
> Key: HIVE-18633
> URL: https://issues.apache.org/jira/browse/HIVE-18633
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18633.1.patch, HIVE-18633.2.patch
>
>
> HIVE-18281 adds Active/Passive HA mode for HS2. This jira is to extend the 
> feature so that clients will connect only to leader instance via JDBC + 
> service discovery. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18633) Service discovery for Active/Passive HA mode

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402597#comment-16402597
 ] 

Sergey Shelukhin commented on HIVE-18633:
-

Minor comments
+1 pending tests

> Service discovery for Active/Passive HA mode
> 
>
> Key: HIVE-18633
> URL: https://issues.apache.org/jira/browse/HIVE-18633
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18633.1.patch
>
>
> HIVE-18281 adds Active/Passive HA mode for HS2. This jira is to extend the 
> feature so that clients will connect only to leader instance via JDBC + 
> service discovery. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18952) Tez session disconnect and reconnect on HS2 HA failover

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402594#comment-16402594
 ] 

Sergey Shelukhin commented on HIVE-18952:
-

Minor comments
+1 pending tests

> Tez session disconnect and reconnect on HS2 HA failover
> ---
>
> Key: HIVE-18952
> URL: https://issues.apache.org/jira/browse/HIVE-18952
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
>Priority: Major
>
> Now that TEZ-3892 is committed, HIVE-18281 can make use of tez session 
> disconnect and reconnect on HA failover.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Issue Comment Deleted] (HIVE-18952) Tez session disconnect and reconnect on HS2 HA failover

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18952:

Comment: was deleted

(was: Minor comments
+1 pending tests)

> Tez session disconnect and reconnect on HS2 HA failover
> ---
>
> Key: HIVE-18952
> URL: https://issues.apache.org/jira/browse/HIVE-18952
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
>Priority: Major
>
> Now that TEZ-3892 is committed, HIVE-18281 can make use of tez session 
> disconnect and reconnect on HA failover.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-14388) Add number of rows inserted message after insert command in Beeline

[Bharathkrishna Guruvayoor Murali (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402557#comment-16402557
 ] 

Bharathkrishna Guruvayoor Murali commented on HIVE-14388:
-

Hi [~Jk_Self] ,

Are you still working on this JIRA?

> Add number of rows inserted message after insert command in Beeline
> ---
>
> Key: HIVE-14388
> URL: https://issues.apache.org/jira/browse/HIVE-14388
> Project: Hive
>  Issue Type: Improvement
>  Components: Beeline
>Reporter: Vihang Karajgaonkar
>Assignee: Ke Jia
>Priority: Minor
> Attachments: HIVE-14388-WIP.patch
>
>
> Currently, when you run insert command on beeline, it returns a message 
> saying "No rows affected .."
> A better and more intuitive msg would be "xxx rows inserted (26.068 seconds)"



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18975) NPE when inserting NULL value in structure and array with HBase table

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402551#comment-16402551
 ] 

Hive QA commented on HIVE-18975:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
11s{color} | {color:red} hbase-handler: The patch generated 6 new + 79 
unchanged - 2 fixed = 85 total (was 81) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}  9m  2s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9670/dev-support/hive-personality.sh
 |
| git revision | master / 6218d8a |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9670/yetus/diff-checkstyle-hbase-handler.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9670/yetus/patch-asflicense-problems.txt
 |
| modules | C: hbase-handler U: hbase-handler |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9670/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> NPE when inserting NULL value in structure and array with HBase table
> -
>
> Key: HIVE-18975
> URL: https://issues.apache.org/jira/browse/HIVE-18975
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18975.1.patch
>
>
> STR (Structure)
> *STEP 1. Create tables*
> {code}
> CREATE TABLE IF NOT EXISTS t1 (id INT);
> INSERT INTO TABLE t1 VALUES (1),(2),(3),(4),(5);
> CREATE TABLE IF NOT EXISTS `htable`(
>   `id` INT, 
>   `map_column` STRUCT) ROW FORMAT 
> SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe'  STORED BY 
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'  WITH SERDEPROPERTIES (   
> 'hbase.columns.mapping'=':key,id:id','serialization.format'='1') 
> TBLPROPERTIES ( 'hbase.table.name'='tmp/h');
> {code}
> *STEP 2. Insert into table stored in HBase the struct with NULL value in it*
> {code}
> INSERT INTO `htable` SELECT 2,NAMED_STRUCT("s_int",CAST(NULL AS 
> INT),"s_string","s1","s_date",CAST('2018-03-12' AS DATE)) FROM t1 LIMIT 1;
> {code}
> *ACTUAL RESULT*
> The query fails with NPE.
> {code}
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row (tag=0) 
> 

[jira] [Commented] (HIVE-18633) Service discovery for Active/Passive HA mode

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402533#comment-16402533
 ] 

Hive QA commented on HIVE-18633:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914826/HIVE-18633.1.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 13027 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=94)

[jira] [Assigned] (HIVE-18983) Add support for table properties inheritance in Create table like

[Kryvenko Igor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kryvenko Igor reassigned HIVE-18983:



> Add support for table properties inheritance in Create table like
> -
>
> Key: HIVE-18983
> URL: https://issues.apache.org/jira/browse/HIVE-18983
> Project: Hive
>  Issue Type: Improvement
>Reporter: Kryvenko Igor
>Assignee: Kryvenko Igor
>Priority: Minor
> Fix For: 3.0.0
>
>
> Currently, Create table like support table properties.
> But it doesn't inherit table properties from original table.
> {code}
> create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc 
> TBLPROPERTIES ('comment'='comm');
> create table T like T1;
> show create table T;
>  {code}
> *Output:*
> {code}
> CREATE TABLE `T`(
>   `a` int, 
>   `b` int)
> CLUSTERED BY ( 
>   a) 
> INTO 2 BUCKETS
> ROW FORMAT SERDE 
>   'org.apache.hadoop.hive.ql.io.orc.OrcSerde' 
> STORED AS INPUTFORMAT 
>   'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' 
> OUTPUTFORMAT 
>   'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>   'maprfs:/user/hive/warehouse/t'
> TBLPROPERTIES (
>   'COLUMN_STATS_ACCURATE'='{\"BASIC_STATS\":\"true\"}', 
>   'numFiles'='0', 
>   'numRows'='0', 
>   'rawDataSize'='0', 
>   'totalSize'='0', 
>   'transient_lastDdlTime'='1521230300')
> {code}
> It uses just default table properties and doesn't inherit properties from 
> original table.
> It would be great if create table like will inherit origin table properties 
> and will override them if they are specified in query .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18975) NPE when inserting NULL value in structure and array with HBase table

[Yongzhi Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402517#comment-16402517
 ] 

Yongzhi Chen commented on HIVE-18975:
-

The change looks fine. +1

> NPE when inserting NULL value in structure and array with HBase table
> -
>
> Key: HIVE-18975
> URL: https://issues.apache.org/jira/browse/HIVE-18975
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18975.1.patch
>
>
> STR (Structure)
> *STEP 1. Create tables*
> {code}
> CREATE TABLE IF NOT EXISTS t1 (id INT);
> INSERT INTO TABLE t1 VALUES (1),(2),(3),(4),(5);
> CREATE TABLE IF NOT EXISTS `htable`(
>   `id` INT, 
>   `map_column` STRUCT) ROW FORMAT 
> SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe'  STORED BY 
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'  WITH SERDEPROPERTIES (   
> 'hbase.columns.mapping'=':key,id:id','serialization.format'='1') 
> TBLPROPERTIES ( 'hbase.table.name'='tmp/h');
> {code}
> *STEP 2. Insert into table stored in HBase the struct with NULL value in it*
> {code}
> INSERT INTO `htable` SELECT 2,NAMED_STRUCT("s_int",CAST(NULL AS 
> INT),"s_string","s1","s_date",CAST('2018-03-12' AS DATE)) FROM t1 LIMIT 1;
> {code}
> *ACTUAL RESULT*
> The query fails with NPE.
> {code}
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row (tag=0) 
> {"key":{},"value":{"_col0":2,"_col1":{"s_int":null,"s_string":"s1","s_date":"2018-03-12"}}}
>   at 
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:257)
>   at 
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
>   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
>   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1631)
>   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row (tag=0) 
> {"key":{},"value":{"_col0":2,"_col1":{"s_int":null,"s_string":"s1","s_date":"2018-03-12"}}}
>   at 
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:245)
>   ... 7 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:787)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
>   at 
> org.apache.hadoop.hive.ql.exec.LimitOperator.process(LimitOperator.java:63)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
>   at 
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
>   at 
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:236)
>   ... 7 more
> Caused by: org.apache.hadoop.hive.serde2.SerDeException: 
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:301)
>   at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:714)
>   ... 12 more
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:36)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:239)
>   at 
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:236)
>   at 
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:295)
>   at 
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:222)
>   at 
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serializeField(HBaseRowSerializer.java:194)
>   at 
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:118)
>   at 
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:297)
>   ... 13 more
> {code}
> *EXPECTED RESULT*
> The query finished successfully.
> STR (arrays)
> *STEP 1. Create tables*
> {code}
> CREATE TABLE hbase_list(id INT, list_column array) ROW FORMAT SERDE 
> 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY 
> 

[jira] [Updated] (HIVE-18978) ConditionalTask.addDependentTask(Task t) adds t in the wrong place

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18978:
--
Description: 
{\{ConditionalTask.addDependentTask(Task t) }} is implemented like this:
{noformat}
/**
* Add a dependent task on the current conditional task. The task will not be a 
direct child of
* conditional task. Actually it will be added as child task of associated tasks.
*
* @return true if the task got added false if it already existed
*/
@Override
public boolean addDependentTask(Task dependent) {
  boolean ret = false;
  if (getListTasks() != null) {
ret = true;
for (Task tsk : getListTasks()) {
  ret = ret & tsk.addDependentTask(dependent);
}
  }
  return ret;
}
{noformat}
So let’s say, the tasks in the ConditionalTask are A,B,C, but they have 
children.
{noformat}
CondTask
  |--A
 |--A1
|-A2
  |--B
 |--B1
  |--C
|--C1
{noformat}
The way ConditionalTask.addDependent() is implemented, MyTask becomes a sibling 
of A1,
 B1 and C1. So even if only 1 branch of ConditionalTask is executed (and 
parallel task
 execution is enabled), there is no guarantee (as I see) that MyTask runs after 
A2 or
 B1 or C1, which is really what is needed.

 

Once this is done add a .q file test that records a plan for Export from Acid: 
HIVE-18978

  was:
{{ConditionalTask.addDependentTask(Task t) }} is implemented like this:
{noformat}
/**
* Add a dependent task on the current conditional task. The task will not be a 
direct child of
* conditional task. Actually it will be added as child task of associated tasks.
*
* @return true if the task got added false if it already existed
*/
@Override
public boolean addDependentTask(Task dependent) {
  boolean ret = false;
  if (getListTasks() != null) {
ret = true;
for (Task tsk : getListTasks()) {
  ret = ret & tsk.addDependentTask(dependent);
}
  }
  return ret;
}
{noformat}


 So let’s say, the tasks in the ConditionalTask are A,B,C, but they have 
children.
{noformat}
CondTask
  |--A
 |--A1
|-A2
  |--B
 |--B1
  |--C
|--C1
{noformat}

 The way ConditionalTask.addDependent() is implemented, MyTask becomes a 
sibling of A1,
 B1 and C1.  So even if only 1 branch of ConditionalTask is executed (and 
parallel task
 execution is enabled), there is no guarantee (as I see) that MyTask runs 
after A2 or
 B1 or C1, which is really what is needed.



> ConditionalTask.addDependentTask(Task t) adds t in the wrong place
> --
>
> Key: HIVE-18978
> URL: https://issues.apache.org/jira/browse/HIVE-18978
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> {\{ConditionalTask.addDependentTask(Task t) }} is implemented like this:
> {noformat}
> /**
> * Add a dependent task on the current conditional task. The task will not be 
> a direct child of
> * conditional task. Actually it will be added as child task of associated 
> tasks.
> *
> * @return true if the task got added false if it already existed
> */
> @Override
> public boolean addDependentTask(Task dependent) {
>   boolean ret = false;
>   if (getListTasks() != null) {
> ret = true;
> for (Task tsk : getListTasks()) {
>   ret = ret & tsk.addDependentTask(dependent);
> }
>   }
>   return ret;
> }
> {noformat}
> So let’s say, the tasks in the ConditionalTask are A,B,C, but they have 
> children.
> {noformat}
> CondTask
>   |--A
>  |--A1
> |-A2
>   |--B
>  |--B1
>   |--C
> |--C1
> {noformat}
> The way ConditionalTask.addDependent() is implemented, MyTask becomes a 
> sibling of A1,
>  B1 and C1. So even if only 1 branch of ConditionalTask is executed (and 
> parallel task
>  execution is enabled), there is no guarantee (as I see) that MyTask runs 
> after A2 or
>  B1 or C1, which is really what is needed.
>  
> Once this is done add a .q file test that records a plan for Export from 
> Acid: HIVE-18978



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18959) Avoid creating extra pool of threads within LLAP

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-18959:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Pushed to master. Thanks, Slim!

> Avoid creating extra pool of threads within LLAP
> 
>
> Key: HIVE-18959
> URL: https://issues.apache.org/jira/browse/HIVE-18959
> Project: Hive
>  Issue Type: Task
>  Components: Druid integration
> Environment: Kerberos Cluster
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18959.patch
>
>
> The current Druid-Kerberos-Http client is using an external single threaded 
> pool to handle retry auth calls (eg when a cookie expire or other transient 
> auth issues). 
> First, this is not buying us anything since all the Druid Task is executed as 
> one synchronous task.
> Second, this can cause a major issue if an exception occurs that leads to 
> shutting down the LLAP main thread with an exception like: 
> {code} org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Thread 
> Thread[KerberosHttpClient... threw an Exception. Shutting down now...{code}
>  Thus to fix this we should avoid using an external thread pool and handle 
> retrying in a synchronous way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18982) Provide a CLI option to manually trigger failover

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-18982:
-
Summary: Provide a CLI option to manually trigger failover  (was: Provide 
an option to manually trigger failover via deregister)

> Provide a CLI option to manually trigger failover
> -
>
> Key: HIVE-18982
> URL: https://issues.apache.org/jira/browse/HIVE-18982
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
>
> HIVE-18281 added active-passive HA. There might be a administrative need to 
> trigger a manual failover of HS2 Active server. Add command line tool to view 
> list of all HS2 instances and trigger manual failover (only under force 
> mode). The clients currently connected to active HS2 will be closed. In 
> future, more options to existing clients connections can be handled via 
> configs/options (like wait until timeout, wait until current sessions are 
> closed etc.)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-18982) Provide an option to manually trigger failover via deregister

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran reassigned HIVE-18982:



> Provide an option to manually trigger failover via deregister
> -
>
> Key: HIVE-18982
> URL: https://issues.apache.org/jira/browse/HIVE-18982
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
>
> HIVE-18281 added active-passive HA. There might be a administrative need to 
> trigger a manual failover of HS2 Active server. Add command line tool to view 
> list of all HS2 instances and trigger manual failover (only under force 
> mode). The clients currently connected to active HS2 will be closed. In 
> future, more options to existing clients connections can be handled via 
> configs/options (like wait until timeout, wait until current sessions are 
> closed etc.)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18633) Service discovery for Active/Passive HA mode

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402444#comment-16402444
 ] 

Prasanth Jayachandran commented on HIVE-18633:
--

rbt post worked. attached RB link.

> Service discovery for Active/Passive HA mode
> 
>
> Key: HIVE-18633
> URL: https://issues.apache.org/jira/browse/HIVE-18633
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18633.1.patch
>
>
> HIVE-18281 adds Active/Passive HA mode for HS2. This jira is to extend the 
> feature so that clients will connect only to leader instance via JDBC + 
> service discovery. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18633) Service discovery for Active/Passive HA mode

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402430#comment-16402430
 ] 

Hive QA commented on HIVE-18633:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
24s{color} | {color:red} util in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
19s{color} | {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
19s{color} | {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 19s{color} 
| {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
15s{color} | {color:red} itests/hive-unit: The patch generated 7 new + 4 
unchanged - 0 fixed = 11 total (was 4) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
10s{color} | {color:red} jdbc: The patch generated 7 new + 55 unchanged - 0 
fixed = 62 total (was 55) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m  
9s{color} | {color:red} llap-client: The patch generated 2 new + 42 unchanged - 
0 fixed = 44 total (was 42) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
11s{color} | {color:red} service: The patch generated 1 new + 24 unchanged - 0 
fixed = 25 total (was 24) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 27s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9669/dev-support/hive-personality.sh
 |
| git revision | master / f40d944 |
| Default Java | 1.8.0_111 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9669/yetus/patch-mvninstall-itests_util.txt
 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9669/yetus/patch-mvninstall-service.txt
 |
| compile | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9669/yetus/patch-compile-service.txt
 |
| javac | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9669/yetus/patch-compile-service.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9669/yetus/diff-checkstyle-itests_hive-unit.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9669/yetus/diff-checkstyle-jdbc.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9669/yetus/diff-checkstyle-llap-client.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9669/yetus/diff-checkstyle-service.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9669/yetus/patch-asflicense-problems.txt
 |
| modules | C: itests/hive-unit itests/util jdbc llap-client service 

[jira] [Updated] (HIVE-18968) LLAP: report guaranteed tasks count in AM registry to check for consistency

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18968:

Attachment: HIVE-18968.01.patch

> LLAP: report guaranteed tasks count in AM registry to check for consistency
> ---
>
> Key: HIVE-18968
> URL: https://issues.apache.org/jira/browse/HIVE-18968
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18968.01.patch, HIVE-18968.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18968) LLAP: report guaranteed tasks count in AM registry to check for consistency

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402407#comment-16402407
 ] 

Sergey Shelukhin commented on HIVE-18968:
-

Rebased and addressed feedback

> LLAP: report guaranteed tasks count in AM registry to check for consistency
> ---
>
> Key: HIVE-18968
> URL: https://issues.apache.org/jira/browse/HIVE-18968
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18968.01.patch, HIVE-18968.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402392#comment-16402392
 ] 

Hive QA commented on HIVE-18908:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914816/HIVE-18908.08.patch

{color:green}SUCCESS:{color} +1 due to 37 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 121 failed/errored test(s), 13050 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=94)

[jira] [Updated] (HIVE-18981) Hive beeline ignores throwable

[Bharathkrishna Guruvayoor Murali (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharathkrishna Guruvayoor Murali updated HIVE-18981:

Description: 
Beeline ignores the exception in catch block here:

 
[https://github.com/apache/hive/blob/f40d94476718df6de42caef3b18811f024fe8713/beeline/src/java/org/apache/hive/beeline/BeeLine.java#L1021]

here:

[https://github.com/apache/hive/blob/f40d94476718df6de42caef3b18811f024fe8713/beeline/src/java/org/apache/hive/beeline/BeeLine.java#L1055]

  was:
Beeline ignores the exception in catch block here:

[https://github.com/apache/hive/blob/f40d94476718df6de42caef3b18811f024fe8713/beeline/src/java/org/apache/hive/beeline/BeeLine.java#L1017]

here:

[https://github.com/apache/hive/blob/f40d94476718df6de42caef3b18811f024fe8713/beeline/src/java/org/apache/hive/beeline/BeeLine.java#L1055]


> Hive beeline ignores throwable
> --
>
> Key: HIVE-18981
> URL: https://issues.apache.org/jira/browse/HIVE-18981
> Project: Hive
>  Issue Type: Bug
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
>
> Beeline ignores the exception in catch block here:
>  
> [https://github.com/apache/hive/blob/f40d94476718df6de42caef3b18811f024fe8713/beeline/src/java/org/apache/hive/beeline/BeeLine.java#L1021]
> here:
> [https://github.com/apache/hive/blob/f40d94476718df6de42caef3b18811f024fe8713/beeline/src/java/org/apache/hive/beeline/BeeLine.java#L1055]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18981) Hive beeline ignores throwable

[Bharathkrishna Guruvayoor Murali (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharathkrishna Guruvayoor Murali updated HIVE-18981:

Description: 
Beeline ignores the exception in catch block here:

[https://github.com/apache/hive/blob/f40d94476718df6de42caef3b18811f024fe8713/beeline/src/java/org/apache/hive/beeline/BeeLine.java#L1017]

here:

[https://github.com/apache/hive/blob/f40d94476718df6de42caef3b18811f024fe8713/beeline/src/java/org/apache/hive/beeline/BeeLine.java#L1055]

  was:
Beeline ignores the exception in catch block here:

[https://github.com/apache/hive/blob/f40d94476718df6de42caef3b18811f024fe8713/beeline/src/java/org/apache/hive/beeline/BeeLine.java#L1017

]

here:

https://github.com/apache/hive/blob/f40d94476718df6de42caef3b18811f024fe8713/beeline/src/java/org/apache/hive/beeline/BeeLine.java#L1055


> Hive beeline ignores throwable
> --
>
> Key: HIVE-18981
> URL: https://issues.apache.org/jira/browse/HIVE-18981
> Project: Hive
>  Issue Type: Bug
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
>
> Beeline ignores the exception in catch block here:
> [https://github.com/apache/hive/blob/f40d94476718df6de42caef3b18811f024fe8713/beeline/src/java/org/apache/hive/beeline/BeeLine.java#L1017]
> here:
> [https://github.com/apache/hive/blob/f40d94476718df6de42caef3b18811f024fe8713/beeline/src/java/org/apache/hive/beeline/BeeLine.java#L1055]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-18981) Hive beeline ignores throwable

[Bharathkrishna Guruvayoor Murali (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharathkrishna Guruvayoor Murali reassigned HIVE-18981:
---


> Hive beeline ignores throwable
> --
>
> Key: HIVE-18981
> URL: https://issues.apache.org/jira/browse/HIVE-18981
> Project: Hive
>  Issue Type: Bug
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
>
> Beeline ignores the exception in catch block here:
> [https://github.com/apache/hive/blob/f40d94476718df6de42caef3b18811f024fe8713/beeline/src/java/org/apache/hive/beeline/BeeLine.java#L1017
> ]
> here:
> https://github.com/apache/hive/blob/f40d94476718df6de42caef3b18811f024fe8713/beeline/src/java/org/apache/hive/beeline/BeeLine.java#L1055



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402372#comment-16402372
 ] 

Hive QA commented on HIVE-18908:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  4m 
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
10s{color} | {color:green} The patch storage-api passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
16s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
20s{color} | {color:green} The patch serde passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  2m 
25s{color} | {color:red} root: The patch generated 376 new + 4248 unchanged - 
177 fixed = 4624 total (was 4425) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 9s{color} | {color:green} itests/hive-jmh: The patch generated 0 new + 11 
unchanged - 6 fixed = 11 total (was 17) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m 
12s{color} | {color:red} ql: The patch generated 376 new + 3170 unchanged - 171 
fixed = 3546 total (was 3341) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
1s{color} | {color:red} The patch has 13 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m  7s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9668/dev-support/hive-personality.sh
 |
| git revision | master / 6f38c9f |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9668/yetus/diff-checkstyle-root.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9668/yetus/diff-checkstyle-ql.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9668/yetus/whitespace-eol.txt 
|
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9668/yetus/patch-asflicense-problems.txt
 |
| modules | C: storage-api common serde . itests itests/hive-jmh ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9668/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Add support for FULL OUTER JOIN to MapJoin
> --
>
> Key: HIVE-18908
>

[jira] [Commented] (HIVE-18633) Service discovery for Active/Passive HA mode

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402358#comment-16402358
 ] 

Prasanth Jayachandran commented on HIVE-18633:
--

Attaching RB link. [~sershe] can you please review?

> Service discovery for Active/Passive HA mode
> 
>
> Key: HIVE-18633
> URL: https://issues.apache.org/jira/browse/HIVE-18633
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18633.1.patch
>
>
> HIVE-18281 adds Active/Passive HA mode for HS2. This jira is to extend the 
> feature so that clients will connect only to leader instance via JDBC + 
> service discovery. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18633) Service discovery for Active/Passive HA mode

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402357#comment-16402357
 ] 

Sergey Shelukhin commented on HIVE-18633:
-

[~prasanth_j] try rbt post... sometimes works when the UI doesn't

> Service discovery for Active/Passive HA mode
> 
>
> Key: HIVE-18633
> URL: https://issues.apache.org/jira/browse/HIVE-18633
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18633.1.patch
>
>
> HIVE-18281 adds Active/Passive HA mode for HS2. This jira is to extend the 
> feature so that clients will connect only to leader instance via JDBC + 
> service discovery. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18970) improve AM WM metrics for use in Grafana and such

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18970:

   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Committed to master. Thanks!

> improve AM WM metrics for use in Grafana and such
> -
>
> Key: HIVE-18970
> URL: https://issues.apache.org/jira/browse/HIVE-18970
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18970.patch
>
>
> AM metrics need to be tagged with something like dag ID.
> Also looks like they are not updated correctly in some cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18939) LLAP cache manageability improvements (local)

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18939:

Summary: LLAP cache manageability improvements (local)  (was: LLAP cache 
improvements (local))

> LLAP cache manageability improvements (local)
> -
>
> Key: HIVE-18939
> URL: https://issues.apache.org/jira/browse/HIVE-18939
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-16992) LLAP: monitoring and better default lambda for LRFU policy

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-16992:

   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Committed to master after some cluster smoke testing. Thanks for the review!

> LLAP: monitoring and better default lambda for LRFU policy
> --
>
> Key: HIVE-16992
> URL: https://issues.apache.org/jira/browse/HIVE-16992
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-16992.patch
>
>
> LRFU is currently skewed heavily towards LRU; there are 10k-s or 100k-s of 
> buffers tracked during a typical workload, but the heap size is around 700. 
> We should see if making it closer to LFU (by tweaking the lambda) will 
> improve hit rate with small queries infrequently interleaved with large 
> scans; and whether it will have negative effects due to perf overhead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18970) improve AM WM metrics for use in Grafana and such

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402349#comment-16402349
 ] 

Sergey Shelukhin commented on HIVE-18970:
-

There are some metrics available via LLAP tasks that are output in the console, 
but not from the AM...

> improve AM WM metrics for use in Grafana and such
> -
>
> Key: HIVE-18970
> URL: https://issues.apache.org/jira/browse/HIVE-18970
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18970.patch
>
>
> AM metrics need to be tagged with something like dag ID.
> Also looks like they are not updated correctly in some cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18930) Fix ppd_join5 test

[Kryvenko Igor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kryvenko Igor updated HIVE-18930:
-
Status: Patch Available  (was: Open)

> Fix ppd_join5 test
> --
>
> Key: HIVE-18930
> URL: https://issues.apache.org/jira/browse/HIVE-18930
> Project: Hive
>  Issue Type: Bug
>  Components: Test
>Reporter: Kryvenko Igor
>Assignee: Kryvenko Igor
>Priority: Major
> Attachments: HIVE-18930.01.patch
>
>
> It seems that HIVE-18068 have broken this test.
> Log from Jenkins
> {code:java}
>  type="java.lang.AssertionError">
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18930) Fix ppd_join5 test

[Kryvenko Igor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kryvenko Igor updated HIVE-18930:
-
Attachment: HIVE-18930.01.patch

> Fix ppd_join5 test
> --
>
> Key: HIVE-18930
> URL: https://issues.apache.org/jira/browse/HIVE-18930
> Project: Hive
>  Issue Type: Bug
>  Components: Test
>Reporter: Kryvenko Igor
>Assignee: Kryvenko Igor
>Priority: Major
> Attachments: HIVE-18930.01.patch
>
>
> It seems that HIVE-18068 have broken this test.
> Log from Jenkins
> {code:java}
>  type="java.lang.AssertionError">
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18976) Add ability to setup Druid Kafka Ingestion from Hive

[Nishant Bangarwa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nishant Bangarwa updated HIVE-18976:

Attachment: HIVE-18976.patch

> Add ability to setup Druid Kafka Ingestion from Hive
> 
>
> Key: HIVE-18976
> URL: https://issues.apache.org/jira/browse/HIVE-18976
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-18976.patch
>
>
> Add Ability to setup druid kafka Ingestion using Hive CREATE TABLE statement
> e.g. Below query can submit a kafka supervisor spec to the druid overlord and 
> druid can start ingesting events from kafka. 
> {code:java}
>  
> CREATE TABLE druid_kafka_test(`__time` timestamp, page string, language 
> string, `user` string, added int, deleted int, delta int)
> STORED BY 
> 'org.apache.hadoop.hive.druid.DruidKafkaStreamingStorageHandler'
> TBLPROPERTIES (
> "druid.segment.granularity" = "HOUR",
> "druid.query.granularity" = "MINUTE",
> "kafka.bootstrap.servers" = "localhost:9092",
> "kafka.topic" = "test-topic",
> "druid.kafka.ingest.useEarliestOffset" = "true"
> );
> {code}
> Design - This can be done via a DruidKafkaStreamingStorageHandler that 
> extends existing DruidStorageHandler and adds the additional functionality 
> for Streaming. 
> Testing - Add a DruidKafkaMiniCluster which will consist of DruidMiniCluster 
> + Single Node Kafka Broker. The broker can be populated with a test topic 
> that has some predefined data. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18976) Add ability to setup Druid Kafka Ingestion from Hive

[Nishant Bangarwa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nishant Bangarwa updated HIVE-18976:

Attachment: (was: HIVE-18976.patch)

> Add ability to setup Druid Kafka Ingestion from Hive
> 
>
> Key: HIVE-18976
> URL: https://issues.apache.org/jira/browse/HIVE-18976
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-18976.patch
>
>
> Add Ability to setup druid kafka Ingestion using Hive CREATE TABLE statement
> e.g. Below query can submit a kafka supervisor spec to the druid overlord and 
> druid can start ingesting events from kafka. 
> {code:java}
>  
> CREATE TABLE druid_kafka_test(`__time` timestamp, page string, language 
> string, `user` string, added int, deleted int, delta int)
> STORED BY 
> 'org.apache.hadoop.hive.druid.DruidKafkaStreamingStorageHandler'
> TBLPROPERTIES (
> "druid.segment.granularity" = "HOUR",
> "druid.query.granularity" = "MINUTE",
> "kafka.bootstrap.servers" = "localhost:9092",
> "kafka.topic" = "test-topic",
> "druid.kafka.ingest.useEarliestOffset" = "true"
> );
> {code}
> Design - This can be done via a DruidKafkaStreamingStorageHandler that 
> extends existing DruidStorageHandler and adds the additional functionality 
> for Streaming. 
> Testing - Add a DruidKafkaMiniCluster which will consist of DruidMiniCluster 
> + Single Node Kafka Broker. The broker can be populated with a test topic 
> that has some predefined data. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18739) Add support for Export from unpartitioned Acid table

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402309#comment-16402309
 ] 

Hive QA commented on HIVE-18739:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914807/HIVE-18739.08.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 13427 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)

[jira] [Commented] (HIVE-18975) NPE when inserting NULL value in structure and array with HBase table

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402301#comment-16402301
 ] 

Ashutosh Chauhan commented on HIVE-18975:
-

+1 pending test.

> NPE when inserting NULL value in structure and array with HBase table
> -
>
> Key: HIVE-18975
> URL: https://issues.apache.org/jira/browse/HIVE-18975
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18975.1.patch
>
>
> STR (Structure)
> *STEP 1. Create tables*
> {code}
> CREATE TABLE IF NOT EXISTS t1 (id INT);
> INSERT INTO TABLE t1 VALUES (1),(2),(3),(4),(5);
> CREATE TABLE IF NOT EXISTS `htable`(
>   `id` INT, 
>   `map_column` STRUCT) ROW FORMAT 
> SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe'  STORED BY 
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'  WITH SERDEPROPERTIES (   
> 'hbase.columns.mapping'=':key,id:id','serialization.format'='1') 
> TBLPROPERTIES ( 'hbase.table.name'='tmp/h');
> {code}
> *STEP 2. Insert into table stored in HBase the struct with NULL value in it*
> {code}
> INSERT INTO `htable` SELECT 2,NAMED_STRUCT("s_int",CAST(NULL AS 
> INT),"s_string","s1","s_date",CAST('2018-03-12' AS DATE)) FROM t1 LIMIT 1;
> {code}
> *ACTUAL RESULT*
> The query fails with NPE.
> {code}
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row (tag=0) 
> {"key":{},"value":{"_col0":2,"_col1":{"s_int":null,"s_string":"s1","s_date":"2018-03-12"}}}
>   at 
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:257)
>   at 
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
>   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
>   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1631)
>   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row (tag=0) 
> {"key":{},"value":{"_col0":2,"_col1":{"s_int":null,"s_string":"s1","s_date":"2018-03-12"}}}
>   at 
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:245)
>   ... 7 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:787)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
>   at 
> org.apache.hadoop.hive.ql.exec.LimitOperator.process(LimitOperator.java:63)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
>   at 
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
>   at 
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:236)
>   ... 7 more
> Caused by: org.apache.hadoop.hive.serde2.SerDeException: 
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:301)
>   at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:714)
>   ... 12 more
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:36)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:239)
>   at 
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:236)
>   at 
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:295)
>   at 
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:222)
>   at 
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serializeField(HBaseRowSerializer.java:194)
>   at 
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:118)
>   at 
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:297)
>   ... 13 more
> {code}
> *EXPECTED RESULT*
> The query finished successfully.
> STR (arrays)
> *STEP 1. Create tables*
> {code}
> CREATE TABLE hbase_list(id INT, list_column array) ROW FORMAT SERDE 
> 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY 
> 

[jira] [Updated] (HIVE-18962) add WM task state to Tez AM heartbeat

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18962:

   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Committed to master. Thanks for the review!

> add WM task state to Tez AM heartbeat
> -
>
> Key: HIVE-18962
> URL: https://issues.apache.org/jira/browse/HIVE-18962
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18962.01.patch, HIVE-18962.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (HIVE-18979) Enable AggregateReduceFunctionsRule from Calcite

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-18979 started by Jesus Camacho Rodriguez.
--
> Enable AggregateReduceFunctionsRule from Calcite
> 
>
> Key: HIVE-18979
> URL: https://issues.apache.org/jira/browse/HIVE-18979
> Project: Hive
>  Issue Type: Task
>  Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> The rule rewrites aggregate functions into simpler ones. It should help us 
> gathering more common sub-expressions for a given plan, and it should have a 
> positive effect on materialized view rewriting coverage too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-18979) Enable AggregateReduceFunctionsRule from Calcite

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez reassigned HIVE-18979:
--


> Enable AggregateReduceFunctionsRule from Calcite
> 
>
> Key: HIVE-18979
> URL: https://issues.apache.org/jira/browse/HIVE-18979
> Project: Hive
>  Issue Type: Task
>  Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> The rule rewrites aggregate functions into simpler ones. It should help us 
> gathering more common sub-expressions for a given plan, and it should have a 
> positive effect on materialized view rewriting coverage too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18264) CachedStore: Store cached partitions/col stats within the table cache and make prewarm non-blocking

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-18264:

Attachment: HIVE-18264.8.patch

> CachedStore: Store cached partitions/col stats within the table cache and 
> make prewarm non-blocking
> ---
>
> Key: HIVE-18264
> URL: https://issues.apache.org/jira/browse/HIVE-18264
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-18264.1.patch, HIVE-18264.2.patch, 
> HIVE-18264.3.patch, HIVE-18264.4.patch, HIVE-18264.5.patch, 
> HIVE-18264.6.patch, HIVE-18264.7.patch, HIVE-18264.8.patch, HIVE-18264.8.patch
>
>
> Currently we have a separate cache for partitions and partition col stats 
> which results in some calls iterating through each of these for 
> retrieving/updating. For example, to modify a partition col stat, currently 
> we need to lock table, partition and partition col stats caches which are all 
> separate hashmaps. We can get better performance by organizing 
> hierarchically. For example, we can have a partition, partition col stats and 
> table col stats cache per table to improve on the previous mechanisms. This 
> will also result in better concurrency, since now instead of locking the 
> whole cache, we can selectively lock the table cache and modify multiple 
> tables in parallel. 
> In addition, currently, the prewarm mechanism populates all the caches 
> initially (it skips tables that do not pass whitelist/blacklist filter) and 
> it is a blocking call. This patch also makes prewarm non-blocking so that the 
> calls for tables that are already cached can be served from the memory and 
> the ones that are not can be served from the rdbms. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18264) CachedStore: Store cached partitions/col stats within the table cache and make prewarm non-blocking

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402237#comment-16402237
 ] 

Vaibhav Gumashta commented on HIVE-18264:
-

Reattaching v8 for test run. Looks like last one wasn't picked

> CachedStore: Store cached partitions/col stats within the table cache and 
> make prewarm non-blocking
> ---
>
> Key: HIVE-18264
> URL: https://issues.apache.org/jira/browse/HIVE-18264
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-18264.1.patch, HIVE-18264.2.patch, 
> HIVE-18264.3.patch, HIVE-18264.4.patch, HIVE-18264.5.patch, 
> HIVE-18264.6.patch, HIVE-18264.7.patch, HIVE-18264.8.patch, HIVE-18264.8.patch
>
>
> Currently we have a separate cache for partitions and partition col stats 
> which results in some calls iterating through each of these for 
> retrieving/updating. For example, to modify a partition col stat, currently 
> we need to lock table, partition and partition col stats caches which are all 
> separate hashmaps. We can get better performance by organizing 
> hierarchically. For example, we can have a partition, partition col stats and 
> table col stats cache per table to improve on the previous mechanisms. This 
> will also result in better concurrency, since now instead of locking the 
> whole cache, we can selectively lock the table cache and modify multiple 
> tables in parallel. 
> In addition, currently, the prewarm mechanism populates all the caches 
> initially (it skips tables that do not pass whitelist/blacklist filter) and 
> it is a blocking call. This patch also makes prewarm non-blocking so that the 
> calls for tables that are already cached can be served from the memory and 
> the ones that are not can be served from the rdbms. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18957) Upgrade Calcite version to 1.16.0

[Jesus Camacho Rodriguez (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402231#comment-16402231
 ] 

Jesus Camacho Rodriguez commented on HIVE-18957:


[~bslim], I have regenerated a few q files and uploaded a new patch 
{{HIVE-18957.02.patch}}. The following tests still have issues:
{noformat}
* org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_timeseries]
  => org.apache.hadoop.hive.ql.parse.SemanticException: Line 0:-1 Invalid 
column reference 'timestamp'
* 
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1]
  => Same exception as previous example
* 
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz]
  => Wrong timestamp printing
* org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_mv]
  => q file changes in L617, were there any changes to 'double' type handling 
for Druid that need changes in Hive?
{noformat}
Please let me know if your follow-up patches for Hive fix these issues and if 
we can bring part of them to fix these tests at least temporarily? If you want 
me to look at any of the failures, let me know too.

> Upgrade Calcite version to 1.16.0
> -
>
> Key: HIVE-18957
> URL: https://issues.apache.org/jira/browse/HIVE-18957
> Project: Hive
>  Issue Type: Task
>  Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18957.01.patch, HIVE-18957.02.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18957) Upgrade Calcite version to 1.16.0

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-18957:
---
Attachment: HIVE-18957.02.patch

> Upgrade Calcite version to 1.16.0
> -
>
> Key: HIVE-18957
> URL: https://issues.apache.org/jira/browse/HIVE-18957
> Project: Hive
>  Issue Type: Task
>  Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18957.01.patch, HIVE-18957.02.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18739) Add support for Export from unpartitioned Acid table

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402216#comment-16402216
 ] 

Hive QA commented on HIVE-18739:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
41s{color} | {color:red} ql: The patch generated 31 new + 669 unchanged - 7 
fixed = 700 total (was 676) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 40s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9667/dev-support/hive-personality.sh
 |
| git revision | master / 21c6a54 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9667/yetus/diff-checkstyle-ql.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9667/yetus/patch-asflicense-problems.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9667/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Add support for Export from unpartitioned Acid table
> 
>
> Key: HIVE-18739
> URL: https://issues.apache.org/jira/browse/HIVE-18739
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18739.01.patch, HIVE-18739.04.patch, 
> HIVE-18739.04.patch, HIVE-18739.06.patch, HIVE-18739.08.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-18978) ConditionalTask.addDependentTask(Task t) adds t in the wrong place

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman reassigned HIVE-18978:
-

Assignee: Eugene Koifman

> ConditionalTask.addDependentTask(Task t) adds t in the wrong place
> --
>
> Key: HIVE-18978
> URL: https://issues.apache.org/jira/browse/HIVE-18978
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> {{ConditionalTask.addDependentTask(Task t) }} is implemented like this:
> {noformat}
> /**
> * Add a dependent task on the current conditional task. The task will not be 
> a direct child of
> * conditional task. Actually it will be added as child task of associated 
> tasks.
> *
> * @return true if the task got added false if it already existed
> */
> @Override
> public boolean addDependentTask(Task dependent) {
>   boolean ret = false;
>   if (getListTasks() != null) {
> ret = true;
> for (Task tsk : getListTasks()) {
>   ret = ret & tsk.addDependentTask(dependent);
> }
>   }
>   return ret;
> }
> {noformat}
>  So let’s say, the tasks in the ConditionalTask are A,B,C, but they have 
> children.
> {noformat}
> CondTask
>   |--A
>  |--A1
> |-A2
>   |--B
>  |--B1
>   |--C
> |--C1
> {noformat}
>  The way ConditionalTask.addDependent() is implemented, MyTask becomes a 
> sibling of A1,
>  B1 and C1.  So even if only 1 branch of ConditionalTask is executed (and 
> parallel task
>  execution is enabled), there is no guarantee (as I see) that MyTask runs 
> after A2 or
>  B1 or C1, which is really what is needed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18972) beeline command suggestion to kill job deprecated

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402185#comment-16402185
 ] 

Hive QA commented on HIVE-18972:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914805/HIVE-18972.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 13025 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=94)

[jira] [Commented] (HIVE-18977) Listing partitions returns different results with JDO and direct SQL

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402149#comment-16402149
 ] 

Alan Gates commented on HIVE-18977:
---

The following tests fail:

testListPartitionsAll: listPartitions(x, y, 0) returns no partitions with 
directSQL but does return partitions in JDO

testListPartitionSpecs: same as testListPartitionsAll

testListPartitionsByFilter: listPartitionsByFilter is not case sensitive in 
directSQL but is in JDO

testListPartitionSpecsByFilter and testGetNumPartitionsByFilter: same as 
testListPartitionsByFilter

> Listing partitions returns different results with JDO and direct SQL
> 
>
> Key: HIVE-18977
> URL: https://issues.apache.org/jira/browse/HIVE-18977
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Priority: Major
>
> Some of the tests in TestListPartitions fail when using JDO instead of direct 
> SQL.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18781) Create/Replicate Open, Commit (without writes) and Abort Txn events

[mahesh kumar behera (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera updated HIVE-18781:
---
Attachment: HIVE-18781.04.patch

> Create/Replicate Open, Commit (without writes) and Abort Txn events
> ---
>
> Key: HIVE-18781
> URL: https://issues.apache.org/jira/browse/HIVE-18781
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl, Transactions
>Affects Versions: 3.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18781.01.patch, HIVE-18781.02.patch, 
> HIVE-18781.03.patch, HIVE-18781.04.patch
>
>
> *EVENT_OPEN_TXN:*
>  *Source Warehouse:*
>  - Create new event type EVENT_OPEN_TXN with related message format etc.
>  - When any transaction is opened either by auto-commit mode or 
> multi-statement mode, need to capture this event.
>  - Repl dump should read this event from EventNotificationTable and dump the 
> message.
> *Target Warehouse:*
>  - Repl load should read the event from the dump and get the message.
>  - Open a txn in target warehouse.
>  - Create a map of source txn ID against target txn ID and persist the same 
> in metastore. There should be one map per replication policy (DBName.* incase 
> of DB level replication, DBName.TableName incase of table level replication)
>  
> *EVENT_COMMIT_TXN (Without writes)*
> Add new EVENT_COMMIT_TXN to log the metadata/data of all tables/partitions 
> modified within the txn.
> *Source warehouse:*
>  - Create EVENT_COMMIT_TXN event type with corresponding message format etc.
> *Target warehouse:*
>  - Repl load should read this event from the dump.
>  - Validate the source txn ID from the event using the Source-Target Txn ID 
> map maintained in target metastore. Also, need to check if corresponding 
> target txn ID is valid.
>  - If valid, then apply the event and commit the corresponding target 
> transaction.
>  - This new event should be idempotent such that if it is applied twice, then 
> second time it should be loop.
>  
> *EVENT_ABORT_TXN*
>  Source Warehouse:
>  - Create new event type EVENT_ABORT_TXN with related message format etc.
>  - Capture this event when abort the txn.
>  - Repl dump should read this event from EventNotificationTable and dump the 
> message.
> *Target Warehouse:*
>  - Repl load should read the event from the dump and get the message.
>  - Validate if source txn ID from the event is there in the source-target txn 
> ID map. If not there, just noop the event.
>  - If valid, then Abort the corresponding target txn and remove the entry 
> from source-target txn ID map.
> All these new events should be idempotent such that if it is applied twice, 
> then second time it should be noop.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18972) beeline command suggestion to kill job deprecated

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402093#comment-16402093
 ] 

Hive QA commented on HIVE-18972:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 16s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9666/dev-support/hive-personality.sh
 |
| git revision | master / 21c6a54 |
| Default Java | 1.8.0_111 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9666/yetus/patch-asflicense-problems.txt
 |
| modules | C: common ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9666/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> beeline command suggestion to kill job deprecated
> -
>
> Key: HIVE-18972
> URL: https://issues.apache.org/jira/browse/HIVE-18972
> Project: Hive
>  Issue Type: Bug
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Minor
> Attachments: HIVE-18972.1.patch
>
>
> When I run a beeline command that uses YARN:
> {code}
> INFO : The url to track the job: 
> http://vd0514.halxg.cloudera.com:8088/proxy/application_1488996234407_0010/
> INFO : Starting Job = job_1488996234407_0010, Tracking URL = 
> http://vd0514.halxg.cloudera.com:8088/proxy/application_1488996234407_0010/
> INFO : Kill Command = 
> /opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.9/lib/hadoop/bin/hadoop job 
> -kill job_1488996234407_0010
> {code}
> If I then try to kill the job using that command:
> {code}
> [systest@vd0514 ~]$ 
> /opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.9/lib/hadoop/bin/hadoop job 
> -kill job_1488996234407_0010
> DEPRECATED: Use of this script to execute mapred command is deprecated.
> Instead use the mapred command for it.
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18976) Add ability to setup Druid Kafka Ingestion from Hive

[Nishant Bangarwa (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402089#comment-16402089
 ] 

Nishant Bangarwa commented on HIVE-18976:
-

+cc [~ashutoshc] [~jcamachorodriguez] @b-slim Please review. 

> Add ability to setup Druid Kafka Ingestion from Hive
> 
>
> Key: HIVE-18976
> URL: https://issues.apache.org/jira/browse/HIVE-18976
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-18976.patch
>
>
> Add Ability to setup druid kafka Ingestion using Hive CREATE TABLE statement
> e.g. Below query can submit a kafka supervisor spec to the druid overlord and 
> druid can start ingesting events from kafka. 
> {code:java}
>  
> CREATE TABLE druid_kafka_test(`__time` timestamp, page string, language 
> string, `user` string, added int, deleted int, delta int)
> STORED BY 
> 'org.apache.hadoop.hive.druid.DruidKafkaStreamingStorageHandler'
> TBLPROPERTIES (
> "druid.segment.granularity" = "HOUR",
> "druid.query.granularity" = "MINUTE",
> "kafka.bootstrap.servers" = "localhost:9092",
> "kafka.topic" = "test-topic",
> "druid.kafka.ingest.useEarliestOffset" = "true"
> );
> {code}
> Design - This can be done via a DruidKafkaStreamingStorageHandler that 
> extends existing DruidStorageHandler and adds the additional functionality 
> for Streaming. 
> Testing - Add a DruidKafkaMiniCluster which will consist of DruidMiniCluster 
> + Single Node Kafka Broker. The broker can be populated with a test topic 
> that has some predefined data. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18976) Add ability to setup Druid Kafka Ingestion from Hive

[Nishant Bangarwa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nishant Bangarwa updated HIVE-18976:

Attachment: HIVE-18976.patch

> Add ability to setup Druid Kafka Ingestion from Hive
> 
>
> Key: HIVE-18976
> URL: https://issues.apache.org/jira/browse/HIVE-18976
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-18976.patch
>
>
> Add Ability to setup druid kafka Ingestion using Hive CREATE TABLE statement
> e.g. Below query can submit a kafka supervisor spec to the druid overlord and 
> druid can start ingesting events from kafka. 
> {code:java}
>  
> CREATE TABLE druid_kafka_test(`__time` timestamp, page string, language 
> string, `user` string, added int, deleted int, delta int)
> STORED BY 
> 'org.apache.hadoop.hive.druid.DruidKafkaStreamingStorageHandler'
> TBLPROPERTIES (
> "druid.segment.granularity" = "HOUR",
> "druid.query.granularity" = "MINUTE",
> "kafka.bootstrap.servers" = "localhost:9092",
> "kafka.topic" = "test-topic",
> "druid.kafka.ingest.useEarliestOffset" = "true"
> );
> {code}
> Design - This can be done via a DruidKafkaStreamingStorageHandler that 
> extends existing DruidStorageHandler and adds the additional functionality 
> for Streaming. 
> Testing - Add a DruidKafkaMiniCluster which will consist of DruidMiniCluster 
> + Single Node Kafka Broker. The broker can be populated with a test topic 
> that has some predefined data. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18976) Add ability to setup Druid Kafka Ingestion from Hive

[Nishant Bangarwa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nishant Bangarwa updated HIVE-18976:

Status: Patch Available  (was: Open)

> Add ability to setup Druid Kafka Ingestion from Hive
> 
>
> Key: HIVE-18976
> URL: https://issues.apache.org/jira/browse/HIVE-18976
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-18976.patch
>
>
> Add Ability to setup druid kafka Ingestion using Hive CREATE TABLE statement
> e.g. Below query can submit a kafka supervisor spec to the druid overlord and 
> druid can start ingesting events from kafka. 
> {code:java}
>  
> CREATE TABLE druid_kafka_test(`__time` timestamp, page string, language 
> string, `user` string, added int, deleted int, delta int)
> STORED BY 
> 'org.apache.hadoop.hive.druid.DruidKafkaStreamingStorageHandler'
> TBLPROPERTIES (
> "druid.segment.granularity" = "HOUR",
> "druid.query.granularity" = "MINUTE",
> "kafka.bootstrap.servers" = "localhost:9092",
> "kafka.topic" = "test-topic",
> "druid.kafka.ingest.useEarliestOffset" = "true"
> );
> {code}
> Design - This can be done via a DruidKafkaStreamingStorageHandler that 
> extends existing DruidStorageHandler and adds the additional functionality 
> for Streaming. 
> Testing - Add a DruidKafkaMiniCluster which will consist of DruidMiniCluster 
> + Single Node Kafka Broker. The broker can be populated with a test topic 
> that has some predefined data. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-18976) Add ability to setup Druid Kafka Ingestion from Hive

[Nishant Bangarwa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nishant Bangarwa reassigned HIVE-18976:
---


> Add ability to setup Druid Kafka Ingestion from Hive
> 
>
> Key: HIVE-18976
> URL: https://issues.apache.org/jira/browse/HIVE-18976
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
>
> Add Ability to setup druid kafka Ingestion using Hive CREATE TABLE statement
> e.g. Below query can submit a kafka supervisor spec to the druid overlord and 
> druid can start ingesting events from kafka. 
> {code:java}
>  
> CREATE TABLE druid_kafka_test(`__time` timestamp, page string, language 
> string, `user` string, added int, deleted int, delta int)
> STORED BY 
> 'org.apache.hadoop.hive.druid.DruidKafkaStreamingStorageHandler'
> TBLPROPERTIES (
> "druid.segment.granularity" = "HOUR",
> "druid.query.granularity" = "MINUTE",
> "kafka.bootstrap.servers" = "localhost:9092",
> "kafka.topic" = "test-topic",
> "druid.kafka.ingest.useEarliestOffset" = "true"
> );
> {code}
> Design - This can be done via a DruidKafkaStreamingStorageHandler that 
> extends existing DruidStorageHandler and adds the additional functionality 
> for Streaming. 
> Testing - Add a DruidKafkaMiniCluster which will consist of DruidMiniCluster 
> + Single Node Kafka Broker. The broker can be populated with a test topic 
> that has some predefined data. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18957) Upgrade Calcite version to 1.16.0

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402062#comment-16402062
 ] 

Hive QA commented on HIVE-18957:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914532/HIVE-18957.01.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 120 failed/errored test(s), 13422 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=94)

[jira] [Commented] (HIVE-18747) Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402028#comment-16402028
 ] 

ASF GitHub Bot commented on HIVE-18747:
---

GitHub user sankarh opened a pull request:

https://github.com/apache/hive/pull/321

HIVE-18747: Cleaner for TXN_TO_WRITE_ID table entries/MIN_HISTORY_LEVEL.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sankarh/hive HIVE-18747

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/321.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #321


commit 3cba53bbf711dd6ea17c271c38c7fb6aefb17925
Author: Sankar Hariappan 
Date:   2018-03-12T20:58:53Z

HIVE-18747: Cleaner for TXN_TO_WRITE_ID table entries/MIN_HISTORY_LEVEL.




> Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.
> --
>
> Key: HIVE-18747
> URL: https://issues.apache.org/jira/browse/HIVE-18747
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Minor
>  Labels: ACID, pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18747.01.patch
>
>
> Per table write ID implementation (HIVE-18192) maintains a map between txn ID 
> and table write ID in TXN_TO_WRITE_ID meta table. 
> The entries in this table is used to generate ValidWriteIdList for the given 
> ValidTxnList to ensure snapshot isolation. 
> When table or database is dropped, then these entries are cleaned-up. But, it 
> is necessary to clean-up for active tables too for better performance.
> Need to have another table MIN_HISTORY_LEVEL to maintain the least txn which 
> is referred by any active ValidTxnList snapshot as open/aborted txn. If no 
> references found in this table for any txn, then it is eligible for cleanup.
> After clean-up, need to maintain just one entry (highest txn <= 
> min_uncommitted_txn) per table to mark as LWM (low water mark).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18747) Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-18747:
--
Labels: ACID pull-request-available  (was: ACID)

> Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.
> --
>
> Key: HIVE-18747
> URL: https://issues.apache.org/jira/browse/HIVE-18747
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Minor
>  Labels: ACID, pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18747.01.patch
>
>
> Per table write ID implementation (HIVE-18192) maintains a map between txn ID 
> and table write ID in TXN_TO_WRITE_ID meta table. 
> The entries in this table is used to generate ValidWriteIdList for the given 
> ValidTxnList to ensure snapshot isolation. 
> When table or database is dropped, then these entries are cleaned-up. But, it 
> is necessary to clean-up for active tables too for better performance.
> Need to have another table MIN_HISTORY_LEVEL to maintain the least txn which 
> is referred by any active ValidTxnList snapshot as open/aborted txn. If no 
> references found in this table for any txn, then it is eligible for cleanup.
> After clean-up, need to maintain just one entry (highest txn <= 
> min_uncommitted_txn) per table to mark as LWM (low water mark).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18747) Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.

[Sankar Hariappan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402027#comment-16402027
 ] 

Sankar Hariappan commented on HIVE-18747:
-

Attached 01.patch with below changes.
 # Added MIN_HISTORY_LEVEL to mark the min_uncommitted_txn referred by any 
ongoing txn.
 # If any txn aborted, it would retain an entry in MIN_HISTORY_LEVEL to avoid 
any future txn to read aborted data. Now, compactor would remove this entry 
once the aborted delta directories are deleted by cleaner.
 # If no uncommitted txn in the system when we take the snapshot, it means all 
data are committed but to avoid future txn from reading data written by current 
txn, an entry would be added by current txn to mark itself as uncommitted.
 # Cleaner would use MIN_HISTORY_LEVEL table and cleanup the entries in 
TXN_TO_WRITE_ID.
 ## It would retain one entry (highest txn <= min_uncommitted_txn) as LWM for 
preserving writeId_hwm for future txns.
 ## The min_uncommitted_txn calculation follows the below elimination logic.
 ### Assume  min_uncommitted_txn = latest txn Id allocated in the system.
 ### If MIN_HISTORY_LEVEL is non-empty, then overwrite min_uncommitted_txn = 
min(uncommitted_txn).
 ### Else if MIN_HISTORY_LEVEL is empty (if all txns are committed), then 
traverse through TXNS table to find min_uncommitted_txn = min(open_txn). 

[~ekoifman], can you please review the patch?

> Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.
> --
>
> Key: HIVE-18747
> URL: https://issues.apache.org/jira/browse/HIVE-18747
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Minor
>  Labels: ACID
> Fix For: 3.0.0
>
> Attachments: HIVE-18747.01.patch
>
>
> Per table write ID implementation (HIVE-18192) maintains a map between txn ID 
> and table write ID in TXN_TO_WRITE_ID meta table. 
> The entries in this table is used to generate ValidWriteIdList for the given 
> ValidTxnList to ensure snapshot isolation. 
> When table or database is dropped, then these entries are cleaned-up. But, it 
> is necessary to clean-up for active tables too for better performance.
> Need to have another table MIN_HISTORY_LEVEL to maintain the least txn which 
> is referred by any active ValidTxnList snapshot as open/aborted txn. If no 
> references found in this table for any txn, then it is eligible for cleanup.
> After clean-up, need to maintain just one entry (highest txn <= 
> min_uncommitted_txn) per table to mark as LWM (low water mark).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18747) Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-18747:

Status: Patch Available  (was: Open)

> Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.
> --
>
> Key: HIVE-18747
> URL: https://issues.apache.org/jira/browse/HIVE-18747
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Minor
>  Labels: ACID
> Fix For: 3.0.0
>
> Attachments: HIVE-18747.01.patch
>
>
> Per table write ID implementation (HIVE-18192) maintains a map between txn ID 
> and table write ID in TXN_TO_WRITE_ID meta table. 
> The entries in this table is used to generate ValidWriteIdList for the given 
> ValidTxnList to ensure snapshot isolation. 
> When table or database is dropped, then these entries are cleaned-up. But, it 
> is necessary to clean-up for active tables too for better performance.
> Need to have another table MIN_HISTORY_LEVEL to maintain the least txn which 
> is referred by any active ValidTxnList snapshot as open/aborted txn. If no 
> references found in this table for any txn, then it is eligible for cleanup.
> After clean-up, need to maintain just one entry (highest txn <= 
> min_uncommitted_txn) per table to mark as LWM (low water mark).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18747) Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-18747:

Description: 
Per table write ID implementation (HIVE-18192) maintains a map between txn ID 
and table write ID in TXN_TO_WRITE_ID meta table. 

The entries in this table is used to generate ValidWriteIdList for the given 
ValidTxnList to ensure snapshot isolation. 

When table or database is dropped, then these entries are cleaned-up. But, it 
is necessary to clean-up for active tables too for better performance.

Need to have another table MIN_HISTORY_LEVEL to maintain the least txn which is 
referred by any active ValidTxnList snapshot as open/aborted txn. If no 
references found in this table for any txn, then it is eligible for cleanup.

After clean-up, need to maintain just one entry (highest txn <= 
min_uncommitted_txn) per table to mark as LWM (low water mark).

  was:
Per table write ID implementation (HIVE-18192) maintains a map between txn ID 
and table write ID in TXN_TO_WRITE_ID meta table. 

The entries in this table is used to generate ValidWriteIdList for the given 
ValidTxnList to ensure snapshot isolation. 

When table or database is dropped, then these entries are cleaned-up. But, it 
is necessary to clean-up for active tables too for better performance.

Need to have another table MIN_HISTORY_LEVEL to maintain the least txn which is 
referred by any active ValidTxnList snapshot as open/aborted txn. If no 
references found in this table for any txn, then it is eligible for cleanup.

After clean-up, need to maintain just one entry (highest committed txn) per 
table to mark as LWM (low water mark).


> Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.
> --
>
> Key: HIVE-18747
> URL: https://issues.apache.org/jira/browse/HIVE-18747
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Minor
>  Labels: ACID
> Fix For: 3.0.0
>
> Attachments: HIVE-18747.01.patch
>
>
> Per table write ID implementation (HIVE-18192) maintains a map between txn ID 
> and table write ID in TXN_TO_WRITE_ID meta table. 
> The entries in this table is used to generate ValidWriteIdList for the given 
> ValidTxnList to ensure snapshot isolation. 
> When table or database is dropped, then these entries are cleaned-up. But, it 
> is necessary to clean-up for active tables too for better performance.
> Need to have another table MIN_HISTORY_LEVEL to maintain the least txn which 
> is referred by any active ValidTxnList snapshot as open/aborted txn. If no 
> references found in this table for any txn, then it is eligible for cleanup.
> After clean-up, need to maintain just one entry (highest txn <= 
> min_uncommitted_txn) per table to mark as LWM (low water mark).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18747) Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-18747:

Attachment: HIVE-18747.01.patch

> Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.
> --
>
> Key: HIVE-18747
> URL: https://issues.apache.org/jira/browse/HIVE-18747
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Minor
>  Labels: ACID
> Fix For: 3.0.0
>
> Attachments: HIVE-18747.01.patch
>
>
> Per table write ID implementation (HIVE-18192) maintains a map between txn ID 
> and table write ID in TXN_TO_WRITE_ID meta table. 
> The entries in this table is used to generate ValidWriteIdList for the given 
> ValidTxnList to ensure snapshot isolation. 
> When table or database is dropped, then these entries are cleaned-up. But, it 
> is necessary to clean-up for active tables too for better performance.
> Need to have another table MIN_HISTORY_LEVEL to maintain the least txn which 
> is referred by any active ValidTxnList snapshot as open/aborted txn. If no 
> references found in this table for any txn, then it is eligible for cleanup.
> After clean-up, need to maintain just one entry (highest committed txn) per 
> table to mark as LWM (low water mark).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18957) Upgrade Calcite version to 1.16.0

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402013#comment-16402013
 ] 

Hive QA commented on HIVE-18957:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m  
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  5m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
20s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 26s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  xml  compile  findbugs  
checkstyle  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9665/dev-support/hive-personality.sh
 |
| git revision | master / 21c6a54 |
| Default Java | 1.8.0_111 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9665/yetus/patch-asflicense-problems.txt
 |
| modules | C: . ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9665/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Upgrade Calcite version to 1.16.0
> -
>
> Key: HIVE-18957
> URL: https://issues.apache.org/jira/browse/HIVE-18957
> Project: Hive
>  Issue Type: Task
>  Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18957.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work stopped] (HIVE-18747) Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-18747 stopped by Sankar Hariappan.
---
> Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.
> --
>
> Key: HIVE-18747
> URL: https://issues.apache.org/jira/browse/HIVE-18747
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Minor
>  Labels: ACID
> Fix For: 3.0.0
>
>
> Per table write ID implementation (HIVE-18192) maintains a map between txn ID 
> and table write ID in TXN_TO_WRITE_ID meta table. 
> The entries in this table is used to generate ValidWriteIdList for the given 
> ValidTxnList to ensure snapshot isolation. 
> When table or database is dropped, then these entries are cleaned-up. But, it 
> is necessary to clean-up for active tables too for better performance.
> Need to have another table MIN_HISTORY_LEVEL to maintain the least txn which 
> is referred by any active ValidTxnList snapshot as open/aborted txn. If no 
> references found in this table for any txn, then it is eligible for cleanup.
> After clean-up, need to maintain just one entry (highest committed txn) per 
> table to mark as LWM (low water mark).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18970) improve AM WM metrics for use in Grafana and such

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401955#comment-16401955
 ] 

Hive QA commented on HIVE-18970:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914789/HIVE-18970.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 13025 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=94)

[jira] [Commented] (HIVE-18970) improve AM WM metrics for use in Grafana and such

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401850#comment-16401850
 ] 

Hive QA commented on HIVE-18970:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m  
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
10s{color} | {color:red} llap-tez: The patch generated 1 new + 74 unchanged - 0 
fixed = 75 total (was 74) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m  
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}  9m  5s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9663/dev-support/hive-personality.sh
 |
| git revision | master / 21c6a54 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9663/yetus/diff-checkstyle-llap-tez.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9663/yetus/patch-asflicense-problems.txt
 |
| modules | C: llap-tez U: llap-tez |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9663/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> improve AM WM metrics for use in Grafana and such
> -
>
> Key: HIVE-18970
> URL: https://issues.apache.org/jira/browse/HIVE-18970
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18970.patch
>
>
> AM metrics need to be tagged with something like dag ID.
> Also looks like they are not updated correctly in some cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18915) Better client logging when a HoS session can't be opened

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401831#comment-16401831
 ] 

Hive QA commented on HIVE-18915:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914783/HIVE-18915.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 13025 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=94)

[jira] [Commented] (HIVE-18975) NPE when inserting NULL value in structure and array with HBase table

[Oleksiy Sayankin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401790#comment-16401790
 ] 

Oleksiy Sayankin commented on HIVE-18975:
-

[~ychena], [~Yibing], [~aihuaxu] please review.

> NPE when inserting NULL value in structure and array with HBase table
> -
>
> Key: HIVE-18975
> URL: https://issues.apache.org/jira/browse/HIVE-18975
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18975.1.patch
>
>
> STR (Structure)
> *STEP 1. Create tables*
> {code}
> CREATE TABLE IF NOT EXISTS t1 (id INT);
> INSERT INTO TABLE t1 VALUES (1),(2),(3),(4),(5);
> CREATE TABLE IF NOT EXISTS `htable`(
>   `id` INT, 
>   `map_column` STRUCT) ROW FORMAT 
> SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe'  STORED BY 
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'  WITH SERDEPROPERTIES (   
> 'hbase.columns.mapping'=':key,id:id','serialization.format'='1') 
> TBLPROPERTIES ( 'hbase.table.name'='tmp/h');
> {code}
> *STEP 2. Insert into table stored in HBase the struct with NULL value in it*
> {code}
> INSERT INTO `htable` SELECT 2,NAMED_STRUCT("s_int",CAST(NULL AS 
> INT),"s_string","s1","s_date",CAST('2018-03-12' AS DATE)) FROM t1 LIMIT 1;
> {code}
> *ACTUAL RESULT*
> The query fails with NPE.
> {code}
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row (tag=0) 
> {"key":{},"value":{"_col0":2,"_col1":{"s_int":null,"s_string":"s1","s_date":"2018-03-12"}}}
>   at 
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:257)
>   at 
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
>   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
>   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1631)
>   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row (tag=0) 
> {"key":{},"value":{"_col0":2,"_col1":{"s_int":null,"s_string":"s1","s_date":"2018-03-12"}}}
>   at 
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:245)
>   ... 7 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:787)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
>   at 
> org.apache.hadoop.hive.ql.exec.LimitOperator.process(LimitOperator.java:63)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
>   at 
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
>   at 
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:236)
>   ... 7 more
> Caused by: org.apache.hadoop.hive.serde2.SerDeException: 
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:301)
>   at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:714)
>   ... 12 more
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:36)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:239)
>   at 
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:236)
>   at 
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:295)
>   at 
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:222)
>   at 
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serializeField(HBaseRowSerializer.java:194)
>   at 
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:118)
>   at 
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:297)
>   ... 13 more
> {code}
> *EXPECTED RESULT*
> The query finished successfully.
> STR (arrays)
> *STEP 1. Create tables*
> {code}
> CREATE TABLE hbase_list(id INT, list_column array) ROW FORMAT SERDE 
> 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY 
>