[jira] [Updated] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-18622:

Status: Patch Available  (was: In Progress)

> Vectorization: IF Statements, Comparisons, and more do not handle NULLs 
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.03.patch, HIVE-18622.04.patch, 
> HIVE-18622.05.patch, HIVE-18622.06.patch, HIVE-18622.07.patch, 
> HIVE-18622.08.patch, HIVE-18622.09.patch, HIVE-18622.091.patch, 
> HIVE-18622.092.patch, HIVE-18622.093.patch, HIVE-18622.094.patch, 
> HIVE-18622.095.patch, HIVE-18622.096.patch, HIVE-18622.097.patch, 
> HIVE-18622.098.patch, HIVE-18622.099.patch, HIVE-18622.0992.patch, 
> HIVE-18622.0993.patch
>
>
>  
>  Many vector expression classes are setting noNulls to true which does not 
> work if the VRB is a scratch column being reused. The previous use may have 
> set noNulls to false and the isNull array will have some rows marked as NULL. 
> The result is wrong query results and sometimes NPEs (for BytesColumnVector).
> So, many vector expressions need this:
> {code:java}
>   // Carefully handle NULLs...
>   /*
>* For better performance on LONG/DOUBLE we don't want the conditional
>* statements inside the for loop.
>*/
>   outputColVector.noNulls = false;
>  {code}
> And, vector expressions need to make sure the isNull array entry is set when 
> outputColVector.noNulls is false.
> And, all place that assign column value need to set noNulls to false when the 
> value is NULL.
> Almost all cases where noNulls is set to true are incorrect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-18622:

Attachment: HIVE-18622.0993.patch

> Vectorization: IF Statements, Comparisons, and more do not handle NULLs 
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.03.patch, HIVE-18622.04.patch, 
> HIVE-18622.05.patch, HIVE-18622.06.patch, HIVE-18622.07.patch, 
> HIVE-18622.08.patch, HIVE-18622.09.patch, HIVE-18622.091.patch, 
> HIVE-18622.092.patch, HIVE-18622.093.patch, HIVE-18622.094.patch, 
> HIVE-18622.095.patch, HIVE-18622.096.patch, HIVE-18622.097.patch, 
> HIVE-18622.098.patch, HIVE-18622.099.patch, HIVE-18622.0992.patch, 
> HIVE-18622.0993.patch
>
>
>  
>  Many vector expression classes are setting noNulls to true which does not 
> work if the VRB is a scratch column being reused. The previous use may have 
> set noNulls to false and the isNull array will have some rows marked as NULL. 
> The result is wrong query results and sometimes NPEs (for BytesColumnVector).
> So, many vector expressions need this:
> {code:java}
>   // Carefully handle NULLs...
>   /*
>* For better performance on LONG/DOUBLE we don't want the conditional
>* statements inside the for loop.
>*/
>   outputColVector.noNulls = false;
>  {code}
> And, vector expressions need to make sure the isNull array entry is set when 
> outputColVector.noNulls is false.
> And, all place that assign column value need to set noNulls to false when the 
> value is NULL.
> Almost all cases where noNulls is set to true are incorrect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-18622:

Status: In Progress  (was: Patch Available)

> Vectorization: IF Statements, Comparisons, and more do not handle NULLs 
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.03.patch, HIVE-18622.04.patch, 
> HIVE-18622.05.patch, HIVE-18622.06.patch, HIVE-18622.07.patch, 
> HIVE-18622.08.patch, HIVE-18622.09.patch, HIVE-18622.091.patch, 
> HIVE-18622.092.patch, HIVE-18622.093.patch, HIVE-18622.094.patch, 
> HIVE-18622.095.patch, HIVE-18622.096.patch, HIVE-18622.097.patch, 
> HIVE-18622.098.patch, HIVE-18622.099.patch, HIVE-18622.0992.patch
>
>
>  
>  Many vector expression classes are setting noNulls to true which does not 
> work if the VRB is a scratch column being reused. The previous use may have 
> set noNulls to false and the isNull array will have some rows marked as NULL. 
> The result is wrong query results and sometimes NPEs (for BytesColumnVector).
> So, many vector expressions need this:
> {code:java}
>   // Carefully handle NULLs...
>   /*
>* For better performance on LONG/DOUBLE we don't want the conditional
>* statements inside the for loop.
>*/
>   outputColVector.noNulls = false;
>  {code}
> And, vector expressions need to make sure the isNull array entry is set when 
> outputColVector.noNulls is false.
> And, all place that assign column value need to set noNulls to false when the 
> value is NULL.
> Almost all cases where noNulls is set to true are incorrect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18693) Snapshot Isolation does not work for Micromanaged table when a insert transaction is aborted

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1639#comment-1639
 ] 

Hive QA commented on HIVE-18693:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12910699/HIVE-18693.01.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9243/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9243/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9243/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-02-16 07:16:28.450
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-9243/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-02-16 07:16:28.453
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 01f34e4 Addendum to HIVE-18553 : Support schema evolution in 
Parquet Vectorization reader. Removes extra q.out file
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 01f34e4 Addendum to HIVE-18553 : Support schema evolution in 
Parquet Vectorization reader. Removes extra q.out file
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-02-16 07:16:32.376
+ rm -rf ../yetus
+ mkdir ../yetus
+ git gc
+ cp -R . ../yetus
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-9243/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: patch failed: 
ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java:426
Falling back to three-way merge...
Applied patch to 
'ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java' with 
conflicts.
Going to apply patch with: git apply -p0
error: patch failed: 
ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java:426
Falling back to three-way merge...
Applied patch to 
'ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java' with 
conflicts.
U ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12910699 - PreCommit-HIVE-Build

> Snapshot Isolation does not work for Micromanaged table when a insert 
> transaction is aborted
> 
>
> Key: HIVE-18693
> URL: https://issues.apache.org/jira/browse/HIVE-18693
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-18693.01.patch
>
>
> TestTxnCommands2#writeBetweenWorkerAndCleaner with minor 
> changes (changing delete command to insert command) fails on MM table.
> Specifically the last SELECT commands returns wrong results. 
> But this test works fine with full ACID table. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1637#comment-1637
 ] 

Hive QA commented on HIVE-18622:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12910847/HIVE-18622.0992.patch

{color:green}SUCCESS:{color} +1 due to 29 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 115 failed/errored test(s), 13783 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
 (batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=78)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
 (batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] 
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
 (batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
 (batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1] 
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
 (batchId=179)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=121)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut 
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestAddAlterDropIndexes.testDropIndexInvalidDB[Embedded]
 (batchId=205)
org.apache.hadoop.hive.ql.TestAcidOnTez.testGetSplitsLocks (batchId=224)
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.TestColumnScalarOperationVectorExpressionCheckedEvaluation.testDoubleColAddDoubleScalarCheckedOutNullsColRepeatsRetDouble
 (batchId=285)
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.TestColumnScalarOperationVectorExpressionCheckedEvaluation.testDoubleColAddDoubleScalarCheckedOutNullsColRepeatsRetFloat
 (batchId=285)
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.TestColumnScalarOperationVectorExpressionCheckedEvaluation.testDoubleColAddLongScalarCheckedOutNullsColRepeatsRetDouble
 (batchId=285)
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.TestColumnScalarOperationVectorExpressionCheckedEvaluation.testDoubleColAddLongScalarCheckedOutNullsColRepeatsRetFloat
 (batchId=285)
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.TestColumnScalarOperationVectorExpressionCheckedEvaluation.testDoubleColModuloDoubleScalarCheckedOutNullsColRepeatsRetDouble
 (batchId=285)
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.TestColumnScalarOperationVectorExpressionCheckedEvaluation.testDoubleColModuloDoubleScalarCheckedOutNullsColRepeatsRetFloat
 (batchId=285)
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.TestColumnScalarOperationVectorExpressionCheckedEvaluation.testDoubleColModuloLongScalarCheckedOutNullsColRepeatsRetDouble
 (batchId=285)
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.TestColumnScalarOperationVectorExpressionCheckedEvaluation.testDoubleColModuloLongScalarCheckedOutNullsColRepeatsRetFloat
 (batchId=285)
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.TestColumnScalarOperationVectorExpressionCheckedEvaluation.testDoubleColMultiplyDoubleScalarCheckedOutNullsColRepeatsRetDouble
 (batchId=285)
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.TestColumnScalarOperationVectorExpressionCheckedEvaluation.testDoubleColMultiplyDoubleScalarCheckedOutNullsColRepeatsRetFloat
 (batchId=285)
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.TestColumnScalarOperationVectorExpressionCheckedEvaluation.testDoubleColMultiplyLongScalarCheckedOutNullsColRepeatsRetDouble
 (batchId=285)
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.TestColumnScalarOperationVectorExpressionCheckedEvaluation.testDoubleColMultiplyLongScalarCheckedOutNullsColRepeatsRetFloat
 (batchId=285)
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.TestColumnScalarOperationVectorExpressionCheckedEvaluation.testDoubleColSubtractDoubleScalarCheckedOutNullsColRepeatsRetDouble
 (batchId=285)
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.TestColumnScalarOperationVectorExpressionCheckedEvaluation.testDoubleColSubtractDoubleScalarCheckedOutNullsColRepeatsRetFloat
 (batchId=285)

[jira] [Commented] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1631#comment-1631
 ] 

Hive QA commented on HIVE-18622:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  5m 
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  7m  
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
6s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
 4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m  
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
12s{color} | {color:red} storage-api: The patch generated 41 new + 57 unchanged 
- 77 fixed = 98 total (was 134) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  3m 
17s{color} | {color:red} root: The patch generated 812 new + 5549 unchanged - 
551 fixed = 6361 total (was 6100) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m 
50s{color} | {color:red} ql: The patch generated 769 new + 5104 unchanged - 474 
fixed = 5873 total (was 5578) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
11s{color} | {color:red} vector-code-gen: The patch generated 2 new + 314 
unchanged - 0 fixed = 316 total (was 314) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
1s{color} | {color:red} The patch has 18 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
10s{color} | {color:red} storage-api generated 1 new + 26 unchanged - 0 fixed = 
27 total (was 26) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  5m 
35s{color} | {color:red} root generated 1 new + 336 unchanged - 0 fixed = 337 
total (was 336) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 50s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 01f34e4 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9242/yetus/diff-checkstyle-storage-api.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9242/yetus/diff-checkstyle-root.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9242/yetus/diff-checkstyle-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9242/yetus/diff-checkstyle-vector-code-gen.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9242/yetus/whitespace-eol.txt 
|
| javadoc | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9242/yetus/diff-javadoc-javadoc-storage-api.txt
 |
| javadoc | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9242/yetus/diff-javadoc-javadoc-root.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9242/yetus/patch-asflicense-problems.txt
 |
| modules | C: storage-api 

[jira] [Updated] (HIVE-18708) Vectorization: Delay out-of-tree fixups till whole work is vectorized

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-18708:
---
Attachment: HIVE-18708.3.patch

> Vectorization: Delay out-of-tree fixups till whole work is vectorized
> -
>
> Key: HIVE-18708
> URL: https://issues.apache.org/jira/browse/HIVE-18708
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-18708.1.patch, HIVE-18708.2.patch, 
> HIVE-18708.2.patch, HIVE-18708.3.patch
>
>
> The vectorization validation codepath should treat the existing operator tree 
> as immutable, so that the VectorizerCannotVectorizeException does not have to 
> undo any changes to the operator tree when caught.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18708) Vectorization: Delay out-of-tree fixups till whole work is vectorized

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-18708:
---
Attachment: (was: HIVE-18708.3.patch)

> Vectorization: Delay out-of-tree fixups till whole work is vectorized
> -
>
> Key: HIVE-18708
> URL: https://issues.apache.org/jira/browse/HIVE-18708
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-18708.1.patch, HIVE-18708.2.patch, 
> HIVE-18708.2.patch, HIVE-18708.3.patch
>
>
> The vectorization validation codepath should treat the existing operator tree 
> as immutable, so that the VectorizerCannotVectorizeException does not have to 
> undo any changes to the operator tree when caught.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18693) Snapshot Isolation does not work for Micromanaged table when a insert transaction is aborted

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366634#comment-16366634
 ] 

Hive QA commented on HIVE-18693:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12910699/HIVE-18693.01.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9241/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9241/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9241/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-02-16 05:59:52.389
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-9241/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-02-16 05:59:52.396
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 01f34e4 Addendum to HIVE-18553 : Support schema evolution in 
Parquet Vectorization reader. Removes extra q.out file
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 01f34e4 Addendum to HIVE-18553 : Support schema evolution in 
Parquet Vectorization reader. Removes extra q.out file
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-02-16 05:59:56.052
+ rm -rf ../yetus
+ mkdir ../yetus
+ git gc
+ cp -R . ../yetus
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-9241/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: patch failed: 
ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java:426
Falling back to three-way merge...
Applied patch to 
'ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java' with 
conflicts.
Going to apply patch with: git apply -p0
error: patch failed: 
ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java:426
Falling back to three-way merge...
Applied patch to 
'ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java' with 
conflicts.
U ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12910699 - PreCommit-HIVE-Build

> Snapshot Isolation does not work for Micromanaged table when a insert 
> transaction is aborted
> 
>
> Key: HIVE-18693
> URL: https://issues.apache.org/jira/browse/HIVE-18693
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-18693.01.patch
>
>
> TestTxnCommands2#writeBetweenWorkerAndCleaner with minor 
> changes (changing delete command to insert command) fails on MM table.
> Specifically the last SELECT commands returns wrong results. 
> But this test works fine with full ACID table. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18726) Implement DEFAULT constraint

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366633#comment-16366633
 ] 

Hive QA commented on HIVE-18726:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12910841/HIVE-18726.1.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 13760 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
 (batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=78)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
 (batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] 
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
 (batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
 (batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
 (batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=160)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=121)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut 
(batchId=221)
org.apache.hadoop.hive.metastore.TestMarkPartition.testMarkingPartitionSet 
(batchId=214)
org.apache.hadoop.hive.metastore.client.TestTablesList.testListTableNamesByFilterNullDatabase[Embedded]
 (batchId=205)
org.apache.hadoop.hive.ql.TestAcidOnTez.testGetSplitsLocks (batchId=224)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=187)
org.apache.hive.hcatalog.common.TestHiveClientCache.testCloseAllClients 
(batchId=199)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.alterIndex 
(batchId=242)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.createIndex 
(batchId=242)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropIndex 
(batchId=242)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd 
(batchId=235)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9240/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9240/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9240/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 27 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12910841 - PreCommit-HIVE-Build

> Implement DEFAULT constraint
> 
>
> Key: HIVE-18726
> URL: https://issues.apache.org/jira/browse/HIVE-18726
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Planning, Query Processor
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18726.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18726) Implement DEFAULT constraint

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366621#comment-16366621
 ] 

Hive QA commented on HIVE-18726:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
17s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
32s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
10s{color} | {color:red} itests/hcatalog-unit: The patch generated 2 new + 20 
unchanged - 0 fixed = 22 total (was 20) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
54s{color} | {color:red} ql: The patch generated 33 new + 1404 unchanged - 25 
fixed = 1437 total (was 1429) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
33s{color} | {color:red} standalone-metastore: The patch generated 53 new + 
1588 unchanged - 11 fixed = 1641 total (was 1599) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
1s{color} | {color:red} The patch has 94 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 40s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 01f34e4 |
| Default Java | 1.8.0_111 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9240/yetus/patch-mvninstall-itests_hcatalog-unit.txt
 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9240/yetus/patch-mvninstall-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9240/yetus/diff-checkstyle-itests_hcatalog-unit.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9240/yetus/diff-checkstyle-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9240/yetus/diff-checkstyle-standalone-metastore.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9240/yetus/whitespace-eol.txt 
|
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9240/yetus/patch-asflicense-problems.txt
 |
| modules | C: itests itests/hcatalog-unit metastore ql standalone-metastore U: 
. |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9240/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Implement DEFAULT constraint
> 
>
> Key: HIVE-18726
> 

[jira] [Commented] (HIVE-17990) Add Thrift and DB storage for Schema Registry objects

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366607#comment-16366607
 ] 

Hive QA commented on HIVE-17990:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12910845/HIVE-17990.patch

{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 13826 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
 (batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=78)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
 (batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] 
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
 (batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
 (batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=160)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=121)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut 
(batchId=221)
org.apache.hadoop.hive.metastore.TestMarkPartition.testMarkingPartitionSet 
(batchId=214)
org.apache.hadoop.hive.metastore.client.TestAddPartitions.testAddPartitionsNullColTypeInSd[Embedded]
 (batchId=205)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testGetIndexEmptyTableName[Embedded]
 (batchId=205)
org.apache.hadoop.hive.ql.TestAcidOnTez.testGetSplitsLocks (batchId=224)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=187)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.alterIndex 
(batchId=242)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.createIndex 
(batchId=242)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropIndex 
(batchId=242)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd 
(batchId=235)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9239/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9239/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9239/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12910845 - PreCommit-HIVE-Build

> Add Thrift and DB storage for Schema Registry objects
> -
>
> Key: HIVE-17990
> URL: https://issues.apache.org/jira/browse/HIVE-17990
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Attachments: Adding-Schema-Registry-to-Metastore.pdf, HIVE-17990.patch
>
>
> This JIRA tracks changes to Thrift, RawStore, and DB scripts to support 
> objects in the Schema Registry.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-18693) Snapshot Isolation does not work for Micromanaged table when a insert transaction is aborted

[Steve Yeom (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366597#comment-16366597
 ] 

Steve Yeom edited comment on HIVE-18693 at 2/16/18 4:31 AM:


Eugene also mentioned possible bottleneck in Cleaner. 
If possible, we can introduce parallelism whereever possible, if 
it is better in performance. 


was (Author: steveyeom2017):
Eugene also mentioned possible bottleneck in Cleaner. 
If possible, we can introduce parallelism whereever possible. 

> Snapshot Isolation does not work for Micromanaged table when a insert 
> transaction is aborted
> 
>
> Key: HIVE-18693
> URL: https://issues.apache.org/jira/browse/HIVE-18693
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-18693.01.patch
>
>
> TestTxnCommands2#writeBetweenWorkerAndCleaner with minor 
> changes (changing delete command to insert command) fails on MM table.
> Specifically the last SELECT commands returns wrong results. 
> But this test works fine with full ACID table. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18693) Snapshot Isolation does not work for Micromanaged table when a insert transaction is aborted

[Steve Yeom (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366597#comment-16366597
 ] 

Steve Yeom commented on HIVE-18693:
---

Eugene also mentioned possible bottleneck in Cleaner. 
If possible, we can introduce parallelism whereever possible. 

> Snapshot Isolation does not work for Micromanaged table when a insert 
> transaction is aborted
> 
>
> Key: HIVE-18693
> URL: https://issues.apache.org/jira/browse/HIVE-18693
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-18693.01.patch
>
>
> TestTxnCommands2#writeBetweenWorkerAndCleaner with minor 
> changes (changing delete command to insert command) fails on MM table.
> Specifically the last SELECT commands returns wrong results. 
> But this test works fine with full ACID table. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-17871) Add non nullability flag to druid time column

[slim bouguerra (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

slim bouguerra updated HIVE-17871:
--
Attachment: HIVE-17871.4.patch

> Add non nullability flag to druid time column
> -
>
> Key: HIVE-17871
> URL: https://issues.apache.org/jira/browse/HIVE-17871
> Project: Hive
>  Issue Type: Improvement
>  Components: Druid integration
>Reporter: slim bouguerra
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-17871.2.patch, HIVE-17871.3.patch, 
> HIVE-17871.4.patch, HIVE-17871.patch
>
>
> Druid time column is non null all the time.
> Adding the non nullability flag will enable extra calcite goodness  like 
> transforming 
> {code} select count(`__time`) from table {code} to {code} select count(*) 
> from table {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17990) Add Thrift and DB storage for Schema Registry objects

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366576#comment-16366576
 ] 

Hive QA commented on HIVE-17990:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
20s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
32s{color} | {color:red} standalone-metastore: The patch generated 135 new + 
1327 unchanged - 6 fixed = 1462 total (was 1333) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
1s{color} | {color:red} The patch has 281 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
15s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 19s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  
xml  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 01f34e4 |
| Default Java | 1.8.0_111 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9239/yetus/patch-mvninstall-itests_hcatalog-unit.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9239/yetus/diff-checkstyle-standalone-metastore.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9239/yetus/whitespace-eol.txt 
|
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9239/yetus/patch-asflicense-problems.txt
 |
| modules | C: itests/hcatalog-unit standalone-metastore U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9239/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Add Thrift and DB storage for Schema Registry objects
> -
>
> Key: HIVE-17990
> URL: https://issues.apache.org/jira/browse/HIVE-17990
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Attachments: Adding-Schema-Registry-to-Metastore.pdf, HIVE-17990.patch
>
>
> This JIRA tracks changes to Thrift, RawStore, and DB scripts to support 
> objects in the Schema Registry.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366570#comment-16366570
 ] 

Vihang Karajgaonkar commented on HIVE-18622:


Thanks [~mmccline] for fixing that. Shouldn't this be removed from 
LongColModuloLongColumnChecked like you did in LongColModuloLongColumn.java? 
Just wanted to make sure. Thanks!
{code:java}
outputColVector.isRepeating =
 inputColVector1.isRepeating && inputColVector2.isRepeating
  || inputColVector1.isRepeating && !inputColVector1.noNulls && 
inputColVector1.isNull[0]
  || inputColVector2.isRepeating && !inputColVector2.noNulls && 
inputColVector2.isNull[0];
{code}

> Vectorization: IF Statements, Comparisons, and more do not handle NULLs 
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.03.patch, HIVE-18622.04.patch, 
> HIVE-18622.05.patch, HIVE-18622.06.patch, HIVE-18622.07.patch, 
> HIVE-18622.08.patch, HIVE-18622.09.patch, HIVE-18622.091.patch, 
> HIVE-18622.092.patch, HIVE-18622.093.patch, HIVE-18622.094.patch, 
> HIVE-18622.095.patch, HIVE-18622.096.patch, HIVE-18622.097.patch, 
> HIVE-18622.098.patch, HIVE-18622.099.patch, HIVE-18622.0992.patch
>
>
>  
>  Many vector expression classes are setting noNulls to true which does not 
> work if the VRB is a scratch column being reused. The previous use may have 
> set noNulls to false and the isNull array will have some rows marked as NULL. 
> The result is wrong query results and sometimes NPEs (for BytesColumnVector).
> So, many vector expressions need this:
> {code:java}
>   // Carefully handle NULLs...
>   /*
>* For better performance on LONG/DOUBLE we don't want the conditional
>* statements inside the for loop.
>*/
>   outputColVector.noNulls = false;
>  {code}
> And, vector expressions need to make sure the isNull array entry is set when 
> outputColVector.noNulls is false.
> And, all place that assign column value need to set noNulls to false when the 
> value is NULL.
> Almost all cases where noNulls is set to true are incorrect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17871) Add non nullability flag to druid time column

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366565#comment-16366565
 ] 

Hive QA commented on HIVE-17871:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12910836/HIVE-17871.3.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 13783 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
 (batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_intervals] 
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_timeseries] 
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=78)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
 (batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
 (batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
 (batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=160)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=121)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut 
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullStorageDescriptorInNew[Embedded]
 (batchId=205)
org.apache.hadoop.hive.ql.TestAcidOnTez.testGetSplitsLocks (batchId=224)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=187)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.alterIndex 
(batchId=242)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.createIndex 
(batchId=242)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropIndex 
(batchId=242)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd 
(batchId=235)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveConflictKill
 (batchId=235)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9238/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9238/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9238/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12910836 - PreCommit-HIVE-Build

> Add non nullability flag to druid time column
> -
>
> Key: HIVE-17871
> URL: https://issues.apache.org/jira/browse/HIVE-17871
> Project: Hive
>  Issue Type: Improvement
>  Components: Druid integration
>Reporter: slim bouguerra
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-17871.2.patch, HIVE-17871.3.patch, HIVE-17871.patch
>
>
> Druid time column is non null all the time.
> Adding the non nullability flag will enable extra calcite goodness  like 
> transforming 
> {code} select count(`__time`) from table {code} to {code} select count(*) 
> from table {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18659) add acid version marker to acid files/directories

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366558#comment-16366558
 ] 

Prasanth Jayachandran commented on HIVE-18659:
--

nit: 
deltaDest is getting added in createdDeltaDirs.add(deltaDest)  even if mkdirs 
fails in the if condition above. Not sure if that is a desired behaviour.

+1 otherwise

> add acid version marker to acid files/directories
> -
>
> Key: HIVE-18659
> URL: https://issues.apache.org/jira/browse/HIVE-18659
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18659.01.patch, HIVE-18659.04.patch, 
> HIVE-18659.05.patch, HIVE-18659.06.patch
>
>
> add acid version marker to acid files so that we know which version of acid 
> wrote the file



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18693) Snapshot Isolation does not work for Micromanaged table when a insert transaction is aborted

[Steve Yeom (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366552#comment-16366552
 ] 

Steve Yeom commented on HIVE-18693:
---

Talked with Eugene.
The patch has the following requirements: 
1. Allow Snapshot Isolation to work. Esp., we should keep TXNS row for aborted 
transaction until the related delta directories
  are removed (which can contain many components). For this we will add rows 
for components (partitions/table)
  intoTXN_COMPONENTS. 
2. Remove aborted transaction delta/base directories for MM tables.
3. Do not initiate compaction for MM tables. So a compaction request for MM 
table may have 
  INITIATING/INITIATED, CLEANING, FINISHED state without COMPACTING/COMPACTED 
state.
3. Clean obsolete directories, and Metastore metadata rows for transaction 
tables. 
 

> Snapshot Isolation does not work for Micromanaged table when a insert 
> transaction is aborted
> 
>
> Key: HIVE-18693
> URL: https://issues.apache.org/jira/browse/HIVE-18693
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-18693.01.patch
>
>
> TestTxnCommands2#writeBetweenWorkerAndCleaner with minor 
> changes (changing delete command to insert command) fails on MM table.
> Specifically the last SELECT commands returns wrong results. 
> But this test works fine with full ACID table. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17871) Add non nullability flag to druid time column

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366551#comment-16366551
 ] 

Hive QA commented on HIVE-17871:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m  
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m 
51s{color} | {color:red} root: The patch generated 11 new + 172 unchanged - 0 
fixed = 183 total (was 172) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
38s{color} | {color:red} ql: The patch generated 11 new + 172 unchanged - 0 
fixed = 183 total (was 172) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 38s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 01f34e4 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9238/yetus/diff-checkstyle-root.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9238/yetus/diff-checkstyle-ql.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9238/yetus/patch-asflicense-problems.txt
 |
| modules | C: . ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9238/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Add non nullability flag to druid time column
> -
>
> Key: HIVE-17871
> URL: https://issues.apache.org/jira/browse/HIVE-17871
> Project: Hive
>  Issue Type: Improvement
>  Components: Druid integration
>Reporter: slim bouguerra
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-17871.2.patch, HIVE-17871.3.patch, HIVE-17871.patch
>
>
> Druid time column is non null all the time.
> Adding the non nullability flag will enable extra calcite goodness  like 
> transforming 
> {code} select count(`__time`) from table {code} to {code} select count(*) 
> from table {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18693) Snapshot Isolation does not work for Micromanaged table when a insert transaction is aborted

[Steve Yeom (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Yeom updated HIVE-18693:
--
Attachment: (was: HIVE-18693.02.patch)

> Snapshot Isolation does not work for Micromanaged table when a insert 
> transaction is aborted
> 
>
> Key: HIVE-18693
> URL: https://issues.apache.org/jira/browse/HIVE-18693
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-18693.01.patch
>
>
> TestTxnCommands2#writeBetweenWorkerAndCleaner with minor 
> changes (changing delete command to insert command) fails on MM table.
> Specifically the last SELECT commands returns wrong results. 
> But this test works fine with full ACID table. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly

[Matt McCline (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366514#comment-16366514
 ] 

Matt McCline commented on HIVE-18622:
-

[~vihangk1] Ok I fixed ColumnArithmeticScalar.txt, ColumnUnaryMinus.txt, and 
ScalarArithmeticColumn.txt

Thanks for spotting that!

LongColModuloLongColumnChecked looks ok to me as is.

Now it is patch #0992

> Vectorization: IF Statements, Comparisons, and more do not handle NULLs 
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.03.patch, HIVE-18622.04.patch, 
> HIVE-18622.05.patch, HIVE-18622.06.patch, HIVE-18622.07.patch, 
> HIVE-18622.08.patch, HIVE-18622.09.patch, HIVE-18622.091.patch, 
> HIVE-18622.092.patch, HIVE-18622.093.patch, HIVE-18622.094.patch, 
> HIVE-18622.095.patch, HIVE-18622.096.patch, HIVE-18622.097.patch, 
> HIVE-18622.098.patch, HIVE-18622.099.patch, HIVE-18622.0992.patch
>
>
>  
>  Many vector expression classes are setting noNulls to true which does not 
> work if the VRB is a scratch column being reused. The previous use may have 
> set noNulls to false and the isNull array will have some rows marked as NULL. 
> The result is wrong query results and sometimes NPEs (for BytesColumnVector).
> So, many vector expressions need this:
> {code:java}
>   // Carefully handle NULLs...
>   /*
>* For better performance on LONG/DOUBLE we don't want the conditional
>* statements inside the for loop.
>*/
>   outputColVector.noNulls = false;
>  {code}
> And, vector expressions need to make sure the isNull array entry is set when 
> outputColVector.noNulls is false.
> And, all place that assign column value need to set noNulls to false when the 
> value is NULL.
> Almost all cases where noNulls is set to true are incorrect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18693) Snapshot Isolation does not work for Micromanaged table when a insert transaction is aborted

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366513#comment-16366513
 ] 

Eugene Koifman commented on HIVE-18693:
---

this won't enqueue any compaction requests so CompactorMR will never run, i.e. 
it won't clean aborted dirs
also, it will not clean relevant entries from txn_components or 
completed_txn_componentes.

The test should look at the directories on the file system and make sure that 
they are actually deleted.

> Snapshot Isolation does not work for Micromanaged table when a insert 
> transaction is aborted
> 
>
> Key: HIVE-18693
> URL: https://issues.apache.org/jira/browse/HIVE-18693
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-18693.01.patch, HIVE-18693.02.patch
>
>
> TestTxnCommands2#writeBetweenWorkerAndCleaner with minor 
> changes (changing delete command to insert command) fails on MM table.
> Specifically the last SELECT commands returns wrong results. 
> But this test works fine with full ACID table. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-18622:

Attachment: HIVE-18622.0992.patch

> Vectorization: IF Statements, Comparisons, and more do not handle NULLs 
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.03.patch, HIVE-18622.04.patch, 
> HIVE-18622.05.patch, HIVE-18622.06.patch, HIVE-18622.07.patch, 
> HIVE-18622.08.patch, HIVE-18622.09.patch, HIVE-18622.091.patch, 
> HIVE-18622.092.patch, HIVE-18622.093.patch, HIVE-18622.094.patch, 
> HIVE-18622.095.patch, HIVE-18622.096.patch, HIVE-18622.097.patch, 
> HIVE-18622.098.patch, HIVE-18622.099.patch, HIVE-18622.0992.patch
>
>
>  
>  Many vector expression classes are setting noNulls to true which does not 
> work if the VRB is a scratch column being reused. The previous use may have 
> set noNulls to false and the isNull array will have some rows marked as NULL. 
> The result is wrong query results and sometimes NPEs (for BytesColumnVector).
> So, many vector expressions need this:
> {code:java}
>   // Carefully handle NULLs...
>   /*
>* For better performance on LONG/DOUBLE we don't want the conditional
>* statements inside the for loop.
>*/
>   outputColVector.noNulls = false;
>  {code}
> And, vector expressions need to make sure the isNull array entry is set when 
> outputColVector.noNulls is false.
> And, all place that assign column value need to set noNulls to false when the 
> value is NULL.
> Almost all cases where noNulls is set to true are incorrect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-18622:

Attachment: (was: HIVE-18622.0991.patch)

> Vectorization: IF Statements, Comparisons, and more do not handle NULLs 
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.03.patch, HIVE-18622.04.patch, 
> HIVE-18622.05.patch, HIVE-18622.06.patch, HIVE-18622.07.patch, 
> HIVE-18622.08.patch, HIVE-18622.09.patch, HIVE-18622.091.patch, 
> HIVE-18622.092.patch, HIVE-18622.093.patch, HIVE-18622.094.patch, 
> HIVE-18622.095.patch, HIVE-18622.096.patch, HIVE-18622.097.patch, 
> HIVE-18622.098.patch, HIVE-18622.099.patch
>
>
>  
>  Many vector expression classes are setting noNulls to true which does not 
> work if the VRB is a scratch column being reused. The previous use may have 
> set noNulls to false and the isNull array will have some rows marked as NULL. 
> The result is wrong query results and sometimes NPEs (for BytesColumnVector).
> So, many vector expressions need this:
> {code:java}
>   // Carefully handle NULLs...
>   /*
>* For better performance on LONG/DOUBLE we don't want the conditional
>* statements inside the for loop.
>*/
>   outputColVector.noNulls = false;
>  {code}
> And, vector expressions need to make sure the isNull array entry is set when 
> outputColVector.noNulls is false.
> And, all place that assign column value need to set noNulls to false when the 
> value is NULL.
> Almost all cases where noNulls is set to true are incorrect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366480#comment-16366480
 ] 

Vihang Karajgaonkar edited comment on HIVE-18622 at 2/16/18 1:17 AM:
-

Thanks [~mmccline] for fixing this. I left minor comments which need to be 
addressed else this patch would regress fix for HIVE-18421. Would appreciate if 
you could take a look at them. We basically need to handle the overflows before 
returning in case the inputVector.isRepeating is true. Also, should the checked 
version of {{LongColModuloLongColumnChecked.java}} should also be modified 
similar to {{LongColModuloLongColumn.java}}?


was (Author: vihangk1):
Thanks [~mmccline] for fixing this. I left minor comments which need to be 
addressed else this patch would regress fix for HIVE-18421. Would appreciate if 
you could take a look at them. We basically need to handle the overflows before 
returning in case the inputVector.isRepeating is true.

> Vectorization: IF Statements, Comparisons, and more do not handle NULLs 
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.03.patch, HIVE-18622.04.patch, 
> HIVE-18622.05.patch, HIVE-18622.06.patch, HIVE-18622.07.patch, 
> HIVE-18622.08.patch, HIVE-18622.09.patch, HIVE-18622.091.patch, 
> HIVE-18622.092.patch, HIVE-18622.093.patch, HIVE-18622.094.patch, 
> HIVE-18622.095.patch, HIVE-18622.096.patch, HIVE-18622.097.patch, 
> HIVE-18622.098.patch, HIVE-18622.099.patch, HIVE-18622.0991.patch
>
>
>  
>  Many vector expression classes are setting noNulls to true which does not 
> work if the VRB is a scratch column being reused. The previous use may have 
> set noNulls to false and the isNull array will have some rows marked as NULL. 
> The result is wrong query results and sometimes NPEs (for BytesColumnVector).
> So, many vector expressions need this:
> {code:java}
>   // Carefully handle NULLs...
>   /*
>* For better performance on LONG/DOUBLE we don't want the conditional
>* statements inside the for loop.
>*/
>   outputColVector.noNulls = false;
>  {code}
> And, vector expressions need to make sure the isNull array entry is set when 
> outputColVector.noNulls is false.
> And, all place that assign column value need to set noNulls to false when the 
> value is NULL.
> Almost all cases where noNulls is set to true are incorrect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-17990) Add Thrift and DB storage for Schema Registry objects

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-17990:
--
Status: Patch Available  (was: In Progress)

> Add Thrift and DB storage for Schema Registry objects
> -
>
> Key: HIVE-17990
> URL: https://issues.apache.org/jira/browse/HIVE-17990
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Attachments: Adding-Schema-Registry-to-Metastore.pdf, HIVE-17990.patch
>
>
> This JIRA tracks changes to Thrift, RawStore, and DB scripts to support 
> objects in the Schema Registry.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-17990) Add Thrift and DB storage for Schema Registry objects

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-17990:
--
Labels: pull-request-available  (was: )

> Add Thrift and DB storage for Schema Registry objects
> -
>
> Key: HIVE-17990
> URL: https://issues.apache.org/jira/browse/HIVE-17990
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Attachments: Adding-Schema-Registry-to-Metastore.pdf, HIVE-17990.patch
>
>
> This JIRA tracks changes to Thrift, RawStore, and DB scripts to support 
> objects in the Schema Registry.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17990) Add Thrift and DB storage for Schema Registry objects

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366495#comment-16366495
 ] 

ASF GitHub Bot commented on HIVE-17990:
---

GitHub user alanfgates opened a pull request:

https://github.com/apache/hive/pull/308

HIVE-17990 Add Thrift and DB storage for Schema Registry objects



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alanfgates/hive hive17990

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/308.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #308


commit a5a49e9a0fcca40c298849e5b9d2eada00cebd9a
Author: Alan Gates 
Date:   2017-10-19T23:49:38Z

HIVE-17990 Add Thrift and DB storage for Schema Registry objects




> Add Thrift and DB storage for Schema Registry objects
> -
>
> Key: HIVE-17990
> URL: https://issues.apache.org/jira/browse/HIVE-17990
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Attachments: Adding-Schema-Registry-to-Metastore.pdf, HIVE-17990.patch
>
>
> This JIRA tracks changes to Thrift, RawStore, and DB scripts to support 
> objects in the Schema Registry.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-17990) Add Thrift and DB storage for Schema Registry objects

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-17990:
--
Attachment: HIVE-17990.patch

> Add Thrift and DB storage for Schema Registry objects
> -
>
> Key: HIVE-17990
> URL: https://issues.apache.org/jira/browse/HIVE-17990
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
>  Labels: pull-request-available
> Attachments: Adding-Schema-Registry-to-Metastore.pdf, HIVE-17990.patch
>
>
> This JIRA tracks changes to Thrift, RawStore, and DB scripts to support 
> objects in the Schema Registry.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18553) Support schema evolution in Parquet Vectorization reader

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366491#comment-16366491
 ] 

Vihang Karajgaonkar commented on HIVE-18553:


Merged an addendum which removes the extraneous q.out file

commit 01f34e49b352bd06ad8e65a1da613de45773c1c6
Author: Vihang Karajgaonkar 
Date:   Thu Feb 15 17:04:44 2018 -0800

Addendum to HIVE-18553 : Support schema evolution in Parquet Vectorization 
reader. Removes extra q.out file

> Support schema evolution in Parquet Vectorization reader
> 
>
> Key: HIVE-18553
> URL: https://issues.apache.org/jira/browse/HIVE-18553
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Ferdinand Xu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18553.10.patch, HIVE-18553.11.patch, 
> HIVE-18553.2.patch, HIVE-18553.3.patch, HIVE-18553.4.patch, 
> HIVE-18553.5.patch, HIVE-18553.6.patch, HIVE-18553.7.patch, 
> HIVE-18553.8.patch, HIVE-18553.9.patch, HIVE-18553.91.patch, 
> HIVE-18553.patch, test_result_based_on_HIVE-18553.xlsx
>
>
> For schema evolution, it includes the following points:
> 1. column changes
> column reorder
> column add, column delete
> column rename
> 2. type conversion
> low precision to high precision
> type to String
> For 1st type, current the code is not supporting the column addition 
> operation. Detailed error is as follows:
> {code}
> 0: jdbc:hive2://localhost:1/default> desc test_p;
> +---++--+
> | col_name  | data_type  | comment  |
> +---++--+
> | t1| tinyint|  |
> | t2| tinyint|  |
> | i1| int|  |
> | i2| int|  |
> +---++--+
> 0: jdbc:hive2://localhost:1/default> set hive.fetch.task.conversion=none;
> 0: jdbc:hive2://localhost:1/default> set 
> hive.vectorized.execution.enabled=true;
> 0: jdbc:hive2://localhost:1/default> alter table test_p add columns (ts 
> timestamp);
> 0: jdbc:hive2://localhost:1/default> select * from test_p;
> Error: Error while processing statement: FAILED: Execution Error, return code 
> 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=2)
> {code}
> Following exception is seen in the logs
> {code}
> Caused by: java.lang.IllegalArgumentException: [ts] BINARY is not in the 
> store: [[i1] INT32, [i2] INT32, [t1] INT32, [t2] INT32] 3
> at 
> org.apache.parquet.hadoop.ColumnChunkPageReadStore.getPageReader(ColumnChunkPageReadStore.java:160)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:479)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:432)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:393)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:345)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:88)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:360)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:167)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:52)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:229)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:142)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> 

[jira] [Commented] (HIVE-18553) Support schema evolution in Parquet Vectorization reader

[KaiXu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366489#comment-16366489
 ] 

KaiXu commented on HIVE-18553:
--

Thanks for your email. I am taking annual leave, email responses can be 
delayed. Sorry for any inconveniences.


> Support schema evolution in Parquet Vectorization reader
> 
>
> Key: HIVE-18553
> URL: https://issues.apache.org/jira/browse/HIVE-18553
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Ferdinand Xu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18553.10.patch, HIVE-18553.11.patch, 
> HIVE-18553.2.patch, HIVE-18553.3.patch, HIVE-18553.4.patch, 
> HIVE-18553.5.patch, HIVE-18553.6.patch, HIVE-18553.7.patch, 
> HIVE-18553.8.patch, HIVE-18553.9.patch, HIVE-18553.91.patch, 
> HIVE-18553.patch, test_result_based_on_HIVE-18553.xlsx
>
>
> For schema evolution, it includes the following points:
> 1. column changes
> column reorder
> column add, column delete
> column rename
> 2. type conversion
> low precision to high precision
> type to String
> For 1st type, current the code is not supporting the column addition 
> operation. Detailed error is as follows:
> {code}
> 0: jdbc:hive2://localhost:1/default> desc test_p;
> +---++--+
> | col_name  | data_type  | comment  |
> +---++--+
> | t1| tinyint|  |
> | t2| tinyint|  |
> | i1| int|  |
> | i2| int|  |
> +---++--+
> 0: jdbc:hive2://localhost:1/default> set hive.fetch.task.conversion=none;
> 0: jdbc:hive2://localhost:1/default> set 
> hive.vectorized.execution.enabled=true;
> 0: jdbc:hive2://localhost:1/default> alter table test_p add columns (ts 
> timestamp);
> 0: jdbc:hive2://localhost:1/default> select * from test_p;
> Error: Error while processing statement: FAILED: Execution Error, return code 
> 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=2)
> {code}
> Following exception is seen in the logs
> {code}
> Caused by: java.lang.IllegalArgumentException: [ts] BINARY is not in the 
> store: [[i1] INT32, [i2] INT32, [t1] INT32, [t2] INT32] 3
> at 
> org.apache.parquet.hadoop.ColumnChunkPageReadStore.getPageReader(ColumnChunkPageReadStore.java:160)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:479)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:432)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:393)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:345)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:88)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:360)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:167)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:52)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:229)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:142)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
>  ~[hadoop-mapreduce-client-core-3.0.0-alpha3-cdh6.x-SNAPSHOT.jar:?]
> at 
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185) 
> 

[jira] [Commented] (HIVE-18553) Support schema evolution in Parquet Vectorization reader

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366488#comment-16366488
 ] 

Vihang Karajgaonkar commented on HIVE-18553:


Looks like {{schema_evol_par_vec_table.q.out}} doesn't need to be in this 
patch. There is a no corresponding .q file introduced and it looks like a copy 
of newly added {{schema_evol_par_vec_table_dictionary_encoding.q}}

> Support schema evolution in Parquet Vectorization reader
> 
>
> Key: HIVE-18553
> URL: https://issues.apache.org/jira/browse/HIVE-18553
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Ferdinand Xu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18553.10.patch, HIVE-18553.11.patch, 
> HIVE-18553.2.patch, HIVE-18553.3.patch, HIVE-18553.4.patch, 
> HIVE-18553.5.patch, HIVE-18553.6.patch, HIVE-18553.7.patch, 
> HIVE-18553.8.patch, HIVE-18553.9.patch, HIVE-18553.91.patch, 
> HIVE-18553.patch, test_result_based_on_HIVE-18553.xlsx
>
>
> For schema evolution, it includes the following points:
> 1. column changes
> column reorder
> column add, column delete
> column rename
> 2. type conversion
> low precision to high precision
> type to String
> For 1st type, current the code is not supporting the column addition 
> operation. Detailed error is as follows:
> {code}
> 0: jdbc:hive2://localhost:1/default> desc test_p;
> +---++--+
> | col_name  | data_type  | comment  |
> +---++--+
> | t1| tinyint|  |
> | t2| tinyint|  |
> | i1| int|  |
> | i2| int|  |
> +---++--+
> 0: jdbc:hive2://localhost:1/default> set hive.fetch.task.conversion=none;
> 0: jdbc:hive2://localhost:1/default> set 
> hive.vectorized.execution.enabled=true;
> 0: jdbc:hive2://localhost:1/default> alter table test_p add columns (ts 
> timestamp);
> 0: jdbc:hive2://localhost:1/default> select * from test_p;
> Error: Error while processing statement: FAILED: Execution Error, return code 
> 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=2)
> {code}
> Following exception is seen in the logs
> {code}
> Caused by: java.lang.IllegalArgumentException: [ts] BINARY is not in the 
> store: [[i1] INT32, [i2] INT32, [t1] INT32, [t2] INT32] 3
> at 
> org.apache.parquet.hadoop.ColumnChunkPageReadStore.getPageReader(ColumnChunkPageReadStore.java:160)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:479)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:432)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:393)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:345)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:88)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:360)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:167)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:52)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:229)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:142)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
>  ~[hadoop-mapreduce-client-core-3.0.0-alpha3-cdh6.x-SNAPSHOT.jar:?]
> 

[jira] [Comment Edited] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366480#comment-16366480
 ] 

Vihang Karajgaonkar edited comment on HIVE-18622 at 2/16/18 12:56 AM:
--

Thanks [~mmccline] for fixing this. I left minor comments which need to be 
addressed else this patch would regress fix for HIVE-18421. Would appreciate if 
you could take a look at them. We basically need to handle the overflows before 
returning in case the inputVector.isRepeating is true.


was (Author: vihangk1):
Thanks [~mmccline] for fixing this. I left minor comments which would regress 
fix for HIVE-18421 unless fixed. Would appreciate if you could take a look at 
them. We basically need to handle the overflows before returning in case the 
inputVector.isRepeating is true.

> Vectorization: IF Statements, Comparisons, and more do not handle NULLs 
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.03.patch, HIVE-18622.04.patch, 
> HIVE-18622.05.patch, HIVE-18622.06.patch, HIVE-18622.07.patch, 
> HIVE-18622.08.patch, HIVE-18622.09.patch, HIVE-18622.091.patch, 
> HIVE-18622.092.patch, HIVE-18622.093.patch, HIVE-18622.094.patch, 
> HIVE-18622.095.patch, HIVE-18622.096.patch, HIVE-18622.097.patch, 
> HIVE-18622.098.patch, HIVE-18622.099.patch, HIVE-18622.0991.patch
>
>
>  
>  Many vector expression classes are setting noNulls to true which does not 
> work if the VRB is a scratch column being reused. The previous use may have 
> set noNulls to false and the isNull array will have some rows marked as NULL. 
> The result is wrong query results and sometimes NPEs (for BytesColumnVector).
> So, many vector expressions need this:
> {code:java}
>   // Carefully handle NULLs...
>   /*
>* For better performance on LONG/DOUBLE we don't want the conditional
>* statements inside the for loop.
>*/
>   outputColVector.noNulls = false;
>  {code}
> And, vector expressions need to make sure the isNull array entry is set when 
> outputColVector.noNulls is false.
> And, all place that assign column value need to set noNulls to false when the 
> value is NULL.
> Almost all cases where noNulls is set to true are incorrect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366480#comment-16366480
 ] 

Vihang Karajgaonkar commented on HIVE-18622:


Thanks [~mmccline] for fixing this. I left minor comments which would regress 
fix for HIVE-18421 unless fixed. Would appreciate if you could take a look at 
them. We basically need to handle the overflows before returning in case the 
inputVector.isRepeating is true.

> Vectorization: IF Statements, Comparisons, and more do not handle NULLs 
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.03.patch, HIVE-18622.04.patch, 
> HIVE-18622.05.patch, HIVE-18622.06.patch, HIVE-18622.07.patch, 
> HIVE-18622.08.patch, HIVE-18622.09.patch, HIVE-18622.091.patch, 
> HIVE-18622.092.patch, HIVE-18622.093.patch, HIVE-18622.094.patch, 
> HIVE-18622.095.patch, HIVE-18622.096.patch, HIVE-18622.097.patch, 
> HIVE-18622.098.patch, HIVE-18622.099.patch, HIVE-18622.0991.patch
>
>
>  
>  Many vector expression classes are setting noNulls to true which does not 
> work if the VRB is a scratch column being reused. The previous use may have 
> set noNulls to false and the isNull array will have some rows marked as NULL. 
> The result is wrong query results and sometimes NPEs (for BytesColumnVector).
> So, many vector expressions need this:
> {code:java}
>   // Carefully handle NULLs...
>   /*
>* For better performance on LONG/DOUBLE we don't want the conditional
>* statements inside the for loop.
>*/
>   outputColVector.noNulls = false;
>  {code}
> And, vector expressions need to make sure the isNull array entry is set when 
> outputColVector.noNulls is false.
> And, all place that assign column value need to set noNulls to false when the 
> value is NULL.
> Almost all cases where noNulls is set to true are incorrect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18541) Secure HS2 web UI with PAM

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366465#comment-16366465
 ] 

Hive QA commented on HIVE-18541:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12910762/HIVE-18541.8.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 13790 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
 (batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=78)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
 (batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] 
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
 (batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
 (batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=160)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=121)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut 
(batchId=221)
org.apache.hadoop.hive.ql.TestAcidOnTez.testGetSplitsLocks (batchId=224)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=187)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.alterIndex 
(batchId=242)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.createIndex 
(batchId=242)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropIndex 
(batchId=242)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd 
(batchId=235)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveAndKill 
(batchId=235)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveBackKill 
(batchId=235)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveConflictKill
 (batchId=235)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9237/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9237/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9237/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12910762 - PreCommit-HIVE-Build

> Secure HS2 web UI with PAM
> --
>
> Key: HIVE-18541
> URL: https://issues.apache.org/jira/browse/HIVE-18541
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18541.1.patch, HIVE-18541.2.patch, 
> HIVE-18541.5.patch, HIVE-18541.6.patch, HIVE-18541.7.patch, HIVE-18541.8.patch
>
>
> Secure HS2 web UI with PAM. Add  property
>  * {{hive.server2.webui.use.pam}}
>  * Default value: {{false}}
>  * Description: If {{true}}, the HiveServer2 WebUI will be secured with PAM



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18588) Add 'checkin' profile that runs slower tests in standalone-metastore

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-18588:
--
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Patch committed.  Thanks [~pvary] and [~szita] for your help with this.

> Add 'checkin' profile that runs slower tests in standalone-metastore
> 
>
> Key: HIVE-18588
> URL: https://issues.apache.org/jira/browse/HIVE-18588
> Project: Hive
>  Issue Type: Test
>  Components: Standalone Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18588.2.patch, HIVE-18588.3.patch, HIVE-18588.patch
>
>
> Runtime for unit tests in standalone-metastore are now exceeding 25 minutes.  
> Ideally unit tests should finish within 2-3 minutes so users will run them 
> frequently.  To solve this I propose to carve off many of the slower tests to 
> run in a new 'checkin' profile.  This profile should be run before checkin 
> and by the ptest infrastructure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-6905) Implement Auto increment, primary-foreign Key, not null constraints and default value in Hive Table columns

[Vineet Garg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366449#comment-16366449
 ] 

Vineet Garg commented on HIVE-6905:
---

[~grw], [~patelmanjun...@gmail.com] [~ppm] [~richardlloyd] FYI I recently 
committed a patch(HIVE-16605) which enforces NOT NULL constraint. I am also 
working on adding DEFAULT (HIVE-18726)

> Implement  Auto increment, primary-foreign Key, not null constraints and 
> default value in Hive Table columns
> 
>
> Key: HIVE-6905
> URL: https://issues.apache.org/jira/browse/HIVE-6905
> Project: Hive
>  Issue Type: New Feature
>  Components: Database/Schema
>Affects Versions: 0.14.0
>Reporter: Pardeep Kumar
>Priority: Major
>
> For Hive to replace a modern datawarehouse based on RDBMS, it must have 
> support for keys, constraints, auto-increment values, surrogate keys and not 
> null features etc. Many customers do not move their EDW to Hive due to these 
> reasons as these have been challenging to maintain in Hive.
> This must be implemented once https://issues.apache.org/jira/browse/HIVE-5317 
> for Updates, Deletes and Inserts are done in Hive. This should be next step 
> for Hive enhancement to take it closer to a very wide mainstream adoption..



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18726) Implement DEFAULT constraint

[Vineet Garg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366445#comment-16366445
 ] 

Vineet Garg commented on HIVE-18726:


Attached is initial patch which adds support for DEFAULT constraint for DDL 
statements (CREATE TABLE, ALTER etc.)
This patch has yet to implement the following:
* Metastore schema changes for all databases beside DERBY.
* Planner changes to change the plan to insert DEFAULT
* Type checking to make sure DEFAULT expression/value type is compatible with 
the column values.
* More tests

> Implement DEFAULT constraint
> 
>
> Key: HIVE-18726
> URL: https://issues.apache.org/jira/browse/HIVE-18726
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Planning, Query Processor
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18726.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18726) Implement DEFAULT constraint

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-18726:
---
Status: Patch Available  (was: Open)

> Implement DEFAULT constraint
> 
>
> Key: HIVE-18726
> URL: https://issues.apache.org/jira/browse/HIVE-18726
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Planning, Query Processor
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18726.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18726) Implement DEFAULT constraint

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-18726:
---
Attachment: HIVE-18726.1.patch

> Implement DEFAULT constraint
> 
>
> Key: HIVE-18726
> URL: https://issues.apache.org/jira/browse/HIVE-18726
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Planning, Query Processor
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18726.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-18727) Update GenericUDFEnforceNotNullConstraint to throw an ERROR instead of Exception on failure

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg reassigned HIVE-18727:
--


> Update GenericUDFEnforceNotNullConstraint to throw an ERROR instead of 
> Exception on failure
> ---
>
> Key: HIVE-18727
> URL: https://issues.apache.org/jira/browse/HIVE-18727
> Project: Hive
>  Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 3.0.0
>
>
> Throwing an exception makes TezProcessor stop retrying the task. Since this 
> is NOT NULL constraint violation we don't want TezProcessor to keep retrying 
> on failure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-18726) Implement DEFAULT constraint

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg reassigned HIVE-18726:
--


> Implement DEFAULT constraint
> 
>
> Key: HIVE-18726
> URL: https://issues.apache.org/jira/browse/HIVE-18726
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Planning, Query Processor
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 3.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18693) Snapshot Isolation does not work for Micromanaged table when a insert transaction is aborted

[Steve Yeom (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Yeom updated HIVE-18693:
--
Attachment: HIVE-18693.02.patch

> Snapshot Isolation does not work for Micromanaged table when a insert 
> transaction is aborted
> 
>
> Key: HIVE-18693
> URL: https://issues.apache.org/jira/browse/HIVE-18693
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-18693.01.patch, HIVE-18693.02.patch
>
>
> TestTxnCommands2#writeBetweenWorkerAndCleaner with minor 
> changes (changing delete command to insert command) fails on MM table.
> Specifically the last SELECT commands returns wrong results. 
> But this test works fine with full ACID table. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18693) Snapshot Isolation does not work for Micromanaged table when a insert transaction is aborted

[Steve Yeom (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Yeom updated HIVE-18693:
--
Status: Patch Available  (was: Open)

> Snapshot Isolation does not work for Micromanaged table when a insert 
> transaction is aborted
> 
>
> Key: HIVE-18693
> URL: https://issues.apache.org/jira/browse/HIVE-18693
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-18693.01.patch, HIVE-18693.02.patch
>
>
> TestTxnCommands2#writeBetweenWorkerAndCleaner with minor 
> changes (changing delete command to insert command) fails on MM table.
> Specifically the last SELECT commands returns wrong results. 
> But this test works fine with full ACID table. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17871) Add non nullability flag to druid time column

[slim bouguerra (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366432#comment-16366432
 ] 

slim bouguerra commented on HIVE-17871:
---

[~jcamachorodriguez] and [~ashutoshc] uploaded new patch please take a look.

> Add non nullability flag to druid time column
> -
>
> Key: HIVE-17871
> URL: https://issues.apache.org/jira/browse/HIVE-17871
> Project: Hive
>  Issue Type: Improvement
>  Components: Druid integration
>Reporter: slim bouguerra
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-17871.2.patch, HIVE-17871.3.patch, HIVE-17871.patch
>
>
> Druid time column is non null all the time.
> Adding the non nullability flag will enable extra calcite goodness  like 
> transforming 
> {code} select count(`__time`) from table {code} to {code} select count(*) 
> from table {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-17871) Add non nullability flag to druid time column

[slim bouguerra (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

slim bouguerra updated HIVE-17871:
--
Attachment: HIVE-17871.3.patch

> Add non nullability flag to druid time column
> -
>
> Key: HIVE-17871
> URL: https://issues.apache.org/jira/browse/HIVE-17871
> Project: Hive
>  Issue Type: Improvement
>  Components: Druid integration
>Reporter: slim bouguerra
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-17871.2.patch, HIVE-17871.3.patch, HIVE-17871.patch
>
>
> Druid time column is non null all the time.
> Adding the non nullability flag will enable extra calcite goodness  like 
> transforming 
> {code} select count(`__time`) from table {code} to {code} select count(*) 
> from table {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18387) Minimize time that REBUILD locks the materialized view

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-18387:
---
Attachment: (was: HIVE-18387.10.patch)

> Minimize time that REBUILD locks the materialized view
> --
>
> Key: HIVE-18387
> URL: https://issues.apache.org/jira/browse/HIVE-18387
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18387.01.patch, HIVE-18387.02.patch, 
> HIVE-18387.03.patch, HIVE-18387.04.patch, HIVE-18387.05.patch, 
> HIVE-18387.07.patch, HIVE-18387.08.patch, HIVE-18387.09.patch, 
> HIVE-18387.11.patch, HIVE-18387.patch
>
>
> Currently, REBUILD will block the materialized view while the final move task 
> is being executed. The idea for this improvement is to create the new 
> materialization in a new folder (new version) and then just flip the pointer 
> to the folder in the MV definition in the metastore. REBUILD operations for a 
> given MV should get an exclusive lock though, i.e., they cannot be executed 
> concurrently.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18387) Minimize time that REBUILD locks the materialized view

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-18387:
---
Attachment: HIVE-18387.11.patch

> Minimize time that REBUILD locks the materialized view
> --
>
> Key: HIVE-18387
> URL: https://issues.apache.org/jira/browse/HIVE-18387
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18387.01.patch, HIVE-18387.02.patch, 
> HIVE-18387.03.patch, HIVE-18387.04.patch, HIVE-18387.05.patch, 
> HIVE-18387.07.patch, HIVE-18387.08.patch, HIVE-18387.09.patch, 
> HIVE-18387.11.patch, HIVE-18387.patch
>
>
> Currently, REBUILD will block the materialized view while the final move task 
> is being executed. The idea for this improvement is to create the new 
> materialization in a new folder (new version) and then just flip the pointer 
> to the folder in the MV definition in the metastore. REBUILD operations for a 
> given MV should get an exclusive lock though, i.e., they cannot be executed 
> concurrently.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18541) Secure HS2 web UI with PAM

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366415#comment-16366415
 ] 

Hive QA commented on HIVE-18541:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
20s{color} | {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
20s{color} | {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 20s{color} 
| {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
15s{color} | {color:red} common: The patch generated 18 new + 435 unchanged - 0 
fixed = 453 total (was 435) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
13s{color} | {color:red} service: The patch generated 20 new + 20 unchanged - 0 
fixed = 40 total (was 20) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 23s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  xml  compile  findbugs  
checkstyle  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 2178112 |
| Default Java | 1.8.0_111 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9237/yetus/patch-mvninstall-service.txt
 |
| compile | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9237/yetus/patch-compile-service.txt
 |
| javac | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9237/yetus/patch-compile-service.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9237/yetus/diff-checkstyle-common.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9237/yetus/diff-checkstyle-service.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9237/yetus/patch-asflicense-problems.txt
 |
| modules | C: common service U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9237/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Secure HS2 web UI with PAM
> --
>
> Key: HIVE-18541
> URL: https://issues.apache.org/jira/browse/HIVE-18541
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18541.1.patch, HIVE-18541.2.patch, 
> HIVE-18541.5.patch, HIVE-18541.6.patch, HIVE-18541.7.patch, HIVE-18541.8.patch
>
>
> Secure HS2 web UI with 

[jira] [Updated] (HIVE-18724) Improve error handling for subqueries referencing columns(correlated) of its grand-parent query

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-18724:
---
Summary: Improve error handling for subqueries referencing 
columns(correlated) of its grand-parent query  (was: Improve error handling for 
subqueries referencing columns(correlated) of it's grand-parent query)

> Improve error handling for subqueries referencing columns(correlated) of its 
> grand-parent query
> ---
>
> Key: HIVE-18724
> URL: https://issues.apache.org/jira/browse/HIVE-18724
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Vineet Garg
>Priority: Major
>
> Currently Hive doesn't allow correlated subqueries which refer to columns of 
> grand parent query e.g.
> {code:sql}
> select t1.a from t t1 where t1.b IN (select t2.c from t t2 where t2.a IN 
> (select max(t3.b) from t t3, t t4 where t3.a=t2.a and t3.b=t1.b))
>  {code}
> In this example inner most query has column reference {{t3.b=t1.b}} where 
> {{t1.b}} references outer most query.
> The error which Hive compiler throw is 
> {noformat}
> FAILED: SemanticException Line 0:-1 Unsupported SubQuery Expression 'a': 
> Nested SubQuery expressions are not supported.
> {noformat}
> This is very misleading since Hive does support Nested subqueries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18724) Improve error handling for subqueries referencing columns(correlated) of it's grand-parent query

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-18724:
---
Description: 
Currently Hive doesn't allow correlated subqueries which refer to columns of 
grand parent query e.g.

{code:sql}
select t1.a from t t1 where t1.b IN (select t2.c from t t2 where t2.a IN 
(select max(t3.b) from t t3, t t4 where t3.a=t2.a and t3.b=t1.b))
 {code}

In this example inner most query has column reference {{t3.b=t1.b}} where 
{{t1.b}} references outer most query.

The error which Hive compiler throw is 
{noformat}
FAILED: SemanticException Line 0:-1 Unsupported SubQuery Expression 'a': Nested 
SubQuery expressions are not supported.
{noformat}

This is very misleading since Hive does support Nested subqueries.

  was:
Currently Hive doesn't allow correlated subqueries which refer to columns of 
grand parent query e.g.

{code:sql}
select t1.a from t t1 where t1.b IN (select t2.c from t t2 where t2.a IN 
(select max(t3.b) from t t3, t t4 where t3.a=t2.a and t3.b=t1.b))
 {code}

In this example inner most query has column reference {{t3.b=t1.b}} where 
{{t1.b}} references outer most query.

The error which Hive compiler throw is 
{noformat}
FAILED: SemanticException Line 0:-1 Unsupported SubQuery Expression 'a': Nested 
SubQuery expressions are not supported.
{format}

This is very misleading since Hive does support Nested subqueries.


> Improve error handling for subqueries referencing columns(correlated) of it's 
> grand-parent query
> 
>
> Key: HIVE-18724
> URL: https://issues.apache.org/jira/browse/HIVE-18724
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Vineet Garg
>Priority: Major
>
> Currently Hive doesn't allow correlated subqueries which refer to columns of 
> grand parent query e.g.
> {code:sql}
> select t1.a from t t1 where t1.b IN (select t2.c from t t2 where t2.a IN 
> (select max(t3.b) from t t3, t t4 where t3.a=t2.a and t3.b=t1.b))
>  {code}
> In this example inner most query has column reference {{t3.b=t1.b}} where 
> {{t1.b}} references outer most query.
> The error which Hive compiler throw is 
> {noformat}
> FAILED: SemanticException Line 0:-1 Unsupported SubQuery Expression 'a': 
> Nested SubQuery expressions are not supported.
> {noformat}
> This is very misleading since Hive does support Nested subqueries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18387) Minimize time that REBUILD locks the materialized view

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-18387:
---
Attachment: HIVE-18387.10.patch

> Minimize time that REBUILD locks the materialized view
> --
>
> Key: HIVE-18387
> URL: https://issues.apache.org/jira/browse/HIVE-18387
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18387.01.patch, HIVE-18387.02.patch, 
> HIVE-18387.03.patch, HIVE-18387.04.patch, HIVE-18387.05.patch, 
> HIVE-18387.07.patch, HIVE-18387.08.patch, HIVE-18387.09.patch, 
> HIVE-18387.10.patch, HIVE-18387.patch
>
>
> Currently, REBUILD will block the materialized view while the final move task 
> is being executed. The idea for this improvement is to create the new 
> materialization in a new folder (new version) and then just flip the pointer 
> to the folder in the MV definition in the metastore. REBUILD operations for a 
> given MV should get an exclusive lock though, i.e., they cannot be executed 
> concurrently.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18672) Printed state in RemoteSparkJobMonitor is ambiguous

[Sahil Takiar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar updated HIVE-18672:

   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Pushed to master. Thanks Peter for the review.

> Printed state in RemoteSparkJobMonitor is ambiguous
> ---
>
> Key: HIVE-18672
> URL: https://issues.apache.org/jira/browse/HIVE-18672
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18672.1.patch, HIVE-18672.2.patch, 
> HIVE-18672.3.patch
>
>
> There are a few places in {{RemoteSparkJobMonitor}} (e.g. when the Spark job 
> is in state QUEUED) where the state of the Spark job is printed, but the info 
> is ambiguous (no reference to HoS, or the id of the Spark job).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18447) JDBC: Provide a way for JDBC users to pass cookie info via connection string

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-18447:

Release Note:   (was: Cookies can now be passed like:
{code}
jdbc:hive2://:/;transportMode=http;httpPath=;http.cookie.=;http.cookie.=
{code})

> JDBC: Provide a way for JDBC users to pass cookie info via connection string
> 
>
> Key: HIVE-18447
> URL: https://issues.apache.org/jira/browse/HIVE-18447
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-18447.1.patch, HIVE-18447.2.patch
>
>
> Some authentication mechanisms like Single Sign On, need the ability to pass 
> a cookie to some intermediate authentication service like Knox via the JDBC 
> driver. We need to add the mechanism in Hive's JDBC driver (when used in HTTP 
> transport mode).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18447) JDBC: Provide a way for JDBC users to pass cookie info via connection string

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-18447:

Description: 
Some authentication mechanisms like Single Sign On, need the ability to pass a 
cookie to some intermediate authentication service like Knox via the JDBC 
driver. We need to add the mechanism in Hive's JDBC driver (when used in HTTP 
transport mode).

Cookies can now be passed like:
{code}
jdbc:hive2://:/;transportMode=http;httpPath=;http.cookie.=;http.cookie.=
{code}

  was:Some authentication mechanisms like Single Sign On, need the ability to 
pass a cookie to some intermediate authentication service like Knox via the 
JDBC driver. We need to add the mechanism in Hive's JDBC driver (when used in 
HTTP transport mode).


> JDBC: Provide a way for JDBC users to pass cookie info via connection string
> 
>
> Key: HIVE-18447
> URL: https://issues.apache.org/jira/browse/HIVE-18447
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-18447.1.patch, HIVE-18447.2.patch
>
>
> Some authentication mechanisms like Single Sign On, need the ability to pass 
> a cookie to some intermediate authentication service like Knox via the JDBC 
> driver. We need to add the mechanism in Hive's JDBC driver (when used in HTTP 
> transport mode).
> Cookies can now be passed like:
> {code}
> jdbc:hive2://:/;transportMode=http;httpPath=;http.cookie.=;http.cookie.=
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-16125) Split work between reducers.

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366363#comment-16366363
 ] 

Ashutosh Chauhan commented on HIVE-16125:
-

+1

> Split work between reducers.
> 
>
> Key: HIVE-16125
> URL: https://issues.apache.org/jira/browse/HIVE-16125
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-16125.4.patch, HIVE-16125.5.patch, 
> HIVE-16125.6.patch, HIVE-16125.7.patch, HIVE-16125.patch
>
>
> Split work between reducer.
> currently we have one reducer per segment granularity even if the interval 
> will be partitioned over multiple partitions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18573) Use proper Calcite operator instead of UDFs

[slim bouguerra (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

slim bouguerra updated HIVE-18573:
--
Attachment: (was: HIVE-18573.3.patch)

> Use proper Calcite operator instead of UDFs
> ---
>
> Key: HIVE-18573
> URL: https://issues.apache.org/jira/browse/HIVE-18573
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: slim bouguerra
>Assignee: Ashutosh Chauhan
>Priority: Major
> Attachments: HIVE-18573.4.patch, HIVE-18573.5.patch, 
> HIVE-18573.6.patch, HIVE-18573.7.patch, HIVE-18573.7.patch, 
> HIVE-18573.8.patch, HIVE-18573.9.patch, HIVE-18573.patch
>
>
> Currently, Hive is mostly using user-defined black box sql operators during 
> Query planning. It will be more beneficial to use proper calcite operators, 
> this is to prepare the ground for pushing complex expressions to 
> Druid-calcite adapter.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18573) Use proper Calcite operator instead of UDFs

[slim bouguerra (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

slim bouguerra updated HIVE-18573:
--
Attachment: HIVE-18573.9.patch

> Use proper Calcite operator instead of UDFs
> ---
>
> Key: HIVE-18573
> URL: https://issues.apache.org/jira/browse/HIVE-18573
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: slim bouguerra
>Assignee: Ashutosh Chauhan
>Priority: Major
> Attachments: HIVE-18573.4.patch, HIVE-18573.5.patch, 
> HIVE-18573.6.patch, HIVE-18573.7.patch, HIVE-18573.7.patch, 
> HIVE-18573.8.patch, HIVE-18573.9.patch, HIVE-18573.patch
>
>
> Currently, Hive is mostly using user-defined black box sql operators during 
> Query planning. It will be more beneficial to use proper calcite operators, 
> this is to prepare the ground for pushing complex expressions to 
> Druid-calcite adapter.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18573) Use proper Calcite operator instead of UDFs

[slim bouguerra (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

slim bouguerra updated HIVE-18573:
--
Attachment: (was: HIVE-18573.2.patch)

> Use proper Calcite operator instead of UDFs
> ---
>
> Key: HIVE-18573
> URL: https://issues.apache.org/jira/browse/HIVE-18573
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: slim bouguerra
>Assignee: Ashutosh Chauhan
>Priority: Major
> Attachments: HIVE-18573.4.patch, HIVE-18573.5.patch, 
> HIVE-18573.6.patch, HIVE-18573.7.patch, HIVE-18573.7.patch, 
> HIVE-18573.8.patch, HIVE-18573.9.patch, HIVE-18573.patch
>
>
> Currently, Hive is mostly using user-defined black box sql operators during 
> Query planning. It will be more beneficial to use proper calcite operators, 
> this is to prepare the ground for pushing complex expressions to 
> Druid-calcite adapter.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18693) Snapshot Isolation does not work for Micromanaged table when a insert transaction is aborted

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366352#comment-16366352
 ] 

Eugene Koifman commented on HIVE-18693:
---

Suppose you have 200 txns all of which committed (with this patch).  How will 
Initiator handle all the data in completed_txn_components?  

> Snapshot Isolation does not work for Micromanaged table when a insert 
> transaction is aborted
> 
>
> Key: HIVE-18693
> URL: https://issues.apache.org/jira/browse/HIVE-18693
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-18693.01.patch
>
>
> TestTxnCommands2#writeBetweenWorkerAndCleaner with minor 
> changes (changing delete command to insert command) fails on MM table.
> Specifically the last SELECT commands returns wrong results. 
> But this test works fine with full ACID table. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18595) UNIX_TIMESTAMP UDF fails when type is Timestamp with local timezone

[slim bouguerra (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

slim bouguerra updated HIVE-18595:
--
Attachment: HIVE-18595.3.patch

> UNIX_TIMESTAMP  UDF fails when type is Timestamp with local timezone
> 
>
> Key: HIVE-18595
> URL: https://issues.apache.org/jira/browse/HIVE-18595
> Project: Hive
>  Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18595.3.patch, HIVE-18595.3.patch, 
> HIVE-18595.patch, HIVE-18595.patch
>
>
> {code}
> 2018-01-31T12:59:45,464 ERROR [10e97c86-7f90-406b-a8fa-38be5d3529cc main] 
> ql.Driver: FAILED: SemanticException [Error 10014]: Line 3:456 Wrong 
> arguments ''-MM-dd HH:mm:ss'': The function UNIX_TIMESTAMP takes only 
> string/date/timestamp types
> org.apache.hadoop.hive.ql.parse.SemanticException: Line 3:456 Wrong arguments 
> ''-MM-dd HH:mm:ss'': The function UNIX_TIMESTAMP takes only 
> string/date/timestamp types
>  at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1394)
>  at 
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
>  at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
>  at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
>  at 
> org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76)
>  at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
>  at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:235)
>  at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:181)
>  at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:11847)
>  at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:11780)
>  at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genGBLogicalPlan(CalcitePlanner.java:3140)
>  at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:4330)
>  at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1407)
>  at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1354)
>  at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:118)
>  at 
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:1052)
>  at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:154)
>  at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:111)
>  at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1159)
>  at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:1175)
>  at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:422)
>  at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11393)
>  at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:304)
>  at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:268)
>  at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:163)
>  at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:268)
>  at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:639)
>  at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1504)
>  at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1632)
>  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1395)
>  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1382)
>  at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:240)
>  at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
>  at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:410)
>  at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:343)
>  at 
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1331)
>  at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1305)
>  at 
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:173)
>  at org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
>  at 
> org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver(TestMiniDruidCliDriver.java:59)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at 
> 

[jira] [Updated] (HIVE-16125) Split work between reducers.

[slim bouguerra (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

slim bouguerra updated HIVE-16125:
--
Attachment: HIVE-16125.7.patch

> Split work between reducers.
> 
>
> Key: HIVE-16125
> URL: https://issues.apache.org/jira/browse/HIVE-16125
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-16125.4.patch, HIVE-16125.5.patch, 
> HIVE-16125.6.patch, HIVE-16125.7.patch, HIVE-16125.patch
>
>
> Split work between reducer.
> currently we have one reducer per segment granularity even if the interval 
> will be partitioned over multiple partitions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly

[Matt McCline (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366328#comment-16366328
 ] 

Matt McCline commented on HIVE-18622:
-

Yes, the bugs have been around for years.

> Vectorization: IF Statements, Comparisons, and more do not handle NULLs 
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.03.patch, HIVE-18622.04.patch, 
> HIVE-18622.05.patch, HIVE-18622.06.patch, HIVE-18622.07.patch, 
> HIVE-18622.08.patch, HIVE-18622.09.patch, HIVE-18622.091.patch, 
> HIVE-18622.092.patch, HIVE-18622.093.patch, HIVE-18622.094.patch, 
> HIVE-18622.095.patch, HIVE-18622.096.patch, HIVE-18622.097.patch, 
> HIVE-18622.098.patch, HIVE-18622.099.patch, HIVE-18622.0991.patch
>
>
>  
>  Many vector expression classes are setting noNulls to true which does not 
> work if the VRB is a scratch column being reused. The previous use may have 
> set noNulls to false and the isNull array will have some rows marked as NULL. 
> The result is wrong query results and sometimes NPEs (for BytesColumnVector).
> So, many vector expressions need this:
> {code:java}
>   // Carefully handle NULLs...
>   /*
>* For better performance on LONG/DOUBLE we don't want the conditional
>* statements inside the for loop.
>*/
>   outputColVector.noNulls = false;
>  {code}
> And, vector expressions need to make sure the isNull array entry is set when 
> outputColVector.noNulls is false.
> And, all place that assign column value need to set noNulls to false when the 
> value is NULL.
> Almost all cases where noNulls is set to true are incorrect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366326#comment-16366326
 ] 

Vihang Karajgaonkar commented on HIVE-18622:


Hi [~mmccline] do you know if this issue applies to branch-2 as well? Thanks

> Vectorization: IF Statements, Comparisons, and more do not handle NULLs 
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.03.patch, HIVE-18622.04.patch, 
> HIVE-18622.05.patch, HIVE-18622.06.patch, HIVE-18622.07.patch, 
> HIVE-18622.08.patch, HIVE-18622.09.patch, HIVE-18622.091.patch, 
> HIVE-18622.092.patch, HIVE-18622.093.patch, HIVE-18622.094.patch, 
> HIVE-18622.095.patch, HIVE-18622.096.patch, HIVE-18622.097.patch, 
> HIVE-18622.098.patch, HIVE-18622.099.patch, HIVE-18622.0991.patch
>
>
>  
>  Many vector expression classes are setting noNulls to true which does not 
> work if the VRB is a scratch column being reused. The previous use may have 
> set noNulls to false and the isNull array will have some rows marked as NULL. 
> The result is wrong query results and sometimes NPEs (for BytesColumnVector).
> So, many vector expressions need this:
> {code:java}
>   // Carefully handle NULLs...
>   /*
>* For better performance on LONG/DOUBLE we don't want the conditional
>* statements inside the for loop.
>*/
>   outputColVector.noNulls = false;
>  {code}
> And, vector expressions need to make sure the isNull array entry is set when 
> outputColVector.noNulls is false.
> And, all place that assign column value need to set noNulls to false when the 
> value is NULL.
> Almost all cases where noNulls is set to true are incorrect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly

[Deepak Jaiswal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366320#comment-16366320
 ] 

Deepak Jaiswal commented on HIVE-18622:
---

[~sershe] Matt opened a new Jira HIVE-18722 to track the test.

> Vectorization: IF Statements, Comparisons, and more do not handle NULLs 
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.03.patch, HIVE-18622.04.patch, 
> HIVE-18622.05.patch, HIVE-18622.06.patch, HIVE-18622.07.patch, 
> HIVE-18622.08.patch, HIVE-18622.09.patch, HIVE-18622.091.patch, 
> HIVE-18622.092.patch, HIVE-18622.093.patch, HIVE-18622.094.patch, 
> HIVE-18622.095.patch, HIVE-18622.096.patch, HIVE-18622.097.patch, 
> HIVE-18622.098.patch, HIVE-18622.099.patch, HIVE-18622.0991.patch
>
>
>  
>  Many vector expression classes are setting noNulls to true which does not 
> work if the VRB is a scratch column being reused. The previous use may have 
> set noNulls to false and the isNull array will have some rows marked as NULL. 
> The result is wrong query results and sometimes NPEs (for BytesColumnVector).
> So, many vector expressions need this:
> {code:java}
>   // Carefully handle NULLs...
>   /*
>* For better performance on LONG/DOUBLE we don't want the conditional
>* statements inside the for loop.
>*/
>   outputColVector.noNulls = false;
>  {code}
> And, vector expressions need to make sure the isNull array entry is set when 
> outputColVector.noNulls is false.
> And, all place that assign column value need to set noNulls to false when the 
> value is NULL.
> Almost all cases where noNulls is set to true are incorrect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18693) Snapshot Isolation does not work for Micromanaged table when a insert transaction is aborted

[Steve Yeom (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366301#comment-16366301
 ] 

Steve Yeom commented on HIVE-18693:
---

[~ekoifman] can you look at the patch? 
Thanks, 
Steve. 

> Snapshot Isolation does not work for Micromanaged table when a insert 
> transaction is aborted
> 
>
> Key: HIVE-18693
> URL: https://issues.apache.org/jira/browse/HIVE-18693
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-18693.01.patch
>
>
> TestTxnCommands2#writeBetweenWorkerAndCleaner with minor 
> changes (changing delete command to insert command) fails on MM table.
> Specifically the last SELECT commands returns wrong results. 
> But this test works fine with full ACID table. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18693) Snapshot Isolation does not work for Micromanaged table when a insert transaction is aborted

[Steve Yeom (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366296#comment-16366296
 ] 

Steve Yeom commented on HIVE-18693:
---

But from the perspective of Atomicity, in case of having orphan transaction 
after client dies, we need a mechanism.
Keeping table/partition names in the Metastore is easiest given that file 
system directory level metadata is limited 
due to append only operations' being allowed. But recovery manager and metadata 
cache might be an alternative.



> Snapshot Isolation does not work for Micromanaged table when a insert 
> transaction is aborted
> 
>
> Key: HIVE-18693
> URL: https://issues.apache.org/jira/browse/HIVE-18693
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-18693.01.patch
>
>
> TestTxnCommands2#writeBetweenWorkerAndCleaner with minor 
> changes (changing delete command to insert command) fails on MM table.
> Specifically the last SELECT commands returns wrong results. 
> But this test works fine with full ACID table. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-15077) Acid LockManager is unfair

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366255#comment-16366255
 ] 

Eugene Koifman commented on HIVE-15077:
---

no related failures
[~alangates] could you review please

> Acid LockManager is unfair
> --
>
> Key: HIVE-15077
> URL: https://issues.apache.org/jira/browse/HIVE-15077
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 2.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-15077.02.patch
>
>
> HIVE-10242 made the acid LM unfair.
> In TxnHandler.checkLock(), suppose we are trying to acquire SR5  (the number 
> is extLockId).  
> Then 
> LockInfo[] locks = lockSet.toArray(new LockInfo[lockSet.size()]);
> may look like this (all explicitly listed locks are in Waiting state)
> {, SR5 SW3 X4}
> So the algorithm will find SR5 in the list and start looking backwards (to 
> the left).
> According to IDs, SR5 should wait for X4 to be granted but X4 won't even be 
> examined and so SR5 may be granted.
> Theoretically, this could cause starvation.
> The query that generates the list already has
> query.append(" and hl_lock_ext_id <= ").append(extLockId);
> but it should use "<" rather than "<=" to exclude the locks being checked 
> from "locks" list which will make the algorithm look at all locks "in front" 
> of a given lock.
> Here is an example (add to TestDbTxnManager2)
> {noformat}
>   @Test
>   public void testFairness2() throws Exception {
> dropTable(new String[]{"T7"});
> CommandProcessorResponse cpr = driver.run("create table if not exists T7 
> (a int) partitioned by (p int) stored as orc TBLPROPERTIES 
> ('transactional'='true')");
> checkCmdOnDriver(cpr);
> checkCmdOnDriver(driver.run("insert into T7 partition(p) 
> values(1,1),(1,2)"));//create 2 partitions
> cpr = driver.compileAndRespond("select a from T7 ");
> checkCmdOnDriver(cpr);
> txnMgr.acquireLocks(driver.getPlan(), ctx, "Fifer");//gets S lock on T7
> HiveTxnManager txnMgr2 = 
> TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
> swapTxnManager(txnMgr2);
> cpr = driver.compileAndRespond("alter table T7 drop partition (p=1)");
> checkCmdOnDriver(cpr);
> //tries to get X lock on T7.p=1 and gets Waiting state
> LockState lockState = ((DbTxnManager) 
> txnMgr2).acquireLocks(driver.getPlan(), ctx, "Fiddler", false);
> List locks = getLocks();
> Assert.assertEquals("Unexpected lock count", 4, locks.size());
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7", 
> null, locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7", 
> "p=1", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7", 
> "p=2", locks);
> checkLock(LockType.EXCLUSIVE, LockState.WAITING, "default", "T7", "p=1", 
> locks);
> HiveTxnManager txnMgr3 = 
> TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
> swapTxnManager(txnMgr3);
> //this should block behind the X lock on  T7.p=1
> cpr = driver.compileAndRespond("select a from T7");
> checkCmdOnDriver(cpr);
> txnMgr3.acquireLocks(driver.getPlan(), ctx, "Fifer");//gets S lock on T6
> locks = getLocks();
> Assert.assertEquals("Unexpected lock count", 7, locks.size());
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7", 
> null, locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7", 
> "p=1", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7", 
> "p=2", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7", 
> null, locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7", 
> "p=1", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7", 
> "p=2", locks);
> checkLock(LockType.EXCLUSIVE, LockState.WAITING, "default", "T7", "p=1", 
> locks);
>   }
> {noformat}
> The 2nd {{locks = getLocks();}} output shows that all locks for the 2nd 
> {{select * from T7}} are all acquired while they should block behind the X 
> lock to be fair.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18659) add acid version marker to acid files/directories

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18659:
--
Attachment: HIVE-18659.06.patch

> add acid version marker to acid files/directories
> -
>
> Key: HIVE-18659
> URL: https://issues.apache.org/jira/browse/HIVE-18659
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18659.01.patch, HIVE-18659.04.patch, 
> HIVE-18659.05.patch, HIVE-18659.06.patch
>
>
> add acid version marker to acid files so that we know which version of acid 
> wrote the file



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HIVE-18556) bucket_map_join_tez1 is failing with plan diffs

[Deepak Jaiswal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepak Jaiswal resolved HIVE-18556.
---
Resolution: Duplicate

Fixed in HIVE-18698

> bucket_map_join_tez1 is failing with plan diffs
> ---
>
> Key: HIVE-18556
> URL: https://issues.apache.org/jira/browse/HIVE-18556
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
>
> bucket_map_join_tez1 is failing with plan diffs likely due to stats.
> Need to evaluate configs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18541) Secure HS2 web UI with PAM

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366225#comment-16366225
 ] 

Hive QA commented on HIVE-18541:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12910762/HIVE-18541.8.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 13789 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
 (batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] 
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=78)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
 (batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] 
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
 (batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
 (batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_recursive_mapjoin]
 (batchId=179)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=121)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut 
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestFunctions.testGetFunctionNullDatabase[Embedded]
 (batchId=205)
org.apache.hadoop.hive.ql.TestAcidOnTez.testGetSplitsLocks (batchId=224)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=187)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.alterIndex 
(batchId=242)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.createIndex 
(batchId=242)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropIndex 
(batchId=242)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd 
(batchId=235)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveConflictKill
 (batchId=235)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9236/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9236/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9236/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12910762 - PreCommit-HIVE-Build

> Secure HS2 web UI with PAM
> --
>
> Key: HIVE-18541
> URL: https://issues.apache.org/jira/browse/HIVE-18541
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18541.1.patch, HIVE-18541.2.patch, 
> HIVE-18541.5.patch, HIVE-18541.6.patch, HIVE-18541.7.patch, HIVE-18541.8.patch
>
>
> Secure HS2 web UI with PAM. Add  property
>  * {{hive.server2.webui.use.pam}}
>  * Default value: {{false}}
>  * Description: If {{true}}, the HiveServer2 WebUI will be secured with PAM



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18721) Bucket Map Join : Handle empty buckets

[Deepak Jaiswal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepak Jaiswal updated HIVE-18721:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Bucket Map Join : Handle empty buckets
> --
>
> Key: HIVE-18721
> URL: https://issues.apache.org/jira/browse/HIVE-18721
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-18721.1.patch
>
>
> Bucket Map Join needs to bluff Tez by sending empty task list for 
> DataMovementEvent for those buckets for which there is no data.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly

[Matt McCline (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366188#comment-16366188
 ] 

Matt McCline commented on HIVE-18622:
-

I had to fixup patch #091 due to HIVE-18421 minor conflicts on these files:

ColumnArithmeticColumn.txt, 

ColumnArithmeticScalar.txt,

ColumnUnaryMinus.txt

> Vectorization: IF Statements, Comparisons, and more do not handle NULLs 
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.03.patch, HIVE-18622.04.patch, 
> HIVE-18622.05.patch, HIVE-18622.06.patch, HIVE-18622.07.patch, 
> HIVE-18622.08.patch, HIVE-18622.09.patch, HIVE-18622.091.patch, 
> HIVE-18622.092.patch, HIVE-18622.093.patch, HIVE-18622.094.patch, 
> HIVE-18622.095.patch, HIVE-18622.096.patch, HIVE-18622.097.patch, 
> HIVE-18622.098.patch, HIVE-18622.099.patch, HIVE-18622.0991.patch
>
>
>  
>  Many vector expression classes are setting noNulls to true which does not 
> work if the VRB is a scratch column being reused. The previous use may have 
> set noNulls to false and the isNull array will have some rows marked as NULL. 
> The result is wrong query results and sometimes NPEs (for BytesColumnVector).
> So, many vector expressions need this:
> {code:java}
>   // Carefully handle NULLs...
>   /*
>* For better performance on LONG/DOUBLE we don't want the conditional
>* statements inside the for loop.
>*/
>   outputColVector.noNulls = false;
>  {code}
> And, vector expressions need to make sure the isNull array entry is set when 
> outputColVector.noNulls is false.
> And, all place that assign column value need to set noNulls to false when the 
> value is NULL.
> Almost all cases where noNulls is set to true are incorrect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-18722) Vectorization: Adding SUM(HASH(..)) to full query seems to produce flakey results -- need to investiage

[Gopal V (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366166#comment-16366166
 ] 

Gopal V edited comment on HIVE-18722 at 2/15/18 7:44 PM:
-

SUM(hash()) out of a Group-by query is not expected to be consistent (since 
JDK8), because it uses an iterator from a a HashMap

If you're seeing different results with {{set hive.map.aggr=false}}, then it 
might be worth investigating.

Integer operations are commutative, if they are of the same type - except it 
can be (long)(int)(a+b) & (long(a) + long(b)) within the partial result.

The GC canary in the group-by operator is the usual problem.


was (Author: gopalv):
SUM(hash()) out of a Group-by query is not expected to be consistent (since 
JDK8), because it uses an iterator from a a HashMap

If you're seeing different results with {{set hive.map.aggr=false}}, then it 
might be worth investigating.

> Vectorization: Adding SUM(HASH(..)) to full query seems to produce flakey 
> results -- need to investiage
> ---
>
> Key: HIVE-18722
> URL: https://issues.apache.org/jira/browse/HIVE-18722
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
>
> When added to HIVE-18622 changes, the query results vary from laptop results 
> when run on Hive QA cluster.  Need to investigate after HIVE-18622 commits.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18541) Secure HS2 web UI with PAM

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366170#comment-16366170
 ] 

Hive QA commented on HIVE-18541:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
20s{color} | {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
20s{color} | {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 20s{color} 
| {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
15s{color} | {color:red} common: The patch generated 18 new + 435 unchanged - 0 
fixed = 453 total (was 435) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
13s{color} | {color:red} service: The patch generated 20 new + 20 unchanged - 0 
fixed = 40 total (was 20) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 32s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  xml  compile  findbugs  
checkstyle  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / dabb62d |
| Default Java | 1.8.0_111 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9236/yetus/patch-mvninstall-service.txt
 |
| compile | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9236/yetus/patch-compile-service.txt
 |
| javac | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9236/yetus/patch-compile-service.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9236/yetus/diff-checkstyle-common.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9236/yetus/diff-checkstyle-service.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9236/yetus/patch-asflicense-problems.txt
 |
| modules | C: common service U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9236/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Secure HS2 web UI with PAM
> --
>
> Key: HIVE-18541
> URL: https://issues.apache.org/jira/browse/HIVE-18541
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18541.1.patch, HIVE-18541.2.patch, 
> HIVE-18541.5.patch, HIVE-18541.6.patch, HIVE-18541.7.patch, HIVE-18541.8.patch
>
>
> Secure HS2 web UI with 

[jira] [Updated] (HIVE-18693) Snapshot Isolation does not work for Micromanaged table when a insert transaction is aborted

[Steve Yeom (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Yeom updated HIVE-18693:
--
Summary: Snapshot Isolation does not work for Micromanaged table when a 
insert transaction is aborted  (was: 
TestTxnCommands2#writeBetweenWorkerAndCleaner fails on MM table)

> Snapshot Isolation does not work for Micromanaged table when a insert 
> transaction is aborted
> 
>
> Key: HIVE-18693
> URL: https://issues.apache.org/jira/browse/HIVE-18693
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-18693.01.patch
>
>
> TestTxnCommands2#writeBetweenWorkerAndCleaner with minor 
> changes (changing delete command to insert command) fails on MM table.
> Specifically the last SELECT commands returns wrong results. 
> But this test works fine with full ACID table. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18722) Vectorization: Adding SUM(HASH(..)) to full query seems to produce flakey results -- need to investiage

[Gopal V (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366166#comment-16366166
 ] 

Gopal V commented on HIVE-18722:


SUM(hash()) out of a Group-by query is not expected to be consistent (since 
JDK8), because it uses an iterator from a a HashMap

If you're seeing different results with {{set hive.map.aggr=false}}, then it 
might be worth investigating.

> Vectorization: Adding SUM(HASH(..)) to full query seems to produce flakey 
> results -- need to investiage
> ---
>
> Key: HIVE-18722
> URL: https://issues.apache.org/jira/browse/HIVE-18722
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
>
> When added to HIVE-18622 changes, the query results vary from laptop results 
> when run on Hive QA cluster.  Need to investigate after HIVE-18622 commits.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366160#comment-16366160
 ] 

Sergey Shelukhin commented on HIVE-18622:
-

+1 pending build/tests... not sure about the test removal, I guess it's ok to 
do in followup jira. Why was it removed?
It may be worth committing this patch because it's better than prior state and 
we can fix/optimize more stuff later.

> Vectorization: IF Statements, Comparisons, and more do not handle NULLs 
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.03.patch, HIVE-18622.04.patch, 
> HIVE-18622.05.patch, HIVE-18622.06.patch, HIVE-18622.07.patch, 
> HIVE-18622.08.patch, HIVE-18622.09.patch, HIVE-18622.091.patch, 
> HIVE-18622.092.patch, HIVE-18622.093.patch, HIVE-18622.094.patch, 
> HIVE-18622.095.patch, HIVE-18622.096.patch, HIVE-18622.097.patch, 
> HIVE-18622.098.patch, HIVE-18622.099.patch, HIVE-18622.0991.patch
>
>
>  
>  Many vector expression classes are setting noNulls to true which does not 
> work if the VRB is a scratch column being reused. The previous use may have 
> set noNulls to false and the isNull array will have some rows marked as NULL. 
> The result is wrong query results and sometimes NPEs (for BytesColumnVector).
> So, many vector expressions need this:
> {code:java}
>   // Carefully handle NULLs...
>   /*
>* For better performance on LONG/DOUBLE we don't want the conditional
>* statements inside the for loop.
>*/
>   outputColVector.noNulls = false;
>  {code}
> And, vector expressions need to make sure the isNull array entry is set when 
> outputColVector.noNulls is false.
> And, all place that assign column value need to set noNulls to false when the 
> value is NULL.
> Almost all cases where noNulls is set to true are incorrect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-18722) Vectorization: Adding SUM(HASH(..)) to full query seems to produce flakey results -- need to investiage

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline reassigned HIVE-18722:
---


> Vectorization: Adding SUM(HASH(..)) to full query seems to produce flakey 
> results -- need to investiage
> ---
>
> Key: HIVE-18722
> URL: https://issues.apache.org/jira/browse/HIVE-18722
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
>
> When added to HIVE-18622 changes, the query results vary from laptop results 
> when run on Hive QA cluster.  Need to investigate after HIVE-18622 commits.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly

[Deepak Jaiswal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366149#comment-16366149
 ] 

Deepak Jaiswal commented on HIVE-18622:
---

+1.

Please address a minor comment in the review board.

> Vectorization: IF Statements, Comparisons, and more do not handle NULLs 
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.03.patch, HIVE-18622.04.patch, 
> HIVE-18622.05.patch, HIVE-18622.06.patch, HIVE-18622.07.patch, 
> HIVE-18622.08.patch, HIVE-18622.09.patch, HIVE-18622.091.patch, 
> HIVE-18622.092.patch, HIVE-18622.093.patch, HIVE-18622.094.patch, 
> HIVE-18622.095.patch, HIVE-18622.096.patch, HIVE-18622.097.patch, 
> HIVE-18622.098.patch, HIVE-18622.099.patch, HIVE-18622.0991.patch
>
>
>  
>  Many vector expression classes are setting noNulls to true which does not 
> work if the VRB is a scratch column being reused. The previous use may have 
> set noNulls to false and the isNull array will have some rows marked as NULL. 
> The result is wrong query results and sometimes NPEs (for BytesColumnVector).
> So, many vector expressions need this:
> {code:java}
>   // Carefully handle NULLs...
>   /*
>* For better performance on LONG/DOUBLE we don't want the conditional
>* statements inside the for loop.
>*/
>   outputColVector.noNulls = false;
>  {code}
> And, vector expressions need to make sure the isNull array entry is set when 
> outputColVector.noNulls is false.
> And, all place that assign column value need to set noNulls to false when the 
> value is NULL.
> Almost all cases where noNulls is set to true are incorrect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18638) Triggers for multi-pool move, failing to initiate the move event

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-18638:
-
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

> Triggers for multi-pool move, failing to initiate the move event
> 
>
> Key: HIVE-18638
> URL: https://issues.apache.org/jira/browse/HIVE-18638
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Aswathy Chellammal Sreekumar
>Assignee: Prasanth Jayachandran
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18638.1.patch
>
>
> Resource plan with multiple pools and trigger set to move job across those 
> pools seems to be failing to do so
> Resource plan:
> {noformat}
> 1: jdbc:hive2://ctr-e137-1514896590304-51538-> show resource plan plan_2; 
> INFO : Compiling 
> command(queryId=hive_20180202220823_2fb8bca7-5b7a-48cf-8ff9-8d5f3548d334): 
> show resource plan plan_2 INFO : Semantic Analysis Completed INFO : Returning 
> Hive schema: Schema(fieldSchemas:[FieldSchema(name:line, type:string, 
> comment:from deserializer)], properties:null) INFO : Completed compiling 
> command(queryId=hive_20180202220823_2fb8bca7-5b7a-48cf-8ff9-8d5f3548d334); 
> Time taken: 0.008 seconds INFO : Executing 
> command(queryId=hive_20180202220823_2fb8bca7-5b7a-48cf-8ff9-8d5f3548d334): 
> show resource plan plan_2 INFO : Starting task [Stage-0:DDL] in serial mode 
> INFO : Completed executing 
> command(queryId=hive_20180202220823_2fb8bca7-5b7a-48cf-8ff9-8d5f3548d334); 
> Time taken: 0.196 seconds INFO : OK 
> ++ | line | 
> ++ | 
> plan_2[status=ACTIVE,parallelism=null,defaultPool=pool2] | | + 
> pool2[allocFraction=0.5,schedulingPolicy=default,parallelism=3] | | | trigger 
> too_large_write_triger: if (HDFS_BYTES_WRITTEN > 10kb) { MOVE TO pool1 } | | 
> | mapped for default | | + 
> pool1[allocFraction=0.3,schedulingPolicy=default,parallelism=5] | | | trigger 
> slow_pool_trigger: if (ELAPSED_TIME > 3) { MOVE TO pool3 } | | + 
> pool3[allocFraction=0.2,schedulingPolicy=default,parallelism=3] | | + 
> default[allocFraction=0.0,schedulingPolicy=null,parallelism=4] | 
> ++ 8 rows selected (0.25 
> seconds)
> {noformat}
> Workload Manager Events Summary from query run:
> {noformat}
> INFO  : {
>   "queryId" : "hive_20180202213425_9633d7af-4242-4e95-a391-2cd3823e3eac",
>   "queryStartTime" : 1517607265395,
>   "queryEndTime" : 1517607321648,
>   "queryCompleted" : true,
>   "queryWmEvents" : [ {
> "wmTezSessionInfo" : {
>   "sessionId" : "21f8a4ab-511e-4828-a2dd-1d5f2932c492",
>   "poolName" : "pool2",
>   "clusterPercent" : 50.0
> },
> "eventStartTimestamp" : 1517607269660,
> "eventEndTimestamp" : 1517607269661,
> "eventType" : "GET",
> "elapsedTime" : 1
>   }, {
> "wmTezSessionInfo" : {
>   "sessionId" : "21f8a4ab-511e-4828-a2dd-1d5f2932c492",
>   "poolName" : null,
>   "clusterPercent" : 0.0
> },
> "eventStartTimestamp" : 1517607321663,
> "eventEndTimestamp" : 1517607321663,
> "eventType" : "RETURN",
> "elapsedTime" : 0
>   } ],
>   "appliedTriggers" : [ {
> "name" : "too_large_write_triger",
> "expression" : {
>   "counterLimit" : {
> "limit" : 10240,
> "name" : "HDFS_BYTES_WRITTEN"
>   },
>   "predicate" : "GREATER_THAN"
> },
> "action" : {
>   "type" : "MOVE_TO_POOL",
>   "poolName" : "pool1"
> },
> "violationMsg" : null
>   } ],
>   "subscribedCounters" : [ "HDFS_BYTES_WRITTEN" ],
>   "currentCounters" : {
> "HDFS_BYTES_WRITTEN" : 33306829
>   },
>   "elapsedTime" : 56284
> }
> {noformat}
> From the Workload Manager Event Summary it could seen that the 'MOVE' event 
> didn't happen though the limit for counter (10240) HDFS_BYTES_WRITTEN was 
> exceeded



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18638) Triggers for multi-pool move, failing to initiate the move event

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366148#comment-16366148
 ] 

Prasanth Jayachandran commented on HIVE-18638:
--

TestTriggerMoveWorkloadManager test failure will be covered in HIVE-18547. 
Other test failures are unrelated.
Committed to master. 

> Triggers for multi-pool move, failing to initiate the move event
> 
>
> Key: HIVE-18638
> URL: https://issues.apache.org/jira/browse/HIVE-18638
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Aswathy Chellammal Sreekumar
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18638.1.patch
>
>
> Resource plan with multiple pools and trigger set to move job across those 
> pools seems to be failing to do so
> Resource plan:
> {noformat}
> 1: jdbc:hive2://ctr-e137-1514896590304-51538-> show resource plan plan_2; 
> INFO : Compiling 
> command(queryId=hive_20180202220823_2fb8bca7-5b7a-48cf-8ff9-8d5f3548d334): 
> show resource plan plan_2 INFO : Semantic Analysis Completed INFO : Returning 
> Hive schema: Schema(fieldSchemas:[FieldSchema(name:line, type:string, 
> comment:from deserializer)], properties:null) INFO : Completed compiling 
> command(queryId=hive_20180202220823_2fb8bca7-5b7a-48cf-8ff9-8d5f3548d334); 
> Time taken: 0.008 seconds INFO : Executing 
> command(queryId=hive_20180202220823_2fb8bca7-5b7a-48cf-8ff9-8d5f3548d334): 
> show resource plan plan_2 INFO : Starting task [Stage-0:DDL] in serial mode 
> INFO : Completed executing 
> command(queryId=hive_20180202220823_2fb8bca7-5b7a-48cf-8ff9-8d5f3548d334); 
> Time taken: 0.196 seconds INFO : OK 
> ++ | line | 
> ++ | 
> plan_2[status=ACTIVE,parallelism=null,defaultPool=pool2] | | + 
> pool2[allocFraction=0.5,schedulingPolicy=default,parallelism=3] | | | trigger 
> too_large_write_triger: if (HDFS_BYTES_WRITTEN > 10kb) { MOVE TO pool1 } | | 
> | mapped for default | | + 
> pool1[allocFraction=0.3,schedulingPolicy=default,parallelism=5] | | | trigger 
> slow_pool_trigger: if (ELAPSED_TIME > 3) { MOVE TO pool3 } | | + 
> pool3[allocFraction=0.2,schedulingPolicy=default,parallelism=3] | | + 
> default[allocFraction=0.0,schedulingPolicy=null,parallelism=4] | 
> ++ 8 rows selected (0.25 
> seconds)
> {noformat}
> Workload Manager Events Summary from query run:
> {noformat}
> INFO  : {
>   "queryId" : "hive_20180202213425_9633d7af-4242-4e95-a391-2cd3823e3eac",
>   "queryStartTime" : 1517607265395,
>   "queryEndTime" : 1517607321648,
>   "queryCompleted" : true,
>   "queryWmEvents" : [ {
> "wmTezSessionInfo" : {
>   "sessionId" : "21f8a4ab-511e-4828-a2dd-1d5f2932c492",
>   "poolName" : "pool2",
>   "clusterPercent" : 50.0
> },
> "eventStartTimestamp" : 1517607269660,
> "eventEndTimestamp" : 1517607269661,
> "eventType" : "GET",
> "elapsedTime" : 1
>   }, {
> "wmTezSessionInfo" : {
>   "sessionId" : "21f8a4ab-511e-4828-a2dd-1d5f2932c492",
>   "poolName" : null,
>   "clusterPercent" : 0.0
> },
> "eventStartTimestamp" : 1517607321663,
> "eventEndTimestamp" : 1517607321663,
> "eventType" : "RETURN",
> "elapsedTime" : 0
>   } ],
>   "appliedTriggers" : [ {
> "name" : "too_large_write_triger",
> "expression" : {
>   "counterLimit" : {
> "limit" : 10240,
> "name" : "HDFS_BYTES_WRITTEN"
>   },
>   "predicate" : "GREATER_THAN"
> },
> "action" : {
>   "type" : "MOVE_TO_POOL",
>   "poolName" : "pool1"
> },
> "violationMsg" : null
>   } ],
>   "subscribedCounters" : [ "HDFS_BYTES_WRITTEN" ],
>   "currentCounters" : {
> "HDFS_BYTES_WRITTEN" : 33306829
>   },
>   "elapsedTime" : 56284
> }
> {noformat}
> From the Workload Manager Event Summary it could seen that the 'MOVE' event 
> didn't happen though the limit for counter (10240) HDFS_BYTES_WRITTEN was 
> exceeded



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18192) Introduce WriteID per table rather than using global transaction ID

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366141#comment-16366141
 ] 

Hive QA commented on HIVE-18192:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12910731/HIVE-18192.13.patch

{color:green}SUCCESS:{color} +1 due to 28 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 28 failed/errored test(s), 13763 tests 
executed
*Failed tests:*
{noformat}
TestDFSErrorHandling - did not produce a TEST-*.xml file (likely timed out) 
(batchId=235)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
 (batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=82)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_exchangepartition] 
(batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=78)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
 (batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] 
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
 (batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
 (batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=160)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=121)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut 
(batchId=221)
org.apache.hadoop.hive.ql.TestAcidOnTez.testGetSplitsLocks (batchId=224)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=187)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.alterIndex 
(batchId=242)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.createIndex 
(batchId=242)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropIndex 
(batchId=242)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd 
(batchId=235)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveAndKill 
(batchId=235)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveConflictKill
 (batchId=235)
org.apache.hive.service.cli.operation.TestQueryLifeTimeHooksWithSQLOperation.testQueryInfoInHookContext
 (batchId=218)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9235/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9235/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9235/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 28 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12910731 - PreCommit-HIVE-Build

> Introduce WriteID per table rather than using global transaction ID
> ---
>
> Key: HIVE-18192
> URL: https://issues.apache.org/jira/browse/HIVE-18192
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, Transactions
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: ACID, DR, pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18192.01.patch, HIVE-18192.02.patch, 
> HIVE-18192.03.patch, HIVE-18192.04.patch, HIVE-18192.05.patch, 
> HIVE-18192.06.patch, HIVE-18192.07.patch, HIVE-18192.08.patch, 
> HIVE-18192.09.patch, HIVE-18192.10.patch, HIVE-18192.11.patch, 
> HIVE-18192.12.patch, HIVE-18192.13.patch
>
>
> To support ACID replication, we will be introducing a per table write Id 
> which will replace the transaction id in the primary key for each row in a 
> ACID table.
> The current primary key is determined via 
>  
> which will move to 
>  
> For each table modified by the given transaction will have a table level 
> write ID allocated and a persisted map of 

[jira] [Commented] (HIVE-18721) Bucket Map Join : Handle empty buckets

[Gunther Hagleitner (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366126#comment-16366126
 ] 

Gunther Hagleitner commented on HIVE-18721:
---

+1

> Bucket Map Join : Handle empty buckets
> --
>
> Key: HIVE-18721
> URL: https://issues.apache.org/jira/browse/HIVE-18721
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-18721.1.patch
>
>
> Bucket Map Join needs to bluff Tez by sending empty task list for 
> DataMovementEvent for those buckets for which there is no data.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-16924) Support distinct in presence Gby

[Julian Hyde (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366125#comment-16366125
 ] 

Julian Hyde commented on HIVE-16924:


As of the latest patch, many tests on [~ashutoshc]'s list have been fixed; the 
following remain:

{noformat}
TestMiniLlapLocalCliDriver.cbo_rp_unionDistinct_2.q
TestMiniLlapLocalCliDriver.cross_prod_1.q
TestMiniLlapLocalCliDriver.cross_prod_3.q
TestMiniLlapLocalCliDriver.cross_prod_4.q
TestMiniLlapLocalCliDriver.selectDistinctStar.q
TestNegativeCliDriver.selectDistinctStarNeg_2.q
TestNegativeCliDriver.udaf_invalid_place.q
{noformat}

> Support distinct in presence Gby 
> -
>
> Key: HIVE-16924
> URL: https://issues.apache.org/jira/browse/HIVE-16924
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Planning
>Reporter: Carter Shanklin
>Assignee: Julian Hyde
>Priority: Major
> Attachments: HIVE-16924.01.patch, HIVE-16924.02.patch, 
> HIVE-16924.03.patch, HIVE-16924.04.patch, HIVE-16924.05.patch
>
>
> {code:sql}
> create table e011_01 (c1 int, c2 smallint);
> insert into e011_01 values (1, 1), (2, 2);
> {code}
> These queries should work:
> {code:sql}
> select distinct c1, count(*) from e011_01 group by c1;
> select distinct c1, avg(c2) from e011_01 group by c1;
> {code}
> Currently, you get : 
> FAILED: SemanticException 1:52 SELECT DISTINCT and GROUP BY can not be in the 
> same query. Error encountered near token 'c1'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18721) Bucket Map Join : Handle empty buckets

[Deepak Jaiswal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366100#comment-16366100
 ] 

Deepak Jaiswal commented on HIVE-18721:
---

Test failures are unrelated.

> Bucket Map Join : Handle empty buckets
> --
>
> Key: HIVE-18721
> URL: https://issues.apache.org/jira/browse/HIVE-18721
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-18721.1.patch
>
>
> Bucket Map Join needs to bluff Tez by sending empty task list for 
> DataMovementEvent for those buckets for which there is no data.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18192) Introduce WriteID per table rather than using global transaction ID

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366072#comment-16366072
 ] 

Hive QA commented on HIVE-18192:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  3m 
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  3m 
 7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
17s{color} | {color:red} streaming in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
37s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
33s{color} | {color:red} ql in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
19s{color} | {color:red} streaming in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
37s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 19s{color} 
| {color:red} streaming in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 37s{color} 
| {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
11s{color} | {color:red} storage-api: The patch generated 5 new + 19 unchanged 
- 2 fixed = 24 total (was 21) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
16s{color} | {color:red} hcatalog/streaming: The patch generated 9 new + 423 
unchanged - 40 fixed = 432 total (was 463) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
18s{color} | {color:red} itests/hive-unit: The patch generated 39 new + 225 
unchanged - 45 fixed = 264 total (was 270) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m 
31s{color} | {color:red} ql: The patch generated 104 new + 4309 unchanged - 307 
fixed = 4413 total (was 4616) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
30s{color} | {color:red} standalone-metastore: The patch generated 46 new + 
1298 unchanged - 43 fixed = 1344 total (was 1341) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 68 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
1s{color} | {color:red} The patch 3 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
14s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 46s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 974d419 |
| Default Java | 1.8.0_111 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9235/yetus/patch-mvninstall-hcatalog_streaming.txt
 |
| mvninstall | 

[jira] [Updated] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-18622:

Status: Patch Available  (was: In Progress)

> Vectorization: IF Statements, Comparisons, and more do not handle NULLs 
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.03.patch, HIVE-18622.04.patch, 
> HIVE-18622.05.patch, HIVE-18622.06.patch, HIVE-18622.07.patch, 
> HIVE-18622.08.patch, HIVE-18622.09.patch, HIVE-18622.091.patch, 
> HIVE-18622.092.patch, HIVE-18622.093.patch, HIVE-18622.094.patch, 
> HIVE-18622.095.patch, HIVE-18622.096.patch, HIVE-18622.097.patch, 
> HIVE-18622.098.patch, HIVE-18622.099.patch, HIVE-18622.0991.patch
>
>
>  
>  Many vector expression classes are setting noNulls to true which does not 
> work if the VRB is a scratch column being reused. The previous use may have 
> set noNulls to false and the isNull array will have some rows marked as NULL. 
> The result is wrong query results and sometimes NPEs (for BytesColumnVector).
> So, many vector expressions need this:
> {code:java}
>   // Carefully handle NULLs...
>   /*
>* For better performance on LONG/DOUBLE we don't want the conditional
>* statements inside the for loop.
>*/
>   outputColVector.noNulls = false;
>  {code}
> And, vector expressions need to make sure the isNull array entry is set when 
> outputColVector.noNulls is false.
> And, all place that assign column value need to set noNulls to false when the 
> value is NULL.
> Almost all cases where noNulls is set to true are incorrect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-18622:

Attachment: HIVE-18622.0991.patch

> Vectorization: IF Statements, Comparisons, and more do not handle NULLs 
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.03.patch, HIVE-18622.04.patch, 
> HIVE-18622.05.patch, HIVE-18622.06.patch, HIVE-18622.07.patch, 
> HIVE-18622.08.patch, HIVE-18622.09.patch, HIVE-18622.091.patch, 
> HIVE-18622.092.patch, HIVE-18622.093.patch, HIVE-18622.094.patch, 
> HIVE-18622.095.patch, HIVE-18622.096.patch, HIVE-18622.097.patch, 
> HIVE-18622.098.patch, HIVE-18622.099.patch, HIVE-18622.0991.patch
>
>
>  
>  Many vector expression classes are setting noNulls to true which does not 
> work if the VRB is a scratch column being reused. The previous use may have 
> set noNulls to false and the isNull array will have some rows marked as NULL. 
> The result is wrong query results and sometimes NPEs (for BytesColumnVector).
> So, many vector expressions need this:
> {code:java}
>   // Carefully handle NULLs...
>   /*
>* For better performance on LONG/DOUBLE we don't want the conditional
>* statements inside the for loop.
>*/
>   outputColVector.noNulls = false;
>  {code}
> And, vector expressions need to make sure the isNull array entry is set when 
> outputColVector.noNulls is false.
> And, all place that assign column value need to set noNulls to false when the 
> value is NULL.
> Almost all cases where noNulls is set to true are incorrect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-18622:

Status: In Progress  (was: Patch Available)

> Vectorization: IF Statements, Comparisons, and more do not handle NULLs 
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.03.patch, HIVE-18622.04.patch, 
> HIVE-18622.05.patch, HIVE-18622.06.patch, HIVE-18622.07.patch, 
> HIVE-18622.08.patch, HIVE-18622.09.patch, HIVE-18622.091.patch, 
> HIVE-18622.092.patch, HIVE-18622.093.patch, HIVE-18622.094.patch, 
> HIVE-18622.095.patch, HIVE-18622.096.patch, HIVE-18622.097.patch, 
> HIVE-18622.098.patch, HIVE-18622.099.patch
>
>
>  
>  Many vector expression classes are setting noNulls to true which does not 
> work if the VRB is a scratch column being reused. The previous use may have 
> set noNulls to false and the isNull array will have some rows marked as NULL. 
> The result is wrong query results and sometimes NPEs (for BytesColumnVector).
> So, many vector expressions need this:
> {code:java}
>   // Carefully handle NULLs...
>   /*
>* For better performance on LONG/DOUBLE we don't want the conditional
>* statements inside the for loop.
>*/
>   outputColVector.noNulls = false;
>  {code}
> And, vector expressions need to make sure the isNull array entry is set when 
> outputColVector.noNulls is false.
> And, all place that assign column value need to set noNulls to false when the 
> value is NULL.
> Almost all cases where noNulls is set to true are incorrect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366031#comment-16366031
 ] 

Hive QA commented on HIVE-18622:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12910715/HIVE-18622.099.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9234/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9234/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9234/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-02-15 17:58:06.257
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-9234/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-02-15 17:58:06.260
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 974d419 HIVE-18421 : Vectorized execution handles overflows in a 
different manner than non-vectorized execution (Vihang Karajgaonkar, reviewed 
by Sahil Takiar)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 974d419 HIVE-18421 : Vectorized execution handles overflows in a 
different manner than non-vectorized execution (Vihang Karajgaonkar, reviewed 
by Sahil Takiar)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-02-15 17:58:09.679
+ rm -rf ../yetus
+ mkdir ../yetus
+ git gc
+ cp -R . ../yetus
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-9234/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: patch failed: 
ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt:122
Falling back to three-way merge...
Applied patch to 
'ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt' with 
conflicts.
error: patch failed: 
ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticScalar.txt:15
Falling back to three-way merge...
Applied patch to 
'ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticScalar.txt' with 
conflicts.
error: patch failed: 
ql/src/gen/vectorization/ExpressionTemplates/ColumnUnaryMinus.txt:15
Falling back to three-way merge...
Applied patch to 
'ql/src/gen/vectorization/ExpressionTemplates/ColumnUnaryMinus.txt' with 
conflicts.
error: patch failed: 
ql/src/gen/vectorization/ExpressionTemplates/ScalarArithmeticColumn.txt:74
Falling back to three-way merge...
Applied patch to 
'ql/src/gen/vectorization/ExpressionTemplates/ScalarArithmeticColumn.txt' 
cleanly.
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:16494: trailing whitespace.
 */ 
/data/hiveptest/working/scratch/build.patch:16836: trailing whitespace.
 */ 
/data/hiveptest/working/scratch/build.patch:22330: trailing whitespace.
  } 
/data/hiveptest/working/scratch/build.patch:22338: trailing whitespace.

/data/hiveptest/working/scratch/build.patch:22402: trailing whitespace.

error: patch failed: 
ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt:122
Falling back to three-way merge...
Applied patch to 
'ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt' with 
conflicts.
error: patch failed: 
ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticScalar.txt:15
Falling back to three-way merge...
Applied patch to 
'ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticScalar.txt' with 
conflicts.
error: patch failed: 
ql/src/gen/vectorization/ExpressionTemplates/ColumnUnaryMinus.txt:15
Falling back to three-way 

[jira] [Commented] (HIVE-18721) Bucket Map Join : Handle empty buckets

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366023#comment-16366023
 ] 

Hive QA commented on HIVE-18721:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12910710/HIVE-18721.1.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 13783 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
 (batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=78)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
 (batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] 
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
 (batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
 (batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=160)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=121)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query39] 
(batchId=250)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut 
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestTablesGetExists.testGetAllTablesCaseInsensitive[Embedded]
 (batchId=205)
org.apache.hadoop.hive.ql.TestAcidOnTez.testGetSplitsLocks (batchId=224)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=187)
org.apache.hive.hcatalog.cli.TestPermsGrp.testCustomPerms (batchId=194)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.alterIndex 
(batchId=242)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.createIndex 
(batchId=242)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropIndex 
(batchId=242)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd 
(batchId=235)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveConflictKill
 (batchId=235)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9233/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9233/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9233/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12910710 - PreCommit-HIVE-Build

> Bucket Map Join : Handle empty buckets
> --
>
> Key: HIVE-18721
> URL: https://issues.apache.org/jira/browse/HIVE-18721
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-18721.1.patch
>
>
> Bucket Map Join needs to bluff Tez by sending empty task list for 
> DataMovementEvent for those buckets for which there is no data.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-18715) Remove index support from metastore

[Adam Szita (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365972#comment-16365972
 ] 

Adam Szita edited comment on HIVE-18715 at 2/15/18 5:22 PM:


I agree that dropping indices from Hive is worth it since it had no popularity 
among users. But since we're separating HMS into another top level component 
anyway, do we (fore)see any future (or even present) use case for indices with 
other projects? (Just thinking out loud)


was (Author: szita):
I agree that dropping indices from Hive is worth it since it had no popularity 
among users. But since we're separating HMS into another top level component 
anyway, do we (for)see any future (or even present) use case for indices with 
other projects? (Just thinking out loud)

> Remove index support from metastore
> ---
>
> Key: HIVE-18715
> URL: https://issues.apache.org/jira/browse/HIVE-18715
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore, Standalone Metastore
>Reporter: Zoltan Haindrich
>Priority: Major
>
> Hive will not use this feature anymore; so if there are no other uses of it 
> we might remove it from the metastore as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18715) Remove index support from metastore

[Adam Szita (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365972#comment-16365972
 ] 

Adam Szita commented on HIVE-18715:
---

I agree that dropping indices from Hive is worth it since it had no popularity 
among users. But since we're separating HMS into another top level component 
anyway, do we (for)see any future (or even present) use case for indices with 
other projects? (Just thinking out loud)

> Remove index support from metastore
> ---
>
> Key: HIVE-18715
> URL: https://issues.apache.org/jira/browse/HIVE-18715
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore, Standalone Metastore
>Reporter: Zoltan Haindrich
>Priority: Major
>
> Hive will not use this feature anymore; so if there are no other uses of it 
> we might remove it from the metastore as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18705) Improve HiveMetaStoreClient.dropDatabase

[Adam Szita (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365962#comment-16365962
 ] 

Adam Szita commented on HIVE-18705:
---

I guess it is because of client side hooks. I intend to rearrange the order 
things happen with the hooks so that all dropTable hooks run first and then we 
only send one dropDB to HMS server side. But before I upload any patches I also 
want to run some performance tests first to see how much speed-up we'd win.

> Improve HiveMetaStoreClient.dropDatabase
> 
>
> Key: HIVE-18705
> URL: https://issues.apache.org/jira/browse/HIVE-18705
> Project: Hive
>  Issue Type: Improvement
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
>
> {{HiveMetaStoreClient.dropDatabase}} has a strange implementation to ensure 
> dealing with client side hooks (for non-native tables e.g. HBase). Currently 
> it starts by retrieving all the tables from HMS, and then sends {{dropTable}} 
> calls to HMS table-by-table. At the end a {{dropDatabase}} just to be sure :) 
> I believe this could be refactored so that it speeds up the dropDB in 
> situations where the average table count per DB is very high.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18721) Bucket Map Join : Handle empty buckets

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365934#comment-16365934
 ] 

Hive QA commented on HIVE-18721:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
15s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 37s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 974d419 |
| Default Java | 1.8.0_111 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9233/yetus/patch-asflicense-problems.txt
 |
| modules | C: itests ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9233/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Bucket Map Join : Handle empty buckets
> --
>
> Key: HIVE-18721
> URL: https://issues.apache.org/jira/browse/HIVE-18721
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-18721.1.patch
>
>
> Bucket Map Join needs to bluff Tez by sending empty task list for 
> DataMovementEvent for those buckets for which there is no data.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18693) TestTxnCommands2#writeBetweenWorkerAndCleaner fails on MM table

[Steve Yeom (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365914#comment-16365914
 ] 

Steve Yeom commented on HIVE-18693:
---

Because of the requirement of ACID, especially Snapshot Isolation, for MM 
table, 
we need to keep aborted transaction in the Metastore metadata so as for a 
reader 
to get a correct ValidTxnList. The current code deletes an aborted transaction 
in 
the Metastore as indicated from the above test case. 

A solution without too many rows in the Metastore can be possible either in the 
Metastore
or in the directory level.   

> TestTxnCommands2#writeBetweenWorkerAndCleaner fails on MM table
> ---
>
> Key: HIVE-18693
> URL: https://issues.apache.org/jira/browse/HIVE-18693
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-18693.01.patch
>
>
> TestTxnCommands2#writeBetweenWorkerAndCleaner with minor 
> changes (changing delete command to insert command) fails on MM table.
> Specifically the last SELECT commands returns wrong results. 
> But this test works fine with full ACID table. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18679) create/replicate open transaction event

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365868#comment-16365868
 ] 

Hive QA commented on HIVE-18679:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12910706/HIVE-18679.01.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 31 failed/errored test(s), 13104 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
 (batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] 
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=78)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
 (batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] 
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
 (batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
 (batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
 (batchId=179)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=121)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut 
(batchId=221)
org.apache.hadoop.hive.ql.TestAcidOnTez.testGetSplitsLocks (batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testBootStrapDumpOfWarehouse
 (batchId=228)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testCreateFunctionIncrementalReplication
 (batchId=228)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testDropFunctionIncrementalReplication
 (batchId=228)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testIncrementalDumpOfWarehouse
 (batchId=228)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testIncrementalMetadataReplication
 (batchId=228)
org.apache.hive.beeline.TestBeeLineWithArgs.testEscapeCRLFInTSV2Output 
(batchId=231)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=187)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.alterIndex 
(batchId=242)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.createIndex 
(batchId=242)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropDatabase 
(batchId=242)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropIndex 
(batchId=242)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd 
(batchId=235)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9232/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9232/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9232/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 31 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12910706 - PreCommit-HIVE-Build

> create/replicate open transaction event
> ---
>
> Key: HIVE-18679
> URL: https://issues.apache.org/jira/browse/HIVE-18679
> Project: Hive
>  Issue Type: Bug
>  Components: repl, Transactions
>Affects Versions: 3.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18679.01.patch
>
>
> *EVENT_OPEN_TXN:*
> *Source Warehouse:*
>  - Create new event type EVENT_OPEN_TXN with related message format etc.
>  - When any transaction is opened either by auto-commit mode or 
> multi-statement mode, need to capture this event.
>  - Repl dump should read this event from