[jira] [Commented] (HIVE-19508) SparkJobMonitor getReport doesn't print stage progress in order
[
https://issues.apache.org/jira/browse/HIVE-19508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502840#comment-16502840
]
Hive QA commented on HIVE-19508:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
23s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
32s{color} | {color:red} ql: The patch generated 1 new + 26 unchanged - 3 fixed
= 27 total (was 29) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11545/dev-support/hive-personality.sh
|
| git revision | master / 0992d82 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11545/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11545/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11545/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> SparkJobMonitor getReport doesn't print stage progress in order
> ---
>
> Key: HIVE-19508
> URL: https://issues.apache.org/jira/browse/HIVE-19508
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19508.1.patch, HIVE-19508.2.patch,
> HIVE-19508.3.patch
>
>
> You can end up with a progress output like this:
> {code}
> Stage-10_0: 0/29 Stage-11_0: 0/44Stage-12_0: 0/11
> Stage-13_0: 0/1 Stage-8_0: 258(+76)/468 Stage-9_0: 0/165
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19708) Repl copy retrying with cm path even if the failure is due to network issue
[
https://issues.apache.org/jira/browse/HIVE-19708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502834#comment-16502834
]
Hive QA commented on HIVE-19708:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926542/HIVE-19708.06-branch-3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11544/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11544/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11544/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-06-06 05:26:16.499
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-11544/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z branch-3 ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-06-06 05:26:16.501
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 0992d82 HIVE-19773 : CBO exception while running queries with
tables that are not present in materialized views (Jesus Camacho Rodriguez via
Ashutosh Chauhan)
+ git clean -f -d
+ git checkout branch-3
Switched to branch 'branch-3'
Your branch is up-to-date with 'origin/branch-3'.
+ git reset --hard origin/branch-3
HEAD is now at 8d9f749 HIVE-19772: Streaming ingest V2 API can generate invalid
orc file if interrupted (Prasanth Jayachandran reviewed by Gopal V)
+ git merge --ff-only origin/branch-3
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-06-06 05:26:17.544
+ rm -rf ../yetus_PreCommit-HIVE-Build-11544
+ mkdir ../yetus_PreCommit-HIVE-Build-11544
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-11544
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-11544/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/common/src/java/org/apache/hadoop/hive/common/FileUtils.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java: does
not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/io/FileOperations.java:
does not exist in index
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/io/FileOperations.java:20
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/io/FileOperations.java'
with conflicts.
Going to apply patch with: git apply -p1
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/io/FileOperations.java:20
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/io/FileOperations.java'
with conflicts.
U ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/io/FileOperations.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-11544
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926542 - PreCommit-HIVE-Build
> Repl copy retrying with cm path even if the failure is due to network issue
> ---
>
> Key: HIVE-19708
> URL: https://issues.apache.org/jira/browse/HIVE-19708
> Project: Hive
> Issue Type: Task
> Components: Hive, HiveServer2, repl
>Affects Versions: 3.1.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-19708.01.patch, HIVE-19708.02.patch,
> HIVE-19708.04.patch, HIVE-19708.05.patch, HIVE-19708.06-branch-3.patch,
> HIVE-19708.06.patch
>
>
> * During repl load
> **
[jira] [Commented] (HIVE-19267) Create/Replicate ACID Write event
[
https://issues.apache.org/jira/browse/HIVE-19267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502833#comment-16502833
]
Hive QA commented on HIVE-19267:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926275/HIVE-19267.10.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11543/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11543/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11543/
Messages:
{noformat}
This message was trimmed, see log for full details
HEAD is now at 0992d82 HIVE-19773 : CBO exception while running queries with
tables that are not present in materialized views (Jesus Camacho Rodriguez via
Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-06-06 05:25:10.012
+ rm -rf ../yetus_PreCommit-HIVE-Build-11543
+ mkdir ../yetus_PreCommit-HIVE-Build-11543
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-11543
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-11543/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: does not
exist in index
error:
a/hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java:
does not exist in index
error:
a/itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java:
does not exist in index
error:
a/itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java:
does not exist in index
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenarios.java:
does not exist in index
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java:
does not exist in index
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcrossInstances.java:
does not exist in index
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/WarehouseInstance.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/metastore/SynchronizedMetaStoreClient.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ReplTxnTask.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java: does
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java: does
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManager.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveUtils.java: does
not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java: does
not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSemanticAnalyzer.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java: does
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/Utils.java: does
not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/CommitTxnHandler.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/EventHandler.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/InsertHandler.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/UpdatedMetaDataTracker.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/AbortTxnHandler.java:
does not
[jira] [Commented] (HIVE-18875) Enable SMB Join by default in Tez
[
https://issues.apache.org/jira/browse/HIVE-18875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502831#comment-16502831
]
Hive QA commented on HIVE-18875:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926310/HIVE-18875.13.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14467 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11542/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11542/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11542/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926310 - PreCommit-HIVE-Build
> Enable SMB Join by default in Tez
> -
>
> Key: HIVE-18875
> URL: https://issues.apache.org/jira/browse/HIVE-18875
> Project: Hive
> Issue Type: Task
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-18875.1.patch, HIVE-18875.10.patch,
> HIVE-18875.11.patch, HIVE-18875.12.patch, HIVE-18875.13.patch,
> HIVE-18875.2.patch, HIVE-18875.3.patch, HIVE-18875.4.patch,
> HIVE-18875.5.patch, HIVE-18875.6.patch, HIVE-18875.7.patch,
> HIVE-18875.8.patch, HIVE-18875.9.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18875) Enable SMB Join by default in Tez
[
https://issues.apache.org/jira/browse/HIVE-18875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502822#comment-16502822
]
Hive QA commented on HIVE-18875:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
29s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
34s{color} | {color:red} ql: The patch generated 1 new + 75 unchanged - 1 fixed
= 76 total (was 76) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11542/dev-support/hive-personality.sh
|
| git revision | master / 0992d82 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11542/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11542/yetus/patch-asflicense-problems.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11542/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Enable SMB Join by default in Tez
> -
>
> Key: HIVE-18875
> URL: https://issues.apache.org/jira/browse/HIVE-18875
> Project: Hive
> Issue Type: Task
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-18875.1.patch, HIVE-18875.10.patch,
> HIVE-18875.11.patch, HIVE-18875.12.patch, HIVE-18875.13.patch,
> HIVE-18875.2.patch, HIVE-18875.3.patch, HIVE-18875.4.patch,
> HIVE-18875.5.patch, HIVE-18875.6.patch, HIVE-18875.7.patch,
> HIVE-18875.8.patch, HIVE-18875.9.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19776) HiveServer2.startHiveServer2 retries of start has concurrency issues
[
https://issues.apache.org/jira/browse/HIVE-19776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502806#comment-16502806
]
Hive QA commented on HIVE-19776:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926263/HIVE-19776.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14467 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testAsyncSessionInitFailures
(batchId=296)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11540/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11540/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11540/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926263 - PreCommit-HIVE-Build
> HiveServer2.startHiveServer2 retries of start has concurrency issues
>
>
> Key: HIVE-19776
> URL: https://issues.apache.org/jira/browse/HIVE-19776
> Project: Hive
> Issue Type: Improvement
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-19776.1.patch, HIVE-19776.2.patch,
> HIVE-19776.3.patch
>
>
> HS2 starts the thrift binary/http servers in background, while it proceeds to
> do other setup (eg create zookeeper entries). If there is a ZK error and it
> attempts to stop and start in the retry loop within
> HiveServer2.startHiveServer2, the retry fails because the thrift server
> doesn't get stopped if it was still getting initialized.
> The thrift server initialization and stopping needs to be synchronized.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19812) Disable external table replication by default via a configuration property
[
https://issues.apache.org/jira/browse/HIVE-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera updated HIVE-19812:
---
Summary: Disable external table replication by default via a configuration
property (was: External table replication is disabled by default via a
configuration property)
> Disable external table replication by default via a configuration property
> --
>
> Key: HIVE-19812
> URL: https://issues.apache.org/jira/browse/HIVE-19812
> Project: Hive
> Issue Type: Task
> Components: repl
>Affects Versions: 3.1.0, 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Fix For: 3.1.0, 4.0.0
>
>
> use a hive config property to allow external table replication. set this
> property by default to prevent external table replication.
> for metadata only hive repl always export metadata for external tables.
>
> REPL_DUMP_EXTERNAL_TABLES("hive.repl.dump.include.external.tables", false,
> "Indicates if repl dump should include information about external tables. It
> should be \n"
> + "used in conjunction with 'hive.repl.dump.metadata.only' set to false. if
> 'hive.repl.dump.metadata.only' \n"
> + " is set to true then this config parameter has no effect as external table
> meta data is flushed \n"
> + " always by default.")
> This should be done for only replication dump and not for export
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19812) External table replication is disabled by default via a configuration property
[
https://issues.apache.org/jira/browse/HIVE-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera reassigned HIVE-19812:
--
> External table replication is disabled by default via a configuration property
> --
>
> Key: HIVE-19812
> URL: https://issues.apache.org/jira/browse/HIVE-19812
> Project: Hive
> Issue Type: Task
> Components: repl
>Affects Versions: 3.1.0, 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Fix For: 3.1.0, 4.0.0
>
>
> use a hive config property to allow external table replication. set this
> property by default to prevent external table replication.
> for metadata only hive repl always export metadata for external tables.
>
> REPL_DUMP_EXTERNAL_TABLES("hive.repl.dump.include.external.tables", false,
> "Indicates if repl dump should include information about external tables. It
> should be \n"
> + "used in conjunction with 'hive.repl.dump.metadata.only' set to false. if
> 'hive.repl.dump.metadata.only' \n"
> + " is set to true then this config parameter has no effect as external table
> meta data is flushed \n"
> + " always by default.")
> This should be done for only replication dump and not for export
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19776) HiveServer2.startHiveServer2 retries of start has concurrency issues
[
https://issues.apache.org/jira/browse/HIVE-19776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502786#comment-16502786
]
Hive QA commented on HIVE-19776:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} service in master has 49 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} service: The patch generated 4 new + 30 unchanged - 6
fixed = 34 total (was 36) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
40s{color} | {color:red} service generated 1 new + 47 unchanged - 2 fixed = 48
total (was 49) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 21s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:service |
| | Exception is caught when Exception is not thrown in
org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.initServer() At
ThriftBinaryCLIService.java:is not thrown in
org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.initServer() At
ThriftBinaryCLIService.java:[line 161] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11540/dev-support/hive-personality.sh
|
| git revision | master / 0992d82 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11540/yetus/diff-checkstyle-service.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11540/yetus/whitespace-eol.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11540/yetus/new-findbugs-service.html
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11540/yetus/patch-asflicense-problems.txt
|
| modules | C: service U: service |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11540/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HiveServer2.startHiveServer2 retries of start has concurrency issues
>
>
> Key: HIVE-19776
> URL: https://issues.apache.org/jira/browse/HIVE-19776
> Project: Hive
> Issue Type: Improvement
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-19776.1.patch, HIVE-19776.2.patch,
> HIVE-19776.3.patch
>
>
> HS2 starts the thrift binary/http servers in background, while it proceeds to
> do other setup (eg create zookeeper entries). If there is a ZK error and it
> attempts to stop and start in the retry loop within
[jira] [Commented] (HIVE-19750) Initialize NEXT_WRITE_ID. NWI_NEXT on converting an existing table to full acid
[
https://issues.apache.org/jira/browse/HIVE-19750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502774#comment-16502774
]
Hive QA commented on HIVE-19750:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926304/HIVE-19750.03.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14467 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11539/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11539/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11539/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926304 - PreCommit-HIVE-Build
> Initialize NEXT_WRITE_ID. NWI_NEXT on converting an existing table to full
> acid
> ---
>
> Key: HIVE-19750
> URL: https://issues.apache.org/jira/browse/HIVE-19750
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Fix For: 3.1.0
>
> Attachments: HIVE-19750.01.patch, HIVE-19750.02.patch,
> HIVE-19750.03.patch
>
>
> Need to set this to a reasonably high value the the table.
> This will reserve a range of write IDs that will be treated by the system as
> committed.
> This is needed so that we can assign unique ROW__IDs to each row in files
> that already exist in the table. For example, if the value is initialized to
> the number of files currently in the table, we can think of each file as
> written by a separate transaction and thus a free to assign bucketProperty
> (BucketCodec) of ROW_ID in whichever way is convenient.
> it's guaranteed that all rows get unique ROW_IDs this way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19750) Initialize NEXT_WRITE_ID. NWI_NEXT on converting an existing table to full acid
[
https://issues.apache.org/jira/browse/HIVE-19750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502757#comment-16502757
]
Hive QA commented on HIVE-19750:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
23s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
36s{color} | {color:blue} standalone-metastore in master has 214 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 2 new + 1016 unchanged - 3
fixed = 1018 total (was 1019) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
19s{color} | {color:red} standalone-metastore: The patch generated 3 new + 566
unchanged - 0 fixed = 569 total (was 566) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
39s{color} | {color:red} standalone-metastore generated 1 new + 214 unchanged -
0 fixed = 215 total (was 214) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore |
| |
org.apache.hadoop.hive.metastore.txn.TxnHandler.seedWriteIdOnAcidConversion(InitializeTableWriteIdsRequest)
passes a nonconstant String to an execute or addBatch method on an SQL
statement At TxnHandler.java:to an execute or addBatch method on an SQL
statement At TxnHandler.java:[line 1516] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11539/dev-support/hive-personality.sh
|
| git revision | master / 0992d82 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11539/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11539/yetus/diff-checkstyle-standalone-metastore.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11539/yetus/new-findbugs-standalone-metastore.html
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11539/yetus/patch-asflicense-problems.txt
|
| modules | C: ql standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11539/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Initialize NEXT_WRITE_ID. NWI_NEXT on converting an existing table to full
> acid
>
[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)
[
https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502756#comment-16502756
]
Saisai Shao commented on HIVE-16391:
{quote}The problem with that is that it changes the meaning of Hive's
artifacts, so anybody currently importing hive-exec would see a breakage, and
that's probably not desired.
{quote}
This might not be acceptable from Hive community, because it will break the
current user as you mentioned.
As [~joshrosen] mentioned, Spark wants the hive-exec jar which shades kryo and
prototuf-java, not a pure non-shaded jar.
{quote}Another option is to change the artifact name of the current "hive-exec"
pom. Then you'd publish the normal jar under the new artifact name, then have a
separate module that imports that jar, shades dependencies, and publishes the
result as "hive-exec". That would maintain compatibility with existing
artifacts.
{quote}
I can try this approach, but it seems not a small change for Hive, I'm not sure
if Hive community will accept such approach (at least for branch 1.2).
> Publish proper Hive 1.2 jars (without including all dependencies in uber jar)
> -
>
> Key: HIVE-16391
> URL: https://issues.apache.org/jira/browse/HIVE-16391
> Project: Hive
> Issue Type: Task
> Components: Build Infrastructure
>Affects Versions: 1.2.2
>Reporter: Reynold Xin
>Assignee: Saisai Shao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.2.3
>
> Attachments: HIVE-16391.patch
>
>
> Apache Spark currently depends on a forked version of Apache Hive. AFAIK, the
> only change in the fork is to work around the issue that Hive publishes only
> two sets of jars: one set with no dependency declared, and another with all
> the dependencies included in the published uber jar. That is to say, Hive
> doesn't publish a set of jars with the proper dependencies declared.
> There is general consensus on both sides that we should remove the forked
> Hive.
> The change in the forked version is recorded here
> https://github.com/JoshRosen/hive/tree/release-1.2.1-spark2
> Note that the fork in the past included other fixes but those have all become
> unnecessary.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19525) Spark task logs print PLAN PATH excessive number of times
[
https://issues.apache.org/jira/browse/HIVE-19525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502742#comment-16502742
]
Hive QA commented on HIVE-19525:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926235/HIVE-19525.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14467 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11538/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11538/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11538/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926235 - PreCommit-HIVE-Build
> Spark task logs print PLAN PATH excessive number of times
> -
>
> Key: HIVE-19525
> URL: https://issues.apache.org/jira/browse/HIVE-19525
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19525.1.patch, HIVE-19525.2.patch
>
>
> A ton of logs with this {{Utilities - PLAN PATH =
> hdfs://localhost:59527/.../apache-hive/itests/qtest-spark/target/tmp/scratchdir/stakiar/6ebceb49-7a76-4159-9082-5bba44391e30/hive_2018-05-14_07-28-44_672_8205774950452575544-1/-mr-10006/bf14c0b5-a014-4ee8-8ddf-fdb7453eb0f0/map.xml}}
> Seems it print multiple times per task exception, not sure where it is coming
> from, but its too verbose. It should be changed to DEBUG level. Furthermore,
> given that we are using {{Utilities#getBaseWork}} anytime we need to access a
> {{MapWork}} or {{ReduceWork}} object, we should make the method slightly more
> efficient. Right now it borrows a {{Kryo}} from a pool and does a bunch of
> stuff to set the classloader, then it checks the cache to see if the work
> object has already been created. It should check the cache before doing any
> of that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19525) Spark task logs print PLAN PATH excessive number of times
[
https://issues.apache.org/jira/browse/HIVE-19525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502719#comment-16502719
]
Hive QA commented on HIVE-19525:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
26s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} ql: The patch generated 0 new + 119 unchanged - 2
fixed = 119 total (was 121) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11538/dev-support/hive-personality.sh
|
| git revision | master / 0992d82 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11538/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11538/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Spark task logs print PLAN PATH excessive number of times
> -
>
> Key: HIVE-19525
> URL: https://issues.apache.org/jira/browse/HIVE-19525
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19525.1.patch, HIVE-19525.2.patch
>
>
> A ton of logs with this {{Utilities - PLAN PATH =
> hdfs://localhost:59527/.../apache-hive/itests/qtest-spark/target/tmp/scratchdir/stakiar/6ebceb49-7a76-4159-9082-5bba44391e30/hive_2018-05-14_07-28-44_672_8205774950452575544-1/-mr-10006/bf14c0b5-a014-4ee8-8ddf-fdb7453eb0f0/map.xml}}
> Seems it print multiple times per task exception, not sure where it is coming
> from, but its too verbose. It should be changed to DEBUG level. Furthermore,
> given that we are using {{Utilities#getBaseWork}} anytime we need to access a
> {{MapWork}} or {{ReduceWork}} object, we should make the method slightly more
> efficient. Right now it borrows a {{Kryo}} from a pool and does a bunch of
> stuff to set the classloader, then it checks the cache to see if the work
> object has already been created. It should check the cache before doing any
> of that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502707#comment-16502707
]
Hive QA commented on HIVE-19646:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926308/HIVE-19646.07-branch-3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 14403 tests
executed
*Failed tests:*
{noformat}
TestOperationLoggingAPIWithMr - did not produce a TEST-*.xml file (likely timed
out) (batchId=238)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidkafkamini_basic]
(batchId=253)
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection (batchId=305)
org.apache.hive.spark.client.rpc.TestRpc.testServerPort (batchId=305)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11537/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11537/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11537/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926308 - PreCommit-HIVE-Build
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HIVE-19646.01-branch-3.patch, HIVE-19646.01.patch,
> HIVE-19646.02.patch, HIVE-19646.03-branch-3.patch, HIVE-19646.03.patch,
> HIVE-19646.04.patch, HIVE-19646.05-branch-3.patch, HIVE-19646.05.patch,
> HIVE-19646.06-branch-3.patch, HIVE-19646.07-branch-3.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18118) Explain Extended should indicate if a file being read is an EC file
[ https://issues.apache.org/jira/browse/HIVE-18118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18118: -- Attachment: HIVE-18118.1.patch Status: Patch Available (was: Open) > Explain Extended should indicate if a file being read is an EC file > --- > > Key: HIVE-18118 > URL: https://issues.apache.org/jira/browse/HIVE-18118 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18118.1.patch > > > We already print out the files Hive will read in the explain extended > command, we just have to modify it to say whether or not its an EC file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19801) JDBC: Add some missing classes to jdbc standalone jar and remove hbase classes
[ https://issues.apache.org/jira/browse/HIVE-19801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-19801: Attachment: (was: HIVE-19801.1.patch) > JDBC: Add some missing classes to jdbc standalone jar and remove hbase classes > -- > > Key: HIVE-19801 > URL: https://issues.apache.org/jira/browse/HIVE-19801 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 3.0.0, 3.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-19801.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19801) JDBC: Add some missing classes to jdbc standalone jar and remove hbase classes
[ https://issues.apache.org/jira/browse/HIVE-19801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-19801: Attachment: HIVE-19801.1.patch > JDBC: Add some missing classes to jdbc standalone jar and remove hbase classes > -- > > Key: HIVE-19801 > URL: https://issues.apache.org/jira/browse/HIVE-19801 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 3.0.0, 3.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-19801.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19801) JDBC: Add some missing classes to jdbc standalone jar and remove hbase classes
[ https://issues.apache.org/jira/browse/HIVE-19801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-19801: Status: Patch Available (was: Open) > JDBC: Add some missing classes to jdbc standalone jar and remove hbase classes > -- > > Key: HIVE-19801 > URL: https://issues.apache.org/jira/browse/HIVE-19801 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 3.0.0, 3.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-19801.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19801) JDBC: Add some missing classes to jdbc standalone jar and remove hbase classes
[ https://issues.apache.org/jira/browse/HIVE-19801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-19801: Attachment: HIVE-19801.1.patch > JDBC: Add some missing classes to jdbc standalone jar and remove hbase classes > -- > > Key: HIVE-19801 > URL: https://issues.apache.org/jira/browse/HIVE-19801 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 3.0.0, 3.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-19801.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19773) CBO exception while running queries with tables that are not present in materialized views
[ https://issues.apache.org/jira/browse/HIVE-19773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-19773: Resolution: Fixed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) Pushed to master and branch-3. > CBO exception while running queries with tables that are not present in > materialized views > -- > > Key: HIVE-19773 > URL: https://issues.apache.org/jira/browse/HIVE-19773 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 3.1.0, 4.0.0 >Reporter: Aswathy Chellammal Sreekumar >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Fix For: 3.1.0 > > Attachments: HIVE-19773.patch > > > When we obtain the valid list of write ids, some tables in the materialized > views may not be present in the list because they are not present in the > query, which leads to exceptions (hidden in logs) when we try to load the > materialized views in the planner, as we need to verify whether they are > outdated or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19808) GenericUDTFGetSplits should support ACID reads in the temp. table read path
[
https://issues.apache.org/jira/browse/HIVE-19808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502658#comment-16502658
]
Jason Dere commented on HIVE-19808:
---
+1
> GenericUDTFGetSplits should support ACID reads in the temp. table read path
> ---
>
> Key: HIVE-19808
> URL: https://issues.apache.org/jira/browse/HIVE-19808
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19808.1.patch, HIVE-19808.2.patch
>
>
> 1. Map-only reads work on ACID tables.
> 2. Temp. table reads (for multi-vertex queries) work on non-ACID tables.
> 3. But temp. table reads don't work on ACID tables.
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
> temp table: java.lang.IllegalStateException: calling recordValidTxn() more
> than once in the same txnid:420
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:303)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:202)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:918)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:492)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:484)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502645#comment-16502645
]
Hive QA commented on HIVE-19646:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red}
/data/hiveptest/logs/PreCommit-HIVE-Build-11537/patches/PreCommit-HIVE-Build-11537.patch
does not apply to master. Rebase required? Wrong Branch? See
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11537/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HIVE-19646.01-branch-3.patch, HIVE-19646.01.patch,
> HIVE-19646.02.patch, HIVE-19646.03-branch-3.patch, HIVE-19646.03.patch,
> HIVE-19646.04.patch, HIVE-19646.05-branch-3.patch, HIVE-19646.05.patch,
> HIVE-19646.06-branch-3.patch, HIVE-19646.07-branch-3.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19668) Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's and duplicate strings
[
https://issues.apache.org/jira/browse/HIVE-19668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502638#comment-16502638
]
Hive QA commented on HIVE-19668:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926217/HIVE-19668.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 14466 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_view_delete]
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_multiinsert]
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_unqual_corr_expr]
(batchId=8)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multiinsert]
(batchId=145)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11536/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11536/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11536/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926217 - PreCommit-HIVE-Build
> Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's and
> duplicate strings
> --
>
> Key: HIVE-19668
> URL: https://issues.apache.org/jira/browse/HIVE-19668
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HIVE-19668.01.patch, image-2018-05-22-17-41-39-572.png
>
>
> I've recently analyzed a HS2 heap dump, obtained when there was a huge memory
> spike during compilation of some big query. The analysis was done with jxray
> ([www.jxray.com).|http://www.jxray.com)./] It turns out that more than 90% of
> the 20G heap was used by data structures associated with query parsing
> ({{org.apache.hadoop.hive.ql.parse.QBExpr}}). There are probably multiple
> opportunities for optimizations here. One of them is to stop the code from
> creating duplicate instances of {{org.antlr.runtime.CommonToken}} class. See
> a sample of these objects in the attached image:
> !image-2018-05-22-17-41-39-572.png|width=879,height=399!
> Looks like these particular {{CommonToken}} objects are constants, that don't
> change once created. I see some code, e.g. in
> {{org.apache.hadoop.hive.ql.parse.CalcitePlanner}}, where such objects are
> apparently repeatedly created with e.g. {{new
> CommonToken(HiveParser.TOK_INSERT, "TOK_INSERT")}} If these 33 token kinds
> are instead created once and reused, we will save more than 1/10th of the
> heap in this scenario. Plus, since these objects are small but very numerous,
> getting rid of them will remove a gread deal of pressure from the GC.
> Another source of waste are duplicate strings, that collectively waste 26.1%
> of memory. Some of them come from CommonToken objects that have the same text
> (i.e. for multiple CommonToken objects the contents of their 'text' Strings
> are the same, but each has its own copy of that String). Other duplicate
> strings come from other sources, that are easy enough to fix by adding
> String.intern() calls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19685) OpenTracing support for HMS
[ https://issues.apache.org/jira/browse/HIVE-19685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502634#comment-16502634 ] Alexander Kolbasov commented on HIVE-19685: --- [~tlipcon] can you add reviewboard link for your changes? > OpenTracing support for HMS > --- > > Key: HIVE-19685 > URL: https://issues.apache.org/jira/browse/HIVE-19685 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Attachments: HIVE-19685.02.patch, hive-19685.patch, hive-19685.patch, > trace.png > > > When diagnosing performance of metastore operations it isn't always obvious > why something took a long time. Using a tracing framework can provide an > end-to-end view of an operation including time spent in dependent systems (eg > filesystem operations, RDBMS queries, etc). This JIRA proposes to integrate > OpenTracing, which is a vendor-neutral tracing API into the HMS server and > client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19808) GenericUDTFGetSplits should support ACID reads in the temp. table read path
[
https://issues.apache.org/jira/browse/HIVE-19808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated HIVE-19808:
Status: Patch Available (was: Open)
> GenericUDTFGetSplits should support ACID reads in the temp. table read path
> ---
>
> Key: HIVE-19808
> URL: https://issues.apache.org/jira/browse/HIVE-19808
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19808.1.patch, HIVE-19808.2.patch
>
>
> 1. Map-only reads work on ACID tables.
> 2. Temp. table reads (for multi-vertex queries) work on non-ACID tables.
> 3. But temp. table reads don't work on ACID tables.
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
> temp table: java.lang.IllegalStateException: calling recordValidTxn() more
> than once in the same txnid:420
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:303)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:202)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:918)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:492)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:484)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19808) GenericUDTFGetSplits should support ACID reads in the temp. table read path
[
https://issues.apache.org/jira/browse/HIVE-19808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated HIVE-19808:
Attachment: HIVE-19808.2.patch
> GenericUDTFGetSplits should support ACID reads in the temp. table read path
> ---
>
> Key: HIVE-19808
> URL: https://issues.apache.org/jira/browse/HIVE-19808
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19808.1.patch, HIVE-19808.2.patch
>
>
> 1. Map-only reads work on ACID tables.
> 2. Temp. table reads (for multi-vertex queries) work on non-ACID tables.
> 3. But temp. table reads don't work on ACID tables.
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
> temp table: java.lang.IllegalStateException: calling recordValidTxn() more
> than once in the same txnid:420
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:303)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:202)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:918)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:492)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:484)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19808) GenericUDTFGetSplits should support ACID reads in the temp. table read path
[
https://issues.apache.org/jira/browse/HIVE-19808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated HIVE-19808:
Status: Open (was: Patch Available)
> GenericUDTFGetSplits should support ACID reads in the temp. table read path
> ---
>
> Key: HIVE-19808
> URL: https://issues.apache.org/jira/browse/HIVE-19808
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19808.1.patch
>
>
> 1. Map-only reads work on ACID tables.
> 2. Temp. table reads (for multi-vertex queries) work on non-ACID tables.
> 3. But temp. table reads don't work on ACID tables.
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
> temp table: java.lang.IllegalStateException: calling recordValidTxn() more
> than once in the same txnid:420
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:303)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:202)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:918)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:492)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:484)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18119) show partitions should say whether a partition is stored via EC
[ https://issues.apache.org/jira/browse/HIVE-18119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman reassigned HIVE-18119: - Assignee: Andrew Sherman (was: Sahil Takiar) > show partitions should say whether a partition is stored via EC > --- > > Key: HIVE-18119 > URL: https://issues.apache.org/jira/browse/HIVE-18119 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > > Not sure what the criteria should be here because technically any single file > in a directory can be stored via EC. So a partition may contain both EC files > and regular files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18118) Explain Extended should indicate if a file being read is an EC file
[ https://issues.apache.org/jira/browse/HIVE-18118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman reassigned HIVE-18118: - Assignee: Andrew Sherman (was: Sahil Takiar) > Explain Extended should indicate if a file being read is an EC file > --- > > Key: HIVE-18118 > URL: https://issues.apache.org/jira/browse/HIVE-18118 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > > We already print out the files Hive will read in the explain extended > command, we just have to modify it to say whether or not its an EC file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19810) StorageHandler fail to ship jars in Tez intermittently
[ https://issues.apache.org/jira/browse/HIVE-19810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502619#comment-16502619 ] Daniel Dai commented on HIVE-19810: --- Unit test is tricky, seems tez container reuse properties are not controllable in the client side. I don't want to touch MiniTezCluster setting to affect mass tests. The attached testcase fails by chance in MiniTezCluster without the patch. Similar test on single node cluster failed reliably for me. > StorageHandler fail to ship jars in Tez intermittently > -- > > Key: HIVE-19810 > URL: https://issues.apache.org/jira/browse/HIVE-19810 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-19810.1.patch, testcase.patch > > > Hive relies on StorageHandler to ship jars to backend automatically in > several cases: JdbcStorageHandler, HBaseStorageHandler, > AccumuloStorageHandler. This does not work reliably, in particular, the first > dag in the session will have those jars, the second will not unless container > is reused. In the later case, the containers allocated to first dag will be > reused in the second dag so the container will have additional resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19810) StorageHandler fail to ship jars in Tez intermittently
[ https://issues.apache.org/jira/browse/HIVE-19810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-19810: -- Attachment: testcase.patch > StorageHandler fail to ship jars in Tez intermittently > -- > > Key: HIVE-19810 > URL: https://issues.apache.org/jira/browse/HIVE-19810 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-19810.1.patch, testcase.patch > > > Hive relies on StorageHandler to ship jars to backend automatically in > several cases: JdbcStorageHandler, HBaseStorageHandler, > AccumuloStorageHandler. This does not work reliably, in particular, the first > dag in the session will have those jars, the second will not unless container > is reused. In the later case, the containers allocated to first dag will be > reused in the second dag so the container will have additional resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19810) StorageHandler fail to ship jars in Tez intermittently
[ https://issues.apache.org/jira/browse/HIVE-19810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-19810: -- Attachment: HIVE-19810.1.patch > StorageHandler fail to ship jars in Tez intermittently > -- > > Key: HIVE-19810 > URL: https://issues.apache.org/jira/browse/HIVE-19810 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-19810.1.patch > > > Hive relies on StorageHandler to ship jars to backend automatically in > several cases: JdbcStorageHandler, HBaseStorageHandler, > AccumuloStorageHandler. This does not work reliably, in particular, the first > dag in the session will have those jars, the second will not unless container > is reused. In the later case, the containers allocated to first dag will be > reused in the second dag so the container will have additional resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19810) StorageHandler fail to ship jars in Tez intermittently
[ https://issues.apache.org/jira/browse/HIVE-19810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai reassigned HIVE-19810: - > StorageHandler fail to ship jars in Tez intermittently > -- > > Key: HIVE-19810 > URL: https://issues.apache.org/jira/browse/HIVE-19810 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > > Hive relies on StorageHandler to ship jars to backend automatically in > several cases: JdbcStorageHandler, HBaseStorageHandler, > AccumuloStorageHandler. This does not work reliably, in particular, the first > dag in the session will have those jars, the second will not unless container > is reused. In the later case, the containers allocated to first dag will be > reused in the second dag so the container will have additional resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19668) Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's and duplicate strings
[
https://issues.apache.org/jira/browse/HIVE-19668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502604#comment-16502604
]
Hive QA commented on HIVE-19668:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
19s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 5 new + 720 unchanged - 0
fixed = 725 total (was 720) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11536/dev-support/hive-personality.sh
|
| git revision | master / afc5fa4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11536/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11536/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11536/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's and
> duplicate strings
> --
>
> Key: HIVE-19668
> URL: https://issues.apache.org/jira/browse/HIVE-19668
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HIVE-19668.01.patch, image-2018-05-22-17-41-39-572.png
>
>
> I've recently analyzed a HS2 heap dump, obtained when there was a huge memory
> spike during compilation of some big query. The analysis was done with jxray
> ([www.jxray.com).|http://www.jxray.com)./] It turns out that more than 90% of
> the 20G heap was used by data structures associated with query parsing
> ({{org.apache.hadoop.hive.ql.parse.QBExpr}}). There are probably multiple
> opportunities for optimizations here. One of them is to stop the code from
> creating duplicate instances of {{org.antlr.runtime.CommonToken}} class. See
> a sample of these objects in the attached image:
> !image-2018-05-22-17-41-39-572.png|width=879,height=399!
> Looks like these particular {{CommonToken}} objects are constants, that don't
> change once created. I see some code, e.g. in
> {{org.apache.hadoop.hive.ql.parse.CalcitePlanner}}, where
[jira] [Updated] (HIVE-19772) Streaming ingest V2 API can generate invalid orc file if interrupted
[
https://issues.apache.org/jira/browse/HIVE-19772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-19772:
-
Resolution: Fixed
Fix Version/s: 4.0.0
3.0.1
3.1.0
Status: Resolved (was: Patch Available)
Committed to branch-3, branch-3.0 and master. Thanks for the review!
> Streaming ingest V2 API can generate invalid orc file if interrupted
>
>
> Key: HIVE-19772
> URL: https://issues.apache.org/jira/browse/HIVE-19772
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0, 3.0.1, 4.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Fix For: 3.1.0, 3.0.1, 4.0.0
>
> Attachments: HIVE-19772.1.patch, HIVE-19772.2.patch,
> HIVE-19772.3.patch
>
>
> Hive streaming ingest generated 0 length and 3 byte files which are invalid
> orc files. This will throw the following exception during compaction
> {code}
> Error: org.apache.orc.FileFormatException: Not a valid ORC file
> hdfs://cn105-10.l42scl.hortonworks.com:8020/apps/hive/warehouse/culvert/year=2018/month=7/delta_025_025/bucket_5
> (maxFileLength= 3) at
> org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:546) at
> org.apache.orc.impl.ReaderImpl.(ReaderImpl.java:370) at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:60) at
> org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:90) at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:1124)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:2373)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:1000)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:977)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:460) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:344) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19772) Streaming ingest V2 API can generate invalid orc file if interrupted
[
https://issues.apache.org/jira/browse/HIVE-19772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502595#comment-16502595
]
Prasanth Jayachandran commented on HIVE-19772:
--
Replaced FS.get with FS.newInstance. Also increased the shutdown hook priority
of streaming connection to be > filesystem so that streaming connection gets
closed before filesystem close.
The change shouldn't affect any tests. Will commit it shortly.
> Streaming ingest V2 API can generate invalid orc file if interrupted
>
>
> Key: HIVE-19772
> URL: https://issues.apache.org/jira/browse/HIVE-19772
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0, 3.0.1, 4.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-19772.1.patch, HIVE-19772.2.patch,
> HIVE-19772.3.patch
>
>
> Hive streaming ingest generated 0 length and 3 byte files which are invalid
> orc files. This will throw the following exception during compaction
> {code}
> Error: org.apache.orc.FileFormatException: Not a valid ORC file
> hdfs://cn105-10.l42scl.hortonworks.com:8020/apps/hive/warehouse/culvert/year=2018/month=7/delta_025_025/bucket_5
> (maxFileLength= 3) at
> org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:546) at
> org.apache.orc.impl.ReaderImpl.(ReaderImpl.java:370) at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:60) at
> org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:90) at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:1124)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:2373)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:1000)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:977)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:460) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:344) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19772) Streaming ingest V2 API can generate invalid orc file if interrupted
[
https://issues.apache.org/jira/browse/HIVE-19772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-19772:
-
Attachment: HIVE-19772.3.patch
> Streaming ingest V2 API can generate invalid orc file if interrupted
>
>
> Key: HIVE-19772
> URL: https://issues.apache.org/jira/browse/HIVE-19772
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0, 3.0.1, 4.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-19772.1.patch, HIVE-19772.2.patch,
> HIVE-19772.3.patch
>
>
> Hive streaming ingest generated 0 length and 3 byte files which are invalid
> orc files. This will throw the following exception during compaction
> {code}
> Error: org.apache.orc.FileFormatException: Not a valid ORC file
> hdfs://cn105-10.l42scl.hortonworks.com:8020/apps/hive/warehouse/culvert/year=2018/month=7/delta_025_025/bucket_5
> (maxFileLength= 3) at
> org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:546) at
> org.apache.orc.impl.ReaderImpl.(ReaderImpl.java:370) at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:60) at
> org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:90) at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:1124)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:2373)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:1000)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:977)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:460) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:344) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19808) GenericUDTFGetSplits should support ACID reads in the temp. table read path
[
https://issues.apache.org/jira/browse/HIVE-19808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-19808:
--
Component/s: Transactions
> GenericUDTFGetSplits should support ACID reads in the temp. table read path
> ---
>
> Key: HIVE-19808
> URL: https://issues.apache.org/jira/browse/HIVE-19808
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19808.1.patch
>
>
> 1. Map-only reads work on ACID tables.
> 2. Temp. table reads (for multi-vertex queries) work on non-ACID tables.
> 3. But temp. table reads don't work on ACID tables.
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
> temp table: java.lang.IllegalStateException: calling recordValidTxn() more
> than once in the same txnid:420
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:303)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:202)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:918)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:492)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:484)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19808) GenericUDTFGetSplits should support ACID reads in the temp. table read path
[
https://issues.apache.org/jira/browse/HIVE-19808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502579#comment-16502579
]
Eric Wohlstadter commented on HIVE-19808:
-
[~jdere] [~prasanth_j]
[https://reviews.apache.org/r/67462/]
> GenericUDTFGetSplits should support ACID reads in the temp. table read path
> ---
>
> Key: HIVE-19808
> URL: https://issues.apache.org/jira/browse/HIVE-19808
> Project: Hive
> Issue Type: Bug
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19808.1.patch
>
>
> 1. Map-only reads work on ACID tables.
> 2. Temp. table reads (for multi-vertex queries) work on non-ACID tables.
> 3. But temp. table reads don't work on ACID tables.
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
> temp table: java.lang.IllegalStateException: calling recordValidTxn() more
> than once in the same txnid:420
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:303)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:202)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:918)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:492)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:484)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19493) VectorUDFDateDiffColCol copySelected does not handle nulls correctly
[
https://issues.apache.org/jira/browse/HIVE-19493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502572#comment-16502572
]
Hive QA commented on HIVE-19493:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926212/HIVE-19493.04.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11535/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11535/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11535/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12926212/HIVE-19493.04.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926212 - PreCommit-HIVE-Build
> VectorUDFDateDiffColCol copySelected does not handle nulls correctly
>
>
> Key: HIVE-19493
> URL: https://issues.apache.org/jira/browse/HIVE-19493
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Vihang Karajgaonkar
>Assignee: Matt McCline
>Priority: Major
> Fix For: 3.1.0, 4.0.0
>
> Attachments: HIVE-19493.01.patch, HIVE-19493.02.patch,
> HIVE-19493.04.patch
>
>
> The {{copySelected}} method in {{VectorUDFDateDiffColCol}} class was missed
> during HIVE-18622
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19773) CBO exception while running queries with tables that are not present in materialized views
[
https://issues.apache.org/jira/browse/HIVE-19773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502568#comment-16502568
]
Hive QA commented on HIVE-19773:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926204/HIVE-19773.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14467 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testAsyncSessionInitFailures
(batchId=296)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11533/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11533/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11533/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926204 - PreCommit-HIVE-Build
> CBO exception while running queries with tables that are not present in
> materialized views
> --
>
> Key: HIVE-19773
> URL: https://issues.apache.org/jira/browse/HIVE-19773
> Project: Hive
> Issue Type: Bug
> Components: Materialized views
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Aswathy Chellammal Sreekumar
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Attachments: HIVE-19773.patch
>
>
> When we obtain the valid list of write ids, some tables in the materialized
> views may not be present in the list because they are not present in the
> query, which leads to exceptions (hidden in logs) when we try to load the
> materialized views in the planner, as we need to verify whether they are
> outdated or not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19808) GenericUDTFGetSplits should support ACID reads in the temp. table read path
[
https://issues.apache.org/jira/browse/HIVE-19808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated HIVE-19808:
Status: Patch Available (was: Open)
> GenericUDTFGetSplits should support ACID reads in the temp. table read path
> ---
>
> Key: HIVE-19808
> URL: https://issues.apache.org/jira/browse/HIVE-19808
> Project: Hive
> Issue Type: Bug
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19808.1.patch
>
>
> 1. Map-only reads work on ACID tables.
> 2. Temp. table reads (for multi-vertex queries) work on non-ACID tables.
> 3. But temp. table reads don't work on ACID tables.
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
> temp table: java.lang.IllegalStateException: calling recordValidTxn() more
> than once in the same txnid:420
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:303)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:202)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:918)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:492)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:484)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19808) GenericUDTFGetSplits should support ACID reads in the temp. table read path
[
https://issues.apache.org/jira/browse/HIVE-19808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated HIVE-19808:
Attachment: HIVE-19808.1.patch
> GenericUDTFGetSplits should support ACID reads in the temp. table read path
> ---
>
> Key: HIVE-19808
> URL: https://issues.apache.org/jira/browse/HIVE-19808
> Project: Hive
> Issue Type: Bug
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19808.1.patch
>
>
> 1. Map-only reads work on ACID tables.
> 2. Temp. table reads (for multi-vertex queries) work on non-ACID tables.
> 3. But temp. table reads don't work on ACID tables.
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
> temp table: java.lang.IllegalStateException: calling recordValidTxn() more
> than once in the same txnid:420
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:303)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:202)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:918)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:492)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:484)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19808) GenericUDTFGetSplits should support ACID reads in the temp. table read path
[
https://issues.apache.org/jira/browse/HIVE-19808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502563#comment-16502563
]
Eric Wohlstadter commented on HIVE-19808:
-
[~ekoifman]
Yeah.
What happens in GenericUDTFGetSplits is:
{code:java}
"create temporary table " + tableName + " as " + query
{code}
and then the temp table is read into LLAP and exported by the
{{LlapOutputFormatService}}.
Currently if {{query}} references an ACID table, then it fails.
The temp table itself is not ACID, the issue occurs when any of the source
tables are ACID.
> GenericUDTFGetSplits should support ACID reads in the temp. table read path
> ---
>
> Key: HIVE-19808
> URL: https://issues.apache.org/jira/browse/HIVE-19808
> Project: Hive
> Issue Type: Bug
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
>
> 1. Map-only reads work on ACID tables.
> 2. Temp. table reads (for multi-vertex queries) work on non-ACID tables.
> 3. But temp. table reads don't work on ACID tables.
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
> temp table: java.lang.IllegalStateException: calling recordValidTxn() more
> than once in the same txnid:420
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:303)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:202)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:918)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:492)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:484)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19808) GenericUDTFGetSplits should support ACID reads in the temp. table read path
[
https://issues.apache.org/jira/browse/HIVE-19808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502563#comment-16502563
]
Eric Wohlstadter edited comment on HIVE-19808 at 6/5/18 9:59 PM:
-
[~ekoifman] [~jdere]
Yeah.
What happens in GenericUDTFGetSplits is:
{code:java}
"create temporary table " + tableName + " as " + query
{code}
and then the temp table is read into LLAP and exported by the
{{LlapOutputFormatService}}.
Currently if {{query}} references an ACID table, then it fails.
The temp table itself is not ACID, the issue occurs when any of the source
tables are ACID.
was (Author: ewohlstadter):
[~ekoifman]
Yeah.
What happens in GenericUDTFGetSplits is:
{code:java}
"create temporary table " + tableName + " as " + query
{code}
and then the temp table is read into LLAP and exported by the
{{LlapOutputFormatService}}.
Currently if {{query}} references an ACID table, then it fails.
The temp table itself is not ACID, the issue occurs when any of the source
tables are ACID.
> GenericUDTFGetSplits should support ACID reads in the temp. table read path
> ---
>
> Key: HIVE-19808
> URL: https://issues.apache.org/jira/browse/HIVE-19808
> Project: Hive
> Issue Type: Bug
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
>
> 1. Map-only reads work on ACID tables.
> 2. Temp. table reads (for multi-vertex queries) work on non-ACID tables.
> 3. But temp. table reads don't work on ACID tables.
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
> temp table: java.lang.IllegalStateException: calling recordValidTxn() more
> than once in the same txnid:420
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:303)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:202)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:918)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:492)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:484)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19808) GenericUDTFGetSplits should support ACID reads in the temp. table read path
[
https://issues.apache.org/jira/browse/HIVE-19808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502548#comment-16502548
]
Eugene Koifman commented on HIVE-19808:
---
I don't think temp tables can properly support acid
> GenericUDTFGetSplits should support ACID reads in the temp. table read path
> ---
>
> Key: HIVE-19808
> URL: https://issues.apache.org/jira/browse/HIVE-19808
> Project: Hive
> Issue Type: Bug
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
>
> 1. Map-only reads work on ACID tables.
> 2. Temp. table reads (for multi-vertex queries) work on non-ACID tables.
> 3. But temp. table reads don't work on ACID tables.
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
> temp table: java.lang.IllegalStateException: calling recordValidTxn() more
> than once in the same txnid:420
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:303)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:202)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:918)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:492)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:484)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19769) Create dedicated objects for DB and Table names
[ https://issues.apache.org/jira/browse/HIVE-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502542#comment-16502542 ] Alan Gates commented on HIVE-19769: --- I took a quick look at what's in MetaStoreDirectSql and it looks pretty similar. If you're ok with it I'll just move its functionality into these new objects in common. > Create dedicated objects for DB and Table names > --- > > Key: HIVE-19769 > URL: https://issues.apache.org/jira/browse/HIVE-19769 > Project: Hive > Issue Type: Sub-task > Components: storage-api >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > > Currently table names are always strings. Sometimes that string is just > tablename, sometimes it is dbname.tablename. Sometimes the code expects one > or the other, sometimes it handles either. This is burdensome for developers > and error prone. With the addition of catalog to the hierarchy, this becomes > even worse. > I propose to add two objects, DatabaseName and TableName. These will track > full names of each object. They will handle inserting default catalog and > database names when those are not provided. They will handle the conversions > to and from strings. > These will need to be added to storage-api because ValidTxnList will use it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19804) msck repair should hold locks
[
https://issues.apache.org/jira/browse/HIVE-19804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502541#comment-16502541
]
Sergey Shelukhin commented on HIVE-19804:
-
Sounds good to me
> msck repair should hold locks
> -
>
> Key: HIVE-19804
> URL: https://issues.apache.org/jira/browse/HIVE-19804
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Priority: Major
>
> {msck repair table }} does not hold locks. This can lead to weird
> race conditions when concurrent sessions are running on the same table. For
> example if two sessions run msck on the same table at the same time, they
> both try to add partitions and they might both end up with failures due to
> AlreadyExistsException. Another example would be if a query is running on a
> partitioned table while some other session issues msck repair which add/drops
> the partitions, it could trigger errors during query execution.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19809) Remove Deprecated Code From Utilities Class
[
https://issues.apache.org/jira/browse/HIVE-19809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-19809:
---
Attachment: HIVE-19809.1.patch
> Remove Deprecated Code From Utilities Class
> ---
>
> Key: HIVE-19809
> URL: https://issues.apache.org/jira/browse/HIVE-19809
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-19809.1.patch
>
>
> {quote}
> This can go away once hive moves to support only JDK 7 and can use
> Files.createTempDirectory
> {quote}
> Remove the {{createTempDir}} method from the {{Utilities}} class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19809) Remove Deprecated Code From Utilities Class
[
https://issues.apache.org/jira/browse/HIVE-19809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-19809:
---
Status: Patch Available (was: Open)
> Remove Deprecated Code From Utilities Class
> ---
>
> Key: HIVE-19809
> URL: https://issues.apache.org/jira/browse/HIVE-19809
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-19809.1.patch
>
>
> {quote}
> This can go away once hive moves to support only JDK 7 and can use
> Files.createTempDirectory
> {quote}
> Remove the {{createTempDir}} method from the {{Utilities}} class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19809) Remove Deprecated Code From Utilities Class
[
https://issues.apache.org/jira/browse/HIVE-19809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR reassigned HIVE-19809:
--
Assignee: BELUGA BEHR
> Remove Deprecated Code From Utilities Class
> ---
>
> Key: HIVE-19809
> URL: https://issues.apache.org/jira/browse/HIVE-19809
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-19809.1.patch
>
>
> {quote}
> This can go away once hive moves to support only JDK 7 and can use
> Files.createTempDirectory
> {quote}
> Remove the {{createTempDir}} method from the {{Utilities}} class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19809) Remove Deprecated Code From Utilities Class
[
https://issues.apache.org/jira/browse/HIVE-19809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-19809:
---
Affects Version/s: 4.0.0
3.0.0
> Remove Deprecated Code From Utilities Class
> ---
>
> Key: HIVE-19809
> URL: https://issues.apache.org/jira/browse/HIVE-19809
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Priority: Minor
>
> {quote}
> This can go away once hive moves to support only JDK 7 and can use
> Files.createTempDirectory
> {quote}
> Remove the {{createTempDir}} method from the {{Utilities}} class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19773) CBO exception while running queries with tables that are not present in materialized views
[
https://issues.apache.org/jira/browse/HIVE-19773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502533#comment-16502533
]
Hive QA commented on HIVE-19773:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
24s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 3 new + 388 unchanged - 2
fixed = 391 total (was 390) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
35s{color} | {color:red} ql generated 2 new + 2278 unchanged - 2 fixed = 2280
total (was 2280) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Exception is caught when Exception is not thrown in
org.apache.hadoop.hive.ql.metadata.Hive.getValidMaterializedViews(String, List,
boolean, String) At Hive.java:is not thrown in
org.apache.hadoop.hive.ql.metadata.Hive.getValidMaterializedViews(String, List,
boolean, String) At Hive.java:[line 1492] |
| | Dead store to group in org.apache.hadoop.hive.ql.metadata.Hive$8.call()
At Hive.java:org.apache.hadoop.hive.ql.metadata.Hive$8.call() At
Hive.java:[line 3686] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11533/dev-support/hive-personality.sh
|
| git revision | master / afc5fa4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11533/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11533/yetus/new-findbugs-ql.html
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11533/yetus/patch-asflicense-problems.txt
|
| modules | C: itests ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11533/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CBO exception while running queries with tables that are not present in
> materialized views
> --
>
> Key: HIVE-19773
> URL: https://issues.apache.org/jira/browse/HIVE-19773
> Project: Hive
> Issue Type: Bug
> Components: Materialized views
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Aswathy Chellammal Sreekumar
>
[jira] [Assigned] (HIVE-19808) GenericUDTFGetSplits should support ACID reads in the temp. table read path
[
https://issues.apache.org/jira/browse/HIVE-19808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter reassigned HIVE-19808:
---
> GenericUDTFGetSplits should support ACID reads in the temp. table read path
> ---
>
> Key: HIVE-19808
> URL: https://issues.apache.org/jira/browse/HIVE-19808
> Project: Hive
> Issue Type: Bug
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
>
> 1. Map-only reads work on ACID tables.
> 2. Temp. table reads (for multi-vertex queries) work on non-ACID tables.
> 3. But temp. table reads don't work on ACID tables.
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
> temp table: java.lang.IllegalStateException: calling recordValidTxn() more
> than once in the same txnid:420
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:303)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:202)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:918)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:492)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:484)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19807) Add Useful Error Message To Table Header/Footer Parsing
[
https://issues.apache.org/jira/browse/HIVE-19807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-19807:
---
Description:
Add some useful logging messages to invalid value parsing of
{{skip.header.line.count}} and {{skip.footer.line.count}} for better
troubleshooting.
Also, parsing error resulted in {{java.io.IOException}} which:
{quote}
Signals that an I/O exception of some sort has occurred. This class is the
general class of exceptions produced by failed or interrupted I/O operations.
{quote}
A parsing error is not the result of a failed or interrupted I/O operation so I
dropped it and replaced with runtime exception {{IllegalArgumentException}}.
was:Add some useful logging messages to invalid value parsing of
{{skip.header.line.count}} and {{skip.footer.line.count}} for better
troubleshooting.
> Add Useful Error Message To Table Header/Footer Parsing
> ---
>
> Key: HIVE-19807
> URL: https://issues.apache.org/jira/browse/HIVE-19807
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-19807.1.patch
>
>
> Add some useful logging messages to invalid value parsing of
> {{skip.header.line.count}} and {{skip.footer.line.count}} for better
> troubleshooting.
> Also, parsing error resulted in {{java.io.IOException}} which:
> {quote}
> Signals that an I/O exception of some sort has occurred. This class is the
> general class of exceptions produced by failed or interrupted I/O operations.
> {quote}
> A parsing error is not the result of a failed or interrupted I/O operation so
> I dropped it and replaced with runtime exception {{IllegalArgumentException}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19807) Add Useful Error Message To Table Header/Footer Parsing
[
https://issues.apache.org/jira/browse/HIVE-19807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-19807:
---
Attachment: HIVE-19807.1.patch
> Add Useful Error Message To Table Header/Footer Parsing
> ---
>
> Key: HIVE-19807
> URL: https://issues.apache.org/jira/browse/HIVE-19807
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.3.2
>Reporter: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-19807.1.patch
>
>
> Add some useful logging messages to invalid value parsing of
> {{skip.header.line.count}} and {{skip.footer.line.count}} for better
> troubleshooting.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19807) Add Useful Error Message To Table Header/Footer Parsing
[
https://issues.apache.org/jira/browse/HIVE-19807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR reassigned HIVE-19807:
--
Assignee: BELUGA BEHR
> Add Useful Error Message To Table Header/Footer Parsing
> ---
>
> Key: HIVE-19807
> URL: https://issues.apache.org/jira/browse/HIVE-19807
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-19807.1.patch
>
>
> Add some useful logging messages to invalid value parsing of
> {{skip.header.line.count}} and {{skip.footer.line.count}} for better
> troubleshooting.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19807) Add Useful Error Message To Table Header/Footer Parsing
[
https://issues.apache.org/jira/browse/HIVE-19807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-19807:
---
Status: Patch Available (was: Open)
> Add Useful Error Message To Table Header/Footer Parsing
> ---
>
> Key: HIVE-19807
> URL: https://issues.apache.org/jira/browse/HIVE-19807
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 2.3.2, 3.0.0
>Reporter: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-19807.1.patch
>
>
> Add some useful logging messages to invalid value parsing of
> {{skip.header.line.count}} and {{skip.footer.line.count}} for better
> troubleshooting.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19772) Streaming ingest V2 API can generate invalid orc file if interrupted
[
https://issues.apache.org/jira/browse/HIVE-19772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502506#comment-16502506
]
Hive QA commented on HIVE-19772:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926486/HIVE-19772.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14466 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11532/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11532/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11532/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926486 - PreCommit-HIVE-Build
> Streaming ingest V2 API can generate invalid orc file if interrupted
>
>
> Key: HIVE-19772
> URL: https://issues.apache.org/jira/browse/HIVE-19772
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0, 3.0.1, 4.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-19772.1.patch, HIVE-19772.2.patch
>
>
> Hive streaming ingest generated 0 length and 3 byte files which are invalid
> orc files. This will throw the following exception during compaction
> {code}
> Error: org.apache.orc.FileFormatException: Not a valid ORC file
> hdfs://cn105-10.l42scl.hortonworks.com:8020/apps/hive/warehouse/culvert/year=2018/month=7/delta_025_025/bucket_5
> (maxFileLength= 3) at
> org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:546) at
> org.apache.orc.impl.ReaderImpl.(ReaderImpl.java:370) at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:60) at
> org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:90) at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:1124)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:2373)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:1000)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:977)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:460) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:344) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19558) HiveAuthorizationProviderBase gets catalog name from config rather than db object
[ https://issues.apache.org/jira/browse/HIVE-19558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502497#comment-16502497 ] Alan Gates commented on HIVE-19558: --- I have, there's a 1take9 version of the patch. I submitted it 2 days ago, and we're running at about 3 day waits right now. So it should be through by tomorrow. > HiveAuthorizationProviderBase gets catalog name from config rather than db > object > - > > Key: HIVE-19558 > URL: https://issues.apache.org/jira/browse/HIVE-19558 > Project: Hive > Issue Type: Bug > Components: Authorization >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Fix For: 3.0.1 > > Attachments: HIVE-19558.1take2.patch, HIVE-19558.1take3.patch, > HIVE-19558.1take4.patch, HIVE-19558.1take5.patch, HIVE-19558.1take6.patch, > HIVE-19558.1take7.patch, HIVE-19558.1take8.patch, HIVE-19558.1take9.patch, > HIVE-19558.patch > > > HiveAuthorizationProviderBase.getDatabase uses just the database name to > fetch the database, relying on getDefaultCatalog() to fetch the catalog name > from the conf file. This does not work when the client has passed in an > object for a different catalog. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19806) Several tests do not properly sort their output
[ https://issues.apache.org/jira/browse/HIVE-19806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-19806: -- Attachment: HIVE-19806.patch > Several tests do not properly sort their output > --- > > Key: HIVE-19806 > URL: https://issues.apache.org/jira/browse/HIVE-19806 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Attachments: HIVE-19806.patch > > > A number of the tests produce unsorted output that happens to come out the > same on people's laptops and the ptest infrastructure. But when run on a > separate linux box the sort differences show up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19806) Several tests do not properly sort their output
[ https://issues.apache.org/jira/browse/HIVE-19806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-19806: -- Status: Patch Available (was: Open) > Several tests do not properly sort their output > --- > > Key: HIVE-19806 > URL: https://issues.apache.org/jira/browse/HIVE-19806 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Attachments: HIVE-19806.patch > > > A number of the tests produce unsorted output that happens to come out the > same on people's laptops and the ptest infrastructure. But when run on a > separate linux box the sort differences show up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19558) HiveAuthorizationProviderBase gets catalog name from config rather than db object
[ https://issues.apache.org/jira/browse/HIVE-19558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502494#comment-16502494 ] Dongjoon Hyun commented on HIVE-19558: -- [~alangates] and [~thejas] . If this fails 8 times with different flakiness reasons, can we retrigger once more? cc [~bikassaha] . > HiveAuthorizationProviderBase gets catalog name from config rather than db > object > - > > Key: HIVE-19558 > URL: https://issues.apache.org/jira/browse/HIVE-19558 > Project: Hive > Issue Type: Bug > Components: Authorization >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Fix For: 3.0.1 > > Attachments: HIVE-19558.1take2.patch, HIVE-19558.1take3.patch, > HIVE-19558.1take4.patch, HIVE-19558.1take5.patch, HIVE-19558.1take6.patch, > HIVE-19558.1take7.patch, HIVE-19558.1take8.patch, HIVE-19558.1take9.patch, > HIVE-19558.patch > > > HiveAuthorizationProviderBase.getDatabase uses just the database name to > fetch the database, relying on getDefaultCatalog() to fetch the catalog name > from the conf file. This does not work when the client has passed in an > object for a different catalog. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19806) Several tests do not properly sort their output
[ https://issues.apache.org/jira/browse/HIVE-19806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates reassigned HIVE-19806: - > Several tests do not properly sort their output > --- > > Key: HIVE-19806 > URL: https://issues.apache.org/jira/browse/HIVE-19806 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > > A number of the tests produce unsorted output that happens to come out the > same on people's laptops and the ptest infrastructure. But when run on a > separate linux box the sort differences show up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19053) RemoteSparkJobStatus#getSparkJobInfo treats all exceptions as timeout errors
[
https://issues.apache.org/jira/browse/HIVE-19053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502449#comment-16502449
]
Aihua Xu edited comment on HIVE-19053 at 6/5/18 8:50 PM:
-
That makes sense. Can you check the patch-2?
was (Author: aihuaxu):
That makes sense. Let me upload a new patch.
> RemoteSparkJobStatus#getSparkJobInfo treats all exceptions as timeout errors
>
>
> Key: HIVE-19053
> URL: https://issues.apache.org/jira/browse/HIVE-19053
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19053.1.patch, HIVE-19053.2.patch
>
>
> {code}
> Future getJobInfo = sparkClient.run(
> new GetJobInfoJob(jobHandle.getClientJobId(), sparkJobId));
> try {
> return getJobInfo.get(sparkClientTimeoutInSeconds, TimeUnit.SECONDS);
> } catch (Exception e) {
> LOG.warn("Failed to get job info.", e);
> throw new HiveException(e, ErrorMsg.SPARK_GET_JOB_INFO_TIMEOUT,
> Long.toString(sparkClientTimeoutInSeconds));
> }
> {code}
> It should only throw {{ErrorMsg.SPARK_GET_JOB_INFO_TIMEOUT}} if a
> {{TimeoutException}} is thrown. Other exceptions should be handled
> independently.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19053) RemoteSparkJobStatus#getSparkJobInfo treats all exceptions as timeout errors
[
https://issues.apache.org/jira/browse/HIVE-19053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-19053:
Attachment: HIVE-19053.2.patch
> RemoteSparkJobStatus#getSparkJobInfo treats all exceptions as timeout errors
>
>
> Key: HIVE-19053
> URL: https://issues.apache.org/jira/browse/HIVE-19053
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19053.1.patch, HIVE-19053.2.patch
>
>
> {code}
> Future getJobInfo = sparkClient.run(
> new GetJobInfoJob(jobHandle.getClientJobId(), sparkJobId));
> try {
> return getJobInfo.get(sparkClientTimeoutInSeconds, TimeUnit.SECONDS);
> } catch (Exception e) {
> LOG.warn("Failed to get job info.", e);
> throw new HiveException(e, ErrorMsg.SPARK_GET_JOB_INFO_TIMEOUT,
> Long.toString(sparkClientTimeoutInSeconds));
> }
> {code}
> It should only throw {{ErrorMsg.SPARK_GET_JOB_INFO_TIMEOUT}} if a
> {{TimeoutException}} is thrown. Other exceptions should be handled
> independently.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19805) TableScanDesc Use Commons Library
[ https://issues.apache.org/jira/browse/HIVE-19805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-19805: --- Attachment: HIVE-19805.1.patch > TableScanDesc Use Commons Library > - > > Key: HIVE-19805 > URL: https://issues.apache.org/jira/browse/HIVE-19805 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: BELUGA BEHR >Priority: Trivial > Attachments: HIVE-19805.1.patch > > > Use commons library and remove some code -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19805) TableScanDesc Use Commons Library
[ https://issues.apache.org/jira/browse/HIVE-19805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-19805: --- Status: Patch Available (was: Open) > TableScanDesc Use Commons Library > - > > Key: HIVE-19805 > URL: https://issues.apache.org/jira/browse/HIVE-19805 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: BELUGA BEHR >Priority: Trivial > Attachments: HIVE-19805.1.patch > > > Use commons library and remove some code -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19805) TableScanDesc Use Commons Library
[ https://issues.apache.org/jira/browse/HIVE-19805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR reassigned HIVE-19805: -- Assignee: BELUGA BEHR > TableScanDesc Use Commons Library > - > > Key: HIVE-19805 > URL: https://issues.apache.org/jira/browse/HIVE-19805 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: HIVE-19805.1.patch > > > Use commons library and remove some code -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19804) msck repair should hold locks
[
https://issues.apache.org/jira/browse/HIVE-19804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502472#comment-16502472
]
Vihang Karajgaonkar commented on HIVE-19804:
[~ashutoshc] [~sershe] any thoughts?
> msck repair should hold locks
> -
>
> Key: HIVE-19804
> URL: https://issues.apache.org/jira/browse/HIVE-19804
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Priority: Major
>
> {msck repair table }} does not hold locks. This can lead to weird
> race conditions when concurrent sessions are running on the same table. For
> example if two sessions run msck on the same table at the same time, they
> both try to add partitions and they might both end up with failures due to
> AlreadyExistsException. Another example would be if a query is running on a
> partitioned table while some other session issues msck repair which add/drops
> the partitions, it could trigger errors during query execution.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19053) RemoteSparkJobStatus#getSparkJobInfo treats all exceptions as timeout errors
[
https://issues.apache.org/jira/browse/HIVE-19053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502449#comment-16502449
]
Aihua Xu commented on HIVE-19053:
-
That makes sense. Let me upload a new patch.
> RemoteSparkJobStatus#getSparkJobInfo treats all exceptions as timeout errors
>
>
> Key: HIVE-19053
> URL: https://issues.apache.org/jira/browse/HIVE-19053
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19053.1.patch
>
>
> {code}
> Future getJobInfo = sparkClient.run(
> new GetJobInfoJob(jobHandle.getClientJobId(), sparkJobId));
> try {
> return getJobInfo.get(sparkClientTimeoutInSeconds, TimeUnit.SECONDS);
> } catch (Exception e) {
> LOG.warn("Failed to get job info.", e);
> throw new HiveException(e, ErrorMsg.SPARK_GET_JOB_INFO_TIMEOUT,
> Long.toString(sparkClientTimeoutInSeconds));
> }
> {code}
> It should only throw {{ErrorMsg.SPARK_GET_JOB_INFO_TIMEOUT}} if a
> {{TimeoutException}} is thrown. Other exceptions should be handled
> independently.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19772) Streaming ingest V2 API can generate invalid orc file if interrupted
[
https://issues.apache.org/jira/browse/HIVE-19772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502440#comment-16502440
]
Hive QA commented on HIVE-19772:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
34s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
23s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} streaming: The patch generated 1 new + 54 unchanged -
1 fixed = 55 total (was 55) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11532/dev-support/hive-personality.sh
|
| git revision | master / afc5fa4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11532/yetus/diff-checkstyle-streaming.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11532/yetus/patch-asflicense-problems.txt
|
| modules | C: ql streaming U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11532/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Streaming ingest V2 API can generate invalid orc file if interrupted
>
>
> Key: HIVE-19772
> URL: https://issues.apache.org/jira/browse/HIVE-19772
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0, 3.0.1, 4.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-19772.1.patch, HIVE-19772.2.patch
>
>
> Hive streaming ingest generated 0 length and 3 byte files which are invalid
> orc files. This will throw the following exception during compaction
> {code}
> Error: org.apache.orc.FileFormatException: Not a valid ORC file
> hdfs://cn105-10.l42scl.hortonworks.com:8020/apps/hive/warehouse/culvert/year=2018/month=7/delta_025_025/bucket_5
> (maxFileLength= 3) at
>
[jira] [Commented] (HIVE-18766) Race condition during shutdown of RemoteDriver, error messages aren't always sent
[
https://issues.apache.org/jira/browse/HIVE-18766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502439#comment-16502439
]
Aihua Xu commented on HIVE-18766:
-
[~stakiar] I don't see another RpcConfiguration object gets created there. Can
you check the patch-2 if it makes sense?
> Race condition during shutdown of RemoteDriver, error messages aren't always
> sent
> -
>
> Key: HIVE-18766
> URL: https://issues.apache.org/jira/browse/HIVE-18766
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-18766.1.patch, HIVE-18766.2.patch
>
>
> When we send an error during shutdown of the {{RemoteDriver}}, we don't wait
> for the error message to be sent. We just send it and then close the RPC
> channel. For a graceful shutdown, that doesn't seem ideal. We should at least
> wait a bit for the RPC message to be sent before shutting things down. I ran
> some tests locally and its pretty easy to hit a situation where the error
> message doesn't even get sent to the {{SparkClientImpl}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18766) Race condition during shutdown of RemoteDriver, error messages aren't always sent
[
https://issues.apache.org/jira/browse/HIVE-18766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-18766:
Attachment: HIVE-18766.2.patch
> Race condition during shutdown of RemoteDriver, error messages aren't always
> sent
> -
>
> Key: HIVE-18766
> URL: https://issues.apache.org/jira/browse/HIVE-18766
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-18766.1.patch, HIVE-18766.2.patch
>
>
> When we send an error during shutdown of the {{RemoteDriver}}, we don't wait
> for the error message to be sent. We just send it and then close the RPC
> channel. For a graceful shutdown, that doesn't seem ideal. We should at least
> wait a bit for the RPC message to be sent before shutting things down. I ran
> some tests locally and its pretty easy to hit a situation where the error
> message doesn't even get sent to the {{SparkClientImpl}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19770) Support for CBO for queries with multiple same columns in select
[
https://issues.apache.org/jira/browse/HIVE-19770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502401#comment-16502401
]
Hive QA commented on HIVE-19770:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926191/HIVE-19770.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 14466 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[explainuser_2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainanalyze_2]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[limit_join_transpose]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_nonvec_part_all_primitive_llap_io]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_nonvec_table]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_vec_part]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_nonvec_table_llap_io]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[varchar_udf1]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_udf1]
(batchId=169)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[ambiguous_col]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[create_view_failure5]
(batchId=96)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketsortoptimize_insert_4]
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_remove_22]
(batchId=143)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11531/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11531/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11531/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926191 - PreCommit-HIVE-Build
> Support for CBO for queries with multiple same columns in select
>
>
> Key: HIVE-19770
> URL: https://issues.apache.org/jira/browse/HIVE-19770
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19770.1.patch
>
>
> Currently queries such as {code:sql} select a,a from t1 where b > 10 {code}
> are not supported for CBO.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19733) RemoteSparkJobStatus#getSparkStageProgress inefficient implementation
[
https://issues.apache.org/jira/browse/HIVE-19733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali reassigned HIVE-19733:
---
Assignee: Bharathkrishna Guruvayoor Murali
> RemoteSparkJobStatus#getSparkStageProgress inefficient implementation
> -
>
> Key: HIVE-19733
> URL: https://issues.apache.org/jira/browse/HIVE-19733
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
>
> The implementation of {{RemoteSparkJobStatus#getSparkStageProgress}} is a bit
> inefficient. There is one RPC call to get the {{SparkJobInfo}} and then for
> every stage there is another RPC call to get each {{SparkStageInfo}}. This
> could all be done in a single RPC call.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19758) Set hadoop.version=3.1.0 in standalone-metastore
[ https://issues.apache.org/jira/browse/HIVE-19758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-19758: --- Resolution: Fixed Fix Version/s: 4.0.0 3.1.0 Status: Resolved (was: Patch Available) Patch merged into master and branch-3. The patch was very small and didn't warrant a HiveQA run on branch-3 given that master came through without any issues. > Set hadoop.version=3.1.0 in standalone-metastore > > > Key: HIVE-19758 > URL: https://issues.apache.org/jira/browse/HIVE-19758 > Project: Hive > Issue Type: Sub-task >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19758.1.patch > > > When HIVE-19243 set hadoop.version=3.1.0 it did not change the value used in > standalone-metastore which still uses 3.0.0-beta1. > At the moment standalone-metastore is still a module of hive and so this can > suck in the wrong code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19758) Set hadoop.version=3.1.0 in standalone-metastore
[ https://issues.apache.org/jira/browse/HIVE-19758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502385#comment-16502385 ] Vihang Karajgaonkar commented on HIVE-19758: Thanks for your contribution [~asherman] > Set hadoop.version=3.1.0 in standalone-metastore > > > Key: HIVE-19758 > URL: https://issues.apache.org/jira/browse/HIVE-19758 > Project: Hive > Issue Type: Sub-task >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19758.1.patch > > > When HIVE-19243 set hadoop.version=3.1.0 it did not change the value used in > standalone-metastore which still uses 3.0.0-beta1. > At the moment standalone-metastore is still a module of hive and so this can > suck in the wrong code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[ https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502380#comment-16502380 ] Steve Yeom commented on HIVE-19417: --- Create a patch 05 for this jira reflecting the talk of last Thursday. See HIVE-19416 for the talk and the change items. > Modify metastore to have/access persistent tables for stats > --- > > Key: HIVE-19417 > URL: https://issues.apache.org/jira/browse/HIVE-19417 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-19417.01.patch, HIVE-19417.02.patch, > HIVE-19417.03.patch, HIVE-19417.04.patch, HIVE-19417.05.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[ https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Yeom updated HIVE-19417: -- Attachment: HIVE-19417.05.patch > Modify metastore to have/access persistent tables for stats > --- > > Key: HIVE-19417 > URL: https://issues.apache.org/jira/browse/HIVE-19417 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-19417.01.patch, HIVE-19417.02.patch, > HIVE-19417.03.patch, HIVE-19417.04.patch, HIVE-19417.05.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19685) OpenTracing support for HMS
[ https://issues.apache.org/jira/browse/HIVE-19685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502374#comment-16502374 ] Todd Lipcon commented on HIVE-19685: Thanks Vihang, was OOO yesterday. > OpenTracing support for HMS > --- > > Key: HIVE-19685 > URL: https://issues.apache.org/jira/browse/HIVE-19685 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Attachments: HIVE-19685.02.patch, hive-19685.patch, hive-19685.patch, > trace.png > > > When diagnosing performance of metastore operations it isn't always obvious > why something took a long time. Using a tracing framework can provide an > end-to-end view of an operation including time spent in dependent systems (eg > filesystem operations, RDBMS queries, etc). This JIRA proposes to integrate > OpenTracing, which is a vendor-neutral tracing API into the HMS server and > client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19053) RemoteSparkJobStatus#getSparkJobInfo treats all exceptions as timeout errors
[
https://issues.apache.org/jira/browse/HIVE-19053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502363#comment-16502363
]
Sahil Takiar commented on HIVE-19053:
-
Makes sense, just a few comments:
* Can you remove the call to {{LOG#warn}} - its redundant, the exception thrown
will already get logged
* For the handling of the {{ExecutionException}} can you add in the underlying
exception message to the {{ErrorMsg}} message - e.g. add in "... due to
exception {0}" and use {{Throwables.getRootCause(e).getMessage()}} to extract
the underlying message cause
> RemoteSparkJobStatus#getSparkJobInfo treats all exceptions as timeout errors
>
>
> Key: HIVE-19053
> URL: https://issues.apache.org/jira/browse/HIVE-19053
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19053.1.patch
>
>
> {code}
> Future getJobInfo = sparkClient.run(
> new GetJobInfoJob(jobHandle.getClientJobId(), sparkJobId));
> try {
> return getJobInfo.get(sparkClientTimeoutInSeconds, TimeUnit.SECONDS);
> } catch (Exception e) {
> LOG.warn("Failed to get job info.", e);
> throw new HiveException(e, ErrorMsg.SPARK_GET_JOB_INFO_TIMEOUT,
> Long.toString(sparkClientTimeoutInSeconds));
> }
> {code}
> It should only throw {{ErrorMsg.SPARK_GET_JOB_INFO_TIMEOUT}} if a
> {{TimeoutException}} is thrown. Other exceptions should be handled
> independently.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19643) MM table conversion doesn't need full ACID structure checks
[ https://issues.apache.org/jira/browse/HIVE-19643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19643: Fix Version/s: 3.1.0 > MM table conversion doesn't need full ACID structure checks > --- > > Key: HIVE-19643 > URL: https://issues.apache.org/jira/browse/HIVE-19643 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Jason Dere >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19643.01.patch, HIVE-19643.02.patch, > HIVE-19643.03.patch, HIVE-19643.04.patch, HIVE-19643.05.patch, > HIVE-19643.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19258) add originals support to MM tables (and make the conversion a metadata only operation)
[ https://issues.apache.org/jira/browse/HIVE-19258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19258: Fix Version/s: 3.1.0 > add originals support to MM tables (and make the conversion a metadata only > operation) > -- > > Key: HIVE-19258 > URL: https://issues.apache.org/jira/browse/HIVE-19258 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19258.01.patch, HIVE-19258.02.patch, > HIVE-19258.03.patch, HIVE-19258.04.patch, HIVE-19258.05.patch, > HIVE-19258.06.patch, HIVE-19258.07.patch, HIVE-19258.08.patch, > HIVE-19258.08.patch, HIVE-19258.09.patch, HIVE-19258.10.patch, > HIVE-19258.11.patch, HIVE-19258.12.patch, HIVE-19258.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19476) Fix failures in TestReplicationScenariosAcidTables, TestReplicationOnHDFSEncryptedZones and TestCopyUtils
[ https://issues.apache.org/jira/browse/HIVE-19476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19476: Fix Version/s: (was: 3.1.0) > Fix failures in TestReplicationScenariosAcidTables, > TestReplicationOnHDFSEncryptedZones and TestCopyUtils > - > > Key: HIVE-19476 > URL: https://issues.apache.org/jira/browse/HIVE-19476 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: Tests, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-19476.01-branch-3.patch, HIVE-19476.01.patch, > HIVE-19476.02.patch > > > TestReplicationScenariosAcidTables broken due to HIVE-17657. > cc [~sershe] > TestReplicationOnHDFSEncryptedZones and TestCopyUtils are failing due to > auto-stats gathered which thrown exception irrelevant to theses tests. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17657) export/import for MM tables is broken
[ https://issues.apache.org/jira/browse/HIVE-17657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17657: Fix Version/s: 3.1.0 > export/import for MM tables is broken > - > > Key: HIVE-17657 > URL: https://issues.apache.org/jira/browse/HIVE-17657 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Sergey Shelukhin >Priority: Major > Labels: mm-gap-2 > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-17657.01.patch, HIVE-17657.02.patch, > HIVE-17657.03.patch, HIVE-17657.04.patch, HIVE-17657.05.patch, > HIVE-17657.06.patch, HIVE-17657.07.patch, HIVE-17657.08.patch, > HIVE-17657.09.patch, HIVE-17657.10.patch, HIVE-17657.patch > > > there is mm_exim.q but it's not clear from the tests what file structure it > creates > On import the txnids in the directory names would have to be remapped if > importing to a different cluster. Perhaps export can be smart and export > highest base_x and accretive deltas (minus aborted ones). Then import can > ...? It would have to remap txn ids from the archive to new txn ids. This > would then mean that import is made up of several transactions rather than 1 > atomic op. (all locks must belong to a transaction) > One possibility is to open a new txn for each dir in the archive (where > start/end txn of file name is the same) and commit all of them at once (need > new TMgr API for that). This assumes using a shared lock (if any!) and thus > allows other inserts (not related to import) to occur. > What if you have delta_6_9, such as a result of concatenate? If we stipulate > that this must mean that there is no delta_6_6 or any other "obsolete" delta > in the archive we can map it to a new single txn delta_x_x. > Add read_only mode for tables (useful in general, may be needed for upgrade > etc) and use that to make the above atomic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19720) backport multiple MM commits to branch-3
[ https://issues.apache.org/jira/browse/HIVE-19720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19720: Resolution: Fixed Status: Resolved (was: Patch Available) > backport multiple MM commits to branch-3 > > > Key: HIVE-19720 > URL: https://issues.apache.org/jira/browse/HIVE-19720 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19720.01-branch-3.patch, > HIVE-19720.02-branch-3.patch, HIVE-19720.03-branch-3.patch, > HIVE-19720.04-branch-3.patch, HIVE-19720.05-branch-3.patch > > > To avoid chained test runs of branch-3 backporting one by one, I will run > HiveQA on an epic combined patch, then commit patches w/proper commit > separation via cherry-pick: > 0930aec69b HIVE-19312 : MM tables don't work with BucketizedHIF (Sergey > Shelukhin, reviewed by Gunther Hagleitner) > 99a2b8bd6b HIVE-19312 : MM tables don't work with BucketizedHIF (Sergey > Shelukhin, reviewed by Gunther Hagleitner) ADDENDUM > 7ebcdeb951 HIVE-17657 : export/import for MM tables is broken (Sergey > Shelukhin, reviewed by Eugene Koifman) > 8db979f1ff (part not previously backported) HIVE-19476: Fix failures in > TestReplicationScenariosAcidTables, TestReplicationOnHDFSEncryptedZones and > TestCopyUtils (Sankar Hariappan, reviewed by Sergey Shelukhin) > f4352e5339 HIVE-19258 : add originals support to MM tables (and make the > conversion a metadata only operation) (Sergey Shelukhin, reviewed by Jason > Dere) > 36d66f0cf27 HIVE-19643 : MM table conversion doesn't need full ACID structure > checks (Sergey Shelukhin, reviewed by Eugene Koifman) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19312) MM tables don't work with BucketizedHIF
[ https://issues.apache.org/jira/browse/HIVE-19312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19312: Fix Version/s: 3.1.0 > MM tables don't work with BucketizedHIF > --- > > Key: HIVE-19312 > URL: https://issues.apache.org/jira/browse/HIVE-19312 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19312.01.patch, HIVE-19312.02.patch, > HIVE-19312.03.patch, HIVE-19312.04.patch, HIVE-19312.05.patch, > HIVE-19312.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19418) add background stats updater similar to compactor
[ https://issues.apache.org/jira/browse/HIVE-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19418: Attachment: HIVE-19418.07.patch > add background stats updater similar to compactor > - > > Key: HIVE-19418 > URL: https://issues.apache.org/jira/browse/HIVE-19418 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19418.01.patch, HIVE-19418.02.patch, > HIVE-19418.03.patch, HIVE-19418.04.patch, HIVE-19418.05.patch, > HIVE-19418.06.patch, HIVE-19418.06.patch, HIVE-19418.07.patch, > HIVE-19418.07.patch, HIVE-19418.patch > > > There's a JIRA HIVE-19416 to add snapshot version to stats for MM/ACID tables > to make them usable in a transaction without breaking ACID (for metadata-only > optimization). However, stats for ACID tables can still become unusable if > e.g. two parallel inserts run - neither sees the data written by the other, > so after both finish, the snapshots on either set of stats won't match the > current snapshot and the stats will be unusable. > Additionally, for ACID and non-ACID tables alike, a lot of the stats, with > some exceptions like numRows, cannot be aggregated (i.e. you cannot combine > ndvs from two inserts), and for ACID even less can be aggregated (you cannot > derive min/max if some rows are deleted but you don't scan the rest of the > dataset). > Therefore we will add background logic to metastore (similar to, and > partially inside, the ACID compactor) to update stats. > It will have 3 modes of operation. > 1) Off. > 2) Update only the stats that exist but are out of date (generating stats can > be expensive, so if the user is only analyzing a subset of tables it should > be able to only update that subset). We can simply look at existing stats and > only analyze for the relevant partitions and columns. > 3) On: 2 + create stats for all tables and columns missing stats. > There will also be a table parameter to skip stats update. > In phase 1, the process will operate outside of compactor, and run analyze > command on the table. The analyze command will automatically save the stats > with ACID snapshot information if needed, based on HIVE-19416, so we don't > need to do any special state management and this will work for all table > types. However it's also more expensive. > In phase 2, we can explore adding stats collection during MM compaction that > uses a temp table. If we don't have open writers during major compaction (so > we overwrite all of the data), the temp table stats can simply be copied over > to the main table with correct snapshot information, saving us a table scan. > In phase 3, we can add custom stats collection logic to full ACID compactor > that is not query based, the same way as we'd do for (2). Alternatively we > can wait for ACID compactor to become query based and just reuse (2). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19770) Support for CBO for queries with multiple same columns in select
[
https://issues.apache.org/jira/browse/HIVE-19770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502350#comment-16502350
]
Hive QA commented on HIVE-19770:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
26s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 2 new + 647 unchanged - 0
fixed = 649 total (was 647) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11531/dev-support/hive-personality.sh
|
| git revision | master / 98982e6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11531/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11531/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11531/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support for CBO for queries with multiple same columns in select
>
>
> Key: HIVE-19770
> URL: https://issues.apache.org/jira/browse/HIVE-19770
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19770.1.patch
>
>
> Currently queries such as {code:sql} select a,a from t1 where b > 10 {code}
> are not supported for CBO.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19769) Create dedicated objects for DB and Table names
[ https://issues.apache.org/jira/browse/HIVE-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502343#comment-16502343 ] Alan Gates commented on HIVE-19769: --- Unfortunately I need the objects to be in the storage-api, as they need to be used in ValidReaderWriteIdList. > Create dedicated objects for DB and Table names > --- > > Key: HIVE-19769 > URL: https://issues.apache.org/jira/browse/HIVE-19769 > Project: Hive > Issue Type: Sub-task > Components: storage-api >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > > Currently table names are always strings. Sometimes that string is just > tablename, sometimes it is dbname.tablename. Sometimes the code expects one > or the other, sometimes it handles either. This is burdensome for developers > and error prone. With the addition of catalog to the hierarchy, this becomes > even worse. > I propose to add two objects, DatabaseName and TableName. These will track > full names of each object. They will handle inserting default catalog and > database names when those are not provided. They will handle the conversions > to and from strings. > These will need to be added to storage-api because ValidTxnList will use it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18766) Race condition during shutdown of RemoteDriver, error messages aren't always sent
[
https://issues.apache.org/jira/browse/HIVE-18766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502337#comment-16502337
]
Sahil Takiar commented on HIVE-18766:
-
Few minor comments, overall LGTM:
* Line 138 of {{RemoteDriver}} is already creating a {{RpcConfiguration}}
object, so no need to create another one for {{getFutureTimeoutMs}}
* Instead of swallowing an exception from {{sendError}} can we just log it at
WARN level
> Race condition during shutdown of RemoteDriver, error messages aren't always
> sent
> -
>
> Key: HIVE-18766
> URL: https://issues.apache.org/jira/browse/HIVE-18766
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-18766.1.patch
>
>
> When we send an error during shutdown of the {{RemoteDriver}}, we don't wait
> for the error message to be sent. We just send it and then close the RPC
> channel. For a graceful shutdown, that doesn't seem ideal. We should at least
> wait a bit for the RPC message to be sent before shutting things down. I ran
> some tests locally and its pretty easy to hit a situation where the error
> message doesn't even get sent to the {{SparkClientImpl}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19763) Prevent execution of very large queries
[ https://issues.apache.org/jira/browse/HIVE-19763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502317#comment-16502317 ] Sergey Shelukhin commented on HIVE-19763: - Hive already has configs (strict checks) that disable certain kinda of obvious query patterns. Workload management (new in Hive 3/3.1) provides triggers that can kill queries on certain counters (time, split size, etc.) although it can only be used with Tez or LLAP. > Prevent execution of very large queries > --- > > Key: HIVE-19763 > URL: https://issues.apache.org/jira/browse/HIVE-19763 > Project: Hive > Issue Type: New Feature > Components: HiveServer2 >Affects Versions: 2.3.2 >Reporter: Luis E Martinez-Poblete >Priority: Minor > > Synopsis: > = > Prevent execution of very large queries. > > Feature Request: > > Please enhance Hive with a parameter to restrict the execution of very large > queries. > Use case: User is trying to create a view with a size of 8 MB. Creation of > this view was possible after increasing heap memory in several components > (HMS, HS2, Zookeeper). However, this view caused major issues when it was > used in a CTE query which resulted in GC pauses and eventually OOM of the HS2 > process. > > Although, it is possible to create the view, it may cause other issues when > used in queries. From the Hadoop administrator point of view, it would be > good to restrict this type of queries. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19769) Create dedicated objects for DB and Table names
[ https://issues.apache.org/jira/browse/HIVE-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502312#comment-16502312 ] Sergey Shelukhin commented on HIVE-19769: - For some new APIs, MetaStoreDirectSql already contains a fulltablename object. Its scope could be changed... > Create dedicated objects for DB and Table names > --- > > Key: HIVE-19769 > URL: https://issues.apache.org/jira/browse/HIVE-19769 > Project: Hive > Issue Type: Sub-task > Components: storage-api >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > > Currently table names are always strings. Sometimes that string is just > tablename, sometimes it is dbname.tablename. Sometimes the code expects one > or the other, sometimes it handles either. This is burdensome for developers > and error prone. With the addition of catalog to the hierarchy, this becomes > even worse. > I propose to add two objects, DatabaseName and TableName. These will track > full names of each object. They will handle inserting default catalog and > database names when those are not provided. They will handle the conversions > to and from strings. > These will need to be added to storage-api because ValidTxnList will use it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19771) allowNullColumnForMissingStats should not be false when column stats are estimated
[
https://issues.apache.org/jira/browse/HIVE-19771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502298#comment-16502298
]
Hive QA commented on HIVE-19771:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926192/HIVE-19771.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11530/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11530/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11530/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-06-05 18:50:45.390
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-11530/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-06-05 18:50:45.392
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 98982e6 HIVE-19079: Add extended query string to Spark job
description (Sahil Takiar, reviewed by Aihua Xu)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 98982e6 HIVE-19079: Add extended query string to Spark job
description (Sahil Takiar, reviewed by Aihua Xu)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-06-05 18:50:46.243
+ rm -rf ../yetus_PreCommit-HIVE-Build-11530
+ mkdir ../yetus_PreCommit-HIVE-Build-11530
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-11530
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-11530/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsWithStatsRule.java:
does not exist in index
error:
a/ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/rules/TestHiveReduceExpressionsWithStatsRule.java:
does not exist in index
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsWithStatsRule.java:295
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsWithStatsRule.java'
with conflicts.
Going to apply patch with: git apply -p1
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsWithStatsRule.java:295
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsWithStatsRule.java'
with conflicts.
U
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsWithStatsRule.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-11530
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926192 - PreCommit-HIVE-Build
> allowNullColumnForMissingStats should not be false when column stats are
> estimated
> --
>
> Key: HIVE-19771
> URL: https://issues.apache.org/jira/browse/HIVE-19771
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19771.patch
>
>
> Otherwise we may throw an Exception.
> {noformat}
> 2018-05-26T00:30:22,335 DEBUG [HiveServer2-Background-Pool:
[jira] [Commented] (HIVE-19418) add background stats updater similar to compactor
[
https://issues.apache.org/jira/browse/HIVE-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502295#comment-16502295
]
Hive QA commented on HIVE-19418:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926449/HIVE-19418.07.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 14475 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitions
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerVertexRawInputSplitsNoKill
(batchId=241)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11529/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11529/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11529/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926449 - PreCommit-HIVE-Build
> add background stats updater similar to compactor
> -
>
> Key: HIVE-19418
> URL: https://issues.apache.org/jira/browse/HIVE-19418
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19418.01.patch, HIVE-19418.02.patch,
> HIVE-19418.03.patch, HIVE-19418.04.patch, HIVE-19418.05.patch,
> HIVE-19418.06.patch, HIVE-19418.06.patch, HIVE-19418.07.patch,
> HIVE-19418.patch
>
>
> There's a JIRA HIVE-19416 to add snapshot version to stats for MM/ACID tables
> to make them usable in a transaction without breaking ACID (for metadata-only
> optimization). However, stats for ACID tables can still become unusable if
> e.g. two parallel inserts run - neither sees the data written by the other,
> so after both finish, the snapshots on either set of stats won't match the
> current snapshot and the stats will be unusable.
> Additionally, for ACID and non-ACID tables alike, a lot of the stats, with
> some exceptions like numRows, cannot be aggregated (i.e. you cannot combine
> ndvs from two inserts), and for ACID even less can be aggregated (you cannot
> derive min/max if some rows are deleted but you don't scan the rest of the
> dataset).
> Therefore we will add background logic to metastore (similar to, and
> partially inside, the ACID compactor) to update stats.
> It will have 3 modes of operation.
> 1) Off.
> 2) Update only the stats that exist but are out of date (generating stats can
> be expensive, so if the user is only analyzing a subset of tables it should
> be able to only update that subset). We can simply look at existing stats and
> only analyze for the relevant partitions and columns.
> 3) On: 2 + create stats for all tables and columns missing stats.
> There will also be a table parameter to skip stats update.
> In phase 1, the process will operate outside of compactor, and run analyze
> command on the table. The analyze command will automatically save the stats
> with ACID snapshot information if needed, based on HIVE-19416, so we don't
> need to do any special state management and this will work for all table
> types. However it's also more expensive.
> In phase 2, we can explore adding stats collection during MM compaction that
> uses a temp table. If we don't have open writers during major compaction (so
> we overwrite all of the data), the temp table stats can simply be copied over
> to the main table with correct snapshot information, saving us a table scan.
> In phase 3, we can add custom stats collection logic to full ACID compactor
> that is not query based, the same way as we'd do for (2). Alternatively we
> can wait for ACID compactor to become query based and just reuse (2).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19772) Streaming ingest V2 API can generate invalid orc file if interrupted
[
https://issues.apache.org/jira/browse/HIVE-19772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502272#comment-16502272
]
Gopal V commented on HIVE-19772:
Left 2 comments on RB, otherwise looks good +1 pending.
The code comments don't change the code flow functionally, just easier to
understand the actual intent.
> Streaming ingest V2 API can generate invalid orc file if interrupted
>
>
> Key: HIVE-19772
> URL: https://issues.apache.org/jira/browse/HIVE-19772
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0, 3.0.1, 4.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-19772.1.patch, HIVE-19772.2.patch
>
>
> Hive streaming ingest generated 0 length and 3 byte files which are invalid
> orc files. This will throw the following exception during compaction
> {code}
> Error: org.apache.orc.FileFormatException: Not a valid ORC file
> hdfs://cn105-10.l42scl.hortonworks.com:8020/apps/hive/warehouse/culvert/year=2018/month=7/delta_025_025/bucket_5
> (maxFileLength= 3) at
> org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:546) at
> org.apache.orc.impl.ReaderImpl.(ReaderImpl.java:370) at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:60) at
> org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:90) at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:1124)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:2373)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:1000)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:977)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:460) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:344) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19800) Handle rename files post HIVE-19751
[
https://issues.apache.org/jira/browse/HIVE-19800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-19800:
--
Status: Patch Available (was: Open)
patch 2 fixes
{noformat}
[ERROR] Failed to execute goal on project hive: Could not resolve dependencies
for project org.apache.hive:hive:pom:4.0.0-SNAPSHOT: Could not find artifact
org.apache.hive:hive-upgrade-acid:jar:4.0.0-SNAPSHOT in apache.snapshots
(https://repository.apache.org/snapshots) -> [Help 1] {noformat} from HIVE-19751
> Handle rename files post HIVE-19751
> ---
>
> Key: HIVE-19800
> URL: https://issues.apache.org/jira/browse/HIVE-19800
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19800.01.patch, HIVE-19800.02.patch
>
>
> this is a followup to HIVE-19751 which includes HIVE-19751 since it hasn't
> landed yet
> this includes file rename logic and HIVE-19750 since it hasn't landed yet
> either
>
> cc [~jdere]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19800) Handle rename files post HIVE-19751
[ https://issues.apache.org/jira/browse/HIVE-19800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-19800: -- Attachment: HIVE-19800.02.patch > Handle rename files post HIVE-19751 > --- > > Key: HIVE-19800 > URL: https://issues.apache.org/jira/browse/HIVE-19800 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-19800.01.patch, HIVE-19800.02.patch > > > this is a followup to HIVE-19751 which includes HIVE-19751 since it hasn't > landed yet > this includes file rename logic and HIVE-19750 since it hasn't landed yet > either > > cc [~jdere] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
