[jira] [Commented] (HIVE-18271) Druid Insert into fails with exception when committing files
[
https://issues.apache.org/jira/browse/HIVE-18271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290476#comment-16290476
]
Hive QA commented on HIVE-18271:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12901961/HIVE-18271.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 11529 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[quotedid_smb]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_10]
(batchId=138)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketsortoptimize_insert_7]
(batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=113)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=226)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8234/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8234/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8234/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12901961 - PreCommit-HIVE-Build
> Druid Insert into fails with exception when committing files
>
>
> Key: HIVE-18271
> URL: https://issues.apache.org/jira/browse/HIVE-18271
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
> Fix For: 3.0.0
>
> Attachments: HIVE-18271.2.patch, HIVE-18271.patch
>
>
> Exception -
> {code}
> 03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty to:
> hdfs://ctr-e136-1513029738776-2163-01-03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty.moved)'
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move:
> hdfs://ctr-e136-1513029738776-2163-01-03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty
> to:
> hdfs://ctr-e136-1513029738776-2163-01-03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty.moved
> at org.apache.hadoop.hive.ql.exec.Utilities.rename(Utilities.java:1129)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.mvFileToFinalPath(Utilities.java:1460)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.jobCloseOp(FileSinkOperator.java:1135)
> at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:765)
> at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:770)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.close(TezTask.java:588)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:286)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1414)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1211)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1204)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:242)
> at
>
[jira] [Commented] (HIVE-18271) Druid Insert into fails with exception when committing files
[
https://issues.apache.org/jira/browse/HIVE-18271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290417#comment-16290417
]
Hive QA commented on HIVE-18271:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 8ab523b |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8234/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Druid Insert into fails with exception when committing files
>
>
> Key: HIVE-18271
> URL: https://issues.apache.org/jira/browse/HIVE-18271
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
> Fix For: 3.0.0
>
> Attachments: HIVE-18271.2.patch, HIVE-18271.patch
>
>
> Exception -
> {code}
> 03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty to:
> hdfs://ctr-e136-1513029738776-2163-01-03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty.moved)'
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move:
> hdfs://ctr-e136-1513029738776-2163-01-03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty
> to:
> hdfs://ctr-e136-1513029738776-2163-01-03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty.moved
> at org.apache.hadoop.hive.ql.exec.Utilities.rename(Utilities.java:1129)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.mvFileToFinalPath(Utilities.java:1460)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.jobCloseOp(FileSinkOperator.java:1135)
> at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:765)
> at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:770)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.close(TezTask.java:588)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:286)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667)
> at
[jira] [Commented] (HIVE-18271) Druid Insert into fails with exception when committing files
[
https://issues.apache.org/jira/browse/HIVE-18271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290136#comment-16290136
]
Ashutosh Chauhan commented on HIVE-18271:
-
+1 pending tests
> Druid Insert into fails with exception when committing files
>
>
> Key: HIVE-18271
> URL: https://issues.apache.org/jira/browse/HIVE-18271
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
> Fix For: 3.0.0
>
> Attachments: HIVE-18271.2.patch, HIVE-18271.patch
>
>
> Exception -
> {code}
> 03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty to:
> hdfs://ctr-e136-1513029738776-2163-01-03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty.moved)'
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move:
> hdfs://ctr-e136-1513029738776-2163-01-03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty
> to:
> hdfs://ctr-e136-1513029738776-2163-01-03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty.moved
> at org.apache.hadoop.hive.ql.exec.Utilities.rename(Utilities.java:1129)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.mvFileToFinalPath(Utilities.java:1460)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.jobCloseOp(FileSinkOperator.java:1135)
> at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:765)
> at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:770)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.close(TezTask.java:588)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:286)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1414)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1211)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1204)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:242)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:350)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18271) Druid Insert into fails with exception when committing files
[
https://issues.apache.org/jira/browse/HIVE-18271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290096#comment-16290096
]
Hive QA commented on HIVE-18271:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12901885/HIVE-18271.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 11529 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[quotedid_smb]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_10]
(batchId=138)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketsortoptimize_insert_7]
(batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=113)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=226)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8225/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8225/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8225/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12901885 - PreCommit-HIVE-Build
> Druid Insert into fails with exception when committing files
>
>
> Key: HIVE-18271
> URL: https://issues.apache.org/jira/browse/HIVE-18271
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
> Fix For: 3.0.0
>
> Attachments: HIVE-18271.patch
>
>
> Exception -
> {code}
> 03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty to:
> hdfs://ctr-e136-1513029738776-2163-01-03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty.moved)'
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move:
> hdfs://ctr-e136-1513029738776-2163-01-03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty
> to:
> hdfs://ctr-e136-1513029738776-2163-01-03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty.moved
> at org.apache.hadoop.hive.ql.exec.Utilities.rename(Utilities.java:1129)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.mvFileToFinalPath(Utilities.java:1460)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.jobCloseOp(FileSinkOperator.java:1135)
> at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:765)
> at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:770)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.close(TezTask.java:588)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:286)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1414)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1211)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1204)
> at
>
[jira] [Commented] (HIVE-18271) Druid Insert into fails with exception when committing files
[
https://issues.apache.org/jira/browse/HIVE-18271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290076#comment-16290076
]
Jason Dere commented on HIVE-18271:
---
Dug into some of the details with [~ashutoshc] and [~bslim]. The
FileSinkOperator should not even be trying to call Utilities.mvToFinalPath(),
because it is a non-native table - there is actually logic in place for that.
The problem is that FileSinkOperator.isNativeTable is dependent on being set
during initializeOp() .. and it appears that initializeOp() is never being
called for the operators in the TezTask (client side)! Making a patch to fix
the immediate problem, by making isNativeTable into a method, that does not
depend on initializeOp().
We may want to look into whether the operators within TezTask should have
initialize() called on them as a future item.
> Druid Insert into fails with exception when committing files
>
>
> Key: HIVE-18271
> URL: https://issues.apache.org/jira/browse/HIVE-18271
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
> Fix For: 3.0.0
>
> Attachments: HIVE-18271.patch
>
>
> Exception -
> {code}
> 03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty to:
> hdfs://ctr-e136-1513029738776-2163-01-03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty.moved)'
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move:
> hdfs://ctr-e136-1513029738776-2163-01-03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty
> to:
> hdfs://ctr-e136-1513029738776-2163-01-03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty.moved
> at org.apache.hadoop.hive.ql.exec.Utilities.rename(Utilities.java:1129)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.mvFileToFinalPath(Utilities.java:1460)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.jobCloseOp(FileSinkOperator.java:1135)
> at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:765)
> at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:770)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.close(TezTask.java:588)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:286)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1414)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1211)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1204)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:242)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:350)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18271) Druid Insert into fails with exception when committing files
[
https://issues.apache.org/jira/browse/HIVE-18271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290052#comment-16290052
]
Hive QA commented on HIVE-18271:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
35s{color} | {color:red} ql: The patch generated 1 new + 184 unchanged - 0
fixed = 185 total (was 184) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 8ab523b |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8225/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8225/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Druid Insert into fails with exception when committing files
>
>
> Key: HIVE-18271
> URL: https://issues.apache.org/jira/browse/HIVE-18271
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
> Fix For: 3.0.0
>
> Attachments: HIVE-18271.patch
>
>
> Exception -
> {code}
> 03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty to:
> hdfs://ctr-e136-1513029738776-2163-01-03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty.moved)'
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move:
> hdfs://ctr-e136-1513029738776-2163-01-03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty
> to:
> hdfs://ctr-e136-1513029738776-2163-01-03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty.moved
> at org.apache.hadoop.hive.ql.exec.Utilities.rename(Utilities.java:1129)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.mvFileToFinalPath(Utilities.java:1460)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.jobCloseOp(FileSinkOperator.java:1135)
> at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:765)
> at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:770)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.close(TezTask.java:588)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:286)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at
[jira] [Commented] (HIVE-18271) Druid Insert into fails with exception when committing files
[
https://issues.apache.org/jira/browse/HIVE-18271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289799#comment-16289799
]
Jason Dere commented on HIVE-18271:
---
Agree with the patch .. the logic should ensure the new tmpPath is a unique FS
path, and since Utilities.mvFileToFinalPath() has renamed the original tmpPath,
then anything that tried to delete the original path later would not work
because it has been renamed to tmpPath.moved. So there might need to be cleanup
of tmpPath if we are renaming it here.
One thing - can you move the call to fs.delete(tmpPath, true) down one line, to
just after the if/else statement? That way tmpPath gets cleaned up for both
BlobStore/non-BlobStore FS cases. And maybe add a comment to the fs.delete()
call - reference this use case or Jira.
> Druid Insert into fails with exception when committing files
>
>
> Key: HIVE-18271
> URL: https://issues.apache.org/jira/browse/HIVE-18271
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
> Fix For: 3.0.0
>
> Attachments: HIVE-18271.patch
>
>
> Exception -
> {code}
> 03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty to:
> hdfs://ctr-e136-1513029738776-2163-01-03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty.moved)'
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move:
> hdfs://ctr-e136-1513029738776-2163-01-03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty
> to:
> hdfs://ctr-e136-1513029738776-2163-01-03.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty.moved
> at org.apache.hadoop.hive.ql.exec.Utilities.rename(Utilities.java:1129)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.mvFileToFinalPath(Utilities.java:1460)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.jobCloseOp(FileSinkOperator.java:1135)
> at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:765)
> at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:770)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.close(TezTask.java:588)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:286)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1414)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1211)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1204)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:242)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:350)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
