[
https://issues.apache.org/jira/browse/HIVE-18271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289799#comment-16289799
]
Jason Dere commented on HIVE-18271:
-----------------------------------
Agree with the patch .. the logic should ensure the new tmpPath is a unique FS
path, and since Utilities.mvFileToFinalPath() has renamed the original tmpPath,
then anything that tried to delete the original path later would not work
because it has been renamed to tmpPath.moved. So there might need to be cleanup
of tmpPath if we are renaming it here.
One thing - can you move the call to fs.delete(tmpPath, true) down one line, to
just after the if/else statement? That way tmpPath gets cleaned up for both
BlobStore/non-BlobStore FS cases. And maybe add a comment to the fs.delete()
call - reference this use case or Jira.
> Druid Insert into fails with exception when committing files
> ------------------------------------------------------------
>
> Key: HIVE-18271
> URL: https://issues.apache.org/jira/browse/HIVE-18271
> Project: Hive
> Issue Type: Bug
> Reporter: Nishant Bangarwa
> Assignee: Nishant Bangarwa
> Fix For: 3.0.0
>
> Attachments: HIVE-18271.patch
>
>
> Exception -
> {code}
> 000003.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty to:
> hdfs://ctr-e136-1513029738776-2163-01-000003.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty.moved)'
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move:
> hdfs://ctr-e136-1513029738776-2163-01-000003.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty
> to:
> hdfs://ctr-e136-1513029738776-2163-01-000003.hwx.site:8020/apps/hive/warehouse/_tmp.all100k_druid_initial_empty.moved
> at org.apache.hadoop.hive.ql.exec.Utilities.rename(Utilities.java:1129)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.mvFileToFinalPath(Utilities.java:1460)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.jobCloseOp(FileSinkOperator.java:1135)
> at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:765)
> at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:770)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.close(TezTask.java:588)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:286)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1414)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1211)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1204)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:242)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:350)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
