[
https://issues.apache.org/jira/browse/HIVE-7273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040392#comment-14040392
]
George Wong commented on HIVE-7273:
-----------------------------------
I went through the code of FileSink operator
The code is like this.
{code:java}
if ((bDynParts || isSkewedStoredAsSubDirectories)
&& !fs.exists(finalPaths[idx].getParent())) {
fs.mkdirs(finalPaths[idx].getParent());
}
{code}
I am wondering why we should check bDynParts and
isSkewedStoredAsSubDirectories. In the code, the output is move to finalPath no
matter what the values of bDynParts and isSkewedStoredAsSubDirectories are.
Since the date move is not avoidable, why not change the code to the following
to make sure the path exists before moving the file.
{code:java}
if (!fs.exists(finalPaths[idx].getParent())) {
fs.mkdirs(finalPaths[idx].getParent());
}
{code}
> Hive job fails due to unable to rename reducer output file
> ----------------------------------------------------------
>
> Key: HIVE-7273
> URL: https://issues.apache.org/jira/browse/HIVE-7273
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.12.0
> Reporter: George Wong
>
> We ran into this issue in our cluster.
> The error message is like this
> {noformat}
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output
> from:
> hdfs://***:8020/tmp/hive-svcckppi/hive_2014-06-16_20-24-09_584_6615934756634587679/_task_tmp.-ext-10002/_tmp.000000_3
> to:
> hdfs://***:8020/tmp/hive-svcckppi/hive_2014-06-16_20-24-09_584_6615934756634587679/_tmp.-ext-10002/000000_3
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:470)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:407)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153)
> {noformat}
> The log of NameNode shows
> {noformat}
> 2014-06-16 20:43:38,582 WARN org.apache.hadoop.hdfs.StateChange: DIR*
> FSDirectory.unprotectedRenameTo: failed to rename
> /tmp/hive-svcckppi/hive_2014-06-16_20-24-09_584_6615934756634587679/_task_tmp.-ext-10002/_tmp.000000_3
> to
> /tmp/hive-svcckppi/hive_2014-06-16_20-24-09_584_6615934756634587679/_tmp.-ext-10002/000000_3
> because destination's parent does not exist
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
