[
https://issues.apache.org/jira/browse/HDFS-12688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212035#comment-16212035
]
Shriya Gupta commented on HDFS-12688:
-------------------------------------
I suspected that same asynchronous behaviour when this issue was brought to my
attention by a user. However, at no point in time am I running two copies of
the job at the same time. Each time, I have launched my script only after the
previous execution would end.
I suggested running it multiple times only because the error occurs randomly --
it has also shown up at the very first run of the script sometimes.
> HDFS File Not Removed Despite Successful "Moved to .Trash" Message
> ------------------------------------------------------------------
>
> Key: HDFS-12688
> URL: https://issues.apache.org/jira/browse/HDFS-12688
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
> Affects Versions: 2.6.0
> Reporter: Shriya Gupta
> Priority: Critical
>
> Wrote a simple script to delete and create a file and ran it multiple times.
> However, some executions of the script randomly threw a FileAlreadyExists
> error while the others succeeded despite successful hdfs dfs -rm command. The
> script is as below, I have reproduced it in two different environments --
> hdfs dfs -ls /user/shriya/shell_test/
> echo "starting hdfs remove **************"
> hdfs dfs -rm -r -f /user/shriya/shell_test/wordcountOutput
> echo "hdfs compeleted!"
> hdfs dfs -ls /user/shriya/shell_test/
> echo "starting mapReduce*******************************"
> mapred job -libjars
> /data/home/shriya/shell_test/hadoop-mapreduce-client-jobclient-2.7.1.jar
> -submit /data/home/shriya/shell_test/wordcountJob.xml
> The message confirming successful move --
> 17/10/19 14:49:12 INFO fs.TrashPolicyDefault: Moved:
> 'hdfs://nameservice1/user/shriya/shell_test/wordcountOutput' to trash at:
> hdfs://nameservice1/user/shriya/.Trash/Current/user/shriya/shell_test/wordcountOutput1508438952728
> The contents of subsequent -ls after -rm also showed that the file still
> existed)
> The error I got when my MapReduce job tried to create the file --
> 17/10/19 14:50:00 WARN security.UserGroupInformation:
> PriviledgedActionException as:<REDACTED> (auth:KERBEROS)
> cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory
> hdfs://nameservice1/user/shriya/shell_test/wordcountOutput already exists
> Exception in thread "main"
> org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory
> hdfs://nameservice1/user/shriya/shell_test/wordcountOutput already exists
> at
> org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:131)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:272)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:143)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304)
> at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:315)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1277)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
