[
https://issues.apache.org/jira/browse/HADOOP-18081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sneha Vijayarajan reassigned HADOOP-18081:
------------------------------------------
Assignee: Sneha Vijayarajan
> FileNotFoundException in abfs mkdirs() call
> -------------------------------------------
>
> Key: HADOOP-18081
> URL: https://issues.apache.org/jira/browse/HADOOP-18081
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.3.1
> Reporter: Steve Loughran
> Assignee: Sneha Vijayarajan
> Priority: Major
>
> seen in production: calling mkdirs in FileOutputCommitter setupJob is
> triggering an FNFE
> {code}
> java.io.FileNotFoundException: Operation failed: "The specified path does
> not exist.", 404, PUT,
> https://bcket.dfs.core.windows.net/table1/_temporary/0?resource=directory&timeout=90,
> PathNotFound, "The specified path does not exist."
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.checkException(AzureBlobFileSystem.java:1131)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.mkdirs(AzureBlobFileSystem.java:445)
> at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2347)
> {code}
> I suspect what is happening is that while this job is setting up, a previous
> job is doing cleanup/abort on the same path
> assuming that abfs mkdirs is like the posix one -nonatomic, as it goes
> up/down the chain of parent dirs, something else gets in the way.
> if so, this is something which can be handled in the client -when we get an
> FNFE we could warn and retry.
> in the manifest committer each job will have a unique id under _temporary and
> there will be the option to skip deleting the temp dir entirely, for better
> coexistence of active jobs.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
