[
https://issues.apache.org/jira/browse/HADOOP-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zamil Majdy updated HADOOP-17990:
---------------------------------
Description:
*Bug description:*
When {{fs.azure.createRemoteFileSystemDuringInitialization}} is enabled, the
filesystem will create a container if it does not already exist inside the
{{initialize}} method. The current flow of creating the container will fail in
the case of concurrent {{initialize}} methods being executed simultaneously
(only one request can create the container, the rest will fail instead of
moving on). This is happen due to the `checkException` method that is not
catching the Hadoop `FileAlreadyExists` exception.
Stacktrace:
{{Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: Operation failed:
"The specified filesystem already exists.", 409, PUT,
https://<REDACTED>.dfs.core.windows.net/project?resource=filesystem,
FilesystemAlreadyExists, "The specified filesystem already exists.
RequestId:<REDACTED> Time:2021-10-18T13:46:05.7504906Z" }}
{{at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.checkException(AzureBlobFileSystem.java:1182)
}}
{{at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.createFileSystem(AzureBlobFileSystem.java:1067)
}}
{{at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:126)
}}
{{at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669) }}
*To reproduce:*
* Set `fs.azure.createRemoteFileSystemDuringInitialization` to `true`
* Run two concurrent `initialize` commands with the root to the non existing
container/filesystem.
was:
*Bug description:*
`fs.mkdirs` command for `RawLocalFileSystem` doesn't work in Hadoop 3 with
NativeIO enabled.
The failure was happening when doing the native `chmod` command to the file
(the `mkdir` command itself is working).
Stacktrace:
{{ENOENT: No such file or directory ENOENT: No such file or directory at
org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method) at
org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:382) at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:974)
at
org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:660)
at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:700)
at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:672)}}
*To reproduce:*
* Add `fs.mkdirs` in RawLocalFileSystem with NativeIO enabled.
* Sample: [https://github.com/apache/hadoop/pull/3391]
> Failing concurrent FS.initialize commands when
> fs.azure.createRemoteFileSystemDuringInitialization is enabled on
> hadoop-azure ABFS
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-17990
> URL: https://issues.apache.org/jira/browse/HADOOP-17990
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, fs
> Affects Versions: 3.3.1
> Reporter: Zamil Majdy
> Priority: Major
> Labels: pull-request-available
>
> *Bug description:*
> When {{fs.azure.createRemoteFileSystemDuringInitialization}} is enabled, the
> filesystem will create a container if it does not already exist inside the
> {{initialize}} method. The current flow of creating the container will fail
> in the case of concurrent {{initialize}} methods being executed
> simultaneously (only one request can create the container, the rest will fail
> instead of moving on). This is happen due to the `checkException` method that
> is not catching the Hadoop `FileAlreadyExists` exception.
> Stacktrace:
> {{Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: Operation
> failed: "The specified filesystem already exists.", 409, PUT,
> https://<REDACTED>.dfs.core.windows.net/project?resource=filesystem,
> FilesystemAlreadyExists, "The specified filesystem already exists.
> RequestId:<REDACTED> Time:2021-10-18T13:46:05.7504906Z" }}
> {{at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.checkException(AzureBlobFileSystem.java:1182)
> }}
> {{at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.createFileSystem(AzureBlobFileSystem.java:1067)
> }}
> {{at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:126)
> }}
> {{at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669) }}
> *To reproduce:*
> * Set `fs.azure.createRemoteFileSystemDuringInitialization` to `true`
> * Run two concurrent `initialize` commands with the root to the non existing
> container/filesystem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
