[
https://issues.apache.org/jira/browse/HADOOP-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zamil Majdy updated HADOOP-17990:
---------------------------------
Description:
*Bug description:*
When {{fs.azure.createRemoteFileSystemDuringInitialization}} is enabled, the
filesystem will create a container if it does not already exist inside the
{{initialize}} method. The current flow of creating the container will fail in
the case of concurrent {{initialize}} methods being executed simultaneously
(only one request can create the container, the rest will fail instead of
moving on). This is happen due to the `checkException` method that is not
catching the Hadoop `FileAlreadyExists` exception.
Stacktrace:
{{Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: Operation failed:
"The specified filesystem already exists.", 409, PUT,
https://<REDACTED>.dfs.core.windows.net/project?resource=filesystem,
FilesystemAlreadyExists, "The specified filesystem already exists.
RequestId:<REDACTED> Time:2021-10-18T13:46:05.7504906Z"}}
{{ {{at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.checkException(AzureBlobFileSystem.java:1182)}}}}
{{ {{at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.createFileSystem(AzureBlobFileSystem.java:1067)}}}}
{{ {{at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:126)}}}}
{{ {{at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)}}}}
*To reproduce:*
* Set `fs.azure.createRemoteFileSystemDuringInitialization` to `true`
* Run two concurrent `initialize` commands with the root to the non existing
container/filesystem.
was:
*Bug description:*
When {{fs.azure.createRemoteFileSystemDuringInitialization}} is enabled, the
filesystem will create a container if it does not already exist inside the
{{initialize}} method. The current flow of creating the container will fail in
the case of concurrent {{initialize}} methods being executed simultaneously
(only one request can create the container, the rest will fail instead of
moving on). This is happen due to the `checkException` method that is not
catching the Hadoop `FileAlreadyExists` exception.
Stacktrace:
{{Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: Operation failed:
"The specified filesystem already exists.", 409, PUT,
https://<REDACTED>.dfs.core.windows.net/project?resource=filesystem,
FilesystemAlreadyExists, "The specified filesystem already exists.
RequestId:<REDACTED> Time:2021-10-18T13:46:05.7504906Z" }}}}
{{at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.checkException(AzureBlobFileSystem.java:1182)}}
{{at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.createFileSystem(AzureBlobFileSystem.java:1067)}}
{{at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:126)}}
{{at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)}}
*To reproduce:*
* Set `fs.azure.createRemoteFileSystemDuringInitialization` to `true`
* Run two concurrent `initialize` commands with the root to the non existing
container/filesystem.
> Failing concurrent FS.initialize commands when
> fs.azure.createRemoteFileSystemDuringInitialization is enabled on
> hadoop-azure ABFS
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-17990
> URL: https://issues.apache.org/jira/browse/HADOOP-17990
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, fs
> Affects Versions: 3.3.1
> Reporter: Zamil Majdy
> Priority: Major
> Labels: pull-request-available
>
> *Bug description:*
> When {{fs.azure.createRemoteFileSystemDuringInitialization}} is enabled, the
> filesystem will create a container if it does not already exist inside the
> {{initialize}} method. The current flow of creating the container will fail
> in the case of concurrent {{initialize}} methods being executed
> simultaneously (only one request can create the container, the rest will fail
> instead of moving on). This is happen due to the `checkException` method that
> is not catching the Hadoop `FileAlreadyExists` exception.
> Stacktrace:
> {{Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: Operation
> failed: "The specified filesystem already exists.", 409, PUT,
> https://<REDACTED>.dfs.core.windows.net/project?resource=filesystem,
> FilesystemAlreadyExists, "The specified filesystem already exists.
> RequestId:<REDACTED> Time:2021-10-18T13:46:05.7504906Z"}}
> {{ {{at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.checkException(AzureBlobFileSystem.java:1182)}}}}
> {{ {{at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.createFileSystem(AzureBlobFileSystem.java:1067)}}}}
> {{ {{at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:126)}}}}
> {{ {{at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)}}}}
> *To reproduce:*
> * Set `fs.azure.createRemoteFileSystemDuringInitialization` to `true`
> * Run two concurrent `initialize` commands with the root to the non existing
> container/filesystem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
