[
https://issues.apache.org/jira/browse/HADOOP-17990?focusedWorklogId=676985&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-676985
]
ASF GitHub Bot logged work on HADOOP-17990:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 05/Nov/21 11:37
Start Date: 05/Nov/21 11:37
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#3620:
URL: https://github.com/apache/hadoop/pull/3620#discussion_r743586220
##########
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##########
@@ -1411,6 +1405,30 @@ AzureBlobFileSystemStore getAbfsStore() {
return abfsStore;
}
+ @VisibleForTesting
+ void setAbfsStore(AzureBlobFileSystemStore abfsStore) {
+ this.abfsStore = abfsStore;
+ }
+
+ @VisibleForTesting
+ void createFileSystemIfNotExist(TracingContext tracingContext) throws
IOException {
+ if (this.tryGetFileStatus(new Path(AbfsHttpConstants.ROOT_PATH),
tracingContext) == null) {
+ try {
+ this.createFileSystem(tracingContext);
+ } catch (IOException ex) {
+ if (ex instanceof AzureBlobFileSystemException) {
+ checkException(null, (AzureBlobFileSystemException) ex,
+ AzureServiceErrorCode.FILE_SYSTEM_ALREADY_EXISTS);
+ } else if (ex.getCause() != null && ex.getCause() instanceof
AzureBlobFileSystemException) {
Review comment:
instanceof always returns false if source is null, so the null check
here can be removed
##########
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemInitAndCreate.java
##########
@@ -49,4 +56,39 @@ public void
ensureFilesystemWillNotBeCreatedIfCreationConfigIsNotSet() throws Ex
final AzureBlobFileSystem fs = this.createFileSystem();
FileStatus[] fileStatuses = fs.listStatus(new Path("/"));
}
+
+ @Test
+ public void ensureFilesystemWillBeCreatedIfCreationConfigIsSet() throws
Exception {
+ final AzureBlobFileSystem fs = createFileSystem();
Review comment:
use try with resources to close afterwards
##########
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemInitAndCreate.java
##########
@@ -49,4 +56,39 @@ public void
ensureFilesystemWillNotBeCreatedIfCreationConfigIsNotSet() throws Ex
final AzureBlobFileSystem fs = this.createFileSystem();
FileStatus[] fileStatuses = fs.listStatus(new Path("/"));
}
+
+ @Test
+ public void ensureFilesystemWillBeCreatedIfCreationConfigIsSet() throws
Exception {
+ final AzureBlobFileSystem fs = createFileSystem();
+ Configuration config = getRawConfiguration();
+ fs.initialize(FileSystem.getDefaultUri(config), config);
+
+ // Make sure createFileSystemIfNotExists is working as intended.
+ final MockAzureBlobFileSystemStore store = new
MockAzureBlobFileSystemStore(config);
+ fs.setAbfsStore(store);
+ fs.createFileSystemIfNotExist(getTestTracingContext(fs, true));
+ assert(store.isCreateFileSystemCalled);
Review comment:
assertj assertions with useful error text
##########
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##########
@@ -1411,6 +1405,30 @@ AzureBlobFileSystemStore getAbfsStore() {
return abfsStore;
}
+ @VisibleForTesting
+ void setAbfsStore(AzureBlobFileSystemStore abfsStore) {
+ this.abfsStore = abfsStore;
+ }
+
+ @VisibleForTesting
+ void createFileSystemIfNotExist(TracingContext tracingContext) throws
IOException {
+ if (this.tryGetFileStatus(new Path(AbfsHttpConstants.ROOT_PATH),
tracingContext) == null) {
+ try {
+ this.createFileSystem(tracingContext);
+ } catch (IOException ex) {
+ if (ex instanceof AzureBlobFileSystemException) {
Review comment:
better to add a catch(AzureBlobFileSystemException) clause
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 676985)
Time Spent: 1h 10m (was: 1h)
> Failing concurrent FS.initialize commands when
> fs.azure.createRemoteFileSystemDuringInitialization is enabled on
> hadoop-azure ABFS
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-17990
> URL: https://issues.apache.org/jira/browse/HADOOP-17990
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
> Affects Versions: 3.3.1
> Reporter: Zamil Majdy
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> *Bug description:*
> When {{fs.azure.createRemoteFileSystemDuringInitialization}} is enabled, the
> filesystem will create a container if it does not already exist inside the
> {{initialize}} method. The current flow of creating the container will fail
> in the case of concurrent {{initialize}} methods being executed
> simultaneously (only one request can create the container, the rest will fail
> instead of moving on). This is happen due to the `checkException` method that
> is not catching the Hadoop `FileAlreadyExists` exception.
> Stacktrace:
> {{Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: Operation
> failed: "The specified filesystem already exists.", 409, PUT,
> https://<REDACTED>.dfs.core.windows.net/project?resource=filesystem,
> FilesystemAlreadyExists, "The specified filesystem already exists.
> RequestId:<REDACTED> Time:2021-10-18T13:46:05.7504906Z"}}
> {{ {{at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.checkException(AzureBlobFileSystem.java:1182)}}}}
> {{ {{at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.createFileSystem(AzureBlobFileSystem.java:1067)}}}}
> {{ {{at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:126)}}}}
> {{ {{at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)}}}}
> *To reproduce:*
> * Set `fs.azure.createRemoteFileSystemDuringInitialization` to `true`
> * Run two concurrent `initialize` commands with the root to the non existing
> container/filesystem.
>
> *Proposed fix:*
> [https://github.com/apache/hadoop/pull/3620]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
