[
https://issues.apache.org/jira/browse/HADOOP-19284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17884875#comment-17884875
]
ASF GitHub Bot commented on HADOOP-19284:
-----------------------------------------
surendralilhore commented on code in PR #7062:
URL: https://github.com/apache/hadoop/pull/7062#discussion_r1776440406
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemE2E.java:
##########
@@ -259,6 +260,9 @@ public void testHttpReadTimeout() throws Exception {
public void testHttpTimeouts(int connectionTimeoutMs, int readTimeoutMs)
throws Exception {
+ // This is to make sure File System creation goes through before network
calls start failing.
+ assumeValidTestConfigPresent(this.getRawConfiguration(),
FS_AZURE_ACCOUNT_IS_HNS_ENABLED);
Review Comment:
Why this change is required?, is this failing without this change ?
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##########
@@ -451,7 +451,9 @@ public AbfsConfiguration(final Configuration rawConfig,
String accountName)
}
public Trilean getIsNamespaceEnabledAccount() {
- return Trilean.getTrilean(isNamespaceEnabledAccount);
+ String isNamespaceEnabledAccountString
Review Comment:
Can you change the variable name to something simpler, like
isNamespaceEnabled?
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestGetNameSpaceEnabled.java:
##########
@@ -271,4 +275,52 @@ private void
ensureGetAclDetermineHnsStatusAccuratelyInternal(int statusCode,
Mockito.verify(mockClient, times(1))
.getAclStatus(anyString(), any(TracingContext.class));
}
+
+ @Test
+ public void testAccountSpecificConfig() throws Exception {
+ Configuration rawConfig = new Configuration();
+ rawConfig.addResource(TEST_CONFIGURATION_FILE_NAME);
+ rawConfig.unset(FS_AZURE_ACCOUNT_IS_HNS_ENABLED);
+ rawConfig.unset(accountProperty(FS_AZURE_ACCOUNT_IS_HNS_ENABLED,
+ this.getAccountName()));
+ String accountName1 = "account1.dfs.core.windows.net";
+ String accountName2 = "account2.dfs.core.windows.net";
+ String accountName3 = "account3.dfs.core.windows.net";
+ String defaultUri1 = this.getTestUrl().replace(this.getAccountName(),
accountName1);
+ String defaultUri2 = this.getTestUrl().replace(this.getAccountName(),
accountName2);
+ String defaultUri3 = this.getTestUrl().replace(this.getAccountName(),
accountName3);
+
+ // Set both account specific and account agnostic config for account 1
+ rawConfig.set(accountProperty(FS_AZURE_ACCOUNT_IS_HNS_ENABLED,
accountName1), FALSE_STR);
+ rawConfig.set(FS_AZURE_ACCOUNT_IS_HNS_ENABLED, TRUE_STR);
+ rawConfig.set(CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY,
defaultUri1);
+ AzureBlobFileSystem fs1 = (AzureBlobFileSystem)
FileSystem.newInstance(rawConfig);
+ // Assert that account specific config takes precedence
+ Assertions.assertThat(getIsNamespaceEnabled(fs1)).describedAs(
+ "getIsNamespaceEnabled should return true when the "
+ + "account specific config is set as true").isFalse();
+
+ // Set only the account specific config for account 2
+ rawConfig.set(accountProperty(FS_AZURE_ACCOUNT_IS_HNS_ENABLED,
accountName2), FALSE_STR);
+ rawConfig.unset(FS_AZURE_ACCOUNT_IS_HNS_ENABLED);
+ rawConfig.set(CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY,
defaultUri2);
+ AzureBlobFileSystem fs2 = (AzureBlobFileSystem)
FileSystem.newInstance(rawConfig);
+ // Assert that account specific config is enough.
+ Assertions.assertThat(getIsNamespaceEnabled(fs2)).describedAs(
+ "getIsNamespaceEnabled should return true when the "
+ + "account specific config is set as true").isFalse();
+
+ // Set only account agnostic config for account 3
+ rawConfig.set(FS_AZURE_ACCOUNT_IS_HNS_ENABLED, FALSE_STR);
+ rawConfig.unset(accountProperty(FS_AZURE_ACCOUNT_IS_HNS_ENABLED,
accountName3));
+ rawConfig.set(CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY,
defaultUri3);
+ AzureBlobFileSystem fs3 = (AzureBlobFileSystem)
FileSystem.newInstance(rawConfig);
+ // Assert that account agnostic config is enough.
+ Assertions.assertThat(getIsNamespaceEnabled(fs3)).describedAs(
+ "getIsNamespaceEnabled should return true when the "
+ + "account specific config is not set").isFalse();
+ fs1.close();
+ fs2.close();
+ fs3.close();
Review Comment:
can you check one more condition where account level and common property is
not set? So it will call the *getAclStatus()* to check the namespace enabled or
not.
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestGetNameSpaceEnabled.java:
##########
@@ -271,4 +275,52 @@ private void
ensureGetAclDetermineHnsStatusAccuratelyInternal(int statusCode,
Mockito.verify(mockClient, times(1))
.getAclStatus(anyString(), any(TracingContext.class));
}
+
+ @Test
+ public void testAccountSpecificConfig() throws Exception {
+ Configuration rawConfig = new Configuration();
+ rawConfig.addResource(TEST_CONFIGURATION_FILE_NAME);
+ rawConfig.unset(FS_AZURE_ACCOUNT_IS_HNS_ENABLED);
+ rawConfig.unset(accountProperty(FS_AZURE_ACCOUNT_IS_HNS_ENABLED,
+ this.getAccountName()));
+ String accountName1 = "account1.dfs.core.windows.net";
+ String accountName2 = "account2.dfs.core.windows.net";
+ String accountName3 = "account3.dfs.core.windows.net";
+ String defaultUri1 = this.getTestUrl().replace(this.getAccountName(),
accountName1);
+ String defaultUri2 = this.getTestUrl().replace(this.getAccountName(),
accountName2);
+ String defaultUri3 = this.getTestUrl().replace(this.getAccountName(),
accountName3);
+
+ // Set both account specific and account agnostic config for account 1
+ rawConfig.set(accountProperty(FS_AZURE_ACCOUNT_IS_HNS_ENABLED,
accountName1), FALSE_STR);
+ rawConfig.set(FS_AZURE_ACCOUNT_IS_HNS_ENABLED, TRUE_STR);
+ rawConfig.set(CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY,
defaultUri1);
+ AzureBlobFileSystem fs1 = (AzureBlobFileSystem)
FileSystem.newInstance(rawConfig);
Review Comment:
Can you use try-with-resources, so you don’t need to explicitly close the
filesystem object?
> ABFS: Allow "fs.azure.account.hns.enabled" to be set as Account Specific
> Config
> -------------------------------------------------------------------------------
>
> Key: HADOOP-19284
> URL: https://issues.apache.org/jira/browse/HADOOP-19284
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.4.0, 3.5.0
> Reporter: Anuj Modi
> Assignee: Anuj Modi
> Priority: Major
> Labels: pull-request-available
>
> There are a few reported requirements where users working with multiple file
> systems need to specify this config either only for some accounts or set it
> differently for different account.
> ABFS driver today does not allow this to be set as account specific config.
> This Jira is to allow that as a new support.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
