[
https://issues.apache.org/jira/browse/HADOOP-18542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17640818#comment-17640818
]
ASF GitHub Bot commented on HADOOP-18542:
-----------------------------------------
CLevasseur commented on PR #4262:
URL: https://github.com/apache/hadoop/pull/4262#issuecomment-1330911612
The exception for
`ITestAzureBlobFileSystemCheckAccess.testCheckAccessForAccountWithoutNS` is:
```
[INFO] Running
org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess
[ERROR] Tests run: 12, Failures: 1, Errors: 0, Skipped: 10, Time elapsed:
1.521 s <<< FAILURE! - in
org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess
[ERROR]
testCheckAccessForAccountWithoutNS(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess)
Time elapsed: 1.15 s <<< FAILURE!
java.lang.AssertionError: Expecting
org.apache.hadoop.security.AccessControlException with text "This request is
not authorized to perform this operation using this permission.", 403 but got :
"void"
at
org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499)
at
org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:529)
at
org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess.testCheckAccessForAccountWithoutNS(ITestAzureBlobFileSystemCheckAccess.java:181)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:829)
```
> are you working with a hierarchical store here? as i think those tests are
failures from that detail, not your change (essentially: etags aren't being
preserved across rename).
No, HNS is disabled on Azure's side, and in the account settings as well:
````
cat ./src/test/resources/accountSettings/${ACCOUNT_NAME}_settings.xml | grep
-B1 -A2 namespace
<property>
<name>fs.azure.test.namespace.enabled.****.dfs.core.windows.net</name>
<value>false</value>
</property>
<property>
<name>fs.azure.test.namespace.enabled</name>
<value>false</value>
</property>
```
> do the same test suite against trunk and see if the tests still fail, if
so, create a new jira (or maybe there is one already). it shouldn't block this
pr
The result is the same, here are the logs I'm getting:
```
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsHttpOperation
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsClient
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsInputStream
[INFO] Running
org.apache.hadoop.fs.azurebfs.services.TestAzureADAuthenticator
[INFO] Running
org.apache.hadoop.fs.azurebfs.services.TestShellDecryptionKeyProvider
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsOutputStream
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAccountConfiguration
[INFO] Running
org.apache.hadoop.fs.azurebfs.services.TestAbfsClientThrottlingAnalyzer
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.976
s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsHttpOperation
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.174
s - in org.apache.hadoop.fs.azurebfs.services.TestShellDecryptionKeyProvider
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsPerfTracker
[ERROR] Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
2.921 s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.TestAccountConfiguration
[ERROR]
testConfigPropNotFound(org.apache.hadoop.fs.azurebfs.TestAccountConfiguration)
Time elapsed: 0.102 s <<< FAILURE!
java.lang.AssertionError: Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
at
org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499)
at
org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384)
at
org.apache.hadoop.fs.azurebfs.TestAccountConfiguration.testMissingConfigKey(TestAccountConfiguration.java:399)
at
org.apache.hadoop.fs.azurebfs.TestAccountConfiguration.testConfigPropNotFound(TestAccountConfiguration.java:386)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.396
s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsClient
[INFO] Running
org.apache.hadoop.fs.azurebfs.services.TestTextFileBasedIdentityHandler
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.043
s - in org.apache.hadoop.fs.azurebfs.services.TestTextFileBasedIdentityHandler
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestQueryParams
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.2 s
- in org.apache.hadoop.fs.azurebfs.services.TestQueryParams
[INFO] Running
org.apache.hadoop.fs.azurebfs.services.TestAbfsRenameRetryRecovery
[INFO] Running
org.apache.hadoop.fs.azurebfs.services.TestExponentialRetryPolicy
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.088
s - in org.apache.hadoop.fs.azurebfs.services.TestAzureADAuthenticator
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsInputStreamStatistics
[INFO] Running org.apache.hadoop.fs.azurebfs.utils.TestCachedSASToken
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.651
s - in org.apache.hadoop.fs.azurebfs.utils.TestCachedSASToken
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
8.133 s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsPerfTracker
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.278
s - in org.apache.hadoop.fs.azurebfs.services.TestExponentialRetryPolicy
[INFO] Running org.apache.hadoop.fs.azurebfs.utils.TestUriUtils
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
11.713 s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsOutputStream
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.861
s - in org.apache.hadoop.fs.azurebfs.TestAbfsInputStreamStatistics
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.668
s - in org.apache.hadoop.fs.azurebfs.utils.TestUriUtils
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsOutputStreamStatistics
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.598
s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsRenameRetryRecovery
[INFO] Running
org.apache.hadoop.fs.azurebfs.diagnostics.TestConfigurationValidators
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsNetworkStatistics
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.322 s - in
org.apache.hadoop.fs.azurebfs.diagnostics.TestConfigurationValidators
[INFO] Running org.apache.hadoop.fs.azurebfs.TestTracingContext
[INFO] Running
org.apache.hadoop.fs.azurebfs.TestAbfsConfigurationFieldsValidation
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsErrorTranslation
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.61
s - in org.apache.hadoop.fs.azurebfs.TestAbfsConfigurationFieldsValidation
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsStatistics
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.534
s - in org.apache.hadoop.fs.azurebfs.TestAbfsErrorTranslation
[INFO] Running
org.apache.hadoop.fs.azurebfs.extensions.TestCustomOauthTokenProvider
[INFO] Running
org.apache.hadoop.fs.azurebfs.extensions.TestDTManagerLifecycle
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.441
s - in org.apache.hadoop.fs.azurebfs.TestAbfsOutputStreamStatistics
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.849
s - in org.apache.hadoop.fs.azurebfs.extensions.TestCustomOauthTokenProvider
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.786
s - in org.apache.hadoop.fs.azurebfs.TestAbfsNetworkStatistics
[WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 2, Time elapsed:
6.956 s - in org.apache.hadoop.fs.azurebfs.TestTracingContext
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.59
s - in org.apache.hadoop.fs.azurebfs.extensions.TestDTManagerLifecycle
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsCrc64
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.061
s - in org.apache.hadoop.fs.azurebfs.TestAbfsCrc64
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.686
s - in org.apache.hadoop.fs.azurebfs.TestAbfsStatistics
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
36.877 s - in
org.apache.hadoop.fs.azurebfs.services.TestAbfsClientThrottlingAnalyzer
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
55.459 s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsInputStream
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 107, Failures: 1, Errors: 0, Skipped: 2
[INFO]
[ERROR] There are test failures.
Please refer to /hadoop/hadoop-tools/hadoop-azure/target/surefire-reports
for the individual test results.
Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump
and [date].dumpstream.
[INFO]
[INFO] -
> Azure Token provider requires tenant and client IDs despite being optional
> --------------------------------------------------------------------------
>
> Key: HADOOP-18542
> URL: https://issues.apache.org/jira/browse/HADOOP-18542
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure, hadoop-thirdparty
> Affects Versions: 3.3.2, 3.3.3, 3.3.4
> Reporter: Carl
> Priority: Major
> Labels: pull-request-available
>
> The `AbfsConfiguration` class requires that we provide a tenant and client ID
> when using the `MsiTokenProvider` class to fetch an authentication token. The
> bug is that those fields are not required by the Azure API, which can infer
> those fields when the call is made from an Azure instance.
> The fix is to make tenant and client ID optional when getting an Azure token
> from the Azure Metadata Service.
> A fix has been submitted here: [https://github.com/apache/hadoop/pull/4262]
> The bug was introduced with HADOOP-17725
> ([https://github.com/apache/hadoop/pull/3041/files])
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
