[
https://issues.apache.org/jira/browse/HADOOP-18542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17681318#comment-17681318
]
ASF GitHub Bot commented on HADOOP-18542:
-----------------------------------------
CLevasseur commented on PR #4262:
URL: https://github.com/apache/hadoop/pull/4262#issuecomment-1406479958
Hi, sorry for the delay, it took me a while to set everything up.
Unfortunately I am still getting issues when running the integration tests on
the `apache/hadoop` repo (i'm not even using my fork at this point).
It looks like permission issues, although the roles look correct to me. I
followed the doc asking for:
```
3. A Service principal with RBAC - Storage Blob Data Owner (superuser) role
on account
4. A Service principal with no RBAC roles assigned on account. This is
needed for checkaccess tests (noRBACUser)
5. A Service principal with Storage Blob Data Contributor RBAC roles
assigned on account. This is needed for RBAC tests (contributorRBACUser)
6. A Service principal with Storage Blob Data Reader RBAC roles assigned on
account. This is needed for RBAC tests (readerRBACUser)
```
The credentials of each of these 4 account was set in the configuration file.
Here is the output of running the HNS-OAuth test suite, can you tell me if
those failures are expected, and if not, what I might miss ?
```
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsStatistics
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsErrorTranslation
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAccountConfiguration
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsCrc64
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsPerfTracker
[INFO] Running
org.apache.hadoop.fs.azurebfs.services.TestExponentialRetryPolicy
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsInputStream
[INFO] Running
org.apache.hadoop.fs.azurebfs.services.TestAbfsRenameRetryRecovery
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.392
s - in org.apache.hadoop.fs.azurebfs.TestAbfsCrc64
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.225
s - in org.apache.hadoop.fs.azurebfs.TestAbfsErrorTranslation
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestQueryParams
[ERROR] Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
3.192 s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.TestAccountConfiguration
[ERROR]
testConfigPropNotFound(org.apache.hadoop.fs.azurebfs.TestAccountConfiguration)
Time elapsed: 0.19 s <<< FAILURE!
java.lang.AssertionError: Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
at
org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499)
at
org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384)
at
org.apache.hadoop.fs.azurebfs.TestAccountConfiguration.testMissingConfigKey(TestAccountConfiguration.java:399)
at
org.apache.hadoop.fs.azurebfs.TestAccountConfiguration.testConfigPropNotFound(TestAccountConfiguration.java:386)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.404
s - in org.apache.hadoop.fs.azurebfs.services.TestQueryParams
[INFO] Running
org.apache.hadoop.fs.azurebfs.services.TestAzureADAuthenticator
[INFO] Running
org.apache.hadoop.fs.azurebfs.services.TestTextFileBasedIdentityHandler
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsOutputStream
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.809
s - in org.apache.hadoop.fs.azurebfs.TestAbfsStatistics
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.516
s - in org.apache.hadoop.fs.azurebfs.services.TestTextFileBasedIdentityHandler
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsClient
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsHttpOperation
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.182
s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsRenameRetryRecovery
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.515
s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsHttpOperation
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
10.529 s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsPerfTracker
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.637
s - in org.apache.hadoop.fs.azurebfs.services.TestAzureADAuthenticator
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.092
s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsClient
[INFO] Running
org.apache.hadoop.fs.azurebfs.services.TestAbfsClientThrottlingAnalyzer
[INFO] Running
org.apache.hadoop.fs.azurebfs.services.TestShellDecryptionKeyProvider
[INFO] Running
org.apache.hadoop.fs.azurebfs.TestAbfsConfigurationFieldsValidation
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.458
s - in org.apache.hadoop.fs.azurebfs.services.TestShellDecryptionKeyProvider
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsInputStreamStatistics
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsOutputStreamStatistics
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.03
s - in org.apache.hadoop.fs.azurebfs.TestAbfsConfigurationFieldsValidation
[INFO] Running
org.apache.hadoop.fs.azurebfs.diagnostics.TestConfigurationValidators
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.263 s - in
org.apache.hadoop.fs.azurebfs.diagnostics.TestConfigurationValidators
[INFO] Running
org.apache.hadoop.fs.azurebfs.extensions.TestDTManagerLifecycle
[INFO] Running
org.apache.hadoop.fs.azurebfs.extensions.TestCustomOauthTokenProvider
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.304
s - in org.apache.hadoop.fs.azurebfs.TestAbfsInputStreamStatistics
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.894
s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsOutputStream
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.627
s - in org.apache.hadoop.fs.azurebfs.extensions.TestDTManagerLifecycle
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.953
s - in org.apache.hadoop.fs.azurebfs.extensions.TestCustomOauthTokenProvider
[INFO] Running org.apache.hadoop.fs.azurebfs.utils.TestUriUtils
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.236
s - in org.apache.hadoop.fs.azurebfs.TestAbfsOutputStreamStatistics
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.319
s - in org.apache.hadoop.fs.azurebfs.utils.TestUriUtils
[INFO] Running org.apache.hadoop.fs.azurebfs.utils.TestCachedSASToken
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsNetworkStatistics
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.355
s - in org.apache.hadoop.fs.azurebfs.utils.TestCachedSASToken
[INFO] Running org.apache.hadoop.fs.azurebfs.TestTracingContext
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.834
s - in org.apache.hadoop.fs.azurebfs.TestAbfsNetworkStatistics
[WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
3.102 s - in org.apache.hadoop.fs.azurebfs.TestTracingContext
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
36.466 s - in
org.apache.hadoop.fs.azurebfs.services.TestAbfsClientThrottlingAnalyzer
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
59.788 s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsInputStream
[WARNING] Tests run: 6, Failures: 0, Errors: 0, Skipped: 2, Time elapsed:
99.715 s - in org.apache.hadoop.fs.azurebfs.services.TestExponentialRetryPolicy
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 111, Failures: 1, Errors: 0, Skipped: 3
[INFO]
[ERROR] There are test failures.
Please refer to /hadoop/hadoop-tools/hadoop-azure/target/surefire-reports
for the individual test results.
Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump
and [date].dumpstream.
[INFO]
[INFO] -
> Azure Token provider requires tenant and client IDs despite being optional
> --------------------------------------------------------------------------
>
> Key: HADOOP-18542
> URL: https://issues.apache.org/jira/browse/HADOOP-18542
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure, hadoop-thirdparty
> Affects Versions: 3.3.2, 3.3.3, 3.3.4
> Reporter: Carl
> Priority: Major
> Labels: pull-request-available
>
> The `AbfsConfiguration` class requires that we provide a tenant and client ID
> when using the `MsiTokenProvider` class to fetch an authentication token. The
> bug is that those fields are not required by the Azure API, which can infer
> those fields when the call is made from an Azure instance.
> The fix is to make tenant and client ID optional when getting an Azure token
> from the Azure Metadata Service.
> A fix has been submitted here: [https://github.com/apache/hadoop/pull/4262]
> The bug was introduced with HADOOP-17725
> ([https://github.com/apache/hadoop/pull/3041/files])
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
