I don't think pyspark 3.5.X and python 3.12 work that well together. ons. 21. mai 2025 kl. 21:15 skrev Akram Shaik <akramshaik....@gmail.com>:
> Hello, > > I am trying to access the azure storage account using the delta table from > pyspark shell using azure AAD identity. I see the token is getting > generated successfully but the debug log is showing the unexpected > response. I tried from managed identity using python to access the storage > account. It is able to access the storage account without any issue but > from spark i am getting the following error. > > full log gist: Full Log > <https://gist.github.com/akramshaik541/e231d578403f795adff5e6ecd493d445> > > > Spark version using 3.5.5 > Hadoop-azure 3.4.1 > Hadoop-common 3.4.1 > > > 25/05/21 18:13:15 DEBUG AzureADAuthenticator: AADToken: HTTP connection to > http://169.254.169.254/metadata/identity/oauth2/token failed for getting > token from AzureAD. HTTP response: 200 OK Proxies: none > First 1K of Body: {"access_token":"eyJ0e.... > 25/05/21 18:13:15 DEBUG AbfsClient: Auth failure: HEAD, > https://storageaccount.dfs.core.windows.net/container/?upn=false&action=getAccessControl&timeout=90 > 25/05/21 18:13:15 DEBUG IOStatisticsStoreImpl: Incrementing counter > action_http_head_request.failures by 1 with final value 1 > 25/05/21 18:13:15 DEBUG AzureBlobFileSystemStore: Failed to get ACL status > with non 400. Inferring namespace enabled > Status code: -1 error code: null error message: Auth failure: HTTP Error > 200; url='http://169.254.169.254/metadata/identity/oauth2/token' > AADToken: HTTP connection to > http://169.254.169.254/metadata/identity/oauth2/token failed for getting > token from AzureAD. Unexpected response. Check configuration, URLs and > proxy settings. proxies=none; contentType='application/json'; response > '{"access_token":"eyJ0e....'org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator$UnexpectedResponseException: > HTTP Error 200; url='http://169.254.169.254/metadata/identity/oauth2/token' > AADToken: HTTP connection to > http://169.254.169.254/metadata/identity/oauth2/token failed for getting > token from AzureAD. Unexpected response. Check configuration, URLs and > proxy settings. proxies=none; contentType='application/json'; response > '{"access_token":"eyJ0e....' > at > org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(AbfsRestOperation.java:321) > at > org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.completeExecute(AbfsRestOperation.java:263) > at > org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.lambda$execute$0(AbfsRestOperation.java:235) > at > org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDurationOfInvocation(IOStatisticsBinding.java:464) > at > org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:233) > at > org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:1351) > at > org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:1333) > at > org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getNamespaceEnabledInformationFromServer(AzureBlobFileSystemStore.java:397) > at > org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getIsNamespaceEnabled(AzureBlobFileSystemStore.java:387) > at > org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getFileStatus(AzureBlobFileSystemStore.java:1147) > at > org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:766) > at > org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:756) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1760) > > > Has anyone faced this issue? Appreciate your help on this > > Thanks, > Akram > -- Bjørn Jørgensen Vestre Aspehaug 4, 6010 Ålesund Norge +47 480 94 297
