[
https://issues.apache.org/jira/browse/HUDI-5103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HUDI-5103:
---------------------------------
Labels: features pull-request-available (was: features)
> Does not work with Azure Data lake Gen2
> ---------------------------------------
>
> Key: HUDI-5103
> URL: https://issues.apache.org/jira/browse/HUDI-5103
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Dheeraj Panangat
> Priority: Major
> Labels: features, pull-request-available
>
> Unable to use Hudi with Flink for Azure Data Lake
> It tries to look for
> {code:java}
> Caused by: Configuration property <datalakeaccount>.dfs.core.windows.net not
> found.
> at
> org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getStorageAccountKey(AbfsConfiguration.java:372)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1133)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.<init>(AzureBlobFileSystemStore.java:174)
> {code}
> Following properties are specified when running the Flink code :
> {code:java}
> "fs.azure.account.auth.type": "OAuth",
> "fs.azure.account.oauth.provider.type":
> "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
> "fs.azure.account.oauth2.client.id": "<appId>",
> "fs.azure.account.oauth2.client.secret": "<clientSecret>",
> "fs.azure.account.oauth2.client.endpoint":
> "https://login.microsoftonline.com/<tenant>/oauth2/token",
> "fs.azure.createRemoteFileSystemDuringInitialization": "true" {code}
> as per document : [Microsoft Azure Spark to Data Lake
> Gen2|https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-use-databricks-spark]
>
>
> In AWS it works because it takes the credentials from the environment, but in
> Azure it needs to get it from config, which does not reach till the point
> where the FileSystem is initialized.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
