[
https://issues.apache.org/jira/browse/FLINK-30745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17679166#comment-17679166
]
Dheeraj Panangat commented on FLINK-30745:
------------------------------------------
Hi [~surendralilhore] ,
We were able to get it working on our local machine, where we are just
executing the main class.
When we deployed the same solution on Kubernetes using Flink Native Kubernetes
command, it gives the same error there.
The core-default-shaded.xml is included in the docker image jar which is
deployed, but still seeing the same issues there.
Can you help check what could be the possible reasons ?
Thanks
> Check-pointing with Azure Data Lake Storage
> -------------------------------------------
>
> Key: FLINK-30745
> URL: https://issues.apache.org/jira/browse/FLINK-30745
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
> Affects Versions: 1.15.2, 1.14.6
> Reporter: Dheeraj Panangat
> Priority: Major
>
> Hi,
> While checkpointing to Azure Blob Storage using Flink, we get the following
> error :
> {code:java}
> Caused by: Configuration property <accoutnname>.dfs.core.windows.net not
> found.
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getStorageAccountKey(AbfsConfiguration.java:372)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1133)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.<init>(AzureBlobFileSystemStore.java:174)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:110)
> {code}
> We have given the configurations in core-site.xml too for following
> {code:java}
> fs.hdfs.impl
> fs.abfs.impl -> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem
> fs.file.impl
> fs.azure.account.auth.type
> fs.azure.account.oauth.provider.type
> fs.azure.account.oauth2.client.id
> fs.azure.account.oauth2.client.secret
> fs.azure.account.oauth2.client.endpoint
> fs.azure.createRemoteFileSystemDuringInitialization -> true {code}
> On debugging found that flink reads from core-default-shaded.xml, but even if
> the properties are specified there, the default configs are not loaded and we
> get a different exception as :
> {code:java}
> Caused by: Unable to load key provider class.
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getTokenProvider(AbfsConfiguration.java:540)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1136)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.<init>(AzureBlobFileSystemStore.java:174)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:110)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
