Hi all, I am having the hardest time trying to use an Azure DataLake Gen2 Storage Account using Azure Workload Identity with abfss scheme with Flink 1.20 or 2.0. I keep on getting that either org.apache.hadoop.fs.azurebfs.oauth2.WorkloadIdentityTokenProvider not found or java.lang.IllegalArgumentException: Failed to initialize class org.apache.hadoop.fs.azurebfs.oauth2.WorkloadIdentityTokenProvider. The latter message happens if I add the flink-fs-hadoop-azure to the opt/flink/lib directory of my custom docker image.
The same thing is happening with both Flink 1.20 or Flink 2.0. I have added the jars below in my opt/flink/lib folder: Hadoop-Azure 3.4.1 Hadoop-common 3.4.1 Azure-identity 1.14 Azure-core 1.53 Azure-storage-blob ,,, If I switch the auth provider to Client Credentials and use a client id and secrets flink comes up fine and works. I have verified that the web hook for workflow identity is doing its job, injecting the Azure client id, Azure tenant id and Azure token file as environment variables. In addition, i have checked the token in the azure identity token file to make sure its values are correct. What is the proper way to get flink 1.20 or 2.0 to use Azure Workload Identity as its oauth provider? What should my custom docker image look like? What jar libraries should i include in the image and in what path. I appreciate your help, Ralph Get Outlook for Android<https://aka.ms/AAb9ysg>
