[jira] [Comment Edited] (FLINK-34565) Enhance flink kubernetes configMap to accommodate additional configuration files
[
https://issues.apache.org/jira/browse/FLINK-34565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833440#comment-17833440
]
Surendra Singh Lilhore edited comment on FLINK-34565 at 4/3/24 9:21 AM:
[~zhuzh] If the user files are dynamic, then it is very useful to support them
in ConfigMap, especially in an App Mode cluster.
A similar use case was discussed on the user mailing list: [link to the
mailing list
thread.|https://lists.apache.org/thread/md2zq0dbvt2dxytdfxw16jbfh02yq0w9]
[~wangyang0918] , Any thought about this?
was (Author: surendrasingh):
[~zhuzh] If the user files are dynamic, then it is very useful to support them
in ConfigMap, especially in an App Mode cluster.
A similar use case was discussed on the user mailing list: [link to the
mailing list
thread.|https://lists.apache.org/thread/md2zq0dbvt2dxytdfxw16jbfh02yq0w9]
> Enhance flink kubernetes configMap to accommodate additional configuration
> files
>
>
> Key: FLINK-34565
> URL: https://issues.apache.org/jira/browse/FLINK-34565
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes
>Reporter: Surendra Singh Lilhore
>Priority: Major
> Labels: pull-request-available
>
> Flink kubernetes client currently supports a fixed number of files
> (flink-conf.yaml, logback-console.xml, log4j-console.properties) in the JM
> and TM Pod ConfigMap. In certain scenarios, particularly in app mode,
> additional configuration files are required for jobs to run successfully.
> Presently, users must resort to workarounds to include dynamic configuration
> files in the JM and TM. This proposed improvement allows users to easily add
> extra files by configuring the
> '{*}kubernetes.flink.configmap.additional.resources{*}' property. Users can
> provide a semicolon-separated list of local files in the client Flink config
> directory that should be included in the Flink ConfigMap.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34565) Enhance flink kubernetes configMap to accommodate additional configuration files
[
https://issues.apache.org/jira/browse/FLINK-34565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833440#comment-17833440
]
Surendra Singh Lilhore commented on FLINK-34565:
[~zhuzh] If the user files are dynamic, then it is very useful to support them
in ConfigMap, especially in an App Mode cluster.
A similar use case was discussed on the user mailing list: [link to the
mailing list
thread.|https://lists.apache.org/thread/md2zq0dbvt2dxytdfxw16jbfh02yq0w9]
> Enhance flink kubernetes configMap to accommodate additional configuration
> files
>
>
> Key: FLINK-34565
> URL: https://issues.apache.org/jira/browse/FLINK-34565
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes
>Reporter: Surendra Singh Lilhore
>Priority: Major
> Labels: pull-request-available
>
> Flink kubernetes client currently supports a fixed number of files
> (flink-conf.yaml, logback-console.xml, log4j-console.properties) in the JM
> and TM Pod ConfigMap. In certain scenarios, particularly in app mode,
> additional configuration files are required for jobs to run successfully.
> Presently, users must resort to workarounds to include dynamic configuration
> files in the JM and TM. This proposed improvement allows users to easily add
> extra files by configuring the
> '{*}kubernetes.flink.configmap.additional.resources{*}' property. Users can
> provide a semicolon-separated list of local files in the client Flink config
> directory that should be included in the Flink ConfigMap.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34565) Enhance flink kubernetes configMap to accommodate additional configuration files
[
https://issues.apache.org/jira/browse/FLINK-34565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831263#comment-17831263
]
Surendra Singh Lilhore commented on FLINK-34565:
[~zhuzh] can you review this ?
> Enhance flink kubernetes configMap to accommodate additional configuration
> files
>
>
> Key: FLINK-34565
> URL: https://issues.apache.org/jira/browse/FLINK-34565
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes
>Reporter: Surendra Singh Lilhore
>Priority: Major
> Labels: pull-request-available
>
> Flink kubernetes client currently supports a fixed number of files
> (flink-conf.yaml, logback-console.xml, log4j-console.properties) in the JM
> and TM Pod ConfigMap. In certain scenarios, particularly in app mode,
> additional configuration files are required for jobs to run successfully.
> Presently, users must resort to workarounds to include dynamic configuration
> files in the JM and TM. This proposed improvement allows users to easily add
> extra files by configuring the
> '{*}kubernetes.flink.configmap.additional.resources{*}' property. Users can
> provide a semicolon-separated list of local files in the client Flink config
> directory that should be included in the Flink ConfigMap.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34565) Enhance flink kubernetes configMap to accommodate additional configuration files
Surendra Singh Lilhore created FLINK-34565:
--
Summary: Enhance flink kubernetes configMap to accommodate
additional configuration files
Key: FLINK-34565
URL: https://issues.apache.org/jira/browse/FLINK-34565
Project: Flink
Issue Type: Bug
Components: Deployment / Kubernetes
Reporter: Surendra Singh Lilhore
Flink kubernetes client currently supports a fixed number of files
(flink-conf.yaml, logback-console.xml, log4j-console.properties) in the JM and
TM Pod ConfigMap. In certain scenarios, particularly in app mode, additional
configuration files are required for jobs to run successfully. Presently, users
must resort to workarounds to include dynamic configuration files in the JM and
TM. This proposed improvement allows users to easily add extra files by
configuring the '{*}kubernetes.flink.configmap.additional.resources{*}'
property. Users can provide a semicolon-separated list of local files in the
client Flink config directory that should be included in the Flink ConfigMap.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-24150) Support to configure cpu resource request and limit in pod template
[
https://issues.apache.org/jira/browse/FLINK-24150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793128#comment-17793128
]
Surendra Singh Lilhore commented on FLINK-24150:
[~richard.su] , I have given [patch|https://github.com/apache/flink/pull/23768]
for similar issue in Flink-33609. Check if this can help here.
> Support to configure cpu resource request and limit in pod template
> ---
>
> Key: FLINK-24150
> URL: https://issues.apache.org/jira/browse/FLINK-24150
> Project: Flink
> Issue Type: New Feature
> Components: Deployment / Kubernetes
>Reporter: Yang Wang
>Priority: Major
>
> Why Flink needs to overwrite memory resource defined in pod template?
> The major reason is that Flink need to ensure the consistency between Flink
> configuration
> (\{{taskmanager.memory.process.size}} , {{kubernetes.taskmanager.cpu}}) and
> pod template resource settings. Since users could specify the total process
> memory or detailed memory[2], Flink will calculate the pod resource
> internally.
>
> For the CPU case the template’s requests/limits should have priority if they
> are specified. The factor could still be used if the template doesn’t specify
> anything. The logic could be something like this:
> # To choose CPU request
> # Read pod template first
> # If template doesn’t have anything, read from {{kubernetes.taskmanager.cpu}}
> # If configuration is not specified, fall back to default
> # To choose CPU limit
> # Read from template first
> # If template doesn’t have anything, apply factor to what was chosen in step
> 1, where the default factor is 1.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-33609) Take into account the resource limit specified in the pod template.
[ https://issues.apache.org/jira/browse/FLINK-33609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore resolved FLINK-33609. Resolution: Duplicate There is one more Jira to fix this issue in better way : FLINK-33548 > Take into account the resource limit specified in the pod template. > --- > > Key: FLINK-33609 > URL: https://issues.apache.org/jira/browse/FLINK-33609 > Project: Flink > Issue Type: Bug > Components: Deployment / Kubernetes >Affects Versions: 1.16.0 >Reporter: Surendra Singh Lilhore >Priority: Major > Labels: pull-request-available > > Flink is currently not considering the pod template resource limits and is > only utilizing the limit obtained from the configured or default limit > factor. Flink should consider both the value obtained from the limit factor > and the pod template resource limits. It should take the maximum value of the > pod template resource limits and the value obtained from the limit factor > calculation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33609) Take into account the resource limit specified in the pod template.
Surendra Singh Lilhore created FLINK-33609: -- Summary: Take into account the resource limit specified in the pod template. Key: FLINK-33609 URL: https://issues.apache.org/jira/browse/FLINK-33609 Project: Flink Issue Type: Bug Components: Deployment / Kubernetes Affects Versions: 1.16.0 Reporter: Surendra Singh Lilhore Flink is currently not considering the pod template resource limits and is only utilizing the limit obtained from the configured or default limit factor. Flink should consider both the value obtained from the limit factor and the pod template resource limits. It should take the maximum value of the pod template resource limits and the value obtained from the limit factor calculation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33095) Job jar related issue should be reported as BAD_REQUEST instead of INTERNAL_SERVER_ERROR
[ https://issues.apache.org/jira/browse/FLINK-33095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated FLINK-33095: --- Attachment: image-2023-09-16-13-21-24-634.png > Job jar related issue should be reported as BAD_REQUEST instead of > INTERNAL_SERVER_ERROR > > > Key: FLINK-33095 > URL: https://issues.apache.org/jira/browse/FLINK-33095 > Project: Flink > Issue Type: Bug > Components: Runtime / REST >Affects Versions: 1.16.0 >Reporter: Surendra Singh Lilhore >Priority: Major > Attachments: image-2023-09-16-13-20-15-052.png, > image-2023-09-16-13-21-24-634.png > > > When submitting a job with incorrect parameters, such as an invalid entry > class, the current response is an internal server error. > To enhance the user experience and consistency, it is recommended to throw a > Rest exception and return a BAD_REQUEST response code in such cases. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33095) Job jar related issue should be reported as BAD_REQUEST instead of INTERNAL_SERVER_ERROR
[ https://issues.apache.org/jira/browse/FLINK-33095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17765922#comment-17765922 ] Surendra Singh Lilhore commented on FLINK-33095: Current response from server : !image-2023-09-16-13-20-15-052.png! == It should be something like this : !image-2023-09-16-13-21-24-634.png! > Job jar related issue should be reported as BAD_REQUEST instead of > INTERNAL_SERVER_ERROR > > > Key: FLINK-33095 > URL: https://issues.apache.org/jira/browse/FLINK-33095 > Project: Flink > Issue Type: Bug > Components: Runtime / REST >Affects Versions: 1.16.0 >Reporter: Surendra Singh Lilhore >Priority: Major > Attachments: image-2023-09-16-13-20-15-052.png, > image-2023-09-16-13-21-24-634.png > > > When submitting a job with incorrect parameters, such as an invalid entry > class, the current response is an internal server error. > To enhance the user experience and consistency, it is recommended to throw a > Rest exception and return a BAD_REQUEST response code in such cases. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33095) Job jar related issue should be reported as BAD_REQUEST instead of INTERNAL_SERVER_ERROR
[ https://issues.apache.org/jira/browse/FLINK-33095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated FLINK-33095: --- Attachment: image-2023-09-16-13-20-15-052.png > Job jar related issue should be reported as BAD_REQUEST instead of > INTERNAL_SERVER_ERROR > > > Key: FLINK-33095 > URL: https://issues.apache.org/jira/browse/FLINK-33095 > Project: Flink > Issue Type: Bug > Components: Runtime / REST >Affects Versions: 1.16.0 >Reporter: Surendra Singh Lilhore >Priority: Major > Attachments: image-2023-09-16-13-20-15-052.png > > > When submitting a job with incorrect parameters, such as an invalid entry > class, the current response is an internal server error. > To enhance the user experience and consistency, it is recommended to throw a > Rest exception and return a BAD_REQUEST response code in such cases. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33095) Job jar related issue should be reported as BAD_REQUEST instead of INTERNAL_SERVER_ERROR
[ https://issues.apache.org/jira/browse/FLINK-33095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated FLINK-33095: --- Attachment: (was: image-2023-09-16-13-17-14-869.png) > Job jar related issue should be reported as BAD_REQUEST instead of > INTERNAL_SERVER_ERROR > > > Key: FLINK-33095 > URL: https://issues.apache.org/jira/browse/FLINK-33095 > Project: Flink > Issue Type: Bug > Components: Runtime / REST >Affects Versions: 1.16.0 >Reporter: Surendra Singh Lilhore >Priority: Major > > When submitting a job with incorrect parameters, such as an invalid entry > class, the current response is an internal server error. > To enhance the user experience and consistency, it is recommended to throw a > Rest exception and return a BAD_REQUEST response code in such cases. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33095) Job jar related issue should be reported as BAD_REQUEST instead of INTERNAL_SERVER_ERROR
Surendra Singh Lilhore created FLINK-33095: -- Summary: Job jar related issue should be reported as BAD_REQUEST instead of INTERNAL_SERVER_ERROR Key: FLINK-33095 URL: https://issues.apache.org/jira/browse/FLINK-33095 Project: Flink Issue Type: Bug Components: Runtime / REST Affects Versions: 1.16.0 Reporter: Surendra Singh Lilhore When submitting a job with incorrect parameters, such as an invalid entry class, the current response is an internal server error. To enhance the user experience and consistency, it is recommended to throw a Rest exception and return a BAD_REQUEST response code in such cases. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33095) Job jar related issue should be reported as BAD_REQUEST instead of INTERNAL_SERVER_ERROR
[ https://issues.apache.org/jira/browse/FLINK-33095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated FLINK-33095: --- Attachment: (was: image-2023-09-16-13-17-49-945.png) > Job jar related issue should be reported as BAD_REQUEST instead of > INTERNAL_SERVER_ERROR > > > Key: FLINK-33095 > URL: https://issues.apache.org/jira/browse/FLINK-33095 > Project: Flink > Issue Type: Bug > Components: Runtime / REST >Affects Versions: 1.16.0 >Reporter: Surendra Singh Lilhore >Priority: Major > > When submitting a job with incorrect parameters, such as an invalid entry > class, the current response is an internal server error. > To enhance the user experience and consistency, it is recommended to throw a > Rest exception and return a BAD_REQUEST response code in such cases. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32122) Update the Azure Blob Storage document to assist in configuring the MSI provider with a shaded class name
Surendra Singh Lilhore created FLINK-32122: -- Summary: Update the Azure Blob Storage document to assist in configuring the MSI provider with a shaded class name Key: FLINK-32122 URL: https://issues.apache.org/jira/browse/FLINK-32122 Project: Flink Issue Type: Improvement Components: Connectors / FileSystem, Documentation Reporter: Surendra Singh Lilhore Many users have reported on the mailing list that they are unable to configure the ABFS filesystem as a checkpoint directory. This is often due to ClassNotFoundException errors for Hadoop classes that are configured in the configuration value. For instance, when using MsiTokenProvider for ABFS storage in Flink, it should be configured with the shaded class name. However, many users mistakenly use the Hadoop class name or package instead. fs.azure.account.oauth.provider.type: *org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider* -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-30745) Check-pointing with Azure Data Lake Storage
[
https://issues.apache.org/jira/browse/FLINK-30745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17679514#comment-17679514
]
Surendra Singh Lilhore edited comment on FLINK-30745 at 1/22/23 5:55 AM:
-
[~dheerajpanangat] , sorry for late reply.
As mentioned in Flink doc ([Azure blob Flink configuration
|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/azure/]
), you need to configure abfs properties in *flink-conf.yaml.*
[HadoopConfigLoader
|https://github.com/apache/flink/blob/master/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopConfigLoader.java#L82]
load this configuration from flink configuration.
{quote}Provided the shaded classes instead of Hadoop classes
{quote}
You are correct
Please configure below properties in flink-conf.yaml in Kubernetes cluster and
try.
{noformat}
fs.azure.account.auth.type : OAuth
fs.azure.account.oauth.provider.type :
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
fs.azure.account.oauth2.client.id :
fs.azure.account.oauth2.client.secret :
fs.azure.account.oauth2.client.endpoint :
https://XX.com/X/oauth2/token {noformat}
was (Author: surendrasingh):
[~dheerajpanangat] , sorry for late reply.
As mentioned in Flink doc ([Azure blob Flink configuration
|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/azure/]
), you need to configure abfs properties in *flink-conf.yaml.*
[HadoopConfigLoader
|https://github.com/apache/flink/blob/master/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopConfigLoader.java#L82]
load this configuration from flink configuration.
{quote}Provided the shaded classes instead of Hadoop classes
{quote}
You we correct
Please configure below properties in flink-conf.yaml in Kubernetes cluster and
try.
{noformat}
fs.azure.account.auth.type : OAuth
fs.azure.account.oauth.provider.type :
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
fs.azure.account.oauth2.client.id :
fs.azure.account.oauth2.client.secret :
fs.azure.account.oauth2.client.endpoint :
https://XX.com/X/oauth2/token {noformat}
> Check-pointing with Azure Data Lake Storage
> ---
>
> Key: FLINK-30745
> URL: https://issues.apache.org/jira/browse/FLINK-30745
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.15.2, 1.14.6
>Reporter: Dheeraj Panangat
>Priority: Major
>
> Hi,
> While checkpointing to Azure Blob Storage using Flink, we get the following
> error :
> {code:java}
> Caused by: Configuration property .dfs.core.windows.net not
> found.
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getStorageAccountKey(AbfsConfiguration.java:372)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1133)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.(AzureBlobFileSystemStore.java:174)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:110)
> {code}
> We have given the configurations in core-site.xml too for following
> {code:java}
> fs.hdfs.impl
> fs.abfs.impl -> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem
> fs.file.impl
> fs.azure.account.auth.type
> fs.azure.account.oauth.provider.type
> fs.azure.account.oauth2.client.id
> fs.azure.account.oauth2.client.secret
> fs.azure.account.oauth2.client.endpoint
> fs.azure.createRemoteFileSystemDuringInitialization -> true {code}
> On debugging found that flink reads from core-default-shaded.xml, but even if
> the properties are specified there, the default configs are not loaded and we
> get a different exception as :
> {code:java}
> Caused by: Unable to load key provider class.
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getTokenProvider(AbfsConfiguration.java:540)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1136)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.(AzureBlobFileSystemStore.java:174)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:110)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-30745) Check-pointing with Azure Data Lake Storage
[
https://issues.apache.org/jira/browse/FLINK-30745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17679514#comment-17679514
]
Surendra Singh Lilhore edited comment on FLINK-30745 at 1/22/23 5:54 AM:
-
[~dheerajpanangat] , sorry for late reply.
As mentioned in Flink doc ([Azure blob Flink configuration
|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/azure/]
), you need to configure abfs properties in *flink-conf.yaml.*
[HadoopConfigLoader
|https://github.com/apache/flink/blob/master/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopConfigLoader.java#L82]
load this configuration from flink configuration.
{quote}Provided the shaded classes instead of Hadoop classes
{quote}
You we correct
Please configure below properties in flink-conf.yaml in Kubernetes cluster and
try.
{noformat}
fs.azure.account.auth.type : OAuth
fs.azure.account.oauth.provider.type :
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
fs.azure.account.oauth2.client.id :
fs.azure.account.oauth2.client.secret :
fs.azure.account.oauth2.client.endpoint :
https://XX.com/X/oauth2/token {noformat}
was (Author: surendrasingh):
[~dheerajpanangat] , sorry for late reply.
As mentioned in flink doc ([Azure Blob Storage | Apache
Flink|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/azure/]),
you need to configure abfs properties in *flink-conf.yaml.*
[HadoopConfigLoader
|https://github.com/apache/flink/blob/master/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopConfigLoader.java#L82]
load this configuration from flink configuration.
{quote}Provided the shaded classes instead of Hadoop classes
{quote}
You we correct
Please configure below properties in flink-conf.yaml in Kubernetes cluster and
try.
{noformat}
fs.azure.account.auth.type : OAuth
fs.azure.account.oauth.provider.type :
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
fs.azure.account.oauth2.client.id :
fs.azure.account.oauth2.client.secret :
fs.azure.account.oauth2.client.endpoint :
https://XX.com/X/oauth2/token {noformat}
> Check-pointing with Azure Data Lake Storage
> ---
>
> Key: FLINK-30745
> URL: https://issues.apache.org/jira/browse/FLINK-30745
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.15.2, 1.14.6
>Reporter: Dheeraj Panangat
>Priority: Major
>
> Hi,
> While checkpointing to Azure Blob Storage using Flink, we get the following
> error :
> {code:java}
> Caused by: Configuration property .dfs.core.windows.net not
> found.
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getStorageAccountKey(AbfsConfiguration.java:372)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1133)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.(AzureBlobFileSystemStore.java:174)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:110)
> {code}
> We have given the configurations in core-site.xml too for following
> {code:java}
> fs.hdfs.impl
> fs.abfs.impl -> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem
> fs.file.impl
> fs.azure.account.auth.type
> fs.azure.account.oauth.provider.type
> fs.azure.account.oauth2.client.id
> fs.azure.account.oauth2.client.secret
> fs.azure.account.oauth2.client.endpoint
> fs.azure.createRemoteFileSystemDuringInitialization -> true {code}
> On debugging found that flink reads from core-default-shaded.xml, but even if
> the properties are specified there, the default configs are not loaded and we
> get a different exception as :
> {code:java}
> Caused by: Unable to load key provider class.
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getTokenProvider(AbfsConfiguration.java:540)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1136)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.(AzureBlobFileSystemStore.java:174)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:110)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30745) Check-pointing with Azure Data Lake Storage
[
https://issues.apache.org/jira/browse/FLINK-30745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17679514#comment-17679514
]
Surendra Singh Lilhore commented on FLINK-30745:
[~dheerajpanangat] , sorry for late reply.
As mentioned in flink doc ([Azure Blob Storage | Apache
Flink|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/azure/]),
you need to configure abfs properties in *flink-conf.yaml.*
[HadoopConfigLoader
|https://github.com/apache/flink/blob/master/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopConfigLoader.java#L82]
load this configuration from flink configuration.
{quote}Provided the shaded classes instead of Hadoop classes
{quote}
You we correct
Please configure below properties in flink-conf.yaml in Kubernetes cluster and
try.
{noformat}
fs.azure.account.auth.type : OAuth
fs.azure.account.oauth.provider.type :
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
fs.azure.account.oauth2.client.id :
fs.azure.account.oauth2.client.secret :
fs.azure.account.oauth2.client.endpoint :
https://XX.com/X/oauth2/token {noformat}
> Check-pointing with Azure Data Lake Storage
> ---
>
> Key: FLINK-30745
> URL: https://issues.apache.org/jira/browse/FLINK-30745
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.15.2, 1.14.6
>Reporter: Dheeraj Panangat
>Priority: Major
>
> Hi,
> While checkpointing to Azure Blob Storage using Flink, we get the following
> error :
> {code:java}
> Caused by: Configuration property .dfs.core.windows.net not
> found.
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getStorageAccountKey(AbfsConfiguration.java:372)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1133)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.(AzureBlobFileSystemStore.java:174)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:110)
> {code}
> We have given the configurations in core-site.xml too for following
> {code:java}
> fs.hdfs.impl
> fs.abfs.impl -> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem
> fs.file.impl
> fs.azure.account.auth.type
> fs.azure.account.oauth.provider.type
> fs.azure.account.oauth2.client.id
> fs.azure.account.oauth2.client.secret
> fs.azure.account.oauth2.client.endpoint
> fs.azure.createRemoteFileSystemDuringInitialization -> true {code}
> On debugging found that flink reads from core-default-shaded.xml, but even if
> the properties are specified there, the default configs are not loaded and we
> get a different exception as :
> {code:java}
> Caused by: Unable to load key provider class.
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getTokenProvider(AbfsConfiguration.java:540)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1136)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.(AzureBlobFileSystemStore.java:174)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:110)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30745) Check-pointing with Azure Data Lake Storage
[
https://issues.apache.org/jira/browse/FLINK-30745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17678329#comment-17678329
]
Surendra Singh Lilhore commented on FLINK-30745:
Thanks [~dheerajpanangat] for reporting this issue.
What value you configured for property
"fs.azure.account.keyprovider." ?
> Check-pointing with Azure Data Lake Storage
> ---
>
> Key: FLINK-30745
> URL: https://issues.apache.org/jira/browse/FLINK-30745
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.15.2, 1.14.6
>Reporter: Dheeraj Panangat
>Priority: Major
>
> Hi,
> While checkpointing to Azure Blob Storage using Flink, we get the following
> error :
> {code:java}
> Caused by: Configuration property .dfs.core.windows.net not
> found.
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getStorageAccountKey(AbfsConfiguration.java:372)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1133)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.(AzureBlobFileSystemStore.java:174)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:110)
> {code}
> We have given the configurations in core-site.xml too for following
> {code:java}
> fs.hdfs.impl
> fs.abfs.impl -> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem
> fs.file.impl
> fs.azure.account.auth.type
> fs.azure.account.oauth.provider.type
> fs.azure.account.oauth2.client.id
> fs.azure.account.oauth2.client.secret
> fs.azure.account.oauth2.client.endpoint
> fs.azure.createRemoteFileSystemDuringInitialization -> true {code}
> On debugging found that flink reads from core-default-shaded.xml, but even if
> the properties are specified there, the default configs are not loaded and we
> get a different exception as :
> {code:java}
> Caused by: Unable to load key provider class.
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getTokenProvider(AbfsConfiguration.java:540)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1136)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.(AzureBlobFileSystemStore.java:174)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:110)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
