[
https://issues.apache.org/jira/browse/FLINK-30851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17682597#comment-17682597
]
Vincent Chenal edited comment on FLINK-30851 at 1/31/23 3:16 PM:
-----------------------------------------------------------------
I said it worked but actually it's not working anymore with operator version
1.3.1 and {color:#000000}flink-s3-fs-presto-1.16.0.jar{color}
I get this errors:
{code:java}
2023-01-31 14:26:47,052 c.a.m.CsmConfigurationProviderChain
[DEBUG][newflink/newflink-job] Unable to load configuration from
com.amazonaws.monitoring.ProfileCsmConfigurationProvider@6c96d3c0: Unable to
load config file
Exception in thread "pool-5-thread-9" java.lang.NoClassDefFoundError:
org/apache/flink/core/fs/RefCountedTmpFileCreator
at
org.apache.flink.fs.s3.common.FlinkS3FileSystem.<init>(FlinkS3FileSystem.java:100)
at
org.apache.flink.fs.s3presto.FlinkS3PrestoFileSystem.<init>(FlinkS3PrestoFileSystem.java:51)
at
org.apache.flink.fs.s3presto.S3FileSystemFactory.createFlinkFileSystem(S3FileSystemFactory.java:76)
at
org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.create(AbstractS3FileSystemFactory.java:155)
at
org.apache.flink.core.fs.PluginFileSystemFactory.create(PluginFileSystemFactory.java:62)
at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:508)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:409)
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274)
at
org.apache.flink.kubernetes.operator.artifact.FileSystemBasedArtifactFetcher.fetch(FileSystemBasedArtifactFetcher.java:41)
at
org.apache.flink.kubernetes.operator.artifact.ArtifactManager.fetch(ArtifactManager.java:63)
at
org.apache.flink.kubernetes.operator.service.AbstractFlinkService.uploadJar(AbstractFlinkService.java:711)
at
org.apache.flink.kubernetes.operator.service.AbstractFlinkService.submitJobToSessionCluster(AbstractFlinkService.java:197)
at
org.apache.flink.kubernetes.operator.reconciler.sessionjob.SessionJobReconciler.deploy(SessionJobReconciler.java:107)
at
org.apache.flink.kubernetes.operator.reconciler.sessionjob.SessionJobReconciler.deploy(SessionJobReconciler.java:44)
at
org.apache.flink.kubernetes.operator.reconciler.deployment.AbstractFlinkResourceReconciler.reconcile(AbstractFlinkResourceReconciler.java:116)
at
org.apache.flink.kubernetes.operator.reconciler.deployment.AbstractFlinkResourceReconciler.reconcile(AbstractFlinkResourceReconciler.java:65)
at
org.apache.flink.kubernetes.operator.controller.FlinkSessionJobController.reconcile(FlinkSessionJobController.java:99)
at
org.apache.flink.kubernetes.operator.controller.FlinkSessionJobController.reconcile(FlinkSessionJobController.java:51)
at
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:136)
at
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:94)
at
org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80)
at
io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:93)
at
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:130)
at
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:110)
at
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:81)
at
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:54)
at
io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:406)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base/java.lang.Thread.run(Unknown Source) {code}
It was working with operator version 1.2.0 and
{color:#000000}flink-s3-fs-presto-1.15.2.jar{color}
So I'm afraid there could incompatibilities between filesystems plugins and
operator, what would be the best approach?
EDIT: I just tried to rebuild operator 1.3.1 with
{color:#000000}flink-s3-fs-presto-1.15.2.jar{color} and it's working
was (Author: vincechenal):
I said it worked but actually it's not working anymore with operator version
1.3.1 and {color:#000000}flink-s3-fs-presto-1.16.0.jar{color}
I get this errors:
{code:java}
2023-01-31 14:26:47,052 c.a.m.CsmConfigurationProviderChain
[DEBUG][newflink/newflink-job] Unable to load configuration from
com.amazonaws.monitoring.ProfileCsmConfigurationProvider@6c96d3c0: Unable to
load config file
Exception in thread "pool-5-thread-9" java.lang.NoClassDefFoundError:
org/apache/flink/core/fs/RefCountedTmpFileCreator
at
org.apache.flink.fs.s3.common.FlinkS3FileSystem.<init>(FlinkS3FileSystem.java:100)
at
org.apache.flink.fs.s3presto.FlinkS3PrestoFileSystem.<init>(FlinkS3PrestoFileSystem.java:51)
at
org.apache.flink.fs.s3presto.S3FileSystemFactory.createFlinkFileSystem(S3FileSystemFactory.java:76)
at
org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.create(AbstractS3FileSystemFactory.java:155)
at
org.apache.flink.core.fs.PluginFileSystemFactory.create(PluginFileSystemFactory.java:62)
at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:508)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:409)
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274)
at
org.apache.flink.kubernetes.operator.artifact.FileSystemBasedArtifactFetcher.fetch(FileSystemBasedArtifactFetcher.java:41)
at
org.apache.flink.kubernetes.operator.artifact.ArtifactManager.fetch(ArtifactManager.java:63)
at
org.apache.flink.kubernetes.operator.service.AbstractFlinkService.uploadJar(AbstractFlinkService.java:711)
at
org.apache.flink.kubernetes.operator.service.AbstractFlinkService.submitJobToSessionCluster(AbstractFlinkService.java:197)
at
org.apache.flink.kubernetes.operator.reconciler.sessionjob.SessionJobReconciler.deploy(SessionJobReconciler.java:107)
at
org.apache.flink.kubernetes.operator.reconciler.sessionjob.SessionJobReconciler.deploy(SessionJobReconciler.java:44)
at
org.apache.flink.kubernetes.operator.reconciler.deployment.AbstractFlinkResourceReconciler.reconcile(AbstractFlinkResourceReconciler.java:116)
at
org.apache.flink.kubernetes.operator.reconciler.deployment.AbstractFlinkResourceReconciler.reconcile(AbstractFlinkResourceReconciler.java:65)
at
org.apache.flink.kubernetes.operator.controller.FlinkSessionJobController.reconcile(FlinkSessionJobController.java:99)
at
org.apache.flink.kubernetes.operator.controller.FlinkSessionJobController.reconcile(FlinkSessionJobController.java:51)
at
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:136)
at
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:94)
at
org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80)
at
io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:93)
at
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:130)
at
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:110)
at
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:81)
at
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:54)
at
io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:406)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base/java.lang.Thread.run(Unknown Source) {code}
It was working with operator version 1.2.0 and
{color:#000000}flink-s3-fs-presto-1.15.2.jar{color}
So I'm afraid there could incompatibilities between filesystems plugins and
operator, what would be the best approach?
> flink-kubernetes-operator - operator image should provide s3 support for
> jarUri
> -------------------------------------------------------------------------------
>
> Key: FLINK-30851
> URL: https://issues.apache.org/jira/browse/FLINK-30851
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
> Affects Versions: kubernetes-operator-1.3.1
> Reporter: Vincent Chenal
> Priority: Minor
>
> The docker image does not embed filesystem plugins.
> Neither in plugins/ neither in opt/ folders.
> I'd like to use jarUri this way:
> {code:java}
> apiVersion: flink.apache.org/v1beta1
> kind: FlinkSessionJob
> metadata:
> name: myjob
> spec:
> deploymentName: mydeployment
> job:
> jarURI: s3://somebucket/myjar.jar {code}
> But I'm getting this error:
> {code:java}
> Could not find a file system implementation for scheme 's3'. The scheme is
> directly supported by Flink through the following plugins:
> flink-s3-fs-hadoop, flink-s3-fs-presto. Please ensure that each plugin
> resides within its own subfolder within the plugins directory. See
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html for
> more information. If you want to use a Hadoop file system for that scheme,
> please add the scheme to the configuration fs.allowed-fallback-filesystems.
> For a full list of supported file systems, please see
> https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/. {code}
> I was able to make it work by putting flink 1.16 s3-presto-fs jar within the
> flink-kubernetes-operator image but it would make sense to have it natively.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
