[
https://issues.apache.org/jira/browse/FLINK-38282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18016086#comment-18016086
]
raphaelauv commented on FLINK-38282:
------------------------------------
the error is not related to flink-s3-fs-hadoop it's related to flink-core , as
I shared in my first comment
why flink-core is missing ( org/apache/flink/core/fs/PathsCopyingFileSystem )
from the operator container image ?
> K8S operator : S3 jarURI
> -------------------------
>
> Key: FLINK-38282
> URL: https://issues.apache.org/jira/browse/FLINK-38282
> Project: Flink
> Issue Type: Bug
> Affects Versions: 2.0.0
> Reporter: raphaelauv
> Priority: Major
> Attachments: flink-kubernetes-operator.log
>
>
> Using the flink operator 1.8.2
> with flink 2.0.0
> I can't submit FlinkSessionJob that have a jarURI on a s3
> {code:java}
> apiVersion: flink.apache.org/v1beta1
> kind: FlinkDeployment
> metadata:
> name: basic-example
> spec:
> image:
> flinkVersion: v2_0
> flinkConfiguration:
> taskmanager.numberOfTaskSlots: "2"
> s3.endpoint: http://minio:9000
> s3.path.style.access: "true"
> s3.access-key: admin
> s3.secret-key: adminadmin
> s3.region: eu-west-3
> serviceAccount: flink
> jobManager:
> resource:
> memory: "2048m"
> cpu: 1
> taskManager:
> resource:
> memory: "2048m"
> cpu: 1
> podTemplate:
> spec:
> containers:
> - name: flink-main-container
> env:
> - name: ENABLE_BUILT_IN_PLUGINS
> value: "flink-s3-fs-hadoop-2.0.0.jar" {code}
> {code:java}
> apiVersion: flink.apache.org/v1beta1
> kind: FlinkSessionJob
> metadata:
> name: my-flink-job
> spec:
> deploymentName: basic-example
> job:
> jarURI: s3://test-bucket/myjar-0.0.1.jar
> parallelism: 1
> upgradeMode: stateless {code}
>
>
> the FlinkSessionJob fail with :
> {code:java}
> Could not find a file system implementation for scheme 's3a'. The scheme is
> directly supported by Flink through the following plugin(s):
> flink-s3-fs-hadoop. Please ensure that each plugin resides within its own
> subfolder within the plugins directory. See
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/plugins/
> for more information. If you want to use a Hadoop file system for that
> scheme, please add the scheme to the configuration
> fs.allowed-fallback-filesystems. For a full list of supported file systems,
> please see
> https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/. {code}
> and in the operator pod the logs are ->
>
> {code:java}
> java.util.ServiceConfigurationError:
> org.apache.flink.core.fs.FileSystemFactory:
> org.apache.flink.fs.s3hadoop.S3FileSystemFactory Unable to get public no-arg
> constructor
> at java.base/java.util.ServiceLoader.fail(Unknown Source)
> at java.base/java.util.ServiceLoader.getConstructor(Unknown Source)
> at
> java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown
> Source)
> at
> java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown
> Source)
> at java.base/java.util.ServiceLoader$2.hasNext(Unknown Source)
> at java.base/java.util.ServiceLoader$3.hasNext(Unknown Source)
> at
> org.apache.flink.core.plugin.PluginLoader$ContextClassLoaderSettingIterator.hasNext(PluginLoader.java:130)
> at
> org.apache.flink.shaded.guava31.com.google.common.collect.Iterators$ConcatenatedIterator.hasNext(Iterators.java:1391)
> at
> org.apache.flink.shaded.guava31.com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46)
> at
> org.apache.flink.core.fs.FileSystem.addAllFactoriesToList(FileSystem.java:1088)
> at
> org.apache.flink.core.fs.FileSystem.loadFileSystemFactories(FileSystem.java:1071)
> at org.apache.flink.core.fs.FileSystem.initialize(FileSystem.java:340)
> at
> org.apache.flink.kubernetes.operator.FlinkOperator.<init>(FlinkOperator.java:112)
> at
> org.apache.flink.kubernetes.operator.FlinkOperator.main(FlinkOperator.java:281)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/flink/core/fs/PathsCopyingFileSystem
> at java.base/java.lang.ClassLoader.defineClass1(Native Method)
> at java.base/java.lang.ClassLoader.defineClass(Unknown Source)
> at java.base/java.security.SecureClassLoader.defineClass(Unknown Source)
> at java.base/java.net.URLClassLoader.defineClass(Unknown Source)
> at java.base/java.net.URLClassLoader$1.run(Unknown Source)
> at java.base/java.net.URLClassLoader$1.run(Unknown Source)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/java.net.URLClassLoader.findClass(Unknown Source)
> at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
> at
> org.apache.flink.core.classloading.ComponentClassLoader.loadClassFromComponentOnly(ComponentClassLoader.java:150)
> at
> org.apache.flink.core.classloading.ComponentClassLoader.loadClassFromOwnerFirst(ComponentClassLoader.java:172)
> at
> org.apache.flink.core.classloading.ComponentClassLoader.loadClass(ComponentClassLoader.java:107)
> at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
> at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.base/java.lang.Class.privateGetDeclaredConstructors(Unknown Source)
> at java.base/java.lang.Class.getConstructor0(Unknown Source)
> at java.base/java.lang.Class.getConstructor(Unknown Source)
> at java.base/java.util.ServiceLoader$1.run(Unknown Source)
> at java.base/java.util.ServiceLoader$1.run(Unknown Source)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> ... 13 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
