[
https://issues.apache.org/jira/browse/FLINK-38282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18016041#comment-18016041
]
raphaelauv commented on FLINK-38282:
------------------------------------
If I add the jar flink-core-2.0.0.jar thanks to an initContainers it works
{code:java}
initContainers:
- name: curl
image: curlimages/curl:latest
command:
- /bin/sh
- '-c'
- >-
curl -H "Accept: application/zip"
https://repo1.maven.org/maven2/org/apache/flink/flink-s3-fs-hadoop/2.0.0/flink-s3-fs-hadoop-2.0.0.jar
--output
/opt/flink/plugins/s3-fs-hadoop/flink-s3-fs-hadoop-2.0.0.jar &&
curl -H "Accept: application/zip"
https://repo1.maven.org/maven2/org/apache/flink/flink-core/2.0.0/flink-core-2.0.0.jar
--output /opt/flink/operator-lib/flink-core-2.0.0.jar
resources: {}
volumeMounts:
- name: s3-plugin
mountPath: /opt/flink/plugins/s3-fs-hadoop/
- name: flink-libs
mountPath: /opt/flink/operator-lib/
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: Always {code}
> K8S operator : S3 jarURI
> -------------------------
>
> Key: FLINK-38282
> URL: https://issues.apache.org/jira/browse/FLINK-38282
> Project: Flink
> Issue Type: Bug
> Affects Versions: 2.0.0
> Reporter: raphaelauv
> Priority: Major
> Attachments: flink-kubernetes-operator.log
>
>
> Using the flink operator 1.8.2
> with flink 2.0.0
> I can't submit FlinkSessionJob that have a jarURI on a s3
> {code:java}
> apiVersion: flink.apache.org/v1beta1
> kind: FlinkDeployment
> metadata:
> name: basic-example
> spec:
> image:
> flinkVersion: v2_0
> flinkConfiguration:
> taskmanager.numberOfTaskSlots: "2"
> s3.endpoint: http://minio:9000
> s3.path.style.access: "true"
> s3.access-key: admin
> s3.secret-key: adminadmin
> s3.region: eu-west-3
> serviceAccount: flink
> jobManager:
> resource:
> memory: "2048m"
> cpu: 1
> taskManager:
> resource:
> memory: "2048m"
> cpu: 1
> podTemplate:
> spec:
> containers:
> - name: flink-main-container
> env:
> - name: ENABLE_BUILT_IN_PLUGINS
> value: "flink-s3-fs-hadoop-2.0.0.jar" {code}
> {code:java}
> apiVersion: flink.apache.org/v1beta1
> kind: FlinkSessionJob
> metadata:
> name: my-flink-job
> spec:
> deploymentName: basic-example
> job:
> jarURI: s3://test-bucket/myjar-0.0.1.jar
> parallelism: 1
> upgradeMode: stateless {code}
>
>
> the FlinkSessionJob fail with :
> {code:java}
> Could not find a file system implementation for scheme 's3a'. The scheme is
> directly supported by Flink through the following plugin(s):
> flink-s3-fs-hadoop. Please ensure that each plugin resides within its own
> subfolder within the plugins directory. See
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/plugins/
> for more information. If you want to use a Hadoop file system for that
> scheme, please add the scheme to the configuration
> fs.allowed-fallback-filesystems. For a full list of supported file systems,
> please see
> https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/. {code}
> and in the operator pod the logs are ->
>
> {code:java}
> java.util.ServiceConfigurationError:
> org.apache.flink.core.fs.FileSystemFactory:
> org.apache.flink.fs.s3hadoop.S3FileSystemFactory Unable to get public no-arg
> constructor
> at java.base/java.util.ServiceLoader.fail(Unknown Source)
> at java.base/java.util.ServiceLoader.getConstructor(Unknown Source)
> at
> java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown
> Source)
> at
> java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown
> Source)
> at java.base/java.util.ServiceLoader$2.hasNext(Unknown Source)
> at java.base/java.util.ServiceLoader$3.hasNext(Unknown Source)
> at
> org.apache.flink.core.plugin.PluginLoader$ContextClassLoaderSettingIterator.hasNext(PluginLoader.java:130)
> at
> org.apache.flink.shaded.guava31.com.google.common.collect.Iterators$ConcatenatedIterator.hasNext(Iterators.java:1391)
> at
> org.apache.flink.shaded.guava31.com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46)
> at
> org.apache.flink.core.fs.FileSystem.addAllFactoriesToList(FileSystem.java:1088)
> at
> org.apache.flink.core.fs.FileSystem.loadFileSystemFactories(FileSystem.java:1071)
> at org.apache.flink.core.fs.FileSystem.initialize(FileSystem.java:340)
> at
> org.apache.flink.kubernetes.operator.FlinkOperator.<init>(FlinkOperator.java:112)
> at
> org.apache.flink.kubernetes.operator.FlinkOperator.main(FlinkOperator.java:281)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/flink/core/fs/PathsCopyingFileSystem
> at java.base/java.lang.ClassLoader.defineClass1(Native Method)
> at java.base/java.lang.ClassLoader.defineClass(Unknown Source)
> at java.base/java.security.SecureClassLoader.defineClass(Unknown Source)
> at java.base/java.net.URLClassLoader.defineClass(Unknown Source)
> at java.base/java.net.URLClassLoader$1.run(Unknown Source)
> at java.base/java.net.URLClassLoader$1.run(Unknown Source)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/java.net.URLClassLoader.findClass(Unknown Source)
> at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
> at
> org.apache.flink.core.classloading.ComponentClassLoader.loadClassFromComponentOnly(ComponentClassLoader.java:150)
> at
> org.apache.flink.core.classloading.ComponentClassLoader.loadClassFromOwnerFirst(ComponentClassLoader.java:172)
> at
> org.apache.flink.core.classloading.ComponentClassLoader.loadClass(ComponentClassLoader.java:107)
> at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
> at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.base/java.lang.Class.privateGetDeclaredConstructors(Unknown Source)
> at java.base/java.lang.Class.getConstructor0(Unknown Source)
> at java.base/java.lang.Class.getConstructor(Unknown Source)
> at java.base/java.util.ServiceLoader$1.run(Unknown Source)
> at java.base/java.util.ServiceLoader$1.run(Unknown Source)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> ... 13 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
