It seems I set a wrong high-availability.storageDir, s3://flink-test/recovery can work, but s3:///flink-test/recovery can not, one / be removed.
Joshua Fan <joshuafat...@gmail.com> 于2021年8月5日周四 上午10:43写道： > Hi Robert, Tobias > > I have tried many ways to build and validate the image. > > 1.put the s3 dependency to plugin subdirectory, the Dockerfile content is > below: > >> FROM apache/flink:1.13.1-scala_2.11 >> ADD ./flink-s3-fs-hadoop-1.13.1.jar >> /opt/flink/plugins/s3-hadoop/flink-s3-fs-hadoop-1.13.1.jar >> ADD ./flink-s3-fs-presto-1.13.1.jar >> /opt/flink/plugins/s3-presto/flink-s3-fs-presto-1.13.1.jar >> > This time the image can be run on k8s but would also hit a error like > "org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not > find a file system implementation for scheme 's3'.", it seems like flink > can not find the s3 filesystem supports dynamically. When I want to run the > image using 'docker run -it ', it would also report > 'standard_init_linux.go:190: exec user process caused "exec format error"'. > > 2.put the s3 dependency to plugin directly, the Dockerfile content is > below: > >> FROM apache/flink:1.13.1-scala_2.11 >> ADD ./flink-s3-fs-hadoop-1.13.1.jar /opt/flink/plugins >> ADD ./flink-s3-fs-presto-1.13.1.jar /opt/flink/plugins >> > The image can not run on the k8s and report error just the same as run > the image using 'docker run -it ', 'standard_init_linux.go:190: exec user > process caused "exec format error"'. > > 3.just run the community edition image flink:1.13.1-scala_2.11 locally as > docker run -it, it will also hit the same error > 'standard_init_linux.go:190: exec user process caused "exec format error"', > but the flink:1.13.1-scala_2.11 can be run on the k8s without s3 > requirement. > > 4.import the s3 dependency as a kubernetes parameter > I submit the session with ' > -Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-1.13.0.jar > \ > -Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS= > flink-s3-fs-hadoop-1.13.0.jar', the session can be start, but report > error as below > >> Caused by: java.lang.NullPointerException: null uri host. >> at java.util.Objects.requireNonNull(Objects.java:228) >> at >> org.apache.hadoop.fs.s3native.S3xLoginHelper.buildFSURI(S3xLoginHelper.java:72) >> at >> org.apache.hadoop.fs.s3a.S3AFileSystem.setUri(S3AFileSystem.java:467) >> at >> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:234) >> at >> org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.create(AbstractS3FileSystemFactory.java:123) >> > but I have set the s3 staff in the flink-conf.yaml as below: > >> high-availability: >> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory > > high-availability.storageDir: s3:///flink-test/recovery > > s3.endpoint: http://xxx.yyy.zzz.net > > s3.path.style.access: true > > s3.access-key: 111111111111111111111111111 > > s3.secret-key: 222222222222222222222222222 > > I think I supplied all the s3 information in the flink-conf.yaml, but it > did not work. > > I will try other ways to complete the s3 ha on k8s. Thank your guys. > > Yours sincerely > Joshua > > Robert Metzger <rmetz...@apache.org> 于2021年8月4日周三 下午11:35写道： > >> Hey Joshua, >> >> Can you first validate if the docker image you've built is valid by >> running it locally on your machine? >> >> I would recommend putting the s3 filesystem files into the plugins [1] >> directory to avoid classloading issues. >> Also, you don't need to build custom images if you want to use build-in >> plugins [2] >> >> [1] >> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/filesystems/plugins/ >> [2] >> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#using-plugins >> >> On Wed, Aug 4, 2021 at 3:06 PM Joshua Fan <joshuafat...@gmail.com> wrote: >> >>> Hi All >>> I want to build a custom flink image to run on k8s, below is my >>> Dockerfile content: >>> >>>> FROM apache/flink:1.13.1-scala_2.11 >>>> ADD ./flink-s3-fs-hadoop-1.13.1.jar /opt/flink/lib >>>> ADD ./flink-s3-fs-presto-1.13.1.jar /opt/flink/lib >>>> >>> I just put the s3 fs dependency to the {flink home}/lib, and then I >>> build the image and push it to the repo. >>> >>> When I submit the flink session from the custom image, a error will be >>> reported like "exec /docker-entrypoint.sh failed: Exec format error". >>> >>> I googled a lot, but it seems no useful information. >>> >>> Thanks for your help. >>> >>> Yours sincerely >>> Joshua >>> >>
