Re: S3 access permission error
Hi, I was facing the same issue, the best way to solve this is to use the IAM role (which is the recommended way) instead of the access keys. Hope this helps. On Wed, Sep 22, 2021 at 1:32 PM Yangze Guo wrote: > I'm not an expert on S3. If it is not a credential issue, have you > finish the checklist of this doc[1]? > > [1] > https://aws.amazon.com/premiumsupport/knowledge-center/emr-s3-403-access-denied/?nc1=h_ls > > Best, > Yangze Guo > > On Wed, Sep 22, 2021 at 3:39 PM Dhiru wrote: > > > > > > Not sure @yangze ... but other services which are deployed in same > places we are able to access s3 bucket, the link which you share are > recommended way, if we have access to s3 then we should not pass > credentials ? > > > > On Wednesday, September 22, 2021, 02:59:05 AM EDT, Yangze Guo < > karma...@gmail.com> wrote: > > > > > > You might need to configure the access credential. [1] > > > > [1] > https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#configure-access-credentials > > > > Best, > > Yangze Guo > > > > On Wed, Sep 22, 2021 at 2:17 PM Dhiru wrote: > > > > > > > > > i see org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2326) > plugin is not able to create folder , not sure if I need to change something > > > Whereas when We are trying to pass from the local laptop and passing > aws credentails its able to create a folder and running as expected > > > On Wednesday, September 22, 2021, 01:39:04 AM EDT, Dhiru < > userdh...@yahoo.com> wrote: > > > > > > > > > flink image I have added both s3 plugin > > > FROM flink:1.11.3-scala_2.12-java11 > > > RUN mkdir ./plugins/flink-s3-fs-presto > > > RUN cp ./opt/flink-s3-fs-presto-1.11.3.jar > ./plugins/flink-s3-fs-presto/ > > > RUN mkdir ./plugins/flink-s3-fs-hadoop > > > RUN cp ./opt/flink-s3-fs-hadoop-1.11.3.jar > ./plugins/flink-s3-fs-hadoop/ > > > > > > some part of flink-conf.yaml ( I tried with both s3a and s3 ) > > ># REQUIRED: set storage location for job metadata in remote storage > > > state.backend: filesystem > > > state.backend.fs.checkpointdir: > s3a://msc-actigraph-test-bucket/flink-checkpointing/checkpoints > > > state.checkpoints.dir: > s3a://msc-actigraph-test-bucket/flink-checkpointing/externalized-checkpoints > > > state.savepoints.dir: > s3a://msc-actigraph-test-bucket/flink-checkpointing/savepoints > > > high-availability.storageDir: > s3a://msc-actigraph-test-bucket/flink-checkpointing/storagedir > > > s3.path.style.access: true > > > > > > org.apache.flink.runtime.rest.handler.RestHandlerException: Could not > execute application. at > org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$1(JarRunHandler.java:103) > at java.base/java.util.concurrent.CompletableFuture.uniHandle(Unknown > Source) at > java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown > Source) at > java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown > Source) at > java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown > Source) at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown > Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at > java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown > Source) at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: > java.util.concurrent.CompletionException: > org.apache.flink.util.FlinkRuntimeException: Could not execute application. > at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown > Source) at > java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown > Source) ... 7 more Caused by: org.apache.flink.util.FlinkRuntimeException: > Could not execute application. at > org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:81) > at > org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:67) > at > org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:100) > ... 7 more Caused by: > org.apache.flink.client.program.ProgramInvocationException: The main method > caused an error: Failed to execute job 'DeduplicationJob'. at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:302) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198) > at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:149) > at > org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:78) > ... 9 more Caused by: org.apache.flink.util.FlinkException: Failed to > execute job
Re: S3 access permission error
I'm not an expert on S3. If it is not a credential issue, have you finish the checklist of this doc[1]? [1] https://aws.amazon.com/premiumsupport/knowledge-center/emr-s3-403-access-denied/?nc1=h_ls Best, Yangze Guo On Wed, Sep 22, 2021 at 3:39 PM Dhiru wrote: > > > Not sure @yangze ... but other services which are deployed in same places we > are able to access s3 bucket, the link which you share are recommended way, > if we have access to s3 then we should not pass credentials ? > > On Wednesday, September 22, 2021, 02:59:05 AM EDT, Yangze Guo > wrote: > > > You might need to configure the access credential. [1] > > [1] > https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#configure-access-credentials > > Best, > Yangze Guo > > On Wed, Sep 22, 2021 at 2:17 PM Dhiru wrote: > > > > > > i see org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2326) plugin > > is not able to create folder , not sure if I need to change something > > Whereas when We are trying to pass from the local laptop and passing aws > > credentails its able to create a folder and running as expected > > On Wednesday, September 22, 2021, 01:39:04 AM EDT, Dhiru > > wrote: > > > > > > flink image I have added both s3 plugin > > FROM flink:1.11.3-scala_2.12-java11 > > RUN mkdir ./plugins/flink-s3-fs-presto > > RUN cp ./opt/flink-s3-fs-presto-1.11.3.jar ./plugins/flink-s3-fs-presto/ > > RUN mkdir ./plugins/flink-s3-fs-hadoop > > RUN cp ./opt/flink-s3-fs-hadoop-1.11.3.jar ./plugins/flink-s3-fs-hadoop/ > > > > some part of flink-conf.yaml ( I tried with both s3a and s3 ) > ># REQUIRED: set storage location for job metadata in remote storage > > state.backend: filesystem > > state.backend.fs.checkpointdir: > > s3a://msc-actigraph-test-bucket/flink-checkpointing/checkpoints > > state.checkpoints.dir: > > s3a://msc-actigraph-test-bucket/flink-checkpointing/externalized-checkpoints > > state.savepoints.dir: > > s3a://msc-actigraph-test-bucket/flink-checkpointing/savepoints > > high-availability.storageDir: > > s3a://msc-actigraph-test-bucket/flink-checkpointing/storagedir > > s3.path.style.access: true > > > > org.apache.flink.runtime.rest.handler.RestHandlerException: Could not > > execute application. at > > org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$1(JarRunHandler.java:103) > > at java.base/java.util.concurrent.CompletableFuture.uniHandle(Unknown > > Source) at > > java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown > > Source) at > > java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown > > Source) at > > java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown > > Source) at > > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown > > Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at > > java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown > > Source) at > > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > > at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > > Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: > > java.util.concurrent.CompletionException: > > org.apache.flink.util.FlinkRuntimeException: Could not execute application. > > at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown > > Source) at > > java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown > > Source) ... 7 more Caused by: org.apache.flink.util.FlinkRuntimeException: > > Could not execute application. at > > org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:81) > > at > > org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:67) > > at > > org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:100) > > ... 7 more Caused by: > > org.apache.flink.client.program.ProgramInvocationException: The main method > > caused an error: Failed to execute job 'DeduplicationJob'. at > > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:302) > > at > > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198) > > at > > org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:149) at > > org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:78) > > ... 9 more Caused by: org.apache.flink.util.FlinkException: Failed to > > execute job 'DeduplicationJob'. at > > org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1829) > > at > >
Re: S3 access permission error
Not sure @yangze ... but other services which are deployed in same places we are able to access s3 bucket, the link which you share are recommended way, if we have access to s3 then we should not pass credentials ? On Wednesday, September 22, 2021, 02:59:05 AM EDT, Yangze Guo wrote: You might need to configure the access credential. [1] [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#configure-access-credentials Best, Yangze Guo On Wed, Sep 22, 2021 at 2:17 PM Dhiru wrote: > > > i see org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2326) plugin is > not able to create folder , not sure if I need to change something > Whereas when We are trying to pass from the local laptop and passing aws > credentails its able to create a folder and running as expected > On Wednesday, September 22, 2021, 01:39:04 AM EDT, Dhiru > wrote: > > > flink image I have added both s3 plugin > FROM flink:1.11.3-scala_2.12-java11 > RUN mkdir ./plugins/flink-s3-fs-presto > RUN cp ./opt/flink-s3-fs-presto-1.11.3.jar ./plugins/flink-s3-fs-presto/ > RUN mkdir ./plugins/flink-s3-fs-hadoop > RUN cp ./opt/flink-s3-fs-hadoop-1.11.3.jar ./plugins/flink-s3-fs-hadoop/ > > some part of flink-conf.yaml ( I tried with both s3a and s3 ) > # REQUIRED: set storage location for job metadata in remote storage > state.backend: filesystem > state.backend.fs.checkpointdir: >s3a://msc-actigraph-test-bucket/flink-checkpointing/checkpoints > state.checkpoints.dir: >s3a://msc-actigraph-test-bucket/flink-checkpointing/externalized-checkpoints > state.savepoints.dir: >s3a://msc-actigraph-test-bucket/flink-checkpointing/savepoints > high-availability.storageDir: >s3a://msc-actigraph-test-bucket/flink-checkpointing/storagedir > s3.path.style.access: true > > org.apache.flink.runtime.rest.handler.RestHandlerException: Could not execute > application. at > org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$1(JarRunHandler.java:103) > at java.base/java.util.concurrent.CompletableFuture.uniHandle(Unknown > Source) at > java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown > Source) at > java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source) > at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown > Source) at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at > java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown > Source) at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: > java.util.concurrent.CompletionException: > org.apache.flink.util.FlinkRuntimeException: Could not execute application. > at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown > Source) at > java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown > Source) ... 7 more Caused by: org.apache.flink.util.FlinkRuntimeException: > Could not execute application. at > org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:81) > at > org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:67) > at > org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:100) > ... 7 more Caused by: > org.apache.flink.client.program.ProgramInvocationException: The main method > caused an error: Failed to execute job 'DeduplicationJob'. at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:302) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198) > at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:149) > at > org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:78) > ... 9 more Caused by: org.apache.flink.util.FlinkException: Failed to > execute job 'DeduplicationJob'. at > org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1829) > at > org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:128) > at > org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:76) > at > org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1700) > at > org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:699) > at
Re: S3 access permission error
You might need to configure the access credential. [1] [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#configure-access-credentials Best, Yangze Guo On Wed, Sep 22, 2021 at 2:17 PM Dhiru wrote: > > > i see org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2326) plugin is > not able to create folder , not sure if I need to change something > Whereas when We are trying to pass from the local laptop and passing aws > credentails its able to create a folder and running as expected > On Wednesday, September 22, 2021, 01:39:04 AM EDT, Dhiru > wrote: > > > flink image I have added both s3 plugin > FROM flink:1.11.3-scala_2.12-java11 > RUN mkdir ./plugins/flink-s3-fs-presto > RUN cp ./opt/flink-s3-fs-presto-1.11.3.jar ./plugins/flink-s3-fs-presto/ > RUN mkdir ./plugins/flink-s3-fs-hadoop > RUN cp ./opt/flink-s3-fs-hadoop-1.11.3.jar ./plugins/flink-s3-fs-hadoop/ > > some part of flink-conf.yaml ( I tried with both s3a and s3 ) > # REQUIRED: set storage location for job metadata in remote storage > state.backend: filesystem > state.backend.fs.checkpointdir: > s3a://msc-actigraph-test-bucket/flink-checkpointing/checkpoints > state.checkpoints.dir: > s3a://msc-actigraph-test-bucket/flink-checkpointing/externalized-checkpoints > state.savepoints.dir: > s3a://msc-actigraph-test-bucket/flink-checkpointing/savepoints > high-availability.storageDir: > s3a://msc-actigraph-test-bucket/flink-checkpointing/storagedir > s3.path.style.access: true > > org.apache.flink.runtime.rest.handler.RestHandlerException: Could not execute > application. at > org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$1(JarRunHandler.java:103) > at java.base/java.util.concurrent.CompletableFuture.uniHandle(Unknown > Source) at > java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown > Source) at > java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source) > at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown > Source) at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at > java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown > Source) at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: > java.util.concurrent.CompletionException: > org.apache.flink.util.FlinkRuntimeException: Could not execute application. > at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown > Source) at > java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown > Source) ... 7 more Caused by: org.apache.flink.util.FlinkRuntimeException: > Could not execute application. at > org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:81) > at > org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:67) > at > org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:100) > ... 7 more Caused by: > org.apache.flink.client.program.ProgramInvocationException: The main method > caused an error: Failed to execute job 'DeduplicationJob'. at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:302) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198) > at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:149) > at > org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:78) > ... 9 more Caused by: org.apache.flink.util.FlinkException: Failed to > execute job 'DeduplicationJob'. at > org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1829) > at > org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:128) > at > org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:76) > at > org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1700) > at > org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:699) > at io.epiphanous.flinkrunner.flink.BaseFlinkJob.run(BaseFlinkJob.scala:45) > at io.epiphanous.flinkrunner.FlinkRunner.process1(FlinkRunner.scala:56) at > io.epiphanous.flinkrunner.FlinkRunner.process(FlinkRunner.scala:33) at > com.mdsol.flink.delivery_streams.Runner$.run(Runner.scala:25) at >
Re: S3 access permission error
i see org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2326) plugin is not able to create folder , not sure if I need to change something Whereas when We are trying to pass from the local laptop and passing aws credentails its able to create a folder and running as expected On Wednesday, September 22, 2021, 01:39:04 AM EDT, Dhiru wrote: flink image I have added both s3 plugin FROM flink:1.11.3-scala_2.12-java11RUN mkdir ./plugins/flink-s3-fs-prestoRUN cp ./opt/flink-s3-fs-presto-1.11.3.jar ./plugins/flink-s3-fs-presto/RUN mkdir ./plugins/flink-s3-fs-hadoopRUN cp ./opt/flink-s3-fs-hadoop-1.11.3.jar ./plugins/flink-s3-fs-hadoop/ some part of flink-conf.yaml ( I tried with both s3a and s3 ) # REQUIRED: set storage location for job metadata in remote storage state.backend: filesystem state.backend.fs.checkpointdir: s3a://msc-actigraph-test-bucket/flink-checkpointing/checkpoints state.checkpoints.dir: s3a://msc-actigraph-test-bucket/flink-checkpointing/externalized-checkpoints state.savepoints.dir: s3a://msc-actigraph-test-bucket/flink-checkpointing/savepoints high-availability.storageDir: s3a://msc-actigraph-test-bucket/flink-checkpointing/storagedir s3.path.style.access: true org.apache.flink.runtime.rest.handler.RestHandlerException: Could not execute application. at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$1(JarRunHandler.java:103) at java.base/java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.CompletionException: org.apache.flink.util.FlinkRuntimeException: Could not execute application. at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ... 7 more Caused by: org.apache.flink.util.FlinkRuntimeException: Could not execute application. at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:81) at org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:67) at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:100) ... 7 more Caused by: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Failed to execute job 'DeduplicationJob'. at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:302) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198) at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:149) at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:78) ... 9 more Caused by: org.apache.flink.util.FlinkException: Failed to execute job 'DeduplicationJob'. at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1829) at org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:128) at org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:76) at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1700) at org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:699) at io.epiphanous.flinkrunner.flink.BaseFlinkJob.run(BaseFlinkJob.scala:45) at io.epiphanous.flinkrunner.FlinkRunner.process1(FlinkRunner.scala:56) at io.epiphanous.flinkrunner.FlinkRunner.process(FlinkRunner.scala:33) at com.mdsol.flink.delivery_streams.Runner$.run(Runner.scala:25) at com.mdsol.flink.delivery_streams.Runner$.main(Runner.scala:7) at com.mdsol.flink.delivery_streams.Runner.main(Runner.scala) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown
S3 access permission error
flink image I have added both s3 plugin FROM flink:1.11.3-scala_2.12-java11RUN mkdir ./plugins/flink-s3-fs-prestoRUN cp ./opt/flink-s3-fs-presto-1.11.3.jar ./plugins/flink-s3-fs-presto/RUN mkdir ./plugins/flink-s3-fs-hadoopRUN cp ./opt/flink-s3-fs-hadoop-1.11.3.jar ./plugins/flink-s3-fs-hadoop/ some part of flink-conf.yaml ( I tried with both s3a and s3 ) # REQUIRED: set storage location for job metadata in remote storage state.backend: filesystem state.backend.fs.checkpointdir: s3a://msc-actigraph-test-bucket/flink-checkpointing/checkpoints state.checkpoints.dir: s3a://msc-actigraph-test-bucket/flink-checkpointing/externalized-checkpoints state.savepoints.dir: s3a://msc-actigraph-test-bucket/flink-checkpointing/savepoints high-availability.storageDir: s3a://msc-actigraph-test-bucket/flink-checkpointing/storagedir s3.path.style.access: true org.apache.flink.runtime.rest.handler.RestHandlerException: Could not execute application. at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$1(JarRunHandler.java:103) at java.base/java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.CompletionException: org.apache.flink.util.FlinkRuntimeException: Could not execute application. at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ... 7 more Caused by: org.apache.flink.util.FlinkRuntimeException: Could not execute application. at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:81) at org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:67) at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:100) ... 7 more Caused by: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Failed to execute job 'DeduplicationJob'. at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:302) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198) at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:149) at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:78) ... 9 more Caused by: org.apache.flink.util.FlinkException: Failed to execute job 'DeduplicationJob'. at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1829) at org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:128) at org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:76) at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1700) at org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:699) at io.epiphanous.flinkrunner.flink.BaseFlinkJob.run(BaseFlinkJob.scala:45) at io.epiphanous.flinkrunner.FlinkRunner.process1(FlinkRunner.scala:56) at io.epiphanous.flinkrunner.FlinkRunner.process(FlinkRunner.scala:33) at com.mdsol.flink.delivery_streams.Runner$.run(Runner.scala:25) at com.mdsol.flink.delivery_streams.Runner$.main(Runner.scala:7) at com.mdsol.flink.delivery_streams.Runner.main(Runner.scala) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:288) ... 12 more Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit job. at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$internalSubmitJob$3(Dispatcher.java:362) at
