Jeremy Lewi created BEAM-12739:
----------------------------------
Summary: Flink JobServer should bundle AWS IO libraries to support
staging on S3
Key: BEAM-12739
URL: https://issues.apache.org/jira/browse/BEAM-12739
Project: Beam
Issue Type: Improvement
Components: jobserver
Affects Versions: 2.31.0
Reporter: Jeremy Lewi
It doesn't look like the AWS Filesystem libraries are included in the Flink
JobServer.
There is a TODO in the code to update the dependencies to support the AWS
filesystem.
[https://github.com/apache/beam/blob/master/runners/flink/job-server/flink_job_server.gradle#L89]
As reported in this [mailing list
thread|https://lists.apache.org/x/thread.html/rec91c120cb5ead637d434ca99ae98e5119721de0789f0b8d814f1306@%3Cuser.beam.apache.org%3E]
trying to use S3 to stage artifacts results in the exception
{code:java}
Aug 06, 2021 1:52:51 AM
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2
finishStaging
SEVERE: Error staging artifacts
java.util.concurrent.ExecutionException:
java.lang.IllegalArgumentException: No filesystem found for scheme s3
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
