Rogan Morrow created BEAM-12845:
-----------------------------------
Summary: Spark JobServer should bundle AWS IO libraries to support
staging on S3
Key: BEAM-12845
URL: https://issues.apache.org/jira/browse/BEAM-12845
Project: Beam
Issue Type: Improvement
Components: jobserver
Affects Versions: 2.32.0
Reporter: Rogan Morrow
Fix For: 2.33.0
The AWS filesystem libraries are not included in the Spark JobServer. These are
necessary to allow staging artifacts to S3.
There is a TODO in the code to update the dependencies to support the AWS
filesystem.
[https://github.com/apache/beam/blob/master/runners/spark/job-server/spark_job_server.gradle#L62]
Additionally, the `amazon-web-services2` library is built without a dependency
on STS. A dependency on STS is needed to support authentication from EKS with
IAM roles for service accounts.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
