Jeremy Lewi created BEAM-12836:
----------------------------------
Summary: Flink JobServer should allow setting the default
environment for the expansion service
Key: BEAM-12836
URL: https://issues.apache.org/jira/browse/BEAM-12836
Project: Beam
Issue Type: Improvement
Components: jobserver
Reporter: Jeremy Lewi
I think the
[FlinkJobServer|https://github.com/apache/beam/blob/3a7b8e757d0c220b5838ca6a0805ee463b24f85d/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java#L72]
needs to expose command line arguments to allow configuring the default
environment for the expansion service.
IIUC the expansion service in the job server is needed when submitting jobs
using the portable runner that have cross language transforms (e.g. a Python
pipeline that uses KafkaIO which is a Java language transform). In this case,
we may need to configure the expansion service to change the default
environment (e.g. the docker image used for Java transforms).
It looks like [https://github.com/apache/beam/pull/15082/files] added
appropriate command line flags to the main entry point for a standalone
expansion service. However, I don't believe this code path is used when running
the expansion service as part of the job server.
I believe the
[FlinkJobServer|https://github.com/apache/beam/blob/3a7b8e757d0c220b5838ca6a0805ee463b24f85d/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java#L72]
would need to take similar command line arguments and use them to configure
the expansion service.
More discussion in this
[thread|https://lists.apache.org/thread.html/r3cf56464c157da840f46e777c1b3f307f65ec2cf6b2278728f2cd1e1%40%3Cuser.beam.apache.org%3E]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
