junaiddshaukat opened a new pull request, #39680:
URL: https://github.com/apache/beam/pull/39680
## Summary
Part of #18479. Follows the review point on #39627: the runner starting its
own job server only helped Java, and a Python user still had to run one by
hand. This adds the wrapper Flink and Spark provide, so a Python pipeline can
select the runner and nothing else.
```
python my_pipeline.py \
--runner=KafkaStreamsRunner \
--bootstrap_servers=localhost:9092 \
--application_id=my-beam-pipeline
```
## The job server jar
A portable runner needs its job server to be launchable without a Beam
source tree, so
`runners/kafka-streams/job-server` packages the runner and its dependencies
into one shaded jar, in
the same shape as the Flink and Spark job server modules.
`KafkaStreamsRunner` on the Python side
builds it if it has to, starts it, and stops it with the pipeline.
The module carries an SLF4J binding. Without one the jar starts and then
logs nothing at all, which
is unhelpful for a process a user runs in the foreground to watch their
pipeline — the Spark module
excludes the binding because Spark supplies its own, and copying that left
us silent.
## The Python runner
`KafkaStreamsRunner` extends `PortableRunner`, resolves the jar through
`JavaJarJobServer`, and
defaults the environment to `LOOPBACK` so a local run needs no Docker.
`KafkaStreamsRunnerOptions`
adds `--bootstrap_servers`, `--application_id` and
`--kafka_streams_job_server_jar`, the last of
which takes a prebuilt jar and skips the build.
## Testing
Beam's Python is not built in my environment, so rather than assert this
works I checked each link
of the chain against something real:
* The shaded jar starts and brings up all three services — job service,
artifact staging and
expansion — on the ports it is given.
* The arguments the Python runner passes (`--artifacts-dir`, `--job-port`,
`--artifact-port`,
`--expansion-port`) are all accepted by
`JobServerDriver.ServerConfiguration`, and our driver
overrides none of them. The jar was run with exactly that argument set.
* Against an installed Beam with these two files grafted in,
`--runner=KafkaStreamsRunner` resolves,
the options parse, `default_environment` sets `LOOPBACK`, and
`java_arguments` produces the flags
above.
* `:runners:kafka-streams:job-server:shadowJar`, the target the runner asks
Beam to build, exists.
What that leaves untested is a Python pipeline actually running through to
completion, which needs a
built Beam Python SDK. If CI does not cover that path I would rather it were
checked before this is
relied on.
The runner documentation gains a Python section, which I will add to #39627
since the page is still
open for review there.
```
./gradlew :runners:kafka-streams:validatesRunner # 59 tests, 0
failures
./gradlew :runners:kafka-streams:build # unit tests,
spotless + checker + errorprone
./gradlew :runners:kafka-streams:job-server:shadowJar # the jar
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
