This starts the Job Server in a container when Python pipelines are run and no Job Server endpoint has been specified. The Job Server has the docker binaries and socket mounted from the host to bring up SDK harness containers directly on the host. This feature is currently only is available for Flink.
Due to host networking not being available on MacOs, we need to explicitly map ports from the host system to the container. On Linux this is not necessary because host networking works as expected. Generally, getting Docker support to work on MacOs proved to be a bit tricky. You can test this via: ``` gradle :beam-sdks-python:portableWordCount ``` Some rough edges on MacOS: - At times, the JobService socket is not available, even though `grpc.channel_ready_future(channel).result()` in the portable_runner returns. - The port range for the SDK harness is restricted to be between 8100 and 8200 (configurable) due to the necessary explicit port forwarding Potentially problematic on Linux: - The Docker binary might not be statically compiled and depend on shared libraries, though this needs to be tested CC @angoenka Post-Commit Tests Status (on master branch) ------------------------------------------------------------------------------------------------ Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) </br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | --- | --- | --- | --- [ Full content available at: https://github.com/apache/beam/pull/6265 ] This message was relayed via gitbox.apache.org for [email protected]
