[
https://issues.apache.org/jira/browse/BEAM-4130?focusedWorklogId=137111&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137111
]
ASF GitHub Bot logged work on BEAM-4130:
----------------------------------------
Author: ASF GitHub Bot
Created on: 22/Aug/18 18:47
Start Date: 22/Aug/18 18:47
Worklog Time Spent: 10m
Work Description: mxm opened a new pull request #6265: [BEAM-4130] Bring
up Job Server container for Python jobs
URL: https://github.com/apache/beam/pull/6265
This starts the Job Server in a container when Python pipelines are run and
no
Job Server endpoint has been specified. The Job Server has the docker
binaries
and socket mounted from the host to bring up SDK harness containers directly
on
the host. This feature is currently only is available for Flink.
Due to host networking not being available on MacOs, we need to explicitly
map
ports from the host system to the container. On Linux this is not necessary
because host networking works as expected. Generally, getting Docker support
to
work on MacOs proved to be a bit tricky.
You can test this via:
```
gradle :beam-sdks-python:portableWordCount
```
Some rough edges on MacOS:
- At times, the JobService socket is not available, even though
`grpc.channel_ready_future(channel).result()` in the portable_runner returns.
- The port range for the SDK harness is restricted to be between 8100 and
8200 (configurable) due to the necessary explicit port forwarding
Potentially problematic on Linux:
- The Docker binary might not be statically compiled and depend on shared
libraries, though this needs to be tested
CC @angoenka
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
| --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
| --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
</br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
| --- | --- | --- | ---
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 137111)
Time Spent: 5h (was: 4h 50m)
> Portable Flink runner JobService entry point in a Docker container
> ------------------------------------------------------------------
>
> Key: BEAM-4130
> URL: https://issues.apache.org/jira/browse/BEAM-4130
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
> Reporter: Ben Sidhom
> Assignee: Maximilian Michels
> Priority: Minor
> Time Spent: 5h
> Remaining Estimate: 0h
>
> The portable Flink runner exists as a Job Service that runs somewhere. We
> need a main entry point that itself spins up the job service (and artifact
> staging service). The main program itself should be packaged into an uberjar
> such that it can be run locally or submitted to a Flink deployment via `flink
> run`.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
