[
https://issues.apache.org/jira/browse/BEAM-10793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17314291#comment-17314291
]
Kenneth Knowles commented on BEAM-10793:
----------------------------------------
I think LOOPBACK is OK to use if you are confident in how you are distributing
and launching the SDK harness subprocesses. But agree that DOCKER is of course
the main path that should almost always "just work" to the greatest extent
possible.
> Incorrect Flink runner documentation
> ------------------------------------
>
> Key: BEAM-10793
> URL: https://issues.apache.org/jira/browse/BEAM-10793
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, sdk-go
> Reporter: Kevin Sijo Puthusseri
> Priority: P3
>
> As per the documentation at
> [https://beam.apache.org/documentation/runners/flink/] under _"Portable
> (Java/Python/Go)"_, a containerized flink job server needs to be started using
> {code:java}
> docker run --net=host apache/beam_flink1.10_job_server:latest
> {code}
> or
> {code:java}
> docker run --net=host apache/beam_flink1.10_job_server:latest
> --flink-master=localhost:8081
> {code}
> If any of the SDKs are run using the DOCKER flag, all crash. As explained by
> [~danoliveira] – _"This command is building and running it locally on your
> machine. I'm not 100% sure why running it in a container is causing the
> error, but my suspicion is that it has to do with writing the
> manifest/artifact files to disk. One thing the job server does is writing
> artifacts to disk and then sending the locations to the SDK harness so it can
> read them. If the job server is in a container, then its probably writing the
> files to the container instead of your local machine, so they're inaccessible
> to the SDK harness."_ In fact, [~lostluck] tracked this to an already
> existing issue https://issues.apache.org/jira/browse/BEAM-5273 which is yet
> to be resolved and addresses this exact problem. Using Daniel's advice, Go
> SDK (and others I'm certain) can be run in DOCKER mode if the flink job
> server is started locally using gradle as follows –
> {code:java}
> ./gradlew :runners:flink:1.10:job-server:runShadow -Djob-host=localhost
> -Dflink-master=local{code}
> Only if the SDK is run using the LOOPBACK flag does it manage to run on a
> containerized flink cluster. Moreoever since the LOOPBACK flag is explicitly
> meant for *local development* purposes only, this makes me wonder how folks
> are deploying their production beam data pipelines on flink (especially on
> managed services like Kubernetes). Overall, the main issue (at least until
> BEAM-5273 is unresolved) is the fact that beam's documentation fails to
> mention these caveats explicitly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
