[
https://issues.apache.org/jira/browse/BEAM-8471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961628#comment-16961628
]
Thomas Weise commented on BEAM-8471:
------------------------------------
If anyone is interested to try this using curl with a local Flink cluster:
Upload the job server jar (can be done ahead of time, when building a
container, for example)
{code:java}
curl -X POST -H "Expect:" -F
"jarfile=@/Users/tweise/src/beam/runners/flink/1.8/job-server/build/libs/beam-runners-flink-1.8-job-server-2.16.0-SNAPSHOT.jar"
http://localhost:8081/jars/upload {code}
The response will be the jar file reference used later.
Job submission payload:
{code:java}
{ "entryClass":"org.apache.beam.runners.flink.FlinkPortableClientEntryPoint",
"programArgs":"--driver-cmd \"python -m apache_beam.dummy_pipeline\"",
"parallelism": 1
} {code}
Run the job:
{code:java}
curl --verbose -X POST -d "@launch.json"
'http://localhost:8081/jars/c1c9747a-42f5-4a97-8871-0dafb809b869_beam-runners-flink-1.8-job-server-2.16.0-SNAPSHOT.jar/run'
{code}
> Flink native job submission for portable pipelines
> --------------------------------------------------
>
> Key: BEAM-8471
> URL: https://issues.apache.org/jira/browse/BEAM-8471
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
> Reporter: Thomas Weise
> Assignee: Thomas Weise
> Priority: Major
> Labels: portability-flink
> Fix For: 2.18.0
>
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> There are currently two methods to run a portable pipeline written in a
> non-JVM language to Flink:
> 1) Run the SDK client entry point which will submit the job server, which in
> turn will submit to a Flink cluster using the Flink remote environment
> 2) Run the SDK client entry point to generate a Flink jar file that can be
> used to start the Flink job using any of the Flink client tooling available.
> Either approach requires the SDK client and the job server dependency to be
> present on the client. This doesn't work well in environments such as
> FlinkK8sOperator that rely on the Flink REST API jar run endpoint (see
> [https://docs.google.com/document/d/1z3LNrRtr8kkiFHonZ5JJM_L4NWNBBNcqRc_yAf6G0VI/edit#heading=h.x7hki4bhh18l]).
> This improvement is to provide a new Flink entry point (main method) that
> invokes the SDK client entry point to generate the pipeline and submits the
> resulting Flink job like any other Flink native driver program would, via the
> optimizer plan environment ("[auto]").
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
