[ https://issues.apache.org/jira/browse/BEAM-8471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961628#comment-16961628 ]
Thomas Weise commented on BEAM-8471: ------------------------------------ If anyone is interested to try this using curl with a local Flink cluster: Upload the job server jar (can be done ahead of time, when building a container, for example) {code:java} curl -X POST -H "Expect:" -F "jarfile=@/Users/tweise/src/beam/runners/flink/1.8/job-server/build/libs/beam-runners-flink-1.8-job-server-2.16.0-SNAPSHOT.jar" http://localhost:8081/jars/upload {code} The response will be the jar file reference used later. Job submission payload: {code:java} { "entryClass":"org.apache.beam.runners.flink.FlinkPortableClientEntryPoint", "programArgs":"--driver-cmd \"python -m apache_beam.dummy_pipeline\"", "parallelism": 1 } {code} Run the job: {code:java} curl --verbose -X POST -d "@launch.json" 'http://localhost:8081/jars/c1c9747a-42f5-4a97-8871-0dafb809b869_beam-runners-flink-1.8-job-server-2.16.0-SNAPSHOT.jar/run' {code} > Flink native job submission for portable pipelines > -------------------------------------------------- > > Key: BEAM-8471 > URL: https://issues.apache.org/jira/browse/BEAM-8471 > Project: Beam > Issue Type: Improvement > Components: runner-flink > Reporter: Thomas Weise > Assignee: Thomas Weise > Priority: Major > Labels: portability-flink > Fix For: 2.18.0 > > Time Spent: 5h 10m > Remaining Estimate: 0h > > There are currently two methods to run a portable pipeline written in a > non-JVM language to Flink: > 1) Run the SDK client entry point which will submit the job server, which in > turn will submit to a Flink cluster using the Flink remote environment > 2) Run the SDK client entry point to generate a Flink jar file that can be > used to start the Flink job using any of the Flink client tooling available. > Either approach requires the SDK client and the job server dependency to be > present on the client. This doesn't work well in environments such as > FlinkK8sOperator that rely on the Flink REST API jar run endpoint (see > [https://docs.google.com/document/d/1z3LNrRtr8kkiFHonZ5JJM_L4NWNBBNcqRc_yAf6G0VI/edit#heading=h.x7hki4bhh18l]). > This improvement is to provide a new Flink entry point (main method) that > invokes the SDK client entry point to generate the pipeline and submits the > resulting Flink job like any other Flink native driver program would, via the > optimizer plan environment ("[auto]"). > -- This message was sent by Atlassian Jira (v8.3.4#803005)