I guess it's about not documented --flink_submit_uber_jar pipeline option. Best regards, Stanislav Porotikov
From: Alexey Romanenko <[email protected]> Sent: Tuesday, December 5, 2023 3:48 PM To: user <[email protected]> Subject: Re: Why do we need Job Server? Oh, interesting. I didn’t know about that possibility, thanks — Alexey On 4 Dec 2023, at 18:14, Robert Bradshaw via user <[email protected]<mailto:[email protected]>> wrote: Note that this shouldn't be strictly necessary, e.g. for Python one can embed the pipeline definition into the jar itself which is then just uploaded as an "ordinary" flink executable jar to the Flink master: https://github.com/apache/beam/blob/release-2.52.0/sdks/python/apache_beam/runners/portability/abstract_job_service.py#L301 If Java doens't do this yet we should probably update it to do so. On Mon, Dec 4, 2023 at 7:10 AM Alexey Romanenko <[email protected]<mailto:[email protected]>> wrote: There are two modes to run a job with FlinkRunner - Portable and Classic. If you run a job server in Portable mode then you meed to start a JobService, configured with your Flink cluster, and submit your job through this. If you run a job in Classical mode (only Java SDK pipeline) then you don’t need it. More information on this is here: Apache Flink Runner<https://beam.apache.org/documentation/runners/flink/> beam.apache.org<https://beam.apache.org/documentation/runners/flink/> <favicon.ico><https://beam.apache.org/documentation/runners/flink/> — Alexey On 4 Dec 2023, at 07:53, Поротиков Станислав Вячеславович via user <[email protected]<mailto:[email protected]>> wrote: Hello! I want to know which cases could lead me to use separate job server for submutting jobs to Flink Cluster? Which cases we don't need it at all? Best regards, Stanislav Porotikov
