The worker pool means `starting a java or python sdk`, to accept the java or python pipeline running. For example, to execute python pipeline, we have to start a python worker pool with `--worker_pool` arguments. For the Java code, besides the docker mode (default one), do we have other better ways to start a java worker pool?
For now, our flink cluster is running on the k8s. If we choose the default sdk harness mode (docker), we will have the docker (java sdk harness) in docker (flink-taskmanager). So, what we are doing is to call org.apache.beam.fn.harness.ExternalWorkerService class with pipeline options as environment variables and fixed two small issues in the FnHarness class to make sure the java sdk harness can run smoothly. Thank you. *Mark Ma* On Tue, Oct 11, 2022 at 12:46 PM Alexey Romanenko <[email protected]> wrote: > I’m not sure that I get it correctly. What do you mean by “worker pool” in > your case? > > — > Alexey > > On 8 Oct 2022, at 03:24, Xiao Ma <[email protected]> wrote: > > Hello, > > I would like to run a pipeline with Java as the main language and python > transformation embedded. The beam pipeline is running on the flink cluster. > Currently, I can run it with a taskmanager + java worker pool and a python > worker pool. Could I ask if there is a way to run the java code on the task > manager directly and keep the python worker pool? > > Current: taskmanager + java worker pool + python worker pool > Desired: taskmanager + python worker pool > > Thank you very much. > > *Mark Ma* > > >
