Thank you very muchfor the reply and explaination. For the Java beam sdk, can it start as a worker pool, like the Python worker pool with --worker_pool option? Or the Java sdk doesn't have the external environment type, it has to be as docker started?
Thank you. Matk On Sat, Oct 8, 2022 at 12:08 AM Chamikara Jayalath via dev < [email protected]> wrote: > > > On Fri, Oct 7, 2022 at 6:29 PM Xiao Ma <[email protected]> wrote: > >> Hello, >> >> I would like to run a pipeline with Java as the main language and python >> transformation embedded. The beam pipeline is running on the flink cluster. >> Currently, I can run it with a taskmanager + java worker pool and a python >> worker pool. Could I ask if there is a way to run the java code on the task >> manager directly and keep the python worker pool? >> >> Current: taskmanager + java worker pool + python worker pool >> Desired: taskmanager + python worker pool >> > > Generally this is not possible. If the transform has to be executed on the > SDK side, the runner usually sets up an environment (for example, a Docker > container) with the corresponding SDK and executes the bundles with the > transform using the Beam Fn API. Runners can choose to override this by > executing the transform within the runner itself, but you'll have to modify > the Flink runner to do this. > > Thanks, > Cham > > >> >> Thank you very much. >> >> *Mark Ma* >> >> -- Xiao Ma Geotab Software Developer, Data Engineering | B.Sc, M.Sc Direct +1 (416) 836 - 3541 Toll-free +1 (877) 436 - 8221 Visit www.geotab.com Twitter | Facebook | YouTube | LinkedIn
