Hi Lukasz, Thanks for your answer. I had a look at the ProcessEnvironmentFactory, and I understand that the factory is expecting to run the boot process which is in the python container (source file boot.go). Is that right?
Best, Andrea From: Lukasz Cwik <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Tuesday, 10 September 2019 at 23:19 To: user <[email protected]> Subject: Re: How do I run Beam Python pipelines using Flink deployed on Kubernetes? [CAUTION - EXTERNAL EMAIL] External environments are mainly used for testing since they represent environments that are already running. There are other clever uses of it as well but unlikely what your looking for. The docker way will be nice once there is integration between the FlinkRunner and its supported cluster managers to automatically manage the SDK container instances. Until that exists you could look at using the process environment (it launches the SDK as a child process beside the Flink workers). On Tue, Sep 10, 2019 at 4:38 AM Andrea Medeghini <[email protected]<mailto:[email protected]>> wrote: Hello, Can I run Beam Python pipelines with Flink deployed on Kubernetes? I have successfully managed to run a Beam Python pipeline using the Portable runner and the job service pointing to a local Flink server running as Docker containers. I was able to achieve that mounting the Docker socket in my Flink container and running Flink as root, so the class DockerEnvironmentFactory can create the Python harness container. Unfortunately, I can't use the same solution when Flink is running in Kubernetes. Moreover, I don't want to create the Python harness container using the Docker command from my pods. I noticed there is an implementation called ExternalEnvironmentFactory, but I am not sure how to use it. Is there a way to deploy a side container and use a different factory to run the Python harness process? What is the correct approach? Thanks, Andrea [Image removed by sender. 10x]<https://www.10xbanking.com/> 10x Banking Please consider the environment before printing this e-mail ********************************************************************************************************************************************************** This e-mail, including any attachments, is confidential. If you are not the intended recipient, you may not copy, forward, disclose or use any part of it. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete this message and any copies and attachments. 10x Future Technologies Services Limited is registered in England & Wales Registered Number: 11262659 Registered Address: 5 Howick Place, Victoria, London, SW1P 1WG [10x]<https://www.10xbanking.com/> 10x Banking Please consider the environment before printing this e-mail ********************************************************************************************************************************************************** This e-mail, including any attachments, is confidential. If you are not the intended recipient, you may not copy, forward, disclose or use any part of it. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete this message and any copies and attachments. 10x Future Technologies Services Limited is registered in England & Wales Registered Number: 11262659 Registered Address: 5 Howick Place, Victoria, London, SW1P 1WG
