Hi Andrea,
You could use the new worker_pool_main.py which was developed for the
Kubernetes use case. It works together with the external environment
factory.
Cheers,
Max
On 11.09.19 18:51, Lukasz Cwik wrote:
Yes.
On Wed, Sep 11, 2019 at 3:12 AM Andrea Medeghini
<[email protected]
<mailto:[email protected]>> wrote:
Hi Lukasz,____
Thanks for your answer. ____
I had a look at the ProcessEnvironmentFactory, and I understand that
the factory is expecting to run the boot process which is in the
python container (source file boot.go). Is that right?____
__ __
Best,____
Andrea____
__ __
*From: *Lukasz Cwik <[email protected] <mailto:[email protected]>>
*Reply-To: *"[email protected] <mailto:[email protected]>"
<[email protected] <mailto:[email protected]>>
*Date: *Tuesday, 10 September 2019 at 23:19
*To: *user <[email protected] <mailto:[email protected]>>
*Subject: *Re: How do I run Beam Python pipelines using Flink
deployed on Kubernetes?____
__ __
[CAUTION - EXTERNAL EMAIL]____
External environments are mainly used for testing since they
represent environments that are already running. There are other
clever uses of it as well but unlikely what your looking for. ____
__ __
The docker way will be nice once there is integration between the
FlinkRunner and its supported cluster managers to automatically
manage the SDK container instances. Until that exists you could look
at using the process environment (it launches the SDK as a child
process beside the Flink workers).____
__ __
On Tue, Sep 10, 2019 at 4:38 AM Andrea Medeghini
<[email protected]
<mailto:[email protected]>> wrote:____
Hello,____
Can I run Beam Python pipelines with Flink deployed on
Kubernetes?____
____
I have successfully managed to run a Beam Python pipeline using
the Portable runner and the job service pointing to a local
Flink server running as Docker containers.____
I was able to achieve that mounting the Docker socket in my
Flink container and running Flink as root, so the class
DockerEnvironmentFactory can create the Python harness
container. ____
Unfortunately, I can't use the same solution when Flink is
running in Kubernetes. Moreover, I don't want to create the
Python harness container using the Docker command from my pods.____
I noticed there is an implementation called
ExternalEnvironmentFactory, but I am not sure how to use it.____
Is there a way to deploy a side container and use a different
factory to run the Python harness process?____
What is the correct approach?____
____
Thanks,____
Andrea____
____
____
__ __
Image removed by sender. 10x <https://www.10xbanking.com/>____
10x Banking
Please consider the environment before printing this e-mail
**********************************************************************************************************************************************************
This e-mail, including any attachments, is confidential. If you
are not the intended recipient, you may not copy, forward,
disclose or use any part of it. If you have received this e-mail
in error, please notify the sender immediately by return e-mail
and delete this message and any copies and attachments.
10x Future Technologies Services Limited is registered in
England & Wales
Registered Number: 11262659
Registered Address: 5 Howick Place, Victoria, London, SW1P 1WG ____
10x <https://www.10xbanking.com/>
10x Banking
Please consider the environment before printing this e-mail
**********************************************************************************************************************************************************
This e-mail, including any attachments, is confidential. If you are
not the intended recipient, you may not copy, forward, disclose or
use any part of it. If you have received this e-mail in error,
please notify the sender immediately by return e-mail and delete
this message and any copies and attachments.
10x Future Technologies Services Limited is registered in England &
Wales
Registered Number: 11262659
Registered Address: 5 Howick Place, Victoria, London, SW1P 1WG
