Severyn Parkhomenko created BEAM-7848:
-----------------------------------------
Summary: Add possibility to manage quantity of instances (threads)
per worker in Python SDK
Key: BEAM-7848
URL: https://issues.apache.org/jira/browse/BEAM-7848
Project: Beam
Issue Type: Improvement
Components: runner-dataflow
Environment: Python SDK
ApacheBeam version==2.13.0
worker_type==n1-standard-4
Reporter: Severyn Parkhomenko
Attachments: Selection_042.png
I'm developing a streaming pipeline with big memory consumption in one of the
PTransforms.
After some period after starting this pipeline fails without any specific logs.
!image-2019-07-30-13-28-34-007.png!
It looks like, that it happens because of OutOfMemory.
It would be great to set a limit of threads that will be used in a single
worker to control memory load.
I found such option in JAVA SDK (--_numberOfWorkerHarnessThreads_), but in
Python SDK it is absent
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
