[
https://issues.apache.org/jira/browse/BEAM-7848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Severyn Parkhomenko updated BEAM-7848:
--------------------------------------
Description:
I'm developing a streaming pipeline with big memory consumption in one of the
PTransforms.
After some period after starting this pipeline fails without any specific logs
(see attachment file)
It looks like, that it happens because of OutOfMemory.
It would be great to set a limit of threads that will be used in a single
worker to control memory load.
I found such option in JAVA SDK (--_numberOfWorkerHarnessThreads_), but in
Python SDK it is absent
was:
I'm developing a streaming pipeline with big memory consumption in one of the
PTransforms.
After some period after starting this pipeline fails without any specific logs.
!image-2019-07-30-13-28-34-007.png!
It looks like, that it happens because of OutOfMemory.
It would be great to set a limit of threads that will be used in a single
worker to control memory load.
I found such option in JAVA SDK (--_numberOfWorkerHarnessThreads_), but in
Python SDK it is absent
> Add possibility to manage quantity of instances (threads) per worker in
> Python SDK
> ----------------------------------------------------------------------------------
>
> Key: BEAM-7848
> URL: https://issues.apache.org/jira/browse/BEAM-7848
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Environment: Python SDK
> ApacheBeam version==2.13.0
> worker_type==n1-standard-4
> Reporter: Severyn Parkhomenko
> Priority: Major
> Attachments: Selection_042.png
>
>
> I'm developing a streaming pipeline with big memory consumption in one of the
> PTransforms.
> After some period after starting this pipeline fails without any specific
> logs (see attachment file)
> It looks like, that it happens because of OutOfMemory.
> It would be great to set a limit of threads that will be used in a single
> worker to control memory load.
> I found such option in JAVA SDK (--_numberOfWorkerHarnessThreads_), but in
> Python SDK it is absent
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
