Thanks for the reply and the help!
At the moment, we are thinking about keeping the Runner outside Beam for
the time being, as there are a number of extra ongoing developments going
regarding a few features for stream processing. I'll submit a PR for the
website in a short time with the details
With latest master the problem seems fixed. Unfortunately that was first
masked by build and docker issues. But I changed multiple things at once
after getting nowhere (the container build "succeeded" when in fact it did
not):
* Update to latest docker
* Increase docker disk space after seeing a
On Fri, Nov 16, 2018 at 3:36 PM Udi Meiri wrote:
> If you're working with Dataflow, it supports this flag:
> https://github.com/apache/beam/blob/75e9f645c7bec940b87b93f416823b020e4c5f69/sdks/python/apache_beam/options/pipeline_options.py#L602
> which uses guppy for heap profiling.
>
This is
On Fri, Nov 16, 2018 at 3:08 PM Ruoyun Huang wrote:
> Even tough the algorithm works on your batch system, did you verify
> anything that can rule out the possibility where it is the underlying ML
> package causing the memory leak?
>
It is possible that ML packages can cause the memory leak but
