I'd just like to close the loop. Josh, did you get an answer/guidance on
how to proceed with your pipeline?
Or maybe we'll need a new thread to figure that out : )

On Fri, Mar 9, 2018 at 1:39 PM Josh Ferge <josh.fe...@bounceexchange.com>

> Hello all:
> Our team has a pipeline that make external network calls. These pipelines
> are currently super slow, and the hypothesis is that they are slow because
> we are not threading for our network calls. The github issue below provides
> some discussion around this:
> https://github.com/apache/beam/pull/957
> In beam 1.0, there was IntraBundleParallelization, which helped with this.
> However, this was removed because it didn't comply with a few BEAM
> paradigms.
> Questions going forward:
> What is advised for jobs that make blocking network calls? It seems
> bundling the elements into groups of size X prior to passing to the DoFn,
> and managing the threading within the function might work. thoughts?
> Are these types of jobs even suitable for beam?
> Are there any plans to develop features that help with this?
> Thanks
Got feedback? go/pabloem-feedback

Reply via email to