Re: Out of Memory errors

Hi, AFAIU you are using dataflow runner right ? Regards JB On 06/09/2018 00:03, asharma...@gmail.com wrote: > Hi > > I am doing some processing and creating local list inside a Pardo function. > Total data size is 2 GB which is executing inside a single instance. I am > running this code on h

Re: Slides from OSCON

Actually I found the speaker release, looks like I can upload it to YouTube - https://youtu.be/ZGyx4GuGEj4 On Tue, Sep 4, 2018 at 10:42 AM Holden Karau wrote: > I think I have permission to share it on YouTube, I'll ask the O'Reilly > folks when I see them @ Strata next week :) > > On Tue, Sep 4

Uber Jar: DataflowRunner + GCP I/O + Pipeline

I have had a few users ask me directly how do I create an uber jar containing the DataflowRunner, GCP I/O and their pipeline. I wanted to shared this example Maven project[1] that created an executable uber jar containing those components and the Apache Beam examples. The instructions in README.md

Out of Memory errors

Hi I am doing some processing and creating local list inside a Pardo function. Total data size is 2 GB which is executing inside a single instance. I am running this code on highmem-64 machine and It gives this error Shutting down JVM after 8 consecutive periods of measured GC thrashing. Memory

Re: INFO:root:Executing Error when executing a pipeline on dataflow

Ella, I believe you are using version 2.0.0. Use of --download flag is fixed at head. (I do not recall the exact version of the fix, could be 2.2 or 2.3). If possible please try to use a newer version of Beam. Also, as Luke suggested, we would welcome any contributions to the documentation. On Fr

[Call for items] September Beam Newsletter

Hi Beamers: Here's [1] the template for the September Beam Newsletter! *Add the highlights from August to now (or planned events and talks) that you want to share with the community by 9/8 11:59 p.m