Re: Reading and writing to external services in DoFns

ds. However, if the index of previous records can fit into > memory on the nodes I would recommend to use a side input instead that you > do the check against in the DoFn. That should both be fast and work well in > streaming. > > Hope it helps. > > Br, > Vilhelm von

Reading and writing to external services in DoFns

Hi everyone, I'm researching how to handle a particular use case in Beam that I imagine is common, but that I haven't been able to find any agreed upon best way of doing yet. *Use case: *I'm processing a stream or batch of records with ids, and for each record I want to check whether I've ever

Re: Reprocessing historic data with streaming jobs

ssed promptly if you use a watermark based trigger; watermarks will be > held until the bounded source is fully processed. > > On Mon, May 1, 2017 at 9:29 AM, Lars BK <larsbkrog...@gmail.com> wrote: > >> I did not see Lukasz reply before I posted, and I will have

Re: Reprocessing historic data with streaming jobs

singTime.pastFirstElementInPane() >.plusDelayOf(TEN_MINUTES))) >.withAllowedLateness(Duration.minutes() >.accumulatingFiredPanes()) > > Thoughts ? > > Regards > JB > > On 05/01/2017 05:12 PM, Lars BK wrote: > &g

Re: Reprocessing historic data with streaming jobs

e, you don't want to re-consume the messages > from > the PubSub topic right ? So, you want to "hold" the PCollections for late > data > processing ? > > Regards > JB > > On 05/01/2017 04:15 PM, Lars BK wrote: > > Hi, > > > > Is there a preferred

Reprocessing historic data with streaming jobs

Hi, Is there a preferred way of approaching reprocessing historic data with streaming jobs? I want to pose this as a general question, but I'm working with Pubsub and Dataflow specifically. I am a fan of the idea of replaying/fast forwarding through historic data to reproduce results (as you

Re: Apache Beam Slack channel

net>: > Aviem already sent the invite on your gmail address. You should have it in > your > mbox. > > Can you check please ? > > Thanks, > Regards > JB > > On 04/30/2017 05:31 PM, Lars BK wrote: > > Yes please. > > > > Thanks, > > Lars

Re: Apache Beam Slack channel

Yes please. Thanks, Lars On Sun, Apr 30, 2017 at 5:22 PM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Hi > > Should I use your Gmail address ? > > Regards > JB > On Apr 30, 2017, at 16:58, Lars BK <larsbkrog...@gmail.com> wrote: >> >> In

Re: Apache Beam Slack channel

Hi, I would like to request an invite to the Slack team too. Regards, Lars On 2017-04-28 15:40 (+0200), Ismaël Mejía wrote: > Done. > > On Fri, Apr 28, 2017 at 3:32 PM, Andrew Psaltis > wrote: > > > Please add me as well. Thanks, > > > > On