Re: Conceptually, what are bundles?

2017-01-26 Thread Jean-Baptiste Onofré
It makes sense. Agreed. Regards JB On 01/25/2017 08:34 PM, Kenneth Knowles wrote: There's actually not a JIRA filed beyond BEAM-25 for what Eugene is referring to. Context: Prior to windowing and streaming it was safe to buffer elements in @ProcessElement and then actually perform output in

Re: Conceptually, what are bundles?

2017-01-26 Thread Etienne Chauchot
Le 25/01/2017 à 20:34, Kenneth Knowles a écrit : There's actually not a JIRA filed beyond BEAM-25 for what Eugene is referring to. Context: Prior to windowing and streaming it was safe to buffer elements in @ProcessElement and then actually perform output in @FinishBundle. This pattern is only

Re: Conceptually, what are bundles?

2017-01-25 Thread Eugene Kirpichov
One more thing. I think ideally, bundles should not leak into the model at all - e.g. ideally, startBundle/finishBundle methods in DoFn should not exist. They interact poorly with windowing. The proper way to address what is commonly done in these methods is either Setup/Teardown methods, or a

Re: Conceptually, what are bundles?

2017-01-25 Thread Amit Sela
On Wed, Jan 25, 2017 at 8:23 PM Thomas Groh wrote: > I have a couple of points in addition to what Robert said > > Runners are permitted to determine bundle sizes as appropriate to their > implementation, so long as bundles are atomically committed. The contents > of a

Re: Conceptually, what are bundles?

2017-01-25 Thread Thomas Groh
I have a couple of points in addition to what Robert said Runners are permitted to determine bundle sizes as appropriate to their implementation, so long as bundles are atomically committed. The contents of a PCollection are independent of the bundling of that PCollection. Runners can process