Re: How does Beam set up the bundle size in streaming mode (like Pub/Sub)?

[Kenneth Knowles]

Hi Derek,

I agree, that phrasing is simply incorrect and mightily confusing. Would
you be up for filing a JIRA with some hyperlinks to the pages that say that?

Kenn

On Sun, Oct 22, 2017 at 9:54 AM, Derek Hao Hu 
wrote:

> Thanks Kenneth! I sort of feel the notions of bundles and windows are a
> bit confusing in Beam.
>
> For example, here is what the Beam Programming Guide says:
>
> "When performing an operation that groups elements in an unbounded
> PCollection, Beam requires a concept called *windowing* to divide a
> continuously updating data set into logical windows of finite size. Beam
> processes each window as a bundle, and processing continues as the data set
> is generated."
>
> So then I would assume "bundles" and "windows" are terms that can be used
> almost interchangeably.
>
> Do you know if there's any good posts / documentations about bundles?
>
> Cheers,
>
> Derek
>
> On Wed, Oct 18, 2017 at 6:59 AM, Kenneth Knowles  wrote:
>
>> Bundles are decidedly not windows, so let's keep the two terms separate.
>> It sounds like you are asking about bundles.
>>
>> The bundle size is a performance tuning parameter and is arbitrarily
>> chosen arbitrarily and dynamically chosen by a runner. The runner chooses
>> based on its best effort to amortize @StartBundle/@FinishBundle operations
>> across multiple @ProcessElement/@OnTimer calls. Your code must yield
>> correct results for for any bundling - you should be implementing
>> per-element logic, where @StartBundle/@FinishBundle are implementation
>> details.
>>
>> Kenn
>>
>> On Tue, Oct 17, 2017 at 5:37 PM, Derek Hao Hu 
>> wrote:
>>
>>> Hi,
>>>
>>> Is there any more detailed explanation on how Beam chooses the window
>>> size (bundle size) in streaming mode? It seems there is no clear answer in
>>> the [Beam Programming Guide](https://beam.apache.org
>>> /documentation/programming-guide/) and I can't find how PubsubIO
>>> implements this windowing strategy as well. :(
>>>
>>> Could someone kindly provide some pointers? Thanks!
>>> --
>>> Derek Hao Hu
>>>
>>> Software Engineer | Snapchat
>>> Snap Inc.
>>>
>>
>>
>
>
> --
> Derek Hao Hu
>
> Software Engineer | Snapchat
> Snap Inc.
>

Re: Regarding Beam Slack Channel

[Jean-Baptiste Onofré]

Done,

you should have received an invite.

Welcome aboard !

Regards
JB

On 10/26/2017 02:20 PM, Lovis Dahl wrote:

I'd love an invite too!

/L

On Sun, 15 Oct 2017 at 08:09 Jean-Baptiste Onofré > wrote:


Invitation sent,

Welcome !
Regards
JB

On 10/15/2017 07:44 AM, Sushil Ks wrote:
 > +1
 > Kindly add me to beam slack channel.
 >
 >
 > On Oct 14, 2017 5:02 AM, "Lukasz Cwik" 
 > >> wrote:
 >
 >     Invite sent, welcome.
 >
 >     On Fri, Oct 13, 2017 at 3:07 PM, NerdyNick 
 >     >> wrote:
 >
 >         Hello
 >
 >         Can someone please add me to the Beam slack channel?
 >
 >         Thanks.
 >
 >

--
Jean-Baptiste Onofré
jbono...@apache.org 
http://blog.nanthrax.net
Talend - http://www.talend.com

--
Lovis Dahl
Backend engineer, Greta.io


--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Regarding Beam Slack Channel

[Lovis Dahl]

I'd love an invite too!

/L

On Sun, 15 Oct 2017 at 08:09 Jean-Baptiste Onofré  wrote:

> Invitation sent,
>
> Welcome !
> Regards
> JB
>
> On 10/15/2017 07:44 AM, Sushil Ks wrote:
> > +1
> > Kindly add me to beam slack channel.
> >
> >
> > On Oct 14, 2017 5:02 AM, "Lukasz Cwik"  > > wrote:
> >
> > Invite sent, welcome.
> >
> > On Fri, Oct 13, 2017 at 3:07 PM, NerdyNick  > > wrote:
> >
> > Hello
> >
> > Can someone please add me to the Beam slack channel?
> >
> > Thanks.
> >
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
-- 
Lovis Dahl
Backend engineer, Greta.io