Re: IO Authoring Guide - first draft

Hi Stephen, Great work ! I will take a look during the week end and will provide feedbacks. Thanks ! Regards JB On 01/28/2017 02:29 AM, Stephen Sisk wrote: Here's the doc I've been working on: https://docs.google.com/document/d/1nGGP2sLb5fLamB_dnkHVHC8BVjDD_SE46mQPIPkK5cQ/edit?usp=sharing

IO Authoring Guide - first draft

Here's the doc I've been working on: https://docs.google.com/document/d/1nGGP2sLb5fLamB_dnkHVHC8BVjDD_SE46mQPIPkK5cQ/edit?usp=sharing The general purpose of the doc is to share high level design thoughts and process for authoring an IO transform. Topics include: * List of example IOs that people

Re: [ANNOUNCEMENT] New committers, January 2017 edition!

Thanks all! And, I am very exited to join and look forward. On Fri, Jan 27, 2017 at 9:40 AM, Thomas Groh wrote: > Congratulations all! > > On Fri, Jan 27, 2017 at 9:34 AM, Chamikara Jayalath > wrote: > > > Congrats all !! :) > > > > - Cham > > >

[VOTE] Apache Beam, version 0.5.0, release candidate #1

Hi everyone, Please review and vote on the release candidate #1 for the version 0.5.0 as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes

Re: [BEAM-135] Utilities for "batching" elements in a DoFn

It makes sense with the PTransform, less invasive, but the user has to define to define two functions (one perElement, the other perBatch). I like the DoFn approach with annotations, and I would do the same for the batch. The "trigger/window" is a key part as well, as even if the batch is not

Re: [BEAM-135] Utilities for "batching" elements in a DoFn

On Fri, Jan 27, 2017 at 6:55 AM, Etienne Chauchot wrote: > Hi Robert, > > Le 26/01/2017 à 18:17, Robert Bradshaw a écrit : >> >> First off, let me say that a *correctly* batching DoFn is a lot of >> value, especially because it's (too) easy to (often unknowingly) >> implement

Re: [ANNOUNCEMENT] New committers, January 2017 edition!

Congratulations all! On Fri, Jan 27, 2017 at 9:34 AM, Chamikara Jayalath wrote: > Congrats all !! :) > > - Cham > > On Fri, Jan 27, 2017 at 4:13 AM Stas Levin wrote: > > > Thanks all, glad to be joining! > > > > On Fri, Jan 27, 2017, 13:07 Aljoscha

Re: [ANNOUNCEMENT] New committers, January 2017 edition!

Congrats all !! :) - Cham On Fri, Jan 27, 2017 at 4:13 AM Stas Levin wrote: > Thanks all, glad to be joining! > > On Fri, Jan 27, 2017, 13:07 Aljoscha Krettek wrote: > > > Welcome aboard! :-) > > > > On Fri, 27 Jan 2017 at 11:27 Ismaël Mejía

Re: Consistent Placement

Hi Jesse, +1 to Dan - I think it makes sense to return the specific type corresponding to the given transform (e.g. returning Combine.Globally from Combine.globally()), because it very often serves as a builder for adding more parameters. You mentioned users extending transform classes. I

Re: Consistent Placement

On Fri, Jan 27, 2017 at 8:41 AM, Jesse Anderson wrote: > @dan I thought you were talking about the transform class definition: > public static class GroupedValues > extends PTransform > >, >

Re: Consistent Placement

@dan I thought you were talking about the transform class definition: public static class GroupedValues extends PTransform >, PCollection>> { On Fri, Jan 27, 2017 at 11:30 AM Dan Halperin

Re: Consistent Placement

Hi Jesse, can you specifically say which functions on Combine and Count you're thinking of? I believe these transforms are consistent with the "principle of least visibility" -- make nothing more public than it needs to be. Look at Combine.globally

Re: [BEAM-135] Utilities for "batching" elements in a DoFn

Hi, Indeed, I did not want to be invasive in DoFn either. So I chose to implement it as a PTransform. Please be aware that it is just the very beginning, it is still in the simple naive approach (no window and no buffering trans-bundle support), I plan to use state API to buffer

Re: [BEAM-135] Utilities for "batching" elements in a DoFn

Hi Robert, Le 26/01/2017 à 18:17, Robert Bradshaw a écrit : First off, let me say that a *correctly* batching DoFn is a lot of value, especially because it's (too) easy to (often unknowingly) implement it incorrectly. I definitely agree, I put a similar comment in another email. As an example

Re: Consistent Placement

One con to making transform classes be private would be that it is a breaking change. If anyone uses that class directly or extends that class, we'd be breaking that. On Fri, Jan 27, 2017 at 9:37 AM Jesse Anderson wrote: > Continuing a discussion

Consistent Placement

Continuing a discussion Dan, Kenn, and I were having here since the bug is closed. They pointed out three things: - Where the private constructor gets placed in the class - Where the code samples of how to use the class get placed (in the

Re: How to implement Timer in runner

I think the biggest challenge here in terms of using external store is to be able to (periodically) scan through keys even if their corresponding values are not updated and check if they should be fired/GCed. The implementation I'm working on in Spark runner does that by using Spark's

回复：How to implement Timer in runner

Thanks for the reply.Maybe we need a external priority queue.Happy Chinese New Year! --发件人：Aljoscha Krettek 发送时间：2017年1月25日(星期三) 18:38收件人：dev ; lzljs3620320 ;

Re: [ANNOUNCEMENT] New committers, January 2017 edition!

Thanks all, glad to be joining! On Fri, Jan 27, 2017, 13:07 Aljoscha Krettek wrote: > Welcome aboard! :-) > > On Fri, 27 Jan 2017 at 11:27 Ismaël Mejía wrote: > > > Congratulations, well deserved guys ! > > > > > > On Fri, Jan 27, 2017 at 9:28 AM, Amit

Re: [ANNOUNCEMENT] New committers, January 2017 edition!

Welcome aboard! :-) On Fri, 27 Jan 2017 at 11:27 Ismaël Mejía wrote: > Congratulations, well deserved guys ! > > > On Fri, Jan 27, 2017 at 9:28 AM, Amit Sela wrote: > > > Welcome and congratulations to all! > > > > On Fri, Jan 27, 2017, 10:12 Ahmet

Re: [ANNOUNCEMENT] New committers, January 2017 edition!

Congratulations, well deserved guys ! On Fri, Jan 27, 2017 at 9:28 AM, Amit Sela wrote: > Welcome and congratulations to all! > > On Fri, Jan 27, 2017, 10:12 Ahmet Altay wrote: > > > Thank you all! And congratulations to other new committers. >

Re: Release 0.5.0

Hi, I will be the release manager for the 0.5.0. I plan to start the release process pretty soon (later today or tomorrow morning my time). It seems we don't have any blocking Jira targeted on 0.5.0. Please, if you have some pending Jira or code to merge, let me know asap. Thanks ! Regards

Re: [ANNOUNCEMENT] New committers, January 2017 edition!

Welcome and congratulations to all! On Fri, Jan 27, 2017, 10:12 Ahmet Altay wrote: > Thank you all! And congratulations to other new committers. > > Ahmet > > On Thu, Jan 26, 2017 at 9:45 PM, Kobi Salant > wrote: > > > Congrats! Well deserved

Re: [ANNOUNCEMENT] New committers, January 2017 edition!

Thank you all! And congratulations to other new committers. Ahmet On Thu, Jan 26, 2017 at 9:45 PM, Kobi Salant wrote: > Congrats! Well deserved Stas > > בתאריך 27 בינו' 2017 7:26,‏ "Frances Perry" כתב: > > > Woohoo! Congrats ;-) > > > > On Thu, Jan