Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

hough. I only > need > > > > to do the last touch ups. > > > > > > > > On Tue, Mar 28, 2017, at 09:16, JingsongLee wrote: > > > > > Hi Aljoscha, > > > > > I would like to work on the Flink runner with you. > > > > > > > > > &

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

st,JingsongLee---------------- > ----------From:Jean-Baptiste > > > > Onofré Time:2017 Mar 28 (Tue) 14:04To:dev > > > > Subject:Re: Call for help: let's add Splittable > > > DoFn > > > > to Spark, Flink and

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

ngsongLee------From:Jean-Baptiste > > > Onofré Time:2017 Mar 28 (Tue) 14:04To:dev > > > Subject:Re: Call for help: let's add Splittable > > DoFn > > > to Spark, Flink and Apex runners > > > Hi A

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

8 (Tue) 14:04To:dev > > Subject:Re: Call for help: let's add Splittable > DoFn > > to Spark, Flink and Apex runners > > Hi Aljoscha, > > > > do you need some help on this ? > > > > Regards > > JB > > > > On 03/28/2017 08:00 AM, Aljoscha

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

ngsongLee--From:Jean-Baptiste > Onofré Time:2017 Mar 28 (Tue) 14:04To:dev > Subject:Re: Call for help: let's add Splittable DoFn > to Spark, Flink and Apex runners > Hi Aljoscha, > > do you need some help on this ? > > Regards > JB > > On 03/28/2017 08:00 AM

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Hi Aljoscha, I would like to work on the Flink runner with you. Best,JingsongLee--From:Jean-Baptiste Onofré Time:2017 Mar 28 (Tue) 14:04To:dev Subject:Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runne

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Hi Aljoscha, do you need some help on this ? Regards JB On 03/28/2017 08:00 AM, Aljoscha Krettek wrote: Hi, sorry for being so slow but I’m currently traveling. The Flink code works but I think it could benefit from some refactoring to make the code nice and maintainable. Best, Aljoscha On

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Hi, sorry for being so slow but I’m currently traveling. The Flink code works but I think it could benefit from some refactoring to make the code nice and maintainable. Best, Aljoscha On Tue, Mar 28, 2017, at 07:40, Jean-Baptiste Onofré wrote: > I add myself on the Spark runner. > > Regards > J

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

I add myself on the Spark runner. Regards JB On 03/27/2017 08:18 PM, Eugene Kirpichov wrote: Hi all, Let's continue the ~bi-weekly sync-ups about state of SDF support in Spark/Flink/Apex runners. Spark: Amit, Aviem, Ismaël - when would be a good time for you; does same time work (8am PST this

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Hi all, Let's continue the ~bi-weekly sync-ups about state of SDF support in Spark/Flink/Apex runners. Spark: Amit, Aviem, Ismaël - when would be a good time for you; does same time work (8am PST this Friday)? Who else would like to join? Flink: I pinged the PR, but - Aljoscha, do you think it's

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Meeting notes: Me and Thomas had a video call and we pretty much walked through the implementation of SDF in the runner-agnostic part and in the direct runner. Flink and Apex are pretty similar, so likely https://github.com/apache/beam/pull/2235 (the Flink PR) will give a very good guideline as to

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Thomas - yes, 9:30 works, shall we do that? JB - excellent! You can start experimenting already, using direct runner! On Mon, Mar 20, 2017, 2:26 AM Jean-Baptiste Onofré wrote: > Hi Eugene, > > Thanks for the meeting notes ! > > I will be in the next call and Ismaël also provided to me some upda

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Hi Eugene, Thanks for the meeting notes ! I will be in the next call and Ismaël also provided to me some updates. I will sync with Amit on Spark runner and start to experiment and test SDF on the JMS IO. Thanks ! Regards JB On 03/17/2017 04:36 PM, Eugene Kirpichov wrote: Meeting notes from

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Meeting notes from today's call with Amit, Aviem and Ismaël: Spark has 2 types of stateful operators; a cheap one intended for updating elements (works with state but not with timers) and an expensive one. I.e. there's no efficient direct counterpart to Beam's keyed state model. In implementation

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Yes, Monday morning works! How about also 8am PST, same Hangout link - does that work for you? On Fri, Mar 17, 2017 at 7:50 AM Thomas Weise wrote: > Eugene, > > I cannot make it for the call today. Would Monday morning work for you to > discuss the Apex changes? > > Thanks > > On Tue, Mar 14, 20

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Eugene, I cannot make it for the call today. Would Monday morning work for you to discuss the Apex changes? Thanks On Tue, Mar 14, 2017 at 7:27 PM, Eugene Kirpichov < kirpic...@google.com.invalid> wrote: > Hi! Please feel free to join this call, but I think we'd be mostly > discussing how to do

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Great! so we'll use the hangout you added here, see you then. On Wed, Mar 15, 2017 at 7:22 PM Eugene Kirpichov wrote: > Amit - 8am is fine with me, let's do that. > > On Wed, Mar 15, 2017 at 6:00 AM Jean-Baptiste Onofré > wrote: > > > Hi, > > > > Anyway, I hope it will result with some notes on

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Amit - 8am is fine with me, let's do that. On Wed, Mar 15, 2017 at 6:00 AM Jean-Baptiste Onofré wrote: > Hi, > > Anyway, I hope it will result with some notes on the mailing list as it > could be > helpful. > > I'm not against a video call to move forward, but, from ma community > perspective,

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Hi, Anyway, I hope it will result with some notes on the mailing list as it could be helpful. I'm not against a video call to move forward, but, from ma community perspective, we should always provide minute notes on the mailing list. Unfortunately, next Friday, I will still be in China, s

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

I have dinner at 9am.. which doesn't sound like a real thing if you forget about timezones J How about 8am ? or something later like 12pm mid-day ? Apex can take the 9am time slot ;-) On Wed, Mar 15, 2017 at 4:28 AM Eugene Kirpichov wrote: > Hi! Please feel free to join this call, but I think we

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Hi! Please feel free to join this call, but I think we'd be mostly discussing how to do it in the Spark runner in particular; so we'll probably need another call for Apex anyway. On Tue, Mar 14, 2017 at 6:54 PM Thomas Weise wrote: > Hi Eugene, > > This would work for me also. Please let me know

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Hi Eugene, This would work for me also. Please let me know if you want to keep the Apex related discussion separate or want me to join this call. Thanks, Thomas On Tue, Mar 14, 2017 at 1:56 PM, Eugene Kirpichov < kirpic...@google.com.invalid> wrote: > Sure, Friday morning sounds good. How abou

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Sure, Friday morning sounds good. How about 9am Friday PST, at videocall by link https://hangouts.google.com/hangouts/_/google.com/splittabledofn ? On Mon, Mar 13, 2017 at 10:30 PM Amit Sela wrote: > PST mornings are better, because they are evening/nights for me. Friday > would work-out best fo

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

PST mornings are better, because they are evening/nights for me. Friday would work-out best for me. On Mon, Mar 13, 2017 at 11:46 PM Eugene Kirpichov wrote: > Awesome!!! > > Amit - remind me your time zone? JB, do you want to join? > I'm free this week all afternoons (say after 2pm) in Pacific T

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Awesome!!! Amit - remind me your time zone? JB, do you want to join? I'm free this week all afternoons (say after 2pm) in Pacific Time, and mornings of Wed & Fri. We'll probably need half an hour to an hour. On Mon, Mar 13, 2017 at 1:29 PM Aljoscha Krettek wrote: > I whipped up a quick version

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

I whipped up a quick version for Flink that seems to work: https://github.com/apache/beam/pull/2235 There are still two failing tests, as described in the PR. On Mon, Mar 13, 2017, at 20:10, Amit Sela wrote: > +1 for a video call. I think it should be pretty straight forward for the > Spark runn

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

+1 for a video call. I think it should be pretty straight forward for the Spark runner after the work on read from UnboundedSource and after GroupAlsoByWindow, but from my experience such a call could move us forward fast enough. On Mon, Mar 13, 2017, 20:37 Eugene Kirpichov wrote: > Hi all, > >

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Hi all, Let us continue working on this. I am back from various travels and am eager to help. Amit, JB - would you like to perhaps have a videocall to hash this out for the Spark runner? Aljoscha - are the necessary Flink changes done / or is the need for them obviated by using the (existing) ru

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

I recommend proceeding with the runner-facing state & timer APIs; they are lower-level and more appropriate for this. All runners provide them or use runners/core implementations, as they are needed for triggering. On Wed, Feb 8, 2017 at 10:34 AM, Eugene Kirpichov wrote: > Thanks Aljoscha! > > M

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Thanks Aljoscha! Minor note: I'm not familiar with what level of support for timers Flink currently has - however SDF in Direct and Dataflow runner currently does not use the user-facing state/timer APIs - rather, it uses the runner-facing APIs (StateInternals and TimerInternals) - perhaps Flink a

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Thanks for the motivation, Eugene! :-) I've wanted to do this for a while now but was waiting for the Flink 1.2 release (which happened this week)! There's some prerequisite work to be done on the Flink runner: we'll move to the new timer interfaces introduced in Flink 1.2 and implement support fo

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Thanks! Looking forward to this work. On Wed, Feb 8, 2017 at 3:50 AM Jean-Baptiste Onofré wrote: > Thanks for the update Eugene. > > I will work on the spark runner with Amit. > > Regards > JB > > On Feb 7, 2017, 19:12, at 19:12, Eugene Kirpichov > wrote: > >Hello, > > > >I'm almost done adding

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Thanks for the update Eugene. I will work on the spark runner with Amit. Regards JB On Feb 7, 2017, 19:12, at 19:12, Eugene Kirpichov wrote: >Hello, > >I'm almost done adding support for Splittable DoFn >http://s.apache.org/splittable-do-fn to Dataflow streaming runner*, and >very excited abou

Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Hello, I'm almost done adding support for Splittable DoFn http://s.apache.org/splittable-do-fn to Dataflow streaming runner*, and very excited about that. There's only 1 PR remaining, plus enabling some tests. * (batch runner is much harder because it's