Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Hi all, Splittable DoFn is now available in Dataflow streaming runner, as of https://github.com/apache/beam/pull/1898 ! Meanwhile, Flink support got disabled due to some churn as part of First Stable Release, but it should be not hard to fix - tracked in

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Nice work Aljoscha! Update WRT ApexRunner: We merged some prep work in the ParDoOperator to weed out remnants of OldDoFn. I have almost all the changes ready to add the support for Splittable DoFn (for most part those follow the Flink runner changes). The final piece missing to support the

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Hey all, The Flink PR has been merged, and thus - Flink becomes the first distributed runner to support Splittable DoFn!!! Thank you, Aljoscha! Looking forward to Spark and Apex, and continuing work on Dataflow. I'll also send proposals about a couple of new ideas related to SDF next week. On

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

I will not be able to make it this weekend, too busy. Let's chat at the beginning of next week and see what's on my plate. On Tue, Mar 28, 2017 at 5:44 PM Aljoscha Krettek wrote: > Thanks for the offers, guys! The code is finished, though. I only need > to do the last touch

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Thanks for the offers, guys! The code is finished, though. I only need to do the last touch ups. On Tue, Mar 28, 2017, at 09:16, JingsongLee wrote: > Hi Aljoscha, > I would like to work on the Flink runner with you. >

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Hi Aljoscha, I would like to work on the Flink runner with you. Best,JingsongLee--From:Jean-Baptiste Onofré Time:2017 Mar 28 (Tue) 14:04To:dev Subject:Re: Call for help: let's add Splittable

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

I add myself on the Spark runner. Regards JB On 03/27/2017 08:18 PM, Eugene Kirpichov wrote: Hi all, Let's continue the ~bi-weekly sync-ups about state of SDF support in Spark/Flink/Apex runners. Spark: Amit, Aviem, Ismaël - when would be a good time for you; does same time work (8am PST

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Hi all, Let's continue the ~bi-weekly sync-ups about state of SDF support in Spark/Flink/Apex runners. Spark: Amit, Aviem, Ismaël - when would be a good time for you; does same time work (8am PST this Friday)? Who else would like to join? Flink: I pinged the PR, but - Aljoscha, do you think

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Thomas - yes, 9:30 works, shall we do that? JB - excellent! You can start experimenting already, using direct runner! On Mon, Mar 20, 2017, 2:26 AM Jean-Baptiste Onofré wrote: > Hi Eugene, > > Thanks for the meeting notes ! > > I will be in the next call and Ismaël also

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Hi Eugene, Thanks for the meeting notes ! I will be in the next call and Ismaël also provided to me some updates. I will sync with Amit on Spark runner and start to experiment and test SDF on the JMS IO. Thanks ! Regards JB On 03/17/2017 04:36 PM, Eugene Kirpichov wrote: Meeting notes

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Meeting notes from today's call with Amit, Aviem and Ismaël: Spark has 2 types of stateful operators; a cheap one intended for updating elements (works with state but not with timers) and an expensive one. I.e. there's no efficient direct counterpart to Beam's keyed state model. In implementation

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Yes, Monday morning works! How about also 8am PST, same Hangout link - does that work for you? On Fri, Mar 17, 2017 at 7:50 AM Thomas Weise wrote: > Eugene, > > I cannot make it for the call today. Would Monday morning work for you to > discuss the Apex changes? > >

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Eugene, I cannot make it for the call today. Would Monday morning work for you to discuss the Apex changes? Thanks On Tue, Mar 14, 2017 at 7:27 PM, Eugene Kirpichov < kirpic...@google.com.invalid> wrote: > Hi! Please feel free to join this call, but I think we'd be mostly > discussing how to

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Great! so we'll use the hangout you added here, see you then. On Wed, Mar 15, 2017 at 7:22 PM Eugene Kirpichov wrote: > Amit - 8am is fine with me, let's do that. > > On Wed, Mar 15, 2017 at 6:00 AM Jean-Baptiste Onofré > wrote: > > > Hi, > > >

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Amit - 8am is fine with me, let's do that. On Wed, Mar 15, 2017 at 6:00 AM Jean-Baptiste Onofré wrote: > Hi, > > Anyway, I hope it will result with some notes on the mailing list as it > could be > helpful. > > I'm not against a video call to move forward, but, from ma

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Hi! Please feel free to join this call, but I think we'd be mostly discussing how to do it in the Spark runner in particular; so we'll probably need another call for Apex anyway. On Tue, Mar 14, 2017 at 6:54 PM Thomas Weise wrote: > Hi Eugene, > > This would work for me also.

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Hi Eugene, This would work for me also. Please let me know if you want to keep the Apex related discussion separate or want me to join this call. Thanks, Thomas On Tue, Mar 14, 2017 at 1:56 PM, Eugene Kirpichov < kirpic...@google.com.invalid> wrote: > Sure, Friday morning sounds good. How

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

PST mornings are better, because they are evening/nights for me. Friday would work-out best for me. On Mon, Mar 13, 2017 at 11:46 PM Eugene Kirpichov wrote: > Awesome!!! > > Amit - remind me your time zone? JB, do you want to join? > I'm free this week all

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Awesome!!! Amit - remind me your time zone? JB, do you want to join? I'm free this week all afternoons (say after 2pm) in Pacific Time, and mornings of Wed & Fri. We'll probably need half an hour to an hour. On Mon, Mar 13, 2017 at 1:29 PM Aljoscha Krettek wrote: > I

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

+1 for a video call. I think it should be pretty straight forward for the Spark runner after the work on read from UnboundedSource and after GroupAlsoByWindow, but from my experience such a call could move us forward fast enough. On Mon, Mar 13, 2017, 20:37 Eugene Kirpichov