Re: [DISCUSS] FLIP-14: Loops API and Termination

That was fast! Seems to be working. Thank you Fabian! > On 17 Nov 2016, at 13:58, Fabian Hueske wrote: > > Hi Paris, > > just gave you the permissions (I hope). > Let me know if something does not work. > > Cheers, Fabian > > 2016-11-17 13:48 GMT+01:00 Paris Carbone

Re: [DISCUSS] FLIP-14: Loops API and Termination

Hi Paris, just gave you the permissions (I hope). Let me know if something does not work. Cheers, Fabian 2016-11-17 13:48 GMT+01:00 Paris Carbone : > We do not have to schedule this for an early Flink release, just saying. > I would just like to get the changes out and you

Re: [DISCUSS] FLIP-14: Loops API and Termination

We do not have to schedule this for an early Flink release, just saying. I would just like to get the changes out and you people can review it and integrate it anytime at your own pace. Who is the admin of the wiki? It would be nice to get write access. > On 17 Nov 2016, at 13:45, Paris Carbone

Re: [DISCUSS] FLIP-14: Loops API and Termination

Sounds like a plan! Can someone grant me access to write in the wiki please? My username is “senorcarbone”. Paris > On 16 Nov 2016, at 14:30, Gyula Fóra wrote: > > I am not completely sure whether we should deprecate the old API for 1.2 or > remove it completely.

Re: [DISCUSS] FLIP-14: Loops API and Termination

I am not completely sure whether we should deprecate the old API for 1.2 or remove it completely. Personally I am in favor of removing it, I don't think it is a huge burden to move to the new one if it makes for a much nicer user experience. I think you can go ahead add the FLIP to the wiki and

Re: [DISCUSS] FLIP-14: Loops API and Termination

Thanks for reviewing, Gyula. One thing that is still up to discussion is whether we should remove completely the old iterations API or simply mark it as deprecated till v2.0. Also, not sure what is the best process now. We have the changes ready. Should I copy the FLIP to the wiki and trigger

Re: [DISCUSS] FLIP-14: Loops API and Termination

Hi Paris, I like the proposed changes to the iteration API, this cleans up things in the Java API without any strict restriction I think (it was never a problem in the Scala API). The termination algorithm based on the proposed scoped loops seems to be fairly simple and looks good :) Cheers,

Re: [DISCUSS] FLIP-14: Loops API and Termination

That would be great Shi! Let's take that offline. Anyone else interested in the iteration changes? It would be nice to incorporate these to v1.2 if possible so I count on your review asap. cheers, Paris On Nov 14, 2016, at 2:59 AM, xiaogang.sxg

Re: [DISCUSS] FLIP-14: Loops API and Termination

Hi Shi, Naiad/Timely Dataflow and other projects use global coordination which is very convenient for asynchronous progress tracking in general but it has some downsides in a production systems that count on in-flight transactional control mechanisms and rollback recovery guarantees. This is

Re: [DISCUSS] FLIP-14: Loops API and Termination

Hi, Fouad Thank you for the explanation. Now the centralized method seems correct to me. The passing of StatusUpdate events will lead to synchronous iterations and we are using the information in each iterations to terminate the computation. Actually, i prefer the centralized method because in

Re: [DISCUSS] FLIP-14: Loops API and Termination

Hi Shi, It seems that you are referring to the centralized algorithm which is no longer the proposed version. In the decentralized version (check last doc) there is no master node or global coordination involved. Let us keep this discussion to the decentralized one if possible. To answer your

Re: [DISCUSS] FLIP-14: Loops API and Termination

Hi Paris I have several concerns about the correctness of the termination protocol. I think the termination protocol put an end to the computation even when the computation has not converged. Suppose there exists a loop context constructed by a OP operator, a Head operator and a Tail operator