The two pull requests do not go all the way, unfortunately. They cover only the runtime, the API integration part is missing still, unfortunately...
On Mon, Jul 20, 2015 at 5:53 PM, Maximilian Michels <m...@apache.org> wrote: > You could do that but you might run into merge conflicts. Also keep in > mind that it is work in progress :) > > On Mon, Jul 20, 2015 at 4:15 PM, Maximilian Alber < > alber.maximil...@gmail.com> wrote: > >> Thanks! >> >> Ok, cool. If I would like to test it, I just need to merge those two pull >> requests into my current branch? >> >> Cheers, >> Max >> >> On Mon, Jul 20, 2015 at 4:02 PM, Maximilian Michels <m...@apache.org> >> wrote: >> >>> Now that makes more sense :) I thought by "nested iterations" you meant >>> iterations in Flink that can be nested, i.e. starting an iteration inside >>> an iteration. >>> >>> The caching/pinning of intermediate results is still a work in progress >>> in Flink. It is actually in a state where it could be merged but some >>> pending pull requests got delayed because priorities changed a bit. >>> >>> Essentially, we need to merge these two pull requests: >>> >>> https://github.com/apache/flink/pull/858 >>> This introduces a session management which allows to keep the >>> ExecutionGraph for the session. >>> >>> https://github.com/apache/flink/pull/640 >>> Implements the actual backtracking and caching of the results. >>> >>> Once these are in, we can change the Java/Scala API to support >>> backtracking. I don't exactly know how Spark's API does it but, essentially >>> it should work then by just creating new operations on an existing DataSet >>> and submit to the cluster again. >>> >>> Cheers, >>> Max >>> >>> On Mon, Jul 20, 2015 at 3:31 PM, Maximilian Alber < >>> alber.maximil...@gmail.com> wrote: >>> >>>> Oh sorry, my fault. When I wrote it, I had iterations in mind. >>>> >>>> What I actually wanted to say, how "resuming from intermediate >>>> results" will work with (non-nested) "non-Flink" iterations? And with >>>> iterations I mean something like this: >>>> >>>> while(...): >>>> - change params >>>> - submit to cluster >>>> >>>> where the executed Flink-program is more or less the same at each >>>> iterations. But with changing input sets, which are reused between >>>> different loop iterations. >>>> >>>> I might got something wrong, because in our group we mentioned caching >>>> a lá Spark for Flink and someone came up that "pinning" will do that. Is >>>> that somewhat right? >>>> >>>> Thanks and Cheers, >>>> Max >>>> >>>> On Mon, Jul 20, 2015 at 1:06 PM, Maximilian Michels <m...@apache.org> >>>> wrote: >>>> >>>>> "So it is up to debate how the support for resuming from intermediate >>>>> results will look like." -> What's the current state of that debate? >>>>> >>>>> Since there is no support for nested iterations that I know of, the >>>>> debate how intermediate results are integrated has not started yet. >>>>> >>>>> >>>>>> "Intermediate results are not produced within the iterations cycles." >>>>>> -> Ok, if there are none, what does it have to do with that debate? :-) >>>>>> >>>>> >>>>> I was referring to the existing support for intermediate results >>>>> within iterations. If we were to implement nested iterations, this could >>>>> (possibly) change. This is all very theoretical because there are no plans >>>>> to support nested iterations. >>>>> >>>>> Hope this clarifies. Otherwise, please restate your question because I >>>>> might have misunderstood. >>>>> >>>>> Cheers, >>>>> Max >>>>> >>>>> >>>>> On Mon, Jul 20, 2015 at 12:11 PM, Maximilian Alber < >>>>> alber.maximil...@gmail.com> wrote: >>>>> >>>>>> Thanks for the answer! But I need some clarification: >>>>>> >>>>>> "So it is up to debate how the support for resuming from intermediate >>>>>> results will look like." -> What's the current state of that debate? >>>>>> "Intermediate results are not produced within the iterations cycles." >>>>>> -> Ok, if there are none, what does it have to do with that debate? :-) >>>>>> >>>>>> Cheers, >>>>>> Max >>>>>> >>>>>> On Mon, Jul 20, 2015 at 10:50 AM, Maximilian Michels <m...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> Hi Max, >>>>>>> >>>>>>> You are right, there is no support for nested iterations yet. As far >>>>>>> as I know, there are no concrete plans to add support for it. So it is >>>>>>> up >>>>>>> to debate how the support for resuming from intermediate results will >>>>>>> look >>>>>>> like. Intermediate results are not produced within the iterations >>>>>>> cycles. >>>>>>> Same would be true for nested iterations. So the behavior for resuming >>>>>>> from >>>>>>> intermediate results should be alike for nested iterations. >>>>>>> >>>>>>> Cheers, >>>>>>> Max >>>>>>> >>>>>>> On Fri, Jul 17, 2015 at 4:26 PM, Maximilian Alber < >>>>>>> alber.maximil...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi Flinksters, >>>>>>>> >>>>>>>> as far as I know, there is still no support for nested iterations >>>>>>>> planned. Am I right? >>>>>>>> >>>>>>>> So my question is how such use cases should be handled in the >>>>>>>> future. >>>>>>>> More specific: when pinning/caching will be available, you suggest >>>>>>>> to use that feature and program in "Spark" style? Or is there some >>>>>>>> other, >>>>>>>> more flexible, mechanism planned for loops? >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Max >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
