Hey, sure, executing this as separate jobs works fine! For now, I just wanted to make sure I'm not missing something obvious. When ready for experiments, I will revisit this and try to avoid the file system.
Is there any description or issue related to the incremental rollout? Let me know if I can help in any way! Thanks, V. On 11 July 2014 18:01, Kostas Tzoumas <kostas.tzou...@tu-berlin.de> wrote: > As a first prototype, you can (using env.execute): > > - execute the bulk iteration as one job > - grab the number of elapsed iterations using the static variable that > Stephan mentioned > - execute the delta iteration as a second job if needed > > On Fri, Jul 11, 2014 at 5:06 PM, Stephan Ewen <se...@apache.org> wrote: > > > Hi! > > > > That is exactly the incremental rollout variant. > > > > You can do this with intermediate file writing, currently. Would that be > a > > way to start prototyping? For benchmarking, we can even hack a caching > > output and input format that keeps the data in the task manager's memory. > > > > Stephan > > >
