Yes I was referring exactly to that, I was also involved in the Dopa project :) So, at the moment what is the suggested way to schedule jobs with Flink?
On Thu, Sep 18, 2014 at 9:48 AM, Robert Metzger <[email protected]> wrote: > Are you referring to this project? > https://github.com/TU-Berlin/dopa-scheduler > Its not an official repository of the Flink (Stratosphere) project. I > think a PhD student at TU Berlin created the code there. > > > > On Thu, Sep 11, 2014 at 4:29 PM, Flavio Pompermaier <[email protected]> > wrote: > >> Of course with Flink I could in principle execute almost everything with >> a single Job but, in general, I could write 2 different jobs and decide >> from time to time when the second should be run. >> That's why also metheor scripts are very useful :) >> From what I know there was a scheduler in Stratosphere that was using >> RabbitMQ, right? >> >> I would like to avoid to run linux commands and instead use some REST >> interface to trigger or schedule jobs. >> >> Best, >> Flavio >> >> >> On Thu, Sep 11, 2014 at 4:07 PM, Fabian Hueske <[email protected]> >> wrote: >> >>> Hi Flavio, >>> >>> what exactly do you mean by scheduling? >>> Do you want to run a job in regular intervals or execute a complex >>> workflow? >>> >>> Oozie is primarily used to orchestrate the execution of MapReduce >>> workflows. Since, MR is a rather inflexible programming model, complex >>> tasks need to split up into multiple dependent jobs that are executed once >>> their predecessors have finished. Oozie orchestrates this execution. >>> In Flink, you can build a complex analysis flow as a single program and >>> execute it. Hence, there is no need for a workflow scheduler such as Oozie. >>> >>> If you want to run a job in regular intervals, you can configure a cron >>> job, that starts executes the CLI client or implement a Java or Scala >>> program that submits jobs a certain points in time. >>> >>> Best, Fabian >>> >>> 2014-09-11 15:36 GMT+02:00 Flavio Pompermaier <[email protected]>: >>> >>>> Hi to all, >>>> >>>> I'd like to know if there's an example of how to schedule a Job in >>>> Flink. >>>> Do we still need something like Oozie or Quartz or we can avoid them? >>>> >>>> Best, >>>> Flavio >>>> >>> >>
