I have problems using Oozie. For example it doesn't sustain spark context like ooyola job server does. Other than GUI interfaces like HUE it's hard to work with - scoozie stopped in development year ago(I spoke with creator) and oozie xml very hard to write. Oozie still have all documentation and code in MR model rather than in yarn model. And based on it's current speed of development I can't expect radical changes in nearest future. There is no "Databricks" for oozie, which would have people on salary to develop this kind of radical changes. It's dinosaur.
Reunold, can you help finding this doc? Do you mean just pipelining spark code or additional logic of persistence tasks, job server, task retry, data availability and extra? 2014-09-17 11:21 GMT+04:00 Reynold Xin <r...@databricks.com>: > Hi Egor, > > I think the design doc for the pipeline feature has been posted. > > For the workflow, I believe Oozie actually works fine with Spark if you > want some external workflow system. Do you have any trouble using that? > > > On Tue, Sep 16, 2014 at 11:45 PM, Egor Pahomov <pahomov.e...@gmail.com> > wrote: > >> There are two things we(Yandex) miss in Spark: MLlib good abstractions and >> good workflow job scheduler. From threads "Adding abstraction in MlLib" >> and >> "[mllib] State of Multi-Model training" I got the idea, that databricks >> working on it and we should wait until first post doc, which would lead >> us. >> What about workflow scheduler? Is there anyone already working on it? Does >> anyone have a plan on doing it? >> >> P.S. We thought that MLlib abstractions about multiple algorithms run with >> same data would need such scheduler, which would rerun algorithm in case >> of >> failure. I understand, that spark provide fault tolerance out of the box, >> but we found some "Ooozie-like" scheduler more reliable for such long >> living workflows. >> >> -- >> >> >> >> *Sincerely yoursEgor PakhomovScala Developer, Yandex* >> > > -- *Sincerely yoursEgor PakhomovScala Developer, Yandex*
