+1
On Tue, Apr 7, 2015 at 11:27 AM, Pallavi Rao <[email protected]> wrote: > Hi, > I was recently looking at some of the use cases at InMobi and how to > enhance Falcon to accommodate those and I realized that due to our > dependency on Oozie coordinator, some of these cannot be easily achieved or > take a much longer cycle as we have to wait for Oozie to add some > functionality. > > I was pointed to this thread that dates slightly before my time in Falcon ( > https://www.mail-archive.com/[email protected]/msg09268.html > ). > I wanted to reopen the thread for discussion, with my 2 cents: > > 1. Some of the scheduling primitives that are already mentioned in the > thread, especially, support for a-periodic datasets or external > triggering > mechanisms are not available in Oozie. It might not even be a natural > fit > for Oozie to add these. > 2. Adding new primitives in Falcon becomes harder and longer as we > completely depend on Oozie for the same. Extensibility of Falcon is > stunted. > 3. Oozie has very limited support for throttling resource utilization. > We can only control the no. of parallel instances of a coordinator job. > 4. Oozie currently has no notion of inter dependency of > instances/workflows, whereas, in Falcon, it will be very useful to > gate/throttle based on the interdependency. For example, re-run a > pipeline > (or a subset) or throttle resource utilization of a pipeline when in > "backlog catchup" mode. > 5. We end up with bugs like FALCON-1127 > <https://issues.apache.org/jira/browse/FALCON-1127>, because Falcon > constantly needs to play catchup with Oozie changes. > > On the thread, most people did seem to be in favor of a native scheduler in > Falcon. If you all think this is useful, I'll volunteer to start work on > this and we can build out a scheduler/orchestrator in Falcon that can open > up a whole lot of possibilities for Falcon users. > > Thanks, > Pallavi > > -- > _____________________________________________________________ > The information contained in this communication is intended solely for the > use of the individual or entity to whom it is addressed and others > authorized to receive it. It may contain confidential or legally privileged > information. If you are not the intended recipient you are hereby notified > that any disclosure, copying, distribution or taking any action in reliance > on the contents of this information is strictly prohibited and may be > unlawful. If you have received this communication in error, please notify > us immediately by responding to this email and then delete it from your > system. The firm is neither liable for the proper and complete transmission > of the information contained in this communication nor for any delay in its > receipt. >
