+1
I would like to help there ! ;)
Regards
JB
On 04/07/2015 08:47 AM, Zulfiqar Ali wrote:
+1
On Tue, Apr 7, 2015 at 11:42 AM, Ajay Yadav <[email protected]> wrote:
+1
On Tue, Apr 7, 2015 at 11:27 AM, Pallavi Rao <[email protected]>
wrote:
Hi,
I was recently looking at some of the use cases at InMobi and how to
enhance Falcon to accommodate those and I realized that due to our
dependency on Oozie coordinator, some of these cannot be easily achieved
or
take a much longer cycle as we have to wait for Oozie to add some
functionality.
I was pointed to this thread that dates slightly before my time in
Falcon (
https://www.mail-archive.com/[email protected]/msg09268.html
).
I wanted to reopen the thread for discussion, with my 2 cents:
1. Some of the scheduling primitives that are already mentioned in the
thread, especially, support for a-periodic datasets or external
triggering
mechanisms are not available in Oozie. It might not even be a natural
fit
for Oozie to add these.
2. Adding new primitives in Falcon becomes harder and longer as we
completely depend on Oozie for the same. Extensibility of Falcon is
stunted.
3. Oozie has very limited support for throttling resource utilization.
We can only control the no. of parallel instances of a coordinator
job.
4. Oozie currently has no notion of inter dependency of
instances/workflows, whereas, in Falcon, it will be very useful to
gate/throttle based on the interdependency. For example, re-run a
pipeline
(or a subset) or throttle resource utilization of a pipeline when in
"backlog catchup" mode.
5. We end up with bugs like FALCON-1127
<https://issues.apache.org/jira/browse/FALCON-1127>, because Falcon
constantly needs to play catchup with Oozie changes.
On the thread, most people did seem to be in favor of a native scheduler
in
Falcon. If you all think this is useful, I'll volunteer to start work on
this and we can build out a scheduler/orchestrator in Falcon that can
open
up a whole lot of possibilities for Falcon users.
Thanks,
Pallavi
--
_____________________________________________________________
The information contained in this communication is intended solely for
the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally
privileged
information. If you are not the intended recipient you are hereby
notified
that any disclosure, copying, distribution or taking any action in
reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete
transmission
of the information contained in this communication nor for any delay in
its
receipt.
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
