Hello Team,
Since its inception Falcon has used Oozie for process orchestration as well as
feed life cycle phase executions, while this has worked reasonably and allowed
to make higher level capabilities available through Falcon, we are increasing
seeing scenarios where this is proving to be a limiting factor. In its current
form, Falcon relies on Oozie for both scheduling and for workflow execution,
due to which the scheduling is limited to time based/cron based scheduling with
additional gating conditions on data availability. Also this imposes
restrictions on datesets being periodic/cyclic in nature.
>From an orchestration stand point, it would help if we can support standard
>gating / scheduling primitives via Falcon:
1. Simple periodic scheduling with no gating conditions
2. Cron based scheduling (day of week, day of the month, specific hours and
non-periodic) with no gating conditions
3. Availability of new data (assuming monotonically increasing data version,
availavility of new versions)
4. Changes to existing data (reinstatement - similar to late data handling)
5. External trigger/notifications
6. Availability of specific instances of data as declared as mandatory
dependency
7. Availability of a minimum subset of instances of data declared as mandatory
depedency (at least 10 hourly instances of a day with 24 instances for ex)
8. Valid combinations of the above.
In this context, I would like to propose that we move away from Oozie for the
orchestration requirements and have them implemented natively within Falcon. It
will no doubt make Falcon server bulkier and heavier in both code and
deployment, but seems like without it, the orchestration within Falcon will be
limited by capabilities available within Oozie.
Please do note that this suggestion is restricted to the scheduling and not to
the workflow execution.
Would like to hear from fellow developers and users on what your thoughts are.
Please do chime in with your views.
Regards
Srikanth Sundarrajan
