[
https://issues.apache.org/jira/browse/FALCON-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120698#comment-15120698
]
Venkat Ranganathan commented on FALCON-634:
-------------------------------------------
Myself, [~sowmyaramesh] and [~yzheng-hortonworks] discussed this today.
Basically
* There should be two classes of recipes - core recipes - are those that are
designed, developed and released as part of Falcon - these are essentially
upto-date with the release of Falcon, have regression and integration tests
that are run as part of the falcon release and thoroughly tested. These
qualify to run within the Falcon server and typically live under the _addons_
directory. Examples are the HDFS and Hive mirroring, and future such addons
* Optional recipes are those that are customer/administrator written, not
shipped with the product and may need updates with newer versions of the
product. Their execution model (whether they get scheduled in a managed
workflow or on a separate process) are things that need further discussion and
clarity.
The packaging of the recipes for execution, their directory structure, how they
can be introspected so that a UI can render the invocation of these REST APIs
are common to both core and optional recipes.
There will be a new design document that we will publish. Just providing the
gist of the discussion to kick off early discussions.
> Add recipes in Falcon
> ---------------------
>
> Key: FALCON-634
> URL: https://issues.apache.org/jira/browse/FALCON-634
> Project: Falcon
> Issue Type: New Feature
> Affects Versions: 0.6
> Reporter: Venkatesh Seetharam
> Labels: recipes
>
> Falcon offers many services OOTB and caters to a wide array of use cases.
> However, there has been many asks that does not fit the functionality offered
> by Falcon. I'm proposing that we add recipes to Falcon which is similar to
> recipes in Whirr and other management solutions such as puppet and chef.
> Overview:
> A recipe essentially is a static process template with parameterized workflow
> to realize a specific use case. For example:
> * replicating directories from one HDFS cluster to another (not timed
> partitions)
> * replicating hive metadata (database, table, views, etc.)
> * replicating between HDFS and Hive - either way
> * anonymization of data based on schema
> * data masking
> * etc.
> Proposal:
> Falcon provides a Process abstraction that encapsulates the configuration
> for a user workflow with scheduling controls. All recipes can be modeled
> as a Process with in Falcon which executes the user workflow
> periodically. The process and its associated workflow are parameterized. The
> user will provide a properties file with name value pairs that are
> substituted by falcon before scheduling it.
> This is a client side concept. The server does not know about a recipe but
> only accepts the cooked recipe as a process entity.
> The CLI would look something like this:
> falcon -recipe $recipe_name -properties $properties_file
> Recipes will reside inside addons (contrib) with source code and will have an
> option to package 'em.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
