Repository: falcon Updated Branches: refs/heads/master 45808eff7 -> e89aa7779
FALCON-1057 recipes.twiki should be renamed Recipes.twiki for consistency and to fix a dead link. Contributed by Aaron Dossett Project: http://git-wip-us.apache.org/repos/asf/falcon/repo Commit: http://git-wip-us.apache.org/repos/asf/falcon/commit/e89aa777 Tree: http://git-wip-us.apache.org/repos/asf/falcon/tree/e89aa777 Diff: http://git-wip-us.apache.org/repos/asf/falcon/diff/e89aa777 Branch: refs/heads/master Commit: e89aa777947165c816310128d5799ba50c4ded7f Parents: 45808ef Author: Srikanth Sundarrajan <[email protected]> Authored: Tue Mar 3 21:39:38 2015 +0530 Committer: Srikanth Sundarrajan <[email protected]> Committed: Tue Mar 3 21:39:38 2015 +0530 ---------------------------------------------------------------------- CHANGES.txt | 3 ++ docs/src/site/twiki/Recipes.twiki | 77 ++++++++++++++++++++++++++++++++++ docs/src/site/twiki/recipes.twiki | 77 ---------------------------------- 3 files changed, 80 insertions(+), 77 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/falcon/blob/e89aa777/CHANGES.txt ---------------------------------------------------------------------- diff --git a/CHANGES.txt b/CHANGES.txt index d88e256..a26376c 100755 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -92,6 +92,9 @@ Trunk (Unreleased) (Suhas vasu) BUG FIXES + FALCON-1057 recipes.twiki should be renamed Recipes.twiki for consistency + and to fix a dead link (Aaron Dossett via Srikanth Sundarrajan) + FALCON-1052 pipeline feature is not able to group by through CLI (Pallavi Rao via Suhas Vasu) http://git-wip-us.apache.org/repos/asf/falcon/blob/e89aa777/docs/src/site/twiki/Recipes.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/Recipes.twiki b/docs/src/site/twiki/Recipes.twiki new file mode 100644 index 0000000..c6cfec3 --- /dev/null +++ b/docs/src/site/twiki/Recipes.twiki @@ -0,0 +1,77 @@ +---+ Falcon Recipes + +---++ Overview + +A Falcon recipe is a static process template with parameterized workflow to realize a specific use case. Recipes are +defined in user space. Recipes will not have support for update or lifecycle management. + +For example: + + * Replicating directories from one HDFS cluster to another (not timed partitions) + * Replicating hive metadata (database, table, views, etc.) + * Replicating between HDFS and Hive - either way + * Data masking etc. + +---++ Proposal + +Falcon provides a Process abstraction that encapsulates the configuration for a user workflow with scheduling +controls. All recipes can be modeled as a Process with in Falcon which executes the user workflow periodically. The +process and its associated workflow are parameterized. The user will provide a properties file with name value pairs +that are substituted by falcon before scheduling it. Falcon translates these recipes as a process entity by +replacing the parameters in the workflow definition. + +---++ Falcon CLI recipe support + +Falcon CLI functionality to support recipes has been added. +<a href="./FalconCLI.html">Recipe command usage is defined here.</a> + +CLI accepts recipe option with a recipe name and optional tool and does the following: + * Validates the options; name option is mandatory and tool is optional and should be provided if user wants to override the base recipe tool + * Looks for <name>-workflow.xml, <name>-template.xml and <name>.properties file in the path specified by falcon.recipe.path in client.properties. If files cannot be found then Falcon CLI will fail + * Invokes a Tool to substitute the properties in the templated process for the recipe. By default invokes base tool if tool option is not passed. Tool is responsible for generating process entity at the path specified by FalconCLI + * Validates the generated entity + * Submit and schedule this entity + * Generated process entity files are stored in tmp directory + +---++ Base Recipe tool + +Falcon provides a base tool that recipes can override. Base Recipe tool does the following: + * Expects recipe template file path, recipe properties file path and path where process entity to be submitted should be generated. Validates these arguments + * Validates the artifacts i.e. workflow and/or lib files specified in the recipe template exists on local filesystem or HDFS at the specified path else returns error + * Copies if the artifacts exists on local filesystem + * If workflow is on local FS then falcon.recipe.workflow.path in recipe property file is mandatory for it to be copied to HDFS. If templated process requires custom libs falcon.recipe.workflow.lib.path property is mandatory for them to be copied from Local FS to HDFS. Recipe tool will copy the local artifacts only if these properties are set in properties file + * Looks for the patten ##[A-Za-z0-9_.]*## in the templated process and substitutes it with the properties. Process entity generated after the substitution is written to the empty file passed by FalconCLI + +---++ Recipe template file format + + * Any templatized string should be in the format ##[A-Za-z0-9_.]*##. + * There should be a corresponding entry in the recipe properties file "falcon.recipe.<templatized-string> = <value to be substituted>" + +<verbatim> +Example: If the entry in recipe template is <workflow name="##workflow.name##"> there should be a corresponding entry in the recipe properties file falcon.recipe.workflow.name=hdfs-dr-workflow +</verbatim> + +---++ Recipe properties file format + + * Regular key value pair properties file + * Property key should be prefixed by "falcon.recipe." + +<verbatim> +Example: falcon.recipe.workflow.name=hdfs-dr-workflow +Recipe template will have <workflow name="##workflow.name##">. Recipe tool will look for the patten ##workflow.name## +and replace it with the property value "hdfs-dr-workflow". Substituted template will have <workflow name="hdfs-dr-workflow"> +</verbatim> + +---++ Managing the scheduled recipe process + * Scheduled recipe process is similar to regular process + * List : falcon entity -type process -name <recipe-process-name> -list + * Status : falcon entity -type process -name <recipe-process-name> -status + * Delete : falcon entity -type process -name <recipe-process-name> -delete + +---++ Sample recipes + + * Sample recipes are published in addons/recipes + +---++ Packaging + + * There is no packaging for recipes at this time but will be added soon. http://git-wip-us.apache.org/repos/asf/falcon/blob/e89aa777/docs/src/site/twiki/recipes.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/recipes.twiki b/docs/src/site/twiki/recipes.twiki deleted file mode 100644 index c6cfec3..0000000 --- a/docs/src/site/twiki/recipes.twiki +++ /dev/null @@ -1,77 +0,0 @@ ----+ Falcon Recipes - ----++ Overview - -A Falcon recipe is a static process template with parameterized workflow to realize a specific use case. Recipes are -defined in user space. Recipes will not have support for update or lifecycle management. - -For example: - - * Replicating directories from one HDFS cluster to another (not timed partitions) - * Replicating hive metadata (database, table, views, etc.) - * Replicating between HDFS and Hive - either way - * Data masking etc. - ----++ Proposal - -Falcon provides a Process abstraction that encapsulates the configuration for a user workflow with scheduling -controls. All recipes can be modeled as a Process with in Falcon which executes the user workflow periodically. The -process and its associated workflow are parameterized. The user will provide a properties file with name value pairs -that are substituted by falcon before scheduling it. Falcon translates these recipes as a process entity by -replacing the parameters in the workflow definition. - ----++ Falcon CLI recipe support - -Falcon CLI functionality to support recipes has been added. -<a href="./FalconCLI.html">Recipe command usage is defined here.</a> - -CLI accepts recipe option with a recipe name and optional tool and does the following: - * Validates the options; name option is mandatory and tool is optional and should be provided if user wants to override the base recipe tool - * Looks for <name>-workflow.xml, <name>-template.xml and <name>.properties file in the path specified by falcon.recipe.path in client.properties. If files cannot be found then Falcon CLI will fail - * Invokes a Tool to substitute the properties in the templated process for the recipe. By default invokes base tool if tool option is not passed. Tool is responsible for generating process entity at the path specified by FalconCLI - * Validates the generated entity - * Submit and schedule this entity - * Generated process entity files are stored in tmp directory - ----++ Base Recipe tool - -Falcon provides a base tool that recipes can override. Base Recipe tool does the following: - * Expects recipe template file path, recipe properties file path and path where process entity to be submitted should be generated. Validates these arguments - * Validates the artifacts i.e. workflow and/or lib files specified in the recipe template exists on local filesystem or HDFS at the specified path else returns error - * Copies if the artifacts exists on local filesystem - * If workflow is on local FS then falcon.recipe.workflow.path in recipe property file is mandatory for it to be copied to HDFS. If templated process requires custom libs falcon.recipe.workflow.lib.path property is mandatory for them to be copied from Local FS to HDFS. Recipe tool will copy the local artifacts only if these properties are set in properties file - * Looks for the patten ##[A-Za-z0-9_.]*## in the templated process and substitutes it with the properties. Process entity generated after the substitution is written to the empty file passed by FalconCLI - ----++ Recipe template file format - - * Any templatized string should be in the format ##[A-Za-z0-9_.]*##. - * There should be a corresponding entry in the recipe properties file "falcon.recipe.<templatized-string> = <value to be substituted>" - -<verbatim> -Example: If the entry in recipe template is <workflow name="##workflow.name##"> there should be a corresponding entry in the recipe properties file falcon.recipe.workflow.name=hdfs-dr-workflow -</verbatim> - ----++ Recipe properties file format - - * Regular key value pair properties file - * Property key should be prefixed by "falcon.recipe." - -<verbatim> -Example: falcon.recipe.workflow.name=hdfs-dr-workflow -Recipe template will have <workflow name="##workflow.name##">. Recipe tool will look for the patten ##workflow.name## -and replace it with the property value "hdfs-dr-workflow". Substituted template will have <workflow name="hdfs-dr-workflow"> -</verbatim> - ----++ Managing the scheduled recipe process - * Scheduled recipe process is similar to regular process - * List : falcon entity -type process -name <recipe-process-name> -list - * Status : falcon entity -type process -name <recipe-process-name> -status - * Delete : falcon entity -type process -name <recipe-process-name> -delete - ----++ Sample recipes - - * Sample recipes are published in addons/recipes - ----++ Packaging - - * There is no packaging for recipes at this time but will be added soon.
