Repository: falcon Updated Branches: refs/heads/0.8 9d201294f -> 517718bfb
FALCON-1585 Documentation for HDFS and Hive DR. Contributed by Peeyush Bishnoi Project: http://git-wip-us.apache.org/repos/asf/falcon/repo Commit: http://git-wip-us.apache.org/repos/asf/falcon/commit/517718bf Tree: http://git-wip-us.apache.org/repos/asf/falcon/tree/517718bf Diff: http://git-wip-us.apache.org/repos/asf/falcon/diff/517718bf Branch: refs/heads/0.8 Commit: 517718bfbdd30b09785de303b7b8b9a8f35dae29 Parents: 9d20129 Author: Sowmya Ramesh <[email protected]> Authored: Fri Nov 6 11:48:19 2015 -0800 Committer: Sowmya Ramesh <[email protected]> Committed: Fri Nov 6 11:48:19 2015 -0800 ---------------------------------------------------------------------- CHANGES.txt | 2 ++ docs/src/site/twiki/HDFSDR.twiki | 21 +++++++++++ docs/src/site/twiki/HiveDR.twiki | 64 ++++++++++++++++++++++++++++++++++ docs/src/site/twiki/Recipes.twiki | 4 +++ 4 files changed, 91 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/falcon/blob/517718bf/CHANGES.txt ---------------------------------------------------------------------- diff --git a/CHANGES.txt b/CHANGES.txt index 9eec6ea..7bf823d 100755 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -27,6 +27,8 @@ Release version: 0.8 FALCON-1027 Falcon proxy user support(Sowmya Ramesh) IMPROVEMENTS + FALCON-1585 Documentation for HDFS and Hive DR(Peeyush Bishnoi via Sowmya Ramesh) + FALCON-1582 Documentation for globally disabling retries (Pallavi Rao) FALCON-1574 Document behavior of lifecycle retention stage frequency behavior(Sowmya Ramesh via Ajay Yadava). http://git-wip-us.apache.org/repos/asf/falcon/blob/517718bf/docs/src/site/twiki/HDFSDR.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/HDFSDR.twiki b/docs/src/site/twiki/HDFSDR.twiki new file mode 100644 index 0000000..4a42c4c --- /dev/null +++ b/docs/src/site/twiki/HDFSDR.twiki @@ -0,0 +1,21 @@ +---+ HDFS DR Recipe +---++ Overview +Falcon supports HDFS DR recipe to replicate data from source cluster to destination cluster. + +---++ Usage +---+++ Setup cluster definition. + <verbatim> + $FALCON_HOME/bin/falcon entity -submit -type cluster -file /cluster/definition.xml + </verbatim> + +---+++ Update recipes properties + Update recipe properties file in addons/recipes/hdfs-replication with required attributes for replicating + data from source cluster to destination cluster. + +---+++ Submit HDFS DR recipe + <verbatim> + $FALCON_HOME/bin/falcon recipe -name hdfs-replication -operation HDFS_REPLICATION + </verbatim> + +Recipe templates for HDFS DR is available in addons/recipes/hdfs-replication and copy it to +recipe path (*falcon.recipe.path=<recipe directory path>*) by specifying in client.properties. http://git-wip-us.apache.org/repos/asf/falcon/blob/517718bf/docs/src/site/twiki/HiveDR.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/HiveDR.twiki b/docs/src/site/twiki/HiveDR.twiki new file mode 100644 index 0000000..ca039ce --- /dev/null +++ b/docs/src/site/twiki/HiveDR.twiki @@ -0,0 +1,64 @@ +---+Hive Disaster Recovery + + +---++Overview +Falcon provides feature to replicate Hive metadata and data events from source cluster +to destination cluster. This is supported for secure and unsecure cluster through Falcon Recipes. + + +---++Prerequisites +Following is the prerequisites to use Hive DR + + * *Hive 1.2.0+* + * *Oozie 4.2.0+* + +*Note:* Set following properties in hive-site.xml for replicating the Hive events on source and destination Hive cluster: +<verbatim> + <property> + <name>hive.metastore.event.listeners</name> + <value>org.apache.hive.hcatalog.listener.DbNotificationListener</value> + <description>event listeners that are notified of any metastore changes</description> + </property> + + <property> + <name>hive.metastore.dml.events</name> + <value>true</value> + </property> +</verbatim> + +---++ Usage +---+++ Bootstrap + Perform initial bootstrap of Table and Database from source cluster to destination cluster + * *Database Bootstrap* + For bootstrapping DB replication, first destination DB should be created. This step is expected, + since DB replication definitions can be set up by users only on pre-existing DBâs. Second, Export all tables in + the source db and Import it in the destination db, as described in Table bootstrap. + + * *Table Bootstrap* + For bootstrapping table replication, essentially after having turned on the !DbNotificationListener + on the source db, perform an Export of the table, distcp the Export over to the destination + warehouse and do an Import over there. Check the following [[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport][Hive Export-Import]] for syntax details + and examples. + This will set up the destination table so that the events on the source cluster that modify the table + will then be replicated. + +---+++ Setup cluster definition + <verbatim> + $FALCON_HOME/bin/falcon entity -submit -type cluster -file /cluster/definition.xml + </verbatim> + +---+++ Update recipes properties + Update recipe properties file in addons/recipes/hive-disaster-recovery with required attributes for replicating + Hive data and metadata from source cluster to destination cluster. + +---+++ Submit Hive DR recipe + <verbatim> + $FALCON_HOME/bin/falcon recipe -name hive-disaster-recovery -operation HIVE_DISASTER_RECOVERY + </verbatim> + + +Recipe templates for Hive DR is available in addons/recipes/hive-disaster-recovery and copy it to +recipe path (*falcon.recipe.path=<recipe directory path>*) by specifying in client.properties. + +*Note:* If kerberos security is enabled on cluster, use the secure templates for Hive DR from + addons/recipes/hive-disaster-recovery http://git-wip-us.apache.org/repos/asf/falcon/blob/517718bf/docs/src/site/twiki/Recipes.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/Recipes.twiki b/docs/src/site/twiki/Recipes.twiki index ea29d94..792bf1b 100644 --- a/docs/src/site/twiki/Recipes.twiki +++ b/docs/src/site/twiki/Recipes.twiki @@ -76,6 +76,10 @@ GraphDB for display on UI. * Sample recipes are published in addons/recipes +---++ Types of recipes + * [[HDFSDR][HDFS Recipe]] + * [[HiveDR][HiveDR Recipe]] + ---++ Packaging * There is no packaging for recipes at this time but will be added soon.
