Added: falcon/site/Configuration.html URL: http://svn.apache.org/viewvc/falcon/site/Configuration.html?rev=1730450&view=auto ============================================================================== --- falcon/site/Configuration.html (added) +++ falcon/site/Configuration.html Mon Feb 15 06:02:08 2016 @@ -0,0 +1,538 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-02-09 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160209" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Falcon - Configuring Falcon</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="../index.html" id="bannerLeft"> + <img src="images/falcon-logo.png" alt="Falcon" width="200px" height="45px"/> + </a> + </div> + <div class="pull-right"> <a href="http://www.apache.org"; id="bannerRight"> + <img src="images/apache-feather-tm.gif" alt="Falcon" height="45px"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="index.html" title="Falcon"> + Falcon</a> + </li> + <li class="divider ">/</li> + <li class="">Configuring Falcon</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2016-02-09</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">Falcon</li> + + <li> + + <a href="index.html" title="About"> + <i class="none"></i> + About</a> + </li> + + <li> + + <a href="slides/falcon-overview.html" title="Overview"> + <i class="none"></i> + Overview</a> + </li> + + <li> + + <a href="slides/falcon-user-guide.html" title="User Guide"> + <i class="none"></i> + User Guide</a> + </li> + + <li> + + <a href="GettingStarted.html" title="Getting Started"> + <i class="none"></i> + Getting Started</a> + </li> + + <li> + + <a href="FalconDocumentation.html" title="Architecture"> + <i class="none"></i> + Architecture</a> + </li> + + <li> + + <a href="InstallationSteps.html" title="Installation"> + <i class="none"></i> + Installation</a> + </li> + + <li> + + <a href="OnBoarding.html" title="On Boarding"> + <i class="none"></i> + On Boarding</a> + </li> + + <li> + + <a href="Operability.html" title="Operability"> + <i class="none"></i> + Operability</a> + </li> + + <li> + + <a href="EntitySpecification.html" title="Entity Specification"> + <i class="none"></i> + Entity Specification</a> + </li> + + <li> + + <a href="FalconCLI.html" title="Client (Falcon CLI)"> + <i class="none"></i> + Client (Falcon CLI)</a> + </li> + + <li> + + <a href="restapi/ResourceList.html" title="Rest API"> + <i class="icon-chevron-right"></i> + Rest API</a> + </li> + + <li> + + <a href="HiveIntegration.html" title="Hive Integration"> + <i class="none"></i> + Hive Integration</a> + </li> + + <li> + + <a href="Security.html" title="Security"> + <i class="none"></i> + Security</a> + </li> + <li class="nav-header">Project Information</li> + + <li> + + <a href="project-info.html" title="Summary"> + <i class="none"></i> + Summary</a> + </li> + + <li> + + <a href="mail-lists.html" title="Mailing Lists"> + <i class="none"></i> + Mailing Lists</a> + </li> + + <li> + + <a href="http://webchat.freenode.net?channels=apachefalcon&uio=d4"; class="externalLink" title="IRC"> + <i class="none"></i> + IRC</a> + </li> + + <li> + + <a href="team-list.html" title="Team"> + <i class="none"></i> + Team</a> + </li> + + <li> + + <a href="issue-tracking.html" title="Issue Tracking"> + <i class="none"></i> + Issue Tracking</a> + </li> + + <li> + + <a href="source-repository.html" title="Source Repository"> + <i class="none"></i> + Source Repository</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/Index"; class="externalLink" title="Wiki"> + <i class="none"></i> + Wiki</a> + </li> + + <li> + + <a href="license.html" title="License"> + <i class="none"></i> + License</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/News"; class="externalLink" title="News"> + <i class="none"></i> + News</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/PoweredBy"; class="externalLink" title="Powered by"> + <i class="none"></i> + Powered by</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/Acknowledgements"; class="externalLink" title="Acknowledgements"> + <i class="none"></i> + Acknowledgements</a> + </li> + + <li> + + <a href="http://blogs.apache.org/falcon/"; class="externalLink" title="Blog"> + <i class="none"></i> + Blog</a> + </li> + <li class="nav-header">Releases</li> + + <li> + + <a href="http://www.apache.org/dyn/closer.lua/falcon/0.8"; class="externalLink" title="0.8"> + <i class="none"></i> + 0.8</a> + </li> + + <li> + + <a href="http://www.apache.org/dyn/closer.lua/falcon/0.7"; class="externalLink" title="0.7"> + <i class="none"></i> + 0.7</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/falcon/0.6.1"; class="externalLink" title="0.6.1"> + <i class="none"></i> + 0.6.1</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/incubator/falcon/0.6-incubating"; class="externalLink" title="0.6-incubating"> + <i class="none"></i> + 0.6-incubating</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/incubator/falcon/0.5-incubating"; class="externalLink" title="0.5-incubating"> + <i class="none"></i> + 0.5-incubating</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/incubator/falcon/0.4-incubating"; class="externalLink" title="0.4-incubating"> + <i class="none"></i> + 0.4-incubating</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/incubator/falcon/0.3-incubating"; class="externalLink" title="0.3-incubating"> + <i class="none"></i> + 0.3-incubating</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/Roadmap"; class="externalLink" title="Coming soon"> + <i class="none"></i> + Coming soon</a> + </li> + <li class="nav-header">Documentation</li> + + <li> + + <a href="0.9/index.html" title="0.9 (Current)"> + <i class="none"></i> + 0.9 (Current)</a> + </li> + + <li> + + <a href="0.8/index.html" title="0.8"> + <i class="none"></i> + 0.8</a> + </li> + + <li> + + <a href="0.7/index.html" title="0.7"> + <i class="none"></i> + 0.7</a> + </li> + + <li> + + <a href="0.6.1/index.html" title="0.6.1"> + <i class="none"></i> + 0.6.1</a> + </li> + + <li> + + <a href="0.6-incubating/index.html" title="0.6-incubating"> + <i class="none"></i> + 0.6-incubating</a> + </li> + + <li> + + <a href="0.5-incubating/index.html" title="0.5-incubating"> + <i class="none"></i> + 0.5-incubating</a> + </li> + + <li> + + <a href="0.4-incubating/index.html" title="0.4-incubating"> + <i class="none"></i> + 0.4-incubating</a> + </li> + + <li> + + <a href="0.3-incubating/index.html" title="0.3-incubating"> + <i class="none"></i> + 0.3-incubating</a> + </li> + <li class="nav-header">ASF</li> + + <li> + + <a href="http://www.apache.org/foundation/how-it-works.html"; class="externalLink" title="How Apache Works"> + <i class="none"></i> + How Apache Works</a> + </li> + + <li> + + <a href="http://www.apache.org/foundation/"; class="externalLink" title="Foundation"> + <i class="none"></i> + Foundation</a> + </li> + + <li> + + <a href="http://www.apache.org/foundation/sponsorship.html"; class="externalLink" title="Sponsoring Apache"> + <i class="none"></i> + Sponsoring Apache</a> + </li> + + <li> + + <a href="http://www.apache.org/foundation/thanks.html"; class="externalLink" title="Thanks"> + <i class="none"></i> + Thanks</a> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <div class="section"> +<h2>Configuring Falcon<a name="Configuring_Falcon"></a></h2> +<p>By default config directory used by falcon is {package dir}/conf. To override this (to use the same conf with multiple falcon upgrades), set environment variable FALCON_CONF to the path of the conf dir.</p> +<p>falcon-env.sh has been added to the falcon conf. This file can be used to set various environment variables that you need for you services. In addition you can set any other environment variables you might need. This file will be sourced by falcon scripts before any commands are executed. The following environment variables are available to set.</p> +<div class="source"> +<pre> +# The java implementation to use. If JAVA_HOME is not found we expect java and jar to be in path +#export JAVA_HOME= + +# any additional java opts you want to set. This will apply to both client and server operations +#export FALCON_OPTS= + +# any additional java opts that you want to set for client only +#export FALCON_CLIENT_OPTS= + +# java heap size we want to set for the client. Default is 1024MB +#export FALCON_CLIENT_HEAP= + +# any additional opts you want to set for prism service. +#export FALCON_PRISM_OPTS= + +# java heap size we want to set for the prism service. Default is 1024MB +#export FALCON_PRISM_HEAP= + +# any additional opts you want to set for falcon service. +#export FALCON_SERVER_OPTS= + +# java heap size we want to set for the falcon server. Default is 1024MB +#export FALCON_SERVER_HEAP= + +# What is is considered as falcon home dir. Default is the base location of the installed software +#export FALCON_HOME_DIR= + +# Where log files are stored. Default is logs directory under the base install location +#export FALCON_LOG_DIR= + +# Where pid files are stored. Default is logs directory under the base install location +#export FALCON_PID_DIR= + +# where the falcon active mq data is stored. Default is logs/data directory under the base install location +#export FALCON_DATA_DIR= + +# Where do you want to expand the war file. By Default it is in /server/webapp dir under the base install dir. +#export FALCON_EXPANDED_WEBAPP_DIR= + +</pre></div></div> +<div class="section"> +<h3>Advanced Configurations<a name="Advanced_Configurations"></a></h3></div> +<div class="section"> +<h4>Configuring Monitoring plugin to register catalog partitions<a name="Configuring_Monitoring_plugin_to_register_catalog_partitions"></a></h4> +<p>Falcon comes with a monitoring plugin that registers catalog partition. This comes in really handy during migration from filesystem based feeds to hcatalog based feeds. This plugin enables the user to de-couple the partition registration and assume that all partitions are already on hcatalog even before the migration, simplifying the hcatalog migration.</p> +<p>By default this plugin is disabled. To enable this plugin and leverage the feature, there are 3 pre-requisites:</p> +<div class="source"> +<pre> +In {package dir}/conf/startup.properties, add +*.workflow.execution.listeners=org.apache.falcon.catalog.CatalogPartitionHandler + +In the cluster definition, ensure registry endpoint is defined. +Ex: +<interface type="registry" endpoint="thrift://localhost:1109" version="0.13.3"/> + +In the feed definition, ensure the corresponding catalog table is mentioned in feed-properties +Ex: +<properties> + <property name="catalog.table" value="catalog:default:in_table#year={YEAR};month={MONTH};day={DAY};hour={HOUR}; + minute={MINUTE}"/> +</properties> + +</pre></div> +<p><b>NOTE : for Mac OS users</b></p> +<div class="source"> +<pre> +If you are using a Mac OS, you will need to configure the FALCON_SERVER_OPTS (explained above). + +In {package dir}/conf/falcon-env.sh uncomment the following line +#export FALCON_SERVER_OPTS= + +and change it to look as below +export FALCON_SERVER_OPTS="-Djava.awt.headless=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc=" + +</pre></div></div> +<div class="section"> +<h4>Activemq<a name="Activemq"></a></h4> +<p>* falcon server starts embedded active mq. To control this behaviour, set the following system properties using -D option in environment variable FALCON_OPTS:</p> +<ul> +<li>falcon.embeddedmq=<true/false> - Should server start embedded active mq, default true</li> +<li>falcon.embeddedmq.port=<port> - Port for embedded active mq, default 61616</li> +<li>falcon.embeddedmq.data=<path> - Data path for embedded active mq, default {package dir}/logs/data</li></ul></div> +<div class="section"> +<h4>Falcon System Notifications<a name="Falcon_System_Notifications"></a></h4> +<p>Some Falcon features such as late data handling, retries, metadata service, depend on JMS notifications sent when the Oozie workflow completes. These system notifications are sent as part of Falcon Post Processing action. Given that the post processing action is also a job, it is prone to failures and in case of failures, Falcon is blind to the status of the workflow. To alleviate this problem and make the notifications more reliable, you can enable Oozie's JMS notification feature and disable Falcon post-processing notification by making the following changes:</p> +<ul> +<li>In Falcon runtime.properties, set *.falcon.jms.notification.enabled to false. This will turn off JMS notification in post-processing.</li> +<li>Copy notification related properties in oozie/conf/oozie-site.xml to oozie-site.xml of the Oozie installation. Restart Oozie so changes get reflected.</li></ul> +<p><b>NOTE : If you disable Falcon post-processing JMS notification and not enable Oozie JMS notification, features such as failure retry, late data handling and metadata service will be disabled for all entities on the server.</b></p></div> +<div class="section"> +<h4>Enabling Falcon Native Scheudler<a name="Enabling_Falcon_Native_Scheudler"></a></h4> +<p><verbatim>$FALCON_HOME/conf/startup.properties</verbatim> before starting the Falcon Server. For details on the same, refer to <a href="./FalconNativeScheduler.html">Falcon Native Scheduler</a></p></div> +<div class="section"> +<h4>Adding Extension Libraries<a name="Adding_Extension_Libraries"></a></h4> +<p>Library extensions allows users to add custom libraries to entity lifecycles such as feed retention, feed replication and process execution. This is useful for usecases such as adding filesystem extensions. To enable this, add the following configs to startup.properties: *.libext.paths=<paths to be added to all entity lifecycles></p> +<p>*.libext.feed.paths=<paths to be added to all feed lifecycles></p> +<p>*.libext.feed.retentions.paths=<paths to be added to feed retention workflow></p> +<p>*.libext.feed.replication.paths=<paths to be added to feed replication workflow></p> +<p>*.libext.process.paths=<paths to be added to process workflow></p> +<p>The configured jars are added to falcon classpath and the corresponding workflows.</p></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2013-2016 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html>
Added: falcon/site/Distributed-mode.html URL: http://svn.apache.org/viewvc/falcon/site/Distributed-mode.html?rev=1730450&view=auto ============================================================================== --- falcon/site/Distributed-mode.html (added) +++ falcon/site/Distributed-mode.html Mon Feb 15 06:02:08 2016 @@ -0,0 +1,595 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-02-09 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160209" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Falcon - Distributed Mode</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="../index.html" id="bannerLeft"> + <img src="images/falcon-logo.png" alt="Falcon" width="200px" height="45px"/> + </a> + </div> + <div class="pull-right"> <a href="http://www.apache.org"; id="bannerRight"> + <img src="images/apache-feather-tm.gif" alt="Falcon" height="45px"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="index.html" title="Falcon"> + Falcon</a> + </li> + <li class="divider ">/</li> + <li class="">Distributed Mode</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2016-02-09</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">Falcon</li> + + <li> + + <a href="index.html" title="About"> + <i class="none"></i> + About</a> + </li> + + <li> + + <a href="slides/falcon-overview.html" title="Overview"> + <i class="none"></i> + Overview</a> + </li> + + <li> + + <a href="slides/falcon-user-guide.html" title="User Guide"> + <i class="none"></i> + User Guide</a> + </li> + + <li> + + <a href="GettingStarted.html" title="Getting Started"> + <i class="none"></i> + Getting Started</a> + </li> + + <li> + + <a href="FalconDocumentation.html" title="Architecture"> + <i class="none"></i> + Architecture</a> + </li> + + <li> + + <a href="InstallationSteps.html" title="Installation"> + <i class="none"></i> + Installation</a> + </li> + + <li> + + <a href="OnBoarding.html" title="On Boarding"> + <i class="none"></i> + On Boarding</a> + </li> + + <li> + + <a href="Operability.html" title="Operability"> + <i class="none"></i> + Operability</a> + </li> + + <li> + + <a href="EntitySpecification.html" title="Entity Specification"> + <i class="none"></i> + Entity Specification</a> + </li> + + <li> + + <a href="FalconCLI.html" title="Client (Falcon CLI)"> + <i class="none"></i> + Client (Falcon CLI)</a> + </li> + + <li> + + <a href="restapi/ResourceList.html" title="Rest API"> + <i class="icon-chevron-right"></i> + Rest API</a> + </li> + + <li> + + <a href="HiveIntegration.html" title="Hive Integration"> + <i class="none"></i> + Hive Integration</a> + </li> + + <li> + + <a href="Security.html" title="Security"> + <i class="none"></i> + Security</a> + </li> + <li class="nav-header">Project Information</li> + + <li> + + <a href="project-info.html" title="Summary"> + <i class="none"></i> + Summary</a> + </li> + + <li> + + <a href="mail-lists.html" title="Mailing Lists"> + <i class="none"></i> + Mailing Lists</a> + </li> + + <li> + + <a href="http://webchat.freenode.net?channels=apachefalcon&uio=d4"; class="externalLink" title="IRC"> + <i class="none"></i> + IRC</a> + </li> + + <li> + + <a href="team-list.html" title="Team"> + <i class="none"></i> + Team</a> + </li> + + <li> + + <a href="issue-tracking.html" title="Issue Tracking"> + <i class="none"></i> + Issue Tracking</a> + </li> + + <li> + + <a href="source-repository.html" title="Source Repository"> + <i class="none"></i> + Source Repository</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/Index"; class="externalLink" title="Wiki"> + <i class="none"></i> + Wiki</a> + </li> + + <li> + + <a href="license.html" title="License"> + <i class="none"></i> + License</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/News"; class="externalLink" title="News"> + <i class="none"></i> + News</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/PoweredBy"; class="externalLink" title="Powered by"> + <i class="none"></i> + Powered by</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/Acknowledgements"; class="externalLink" title="Acknowledgements"> + <i class="none"></i> + Acknowledgements</a> + </li> + + <li> + + <a href="http://blogs.apache.org/falcon/"; class="externalLink" title="Blog"> + <i class="none"></i> + Blog</a> + </li> + <li class="nav-header">Releases</li> + + <li> + + <a href="http://www.apache.org/dyn/closer.lua/falcon/0.8"; class="externalLink" title="0.8"> + <i class="none"></i> + 0.8</a> + </li> + + <li> + + <a href="http://www.apache.org/dyn/closer.lua/falcon/0.7"; class="externalLink" title="0.7"> + <i class="none"></i> + 0.7</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/falcon/0.6.1"; class="externalLink" title="0.6.1"> + <i class="none"></i> + 0.6.1</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/incubator/falcon/0.6-incubating"; class="externalLink" title="0.6-incubating"> + <i class="none"></i> + 0.6-incubating</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/incubator/falcon/0.5-incubating"; class="externalLink" title="0.5-incubating"> + <i class="none"></i> + 0.5-incubating</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/incubator/falcon/0.4-incubating"; class="externalLink" title="0.4-incubating"> + <i class="none"></i> + 0.4-incubating</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/incubator/falcon/0.3-incubating"; class="externalLink" title="0.3-incubating"> + <i class="none"></i> + 0.3-incubating</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/Roadmap"; class="externalLink" title="Coming soon"> + <i class="none"></i> + Coming soon</a> + </li> + <li class="nav-header">Documentation</li> + + <li> + + <a href="0.9/index.html" title="0.9 (Current)"> + <i class="none"></i> + 0.9 (Current)</a> + </li> + + <li> + + <a href="0.8/index.html" title="0.8"> + <i class="none"></i> + 0.8</a> + </li> + + <li> + + <a href="0.7/index.html" title="0.7"> + <i class="none"></i> + 0.7</a> + </li> + + <li> + + <a href="0.6.1/index.html" title="0.6.1"> + <i class="none"></i> + 0.6.1</a> + </li> + + <li> + + <a href="0.6-incubating/index.html" title="0.6-incubating"> + <i class="none"></i> + 0.6-incubating</a> + </li> + + <li> + + <a href="0.5-incubating/index.html" title="0.5-incubating"> + <i class="none"></i> + 0.5-incubating</a> + </li> + + <li> + + <a href="0.4-incubating/index.html" title="0.4-incubating"> + <i class="none"></i> + 0.4-incubating</a> + </li> + + <li> + + <a href="0.3-incubating/index.html" title="0.3-incubating"> + <i class="none"></i> + 0.3-incubating</a> + </li> + <li class="nav-header">ASF</li> + + <li> + + <a href="http://www.apache.org/foundation/how-it-works.html"; class="externalLink" title="How Apache Works"> + <i class="none"></i> + How Apache Works</a> + </li> + + <li> + + <a href="http://www.apache.org/foundation/"; class="externalLink" title="Foundation"> + <i class="none"></i> + Foundation</a> + </li> + + <li> + + <a href="http://www.apache.org/foundation/sponsorship.html"; class="externalLink" title="Sponsoring Apache"> + <i class="none"></i> + Sponsoring Apache</a> + </li> + + <li> + + <a href="http://www.apache.org/foundation/thanks.html"; class="externalLink" title="Thanks"> + <i class="none"></i> + Thanks</a> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <div class="section"> +<h2>Distributed Mode<a name="Distributed_Mode"></a></h2> +<p>Following are the steps needed to package and deploy Falcon in Embedded Mode. You need to complete Steps 1-3 mentioned <a href="./InstallationSteps.html">here</a> before proceeding further.</p></div> +<div class="section"> +<h3>Package Falcon<a name="Package_Falcon"></a></h3> +<p>Ensure that you are in the base directory (where you cloned Falcon). Let’s call it {project dir}</p> +<div class="source"> +<pre> +$mvn clean assembly:assembly -DskipTests -DskipCheck=true -Pdistributed,hadoop-2 + +</pre></div> +<div class="source"> +<pre> +$ls {project dir}/target/ + +</pre></div> +<p>It should give an output like below :</p> +<div class="source"> +<pre> +apache-falcon-distributed-${project.version}-server.tar.gz +apache-falcon-distributed-${project.version}-sources.tar.gz +archive-tmp +maven-shared-archive-resources + +</pre></div> +<p></p> +<ul> +<li>apache-falcon-distributed-${project.version}-sources.tar.gz contains source files of Falcon repo.</li></ul> +<p></p> +<ul> +<li>apache-falcon-distributed-${project.version}-server.tar.gz package contains project artifacts along with it's</li></ul>dependencies, configuration files and scripts required to deploy Falcon. +<p>Tar can be found in {project dir}/target/apache-falcon-distributed-${project.version}-server.tar.gz . This is the tar used for installing Falcon. Lets call it {falcon package}</p> +<p>Tar is structured as follows.</p> +<div class="source"> +<pre> + +|- bin + |- falcon + |- falcon-start + |- falcon-stop + |- falcon-status + |- falcon-config.sh + |- service-start.sh + |- service-stop.sh + |- service-status.sh + |- prism-stop + |- prism-start + |- prism-status +|- conf + |- startup.properties + |- runtime.properties + |- client.properties + |- prism.keystore + |- log4j.xml + |- falcon-env.sh +|- docs +|- client + |- lib (client support libs) +|- server + |- webapp + |- falcon.war + |- prism.war +|- oozie + |- conf + |- libext +|- hadooplibs +|- README +|- NOTICE.txt +|- LICENSE.txt +|- DISCLAIMER.txt +|- CHANGES.txt + +</pre></div></div> +<div class="section"> +<h3>Installing & running Falcon<a name="Installing__running_Falcon"></a></h3></div> +<div class="section"> +<h4>Installing Falcon<a name="Installing_Falcon"></a></h4> +<p>Running Falcon in distributed mode requires bringing up both prism and server.As the name suggests Falcon prism splits the request it gets to the Falcon servers. It is a good practice to start prism and server with their corresponding configurations separately. Create separate directory for prism and server. Let's call them {falcon-prism-dir} and {falcon-server-dir} respectively.</p> +<p><b>For prism</b></p> +<div class="source"> +<pre> +$mkdir {falcon-prism-dir} +$tar -xzvf {falcon package} + +</pre></div> +<p><b>For server</b></p> +<div class="source"> +<pre> +$mkdir {falcon-server-dir} +$tar -xzvf {falcon package} + +</pre></div></div> +<div class="section"> +<h4>Starting Prism<a name="Starting_Prism"></a></h4> +<div class="source"> +<pre> +cd {falcon-prism-dir}/falcon-distributed-${project.version} +bin/prism-start [-port <port>] + +</pre></div> +<p>By default, * prism server starts at port 16443. To change the port, use -port option</p> +<p>* falcon.enableTLS can be set to true or false explicitly to enable SSL, if not port that end with 443 will automatically put prism on <a class="externalLink" href="https://";>https://</a></p> +<p>* prism starts with conf from {falcon-prism-dir}/falcon-distributed-${project.version}/conf. To override this (to use the same conf with multiple prism upgrades), set environment variable FALCON_CONF to the path of conf dir. You can find the instructions for configuring Falcon <a href="./Configuration.html">here</a>.</p> +<p><b>Enabling prism-client</b> *If prism is not started using default-port 16443 then edit the following property in {falcon-prism-dir}/falcon-distributed-${project.version}/conf/client.properties falcon.url=http://{machine-ip}:{prism-port}/</p></div> +<div class="section"> +<h4>Starting Falcon Server<a name="Starting_Falcon_Server"></a></h4> +<div class="source"> +<pre> +$cd {falcon-server-dir}/falcon-distributed-${project.version} +$bin/falcon-start [-port <port>] + +</pre></div> +<p>By default, * If falcon.enableTLS is set to true explicitly or not set at all, Falcon starts at port 15443 on <a class="externalLink" href="https://";>https://</a> by default.</p> +<p>* If falcon.enableTLS is set to false explicitly, Falcon starts at port 15000 on <a class="externalLink" href="http://.";>http://.</a></p> +<p>* To change the port, use -port option.</p> +<p>* If falcon.enableTLS is not set explicitly, port that ends with 443 will automatically put Falcon on <a class="externalLink" href="https://.";>https://.</a> Any other port will put Falcon on <a class="externalLink" href="http://.";>http://.</a></p> +<p>* server starts with conf from {falcon-server-dir}/falcon-distributed-${project.version}/conf. To override this (to use the same conf with multiple server upgrades), set environment variable FALCON_CONF to the path of conf dir. You can find the instructions for configuring Falcon <a href="./Configuration.html">here</a>.</p> +<p><b>Enabling server-client</b> *If server is not started using default-port 15443 then edit the following property in {falcon-server-dir}/falcon-distributed-${project.version}/conf/client.properties. You can find the instructions for configuring Falcon here. falcon.url=http://{machine-ip}:{server-port}/</p> +<p><b>NOTE</b> : https is the secure version of HTTP, the protocol over which data is sent between your browser and the website that you are connected to. By default Falcon runs in https mode. But user can configure it to http.</p></div> +<div class="section"> +<h4>Using Falcon<a name="Using_Falcon"></a></h4> +<div class="source"> +<pre> +$cd {falcon-prism-dir}/falcon-distributed-${project.version} +$bin/falcon admin -version +Falcon server build version: {Version:"${project.version}-SNAPSHOT-rd7e2be9afa2a5dc96acd1ec9e325f39c6b2f17f7", +Mode:"embedded"} + +$bin/falcon help +(for more details about Falcon cli usage) + +</pre></div></div> +<div class="section"> +<h4>Dashboard<a name="Dashboard"></a></h4> +<p>Once Falcon / prism is started, you can view the status of Falcon entities using the Web-based dashboard. You can open your browser at the corresponding port to use the web UI.</p> +<p>Falcon dashboard makes the REST api calls as user "falcon-dashboard". If this user does not exist on your Falcon and Oozie servers, please create the user.</p> +<div class="source"> +<pre> +## create user. +[root@falconhost ~] useradd -U -m falcon-dashboard -G users + +## verify user is created with membership in correct groups. +[root@falconhost ~] groups falcon-dashboard +falcon-dashboard : falcon-dashboard users +[root@falconhost ~] + +</pre></div></div> +<div class="section"> +<h4>Stopping Falcon Server<a name="Stopping_Falcon_Server"></a></h4> +<div class="source"> +<pre> +$cd {falcon-server-dir}/falcon-distributed-${project.version} +$bin/falcon-stop + +</pre></div></div> +<div class="section"> +<h4>Stopping Falcon Prism<a name="Stopping_Falcon_Prism"></a></h4> +<div class="source"> +<pre> +$cd {falcon-prism-dir}/falcon-distributed-${project.version} +$bin/prism-stop + +</pre></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2013-2016 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html> Added: falcon/site/Embedded-mode.html URL: http://svn.apache.org/viewvc/falcon/site/Embedded-mode.html?rev=1730450&view=auto ============================================================================== --- falcon/site/Embedded-mode.html (added) +++ falcon/site/Embedded-mode.html Mon Feb 15 06:02:08 2016 @@ -0,0 +1,621 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-02-09 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160209" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Falcon - Embedded Mode</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="../index.html" id="bannerLeft"> + <img src="images/falcon-logo.png" alt="Falcon" width="200px" height="45px"/> + </a> + </div> + <div class="pull-right"> <a href="http://www.apache.org"; id="bannerRight"> + <img src="images/apache-feather-tm.gif" alt="Falcon" height="45px"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="index.html" title="Falcon"> + Falcon</a> + </li> + <li class="divider ">/</li> + <li class="">Embedded Mode</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2016-02-09</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">Falcon</li> + + <li> + + <a href="index.html" title="About"> + <i class="none"></i> + About</a> + </li> + + <li> + + <a href="slides/falcon-overview.html" title="Overview"> + <i class="none"></i> + Overview</a> + </li> + + <li> + + <a href="slides/falcon-user-guide.html" title="User Guide"> + <i class="none"></i> + User Guide</a> + </li> + + <li> + + <a href="GettingStarted.html" title="Getting Started"> + <i class="none"></i> + Getting Started</a> + </li> + + <li> + + <a href="FalconDocumentation.html" title="Architecture"> + <i class="none"></i> + Architecture</a> + </li> + + <li> + + <a href="InstallationSteps.html" title="Installation"> + <i class="none"></i> + Installation</a> + </li> + + <li> + + <a href="OnBoarding.html" title="On Boarding"> + <i class="none"></i> + On Boarding</a> + </li> + + <li> + + <a href="Operability.html" title="Operability"> + <i class="none"></i> + Operability</a> + </li> + + <li> + + <a href="EntitySpecification.html" title="Entity Specification"> + <i class="none"></i> + Entity Specification</a> + </li> + + <li> + + <a href="FalconCLI.html" title="Client (Falcon CLI)"> + <i class="none"></i> + Client (Falcon CLI)</a> + </li> + + <li> + + <a href="restapi/ResourceList.html" title="Rest API"> + <i class="icon-chevron-right"></i> + Rest API</a> + </li> + + <li> + + <a href="HiveIntegration.html" title="Hive Integration"> + <i class="none"></i> + Hive Integration</a> + </li> + + <li> + + <a href="Security.html" title="Security"> + <i class="none"></i> + Security</a> + </li> + <li class="nav-header">Project Information</li> + + <li> + + <a href="project-info.html" title="Summary"> + <i class="none"></i> + Summary</a> + </li> + + <li> + + <a href="mail-lists.html" title="Mailing Lists"> + <i class="none"></i> + Mailing Lists</a> + </li> + + <li> + + <a href="http://webchat.freenode.net?channels=apachefalcon&uio=d4"; class="externalLink" title="IRC"> + <i class="none"></i> + IRC</a> + </li> + + <li> + + <a href="team-list.html" title="Team"> + <i class="none"></i> + Team</a> + </li> + + <li> + + <a href="issue-tracking.html" title="Issue Tracking"> + <i class="none"></i> + Issue Tracking</a> + </li> + + <li> + + <a href="source-repository.html" title="Source Repository"> + <i class="none"></i> + Source Repository</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/Index"; class="externalLink" title="Wiki"> + <i class="none"></i> + Wiki</a> + </li> + + <li> + + <a href="license.html" title="License"> + <i class="none"></i> + License</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/News"; class="externalLink" title="News"> + <i class="none"></i> + News</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/PoweredBy"; class="externalLink" title="Powered by"> + <i class="none"></i> + Powered by</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/Acknowledgements"; class="externalLink" title="Acknowledgements"> + <i class="none"></i> + Acknowledgements</a> + </li> + + <li> + + <a href="http://blogs.apache.org/falcon/"; class="externalLink" title="Blog"> + <i class="none"></i> + Blog</a> + </li> + <li class="nav-header">Releases</li> + + <li> + + <a href="http://www.apache.org/dyn/closer.lua/falcon/0.8"; class="externalLink" title="0.8"> + <i class="none"></i> + 0.8</a> + </li> + + <li> + + <a href="http://www.apache.org/dyn/closer.lua/falcon/0.7"; class="externalLink" title="0.7"> + <i class="none"></i> + 0.7</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/falcon/0.6.1"; class="externalLink" title="0.6.1"> + <i class="none"></i> + 0.6.1</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/incubator/falcon/0.6-incubating"; class="externalLink" title="0.6-incubating"> + <i class="none"></i> + 0.6-incubating</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/incubator/falcon/0.5-incubating"; class="externalLink" title="0.5-incubating"> + <i class="none"></i> + 0.5-incubating</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/incubator/falcon/0.4-incubating"; class="externalLink" title="0.4-incubating"> + <i class="none"></i> + 0.4-incubating</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/incubator/falcon/0.3-incubating"; class="externalLink" title="0.3-incubating"> + <i class="none"></i> + 0.3-incubating</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/Roadmap"; class="externalLink" title="Coming soon"> + <i class="none"></i> + Coming soon</a> + </li> + <li class="nav-header">Documentation</li> + + <li> + + <a href="0.9/index.html" title="0.9 (Current)"> + <i class="none"></i> + 0.9 (Current)</a> + </li> + + <li> + + <a href="0.8/index.html" title="0.8"> + <i class="none"></i> + 0.8</a> + </li> + + <li> + + <a href="0.7/index.html" title="0.7"> + <i class="none"></i> + 0.7</a> + </li> + + <li> + + <a href="0.6.1/index.html" title="0.6.1"> + <i class="none"></i> + 0.6.1</a> + </li> + + <li> + + <a href="0.6-incubating/index.html" title="0.6-incubating"> + <i class="none"></i> + 0.6-incubating</a> + </li> + + <li> + + <a href="0.5-incubating/index.html" title="0.5-incubating"> + <i class="none"></i> + 0.5-incubating</a> + </li> + + <li> + + <a href="0.4-incubating/index.html" title="0.4-incubating"> + <i class="none"></i> + 0.4-incubating</a> + </li> + + <li> + + <a href="0.3-incubating/index.html" title="0.3-incubating"> + <i class="none"></i> + 0.3-incubating</a> + </li> + <li class="nav-header">ASF</li> + + <li> + + <a href="http://www.apache.org/foundation/how-it-works.html"; class="externalLink" title="How Apache Works"> + <i class="none"></i> + How Apache Works</a> + </li> + + <li> + + <a href="http://www.apache.org/foundation/"; class="externalLink" title="Foundation"> + <i class="none"></i> + Foundation</a> + </li> + + <li> + + <a href="http://www.apache.org/foundation/sponsorship.html"; class="externalLink" title="Sponsoring Apache"> + <i class="none"></i> + Sponsoring Apache</a> + </li> + + <li> + + <a href="http://www.apache.org/foundation/thanks.html"; class="externalLink" title="Thanks"> + <i class="none"></i> + Thanks</a> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <div class="section"> +<h2>Embedded Mode<a name="Embedded_Mode"></a></h2> +<p>Following are the steps needed to package and deploy Falcon in Embedded Mode. You need to complete Steps 1-3 mentioned <a href="./InstallationSteps.html">here</a> before proceeding further.</p></div> +<div class="section"> +<h3>Package Falcon<a name="Package_Falcon"></a></h3> +<p>Ensure that you are in the base directory (where you cloned Falcon). Let’s call it {project dir}</p> +<div class="source"> +<pre> +$mvn clean assembly:assembly -DskipTests -DskipCheck=true + +</pre></div> +<div class="source"> +<pre> +$ls {project dir}/target/ + +</pre></div> +<p>It should give an output like below :</p> +<div class="source"> +<pre> +apache-falcon-${project.version}-bin.tar.gz +apache-falcon-${project.version}-sources.tar.gz +archive-tmp +maven-shared-archive-resources + +</pre></div> +<p>* apache-falcon-${project.version}-sources.tar.gz contains source files of Falcon repo.</p> +<p>* apache-falcon-${project.version}-bin.tar.gz package contains project artifacts along with it's dependencies, configuration files and scripts required to deploy Falcon.</p> +<p>Tar can be found in {project dir}/target/apache-falcon-${project.version}-bin.tar.gz</p> +<p>Tar is structured as follows :</p> +<div class="source"> +<pre> + +|- bin + |- falcon + |- falcon-start + |- falcon-stop + |- falcon-status + |- falcon-config.sh + |- service-start.sh + |- service-stop.sh + |- service-status.sh +|- conf + |- startup.properties + |- runtime.properties + |- prism.keystore + |- client.properties + |- log4j.xml + |- falcon-env.sh +|- docs +|- client + |- lib (client support libs) +|- server + |- webapp + |- falcon.war +|- data + |- falcon-store + |- graphdb + |- localhost +|- examples + |- app + |- hive + |- oozie-mr + |- pig + |- data + |- entity + |- filesystem + |- hcat +|- oozie + |- conf + |- libext +|- logs +|- hadooplibs +|- README +|- NOTICE.txt +|- LICENSE.txt +|- DISCLAIMER.txt +|- CHANGES.txt + +</pre></div></div> +<div class="section"> +<h3>Installing & running Falcon<a name="Installing__running_Falcon"></a></h3> +<p>Running Falcon in embedded mode requires bringing up server.</p> +<div class="source"> +<pre> +$tar -xzvf {falcon package} +$cd falcon-${project.version} + +</pre></div></div> +<div class="section"> +<h4>Starting Falcon Server<a name="Starting_Falcon_Server"></a></h4> +<div class="source"> +<pre> +$cd falcon-${project.version} +$bin/falcon-start [-port <port>] + +</pre></div> +<p>By default, * If falcon.enableTLS is set to true explicitly or not set at all, Falcon starts at port 15443 on <a class="externalLink" href="https://";>https://</a> by default.</p> +<p>* If falcon.enableTLS is set to false explicitly, Falcon starts at port 15000 on <a class="externalLink" href="http://.";>http://.</a></p> +<p>* To change the port, use -port option.</p> +<p>* If falcon.enableTLS is not set explicitly, port that ends with 443 will automatically put Falcon on <a class="externalLink" href="https://.";>https://.</a> Any other port will put Falcon on <a class="externalLink" href="http://.";>http://.</a></p> +<p>* Server starts with conf from {falcon-server-dir}/falcon-distributed-${project.version}/conf. To override this (to use the same conf with multiple server upgrades), set environment variable FALCON_CONF to the path of conf dir. You can find the instructions for configuring Falcon <a href="./Configuration.html">here</a>.</p></div> +<div class="section"> +<h4>Enabling server-client<a name="Enabling_server-client"></a></h4> +<p>If server is not started using default-port 15443 then edit the following property in {falcon-server-dir}/falcon-${project.version}/conf/client.properties</p> +<p>falcon.url=http://{machine-ip}:{server-port}/</p></div> +<div class="section"> +<h4>Using Falcon<a name="Using_Falcon"></a></h4> +<div class="source"> +<pre> +$cd falcon-${project.version} +$bin/falcon admin -version +Falcon server build version: {Version:"${project.version}-SNAPSHOT-rd7e2be9afa2a5dc96acd1ec9e325f39c6b2f17f7",Mode: +"embedded",Hadoop:"${hadoop.version}"} + +$bin/falcon help +(for more details about Falcon cli usage) + +</pre></div> +<p><b>Note</b> : https is the secure version of HTTP, the protocol over which data is sent between your browser and the website that you are connected to. By default Falcon runs in https mode. But user can configure it to http.</p></div> +<div class="section"> +<h4>Dashboard<a name="Dashboard"></a></h4> +<p>Once Falcon server is started, you can view the status of Falcon entities using the Web-based dashboard. You can open your browser at the corresponding port to use the web UI.</p> +<p>Falcon dashboard makes the REST api calls as user "falcon-dashboard". If this user does not exist on your Falcon and Oozie servers, please create the user.</p> +<div class="source"> +<pre> +## create user. +[root@falconhost ~] useradd -U -m falcon-dashboard -G users + +## verify user is created with membership in correct groups. +[root@falconhost ~] groups falcon-dashboard +falcon-dashboard : falcon-dashboard users +[root@falconhost ~] + +</pre></div></div> +<div class="section"> +<h3>Running Examples using embedded package<a name="Running_Examples_using_embedded_package"></a></h3> +<div class="source"> +<pre> +$cd falcon-${project.version} +$bin/falcon-start + +</pre></div> +<p>Make sure the Hadoop and Oozie endpoints are according to your setup in examples/entity/filesystem/standalone-cluster.xml The cluster locations,staging and working dirs, MUST be created prior to submitting a cluster entity to Falcon. <b>staging</b> must have 777 permissions and the parent dirs must have execute permissions <b>working</b> must have 755 permissions and the parent dirs must have execute permissions</p> +<div class="source"> +<pre> +$bin/falcon entity -submit -type cluster -file examples/entity/filesystem/standalone-cluster.xml + +</pre></div> +<p>Submit input and output feeds:</p> +<div class="source"> +<pre> +$bin/falcon entity -submit -type feed -file examples/entity/filesystem/in-feed.xml +$bin/falcon entity -submit -type feed -file examples/entity/filesystem/out-feed.xml + +</pre></div> +<p>Set-up workflow for the process:</p> +<div class="source"> +<pre> +$hadoop fs -put examples/app / + +</pre></div> +<p>Submit and schedule the process:</p> +<div class="source"> +<pre> +$bin/falcon entity -submitAndSchedule -type process -file examples/entity/filesystem/oozie-mr-process.xml +$bin/falcon entity -submitAndSchedule -type process -file examples/entity/filesystem/pig-process.xml + +</pre></div> +<p>Generate input data:</p> +<div class="source"> +<pre> +$examples/data/generate.sh <<hdfs endpoint>> + +</pre></div> +<p>Get status of instances:</p> +<div class="source"> +<pre> +$bin/falcon instance -status -type process -name oozie-mr-process -start 2013-11-15T00:05Z -end 2013-11-15T01:00Z + +</pre></div> +<p>HCat based example entities are in examples/entity/hcat.</p></div> +<div class="section"> +<h4>Stopping Falcon Server<a name="Stopping_Falcon_Server"></a></h4> +<div class="source"> +<pre> +$cd falcon-${project.version} +$bin/falcon-stop + +</pre></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2013-2016 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html> Modified: falcon/site/EntitySpecification.html URL: http://svn.apache.org/viewvc/falcon/site/EntitySpecification.html?rev=1730450&r1=1730449&r2=1730450&view=diff ============================================================================== --- falcon/site/EntitySpecification.html (original) +++ falcon/site/EntitySpecification.html Mon Feb 15 06:02:08 2016 @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-12-08 + | Generated by Apache Maven Doxia at 2016-02-09 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20151208" /> + <meta name="Date-Revision-yyyymmdd" content="20160209" /> <meta http-equiv="Content-Language" content="en" /> <title>Falcon - Contents</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -58,7 +58,7 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-12-08</li> + <li id="publishDate" class="pull-right">Last Published: 2016-02-09</li> </ul> </div> @@ -306,6 +306,13 @@ <li> + <a href="0.9/index.html" title="0.9 (Current)"> + <i class="none"></i> + 0.9 (Current)</a> + </li> + + <li> + <a href="0.8/index.html" title="0.8"> <i class="none"></i> 0.8</a> @@ -467,7 +474,7 @@ <location name="working" path="/projects/falcon/working" /> <!--optional--> </pre></div> -<p>Location has the name and the path, name is the type of locations .Allowed values of name are staging, temp and working. Path is the hdfs path for each location. Falcon would use the location to do intermediate processing of entities in hdfs and hence Falcon should have read/write/execute permission on these locations. These locations MUST be created prior to submitting a cluster entity to Falcon. <b>staging</b> should have atleast 755 permissions and is a mandatory location .The parent dirs must have execute permissions so multiple users can write to this location. <b>working</b> must have 755 permissions and is a optional location. If <b>working</b> is not specified, falcon creates a sub directory in the <b>staging</b> location with 755 perms. The parent dir for <b>working</b> must have execute permissions so multiple users can read from this location</p></div> +<p>Location has the name and the path, name is the type of locations .Allowed values of name are staging, temp and working. Path is the hdfs path for each location. Falcon would use the location to do intermediate processing of entities in hdfs and hence Falcon should have read/write/execute permission on these locations. These locations MUST be created prior to submitting a cluster entity to Falcon. <b>staging</b> should have 777 permissions and is a mandatory location .The parent dirs must have execute permissions so multiple users can write to this location. <b>working</b> must have 755 permissions and is a optional location. If <b>working</b> is not specified, falcon creates a sub directory in the <b>staging</b> location with 755 perms. The parent dir for <b>working</b> must have execute permissions so multiple users can read from this location</p></div> <div class="section"> <h4>ACL<a name="ACL"></a></h4> <p>A cluster has ACL (Access Control List) useful for implementing permission requirements and provide a way to set different permissions for specific users or named groups.</p> @@ -487,6 +494,50 @@ </pre></div> <p>Ideally JMS impl class name of messaging engine (brokerImplClass) should be defined here.</p></div> <div class="section"> +<h3>Datasource Specification<a name="Datasource_Specification"></a></h3> +<p>The datasource entity contains connection information required to connect to a data source like MySQL database. The datasource XSD specification is available here: A datasource contains read and write interfaces which are used by Falcon to import or export data from or to datasources respectively. A datasource is referenced by feeds which are on-boarded to Falcon by its name.</p> +<p>Following are the tags defined in a datasource.xml:</p> +<div class="source"> +<pre> +<datasource colo="west-coast" description="Customer database on west coast" type="mysql" + name="test-hsql-db" xmlns="uri:falcon:datasource:0.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> + +</pre></div> +<p>The colo specifies the colo to which the datasource belongs to and name is the name of the datasource which has to be unique.</p></div> +<div class="section"> +<h4>Interfaces<a name="Interfaces"></a></h4> +<p>A datasource has two interfaces as described below:</p> +<div class="source"> +<pre> + <interface type="readonly" endpoint="jdbc:hsqldb:localhost/db"/> + +</pre></div> +<p>A readonly interface specifies the endpoint and protocol to connect to a datasource. This would be used in the context of import from datasource into HDFS.</p> +<div class="source"> +<pre> +<interface type="write" endpoint="jdbc:hsqldb:localhost/db1"> + +</pre></div> +<p>A write interface specifies the endpoint and protocol to to write to the datasource. Falcon uses this interface to export data from hdfs to datasource.</p> +<div class="source"> +<pre> +<credential type="password-text"> + <userName>SA</userName> + <passwordText></passwordText> +</credential> + +</pre></div> +<p>A credential is associated with an interface (read or write) providing user name and password to authenticate to the datasource.</p> +<div class="source"> +<pre> +<credential type="password-text"> + <userName>SA</userName> + <passwordFile>hdfs-file-path</passwordText> +</credential> + +</pre></div> +<p>The credential can be specified via a password file present in the HDFS. This file should only be accessible by the user.</p></div> +<div class="section"> <h3>Feed Specification<a name="Feed_Specification"></a></h3> <p>The Feed XSD specification is available here. A Feed defines various attributes of feed like feed location, frequency, late-arrival handling and retention policies. A feed can be scheduled on a cluster, once a feed is scheduled its retention and replication process are triggered in a given cluster.</p> <div class="source"> @@ -612,15 +663,53 @@ catalog:$database-name:$table-name#parti </pre></div> <p>A feed can have SLA and each SLA has two properties - slaLow and slaHigh. Both slaLow and slaHigh are written using expressions like frequency. slaLow is intended to serve for alerting for feed instances which are in danger of missing their availability SLAs. slaHigh is intended to serve for reporting the feeds which missed their SLAs. SLAs are relative to feed instance time.</p></div> <div class="section"> -<h4>Late Arrival<a name="Late_Arrival"></a></h4> +<h4>Import<a name="Import"></a></h4> <div class="source"> <pre> +<import> + <source name="test-hsql-db" tableName="customer"> + <extract type="full"> + <mergepolicy>snapshot</mergepolicy> + </extract> + <fields> + <includes> + <field>id</field> + <field>name</field> + </includes> + </fields> + </source> + <arguments> + <argument name="--split-by" value="id"/> + <argument name="--num-mappers" value="2"/> + </arguments> +</import> + +A feed can have an import policy associated with it. The souce name specified the datasource reference to the +datasource entity from which the data will be imported to HDFS. The tableName spcified the table or topic to be +imported from the datasource. The extract type specifies the pull mechanism (full or +incremental extract). Full extract method extracts all the data from the datasource. The incremental extraction +method feature implementation is in progress. The mergeplocy determines how the data is to be layed out on HDFS. +The snapshot layout creates a snapshot of the data on HDFS using the feed's location specification. Fields is used +to specify the projection columns. Feed import from database underneath uses sqoop to achieve the task. Any advanced +Sqoop options can be specified via the arguments. + +---+++ Late Arrival + +<verbatim> <late-arrival cut-off="hours(6)" /> </pre></div> <p>A late-arrival specifies the cut-off period till which the feed is expected to arrive late and should be honored be processes referring to it as input feed by rerunning the instances in case the data arrives late with in a cut-off period. The cut-off period is specified by expression frequency(times), ex: if the feed can arrive late upto 8 hours then late-arrival's cut-off="hours(8)"</p> <p><b>Note:</b> This will only apply for FileSystem storage but not Table storage until a future time.</p></div> <div class="section"> +<h4>Email Notification<a name="Email_Notification"></a></h4> +<div class="source"> +<pre> + <notification type="email" to="[email protected]"/> + +</pre></div> +<p>Specifying the notification element with "type" property allows users to receive email notification when a scheduled feed instance completes. Multiple recipients of an email can be provided as comma separated addresses with "to" property. To send email notification ensure that SMTP parameters are defined in Falcon startup.properties. Refer to <a href="./FalconEmailNotification.html">Falcon Email Notification</a> for more details.</p></div> +<div class="section"> <h4>ACL<a name="ACL"></a></h4> <p>A feed has ACL (Access Control List) useful for implementing permission requirements and provide a way to set different permissions for specific users or named groups.</p> <div class="source"> @@ -642,11 +731,41 @@ catalog:$database-name:$table-name#parti <property name="parallel" value="3"/> <property name="maxMaps" value="8"/> <property name="mapBandwidth" value="1"/> + <property name="overwrite" value="true"/> + <property name="ignoreErrors" value="false"/> + <property name="skipChecksum" value="false"/> + <property name="removeDeletedFiles" value="true"/> + <property name="preserveBlockSize" value="true"/> + <property name="preserveReplicationNumber" value="true"/> + <property name="preservePermission" value="true"/> <property name="order" value="LIFO"/> </properties> </pre></div> -<p>A key-value pair, which are propagated to the workflow engine. "queueName" and "jobPriority" are special properties available to user to specify the Hadoop job queue and priority, the same values are used by Falcon's launcher job. "timeout", "parallel" and "order" are other special properties which decides replication instance's timeout value while waiting for the feed instance, parallel decides the concurrent replication instances that can run at any given time and order decides the execution order for replication instances like FIFO, LIFO and LAST_ONLY. "maxMaps" represents the maximum number of maps used during replication. "mapBandwidth" represents the bandwidth in MB/s used by each mapper during replication.</p></div> +<p>A key-value pair, which are propagated to the workflow engine. "queueName" and "jobPriority" are special properties available to user to specify the Hadoop job queue and priority, the same values are used by Falcon's launcher job. "timeout", "parallel" and "order" are other special properties which decides replication instance's timeout value while waiting for the feed instance, parallel decides the concurrent replication instances that can run at any given time and order decides the execution order for replication instances like FIFO, LIFO and LAST_ONLY. <a href="./DistCp.html">DistCp</a> options can be passed as custom properties, which will be propagated to the <a href="./DistCp.html">DistCp</a> tool. "maxMaps" represents the maximum number of maps used during replication. "mapBandwidth" represents the bandwidth in MB/s used by each mapper during replication. "overwrite" represents overwrite destin ation during replication. "ignoreErrors" represents ignore failures not causing the job to fail during replication. "skipChecksum" represents bypassing checksum verification during replication. "removeDeletedFiles" represents deleting the files existing in the destination but not in source during replication. "preserveBlockSize" represents preserving block size during replication. "preserveReplicationNumber" represents preserving replication number during replication. "preservePermission" represents preserving permission during</p></div> +<div class="section"> +<h4>Lifecycle<a name="Lifecycle"></a></h4> +<div class="source"> +<pre> + +<lifecycle> + <retention-stage> + <frequency>hours(10)</frequency> + <queue>reports</queue> + <priority>NORMAL</priority> + <properties> + <property name="retention.policy.agebaseddelete.limit" value="hours(9)"></property> + </properties> + </retention-stage> +</lifecycle> + + +</pre></div> +<p>lifecycle tag is the new way to define various stages of a feed's lifecycle. In the example above we have defined a retention-stage using lifecycle tag. You may define lifecycle at global level or a cluster level or both. Cluster level configuration takes precedence and falcon falls back to global definition if cluster level specification is missing.</p> +<p>----++++ Retention Stage As of now there are two ways to specify retention. One is through the <retention> tag in the cluster and another is the new way through <retention-stage> tag in <lifecycle> tag. If both are defined for a feed, then the lifecycle tag will be considered effective and falcon will ignore the <retention> tag in the cluster. If there is an invalid configuration of retention-stage in lifecycle tag, then falcon will <b>NOT</b> fall back to retention tag even if it is defined and will throw validation error.</p> +<p>In this new method of defining retention you can specify the frequency at which the retention should occur, you can also define the queue and priority parameters for retention jobs. The default behavior of retention-stage is same as the existing one which is to delete all instances corresponding to instance-time earlier than the duration provided in "retention.policy.agebaseddelete.limit"</p> +<p>Property "retention.policy.agebaseddelete.limit" is a mandatory property and must contain a valid duration e.g. "hours(1)" Retention frequency is not a mandatory parameter. If user doesn't specify the frequency in the retention stage then it doesn't fallback to old retention policy frequency. Its default value is set to 6 hours if feed frequency is less than 6 hours else its set to feed frequency as retention shouldn't be more frequent than data availability to avoid wastage of compute resources.</p> +<p>In future, we will allow more customisation like customising how to choose instances to be deleted through this method.</p></div> <div class="section"> <h3>Process Specification<a name="Process_Specification"></a></h3> <p>A process defines configuration for a workflow. A workflow is a directed acyclic graph(DAG) which defines the job for the workflow engine. A process definition defines the configurations required to run the workflow job. For example, process defines the frequency at which the workflow should run, the clusters on which the workflow should run, the inputs and outputs for the workflow, how the workflow failures should be handled, how the late inputs should be handled and so on.</p> @@ -1016,18 +1135,20 @@ catalog:$database-name:$table-name#parti </process> </pre></div> -<p>queueName and jobPriority are special properties, which when present are used by the Falcon's launcher job, the same property is also available in workflow which can be used to propagate to pig or M/R job.</p> +<p>The following are some special properties, which when present are used by the Falcon's launcher job, the same property is also available in workflow which can be used to propagate to pig or M/R job.</p> <div class="source"> <pre> <property name="queueName" value="hadoopQueue"/> <property name="jobPriority" value="VERY_HIGH"/> + <!-- This property is used to turn off JMS notifications for this process. JMS notifications are enabled by default. --> + <property name="userJMSNotificationEnabled" value="false"/> </pre></div></div> <div class="section"> <h4>Workflow<a name="Workflow"></a></h4> <p>The workflow defines the workflow engine that should be used and the path to the workflow on hdfs. The workflow definition on hdfs contains the actual job that should run and it should confirm to the workflow specification of the engine specified. The libraries required by the workflow should be in lib folder inside the workflow path.</p> <p>The properties defined in the cluster and cluster properties(nameNode and jobTracker) will also be available for the workflow.</p> -<p>There are 2 engines supported today.</p></div> +<p>There are 3 engines supported today.</p></div> <div class="section"> <h5>Oozie<a name="Oozie"></a></h5> <p>As part of oozie workflow engine support, users can embed a oozie workflow. Refer to oozie <a class="externalLink" href="http://oozie.apache.org/docs/4.0.1/DG_Overview.html";>workflow overview</a> and <a class="externalLink" href="http://oozie.apache.org/docs/4.0.1/WorkflowFunctionalSpec.html";>workflow specification</a> for details.</p> @@ -1090,7 +1211,7 @@ catalog:$database-name:$table-name#parti </pre></div></div> <div class="section"> <h4>Retry<a name="Retry"></a></h4> -<p>Retry policy defines how the workflow failures should be handled. Two retry policies are defined: backoff and exp-backoff(exponential backoff). Depending on the delay and number of attempts, the workflow is re-tried after specific intervals. Syntax:</p> +<p>Retry policy defines how the workflow failures should be handled. Three retry policies are defined: periodic, exp-backoff(exponential backoff) and final. Depending on the delay and number of attempts, the workflow is re-tried after specific intervals. Syntax:</p> <div class="source"> <pre> <process name="[process name]"> @@ -1105,7 +1226,7 @@ catalog:$database-name:$table-name#parti <pre> <process name="sample-process"> ... - <retry policy="backoff" delay="minutes(10)" attempts="3"/> + <retry policy="periodic" delay="minutes(10)" attempts="3"/> ... </process> @@ -1155,6 +1276,14 @@ catalog:$database-name:$table-name#parti <p>This late handling specifies that late data detection should run at feed's late cut-off which is 6 hours in this case. If there is late data, Falcon should run the workflow specified at /projects/bootcamp/workflow/lateinput1/workflow.xml</p> <p><b>Note:</b> This is only supported for FileSystem storage but not Table storage at this point.</p></div> <div class="section"> +<h4>Email Notification<a name="Email_Notification"></a></h4> +<div class="source"> +<pre> + <notification type="email" to="bob@@xyz.com"/> + +</pre></div> +<p>Specifying the notification element with "type" property allows users to receive email notification when a scheduled process instance completes. Multiple recipients of an email can be provided as comma separated addresses with "to" property. To send email notification ensure that SMTP parameters are defined in Falcon startup.properties. Refer to <a href="./FalconEmailNotification.html">Falcon Email Notification</a> for more details.</p></div> +<div class="section"> <h4>ACL<a name="ACL"></a></h4> <p>A process has ACL (Access Control List) useful for implementing permission requirements and provide a way to set different permissions for specific users or named groups.</p> <div class="source"> @@ -1171,7 +1300,7 @@ catalog:$database-name:$table-name#parti <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2013-2015 + <div class="row span12">Copyright © 2013-2016 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved.
