Added: falcon/site/ImportExport.html URL: http://svn.apache.org/viewvc/falcon/site/ImportExport.html?rev=1730450&view=auto ============================================================================== --- falcon/site/ImportExport.html (added) +++ falcon/site/ImportExport.html Mon Feb 15 06:02:08 2016 @@ -0,0 +1,636 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-02-09 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160209" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Falcon - Falcon Data Import and Export</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="../index.html" id="bannerLeft"> + <img src="images/falcon-logo.png" alt="Falcon" width="200px" height="45px"/> + </a> + </div> + <div class="pull-right"> <a href="http://www.apache.org"; id="bannerRight"> + <img src="images/apache-feather-tm.gif" alt="Falcon" height="45px"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="index.html" title="Falcon"> + Falcon</a> + </li> + <li class="divider ">/</li> + <li class="">Falcon Data Import and Export</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2016-02-09</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">Falcon</li> + + <li> + + <a href="index.html" title="About"> + <i class="none"></i> + About</a> + </li> + + <li> + + <a href="slides/falcon-overview.html" title="Overview"> + <i class="none"></i> + Overview</a> + </li> + + <li> + + <a href="slides/falcon-user-guide.html" title="User Guide"> + <i class="none"></i> + User Guide</a> + </li> + + <li> + + <a href="GettingStarted.html" title="Getting Started"> + <i class="none"></i> + Getting Started</a> + </li> + + <li> + + <a href="FalconDocumentation.html" title="Architecture"> + <i class="none"></i> + Architecture</a> + </li> + + <li> + + <a href="InstallationSteps.html" title="Installation"> + <i class="none"></i> + Installation</a> + </li> + + <li> + + <a href="OnBoarding.html" title="On Boarding"> + <i class="none"></i> + On Boarding</a> + </li> + + <li> + + <a href="Operability.html" title="Operability"> + <i class="none"></i> + Operability</a> + </li> + + <li> + + <a href="EntitySpecification.html" title="Entity Specification"> + <i class="none"></i> + Entity Specification</a> + </li> + + <li> + + <a href="FalconCLI.html" title="Client (Falcon CLI)"> + <i class="none"></i> + Client (Falcon CLI)</a> + </li> + + <li> + + <a href="restapi/ResourceList.html" title="Rest API"> + <i class="icon-chevron-right"></i> + Rest API</a> + </li> + + <li> + + <a href="HiveIntegration.html" title="Hive Integration"> + <i class="none"></i> + Hive Integration</a> + </li> + + <li> + + <a href="Security.html" title="Security"> + <i class="none"></i> + Security</a> + </li> + <li class="nav-header">Project Information</li> + + <li> + + <a href="project-info.html" title="Summary"> + <i class="none"></i> + Summary</a> + </li> + + <li> + + <a href="mail-lists.html" title="Mailing Lists"> + <i class="none"></i> + Mailing Lists</a> + </li> + + <li> + + <a href="http://webchat.freenode.net?channels=apachefalcon&uio=d4"; class="externalLink" title="IRC"> + <i class="none"></i> + IRC</a> + </li> + + <li> + + <a href="team-list.html" title="Team"> + <i class="none"></i> + Team</a> + </li> + + <li> + + <a href="issue-tracking.html" title="Issue Tracking"> + <i class="none"></i> + Issue Tracking</a> + </li> + + <li> + + <a href="source-repository.html" title="Source Repository"> + <i class="none"></i> + Source Repository</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/Index"; class="externalLink" title="Wiki"> + <i class="none"></i> + Wiki</a> + </li> + + <li> + + <a href="license.html" title="License"> + <i class="none"></i> + License</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/News"; class="externalLink" title="News"> + <i class="none"></i> + News</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/PoweredBy"; class="externalLink" title="Powered by"> + <i class="none"></i> + Powered by</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/Acknowledgements"; class="externalLink" title="Acknowledgements"> + <i class="none"></i> + Acknowledgements</a> + </li> + + <li> + + <a href="http://blogs.apache.org/falcon/"; class="externalLink" title="Blog"> + <i class="none"></i> + Blog</a> + </li> + <li class="nav-header">Releases</li> + + <li> + + <a href="http://www.apache.org/dyn/closer.lua/falcon/0.8"; class="externalLink" title="0.8"> + <i class="none"></i> + 0.8</a> + </li> + + <li> + + <a href="http://www.apache.org/dyn/closer.lua/falcon/0.7"; class="externalLink" title="0.7"> + <i class="none"></i> + 0.7</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/falcon/0.6.1"; class="externalLink" title="0.6.1"> + <i class="none"></i> + 0.6.1</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/incubator/falcon/0.6-incubating"; class="externalLink" title="0.6-incubating"> + <i class="none"></i> + 0.6-incubating</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/incubator/falcon/0.5-incubating"; class="externalLink" title="0.5-incubating"> + <i class="none"></i> + 0.5-incubating</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/incubator/falcon/0.4-incubating"; class="externalLink" title="0.4-incubating"> + <i class="none"></i> + 0.4-incubating</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/incubator/falcon/0.3-incubating"; class="externalLink" title="0.3-incubating"> + <i class="none"></i> + 0.3-incubating</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/Roadmap"; class="externalLink" title="Coming soon"> + <i class="none"></i> + Coming soon</a> + </li> + <li class="nav-header">Documentation</li> + + <li> + + <a href="0.9/index.html" title="0.9 (Current)"> + <i class="none"></i> + 0.9 (Current)</a> + </li> + + <li> + + <a href="0.8/index.html" title="0.8"> + <i class="none"></i> + 0.8</a> + </li> + + <li> + + <a href="0.7/index.html" title="0.7"> + <i class="none"></i> + 0.7</a> + </li> + + <li> + + <a href="0.6.1/index.html" title="0.6.1"> + <i class="none"></i> + 0.6.1</a> + </li> + + <li> + + <a href="0.6-incubating/index.html" title="0.6-incubating"> + <i class="none"></i> + 0.6-incubating</a> + </li> + + <li> + + <a href="0.5-incubating/index.html" title="0.5-incubating"> + <i class="none"></i> + 0.5-incubating</a> + </li> + + <li> + + <a href="0.4-incubating/index.html" title="0.4-incubating"> + <i class="none"></i> + 0.4-incubating</a> + </li> + + <li> + + <a href="0.3-incubating/index.html" title="0.3-incubating"> + <i class="none"></i> + 0.3-incubating</a> + </li> + <li class="nav-header">ASF</li> + + <li> + + <a href="http://www.apache.org/foundation/how-it-works.html"; class="externalLink" title="How Apache Works"> + <i class="none"></i> + How Apache Works</a> + </li> + + <li> + + <a href="http://www.apache.org/foundation/"; class="externalLink" title="Foundation"> + <i class="none"></i> + Foundation</a> + </li> + + <li> + + <a href="http://www.apache.org/foundation/sponsorship.html"; class="externalLink" title="Sponsoring Apache"> + <i class="none"></i> + Sponsoring Apache</a> + </li> + + <li> + + <a href="http://www.apache.org/foundation/thanks.html"; class="externalLink" title="Thanks"> + <i class="none"></i> + Thanks</a> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <div class="section"> +<h2>Falcon Data Import and Export<a name="Falcon_Data_Import_and_Export"></a></h2></div> +<div class="section"> +<h3>Overview<a name="Overview"></a></h3> +<p>Falcon provides constructs to periodically bring raw data from external data sources (like databases, drop boxes etc) onto Hadoop and push derived data computed on Hadoop onto external data sources.</p> +<p>As of this release, Falcon only supports Relational Databases (e.g. Oracle, MySQL etc) via JDBC as external data source. The future releases will add support for other external data sources.</p></div> +<div class="section"> +<h3>Prerequisites<a name="Prerequisites"></a></h3> +<p>Following are the prerequisites to import external data from and export to databases.</p> +<p></p> +<ul> +<li><b>Sqoop 1.4.6+</b></li> +<li><b>Oozie 4.2.0+</b></li> +<li><b>Appropriate database connector</b></li></ul> +<p><b>Note:</b> Falcon uses Sqoop for import/export operation. Sqoop will require appropriate database driver to connect to the relational database. Please refer to the Sqoop documentation for any Sqoop related question. Please make sure the database driver jar is copied into oozie share lib for Sqoop.</p> +<div class="source"> +<pre> +For example, in order to import and export with MySQL, please make sure the latest MySQL connector +*mysql-connector-java-5.1.31.jar+* is copied into oozie's Sqoop share lib + +/user/oozie/share/lib/{lib-dir}/sqoop/mysql-connector-java-5.1.31.jar+ + +where {lib-dir} value varies in oozie deployments. + + +</pre></div></div> +<div class="section"> +<h3>Usage<a name="Usage"></a></h3></div> +<div class="section"> +<h4>Entity Definition and Setup<a name="Entity_Definition_and_Setup"></a></h4> +<p></p> +<ul> +<li><b>Datasource Entity</b></li></ul>Datasource entity abstracts connection and credential details to external data sources. The Datasource entity supports read and write interfaces with specific credentials. The default credential will be used if the read or write interface does not have its own credentials. In general, the Datasource entity will be defined by system administrator. Please refer to datasource XSD for more details. +<p>The following example defines a Datasource entity for a MySQL database. The import operation will use the read interface with url "jdbc:mysql://dbhost/test", user name "import_usr" and password text "sqoop". Where as, the export operation will use the write interface with url "jdbc:mysql://dbhost/test" with user name "export_usr" and password specified in a HDFS file at the location "/user/ambari-qa/password-store/password_write_user".</p> +<p>The default credential specifies the password using password text and will be used if either read or write interface does not provide credentials.</p> +<p>The available read and write interfaces enable database administrators to segregate read and write workloads.</p> +<div class="source"> +<pre> + + File: mysql-database.xml + + <?xml version="1.0" encoding="UTF-8"?> + <datasource colo="west-coast" description="MySQL database on west coast" type="mysql" name="mysql-db" xmlns="uri:falcon:datasource:0.1"> + <tags>[email protected], [email protected]</tags> + <interfaces> + <!-- ***** read interface ***** --> + <interface type="readonly" endpoint="jdbc:mysql://dbhost/test"> + <credential type="password-text"> + <userName>import_usr</userName> + <passwordText>sqoop</passwordFile> + </credential> + </interface> + + <!-- ***** write interface ***** --> + <interface type="write" endpoint="jdbc:mysql://dbhost/test"> + <credential type="password-file"> + <userName>export_usr</userName> + <passwordFile>/user/ambari-qa/password-store/password_write_user</passwordFile> + </credential> + </interface> + + <!-- *** default credential *** --> + <credential type="password-text"> + <userName>sqoop2_user</userName> + <passwordText>sqoop</passwordText> + </credential> + + </interfaces> + + <driver> + <clazz>com.mysql.jdbc.Driver</clazz> + <jar>/user/oozie/share/lib/lib_20150721010816/sqoop/mysql-connector-java-5.1.31</jar> + </driver> + </datasource> + +</pre></div> +<p></p> +<ul> +<li><b>Feed Entity</b></li></ul>Feed entity now enables users to define IMPORT and EXPORT policies in addition to RETENTION and REPLICATION. The IMPORT and EXPORT policies will refer to a already defined Datasource entity for connection and credential details and take a table name from the policy to operate on. Please refer to feed entity XSD for details. +<p>The following example defines a Feed entity with IMPORT and EXPORT policies. Both the IMPORT and EXPORT operations refer to a datasource entity "mysql-db". The IMPORT operation will use the read interface and credentials while the EXPORT operation will use the write interface and credentials. A feed instance is created every 1 hour since the frequency of the Feed is hour(1) and the Feed instances are deleted after 90 days because of the retention policy.</p> +<div class="source"> +<pre> + + File: customer_email_feed.xml + + <?xml version="1.0" encoding="UTF-8"?> + <!-- + A feed representing Hourly customer email data retained for 90 days + --> + <feed description="Raw customer email feed" name="customer_feed" xmlns="uri:falcon:feed:0.1"> + <tags>externalSystem=USWestEmailServers,classification=secure</tags> + <groups>DataImportPipeline</groups> + <frequency>hours(1)</frequency> + <late-arrival cut-off="hours(4)"/> + <clusters> + <cluster name="primaryCluster" type="source"> + <validity start="2015-12-15T00:00Z" end="2016-03-31T00:00Z"/> + <retention limit="days(90)" action="delete"/> + <import> + <source name="mysql-db" tableName="simple"> + <extract type="full"> + <mergepolicy>snapshot</mergepolicy> + </extract> + <fields> + <includes> + <field>id</field> + <field>name</field> + </includes> + </fields> + </source> + <arguments> + <argument name="--split-by" value="id"/> + <argument name="--num-mappers" value="2"/> + </arguments> + </import> + <export> + <target name="mysql-db" tableName="simple_export"> + <load type="insert"/> + <fields> + <includes> + <field>id</field> + <field>name</field> + </includes> + </fields> + </target> + <arguments> + <argument name="--update-key" value="id"/> + </arguments> + </export> + </cluster> + </clusters> + + <locations> + <location type="data" path="/user/ambari-qa/falcon/demo/primary/importfeed/${YEAR}-${MONTH}-${DAY}-${HOUR}-${MINUTE}"/> + <location type="stats" path="/none"/> + <location type="meta" path="/none"/> + </locations> + + <ACL owner="ambari-qa" group="users" permission="0755"/> + <schema location="/none" provider="none"/> + + </feed> + +</pre></div> +<p></p> +<ul> +<li><b>Import policy</b></li></ul>The import policy uses the datasource entity specified in the "source" to connect to the database. The tableName specified should exist in the source datasource. +<p>Extraction type specifies whether to pull data from external datasource "full" everytime or "incrementally". The mergepolicy specifies how to organize (snapshot or append, i.e time series partiitons) the data on hadoop. The valid combinations are:</p> +<ul> +<li> +<ul> +<li>[full,snapshot] - data is extracted in full and dumped into the feed instance location.</li> +<li>[incremental, append] - data is extracted incrementally using the key specified in the <b>deltacolumn</b></li></ul></li></ul>and added as a partition to the feed instance location. +<ul> +<li> +<ul> +<li>[incremental, snapshot] - data is extracted incrementally and merged with already existing data on hadoop to</li></ul></li></ul>produce one latest feed instance.*This feature is not supported currently*. The use case for this feature is to efficiently import very large dimention tables that have updates and inserts onto hadoop and make it available as a snapshot with latest updates to consumers. +<p>The following example defines an incremental extraction with append organization:</p> +<div class="source"> +<pre> + <import> + <source name="mysql-db" tableName="simple"> + <extract type="incremental"> + <deltacolumn>modified_time</deltacolumn> + <mergepolicy>append</mergepolicy> + </extract> + <fields> + <includes> + <field>id</field> + <field>name</field> + </includes> + </fields> + </source> + <arguments> + <argument name="--split-by" value="id"/> + <argument name="--num-mappers" value="2"/> + </arguments> + </import> + +</pre></div> +<p>The fields option enables to control what fields get imported. By default, all fields get import. The "includes" option brings only those fields specified. The "excludes" option brings all the fields other than specified.</p> +<p>The arguments section enables to pass in any extra arguments needed for fine control on the underlying implementation -- in this case, Sqoop.</p> +<p></p> +<ul> +<li><b>Export policy</b></li></ul> +<p>The export, like import, uses the datasource for connecting to the database. Load type specifies whether to insert or only update data onto the external table. Fields option behaves the same way as in import policy. The tableName specified should exist in the external datasource.</p></div> +<div class="section"> +<h4>Operation<a name="Operation"></a></h4> +<p>Once the Datasource and Feed entity with import and export policies are defined, Users can submit and schedule the Import and Export operations via CLI and REST API as below:</p> +<div class="source"> +<pre> + + ## submit the mysql-db datasource defined in the file mysql_datasource.xml + falcon entity -submit -type datasource -file mysql_datasource.xml + + ## submit the customer_feed specified in the customer_email_feed.xml + falcon entity -submit -type feed -file customer_email_feed.xml + + ## schedule the customer_feed + falcon entity -schedule -type feed -name customer_feed + + +</pre></div> +<p>Falcon will create corresponding oozie bundles with coordinator and workflow for import and export operation.</p></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2013-2016 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html>
Modified: falcon/site/InstallationSteps.html URL: http://svn.apache.org/viewvc/falcon/site/InstallationSteps.html?rev=1730450&r1=1730449&r2=1730450&view=diff ============================================================================== --- falcon/site/InstallationSteps.html (original) +++ falcon/site/InstallationSteps.html Mon Feb 15 06:02:08 2016 @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-12-08 + | Generated by Apache Maven Doxia at 2016-02-09 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20151208" /> + <meta name="Date-Revision-yyyymmdd" content="20160209" /> <meta http-equiv="Content-Language" content="en" /> <title>Falcon - Building & Installing Falcon</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -58,7 +58,7 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-12-08</li> + <li id="publishDate" class="pull-right">Last Published: 2016-02-09</li> </ul> </div> @@ -306,6 +306,13 @@ <li> + <a href="0.9/index.html" title="0.9 (Current)"> + <i class="none"></i> + 0.9 (Current)</a> + </li> + + <li> + <a href="0.8/index.html" title="0.8"> <i class="none"></i> 0.8</a> @@ -402,319 +409,66 @@ <div id="bodyColumn" class="span9" > <div class="section"> -<h3>Building & Installing Falcon<a name="Building__Installing_Falcon"></a></h3></div> +<h2>Building & Installing Falcon<a name="Building__Installing_Falcon"></a></h2></div> <div class="section"> -<h4>Building Falcon<a name="Building_Falcon"></a></h4> -<div class="source"> -<pre> -You would need the following installed to build Falcon - -* JDK 1.7 -* Maven 3.x - -git clone https://git-wip-us.apache.org/repos/asf/falcon.git falcon - -cd falcon - -export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m -noverify" && mvn clean install - -[optionally -Dhadoop.version=<<hadoop.version>> can be appended to build for a specific version of hadoop] -*Note:* Falcon drops support for Hadoop-1 and only supports Hadoop-2 from Falcon 0.6 onwards -[optionally -Doozie.version=<<oozie version>> can be appended to build with a specific version of oozie. -Oozie versions >= 4 are supported] -Falcon build with JDK 1.7 using -noverify option - - -</pre></div> -<p>Once the build successfully completes, artifacts can be packaged for deployment. The package can be built in embedded or distributed mode.</p> -<p><b>Embedded Mode</b></p> -<div class="source"> -<pre> - -mvn clean assembly:assembly -DskipTests -DskipCheck=true - - -</pre></div> -<p>Tar can be found in {project dir}/target/apache-falcon-${project.version}-bin.tar.gz</p> -<p>Tar is structured as follows</p> -<div class="source"> -<pre> - -|- bin - |- falcon - |- falcon-start - |- falcon-stop - |- falcon-config.sh - |- service-start.sh - |- service-stop.sh -|- conf - |- startup.properties - |- runtime.properties - |- client.properties - |- log4j.xml - |- falcon-env.sh -|- docs -|- client - |- lib (client support libs) -|- server - |- webapp - |- falcon.war -|- hadooplibs -|- README -|- NOTICE.txt -|- LICENSE.txt -|- DISCLAIMER.txt -|- CHANGES.txt - -</pre></div> -<p><b>Distributed Mode</b></p> -<div class="source"> -<pre> - -mvn clean assembly:assembly -DskipTests -DskipCheck=true -Pdistributed,hadoop-2 - - -</pre></div> -<p>Tar can be found in {project dir}/target/apache-falcon-distributed-${project.version}-server.tar.gz</p> -<p>Tar is structured as follows</p> -<div class="source"> -<pre> - -|- bin - |- falcon - |- falcon-start - |- falcon-stop - |- falcon-config.sh - |- service-start.sh - |- service-stop.sh - |- prism-stop - |- prism-start -|- conf - |- startup.properties - |- runtime.properties - |- client.properties - |- log4j.xml - |- falcon-env.sh -|- docs -|- client - |- lib (client support libs) -|- server - |- webapp - |- falcon.war - |- prism.war -|- hadooplibs -|- README -|- NOTICE.txt -|- LICENSE.txt -|- DISCLAIMER.txt -|- CHANGES.txt - -</pre></div></div> +<h3>Building Falcon<a name="Building_Falcon"></a></h3></div> <div class="section"> -<h4>Installing & running Falcon<a name="Installing__running_Falcon"></a></h4> -<p><b>Installing falcon</b></p> -<div class="source"> -<pre> -tar -xzvf {falcon package} -cd falcon-distributed-${project.version} or falcon-${project.version} - -</pre></div> -<p><b>Configuring Falcon</b></p> -<p>By default config directory used by falcon is {package dir}/conf. To override this set environment variable FALCON_CONF to the path of the conf dir.</p> -<p>falcon-env.sh has been added to the falcon conf. This file can be used to set various environment variables that you need for you services. In addition you can set any other environment variables you might need. This file will be sourced by falcon scripts before any commands are executed. The following environment variables are available to set.</p> -<div class="source"> -<pre> -# The java implementation to use. If JAVA_HOME is not found we expect java and jar to be in path -#export JAVA_HOME= - -# any additional java opts you want to set. This will apply to both client and server operations -#export FALCON_OPTS= - -# any additional java opts that you want to set for client only -#export FALCON_CLIENT_OPTS= - -# java heap size we want to set for the client. Default is 1024MB -#export FALCON_CLIENT_HEAP= - -# any additional opts you want to set for prism service. -#export FALCON_PRISM_OPTS= - -# java heap size we want to set for the prism service. Default is 1024MB -#export FALCON_PRISM_HEAP= - -# any additional opts you want to set for falcon service. -#export FALCON_SERVER_OPTS= - -# java heap size we want to set for the falcon server. Default is 1024MB -#export FALCON_SERVER_HEAP= - -# What is is considered as falcon home dir. Default is the base location of the installed software -#export FALCON_HOME_DIR= - -# Where log files are stored. Default is logs directory under the base install location -#export FALCON_LOG_DIR= - -# Where pid files are stored. Default is logs directory under the base install location -#export FALCON_PID_DIR= - -# where the falcon active mq data is stored. Default is logs/data directory under the base install location -#export FALCON_DATA_DIR= - -# Where do you want to expand the war file. By Default it is in /server/webapp dir under the base install dir. -#export FALCON_EXPANDED_WEBAPP_DIR= - -</pre></div> -<p><b>Configuring Monitoring plugin to register catalog partitions</b> Falcon comes with a monitoring plugin that registers catalog partition. This comes in really handy during migration from filesystem based feeds to hcatalog based feeds. This plugin enables the user to de-couple the partition registration and assume that all partitions are already on hcatalog even before the migration, simplifying the hcatalog migration.</p> -<p>By default this plugin is disabled. To enable this plugin and leverage the feature, there are 3 pre-requisites:</p> -<div class="source"> -<pre> -In {package dir}/conf/startup.properties, add -*.workflow.execution.listeners=org.apache.falcon.catalog.CatalogPartitionHandler - -In the cluster definition, ensure registry endpoint is defined. -Ex: -<interface type="registry" endpoint="thrift://localhost:1109" version="0.13.3"/> - -In the feed definition, ensure the corresponding catalog table is mentioned in feed-properties -Ex: -<properties> - <property name="catalog.table" value="catalog:default:in_table#year={YEAR};month={MONTH};day={DAY};hour={HOUR};minute={MINUTE}"/> -</properties> - -</pre></div> -<p><b>NOTE for Mac OS users</b></p> -<div class="source"> -<pre> -If you are using a Mac OS, you will need to configure the FALCON_SERVER_OPTS (explained above). - -In {package dir}/conf/falcon-env.sh uncomment the following line -#export FALCON_SERVER_OPTS= - -and change it to look as below -export FALCON_SERVER_OPTS="-Djava.awt.headless=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc=" - -</pre></div> -<p><b>Starting Falcon Server</b></p> -<div class="source"> -<pre> -bin/falcon-start [-port <port>] - -</pre></div> -<p>By default, * If falcon.enableTLS is set to true explicitly or not set at all, falcon starts at port 15443 on <a class="externalLink" href="https://";>https://</a> by default. * If falcon.enableTLS is set to false explicitly, falcon starts at port 15000 on <a class="externalLink" href="http://.";>http://.</a> * To change the port, use -port option.</p> -<ul> -<li>If falcon.enableTLS is not set explicitly, port that ends with 443 will automatically put falcon on <a class="externalLink" href="https://.";>https://.</a> Any other port will put falcon on <a class="externalLink" href="http://.";>http://.</a></li></ul>* falcon server starts embedded active mq. To control this behaviour, set the following system properties using -D option in environment variable FALCON_OPTS: -<ul> -<li>falcon.embeddedmq=<true/false> - Should server start embedded active mq, default true</li> -<li>falcon.embeddedmq.port=<port> - Port for embedded active mq, default 61616</li> -<li>falcon.embeddedmq.data=<path> - Data path for embedded active mq, default {package dir}/logs/data</li></ul>* falcon server starts with conf from {package dir}/conf. To override this (to use the same conf with multiple falcon upgrades), set environment variable FALCON_CONF to the path of conf dir -<p><b><i>Adding Extension Libraries</i></b> Library extensions allows users to add custom libraries to entity lifecycles such as feed retention, feed replication and process execution. This is useful for usecases such as adding filesystem extensions. To enable this, add the following configs to startup.properties: *.libext.paths=<paths to be added to all entity lifecycles> *.libext.feed.paths=<paths to be added to all feed lifecycles> *.libext.feed.retentions.paths=<paths to be added to feed retention workflow> *.libext.feed.replication.paths=<paths to be added to feed replication workflow> *.libext.process.paths=<paths to be added to process workflow></p> -<p>The configured jars are added to falcon classpath and the corresponding workflows</p> -<p><b>Starting Prism</b></p> -<div class="source"> -<pre> -bin/prism-start [-port <port>] - -</pre></div> -<p>By default, * prism server starts at port 16443. To change the port, use -port option</p> +<h4>Prerequisites<a name="Prerequisites"></a></h4> +<p></p> <ul> -<li>falcon.enableTLS can be set to true or false explicitly to enable SSL, if not port that end with 443 will automatically put prism on <a class="externalLink" href="https://";>https://</a></li></ul>* prism starts with conf from {package dir}/conf. To override this (to use the same conf with multiple prism upgrades), set environment variable FALCON_CONF to the path of conf dir -<p><b>Using Falcon</b></p> -<div class="source"> -<pre> -bin/falcon admin -version -Falcon server build version: {Version:"0.3-SNAPSHOT-rd7e2be9afa2a5dc96acd1ec9e325f39c6b2f17f7",Mode:"embedded"} - ----- - -bin/falcon help -(for more details about falcon cli usage) - -</pre></div> -<p><b>Dashboard</b></p> -<p>Once falcon / prism is started, you can view the status of falcon entities using the Web-based dashboard. The web UI works in both distributed and embedded mode. You can open your browser at the corresponding port to use the web UI.</p> -<p>Falcon dashboard makes the REST api calls as user "falcon-dashboard". If this user does not exist on your falcon and oozie servers, please create the user.</p> +<li>JDK 1.7/1.8</li> +<li>Maven 3.2.x</li></ul></div> +<div class="section"> +<h4>Step 1 - Clone the Falcon repository<a name="Step_1_-_Clone_the_Falcon_repository"></a></h4> <div class="source"> <pre> -## create user. -[root@falconhost ~] useradd -U -m falcon-dashboard -G users +$git clone https://git-wip-us.apache.org/repos/asf/falcon.git falcon -## verify user is created with membership in correct groups. -[root@falconhost ~] groups falcon-dashboard -falcon-dashboard : falcon-dashboard users -[root@falconhost ~] - -</pre></div> -<p><b>Stopping Falcon Server</b></p> +</pre></div></div> +<div class="section"> +<h4>Step 2 - Build Falcon<a name="Step_2_-_Build_Falcon"></a></h4> <div class="source"> <pre> -bin/falcon-stop +$cd falcon +$export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m -noverify" && mvn clean install </pre></div> -<p><b>Stopping Prism</b></p> -<div class="source"> -<pre> -bin/prism-stop - -</pre></div></div> +<p>It builds and installs the package into the local repository, for use as a dependency in other projects locally.</p> +<p>[optionally -Dhadoop.version=<<hadoop.version>> can be appended to build for a specific version of Hadoop]</p> +<p><b>NOTE:</b> Falcon drops support for Hadoop-1 and only supports Hadoop-2 from Falcon 0.6 onwards [optionally -Doozie.version=<<oozie version>> can be appended to build with a specific version of Oozie. Oozie versions >= 4 are supported] NOTE: Falcon builds with JDK 1.7/1.8 using -noverify option To compile Falcon with Hive Replication, optionally "-P hadoop-2,hivedr" can be appended. For this Hive >= 1.2.0 and Oozie >= 4.2.0 should be available.</p></div> +<div class="section"> +<h4>Step 3 - Package and Deploy Falcon<a name="Step_3_-_Package_and_Deploy_Falcon"></a></h4> +<p>Once the build successfully completes, artifacts can be packaged for deployment using the assembly plugin. The Assembly Plugin for Maven is primarily intended to allow users to aggregate the project output along with its dependencies, modules, site documentation, and other files into a single distributable archive. There are two basic ways in which you can deploy Falcon - Embedded mode(also known as Stand Alone Mode) and Distributed mode. Your next steps will vary based on the mode in which you want to deploy Falcon.</p> +<p><b>NOTE</b> : Oozie is being extended by Falcon (particularly on el-extensions) and hence the need for Falcon to build & re-package Oozie, so that users of Falcon can work with the right Oozie setup. Though Oozie is packaged by Falcon, it needs to be deployed separately by the administrator and is not auto deployed along with Falcon.</p></div> +<div class="section"> +<h5>Embedded/Stand Alone Mode<a name="EmbeddedStand_Alone_Mode"></a></h5> +<p>Embedded mode is useful when the Hadoop jobs and relevant data processing involve only one Hadoop cluster. In this mode there is a single Falcon server that contacts the scheduler to schedule jobs on Hadoop. All the process/feed requests like submit, schedule, suspend, kill etc. are sent to this server. For running Falcon in this mode one should use the Falcon which has been built using standalone option. You can find the instructions for Embedded mode setup <a href="./Embedded-mode.html">here</a>.</p></div> +<div class="section"> +<h5>Distributed Mode<a name="Distributed_Mode"></a></h5> +<p>Distributed mode is for multiple (colos) instances of Hadoop clusters, and multiple workflow schedulers to handle them. In this mode Falcon has 2 components: Prism and Server(s). Both Prism and Server(s) have their own their own config locations(startup and runtime properties). In this mode Prism acts as a contact point for Falcon servers. While all commands are available through Prism, only read and instance api's are available through Server. You can find the instructions for Distributed Mode setup <a href="./Distributed-mode.html">here</a>.</p></div> <div class="section"> <h4>Preparing Oozie and Falcon packages for deployment<a name="Preparing_Oozie_and_Falcon_packages_for_deployment"></a></h4> <div class="source"> <pre> -cd <<project home>> -src/bin/package.sh <<hadoop-version>> <<oozie-version>> +$cd <<project home>> +$src/bin/package.sh <<hadoop-version>> <<oozie-version>> >> ex. src/bin/package.sh 1.1.2 4.0.1 or src/bin/package.sh 0.20.2-cdh3u5 4.0.1 >> ex. src/bin/package.sh 2.5.0 4.0.0 >> Falcon package is available in <<falcon home>>/target/apache-falcon-<<version>>-bin.tar.gz >> Oozie package is available in <<falcon home>>/target/oozie-4.0.1-distro.tar.gz -</pre></div></div> -<div class="section"> -<h4>Running Examples using embedded package<a name="Running_Examples_using_embedded_package"></a></h4> -<div class="source"> -<pre> -bin/falcon-start - -</pre></div> -<p>Make sure the hadoop and oozie endpoints are according to your setup in examples/entity/filesystem/standalone-cluster.xml The cluster locations,staging and working dirs, MUST be created prior to submitting a cluster entity to Falcon. <b>staging</b> must have 777 permissions and the parent dirs must have execute permissions <b>working</b> must have 755 permissions and the parent dirs must have execute permissions</p> -<div class="source"> -<pre> -bin/falcon entity -submit -type cluster -file examples/entity/filesystem/standalone-cluster.xml - -</pre></div> -<p>Submit input and output feeds:</p> -<div class="source"> -<pre> -bin/falcon entity -submit -type feed -file examples/entity/filesystem/in-feed.xml -bin/falcon entity -submit -type feed -file examples/entity/filesystem/out-feed.xml - -</pre></div> -<p>Set-up workflow for the process:</p> -<div class="source"> -<pre> -hadoop fs -put examples/app / - -</pre></div> -<p>Submit and schedule the process:</p> -<div class="source"> -<pre> -bin/falcon entity -submitAndSchedule -type process -file examples/entity/filesystem/oozie-mr-process.xml -bin/falcon entity -submitAndSchedule -type process -file examples/entity/filesystem/pig-process.xml - -</pre></div> -<p>Generate input data:</p> -<div class="source"> -<pre> -examples/data/generate.sh <<hdfs endpoint>> - -</pre></div> -<p>Get status of instances:</p> -<div class="source"> -<pre> -bin/falcon instance -status -type process -name oozie-mr-process -start 2013-11-15T00:05Z -end 2013-11-15T01:00Z - </pre></div> -<p>HCat based example entities are in examples/entity/hcat.</p></div> +<p><b>NOTE:</b> If you have a separate Apache Oozie installation, you will need to follow some additional steps:</p> +<ol style="list-style-type: decimal"> +<li>Once you have setup the Falcon Server, copy libraries under {falcon-server-dir}/oozie/libext/ to {oozie-install-dir}/libext.</li> +<li>Modify Oozie's configuration file. Copy all Falcon related properties from {falcon-server-dir}/oozie/conf/oozie-site.xml to {oozie-install-dir}/conf/oozie-site.xml</li> +<li>Restart oozie: +<ol style="list-style-type: decimal"> +<li>cd {oozie-install-dir}</li> +<li>sudo -u oozie ./bin/oozie-stop.sh</li> +<li>sudo -u oozie ./bin/oozie-setup.sh prepare-war</li> +<li>sudo -u oozie ./bin/oozie-start.sh</li></ol></li></ol></div> </div> </div> </div> @@ -723,7 +477,7 @@ bin/falcon instance -status -type proces <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2013-2015 + <div class="row span12">Copyright © 2013-2016 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. Modified: falcon/site/MigrationInstructions.html URL: http://svn.apache.org/viewvc/falcon/site/MigrationInstructions.html?rev=1730450&r1=1730449&r2=1730450&view=diff ============================================================================== --- falcon/site/MigrationInstructions.html (original) +++ falcon/site/MigrationInstructions.html Mon Feb 15 06:02:08 2016 @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-12-08 + | Generated by Apache Maven Doxia at 2016-02-09 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20151208" /> + <meta name="Date-Revision-yyyymmdd" content="20160209" /> <meta http-equiv="Content-Language" content="en" /> <title>Falcon - Migration Instructions</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -58,7 +58,7 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-12-08</li> + <li id="publishDate" class="pull-right">Last Published: 2016-02-09</li> </ul> </div> @@ -308,6 +308,13 @@ <li> + <a href="0.9/index.html" title="0.9 (Current)"> + <i class="none"></i> + 0.9 (Current)</a> + </li> + + <li> + <a href="0.8/index.html" title="0.8"> <i class="none"></i> 0.8</a> @@ -424,7 +431,7 @@ <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2013-2015 + <div class="row span12">Copyright © 2013-2016 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. Modified: falcon/site/OnBoarding.html URL: http://svn.apache.org/viewvc/falcon/site/OnBoarding.html?rev=1730450&r1=1730449&r2=1730450&view=diff ============================================================================== --- falcon/site/OnBoarding.html (original) +++ falcon/site/OnBoarding.html Mon Feb 15 06:02:08 2016 @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-12-08 + | Generated by Apache Maven Doxia at 2016-02-09 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20151208" /> + <meta name="Date-Revision-yyyymmdd" content="20160209" /> <meta http-equiv="Content-Language" content="en" /> <title>Falcon - Contents</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -58,7 +58,7 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-12-08</li> + <li id="publishDate" class="pull-right">Last Published: 2016-02-09</li> </ul> </div> @@ -306,6 +306,13 @@ <li> + <a href="0.9/index.html" title="0.9 (Current)"> + <i class="none"></i> + 0.9 (Current)</a> + </li> + + <li> + <a href="0.8/index.html" title="0.8"> <i class="none"></i> 0.8</a> @@ -558,7 +565,7 @@ xmlns:xsi="http://www.w3.org/2001/X <workflow engine="oozie" path="/projects/bootcamp/workflow" /> - <retry policy="backoff" delay="minutes(5)" attempts="3" /> + <retry policy="periodic" delay="minutes(5)" attempts="3" /> <late-process policy="exp-backoff" delay="hours(1)"> <late-input input="input" workflow-path="/projects/bootcamp/workflow/lateinput" /> @@ -690,7 +697,7 @@ echo "output=0" <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2013-2015 + <div class="row span12">Copyright © 2013-2016 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. Modified: falcon/site/Operability.html URL: http://svn.apache.org/viewvc/falcon/site/Operability.html?rev=1730450&r1=1730449&r2=1730450&view=diff ============================================================================== --- falcon/site/Operability.html (original) +++ falcon/site/Operability.html Mon Feb 15 06:02:08 2016 @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-12-08 + | Generated by Apache Maven Doxia at 2016-02-09 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20151208" /> + <meta name="Date-Revision-yyyymmdd" content="20160209" /> <meta http-equiv="Content-Language" content="en" /> <title>Falcon - Operationalizing Falcon</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -58,7 +58,7 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-12-08</li> + <li id="publishDate" class="pull-right">Last Published: 2016-02-09</li> </ul> </div> @@ -306,6 +306,13 @@ <li> + <a href="0.9/index.html" title="0.9 (Current)"> + <i class="none"></i> + 0.9 (Current)</a> + </li> + + <li> + <a href="0.8/index.html" title="0.8"> <i class="none"></i> 0.8</a> @@ -405,10 +412,13 @@ <h2>Operationalizing Falcon<a name="Operationalizing_Falcon"></a></h2></div> <div class="section"> <h3>Overview<a name="Overview"></a></h3> -<p>Apache Falcon provides various tools to operationalize Falcon consisting of Alerts for unrecoverable errors, Audits of user actions, Metrics, and Notifications. They are detailed below.</p></div> +<p>Apache Falcon provides various tools to operationalize Falcon consisting of Alerts for unrecoverable errors, Audits of user actions, Metrics, and Notifications. They are detailed below.</p> +<p>++ Lineage</p> +<p>Currently Lineage has no way to access or restore information about entity instances created during the time lineage was disabled. Information about entities however, is preserved and bootstrapped when lineage is enabled. If you have to reset the graph db then you can delete the graph db files as specified in the startup.properties and restart the falcon. Please note: you will loose all the information about the instances if you delete the graph db.</p></div> <div class="section"> <h3>Monitoring<a name="Monitoring"></a></h3> <p>Falcon provides monitoring of various events by capturing metrics of those events. The metric numbers can then be used to monitor performance and health of the Falcon system and the entire processing pipelines.</p> +<p>Falcon also exposes <a class="externalLink" href="https://github.com/thinkaurelius/titan/wiki/Titan-Performance-and-Monitoring";>metrics for titandb</a></p> <p>Users can view the logs of these events in the metric.log file, by default this file is created under ${user.dir}/logs/ directory. Users may also extend the Falcon monitoring framework to send events to systems like Mondemand/lwes by implementingorg.apache.falcon.plugin.MonitoringPlugin interface.</p> <p>The following events are captured by Falcon for logging the metrics:</p> <ol style="list-style-type: decimal"> @@ -472,7 +482,7 @@ <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2013-2015 + <div class="row span12">Copyright © 2013-2016 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. Modified: falcon/site/Recipes.html URL: http://svn.apache.org/viewvc/falcon/site/Recipes.html?rev=1730450&r1=1730449&r2=1730450&view=diff ============================================================================== --- falcon/site/Recipes.html (original) +++ falcon/site/Recipes.html Mon Feb 15 06:02:08 2016 @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-12-08 + | Generated by Apache Maven Doxia at 2016-02-09 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20151208" /> + <meta name="Date-Revision-yyyymmdd" content="20160209" /> <meta http-equiv="Content-Language" content="en" /> <title>Falcon - Falcon Recipes</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -58,7 +58,7 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-12-08</li> + <li id="publishDate" class="pull-right">Last Published: 2016-02-09</li> </ul> </div> @@ -308,6 +308,13 @@ <li> + <a href="0.9/index.html" title="0.9 (Current)"> + <i class="none"></i> + 0.9 (Current)</a> + </li> + + <li> + <a href="0.8/index.html" title="0.8"> <i class="none"></i> 0.8</a> @@ -464,6 +471,9 @@ and replace it with the property value & </pre></div></div> <div class="section"> +<h3>Metrics<a name="Metrics"></a></h3> +<p>HDFS DR and Hive DR recipes will capture the replication metrics like TIMETAKEN, BYTESCOPIED, COPY (number of files copied) for an instance and populate to the GraphDB.</p></div> +<div class="section"> <h3>Managing the scheduled recipe process<a name="Managing_the_scheduled_recipe_process"></a></h3> <p></p> <ul> @@ -478,6 +488,12 @@ and replace it with the property value & <ul> <li>Sample recipes are published in addons/recipes</li></ul></div> <div class="section"> +<h3>Types of recipes<a name="Types_of_recipes"></a></h3> +<p></p> +<ul> +<li><a href="./HDFSDR.html">HDFS Recipe</a></li> +<li><a href="./HiveDR.html">HiveDR Recipe</a></li></ul></div> +<div class="section"> <h3>Packaging<a name="Packaging"></a></h3> <p></p> <ul> @@ -490,7 +506,7 @@ and replace it with the property value & <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2013-2015 + <div class="row span12">Copyright © 2013-2016 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. Modified: falcon/site/Security.html URL: http://svn.apache.org/viewvc/falcon/site/Security.html?rev=1730450&r1=1730449&r2=1730450&view=diff ============================================================================== --- falcon/site/Security.html (original) +++ falcon/site/Security.html Mon Feb 15 06:02:08 2016 @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-12-08 + | Generated by Apache Maven Doxia at 2016-02-09 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20151208" /> + <meta name="Date-Revision-yyyymmdd" content="20160209" /> <meta http-equiv="Content-Language" content="en" /> <title>Falcon - Securing Falcon</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -58,7 +58,7 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-12-08</li> + <li id="publishDate" class="pull-right">Last Published: 2016-02-09</li> </ul> </div> @@ -306,6 +306,13 @@ <li> + <a href="0.9/index.html" title="0.9 (Current)"> + <i class="none"></i> + 0.9 (Current)</a> + </li> + + <li> + <a href="0.8/index.html" title="0.8"> <i class="none"></i> 0.8</a> @@ -478,51 +485,51 @@ <th>Description</th> <th>Authorization</th></tr> <tr class="b"> -<td><a href="././restapi/EntityValidate.html">api/entities/validate/:entity-type</a></td> +<td><a href="./Restapi/EntityValidate.html">api/entities/validate/:entity-type</a></td> <td>Validate the entity</td> <td>Owner/Group</td></tr> <tr class="a"> -<td><a href="././restapi/EntitySubmit.html">api/entities/submit/:entity-type</a></td> +<td><a href="./Restapi/EntitySubmit.html">api/entities/submit/:entity-type</a></td> <td>Submit the entity</td> <td>Owner/Group</td></tr> <tr class="b"> -<td><a href="././restapi/EntityUpdate.html">api/entities/update/:entity-type/:entity-name</a></td> +<td><a href="./Restapi/EntityUpdate.html">api/entities/update/:entity-type/:entity-name</a></td> <td>Update the entity</td> <td>Owner/Group</td></tr> <tr class="a"> -<td><a href="././restapi/EntitySubmitAndSchedule.html">api/entities/submitAndSchedule/:entity-type</a></td> +<td><a href="./Restapi/EntitySubmitAndSchedule.html">api/entities/submitAndSchedule/:entity-type</a></td> <td>Submit & Schedule the entity</td> <td>Owner/Group</td></tr> <tr class="b"> -<td><a href="././restapi/EntitySchedule.html">api/entities/schedule/:entity-type/:entity-name</a></td> +<td><a href="./Restapi/EntitySchedule.html">api/entities/schedule/:entity-type/:entity-name</a></td> <td>Schedule the entity</td> <td>Owner/Group</td></tr> <tr class="a"> -<td><a href="././restapi/EntitySuspend.html">api/entities/suspend/:entity-type/:entity-name</a></td> +<td><a href="./Restapi/EntitySuspend.html">api/entities/suspend/:entity-type/:entity-name</a></td> <td>Suspend the entity</td> <td>Owner/Group</td></tr> <tr class="b"> -<td><a href="././restapi/EntityResume.html">api/entities/resume/:entity-type/:entity-name</a></td> +<td><a href="./Restapi/EntityResume.html">api/entities/resume/:entity-type/:entity-name</a></td> <td>Resume the entity</td> <td>Owner/Group</td></tr> <tr class="a"> -<td><a href="././restapi/EntityDelete.html">api/entities/delete/:entity-type/:entity-name</a></td> +<td><a href="./Restapi/EntityDelete.html">api/entities/delete/:entity-type/:entity-name</a></td> <td>Delete the entity</td> <td>Owner/Group</td></tr> <tr class="b"> -<td><a href="././restapi/EntityStatus.html">api/entities/status/:entity-type/:entity-name</a></td> +<td><a href="./Restapi/EntityStatus.html">api/entities/status/:entity-type/:entity-name</a></td> <td>Get the status of the entity</td> <td>Owner/Group</td></tr> <tr class="a"> -<td><a href="././restapi/EntityDefinition.html">api/entities/definition/:entity-type/:entity-name</a></td> +<td><a href="./Restapi/EntityDefinition.html">api/entities/definition/:entity-type/:entity-name</a></td> <td>Get the definition of the entity</td> <td>Owner/Group</td></tr> <tr class="b"> -<td><a href="././restapi/EntityList.html">api/entities/list/:entity-type?fields=:fields</a></td> +<td><a href="./Restapi/EntityList.html">api/entities/list/:entity-type?fields=:fields</a></td> <td>Get the list of entities</td> <td>Owner/Group</td></tr> <tr class="a"> -<td><a href="././restapi/EntityDependencies.html">api/entities/dependencies/:entity-type/:entity-name</a></td> +<td><a href="./Restapi/EntityDependencies.html">api/entities/dependencies/:entity-type/:entity-name</a></td> <td>Get the dependencies of the entity</td> <td>Owner/Group</td></tr></table><b>REST Call on Feed and Process Instances</b> <p></p> @@ -532,27 +539,27 @@ <th>Description</th> <th>Authorization</th></tr> <tr class="b"> -<td><a href="././restapi/InstanceRunning.html">api/instance/running/:entity-type/:entity-name</a></td> +<td><a href="./Restapi/InstanceRunning.html">api/instance/running/:entity-type/:entity-name</a></td> <td>List of running instances.</td> <td>Owner/Group</td></tr> <tr class="a"> -<td><a href="././restapi/InstanceStatus.html">api/instance/status/:entity-type/:entity-name</a></td> +<td><a href="./Restapi/InstanceStatus.html">api/instance/status/:entity-type/:entity-name</a></td> <td>Status of a given instance</td> <td>Owner/Group</td></tr> <tr class="b"> -<td><a href="././restapi/InstanceKill.html">api/instance/kill/:entity-type/:entity-name</a></td> +<td><a href="./Restapi/InstanceKill.html">api/instance/kill/:entity-type/:entity-name</a></td> <td>Kill a given instance</td> <td>Owner/Group</td></tr> <tr class="a"> -<td><a href="././restapi/InstanceSuspend.html">api/instance/suspend/:entity-type/:entity-name</a></td> +<td><a href="./Restapi/InstanceSuspend.html">api/instance/suspend/:entity-type/:entity-name</a></td> <td>Suspend a running instance</td> <td>Owner/Group</td></tr> <tr class="b"> -<td><a href="././restapi/InstanceResume.html">api/instance/resume/:entity-type/:entity-name</a></td> +<td><a href="./Restapi/InstanceResume.html">api/instance/resume/:entity-type/:entity-name</a></td> <td>Resume a given instance</td> <td>Owner/Group</td></tr> <tr class="a"> -<td><a href="././restapi/InstanceRerun.html">api/instance/rerun/:entity-type/:entity-name</a></td> +<td><a href="./Restapi/InstanceRerun.html">api/instance/rerun/:entity-type/:entity-name</a></td> <td>Rerun a given instance</td> <td>Owner/Group</td></tr> <tr class="b"> @@ -569,15 +576,15 @@ <th>Description</th> <th>Authorization</th></tr> <tr class="b"> -<td><a href="././restapi/AdminVersion.html">api/admin/version</a></td> +<td><a href="./Restapi/AdminVersion.html">api/admin/version</a></td> <td>Get version of the server</td> <td>No restriction</td></tr> <tr class="a"> -<td><a href="././restapi/AdminStack.html">api/admin/stack</a></td> +<td><a href="./Restapi/AdminStack.html">api/admin/stack</a></td> <td>Get stack of the server</td> <td>Admin User/Group</td></tr> <tr class="b"> -<td><a href="././restapi/AdminConfig.html">api/admin/config/:config-type</a></td> +<td><a href="./Restapi/AdminConfig.html">api/admin/config/:config-type</a></td> <td>Get configuration information of the server</td> <td>Admin User/Group</td></tr></table></div> <div class="section"> @@ -609,6 +616,9 @@ # name node principal to talk to config store *.dfs.namenode.kerberos.principal=nn/[email protected] +# Indicates how long (in seconds) falcon authentication token is valid before it has to be renewed. +*.falcon.service.authentication.token.validity=86400 + ##### SPNEGO Configuration # Authentication type must be specified: simple|kerberos|<class> @@ -829,7 +839,7 @@ $ curl --negotiate -u foo -b ~/cookiejar <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2013-2015 + <div class="row span12">Copyright © 2013-2016 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. Added: falcon/site/falconcli/CommonCLI.html URL: http://svn.apache.org/viewvc/falcon/site/falconcli/CommonCLI.html?rev=1730450&view=auto ============================================================================== --- falcon/site/falconcli/CommonCLI.html (added) +++ falcon/site/falconcli/CommonCLI.html Mon Feb 15 06:02:08 2016 @@ -0,0 +1,445 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-02-09 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160209" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Falcon - Common CLI Options</title> + <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../css/site.css" /> + <link rel="stylesheet" href="../css/print.css" media="print" /> + + + <script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="../../index.html" id="bannerLeft"> + <img src="../images/falcon-logo.png" alt="Falcon" width="200px" height="45px"/> + </a> + </div> + <div class="pull-right"> <a href="http://www.apache.org"; id="bannerRight"> + <img src="../images/apache-feather-tm.gif" alt="Falcon" height="45px"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../index.html" title="Falcon"> + Falcon</a> + </li> + <li class="divider ">/</li> + <li class="">Common CLI Options</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2016-02-09</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">Falcon</li> + + <li> + + <a href="../index.html" title="About"> + <i class="none"></i> + About</a> + </li> + + <li> + + <a href="../slides/falcon-overview.html" title="Overview"> + <i class="none"></i> + Overview</a> + </li> + + <li> + + <a href="../slides/falcon-user-guide.html" title="User Guide"> + <i class="none"></i> + User Guide</a> + </li> + + <li> + + <a href="../GettingStarted.html" title="Getting Started"> + <i class="none"></i> + Getting Started</a> + </li> + + <li> + + <a href="../FalconDocumentation.html" title="Architecture"> + <i class="none"></i> + Architecture</a> + </li> + + <li> + + <a href="../InstallationSteps.html" title="Installation"> + <i class="none"></i> + Installation</a> + </li> + + <li> + + <a href="../OnBoarding.html" title="On Boarding"> + <i class="none"></i> + On Boarding</a> + </li> + + <li> + + <a href="../Operability.html" title="Operability"> + <i class="none"></i> + Operability</a> + </li> + + <li> + + <a href="../EntitySpecification.html" title="Entity Specification"> + <i class="none"></i> + Entity Specification</a> + </li> + + <li> + + <a href="../FalconCLI.html" title="Client (Falcon CLI)"> + <i class="none"></i> + Client (Falcon CLI)</a> + </li> + + <li> + + <a href="../restapi/ResourceList.html" title="Rest API"> + <i class="icon-chevron-right"></i> + Rest API</a> + </li> + + <li> + + <a href="../HiveIntegration.html" title="Hive Integration"> + <i class="none"></i> + Hive Integration</a> + </li> + + <li> + + <a href="../Security.html" title="Security"> + <i class="none"></i> + Security</a> + </li> + <li class="nav-header">Project Information</li> + + <li> + + <a href="../project-info.html" title="Summary"> + <i class="none"></i> + Summary</a> + </li> + + <li> + + <a href="../mail-lists.html" title="Mailing Lists"> + <i class="none"></i> + Mailing Lists</a> + </li> + + <li> + + <a href="http://webchat.freenode.net?channels=apachefalcon&uio=d4"; class="externalLink" title="IRC"> + <i class="none"></i> + IRC</a> + </li> + + <li> + + <a href="../team-list.html" title="Team"> + <i class="none"></i> + Team</a> + </li> + + <li> + + <a href="../issue-tracking.html" title="Issue Tracking"> + <i class="none"></i> + Issue Tracking</a> + </li> + + <li> + + <a href="../source-repository.html" title="Source Repository"> + <i class="none"></i> + Source Repository</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/Index"; class="externalLink" title="Wiki"> + <i class="none"></i> + Wiki</a> + </li> + + <li> + + <a href="../license.html" title="License"> + <i class="none"></i> + License</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/News"; class="externalLink" title="News"> + <i class="none"></i> + News</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/PoweredBy"; class="externalLink" title="Powered by"> + <i class="none"></i> + Powered by</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/Acknowledgements"; class="externalLink" title="Acknowledgements"> + <i class="none"></i> + Acknowledgements</a> + </li> + + <li> + + <a href="http://blogs.apache.org/falcon/"; class="externalLink" title="Blog"> + <i class="none"></i> + Blog</a> + </li> + <li class="nav-header">Releases</li> + + <li> + + <a href="http://www.apache.org/dyn/closer.lua/falcon/0.8"; class="externalLink" title="0.8"> + <i class="none"></i> + 0.8</a> + </li> + + <li> + + <a href="http://www.apache.org/dyn/closer.lua/falcon/0.7"; class="externalLink" title="0.7"> + <i class="none"></i> + 0.7</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/falcon/0.6.1"; class="externalLink" title="0.6.1"> + <i class="none"></i> + 0.6.1</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/incubator/falcon/0.6-incubating"; class="externalLink" title="0.6-incubating"> + <i class="none"></i> + 0.6-incubating</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/incubator/falcon/0.5-incubating"; class="externalLink" title="0.5-incubating"> + <i class="none"></i> + 0.5-incubating</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/incubator/falcon/0.4-incubating"; class="externalLink" title="0.4-incubating"> + <i class="none"></i> + 0.4-incubating</a> + </li> + + <li> + + <a href="http://archive.apache.org/dist/incubator/falcon/0.3-incubating"; class="externalLink" title="0.3-incubating"> + <i class="none"></i> + 0.3-incubating</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/FALCON/Roadmap"; class="externalLink" title="Coming soon"> + <i class="none"></i> + Coming soon</a> + </li> + <li class="nav-header">Documentation</li> + + <li> + + <a href="../0.9/index.html" title="0.9 (Current)"> + <i class="none"></i> + 0.9 (Current)</a> + </li> + + <li> + + <a href="../0.8/index.html" title="0.8"> + <i class="none"></i> + 0.8</a> + </li> + + <li> + + <a href="../0.7/index.html" title="0.7"> + <i class="none"></i> + 0.7</a> + </li> + + <li> + + <a href="../0.6.1/index.html" title="0.6.1"> + <i class="none"></i> + 0.6.1</a> + </li> + + <li> + + <a href="../0.6-incubating/index.html" title="0.6-incubating"> + <i class="none"></i> + 0.6-incubating</a> + </li> + + <li> + + <a href="../0.5-incubating/index.html" title="0.5-incubating"> + <i class="none"></i> + 0.5-incubating</a> + </li> + + <li> + + <a href="../0.4-incubating/index.html" title="0.4-incubating"> + <i class="none"></i> + 0.4-incubating</a> + </li> + + <li> + + <a href="../0.3-incubating/index.html" title="0.3-incubating"> + <i class="none"></i> + 0.3-incubating</a> + </li> + <li class="nav-header">ASF</li> + + <li> + + <a href="http://www.apache.org/foundation/how-it-works.html"; class="externalLink" title="How Apache Works"> + <i class="none"></i> + How Apache Works</a> + </li> + + <li> + + <a href="http://www.apache.org/foundation/"; class="externalLink" title="Foundation"> + <i class="none"></i> + Foundation</a> + </li> + + <li> + + <a href="http://www.apache.org/foundation/sponsorship.html"; class="externalLink" title="Sponsoring Apache"> + <i class="none"></i> + Sponsoring Apache</a> + </li> + + <li> + + <a href="http://www.apache.org/foundation/thanks.html"; class="externalLink" title="Thanks"> + <i class="none"></i> + Thanks</a> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <div class="section"> +<h3>Common CLI Options<a name="Common_CLI_Options"></a></h3></div> +<div class="section"> +<h4>Falcon URL<a name="Falcon_URL"></a></h4> +<p>Optional -url option indicating the URL of the Falcon system to run the command against can be provided. If not mentioned it will be picked from the system environment variable FALCON_URL. If FALCON_URL is not set then it will be picked from client.properties file. If the option is not provided and also not set in client.properties, Falcon CLI will fail.</p></div> +<div class="section"> +<h4>Proxy user support<a name="Proxy_user_support"></a></h4> +<p>The -doAs option allows the current user to impersonate other users when interacting with the Falcon system. The current user must be configured as a proxyuser in the Falcon system. The proxyuser configuration may restrict from which hosts a user may impersonate users, as well as users of which groups can be impersonated.</p> +<p><a href="../FalconDocumentation.html#Proxyuser_support">Proxyuser support described here.</a></p></div> +<div class="section"> +<h4>Debug Mode<a name="Debug_Mode"></a></h4> +<p>If you export FALCON_DEBUG=true then the Falcon CLI will output the Web Services API details used by any commands you execute. This is useful for debugging purposes to or see how the Falcon CLI works with the WS API. Alternately, you can specify '-debug' through the CLI arguments to get the debug statements.</p> +<p>Example: $FALCON_HOME/bin/falcon entity -submit -type cluster -file /cluster/definition.xml -debug</p></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2013-2016 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html>
