Added: falcon/site/restapi/MetadataRelations.html URL: http://svn.apache.org/viewvc/falcon/site/restapi/MetadataRelations.html?rev=1660589&view=auto ============================================================================== --- falcon/site/restapi/MetadataRelations.html (added) +++ falcon/site/restapi/MetadataRelations.html Wed Feb 18 10:55:56 2015 @@ -0,0 +1,336 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2015-02-18 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20150218" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Falcon - GET api/metadata/discovery/:dimension-type/:dimension-name/relations</title> + <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../css/site.css" /> + <link rel="stylesheet" href="../css/print.css" media="print" /> + + + <script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarEnabled"> + + + + + + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Falcon <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../index.html" title="About">About</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/FALCON/Index"; title="Wiki">Wiki</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/FALCON/News"; title="News">News</a> +</li> + + <li> <a href="https://git-wip-us.apache.org/repos/asf/falcon.git"; title="Git">Git</a> +</li> + + <li> <a href="https://svn.apache.org/repos/asf/falcon"; title="Svn">Svn</a> +</li> + + <li> <a href="https://issues.apache.org/jira/browse/FALCON"; title="Jira">Jira</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/FALCON/PoweredBy"; title="Powered by">Powered by</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/FALCON/Acknowledgements"; title="Acknowledgements">Acknowledgements</a> +</li> + + <li> <a href="http://blogs.apache.org/falcon/"; title="Blog">Blog</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../project-info.html" title="Summary">Summary</a> +</li> + + <li> <a href="../mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="http://webchat.freenode.net?channels=apachefalcon&uio=d4"; title="IRC">IRC</a> +</li> + + <li> <a href="../team-list.html" title="Team">Team</a> +</li> + + <li> <a href="../issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="../source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="../license.html" title="License">License</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/dyn/closer.cgi/falcon/0.6-incubating"; title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/falcon/0.5-incubating"; title="0.5-incubating">0.5-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/falcon/0.4-incubating"; title="0.4-incubating">0.4-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/falcon/0.3-incubating"; title="0.3-incubating">0.3-incubating</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/FALCON/Roadmap"; title="Coming soon">Coming soon</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../0.6-incubating/index.html" title="current">current</a> +</li> + + <li> <a href="../0.6-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="../0.5-incubating/index.html" title="0.5-incubating">0.5-incubating</a> +</li> + + <li> <a href="../0.4-incubating/index.html" title="0.4-incubating">0.4-incubating</a> +</li> + + <li> <a href="../0.3-incubating/index.html" title="0.3-incubating">0.3-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Resources <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../slides/falcon-overview.html" title="Overview">Overview</a> +</li> + + <li> <a href="../slides/falcon-user-guide.html" title="User Guide">User Guide</a> +</li> + + <li> <a href="../GettingStarted.html" title="Getting Started">Getting Started</a> +</li> + + <li> <a href="../FalconDocumentation.html" title="Architecture">Architecture</a> +</li> + + <li> <a href="../InstallationSteps.html" title="Installation">Installation</a> +</li> + + <li> <a href="../OnBoarding.html" title="On Boarding">On Boarding</a> +</li> + + <li> <a href="../EntitySpecification.html" title="Entity Specification">Entity Specification</a> +</li> + + <li> <a href="../FalconCLI.html" title="Client (Falcon CLI)">Client (Falcon CLI)</a> +</li> + + <li> <a href="../restapi/ResourceList.html" title="Rest API">Rest API</a> +</li> + + <li> <a href="../HiveIntegration.html" title="Hive Integration">Hive Integration</a> +</li> + + <li> <a href="../Security.html" title="Security">Security</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> +</li> + + <li> <a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a> +</li> + + <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a> +</li> + + <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> +</li> + </ul> + </li> + </ul> + + + + + + + <iframe src="http://www.facebook.com/plugins/like.php?href=http://falcon.apache.org/falcon-website-general&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:80px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + + <script type="text/javascript" src="https://apis.google.com/js/plusone.js";></script> + + <ul class="nav pull-right"><li style="margin-top: 10px;"> + + <div class="g-plusone" data-href="http://falcon.apache.org/falcon-website-general"; data-size="medium" width="60px" align="right" ></div> + + </li></ul> + + + </div> + + </div> + </div> + </div> + + <div class="container"> + <div id="banner"> + <div class="pull-left"> + <a href="../../index.html" id="bannerLeft"> + <img src="../images/falcon-logo.png" alt="Falcon" width="200px" height="45px"/> + </a> + </div> + <div class="pull-right"> <a href="http://www.apache.org"; id="bannerRight"> + <img src="../images/apache-feather-tm.gif" alt="Falcon" height="45px"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../index.html" title="Falcon"> + Falcon</a> + </li> + <li class="divider ">/</li> + <li class="">GET api/metadata/discovery/:dimension-type/:dimension-name/relations</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2015-02-18</li> + + </ul> + </div> + + + + <div id="bodyColumn" > + + <div class="section"> +<h3>GET api/metadata/discovery/:dimension-type/:dimension-name/relations<a name="GET_apimetadatadiscovery:dimension-type:dimension-namerelations"></a></h3> +<p></p> +<ul> +<li><a href="#Description">Description</a></li> +<li><a href="#Parameters">Parameters</a></li> +<li><a href="#Results">Results</a></li> +<li><a href="#Examples">Examples</a></li></ul></div> +<div class="section"> +<h3>Description<a name="Description"></a></h3> +<p>Get all relations of a specific dimension.</p></div> +<div class="section"> +<h3>Parameters<a name="Parameters"></a></h3> +<p></p> +<ul> +<li>:type Valid dimension types are cluster_entity,feed_entity, process_entity, user, colo, tags, groups, pipelines</li> +<li>:name Name of the dimension.</li></ul></div> +<div class="section"> +<h3>Results<a name="Results"></a></h3> +<p>Get all relations of a specific dimension.</p></div> +<div class="section"> +<h3>Examples<a name="Examples"></a></h3></div> +<div class="section"> +<h4>Rest Call<a name="Rest_Call"></a></h4> +<div class="source"> +<pre> +GET http://localhost:15000/api/metadata/discovery/process_entity/sample-process/relations + +</pre></div></div> +<div class="section"> +<h4>Result<a name="Result"></a></h4> +<div class="source"> +<pre> +{ + "timestamp":"2014-09-09T01:31Z", + "userWorkflowEngine":"pig", + "name":"sample-process", + "type":"PROCESS_ENTITY", + "userWorkflowName":"imp-click-join-workflow", + "version":"1.0.9", + "inVertices":[ + {"name":"clicks-feed","type":"FEED_ENTITY","label":"input"}, + {"name":"impression-feed","type":"FEED_ENTITY","label":"input"}, + {"name":"sample-process\/2014-01-01T01:00Z","type":"PROCESS_INSTANCE","label":"instance-of"} + ], + "outVertices":[ + {"name":"Critical","type":"TAGS","label":"classified-as"}, + {"name":"testPipeline","type":"PIPELINES","label":"pipeline"}, + {"name":"primary-cluster","type":"CLUSTER_ENTITY","label":"runs-on"}, + {"name":"imp-click-join2","type":"FEED_ENTITY","label":"output"}, + {"name":"imp-click-join1","type":"FEED_ENTITY","label":"output"}, + {"name":"falcon-user","type":"USER","label":"owned-by"} + ] +} + +</pre></div></div> + </div> + </div> + + <hr/> + + <footer> + <div class="container"> + <div class="row span12">Copyright © 2013-2015 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + <p id="poweredBy" class="pull-right"> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" /> + </a> + </p> + + </div> + </footer> + </body> +</html>
Modified: falcon/site/restapi/ResourceList.html URL: http://svn.apache.org/viewvc/falcon/site/restapi/ResourceList.html?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/site/restapi/ResourceList.html (original) +++ falcon/site/restapi/ResourceList.html Wed Feb 18 10:55:56 2015 @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-01-11 + | Generated by Apache Maven Doxia at 2015-02-18 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20150111" /> + <meta name="Date-Revision-yyyymmdd" content="20150218" /> <meta http-equiv="Content-Language" content="en" /> <title>Falcon - RESTful Resources</title> <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" /> @@ -245,7 +245,7 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-01-11</li> + <li id="publishDate" class="pull-right">Last Published: 2015-02-18</li> </ul> </div> @@ -263,7 +263,8 @@ <li><a href="#REST_Call_on_Entity_Resource">REST Call on Entity Resource</a></li> <li><a href="#REST_Call_on_Feed_and_Process_Instances">REST Call on Feed/Process Instances</a></li> <li><a href="#REST_Call_on_Admin_Resource">REST Call on Admin Resource</a></li> -<li><a href="#REST_Call_on_Lineage_Graph">REST Call on Lineage Graph Resource</a></li></ul></div> +<li><a href="#REST_Call_on_Lineage_Graph">REST Call on Lineage Graph Resource</a></li> +<li><a href="#REST_Call_on_Metadata_Resource">REST Call on Metadata Resource</a></li></ul></div> <div class="section"> <h3>Authentication<a name="Authentication"></a></h3> <p>When security is off (Pseudo/Simple), the authenticated user is the username specified in the user.name query parameter. If the user.name parameter is not set, the server may either set the authenticated user to a default web user, if there is any, or return an error response.</p> @@ -376,38 +377,42 @@ curl -i --negotiate -u : "http://&l <td>List of running instances.</td></tr> <tr class="a"> <td>GET</td> +<td><a href="./InstanceParams.html">api/instance/params/:entity-type/:entity-name</a></td> +<td>List of entity instances along with their workflow params.</td></tr> +<tr class="b"> +<td>GET</td> <td><a href="./InstanceList.html">api/instance/list/:entity-type/:entity-name</a></td> <td>List of instances</td></tr> -<tr class="b"> +<tr class="a"> <td>GET</td> <td><a href="./InstanceStatus.html">api/instance/status/:entity-type/:entity-name</a></td> <td>Status of a given instance</td></tr> -<tr class="a"> +<tr class="b"> <td>POST</td> <td><a href="./InstanceKill.html">api/instance/kill/:entity-type/:entity-name</a></td> <td>Kill a given instance</td></tr> -<tr class="b"> +<tr class="a"> <td>POST</td> <td><a href="./InstanceSuspend.html">api/instance/suspend/:entity-type/:entity-name</a></td> <td>Suspend a running instance</td></tr> -<tr class="a"> +<tr class="b"> <td>POST</td> <td><a href="./InstanceResume.html">api/instance/resume/:entity-type/:entity-name</a></td> <td>Resume a given instance</td></tr> -<tr class="b"> +<tr class="a"> <td>POST</td> <td><a href="./InstanceRerun.html">api/instance/rerun/:entity-type/:entity-name</a></td> <td>Rerun a given instance</td></tr> -<tr class="a"> +<tr class="b"> <td>GET</td> <td><a href="./InstanceLogs.html">api/instance/logs/:entity-type/:entity-name</a></td> <td>Get logs of a given instance</td></tr> -<tr class="b"> +<tr class="a"> <td>GET</td> <td><a href="./InstanceSummary.html">api/instance/summary/:entity-type/:entity-name</a></td> <td>Return summary of instances for an entity</td></tr></table></div> <div class="section"> -<h3>REST Call on Lineage Graph<a name="REST_Call_on_Lineage_Graph"></a></h3> +<h3>REST Call on Metadata Lineage Resource<a name="REST_Call_on_Metadata_Lineage_Resource"></a></h3> <p></p> <table border="0" class="table table-striped"> <tr class="a"> @@ -416,36 +421,56 @@ curl -i --negotiate -u : "http://&l <th>Description</th></tr> <tr class="b"> <td>GET</td> -<td><a href="./Graph.html">api/graphs/lineage/serialize</a></td> +<td><a href="./Graph.html">api/metadata/lineage/serialize</a></td> <td>dump the graph</td></tr> <tr class="a"> <td>GET</td> -<td><a href="./AllVertices.html">api/graphs/lineage/vertices/all</a></td> +<td><a href="./AllVertices.html">api/metadata/lineage/vertices/all</a></td> <td>get all vertices</td></tr> <tr class="b"> <td>GET</td> -<td><a href="./Vertices.html">api/graphs/lineage/vertices?key=:key&value=:value</a></td> +<td><a href="./Vertices.html">api/metadata/lineage/vertices?key=:key&value=:value</a></td> <td>get all vertices for a key index</td></tr> <tr class="a"> <td>GET</td> -<td><a href="./Vertex.html">api/graphs/lineage/vertices/:id</a></td> +<td><a href="./Vertex.html">api/metadata/lineage/vertices/:id</a></td> <td>get the vertex with the specified id</td></tr> <tr class="b"> <td>GET</td> -<td><a href="./VertexProperties.html">api/graphs/lineage/vertices/properties/:id?relationships=:true</a></td> +<td><a href="./VertexProperties.html">api/metadata/lineage/vertices/properties/:id?relationships=:true</a></td> <td>get the properties of the vertex with the specified id</td></tr> <tr class="a"> <td>GET</td> -<td><a href="./AdjacentVertices.html">api/graphs/lineage/vertices/:id/:direction</a></td> +<td><a href="./AdjacentVertices.html">api/metadata/lineage/vertices/:id/:direction</a></td> <td>get the adjacent vertices or edges of the vertex with the specified direction</td></tr> <tr class="b"> <td>GET</td> -<td><a href="./AllEdges.html">api/graphs/lineage//edges/all</a></td> +<td><a href="./AllEdges.html">api/metadata/lineage/edges/all</a></td> <td>get all edges</td></tr> <tr class="a"> <td>GET</td> -<td><a href="./Edge.html">api/graphs/lineage/edges/:id</a></td> -<td>get the edge with the specified id</td></tr></table></div> +<td><a href="./Edge.html">api/metadata/lineage/edges/:id</a></td> +<td>get the edge with the specified id</td></tr> +<tr class="b"> +<td>GET</td> +<td><a href="./EntityLineage.html">api/metadata/lineage/entities?pipeline=:name</a></td> +<td>Get lineage graph for processes and feeds in the specified pipeline</td></tr></table></div> +<div class="section"> +<h3>REST Call on Metadata Discovery Resource<a name="REST_Call_on_Metadata_Discovery_Resource"></a></h3> +<p></p> +<table border="0" class="table table-striped"> +<tr class="a"> +<th>Call Type</th> +<th>Resource</th> +<th>Description</th></tr> +<tr class="b"> +<td>GET</td> +<td><a href="./MetadataList.html">api/metadata/discovery/:dimension-type/list</a></td> +<td>list of dimensions</td></tr> +<tr class="a"> +<td>GET</td> +<td>[MetadataRelations][api/metadata/discovery/:dimension-type/:dimension-name/relations]]</td> +<td>Return all relations of a dimension</td></tr></table></div> </div> </div> Modified: falcon/site/restapi/Vertex.html URL: http://svn.apache.org/viewvc/falcon/site/restapi/Vertex.html?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/site/restapi/Vertex.html (original) +++ falcon/site/restapi/Vertex.html Wed Feb 18 10:55:56 2015 @@ -1,15 +1,15 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-01-11 + | Generated by Apache Maven Doxia at 2015-02-18 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20150111" /> + <meta name="Date-Revision-yyyymmdd" content="20150218" /> <meta http-equiv="Content-Language" content="en" /> - <title>Falcon - GET api/graphs/lineage/vertices/:id</title> + <title>Falcon - GET api/metadata/lineage/vertices/:id</title> <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" /> <link rel="stylesheet" href="../css/site.css" /> <link rel="stylesheet" href="../css/print.css" media="print" /> @@ -241,11 +241,11 @@ Falcon</a> </li> <li class="divider ">/</li> - <li class="">GET api/graphs/lineage/vertices/:id</li> + <li class="">GET api/metadata/lineage/vertices/:id</li> - <li id="publishDate" class="pull-right">Last Published: 2015-01-11</li> + <li id="publishDate" class="pull-right">Last Published: 2015-02-18</li> </ul> </div> @@ -255,7 +255,7 @@ <div id="bodyColumn" > <div class="section"> -<h3>GET api/graphs/lineage/vertices/:id<a name="GET_apigraphslineagevertices:id"></a></h3> +<h3>GET api/metadata/lineage/vertices/:id<a name="GET_apimetadatalineagevertices:id"></a></h3> <p></p> <ul> <li><a href="#Description">Description</a></li> @@ -279,7 +279,7 @@ <h4>Rest Call<a name="Rest_Call"></a></h4> <div class="source"> <pre> -GET http://localhost:15000/api/graphs/lineage/vertices/4 +GET http://localhost:15000/api/metadata/lineage/vertices/4 </pre></div></div> <div class="section"> Modified: falcon/site/restapi/VertexProperties.html URL: http://svn.apache.org/viewvc/falcon/site/restapi/VertexProperties.html?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/site/restapi/VertexProperties.html (original) +++ falcon/site/restapi/VertexProperties.html Wed Feb 18 10:55:56 2015 @@ -1,15 +1,15 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-01-11 + | Generated by Apache Maven Doxia at 2015-02-18 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20150111" /> + <meta name="Date-Revision-yyyymmdd" content="20150218" /> <meta http-equiv="Content-Language" content="en" /> - <title>Falcon - GET api/graphs/lineage/vertices/properties/:id?relationships=:true</title> + <title>Falcon - GET api/metadata/lineage/vertices/properties/:id?relationships=:true</title> <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" /> <link rel="stylesheet" href="../css/site.css" /> <link rel="stylesheet" href="../css/print.css" media="print" /> @@ -241,11 +241,11 @@ Falcon</a> </li> <li class="divider ">/</li> - <li class="">GET api/graphs/lineage/vertices/properties/:id?relationships=:true</li> + <li class="">GET api/metadata/lineage/vertices/properties/:id?relationships=:true</li> - <li id="publishDate" class="pull-right">Last Published: 2015-01-11</li> + <li id="publishDate" class="pull-right">Last Published: 2015-02-18</li> </ul> </div> @@ -255,7 +255,7 @@ <div id="bodyColumn" > <div class="section"> -<h3>GET api/graphs/lineage/vertices/properties/:id?relationships=:true<a name="GET_apigraphslineageverticesproperties:idrelationships:true"></a></h3> +<h3>GET api/metadata/lineage/vertices/properties/:id?relationships=:true<a name="GET_apimetadatalineageverticesproperties:idrelationships:true"></a></h3> <p></p> <ul> <li><a href="#Description">Description</a></li> @@ -280,7 +280,7 @@ <h4>Rest Call<a name="Rest_Call"></a></h4> <div class="source"> <pre> -GET http://localhost:15000/api/graphs/lineage/vertices/properties/40004?relationships=true +GET http://localhost:15000/api/metadata/lineage/vertices/properties/40004?relationships=true </pre></div></div> <div class="section"> Modified: falcon/site/restapi/Vertices.html URL: http://svn.apache.org/viewvc/falcon/site/restapi/Vertices.html?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/site/restapi/Vertices.html (original) +++ falcon/site/restapi/Vertices.html Wed Feb 18 10:55:56 2015 @@ -1,15 +1,15 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-01-11 + | Generated by Apache Maven Doxia at 2015-02-18 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20150111" /> + <meta name="Date-Revision-yyyymmdd" content="20150218" /> <meta http-equiv="Content-Language" content="en" /> - <title>Falcon - GET api/graphs/lineage/vertices?key=:key&value=:value</title> + <title>Falcon - GET api/metadata/lineage/vertices?key=:key&value=:value</title> <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" /> <link rel="stylesheet" href="../css/site.css" /> <link rel="stylesheet" href="../css/print.css" media="print" /> @@ -241,11 +241,11 @@ Falcon</a> </li> <li class="divider ">/</li> - <li class="">GET api/graphs/lineage/vertices?key=:key&value=:value</li> + <li class="">GET api/metadata/lineage/vertices?key=:key&value=:value</li> - <li id="publishDate" class="pull-right">Last Published: 2015-01-11</li> + <li id="publishDate" class="pull-right">Last Published: 2015-02-18</li> </ul> </div> @@ -255,7 +255,7 @@ <div id="bodyColumn" > <div class="section"> -<h3>GET api/graphs/lineage/vertices?key=:key&value=:value<a name="GET_apigraphslineageverticeskey:keyvalue:value"></a></h3> +<h3>GET api/metadata/lineage/vertices?key=:key&value=:value<a name="GET_apimetadatalineageverticeskey:keyvalue:value"></a></h3> <p></p> <ul> <li><a href="#Description">Description</a></li> @@ -280,7 +280,7 @@ <h4>Rest Call<a name="Rest_Call"></a></h4> <div class="source"> <pre> -GET http://localhost:15000/api/graphs/lineage/vertices?key=name&value=sampleIngestProcess +GET http://localhost:15000/api/metadata/lineage/vertices?key=name&value=sampleIngestProcess </pre></div></div> <div class="section"> Modified: falcon/site/source-repository.html URL: http://svn.apache.org/viewvc/falcon/site/source-repository.html?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/site/source-repository.html (original) +++ falcon/site/source-repository.html Wed Feb 18 10:55:56 2015 @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-01-11 + | Generated by Apache Maven Doxia at 2015-02-18 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20150111" /> + <meta name="Date-Revision-yyyymmdd" content="20150218" /> <meta http-equiv="Content-Language" content="en" /> <title>Falcon - Source Repository</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -245,7 +245,7 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-01-11</li> + <li id="publishDate" class="pull-right">Last Published: 2015-02-18</li> </ul> </div> Modified: falcon/site/team-list.html URL: http://svn.apache.org/viewvc/falcon/site/team-list.html?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/site/team-list.html (original) +++ falcon/site/team-list.html Wed Feb 18 10:55:56 2015 @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-01-11 + | Generated by Apache Maven Doxia at 2015-02-18 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20150111" /> + <meta name="Date-Revision-yyyymmdd" content="20150218" /> <meta http-equiv="Content-Language" content="en" /> <title>Falcon - Team list</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -245,7 +245,7 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-01-11</li> + <li id="publishDate" class="pull-right">Last Published: 2015-02-18</li> </ul> </div> Added: falcon/trunk/general/src/site/resources/images/accessories-text-editor.png URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/accessories-text-editor.png?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/accessories-text-editor.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/add.gif URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/add.gif?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/add.gif ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/apache-incubator-logo.png URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/apache-incubator-logo.png?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/apache-incubator-logo.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/apache-maven-project-2.png URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/apache-maven-project-2.png?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/apache-maven-project-2.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/application-certificate.png URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/application-certificate.png?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/application-certificate.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/contact-new.png URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/contact-new.png?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/contact-new.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/document-properties.png URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/document-properties.png?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/document-properties.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/drive-harddisk.png URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/drive-harddisk.png?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/drive-harddisk.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/fix.gif URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/fix.gif?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/fix.gif ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/icon_error_sml.gif URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/icon_error_sml.gif?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/icon_error_sml.gif ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/icon_help_sml.gif URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/icon_help_sml.gif?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/icon_help_sml.gif ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/icon_info_sml.gif URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/icon_info_sml.gif?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/icon_info_sml.gif ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/icon_success_sml.gif URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/icon_success_sml.gif?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/icon_success_sml.gif ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/icon_warning_sml.gif URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/icon_warning_sml.gif?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/icon_warning_sml.gif ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/image-x-generic.png URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/image-x-generic.png?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/image-x-generic.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/internet-web-browser.png URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/internet-web-browser.png?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/internet-web-browser.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/logos/build-by-maven-black.png URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/logos/build-by-maven-black.png?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/logos/build-by-maven-black.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/logos/build-by-maven-white.png URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/logos/build-by-maven-white.png?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/logos/build-by-maven-white.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/logos/maven-feather.png URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/logos/maven-feather.png?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/logos/maven-feather.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/network-server.png URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/network-server.png?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/network-server.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/package-x-generic.png URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/package-x-generic.png?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/package-x-generic.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/profiles/pre-release.png URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/profiles/pre-release.png?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/profiles/pre-release.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/profiles/retired.png URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/profiles/retired.png?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/profiles/retired.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/profiles/sandbox.png URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/profiles/sandbox.png?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/profiles/sandbox.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/remove.gif URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/remove.gif?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/remove.gif ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/rss.png URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/rss.png?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/rss.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/update.gif URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/update.gif?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/update.gif ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/resources/images/window-new.png URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/resources/images/window-new.png?rev=1660589&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/trunk/general/src/site/resources/images/window-new.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/trunk/general/src/site/twiki/Appendix.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/Appendix.twiki?rev=1660589&view=auto ============================================================================== --- falcon/trunk/general/src/site/twiki/Appendix.twiki (added) +++ falcon/trunk/general/src/site/twiki/Appendix.twiki Wed Feb 18 10:55:56 2015 @@ -0,0 +1,55 @@ +---+ Compatibility + +---++ 0.6-incubating Version Compatibility Matrix + + * Hadoop 2.5.0 and above + * Oozie 4.0.0 and above + * Hive 0.11.0 and above + * HCatalog 0.11.0 and above + * Active MQ 5.4.3 and above + * Titan 0.4.2 and above but below 0.5 + + +---++ 0.6-incubating Tested Compatibility + + * Hadoop 2.6.0 + * Oozie 4.1.0 + * Hive 0.14.0 + * HCatalog 0.14.0 + * Active MQ 5.4.3 + * Titan 0.4.2 + * Java 1.6, Java 1.7 + + Note : Oozie versions below 4.1.0 are not compatible with Java 1.7 + +---++ 0.6-incubating Release Notes + +Major additions are listed below. Refer to CHANGES.txt for detailed issues addressed in this release. + + * Security - Authorization, SSL + * Lineage - More complete with better API + * Recipes + * Usability improvements - Dry run, entity summary, Pagination, etc. + * Operability - Alerts, Audits, etc. + * Refactoring - Messaging, Orchestration of workflows, etc. + * Extension points for developers + * Many bug fixes + + +---++ 0.6-incubating Upgrade Instructions + +Please follow these instructions when upgrading from an older release. + +---+++ Upgrading from 0.5-incubating + +0.6-incubating is backwards *incompatible* with 0.5-incubating. It is recommended that user do not +migrate from 0.5 to 0.6. However if the user must migrate from 0.5-incubating to 0.6-incubating, +user should [[https://cwiki.apache.org/confluence/display/FALCON/Index][follow these instructions] + +---+++ Upgrading from 0.4-incubating + +It is not possible to upgrade to 0.6-incubating from 0.4-incubating. + +---+++ Upgrading from 0.3-incubating + +It is not possible to upgrade to 0.6-incubating from 0.3-incubating. Modified: falcon/trunk/general/src/site/twiki/EntitySpecification.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/EntitySpecification.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/EntitySpecification.twiki (original) +++ falcon/trunk/general/src/site/twiki/EntitySpecification.twiki Wed Feb 18 10:55:56 2015 @@ -29,18 +29,18 @@ this would be used in the context of fee <verbatim> <interface type="write" endpoint="hdfs://localhost:8020" version="0.20.2" /> </verbatim> -A write interface specifies the interface to write to hdfs, it's endpoint is the value of fs.default.name. +A write interface specifies the interface to write to hdfs, it's endpoint is the value of fs.defaultFS. Falcon uses this interface to write system data to hdfs and feeds referencing this cluster are written to hdfs using the same write interface. <verbatim> <interface type="execute" endpoint="localhost:8021" version="0.20.2" /> </verbatim> -An execute interface specifies the interface for job tracker, it's endpoint is the value of mapred.job.tracker. +An execute interface specifies the interface for job tracker, it's endpoint is the value of mapreduce.jobtracker.address. Falcon uses this interface to submit the processes as jobs on !JobTracker defined here. <verbatim> -<interface type="workflow" endpoint="http://localhost:11000/oozie/"; version="3.1" /> +<interface type="workflow" endpoint="http://localhost:11000/oozie/"; version="4.0" /> </verbatim> A workflow interface specifies the interface for workflow engine, example of its endpoint is the value for OOZIE_URL. Falcon uses this interface to schedule the processes referencing this cluster on workflow engine defined here. @@ -69,6 +69,11 @@ Location has the name and the path, name and path is the hdfs path for each location. Falcon would use the location to do intermediate processing of entities in hdfs and hence Falcon should have read/write/execute permission on these locations. +These locations MUST be created prior to submitting a cluster entity to Falcon. +*staging* must have 777 permissions and the parent dirs must have execute permissions so multiple +users can write to this location +*working* must have 755 permissions and the parent dirs must have execute permissions so multiple +users can read from this location ---+++ ACL @@ -94,8 +99,7 @@ should be defined here. ---++ Feed Specification The Feed XSD specification is available here. -a Feed defines various attributes of feed like feed location, frequency, late-arrival handling -and retention policies. +A Feed defines various attributes of feed like feed location, frequency, late-arrival handling and retention policies. A feed can be scheduled on a cluster, once a feed is scheduled its retention and replication process are triggered in a given cluster. <verbatim> <feed description="clicks log" name="clicks" xmlns="uri:falcon:feed:0.1" @@ -123,6 +127,7 @@ Feed should contain one of the two stora <cluster name="test-cluster"> <validity start="2012-07-20T03:00Z" end="2099-07-16T00:00Z"/> <retention limit="days(10)" action="delete"/> + <sla slaLow="hours(3)" slaHigh="hours(4)"/> <locations> <location type="data" path="/hdfsDataLocation/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/> <location type="stats" path="/projects/falcon/clicksStats" /> @@ -137,8 +142,11 @@ Validity of a feed on cluster specifies Retention specifies how long the feed is retained on this cluster and the action to be taken on the feed after the expiry of retention period. The retention limit is specified by expression frequency(times), ex: if feed should be retained for at least 6 hours then retention's limit="hours(6)". The field partitionExp contains partition tags. Number of partition tags has to be equal to number of partitions specified in feed schema. A partition tag can be a wildcard(*), a static string or an expression. Atleast one of the strings has to be an expression. +sla specifies sla for the feed on this cluster. This is an optional parameter and sla can be same or different from the +global sla tag (mentioned outside the clusters tag ). This tag provides the user to flexibility to have +different sla for different clusters e.g. in case of replication. If this attribute is missing then the default global +sla is picked from the feed definition. Location specifies where the feed is available on this cluster. This is an optional parameter and path can be same or different from the global locations tag value ( it is mentioned outside the clusters tag ) . This tag provides the user to flexibility to have feed at different locations on different clusters. If this attribute is missing then the default global location is picked from the feed definition. Also the individual location tags data, stats, meta are optional. - <verbatim> <location type="data" path="/projects/falcon/clicks" /> <location type="stats" path="/projects/falcon/clicksStats" /> @@ -225,6 +233,17 @@ A feed has a frequency which specifies t ex: it can be generated every hour, every 5 minutes, daily, weekly etc. valid frequency type for a feed are minutes, hours, days, months. The values can be negative, zero or positive. +---+++ SLA +<verbatim> + <sla slaLow="hours(40)" slaHigh="hours(44)" /> +</verbatim> + +A feed can have SLA and each SLA has two properties - slaLow and slaHigh. Both slaLow and slaHigh are written using +expressions like frequency. slaLow is intended to serve for alerting for feed instances which are in danger of missing their +availability SLAs. slaHigh is intended to serve for reporting the feeds which missed their SLAs. SLAs are relative to +feed instance time. + + ---+++ Late Arrival <verbatim> @@ -259,13 +278,13 @@ permission indicates the permission. <property name="timeout" value="hours(1)"/> <property name="parallel" value="3"/> <property name="maxMaps" value="8"/> - <property name="mapBandwidthKB" value="1024"/> + <property name="mapBandwidth" value="1"/> </properties> </verbatim> A key-value pair, which are propagated to the workflow engine. "queueName" and "jobPriority" are special properties available to user to specify the Hadoop job queue and priority, the same value is used by Falcons launcher job. "timeout" and "parallel" are other special properties which decides replication instance's timeout value while -waiting for the feed instance and parallel decides the concurrent replication instances that can run at any given time. "maxMaps" represents the maximum number of maps used during replication. "mapBandwidthKB" represents the bandwidth in KB/s used by each mapper during replication. +waiting for the feed instance and parallel decides the concurrent replication instances that can run at any given time. "maxMaps" represents the maximum number of maps used during replication. "mapBandwidth" represents the bandwidth in MB/s used by each mapper during replication. ---++ Process Specification A process defines configuration for a workflow. A workflow is a directed acyclic graph(DAG) which defines the job for the workflow engine. A process definition defines the configurations required to run the workflow job. For example, process defines the frequency at which the workflow should run, the clusters on which the workflow should run, the inputs and outputs for the workflow, how the workflow failures should be handled, how the late inputs should be handled and so on. @@ -363,6 +382,15 @@ Syntax: </process> </verbatim> +---+++ SLA +<verbatim> + <sla shouldStartIn="hours(2)" shouldEndIn="hours(4)"/> +</verbatim> +A process can have SLA which is defined by 2 optional attributes - shouldStartIn and shouldEndIn. All the attributes +are written using expressions like frequency. shouldStartIn is the time by which the process should have started. +shouldEndIn is the time by which the process should have finished. + + ---+++ Validity Validity defines how long the workflow should run. It has 3 components - start time, end time and timezone. Start time and end time are timestamps defined in yyyy-MM-dd'T'HH:mm'Z' format and should always be in UTC. Timezone is used to compute the next instances starting from start time. The workflow will start at start time and end before end time specified on a given cluster. So, there will not be a workflow instance at end time. Syntax: @@ -645,8 +673,8 @@ There are 2 engines supported today. ---++++ Oozie As part of oozie workflow engine support, users can embed a oozie workflow. -Refer to oozie [[http://oozie.apache.org/docs/4.0.0/DG_Overview.html][workflow overview]] and -[[http://oozie.apache.org/docs/4.0.0/WorkflowFunctionalSpec.html][workflow specification]] for details. +Refer to oozie [[http://oozie.apache.org/docs/4.0.1/DG_Overview.html][workflow overview]] and +[[http://oozie.apache.org/docs/4.0.1/WorkflowFunctionalSpec.html][workflow specification]] for details. Syntax: <verbatim> Modified: falcon/trunk/general/src/site/twiki/FalconCLI.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/FalconCLI.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/FalconCLI.twiki (original) +++ falcon/trunk/general/src/site/twiki/FalconCLI.twiki Wed Feb 18 10:55:56 2015 @@ -56,12 +56,15 @@ Optional Args : -fields <<field1,field2> <a href="./Restapi/EntityList.html">Optional params described here.</a> + + + ---+++Summary Summary of entities of a particular type and a cluster will be listed. Entity summary has N most recent instances of entity. Usage: -$FALCON_HOME/bin/falcon entity -type [cluster|feed|process] -summary +$FALCON_HOME/bin/falcon entity -type [feed|process] -summary Optional Args : -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" -fields <<field1,field2>> -filterBy <<field1:value1,field2:value2>> -tags <<tagkey=tagvalue,tagkey=tagvalue>> @@ -75,7 +78,17 @@ Update operation allows an already submi not allowed. Usage: -$FALCON_HOME/bin/falcon entity -type [feed|process] -name <<name>> -update [-effective <<effective time>>] +$FALCON_HOME/bin/falcon entity -type [feed|process] -name <<name>> -update -file <<path_to_file>> + +Example: +$FALCON_HOME/bin/falcon entity -type process -name HourlyReportsGenerator -update -file /process/definition.xml + +---+++Touch + +Force Update operation allows an already submitted/scheduled entity to be updated. + +Usage: +$FALCON_HOME/bin/falcon entity -type [feed|process] -name <<name>> -touch ---+++Status @@ -105,9 +118,7 @@ $FALCON_HOME/bin/falcon entity -type [cl Kill sub-command is used to kill all the instances of the specified process whose nominal time is between the given start time and end time. Note: -1. For all the instance management sub-commands, if end time is not specified, Falcon will perform the actions on all the instances whose instance time falls after the start time. - -2. The start time and end time needs to be specified in TZ format. +1. The start time and end time needs to be specified in TZ format. Example: 01 Jan 2012 01:00 => 2012-01-01T01:00Z 3. Process name is compulsory parameter for each instance management command. @@ -124,17 +135,17 @@ $FALCON_HOME/bin/falcon instance -type < ---+++Continue -Continue option is used to continue the failed workflow instance. This option is valid only for process instances in terminal state, i.e. SUCCEDDED, KILLED or FAILED. +Continue option is used to continue the failed workflow instance. This option is valid only for process instances in terminal state, i.e. KILLED or FAILED. Usage: -$FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -re-run -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" +$FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -continue -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" ---+++Rerun Rerun option is used to rerun instances of a given process. This option is valid only for process instances in terminal state, i.e. SUCCEDDED, KILLED or FAILED. Optionally, you can specify the properties to override. Usage: -$FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -re-run -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" [-file <<properties file>>] +$FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -rerun -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" [-file <<properties file>>] ---+++Resume @@ -146,7 +157,8 @@ $FALCON_HOME/bin/falcon instance -type < ---+++Status Status option via CLI can be used to get the status of a single or multiple instances. If the instance is not yet materialized but is within the process validity range, WAITING is returned as the state. Along with the status of the instance time is also returned. Log location gives the oozie workflow url -If the instance is in WAITING state, missing dependencies are listed +If the instance is in WAITING state, missing dependencies are listed. +The job urls are populated for all actions of user workflow and non-succeeded actions of the main-workflow. The user then need not go to the underlying scheduler to get the job urls when needed to debug an issue in the job. Example : Suppose a process has 3 instance, one has succeeded,one is in running state and other one is waiting, the expected output is: @@ -209,6 +221,18 @@ Optional Args : -colo <<colo>> -lifecycl <a href="./Restapi/InstanceRunning.html">Optional params described here.</a> +---+++FeedInstanceListing + +Get falcon feed instance availability. + +Usage: +$FALCON_HOME/bin/falcon instance -entity feed -name <<name>> -listing + +Optional Args : -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" +-colo <<colo>> + +<a href="./Restapi/FeedInstanceListing.html">Optional params described here.</a> + ---+++Logs Get logs for instance actions @@ -238,48 +262,85 @@ Usage: $FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -params -start "yyyy-MM-dd'T'HH:mm'Z'" ----++ Graphs Options +---++ Metadata Lineage Options + +---+++Lineage + +Returns the relationship between processes and feeds in a given pipeline in <a href="http://www.graphviz.org/content/dot-language";>dot</a> format. +You can use the output and view a graphical representation of DAG using an online graphviz viewer like <a href="http://graphviz-dev.appspot.com/";>this</a>. + + +Usage: + +$FALCON_HOME/bin/falcon metadata -lineage -pipeline my-pipeline + +pipeline is a mandatory option. + + ---+++ Vertex Get the vertex with the specified id. Usage: -$FALCON_HOME/bin/falcon graph -vertex -id <<id>> +$FALCON_HOME/bin/falcon metadata -vertex -id <<id>> Example: -$FALCON_HOME/bin/falcon graph -vertex -id 4 +$FALCON_HOME/bin/falcon metadata -vertex -id 4 ---+++ Vertices Get all vertices for a key index given the specified value. Usage: -$FALCON_HOME/bin/falcon graph -vertices -key <<key>> -value <<value>> +$FALCON_HOME/bin/falcon metadata -vertices -key <<key>> -value <<value>> Example: -$FALCON_HOME/bin/falcon graph -vertices -key type -value feed-instance +$FALCON_HOME/bin/falcon metadata -vertices -key type -value feed-instance ---+++ Vertex Edges Get the adjacent vertices or edges of the vertex with the specified direction. Usage: -$FALCON_HOME/bin/falcon graph -edges -id <<vertex-id>> -direction <<direction>> +$FALCON_HOME/bin/falcon metadata -edges -id <<vertex-id>> -direction <<direction>> Example: -$FALCON_HOME/bin/falcon graph -edges -id 4 -direction both -$FALCON_HOME/bin/falcon graph -edges -id 4 -direction inE +$FALCON_HOME/bin/falcon metadata -edges -id 4 -direction both +$FALCON_HOME/bin/falcon metadata -edges -id 4 -direction inE ---+++ Edge Get the edge with the specified id. Usage: -$FALCON_HOME/bin/falcon graph -edge -id <<id>> +$FALCON_HOME/bin/falcon metadata -edge -id <<id>> + +Example: +$FALCON_HOME/bin/falcon metadata -edge -id Q9n-Q-5g + +---++ Metadata Discovery Options + +---+++ List + +Lists of all dimensions of given type. If the user provides optional param cluster, only the dimensions related to the cluster are listed. +Usage: +$FALCON_HOME/bin/falcon metadata -list -type [cluster_entity|feed_entity|process_entity|user|colo|tags|groups|pipelines] + +Optional Args : -cluster <<cluster name>> + +Example: +$FALCON_HOME/bin/falcon metadata -list -type process_entity -cluster primary-cluster +$FALCON_HOME/bin/falcon metadata -list -type tags + +---+++ Relations + +List all dimensions related to specified Dimension identified by dimension-type and dimension-name. +Usage: +$FALCON_HOME/bin/falcon metadata -relations -type [cluster_entity|feed_entity|process_entity|user|colo|tags|groups|pipelines] -name <<Dimension Name>> Example: -$FALCON_HOME/bin/falcon graph -edge -id Q9n-Q-5g +$FALCON_HOME/bin/falcon metadata -relations -type process_entity -name sample-process ---++Admin Options @@ -300,3 +361,23 @@ $FALCON_HOME/bin/falcon admin -version Status returns the current state of Falcon (running or stopped). Usage: $FALCON_HOME/bin/falcon admin -status + + +---++ Recipe Options + +---+++ Submit Recipe + +Submit the specified recipe. + +Usage: +$FALCON_HOME/bin/falcon recipe -name <name> +Name of the recipe. User should have defined <name>-template.xml and <name>.properties in the path specified by falcon.recipe.path in client.properties file. falcon.home path is used if its not specified in client.properties file. +If its not specified in client.properties file and also if files cannot be found at falcon.home, Falcon CLI will fail. + +Optional Args : -tool <recipeToolClassName> +Falcon provides a base tool that recipes can override. If this option is not specified the default Recipe Tool +RecipeTool defined is used. This option is required if user defines his own recipe tool class. + +Example: +$FALCON_HOME/bin/falcon recipe -name hdfs-replication + Modified: falcon/trunk/general/src/site/twiki/FalconDocumentation.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/FalconDocumentation.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/FalconDocumentation.twiki (original) +++ falcon/trunk/general/src/site/twiki/FalconDocumentation.twiki Wed Feb 18 10:55:56 2015 @@ -10,12 +10,15 @@ * <a href="#Updating_process_and_feed_definition">Updating process and feed definition</a> * <a href="#Handling_late_input_data">Handling late input data</a> * <a href="#Idempotency">Idempotency</a> - * <a href="#Alerting_and_Monitoring">Alerting and Monitoring</a> * <a href="#Falcon_EL_Expressions">Falcon EL Expressions</a> * <a href="#Lineage">Lineage</a> * <a href="#Security">Security</a> + * <a href="#Recipes">Recipes</a> + * <a href="#Monitoring">Monitoring</a> + * <a href="#Backwards_Compatibility">Backwards Compatibility Instructions</a> ---++ Architecture + ---+++ Introduction Falcon is a feed and process management platform over hadoop. Falcon essentially transforms user's feed and process configurations into repeated actions through a standard workflow engine. Falcon by itself @@ -53,11 +56,18 @@ There are two basic components of Falcon As the name suggests Falcon Prism splits the request it gets to the Falcon Servers. More details below: ---+++ Stand Alone Mode -Stand alone mode is useful when the hadoop jobs and relevant data processing involves only one hadoop cluster. In this mode there is single Falcon server that contacts with oozie to schedule jobs on Hadoop. All the process / feed request like submit, schedule, suspend, kill are sent to this server only. For running in this mode one should use the falcon which has been built for standalone mode, or build using standalone option if using source code. +Stand alone mode is useful when the hadoop jobs and relevant data processing involves only one hadoop cluster. +In this mode there is a single Falcon server that contacts Oozie to schedule jobs on Hadoop. +All the process/feed requests like submit, schedule, suspend, kill etc. are sent to this server. +For running falcon in this mode one should use the falcon which has been built using standalone option. ---+++ Distributed Mode -Distributed mode is the mode which you might me using most of the time. This is for organisations which have multiple instances of hadoop clusters, and multiple workflow schedulers to handle them. Here we have 2 components: Prism and Server. Both Prism and server have there own setup (runtime and startup properties) and there config locations. -In this mode Prism acts as a contact point for Falcon servers. Below are the requests that can be sent to prism and server in this mode: +Distributed mode is for multiple (colos) instances of hadoop clusters, and multiple workflow schedulers to handle them. +In this mode falcon has 2 components: Prism and Server(s). +Both Prism and servers have their own setup (runtime and startup properties) and their own config locations. +In this mode Prism acts as a contact point for Falcon servers. +While all commands are available through Prism, only read and instance api's are available through Server. +Below are the requests that can be sent to each of these: Prism: submit, schedule, submitAndSchedule, Suspend, Resume, Kill, instance management Server: schedule, suspend, resume, instance management @@ -192,20 +202,12 @@ no dependent entities on the deleted ent ---+++ Update Update operation allows an already submitted/scheduled entity to be updated. Cluster update is currently not allowed. Feed update can cause cascading update to all the processes already scheduled. Process update triggers -update in falcon if entity is updated/the user specified workflow/lib is updated. The following set of actions are -performed in Oozie to realize an update: - * Suspend the previously scheduled Oozie coordinator. This is to prevent any new action from being triggered. - * Update the coordinator to set the end time to "now" - * Resume the suspended coordinators +update in falcon if entity is updated. The following set of actions are performed in scheduler to realize an update: + * Update the old scheduled entity to set the end time to "now" * Schedule as per the new process/feed definition with the start time as "now" -Update optionally takes effective time as a parameter which is used as the end time of previously scheduled coordinator. -So, the updated configuration will be effective since the given timestamp. - - ---++ Instance Management actions - Instance Manager gives user the option to control individual instances of the process based on their instance start time (start time of that instance). Start time needs to be given in standard TZ format. Example: 01 Jan 2012 01:00 => 2012-01-01T01:00Z All the instance management operations (except running) allow single instance or list of instance within a Date range to be acted on. Make sure the dates are valid. i.e. are within the start and end time of process itself. @@ -257,7 +259,7 @@ With the integration of Hive, Falcon als ---+++ Example: If retention period is 10 hours, and the policy kicks in at time 't', the data retained by system is essentially the -one falling in between [t-10h,t]. Any data in the boundaries [-�,t-10h) and (t,�] is removed from the system. +one in range [t-10h, t]. Any data before t-10h and after t is removed from the system. The 'action' attribute can attain values of DELETE/ARCHIVE. Based upon the tag value, the data eligible for removal is either deleted/archived. @@ -368,6 +370,28 @@ such as schema, storage format, etc. cluster, (no dirty reads) +---+++ Archival as Replication + +Falcon allows users to archive data from on-premice to cloud, either Azure WASB or S3. +It uses the underlying replication for archiving data from source to target. The archival URI is +specified as the overridden location for the target cluster. + +*Example:* +<verbatim> + <clusters> + <cluster name="on-premise-cluster" type="source"> + <validity start="2021-11-01T00:00Z" end="2021-12-31T00:00Z"/> + </cluster> + <cluster name="cloud-cluster" type="target"> + <validity start="2011-11-01T00:00Z" end="2011-12-31T00:00Z"/> + <locations> + <location type="data" + path="wasb://[email protected]/data/${YEAR}-${MONTH}-${DAY}-${HOUR}"/> + </locations> + </cluster> + </clusters> +</verbatim> + ---+++ Relation between feed's retention limit and feed's late arrival cut off period: For reasons that are obvious, Falcon has an external validation that ensures that the user @@ -556,76 +580,6 @@ All the operations in Falcon are Idempot Idempotency also by takes care of the condition when request is sent through prism and fails on one or more servers. For example prism is configured to send request to 3 servers. First user sends a request to SUBMIT a process on all 3 of them, and receives a response SUCCESSFUL from all of them. Then due to some issue one of the servers goes down, and user send a request to schedule the submitted process. This time he will receive a response with PARTIAL status and a FAILURE message from the server that has gone down. If the users check he will find the process would have been started and running on the 2 SUCCESSFUL servers. Now the issue with server is figured out and it is brought up. Sending the SCHEDULE request again through prism will result in a SUCCESSFUL response from prism as well as other three servers, but this time PROCESS will be SCHEDULED only on the server which had failed earlier and other two will keep running as before. ----++ Alerting and Monitoring ----+++ Alerting -Falcon provides monitoring of various events by capturing metrics of those events. -The metric numbers can then be used to monitor performance and health of the Falcon system and the entire processing pipelines. - -Users can view the logs of these events in the metric.log file, by default this file is created under ${user.dir}/logs/ directory. -Users may also extend the Falcon monitoring framework to send events to systems like Mondemand/lwes. - -The following events are captured by Falcon for logging the metrics: - 1. New cluster definitions posted to Falcon (success & failures) - 1. New feed definition posted to Falcon (success & failures) - 1. New process definition posted to Falcon (success & failures) - 1. Process update events (success & failures) - 1. Feed update events (success & failures) - 1. Cluster update events (success & failures) - 1. Process suspend events (success & failures) - 1. Feed suspend events (success & failures) - 1. Process resume events (success & failures) - 1. Feed resume events (success & failures) - 1. Process remove events (success & failures) - 1. Feed remove events (success & failures) - 1. Cluster remove events (success & failures) - 1. Process instance kill events (success & failures) - 1. Process instance re-run events (success & failures) - 1. Process instance generation events - 1. Process instance failure events - 1. Process instance auto-retry events - 1. Process instance retry exhaust events - 1. Feed instance deletion event - 1. Feed instance deletion failure event (no retries) - 1. Feed instance replication event - 1. Feed instance replication failure event - 1. Feed instance replication auto-retry event - 1. Feed instance replication retry exhaust event - 1. Feed instance late arrival event - 1. Feed instance post cut-off arrival event - 1. Process re-run due to late feed event - 1. Transaction rollback failed event - -The metric logged for an event has the following properties: - 1. Action - Name of the event. - 2. Dimensions - A list of name/value pairs of various attributes for a given action. - 3. Status- Status of an action FAILED/SUCCEEDED. - 4. Time-taken - Time taken in nanoseconds for a given action. - -An example for an event logged for a submit of a new process definition: - - 2012-05-04 12:23:34,026 {Action:submit, Dimensions:{entityType=process}, Status: SUCCEEDED, Time-taken:97087000 ns} - -Users may parse the metric.log or capture these events from custom monitoring frameworks and can plot various graphs -or send alerts according to their requirements. - ----+++ Notifications -Falcon creates a JMS topic for every process/feed that is scheduled in Falcon. -The implementation class and the broker url of the JMS engine are read from the dependent cluster's definition. -Users may register consumers on the required topic to check the availability or status of feed instances. - -For a given process that is scheduled, the name of the topic is same as the process name. -Falcon sends a Map message for every feed produced by the instance of a process to the JMS topic. -The JMS !MapMessage sent to a topic has the following properties: -entityName, feedNames, feedInstancePath, workflowId, runId, nominalTime, timeStamp, brokerUrl, brokerImplClass, entityType, operation, logFile, topicName, status, brokerTTL; - -For a given feed that is scheduled, the name of the topic is same as the feed name. -Falcon sends a map message for every feed instance that is deleted/archived/replicated depending upon the retention policy set in the feed definition. -The JMS !MapMessage sent to a topic has the following properties: -entityName, feedNames, feedInstancePath, workflowId, runId, nominalTime, timeStamp, brokerUrl, brokerImplClass, entityType, operation, logFile, topicName, status, brokerTTL; - -The JMS messages are automatically purged after a certain period (default 3 days) by the Falcon JMS house-keeping service.TTL (Time-to-live) for JMS message -can be configured in the Falcon's startup.properties file. - ---++ Falcon EL Expressions @@ -740,11 +694,23 @@ This feature is enabled by default but c <verbatim> config name: *.application.services config value: org.apache.falcon.metadata.MetadataMappingService -<verbatim> +</verbatim> Lineage is only captured for Process executions. A future release will capture lineage for lifecycle policies such as replication and retention. ---++ Security +---++Security Security is detailed in [[Security][Security]]. + +---++ Recipes + +Recipes is detailed in [[Recipes][Recipes]]. + +---++ Monitoring + +Monitoring and Operationalizing Falcon is detailed in [[Operability][Operability]]. + +---++ Backwards Compatibility + +Backwards compatibility instructions are [[Compatibility][detailed here.]] Modified: falcon/trunk/general/src/site/twiki/HiveIntegration.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/HiveIntegration.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/HiveIntegration.twiki (original) +++ falcon/trunk/general/src/site/twiki/HiveIntegration.twiki Wed Feb 18 10:55:56 2015 @@ -41,11 +41,11 @@ Hence, Falcon for Hive support needs Ooz ---++ Oozie Shared Library setup -Falcon post Hive integration depends heavily on the [[http://oozie.apache.org/docs/4.0.0/WorkflowFunctionalSpec.html#a17_HDFS_Share_Libraries_for_Workflow_Applications_since_Oozie_2.3][shared library feature of Oozie]]. +Falcon post Hive integration depends heavily on the [[http://oozie.apache.org/docs/4.0.1/WorkflowFunctionalSpec.html#a17_HDFS_Share_Libraries_for_Workflow_Applications_since_Oozie_2.3][shared library feature of Oozie]]. Since the sheer number of jars for HCatalog, Pig and Hive are in the many 10s in numbers, its quite daunting to redistribute the dependent jars from Falcon. -[[http://oozie.apache.org/docs/4.0.0/DG_QuickStart.html#Oozie_Share_Lib_Installation][This is a one time effort in Oozie setup and is quite straightforward.]] +[[http://oozie.apache.org/docs/4.0.1/DG_QuickStart.html#Oozie_Share_Lib_Installation][This is a one time effort in Oozie setup and is quite straightforward.]] ---++ Approach @@ -195,7 +195,7 @@ Following is an example entity configura <interface type="execute" endpoint="localhost:10300" version="1.1.1" /> <interface type="workflow" endpoint="http://localhost:11010/oozie/"; - version="3.3.0" /> + version="4.0.1" /> <interface type="registry" endpoint="thrift://localhost:19083" version="0.11.0" /> <interface type="messaging" endpoint="tcp://localhost:61616?daemon=true" @@ -227,7 +227,7 @@ Following is an example entity configura <interface type="execute" endpoint="localhost:20300" version="1.1.1" /> <interface type="workflow" endpoint="http://localhost:11020/oozie/"; - version="3.3.0" /> + version="4.0.1" /> <interface type="registry" endpoint="thrift://localhost:29083" version="0.11.0" /> <interface type="messaging" endpoint="tcp://localhost:61616?daemon=true" Modified: falcon/trunk/general/src/site/twiki/InstallationSteps.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/InstallationSteps.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/InstallationSteps.twiki (original) +++ falcon/trunk/general/src/site/twiki/InstallationSteps.twiki Wed Feb 18 10:55:56 2015 @@ -4,15 +4,22 @@ ---+++ Building Falcon <verbatim> +You would need the following installed to build Falcon + +* JDK 1.7 +* Maven 3.x + git clone https://git-wip-us.apache.org/repos/asf/falcon.git falcon cd falcon -export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m" && mvn clean install [For hadoop 1] -export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m" && mvn clean install -Phadoop-2 [For hadoop 2] +export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m -noverify" && mvn clean install [optionally -Dhadoop.version=<<hadoop.version>> can be appended to build for a specific version of hadoop] -[optionally -Doozie.version=<<oozie version>> can be appended to build with a specific version of oozie. Oozie versions >= 3.oozie-3.2.0-incubating are supported] +*Note:* Falcon drops support for Hadoop-1 and only supports Hadoop-2 from Falcon 0.6 onwards +[optionally -Doozie.version=<<oozie version>> can be appended to build with a specific version of oozie. +Oozie versions >= 4 are supported] +Falcon build with JDK 1.7 using -noverify option </verbatim> @@ -21,12 +28,11 @@ Once the build successfully completes, a *Embedded Mode* <verbatim> -mvn clean assembly:assembly -DskipTests -DskipCheck=true [For hadoop 1] -mvn clean assembly:assembly -DskipTests -DskipCheck=true -P hadoop-2 [For hadoop 2] +mvn clean assembly:assembly -DskipTests -DskipCheck=true </verbatim> -Tar can be found in {project dir}/target/falcon-${project.version}-bin.tar.gz +Tar can be found in {project dir}/target/apache-falcon-${project.version}-bin.tar.gz Tar is structured as follows @@ -63,12 +69,11 @@ Tar is structured as follows <verbatim> -mvn clean assembly:assembly -DskipTests -DskipCheck=true -Pdistributed,hadoop-1 [For hadoop 1] -mvn clean assembly:assembly -DskipTests -DskipCheck=true -Pdistributed,hadoop-2 [For hadoop 2] +mvn clean assembly:assembly -DskipTests -DskipCheck=true -Pdistributed,hadoop-2 </verbatim> -Tar can be found in {project dir}/target/falcon-distributed-${project.version}-server.tar.gz +Tar can be found in {project dir}/target/apache-falcon-distributed-${project.version}-server.tar.gz Tar is structured as follows @@ -160,14 +165,28 @@ In addition you can set any other enviro #export FALCON_EXPANDED_WEBAPP_DIR= </verbatim> + +*NOTE for Mac OS users* +<verbatim> +If you are using a Mac OS, you will need to configure the FALCON_SERVER_OPTS (explained above). + +In {package dir}/conf/falcon-env.sh uncomment the following line +#export FALCON_SERVER_OPTS= + +and change it to look as below +export FALCON_SERVER_OPTS="-Djava.awt.headless=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc=" +</verbatim> + *Starting Falcon Server* <verbatim> bin/falcon-start [-port <port>] </verbatim> -By default, -* falcon server starts at port 15443 (https) by default . To change the port, use -port option - * falcon.enableTLS can be set to true or false explicitly to enable SSL, if not port that end with 443 will automatically put falcon on https:// +By default, +* If falcon.enableTLS is set to true explicitly or not set at all, falcon starts at port 15443 on https:// by default. +* If falcon.enableTLS is set to false explicitly, falcon starts at port 15000 on http://. +* To change the port, use -port option. + * If falcon.enableTLS is not set explicitly, port that ends with 443 will automatically put falcon on https://. Any other port will put falcon on http://. * falcon server starts embedded active mq. To control this behaviour, set the following system properties using -D option in environment variable FALCON_OPTS: * falcon.embeddedmq=<true/false> - Should server start embedded active mq, default true * falcon.embeddedmq.port=<port> - Port for embedded active mq, default 61616 @@ -210,6 +229,18 @@ bin/falcon help Once falcon / prism is started, you can view the status of falcon entities using the Web-based dashboard. The web UI works in both distributed and embedded mode. You can open your browser at the corresponding port to use the web UI. +Falcon dashboard makes the REST api calls as user "falcon-dashboard". If this user does not exist on your falcon and oozie servers, please create the user. + +<verbatim> +## create user. +[root@falconhost ~] useradd -U -m falcon-dashboard -G users + +## verify user is created with membership in correct groups. +[root@falconhost ~] groups falcon-dashboard +falcon-dashboard : falcon-dashboard users +[root@falconhost ~] +</verbatim> + *Stopping Falcon Server* <verbatim> bin/falcon-stop @@ -225,9 +256,10 @@ bin/prism-stop cd <<project home>> src/bin/package.sh <<hadoop-version>> <<oozie-version>> ->> ex. src/bin/package.sh 1.1.2 3.1.3-incubating or src/bin/package.sh 0.20.2-cdh3u5 4.0.0 ->> Falcon package is available in <<falcon home>>/target/falcon-<<version>>-bin.tar.gz ->> Oozie package is available in <<falcon home>>/target/oozie-3.3.2-distro.tar.gz +>> ex. src/bin/package.sh 1.1.2 4.0.1 or src/bin/package.sh 0.20.2-cdh3u5 4.0.1 +>> ex. src/bin/package.sh 2.5.0 4.0.0 +>> Falcon package is available in <<falcon home>>/target/apache-falcon-<<version>>-bin.tar.gz +>> Oozie package is available in <<falcon home>>/target/oozie-4.0.1-distro.tar.gz </verbatim> ---+++ Running Examples using embedded package @@ -235,6 +267,9 @@ src/bin/package.sh <<hadoop-version>> << bin/falcon-start </verbatim> Make sure the hadoop and oozie endpoints are according to your setup in examples/entity/filesystem/standalone-cluster.xml +The cluster locations,staging and working dirs, MUST be created prior to submitting a cluster entity to Falcon. +*staging* must have 777 permissions and the parent dirs must have execute permissions +*working* must have 755 permissions and the parent dirs must have execute permissions <verbatim> bin/falcon entity -submit -type cluster -file examples/entity/filesystem/standalone-cluster.xml </verbatim>
