Modified: falcon/site/FalconCLI.html URL: http://svn.apache.org/viewvc/falcon/site/FalconCLI.html?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/site/FalconCLI.html (original) +++ falcon/site/FalconCLI.html Wed Feb 18 10:55:56 2015 @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-01-11 + | Generated by Apache Maven Doxia at 2015-02-18 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20150111" /> + <meta name="Date-Revision-yyyymmdd" content="20150218" /> <meta http-equiv="Content-Language" content="en" /> <title>Falcon - FalconCLI</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -245,7 +245,7 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-01-11</li> + <li id="publishDate" class="pull-right">Last Published: 2015-02-18</li> </ul> </div> @@ -290,13 +290,18 @@ <div class="section"> <h4>Summary<a name="Summary"></a></h4> <p>Summary of entities of a particular type and a cluster will be listed. Entity summary has N most recent instances of entity.</p> -<p>Usage: $FALCON_HOME/bin/falcon entity -type [cluster|feed|process] -summary</p> +<p>Usage: $FALCON_HOME/bin/falcon entity -type [feed|process] -summary</p> <p>Optional Args : -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" -fields <<field1,field2>> -filterBy <<field1:value1,field2:value2>> -tags <<tagkey=tagvalue,tagkey=tagvalue>> -orderBy <<field>> -sortOrder <<sortOrder>> -offset 0 -numResults 10 -numInstances 7</p> <p><a href="./Restapi/EntitySummary.html">Optional params described here.</a></p></div> <div class="section"> <h4>Update<a name="Update"></a></h4> <p>Update operation allows an already submitted/scheduled entity to be updated. Cluster update is currently not allowed.</p> -<p>Usage: $FALCON_HOME/bin/falcon entity -type [feed|process] -name <<name>> -update [-effective <<effective time>>]</p></div> +<p>Usage: $FALCON_HOME/bin/falcon entity -type [feed|process] -name <<name>> -update -file <<path_to_file>></p> +<p>Example: $FALCON_HOME/bin/falcon entity -type process -name <a href="./HourlyReportsGenerator.html">HourlyReportsGenerator</a> -update -file /process/definition.xml</p></div> +<div class="section"> +<h4>Touch<a name="Touch"></a></h4> +<p>Force Update operation allows an already submitted/scheduled entity to be updated.</p> +<p>Usage: $FALCON_HOME/bin/falcon entity -type [feed|process] -name <<name>> -touch</p></div> <div class="section"> <h4>Status<a name="Status"></a></h4> <p>Status returns the current status of the entity.</p> @@ -314,8 +319,7 @@ <div class="section"> <h4>Kill<a name="Kill"></a></h4> <p>Kill sub-command is used to kill all the instances of the specified process whose nominal time is between the given start time and end time.</p> -<p>Note: 1. For all the instance management sub-commands, if end time is not specified, Falcon will perform the actions on all the instances whose instance time falls after the start time.</p> -<p>2. The start time and end time needs to be specified in TZ format. Example: 01 Jan 2012 01:00 => 2012-01-01T01:00Z</p> +<p>Note: 1. The start time and end time needs to be specified in TZ format. Example: 01 Jan 2012 01:00 => 2012-01-01T01:00Z</p> <p>3. Process name is compulsory parameter for each instance management command.</p> <p>Usage: $FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -kill -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'"</p></div> <div class="section"> @@ -324,19 +328,19 @@ <p>Usage: $FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -suspend -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'"</p></div> <div class="section"> <h4>Continue<a name="Continue"></a></h4> -<p>Continue option is used to continue the failed workflow instance. This option is valid only for process instances in terminal state, i.e. SUCCEDDED, KILLED or FAILED.</p> -<p>Usage: $FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -re-run -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'"</p></div> +<p>Continue option is used to continue the failed workflow instance. This option is valid only for process instances in terminal state, i.e. KILLED or FAILED.</p> +<p>Usage: $FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -continue -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'"</p></div> <div class="section"> <h4>Rerun<a name="Rerun"></a></h4> <p>Rerun option is used to rerun instances of a given process. This option is valid only for process instances in terminal state, i.e. SUCCEDDED, KILLED or FAILED. Optionally, you can specify the properties to override.</p> -<p>Usage: $FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -re-run -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" [-file <<properties file>>]</p></div> +<p>Usage: $FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -rerun -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" [-file <<properties file>>]</p></div> <div class="section"> <h4>Resume<a name="Resume"></a></h4> <p>Resume option is used to resume any instance that is in suspended state.</p> <p>Usage: $FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -resume -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'"</p></div> <div class="section"> <h4>Status<a name="Status"></a></h4> -<p>Status option via CLI can be used to get the status of a single or multiple instances. If the instance is not yet materialized but is within the process validity range, WAITING is returned as the state. Along with the status of the instance time is also returned. Log location gives the oozie workflow url If the instance is in WAITING state, missing dependencies are listed</p> +<p>Status option via CLI can be used to get the status of a single or multiple instances. If the instance is not yet materialized but is within the process validity range, WAITING is returned as the state. Along with the status of the instance time is also returned. Log location gives the oozie workflow url If the instance is in WAITING state, missing dependencies are listed. The job urls are populated for all actions of user workflow and non-succeeded actions of the main-workflow. The user then need not go to the underlying scheduler to get the job urls when needed to debug an issue in the job.</p> <p>Example : Suppose a process has 3 instance, one has succeeded,one is in running state and other one is waiting, the expected output is:</p> <p>{"status":"SUCCEEDED","message":"getStatus is successful","instances":[{"instance":"2012-05-07T05:02Z","status":"SUCCEEDED","logFile":"http://oozie-dashboard-url"},{"instance":"2012-05-07T05:07Z","status":"RUNNING","logFile":"http://oozie-dashboard-url"}, {"instance":"2010-01-02T11:05Z","status":"WAITING"}]</p> <p>Usage: $FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -status</p> @@ -365,6 +369,12 @@ <p>Optional Args : -colo <<colo>> -lifecycle <<lifecycles>> -filterBy <<field1:value1,field2:value2>> -orderBy <<field>> -sortOrder <<sortOrder>> -offset 0 -numResults 10</p> <p><a href="./Restapi/InstanceRunning.html">Optional params described here.</a></p></div> <div class="section"> +<h4>FeedInstanceListing<a name="FeedInstanceListing"></a></h4> +<p>Get falcon feed instance availability.</p> +<p>Usage: $FALCON_HOME/bin/falcon instance -entity feed -name <<name>> -listing</p> +<p>Optional Args : -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" -colo <<colo>></p> +<p><a href="./Restapi/FeedInstanceListing.html">Optional params described here.</a></p></div> +<div class="section"> <h4>Logs<a name="Logs"></a></h4> <p>Get logs for instance actions</p> <p>Usage: $FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -logs</p> @@ -379,27 +389,44 @@ <p>Displays the workflow params of a given instance. Where start time is considered as nominal time of that instance.</p> <p>Usage: $FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -params -start "yyyy-MM-dd'T'HH:mm'Z'"</p></div> <div class="section"> -<h3>Graphs Options<a name="Graphs_Options"></a></h3></div> +<h3>Metadata Lineage Options<a name="Metadata_Lineage_Options"></a></h3></div> +<div class="section"> +<h4>Lineage<a name="Lineage"></a></h4> +<p><a href="http://www.graphviz.org/content/dot-language";>dot</a> format. You can use the output and view a graphical representation of DAG using an online graphviz viewer like <a href="http://graphviz-dev.appspot.com/";>this</a>.</p> +<p>Usage:</p> +<p>$FALCON_HOME/bin/falcon metadata -lineage -pipeline my-pipeline</p> +<p>pipeline is a mandatory option.</p></div> <div class="section"> <h4>Vertex<a name="Vertex"></a></h4> <p>Get the vertex with the specified id.</p> -<p>Usage: $FALCON_HOME/bin/falcon graph -vertex -id <<id>></p> -<p>Example: $FALCON_HOME/bin/falcon graph -vertex -id 4</p></div> +<p>Usage: $FALCON_HOME/bin/falcon metadata -vertex -id <<id>></p> +<p>Example: $FALCON_HOME/bin/falcon metadata -vertex -id 4</p></div> <div class="section"> <h4>Vertices<a name="Vertices"></a></h4> <p>Get all vertices for a key index given the specified value.</p> -<p>Usage: $FALCON_HOME/bin/falcon graph -vertices -key <<key>> -value <<value>></p> -<p>Example: $FALCON_HOME/bin/falcon graph -vertices -key type -value feed-instance</p></div> +<p>Usage: $FALCON_HOME/bin/falcon metadata -vertices -key <<key>> -value <<value>></p> +<p>Example: $FALCON_HOME/bin/falcon metadata -vertices -key type -value feed-instance</p></div> <div class="section"> <h4>Vertex Edges<a name="Vertex_Edges"></a></h4> <p>Get the adjacent vertices or edges of the vertex with the specified direction.</p> -<p>Usage: $FALCON_HOME/bin/falcon graph -edges -id <<vertex-id>> -direction <<direction>></p> -<p>Example: $FALCON_HOME/bin/falcon graph -edges -id 4 -direction both $FALCON_HOME/bin/falcon graph -edges -id 4 -direction inE</p></div> +<p>Usage: $FALCON_HOME/bin/falcon metadata -edges -id <<vertex-id>> -direction <<direction>></p> +<p>Example: $FALCON_HOME/bin/falcon metadata -edges -id 4 -direction both $FALCON_HOME/bin/falcon metadata -edges -id 4 -direction inE</p></div> <div class="section"> <h4>Edge<a name="Edge"></a></h4> <p>Get the edge with the specified id.</p> -<p>Usage: $FALCON_HOME/bin/falcon graph -edge -id <<id>></p> -<p>Example: $FALCON_HOME/bin/falcon graph -edge -id Q9n-Q-5g</p></div> +<p>Usage: $FALCON_HOME/bin/falcon metadata -edge -id <<id>></p> +<p>Example: $FALCON_HOME/bin/falcon metadata -edge -id Q9n-Q-5g</p></div> +<div class="section"> +<h3>Metadata Discovery Options<a name="Metadata_Discovery_Options"></a></h3></div> +<div class="section"> +<h4>List<a name="List"></a></h4> +<p>Lists of all dimensions of given type. If the user provides optional param cluster, only the dimensions related to the cluster are listed. Usage: $FALCON_HOME/bin/falcon metadata -list -type [cluster_entity|feed_entity|process_entity|user|colo|tags|groups|pipelines]</p> +<p>Optional Args : -cluster <<cluster name>></p> +<p>Example: $FALCON_HOME/bin/falcon metadata -list -type process_entity -cluster primary-cluster $FALCON_HOME/bin/falcon metadata -list -type tags</p></div> +<div class="section"> +<h4>Relations<a name="Relations"></a></h4> +<p>List all dimensions related to specified Dimension identified by dimension-type and dimension-name. Usage: $FALCON_HOME/bin/falcon metadata -relations -type [cluster_entity|feed_entity|process_entity|user|colo|tags|groups|pipelines] -name <<Dimension Name>></p> +<p>Example: $FALCON_HOME/bin/falcon metadata -relations -type process_entity -name sample-process</p></div> <div class="section"> <h3>Admin Options<a name="Admin_Options"></a></h3></div> <div class="section"> @@ -411,6 +438,14 @@ <div class="section"> <h4>Status<a name="Status"></a></h4> <p>Status returns the current state of Falcon (running or stopped). Usage: $FALCON_HOME/bin/falcon admin -status</p></div> +<div class="section"> +<h3>Recipe Options<a name="Recipe_Options"></a></h3></div> +<div class="section"> +<h4>Submit Recipe<a name="Submit_Recipe"></a></h4> +<p>Submit the specified recipe.</p> +<p>Usage: $FALCON_HOME/bin/falcon recipe -name <name> Name of the recipe. User should have defined <name>-template.xml and <name>.properties in the path specified by falcon.recipe.path in client.properties file. falcon.home path is used if its not specified in client.properties file. If its not specified in client.properties file and also if files cannot be found at falcon.home, Falcon CLI will fail.</p> +<p>Optional Args : -tool <recipeToolClassName> Falcon provides a base tool that recipes can override. If this option is not specified the default Recipe Tool RecipeTool defined is used. This option is required if user defines his own recipe tool class.</p> +<p>Example: $FALCON_HOME/bin/falcon recipe -name hdfs-replication</p></div> </div> </div>
Modified: falcon/site/FalconDocumentation.html URL: http://svn.apache.org/viewvc/falcon/site/FalconDocumentation.html?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/site/FalconDocumentation.html (original) +++ falcon/site/FalconDocumentation.html Wed Feb 18 10:55:56 2015 @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-01-11 + | Generated by Apache Maven Doxia at 2015-02-18 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20150111" /> + <meta name="Date-Revision-yyyymmdd" content="20150218" /> <meta http-equiv="Content-Language" content="en" /> <title>Falcon - Contents</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -245,7 +245,7 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-01-11</li> + <li id="publishDate" class="pull-right">Last Published: 2015-02-18</li> </ul> </div> @@ -269,10 +269,12 @@ <li><a href="#Updating_process_and_feed_definition">Updating process and feed definition</a></li> <li><a href="#Handling_late_input_data">Handling late input data</a></li> <li><a href="#Idempotency">Idempotency</a></li> -<li><a href="#Alerting_and_Monitoring">Alerting and Monitoring</a></li> <li><a href="#Falcon_EL_Expressions">Falcon EL Expressions</a></li> <li><a href="#Lineage">Lineage</a></li> -<li><a href="#Security">Security</a></li></ul></div> +<li><a href="#Security">Security</a></li> +<li><a href="#Recipes">Recipes</a></li> +<li><a href="#Monitoring">Monitoring</a></li> +<li><a href="#Backwards_Compatibility">Backwards Compatibility Instructions</a></li></ul></div> <div class="section"> <h3>Architecture<a name="Architecture"></a></h3></div> <div class="section"> @@ -298,10 +300,10 @@ <p>There are two basic components of Falcon set up. Falcon Prism and Falcon Server. As the name suggests Falcon Prism splits the request it gets to the Falcon Servers. More details below:</p></div> <div class="section"> <h4>Stand Alone Mode<a name="Stand_Alone_Mode"></a></h4> -<p>Stand alone mode is useful when the hadoop jobs and relevant data processing involves only one hadoop cluster. In this mode there is single Falcon server that contacts with oozie to schedule jobs on Hadoop. All the process / feed request like submit, schedule, suspend, kill are sent to this server only. For running in this mode one should use the falcon which has been built for standalone mode, or build using standalone option if using source code.</p></div> +<p>Stand alone mode is useful when the hadoop jobs and relevant data processing involves only one hadoop cluster. In this mode there is a single Falcon server that contacts Oozie to schedule jobs on Hadoop. All the process/feed requests like submit, schedule, suspend, kill etc. are sent to this server. For running falcon in this mode one should use the falcon which has been built using standalone option.</p></div> <div class="section"> <h4>Distributed Mode<a name="Distributed_Mode"></a></h4> -<p>Distributed mode is the mode which you might me using most of the time. This is for organisations which have multiple instances of hadoop clusters, and multiple workflow schedulers to handle them. Here we have 2 components: Prism and Server. Both Prism and server have there own setup (runtime and startup properties) and there config locations. In this mode Prism acts as a contact point for Falcon servers. Below are the requests that can be sent to prism and server in this mode:</p> +<p>Distributed mode is for multiple (colos) instances of hadoop clusters, and multiple workflow schedulers to handle them. In this mode falcon has 2 components: Prism and Server(s). Both Prism and servers have their own setup (runtime and startup properties) and their own config locations. In this mode Prism acts as a contact point for Falcon servers. While all commands are available through Prism, only read and instance api's are available through Server. Below are the requests that can be sent to each of these:</p> <p>Prism: submit, schedule, submitAndSchedule, Suspend, Resume, Kill, instance management Server: schedule, suspend, resume, instance management</p> <p>As observed above submit and kill are kept exclusively as Prism operations to keep all the config stores in sync and to support feature of idempotency. Request may also be sent from prism but directed to a specific server using the option "-colo" from CLI or append the same in web request, if using API.</p> <p>When a cluster is submitted it is by default sent to all the servers configured in the prism. When is feed is SUBMIT / SCHEDULED request is only sent to the servers specified in the feed / process definitions. Servers are mentioned in the feed / process via CLUSTER tags in xml definition.</p> @@ -389,13 +391,10 @@ catalog:$database-name:$table-name#parti <p>Delete operation on the entity removes any scheduled activity on the workflow engine, besides removing the entity from the falcon configuration store. Delete operation on an entity would only succeed if there are no dependent entities on the deleted entity.</p></div> <div class="section"> <h4>Update<a name="Update"></a></h4> -<p>Update operation allows an already submitted/scheduled entity to be updated. Cluster update is currently not allowed. Feed update can cause cascading update to all the processes already scheduled. Process update triggers update in falcon if entity is updated/the user specified workflow/lib is updated. The following set of actions are performed in Oozie to realize an update:</p> +<p>Update operation allows an already submitted/scheduled entity to be updated. Cluster update is currently not allowed. Feed update can cause cascading update to all the processes already scheduled. Process update triggers update in falcon if entity is updated. The following set of actions are performed in scheduler to realize an update:</p> <ul> -<li>Suspend the previously scheduled Oozie coordinator. This is to prevent any new action from being triggered.</li> -<li>Update the coordinator to set the end time to "now"</li> -<li>Resume the suspended coordinators</li> -<li>Schedule as per the new process/feed definition with the start time as "now"</li></ul> -<p>Update optionally takes effective time as a parameter which is used as the end time of previously scheduled coordinator. So, the updated configuration will be effective since the given timestamp.</p></div> +<li>Update the old scheduled entity to set the end time to "now"</li> +<li>Schedule as per the new process/feed definition with the start time as "now"</li></ul></div> <div class="section"> <h3>Instance Management actions<a name="Instance_Management_actions"></a></h3> <p>Instance Manager gives user the option to control individual instances of the process based on their instance start time (start time of that instance). Start time needs to be given in standard TZ format. Example: 01 Jan 2012 01:00 => 2012-01-01T01:00Z</p> @@ -442,7 +441,7 @@ catalog:$database-name:$table-name#parti <p>With the integration of Hive, Falcon also provides retention for tables in Hive catalog.</p></div> <div class="section"> <h4>Example:<a name="Example:"></a></h4> -<p>If retention period is 10 hours, and the policy kicks in at time 't', the data retained by system is essentially the one falling in between [t-10h,t]. Any data in the boundaries [-�,t-10h) and (t,�] is removed from the system.</p> +<p>If retention period is 10 hours, and the policy kicks in at time 't', the data retained by system is essentially the one in range [t-10h, t]. Any data before t-10h and after t is removed from the system.</p> <p>The 'action' attribute can attain values of DELETE/ARCHIVE. Based upon the tag value, the data eligible for removal is either deleted/archived.</p></div> <div class="section"> <h4>NOTE: Falcon 0.1/0.2 releases support Delete operation only<a name="NOTE:_Falcon_0.10.2_releases_support_Delete_operation_only"></a></h4></div> @@ -520,6 +519,26 @@ catalog:$database-name:$table-name#parti <ul> <li>The partition is not complete and hence not visible to users until all the data is committed on the secondary</li></ul>cluster, (no dirty reads)</div> <div class="section"> +<h4>Archival as Replication<a name="Archival_as_Replication"></a></h4> +<p>Falcon allows users to archive data from on-premice to cloud, either Azure WASB or S3. It uses the underlying replication for archiving data from source to target. The archival URI is specified as the overridden location for the target cluster.</p> +<p><b>Example:</b></p> +<div class="source"> +<pre> + <clusters> + <cluster name="on-premise-cluster" type="source"> + <validity start="2021-11-01T00:00Z" end="2021-12-31T00:00Z"/> + </cluster> + <cluster name="cloud-cluster" type="target"> + <validity start="2011-11-01T00:00Z" end="2011-12-31T00:00Z"/> + <locations> + <location type="data" + path="wasb://[email protected]/data/${YEAR}-${MONTH}-${DAY}-${HOUR}"/> + </locations> + </cluster> + </clusters> + +</pre></div></div> +<div class="section"> <h4>Relation between feed's retention limit and feed's late arrival cut off period:<a name="Relation_between_feeds_retention_limit_and_feeds_late_arrival_cut_off_period:"></a></h4> <p>For reasons that are obvious, Falcon has an external validation that ensures that the user always specifies the feed retention limit to be more than the feed's allowed late arrival period. If this rule is violated by the user, the feed submission call itself throws back an error.</p></div> <div class="section"> @@ -656,58 +675,6 @@ validity start="2009-01-01T00:00Z&q <h3>Idempotency<a name="Idempotency"></a></h3> <p>All the operations in Falcon are Idempotent. That is if you make same request to the falcon server / prism again you will get a SUCCESSFUL return if it was SUCCESSFUL in the first attempt. For example, you submit a new process / feed and get SUCCESSFUL message return. Now if you run the same command / api request on same entity you will again get a SUCCESSFUL message. Same is true for other operations like schedule, kill, suspend and resume. Idempotency also by takes care of the condition when request is sent through prism and fails on one or more servers. For example prism is configured to send request to 3 servers. First user sends a request to SUBMIT a process on all 3 of them, and receives a response SUCCESSFUL from all of them. Then due to some issue one of the servers goes down, and user send a request to schedule the submitted process. This time he will receive a response with PARTIAL status and a FAILURE message from the server that has gone down. If the users check he wi ll find the process would have been started and running on the 2 SUCCESSFUL servers. Now the issue with server is figured out and it is brought up. Sending the SCHEDULE request again through prism will result in a SUCCESSFUL response from prism as well as other three servers, but this time PROCESS will be SCHEDULED only on the server which had failed earlier and other two will keep running as before.</p></div> <div class="section"> -<h3>Alerting and Monitoring<a name="Alerting_and_Monitoring"></a></h3></div> -<div class="section"> -<h4>Alerting<a name="Alerting"></a></h4> -<p>Falcon provides monitoring of various events by capturing metrics of those events. The metric numbers can then be used to monitor performance and health of the Falcon system and the entire processing pipelines.</p> -<p>Users can view the logs of these events in the metric.log file, by default this file is created under ${user.dir}/logs/ directory. Users may also extend the Falcon monitoring framework to send events to systems like Mondemand/lwes.</p> -<p>The following events are captured by Falcon for logging the metrics:</p> -<ol style="list-style-type: decimal"> -<li>New cluster definitions posted to Falcon (success & failures)</li> -<li>New feed definition posted to Falcon (success & failures)</li> -<li>New process definition posted to Falcon (success & failures)</li> -<li>Process update events (success & failures)</li> -<li>Feed update events (success & failures)</li> -<li>Cluster update events (success & failures)</li> -<li>Process suspend events (success & failures)</li> -<li>Feed suspend events (success & failures)</li> -<li>Process resume events (success & failures)</li> -<li>Feed resume events (success & failures)</li> -<li>Process remove events (success & failures)</li> -<li>Feed remove events (success & failures)</li> -<li>Cluster remove events (success & failures)</li> -<li>Process instance kill events (success & failures)</li> -<li>Process instance re-run events (success & failures)</li> -<li>Process instance generation events</li> -<li>Process instance failure events</li> -<li>Process instance auto-retry events</li> -<li>Process instance retry exhaust events</li> -<li>Feed instance deletion event</li> -<li>Feed instance deletion failure event (no retries)</li> -<li>Feed instance replication event</li> -<li>Feed instance replication failure event</li> -<li>Feed instance replication auto-retry event</li> -<li>Feed instance replication retry exhaust event</li> -<li>Feed instance late arrival event</li> -<li>Feed instance post cut-off arrival event</li> -<li>Process re-run due to late feed event</li> -<li>Transaction rollback failed event</li></ol> -<p>The metric logged for an event has the following properties:</p> -<ol style="list-style-type: decimal"> -<li>Action - Name of the event.</li> -<li>Dimensions - A list of name/value pairs of various attributes for a given action.</li> -<li>Status- Status of an action FAILED/SUCCEEDED.</li> -<li>Time-taken - Time taken in nanoseconds for a given action.</li></ol> -<p>An example for an event logged for a submit of a new process definition:</p> -<p>2012-05-04 12:23:34,026 {Action:submit, Dimensions:{entityType=process}, Status: SUCCEEDED, Time-taken:97087000 ns}</p> -<p>Users may parse the metric.log or capture these events from custom monitoring frameworks and can plot various graphs or send alerts according to their requirements.</p></div> -<div class="section"> -<h4>Notifications<a name="Notifications"></a></h4> -<p>Falcon creates a JMS topic for every process/feed that is scheduled in Falcon. The implementation class and the broker url of the JMS engine are read from the dependent cluster's definition. Users may register consumers on the required topic to check the availability or status of feed instances.</p> -<p>For a given process that is scheduled, the name of the topic is same as the process name. Falcon sends a Map message for every feed produced by the instance of a process to the JMS topic. The JMS MapMessage sent to a topic has the following properties: entityName, feedNames, feedInstancePath, workflowId, runId, nominalTime, timeStamp, brokerUrl, brokerImplClass, entityType, operation, logFile, topicName, status, brokerTTL;</p> -<p>For a given feed that is scheduled, the name of the topic is same as the feed name. Falcon sends a map message for every feed instance that is deleted/archived/replicated depending upon the retention policy set in the feed definition. The JMS MapMessage sent to a topic has the following properties: entityName, feedNames, feedInstancePath, workflowId, runId, nominalTime, timeStamp, brokerUrl, brokerImplClass, entityType, operation, logFile, topicName, status, brokerTTL;</p> -<p>The JMS messages are automatically purged after a certain period (default 3 days) by the Falcon JMS house-keeping service.TTL (Time-to-live) for JMS message can be configured in the Falcon's startup.properties file.</p></div> -<div class="section"> <h3>Falcon EL Expressions<a name="Falcon_EL_Expressions"></a></h3> <p>Falcon expression language can be used in process definition for giving the start and end instance for various feeds.</p> <p>Before going into how to use falcon EL expressions it is necessary to understand what does instance and instance start time refer to with respect to Falcon.</p> @@ -805,15 +772,21 @@ validity start="2009-01-01T00:00Z&q <pre> config name: *.application.services config value: org.apache.falcon.metadata.MetadataMappingService -<verbatim> -Lineage is only captured for Process executions. A future release will capture lineage for -lifecycle policies such as replication and retention. - ---++ Security - -Security is detailed in [[Security][Security]]. -</pre></div></div> +</pre></div> +<p>Lineage is only captured for Process executions. A future release will capture lineage for lifecycle policies such as replication and retention.</p></div> +<div class="section"> +<h3>Security<a name="Security"></a></h3> +<p>Security is detailed in <a href="./Security.html">Security</a>.</p></div> +<div class="section"> +<h3>Recipes<a name="Recipes"></a></h3> +<p>Recipes is detailed in <a href="./Recipes.html">Recipes</a>.</p></div> +<div class="section"> +<h3>Monitoring<a name="Monitoring"></a></h3> +<p>Monitoring and Operationalizing Falcon is detailed in <a href="./Operability.html">Operability</a>.</p></div> +<div class="section"> +<h3>Backwards Compatibility<a name="Backwards_Compatibility"></a></h3> +<p>Backwards compatibility instructions are <a href="./Compatibility.html">detailed here.</a></p></div> </div> </div> Modified: falcon/site/HiveIntegration.html URL: http://svn.apache.org/viewvc/falcon/site/HiveIntegration.html?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/site/HiveIntegration.html (original) +++ falcon/site/HiveIntegration.html Wed Feb 18 10:55:56 2015 @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-01-11 + | Generated by Apache Maven Doxia at 2015-02-18 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20150111" /> + <meta name="Date-Revision-yyyymmdd" content="20150218" /> <meta http-equiv="Content-Language" content="en" /> <title>Falcon - Hive Integration</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -245,7 +245,7 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-01-11</li> + <li id="publishDate" class="pull-right">Last Published: 2015-02-18</li> </ul> </div> @@ -288,8 +288,8 @@ catalog.service.impl=org.apache.falcon.c <p>Hence, Falcon for Hive support needs Oozie 4.x.</p></div> <div class="section"> <h3>Oozie Shared Library setup<a name="Oozie_Shared_Library_setup"></a></h3> -<p>Falcon post Hive integration depends heavily on the <a class="externalLink" href="http://oozie.apache.org/docs/4.0.0/WorkflowFunctionalSpec.html#a17_HDFS_Share_Libraries_for_Workflow_Applications_since_Oozie_2.3";>shared library feature of Oozie</a>. Since the sheer number of jars for HCatalog, Pig and Hive are in the many 10s in numbers, its quite daunting to redistribute the dependent jars from Falcon.</p> -<p><a class="externalLink" href="http://oozie.apache.org/docs/4.0.0/DG_QuickStart.html#Oozie_Share_Lib_Installation";>This is a one time effort in Oozie setup and is quite straightforward.</a></p></div> +<p>Falcon post Hive integration depends heavily on the <a class="externalLink" href="http://oozie.apache.org/docs/4.0.1/WorkflowFunctionalSpec.html#a17_HDFS_Share_Libraries_for_Workflow_Applications_since_Oozie_2.3";>shared library feature of Oozie</a>. Since the sheer number of jars for HCatalog, Pig and Hive are in the many 10s in numbers, its quite daunting to redistribute the dependent jars from Falcon.</p> +<p><a class="externalLink" href="http://oozie.apache.org/docs/4.0.1/DG_QuickStart.html#Oozie_Share_Lib_Installation";>This is a one time effort in Oozie setup and is quite straightforward.</a></p></div> <div class="section"> <h3>Approach<a name="Approach"></a></h3></div> <div class="section"> @@ -442,7 +442,7 @@ org.apache.hadoop.hive.ql.parse.ImportSe <interface type="execute" endpoint="localhost:10300" version="1.1.1" /> <interface type="workflow" endpoint="http://localhost:11010/oozie/" - version="3.3.0" /> + version="4.0.1" /> <interface type="registry" endpoint="thrift://localhost:19083" version="0.11.0" /> <interface type="messaging" endpoint="tcp://localhost:61616?daemon=true" @@ -475,7 +475,7 @@ org.apache.hadoop.hive.ql.parse.ImportSe <interface type="execute" endpoint="localhost:20300" version="1.1.1" /> <interface type="workflow" endpoint="http://localhost:11020/oozie/" - version="3.3.0" /> + version="4.0.1" /> <interface type="registry" endpoint="thrift://localhost:29083" version="0.11.0" /> <interface type="messaging" endpoint="tcp://localhost:61616?daemon=true" Modified: falcon/site/InstallationSteps.html URL: http://svn.apache.org/viewvc/falcon/site/InstallationSteps.html?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/site/InstallationSteps.html (original) +++ falcon/site/InstallationSteps.html Wed Feb 18 10:55:56 2015 @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-01-11 + | Generated by Apache Maven Doxia at 2015-02-18 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20150111" /> + <meta name="Date-Revision-yyyymmdd" content="20150218" /> <meta http-equiv="Content-Language" content="en" /> <title>Falcon - Building & Installing Falcon</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -245,7 +245,7 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-01-11</li> + <li id="publishDate" class="pull-right">Last Published: 2015-02-18</li> </ul> </div> @@ -260,15 +260,22 @@ <h4>Building Falcon<a name="Building_Falcon"></a></h4> <div class="source"> <pre> +You would need the following installed to build Falcon + +* JDK 1.7 +* Maven 3.x + git clone https://git-wip-us.apache.org/repos/asf/falcon.git falcon cd falcon -export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m" && mvn clean install [For hadoop 1] -export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m" && mvn clean install -Phadoop-2 [For hadoop 2] +export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m -noverify" && mvn clean install [optionally -Dhadoop.version=<<hadoop.version>> can be appended to build for a specific version of hadoop] -[optionally -Doozie.version=<<oozie version>> can be appended to build with a specific version of oozie. Oozie versions >= 3.oozie-3.2.0-incubating are supported] +*Note:* Falcon drops support for Hadoop-1 and only supports Hadoop-2 from Falcon 0.6 onwards +[optionally -Doozie.version=<<oozie version>> can be appended to build with a specific version of oozie. +Oozie versions >= 4 are supported] +Falcon build with JDK 1.7 using -noverify option </pre></div> @@ -277,12 +284,11 @@ export MAVEN_OPTS="-Xmx1024m -XX:Ma <div class="source"> <pre> -mvn clean assembly:assembly -DskipTests -DskipCheck=true [For hadoop 1] -mvn clean assembly:assembly -DskipTests -DskipCheck=true -P hadoop-2 [For hadoop 2] +mvn clean assembly:assembly -DskipTests -DskipCheck=true </pre></div> -<p>Tar can be found in {project dir}/target/falcon-${project.version}-bin.tar.gz</p> +<p>Tar can be found in {project dir}/target/apache-falcon-${project.version}-bin.tar.gz</p> <p>Tar is structured as follows</p> <div class="source"> <pre> @@ -318,12 +324,11 @@ mvn clean assembly:assembly -DskipTests <div class="source"> <pre> -mvn clean assembly:assembly -DskipTests -DskipCheck=true -Pdistributed,hadoop-1 [For hadoop 1] -mvn clean assembly:assembly -DskipTests -DskipCheck=true -Pdistributed,hadoop-2 [For hadoop 2] +mvn clean assembly:assembly -DskipTests -DskipCheck=true -Pdistributed,hadoop-2 </pre></div> -<p>Tar can be found in {project dir}/target/falcon-distributed-${project.version}-server.tar.gz</p> +<p>Tar can be found in {project dir}/target/apache-falcon-distributed-${project.version}-server.tar.gz</p> <p>Tar is structured as follows</p> <div class="source"> <pre> @@ -412,15 +417,27 @@ cd falcon-distributed-${project.version} #export FALCON_EXPANDED_WEBAPP_DIR= </pre></div> +<p><b>NOTE for Mac OS users</b></p> +<div class="source"> +<pre> +If you are using a Mac OS, you will need to configure the FALCON_SERVER_OPTS (explained above). + +In {package dir}/conf/falcon-env.sh uncomment the following line +#export FALCON_SERVER_OPTS= + +and change it to look as below +export FALCON_SERVER_OPTS="-Djava.awt.headless=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc=" + +</pre></div> <p><b>Starting Falcon Server</b></p> <div class="source"> <pre> bin/falcon-start [-port <port>] </pre></div> -<p>By default, * falcon server starts at port 15443 (https) by default . To change the port, use -port option</p> +<p>By default, * If falcon.enableTLS is set to true explicitly or not set at all, falcon starts at port 15443 on <a class="externalLink" href="https://";>https://</a> by default. * If falcon.enableTLS is set to false explicitly, falcon starts at port 15000 on <a class="externalLink" href="http://.";>http://.</a> * To change the port, use -port option.</p> <ul> -<li>falcon.enableTLS can be set to true or false explicitly to enable SSL, if not port that end with 443 will automatically put falcon on <a class="externalLink" href="https://";>https://</a></li></ul>* falcon server starts embedded active mq. To control this behaviour, set the following system properties using -D option in environment variable FALCON_OPTS: +<li>If falcon.enableTLS is not set explicitly, port that ends with 443 will automatically put falcon on <a class="externalLink" href="https://.";>https://.</a> Any other port will put falcon on <a class="externalLink" href="http://.";>http://.</a></li></ul>* falcon server starts embedded active mq. To control this behaviour, set the following system properties using -D option in environment variable FALCON_OPTS: <ul> <li>falcon.embeddedmq=<true/false> - Should server start embedded active mq, default true</li> <li>falcon.embeddedmq.port=<port> - Port for embedded active mq, default 61616</li> @@ -450,6 +467,18 @@ bin/falcon help </pre></div> <p><b>Dashboard</b></p> <p>Once falcon / prism is started, you can view the status of falcon entities using the Web-based dashboard. The web UI works in both distributed and embedded mode. You can open your browser at the corresponding port to use the web UI.</p> +<p>Falcon dashboard makes the REST api calls as user "falcon-dashboard". If this user does not exist on your falcon and oozie servers, please create the user.</p> +<div class="source"> +<pre> +## create user. +[root@falconhost ~] useradd -U -m falcon-dashboard -G users + +## verify user is created with membership in correct groups. +[root@falconhost ~] groups falcon-dashboard +falcon-dashboard : falcon-dashboard users +[root@falconhost ~] + +</pre></div> <p><b>Stopping Falcon Server</b></p> <div class="source"> <pre> @@ -469,9 +498,10 @@ bin/prism-stop cd <<project home>> src/bin/package.sh <<hadoop-version>> <<oozie-version>> ->> ex. src/bin/package.sh 1.1.2 3.1.3-incubating or src/bin/package.sh 0.20.2-cdh3u5 4.0.0 ->> Falcon package is available in <<falcon home>>/target/falcon-<<version>>-bin.tar.gz ->> Oozie package is available in <<falcon home>>/target/oozie-3.3.2-distro.tar.gz +>> ex. src/bin/package.sh 1.1.2 4.0.1 or src/bin/package.sh 0.20.2-cdh3u5 4.0.1 +>> ex. src/bin/package.sh 2.5.0 4.0.0 +>> Falcon package is available in <<falcon home>>/target/apache-falcon-<<version>>-bin.tar.gz +>> Oozie package is available in <<falcon home>>/target/oozie-4.0.1-distro.tar.gz </pre></div></div> <div class="section"> @@ -481,7 +511,7 @@ src/bin/package.sh <<hadoop-versio bin/falcon-start </pre></div> -<p>Make sure the hadoop and oozie endpoints are according to your setup in examples/entity/filesystem/standalone-cluster.xml</p> +<p>Make sure the hadoop and oozie endpoints are according to your setup in examples/entity/filesystem/standalone-cluster.xml The cluster locations,staging and working dirs, MUST be created prior to submitting a cluster entity to Falcon. <b>staging</b> must have 777 permissions and the parent dirs must have execute permissions <b>working</b> must have 755 permissions and the parent dirs must have execute permissions</p> <div class="source"> <pre> bin/falcon entity -submit -type cluster -file examples/entity/filesystem/standalone-cluster.xml Added: falcon/site/MigrationInstructions.html URL: http://svn.apache.org/viewvc/falcon/site/MigrationInstructions.html?rev=1660589&view=auto ============================================================================== --- falcon/site/MigrationInstructions.html (added) +++ falcon/site/MigrationInstructions.html Wed Feb 18 10:55:56 2015 @@ -0,0 +1,293 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2015-02-18 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20150218" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Falcon - Migration Instructions</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarEnabled"> + + + + + + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Falcon <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="About">About</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/FALCON/Index"; title="Wiki">Wiki</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/FALCON/News"; title="News">News</a> +</li> + + <li> <a href="https://git-wip-us.apache.org/repos/asf/falcon.git"; title="Git">Git</a> +</li> + + <li> <a href="https://svn.apache.org/repos/asf/falcon"; title="Svn">Svn</a> +</li> + + <li> <a href="https://issues.apache.org/jira/browse/FALCON"; title="Jira">Jira</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/FALCON/PoweredBy"; title="Powered by">Powered by</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/FALCON/Acknowledgements"; title="Acknowledgements">Acknowledgements</a> +</li> + + <li> <a href="http://blogs.apache.org/falcon/"; title="Blog">Blog</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="project-info.html" title="Summary">Summary</a> +</li> + + <li> <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="http://webchat.freenode.net?channels=apachefalcon&uio=d4"; title="IRC">IRC</a> +</li> + + <li> <a href="team-list.html" title="Team">Team</a> +</li> + + <li> <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="license.html" title="License">License</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/dyn/closer.cgi/falcon/0.6-incubating"; title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/falcon/0.5-incubating"; title="0.5-incubating">0.5-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/falcon/0.4-incubating"; title="0.4-incubating">0.4-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/falcon/0.3-incubating"; title="0.3-incubating">0.3-incubating</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/FALCON/Roadmap"; title="Coming soon">Coming soon</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="0.6-incubating/index.html" title="current">current</a> +</li> + + <li> <a href="0.6-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="0.5-incubating/index.html" title="0.5-incubating">0.5-incubating</a> +</li> + + <li> <a href="0.4-incubating/index.html" title="0.4-incubating">0.4-incubating</a> +</li> + + <li> <a href="0.3-incubating/index.html" title="0.3-incubating">0.3-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Resources <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="slides/falcon-overview.html" title="Overview">Overview</a> +</li> + + <li> <a href="slides/falcon-user-guide.html" title="User Guide">User Guide</a> +</li> + + <li> <a href="GettingStarted.html" title="Getting Started">Getting Started</a> +</li> + + <li> <a href="FalconDocumentation.html" title="Architecture">Architecture</a> +</li> + + <li> <a href="InstallationSteps.html" title="Installation">Installation</a> +</li> + + <li> <a href="OnBoarding.html" title="On Boarding">On Boarding</a> +</li> + + <li> <a href="EntitySpecification.html" title="Entity Specification">Entity Specification</a> +</li> + + <li> <a href="FalconCLI.html" title="Client (Falcon CLI)">Client (Falcon CLI)</a> +</li> + + <li> <a href="restapi/ResourceList.html" title="Rest API">Rest API</a> +</li> + + <li> <a href="HiveIntegration.html" title="Hive Integration">Hive Integration</a> +</li> + + <li> <a href="Security.html" title="Security">Security</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> +</li> + + <li> <a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a> +</li> + + <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a> +</li> + + <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> +</li> + </ul> + </li> + </ul> + + + + + + + <iframe src="http://www.facebook.com/plugins/like.php?href=http://falcon.apache.org/falcon-website-general&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:80px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + + <script type="text/javascript" src="https://apis.google.com/js/plusone.js";></script> + + <ul class="nav pull-right"><li style="margin-top: 10px;"> + + <div class="g-plusone" data-href="http://falcon.apache.org/falcon-website-general"; data-size="medium" width="60px" align="right" ></div> + + </li></ul> + + + </div> + + </div> + </div> + </div> + + <div class="container"> + <div id="banner"> + <div class="pull-left"> + <a href="../index.html" id="bannerLeft"> + <img src="images/falcon-logo.png" alt="Falcon" width="200px" height="45px"/> + </a> + </div> + <div class="pull-right"> <a href="http://www.apache.org"; id="bannerRight"> + <img src="images/apache-feather-tm.gif" alt="Falcon" height="45px"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="index.html" title="Falcon"> + Falcon</a> + </li> + <li class="divider ">/</li> + <li class="">Migration Instructions</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2015-02-18</li> + + </ul> + </div> + + + + <div id="bodyColumn" > + + <div class="section"> +<h2>Migration Instructions<a name="Migration_Instructions"></a></h2></div> +<div class="section"> +<h3>Migrate from 0.5-incubating to 0.6-incubating<a name="Migrate_from_0.5-incubating_to_0.6-incubating"></a></h3> +<p>This is a placeholder wiki for migration instructions from falcon 0.5-incubating to 0.6-incubating.</p></div> +<div class="section"> +<h4>Update Entities<a name="Update_Entities"></a></h4></div> +<div class="section"> +<h4>Change cluster dir permissions<a name="Change_cluster_dir_permissions"></a></h4></div> +<div class="section"> +<h4>Enable/Disable TLS<a name="EnableDisable_TLS"></a></h4></div> +<div class="section"> +<h4>Authorization<a name="Authorization"></a></h4></div> + </div> + </div> + + <hr/> + + <footer> + <div class="container"> + <div class="row span12">Copyright © 2013-2015 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + <p id="poweredBy" class="pull-right"> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </p> + + </div> + </footer> + </body> +</html> Modified: falcon/site/OnBoarding.html URL: http://svn.apache.org/viewvc/falcon/site/OnBoarding.html?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/site/OnBoarding.html (original) +++ falcon/site/OnBoarding.html Wed Feb 18 10:55:56 2015 @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-01-11 + | Generated by Apache Maven Doxia at 2015-02-18 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20150111" /> + <meta name="Date-Revision-yyyymmdd" content="20150218" /> <meta http-equiv="Content-Language" content="en" /> <title>Falcon - Contents</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -245,7 +245,7 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-01-11</li> + <li id="publishDate" class="pull-right">Last Published: 2015-02-18</li> </ul> </div> @@ -276,7 +276,7 @@ <h4>Sample Pipeline<a name="Sample_Pipeline"></a></h4></div> <div class="section"> <h5>Cluster <a name="Cluster"></a></h5> -<p>Cluster definition that contains end points for name node, job tracker, oozie and jms server:</p> +<p>Cluster definition that contains end points for name node, job tracker, oozie and jms server: The cluster locations MUST be created prior to submitting a cluster entity to Falcon. <b>staging</b> must have 777 permissions and the parent dirs must have execute permissions <b>working</b> must have 755 permissions and the parent dirs must have execute permissions</p> <div class="source"> <pre> <?xml version="1.0"?> @@ -286,13 +286,13 @@ <cluster colo="ua2" description="" name="corp" xmlns="uri:falcon:cluster:0.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <interfaces> - <interface type="readonly" endpoint="hftp://name-node.com:50070" version="0.20.2-cdh3u0" /> + <interface type="readonly" endpoint="hftp://name-node.com:50070" version="2.5.0" /> - <interface type="write" endpoint="hdfs://name-node.com:54310" version="0.20.2-cdh3u0" /> + <interface type="write" endpoint="hdfs://name-node.com:54310" version="2.5.0" /> - <interface type="execute" endpoint="job-tracker:54311" version="0.20.2-cdh3u0" /> + <interface type="execute" endpoint="job-tracker:54311" version="2.5.0" /> - <interface type="workflow" endpoint="http://oozie.com:11000/oozie/" version="3.1.4" /> + <interface type="workflow" endpoint="http://oozie.com:11000/oozie/" version="4.0.1" /> <interface type="messaging" endpoint="tcp://jms-server.com:61616?daemon=true" version="5.1.6" /> </interfaces> Added: falcon/site/Operability.html URL: http://svn.apache.org/viewvc/falcon/site/Operability.html?rev=1660589&view=auto ============================================================================== --- falcon/site/Operability.html (added) +++ falcon/site/Operability.html Wed Feb 18 10:55:56 2015 @@ -0,0 +1,343 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2015-02-18 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20150218" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Falcon - Operationalizing Falcon</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarEnabled"> + + + + + + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Falcon <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="About">About</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/FALCON/Index"; title="Wiki">Wiki</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/FALCON/News"; title="News">News</a> +</li> + + <li> <a href="https://git-wip-us.apache.org/repos/asf/falcon.git"; title="Git">Git</a> +</li> + + <li> <a href="https://svn.apache.org/repos/asf/falcon"; title="Svn">Svn</a> +</li> + + <li> <a href="https://issues.apache.org/jira/browse/FALCON"; title="Jira">Jira</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/FALCON/PoweredBy"; title="Powered by">Powered by</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/FALCON/Acknowledgements"; title="Acknowledgements">Acknowledgements</a> +</li> + + <li> <a href="http://blogs.apache.org/falcon/"; title="Blog">Blog</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="project-info.html" title="Summary">Summary</a> +</li> + + <li> <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="http://webchat.freenode.net?channels=apachefalcon&uio=d4"; title="IRC">IRC</a> +</li> + + <li> <a href="team-list.html" title="Team">Team</a> +</li> + + <li> <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="license.html" title="License">License</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/dyn/closer.cgi/falcon/0.6-incubating"; title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/falcon/0.5-incubating"; title="0.5-incubating">0.5-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/falcon/0.4-incubating"; title="0.4-incubating">0.4-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/falcon/0.3-incubating"; title="0.3-incubating">0.3-incubating</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/FALCON/Roadmap"; title="Coming soon">Coming soon</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="0.6-incubating/index.html" title="current">current</a> +</li> + + <li> <a href="0.6-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="0.5-incubating/index.html" title="0.5-incubating">0.5-incubating</a> +</li> + + <li> <a href="0.4-incubating/index.html" title="0.4-incubating">0.4-incubating</a> +</li> + + <li> <a href="0.3-incubating/index.html" title="0.3-incubating">0.3-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Resources <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="slides/falcon-overview.html" title="Overview">Overview</a> +</li> + + <li> <a href="slides/falcon-user-guide.html" title="User Guide">User Guide</a> +</li> + + <li> <a href="GettingStarted.html" title="Getting Started">Getting Started</a> +</li> + + <li> <a href="FalconDocumentation.html" title="Architecture">Architecture</a> +</li> + + <li> <a href="InstallationSteps.html" title="Installation">Installation</a> +</li> + + <li> <a href="OnBoarding.html" title="On Boarding">On Boarding</a> +</li> + + <li> <a href="EntitySpecification.html" title="Entity Specification">Entity Specification</a> +</li> + + <li> <a href="FalconCLI.html" title="Client (Falcon CLI)">Client (Falcon CLI)</a> +</li> + + <li> <a href="restapi/ResourceList.html" title="Rest API">Rest API</a> +</li> + + <li> <a href="HiveIntegration.html" title="Hive Integration">Hive Integration</a> +</li> + + <li> <a href="Security.html" title="Security">Security</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> +</li> + + <li> <a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a> +</li> + + <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a> +</li> + + <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> +</li> + </ul> + </li> + </ul> + + + + + + + <iframe src="http://www.facebook.com/plugins/like.php?href=http://falcon.apache.org/falcon-website-general&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:80px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + + <script type="text/javascript" src="https://apis.google.com/js/plusone.js";></script> + + <ul class="nav pull-right"><li style="margin-top: 10px;"> + + <div class="g-plusone" data-href="http://falcon.apache.org/falcon-website-general"; data-size="medium" width="60px" align="right" ></div> + + </li></ul> + + + </div> + + </div> + </div> + </div> + + <div class="container"> + <div id="banner"> + <div class="pull-left"> + <a href="../index.html" id="bannerLeft"> + <img src="images/falcon-logo.png" alt="Falcon" width="200px" height="45px"/> + </a> + </div> + <div class="pull-right"> <a href="http://www.apache.org"; id="bannerRight"> + <img src="images/apache-feather-tm.gif" alt="Falcon" height="45px"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="index.html" title="Falcon"> + Falcon</a> + </li> + <li class="divider ">/</li> + <li class="">Operationalizing Falcon</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2015-02-18</li> + + </ul> + </div> + + + + <div id="bodyColumn" > + + <div class="section"> +<h2>Operationalizing Falcon<a name="Operationalizing_Falcon"></a></h2></div> +<div class="section"> +<h3>Overview<a name="Overview"></a></h3> +<p>Apache Falcon provides various tools to operationalize Falcon consisting of Alerts for unrecoverable errors, Audits of user actions, Metrics, and Notifications. They are detailed below.</p></div> +<div class="section"> +<h3>Monitoring<a name="Monitoring"></a></h3> +<p>Falcon provides monitoring of various events by capturing metrics of those events. The metric numbers can then be used to monitor performance and health of the Falcon system and the entire processing pipelines.</p> +<p>Users can view the logs of these events in the metric.log file, by default this file is created under ${user.dir}/logs/ directory. Users may also extend the Falcon monitoring framework to send events to systems like Mondemand/lwes by implementingorg.apache.falcon.plugin.MonitoringPlugin interface.</p> +<p>The following events are captured by Falcon for logging the metrics:</p> +<ol style="list-style-type: decimal"> +<li>New cluster definitions posted to Falcon (success & failures)</li> +<li>New feed definition posted to Falcon (success & failures)</li> +<li>New process definition posted to Falcon (success & failures)</li> +<li>Process update events (success & failures)</li> +<li>Feed update events (success & failures)</li> +<li>Cluster update events (success & failures)</li> +<li>Process suspend events (success & failures)</li> +<li>Feed suspend events (success & failures)</li> +<li>Process resume events (success & failures)</li> +<li>Feed resume events (success & failures)</li> +<li>Process remove events (success & failures)</li> +<li>Feed remove events (success & failures)</li> +<li>Cluster remove events (success & failures)</li> +<li>Process instance kill events (success & failures)</li> +<li>Process instance re-run events (success & failures)</li> +<li>Process instance generation events</li> +<li>Process instance failure events</li> +<li>Process instance auto-retry events</li> +<li>Process instance retry exhaust events</li> +<li>Feed instance deletion event</li> +<li>Feed instance deletion failure event (no retries)</li> +<li>Feed instance replication event</li> +<li>Feed instance replication failure event</li> +<li>Feed instance replication auto-retry event</li> +<li>Feed instance replication retry exhaust event</li> +<li>Feed instance late arrival event</li> +<li>Feed instance post cut-off arrival event</li> +<li>Process re-run due to late feed event</li> +<li>Transaction rollback failed event</li></ol> +<p>The metric logged for an event has the following properties:</p> +<ol style="list-style-type: decimal"> +<li>Action - Name of the event.</li> +<li>Dimensions - A list of name/value pairs of various attributes for a given action.</li> +<li>Status- Status of an action FAILED/SUCCEEDED.</li> +<li>Time-taken - Time taken in nanoseconds for a given action.</li></ol> +<p>An example for an event logged for a submit of a new process definition:</p> +<p>2012-05-04 12:23:34,026 {Action:submit, Dimensions:{entityType=process}, Status: SUCCEEDED, Time-taken:97087000 ns}</p> +<p>Users may parse the metric.log or capture these events from custom monitoring frameworks and can plot various graphs or send alerts according to their requirements.</p></div> +<div class="section"> +<h3>Notifications<a name="Notifications"></a></h3> +<p>Falcon creates a JMS topic for every process/feed that is scheduled in Falcon. The implementation class and the broker url of the JMS engine are read from the dependent cluster's definition. Users may register consumers on the required topic to check the availability or status of feed instances.</p> +<p>For a given process that is scheduled, the name of the topic is same as the process name. Falcon sends a Map message for every feed produced by the instance of a process to the JMS topic. The JMS MapMessage sent to a topic has the following properties: entityName, feedNames, feedInstancePath, workflowId, runId, nominalTime, timeStamp, brokerUrl, brokerImplClass, entityType, operation, logFile, topicName, status, brokerTTL;</p> +<p>For a given feed that is scheduled, the name of the topic is same as the feed name. Falcon sends a map message for every feed instance that is deleted/archived/replicated depending upon the retention policy set in the feed definition. The JMS MapMessage sent to a topic has the following properties: entityName, feedNames, feedInstancePath, workflowId, runId, nominalTime, timeStamp, brokerUrl, brokerImplClass, entityType, operation, logFile, topicName, status, brokerTTL;</p> +<p>The JMS messages are automatically purged after a certain period (default 3 days) by the Falcon JMS house-keeping service.TTL (Time-to-live) for JMS message can be configured in the Falcon's startup.properties file.</p></div> +<div class="section"> +<h3>Alerts<a name="Alerts"></a></h3> +<p>Falcon generates alerts for unrecoverable errors into a log file by default. Users can view these alerts in the alerts.log file, by default this file is created under ${user.dir}/logs/ directory.</p> +<p>Users may also extend the Falcon Alerting plugin to send events to systems like Nagios, etc. by extending org.apache.falcon.plugin.AlertingPlugin interface.</p></div> +<div class="section"> +<h3>Audits<a name="Audits"></a></h3> +<p>Falcon audits all user activity and captures them into a log file by default. Users can view these audits in the audit.log file, by default this file is created under ${user.dir}/logs/ directory.</p> +<p>Users may also extend the Falcon Audit plugin to send audits to systems like Apache Argus, etc. by extending org.apache.falcon.plugin.AuditingPlugin interface.</p></div> + </div> + </div> + + <hr/> + + <footer> + <div class="container"> + <div class="row span12">Copyright © 2013-2015 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + <p id="poweredBy" class="pull-right"> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </p> + + </div> + </footer> + </body> +</html> Modified: falcon/site/Security.html URL: http://svn.apache.org/viewvc/falcon/site/Security.html?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/site/Security.html (original) +++ falcon/site/Security.html Wed Feb 18 10:55:56 2015 @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2015-01-11 + | Generated by Apache Maven Doxia at 2015-02-18 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20150111" /> + <meta name="Date-Revision-yyyymmdd" content="20150218" /> <meta http-equiv="Content-Language" content="en" /> <title>Falcon - Securing Falcon</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> @@ -245,7 +245,7 @@ - <li id="publishDate" class="pull-right">Last Published: 2015-01-11</li> + <li id="publishDate" class="pull-right">Last Published: 2015-02-18</li> </ul> </div> @@ -289,19 +289,40 @@ <div class="section"> <h4>Super-User<a name="Super-User"></a></h4> <p>The super-user is the user with the same identity as falcon process itself. Loosely, if you started the falcon, then you are the super-user. The super-user can do anything in that permissions checks never fail for the super-user. There is no persistent notion of who was the super-user; when the falcon is started the process identity determines who is the super-user for now. The Falcon super-user does not have to be the super-user of the falcon host, nor is it necessary that all clusters have the same super-user. Also, an experimenter running Falcon on a personal workstation, conveniently becomes that installation's super-user without any configuration.</p> -<p>Falcon also allows users to configure a super user group and allows users belonging to this group to be a super user.</p></div> +<p>Falcon also allows users to configure a super user group and allows users belonging to this group to be a super user.</p> +<p>ACL owner and group must be valid even if the authenticated user is a super-user.</p></div> <div class="section"> <h4>Group Memberships<a name="Group_Memberships"></a></h4> <p>Once a user has been authenticated and a username has been determined, the list of groups is determined by a group mapping service, configured by the hadoop.security.group.mapping property in Hadoop. The default implementation, org.apache.hadoop.security.ShellBasedUnixGroupsMapping, will shell out to the Unix bash -c groups command to resolve a list of groups for a user.</p> -<p>Note that Falcon stores the user and group of an Entity as strings; there is no conversion from user and group identity numbers as is conventional in Unix.</p></div> +<p>Note that Falcon stores the user and group of an Entity as strings; there is no conversion from user and group identity numbers as is conventional in Unix.</p> +<p>The only limitation is that a user cannot add a group in ACL that he does not belong to.</p></div> <div class="section"> <h4>Authorization Provider<a name="Authorization_Provider"></a></h4> <p>Falcon provides a plugin-able provider interface for Authorization. It also ships with a default implementation that enforces the following authorization policy.</p></div> <div class="section"> <h5>Entity and Instance Management Operations Policy<a name="Entity_and_Instance_Management_Operations_Policy"></a></h5> -<p>* All Entity and Instance operations are authorized for users who created them, Owners and users with group memberships * Reference to entities with in a feed or process is allowed with out enforcing permissions Any Feed or Process can refer to a Cluster entity not owned by the Feed or Process owner Any Process can refer to a Feed entity not owned by the Process owner</p> +<p></p> +<ul> +<li>All Entity and Instance operations are authorized for users who created them, Owners and users with group memberships</li> +<li>Reference to entities with in a feed or process is allowed with out enforcing permissions</li></ul> +<p>Any Feed or Process can refer to a Cluster entity not owned by the Feed or Process owner. Any Process can refer to a Feed entity not owned by the Process owner</p> <p>The authorization is enforced in the following way:</p> -<p>if admin resource, if authenticated user name matches the admin users configuration Else if groups of the authenticated user matches the admin groups configuration Else authorization exception is thrown Else if entities or instance resource if the authenticated user matches the owner in ACL for the entity Else if the groups of the authenticated user matches the group in ACL for the entity Else authorization exception is thrown Else if lineage resource All have read-only permissions, reason being folks should be able to examine the dependency and allow reuse</p> +<p></p> +<ul> +<li>if admin resource, +<ul> +<li>If authenticated user name matches the admin users configuration</li> +<li>Else if groups of the authenticated user matches the admin groups configuration</li> +<li>Else authorization exception is thrown</li></ul></li> +<li>Else if entities or instance resource +<ul> +<li>If the authenticated user matches the owner in ACL for the entity</li> +<li>Else if the groups of the authenticated user matches the group in ACL for the entity</li> +<li>Else authorization exception is thrown</li></ul></li> +<li>Else if lineage resource +<ul> +<li>All have read-only permissions, reason being folks should be able to examine the dependency and allow reuse</li></ul></li></ul> +<p>To authenticate user for REST api calls, user should append "user.name=<username>" to the query.</p> <p><b>operations on Entity Resource</b></p> <p></p> <table border="0" class="table table-striped"> @@ -414,7 +435,7 @@ <td>Admin User/Group</td></tr></table></div> <div class="section"> <h5>Lineage Resource Policy<a name="Lineage_Resource_Policy"></a></h5> -<p>Lineage is read-only and hence all users can look at lineage for their respective entities.</p></div> +<p>Lineage is read-only and hence all users can look at lineage for their respective entities. <b>Note:</b> This gap will be fixed in a later release.</p></div> <div class="section"> <h3>Authentication Configuration<a name="Authentication_Configuration"></a></h3> <p>Following is the Server Side Configuration Setup for Authentication.</p></div> @@ -472,6 +493,9 @@ # Comma separated list of black listed users *.falcon.http.authentication.blacklisted.users= +# Increase Jetty request buffer size to accommodate the generated Kerberos token +*.falcon.jetty.request.buffer.size=16192 + </pre></div></div> <div class="section"> <h4>Pseudo/Simple Configuration<a name="PseudoSimple_Configuration"></a></h4> @@ -572,22 +596,32 @@ <td>Configuration Store</td> <td>${config.store.uri}</td> <td>falcon</td> -<td>750</td></tr> +<td>700</td></tr> <tr class="a"> -<td>Oozie coord/bundle XMLs</td> -<td>${cluster.staging-location}/workflows/{entity}/{entity-name}</td> +<td>Cluster Staging Location</td> +<td>${cluster.staging-location}</td> <td>falcon</td> -<td>644</td></tr> +<td>777</td></tr> <tr class="b"> +<td>Cluster Working Location</td> +<td>${cluster.working-location}</td> +<td>falcon</td> +<td>755</td></tr> +<tr class="a"> <td>Shared libs</td> <td>{cluster.working}/{lib,libext}</td> <td>falcon</td> <td>755</td></tr> +<tr class="b"> +<td>Oozie coord/bundle XMLs</td> +<td>${cluster.staging-location}/workflows/{entity}/{entity-name}</td> +<td>$user</td> +<td>cluster umask</td></tr> <tr class="a"> <td>App logs</td> <td>${cluster.staging-location}/workflows/{entity}/{entity-name}/logs</td> -<td>falcon</td> -<td>777</td></tr></table></div> +<td>$user</td> +<td>cluster umask</td></tr></table><b>Note:</b> Please note that the cluster staging and working locations MUST be created prior to submitting a cluster entity to Falcon. Also, note that the the parent dirs must have execute permissions.</div> <div class="section"> <h3>Backwards compatibility<a name="Backwards_compatibility"></a></h3></div> <div class="section"> @@ -606,13 +640,11 @@ <p>The blacklist users used to have the following super users: hdfs, mapreduce, oozie, and falcon. The list is externalized from code into Startup.properties file and is empty now and needs to be configured specifically in the file.</p></div> <div class="section"> <h4>Falcon Dashboard<a name="Falcon_Dashboard"></a></h4> -<p>The dashboard assumes an anonymous user in Pseudo/Simple method and hence anonymous users must be enabled for it to work.</p> -<div class="source"> -<pre> -# Indicates if anonymous requests are allowed when using 'simple' authentication. -*.falcon.http.authentication.simple.anonymous.allowed=true - -</pre></div> +<p>To initialize the current user for dashboard, user should append query param "user.name=<username>" to the REST api call.</p> +<p>If dashboard user wishes to change the current user, they should do the following.</p> +<ul> +<li>delete the hadoop.auth cookie from browser cache.</li> +<li>append query param "user.name=<new_user>" to the next REST API call.</li></ul> <p>In Kerberos method, the browser must support HTTP Kerberos SPNEGO.</p></div> <div class="section"> <h3>Known Limitations<a name="Known_Limitations"></a></h3>
