http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/content/0.11/OnBoarding.html ---------------------------------------------------------------------- diff --git a/content/0.11/OnBoarding.html b/content/0.11/OnBoarding.html new file mode 100644 index 0000000..2c594b3 --- /dev/null +++ b/content/0.11/OnBoarding.html @@ -0,0 +1,368 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2018-03-12 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20180312" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Falcon - Contents</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container"> + <div id="banner"> + <div class="pull-left"> + <div id="bannerLeft"> + <img src="images/falcon-logo.png" alt="Apache Falcon" width="200px" height="45px"/> + </div> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="index.html" title="Falcon"> + Falcon</a> + </li> + <li class="divider ">/</li> + <li class="">Contents</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2018-03-12</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.11</li> + + </ul> + </div> + + + + <div id="bodyColumn" > + + <div class="section"> +<h3>Contents<a name="Contents"></a></h3> +<p></p> +<ul> +<li><a href="#Onboarding Steps">Onboarding Steps</a></li> +<li><a href="#Sample Pipeline">Sample Pipeline</a></li> +<li><a href="./HiveIntegration.html">Hive Examples</a></li></ul></div> +<div class="section"> +<h4>Onboarding Steps<a name="Onboarding_Steps"></a></h4> +<p></p> +<ul> +<li>Create cluster definition for the cluster, specifying name node, job tracker, workflow engine endpoint, messaging endpoint. Refer to <a href="./EntitySpecification.html">cluster definition</a> for details.</li> +<li>Create Feed definitions for each of the input and output specifying frequency, data path, ownership. Refer to <a href="./EntitySpecification.html">feed definition</a> for details.</li> +<li>Create Process definition for your job. Process defines configuration for the workflow job. Important attributes are frequency, inputs/outputs and workflow path. Refer to <a href="./EntitySpecification.html">process definition</a> for process details.</li> +<li>Define workflow for your job using the workflow engine(only oozie is supported as of now). Refer <a class="externalLink" href="http://oozie.apache.org/docs/3.1.3-incubating/WorkflowFunctionalSpec.html";>Oozie Workflow Specification</a>. The libraries required for the workflow should be available in lib folder in workflow path.</li> +<li>Set-up workflow definition, libraries and referenced scripts on hadoop.</li> +<li>Submit cluster definition</li> +<li>Submit and schedule feed and process definitions</li></ul></div> +<div class="section"> +<h4>Sample Pipeline<a name="Sample_Pipeline"></a></h4></div> +<div class="section"> +<h5>Cluster <a name="Cluster"></a></h5> +<p>Cluster definition that contains end points for name node, job tracker, oozie and jms server: The cluster locations MUST be created prior to submitting a cluster entity to Falcon. <b>staging</b> must have 777 permissions and the parent dirs must have execute permissions <b>working</b> must have 755 permissions and the parent dirs must have execute permissions</p> +<div class="source"> +<pre> +<?xml version="1.0"?> +<!-- + Cluster configuration + --> +<cluster colo="ua2" description="" name="corp" xmlns="uri:falcon:cluster:0.1" + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> + <interfaces> + <interface type="readonly" endpoint="hftp://name-node.com:50070" version="2.5.0" /> + + <interface type="write" endpoint="hdfs://name-node.com:54310" version="2.5.0" /> + + <interface type="execute" endpoint="job-tracker:54311" version="2.5.0" /> + + <interface type="workflow" endpoint="http://oozie.com:11000/oozie/" version="4.0.1" /> + + <interface type="messaging" endpoint="tcp://jms-server.com:61616?daemon=true" version="5.1.6" /> + </interfaces> + + <locations> + <location name="staging" path="/projects/falcon/staging" /> + <location name="temp" path="/tmp" /> + <location name="working" path="/projects/falcon/working" /> + </locations> +</cluster> + +</pre></div></div> +<div class="section"> +<h5>Input Feed<a name="Input_Feed"></a></h5> +<p>Hourly feed that defines feed path, frequency, ownership and validity:</p> +<div class="source"> +<pre> +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Hourly sample input data + --> + +<feed description="sample input data" name="SampleInput" xmlns="uri:falcon:feed:0.1" + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> + <groups>group</groups> + + <frequency>hours(1)</frequency> + + <late-arrival cut-off="hours(6)" /> + + <clusters> + <cluster name="corp" type="source"> + <validity start="2009-01-01T00:00Z" end="2099-12-31T00:00Z" timezone="UTC" /> + <retention limit="months(24)" action="delete" /> + </cluster> + </clusters> + + <locations> + <location type="data" path="/projects/bootcamp/data/${YEAR}-${MONTH}-${DAY}-${HOUR}/SampleInput" /> + <location type="stats" path="/projects/bootcamp/stats/SampleInput" /> + <location type="meta" path="/projects/bootcamp/meta/SampleInput" /> + </locations> + + <ACL owner="suser" group="users" permission="0755" /> + + <schema location="/none" provider="none" /> +</feed> + +</pre></div></div> +<div class="section"> +<h5>Output Feed<a name="Output_Feed"></a></h5> +<p>Daily feed that defines feed path, frequency, ownership and validity:</p> +<div class="source"> +<pre> +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Daily sample output data + --> + +<feed description="sample output data" name="SampleOutput" xmlns="uri:falcon:feed:0.1" +xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> + <groups>group</groups> + + <frequency>days(1)</frequency> + + <late-arrival cut-off="hours(6)" /> + + <clusters> + <cluster name="corp" type="source"> + <validity start="2009-01-01T00:00Z" end="2099-12-31T00:00Z" timezone="UTC" /> + <retention limit="months(24)" action="delete" /> + </cluster> + </clusters> + + <locations> + <location type="data" path="/projects/bootcamp/output/${YEAR}-${MONTH}-${DAY}/SampleOutput" /> + <location type="stats" path="/projects/bootcamp/stats/SampleOutput" /> + <location type="meta" path="/projects/bootcamp/meta/SampleOutput" /> + </locations> + + <ACL owner="suser" group="users" permission="0755" /> + + <schema location="/none" provider="none" /> +</feed> + +</pre></div></div> +<div class="section"> +<h5>Process<a name="Process"></a></h5> +<p>Sample process which runs daily at 6th hour on corp cluster. It takes one input - SampleInput for the previous day(24 instances). It generates one output - SampleOutput for previous day. The workflow is defined at /projects/bootcamp/workflow/workflow.xml. Any libraries available for the workflow should be at /projects/bootcamp/workflow/lib. The process also defines properties queueName, ssh.host, and fileTimestamp which are passed to the workflow. In addition, Falcon exposes the following properties to the workflow: nameNode, jobTracker(hadoop properties), input and output(Input/Output properties).</p> +<div class="source"> +<pre> +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Daily sample process. Runs at 6th hour every day. Input - last day's hourly data. Generates output for yesterday + --> +<process name="SampleProcess"> + <cluster name="corp" /> + + <frequency>days(1)</frequency> + + <validity start="2012-04-03T06:00Z" end="2022-12-30T00:00Z" timezone="UTC" /> + + <inputs> + <input name="input" feed="SampleInput" start="yesterday(0,0)" end="today(-1,0)" /> + </inputs> + + <outputs> + <output name="output" feed="SampleOutput" instance="yesterday(0,0)" /> + </outputs> + + <properties> + <property name="queueName" value="reports" /> + <property name="ssh.host" value="host.com" /> + <property name="fileTimestamp" value="${coord:formatTime(coord:nominalTime(), 'yyyy-MM-dd')}" /> + </properties> + + <workflow engine="oozie" path="/projects/bootcamp/workflow" /> + + <retry policy="periodic" delay="minutes(5)" attempts="3" /> + + <late-process policy="exp-backoff" delay="hours(1)"> + <late-input input="input" workflow-path="/projects/bootcamp/workflow/lateinput" /> + </late-process> +</process> + +</pre></div></div> +<div class="section"> +<h5>Oozie Workflow<a name="Oozie_Workflow"></a></h5> +<p>The sample user workflow contains 3 actions:</p> +<ul> +<li>Pig action - Executes pig script /projects/bootcamp/workflow/script.pig</li> +<li>concatenator - Java action that concatenates part files and generates a single file</li> +<li>file upload - ssh action that gets the concatenated file from hadoop and sends the file to a remote host</li></ul> +<div class="source"> +<pre> +<workflow-app xmlns="uri:oozie:workflow:0.2" name="sample-wf"> + <start to="pig" /> + + <action name="pig"> + <pig> + <job-tracker>${jobTracker}</job-tracker> + <name-node>${nameNode}</name-node> + <prepare> + <delete path="${output}"/> + </prepare> + <configuration> + <property> + <name>mapred.job.queue.name</name> + <value>${queueName}</value> + </property> + <property> + <name>mapreduce.fileoutputcommitter.marksuccessfuljobs</name> + <value>true</value> + </property> + </configuration> + <script>${nameNode}/projects/bootcamp/workflow/script.pig</script> + <param>input=${input}</param> + <param>output=${output}</param> + <file>lib/dependent.jar</file> + </pig> + <ok to="concatenator" /> + <error to="fail" /> + </action> + + <action name="concatenator"> + <java> + <job-tracker>${jobTracker}</job-tracker> + <name-node>${nameNode}</name-node> + <prepare> + <delete path="${nameNode}/projects/bootcamp/concat/data-${fileTimestamp}.csv"/> + </prepare> + <configuration> + <property> + <name>mapred.job.queue.name</name> + <value>${queueName}</value> + </property> + </configuration> + <main-class>com.wf.Concatenator</main-class> + <arg>${output}</arg> + <arg>${nameNode}/projects/bootcamp/concat/data-${fileTimestamp}.csv</arg> + </java> + <ok to="fileupload" /> + <error to="fail"/> + </action> + + <action name="fileupload"> + <ssh> + <host>localhost</host> + <command>/tmp/fileupload.sh</command> + <args>${nameNode}/projects/bootcamp/concat/data-${fileTimestamp}.csv</args> + <args>${wf:conf("ssh.host")}</args> + <capture-output/> + </ssh> + <ok to="fileUploadDecision" /> + <error to="fail"/> + </action> + + <decision name="fileUploadDecision"> + <switch> + <case to="end"> + ${wf:actionData('fileupload')['output'] == '0'} + </case> + <default to="fail"/> + </switch> + </decision> + + <kill name="fail"> + <message>Workflow failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> + </kill> + + <end name="end" /> +</workflow-app> + +</pre></div></div> +<div class="section"> +<h5>File Upload Script<a name="File_Upload_Script"></a></h5> +<p>The script gets the file from hadoop, rsyncs the file to /tmp on remote host and deletes the file from hadoop</p> +<div class="source"> +<pre> +#!/bin/bash + +trap 'echo "output=$?"; exit $?' ERR INT TERM + +echo "Arguments: $@" +SRCFILE=$1 +DESTHOST=$3 + +FILENAME=`basename $SRCFILE` +rm -f /tmp/$FILENAME +hadoop fs -copyToLocal $SRCFILE /tmp/ +echo "Copied $SRCFILE to /tmp" + +rsync -ztv --rsh=ssh --stats /tmp/$FILENAME $DESTHOST:/tmp +echo "rsynced $FILENAME to $DESTUSER@$DESTHOST:$DESTFILE" + +hadoop fs -rmr $SRCFILE +echo "Deleted $SRCFILE" + +rm -f /tmp/$FILENAME +echo "output=0" + +</pre></div></div> + </div> + </div> + + <hr/> + + <footer> + <div class="container"> + <div class="row span12">Copyright © 2013-2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + <p id="poweredBy" class="pull-right"> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </p> + + </div> + </footer> + </body> +</html>
http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/content/0.11/Operability.html ---------------------------------------------------------------------- diff --git a/content/0.11/Operability.html b/content/0.11/Operability.html new file mode 100644 index 0000000..39740a9 --- /dev/null +++ b/content/0.11/Operability.html @@ -0,0 +1,277 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2018-03-12 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20180312" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Falcon - Operationalizing Falcon</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container"> + <div id="banner"> + <div class="pull-left"> + <div id="bannerLeft"> + <img src="images/falcon-logo.png" alt="Apache Falcon" width="200px" height="45px"/> + </div> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="index.html" title="Falcon"> + Falcon</a> + </li> + <li class="divider ">/</li> + <li class="">Operationalizing Falcon</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2018-03-12</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.11</li> + + </ul> + </div> + + + + <div id="bodyColumn" > + + <div class="section"> +<h2>Operationalizing Falcon<a name="Operationalizing_Falcon"></a></h2></div> +<div class="section"> +<h3>Overview<a name="Overview"></a></h3> +<p>Apache Falcon provides various tools to operationalize Falcon consisting of Alerts for unrecoverable errors, Audits of user actions, Metrics, and Notifications. They are detailed below.</p> +<p>++ Lineage</p> +<p>Currently Lineage has no way to access or restore information about entity instances created during the time lineage was disabled. Information about entities however, is preserved and bootstrapped when lineage is enabled. If you have to reset the graph db then you can delete the graph db files as specified in the startup.properties and restart the falcon. Please note: you will loose all the information about the instances if you delete the graph db.</p></div> +<div class="section"> +<h3>Monitoring<a name="Monitoring"></a></h3> +<p>Falcon provides monitoring of various events by capturing metrics of those events. The metric numbers can then be used to monitor performance and health of the Falcon system and the entire processing pipelines.</p> +<p>Falcon also exposes <a class="externalLink" href="https://github.com/thinkaurelius/titan/wiki/Titan-Performance-and-Monitoring";>metrics for titandb</a></p> +<p>Users can view the logs of these events in the metric.log file, by default this file is created under ${user.dir}/logs/ directory. Users may also extend the Falcon monitoring framework to send events to systems like Mondemand/lwes by implementingorg.apache.falcon.plugin.MonitoringPlugin interface.</p> +<p>The following events are captured by Falcon for logging the metrics:</p> +<ol style="list-style-type: decimal"> +<li>New cluster definitions posted to Falcon (success & failures)</li> +<li>New feed definition posted to Falcon (success & failures)</li> +<li>New process definition posted to Falcon (success & failures)</li> +<li>Process update events (success & failures)</li> +<li>Feed update events (success & failures)</li> +<li>Cluster update events (success & failures)</li> +<li>Process suspend events (success & failures)</li> +<li>Feed suspend events (success & failures)</li> +<li>Process resume events (success & failures)</li> +<li>Feed resume events (success & failures)</li> +<li>Process remove events (success & failures)</li> +<li>Feed remove events (success & failures)</li> +<li>Cluster remove events (success & failures)</li> +<li>Process instance kill events (success & failures)</li> +<li>Process instance re-run events (success & failures)</li> +<li>Process instance generation events</li> +<li>Process instance failure events</li> +<li>Process instance auto-retry events</li> +<li>Process instance retry exhaust events</li> +<li>Feed instance deletion event</li> +<li>Feed instance deletion failure event (no retries)</li> +<li>Feed instance replication event</li> +<li>Feed instance replication failure event</li> +<li>Feed instance replication auto-retry event</li> +<li>Feed instance replication retry exhaust event</li> +<li>Feed instance late arrival event</li> +<li>Feed instance post cut-off arrival event</li> +<li>Process re-run due to late feed event</li> +<li>Transaction rollback failed event</li></ol> +<p>The metric logged for an event has the following properties:</p> +<ol style="list-style-type: decimal"> +<li>Action - Name of the event.</li> +<li>Dimensions - A list of name/value pairs of various attributes for a given action.</li> +<li>Status- Status of an action FAILED/SUCCEEDED.</li> +<li>Time-taken - Time taken in nanoseconds for a given action.</li></ol> +<p>An example for an event logged for a submit of a new process definition:</p> +<p>2012-05-04 12:23:34,026 {Action:submit, Dimensions:{entityType=process}, Status: SUCCEEDED, Time-taken:97087000 ns}</p> +<p>Users may parse the metric.log or capture these events from custom monitoring frameworks and can plot various graphs or send alerts according to their requirements.</p></div> +<div class="section"> +<h3>Notifications<a name="Notifications"></a></h3> +<p>Falcon has two types of notifications - System and User notifications.</p></div> +<div class="section"> +<h4>System notifications<a name="System_notifications"></a></h4> +<p>The System notifications are internally generated and used by Falcon to monitor the Falcon orchestrated workflow jobs. By default, Falcon starts an ActiveMQ embedded JMS server on Falcon machine on port 61616 as a daemon. Alternatively, users can make Falcon to use an existing JMS server instead of starting an embedded instance by doing the following 2 steps:</p> +<p></p> +<ul> +<li>Setting the property broker.url in the startup.properties as below</li></ul> +<div class="source"> +<pre> + *.broker.url=tcp://jms-server-host:61616 + +</pre></div> +<p></p> +<ul> +<li>Set the system property falcon.embeddedmq to false as below</li></ul> +<div class="source"> +<pre> + <FALCON-INSTALL-DIR>/bin/falcon-start -Dfalcon.embeddedmq=false + +</pre></div> +<p>Falcon uses FALCON.ENTITY.TOPIC to publish system notifications. This topic and the Map Message fields are internal and could change between releases.</p></div> +<div class="section"> +<h4>User notifications<a name="User_notifications"></a></h4> +<p>Falcon, in addition to the FALCON.ENTITY.TOPIC, also creates a JMS topic for every process/feed that is scheduled in Falcon as part of User notification. To enable User notifications, the broker url and implementation class of the JMS engine need to be specified in the cluster definition associated with the feed/process. Users may register consumers on the required topic to check the availability or status of feed instances. The User notification JMS broker instance can be same as the System notification or different.</p> +<p>The name of the JMS topic is same as the process/feed name. Falcon sends a map message for every feed instance that is created/deleted/replicated/imported/exported to the JMS topic. The JMS Map Message sent to a topic has the following fields:</p> +<p></p> +<ol style="list-style-type: decimal"> +<li>cluster - name of the current cluster the feed/process is dependent on.</li> +<li>entityType - type of the entity (feed or process).</li> +<li>entityName - name of the entity.</li> +<li>nominalTime - instance time (or data date).</li> +<li>operation - operation like generate, delete, replicate, import, export.</li> +<li>feedNames - name of the feeds which are generated/replicated/deleted/imported/exported.</li> +<li>feedInstancePaths - comma separated feed instance paths.</li> +<li>workflowId - current workflow-id of the instance.</li> +<li>workflowUser - user who owns the feed instance (i.e partition).</li> +<li>runId - current run-id of the instance.</li> +<li>status - status of the user workflow instance.</li> +<li>timeStamp - current timestamp.</li> +<li>logDir - log dir where lineage can be recorded.</li></ol> +<p>The JMS messages are automatically purged after a certain period (default 3 days) by the Falcon JMS house-keeping service. TTL (Time-to-live) for JMS message can be configured in the Falcon's startup.properties file.</p> +<p>The following example shows how to enable and read user notification by connecting to the JMS broker.</p> +<p>First, specify the JMS broker url in the cluster definition XML as shown below.</p> +<div class="source"> +<pre> + +<?xml version="1.0"?> +<!-- filename : primaryCluster.xml --> +<cluster colo="USWestOregon" description="oregonHadoopCluster" name="primaryCluster" xmlns="uri:falcon:cluster:0.1"> + <interfaces> + ... + ... + <interface type="messaging" endpoint="tcp://user-jms-broker-host:61616?daemon=true" version="5.1.6" /> + ... + </interfaces> +</cluster> + + +</pre></div> +<p>Next, use a JMS consumer (example below in Java) to read the message from the topic with the name FALCON.<feed-or-process-name></p> +<div class="source"> +<pre> +import org.apache.activemq.ActiveMQConnectionFactory; +import org.apache.activemq.command.ActiveMQMapMessage; +import javax.jms.ConnectionFactory; +import javax.jms.Connection; +import javax.jms.MessageConsumer; +import javax.jms.Topic; +import javax.jms.Session; +import javax.jms.TopicSession; + +public class FalconUserJMSClient { + public static void main(String[] args)throws Exception { + // Note: specify the JMS broker URL + String brokerUrl = "tcp://localhost:61616"; + + ConnectionFactory connectionFactory = new ActiveMQConnectionFactory(brokerUrl); + Connection connection = connectionFactory.createConnection(); + connection.setClientID("Falcon User JMS Consumer"); + TopicSession session = (TopicSession) connection.createSession(false, Session.AUTO_ACKNOWLEDGE); + try { + + // Note: the topic name for the feed will be FALCON.<feed-name> + Topic falconTopic = session.createTopic("FALCON.feed-sample"); + MessageConsumer consumer = session.createConsumer(falconTopic); + connection.start(); + while (true) { + ActiveMQMapMessage msg = (ActiveMQMapMessage) consumer.receive(); + System.out.println("cluster : " + msg.getString("cluster")); + System.out.println("entityType : " + msg.getString("entityType")); + System.out.println("entityName : " + msg.getString("entityName")); + System.out.println("nominalTime : " + msg.getString("nominalTime")); + System.out.println("operation : " + msg.getString("operation")); + + System.out.println("feedNames : " + msg.getString("feedNames")); + System.out.println("feedInstancePaths : " + msg.getString("feedInstancePaths")); + + System.out.println("workflowId : " + msg.getString("workflowId")); + System.out.println("workflowUser : " + msg.getString("workflowUser")); + System.out.println("runId : " + msg.getString("runId")); + System.out.println("status : " + msg.getString("status")); + System.out.println("timeStamp : " + msg.getString("timeStamp")); + System.out.println("logDir : " + msg.getString("logDir")); + + System.out.println("brokerUrl : " + msg.getString("brokerUrl")); + System.out.println("brokerImplClass : " + msg.getString("brokerImplClass")); + System.out.println("logFile : " + msg.getString("logFile")); + System.out.println("topicName : " + msg.getString("topicName")); + System.out.println("brokerTTL : " + msg.getString("brokerTTL")); + } + } finally { + if (session != null) { + session.close(); + } + if (connection != null) { + connection.close(); + } + } + } +} + +</pre></div></div> +<div class="section"> +<h3>Alerts<a name="Alerts"></a></h3> +<p>Falcon generates two type of alerts:</p> +<p>1. By default it logs unrecoverable errors into a log file Users can view these alerts in the alerts.log file, by default this file is created under ${user.dir}/logs/ directory.</p> +<p>Users may also extend the Falcon Alerting plugin to send events to systems like Nagios, etc. by extending org.apache.falcon.plugin.AlertingPlugin interface.</p> +<p>2. Alerts on <b>SLA misses</b> for feeds and process is detailed in <a href="./EntitySLAAlerting.html">Entity SLA Alerting</a>.</p></div> +<div class="section"> +<h3>Audits<a name="Audits"></a></h3> +<p>Falcon audits all user activity and captures them into a log file by default. Users can view these audits in the audit.log file, by default this file is created under ${user.dir}/logs/ directory.</p> +<p>Users may also extend the Falcon Audit plugin to send audits to systems like Apache Argus, etc. by extending org.apache.falcon.plugin.AuditingPlugin interface.</p></div> +<div class="section"> +<h3>Metrics Collection In Graphite and Database<a name="Metrics_Collection_In_Graphite_and_Database"></a></h3> +<p>Falcon has support to send process metrics like waiting time ,exection time and number of failures to graphite and falcon db.</p> +<p>For details go through <a href="./MetricCollection.html">Metric Collection</a></p></div> + </div> + </div> + + <hr/> + + <footer> + <div class="container"> + <div class="row span12">Copyright © 2013-2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + <p id="poweredBy" class="pull-right"> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </p> + + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/content/0.11/PrismSetup.png ---------------------------------------------------------------------- diff --git a/content/0.11/PrismSetup.png b/content/0.11/PrismSetup.png new file mode 100644 index 0000000..b0dc9a5 Binary files /dev/null and b/content/0.11/PrismSetup.png differ http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/content/0.11/ProcessSchedule.png ---------------------------------------------------------------------- diff --git a/content/0.11/ProcessSchedule.png b/content/0.11/ProcessSchedule.png new file mode 100644 index 0000000..a7dd788 Binary files /dev/null and b/content/0.11/ProcessSchedule.png differ http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/content/0.11/Security.html ---------------------------------------------------------------------- diff --git a/content/0.11/Security.html b/content/0.11/Security.html new file mode 100644 index 0000000..195786e --- /dev/null +++ b/content/0.11/Security.html @@ -0,0 +1,535 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2018-03-12 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20180312" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Falcon - Securing Falcon</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container"> + <div id="banner"> + <div class="pull-left"> + <div id="bannerLeft"> + <img src="images/falcon-logo.png" alt="Apache Falcon" width="200px" height="45px"/> + </div> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="index.html" title="Falcon"> + Falcon</a> + </li> + <li class="divider ">/</li> + <li class="">Securing Falcon</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2018-03-12</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.11</li> + + </ul> + </div> + + + + <div id="bodyColumn" > + + <div class="section"> +<h2>Securing Falcon<a name="Securing_Falcon"></a></h2></div> +<div class="section"> +<h3>Overview<a name="Overview"></a></h3> +<p>Apache Falcon provides the following security features:</p> +<ul> +<li>Credential provider alias for passwords used in Falcon server.</li> +<li>Authentication to identify proper users.</li> +<li>Authorization to specify resource access permission for users or groups.</li> +<li>Cross-Site Request Forgery (CSRF) prevention.</li> +<li>SSL to provide transport level security for data confidentiality and integrity.</li></ul></div> +<div class="section"> +<h3>Credential Provider Alias for Passwords<a name="Credential_Provider_Alias_for_Passwords"></a></h3> +<p>Server-side configuration properties (i.e. startup.properties) contain passwords and other sensitive information. In addition to specifying properties in plain text, we provide the user an option to use credential provider alias in the property file.</p> +<p>Take SMTP password for example. The user can store the password in a <a class="externalLink" href="http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CommandsManual.html#credential";>Hadoop credential provider</a> with the alias name <i>SMTPPasswordAlias</i>. In startup.properties where SMTP password is needed, the user can refer to its alias name <i>SMTPPasswordAlias</i> instead of providing the real password.</p> +<p>The alias property to be resolved through Hadoop credential provider should have the format: <i>credential.provider.alias.for.[property-key]</i>. For example, <i>credential.provider.alias.for.falcon.email.smtp.password=SMTPPasswordAlias</i> for SMTP password. Falcon server, during the start, will automatically retrieve the real password provided the alias name.</p> +<p>The user can specify the provider path with the property key <i>credential.provider.path</i>, e.g. <i>credential.provider.path=jceks://file/tmp/test.jceks</i>. If not specified, Falcon will use the default Hadoop credential provider path in core-site.xml.</p></div> +<div class="section"> +<h3>Authentication (User Identity)<a name="Authentication_User_Identity"></a></h3> +<p>Apache Falcon enforces authentication on protected resources. Once authentication has been established it sets a signed HTTP Cookie that contains an authentication token with the user name, user principal, authentication type and expiration time.</p> +<p>It does so by using <a class="externalLink" href="./Http://hadoop.apache.org/docs/current/hadoop-auth/index.html.html">Hadoop Auth</a>. Hadoop Auth is a Java library consisting of a client and a server components to enable Kerberos SPNEGO authentication for HTTP. Hadoop Auth also supports additional authentication mechanisms on the client and the server side via 2 simple interfaces.</p></div> +<div class="section"> +<h4>Authentication Methods<a name="Authentication_Methods"></a></h4> +<p>It supports 2 authentication methods, simple and kerberos out of the box.</p></div> +<div class="section"> +<h5>Pseudo/Simple Authentication<a name="PseudoSimple_Authentication"></a></h5> +<p>Falcon authenticates the user by simply trusting the value of the query string parameter 'user.name'. This is the default mode Falcon is configured with.</p></div> +<div class="section"> +<h5>Kerberos Authentication<a name="Kerberos_Authentication"></a></h5> +<p>Falcon uses HTTP Kerberos SPNEGO to authenticate the user.</p></div> +<div class="section"> +<h3>Authorization<a name="Authorization"></a></h3> +<p>Falcon also enforces authorization on Entities using ACLs (Access Control Lists). ACLs are useful for implementing permission requirements and provide a way to set different permissions for specific users or named groups.</p> +<p>By default, support for authorization is disabled and can be enabled in startup.properties.</p></div> +<div class="section"> +<h4>ACLs in Entity<a name="ACLs_in_Entity"></a></h4> +<p>All Entities now have ACL which needs to be present if authorization is enabled. Only owners who own or created the entity will be allowed to update or delete their entities.</p> +<p>An entity has ACLs (Access Control Lists) that are useful for implementing permission requirements and provide a way to set different permissions for specific users or named groups.</p> +<div class="source"> +<pre> + <ACL owner="test-user" group="test-group" permission="*"/> + +</pre></div> +<p>ACL indicates the Access control list for this cluster. owner is the Owner of this entity. group is the one which has access to read. permission indicates the rwx is not enforced at this time.</p></div> +<div class="section"> +<h4>Super-User<a name="Super-User"></a></h4> +<p>The super-user is the user with the same identity as falcon process itself. Loosely, if you started the falcon, then you are the super-user. The super-user can do anything in that permissions checks never fail for the super-user. There is no persistent notion of who was the super-user; when the falcon is started the process identity determines who is the super-user for now. The Falcon super-user does not have to be the super-user of the falcon host, nor is it necessary that all clusters have the same super-user. Also, an experimenter running Falcon on a personal workstation, conveniently becomes that installation's super-user without any configuration.</p> +<p>Falcon also allows users to configure a super user group and allows users belonging to this group to be a super user.</p> +<p>ACL owner and group must be valid even if the authenticated user is a super-user.</p></div> +<div class="section"> +<h4>Group Memberships<a name="Group_Memberships"></a></h4> +<p>Once a user has been authenticated and a username has been determined, the list of groups is determined by a group mapping service, configured by the hadoop.security.group.mapping property in Hadoop. The default implementation, org.apache.hadoop.security.ShellBasedUnixGroupsMapping, will shell out to the Unix bash -c groups command to resolve a list of groups for a user.</p> +<p>Note that Falcon stores the user and group of an Entity as strings; there is no conversion from user and group identity numbers as is conventional in Unix.</p> +<p>The only limitation is that a user cannot add a group in ACL that he does not belong to.</p></div> +<div class="section"> +<h4>Authorization Provider<a name="Authorization_Provider"></a></h4> +<p>Falcon provides a plugin-able provider interface for Authorization. It also ships with a default implementation that enforces the following authorization policy.</p></div> +<div class="section"> +<h5>Entity and Instance Management Operations Policy<a name="Entity_and_Instance_Management_Operations_Policy"></a></h5> +<p></p> +<ul> +<li>All Entity and Instance operations are authorized for users who created them, Owners and users with group memberships</li> +<li>Reference to entities with in a feed or process is allowed with out enforcing permissions</li></ul> +<p>Any Feed or Process can refer to a Cluster entity not owned by the Feed or Process owner. Any Process can refer to a Feed entity not owned by the Process owner</p> +<p>The authorization is enforced in the following way:</p> +<p></p> +<ul> +<li>if admin resource, +<ul> +<li>If authenticated user name matches the admin users configuration</li> +<li>Else if groups of the authenticated user matches the admin groups configuration</li> +<li>Else authorization exception is thrown</li></ul></li> +<li>Else if entities or instance resource +<ul> +<li>If the authenticated user matches the owner in ACL for the entity</li> +<li>Else if the groups of the authenticated user matches the group in ACL for the entity</li> +<li>Else authorization exception is thrown</li></ul></li> +<li>Else if lineage resource +<ul> +<li>All have read-only permissions, reason being folks should be able to examine the dependency and allow reuse</li></ul></li></ul> +<p>To authenticate user for REST api calls, user should append "user.name=<username>" to the query.</p> +<p><b>operations on Entity Resource</b></p> +<p></p> +<table border="0" class="table table-striped"> +<tr class="a"> +<th>Resource</th> +<th>Description</th> +<th>Authorization</th></tr> +<tr class="b"> +<td><a href="./Restapi/EntityValidate.html">api/entities/validate/:entity-type</a></td> +<td>Validate the entity</td> +<td>Owner/Group</td></tr> +<tr class="a"> +<td><a href="./Restapi/EntitySubmit.html">api/entities/submit/:entity-type</a></td> +<td>Submit the entity</td> +<td>Owner/Group</td></tr> +<tr class="b"> +<td><a href="./Restapi/EntityUpdate.html">api/entities/update/:entity-type/:entity-name</a></td> +<td>Update the entity</td> +<td>Owner/Group</td></tr> +<tr class="a"> +<td><a href="./Restapi/EntitySubmitAndSchedule.html">api/entities/submitAndSchedule/:entity-type</a></td> +<td>Submit & Schedule the entity</td> +<td>Owner/Group</td></tr> +<tr class="b"> +<td><a href="./Restapi/EntitySchedule.html">api/entities/schedule/:entity-type/:entity-name</a></td> +<td>Schedule the entity</td> +<td>Owner/Group</td></tr> +<tr class="a"> +<td><a href="./Restapi/EntitySuspend.html">api/entities/suspend/:entity-type/:entity-name</a></td> +<td>Suspend the entity</td> +<td>Owner/Group</td></tr> +<tr class="b"> +<td><a href="./Restapi/EntityResume.html">api/entities/resume/:entity-type/:entity-name</a></td> +<td>Resume the entity</td> +<td>Owner/Group</td></tr> +<tr class="a"> +<td><a href="./Restapi/EntityDelete.html">api/entities/delete/:entity-type/:entity-name</a></td> +<td>Delete the entity</td> +<td>Owner/Group</td></tr> +<tr class="b"> +<td><a href="./Restapi/EntityStatus.html">api/entities/status/:entity-type/:entity-name</a></td> +<td>Get the status of the entity</td> +<td>Owner/Group</td></tr> +<tr class="a"> +<td><a href="./Restapi/EntityDefinition.html">api/entities/definition/:entity-type/:entity-name</a></td> +<td>Get the definition of the entity</td> +<td>Owner/Group</td></tr> +<tr class="b"> +<td><a href="./Restapi/EntityList.html">api/entities/list/:entity-type?fields=:fields</a></td> +<td>Get the list of entities</td> +<td>Owner/Group</td></tr> +<tr class="a"> +<td><a href="./Restapi/EntityDependencies.html">api/entities/dependencies/:entity-type/:entity-name</a></td> +<td>Get the dependencies of the entity</td> +<td>Owner/Group</td></tr></table><b>REST Call on Feed and Process Instances</b> +<p></p> +<table border="0" class="table table-striped"> +<tr class="a"> +<th>Resource</th> +<th>Description</th> +<th>Authorization</th></tr> +<tr class="b"> +<td><a href="./Restapi/InstanceRunning.html">api/instance/running/:entity-type/:entity-name</a></td> +<td>List of running instances.</td> +<td>Owner/Group</td></tr> +<tr class="a"> +<td><a href="./Restapi/InstanceStatus.html">api/instance/status/:entity-type/:entity-name</a></td> +<td>Status of a given instance</td> +<td>Owner/Group</td></tr> +<tr class="b"> +<td><a href="./Restapi/InstanceKill.html">api/instance/kill/:entity-type/:entity-name</a></td> +<td>Kill a given instance</td> +<td>Owner/Group</td></tr> +<tr class="a"> +<td><a href="./Restapi/InstanceSuspend.html">api/instance/suspend/:entity-type/:entity-name</a></td> +<td>Suspend a running instance</td> +<td>Owner/Group</td></tr> +<tr class="b"> +<td><a href="./Restapi/InstanceResume.html">api/instance/resume/:entity-type/:entity-name</a></td> +<td>Resume a given instance</td> +<td>Owner/Group</td></tr> +<tr class="a"> +<td><a href="./Restapi/InstanceRerun.html">api/instance/rerun/:entity-type/:entity-name</a></td> +<td>Rerun a given instance</td> +<td>Owner/Group</td></tr> +<tr class="b"> +<td><a href="./InstanceLogs.html">api/instance/logs/:entity-type/:entity-name</a></td> +<td>Get logs of a given instance</td> +<td>Owner/Group</td></tr></table></div> +<div class="section"> +<h5>Admin Resources Policy<a name="Admin_Resources_Policy"></a></h5> +<p>Only users belonging to admin users or groups have access to this resource. Admin membership is determined by a static configuration parameter.</p> +<p></p> +<table border="0" class="table table-striped"> +<tr class="a"> +<th>Resource</th> +<th>Description</th> +<th>Authorization</th></tr> +<tr class="b"> +<td><a href="./Restapi/AdminVersion.html">api/admin/version</a></td> +<td>Get version of the server</td> +<td>No restriction</td></tr> +<tr class="a"> +<td><a href="./Restapi/AdminStack.html">api/admin/stack</a></td> +<td>Get stack of the server</td> +<td>Admin User/Group</td></tr> +<tr class="b"> +<td><a href="./Restapi/AdminConfig.html">api/admin/config/:config-type</a></td> +<td>Get configuration information of the server</td> +<td>Admin User/Group</td></tr></table></div> +<div class="section"> +<h5>Lineage Resource Policy<a name="Lineage_Resource_Policy"></a></h5> +<p>Lineage is read-only and hence all users can look at lineage for their respective entities. <b>Note:</b> This gap will be fixed in a later release.</p></div> +<div class="section"> +<h3>Authentication Configuration<a name="Authentication_Configuration"></a></h3> +<p>Following is the Server Side Configuration Setup for Authentication.</p></div> +<div class="section"> +<h4>Common Configuration Parameters<a name="Common_Configuration_Parameters"></a></h4> +<div class="source"> +<pre> +# Authentication type must be specified: simple|kerberos +*.falcon.authentication.type=kerberos + +</pre></div></div> +<div class="section"> +<h4>Kerberos Configuration<a name="Kerberos_Configuration"></a></h4> +<div class="source"> +<pre> +##### Service Configuration + +# Indicates the Kerberos principal to be used in Falcon Service. +*.falcon.service.authentication.kerberos.principal=falcon/[email protected] + +# Location of the keytab file with the credentials for the Service principal. +*.falcon.service.authentication.kerberos.keytab=/etc/security/keytabs/falcon.service.keytab + +# name node principal to talk to config store +*.dfs.namenode.kerberos.principal=nn/[email protected] + +# Indicates how long (in seconds) falcon authentication token is valid before it has to be renewed. +*.falcon.service.authentication.token.validity=86400 + +##### SPNEGO Configuration + +# Authentication type must be specified: simple|kerberos|<class> +# org.apache.falcon.security.RemoteUserInHeaderBasedAuthenticationHandler can be used for backwards compatibility +*.falcon.http.authentication.type=kerberos + +# Indicates how long (in seconds) an authentication token is valid before it has to be renewed. +*.falcon.http.authentication.token.validity=36000 + +# The signature secret for signing the authentication tokens. +*.falcon.http.authentication.signature.secret=falcon + +# The domain to use for the HTTP cookie that stores the authentication token. +*.falcon.http.authentication.cookie.domain= + +# Indicates if anonymous requests are allowed when using 'simple' authentication. +*.falcon.http.authentication.simple.anonymous.allowed=true + +# Indicates the Kerberos principal to be used for HTTP endpoint. +# The principal MUST start with 'HTTP/' as per Kerberos HTTP SPNEGO specification. +*.falcon.http.authentication.kerberos.principal=HTTP/[email protected] + +# Location of the keytab file with the credentials for the HTTP principal. +*.falcon.http.authentication.kerberos.keytab=/etc/security/keytabs/spnego.service.keytab + +# The kerberos names rules is to resolve kerberos principal names, refer to Hadoop's KerberosName for more details. +*.falcon.http.authentication.kerberos.name.rules=DEFAULT + +# Comma separated list of black listed users +*.falcon.http.authentication.blacklisted.users= + +# Increase Jetty request buffer size to accommodate the generated Kerberos token +*.falcon.jetty.request.buffer.size=16192 + +</pre></div></div> +<div class="section"> +<h4>Pseudo/Simple Configuration<a name="PseudoSimple_Configuration"></a></h4> +<div class="source"> +<pre> +##### SPNEGO Configuration + +# Authentication type must be specified: simple|kerberos|<class> +# org.apache.falcon.security.RemoteUserInHeaderBasedAuthenticationHandler can be used for backwards compatibility +*.falcon.http.authentication.type=simple + +# Indicates how long (in seconds) an authentication token is valid before it has to be renewed. +*.falcon.http.authentication.token.validity=36000 + +# The signature secret for signing the authentication tokens. +*.falcon.http.authentication.signature.secret=falcon + +# The domain to use for the HTTP cookie that stores the authentication token. +*.falcon.http.authentication.cookie.domain= + +# Indicates if anonymous requests are allowed when using 'simple' authentication. +*.falcon.http.authentication.simple.anonymous.allowed=true + +# Comma separated list of black listed users +*.falcon.http.authentication.blacklisted.users= + +</pre></div></div> +<div class="section"> +<h3>Authorization Configuration<a name="Authorization_Configuration"></a></h3></div> +<div class="section"> +<h4>Enabling Authorization<a name="Enabling_Authorization"></a></h4> +<p>By default, support for authorization is disabled and specifying ACLs in entities are optional. To enable support for authorization, set falcon.security.authorization.enabled to true in the startup configuration.</p> +<div class="source"> +<pre> +# Authorization Enabled flag: false|true +*.falcon.security.authorization.enabled=true + +</pre></div></div> +<div class="section"> +<h4>Authorization Provider<a name="Authorization_Provider"></a></h4> +<p>Falcon provides a basic implementation for Authorization bundled, org.apache.falcon.security .DefaultFalconAuthorizationProvider. This can be overridden by custom implementations in the startup configuration.</p> +<div class="source"> +<pre> +# Authorization Provider Fully Qualified Class Name +*.falcon.security.authorization.provider=org.apache.falcon.security.DefaultAuthorizationProvider + +</pre></div></div> +<div class="section"> +<h4>Super User Group<a name="Super_User_Group"></a></h4> +<p>Super user group is determined by the configuration:</p> +<div class="source"> +<pre> +# The name of the group of super-users +*.falcon.security.authorization.superusergroup=falcon + +</pre></div></div> +<div class="section"> +<h4>Admin Membership<a name="Admin_Membership"></a></h4> +<p>Administrative users are determined by the configuration:</p> +<div class="source"> +<pre> +# Admin Users, comma separated users +*.falcon.security.authorization.admin.users=falcon,ambari-qa,seetharam + +</pre></div> +<p>Administrative groups are determined by the configuration:</p> +<div class="source"> +<pre> +# Admin Group Membership, comma separated users +*.falcon.security.authorization.admin.groups=falcon,testgroup,staff + +</pre></div></div> +<div class="section"> +<h3>Cross-Site Request Forgery (CSRF) Prevention<a name="Cross-Site_Request_Forgery_CSRF_Prevention"></a></h3> +<p>Cross-Site Request Forgery (CSRF) is an attack that forces an end user to execute unwanted actions on a web application in which they're currently authenticated. Falcon provides an option to prevent CSRF with Hadoop CSRF filter for REST APIs. By default, Falcon CSRF filter is disabled. To enable the support for CSRF prevention, set falcon.security.csrf.enabled to true in the startup configuration. We also provide options to configure custom header and browser user agents.</p> +<div class="source"> +<pre> +# CSRF filter enabled flag: false (default) | true +*.falcon.security.csrf.enabled=true +# Custom header for CSRF filter +*.falcon.security.csrf.header=FALCON-CSRF-FILTER +# Browser user agents to be filtered +*.falcon.security.csrf.browser=^Mozilla.*,^Opera.* + +</pre></div></div> +<div class="section"> +<h3>SSL<a name="SSL"></a></h3> +<p>Falcon provides transport level security ensuring data confidentiality and integrity. This is enabled by default for communicating over HTTP between the client and the server.</p></div> +<div class="section"> +<h4>SSL Configuration<a name="SSL_Configuration"></a></h4> +<div class="source"> +<pre> +*.falcon.enableTLS=true +*.keystore.file=/path/to/keystore/file +*.keystore.password=password + +</pre></div></div> +<div class="section"> +<h4>Distributed Falcon Setup<a name="Distributed_Falcon_Setup"></a></h4> +<p>Falcon should be configured to communicate with Prism over TLS in secure mode. Its not enabled by default.</p></div> +<div class="section"> +<h3>Changes to ownership and permissions of directories managed by Falcon<a name="Changes_to_ownership_and_permissions_of_directories_managed_by_Falcon"></a></h3> +<p></p> +<table border="0" class="table table-striped"> +<tr class="a"> +<th>Directory</th> +<th>Location</th> +<th>Owner</th> +<th>Permissions</th></tr> +<tr class="b"> +<td>Configuration Store</td> +<td>${config.store.uri}</td> +<td>falcon</td> +<td>700</td></tr> +<tr class="a"> +<td>Cluster Staging Location</td> +<td>${cluster.staging-location}</td> +<td>falcon</td> +<td>777</td></tr> +<tr class="b"> +<td>Cluster Working Location</td> +<td>${cluster.working-location}</td> +<td>falcon</td> +<td>755</td></tr> +<tr class="a"> +<td>Shared libs</td> +<td>{cluster.working}/{lib,libext}</td> +<td>falcon</td> +<td>755</td></tr> +<tr class="b"> +<td>Oozie coord/bundle XMLs</td> +<td>${cluster.staging-location}/workflows/{entity}/{entity-name}</td> +<td>$user</td> +<td>cluster umask</td></tr> +<tr class="a"> +<td>App logs</td> +<td>${cluster.staging-location}/workflows/{entity}/{entity-name}/logs</td> +<td>$user</td> +<td>cluster umask</td></tr></table><b>Note:</b> Please note that the cluster staging and working locations MUST be created prior to submitting a cluster entity to Falcon. Also, note that the the parent dirs must have execute permissions.</div> +<div class="section"> +<h3>Backwards compatibility<a name="Backwards_compatibility"></a></h3></div> +<div class="section"> +<h4>Scheduled Entities<a name="Scheduled_Entities"></a></h4> +<p>Entities already scheduled with an earlier version of Falcon are not compatible with this version</p></div> +<div class="section"> +<h4>Falcon Clients<a name="Falcon_Clients"></a></h4> +<p>Older Falcon clients are backwards compatible wrt Authentication and user information sent as part of the HTTP header, Remote-User is still honoured when the authentication type is configured as below:</p> +<div class="source"> +<pre> +*.falcon.http.authentication.type=org.apache.falcon.security.RemoteUserInHeaderBasedAuthenticationHandler + +</pre></div></div> +<div class="section"> +<h4>Blacklisted super users for authentication<a name="Blacklisted_super_users_for_authentication"></a></h4> +<p>The blacklist users used to have the following super users: hdfs, mapreduce, oozie, and falcon. The list is externalized from code into Startup.properties file and is empty now and needs to be configured specifically in the file.</p></div> +<div class="section"> +<h4>Falcon Dashboard<a name="Falcon_Dashboard"></a></h4> +<p>To initialize the current user for dashboard, user should append query param "user.name=<username>" to the REST api call.</p> +<p>If dashboard user wishes to change the current user, they should do the following.</p> +<ul> +<li>delete the hadoop.auth cookie from browser cache.</li> +<li>append query param "user.name=<new_user>" to the next REST API call.</li></ul> +<p>In Kerberos method, the browser must support HTTP Kerberos SPNEGO.</p></div> +<div class="section"> +<h3>Known Limitations<a name="Known_Limitations"></a></h3> +<p></p> +<ul> +<li>ActiveMQ topics are not secure but will be in the near future</li> +<li>Entities already scheduled with an earlier version of Falcon are not compatible with this version as new</li></ul>workflow parameters are being passed back into Falcon such as the user are required +<ul> +<li>Use of hftp as the scheme for read only interface in cluster entity <a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-10215";>will not work in Oozie</a></li></ul>The alternative is to use webhdfs scheme instead and its been tested with <a href="./DistCp.html">DistCp</a>.</div> +<div class="section"> +<h3>Examples<a name="Examples"></a></h3></div> +<div class="section"> +<h4>Accessing the server using Falcon CLI (Java client)<a name="Accessing_the_server_using_Falcon_CLI_Java_client"></a></h4> +<p>There is no change in the way the CLI is used. The CLI has been changed to work with the configured authentication method.</p></div> +<div class="section"> +<h4>Accessing the server using curl<a name="Accessing_the_server_using_curl"></a></h4> +<p>Try accessing protected resources using curl. The protected resources are:</p> +<div class="source"> +<pre> +$ kinit +Please enter the password for venkatesh@LOCALHOST: + +$ curl http://localhost:15000/api/admin/version + +$ curl http://localhost:15000/api/admin/version?user.name=venkatesh + +$ curl --negotiate -u foo -b ~/cookiejar.txt -c ~/cookiejar.txt curl http://localhost:15000/api/admin/version + +</pre></div></div> + </div> + </div> + + <hr/> + + <footer> + <div class="container"> + <div class="row span12">Copyright © 2013-2018 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + <p id="poweredBy" class="pull-right"> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </p> + + </div> + </footer> + </body> +</html>
