http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/HiveMirroring.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/HiveMirroring.twiki b/trunk/general/src/site/twiki/HiveMirroring.twiki new file mode 100644 index 0000000..e28502a --- /dev/null +++ b/trunk/general/src/site/twiki/HiveMirroring.twiki @@ -0,0 +1,63 @@ +---+Hive Mirroring + +---++Overview +Falcon provides feature to replicate Hive metadata and data events from source cluster to destination cluster. This is supported for both secure and unsecure cluster through Falcon extensions. + +---++Prerequisites +Following is the prerequisites to use Hive Mirrroring + + * *Hive 1.2.0+* + * *Oozie 4.2.0+* + +*Note:* Set following properties in hive-site.xml for replicating the Hive events on source and destination Hive cluster: +<verbatim> + <property> + <name>hive.metastore.event.listeners</name> + <value>org.apache.hive.hcatalog.listener.DbNotificationListener</value> + <description>event listeners that are notified of any metastore changes</description> + </property> + + <property> + <name>hive.metastore.dml.events</name> + <value>true</value> + </property> +</verbatim> + +---++ Use Case +* Replicate data/metadata of Hive DB & table from source to target cluster + +---++ Limitations +* Currently Hive doesn't support create database, roles, views, offline tables, direct HDFS writes without registering with metadata and Database/Table name mapping replication events. Hence Hive mirroring extension cannot be used to replicate above mentioned events between warehouses. + +---++ Usage +---+++ Bootstrap + Perform initial bootstrap of Table and Database from source cluster to destination cluster + * *Database Bootstrap* + For bootstrapping DB replication, first destination DB should be created. This step is expected, + since DB replication definitions can be set up by users only on pre-existing DBâs. Second, Export all tables in + the source db and Import it in the destination db, as described in Table bootstrap. + + * *Table Bootstrap* + For bootstrapping table replication, essentially after having turned on the !DbNotificationListener + on the source db, perform an Export of the table, distcp the Export over to the destination + warehouse and do an Import over there. Check the following [[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport][Hive Export-Import]] for syntax details + and examples. + This will set up the destination table so that the events on the source cluster that modify the table + will then be replicated. + +---+++ Setup source and destination clusters + <verbatim> + $FALCON_HOME/bin/falcon entity -submit -type cluster -file /cluster/definition.xml + </verbatim> + +---+++ Hive mirroring extension properties + Extension artifacts are expected to be installed on HDFS at the path specified by "extension.store.uri" in startup properties. hive-mirroring-properties.json file located at "<extension.store.uri>/hive-mirroring/META/hive-mirroring-properties.json" lists all the required and optional parameters/arguments for scheduling Hive mirroring job. + +---+++ Submit and schedule Hive mirroring extension + + <verbatim> + $FALCON_HOME/bin/falcon extension -submitAndSchedule -extensionName hive-mirroring -file /process/definition.xml + </verbatim> + + Please Refer to [[falconcli/FalconCLI][Falcon CLI]] and [[restapi/ResourceList][REST API]] for more details on usage of CLI and REST API's. +
http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/InstallationSteps.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/InstallationSteps.twiki b/trunk/general/src/site/twiki/InstallationSteps.twiki index a5ee2cc..297d88e 100644 --- a/trunk/general/src/site/twiki/InstallationSteps.twiki +++ b/trunk/general/src/site/twiki/InstallationSteps.twiki @@ -20,21 +20,22 @@ $git clone https://git-wip-us.apache.org/repos/asf/falcon.git falcon ---+++Step 2 - Build Falcon <verbatim> -$cd falcon -$export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m -noverify" && mvn clean install +$ cd falcon +$ export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m -noverify" +$ mvn clean install + </verbatim> It builds and installs the package into the local repository, for use as a dependency in other projects locally. -[optionally -Dhadoop.version=<<hadoop.version>> can be appended to build for a specific version of Hadoop] - -*NOTE:* Falcon drops support for Hadoop-1 and only supports Hadoop-2 from Falcon 0.6 onwards -[optionally -Doozie.version=<<oozie version>> can be appended to build with a specific version of Oozie. Oozie versions ->= 4 are supported] -NOTE: Falcon builds with JDK 1.7/1.8 using -noverify option - To compile Falcon with Hive Replication, optionally "-P hadoop-2,hivedr" can be appended. For this Hive >= 1.2.0 - and Oozie >= 4.2.0 should be available. +[optionally -Dhadoop.version=<<hadoop.version>> can be appended to build for a specific version of hadoop] +*Note 1:* Falcon drops support for Hadoop-1 and only supports Hadoop-2 from Falcon 0.6 onwards + Falcon build with JDK 1.7 using -noverify option +*Note 2:* To compile Falcon with addon extensions, append additional profiles to build command using syntax -P<<profile1,profile2>> + For Hive Mirroring extension, use profile"hivedr". Hive >= 1.2.0 and Oozie >= 4.2.0 is required + For HDFS Snapshot mirroring extension, use profile "hdfs-snapshot-mirroring". Hadoop >= 2.7.0 is required + For ADF integration, use profile "adf" ---+++Step 3 - Package and Deploy Falcon @@ -75,6 +76,8 @@ $src/bin/package.sh <<hadoop-version>> <<oozie-version>> >> ex. src/bin/package.sh 2.5.0 4.0.0 >> Falcon package is available in <<falcon >> home>>/target/apache-falcon-<<version>>-bin.tar.gz >> Oozie package is available in <<falcon >> home>>/target/oozie-4.0.1-distro.tar.gz +>> __IMPORTANT: You need to download the je-5.0.73 version from http://download.oracle.com/otn/berkeley-db/je-5.0.73.zip and extract je-5.0.73 under the Falcon webapp directory or provision an HBase cluster for use as Falcon graphdb backend DB. + Depending on the Graphdb backend choice, update the startup.properties appropriately.__ </verbatim> *NOTE:* If you have a separate Apache Oozie installation, you will need to follow some additional steps: http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/MigrationInstructions.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/MigrationInstructions.twiki b/trunk/general/src/site/twiki/MigrationInstructions.twiki index 7c0e027..a11dbc4 100644 --- a/trunk/general/src/site/twiki/MigrationInstructions.twiki +++ b/trunk/general/src/site/twiki/MigrationInstructions.twiki @@ -1,15 +1,32 @@ ---+ Migration Instructions ----++ Migrate from 0.5-incubating to 0.6-incubating +---++ Migrate from 0.9 to 0.10 -This is a placeholder wiki for migration instructions from falcon 0.5-incubating to 0.6-incubating. +FALCON-1333 (Instance Search feature) requires Falcon to use titan-berkeleyje version 0.5.4 to support indexing. +Up until version 0.9 - Falcon used titan-berkeleyje-jre6 version 0.4.2. GraphDB created by version 0.4.2 cannot be +read by version 0.5.4. The solution is to migrate the GraphDB to be compatible with Falcon 0.10 release. Please make +sure that no falcon server is running while performing the migration. ----+++ Update Entities +---+++ 1. Install Falcon 0.10 +Install Falcon 0.10 by following the [[InstallationSteps][Installation Steps]]. Do not start the falcon server yet. +The tool to migrate graphDB is packaged with 0.10 Falcon server in falcon-common-0.10.jar. ----+++ Change cluster dir permissions +---+++ 2. Export GraphDB to JSON file using Falcon 0.9 +Please run the following command to generate the JSON file. ----+++ Enable/Disable TLS +<verbatim> + $FALCON_HOME/bin/graphdbutil.sh export <<java_home> <<hadoop_home>> <<falcon_0.9_home>> <<path_to_falcon-common-0.10.jar>> /jsonFile/dir/ +</verbatim> ----+++ Authorization +This command will create /jsonFile/dir/instanceMetadata.json +---+++ 3. Import GraphDB from JSON file using Falcon 0.10 +Please run the following command to import graphDB the JSON file. The location of graphDB will be based on property +"*.falcon.graph.storage.directory" set in startup.properties file. + +<verbatim> + $FALCON_HOME/bin/graphdbutil.sh export <<java_home> <<hadoop_home>> <<falcon_0.10_home>> <<path_to_falcon-common-0.10.jar>> /jsonFile/dir/ +</verbatim> + +This command will import from /jsonFile/dir/instanceMetadata.json, now start the Falcon 0.10 server. http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/Operability.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/Operability.twiki b/trunk/general/src/site/twiki/Operability.twiki index 05850c1..2bccb51 100644 --- a/trunk/general/src/site/twiki/Operability.twiki +++ b/trunk/general/src/site/twiki/Operability.twiki @@ -72,22 +72,137 @@ or send alerts according to their requirements. ---++ Notifications -Falcon creates a JMS topic for every process/feed that is scheduled in Falcon. -The implementation class and the broker url of the JMS engine are read from the dependent cluster's definition. -Users may register consumers on the required topic to check the availability or status of feed instances. - -For a given process that is scheduled, the name of the topic is same as the process name. -Falcon sends a Map message for every feed produced by the instance of a process to the JMS topic. -The JMS !MapMessage sent to a topic has the following properties: -entityName, feedNames, feedInstancePath, workflowId, runId, nominalTime, timeStamp, brokerUrl, brokerImplClass, entityType, operation, logFile, topicName, status, brokerTTL; - -For a given feed that is scheduled, the name of the topic is same as the feed name. -Falcon sends a map message for every feed instance that is deleted/archived/replicated depending upon the retention policy set in the feed definition. -The JMS !MapMessage sent to a topic has the following properties: -entityName, feedNames, feedInstancePath, workflowId, runId, nominalTime, timeStamp, brokerUrl, brokerImplClass, entityType, operation, logFile, topicName, status, brokerTTL; - -The JMS messages are automatically purged after a certain period (default 3 days) by the Falcon JMS house-keeping service.TTL (Time-to-live) for JMS message -can be configured in the Falcon's startup.properties file. +Falcon has two types of notifications - System and User notifications. + +---+++ System notifications +The System notifications are internally generated and used by Falcon to monitor the Falcon orchestrated workflow jobs. +By default, Falcon starts an ActiveMQ embedded JMS server on Falcon machine on port 61616 as a daemon. Alternatively, +users can make Falcon to use an existing JMS server instead of starting an embedded instance by doing the +following 2 steps: + + * Setting the property broker.url in the startup.properties as below +<verbatim> + *.broker.url=tcp://jms-server-host:61616 +</verbatim> + * Set the system property falcon.embeddedmq to false as below +<verbatim> + <FALCON-INSTALL-DIR>/bin/falcon-start -Dfalcon.embeddedmq=false +</verbatim> + +Falcon uses FALCON.ENTITY.TOPIC to publish system notifications. This topic and the Map Message fields are internal +and could change between releases. + +---+++ User notifications + +Falcon, in addition to the FALCON.ENTITY.TOPIC, also creates a JMS topic for every process/feed that is scheduled in +Falcon as part of User notification. To enable User notifications, the broker url and implementation class of the JMS +engine need to be specified in the cluster definition associated with the feed/process. Users may register consumers +on the required topic to check the availability or status of feed instances. The User notification JMS broker instance +can be same as the System notification or different. + +The name of the JMS topic is same as the process/feed name. Falcon sends a map message for every feed instance that is +created/deleted/replicated/imported/exported to the JMS topic. The JMS Map Message sent to a topic has the following +fields: + + 1. cluster - name of the current cluster the feed/process is dependent on. + 1. entityType - type of the entity (feed or process). + 1. entityName - name of the entity. + 1. nominalTime - instance time (or data date). + 1. operation - operation like generate, delete, replicate, import, export. + 1. feedNames - name of the feeds which are generated/replicated/deleted/imported/exported. + 1. feedInstancePaths - comma separated feed instance paths. + 1. workflowId - current workflow-id of the instance. + 1. workflowUser - user who owns the feed instance (i.e partition). + 1. runId - current run-id of the instance. + 1. status - status of the user workflow instance. + 1. timeStamp - current timestamp. + 1. logDir - log dir where lineage can be recorded. + +The JMS messages are automatically purged after a certain period (default 3 days) by the Falcon JMS house-keeping +service. TTL (Time-to-live) for JMS message can be configured in the Falcon's startup.properties file. + +The following example shows how to enable and read user notification by connecting to the JMS broker. + +First, specify the JMS broker url in the cluster definition XML as shown below. + +<verbatim> + +<?xml version="1.0"?> +<!-- filename : primaryCluster.xml --> +<cluster colo="USWestOregon" description="oregonHadoopCluster" name="primaryCluster" xmlns="uri:falcon:cluster:0.1"> + <interfaces> + ... + ... + <interface type="messaging" endpoint="tcp://user-jms-broker-host:61616?daemon=true" version="5.1.6" /> + ... + </interfaces> +</cluster> + +</verbatim> + +Next, use a JMS consumer (example below in Java) to read the message from the topic with the name +FALCON.<feed-or-process-name> + +<verbatim> +import org.apache.activemq.ActiveMQConnectionFactory; +import org.apache.activemq.command.ActiveMQMapMessage; +import javax.jms.ConnectionFactory; +import javax.jms.Connection; +import javax.jms.MessageConsumer; +import javax.jms.Topic; +import javax.jms.Session; +import javax.jms.TopicSession; + +public class FalconUserJMSClient { + public static void main(String[] args)throws Exception { + // Note: specify the JMS broker URL + String brokerUrl = "tcp://localhost:61616"; + + ConnectionFactory connectionFactory = new ActiveMQConnectionFactory(brokerUrl); + Connection connection = connectionFactory.createConnection(); + connection.setClientID("Falcon User JMS Consumer"); + TopicSession session = (TopicSession) connection.createSession(false, Session.AUTO_ACKNOWLEDGE); + try { + + // Note: the topic name for the feed will be FALCON.<feed-name> + Topic falconTopic = session.createTopic("FALCON.feed-sample"); + MessageConsumer consumer = session.createConsumer(falconTopic); + connection.start(); + while (true) { + ActiveMQMapMessage msg = (ActiveMQMapMessage) consumer.receive(); + System.out.println("cluster : " + msg.getString("cluster")); + System.out.println("entityType : " + msg.getString("entityType")); + System.out.println("entityName : " + msg.getString("entityName")); + System.out.println("nominalTime : " + msg.getString("nominalTime")); + System.out.println("operation : " + msg.getString("operation")); + + System.out.println("feedNames : " + msg.getString("feedNames")); + System.out.println("feedInstancePaths : " + msg.getString("feedInstancePaths")); + + System.out.println("workflowId : " + msg.getString("workflowId")); + System.out.println("workflowUser : " + msg.getString("workflowUser")); + System.out.println("runId : " + msg.getString("runId")); + System.out.println("status : " + msg.getString("status")); + System.out.println("timeStamp : " + msg.getString("timeStamp")); + System.out.println("logDir : " + msg.getString("logDir")); + + System.out.println("brokerUrl : " + msg.getString("brokerUrl")); + System.out.println("brokerImplClass : " + msg.getString("brokerImplClass")); + System.out.println("logFile : " + msg.getString("logFile")); + System.out.println("topicName : " + msg.getString("topicName")); + System.out.println("brokerTTL : " + msg.getString("brokerTTL")); + } + } finally { + if (session != null) { + session.close(); + } + if (connection != null) { + connection.close(); + } + } + } +} +</verbatim> ---++ Alerts @@ -108,3 +223,8 @@ under ${user.dir}/logs/ directory. Users may also extend the Falcon Audit plugin to send audits to systems like Apache Argus, etc. by extending org.apache.falcon.plugin.AuditingPlugin interface. + + +---++ Metrics Collection In Graphite + +Falcon has support to send metrics to graphite more details regarding this can be found on [[GraphiteMetricCollection][Graphite Metric Collection]] \ No newline at end of file http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/Security.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/Security.twiki b/trunk/general/src/site/twiki/Security.twiki index 8955bdc..b17650c 100644 --- a/trunk/general/src/site/twiki/Security.twiki +++ b/trunk/general/src/site/twiki/Security.twiki @@ -2,8 +2,30 @@ ---++ Overview -Apache Falcon enforces authentication and authorization which are detailed below. Falcon also -provides transport level security ensuring data confidentiality and integrity. +Apache Falcon provides the following security features: + * Support credential provider alias for passwords used in Falcon server. + * Support authentication to identify proper users. + * Support authorization to specify resource access permission for users or groups. + * Support SSL to provide transport level security for data confidentiality and integrity. + + +---++ Credential Provider Alias for Passwords +Server-side configuration properties (i.e. startup.properties) contain passwords and other sensitive information. +In addition to specifying properties in plain text, we provide the user an option to use credential provider alias in the property file. + +Take SMTP password for example. The user can store the password in a +[[http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CommandsManual.html#credential][Hadoop credential provider]] +with the alias name _SMTPPasswordAlias_. In startup.properties where SMTP password is needed, the user can refer to its +alias name _SMTPPasswordAlias_ instead of providing the real password. + +The alias property to be resolved through Hadoop credential provider should have the format: +_credential.provider.alias.for.[property-key]_. For example, +_credential.provider.alias.for.falcon.email.smtp.password=SMTPPasswordAlias_ for SMTP password. +Falcon server, during the start, will automatically retrieve the real password provided the alias name. + +The user can specify the provider path with the property key _credential.provider.path_, +e.g. _credential.provider.path=jceks://file/tmp/test.jceks_. +If not specified, Falcon will use the default Hadoop credential provider path in core-site.xml. ---++ Authentication (User Identity) http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/DefineExtension.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/falconcli/DefineExtension.twiki b/trunk/general/src/site/twiki/falconcli/DefineExtension.twiki new file mode 100644 index 0000000..c260911 --- /dev/null +++ b/trunk/general/src/site/twiki/falconcli/DefineExtension.twiki @@ -0,0 +1,8 @@ +---+++Definition + +[[CommonCLI][Common CLI Options]] + +Definition of an extension. Outputs a JSON document describing the extension invocation parameters. + +Usage: +$FALCON_HOME/bin/falcon extension Âdefinition Âname <extensionÂname> http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/DeleteEntity.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/falconcli/DeleteEntity.twiki b/trunk/general/src/site/twiki/falconcli/DeleteEntity.twiki index f2b3080..cc07406 100644 --- a/trunk/general/src/site/twiki/falconcli/DeleteEntity.twiki +++ b/trunk/general/src/site/twiki/falconcli/DeleteEntity.twiki @@ -2,7 +2,7 @@ [[CommonCLI][Common CLI Options]] -Delete removes the submitted entity definition for the specified entity and put it into the archive. +Delete removes the submitted entity definition for the specified entity and put it into the archive.Archive path is defined in startup.properties in variable "config.store.uri". Usage: $FALCON_HOME/bin/falcon entity -type [cluster|datasource|feed|process] -name <<name>> -delete http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/DescribeExtension.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/falconcli/DescribeExtension.twiki b/trunk/general/src/site/twiki/falconcli/DescribeExtension.twiki new file mode 100644 index 0000000..9f9895e --- /dev/null +++ b/trunk/general/src/site/twiki/falconcli/DescribeExtension.twiki @@ -0,0 +1,8 @@ +---+++Describe + +[[CommonCLI][Common CLI Options]] + +Description of an extension. Outputs the README of the specified extension. + +Usage: +$FALCON_HOME/bin/falcon extension Âdescribe Âname <extensionÂname> http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/EnumerateExtension.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/falconcli/EnumerateExtension.twiki b/trunk/general/src/site/twiki/falconcli/EnumerateExtension.twiki new file mode 100644 index 0000000..0b28630 --- /dev/null +++ b/trunk/general/src/site/twiki/falconcli/EnumerateExtension.twiki @@ -0,0 +1,8 @@ +---+++Enumerate + +[[CommonCLI][Common CLI Options]] + +List all the extensions supported. Returns total number of results and a list of server side extensions supported. + +Usage: +$FALCON_HOME/bin/falcon extension Âenumerate \ No newline at end of file http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/ExtensionDelete.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/falconcli/ExtensionDelete.twiki b/trunk/general/src/site/twiki/falconcli/ExtensionDelete.twiki new file mode 100644 index 0000000..bb867b5 --- /dev/null +++ b/trunk/general/src/site/twiki/falconcli/ExtensionDelete.twiki @@ -0,0 +1,15 @@ +---+++Delete + +[[CommonCLI][Common CLI Options]] + +Delete an extension job. + +Usage: +$FALCON_HOME/bin/falcon extension -delete -jobName <<extension-job-name>> + +Optional Args : -doAs <<user-name>> + +<a href="../restapi/ExtensionDelete.html">Parameters and examples described here.</a> + +Example: +$FALCON_HOME/bin/falcon extension -delete -jobName sales-monthly http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/ExtensionInstances.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/falconcli/ExtensionInstances.twiki b/trunk/general/src/site/twiki/falconcli/ExtensionInstances.twiki new file mode 100644 index 0000000..765a5bf --- /dev/null +++ b/trunk/general/src/site/twiki/falconcli/ExtensionInstances.twiki @@ -0,0 +1,18 @@ +---+++Instances + +[[CommonCLI][Common CLI Options]] + +List instances of an extension job. + +Usage: +$FALCON_HOME/bin/falcon extension -instances -jobName <<extension-job-name>> + +Optional Args : -doAs <<user-name>> +-start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" -fields <<field1,field2>> +-instanceStatus <<instance-status>> -orderBy <<order-by-field>> -sortOrder <<sort-order>> +-offset <<offset-number>> -numResults <<number-of-results>> + +<a href="../restapi/ExtensionInstances.html">Parameters and examples described here.</a> + +Example: +$FALCON_HOME/bin/falcon extension -instances -jobName daily-health-bill -start 2012-04-01T00:00 http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/ExtensionList.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/falconcli/ExtensionList.twiki b/trunk/general/src/site/twiki/falconcli/ExtensionList.twiki new file mode 100644 index 0000000..8ddf573 --- /dev/null +++ b/trunk/general/src/site/twiki/falconcli/ExtensionList.twiki @@ -0,0 +1,16 @@ +---+++List + +[[CommonCLI][Common CLI Options]] + +List jobs generated from an extension. + +Usage: +$FALCON_HOME/bin/falcon extension -list -extensionName <<extension-name>> + +Optional Args : -doAs <<user-name>> +-sortOrder <<sortOrder>> -offset <<offset-number>> -numResults <<number-of-results>> -fields <<field1,field2>> + +<a href="../restapi/ExtensionList.html">Parameters and examples described here.</a> + +Example: +$FALCON_HOME/bin/falcon extension -list -extensionName hdfs-mirroring -offset 0 -numResults 10 http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/ExtensionResume.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/falconcli/ExtensionResume.twiki b/trunk/general/src/site/twiki/falconcli/ExtensionResume.twiki new file mode 100644 index 0000000..4a62d04 --- /dev/null +++ b/trunk/general/src/site/twiki/falconcli/ExtensionResume.twiki @@ -0,0 +1,15 @@ +---+++Resume + +[[CommonCLI][Common CLI Options]] + +Resume an extension job. + +Usage: +$FALCON_HOME/bin/falcon extension -resume -jobName <<extension-job-name>> + +Optional Args : -doAs <<user-name>> + +<a href="../restapi/ExtensionResume.html">Parameters and examples described here.</a> + +Example: +$FALCON_HOME/bin/falcon extension -resume -jobName sales-monthly http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/ExtensionSchedule.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/falconcli/ExtensionSchedule.twiki b/trunk/general/src/site/twiki/falconcli/ExtensionSchedule.twiki new file mode 100644 index 0000000..813d7b7 --- /dev/null +++ b/trunk/general/src/site/twiki/falconcli/ExtensionSchedule.twiki @@ -0,0 +1,15 @@ +---+++Schedule + +[[CommonCLI][Common CLI Options]] + +Schedule an extension job. + +Usage: +$FALCON_HOME/bin/falcon extension -schedule -jobName <<extension-job-name>> + +Optional Args : -doAs <<user-name>> + +<a href="../restapi/ExtensionSchedule.html">Parameters and examples described here.</a> + +Example: +$FALCON_HOME/bin/falcon extension -schedule -jobName sales-monthly http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/ExtensionSubmit.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/falconcli/ExtensionSubmit.twiki b/trunk/general/src/site/twiki/falconcli/ExtensionSubmit.twiki new file mode 100644 index 0000000..40a7b44 --- /dev/null +++ b/trunk/general/src/site/twiki/falconcli/ExtensionSubmit.twiki @@ -0,0 +1,15 @@ +---+++Submit + +[[CommonCLI][Common CLI Options]] + +Submit an extension job. + +Usage: +$FALCON_HOME/bin/falcon extension -submit -extensionName <<extension-name>> -file <<path-to-file>> + +Optional Args : -doAs <<user-name>> + +<a href="../restapi/ExtensionSubmit.html">Parameters and examples described here.</a> + +Example: +$FALCON_HOME/bin/falcon extension -submit -extensionName hdfs-mirroring -file /data/hdfs-mirroring-para.txt http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/ExtensionSubmitAndSchedule.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/falconcli/ExtensionSubmitAndSchedule.twiki b/trunk/general/src/site/twiki/falconcli/ExtensionSubmitAndSchedule.twiki new file mode 100644 index 0000000..ea1c0e2 --- /dev/null +++ b/trunk/general/src/site/twiki/falconcli/ExtensionSubmitAndSchedule.twiki @@ -0,0 +1,15 @@ +---+++SubmitAndSchedule + +[[CommonCLI][Common CLI Options]] + +Submit and schedule an extension job. + +Usage: +$FALCON_HOME/bin/falcon extension -submitAndSchedule -extensionName <<extension-name>> -file <<path-to-file>> + +Optional Args : -doAs <<user-name>> + +<a href="../restapi/ExtensionSubmitAndSchedule.html">Parameters and examples described here.</a> + +Example: +$FALCON_HOME/bin/falcon extension -submitAndSchedule -extensionName hdfs-mirroring -file /data/hdfs-mirroring-para.txt http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/ExtensionSuspend.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/falconcli/ExtensionSuspend.twiki b/trunk/general/src/site/twiki/falconcli/ExtensionSuspend.twiki new file mode 100644 index 0000000..dd6b69c --- /dev/null +++ b/trunk/general/src/site/twiki/falconcli/ExtensionSuspend.twiki @@ -0,0 +1,15 @@ +---+++Suspend + +[[CommonCLI][Common CLI Options]] + +Suspend an extension job. + +Usage: +$FALCON_HOME/bin/falcon extension -suspend -jobName <<extension-job-name>> + +Optional Args : -doAs <<user-name>> + +<a href="../restapi/ExtensionSuspend.html">Parameters and examples described here.</a> + +Example: +$FALCON_HOME/bin/falcon extension -suspend -jobName sales-monthly http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/ExtensionUpdate.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/falconcli/ExtensionUpdate.twiki b/trunk/general/src/site/twiki/falconcli/ExtensionUpdate.twiki new file mode 100644 index 0000000..fa0985e --- /dev/null +++ b/trunk/general/src/site/twiki/falconcli/ExtensionUpdate.twiki @@ -0,0 +1,15 @@ +---+++Update + +[[CommonCLI][Common CLI Options]] + +Update an extension entity. Note that we require all the entity names to remain the same. Future work will allow different entity names. + +Usage: +$FALCON_HOME/bin/falcon extension -update -extensionName <<extension-name>> -file <<path-to-file>> + +Optional Args : -doAs <<user-name>> + +<a href="../restapi/ExtensionUpdate.html">Parameters and examples described here.</a> + +Example: +$FALCON_HOME/bin/falcon extension -update -extensionName hdfs-mirroring -file /data/hdfs-mirroring-para.txt http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/ExtensionValidate.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/falconcli/ExtensionValidate.twiki b/trunk/general/src/site/twiki/falconcli/ExtensionValidate.twiki new file mode 100644 index 0000000..d788a20 --- /dev/null +++ b/trunk/general/src/site/twiki/falconcli/ExtensionValidate.twiki @@ -0,0 +1,15 @@ +---+++Validate + +[[CommonCLI][Common CLI Options]] + +Validate the input for extension job submission. + +Usage: +$FALCON_HOME/bin/falcon extension -validate -extensionName <<extension-name>> -file <<path-to-file>> + +Optional Args : -doAs <<user-name>> + +<a href="../restapi/ExtensionValidate.html">Parameters and examples described here.</a> + +Example: +$FALCON_HOME/bin/falcon extension -validate -extensionName hdfs-mirroring -file /data/hdfs-mirroring-para.txt http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/FalconCLI.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/falconcli/FalconCLI.twiki b/trunk/general/src/site/twiki/falconcli/FalconCLI.twiki index 0c0082f..9ae0646 100644 --- a/trunk/general/src/site/twiki/falconcli/FalconCLI.twiki +++ b/trunk/general/src/site/twiki/falconcli/FalconCLI.twiki @@ -11,7 +11,8 @@ CLI options are classified into : * <a href="#Instance_Management_Commands">Instance Management Commands</a> * <a href="#Metadata_Commands">Metadata Commands</a> * <a href="#Admin_Commands">Admin commands</a> - * <a href="#Recipe_Commands">Recipe commands</a> + * <a href="#Extension_Artifacts_Commands">Extension artifacts commands</a> + * <a href="#Extension_Commands">Extension commands</a> @@ -68,7 +69,8 @@ $FALCON_HOME/bin/falcon entity -submit -type cluster -file /cluster/definition.x | [[ContinueInstance][Continue]] | Continue the failed workflow instances | | [[RerunInstance][Rerun]] | Rerun instances of specified process | | [[ResumeInstance][Resume]] | Resume instance of specified process from suspended state | -| [[StatusInstance][Status]] | Gets the status of entity | +| [[SearchInstance][Search]] | Search instances | +| [[StatusInstance][Status]] | Gets the status of instances | | [[ListInstance][List]] | Gets single or multiple instances | | [[SummaryInstance][Summary]] | Gets consolidated status of the instances between the specified time period | | [[RunningInstance][Running]] | Gets running instances of the mentioned process | @@ -100,13 +102,29 @@ $FALCON_HOME/bin/falcon entity -submit -type cluster -file /cluster/definition.x |[[HelpAdmin][Help]] | Return help options | |[[VersionAdmin][Version]] | Return current falcon version | |[[StatusAdmin][Status]] | Return the status of falcon | +|[[SafemodeAdmin][SetSafeMode]] | Set/unset Falcon Server in safemode | ----------- ----++Recipe Commands +---++Extension Artifacts Management Commands | *Command* | *Description* | -|[[SubmitRecipe][Submit]] | Submit the specified Recipe | - +|[[EnumerateExtension][Enumerate]] | Return all the extensions supported | +|[[DescribeExtension][Describe]] | Return description of an extension | +|[[DefineExtension][Definition]] | Return the definition of an extension | +----------- +---++Extension Job Management Commands + +| *Command* | *Description* | +|[[ExtensionList][List]] | List jobs generated from an extension | +|[[ExtensionInstances][Instances]] | List instances of an extension job | +|[[ExtensionValidate][Validate]] | Validate an extension job | +|[[ExtensionSubmit][Submit]] | Submit an extension job | +|[[ExtensionUpdate][Update]] | Update an extension job | +|[[ExtensionSubmitAndSchedule][SubmitAndSchedule]] | Submit and schedule an extension job | +|[[ExtensionSchedule][Schedule]] | Schedule an extension job | +|[[ExtensionSuspend][Suspend]] | Suspend an extension job | +|[[ExtensionResume][Resume]] | Resume an extension job | +|[[ExtensionDelete][Delete]] | Delete an extension job | http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/SafemodeAdmin.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/falconcli/SafemodeAdmin.twiki b/trunk/general/src/site/twiki/falconcli/SafemodeAdmin.twiki new file mode 100644 index 0000000..5eb26c0 --- /dev/null +++ b/trunk/general/src/site/twiki/falconcli/SafemodeAdmin.twiki @@ -0,0 +1,10 @@ +---+++SetSafeMode + +[[CommonCLI][Common CLI Options]] + +When -setsafemode is true, Falcon starts in safemode. +When -setsafemode is false, falcon starts in normal mode. + +Usage: +$FALCON_HOME/bin/falcon admin -setsafemode true/false + http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/SearchInstance.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/falconcli/SearchInstance.twiki b/trunk/general/src/site/twiki/falconcli/SearchInstance.twiki new file mode 100644 index 0000000..35cac74 --- /dev/null +++ b/trunk/general/src/site/twiki/falconcli/SearchInstance.twiki @@ -0,0 +1,17 @@ +---+++Search + +[[CommonCLI][Common CLI Options]] + +Search option via CLI can be used to get instances that meet search criteria, e.g. nominal time, instance status, entity name subsequence, entity tag, etc. +Instance name and status are listed in the output. More instance properties will be added to the output in the future (e.g. log url, running time). +Example usage include: search of instances of all entities with the same tag; search of all current running instances. + +Usage: +$FALCON_HOME/bin/falcon instance -search + +Optional Args : -type <<[feed|process|datasource],[feed|process|datasource]>> +-nameseq <<namesubsequence>> -tagkeys <<tagkeyword1,tagkeyword2>> +-instanceStatus <<status>> -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" +-orderBy <<field>> -offset 0 -numResults 10 + +<a href="../restapi/InstanceSearch.html">Optional params and examples described here.</a> http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/UpdateEntity.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/falconcli/UpdateEntity.twiki b/trunk/general/src/site/twiki/falconcli/UpdateEntity.twiki index ae60559..146a60f 100644 --- a/trunk/general/src/site/twiki/falconcli/UpdateEntity.twiki +++ b/trunk/general/src/site/twiki/falconcli/UpdateEntity.twiki @@ -2,13 +2,15 @@ [[CommonCLI][Common CLI Options]] -Update operation allows an already submitted/scheduled entity to be updated. Cluster and datasource updates are -currently not allowed. +Update operation allows an already submitted/scheduled entity to be updated and put it into the archive.Archive path is defined in startup.properties in variable "config.store.uri". Datasource updates are currently not allowed. Usage: -$FALCON_HOME/bin/falcon entity -type [feed|process] -name <<name>> -update -file <<path_to_file>> +$FALCON_HOME/bin/falcon entity -type [cluster|feed|process] -name <<name>> -update -file <<path_to_file>> Optional Arg : -skipDryRun. When this argument is specified, Falcon skips oozie dryrun. Example: $FALCON_HOME/bin/falcon entity -type process -name hourly-reports-generator -update -file /process/definition.xml + +Note: When a cluster entity is updated, the dependent feed and process bundle+coordinators are updated in the +workflow engine. Hence, only a falcon superuser who has ability to impersonate other users can update a cluster entity. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/AdminSafemode.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/restapi/AdminSafemode.twiki b/trunk/general/src/site/twiki/restapi/AdminSafemode.twiki new file mode 100644 index 0000000..d7c9631 --- /dev/null +++ b/trunk/general/src/site/twiki/restapi/AdminSafemode.twiki @@ -0,0 +1,25 @@ +---++ GET /api/admin/setSafeMode/:mode + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Set falcon server into/out of safemode. + +---++ Parameters + * :mode true/false. + * doAs <optional query param> allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. + +---++ Results +The value of FalconSever safemode. + +---++ Examples +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/admin/setSafeMode/true +</verbatim> +---+++ Result +<verbatim> +true +</verbatim> http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/AdminVersion.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/restapi/AdminVersion.twiki b/trunk/general/src/site/twiki/restapi/AdminVersion.twiki index 7db2d8f..d7615a0 100644 --- a/trunk/general/src/site/twiki/restapi/AdminVersion.twiki +++ b/trunk/general/src/site/twiki/restapi/AdminVersion.twiki @@ -29,6 +29,14 @@ GET http://localhost:15000/api/admin/version?doAs=joe { "key":"Mode", "value":"embedded" + }, + { + "key":"authentication", + "value":"simple" + }, + { + "key":"safemode", + "value":"false" } ] } http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/EntityList.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/restapi/EntityList.twiki b/trunk/general/src/site/twiki/restapi/EntityList.twiki index 2c2a734..a439dc7 100644 --- a/trunk/general/src/site/twiki/restapi/EntityList.twiki +++ b/trunk/general/src/site/twiki/restapi/EntityList.twiki @@ -1,4 +1,4 @@ ----++ GET /api/entities/list/:entity-type?fields=:fields +---++ GET /api/entities/list/{:entity-type} * <a href="#Description">Description</a> * <a href="#Parameters">Parameters</a> * <a href="#Results">Results</a> @@ -8,7 +8,7 @@ Get list of the entities. ---++ Parameters - * :entity-type Comma-separated entity types. Can be empty. Valid entity types are cluster, feed or process. + * :entity-type <optional param> Comma-separated entity types. Valid entity types are cluster, feed or process. * fields <optional param> Fields of entity that the user wants to view, separated by commas. * Valid options are STATUS, TAGS, PIPELINES, CLUSTERS. * nameseq <optional param> Subsequence of entity name. Not case sensitive. @@ -38,20 +38,28 @@ Total number of results and a list of entities. ---++ Examples ---+++ Rest Call <verbatim> -GET http://localhost:15000/api/entities/list/feed +GET http://localhost:15000/api/entities/list </verbatim> ---+++ Result <verbatim> { - "totalResults":"2â, + "totalResults":"4â, "entity": [ { - "name": "SampleOutput", - "type": "feed" + "name" : "SampleCluster1", + "type" : "cluster" + } + { + "name" : "SampleOutput", + "type" : "feed" }, { - "name": "SampleInput", - "type": "feed" + "name" : "SampleInput", + "type" : "feed" + } + { + "name" : "SampleProcess1", + "type" : "process" } ] } @@ -59,28 +67,20 @@ GET http://localhost:15000/api/entities/list/feed ---+++ Rest Call <verbatim> -GET http://localhost:15000/api/entities/list +GET http://localhost:15000/api/entities/list/feed </verbatim> ---+++ Result <verbatim> { - "totalResults":"4â, + "totalResults":"2â, "entity": [ { - "name" : "SampleCluster1", - "type" : "cluster" - } - { - "name" : "SampleOutput", - "type" : "feed" + "name": "SampleOutput", + "type": "feed" }, { - "name" : "SampleInput", - "type" : "feed" - } - { - "name" : "SampleProcess1", - "type" : "process" + "name": "SampleInput", + "type": "feed" } ] } http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/EntityUpdate.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/restapi/EntityUpdate.twiki b/trunk/general/src/site/twiki/restapi/EntityUpdate.twiki index 46b01fc..cbf33db 100644 --- a/trunk/general/src/site/twiki/restapi/EntityUpdate.twiki +++ b/trunk/general/src/site/twiki/restapi/EntityUpdate.twiki @@ -8,8 +8,8 @@ Updates the submitted entity. ---++ Parameters - * :entity-type can be feed or process. - * :entity-name is name of the feed or process. + * :entity-type can be cluster, feed or process. + * :entity-name is name of the cluster, feed or process. * skipDryRun : Optional query param, Falcon skips oozie dryrun when value is set to true. * doAs <optional query param> allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionDefinition.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/restapi/ExtensionDefinition.twiki b/trunk/general/src/site/twiki/restapi/ExtensionDefinition.twiki new file mode 100644 index 0000000..66f6674 --- /dev/null +++ b/trunk/general/src/site/twiki/restapi/ExtensionDefinition.twiki @@ -0,0 +1,160 @@ +---++ GET api/extension/definition/:extensionÂname + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Get definition of the extension. + +---++ Parameters + * :extensionÂname Name of the extension. + +---++ Results +Outputs a JSON document describing the extension invocation parameters. + +---++ Examples +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/extension/definition/hdfsÂmirroring +</verbatim> +---+++ Result +<verbatim> +{ + "shortDescription": "This extension implements replicating arbitrary directories on HDFS from one Hadoop cluster to another Hadoop cluster. This piggy backs on replication solution in Falcon which uses the DistCp tool.", + "properties":[ + { + "propertyName":"jobName", + "required":true, + "description":"Unique job name", + "example":"hdfs-monthly-sales-dr" + }, + { + "propertyName":"jobClusterName", + "required":true, + "description":"Cluster where job should run", + "example":"backupCluster" + }, + { + "propertyName":"jobValidityStart", + "required":true, + "description":"Job validity start time", + "example":"2016-03-03T00:00Z" + }, + { + "propertyName":"jobValidityEnd", + "required":true, + "description":"Job validity end time", + "example":"2018-03-13T00:00Z" + }, + { + "propertyName":"jobFrequency", + "required":true, + "description":"job frequency. Valid frequency types are minutes, hours, days, months", + "example":"months(1)" + }, + { + "propertyName":"jobTimezone", + "required":false, + "description":"Time zone for the job", + "example":"GMT" + }, + { + "propertyName":"jobTags", + "required":false, + "description":"list of comma separated tags. Key Value Pairs, separated by comma", + "example":"[email protected], [email protected], _department_type=forecasting" + }, + { + "propertyName":"jobRetryPolicy", + "required":false, + "description":"Job retry policy", + "example":"periodic" + }, + { + "propertyName":"jobRetryDelay", + "required":false, + "description":"Job retry delay", + "example":"minutes(30)" + }, + { + "propertyName":"jobRetryAttempts", + "required":false, + "description":"Job retry attempts", + "example":"3" + }, + { + "propertyName":"jobRetryOnTimeout", + "required":false, + "description":"Job retry on timeout", + "example":"true" + }, + { + "propertyName":"jobAclOwner", + "required":false, + "description":"ACL owner", + "example":"ambari-qa" + }, + { + "propertyName":"jobAclGroup", + "required":false, + "description":"ACL group", + "example":"users" + }, + { + "propertyName":"jobAclPermission", + "required":false, + "description":"ACL permission", + "example":"0x755" + }, + { + "propertyName":"sourceDir", + "required":true, + "description":"Multiple hdfs comma separated source directories", + "example":"/user/ambari-qa/primaryCluster/dr/input1, /user/ambari-qa/primaryCluster/dr/input2" + }, + { + "propertyName":"sourceCluster", + "required":true, + "description":"Source cluster for hdfs mirroring", + "example":"primaryCluster" + }, + { + "propertyName":"targetDir", + "required":true, + "description":"Target hdfs directory", + "example":"/user/ambari-qa/backupCluster/dr" + }, + { + "propertyName":"targetCluster", + "required":true, + "description":"Target cluster for hdfs mirroring", + "example":"backupCluster" + }, + { + "propertyName":"distcpMaxMaps", + "required":false, + "description":"Maximum number of mappers for DistCP", + "example":"1" + }, + { + "propertyName":"distcpMapBandwidth", + "required":false, + "description":"Bandwidth in MB for each mapper in DistCP", + "example":"100" + }, + { + "propertyName":"jobNotificationType", + "required":false, + "description":"Email Notification for Falcon instance completion", + "example":"email" + }, + { + "propertyName":"jobNotificationReceivers", + "required":false, + "description":"Comma separated email Id's", + "example":"[email protected], [email protected]" + } + ] +} +</verbatim> http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionDelete.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/restapi/ExtensionDelete.twiki b/trunk/general/src/site/twiki/restapi/ExtensionDelete.twiki new file mode 100644 index 0000000..2bdd23c --- /dev/null +++ b/trunk/general/src/site/twiki/restapi/ExtensionDelete.twiki @@ -0,0 +1,29 @@ +---++ POST api/extension/delete/:job-name + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Delete an extension job. + +---++ Parameters + * :job-name Name of the extension job. + * doAs <optional> Impersonate the user. + +---++ Results +Result of the delete operation. + +---++ Examples +---+++ Rest Call +<verbatim> +POST http://localhost:15000/api/extensions/delete/sales-monthly +</verbatim> +---+++ Result +<verbatim> +{ + "requestId": "e5cc8230-f356-4566-9b65-536abdff8aa3", + "message": "Extension job sales-monthly deleted successfully", + "status": "SUCCEEDED" +} +</verbatim> http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionDescription.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/restapi/ExtensionDescription.twiki b/trunk/general/src/site/twiki/restapi/ExtensionDescription.twiki new file mode 100644 index 0000000..5900fbb --- /dev/null +++ b/trunk/general/src/site/twiki/restapi/ExtensionDescription.twiki @@ -0,0 +1,24 @@ +---++ GET api/extension/describe/:extensionÂname + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Description of an extension. + +---++ Parameters + * :extensionÂname Name of the extension. + +---++ Results +Outputs the README of the specified extension. + +---++ Examples +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/extension/describe/hdfsÂmirroring +</verbatim> +---+++ Result +<verbatim> +<README file of the specified extension> +</verbatim> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionEnumeration.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/restapi/ExtensionEnumeration.twiki b/trunk/general/src/site/twiki/restapi/ExtensionEnumeration.twiki new file mode 100644 index 0000000..abd94c8 --- /dev/null +++ b/trunk/general/src/site/twiki/restapi/ExtensionEnumeration.twiki @@ -0,0 +1,38 @@ +---++ GET api/extension/enumerate + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Get list of the supported extensions. + +---++ Parameters +None + +---++ Results +Total number of results and a list of extension server extensions supported. + +---++ Examples +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/extension/enumerate +</verbatim> +---+++ Result +<verbatim> +{ + "totalResults":"2â, + âextensionsâ: [ + { + ânameâ: âHdfsÂmirroringâ + âtypeâ: âTrusted/Provided extensionâ + âdescriptionâ: âThis extension implements replicating arbitrary directories on HDFS from one Hadoop cluster to another Hadoop cluster.â + }, + { + ânameâ: âHiveÂmirroringâ + âtypeâ: âTrusted/Provided extensionâ + âdescriptionâ: âThis extension implements replicating hive metadata and data from one Hadoop cluster to another Hadoop cluster.â + } + ] +} +</verbatim> http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionInstances.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/restapi/ExtensionInstances.twiki b/trunk/general/src/site/twiki/restapi/ExtensionInstances.twiki new file mode 100644 index 0000000..a11a97a --- /dev/null +++ b/trunk/general/src/site/twiki/restapi/ExtensionInstances.twiki @@ -0,0 +1,87 @@ +---++ GET api/extension/instances/:job-name + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +List instances of an extension job. + +---++ Parameters + * :job-name Name of the extension job. + * start <optional> Show instances from this nominal time. Format: yyyy-MM-dd'T'HH:mm'Z'. + * end <optional> Show instances up to this nominal time. Format: yyyy-MM-dd'T'HH:mm'Z'. + * instanceStatus <optional> Valid options are RUNNING, SUCCEEDED, FAILED, WAITING, SUSPENDED, KILLED, TIMEOUT. + * fields <optional> Entity output fields separated by commas. Valid options: STATUS, TAGS, PIPELINES, CLUSTERS. + * orderBy <optional> Column by which results should be ordered. Sorted by descending order. Valid options are nominalTime (default) and status. + * sortOrder <optional> Valid options are âascâ and âdescâ. + * offset <optional> Show results from the offset. Used for pagination. Default is 0. + * numResults <optional> Number of instances per entity to show. Default value is 10. + * doAs <optional> Impersonate the user. + +---++ Results +A list of entities of the job, each followed by a set of instances. + +---++ Examples +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/extensions/instances/daily-health-bill?start=2012-04-01T00:00 +</verbatim> +---+++ Result +<verbatim> +{ + "numEntities": "2", + "entitiesSummary": { + "entitySummary": [ + { + "entityProfile": { + "type": "FEED", + "name": "SampleUSHealthBill" + }, + "instances": [ + { + "instance": "2012-04-02T00:00Z", + "status": "RUNNING", + "cluster": "SampleCluster1", + "sourceCluster": "primaryCluster", + "details": "hftp://sandbox.hortonworks.com:50070/apps/falcon/adf-demo/pig-input-2012-04-02"; + }, + { + "instance": "2012-04-01T00:00Z", + "status": "SUCCEEDED", + "cluster": "SampleCluster1", + "sourceCluster": "primaryCluster", + "details": "hftp://sandbox.hortonworks.com:50070/apps/falcon/adf-demo/pig-input-2012-04-01"; + } + ] + }, + { + "entityProfile": { + "type": "PROCESS", + "name": "SampleBillPay" + }, + "instances": [ + { + "instance": "2012-04-02T00:00Z", + "status": "RUNNING", + "logFile": "http://sandbox.hortonworks.com:11000/oozie?job=0000085-160322203213815-oozie-oozi-W";, + "cluster": "primaryCluster", + "startTime": "2016-04-04T22:37:46Z", + "details": "", + "actions": { + "action": "dr-replication", + "status": "RUNNING", + "logFile": "http://sandbox.hortonworks.com:8088/proxy/application_1458678764964_0052/"; + } + }, + { + "instance": "2012-04-01T00:00Z", + "status": "SKIPPED", + "cluster": "primaryCluster" + } + ] + } + ] + } +} +</verbatim> http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionList.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/restapi/ExtensionList.twiki b/trunk/general/src/site/twiki/restapi/ExtensionList.twiki new file mode 100644 index 0000000..dcd4d8d --- /dev/null +++ b/trunk/general/src/site/twiki/restapi/ExtensionList.twiki @@ -0,0 +1,71 @@ +---++ GET api/extension/list/:extension-name + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +List jobs generated from an extension. + +---++ Parameters + * :extension-name Name of the extension. + * sortOrder <optional> Sort order by job name. Valid options: "asc" (default) and "desc". + * offset <optional> Show results from the offset. Default is 0. + * numResults <optional> Number of results to show per request. Default is 10. + * fields <optional> Output fields separated by commas. Valid options: STATUS, TAGS, PIPELINES, CLUSTERS. + * doAs <optional> Impersonate the user. + +---++ Results +Total number of results and a list of jobs generated from the extension, followed by the associated entities. + +---++ Examples +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/extensions/list/billCollection?fields=status,clusters,tags +</verbatim> +---+++ Result +<verbatim> +{ + "numJobs":"2", + "jobs":{ + "job": [ + { + "jobName": "daily-health-bill", + "jobEntities: { + "totalResults": "2", + "entity": [ + { + "type":"FEED", + "name":"SampleUSHealthBill", + "status":"SUBMITTEDâ, + "tags":{"tag":["related=ushealthcare","department=billingDepartment","_falcon_extension_name=billCoollection","_falcon_extension_job=daily-health-bill"]}, + "clusters": {"cluster":["SampleCluster1","primaryClusterâ]} + }, + { + "type":"PROCESSâ, + "name":"SampleBillPayâ, + "status":"RUNNINGâ, + "tags":{"tag":["related=healthcare","department=billingDepartment","_falcon_extension_name=billCoollection","_falcon_extension_job=daily-health-bill"]}, + "clusters":{"cluster":"primaryClusterâ} + } + ] + } + }, + { + "jobName": "fsa-bill", + "jobEntities": { + "totalResults": "1", + "entity": + { + "type":"PROCESSâ, + "name":"FSAPayâ, + "status":"RUNNINGâ, + "tagsâ:{"tag":["related=healthcare","department=billingDepartment","_falcon_extension_name=billCollection","_falcon_extension_job=fsa-bill"]}, + "clusters":{"cluster":"primaryClusterâ} + } + } + } + ] + } +} +</verbatim> http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionResume.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/restapi/ExtensionResume.twiki b/trunk/general/src/site/twiki/restapi/ExtensionResume.twiki new file mode 100644 index 0000000..70cbbff --- /dev/null +++ b/trunk/general/src/site/twiki/restapi/ExtensionResume.twiki @@ -0,0 +1,29 @@ +---++ POST api/extension/resume/:job-name + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Resume an extension job. + +---++ Parameters + * :job-name Name of the extension job. + * doAs <optional> Impersonate the user. + +---++ Results +Result of the resume operation. + +---++ Examples +---+++ Rest Call +<verbatim> +POST http://localhost:15000/api/extensions/resume/sales-monthly +</verbatim> +---+++ Result +<verbatim> +{ + "requestId": "e5cc8230-f356-4566-9b65-536abdff8aa3", + "message": "Extension job sales-monthly resumed successfully", + "status": "SUCCEEDED" +} +</verbatim> http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionSchedule.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/restapi/ExtensionSchedule.twiki b/trunk/general/src/site/twiki/restapi/ExtensionSchedule.twiki new file mode 100644 index 0000000..c40ef6e --- /dev/null +++ b/trunk/general/src/site/twiki/restapi/ExtensionSchedule.twiki @@ -0,0 +1,29 @@ +---++ POST api/extension/schedule/:job-name + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Schedule an extension job. + +---++ Parameters + * :job-name Name of the extension job. + * doAs <optional> Impersonate the user. + +---++ Results +Result of the schedule operation. + +---++ Examples +---+++ Rest Call +<verbatim> +POST http://localhost:15000/api/extensions/schedule/sales-monthly +</verbatim> +---+++ Result +<verbatim> +{ + "requestId": "e5cc8230-f356-4566-9b65-536abdff8aa3", + "message": "Extension job sales-monthly scheduled successfully", + "status": "SUCCEEDED" +} +</verbatim> http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionSubmit.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/restapi/ExtensionSubmit.twiki b/trunk/general/src/site/twiki/restapi/ExtensionSubmit.twiki new file mode 100644 index 0000000..6b4df32 --- /dev/null +++ b/trunk/general/src/site/twiki/restapi/ExtensionSubmit.twiki @@ -0,0 +1,38 @@ +---++ POST api/extension/submit/:extension-name + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Submit an extension job. + +---++ Parameters + * :extension-name Name of the extension. + * doAs <optional> Impersonate the user. + +---++ Results +Result of submission. + +---++ Examples +---+++ Rest Call +<verbatim> +POST http://localhost:15000/api/extensions/submit/hdfs-mirroring +jobName=sales-monthly +jobClustername=primaryCluster +jobClusterValidityStart=2015-03-13T00:00Z +jobClusterValidityEnd=2016-12-30T00:00Z +jobFrequency=minutes(5) +sourceDir=/user/hrt_qa/dr/test/primaryCluster/input +sourceCluster=primaryCluster +targetDir=/user/hrt_qa/dr/test/backupCluster/input +targetCluster=backupCluster +</verbatim> +---+++ Result +<verbatim> +{ + "requestId": "e5cc8230-f356-4566-9b65-536abdff8aa3", + "message": "Extension job submitted successfully", + "status": "SUCCEEDED" +} +</verbatim> http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionSubmitAndSchedule.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/restapi/ExtensionSubmitAndSchedule.twiki b/trunk/general/src/site/twiki/restapi/ExtensionSubmitAndSchedule.twiki new file mode 100644 index 0000000..8a148ce --- /dev/null +++ b/trunk/general/src/site/twiki/restapi/ExtensionSubmitAndSchedule.twiki @@ -0,0 +1,38 @@ +---++ POST api/extension/submitAndSchedule/:extension-name + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Submit and schedule an extension job. + +---++ Parameters + * :extension-name Name of the extension. + * doAs <optional> Impersonate the user. + +---++ Results +Result of the submit and schedule operation. + +---++ Examples +---+++ Rest Call +<verbatim> +POST http://localhost:15000/api/extensions/submitAndSchedule/hdfs-mirroring +jobName=sales-monthly +jobClustername=primaryCluster +jobClusterValidityStart=2015-03-13T00:00Z +jobClusterValidityEnd=2016-12-30T00:00Z +jobFrequency=minutes(5) +sourceDir=/user/hrt_qa/dr/test/primaryCluster/input +sourceCluster=primaryCluster +targetDir=/user/hrt_qa/dr/test/backupCluster/input +targetCluster=backupCluster +</verbatim> +---+++ Result +<verbatim> +{ + "requestId": "e5cc8230-f356-4566-9b65-536abdff8aa3", + "message": "Extension job submitted and scheduled successfully", + "status": "SUCCEEDED" +} +</verbatim> http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionSuspend.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/restapi/ExtensionSuspend.twiki b/trunk/general/src/site/twiki/restapi/ExtensionSuspend.twiki new file mode 100644 index 0000000..cc3478f --- /dev/null +++ b/trunk/general/src/site/twiki/restapi/ExtensionSuspend.twiki @@ -0,0 +1,29 @@ +---++ POST api/extension/suspend/:job-name + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Suspend an extension job. + +---++ Parameters + * :job-name Name of the extension job. + * doAs <optional> Impersonate the user. + +---++ Results +Result of the suspend operation. + +---++ Examples +---+++ Rest Call +<verbatim> +POST http://localhost:15000/api/extensions/suspend/sales-monthly +</verbatim> +---+++ Result +<verbatim> +{ + "requestId": "e5cc8230-f356-4566-9b65-536abdff8aa3", + "message": "Extension job sales-monthly suspended successfully", + "status": "SUCCEEDED" +} +</verbatim> http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionUpdate.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/restapi/ExtensionUpdate.twiki b/trunk/general/src/site/twiki/restapi/ExtensionUpdate.twiki new file mode 100644 index 0000000..1ac626a --- /dev/null +++ b/trunk/general/src/site/twiki/restapi/ExtensionUpdate.twiki @@ -0,0 +1,38 @@ +---++ POST api/extension/update/:extension-name + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Update an extension entity. Note that we require all the entity names to remain the same. Future work will allow different entity names. + +---++ Parameters + * :extension-name Name of the extension. + * doAs <optional> Impersonate the user. + +---++ Results +Result of update. + +---++ Examples +---+++ Rest Call +<verbatim> +POST http://localhost:15000/api/extensions/update/hdfs-mirroring +jobName=sales-monthly +jobClustername=primaryCluster +jobClusterValidityStart=2015-03-13T00:00Z +jobClusterValidityEnd=2016-12-30T00:00Z +jobFrequency=minutes(5) +sourceDir=/user/hrt_qa/dr/test/primaryCluster/input +sourceCluster=primaryCluster +targetDir=/user/hrt_qa/dr/test/backupCluster/input +targetCluster=backupCluster +</verbatim> +---+++ Result +<verbatim> +{ + "requestId": "e5cc8230-f356-4566-9b65-536abdff8aa3", + "message": "Updated successfully", + "status": "SUCCEEDED" +} +</verbatim> http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionValidate.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/restapi/ExtensionValidate.twiki b/trunk/general/src/site/twiki/restapi/ExtensionValidate.twiki new file mode 100644 index 0000000..c27f2d7 --- /dev/null +++ b/trunk/general/src/site/twiki/restapi/ExtensionValidate.twiki @@ -0,0 +1,38 @@ +---++ POST api/extension/validate/:extension-name + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Validate the input for extension job submission. + +---++ Parameters + * :extension-name Name of the extension. + * doAs <optional> Impersonate the user. + +---++ Results +Result of validation. + +---++ Examples +---+++ Rest Call +<verbatim> +POST http://localhost:15000/api/extensions/validate/hdfs-mirroring +jobName=sales-monthly +jobClustername=primaryCluster +jobClusterValidityStart=2015-03-13T00:00Z +jobClusterValidityEnd=2016-12-30T00:00Z +jobFrequency=minutes(5) +sourceDir=/user/hrt_qa/dr/test/primaryCluster/input +sourceCluster=primaryCluster +targetDir=/user/hrt_qa/dr/test/backupCluster/input +targetCluster=backupCluster +</verbatim> +---+++ Result +<verbatim> +{ + "requestId": "e5cc8230-f356-4566-9b65-536abdff8aa3", + "message": "Validated successfully", + "status": "SUCCEEDED" +} +</verbatim> http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/InstanceSearch.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/restapi/InstanceSearch.twiki b/trunk/general/src/site/twiki/restapi/InstanceSearch.twiki new file mode 100644 index 0000000..4744240 --- /dev/null +++ b/trunk/general/src/site/twiki/restapi/InstanceSearch.twiki @@ -0,0 +1,97 @@ +---++ GET /api/instance/search + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Get the list of instances that meet search criteria. + +---++ Parameters + * :type <optional param> Comma-separated entity types. Valid entity types are feed, process or datasource. + * nameseq <optional param> Subsequence of entity name. Not case sensitive. + * The entity name needs to contain all the characters in the subsequence in the same order. + * Example 1: "sample1" will match the entity named "SampleFeed1-2". + * Example 2: "mhs" will match the entity named "New-My-Hourly-Summary". + * tagkeys <optional param> Keywords in tags, separated by comma. Not case sensitive. + * The returned entities will have tags that match all the tag keywords. + * instanceStatus <optional param> Valid options are RUNNING, SUCCEEDED, FAILED, WAITING, SUSPENDED, KILLED, TIMEOUT. + * start <optional param> Show instances from this nominal time. Format: yyyy-MM-dd'T'HH:mm'Z'. + * By default, it is set to (end - (10 * entityFrequency)). + * end <optional param> Show instances up to this nominal time. Format: yyyy-MM-dd'T'HH:mm'Z'. + * Default is set to now. + * orderBy <optional param> Field by which results should be ordered. Sorted by descending order. + * If not specified, instances of the same entity will be listed together in the result. + * Current valid option: status. Coming soon: sorting by name, nominal time, running time. + * offset <optional param> Show results from the offset, used for pagination. Defaults to 0. + * numResults <optional param> Number of results to show per request, used for pagination. + * Only integers > 0 are valid. Default is 10. + * doAs <optional query param> allows the current user to impersonate the user passed in doAs. + +---++ Results +A list of instances. + +---++ Examples +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/instance/search?nameseq=usbill&start=2012-04-03T00:00Z +</verbatim> +---+++ Result +<verbatim> +{ + "instances": [ + { + âname":"SampleUSHealthBillâ, + âtype":"FEEDâ, + âtagsâ: {"tag":["related=ushealthcare","department=billingDepartment"]}, + ânominalTime": "2012-04-03T07:00Z" + "cluster": "primary-cluster", + "status": "SUCCEEDED", + "logFile": "http:\/\/localhost:11000\/oozie?job=0000070-131021115933395-oozie-rgau-W", + "startTime": "2013-10-21T14:39:56-07:00", + "endTime": "2013-10-21T14:40:26-07:00", + } + ], + "requestId": "default\/e15bb378-d09f-4911-9df2-5334a45153d2\n", + "message": "Instances Search Results", + "status": "SUCCEEDEDâ +} +</verbatim> + +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/instance/search?nameseq=samplebill&tagkeys=billing,healthcare&start=2012-04-03T00:00Z&instanceStatus=succeeded&orderBy=nominalTime&offset=2&numResults=2 +</verbatim> +---+++ Result +<verbatim> + { + "instances": [ + { + "name":"SampleHealthBill", + "type":"PROCESS", + "tagsâ: {"tag":["related=healthcare","department=billingDepartment"]}, + ânominalTime": "2012-04-03T08:00Z" + "cluster": "primary-cluster", + "status": "SUCCEEDED", + "logFile": "http:\/\/localhost:11000\/oozie?job=0000070-131021115933397-oozie-rgau-W", + "startTime": "2013-10-21T14:41:56-07:00", + "endTime": "2013-10-21T14:42:26-07:00", + }, + { + "name":"SampleUSHealthBill", + "type":"FEEDâ, + "tags": {"tag":["related=ushealthcare","department=billingDepartment"]}, + ânominalTime": "2012-04-03T07:00Z" + "cluster": "primary-cluster", + "status": "SUCCEEDED", + "logFile": "http:\/\/localhost:11000\/oozie?job=0000070-131021115933395-oozie-rgau-W", + "startTime": "2013-10-21T14:39:56-07:00", + "endTime": "2013-10-21T14:40:26-07:00", + }, + ], + + "requestId": "default\/e15bb378-d09f-4911-9df2-5334a45153d2\n", + "message": "Instances Search Results", + "status": "SUCCEEDEDâ +} +</verbatim> http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ResourceList.twiki ---------------------------------------------------------------------- diff --git a/trunk/general/src/site/twiki/restapi/ResourceList.twiki b/trunk/general/src/site/twiki/restapi/ResourceList.twiki index 34c2c6f..5a4c253 100644 --- a/trunk/general/src/site/twiki/restapi/ResourceList.twiki +++ b/trunk/general/src/site/twiki/restapi/ResourceList.twiki @@ -6,6 +6,8 @@ * <a href="#REST_Call_on_Admin_Resource">REST Call on Admin Resource</a> * <a href="#REST_Call_on_Lineage_Graph">REST Call on Lineage Graph Resource</a> * <a href="#REST_Call_on_Metadata_Resource">REST Call on Metadata Resource</a> + * <a href="#REST_Call_on_Extension_Artifact">REST Call on Extension artifact</a> + * <a href="#REST_Call_on_Extension_Job Management">REST Call on Extension Job Management</a> ---++ Authentication @@ -37,6 +39,7 @@ The current version of the rest api's documentation is also hosted on the Falcon | GET | [[AdminStack][api/admin/stack]] | Get stack of the server | | GET | [[AdminVersion][api/admin/version]] | Get version of the server | | GET | [[AdminConfig][api/admin/config/:config-type]] | Get configuration information of the server | +| GET | [[AdminSafemode][api/admin/setSafeMode/:mode]] | Set safemode to true/false in falcon server | ---++ REST Call on Entity Resource @@ -51,7 +54,7 @@ The current version of the rest api's documentation is also hosted on the Falcon | DELETE | [[EntityDelete][api/entities/delete/:entity-type/:entity-name]] | Delete the entity | | GET | [[EntityStatus][api/entities/status/:entity-type/:entity-name]] | Get the status of the entity | | GET | [[EntityDefinition][api/entities/definition/:entity-type/:entity-name]] | Get the definition of the entity | -| GET | [[EntityList][api/entities/list/:entity-type]] | Get the list of entities | +| GET | [[EntityList][api/entities/list/{:entity-type}]] | Get the list of entities | | GET | [[EntitySummary][api/entities/summary/:entity-type/:cluster]] | Get instance summary of all entities | | GET | [[EntityDependencies][api/entities/dependencies/:entity-type/:entity-name]] | Get the dependencies of the entity | | GET | [[FeedSLA][api/entities/sla-alert/:entity-type]] | Get pending feed instances which missed sla | @@ -88,6 +91,28 @@ The current version of the rest api's documentation is also hosted on the Falcon ---++ REST Call on Metadata Discovery Resource -| *Call Type* | *Resource* | *Description* | -| GET | [[MetadataList][api/metadata/discovery/:dimension-type/list]] | list of dimensions | -| GET | [MetadataRelations][api/metadata/discovery/:dimension-type/:dimension-name/relations]] | Return all relations of a dimension | +| *Call Type* | *Resource* | *Description* | +| GET | [[MetadataList][api/metadata/discovery/:dimension-type/list]] | list of dimensions | +| GET | [[MetadataRelations][api/metadata/discovery/:dimension-type/:dimension-name/relations]] | Return all relations of a dimension | + +---++ REST Call on Extension Artifact + +| *Call Type* | *Resource* | *Description* | +| GET | [[ExtensionEnumeration][api/extension/enumerate]] | List all the extensions supported | +| GET | [[ExtensionDescription][api/extension/describe/:extensionÂname]] | Return the README of the specified extension | +| GET | [[ExtensionDefinition][api/extension/definition/:extensionÂname]] | Return a JSON document describing the extension invocation parameters | + + +---++ REST Call on Extension Job Management + +| *Call Type* | *Resource* | *Description* | +| GET | [[ExtensionList][api/extension/list/:extensionÂname]] | List jobs generated from an extension | +| GET | [[ExtensionInstances][api/extension/instances/:job-name]] | List instances of an extension job | +| POST | [[ExtensionValidate][api/extension/validate/:extensionÂname]] | Validate an extension job | +| POST | [[ExtensionSubmit][api/extension/submit/:extensionÂname]] | Submit an extension job | +| POST | [[ExtensionUpdate][api/extension/update/:extensionÂname]] | Update an extension job | +| POST | [[ExtensionSubmitAndSchedule][api/extension/submitAndSchedule/:extensionÂname]] | Submit and schedule an extension job | +| POST | [[ExtensionSchedule][api/extension/schedule/:job-name]] | Schedule an extension job | +| POST | [[ExtensionSuspend][api/extension/suspend/:job-name]] | Suspend an extension job | +| POST | [[ExtensionResume][api/extension/resume/:job-name]] | Resume an extension job | +| POST | [[ExtensionDelete][api/extension/delete/:job-name]] | Delete an extension job | \ No newline at end of file http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/pom.xml ---------------------------------------------------------------------- diff --git a/trunk/pom.xml b/trunk/pom.xml index a3e2557..b60eb84 100644 --- a/trunk/pom.xml +++ b/trunk/pom.xml @@ -21,7 +21,7 @@ <modelVersion>4.0.0</modelVersion> <groupId>org.apache.falcon</groupId> <artifactId>falcon-website</artifactId> - <version>0.9-SNAPSHOT</version> + <version>0.11-SNAPSHOT</version> <packaging>pom</packaging> <name>Apache Falcon</name> @@ -254,7 +254,7 @@ <email>[email protected]</email> <timezone>-08:00</timezone> <roles> - <role>committer</role> + <role>PPMC</role> </roles> <organization>Hortonworks</organization> </developer> http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/releases/0.10/pom.xml ---------------------------------------------------------------------- diff --git a/trunk/releases/0.10/pom.xml b/trunk/releases/0.10/pom.xml new file mode 100644 index 0000000..e9e6a21 --- /dev/null +++ b/trunk/releases/0.10/pom.xml @@ -0,0 +1,66 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd";> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>org.apache.falcon</groupId> + <artifactId>falcon-website-releases</artifactId> + <version>0.2</version> + </parent> + <artifactId>falcon-website-0.10</artifactId> + <version>0.10</version> + <packaging>pom</packaging> + + <name>Apache Falcon - Documentation v0.10</name> + + <build> + <plugins> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-site-plugin</artifactId> + <version>3.3</version> + <dependencies> + <dependency> + <groupId>org.apache.maven.doxia</groupId> + <artifactId>doxia-module-twiki</artifactId> + <version>1.3</version> + </dependency> + <dependency> + <groupId>org.apache.maven.wagon</groupId> + <artifactId>wagon-ssh-external</artifactId> + <version>2.6</version> + </dependency> + </dependencies> + <executions> + <execution> + <goals> + <goal>site</goal> + </goals> + <phase>prepare-package</phase> + </execution> + </executions> + <configuration> + <outputDirectory>../../../content/0.10</outputDirectory> + </configuration> + </plugin> + </plugins> + </build> + +</project> http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/releases/0.10/src/site/resources/Architecture.png ---------------------------------------------------------------------- diff --git a/trunk/releases/0.10/src/site/resources/Architecture.png b/trunk/releases/0.10/src/site/resources/Architecture.png new file mode 100644 index 0000000..0378b49 Binary files /dev/null and b/trunk/releases/0.10/src/site/resources/Architecture.png differ http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/releases/0.10/src/site/resources/EntityDependency.png ---------------------------------------------------------------------- diff --git a/trunk/releases/0.10/src/site/resources/EntityDependency.png b/trunk/releases/0.10/src/site/resources/EntityDependency.png new file mode 100644 index 0000000..9f11870 Binary files /dev/null and b/trunk/releases/0.10/src/site/resources/EntityDependency.png differ http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/releases/0.10/src/site/resources/FeedSchedule.png ---------------------------------------------------------------------- diff --git a/trunk/releases/0.10/src/site/resources/FeedSchedule.png b/trunk/releases/0.10/src/site/resources/FeedSchedule.png new file mode 100644 index 0000000..105c6b1 Binary files /dev/null and b/trunk/releases/0.10/src/site/resources/FeedSchedule.png differ
