Added: falcon/trunk/general/src/site/twiki/MigrationInstructions.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/MigrationInstructions.twiki?rev=1660589&view=auto ============================================================================== --- falcon/trunk/general/src/site/twiki/MigrationInstructions.twiki (added) +++ falcon/trunk/general/src/site/twiki/MigrationInstructions.twiki Wed Feb 18 10:55:56 2015 @@ -0,0 +1,15 @@ +---+ Migration Instructions + +---++ Migrate from 0.5-incubating to 0.6-incubating + +This is a placeholder wiki for migration instructions from falcon 0.5-incubating to 0.6-incubating. + +---+++ Update Entities + +---+++ Change cluster dir permissions + +---+++ Enable/Disable TLS + +---+++ Authorization + +
Modified: falcon/trunk/general/src/site/twiki/OnBoarding.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/OnBoarding.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/OnBoarding.twiki (original) +++ falcon/trunk/general/src/site/twiki/OnBoarding.twiki Wed Feb 18 10:55:56 2015 @@ -16,6 +16,10 @@ ---+++ Sample Pipeline ---++++ Cluster Cluster definition that contains end points for name node, job tracker, oozie and jms server: +The cluster locations MUST be created prior to submitting a cluster entity to Falcon. +*staging* must have 777 permissions and the parent dirs must have execute permissions +*working* must have 755 permissions and the parent dirs must have execute permissions + <verbatim> <?xml version="1.0"?> <!-- @@ -24,13 +28,13 @@ Cluster definition that contains end poi <cluster colo="ua2" description="" name="corp" xmlns="uri:falcon:cluster:0.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";> <interfaces> - <interface type="readonly" endpoint="hftp://name-node.com:50070"; version="0.20.2-cdh3u0" /> + <interface type="readonly" endpoint="hftp://name-node.com:50070"; version="2.5.0" /> - <interface type="write" endpoint="hdfs://name-node.com:54310" version="0.20.2-cdh3u0" /> + <interface type="write" endpoint="hdfs://name-node.com:54310" version="2.5.0" /> - <interface type="execute" endpoint="job-tracker:54311" version="0.20.2-cdh3u0" /> + <interface type="execute" endpoint="job-tracker:54311" version="2.5.0" /> - <interface type="workflow" endpoint="http://oozie.com:11000/oozie/"; version="3.1.4" /> + <interface type="workflow" endpoint="http://oozie.com:11000/oozie/"; version="4.0.1" /> <interface type="messaging" endpoint="tcp://jms-server.com:61616?daemon=true" version="5.1.6" /> </interfaces> Added: falcon/trunk/general/src/site/twiki/Operability.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/Operability.twiki?rev=1660589&view=auto ============================================================================== --- falcon/trunk/general/src/site/twiki/Operability.twiki (added) +++ falcon/trunk/general/src/site/twiki/Operability.twiki Wed Feb 18 10:55:56 2015 @@ -0,0 +1,102 @@ +---+ Operationalizing Falcon + +---++ Overview + +Apache Falcon provides various tools to operationalize Falcon consisting of Alerts for +unrecoverable errors, Audits of user actions, Metrics, and Notifications. They are detailed below. + + +---++ Monitoring + +Falcon provides monitoring of various events by capturing metrics of those events. +The metric numbers can then be used to monitor performance and health of the Falcon system and +the entire processing pipelines. + +Users can view the logs of these events in the metric.log file, by default this file is created +under ${user.dir}/logs/ directory. Users may also extend the Falcon monitoring framework to send +events to systems like Mondemand/lwes by implementingorg.apache.falcon.plugin.MonitoringPlugin +interface. + +The following events are captured by Falcon for logging the metrics: + 1. New cluster definitions posted to Falcon (success & failures) + 1. New feed definition posted to Falcon (success & failures) + 1. New process definition posted to Falcon (success & failures) + 1. Process update events (success & failures) + 1. Feed update events (success & failures) + 1. Cluster update events (success & failures) + 1. Process suspend events (success & failures) + 1. Feed suspend events (success & failures) + 1. Process resume events (success & failures) + 1. Feed resume events (success & failures) + 1. Process remove events (success & failures) + 1. Feed remove events (success & failures) + 1. Cluster remove events (success & failures) + 1. Process instance kill events (success & failures) + 1. Process instance re-run events (success & failures) + 1. Process instance generation events + 1. Process instance failure events + 1. Process instance auto-retry events + 1. Process instance retry exhaust events + 1. Feed instance deletion event + 1. Feed instance deletion failure event (no retries) + 1. Feed instance replication event + 1. Feed instance replication failure event + 1. Feed instance replication auto-retry event + 1. Feed instance replication retry exhaust event + 1. Feed instance late arrival event + 1. Feed instance post cut-off arrival event + 1. Process re-run due to late feed event + 1. Transaction rollback failed event + +The metric logged for an event has the following properties: + 1. Action - Name of the event. + 2. Dimensions - A list of name/value pairs of various attributes for a given action. + 3. Status- Status of an action FAILED/SUCCEEDED. + 4. Time-taken - Time taken in nanoseconds for a given action. + +An example for an event logged for a submit of a new process definition: + + 2012-05-04 12:23:34,026 {Action:submit, Dimensions:{entityType=process}, Status: SUCCEEDED, Time-taken:97087000 ns} + +Users may parse the metric.log or capture these events from custom monitoring frameworks and can plot various graphs +or send alerts according to their requirements. + + +---++ Notifications + +Falcon creates a JMS topic for every process/feed that is scheduled in Falcon. +The implementation class and the broker url of the JMS engine are read from the dependent cluster's definition. +Users may register consumers on the required topic to check the availability or status of feed instances. + +For a given process that is scheduled, the name of the topic is same as the process name. +Falcon sends a Map message for every feed produced by the instance of a process to the JMS topic. +The JMS !MapMessage sent to a topic has the following properties: +entityName, feedNames, feedInstancePath, workflowId, runId, nominalTime, timeStamp, brokerUrl, brokerImplClass, entityType, operation, logFile, topicName, status, brokerTTL; + +For a given feed that is scheduled, the name of the topic is same as the feed name. +Falcon sends a map message for every feed instance that is deleted/archived/replicated depending upon the retention policy set in the feed definition. +The JMS !MapMessage sent to a topic has the following properties: +entityName, feedNames, feedInstancePath, workflowId, runId, nominalTime, timeStamp, brokerUrl, brokerImplClass, entityType, operation, logFile, topicName, status, brokerTTL; + +The JMS messages are automatically purged after a certain period (default 3 days) by the Falcon JMS house-keeping service.TTL (Time-to-live) for JMS message +can be configured in the Falcon's startup.properties file. + + +---++ Alerts + +Falcon generates alerts for unrecoverable errors into a log file by default. +Users can view these alerts in the alerts.log file, by default this file is created +under ${user.dir}/logs/ directory. + +Users may also extend the Falcon Alerting plugin to send events to systems like Nagios, etc. by +extending org.apache.falcon.plugin.AlertingPlugin interface. + + +---++ Audits + +Falcon audits all user activity and captures them into a log file by default. +Users can view these audits in the audit.log file, by default this file is created +under ${user.dir}/logs/ directory. + +Users may also extend the Falcon Audit plugin to send audits to systems like Apache Argus, etc. by +extending org.apache.falcon.plugin.AuditingPlugin interface. Modified: falcon/trunk/general/src/site/twiki/Security.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/Security.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/Security.twiki (original) +++ falcon/trunk/general/src/site/twiki/Security.twiki Wed Feb 18 10:55:56 2015 @@ -68,6 +68,8 @@ personal workstation, conveniently becom Falcon also allows users to configure a super user group and allows users belonging to this group to be a super user. +ACL owner and group must be valid even if the authenticated user is a super-user. + ---+++ Group Memberships Once a user has been authenticated and a username has been determined, the list of groups is @@ -78,6 +80,8 @@ will shell out to the Unix bash -c group Note that Falcon stores the user and group of an Entity as strings; there is no conversion from user and group identity numbers as is conventional in Unix. +The only limitation is that a user cannot add a group in ACL that he does not belong to. + ---+++ Authorization Provider Falcon provides a plugin-able provider interface for Authorization. It also ships with a default @@ -85,26 +89,25 @@ implementation that enforces the followi ---++++ Entity and Instance Management Operations Policy -* All Entity and Instance operations are authorized for users who created them, Owners and users -with group memberships -* Reference to entities with in a feed or process is allowed with out enforcing permissions -Any Feed or Process can refer to a Cluster entity not owned by the Feed or Process owner -Any Process can refer to a Feed entity not owned by the Process owner + * All Entity and Instance operations are authorized for users who created them, Owners and users with group memberships + * Reference to entities with in a feed or process is allowed with out enforcing permissions + +Any Feed or Process can refer to a Cluster entity not owned by the Feed or Process owner. Any Process can refer to a Feed entity not owned by the Process owner The authorization is enforced in the following way: -if admin resource, - if authenticated user name matches the admin users configuration - Else if groups of the authenticated user matches the admin groups configuration - Else authorization exception is thrown -Else if entities or instance resource - if the authenticated user matches the owner in ACL for the entity - Else if the groups of the authenticated user matches the group in ACL for the entity - Else authorization exception is thrown -Else if lineage resource - All have read-only permissions, reason being folks should be able to examine the dependency - and allow reuse + * if admin resource, + * If authenticated user name matches the admin users configuration + * Else if groups of the authenticated user matches the admin groups configuration + * Else authorization exception is thrown + * Else if entities or instance resource + * If the authenticated user matches the owner in ACL for the entity + * Else if the groups of the authenticated user matches the group in ACL for the entity + * Else authorization exception is thrown + * Else if lineage resource + * All have read-only permissions, reason being folks should be able to examine the dependency and allow reuse +To authenticate user for REST api calls, user should append "user.name=<username>" to the query. *operations on Entity Resource* @@ -147,6 +150,7 @@ determined by a static configuration par ---++++ Lineage Resource Policy Lineage is read-only and hence all users can look at lineage for their respective entities. +*Note:* This gap will be fixed in a later release. ---++ Authentication Configuration @@ -204,6 +208,9 @@ Following is the Server Side Configurati # Comma separated list of black listed users *.falcon.http.authentication.blacklisted.users= + +# Increase Jetty request buffer size to accommodate the generated Kerberos token +*.falcon.jetty.request.buffer.size=16192 </verbatim> ---+++ Pseudo/Simple Configuration @@ -299,11 +306,17 @@ Falcon should be configured to communica ---++ Changes to ownership and permissions of directories managed by Falcon -| *Directory* | *Location* | *Owner* | *Permissions* | -| Configuration Store | ${config.store.uri} | falcon | 750 | -| Oozie coord/bundle XMLs | ${cluster.staging-location}/workflows/{entity}/{entity-name} | falcon | 644 | -| Shared libs | {cluster.working}/{lib,libext} | falcon | 755 | -| App logs | ${cluster.staging-location}/workflows/{entity}/{entity-name}/logs | falcon | 777 | +| *Directory* | *Location* | *Owner* | *Permissions* | +| Configuration Store | ${config.store.uri} | falcon | 700 | +| Cluster Staging Location | ${cluster.staging-location} | falcon | 777 | +| Cluster Working Location | ${cluster.working-location} | falcon | 755 | +| Shared libs | {cluster.working}/{lib,libext} | falcon | 755 | +| Oozie coord/bundle XMLs | ${cluster.staging-location}/workflows/{entity}/{entity-name} | $user | cluster umask | +| App logs | ${cluster.staging-location}/workflows/{entity}/{entity-name}/logs | $user | cluster umask | + +*Note:* Please note that the cluster staging and working locations MUST be created prior to +submitting a cluster entity to Falcon. Also, note that the the parent dirs must have execute +permissions. ---++ Backwards compatibility @@ -330,12 +343,11 @@ configured specifically in the file. ---+++ Falcon Dashboard -The dashboard assumes an anonymous user in Pseudo/Simple method and hence anonymous users must be enabled for it to -work. -<verbatim> -# Indicates if anonymous requests are allowed when using 'simple' authentication. -*.falcon.http.authentication.simple.anonymous.allowed=true -</verbatim> +To initialize the current user for dashboard, user should append query param "user.name=<username>" to the REST api call. + +If dashboard user wishes to change the current user, they should do the following. + * delete the hadoop.auth cookie from browser cache. + * append query param "user.name=<new_user>" to the next REST API call. In Kerberos method, the browser must support HTTP Kerberos SPNEGO. Modified: falcon/trunk/general/src/site/twiki/index.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/index.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/index.twiki (original) +++ falcon/trunk/general/src/site/twiki/index.twiki Wed Feb 18 10:55:56 2015 @@ -1,101 +1,43 @@ ----+ Apache Falcon - Data management and processing platform +---+ Falcon - Feed management and data processing platform + +Falcon is a feed processing and feed management system aimed at making it +easier for end consumers to onboard their feed processing and feed +management on hadoop clusters. + +---++ Why? + + * Establishes relationship between various data and processing elements on a Hadoop environment + + * Feed management services such as feed retention, replications across clusters, archival etc. + + * Easy to onboard new workflows/pipelines, with support for late data handling, retry policies + + * Integration with metastore/catalog such as Hive/HCatalog + + * Provide notification to end customer based on availability of feed groups + (logical group of related feeds, which are likely to be used together) + + * Enables use cases for local processing in colo and global aggregations + + * Captures Lineage information for feeds and processes + +---+ Getting Started + +Start with these simple steps to install an falcon instance [[InstallationSteps][Simple setup]]. Also refer +to Falcon architecture and documentation in [[FalconDocumentation][Documentation]]. [[OnBoarding][On boarding]] +describes steps to on-board a pipeline to Falcon. It also gives a sample pipeline for reference. +[[EntitySpecification][Entity Specification]] gives complete details of all Falcon entities. + +[[FalconCLI][Falcon CLI]] implements [[restapi/ResourceList][Falcon's RESTful API]] and +describes various options for the command line utility provided by Falcon. + +Falcon provides OOTB [[HiveIntegration][lifecycle management for Tables in Hive (HCatalog)]] +such as table replication for BCP and table eviction. Falcon also enforces +[[Security][Security]] on protected resources and enables SSL. + +#LicenseInfo +---+ Licensing Information + +Falcon is distributed under [[http://www.apache.org/licenses/LICENSE-2.0][Apache License 2.0]]. + -Apache Falcon is a data processing and management solution for Hadoop designed for data motion, -coordination of data pipelines, lifecycle management, and data discovery. Falcon enables end consumers -to quickly onboard their data and its associated processing and management tasks on Hadoop clusters. - -Data Management on Hadoop encompasses data motion, process orchestration, lifecycle management, -data discovery, etc. among other concerns. Falcon is a new data processing and management platform -for Hadoop that solves this problem and creates additional opportunities by building on existing -components within the Hadoop ecosystem without reinventing the wheel. - -Falcon will enable easy data management via declarative mechanism for Hadoop. Users of Falcon platform -simply define infrastructure endpoints, data sets and processing rules declaratively. These declarative -configurations are expressed in such a way that the dependencies between these configured entities are -explicitly described. This information about inter-dependencies between various entities allows Falcon -to orchestrate and manage various data management functions. - -Falcon has gradauted to a top level project in Dec 2014. - - -<div id="components" class="carousel slide"> -<!-- Carousel items --> -<div class="carousel-inner"> - <div class="active item"> - <div class="hero-unit"> - <h3>Provides key services for data processing apps</h3> - <p>Complex data processing logic handled by Falcon instead of hard-coded in apps</p> - <!-- - <p> - <a class="btn btn-primary pull-right" href="././docs/GettingStarted.html">Learn more</a> - </p> - --> - </div> - </div> - <div class="item"> - <div class="hero-unit"> - <h3>Introduces higher layer of abstractions</h3> - <p>Data Set, Infrastructure (Cluster, Database, Filer, etc.), Process</p> - </div> - </div> - <div class="item"> - <div class="hero-unit"> - <h3>Declarative language for expressing data management functions</h3> - <p>Data management expressed as simple directives, instead of verbosely defining it repeatedly</p> - </div> - </div> - <div class="item"> - <div class="hero-unit"> - <h3>Promotes Separation of Concerns</h3> - <p> - Allow process owners to keep application/user workflow specific to their application logic - than muddy them with the common management functions. - </p> - </div> - </div> - <div class="item"> - <div class="hero-unit"> - <h3>Promotes Polyglot programming</h3> - <p> - Does not do any heavy lifting but delegates to tools with in the Hadoop ecosystem, - Enhances productivity - </p> - </div> - </div> - <div class="item"> - <div class="hero-unit"> - <h3>Data Motion</h3> - <p>Data Import & Export from DBs, Filers into HDFS</p> - </div> - </div> - <div class="item"> - <div class="hero-unit"> - <h3>Process orchestration</h3> - <p> - Late Data Handling, Retries, etc. - Multi-cluster management to support Local/Global Aggregations, Rollups, etc. - <a href="http://oozie.apache.org/";>Scheduler</a> integration - </p> - </div> - </div> - <div class="item"> - <div class="hero-unit"> - <h3>Data Lifecycle Management</h3> - <p>Retention, Replication/BCP/DR, Anonymization of PII Data, Archival, etc.</p> - </div> - </div> - <div class="item"> - <div class="hero-unit"> - <h3>Data Discovery</h3> - <p>Data Classification, Audit, Lineage.</p> - </div> - </div> -</div> -</div> - ----++ Getting Involved - -Developers interested in getting involved with Falcon may join the [[./mail-lists][mailing lists]], -[[https://issues.apache.org/jira/browse/FALCON][report bugs]], retrieve code from the -[[./source-repository][version control system]], and make -[[https://cwiki.apache.org/confluence/display/FALCON/How+To+Contribute][contributions]]. Added: falcon/trunk/general/src/site/twiki/recipes.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/recipes.twiki?rev=1660589&view=auto ============================================================================== --- falcon/trunk/general/src/site/twiki/recipes.twiki (added) +++ falcon/trunk/general/src/site/twiki/recipes.twiki Wed Feb 18 10:55:56 2015 @@ -0,0 +1,77 @@ +---+ Falcon Recipes + +---++ Overview + +A Falcon recipe is a static process template with parameterized workflow to realize a specific use case. Recipes are +defined in user space. Recipes will not have support for update or lifecycle management. + +For example: + + * Replicating directories from one HDFS cluster to another (not timed partitions) + * Replicating hive metadata (database, table, views, etc.) + * Replicating between HDFS and Hive - either way + * Data masking etc. + +---++ Proposal + +Falcon provides a Process abstraction that encapsulates the configuration for a user workflow with scheduling +controls. All recipes can be modeled as a Process with in Falcon which executes the user workflow periodically. The +process and its associated workflow are parameterized. The user will provide a properties file with name value pairs +that are substituted by falcon before scheduling it. Falcon translates these recipes as a process entity by +replacing the parameters in the workflow definition. + +---++ Falcon CLI recipe support + +Falcon CLI functionality to support recipes has been added. +<a href="./FalconCLI.html">Recipe command usage is defined here.</a> + +CLI accepts recipe option with a recipe name and optional tool and does the following: + * Validates the options; name option is mandatory and tool is optional and should be provided if user wants to override the base recipe tool + * Looks for <name>-workflow.xml, <name>-template.xml and <name>.properties file in the path specified by falcon.recipe.path in client.properties. If files cannot be found then Falcon CLI will fail + * Invokes a Tool to substitute the properties in the templated process for the recipe. By default invokes base tool if tool option is not passed. Tool is responsible for generating process entity at the path specified by FalconCLI + * Validates the generated entity + * Submit and schedule this entity + * Generated process entity files are stored in tmp directory + +---++ Base Recipe tool + +Falcon provides a base tool that recipes can override. Base Recipe tool does the following: + * Expects recipe template file path, recipe properties file path and path where process entity to be submitted should be generated. Validates these arguments + * Validates the artifacts i.e. workflow and/or lib files specified in the recipe template exists on local filesystem or HDFS at the specified path else returns error + * Copies if the artifacts exists on local filesystem + * If workflow is on local FS then falcon.recipe.workflow.path in recipe property file is mandatory for it to be copied to HDFS. If templated process requires custom libs falcon.recipe.workflow.lib.path property is mandatory for them to be copied from Local FS to HDFS. Recipe tool will copy the local artifacts only if these properties are set in properties file + * Looks for the patten ##[A-Za-z0-9_.]*## in the templated process and substitutes it with the properties. Process entity generated after the substitution is written to the empty file passed by FalconCLI + +---++ Recipe template file format + + * Any templatized string should be in the format ##[A-Za-z0-9_.]*##. + * There should be a corresponding entry in the recipe properties file "falcon.recipe.<templatized-string> = <value to be substituted>" + +<verbatim> +Example: If the entry in recipe template is <workflow name="##workflow.name##"> there should be a corresponding entry in the recipe properties file falcon.recipe.workflow.name=hdfs-dr-workflow +</verbatim> + +---++ Recipe properties file format + + * Regular key value pair properties file + * Property key should be prefixed by "falcon.recipe." + +<verbatim> +Example: falcon.recipe.workflow.name=hdfs-dr-workflow +Recipe template will have <workflow name="##workflow.name##">. Recipe tool will look for the patten ##workflow.name## +and replace it with the property value "hdfs-dr-workflow". Substituted template will have <workflow name="hdfs-dr-workflow"> +</verbatim> + +---++ Managing the scheduled recipe process + * Scheduled recipe process is similar to regular process + * List : falcon entity -type process -name <recipe-process-name> -list + * Status : falcon entity -type process -name <recipe-process-name> -status + * Delete : falcon entity -type process -name <recipe-process-name> -delete + +---++ Sample recipes + + * Sample recipes are published in addons/recipes + +---++ Packaging + + * There is no packaging for recipes at this time but will be added soon. Modified: falcon/trunk/general/src/site/twiki/restapi/AdjacentVertices.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/AdjacentVertices.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/AdjacentVertices.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/AdjacentVertices.twiki Wed Feb 18 10:55:56 2015 @@ -1,4 +1,4 @@ ----++ GET api/graphs/lineage/vertices/:id/:direction +---++ GET api/metadata/lineage/vertices/:id/:direction * <a href="#Description">Description</a> * <a href="#Parameters">Parameters</a> * <a href="#Results">Results</a> @@ -31,7 +31,7 @@ Adjacent vertices of the vertex for the ---++ Examples ---+++ Rest Call <verbatim> -GET http://localhost:15000/api/graphs/lineage/vertices/4/out +GET http://localhost:15000/api/metadata/lineage/vertices/4/out </verbatim> ---+++ Result <verbatim> @@ -51,7 +51,7 @@ GET http://localhost:15000/api/graphs/li ---+++ Rest Call <verbatim> -GET http://localhost:15000/api/graphs/lineage/vertices/4/bothE +GET http://localhost:15000/api/metadata/lineage/vertices/4/bothE </verbatim> ---+++ Result <verbatim> Modified: falcon/trunk/general/src/site/twiki/restapi/AllEdges.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/AllEdges.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/AllEdges.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/AllEdges.twiki Wed Feb 18 10:55:56 2015 @@ -1,4 +1,4 @@ ----++ GET pi/graphs/lineage//edges/all +---++ GET pi/metadata/lineage//edges/all * <a href="#Description">Description</a> * <a href="#Parameters">Parameters</a> * <a href="#Results">Results</a> @@ -16,7 +16,7 @@ All edges in lineage graph. ---++ Examples ---+++ Rest Call <verbatim> -GET http://localhost:15000/api/graphs/lineage/edges/all +GET http://localhost:15000/api/metadata/lineage/edges/all </verbatim> ---+++ Result <verbatim> Modified: falcon/trunk/general/src/site/twiki/restapi/AllVertices.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/AllVertices.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/AllVertices.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/AllVertices.twiki Wed Feb 18 10:55:56 2015 @@ -1,4 +1,4 @@ ----++ GET api/graphs/lineage/vertices/all +---++ GET api/metadata/lineage/vertices/all * <a href="#Description">Description</a> * <a href="#Parameters">Parameters</a> * <a href="#Results">Results</a> @@ -16,7 +16,7 @@ All vertices in lineage graph. ---++ Examples ---+++ Rest Call <verbatim> -GET http://localhost:15000/api/graphs/lineage/vertices/all +GET http://localhost:15000/api/metadata/lineage/vertices/all </verbatim> ---+++ Result <verbatim> Modified: falcon/trunk/general/src/site/twiki/restapi/Edge.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/Edge.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/Edge.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/Edge.twiki Wed Feb 18 10:55:56 2015 @@ -1,4 +1,4 @@ ----++ GET api/graphs/lineage/edges/:id +---++ GET api/metadata/lineage/edges/:id * <a href="#Description">Description</a> * <a href="#Parameters">Parameters</a> * <a href="#Results">Results</a> @@ -16,7 +16,7 @@ Edge with the specified id. ---++ Examples ---+++ Rest Call <verbatim> -GET http://localhost:15000/api/graphs/lineage/edges/Q6t-c-5g +GET http://localhost:15000/api/metadata/lineage/edges/Q6t-c-5g </verbatim> ---+++ Result <verbatim> Added: falcon/trunk/general/src/site/twiki/restapi/EntityLineage.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/EntityLineage.twiki?rev=1660589&view=auto ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/EntityLineage.twiki (added) +++ falcon/trunk/general/src/site/twiki/restapi/EntityLineage.twiki Wed Feb 18 10:55:56 2015 @@ -0,0 +1,39 @@ +---++ GET api/metadata/lineage/entities?pipeline=:pipeline + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +It returns the graph depicting the relationship between the various processes and feeds in a given pipeline. + +---++ Parameters + * :pipeline is the name of the pipeline + +---++ Results +It returns a json graph + +---++ Examples +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/metadata/lineage/entities?pipeline=my-pipeline +</verbatim> +---+++ Result +<verbatim> +{ + "vertices": ["my-minutely-process", "my-hourly-process"], + "edges": + [ + { + "from" : "my-minutely-process", + "to" : "my-hourly-process", + "label" : "my-minutely-feed" + }, + { + "from" : "my-hourly-process", + "to" : "my-minutely-process", + "label" : "my-hourly-feedback" + } + ] +} +</verbatim> Modified: falcon/trunk/general/src/site/twiki/restapi/EntityList.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/EntityList.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/EntityList.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/EntityList.twiki Wed Feb 18 10:55:56 2015 @@ -11,6 +11,9 @@ Get list of the entities. * :entity-type Valid options are cluster, feed or process. * fields <optional param> Fields of entity that the user wants to view, separated by commas. * Valid options are STATUS, TAGS, PIPELINES. + * pattern <optional param> Find string which contains this sequence of characters. Example: pattern=abc + * matching is case insensitive. + * For example a pattern mhs will match a process named New-My-Hourly-Summary. * filterBy <optional param> Filter results by list of field:value pairs. Example: filterBy=STATUS:RUNNING,PIPELINES:clickLogs * Supported filter fields are NAME, STATUS, PIPELINES, CLUSTER. * Query will do an AND among filterBy fields. Modified: falcon/trunk/general/src/site/twiki/restapi/EntitySummary.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/EntitySummary.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/EntitySummary.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/EntitySummary.twiki Wed Feb 18 10:55:56 2015 @@ -1,4 +1,4 @@ ----++ GET /api/entities/summary/:entity-type/:cluster +---++ GET /api/entities/summary/:entity-type * <a href="#Description">Description</a> * <a href="#Parameters">Parameters</a> * <a href="#Results">Results</a> @@ -8,8 +8,8 @@ Given an EntityType and cluster, get list of entities along with summary of N recent instances of each entity ---++ Parameters - * :entity-type Valid options are cluster, feed or process. - * :cluster Show entities that belong to this cluster. + * :entity-type Valid options are feed or process. + * cluster Show entities that belong to this cluster. * start <optional param> Show entity summaries from this date. Date format is yyyy-MM-dd'T'HH:mm'Z'. * By default, it is set to (end - 2 days). * end <optional param> Show entity summary up to this date. Date format is yyyy-MM-dd'T'HH:mm'Z'. @@ -34,7 +34,7 @@ Show entities along with summary of N in ---++ Examples ---+++ Rest Call <verbatim> -GET http://localhost:15000/api/entities/summary/feed/primary-cluster?filterBy=STATUS:RUNNING&fields=status&[email protected]&orderBy=name&offset=0&numResults=1&numInstances=2 +GET http://localhost:15000/api/entities/summary/feed?cluster=primary-cluster&filterBy=STATUS:RUNNING&fields=status&[email protected]&orderBy=name&offset=0&numResults=1&numInstances=2 </verbatim> ---+++ Result <verbatim> Added: falcon/trunk/general/src/site/twiki/restapi/EntityTouch.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/EntityTouch.twiki?rev=1660589&view=auto ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/EntityTouch.twiki (added) +++ falcon/trunk/general/src/site/twiki/restapi/EntityTouch.twiki Wed Feb 18 10:55:56 2015 @@ -0,0 +1,29 @@ +---++ POST api/entities/touch/:entity-type/:entity-name + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Force updates the entity. + +---++ Parameters + * :entity-type can be feed or process. + * :entity-name is name of the feed or process. + +---++ Results +Result of the validation. + +---++ Examples +---+++ Rest Call +<verbatim> +POST http://localhost:15000/api/entities/touch/process/SampleProcess +</verbatim> +---+++ Result +<verbatim> +{ + "requestId": "touch\/default\/d6aaa328-6836-4818-a212-515bb43d8b86\n\n", + "message": "touch\/default\/SampleProcess updated successfully\n\n", + "status": "SUCCEEDED" +} +</verbatim> \ No newline at end of file Modified: falcon/trunk/general/src/site/twiki/restapi/EntityUpdate.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/EntityUpdate.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/EntityUpdate.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/EntityUpdate.twiki Wed Feb 18 10:55:56 2015 @@ -10,7 +10,6 @@ Updates the submitted entity. ---++ Parameters * :entity-type can be feed or process. * :entity-name is name of the feed or process. - * :effective is optional effective time ---++ Results Result of the validation. @@ -18,7 +17,7 @@ Result of the validation. ---++ Examples ---+++ Rest Call <verbatim> -POST http://localhost:15000/api/entities/update/process/SampleProcess?effective=2014-01-01T00:00Z +POST http://localhost:15000/api/entities/update/process/SampleProcess <?xml version="1.0" encoding="UTF-8"?> <!-- Daily sample process. Runs at 6th hour every day. Input - last day's hourly data. Generates output for yesterday --> <process xmlns="uri:falcon:process:0.1" name="SampleProcess" > @@ -58,7 +57,7 @@ POST http://localhost:15000/api/entities <verbatim> { "requestId": "update\/default\/d6aaa328-6836-4818-a212-515bb43d8b86\n\n", - "message": "update\/default\/SampleProcess updated successfully with effective time [(local/2014-01-01T00:00Z)]\n\n", + "message": "update\/default\/SampleProcess updated successfully\n\n", "status": "SUCCEEDED" } </verbatim> Modified: falcon/trunk/general/src/site/twiki/restapi/EntityValidate.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/EntityValidate.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/EntityValidate.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/EntityValidate.twiki Wed Feb 18 10:55:56 2015 @@ -23,7 +23,7 @@ POST http://localhost:15000/api/entities <interface type="readonly" endpoint="hftp://localhost:50070"; version="1.1.1"/> <interface type="write" endpoint="hdfs://localhost:9000" version="1.1.1"/> <interface type="execute" endpoint="localhost:9001" version="1.1.1"/> - <interface type="workflow" endpoint="http://localhost:11000/oozie/"; version="3.3.0"/> + <interface type="workflow" endpoint="http://localhost:11000/oozie/"; version="4.0.0"/> <interface type="messaging" endpoint="tcp://localhost:61616?daemon=true" version="5.4.3"/> </interfaces> <locations> Added: falcon/trunk/general/src/site/twiki/restapi/FeedInstanceListing.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/FeedInstanceListing.twiki?rev=1660589&view=auto ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/FeedInstanceListing.twiki (added) +++ falcon/trunk/general/src/site/twiki/restapi/FeedInstanceListing.twiki Wed Feb 18 10:55:56 2015 @@ -0,0 +1,45 @@ +---++ GET /api/instance/listing/feed/:entity-name + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Get falcon feed instance availability. + +---++ Parameters + * :entity-name Name of the entity. + * start <optional param> Show instances from this date. Date format is yyyy-MM-dd'T'HH:mm'Z'. + * By default, it is set to (end - (10 * entityFrequency)). + * end <optional param> Show instances up to this date. Date format is yyyy-MM-dd'T'HH:mm'Z'. + * Default is set to now. + * colo <optional param> Colo on which the query should be run. + +---++ Results +Feed instance availability status + +---++ Examples +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/instance/listing/feed/SampleFeed?colo=*&start=2012-04-03T07:00Z +</verbatim> +---+++ Result +<verbatim> +{ + "instances": [ + { + "size": "450231212222", + "creationTime": "1236679827365", + "cluster": "primary-cluster", + "uri": "/data/SampleFeed/2012-04-03", + "status": "AVAILABLE", + "instance": "2012-04-03T07:00Z" + } + ], + "requestId": "default\/3527038e-8334-4e50-8173-76c4fa430d0b\n", + "message": "default\/STATUS\n", + "status": "SUCCEEDED" +} +</verbatim> + + Modified: falcon/trunk/general/src/site/twiki/restapi/Graph.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/Graph.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/Graph.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/Graph.twiki Wed Feb 18 10:55:56 2015 @@ -1,4 +1,4 @@ ----++ GET api/graphs/lineage/serialize +---++ GET api/metadata/lineage/serialize * <a href="#Description">Description</a> * <a href="#Parameters">Parameters</a> * <a href="#Results">Results</a> @@ -16,7 +16,7 @@ Serialize graph to a file configured usi ---++ Examples ---+++ Rest Call <verbatim> -GET http://localhost:15000/api/graphs/lineage/serialize +GET http://localhost:15000/api/metadata/lineage/serialize </verbatim> ---+++ Result None. Modified: falcon/trunk/general/src/site/twiki/restapi/InstanceKill.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/InstanceKill.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/InstanceKill.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/InstanceKill.twiki Wed Feb 18 10:55:56 2015 @@ -5,12 +5,13 @@ * <a href="#Examples">Examples</a> ---++ Description -Kill a currently running instance. +Kill currently running instance(s) of an entity. ---++ Parameters * :entity-type can either be a feed or a process. * :entity-name is name of the entity. - * start start time of the entity. + * start is the start time of the instance(s) that you want to refer to + * end is the end time of the instance(s) that you want to refer to * lifecycle <optional param> can be Eviction/Replication(default) for feed and Execution(default) for process. ---++ Results @@ -19,7 +20,7 @@ Result of the kill operation. ---++ Examples ---+++ Rest Call <verbatim> -POST http://localhost:15000/api/instance/kill/process/SampleProcess?colo=*&start=2012-04-03T07:00Z +POST http://localhost:15000/api/instance/kill/process/SampleProcess?colo=*&start=2012-04-03T07:00Z&end=2014-04-03T07:00Z </verbatim> ---+++ Result <verbatim> Added: falcon/trunk/general/src/site/twiki/restapi/InstanceParams.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/InstanceParams.twiki?rev=1660589&view=auto ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/InstanceParams.twiki (added) +++ falcon/trunk/general/src/site/twiki/restapi/InstanceParams.twiki Wed Feb 18 10:55:56 2015 @@ -0,0 +1,82 @@ +---++ GET /api/instance/params/:entity-type/:entity-name + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Get the params passed to the workflow for an instance of feed/process. + +---++ Parameters + * :entity-type Valid options are cluster, feed or process. + * :entity-name Name of the entity. + * start should be the nominal time of the instance for which you want the params to be returned + * colo <optional param> Colo on which the query should be run. + * lifecycle <optional param> Valid lifecycles for feed are Eviction/Replication(default) and for process is Execution(default). + + +---++ Results +List of instances currently running. + +---++ Examples +---+++ Rest Call +<verbatim> +http://userqa.user.com:16000/api/instance/params/process/Sample-Process?start=2014-10-01T11:00Z&colo=* +</verbatim> +---+++ Result +{ + "status": "SUCCEEDED", + "message": "ua1/PARAMS\n", + "requestId": "ua1/807e9fe6-ba60-490e-b720-f8dc8b92063e\n", + "instances": [ + { + "instance": "2014-10-01T11:00Z", + "status": "RUNNING", + "logFile": "http://spyke.user.com:11000/oozie?job=0000211-141117203201940-oozie-oozi-W";, + "cluster": "sample-cluster", + "startTime": "2014-11-19T19:46:29+08:00", + "details": "", + "actions": [ + { + "action": "succeeded-post-processing", + "status": "RUNNING", + "logFile": "http://spyke.user.com:50030/jobdetails.jsp?jobid=job_201411071450_1052"; + } + ], + "params": { + "entry": {"key": "jobTracker", "value": "10.16.114.113:8021"}, + "entry":{"key":"falconInputNames","value":"IGNORE"}, + "entry":{"key":"shouldRecord","value":"false"}, + "entry":{"key":"timeStamp","value":"2014-11-19-11-46"}, + "entry":{"key":"falconInPaths","value":"IGNORE"}, + "entry":{"key":"broker.url","value":"tcp://localhost:61616"}, + "entry":{"key":"feedNames","value":"NONE"}, + "entry":{"key":"falcon.libpath","value":"/path/falcon/sample/lib"}, + "entry":{"key":"ENTITY_PATH","value":"/path/falcon/staging/falcon/workflows/process/Sample-Process/9506be19980e0e6fdb709e1baffff_1416397585511/DEFAULT"}, + "entry":{"key":"entityType","value":"process"}, + "entry":{"key":"nominalTime","value":"2014-10-01-11-00"}, + "entry":{"key":"feedInstancePaths","value":"IGNORE"}, + "entry":{"key":"oozie.bundle.application.path","value":"hdfs://10.16.104.13:8020/path/falcon/staging/falcon/workflows/process/Sample-Process/9506be19980e0e669709e1baffff_1416397585511"}, + "entry":{"key":"logDir","value":"hdfs://10.16.104.13:8020/path/falcon/staging/falcon/workflows/process/Sample-Process/logs"}, + "entry":{"key":"userWorkflowEngine","value":"oozie"}, + "entry":{"key":"broker.ttlInMins","value":"4320"}, + "entry":{"key":"oozie.use.system.libpath","value":"true"}, + "entry":{"key":"queueName","value":"reports"}, + "entry":{"key":"falconDataOperation","value":"GENERATE"}, + "entry":{"key":"oozie.wf.external.id","value":"Sample-Process/DEFAULT/2014-10-01T11:00Z"}, + "entry":{"key":"workflowEngineUrl","value":"http://10.11.100.10:11000/oozie/"}, + "entry":{"key":"userBrokerImplClass","value":"org.apache.activemq.ActiveMQConnectionFactory"}, + "entry":{"key":"ENTITY_NAME","value":"FALCON_PROCESS_DEFAULT_Sample-Process"}, + "entry":{"key":"broker.impl.class","value":"org.apache.activemq.ActiveMQConnectionFactory"}, + "entry":{"key":"userWorkflowName","value":"Sample-workflow"}, + "entry":{"key":"entityName","value":"Sample-Process"}, + "entry":{"key":"srcClusterName","value":"NA"}, + "entry":{"key":"userBrokerUrl","value":"tcp://localhost:61616?daemon=true"}, + "entry":{"key":"falconInputFeeds","value":"NONE"}, + "entry":{"key":"user.name","value":"sampleuser"}, + "entry":{"key":"threedaysback","value":"2014-09-28"}, + "entry":{"key":"userWorkflowVersion","value":"1.0"} + } + } + ] +} \ No newline at end of file Modified: falcon/trunk/general/src/site/twiki/restapi/InstanceRerun.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/InstanceRerun.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/InstanceRerun.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/InstanceRerun.twiki Wed Feb 18 10:55:56 2015 @@ -5,12 +5,13 @@ * <a href="#Examples">Examples</a> ---++ Description -Rerun a specific instance of an entity. +Rerun instances of an entity. ---++ Parameters * :entity-type can either be a feed or a process. * :entity-name is name of the entity. * start is the start time of the instance that you want to refer to + * end is the end time of the instance that you want to refer to * lifecycle <optional param> can be Eviction/Replication(default) for feed and Execution(default) for process. ---++ Results @@ -19,7 +20,7 @@ Results of the rerun command. ---++ Examples ---+++ Rest Call <verbatim> -POST http://localhost:15000/api/instance/rerun/process/SampleProcess?colo=*&start=2012-04-03T07:00Z +POST http://localhost:15000/api/instance/rerun/process/SampleProcess?colo=*&start=2013-04-03T07:00Z&end=2014-04-03T07:00Z </verbatim> ---+++ Result <verbatim> Modified: falcon/trunk/general/src/site/twiki/restapi/InstanceResume.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/InstanceResume.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/InstanceResume.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/InstanceResume.twiki Wed Feb 18 10:55:56 2015 @@ -5,12 +5,13 @@ * <a href="#Examples">Examples</a> ---++ Description -Resume a specific instance of an entity. +Resume suspended instances of an entity. ---++ Parameters * :entity-type can either be a feed or a process. * :entity-name is name of the entity. - * start is the start time of the instance that you want to refer to + * start is the start time of the instance(s) that you want to refer to + * end is the end time of the instance(s) that you want to refer to * lifecycle <optional param> can be Eviction/Replication(default) for feed and Execution(default) for process. ---++ Results Results of the resume command. @@ -18,7 +19,7 @@ Results of the resume command. ---++ Examples ---+++ Rest Call <verbatim> -POST http://localhost:15000/api/instance/resume/process/SampleProcess?colo=*&start=2012-04-03T07:00Z +POST http://localhost:15000/api/instance/resume/process/SampleProcess?colo=*&start=2012-04-03T07:00Z&end=2014-04-03T07:00Z </verbatim> ---+++ Result <verbatim> Modified: falcon/trunk/general/src/site/twiki/restapi/InstanceStatus.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/InstanceStatus.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/InstanceStatus.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/InstanceStatus.twiki Wed Feb 18 10:55:56 2015 @@ -27,12 +27,12 @@ Get status of a specific instance of an ---++ Results -Status of the specified instance. +Status of the specified instance along with job urls for all actions of user workflow and non-succeeded actions of the main-workflow. ---++ Examples ---+++ Rest Call <verbatim> -GET http://localhost:15000/api/instance/status/process/SampleProcess?colo=*&start=2012-04-03T07:00Z +GET https://localhost:15443/api/instance/status/process/WordCount?start=2014-11-04T16:00Z&colo=* </verbatim> ---+++ Result <verbatim> @@ -40,15 +40,22 @@ GET http://localhost:15000/api/instance/ "instances": [ { "details": "", - "endTime": "2013-10-21T14:40:26-07:00", - "startTime": "2013-10-21T14:39:56-07:00", - "cluster": "primary-cluster", - "logFile": "http:\/\/localhost:11000\/oozie?job=0000070-131021115933395-oozie-rgau-W", + "endTime": "2014-11-05T16:08:10+05:30", + "startTime": "2014-11-05T16:07:29+05:30", + "cluster": "local", + "logFile": "http:\/\/localhost:11000\/oozie?job=0000011-141105155430303-oozie-oozi-W", "status": "SUCCEEDED", - "instance": "2012-04-03T07:00Z" + "instance": "2014-11-04T16:00Z", + "actions": [ + { + "action": "wordcount-mr", + "status": "SUCCEEDED", + "logFile": "http:\/\/localhost:50030\/jobdetails.jsp?jobid=job_201411051553_0005" + } + ] } ], - "requestId": "default\/e15bb378-d09f-4911-9df2-5334a45153d2\n", + "requestId": "default\/b9fc3cba-1b46-4d1f-8196-52c795ea3580\n", "message": "default\/STATUS\n", "status": "SUCCEEDED" } Modified: falcon/trunk/general/src/site/twiki/restapi/InstanceSuspend.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/InstanceSuspend.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/InstanceSuspend.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/InstanceSuspend.twiki Wed Feb 18 10:55:56 2015 @@ -5,12 +5,13 @@ * <a href="#Examples">Examples</a> ---++ Description -Suspend a specific instance of an entity. +Suspend instances of an entity. ---++ Parameters * :entity-type can either be a feed or a process. * :entity-name is name of the entity. - * start is the start time of the instance that you want to refer to + * start is the start time of the instance(s) that you want to refer to + * end is the end time of the instance(s) that you want to refer to * lifecycle <optional param> can be Eviction/Replication(default) for feed and Execution(default) for process. ---++ Results @@ -19,7 +20,7 @@ Results of the suspend command. ---++ Examples ---+++ Rest Call <verbatim> -POST http://localhost:15000/api/instance/suspend/process/SampleProcess?colo=*&start=2012-04-03T07:00Z +POST http://localhost:15000/api/instance/suspend/process/SampleProcess?colo=*&start=2012-04-03T07:00Z&end=2014-04-03T07:00Z </verbatim> ---+++ Result <verbatim> Added: falcon/trunk/general/src/site/twiki/restapi/MetadataList.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/MetadataList.twiki?rev=1660589&view=auto ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/MetadataList.twiki (added) +++ falcon/trunk/general/src/site/twiki/restapi/MetadataList.twiki Wed Feb 18 10:55:56 2015 @@ -0,0 +1,30 @@ +---++ GET api/metadata/discovery/:type/list + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Get all dimensions of specified type. + +---++ Parameters + * :type Valid dimension types are cluster_entity,feed_entity, process_entity, user, colo, tags, groups, pipelines + * cluster <optional query param> Show dimensions related to this cluster. + + +---++ Results +List of dimensions that match requested type [and cluster]. + +---++ Examples +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/metadata/discovery/process_entity/list?cluster=primary-cluster +</verbatim> +---+++ Result +<verbatim> +{ + "results": ["sampleIngestProcess","testProcess","anotherProcess"], + "totalSize": 3 +} +</verbatim> + Added: falcon/trunk/general/src/site/twiki/restapi/MetadataRelations.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/MetadataRelations.twiki?rev=1660589&view=auto ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/MetadataRelations.twiki (added) +++ falcon/trunk/general/src/site/twiki/restapi/MetadataRelations.twiki Wed Feb 18 10:55:56 2015 @@ -0,0 +1,45 @@ +---++ GET api/metadata/discovery/:dimension-type/:dimension-name/relations + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Get all relations of a specific dimension. + +---++ Parameters + * :type Valid dimension types are cluster_entity,feed_entity, process_entity, user, colo, tags, groups, pipelines + * :name Name of the dimension. + +---++ Results +Get all relations of a specific dimension. + +---++ Examples +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/metadata/discovery/process_entity/sample-process/relations +</verbatim> +---+++ Result +<verbatim> +{ + "timestamp":"2014-09-09T01:31Z", + "userWorkflowEngine":"pig", + "name":"sample-process", + "type":"PROCESS_ENTITY", + "userWorkflowName":"imp-click-join-workflow", + "version":"1.0.9", + "inVertices":[ + {"name":"clicks-feed","type":"FEED_ENTITY","label":"input"}, + {"name":"impression-feed","type":"FEED_ENTITY","label":"input"}, + {"name":"sample-process\/2014-01-01T01:00Z","type":"PROCESS_INSTANCE","label":"instance-of"} + ], + "outVertices":[ + {"name":"Critical","type":"TAGS","label":"classified-as"}, + {"name":"testPipeline","type":"PIPELINES","label":"pipeline"}, + {"name":"primary-cluster","type":"CLUSTER_ENTITY","label":"runs-on"}, + {"name":"imp-click-join2","type":"FEED_ENTITY","label":"output"}, + {"name":"imp-click-join1","type":"FEED_ENTITY","label":"output"}, + {"name":"falcon-user","type":"USER","label":"owned-by"} + ] +} +</verbatim> Modified: falcon/trunk/general/src/site/twiki/restapi/ResourceList.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/ResourceList.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/ResourceList.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/ResourceList.twiki Wed Feb 18 10:55:56 2015 @@ -5,6 +5,7 @@ * <a href="#REST_Call_on_Feed_and_Process_Instances">REST Call on Feed/Process Instances</a> * <a href="#REST_Call_on_Admin_Resource">REST Call on Admin Resource</a> * <a href="#REST_Call_on_Lineage_Graph">REST Call on Lineage Graph Resource</a> + * <a href="#REST_Call_on_Metadata_Resource">REST Call on Metadata Resource</a> ---++ Authentication @@ -56,6 +57,7 @@ See also: [[../Security.twiki][Security | *Call Type* | *Resource* | *Description* | | GET | [[InstanceRunning][api/instance/running/:entity-type/:entity-name]] | List of running instances. | +| GET | [[InstanceParams][api/instance/params/:entity-type/:entity-name]] | List of entity instances along with their workflow params. | | GET | [[InstanceList][api/instance/list/:entity-type/:entity-name]] | List of instances | | GET | [[InstanceStatus][api/instance/status/:entity-type/:entity-name]] | Status of a given instance | | POST | [[InstanceKill][api/instance/kill/:entity-type/:entity-name]] | Kill a given instance | @@ -65,14 +67,21 @@ See also: [[../Security.twiki][Security | GET | [[InstanceLogs][api/instance/logs/:entity-type/:entity-name]] | Get logs of a given instance | | GET | [[InstanceSummary][api/instance/summary/:entity-type/:entity-name]] | Return summary of instances for an entity | ----++ REST Call on Lineage Graph +---++ REST Call on Metadata Lineage Resource -| *Call Type* | *Resource* | *Description* | -| GET | [[Graph][api/graphs/lineage/serialize]] | dump the graph | -| GET | [[AllVertices][api/graphs/lineage/vertices/all]] | get all vertices | -| GET | [[Vertices][api/graphs/lineage/vertices?key=:key&value=:value]] | get all vertices for a key index | -| GET | [[Vertex][api/graphs/lineage/vertices/:id]] | get the vertex with the specified id | -| GET | [[VertexProperties][api/graphs/lineage/vertices/properties/:id?relationships=:true]] | get the properties of the vertex with the specified id | -| GET | [[AdjacentVertices][api/graphs/lineage/vertices/:id/:direction]] | get the adjacent vertices or edges of the vertex with the specified direction | -| GET | [[AllEdges][api/graphs/lineage//edges/all]] | get all edges | -| GET | [[Edge][api/graphs/lineage/edges/:id]] | get the edge with the specified id | +| *Call Type* | *Resource* | *Description* | +| GET | [[Graph][api/metadata/lineage/serialize]] | dump the graph | +| GET | [[AllVertices][api/metadata/lineage/vertices/all]] | get all vertices | +| GET | [[Vertices][api/metadata/lineage/vertices?key=:key&value=:value]] | get all vertices for a key index | +| GET | [[Vertex][api/metadata/lineage/vertices/:id]] | get the vertex with the specified id | +| GET | [[VertexProperties][api/metadata/lineage/vertices/properties/:id?relationships=:true]] | get the properties of the vertex with the specified id | +| GET | [[AdjacentVertices][api/metadata/lineage/vertices/:id/:direction]] | get the adjacent vertices or edges of the vertex with the specified direction | +| GET | [[AllEdges][api/metadata/lineage/edges/all]] | get all edges | +| GET | [[Edge][api/metadata/lineage/edges/:id]] | get the edge with the specified id | +| GET | [[EntityLineage][api/metadata/lineage/entities?pipeline=:name]] | Get lineage graph for processes and feeds in the specified pipeline | + +---++ REST Call on Metadata Discovery Resource + +| *Call Type* | *Resource* | *Description* | +| GET | [[MetadataList][api/metadata/discovery/:dimension-type/list]] | list of dimensions | +| GET | [MetadataRelations][api/metadata/discovery/:dimension-type/:dimension-name/relations]] | Return all relations of a dimension | Modified: falcon/trunk/general/src/site/twiki/restapi/Vertex.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/Vertex.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/Vertex.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/Vertex.twiki Wed Feb 18 10:55:56 2015 @@ -1,4 +1,4 @@ ----++ GET api/graphs/lineage/vertices/:id +---++ GET api/metadata/lineage/vertices/:id * <a href="#Description">Description</a> * <a href="#Parameters">Parameters</a> * <a href="#Results">Results</a> @@ -16,7 +16,7 @@ Vertex with the specified id. ---++ Examples ---+++ Rest Call <verbatim> -GET http://localhost:15000/api/graphs/lineage/vertices/4 +GET http://localhost:15000/api/metadata/lineage/vertices/4 </verbatim> ---+++ Result <verbatim> Modified: falcon/trunk/general/src/site/twiki/restapi/VertexProperties.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/VertexProperties.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/VertexProperties.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/VertexProperties.twiki Wed Feb 18 10:55:56 2015 @@ -1,4 +1,4 @@ ----++ GET api/graphs/lineage/vertices/properties/:id?relationships=:true +---++ GET api/metadata/lineage/vertices/properties/:id?relationships=:true * <a href="#Description">Description</a> * <a href="#Parameters">Parameters</a> * <a href="#Results">Results</a> @@ -17,7 +17,7 @@ Gets the properties of the vertex with s ---++ Examples ---+++ Rest Call <verbatim> -GET http://localhost:15000/api/graphs/lineage/vertices/properties/40004?relationships=true +GET http://localhost:15000/api/metadata/lineage/vertices/properties/40004?relationships=true </verbatim> ---+++ Result <verbatim> Modified: falcon/trunk/general/src/site/twiki/restapi/Vertices.twiki URL: http://svn.apache.org/viewvc/falcon/trunk/general/src/site/twiki/restapi/Vertices.twiki?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/general/src/site/twiki/restapi/Vertices.twiki (original) +++ falcon/trunk/general/src/site/twiki/restapi/Vertices.twiki Wed Feb 18 10:55:56 2015 @@ -1,4 +1,4 @@ ----++ GET api/graphs/lineage/vertices?key=:key&value=:value +---++ GET api/metadata/lineage/vertices?key=:key&value=:value * <a href="#Description">Description</a> * <a href="#Parameters">Parameters</a> * <a href="#Results">Results</a> @@ -17,7 +17,7 @@ All vertices matching given property key ---++ Examples ---+++ Rest Call <verbatim> -GET http://localhost:15000/api/graphs/lineage/vertices?key=name&value=sampleIngestProcess +GET http://localhost:15000/api/metadata/lineage/vertices?key=name&value=sampleIngestProcess </verbatim> ---+++ Result <verbatim> Modified: falcon/trunk/releases/0.3-incubating/src/site/site.xml URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.3-incubating/src/site/site.xml?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/releases/0.3-incubating/src/site/site.xml (original) +++ falcon/trunk/releases/0.3-incubating/src/site/site.xml Wed Feb 18 10:55:56 2015 @@ -54,7 +54,7 @@ <body> <links> <item name="0.3-incubating" title="0.3-incubating" - href="http://www.apache.org/dist/incubator/falcon"; position="none"/> + href="http://archive.apache.org/dist/incubator/falcon/"; position="none"/> <item name="Released: 2013-08-15" title="Released: 2013-08-15" href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12314429&version=12324423"; position="none"/> </links> Modified: falcon/trunk/releases/0.4-incubating/src/site/site.xml URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.4-incubating/src/site/site.xml?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/releases/0.4-incubating/src/site/site.xml (original) +++ falcon/trunk/releases/0.4-incubating/src/site/site.xml Wed Feb 18 10:55:56 2015 @@ -54,7 +54,7 @@ <body> <links> <item name="0.4-incubating" title="0.4-incubating" - href="http://www.apache.org/dist/incubator/falcon"; position="none"/> + href="http://archive.apache.org/dist/incubator/falcon/"; position="none"/> <item name="Released: 2014-02-02" title="Released: 2014-02-02" href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12314429&version=12324424"; position="none"/> </links> Modified: falcon/trunk/releases/0.5-incubating/src/site/site.xml URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.5-incubating/src/site/site.xml?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/releases/0.5-incubating/src/site/site.xml (original) +++ falcon/trunk/releases/0.5-incubating/src/site/site.xml Wed Feb 18 10:55:56 2015 @@ -54,7 +54,7 @@ <body> <links> <item name="0.5-incubating" title="0.5-incubating" - href="http://www.apache.org/dist/incubator/falcon"; position="none"/> + href="http://archive.apache.org/dist/incubator/falcon/"; position="none"/> <item name="Released: 2014-09-22" title="Released: 2014-09-22" href="http://s.apache.org/falcon-0.5-release-notes"; position="none"/> </links> Modified: falcon/trunk/releases/0.6-incubating/src/site/site.xml URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6-incubating/src/site/site.xml?rev=1660589&r1=1660588&r2=1660589&view=diff ============================================================================== --- falcon/trunk/releases/0.6-incubating/src/site/site.xml (original) +++ falcon/trunk/releases/0.6-incubating/src/site/site.xml Wed Feb 18 10:55:56 2015 @@ -53,10 +53,10 @@ <body> <links> - <item name="0.5-incubating" title="0.5-incubating" - href="http://www.apache.org/dist/incubator/falcon"; position="none"/> - <item name="Released: 2014-09-22" title="Released: 2014-09-22" - href="http://s.apache.org/falcon-0.5-release-notes"; position="none"/> + <item name="0.6-incubating" title="0.6-incubating" + href="http://archive.apache.org/dist/incubator/falcon"; position="none"/> + <item name="Released: 2014-12-05" title="Released: 2014-12-05" + href="http://s.apache.org/falcon-0.6-release-notes"; position="none"/> </links> <breadcrumbs>
