http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/DefineExtension.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/DefineExtension.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/DefineExtension.twiki new file mode 100644 index 0000000..c260911 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/DefineExtension.twiki @@ -0,0 +1,8 @@ +---+++Definition + +[[CommonCLI][Common CLI Options]] + +Definition of an extension. Outputs a JSON document describing the extension invocation parameters. + +Usage: +$FALCON_HOME/bin/falcon extension Âdefinition Âname <extensionÂname>
http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/Definition.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/Definition.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/Definition.twiki new file mode 100644 index 0000000..08d46c7 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/Definition.twiki @@ -0,0 +1,8 @@ +---+++Definition + +[[CommonCLI][Common CLI Options]] + +Definition option returns the entity definition submitted earlier during submit step. + +Usage: +$FALCON_HOME/bin/falcon entity -type [cluster|datasource|feed|process] -name <<name>> -definition http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/DeleteEntity.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/DeleteEntity.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/DeleteEntity.twiki new file mode 100644 index 0000000..cc07406 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/DeleteEntity.twiki @@ -0,0 +1,8 @@ +---+++Delete + +[[CommonCLI][Common CLI Options]] + +Delete removes the submitted entity definition for the specified entity and put it into the archive.Archive path is defined in startup.properties in variable "config.store.uri". + +Usage: +$FALCON_HOME/bin/falcon entity -type [cluster|datasource|feed|process] -name <<name>> -delete http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/DependencyEntity.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/DependencyEntity.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/DependencyEntity.twiki new file mode 100644 index 0000000..bdef1d7 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/DependencyEntity.twiki @@ -0,0 +1,10 @@ +---+++Dependency + +[[CommonCLI][Common CLI Options]] + +With the use of dependency option, we can list all the entities on which the specified entity is dependent. +For example for a feed, dependency return the cluster name and for process it returns all the input feeds, +output feeds and cluster names. + +Usage: +$FALCON_HOME/bin/falcon entity -type [cluster|datasource|feed|process] -name <<name>> -dependency \ No newline at end of file http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/DependencyInstance.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/DependencyInstance.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/DependencyInstance.twiki new file mode 100644 index 0000000..51508cc --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/DependencyInstance.twiki @@ -0,0 +1,33 @@ +---+++Dependency +Display the dependent instances which are dependent on the given instance. For example for a given process instance it will +list all the input feed instances(if any) and the output feed instances(if any). + +An example use case of this command is as follows: +Suppose you find out that the data in a feed instance was incorrect and you need to figure out which all process instances +consumed this feed instance so that you can reprocess them after correcting the feed instance. You can give the feed instance +and it will tell you which process instance produced this feed and which all process instances consumed this feed. + +NOTE: +1. instanceTime must be a valid instanceTime e.g. instanceTime of a feed should be in it's validity range on applicable clusters, + and it should be in the range of instances produced by the producer process(if any) + +2. For processes with inputs like latest() which vary with time the results are not guaranteed to be correct. + +Usage: +$FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -dependency -instanceTime "yyyy-MM-dd'T'HH:mm'Z'" + +For example: +$FALCON_HOME/bin/falcon instance -dependency -type feed -name out -instanceTime 2014-12-15T00:00Z +name: producer, type: PROCESS, cluster: local, instanceTime: 2014-12-15T00:00Z, tags: Output +name: consumer, type: PROCESS, cluster: local, instanceTime: 2014-12-15T00:03Z, tags: Input +name: consumer, type: PROCESS, cluster: local, instanceTime: 2014-12-15T00:04Z, tags: Input +name: consumer, type: PROCESS, cluster: local, instanceTime: 2014-12-15T00:02Z, tags: Input +name: consumer, type: PROCESS, cluster: local, instanceTime: 2014-12-15T00:05Z, tags: Input + + +Response: default/Success! + +Request Id: default/1125035965@qtp-503156953-7 - 447be0ad-1d38-4dce-b438-20f3de69b172 + + +<a href="../Restapi/InstanceDependencies.html">Optional params described here.</a> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/DescribeExtension.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/DescribeExtension.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/DescribeExtension.twiki new file mode 100644 index 0000000..9f9895e --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/DescribeExtension.twiki @@ -0,0 +1,8 @@ +---+++Describe + +[[CommonCLI][Common CLI Options]] + +Description of an extension. Outputs the README of the specified extension. + +Usage: +$FALCON_HOME/bin/falcon extension Âdescribe Âname <extensionÂname> http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/EdgeMetadata.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/EdgeMetadata.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/EdgeMetadata.twiki new file mode 100644 index 0000000..477996e --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/EdgeMetadata.twiki @@ -0,0 +1,11 @@ +---+++ Edge + +[[CommonCLI][Common CLI Options]] + +Get the edge with the specified id. + +Usage: +$FALCON_HOME/bin/falcon metadata -edge -id <<id>> + +Example: +$FALCON_HOME/bin/falcon metadata -edge -id Q9n-Q-5g \ No newline at end of file http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/EnumerateExtension.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/EnumerateExtension.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/EnumerateExtension.twiki new file mode 100644 index 0000000..0b28630 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/EnumerateExtension.twiki @@ -0,0 +1,8 @@ +---+++Enumerate + +[[CommonCLI][Common CLI Options]] + +List all the extensions supported. Returns total number of results and a list of server side extensions supported. + +Usage: +$FALCON_HOME/bin/falcon extension Âenumerate \ No newline at end of file http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionDelete.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionDelete.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionDelete.twiki new file mode 100644 index 0000000..bb867b5 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionDelete.twiki @@ -0,0 +1,15 @@ +---+++Delete + +[[CommonCLI][Common CLI Options]] + +Delete an extension job. + +Usage: +$FALCON_HOME/bin/falcon extension -delete -jobName <<extension-job-name>> + +Optional Args : -doAs <<user-name>> + +<a href="../restapi/ExtensionDelete.html">Parameters and examples described here.</a> + +Example: +$FALCON_HOME/bin/falcon extension -delete -jobName sales-monthly http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionInstances.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionInstances.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionInstances.twiki new file mode 100644 index 0000000..765a5bf --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionInstances.twiki @@ -0,0 +1,18 @@ +---+++Instances + +[[CommonCLI][Common CLI Options]] + +List instances of an extension job. + +Usage: +$FALCON_HOME/bin/falcon extension -instances -jobName <<extension-job-name>> + +Optional Args : -doAs <<user-name>> +-start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" -fields <<field1,field2>> +-instanceStatus <<instance-status>> -orderBy <<order-by-field>> -sortOrder <<sort-order>> +-offset <<offset-number>> -numResults <<number-of-results>> + +<a href="../restapi/ExtensionInstances.html">Parameters and examples described here.</a> + +Example: +$FALCON_HOME/bin/falcon extension -instances -jobName daily-health-bill -start 2012-04-01T00:00 http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionList.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionList.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionList.twiki new file mode 100644 index 0000000..e55f88b --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionList.twiki @@ -0,0 +1,16 @@ +---+++List + +[[CommonCLI][Common CLI Options]] + +List jobs generated from an extension. + +Usage: +$FALCON_HOME/bin/falcon extension -list -extensionName <<extension-name>> + +Optional Args : -doAs <<user-name>> +-sortOrder <<sortOrder>> + +<a href="../restapi/ExtensionList.html">Parameters and examples described here.</a> + +Example: +$FALCON_HOME/bin/falcon extension -list -extensionName hdfs-mirroring http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionResume.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionResume.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionResume.twiki new file mode 100644 index 0000000..4a62d04 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionResume.twiki @@ -0,0 +1,15 @@ +---+++Resume + +[[CommonCLI][Common CLI Options]] + +Resume an extension job. + +Usage: +$FALCON_HOME/bin/falcon extension -resume -jobName <<extension-job-name>> + +Optional Args : -doAs <<user-name>> + +<a href="../restapi/ExtensionResume.html">Parameters and examples described here.</a> + +Example: +$FALCON_HOME/bin/falcon extension -resume -jobName sales-monthly http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionSchedule.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionSchedule.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionSchedule.twiki new file mode 100644 index 0000000..813d7b7 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionSchedule.twiki @@ -0,0 +1,15 @@ +---+++Schedule + +[[CommonCLI][Common CLI Options]] + +Schedule an extension job. + +Usage: +$FALCON_HOME/bin/falcon extension -schedule -jobName <<extension-job-name>> + +Optional Args : -doAs <<user-name>> + +<a href="../restapi/ExtensionSchedule.html">Parameters and examples described here.</a> + +Example: +$FALCON_HOME/bin/falcon extension -schedule -jobName sales-monthly http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionSubmit.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionSubmit.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionSubmit.twiki new file mode 100644 index 0000000..2cda478 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionSubmit.twiki @@ -0,0 +1,15 @@ +---+++Submit + +[[CommonCLI][Common CLI Options]] + +Submit an extension job. + +Usage: +$FALCON_HOME/bin/falcon extension -submit -extensionName <<extension-name>> -jobName <<job-name>> -file <<path-to-file>> + +Optional Args : -doAs <<user-name>> + +<a href="../restapi/ExtensionSubmit.html">Parameters and examples described here.</a> + +Example: +$FALCON_HOME/bin/falcon extension -submit -extensionName hdfs-mirroring -file /data/hdfs-mirroring-para.txt http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionSubmitAndSchedule.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionSubmitAndSchedule.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionSubmitAndSchedule.twiki new file mode 100644 index 0000000..4dbd76d --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionSubmitAndSchedule.twiki @@ -0,0 +1,15 @@ +---+++SubmitAndSchedule + +[[CommonCLI][Common CLI Options]] + +Submit and schedule an extension job. + +Usage: +$FALCON_HOME/bin/falcon extension -submitAndSchedule -extensionName <<extension-name>> -jobName <<job-name>> -file <<path-to-file>> + +Optional Args : -doAs <<user-name>> + +<a href="../restapi/ExtensionSubmitAndSchedule.html">Parameters and examples described here.</a> + +Example: +$FALCON_HOME/bin/falcon extension -submitAndSchedule -extensionName hdfs-mirroring -file /data/hdfs-mirroring-para.txt http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionSuspend.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionSuspend.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionSuspend.twiki new file mode 100644 index 0000000..dd6b69c --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionSuspend.twiki @@ -0,0 +1,15 @@ +---+++Suspend + +[[CommonCLI][Common CLI Options]] + +Suspend an extension job. + +Usage: +$FALCON_HOME/bin/falcon extension -suspend -jobName <<extension-job-name>> + +Optional Args : -doAs <<user-name>> + +<a href="../restapi/ExtensionSuspend.html">Parameters and examples described here.</a> + +Example: +$FALCON_HOME/bin/falcon extension -suspend -jobName sales-monthly http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionUpdate.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionUpdate.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionUpdate.twiki new file mode 100644 index 0000000..fa0985e --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionUpdate.twiki @@ -0,0 +1,15 @@ +---+++Update + +[[CommonCLI][Common CLI Options]] + +Update an extension entity. Note that we require all the entity names to remain the same. Future work will allow different entity names. + +Usage: +$FALCON_HOME/bin/falcon extension -update -extensionName <<extension-name>> -file <<path-to-file>> + +Optional Args : -doAs <<user-name>> + +<a href="../restapi/ExtensionUpdate.html">Parameters and examples described here.</a> + +Example: +$FALCON_HOME/bin/falcon extension -update -extensionName hdfs-mirroring -file /data/hdfs-mirroring-para.txt http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionValidate.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionValidate.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionValidate.twiki new file mode 100644 index 0000000..d788a20 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/ExtensionValidate.twiki @@ -0,0 +1,15 @@ +---+++Validate + +[[CommonCLI][Common CLI Options]] + +Validate the input for extension job submission. + +Usage: +$FALCON_HOME/bin/falcon extension -validate -extensionName <<extension-name>> -file <<path-to-file>> + +Optional Args : -doAs <<user-name>> + +<a href="../restapi/ExtensionValidate.html">Parameters and examples described here.</a> + +Example: +$FALCON_HOME/bin/falcon extension -validate -extensionName hdfs-mirroring -file /data/hdfs-mirroring-para.txt http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/FalconCLI.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/FalconCLI.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/FalconCLI.twiki new file mode 100644 index 0000000..9ae0646 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/FalconCLI.twiki @@ -0,0 +1,130 @@ +---+FalconCLI + +FalconCLI is a interface between user and Falcon. It is a command line utility provided by Falcon. FalconCLI supports Entity Management, Instance Management and Admin operations.There is a set of web services that are used by FalconCLI to interact with Falcon. + +---+++Types of CLI Options + +CLI options are classified into : + + * <a href="#Common_CLI_Options">Common CLI Options</a> + * <a href="#Entity_Management_Commands">Entity Management Commands</a> + * <a href="#Instance_Management_Commands">Instance Management Commands</a> + * <a href="#Metadata_Commands">Metadata Commands</a> + * <a href="#Admin_Commands">Admin commands</a> + * <a href="#Extension_Artifacts_Commands">Extension artifacts commands</a> + * <a href="#Extension_Commands">Extension commands</a> + + + +----------- + +---++Common CLI Options + +---+++Falcon URL + +Optional -url option indicating the URL of the Falcon system to run the command against can be provided. If not mentioned it will be picked from the system environment variable FALCON_URL. If FALCON_URL is not set then it will be picked from client.properties file. If the option is not +provided and also not set in client.properties, Falcon CLI will fail. + +---+++Proxy user support + +The -doAs option allows the current user to impersonate other users when interacting with the Falcon system. The current user must be configured as a proxyuser in the Falcon system. The proxyuser configuration may restrict from +which hosts a user may impersonate users, as well as users of which groups can be impersonated. + +<a href="../FalconDocumentation.html#Proxyuser_support">Proxyuser support described here.</a> + +---+++Debug Mode + +If you export FALCON_DEBUG=true then the Falcon CLI will output the Web Services API details used by any commands you execute. This is useful for debugging purposes to or see how the Falcon CLI works with the WS API. +Alternately, you can specify '-debug' through the CLI arguments to get the debug statements. +Example: +$FALCON_HOME/bin/falcon entity -submit -type cluster -file /cluster/definition.xml -debug + +----------- + +---++Entity Management Commands + +| *Command* | *Description* | +| [[Submit]] | Submit the entity definition. | +| [[Schedule]] | Scheduled the entity | +| [[SuspendEntity][Suspend]] | Suspends the scheduled entity | +| [[ResumeEntity][Resume]] | Puts a suspended entity back in action | +| [[DeleteEntity][Delete]] | Remove the submitted entity | +| [[ListEntity][List]] | Lists the particular type of entity | +| [[SummaryEntity][Summary]] | Shows summary of the type of entity | +| [[UpdateEntity][Update]] | Update already submitted entity | +| [[Touch]] | Force update already submitted entity | +| [[StatusEntity][Status]] | Return's the status of the entity | +| [[DependencyEntity][Dependency]] | List all the entities on which the specified entity is dependent| +| [[Definition]] | Return's the definition of the entity | +| [[Lookup]] | Return's the feed name for a path | +| [[SLAAlert]] | Return's the feed instance which have missed sla| + + +----------- +---++Instance Management Commands + +| *Command* | *Description* | +| [[KillInstance][Kill]] | Kills all the instances of specified process | +| [[SuspendInstance][Suspend]] | Suspends instances of a specified process | +| [[ContinueInstance][Continue]] | Continue the failed workflow instances | +| [[RerunInstance][Rerun]] | Rerun instances of specified process | +| [[ResumeInstance][Resume]] | Resume instance of specified process from suspended state | +| [[SearchInstance][Search]] | Search instances | +| [[StatusInstance][Status]] | Gets the status of instances | +| [[ListInstance][List]] | Gets single or multiple instances | +| [[SummaryInstance][Summary]] | Gets consolidated status of the instances between the specified time period | +| [[RunningInstance][Running]] | Gets running instances of the mentioned process | +| [[FeedInstanceListing]] | Gets falcon feed instance availability | +| [[LogsInstance][Logs]] | Gets logs for instance | +| [[LifeCycleInstance][LifeCycle]] | Describes list of life cycles of a entity | +| [[TriageInstance][Triage]] | Traces entities ancestors for failure | +| [[ParamsInstance][Params]] | Displays workflow params | +| [[DependencyInstance][Dependency]] | Displays the dependent instances | + +----------- + +---++Metadata Commands + +| *Command* | *Description* | +|[[LineageMetadata][Lineage]] | Returns the relationship between processes and feeds | +|[[VertexMetadata][Vertex]] | Gets the vertex with the specified id | +|[[VerticesMetadata][Vertices]] | Gets all vertices for a key | +|[[VertexEdgesMetadata][Vertex Edges]] | Gets the adjacent vertices or edges of the vertex| +|[[EdgeMetadata][Edge]] | Gets the edge with the specified id | +|[[ListMetadata][List]] | Return list of all dimension of given type | +|[[RelationMetadata][Relations]] | Return all dimensions related to specified Dimension | + +----------- + +---++Admin Commands + +| *Command* | *Description* | +|[[HelpAdmin][Help]] | Return help options | +|[[VersionAdmin][Version]] | Return current falcon version | +|[[StatusAdmin][Status]] | Return the status of falcon | +|[[SafemodeAdmin][SetSafeMode]] | Set/unset Falcon Server in safemode | + +----------- + +---++Extension Artifacts Management Commands + +| *Command* | *Description* | +|[[EnumerateExtension][Enumerate]] | Return all the extensions supported | +|[[DescribeExtension][Describe]] | Return description of an extension | +|[[DefineExtension][Definition]] | Return the definition of an extension | + +----------- + +---++Extension Job Management Commands + +| *Command* | *Description* | +|[[ExtensionList][List]] | List jobs generated from an extension | +|[[ExtensionInstances][Instances]] | List instances of an extension job | +|[[ExtensionValidate][Validate]] | Validate an extension job | +|[[ExtensionSubmit][Submit]] | Submit an extension job | +|[[ExtensionUpdate][Update]] | Update an extension job | +|[[ExtensionSubmitAndSchedule][SubmitAndSchedule]] | Submit and schedule an extension job | +|[[ExtensionSchedule][Schedule]] | Schedule an extension job | +|[[ExtensionSuspend][Suspend]] | Suspend an extension job | +|[[ExtensionResume][Resume]] | Resume an extension job | +|[[ExtensionDelete][Delete]] | Delete an extension job | http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/FeedInstanceListing.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/FeedInstanceListing.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/FeedInstanceListing.twiki new file mode 100644 index 0000000..aa60d49 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/FeedInstanceListing.twiki @@ -0,0 +1,11 @@ +---+++FeedInstanceListing + +Get falcon feed instance availability. + +Usage: +$FALCON_HOME/bin/falcon instance -type feed -name <<name>> -listing + +Optional Args : -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" +-colo <<colo>> + +<a href="../Restapi/FeedInstanceListing.html">Optional params described here.</a> http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/HelpAdmin.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/HelpAdmin.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/HelpAdmin.twiki new file mode 100644 index 0000000..69b1378 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/HelpAdmin.twiki @@ -0,0 +1,6 @@ +---+++Help + +[[CommonCLI][Common CLI Options]] + +Usage: +$FALCON_HOME/bin/falcon admin -help http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/KillInstance.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/KillInstance.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/KillInstance.twiki new file mode 100644 index 0000000..623921f --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/KillInstance.twiki @@ -0,0 +1,14 @@ +---+++Kill + +[[CommonCLI][Common CLI Options]] + +Kill sub-command is used to kill all the instances of the specified process whose nominal time is between the given start time and end time. + +Note: +1. The start time and end time needs to be specified in TZ format. +Example: 01 Jan 2012 01:00 => 2012-01-01T01:00Z + +2. Process name is compulsory parameter for each instance management command. + +Usage: +$FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -kill -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/LifeCycleInstance.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/LifeCycleInstance.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/LifeCycleInstance.twiki new file mode 100644 index 0000000..bbcda55 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/LifeCycleInstance.twiki @@ -0,0 +1,9 @@ +---+++LifeCycle + +[[CommonCLI][Common CLI Options]] + +Describes list of life cycles of a entity , for feed it can be replication/retention and for process it can be execution. +This can be used with instance management options. Default values are replication for feed and execution for process. + +Usage: +$FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -status -lifecycle <<lifecycletype>> -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/LineageMetadata.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/LineageMetadata.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/LineageMetadata.twiki new file mode 100644 index 0000000..e668e03 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/LineageMetadata.twiki @@ -0,0 +1,12 @@ +---+++Lineage + + +Returns the relationship between processes and feeds in a given pipeline in [[http://www.graphviz.org/content/dot-language/][dot]] format. +You can use the output and view a graphical representation of DAG using an online graphviz viewer like [[http://www.webgraphviz.com/][this]]. + +Usage: + +$FALCON_HOME/bin/falcon metadata -lineage -pipeline my-pipeline + +pipeline is a mandatory option. + http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/ListEntity.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/ListEntity.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/ListEntity.twiki new file mode 100644 index 0000000..0047c1b --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/ListEntity.twiki @@ -0,0 +1,17 @@ +---+++List + +[[CommonCLI][Common CLI Options]] + +Entities of a particular type can be listed with list sub-command. + +Usage: +$FALCON_HOME/bin/falcon entity -list + +Optional Args : -fields <<field1,field2>> +-type <<[cluster|datasource|feed|process],[cluster|datasource|feed|process]>> +-nameseq <<namesubsequence>> -tagkeys <<tagkeyword1,tagkeyword2>> +-filterBy <<field1:value1,field2:value2>> -tags <<tagkey=tagvalue,tagkey=tagvalue>> +-orderBy <<field>> -sortOrder <<sortOrder>> -offset 0 -numResults 10 + +<a href="../Restapi/EntityList.html">Optional params described here.</a> + http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/ListInstance.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/ListInstance.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/ListInstance.twiki new file mode 100644 index 0000000..1203629 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/ListInstance.twiki @@ -0,0 +1,20 @@ +---+++List + +[[CommonCLI][Common CLI Options]] + +List option via CLI can be used to get single or multiple instances. If the instance is not yet materialized but is within the process validity range, WAITING is returned as the state. Instance time is also returned. Log location gives the oozie workflow url +If the instance is in WAITING state, missing dependencies are listed + +Example : Suppose a process has 3 instance, one has succeeded,one is in running state and other one is waiting, the expected output is: + +{"status":"SUCCEEDED","message":"getStatus is successful","instances":[{"instance":"2012-05-07T05:02Z","status":"SUCCEEDED","logFile":"http://oozie-dashboard-url"},{"instance":"2012-05-07T05:07Z","status":"RUNNING","logFile":"http://oozie-dashboard-url"}, {"instance":"2010-01-02T11:05Z","status":"WAITING"}]} + +Usage: +$FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -list + +Optional Args : -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" +-colo <<colo>> -lifecycle <<lifecycles>> +-filterBy <<field1:value1,field2:value2>> -orderBy field -sortOrder <<sortOrder>> -offset 0 -numResults 10 +-allAttempts To get all the attempts for corresponding instances + +<a href="../Restapi/InstanceList.html">Optional params described here.</a> http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/ListMetadata.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/ListMetadata.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/ListMetadata.twiki new file mode 100644 index 0000000..8adea21 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/ListMetadata.twiki @@ -0,0 +1,13 @@ +---+++ List + +[[CommonCLI][Common CLI Options]] + +Lists of all dimensions of given type. If the user provides optional param cluster, only the dimensions related to the cluster are listed. +Usage: +$FALCON_HOME/bin/falcon metadata -list -type [cluster_entity|datasource_entity|feed_entity|process_entity|user|colo|tags|groups|pipelines] + +Optional Args : -cluster <<cluster name>> + +Example: +$FALCON_HOME/bin/falcon metadata -list -type process_entity -cluster primary-cluster +$FALCON_HOME/bin/falcon metadata -list -type tags http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/LogsInstance.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/LogsInstance.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/LogsInstance.twiki new file mode 100644 index 0000000..ac40ec0 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/LogsInstance.twiki @@ -0,0 +1,14 @@ +---+++Logs + +[[CommonCLI][Common CLI Options]] + +Get logs for instance actions + +Usage: +$FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -logs + +Optional Args : -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" -runid <<runid>> +-colo <<colo>> -lifecycle <<lifecycles>> +-filterBy <<field1:value1,field2:value2>> -orderBy field -sortOrder <<sortOrder>> -offset 0 -numResults 10 + +<a href="../Restapi/InstanceLogs.html">Optional params described here.</a> http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/Lookup.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/Lookup.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/Lookup.twiki new file mode 100644 index 0000000..a9d9c4e --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/Lookup.twiki @@ -0,0 +1,12 @@ +---+++Lookup + +[[CommonCLI][Common CLI Options]] + +Lookup option tells you which feed does a given path belong to. This can be useful in several scenarios e.g. generally you would want to have a single definition for common feeds like metadata with same location +otherwise it can result in a problem (different retention durations can result in surprises for one team) If you want to check if there are multiple definitions of same metadata then you can pick +an instance of that and run through the lookup command like below. + +Usage: +$FALCON_HOME/bin/falcon entity -type feed -lookup -path /data/projects/my-hourly/2014/10/10/23/ + +If you have multiple feeds with location as /data/projects/my-hourly/${YEAR}/${MONTH}/${DAY}/${HOUR} then this command will return all of them. http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/ParamsInstance.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/ParamsInstance.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/ParamsInstance.twiki new file mode 100644 index 0000000..9f217ba --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/ParamsInstance.twiki @@ -0,0 +1,8 @@ +---+++Params + +[[CommonCLI][Common CLI Options]] + +Displays the workflow params of a given instance. Where start time is considered as nominal time of that instance and end time won't be considered. + +Usage: +$FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -params -start "yyyy-MM-dd'T'HH:mm'Z'" http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/RelationMetadata.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/RelationMetadata.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/RelationMetadata.twiki new file mode 100644 index 0000000..e9bc970 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/RelationMetadata.twiki @@ -0,0 +1,10 @@ +---+++ Relations + +[[CommonCLI][Common CLI Options]] + +List all dimensions related to specified Dimension identified by dimension-type and dimension-name. +Usage: +$FALCON_HOME/bin/falcon metadata -relations -type [cluster_entity|feed_entity|process_entity|user|colo|tags|groups|pipelines] -name <<Dimension Name>> + +Example: +$FALCON_HOME/bin/falcon metadata -relations -type process_entity -name sample-process \ No newline at end of file http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/RerunInstance.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/RerunInstance.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/RerunInstance.twiki new file mode 100644 index 0000000..aac844c --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/RerunInstance.twiki @@ -0,0 +1,10 @@ +---+++Rerun + +[[CommonCLI][Common CLI Options]] + +Rerun option is used to rerun instances of a given process. On issuing a rerun, by default the execution resumes from the last failed node in the workflow. This option is valid only for process instances in terminal state, i.e. SUCCEEDED, KILLED or FAILED. +If one wants to forcefully rerun the entire workflow, -force should be passed along with -rerun +Additionally, you can also specify properties to override via a properties file. + +Usage: +$FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -rerun -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" [-force] [-file <<properties file>>] http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/ResumeEntity.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/ResumeEntity.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/ResumeEntity.twiki new file mode 100644 index 0000000..39be411 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/ResumeEntity.twiki @@ -0,0 +1,8 @@ +---+++Resume + +[[CommonCLI][Common CLI Options]] + +Puts a suspended process/feed back to active, which in turn resumes applicable oozie bundle. + +Usage: + $FALCON_HOME/bin/falcon entity -type [feed|process] -name <<name>> -resume http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/ResumeInstance.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/ResumeInstance.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/ResumeInstance.twiki new file mode 100644 index 0000000..3790f47 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/ResumeInstance.twiki @@ -0,0 +1,8 @@ +---+++Resume + +[[CommonCLI][Common CLI Options]] + +Resume option is used to resume any instance that is in suspended state. + +Usage: +$FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -resume -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/RunningInstance.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/RunningInstance.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/RunningInstance.twiki new file mode 100644 index 0000000..f269358 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/RunningInstance.twiki @@ -0,0 +1,13 @@ +---+++Running + +[[CommonCLI][Common CLI Options]] + +Running option provides all the running instances of the mentioned process. + +Usage: +$FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -running + +Optional Args : -colo <<colo>> -lifecycle <<lifecycles>> +-filterBy <<field1:value1,field2:value2>> -orderBy <<field>> -sortOrder <<sortOrder>> -offset 0 -numResults 10 + +<a href="../Restapi/InstanceRunning.html">Optional params described here.</a> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/SLAAlert.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/SLAAlert.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/SLAAlert.twiki new file mode 100644 index 0000000..e5270fa --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/SLAAlert.twiki @@ -0,0 +1,49 @@ +---+++SLAAlert + +[[CommonCLI][Common CLI Options]] + +<verbatim> +Since: 0.8 +</verbatim> + +This command lists all the feed instances which have missed sla and are still not available. If a feed instance missed +sla but is now available, then it will not be reported in results. The purpose of this API is alerting and hence it + doesn't return feed instances which missed SLA but are available as they don't require any action. + +* Currently sla monitoring is supported only for feeds. + +* Option end is optional and will default to current time if missing. + +* Option name is optional, if provided only instances of that feed will be considered. + +Usage: + +*Example 1* + +*$FALCON_HOME/bin/falcon entity -type feed -start 2014-09-05T00:00Z -slaAlert -end 2016-05-03T00:00Z -colo local* + +name: out, type: FEED, cluster: local, instanceTime: 2015-09-26T11:59Z, tags: Missed SLA High +name: out, type: FEED, cluster: local, instanceTime: 2015-09-26T12:00Z, tags: Missed SLA High +name: out, type: FEED, cluster: local, instanceTime: 2015-09-26T12:01Z, tags: Missed SLA High +name: out, type: FEED, cluster: local, instanceTime: 2015-09-26T12:02Z, tags: Missed SLA High +name: out, type: FEED, cluster: local, instanceTime: 2015-09-26T12:03Z, tags: Missed SLA High +name: out, type: FEED, cluster: local, instanceTime: 2015-09-26T12:04Z, tags: Missed SLA High +name: out, type: FEED, cluster: local, instanceTime: 2015-09-26T12:05Z, tags: Missed SLA High +name: out, type: FEED, cluster: local, instanceTime: 2015-09-26T12:06Z, tags: Missed SLA High +name: out, type: FEED, cluster: local, instanceTime: 2015-09-26T12:07Z, tags: Missed SLA High +name: out, type: FEED, cluster: local, instanceTime: 2015-09-26T12:08Z, tags: Missed SLA Low + + +Response: default/Success! + +Request Id: default/216978070@qtp-830047511-4 - f5a6c129-ab42-4feb-a2bf-c3baed356248 + +*Example 2* + +*$FALCON_HOME/bin/falcon entity -type feed -start 2014-09-05T00:00Z -slaAlert -end 2016-05-03T00:00Z -colo local -name in* + +name: in, type: FEED, cluster: local, instanceTime: 2015-09-26T06:00Z, tags: Missed SLA High + +Response: default/Success! + +Request Id: default/1580107885@qtp-830047511-7 - f16cbc51-5070-4551-ad25-28f75e5e4cf2 http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/SafemodeAdmin.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/SafemodeAdmin.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/SafemodeAdmin.twiki new file mode 100644 index 0000000..5eb26c0 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/SafemodeAdmin.twiki @@ -0,0 +1,10 @@ +---+++SetSafeMode + +[[CommonCLI][Common CLI Options]] + +When -setsafemode is true, Falcon starts in safemode. +When -setsafemode is false, falcon starts in normal mode. + +Usage: +$FALCON_HOME/bin/falcon admin -setsafemode true/false + http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/Schedule.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/Schedule.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/Schedule.twiki new file mode 100644 index 0000000..c4422e7 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/Schedule.twiki @@ -0,0 +1,22 @@ +---+++Schedule + +[[CommonCLI][Common CLI Options]] + +Once submitted, an entity can be scheduled using schedule option. Process and feed can only be scheduled. + +Usage: +$FALCON_HOME/bin/falcon entity -type [process|feed] -name <<name>> -schedule + +Optional Args : + +-skipDryRun When this argument is specified, Falcon skips oozie dryrun. + +-doAs <username> + +-properties <<key1:val1,...,keyN:valN>>. Specifying 'falcon.scheduler:native' as a property will schedule the entity on the the native scheduler of Falcon. Else, it will default to the engine specified in startup.properties. For details on Native scheduler, refer to [[FalconNativeScheduler][Falcon Native Scheduler]] + +Examples: + + $FALCON_HOME/bin/falcon entity -type process -name sampleProcess -schedule + + $FALCON_HOME/bin/falcon entity -type process -name sampleProcess -schedule -properties falcon.scheduler:native http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/SearchInstance.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/SearchInstance.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/SearchInstance.twiki new file mode 100644 index 0000000..35cac74 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/SearchInstance.twiki @@ -0,0 +1,17 @@ +---+++Search + +[[CommonCLI][Common CLI Options]] + +Search option via CLI can be used to get instances that meet search criteria, e.g. nominal time, instance status, entity name subsequence, entity tag, etc. +Instance name and status are listed in the output. More instance properties will be added to the output in the future (e.g. log url, running time). +Example usage include: search of instances of all entities with the same tag; search of all current running instances. + +Usage: +$FALCON_HOME/bin/falcon instance -search + +Optional Args : -type <<[feed|process|datasource],[feed|process|datasource]>> +-nameseq <<namesubsequence>> -tagkeys <<tagkeyword1,tagkeyword2>> +-instanceStatus <<status>> -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" +-orderBy <<field>> -offset 0 -numResults 10 + +<a href="../restapi/InstanceSearch.html">Optional params and examples described here.</a> http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/StatusAdmin.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/StatusAdmin.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/StatusAdmin.twiki new file mode 100644 index 0000000..dadb8e5 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/StatusAdmin.twiki @@ -0,0 +1,8 @@ +---+++Status + +[[CommonCLI][Common CLI Options]] + +Status returns the current state of Falcon (running or stopped). +Usage: +$FALCON_HOME/bin/falcon admin -status + http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/StatusEntity.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/StatusEntity.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/StatusEntity.twiki new file mode 100644 index 0000000..56d16f0 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/StatusEntity.twiki @@ -0,0 +1,8 @@ +---+++Status + +[[CommonCLI][Common CLI Options]] + +Status returns the current status of the entity. + +Usage: +$FALCON_HOME/bin/falcon entity -type [cluster|datasource|feed|process] -name <<name>> -status \ No newline at end of file http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/StatusInstance.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/StatusInstance.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/StatusInstance.twiki new file mode 100644 index 0000000..047d334 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/StatusInstance.twiki @@ -0,0 +1,21 @@ +---+++Status + +[[CommonCLI][Common CLI Options]] + +Status option via CLI can be used to get the status of a single or multiple instances. If the instance is not yet materialized but is within the process validity range, WAITING is returned as the state. Along with the status of the instance time is also returned. Log location gives the oozie workflow url +If the instance is in WAITING state, missing dependencies are listed. +The job urls are populated for all actions of user workflow and non-succeeded actions of the main-workflow. The user then need not go to the underlying scheduler to get the job urls when needed to debug an issue in the job. + +Example : Suppose a process has 3 instance, one has succeeded,one is in running state and other one is waiting, the expected output is: + +{"status":"SUCCEEDED","message":"getStatus is successful","instances":[{"instance":"2012-05-07T05:02Z","status":"SUCCEEDED","logFile":"http://oozie-dashboard-url"},{"instance":"2012-05-07T05:07Z","status":"RUNNING","logFile":"http://oozie-dashboard-url"}, {"instance":"2010-01-02T11:05Z","status":"WAITING"}] + +Usage: +$FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -status + +Optional Args : -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" -colo <<colo>> +-filterBy <<field1:value1,field2:value2>> -lifecycle <<lifecycles>> +-orderBy field -sortOrder <<sortOrder>> -offset 0 -numResults 10 +-allAttempts To get all the attempts for corresponding instances + +<a href="../Restapi/InstanceStatus.html"> Optional params described here.</a> http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/Submit.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/Submit.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/Submit.twiki new file mode 100644 index 0000000..f2f7a49 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/Submit.twiki @@ -0,0 +1,13 @@ +---+++Submit + +[[CommonCLI][Common CLI Options]] + +Submit option is used to set up entity definition. + +Usage: +$FALCON_HOME/bin/falcon entity -submit -type [cluster|datasource|feed|process] -file <entity-definition.xml> + +Example: +$FALCON_HOME/bin/falcon entity -submit -type cluster -file /cluster/definition.xml + +Note: The url option in the above and all subsequent commands is optional. If not mentioned it will be picked from client.properties file. If the option is not provided and also not set in client.properties, Falcon CLI will fail. http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/SummaryEntity.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/SummaryEntity.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/SummaryEntity.twiki new file mode 100644 index 0000000..800f9fc --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/SummaryEntity.twiki @@ -0,0 +1,14 @@ +---+++Summary + +[[CommonCLI][Common CLI Options]] + +Summary of entities of a particular type and a cluster will be listed. Entity summary has N most recent instances of entity. + +Usage: +$FALCON_HOME/bin/falcon entity -type [feed|process] -summary + +Optional Args : -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" -fields <<field1,field2>> +-filterBy <<field1:value1,field2:value2>> -tags <<tagkey=tagvalue,tagkey=tagvalue>> +-orderBy <<field>> -sortOrder <<sortOrder>> -offset 0 -numResults 10 -numInstances 7 + +<a href="../Restapi/EntitySummary.html">Optional params described here.</a> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/SummaryInstance.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/SummaryInstance.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/SummaryInstance.twiki new file mode 100644 index 0000000..f7ca0b4 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/SummaryInstance.twiki @@ -0,0 +1,20 @@ +---+++Summary + +[[CommonCLI][Common CLI Options]] + +Summary option via CLI can be used to get the consolidated status of the instances between the specified time period. +Each status along with the corresponding instance count are listed for each of the applicable colos. +The unscheduled instances between the specified time period are included as UNSCHEDULED in the output to provide more clarity. + +Example : Suppose a process has 3 instance, one has succeeded,one is in running state and other one is waiting, the expected output is: + +{"status":"SUCCEEDED","message":"getSummary is successful", instancesSummary:[{"cluster": <<name>> "map":[{"SUCCEEDED":"1"}, {"WAITING":"1"}, {"RUNNING":"1"}]}]} + +Usage: +$FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -summary + +Optional Args : -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" -colo <<colo>> +-filterBy <<field1:value1,field2:value2>> -lifecycle <<lifecycles>> +-orderBy field -sortOrder <<sortOrder>> + +<a href="../Restapi/InstanceSummary.html">Optional params described here.</a> http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/SuspendEntity.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/SuspendEntity.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/SuspendEntity.twiki new file mode 100644 index 0000000..7618e9c --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/SuspendEntity.twiki @@ -0,0 +1,8 @@ +---+++Suspend + +[[CommonCLI][Common CLI Options]] + +Suspend on an entity results in suspension of the oozie bundle that was scheduled earlier through the schedule function. No further instances are executed on a suspended entity. Only schedule-able entities(process/feed) can be suspended. + +Usage: +$FALCON_HOME/bin/falcon entity -type [feed|process] -name <<name>> -suspend \ No newline at end of file http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/SuspendInstance.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/SuspendInstance.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/SuspendInstance.twiki new file mode 100644 index 0000000..221cf5c --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/SuspendInstance.twiki @@ -0,0 +1,8 @@ +---+++Suspend + +[[CommonCLI][Common CLI Options]] + +Suspend is used to suspend a instance or instances for the given process. This option pauses the parent workflow at the state, which it was in at the time of execution of this command. + +Usage: +$FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -suspend -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/Touch.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/Touch.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/Touch.twiki new file mode 100644 index 0000000..afbd848 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/Touch.twiki @@ -0,0 +1,10 @@ +---+++Touch + +[[CommonCLI][Common CLI Options]] + +Force Update operation allows an already submitted/scheduled entity to be updated. + +Usage: +$FALCON_HOME/bin/falcon entity -type [feed|process] -name <<name>> -touch + +Optional Arg : -skipDryRun. When this argument is specified, Falcon skips oozie dryrun. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/TriageInstance.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/TriageInstance.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/TriageInstance.twiki new file mode 100644 index 0000000..c2c32cd --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/TriageInstance.twiki @@ -0,0 +1,9 @@ +---+++Triage + +[[CommonCLI][Common CLI Options]] + +Given a feed/process instance this command traces it's ancestors to find what all ancestors have failed. It's useful if +lot of instances are failing in a pipeline as it then finds out the root cause of the pipeline being stuck. + +Usage: +$FALCON_HOME/bin/falcon instance -triage -type <<feed/process>> -name <<name>> -start "yyyy-MM-dd'T'HH:mm'Z'" http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/UpdateEntity.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/UpdateEntity.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/UpdateEntity.twiki new file mode 100644 index 0000000..146a60f --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/UpdateEntity.twiki @@ -0,0 +1,16 @@ +---+++Update + +[[CommonCLI][Common CLI Options]] + +Update operation allows an already submitted/scheduled entity to be updated and put it into the archive.Archive path is defined in startup.properties in variable "config.store.uri". Datasource updates are currently not allowed. + +Usage: +$FALCON_HOME/bin/falcon entity -type [cluster|feed|process] -name <<name>> -update -file <<path_to_file>> + +Optional Arg : -skipDryRun. When this argument is specified, Falcon skips oozie dryrun. + +Example: +$FALCON_HOME/bin/falcon entity -type process -name hourly-reports-generator -update -file /process/definition.xml + +Note: When a cluster entity is updated, the dependent feed and process bundle+coordinators are updated in the +workflow engine. Hence, only a falcon superuser who has ability to impersonate other users can update a cluster entity. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/VersionAdmin.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/VersionAdmin.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/VersionAdmin.twiki new file mode 100644 index 0000000..453f6a1 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/VersionAdmin.twiki @@ -0,0 +1,7 @@ +---+++Version + +[[CommonCLI][Common CLI Options]] + +Version returns the current version of Falcon installed. +Usage: +$FALCON_HOME/bin/falcon admin -version http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/VertexEdgesMetadata.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/VertexEdgesMetadata.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/VertexEdgesMetadata.twiki new file mode 100644 index 0000000..e9182fc --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/VertexEdgesMetadata.twiki @@ -0,0 +1,12 @@ +---+++ Vertex Edges + +[[CommonCLI][Common CLI Options]] + +Get the adjacent vertices or edges of the vertex with the specified direction. + +Usage: +$FALCON_HOME/bin/falcon metadata -edges -id <<vertex-id>> -direction <<direction>> + +Example: +$FALCON_HOME/bin/falcon metadata -edges -id 4 -direction both +$FALCON_HOME/bin/falcon metadata -edges -id 4 -direction inE http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/VertexMetadata.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/VertexMetadata.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/VertexMetadata.twiki new file mode 100644 index 0000000..b2c62e8 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/VertexMetadata.twiki @@ -0,0 +1,11 @@ +---+++ Vertex + +[[CommonCLI][Common CLI Options]] + +Get the vertex with the specified id. + +Usage: +$FALCON_HOME/bin/falcon metadata -vertex -id <<id>> + +Example: +$FALCON_HOME/bin/falcon metadata -vertex -id 4 http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/falconcli/VerticesMetadata.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/falconcli/VerticesMetadata.twiki b/trunk/releases/0.11/src/site/twiki/falconcli/VerticesMetadata.twiki new file mode 100644 index 0000000..1b32ad5 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/falconcli/VerticesMetadata.twiki @@ -0,0 +1,11 @@ +---+++ Vertices + +[[CommonCLI][Common CLI Options]] + +Get all vertices for a key index given the specified value. + +Usage: +$FALCON_HOME/bin/falcon metadata -vertices -key <<key>> -value <<value>> + +Example: +$FALCON_HOME/bin/falcon metadata -vertices -key type -value feed-instance http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/index.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/index.twiki b/trunk/releases/0.11/src/site/twiki/index.twiki new file mode 100644 index 0000000..a1ee0a3 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/index.twiki @@ -0,0 +1,43 @@ +---+ Falcon - Feed management and data processing platform + +Falcon is a feed processing and feed management system aimed at making it +easier for end consumers to onboard their feed processing and feed +management on hadoop clusters. + +---++ Why? + + * Establishes relationship between various data and processing elements on a Hadoop environment + + * Feed management services such as feed retention, replications across clusters, archival etc. + + * Easy to onboard new workflows/pipelines, with support for late data handling, retry policies + + * Integration with metastore/catalog such as Hive/HCatalog + + * Provide notification to end customer based on availability of feed groups + (logical group of related feeds, which are likely to be used together) + + * Enables use cases for local processing in colo and global aggregations + + * Captures Lineage information for feeds and processes + +---+ Getting Started + +Start with these simple steps to install an falcon instance [[InstallationSteps][Simple setup]]. Also refer +to Falcon architecture and documentation in [[FalconDocumentation][Documentation]]. [[OnBoarding][On boarding]] +describes steps to on-board a pipeline to Falcon. It also gives a sample pipeline for reference. +[[EntitySpecification][Entity Specification]] gives complete details of all Falcon entities. + +[[falconcli/FalconCLI][Falcon CLI]] implements [[restapi/ResourceList][Falcon's RESTful API]] and +describes various options for the command line utility provided by Falcon. + +Falcon provides OOTB [[HiveIntegration][lifecycle management for Tables in Hive (HCatalog)]] +such as table replication for BCP and table eviction. Falcon also enforces +[[Security][Security]] on protected resources and enables SSL. + +#LicenseInfo +---+ Licensing Information + +Falcon is distributed under [[http://www.apache.org/licenses/LICENSE-2.0][Apache License 2.0]]. + + http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/restapi/AdjacentVertices.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/restapi/AdjacentVertices.twiki b/trunk/releases/0.11/src/site/twiki/restapi/AdjacentVertices.twiki new file mode 100644 index 0000000..1e60866 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/restapi/AdjacentVertices.twiki @@ -0,0 +1,91 @@ +---++ GET api/metadata/lineage/vertices/:id/:direction + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Get a list of adjacent vertices or edges with a direction. + +---++ Parameters + * :id is the id of the vertex. + * doAs <optional query param> allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. + * :direction is the direction associated with the edges. + + To get the adjacent out vertices of vertex pass direction as out, in to get adjacent in vertices + and both to get both in and out adjacent vertices. Similarly to get the out edges of vertex + pass outE, inE to get in edges and bothE to get the both in and out edges of vertex. + + * out : get the adjacent out vertices of vertex + * in : get the adjacent in vertices of vertex + * both : get the both adjacent in and out vertices of vertex + * outCount : get the number of out vertices of vertex + * inCount : get the number of in vertices of vertex + * bothCount : get the number of adjacent in and out vertices of vertex + * outIds : get the identifiers of out vertices of vertex + * inIds : get the identifiers of in vertices of vertex + * bothIds : get the identifiers of adjacent in and out vertices of vertex + +---++ Results +Adjacent vertices of the vertex for the specified direction. + +---++ Examples +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/metadata/lineage/vertices/4/out +</verbatim> +---+++ Result +<verbatim> +{ + "results": [ + { + "timestamp":"2014-04-21T20:55Z", + "name":"sampleFeed", + "type":"feed-instance", + "_id":8, + "_type":"vertex" + } + ], + "totalSize":1} +} +</verbatim> + +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/metadata/lineage/vertices/4/bothE +</verbatim> +---+++ Result +<verbatim> +{ + "results":[ + { + "_id":"Q5V-4-5g", + "_type":"edge", + "_outV":4, + "_inV":8, + "_label":"output" + } + ], + "totalSize":1 +} +</verbatim> + +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/metadata/lineage/vertices/4/bothE?doAs=joe +</verbatim> +---+++ Result +<verbatim> +{ + "results":[ + { + "_id":"Q5V-4-5g", + "_type":"edge", + "_outV":4, + "_inV":8, + "_label":"output" + } + ], + "totalSize":1 +} +</verbatim> http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/restapi/AdminConfig.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/restapi/AdminConfig.twiki b/trunk/releases/0.11/src/site/twiki/restapi/AdminConfig.twiki new file mode 100644 index 0000000..675b19e --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/restapi/AdminConfig.twiki @@ -0,0 +1,35 @@ +---++ GET /api/admin/config/:config-type + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Get configuration information of the falcon server. + +---++ Parameters + * :config-type can be build, deploy, startup or runtime + +---++ Results +Configuration information of the server. + +---++ Examples +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/admin/config/deploy +</verbatim> +---+++ Result +<verbatim> +{ + "properties": [ + { + "value": "embedded", + "key": "deploy.mode" + }, + { + "value": "all", + "key": "domain" + } + ] +} +</verbatim> http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/restapi/AdminSafemode.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/restapi/AdminSafemode.twiki b/trunk/releases/0.11/src/site/twiki/restapi/AdminSafemode.twiki new file mode 100644 index 0000000..d7c9631 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/restapi/AdminSafemode.twiki @@ -0,0 +1,25 @@ +---++ GET /api/admin/setSafeMode/:mode + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Set falcon server into/out of safemode. + +---++ Parameters + * :mode true/false. + * doAs <optional query param> allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. + +---++ Results +The value of FalconSever safemode. + +---++ Examples +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/admin/setSafeMode/true +</verbatim> +---+++ Result +<verbatim> +true +</verbatim> http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/restapi/AdminStack.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/restapi/AdminStack.twiki b/trunk/releases/0.11/src/site/twiki/restapi/AdminStack.twiki new file mode 100644 index 0000000..08903a2 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/restapi/AdminStack.twiki @@ -0,0 +1,40 @@ +---++ GET /api/admin/stack + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Get stack trace of the falcon server. + +---++ Parameters + * doAs <optional query param> allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. + +---++ Results +Stack trace of the server. + +---++ Examples +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/admin/stack?doAs=joe +</verbatim> +---+++ Result +<verbatim> +Reference Handler +State: WAITING +java.lang.Object.wait(Native Method) +java.lang.Object.wait(Object.java:485) +java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)Finalizer + +... + +State: TIMED_WAITING +sun.misc.Unsafe.park(Native Method) +java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) +java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424) +java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) +java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874) +java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:955) +java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917) +java.lang.Thread.run(Thread.java:695) +</verbatim> http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/restapi/AdminVersion.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/restapi/AdminVersion.twiki b/trunk/releases/0.11/src/site/twiki/restapi/AdminVersion.twiki new file mode 100644 index 0000000..d7615a0 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/restapi/AdminVersion.twiki @@ -0,0 +1,43 @@ +---++ GET /api/admin/version + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Get version of the falcon server. + +---++ Parameters + * doAs <optional query param> allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. + +---++ Results +Version of the server. + +---++ Examples +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/admin/version?doAs=joe +</verbatim> +---+++ Result +<verbatim> +{ + "properties":[ + { + "key":"Version", + "value":"0.4-incubating-SNAPSHOT-rb47788d1112fcf949c22a3860934167237b395b0" + }, + { + "key":"Mode", + "value":"embedded" + }, + { + "key":"authentication", + "value":"simple" + }, + { + "key":"safemode", + "value":"false" + } + ] +} +</verbatim> http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/restapi/AllEdges.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/restapi/AllEdges.twiki b/trunk/releases/0.11/src/site/twiki/restapi/AllEdges.twiki new file mode 100644 index 0000000..303ac50 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/restapi/AllEdges.twiki @@ -0,0 +1,42 @@ +---++ GET pi/metadata/lineage//edges/all + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Get all edges. + +---++ Parameters + * doAs <optional query param> allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. + +---++ Results +All edges in lineage graph. + +---++ Examples +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/metadata/lineage/edges/all?doAs=joe +</verbatim> +---+++ Result +<verbatim> +{ + "results": [ + { + "_id":"Q5V-4-5g", + "_type":"edge", + "_outV":4, + "_inV":8, + "_label":"output" + }, + { + "_id":"Q6t-c-5g", + "_type":"edge", + "_outV":12, + "_inV":16, + "_label":"output" + } + ], + "totalSize": 2 +} +</verbatim> http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/restapi/AllVertices.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/restapi/AllVertices.twiki b/trunk/releases/0.11/src/site/twiki/restapi/AllVertices.twiki new file mode 100644 index 0000000..d2beb48 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/restapi/AllVertices.twiki @@ -0,0 +1,43 @@ +---++ GET api/metadata/lineage/vertices/all + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Get all vertices. + +---++ Parameters + * doAs <optional query param> allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. + +---++ Results +All vertices in lineage graph. + +---++ Examples +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/metadata/lineage/vertices/all?doAs=joe +</verbatim> +---+++ Result +<verbatim> +{ + "results": [ + { + "timestamp":"2014-04-21T20:55Z", + "name":"sampleIngestProcess\/2014-03-01T10:00Z", + "type":"process-instance", + "version":"2.0.0", + "_id":4, + "_type":"vertex" + }, + { + "timestamp":"2014-04-21T20:55Z", + "name":"rawEmailFeed\/2014-03-01T10:00Z", + "type":"feed-instance", + "_id":8, + "_type":"vertex" + } + ], + "totalSize": 2 +} +</verbatim> http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/restapi/Edge.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/restapi/Edge.twiki b/trunk/releases/0.11/src/site/twiki/restapi/Edge.twiki new file mode 100644 index 0000000..7c4dbe5 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/restapi/Edge.twiki @@ -0,0 +1,34 @@ +---++ GET api/metadata/lineage/edges/:id + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Gets the edge with specified id. + +---++ Parameters + * :id is the unique id of the edge. + * doAs <optional query param> allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. + +---++ Results +Edge with the specified id. + +---++ Examples +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/metadata/lineage/edges/Q6t-c-5g?doAs=joe +</verbatim> +---+++ Result +<verbatim> +{ + "results": + { + "_id":"Q6t-c-5g", + "_type":"edge", + "_outV":12, + "_inV":16, + "_label":"output" + } +} +</verbatim> http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/restapi/EntityDefinition.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/restapi/EntityDefinition.twiki b/trunk/releases/0.11/src/site/twiki/restapi/EntityDefinition.twiki new file mode 100644 index 0000000..5e1165b --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/restapi/EntityDefinition.twiki @@ -0,0 +1,53 @@ +---++ GET /api/entities/definition/:entity-type/:entity-name + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Get definition of the entity. + +---++ Parameters + * :entity-type can be cluster, feed or process. + * :entity-name is name of the entity. + * doAs <optional query param> allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. + +---++ Results +Definition of the entity. + +---++ Examples +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/entities/definition/process/SampleProcess?doAs=joe +</verbatim> +---+++ Result +<verbatim> +<?xml version="1.0" encoding="UTF-8" standalone="yes"?> +<process name="SampleProcess" xmlns="uri:falcon:process:0.1"> + <clusters> + <cluster name="primary-cluster"> + <validity start="2012-04-03T06:00Z" end="2022-12-30T00:00Z"/> + </cluster> + </clusters> + <parallel>1</parallel> + <order>FIFO</order> + <frequency>hours(1)</frequency> + <timezone>UTC</timezone> + <inputs> + <input name="input" feed="SampleInput" start="yesterday(0,0)" end="today(-1,0)"/> + </inputs> + <outputs> + <output name="output" feed="SampleOutput" instance="yesterday(0,0)"/> + </outputs> + <properties> + <property name="queueName" value="default"/> + <property name="ssh.host" value="localhost"/> + <property name="fileTimestamp" value="${coord:formatTime(coord:nominalTime(), 'yyyy-MM-dd')}"/> + </properties> + <workflow engine="oozie" path="/examples/apps/aggregator"/> + <retry policy="exp-backoff" delay="minutes(5)" attempts="3"/> + <late-process policy="exp-backoff" delay="hours(1)"> + <late-input input="input" workflow-path="/projects/bootcamp/workflow/lateinput"/> + </late-process> +</process> +</verbatim> http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/restapi/EntityDelete.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/restapi/EntityDelete.twiki b/trunk/releases/0.11/src/site/twiki/restapi/EntityDelete.twiki new file mode 100644 index 0000000..a488943 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/restapi/EntityDelete.twiki @@ -0,0 +1,31 @@ +---++ DELETE /api/entities/delete/:entity-type/:entity-name + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Delete the specified entity. + +---++ Parameters + * :entity-type can be feed or process. + * :entity-name is name of the feed or process. + * doAs <optional query param> allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. + +---++ Results +Results of the delete operation. + +---++ Examples +---+++ Rest Call +<verbatim> +DELETE http://localhost:15000/api/entities/delete/cluster/SampleProcess?doAs=joe +</verbatim> +---+++ Result +<verbatim> +{ + "requestId": "falcon\/17ff6ca6-1c8a-459f-9ba8-8fec480e384a\n", + "message": "falcon\/SampleProcess(cluster) removed successfully\n", + "status": "SUCCEEDED" +} +</verbatim> + http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/restapi/EntityDependencies.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/restapi/EntityDependencies.twiki b/trunk/releases/0.11/src/site/twiki/restapi/EntityDependencies.twiki new file mode 100644 index 0000000..864b084 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/restapi/EntityDependencies.twiki @@ -0,0 +1,43 @@ +---++ GET /api/entities/dependencies/:entity-type/:entity-name + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +Get dependencies of the entity. + +---++ Parameters + * :entity-type can be cluster, feed or process. + * :entity-name is name of the entity. + * doAs <optional query param> allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. + +---++ Results +Dependenciess of the entity. + +---++ Examples +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/entities/dependencies/process/SampleProcess?doAs=joe +</verbatim> +---+++ Result +<verbatim> +{ + "entity": [ + { + "name": "SampleInput", + "type": "feed", + "tag": [Input] + }, + { + "name": "SampleOutput", + "type": "feed" + "tag": [Output] + }, + { + "name": "primary-cluster", + "type": "cluster" + } + ] +} +</verbatim> http://git-wip-us.apache.org/repos/asf/falcon/blob/91c68bea/trunk/releases/0.11/src/site/twiki/restapi/EntityLineage.twiki ---------------------------------------------------------------------- diff --git a/trunk/releases/0.11/src/site/twiki/restapi/EntityLineage.twiki b/trunk/releases/0.11/src/site/twiki/restapi/EntityLineage.twiki new file mode 100644 index 0000000..f2258f2 --- /dev/null +++ b/trunk/releases/0.11/src/site/twiki/restapi/EntityLineage.twiki @@ -0,0 +1,40 @@ +---++ GET api/metadata/lineage/entities?pipeline=:pipeline + * <a href="#Description">Description</a> + * <a href="#Parameters">Parameters</a> + * <a href="#Results">Results</a> + * <a href="#Examples">Examples</a> + +---++ Description +It returns the graph depicting the relationship between the various processes and feeds in a given pipeline. + +---++ Parameters + * :pipeline is the name of the pipeline + * doAs <optional query param> allows the current user to impersonate the user passed in doAs when interacting with the Falcon system. + +---++ Results +It returns a json graph + +---++ Examples +---+++ Rest Call +<verbatim> +GET http://localhost:15000/api/metadata/lineage/entities?pipeline=my-pipeline&doAs=joe +</verbatim> +---+++ Result +<verbatim> +{ + "vertices": ["my-minutely-process", "my-hourly-process"], + "edges": + [ + { + "from" : "my-minutely-process", + "to" : "my-hourly-process", + "label" : "my-minutely-feed" + }, + { + "from" : "my-hourly-process", + "to" : "my-minutely-process", + "label" : "my-hourly-feedback" + } + ] +} +</verbatim>
