Repository: samza Updated Branches: refs/heads/master 260d1ff96 -> 3902e2047
SAMZA-976 - Samza REST Documentation Project: http://git-wip-us.apache.org/repos/asf/samza/repo Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/3902e204 Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/3902e204 Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/3902e204 Branch: refs/heads/master Commit: 3902e20474fae3bcdd6a28e2b03eaaf303d0d85d Parents: 260d1ff Author: Jacob Maes <[email protected]> Authored: Mon Aug 22 23:50:10 2016 -0700 Committer: Navina Ramesh <[email protected]> Committed: Tue Aug 23 00:09:29 2016 -0700 ---------------------------------------------------------------------- .gitignore | 1 + bin/generate-javadocs.sh | 16 +- .../learn/documentation/rest/JobsResource.png | Bin 0 -> 191199 bytes docs/learn/documentation/versioned/index.html | 9 + .../documentation/versioned/rest/monitors.md | 35 ++ .../documentation/versioned/rest/overview.md | 94 ++++++ .../versioned/rest/resource-directory.md | 27 ++ .../documentation/versioned/rest/resources.md | 61 ++++ .../versioned/rest/resources/jobs.md | 327 +++++++++++++++++++ docs/learn/tutorials/versioned/index.md | 2 + .../versioned/samza-rest-getting-started.md | 102 ++++++ 11 files changed, 670 insertions(+), 4 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/samza/blob/3902e204/.gitignore ---------------------------------------------------------------------- diff --git a/.gitignore b/.gitignore index 3e97416..a8de1b5 100644 --- a/.gitignore +++ b/.gitignore @@ -23,6 +23,7 @@ build **/bin samza-test/state docs/learn/documentation/*/api/javadocs +docs/learn/documentation/*/rest/javadocs .DS_Store out/ *.patch http://git-wip-us.apache.org/repos/asf/samza/blob/3902e204/bin/generate-javadocs.sh ---------------------------------------------------------------------- diff --git a/bin/generate-javadocs.sh b/bin/generate-javadocs.sh index da9f32d..4288059 100755 --- a/bin/generate-javadocs.sh +++ b/bin/generate-javadocs.sh @@ -18,11 +18,19 @@ DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" BASE_DIR=$DIR/.. -JAVADOC_DIR=$BASE_DIR/docs/learn/documentation/versioned/api/javadocs +API_JAVADOC_DIR=$BASE_DIR/docs/learn/documentation/versioned/api/javadocs +REST_JAVADOC_DIR=$BASE_DIR/docs/learn/documentation/versioned/rest/javadocs cd $BASE_DIR ./gradlew javadoc -rm -rf $JAVADOC_DIR -mkdir -p $JAVADOC_DIR -cp -r $BASE_DIR/samza-api/build/docs/javadoc/* $JAVADOC_DIR + +# API Docs +rm -rf $API_JAVADOC_DIR +mkdir -p $API_JAVADOC_DIR +cp -r $BASE_DIR/samza-api/build/docs/javadoc/* $API_JAVADOC_DIR + +# REST Docs +rm -rf $REST_JAVADOC_DIR +mkdir -p $REST_JAVADOC_DIR +cp -r $BASE_DIR/samza-rest/build/docs/javadoc/* $REST_JAVADOC_DIR cd - http://git-wip-us.apache.org/repos/asf/samza/blob/3902e204/docs/img/versioned/learn/documentation/rest/JobsResource.png ---------------------------------------------------------------------- diff --git a/docs/img/versioned/learn/documentation/rest/JobsResource.png b/docs/img/versioned/learn/documentation/rest/JobsResource.png new file mode 100644 index 0000000..a208c3d Binary files /dev/null and b/docs/img/versioned/learn/documentation/rest/JobsResource.png differ http://git-wip-us.apache.org/repos/asf/samza/blob/3902e204/docs/learn/documentation/versioned/index.html ---------------------------------------------------------------------- diff --git a/docs/learn/documentation/versioned/index.html b/docs/learn/documentation/versioned/index.html index 84e15a2..a997445 100644 --- a/docs/learn/documentation/versioned/index.html +++ b/docs/learn/documentation/versioned/index.html @@ -94,4 +94,13 @@ title: Documentation <li><a href="operations/security.html">Security</a></li> <li><a href="operations/kafka.html">Kafka</a></li> </ul> + +<h4>REST Service</h4> + +<ul class="documentation-list"> + <li><a href="rest/overview.html">Overview</a></li> + <li><a href="rest/resources.html">Resources</a></li> + <li><a href="rest/monitors.html">Monitors</a></li> + <li><a href="rest/resource-directory.html">Resource Directory</a></li> +</ul> </div> http://git-wip-us.apache.org/repos/asf/samza/blob/3902e204/docs/learn/documentation/versioned/rest/monitors.md ---------------------------------------------------------------------- diff --git a/docs/learn/documentation/versioned/rest/monitors.md b/docs/learn/documentation/versioned/rest/monitors.md new file mode 100644 index 0000000..9833068 --- /dev/null +++ b/docs/learn/documentation/versioned/rest/monitors.md @@ -0,0 +1,35 @@ +--- +layout: page +title: Monitors +--- +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + + +Samza REST supports the ability to add Monitors to the service. The initial implementation is very basic. Monitors are essentially tasks that can be scheduled to run periodically. They do not read the config and they are all scheduled at the same global interval. More capabilities will be added later, but the initial implementation supports simple cases like monitoring the YARN NodeManager and restarting it if it dies. + +## Implementing a New Monitor +Implement the [Monitor](javadocs/org/apache/samza/monitor/Monitor.html) interface with some behavior that should be executed periodically. The Monitor is Java code that invokes some method on the SAMZA Rest Service, runs a bash script to restart a failed NodeManager, or cleans old RocksDB sst files left by Host Affinity, for example. + +## Adding a New Monitor to the Samza REST Service +Add the fully-qualified class name of the Monitor implementation to the `monitor.classes` property in the service config. + +Set the `monitor.run.interval.ms` property to the appropriate interval. The `monitor()` method will be invoked at this interval. + +For more information on these properties, see the config table in the [Overview page.](overview.html) + +## [Resource Reference »](resource-directory.html) http://git-wip-us.apache.org/repos/asf/samza/blob/3902e204/docs/learn/documentation/versioned/rest/overview.md ---------------------------------------------------------------------- diff --git a/docs/learn/documentation/versioned/rest/overview.md b/docs/learn/documentation/versioned/rest/overview.md new file mode 100644 index 0000000..5b958ac --- /dev/null +++ b/docs/learn/documentation/versioned/rest/overview.md @@ -0,0 +1,94 @@ +--- +layout: page +title: Overview +--- +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Samza provides a REST service that is deployable on any node in the cluster and has pluggable interfaces to add custom [Resources](resources.html) and [Monitors](monitors.html). It is intended to be a node-local delegate for common actions such as starting jobs, sampling the local state, measuring disk usage, taking heap dumps, verifying liveness, and so on. + +The Samza REST Service does not yet have SSL enabled or authentication, so it is initially geared for more backend and operations use cases. It would not be wise to expose it as a user-facing API in environments where users are not allowed to stop each other's jobs, for example. + +Samza REST is packaged and configured very similarly to Samza jobs. A notable difference is Samza REST must be deployed and executed on each host you want it to run on, whereas a Samza Job is typically started on a master node in the cluster manager and the master deploys the job to the other nodes. + +### Deployment +Samza REST is intended to be a proxy for all operations which need to be executed from the nodes of the Samza cluster. It can be deployed to all the hosts in the cluster and may serve different purposes on different hosts. In such cases it may be useful to deploy the same release tarball with different configs to customize the functionality for the role of the hosts. For example, Samza REST may be deployed on a YARN cluster with one config for the ResourceManager (RM) hosts and another config for the NodeManager (NM) hosts. + +Deploying the service is very similar to running a Samza job. First build the tarball using: +{% highlight bash %} +./gradlew samza-rest:clean releaseRestServiceTar +{% endhighlight %} + + +Then from the extracted location, run the service using: +{% highlight bash %} +samza-example/target/bin/run-samza-rest-service.sh \ + --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory \ + --config-path=file://$PWD/config/samza-rest.properties +{% endhighlight %} + +The two config parameters have the same purpose as they do for [run-job.sh](../jobs/job-runner.html). + +Follow the [getting started tutorial](../../../tutorials/{{site.version}}/samza-rest-getting-started.html) to quickly deploy and test the Samza REST Service for the first time. + +### Configuration +The Samza REST Service relies on the same configuration system as Samza Jobs. However, the Samza REST Service config file itself is completely separate and unrelated to the config files for your Samza jobs. + +The configuration may provide values for the core configs as well as any additional configs needed for Resources or Monitors that you may have added to the service. A basic configuration file which includes configs for the core service as well as the [JobsResource](resources/jobs.html#configuration) looks like this: + +{% highlight jproperties %} +# Service port. Set to 0 for a dynamic port. +services.rest.port=9139 + +# JobProxy +job.proxy.factory.class=org.apache.samza.rest.proxy.job.SimpleYarnJobProxyFactory +# Installation path for hello-samza project. Your root may vary. +job.installations.path=/hello-samza-ROOT/deploy/samza +{% endhighlight %} + +###### Core Configuration +<table class="table table-condensed table-bordered table-striped"> + <thead> + <tr> + <th>Name</th> + <th>Default</th> + <th>Description</th> + </tr> + </thead> + <tbody> + <tr> + <td>services.rest.port</td><td></td><td><b>Required:</b> The port to use on the local host for the Samza REST Service. If 0, an available port will be dynamically chosen.</td> + </tr> + <tr> + <td>rest.resource.factory.classes</td><td></td><td>A comma-delimited list of class names that implement ResourceFactory. These factories will be used to create specific instances of resources and can pull whatever properties they need from the provided server config. The instance returned will be used for the lifetime of the server. If no value is provided for this property or <pre>rest.resource.classes</pre> then <pre>org.apache.samza.rest.resources.DefaultResourceFactory</pre> will be used as a default.</td> + </tr> + <tr> + <td>rest.resource.classes</td><td></td><td>A comma-delimited list of class names of resources to register with the server. These classes can be instantiated as often as each request, the life cycle is not guaranteed to match the server. Also, the instances do not receive any config. Note that the lifecycle and ability to receive config are the primary differences between resources added via this property versus rest.resource.factory.classes</td> + </tr> + <tr> + <td>monitor.classes</td><td></td><td>A comma-delimited list of monitor classes to use. These should be fully-qualified (org.apache.samza...) and must implement the Monitor interface.</td> + </tr> + <tr> + <td>monitor.run.interval.ms</td><td>60000</td><td>The interval at which to call the run() method of each monitor. This one value applies to all monitors. They are not individually configurable.</td> + </tr> + </tbody> +</table> + +### Logging +Samza REST uses SLF4J for logging. The `run-samza-rest-service.sh` script mentioned above by default expects a log4j.xml in the package's bin directory and writes the logs to a logs directory in the package root. However, since the script invokes the same `run-class.sh` script used to run Samza jobs, it can be reconfigured very similarly to [logging for Samza jobs](../jobs/logging.html). + +## [Resources »](resources.html) \ No newline at end of file http://git-wip-us.apache.org/repos/asf/samza/blob/3902e204/docs/learn/documentation/versioned/rest/resource-directory.md ---------------------------------------------------------------------- diff --git a/docs/learn/documentation/versioned/rest/resource-directory.md b/docs/learn/documentation/versioned/rest/resource-directory.md new file mode 100644 index 0000000..79746d1 --- /dev/null +++ b/docs/learn/documentation/versioned/rest/resource-directory.md @@ -0,0 +1,27 @@ +--- +layout: page +title: Resource Directory +--- +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +The Samza REST Service ships with the JAX-RS Resources listed below. + +- [JobsResource](resources/jobs.html) +- (Second resource coming soon) + +## [Jobs Resource »](resources/jobs.html) http://git-wip-us.apache.org/repos/asf/samza/blob/3902e204/docs/learn/documentation/versioned/rest/resources.md ---------------------------------------------------------------------- diff --git a/docs/learn/documentation/versioned/rest/resources.md b/docs/learn/documentation/versioned/rest/resources.md new file mode 100644 index 0000000..23d2eb3 --- /dev/null +++ b/docs/learn/documentation/versioned/rest/resources.md @@ -0,0 +1,61 @@ +--- +layout: page +title: Resources +--- +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Samza REST can expose any JAX-RS Resource. By default, it ships with a [JobsResource](./resources/jobs.html) which is discussed below. You can implement your own Resources and specify them in the config. + +## Implementing a New Resource +Samza REST uses the [Jersey](https://jersey.java.net/) implementation of the [JAX-RS specification](https://jax-rs-spec.java.net/). The Jersey documentation and examples are helpful for creating a new Resource. + +In addition, Samza REST imposes the following conventions, which should be followed for all Resources. + +### Versioned Paths +By convention, all resources prefix their path with a version number to enable supporting legacy APIs in the future. For example, the base URL for the JobsResource is + + /v1/jobs +If a future version of JobsResource implements a different API, it will use a different version number in the base path. + + /v2/jobs +All resources should be exposed on the latest version so clients can use a common version base for all requests. + +### Error Messages +Every error response will include a JSON message body containing a single `message` field describing the problem. For example: + + { + "message": "Unrecognized status: null" + } + +### Configuration +There are a few extra steps for Resources that require configuration values. + +1. Implement a configuration class that extends [MapConfig](../api/javadocs/org/apache/samza/config/MapConfig.html) with all the constants and accessors for the Resource's configs. The [SamzaRestConfig](javadocs/org/apache/samza/rest/SamzaRestConfig.html) should not be cluttered with Resource configs. See the [JobsResourceConfig](javadocs/org/apache/samza/rest/resources/JobsResourceConfig.html) for an example. +2. Implement or extend a [ResourceFactory](javadocs/org/apache/samza/rest/resources/ResourceFactory.html) which will use the global Samza REST config file to instantiate the MapConfig implementation from step 1 and use it to construct the Resource that requires the config. See the [DefaultResourceFactory](javadocs/org/apache/samza/rest/resources/DefaultResourceFactory.html) for an example. +3. Add all the necessary resource properties to the Samza REST config file. The [SamzaRestApplication](javadocs/org/apache/samza/rest/SamzaRestApplication.html) passes the global config into the configured ResourceFactories to instantiate the Resources with configs. + +## Adding a New Resource to the Samza REST Service +Resources are added to the Samza REST Service via config. There are two ways to add a new Resource depending on whether the Resource requires properties from the config file or not. In the former case, the Resource is instantiated once with the config and the instance is registered with the SamzaRestApplication. In the latter case, the Resource *class* is registered and it can be instantiated many times over the lifecycle of the application. + +- To add a configured Resource add the fully-qualified class name of the ResourceFactory implementation that instantiates the Resource to the `rest.resource.factory.classes` property in the service config. + +* To add a config-less Resource, add the fully-qualified class name of the Resource implementation to the `rest.resource.classes` property in the service config. + +For more information on these config properties, see the config table in the [Overview page.](overview.html#configuration) + +## [Monitors »](monitors.html) \ No newline at end of file http://git-wip-us.apache.org/repos/asf/samza/blob/3902e204/docs/learn/documentation/versioned/rest/resources/jobs.md ---------------------------------------------------------------------- diff --git a/docs/learn/documentation/versioned/rest/resources/jobs.md b/docs/learn/documentation/versioned/rest/resources/jobs.md new file mode 100644 index 0000000..68ce9c2 --- /dev/null +++ b/docs/learn/documentation/versioned/rest/resources/jobs.md @@ -0,0 +1,327 @@ +--- +layout: page +title: Jobs Resource +--- +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Exposes operations at the job scope (as opposed to the cluster, container, or task scope). The initial implementation includes the ability to list all jobs, get the status of a particular job, and start or stop an individual job. + +#API +The following sections provide general information about the response structure and detailed descriptions of each of the requests. + +## Response Structure +All responses will contain either a job status or an error message. + +**Job Status** + +Job status will be of the form: + +{% highlight json %} +{ + "status":"STOPPED", + "statusDetail":"KILLED", + "jobName":"wikipedia-parser", + "jobId":"1" +} +{% endhighlight %} + +`status` is the abstract Samza status for the job. Initially it will be one of {STARTING, STARTED, STOPPED, UNKNOWN}. + +`statusDetail` is the implementation-specific status for the job. For YARN, it will be one of the values in the YarnApplicationState enum. + +**Error Message** + +Every error response have the following structure: + +{% highlight json %} +{ + "message": "Unrecognized status parameter: null" +} +{% endhighlight %} + +`message` is the only field in the response and contains a description of the problem. +<br/> + +##Get All Jobs +Lists all the jobs installed on the host and provides their status. + +######Request + GET /v1/jobs + +######Response + Status: 200 OK +{% highlight json %}[ + + { + "status":"STOPPED", + "statusDetail":"KILLED", + "jobName":"wikipedia-parser", + "jobId":"1" + }, + { + "status":"STARTED", + "statusDetail":"RUNNING", + "jobName":"wikipedia-feed", + "jobId":"1" + }, + { + "status":"STOPPED", + "statusDetail":null, + "jobName":"wikipedia-stats", + "jobId":"1" + } +] +{% endhighlight %} + +######Response codes +<table class="table table-condensed table-bordered table-striped"> + <thead> + <tr> + <th>Status</th> + <th>Description</th> + </tr> + </thead> + <tbody> + <tr> + <td>200 OK</td><td>The operation completed successfully and the current job status for each job is returned.</td> + </tr> + <tr> + <td>500 Server Error</td><td>There was an error executing the command on the server. e.g. The command timed out.{% highlight json %} +{ + "message": "Timeout waiting for job status." +} +{% endhighlight %}</td> + </tr> + </tbody> +</table> +<br/> + +##Get Job +Gets the status of the specified job. + +######Format + GET /v1/jobs/{jobName}/{jobId} +The `{jobName}` and `{jobId}` path parameters reflect the values of 'job.name' and 'job.id' in the job config. + +######Request + GET /v1/jobs/wikipedia-feed/1 + +######Response + Status: 200 OK +{% highlight json %} +{ + "status":"STARTED", + "statusDetail":"RUNNING", + "jobName":"wikipedia-feed", + "jobId":"1" +} +{% endhighlight %} + +######Response codes +<table class="table table-condensed table-bordered table-striped"> + <thead> + <tr> + <th>Status</th> + <th>Description</th> + </tr> + </thead> + <tbody> + <tr> + <td>200 OK</td><td>The operation completed successfully and the current job status is returned.</td> + </tr> + <tr> + <td>404 Not Found</td><td>The specified job+instance was not found.{% highlight json %} +{ + "message": "Job wikipedia-¯\_(ã)_/¯-feed instance 1 is not installed on this host." +} +{% endhighlight %}</td> + </tr> + <tr> + <td>500 Server Error</td><td>There was an error executing the command on the server. e.g. The command timed out.{% highlight json %} +{ + "message": "Timeout waiting for job status." +} +{% endhighlight %}</td> + </tr> + </tbody> +</table> +<br/> + +##Start Job +Starts the job with the specified app name if it's not already started. The command will return when it has initiated the start operation. + +######Format + PUT /v1/jobs/{jobName}/{jobId}?status=started + +Form parameter `status` is the intended status of the job at the end of the request. + +######Example + PUT /v1/jobs/wikipedia-feed/1?status=started +######Response + Status: 202 Accepted +{% highlight json %} +{ + "status":"STARTING", + "statusDetail":"ACCEPTED", + "jobName": "wikipedia-feed", + "jobId": "1" +} +{% endhighlight %} + +######Response codes + +<table class="table table-condensed table-bordered table-striped"> + <thead> + <tr> + <th>Status</th> + <th>Description</th> + </tr> + </thead> + <tbody> + <tr> + <td>200 OK</td><td>The operation completed successfully and the current job status is returned.</td> + </tr> + <tr> + <td>400 Bad Request</td><td>There was a problem with the request. e.g. an invalid status parameter.{% highlight json %} +{ + "message": "Unrecognized status parameter: null" +} +{% endhighlight %}</td> + <tr> + <td>404 Not Found</td><td>The specified job+instance was not found.{% highlight json %} +{ + "message": "Job wikipedia-¯\_(ã)_/¯-feed instance 1 is not installed on this host." +} +{% endhighlight %}</td> + </tr> + <tr> + <td>500 Server Error</td><td>There was an error executing the command on the server. e.g. The command timed out.{% highlight json %} +{ + "message": "Timeout waiting for job status." +} +{% endhighlight %}</td> + </tr> + </tbody> +</table> +<br/> + +##Stop Job +Stops the job with the specified app name if it's not already stopped. + +######Format + PUT /v1/jobs/{jobName}/{jobId}?status=stopped + +Form parameter `status` is the intended status of the job at the end of the request. + +######Example + PUT /v1/jobs/wikipedia-feed/1?status=stopped +######Response + Status: 202 Accepted +{% highlight json %} +{ + "status":"STOPPED", + "statusDetail":"KILLED", + "jobName": "wikipedia-feed", + "jobId": "1" +} +{% endhighlight %} + +######Response codes +<table class="table table-condensed table-bordered table-striped"> + <thead> + <tr> + <th>Status</th> + <th>Description</th> + </tr> + </thead> + <tbody> + <tr> + <td>200 OK</td><td>The operation completed successfully and the current job status is returned.</td> + </tr> + <tr> + <td>400 Bad Request</td><td>There was a problem with the request. e.g. an invalid status parameter.{% highlight json %} +{ + "message": "Unrecognized status parameter: null" +} +{% endhighlight %}</td> + <tr> + <td>404 Not Found</td><td>The specified job+instance was not found.{% highlight json %} +{ + "message": "Job wikipedia-¯\_(ã)_/¯-feed instance 1 is not installed on this host." +} +{% endhighlight %}</td> + </tr> + <tr> + <td>500 Server Error</td><td>There was an error executing the command on the server. e.g. The command timed out.{% highlight json %} +{ + "message": "Timeout waiting for job status." +} +{% endhighlight %}</td> + </tr> + </tbody> +</table> +<br/> + +# Design +###Abstractions +There are three primary abstractions used by the JobsResource that users can implement to handle any details specific to their environment. + +1. **JobProxy**: The JobProxy is the central point of interacting with Samza jobs. It exposes generic methods to start, stop, and get the status of a Samza job. Implementations of this interface can employ custom code to implement these methods tailored to the specific API for any cluster manager. +2. **InstallationFinder**: The InstallationFinder provides a generic interface to discover all the installed jobs, hiding any customizations in the job package structure and its location (e.g. local vs remote host). The InstallationFinder also resolves the job configuration, which is used to validate and identify the job. +3. **JobStatusProvider**: The JobStatusProvider allows the JobProxy to get the status of a job in a generic way. The same interface can be used to get the job status on Yarn, Mesos, or standalone jobs. It also enables different implementations for the same cluster. With Yarn, for example, one implementation may get job status via command line and another via the ResourceManager REST API. + +The [configuration](jobs.html#configuration) must specify a JobProxy factory class explicitly. By contrast, the InstallationFinder and JobStatusProvider abstractions are natural extensions of the JobProxy and are solely provided to demonstrate a pattern for discovering installed jobs and fetching job status. However, they are not an explicit requirement. + +The `SimpleYarnJobProxy` that ships with Samza REST is intended to demonstrate a functional implementation of a JobProxy which works with the Hello Samza jobs. See the [tutorial](../../../../tutorials/{{site.version}}/samza-rest-getting-started.html) to try it out. You can implement your own JobProxy to adapt the JobsResource to the specifics of your job packaging and deployment model. + + +### Request Flow +After validating each request, the JobsResource invokes the appropriate JobProxy command. The JobProxy uses the InstallationFinder to enumerate the installed jobs and the JobStatusProvider to get the runtime status of the jobs. + +The provided [SimpleInstallationFinder](../javadocs/org/apache/samza/rest/proxy/installation/SimpleInstallationFinder.html) crawls the file system, starting in the directory specified by the `job.installations.path` looking for valid Samza job config files. It extracts the `job.name` and `job.id` property values and creates an [InstallationRecord](../javadocs/org/apache/samza/rest/proxy/installation/InstallationRecord.html) for the each job instance. The InstallationRecord contains all the information needed to start, stop, and get the status for the job. + +The provided [YarnCliJobStatusProvider](../javadocs/org/apache/samza/rest/proxy/job/YarnCliJobStatusProvider.html) leverages a ScriptRunner to fetch job status using the Yarn ApplicationCLI. + +The [SimpleYarnJobProxy](../javadocs/org/apache/samza/rest/proxy/job/SimpleYarnJobProxy.html) relies on the scripts in the InstallationRecord scriptFilePath (`/bin`) directory to start and stop jobs. + +The following is a depiction of the implementation that ships with Samza REST: + + + +## Configuration +The JobsResource properties should be specified in the same file as the Samza REST configuration. They are specified here for clarity. + +<table class="table table-condensed table-bordered table-striped"> + <thead> + <tr> + <th>Name</th> + <th>Description</th> + </tr> + </thead> + <tbody> + <tr> + <td>job.proxy.factory.class</td><td><b>Required:</b> The Job Proxy Factory that will be used to create a Job Proxy to control the job. The value is a fully-qualified class name which must implement JobProxyFactory. Samza ships with one implementation: <pre>org.apache.samza.rest.proxy.job.SimpleYarnJobProxy</pre> <li> Executes shell scripts to start, stop, and get the status of jobs, relying primarily on the YARN ApplicationCLI. It uses the <pre>SimpleInstallationRecord</pre> abstraction to interact with Samza jobs installed on disk.</li></td> + </tr> + <tr> + <td>job.installations.path</td><td><b>Required:</b> The file system path which contains the Samza job installations. The path must be on the same host as the Samza REST Service. Each installation must be a directory with structure conforming to the expectations of the InstallationRecord implementation used by the JobProxy.</td> + </tr> + <tr> + <td>job.config.factory.class</td><td>The config factory to use for reading Samza job configs. This is used to fetch the job.name and job.id properties for each job instance in the InstallationRecord. It's also used to validate that a particular directory within the installations path actually contains Samza jobs. If not specified <pre>org.apache.samza.config.factories.PropertiesConfigFactory</pre> will be used. </td> + </tr> + </tbody> +</table> http://git-wip-us.apache.org/repos/asf/samza/blob/3902e204/docs/learn/tutorials/versioned/index.md ---------------------------------------------------------------------- diff --git a/docs/learn/tutorials/versioned/index.md b/docs/learn/tutorials/versioned/index.md index b4d687a..ca2b08f 100644 --- a/docs/learn/tutorials/versioned/index.md +++ b/docs/learn/tutorials/versioned/index.md @@ -29,6 +29,8 @@ title: Tutorials [Run Hello-samza without Internet](run-hello-samza-without-internet.html) +[Getting Started with Samza REST](samza-rest-getting-started.html) + <!-- TODO a bunch of tutorials [Log Walkthrough](log-walkthrough.html) <a href="configuring-kafka-system.html">Configuring a Kafka System</a><br/> http://git-wip-us.apache.org/repos/asf/samza/blob/3902e204/docs/learn/tutorials/versioned/samza-rest-getting-started.md ---------------------------------------------------------------------- diff --git a/docs/learn/tutorials/versioned/samza-rest-getting-started.md b/docs/learn/tutorials/versioned/samza-rest-getting-started.md new file mode 100644 index 0000000..942329e --- /dev/null +++ b/docs/learn/tutorials/versioned/samza-rest-getting-started.md @@ -0,0 +1,102 @@ +--- +layout: page +title: Getting Started with Samza REST +--- +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +This tutorial depends on [hello-samza](../../../startup/hello-samza/{{site.version}}/) to start some example jobs on a local cluster, which you will then access via the [JobsResource](../../documentation/{{site.version}}/rest/resources/jobs.html). After completing this tutorial, you will have built and deployed the Samza REST resource locally, changed the configuration for the JobsResource, and executed a couple of basic curl requests to verify the service works. + +Lets get started. + +### Run Hello Samza Jobs Locally + +Follow the [hello-samza](../../../startup/hello-samza/{{site.version}}/) tutorial to setup a local grid and run the wikipedia jobs. Skip the [shutdown step](../../../startup/hello-samza/{{site.version}}/#shutdown) because you need the grid to still be running to query the REST service for jobs. You can optionally skip all the ```kafka-console-consumer.sh``` commands if you don't want to verify the output of the jobs. + +Take note of the path where you cloned hello-samza. You will need this to configure the installations path for the JobsResource. + + +#### Build the Samza REST Service package +The source code for Samza REST is in the samza-rest module of the Samza repository. It To build it, execute the following gradle task from the root of the project. +{% highlight bash %} +./gradlew samza-rest:clean releaseRestServiceTar +{% endhighlight %} + +#### Deploy the Samza REST Service Locally +To deploy the service, you simply extract the tarball to the desired location. Here, we will deploy the tarball on the local host in + +``` +SAMZA_ROOT/samza-rest/build/distributions/deploy/samza-rest +``` +where ```SAMZA_ROOT``` is the path to the root of your Samza project. + +Run the following commands: +{% highlight bash %} +cd samza-rest/build/distributions/ +mkdir -p deploy/samza-rest +tar -xvf ./samza-rest-0.10.1-SNAPSHOT.tgz -C deploy/samza-rest +{% endhighlight %} + +#### Configure the Installations Path +The JobsResource has a required config [job.installations.path](../../documentation/{{site.version}}/rest/resources/jobs.html#configuration) which specifies the path where the jobs are installed. Edit the configuration file: + +``` +deploy/samza-rest/config/samza-rest.properties +``` + +Set the job.installations.path to: + +``` +job.installations.path=/hello-samza-ROOT/deploy +``` + +where ```hello-samza-ROOT``` is the path to your hello-samza clone, noted above. This tells the JobsResource to crawl this location to find all the installed jobs. + +#### Start the Samza REST Service +To deploy the service, run the run-samza-rest-service.sh script from the extracted directory. +{% highlight bash %} +cd deploy/samza-rest +./bin/run-samza-rest-service.sh \ + --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory \ + --config-path=file://$PWD/config/samza-rest.properties +{% endhighlight %} + +You provide two parameters to the run-samza-rest-service.sh script. One is the config location, and the other, optional, parameter is a factory class that is used to read your configuration file. The SamzaRestService uses your ConfigFactory to get a Config object from the config path. The ConfigFactory is covered in more detail on the [Job Runner page](../../documentation/{{site.version}}/jobs/job-runner.html). The run-samza-rest-service.sh script will block until the SamzaRestService terminates. + +Note: With the default settings, the JobsResource will expect a YARN cluster with a local Resource Manager accessible via the ApplicationCLI. Without YARN, the JobsResource will not respond to any requests. So it's important to walk through hello-samza demo before the next step. + + +### Curl the Default REST Service +Curl the JobsResource to get all installed jobs + +``` +curl localhost:9139/v1/jobs +[{"jobName":"wikipedia-stats","jobId":"1","status":"STARTED","statusDetail":RUNNING},{"jobName":"wikipedia-parser","jobId":"1","status":"STARTED","statusDetail":RUNNING},{"jobName":"wikipedia-feed","jobId":"1","status":"STARTED","statusDetail":RUNNING} +``` + +Now curl the JobsResource to stop one of the jobs + +``` +curl -X PUT localhost:9139/v1/jobs/wikipedia-feed/1?status=stopped +{"jobId":"1","jobName":"wikipedia-feed","status":"STOPPED","statusDetail":"FINISHED"} +``` + +Congratulations, you've successfully deployed the Samza REST Service and used the JobsResource to list jobs and stop a job! + +See the [JobsResource documentation](../../documentation/{{site.version}}/rest/resources/jobs.html) for the rest of its API. + +See the [Resources documentation](../../documentation/{{site.version}}/rest/resources.html) for more information about Resources and how you can add your own.
