This is an automated email from the ASF dual-hosted git repository.
karthikz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 38f6a57 add documentation about nomad consul integration (#2898)
38f6a57 is described below
commit 38f6a570d081ad1429d444f392a6679737084bcb
Author: Boyang Jerry Peng <jerry.boyang.p...@gmail.com>
AuthorDate: Thu May 17 07:45:57 2018 -0700
add documentation about nomad consul integration (#2898)
* add documentation about nomad consul integration
* add newline
---
heron/config/src/yaml/conf/nomad/scheduler.yaml | 4 +--
.../config/src/yaml/conf/standalone/scheduler.yaml | 2 +-
.../deployment/schedulers/nomad-docker.md | 34 ++++++++++++++++++--
.../deployment/schedulers/nomad-raw-execs.md | 36 ++++++++++++++++++++--
4 files changed, 68 insertions(+), 8 deletions(-)
diff --git a/heron/config/src/yaml/conf/nomad/scheduler.yaml
b/heron/config/src/yaml/conf/nomad/scheduler.yaml
index 8e7e3a5..832ca76 100644
--- a/heron/config/src/yaml/conf/nomad/scheduler.yaml
+++ b/heron/config/src/yaml/conf/nomad/scheduler.yaml
@@ -58,5 +58,5 @@ heron.nomad.metrics.service.check.interval.sec: 10
heron.nomad.metrics.service.check.timeout.sec: 2
# additional tags to be attached to metrics service
-# A tag of <topology-name>-<container-index> with be automaticallu attached
-heron.nomad.metrics.service.additional.tags: "prometheus,metrics,heron"
\ No newline at end of file
+# A tag of <topology-name>-<container-index> with be automatically attached
+heron.nomad.metrics.service.additional.tags: "prometheus,metrics,heron"
diff --git a/heron/config/src/yaml/conf/standalone/scheduler.yaml
b/heron/config/src/yaml/conf/standalone/scheduler.yaml
index fc1cd27..a492ef2 100644
--- a/heron/config/src/yaml/conf/standalone/scheduler.yaml
+++ b/heron/config/src/yaml/conf/standalone/scheduler.yaml
@@ -51,5 +51,5 @@ heron.nomad.metrics.service.check.interval.sec: 10
heron.nomad.metrics.service.check.timeout.sec: 2
# additional tags to be attached to metrics service in a comma delimited list
-# A tag of <topology-name>-<container-index> with be automaticallu attached
+# A tag of <topology-name>-<container-index> with be automatically attached
heron.nomad.metrics.service.additional.tags: "prometheus,metrics,heron"
diff --git
a/website/content/docs/operators/deployment/schedulers/nomad-docker.md
b/website/content/docs/operators/deployment/schedulers/nomad-docker.md
index 063a4cc..cde8320 100644
--- a/website/content/docs/operators/deployment/schedulers/nomad-docker.md
+++ b/website/content/docs/operators/deployment/schedulers/nomad-docker.md
@@ -4,7 +4,7 @@ title: Running Heron via Docker Containers on Nomad
Below are instructions on how to run Heron on Nomad via docker containers. In
this mode, Heron executors will run as docker containers on host machines.
-### Requirements
+## Requirements
When setting up your Nomad cluster, the following are required:
@@ -14,7 +14,7 @@ When setting up your Nomad cluster, the following are
required:
* Docker installed and enabled on every machine
* Each machine must also be able to pull the official Heron docker image from
DockerHub or have the image preloaded.
-### Configuring Heron settings
+## Configuring Heron settings
Before running Heron via Nomad, you'll need to configure some settings. Once
you've [installed Heron](../../../../getting-started), all of the
configurations you'll need to modify will be in the `~/.heron/conf/nomad`
diredctory.
@@ -132,3 +132,33 @@ Heron users can upload their Heron topology packages to
the Heron API server usi
heron.class.uploader: org.apache.heron.uploader.http.HttpUploader
heron.uploader.http.uri: http://localhost:9000/api/v1/file/upload
```
+
+## Integration with Consul for metrics
+Each container part of a Heron topology serves metrics out of a port randomly
generated by Nomad. Thus, Consul is needed for service discovery for users to
determine which port the container is serving the metrics out of.
+Every Heron executor running in a docker container will automatically register
itself as a service with Consul given that there is a Consul cluster running.
The port Heron will be serving metrics will be registered with Consul.
+
+The service will be registered with the name with the following format:
+
+```yaml
+metrics-heron-<TOPOLOGY_NAME>-<CONTAINER_INDEX>
+```
+
+Each heron executor registered with Consul will be tagged with
+
+```yaml
+<TOPOLOGY_NAME>-<CONTAINER_INDEX>
+```
+
+To add additional tags, please add specify them in a comma delimited list via
+
+```yaml
+heron.nomad.metrics.service.additional.tags
+```
+
+in `scheduler.yaml`. For example:
+
+```yaml
+heron.nomad.metrics.service.additional.tags: "prometheus,metrics,heron"
+```
+
+Users can then configure Prometheus to scrape metrics for each container based
on these tags
diff --git
a/website/content/docs/operators/deployment/schedulers/nomad-raw-execs.md
b/website/content/docs/operators/deployment/schedulers/nomad-raw-execs.md
index b362123..79cc02a 100644
--- a/website/content/docs/operators/deployment/schedulers/nomad-raw-execs.md
+++ b/website/content/docs/operators/deployment/schedulers/nomad-raw-execs.md
@@ -6,7 +6,7 @@ Below are instructions on how to to run Heron on Nomad via raw
execs. In this m
The advantages of this mode is that it is incredibly lightweight and likely do
not require sudo privileges to setup and run. However in this mode, the setup
procedure may be a little more complex compared to running via docker since
there are more things to consider. Also in resource allocation is considered
but not enforced.
-### Requirements
+## Requirements
When setting up your Nomad cluster, the following are required:
@@ -14,7 +14,7 @@ When setting up your Nomad cluster, the following are
required:
* Python 2.7, Java 7 or 8, and [curl](https://curl.haxx.se/) must be installed
on every machine in the cluster
* A [ZooKeeper cluster](https://zookeeper.apache.org)
-### Configuring Heron settings
+## Configuring Heron settings
Before running Heron via Nomad, you'll need to configure some settings. Once
you've [installed Heron](../../../../getting-started), all of the
configurations you'll need to modify will be in the `~/.heron/conf/nomad`
diredctory.
@@ -52,7 +52,7 @@ heron.scheduler.local.working.directory:
${HOME}/.herondata/topologies/${CLUSTER
> Heron uses string interpolation to fill in the missing values for `CLUSTER`,
> `ROLE`, etc.
-### Distributing Heron core
+## Distributing Heron core
The Heron core package needs to be made available for every machine in the
cluster to download. You'll need to provide a URI for the Heron core package.
Here are the currently supported protocols:
@@ -214,3 +214,33 @@ $ heron submit nomad \
org.apache.heron.examples.api.WindowedWordCountTopology \
windowed-word-count
```
+
+## Integration with Consul for metrics
+Each Heron executor part of a Heron topology serves metrics out of a port
randomly generated by Nomad. Thus, Consul is needed for service discovery for
users to determine which port the Heron executor is serving the metrics out of.
+Every Heron executor will automatically register itself as a service with
Consul given that there is a Consul cluster running. The port Heron will be
serving metrics will be registered with Consul.
+
+The service will be registered with the name with the following format:
+
+```yaml
+metrics-heron-<TOPOLOGY_NAME>-<CONTAINER_INDEX>
+```
+
+Each heron executor registered with Consul will be tagged with
+
+```yaml
+<TOPOLOGY_NAME>-<CONTAINER_INDEX>
+```
+
+To add additional tags, please add specify them in a comma delimited list via
+
+```yaml
+heron.nomad.metrics.service.additional.tags
+```
+
+in `scheduler.yaml`. For example:
+
+```yaml
+heron.nomad.metrics.service.additional.tags: "prometheus,metrics,heron"
+```
+
+Users can then configure Prometheus to scrape metrics for each Heron executor
based on these tags
--
To stop receiving notification emails like this one, please contact
karth...@apache.org.
