This is an automated email from the ASF dual-hosted git repository.
nwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/master by this push:
new 5e8fb15 Joshfischer/update docs (#3298)
5e8fb15 is described below
commit 5e8fb156ca39628dc94c6a8aa1623d1f4389cede
Author: Josh Fischer <[email protected]>
AuthorDate: Mon Jun 24 11:22:55 2019 -0500
Joshfischer/update docs (#3298)
* updating missed link. Updating cluster config section
* correcting observability
* updating user manuals section
* adding missed heron shell file. Updating links for compiling heron
* update community pages and links. Fix links in extending heron.
Updating resources page.
* fixing broken links
---
website2/docs/cluster-config-instance.md | 4 +-
website2/docs/cluster-config-metrics.md | 2 +-
website2/docs/cluster-config-overview.md | 16 +++---
website2/docs/cluster-config-stream.md | 2 +-
website2/docs/cluster-config-tmaster.md | 2 +-
website2/docs/community-contact.md | 18 +++++++
website2/docs/community-events.md | 9 ++++
website2/docs/compiling-code-organization.md | 60 ++++++++++------------
website2/docs/compiling-docker.md | 4 +-
website2/docs/compiling-osx.md | 2 +-
website2/docs/compiling-overview.md | 22 ++++----
website2/docs/compiling-running-tests.md | 4 +-
website2/docs/extending-heron-metric-sink.md | 14 ++---
website2/docs/extending-heron-scheduler.md | 13 +++--
website2/docs/getting-started-local-single-node.md | 5 +-
website2/docs/heron-architecture.md | 2 +-
website2/docs/heron-resources-resources.md | 2 +-
website2/docs/observability-graphite.md | 4 +-
website2/docs/observability-prometheus.md | 2 +-
website2/docs/observability-scribe.md | 2 +-
website2/docs/user-manuals-heron-cli.md | 4 +-
website2/docs/user-manuals-heron-explorer.md | 8 +--
website2/docs/user-manuals-heron-shell.md | 56 ++++++++++++++++++++
.../docs/user-manuals-heron-tracker-runbook.md | 10 ++--
website2/docs/user-manuals-heron-ui.md | 11 ++--
website2/docs/user-manuals-tracker-rest.md | 6 +--
website2/website/sidebars.json | 3 +-
website2/website/static/js/custom.js | 6 +--
28 files changed, 186 insertions(+), 107 deletions(-)
diff --git a/website2/docs/cluster-config-instance.md
b/website2/docs/cluster-config-instance.md
index fa556e8..e311b54 100644
--- a/website2/docs/cluster-config-instance.md
+++ b/website2/docs/cluster-config-instance.md
@@ -5,7 +5,7 @@ sidebar_label: Heron Instance
---
You can configure the behavior of the [Heron
-Instances](../../../concepts/architecture#heron-instance) (HIs) in a
+Instances](heron-architecture#heron-instance) (HIs) in a
topology using the parameters below.
## Internal Configuration
@@ -50,7 +50,7 @@ Parameter | Meaning | Default
## Metrics Manager Configuration
These parameters deal with how each HI interacts with the topology's [Stream
-Manager](../../../concepts/architecture#stream-manager).
+Manager](heron-architecture#stream-manager).
Parameter | Meaning | Default
:-------- |:------- |:-------
diff --git a/website2/docs/cluster-config-metrics.md
b/website2/docs/cluster-config-metrics.md
index ec2db56..b2c203b 100644
--- a/website2/docs/cluster-config-metrics.md
+++ b/website2/docs/cluster-config-metrics.md
@@ -5,7 +5,7 @@ sidebar_label: Metrics Manager
---
You can configure all of the [Metrics
-Managers](../../../concepts/architecture#metrics-manager) (MMs) in a topology
+Managers](heron-architecture#metrics-manager) (MMs) in a topology
using the parameters below.
## Network Configuration
diff --git a/website2/docs/cluster-config-overview.md
b/website2/docs/cluster-config-overview.md
index de008ea..6ff339c 100644
--- a/website2/docs/cluster-config-overview.md
+++ b/website2/docs/cluster-config-overview.md
@@ -11,7 +11,7 @@ Heron cluster rather than to any specific component (e.g.
logging configurations
2. **The component level** --- Component-level configurations enable you to
establish
default configurations for different components.
These configurations are fixed at any stage of the topology's
-[lifecycle](../../../concepts/topologies#topology-lifecycle), once the topology
+[lifecycle](heron-topology-concepts#topology-lifecycle), once the topology
is deployed.
Neither system- nor component-level configurations can be overridden by
topology developers.
@@ -19,13 +19,13 @@ Neither system- nor component-level configurations can be
overridden by topology
All system-level configs and component-level defaults are declared in a
[YAML](http://www.yaml.org/) config file in
`heron/config/src/yaml/conf/{cluster}/heron_internals.yaml`
in the Heron codebase. You can leave that file as is when [compiling
-Heron](../../../developers/compiling) or modify the values to suit your use
+Heron](compiling-overview) or modify the values to suit your use
case.
## The System Level
There are a small handful of system-level configs for Heron. These are detailed
-in [System-level Configuration](../system).
+in [System-level Configuration](cluster-config-system-level).
## The Component Level
@@ -33,10 +33,10 @@ There is a wide variety of component-level configurations
that you can establish
as defaults in your Heron cluster. These configurations tend to apply to
specific components in a topology and are detailed in the docs below:
-* [Heron Instance](../instance)
-* [Heron Metrics Manager](../metrics-manager)
-* [Heron Stream Manager](../stmgr)
-* [Heron Topology Master](../tmaster)
+* [Heron Instance](cluster-config-instance)
+* [Heron Metrics Manager](cluster-config-metrics)
+* [Heron Stream Manager](cluster-config-stream)
+* [Heron Topology Master](cluster-config-tmaster)
### Overriding Heron Cluster Configuration
@@ -46,4 +46,4 @@ It is not possible to override the Heron configuration
for a topology via Heron client or other Heron tools.
More on Heron's CLI tool can be found in [Managing Heron
-Topologies](../../heron-cli).
+Topologies](user-manuals-heron-cli).
diff --git a/website2/docs/cluster-config-stream.md
b/website2/docs/cluster-config-stream.md
index dbcf088..bfbbc51 100644
--- a/website2/docs/cluster-config-stream.md
+++ b/website2/docs/cluster-config-stream.md
@@ -6,7 +6,7 @@ sidebar_label: Stream Manager
You can configure the [Stream
-Manager](../../../concepts/architecture#stream-manager) (SM) in a
+Manager](heron-architecture#stream-manager) (SM) in a
topology using the parameters below, including how the SM handles [back
pressure](#back-pressure-parameters).
diff --git a/website2/docs/cluster-config-tmaster.md
b/website2/docs/cluster-config-tmaster.md
index 9cdaed3..7891e8c 100644
--- a/website2/docs/cluster-config-tmaster.md
+++ b/website2/docs/cluster-config-tmaster.md
@@ -5,7 +5,7 @@ sidebar_label: Topology Master
---
You can configure the [Topology
-Master](../../../concepts/architecture#metrics-manager) (TM) for a topology
+Master](heron-architecture#topology-master) (TM) for a topology
using the parameters below.
Parameter | Meaning | Default
diff --git a/website2/docs/community-contact.md
b/website2/docs/community-contact.md
new file mode 100644
index 0000000..bed72a8
--- /dev/null
+++ b/website2/docs/community-contact.md
@@ -0,0 +1,18 @@
+---
+id: community-contact
+title: Contact
+sidebar_label: Contact
+---
+
+## Contact
+
+#### Mailing lists
+
+| Name |
Scope |
|
|
|
+|:--------------------------------------------------------------------------|:--------------------------------|:---------------------------------------------------------------|:-------------------------------------------------------------------|:--------------------------------------------------------------------------|
+| [[email protected]](mailto:[email protected]) |
User-related discussions |
[Subscribe](mailto:[email protected]) |
[Unsubscribe](mailto:[email protected]) |
[Archives](http://mail-archives.apache.org/mod_mbox/incubator-heron-user/)|
+| [[email protected]](mailto:[email protected])
| Development-related discussions |
[Subscribe](mailto:[email protected]) |
[Unsubscribe](mailto:[email protected]) |
[Archives](http://mail-archives.apache.org/mod_mbox/incubator-heron-dev/) |
+
+#### Slack
+
+[Self-Register](http://heronstreaming.herokuapp.com/) to our [Heron Slack
Workspace](https://heronstreaming.slack.com/)
\ No newline at end of file
diff --git a/website2/docs/community-events.md
b/website2/docs/community-events.md
new file mode 100644
index 0000000..d7014a3
--- /dev/null
+++ b/website2/docs/community-events.md
@@ -0,0 +1,9 @@
+---
+id: community-events
+title: Community Events
+sidebar_label: Community Events
+---
+
+Groups
+
+[Apache Heron Bay Area Meetup](https://www.meetup.com/Apache-Heron-Bay-Area)
\ No newline at end of file
diff --git a/website2/docs/compiling-code-organization.md
b/website2/docs/compiling-code-organization.md
index 0f7ea24..067561d 100644
--- a/website2/docs/compiling-code-organization.md
+++ b/website2/docs/compiling-code-organization.md
@@ -9,33 +9,29 @@ for developers who want to contribute to Heron. The Heron
codebase lives on
[github]({{% githubMaster %}}).
If you're looking for documentation about developing topologies for a Heron
-cluster, see [Building Topologies](../../developers/java/topologies) instead.
+cluster, see [Building Topologies](topology-development-topology-api-java)
instead.
## Languages
The primary programming languages for Heron are C++, Java, and Python.
* **C++ 11** is used for most of Heron's core components, including the
-[Topology Master](../../concepts/architecture#topology-master), and
-[Stream Manager](../../concepts/architecture#stream-manager).
+[Topology Master](heron-architecture#topology-master), and
+[Stream Manager](heron-architecture#stream-manager).
* **Java 8** is used primarily for Heron's [topology
-API](../../concepts/topologies), and [Heron
Instance](../../concepts/architecture#heron-instance).
+API](heron-topology-concepts), and [Heron
Instance](heron-architecture#heron-instance).
It is currently the only language in which topologies can be written.
Instructions can be found
in [Building Topologies](../../developers/java/topologies), while
documentation for the Java
API can be found
[here](/api/org/apache/heron/api/topology/package-summary.html). Please note
that Heron topologies do not
require Java 8 and can be written in Java 7 or later.
-* **Python 2** (specifically 2.7) is used primarily for Heron's [CLI
-interface](../..//operators/heron-cli) and UI components such as [Heron
-UI](../../operators/heron-ui) and the [Heron
-Tracker](../../operators/heron-tracker).
+* **Python 2** (specifically 2.7) is used primarily for Heron's [CLI
interface](user-manuals-heron-cli) and UI components such as [Heron
UI](user-manuals-heron-ui) and the [Heron
Tracker](user-manuals-heron-tracker-runbook).
## Main Tools
* **Build tool** --- Heron uses [Bazel](http://bazel.io/) as its build tool.
-Information on setting up and using Bazel for Heron can be found in [Compiling
-Heron](../../developers/compiling/compiling).
+Information on setting up and using Bazel for Heron can be found in [Compiling
Heron](compiling-overview).
* **Inter-component communication** --- Heron uses [Protocol
Buffers](https://developers.google.com/protocol-buffers/?hl=en) for
@@ -43,8 +39,8 @@ communication between components. Most `.proto` definition
files can be found in
[`heron/proto`]({{% githubMaster %}}/heron/proto).
* **Cluster coordination** --- Heron relies heavily on ZooKeeper for cluster
-coordination for distributed deployment, be it for
[Aurora](../../operators/deployment/schedulers/aurora) or for a [custom
-scheduler](../custom-scheduler) that you build. More information on ZooKeeper
+coordination for distributed deployment, be it for
[Aurora](schedulers-aurora-cluster) or for a [custom
+scheduler](extending-heron-scheduler) that you build. More information on
ZooKeeper
components in the codebase can be found in the [State
Management](#state-management) section below.
@@ -57,16 +53,16 @@ utilities, networking interfaces, and more.
## Cluster Scheduling
Heron supports two cluster schedulers out of the box:
-[Aurora](../../operators/deployment/schedulers/aurora) and a [local
-scheduler](../../operators/deployment/schedulers/local). The Java code for
each of those
+[Aurora](schedulers-aurora-cluster) and a [local
+scheduler](schedulers-local). The Java code for each of those
schedulers can be found in [`heron/schedulers`]({{% githubMaster
%}}/heron/schedulers)
, while the underlying scheduler API can be found
[here](/api/org/apache/heron/spi/scheduler/package-summary.html)
Info on custom schedulers can be found in [Implementing a Custom
-Scheduler](../custom-scheduler); info on the currently available schedulers
+Scheduler](extending-heron-scheduler); info on the currently available
schedulers
can be found in [Deploying Heron on
-Aurora](../../operators/deployment/schedulers/aurora) and [Local
-Deployment](../../operators/deployment/schedulers/local).
+Aurora](schedulers-aurora-cluster) and [Local
+Deployment](schedulers-local).
## State Management
@@ -83,29 +79,29 @@ Heron components.
### Topology Master
The C++ code for Heron's [Topology
-Master](../../concepts/architecture#topology-master) is written in C++ can be
+Master](heron-architecture#topology-master) is written in C++ can be
found in [`heron/tmaster`]({{% githubMaster %}}/heron/tmaster).
### Stream Manager
The C++ code for Heron's [Stream
-Manager](../../concepts/architecture#stream-manager) can be found in
+Manager](heron-architecture#stream-manager) can be found in
[`heron/stmgr`]({{% githubMaster %}}/heron/stmgr).
### Heron Instance
The Java code for [Heron
-instances](../../concepts/architecture#heron-instance) can be found in
+instances](heron-architecture#heron-instance) can be found in
[`heron/instance`]({{% githubMaster %}}/heron/instance).
### Metrics Manager
The Java code for Heron's [Metrics
-Manager](../../concepts/architecture#metrics-manager) can be found in
+Manager](heron-architecture#metrics-manager) can be found in
[`heron/metricsmgr`]({{% githubMaster %}}/heron/metricsmgr).
If you'd like to implement your own custom metrics handler (known as a
**metrics
-sink**), see [Implementing a Custom Metrics Sink](../custom-metrics-sink).
+sink**), see [Implementing a Custom Metrics Sink](extending-heron-metric-sink).
## Developer APIs
@@ -115,12 +111,12 @@ Heron's API for writing topologies is written in Java.
The code for this API can
be found in [`heron/api`]({{% githubMaster %}}/heron/api).
Documentation for writing topologies can be found in [Building
-Topologies](../../developers/java/topologies), while API documentation can be
found
+Topologies](topology-development-topology-api-java), while API documentation
can be found
[here](/api/org/apache/heron/api/topology/package-summary.html).
### Simulator
-Heron enables you to run topologies in
[`Simulator`](../../developers/simulator-mode)
+Heron enables you to run topologies in [`Simulator`](guides-simulator-mode)
for debugging purposes.
The Java API for simulator can be found in
@@ -129,7 +125,7 @@ The Java API for simulator can be found in
### Example Topologies
Heron's codebase includes a wide variety of example
-[topologies](../../concepts/topologies) built using Heron's topology API for
+[topologies](heron-topology-concepts) built using Heron's topology API for
Java. Those examples can be found in
[`heron/examples`]({{% githubMaster %}}/heron/examples).
@@ -138,19 +134,19 @@ Java. Those examples can be found in
### Heron CLI
Heron has a tool called `heron` that is used to both provide a CLI interface
-for [managing topologies](../../operators/heron-cli) and to perform much of
+for [managing topologies](user-manuals-heron-cli) and to perform much of
the heavy lifting behind assembling physical topologies in your cluster.
The Python code for `heron` can be found in
[`heron/tools/cli`]({{% githubMaster %}}/heron/tools/cli).
Sample configurations for different Heron schedulers
-* [Local scheduler](../../operators/deployment/schedulers/local) config can be
found in [`heron/config/src/yaml/conf/local`]({{% githubMaster
%}}/heron/config/src/yaml/conf/local),
-* [Aurora scheduler](../../operators/deployment/schedulers/aurora) config can
be found [`heron/config/src/yaml/conf/aurora`]({{% githubMaster
%}}/heron/config/src/yaml/conf/aurora).
+* [Local scheduler](schedulers-local) config can be found in
[`heron/config/src/yaml/conf/local`]({{% githubMaster
%}}/heron/config/src/yaml/conf/local),
+* [Aurora scheduler](schedulers-aurora-cluster) config can be found
[`heron/config/src/yaml/conf/aurora`]({{% githubMaster
%}}/heron/config/src/yaml/conf/aurora).
### Heron Tracker
-The Python code for the [Heron Tracker](../../operators/heron-tracker) can be
+The Python code for the [Heron Tracker](user-manuals-heron-tracker-runbook)
can be
found in [`heron/tools/tracker`]({{% githubMaster %}}/heron/tools/tracker).
The Tracker is a web server written in Python. It relies on the
@@ -162,7 +158,7 @@ corresponding handlers in the
### Heron UI
-The Python code for the [Heron UI](../../operators/heron-ui) can be found in
+The Python code for the [Heron UI](user-manuals-heron-ui) can be found in
[`heron/tools/ui`]({{% githubMaster %}}/heron/tools/ui).
Like Heron Tracker, Heron UI is a web server written in Python that relies on
@@ -174,7 +170,7 @@ directory.
### Heron Shell
-The Python code for the [Heron Shell](../../operators/heron-shell) can be
+The Python code for the [Heron Shell](user-manuals-heron-shell) can be
found in [`heron/shell`]({{% githubMaster %}}/heron/shell). The HTTP handlers
and
web server are defined in
[`main.py`]({{% githubMaster %}}/heron/shell/src/python/main.py) while the
HTML,
@@ -184,4 +180,4 @@ JavaScript, CSS, and images for the web UI can be found in
the
## Tests
There are a wide variety of tests for Heron that are scattered throughout the
-codebase. For more info see [Testing Heron](../testing).
+codebase. For more info see [Testing Heron](compiling-running-tests).
diff --git a/website2/docs/compiling-docker.md
b/website2/docs/compiling-docker.md
index e197c58..62830d8 100644
--- a/website2/docs/compiling-docker.md
+++ b/website2/docs/compiling-docker.md
@@ -26,8 +26,8 @@ create the VM), click on the VM, and then click on
**Settings**.
**Note**: You will need to stop the VM before modifying these settings.
-
-
+
+
## Building Heron
diff --git a/website2/docs/compiling-osx.md b/website2/docs/compiling-osx.md
index ba8a556..36742ab 100644
--- a/website2/docs/compiling-osx.md
+++ b/website2/docs/compiling-osx.md
@@ -36,7 +36,7 @@ $ echo $CC $CXX
### Step 4 --- Fetch the latest version of Heron's source code
```bash
-$ git clone https://github.com/apache/incubator-heron.git && cd heron
+$ git clone https://github.com/apache/incubator-heron.git && cd incubator-heron
```
### Step 5 --- Configure Heron for building with Bazel
diff --git a/website2/docs/compiling-overview.md
b/website2/docs/compiling-overview.md
index d6669d4..a463669 100644
--- a/website2/docs/compiling-overview.md
+++ b/website2/docs/compiling-overview.md
@@ -4,20 +4,18 @@ title: Compiling Heron
sidebar_label: Compiling Overview
---
-Heron is currently available for [Mac OS X
10.10](../../../developers/compiling/mac),
-[Ubuntu 14.04](../../../developers/compiling/linux), and [CentOS
-7](../../../developers/compiling/linux). This guide describes the basics of the
+Heron is currently available for [Mac OS X 10.14](compiling-osx),
+[Ubuntu 14.04](compiling-linux), and [CentOS
+7](compiling-linux#building-on-centos-7). This guide describes the basics of
the
Heron build system. For step-by-step build instructions for a specific
platform,
the following guides are available:
-* [Building on Linux Platforms](../../../developers/compiling/linux)
-* [Building on Mac OS X](../../../developers/compiling/mac)
+* [Building on Linux Platforms](compiling-linux)
+* [Building on Mac OS X](compiling-mac)
-Heron can be built either [in its entirety]({{< ref "#building-all-components"
->}}), as [individual components]({{< ref "#building-specific-components" >}}),
-or as a [release package]({{< ref "#building-a-full-release-package" >}}).
+Heron can be built either [in its entirety](#building-all-components), as
[individual components](#building-specific-components).
-Instructions on running unit tests for Heron can also be found in [Testing
Heron](../../../contributors/testing).
+Instructions on running unit tests for Heron can also be found in [Testing
Heron](compiling-running-tests).
## Requirements
@@ -114,8 +112,8 @@ directory. The `heron-tracker` executable, for example, can
be found at
As an alternative to building a full release, you can build Heron executables
for a single Heron component (such as the [Heron
-Tracker](../../../operators/heron-tracker)) by passing a target to the `bazel
-build` command. For example, the following command would build the [Heron
Tracker](../../../operators/heron-tracker):
+Tracker](user-manuals-heron-tracker-runbook)) by passing a target to the `bazel
+build` command. For example, the following command would build the Heron
Tracker:
```bash
$ bazel build --config=darwin heron/tools/tracker/src/python:heron-tracker
@@ -124,4 +122,4 @@ $ bazel build --config=darwin
heron/tools/tracker/src/python:heron-tracker
## Testing Heron
Instructions for running Heron unit tests can be found at [Testing
-Heron](../../../contributors/testing).
+Heron](compiling-running-tests).
diff --git a/website2/docs/compiling-running-tests.md
b/website2/docs/compiling-running-tests.md
index c45faa1..cb4f2dd 100644
--- a/website2/docs/compiling-running-tests.md
+++ b/website2/docs/compiling-running-tests.md
@@ -4,9 +4,9 @@ title: Running Tests
sidebar_label: Running Tests
---
-Heron uses [Bazel](../../developers/compiling#installing-bazel) for building
+Heron uses [Bazel](compiling-overview#installing-bazel) for building
and running unit tests. Before running tests, first set up your build
environment
-as described in [Compiling Heron](../../developers/compiling/compiling).
+as described in [Compiling Heron](compiling-overview).
### Running Unit Tests
diff --git a/website2/docs/extending-heron-metric-sink.md
b/website2/docs/extending-heron-metric-sink.md
index d3fe15e..eaf5e4c 100644
--- a/website2/docs/extending-heron-metric-sink.md
+++ b/website2/docs/extending-heron-metric-sink.md
@@ -5,8 +5,8 @@ sidebar_label: Custom Metrics Sink
---
Each Heron container has its own centralized [Metrics
-Manager](../../concepts/architecture#metrics-manager) (MM), which collects
-metrics from all [Heron Instances](../../concepts/architecture#heron-instance)
in
+Manager](heron-architecture#metrics-manager) (MM), which collects
+metrics from all [Heron Instances](heron-architecture#heron-instance) in
the container. You can define how the MM processes metrics by implementing a
**metrics sink**, which specifies how the MM handles incoming
[`MetricsRecord`](/api/org/apache/heron/spi/metricsmgr/metrics/MetricsRecord.html)
@@ -22,9 +22,9 @@ implementing your own.
Sink | How it works
:----|:------------
-[Prometheus](../../operators/observability/prometheus) |
[`PrometheusSink`](/api/org/apache/heron/metricsmgr/sink/PrometheusSink.html)
sends each `MetricsRecord` object to a specified path in the
[Prometheus](https://prometheus.io) instance.
-[Graphite](../../operators/observability/graphite) |
[`GraphiteSink`](/api/org/apache/heron/metricsmgr/sink/GraphiteSink.html) sends
each `MetricsRecord` object to a [Graphite](http://graphite.wikidot.com/)
instance according to a Graphite prefix.
-[Scribe](../../operators/observability/scribe) |
[`ScribeSink`](/api/org/apache/heron/metricsmgr/sink/ScribeSink.html) sends
each `MetricsRecord` object to a
[Scribe](https://github.com/facebookarchive/scribe) instance according to a
Scribe category and namespace.
+[Prometheus](observability-prometheus) |
[`PrometheusSink`](/api/org/apache/heron/metricsmgr/sink/PrometheusSink.html)
sends each `MetricsRecord` object to a specified path in the
[Prometheus](https://prometheus.io) instance.
+[Graphite](observability-graphite) |
[`GraphiteSink`](/api/org/apache/heron/metricsmgr/sink/GraphiteSink.html) sends
each `MetricsRecord` object to a [Graphite](http://graphite.wikidot.com/)
instance according to a Graphite prefix.
+[Scribe](observability-scribe) |
[`ScribeSink`](/api/org/apache/heron/metricsmgr/sink/ScribeSink.html) sends
each `MetricsRecord` object to a
[Scribe](https://github.com/facebookarchive/scribe) instance according to a
Scribe category and namespace.
Local filesystem |
[`FileSink`](/api/org/apache/heron/metricsmgr/sink/FileSink.html) writes each
`MetricsRecord` object to a JSON file at a specified path.
## Java Setup
@@ -65,7 +65,7 @@ Method | Description
Your implementation of those interfaces will need to be packaged into a JAR
file
and distributed to the `heron-core/lib/metricsmgr` folder of your [Heron
-release](../../developers/compiling/compiling).
+release](compiling-overview).
## Example Implementation
@@ -158,5 +158,5 @@ Once you've made a JAR for your custom Java sink,
distributed that JAR to
`metrics_sinks.yaml` file in the base configuration template, any topology
submitted using that configuration will include the custom sink.
You must [re-compile
-Heron](../../developers/compiling) if you want to include the configuration in
a new distribution of [Heron CLI](../../operators/heron-cli).
+Heron](compiling-overview) if you want to include the configuration in a new
distribution of [Heron CLI](user-manuals-heron-cli).
diff --git a/website2/docs/extending-heron-scheduler.md
b/website2/docs/extending-heron-scheduler.md
index 0b12ffa..881e6d7 100644
--- a/website2/docs/extending-heron-scheduler.md
+++ b/website2/docs/extending-heron-scheduler.md
@@ -8,9 +8,12 @@ To run a Heron topology, you’ll need to set up a scheduler
that is responsible
for topology management. Note: one scheduler is managing only one topology,
for the purpose of better isolation. Heron currently supports the following
schedulers out of the box:
-* [Aurora](../../operators/deployment/schedulers/aurora)
-* [Local scheduler](../../operators/deployment/schedulers/local)
-* [Slurm scheduler](../../operators/deployment/schedulers/slurm)
+* [Aurora](schedulers-aurora-cluster)
+* [Kubernetes](schedulers-k8s-by-hand)
+* [Kubernetes-Helm](schedulers-k8s-with-helm)
+* [Nomad](schedulers-nomad)
+* [Local scheduler](schedulers-local)
+* [Slurm scheduler](schedulers-slurm)
If you'd like to run Heron on a not-yet-supported system, such as
[Amazon ECS](https://aws.amazon.com/ecs/), you can create your own scheduler
@@ -59,10 +62,10 @@ Heron provides a number of built-in implementations out of
box.
## Running the Scheduler
-To run the a custom scheduler, the implementation of the interfaces above must
be specified in the [config](../../operators/deployment/configuration).
+To run the a custom scheduler, the implementation of the interfaces above must
be specified in the [config](deployment-configuration).
By default, the heron-cli looks for configurations under
`${HERON_HOME}/conf/`. The location can be overridden using option
`--config-path`.
Below is an example showing the command for [topology
-submission](../../operators/heron-cli#submitting-a-topology):
+submission](user-manuals-heron-cli#submitting-a-topology):
```bash
$ heron submit [cluster-name-storing-your-new-config]/[role]/[env] \
diff --git a/website2/docs/getting-started-local-single-node.md
b/website2/docs/getting-started-local-single-node.md
index bda9553..bf6e060 100644
--- a/website2/docs/getting-started-local-single-node.md
+++ b/website2/docs/getting-started-local-single-node.md
@@ -12,8 +12,7 @@ The easiest way to get started learning Heron is to install
the Heron client too
* [Ubuntu >= 14.04](#using-installation-scripts)
* [CentOS](#using-installation-scripts)
-For other platforms, you need to build from source. Please refer to the [guide
to compiling Heron]
-(../developers/compiling/compiling).
+For other platforms, you need to build from source. Please refer to the [guide
to compiling Heron](compiling-overview).
## Step 1 --- Download the Heron tools
@@ -55,7 +54,7 @@ $ chmod +x heron-*.sh
> The script will install executables in the `~/bin` folder. You should add
> that folder to your `PATH` using `export PATH=~/bin:$PATH`.
-Now run the [Heron client](../operators/heron-cli) installation script with
the `--user` flag set. Here's an example for Ubuntu:
+Now run the [Heron client](user-manuals-heron-cli) installation script with
the `--user` flag set. Here's an example for Ubuntu:
```bash
$ ./heron-install-{{% heronVersion %}}-ubuntu.sh --user
diff --git a/website2/docs/heron-architecture.md
b/website2/docs/heron-architecture.md
index d514dab..c02e4e0 100644
--- a/website2/docs/heron-architecture.md
+++ b/website2/docs/heron-architecture.md
@@ -108,7 +108,7 @@ components.
#### Topology Master Configuration
TMs have a variety of [configurable
-parameters](../../operators/configuration/tmaster) that you can adjust at each
+parameters](cluster-config-tmaster) that you can adjust at each
phase of a topology's [lifecycle](heron-topology-concepts#topology-lifecycle).
### Containers
diff --git a/website2/docs/heron-resources-resources.md
b/website2/docs/heron-resources-resources.md
index 0c78cb4..f9ac923 100644
--- a/website2/docs/heron-resources-resources.md
+++ b/website2/docs/heron-resources-resources.md
@@ -8,7 +8,7 @@ Heron Resources outside this documentation:
## Books
-* [Stream Processing with
Heron](http://shop.oreilly.com/product/0636920203131.do) (book)
+* [Stream Processing with
Heron](http://shop.oreilly.com/product/0636920203131.do)
## Conference & Journal Papers
diff --git a/website2/docs/observability-graphite.md
b/website2/docs/observability-graphite.md
index c807580..b2239aa 100644
--- a/website2/docs/observability-graphite.md
+++ b/website2/docs/observability-graphite.md
@@ -17,7 +17,7 @@ To accomplish this, you need to do the following:
### Exporting Topology Metrics From Heron
-Heron supports custom metric exporters from the Metrics Manager. You can
either build your own Graphite metrics sink or use the [provided Graphite
sink](/docs/contributors/custom-metrics-sink/).
+Heron supports custom metric exporters from the Metrics Manager. You can
either build your own Graphite metrics sink or use the [provided Graphite
sink](extending-heron-metric-sink).
### Gathering Metrics From Aurora
@@ -80,6 +80,6 @@ PS-*),5), 2,3,6)'
Finally, you can configure the Heron UI to deep link to scripted dashboards by
specifying an
[observability URL format]
-(https://github.com/apache/incubator-heron/blob/master/heron/config/src/yaml/tracker/heron_tracker.yaml)
+(https://github.com/apache/incubator-heron/blob/master/heron/tools/config/src/yaml/tracker/heron_tracker.yaml)
(`viz.url.format`) in the Heron Tracker's configuration. This will add
topology-specific buttons to
the Heron UI enabling you to drill-down into your Grafana dashboards.
diff --git a/website2/docs/observability-prometheus.md
b/website2/docs/observability-prometheus.md
index a405829..5b033bb 100644
--- a/website2/docs/observability-prometheus.md
+++ b/website2/docs/observability-prometheus.md
@@ -8,7 +8,7 @@ You can integrate Heron with
[Prometheus](https://prometheus.io/) to monitor and
## Exporting topology metrics from Heron to Prometheus
-Heron supports custom metric exporters from the Metrics Manager. You can
either build your own Prometheus metrics sink or use the [provided Prometheus
sink](/docs/contributors/custom-metrics-sink/).
+Heron supports custom metric exporters from the Metrics Manager. You can
either build your own Prometheus metrics sink or use the [provided Prometheus
sink](extending-heron-metric-sink).
To set up your Heron cluster to export to Prometheus, you need to make two
changes to the `metrics_sinks.yaml` configuration file:
diff --git a/website2/docs/observability-scribe.md
b/website2/docs/observability-scribe.md
index 338d8c7..dda63de 100644
--- a/website2/docs/observability-scribe.md
+++ b/website2/docs/observability-scribe.md
@@ -8,7 +8,7 @@ You can integrate Heron with
[Scribe](https://github.com/facebookarchive/scribe/
## Exporting topology metrics from Heron to Scribe
-Heron supports custom metric exporters from the Metrics Manager. You can
either build your own Scribe metrics sink or use the [provided Scribe
sink](/docs/contributors/custom-metrics-sink/).
+Heron supports custom metric exporters from the Metrics Manager. You can
either build your own Scribe metrics sink or use the [provided Scribe
sink](extending-heron-metric-sink).
To set up your Heron cluster to export to Scribe, you need to make two changes
to the `metrics_sinks.yaml` configuration file:
diff --git a/website2/docs/user-manuals-heron-cli.md
b/website2/docs/user-manuals-heron-cli.md
index eb2a139..bc43c86 100644
--- a/website2/docs/user-manuals-heron-cli.md
+++ b/website2/docs/user-manuals-heron-cli.md
@@ -5,7 +5,7 @@ sidebar_label: Heron Client
---
The **Heron CLI** us used to to manage every aspect of the
-[topology lifecycle](../../concepts/topologies#topology-lifecycle).
+[topology lifecycle](heron-topology-concepts#topology-lifecycle).
## Deploying the `heron` CLI Executable
@@ -31,7 +31,7 @@ See http://heronstreaming.io/docs/getting-started.html on how
to use Heron!
....
```
-Alternatively, generate a full [Heron
release](../../developers/compiling/compiling) and
+Alternatively, generate a full [Heron release](compiling-overview) and
distribute the resulting `heron` CLI to all machines used to manage topologies.
### Common CLI Args
diff --git a/website2/docs/user-manuals-heron-explorer.md
b/website2/docs/user-manuals-heron-explorer.md
index e049f77..8f85add 100644
--- a/website2/docs/user-manuals-heron-explorer.md
+++ b/website2/docs/user-manuals-heron-explorer.md
@@ -13,9 +13,9 @@ The **Heron Explorer** is a CLI tool that you can use to gain
insight into a Her
* the topologies running in a given cluster, role, or environment
> #### The Heron Explorer vs. Heron CLI
-> There are two important differences between the Heron Explorer and [Heron
CLI](../heron-cli). Unlike Heron CLI, the Heron Explorer (a) requires the
[Heron Tracker](../heron-tracker) and (b) performs read-only,
observation-oriented commands (rather than commands for actions like
submitting, activating, and killing topologies).
+> There are two important differences between the Heron Explorer and [Heron
CLI](user-manuals-heron-cli). Unlike Heron CLI, the Heron Explorer (a) requires
the [Heron Tracker](user-manuals-heron-tracker-runbook) and (b) performs
read-only, observation-oriented commands (rather than commands for actions like
submitting, activating, and killing topologies).
-In order to use the Heron Explorer, the [Heron Tracker](../heron-tracker) will
need to be running. If you've [installed the Tracker](../../getting-started),
you can start it up using just one command:
+In order to use the Heron Explorer, the [Heron
Tracker](user-manuals-heron-tracker-runbook) will need to be running. If you've
[installed the Tracker](getting-started-local-single-node), you can start it up
using just one command:
```shell
$ heron-tracker
@@ -23,8 +23,8 @@ $ heron-tracker
## Installation
-The Heron Explorer is installed automatically if you follow the Heron tools
installation tutorial in the [Quick Start
Guide](../../getting-started#step-1-download-the-heron-tools).
-
+The Heron Explorer is installed automatically if you follow the Heron tools
installation tutorial in the [Quick Start
Guide](getting-started-local-single-node#step-1-download-the-heron-tools).
+x
## Commands
The commands available for the Heron Explorer are listed in the table below.
diff --git a/website2/docs/user-manuals-heron-shell.md
b/website2/docs/user-manuals-heron-shell.md
new file mode 100644
index 0000000..270364a
--- /dev/null
+++ b/website2/docs/user-manuals-heron-shell.md
@@ -0,0 +1,56 @@
+---
+id: user-manuals-heron-shell
+title: Heron Shell
+sidebar_label: Heron Shell
+---
+
+Heron shell helps debugging a heron topology. It is an HTTP server that runs
as a
+separate process in every container. It exposes many utilities through REST
APIs.
+These utilities are described below in more details.
+
+The port to connect to heron shell for each container is stored in the physical
+plan. Heron tracker picks up this port and connects to shell. See the next
+section for more details.
+
+## Shell Utilities
+
+### Log files
+
+This is probably the most useful utility of shell. Since relative paths to log
+files are known, these paths are directly embedded in UI through Tracker. This
+makes the logs available in the browser. The log files start at the end and
+users can scroll up a page at a time, or download them locally.
+
+### Browse
+
+A container is expected to run in a sandbox directory. All the files
+under this directory are accessible through shell. This can be used to download
+the jars or tars for a topology, or browse through, download, or even view the
+files online. Viewing and downloading log files is one special case of browse.
+
+### Pid of a process
+
+Each instance in the topology runs as a separate JVM process. This allows us to
+monitor each spout or bolt instance in isolation. To run more sophisticated
+operations on the process as mentioned in this list below, we need to know the
+process id of the process running as that instance.
+
+### Jstack of a process
+
+This utility runs the `jstack` command on the JVM process that is running an
+instance. The result is passed back through the REST API which can be viewed
+directly in the browser.
+
+### Jmap of a process
+
+This utility runs `jmap` on the process running one instance to capture a heap
+dump. Since a heap dump can be huge, the dump file is created on the host that
+is running the container. This file can be downloaded using the info that is
+returned as the response to this endpoint. The dumps are also accessible
through
+the "browse" utility.
+
+### Memory histogram of a process
+
+This utility runs `jmap` with `-histo` option, to output the memory
+histogram for the JVM process of an instance. The resulting histogram is passed
+back as part of the response, and can be viewed directly in the browser.
diff --git a/website2/docs/user-manuals-heron-tracker-runbook.md
b/website2/docs/user-manuals-heron-tracker-runbook.md
index 528a0a3..8b29c87 100644
--- a/website2/docs/user-manuals-heron-tracker-runbook.md
+++ b/website2/docs/user-manuals-heron-tracker-runbook.md
@@ -6,14 +6,14 @@ sidebar_label: Heron Tracker Runbook
**Heron Tracker** is a web service that continuously gathers a wide
variety of information about Heron topologies and exposes
-that information through a [JSON REST API](../heron-tracker-api).
+that information through a [JSON REST API](user-manuals-tracker-rest).
More on the role of the Tracker can be found
-[here](../../concepts/architecture#heron-tracker).
+[here](heron-architecture#heron-tracker).
## Building Heron Tracker
Heron uses [bazel](http://bazel.io/) for compiling.
-[Compiling](../../developers/compiling/compiling) describes how to setup bazel
+[Compiling](compiling-overview) describes how to setup bazel
for heron.
```bash
@@ -36,8 +36,8 @@ The config file is a `yaml` file that should contain the
following information.
#### 1. State Manager locations
This is a list of locations where topology writes its states. An example of
-[zookeeper state manager](../deployment/statemanagers/zookeeper) and
-[local file state manager](../deployment/statemanagers/localfs) look like this:
+[zookeeper state manager](state-managers-zookeeper) and
+[local file state manager](state-managers-local-fs) look like this:
```yaml
## Contains the sources where the states are stored.
diff --git a/website2/docs/user-manuals-heron-ui.md
b/website2/docs/user-manuals-heron-ui.md
index ea6dc17..09e9a89 100644
--- a/website2/docs/user-manuals-heron-ui.md
+++ b/website2/docs/user-manuals-heron-ui.md
@@ -4,18 +4,17 @@ title: Heron UI Runbook
sidebar_label: Heron UI Runbook
---
-**Heron UI** is a user interface that uses the [Heron
-Tracker](../../concepts/architecture#heron-tracker) to display detailed,
+**Heron UI** is a user interface that uses the [Heron
Tracker](heron-architecture#heron-tracker) to display detailed,
colorful visual representations of topologies, including the
-[logical](../../concepts/topologies/#logical-plan) and [physical
-plan](../../concepts/topologies#physical-plan) for each topology. Check out
-[Heron UI Usage Guide](../../developers/ui-guide) for more information about
+[logical](heron-topology-concepts#logical-plan) and [physical
+plan](heron-topology-concepts#physical-plan) for each topology. Check out
+[Heron UI Usage Guide](guides-ui-guide) for more information about
various elements that UI exposes.
### Building Heron UI
Heron uses [bazel](http://bazel.io/) for compiling.
-[This page](../../developers/compiling/compiling) describes how to setup bazel
+[This page](compiling-overview) describes how to setup bazel
for heron.
```bash
diff --git a/website2/docs/user-manuals-tracker-rest.md
b/website2/docs/user-manuals-tracker-rest.md
index fbc5786..f69975a 100644
--- a/website2/docs/user-manuals-tracker-rest.md
+++ b/website2/docs/user-manuals-tracker-rest.md
@@ -70,7 +70,7 @@ $ curl
"http://heron-tracker-url/topologies?cluster=cluster1&environ=devel";
### <a name="topologies_logicalplan">/topologies/logicalplan</a>
-Returns a JSON representation of the [logical
plan](../../concepts/topologies#logical-plan) of a topology.
+Returns a JSON representation of the [logical
plan](heron-topology-concepts#logical-plan) of a topology.
```bash
$ curl
"http://heron-tracker-url/topologies/logicalplan?cluster=cluster1&environ=devel&topology=topologyName";
@@ -101,7 +101,7 @@ The resulting JSON contains the following
### <a name="topologies_physicalplan">/topologies/physicalplan</a>
-Returns a JSON representation of the [physical
plan](../../concepts/topologies#physical-plan) of a topology.
+Returns a JSON representation of the [physical
plan](heron-topology-concepts#physical-plan) of a topology.
```bash
$ curl
"http://heron-tracker-url/topologies/physicalplan?cluster=datacenter1&environ=prod&topology=topologyName";
@@ -184,7 +184,7 @@ Each execution state object lists the following:
* `has_physical_plan` --- Whether the topology has a physical plan
* `has_tmaster_location` --- Whether the topology has a Topology Master
Location
* `has_scheduler_location` --- Whether the topology has a Scheduler Location
-* `viz` --- Metric visualization UI URL for the topology if it was
[configured](../heron-tracker)
+* `viz` --- Metric visualization UI URL for the topology if it was
[configured](user-manuals-heron-tracker-runbook)
---
diff --git a/website2/website/sidebars.json b/website2/website/sidebars.json
index 848ce10..c95b1e8 100755
--- a/website2/website/sidebars.json
+++ b/website2/website/sidebars.json
@@ -79,7 +79,8 @@
"user-manuals-heron-explorer",
"user-manuals-tracker-rest",
"user-manuals-heron-tracker-runbook",
- "user-manuals-heron-ui-runbook"
+ "user-manuals-heron-ui-runbook",
+ "user-manuals-heron-shell"
],
"Compiling": [
"compiling-overview",
diff --git a/website2/website/static/js/custom.js
b/website2/website/static/js/custom.js
index a89d27f..cd61b65 100644
--- a/website2/website/static/js/custom.js
+++ b/website2/website/static/js/custom.js
@@ -7,10 +7,10 @@ window.addEventListener('load', function() {
'<a id="community-menu" href="#">Community<span style="font-size:
0.75em"> ▼</span></a>' +
'<div id="community-dropdown" class="hide" >' +
'<ul id="community-dropdown-items">' +
- '<li><a href="/contact" style="color:#1d3f5f">Contact</a></li>' +
- '<li><a href="/events" style="color:#1d3f5f">Events</a></li>' +
+ '<li><a href="community-contact"
style="color:#1d3f5f">Contact</a></li>' +
+ '<li><a href="community-events" style="color:#1d3f5f">Events</a></li>'
+
'<li><a href="https://github.com/apache/incubator-heron/issues";
style="color:#1d3f5f">Issue tracking</a></li>' +
- '<li><a href="/resources" style="color:#1d3f5f">Resources</a></li>' +
+ '<li><a href="heron-resources-resources"
style="color:#1d3f5f">Resources</a></li>' +
'<li><a href="/team" style="color:#1d3f5f">Team</a></li>' +
'</ul>' +
'</div>' +
