Author: bobby
Date: Fri Mar 18 01:49:28 2016
New Revision: 1735530
URL: http://svn.apache.org/viewvc?rev=1735530&view=rev
Log:
Added in plugin to generate release version specific page parameters
Added:
storm/branches/bobby-versioned-site/_plugins/
storm/branches/bobby-versioned-site/_plugins/releases.rb
Modified:
storm/branches/bobby-versioned-site/releases/0.10.0/Clojure-DSL.md
storm/branches/bobby-versioned-site/releases/0.10.0/Command-line-client.md
storm/branches/bobby-versioned-site/releases/0.10.0/Common-patterns.md
storm/branches/bobby-versioned-site/releases/0.10.0/Concepts.md
storm/branches/bobby-versioned-site/releases/0.10.0/Configuration.md
storm/branches/bobby-versioned-site/releases/0.10.0/Contributing-to-Storm.md
storm/branches/bobby-versioned-site/releases/0.10.0/Creating-a-new-Storm-project.md
storm/branches/bobby-versioned-site/releases/0.10.0/DSLs-and-multilang-adapters.md
storm/branches/bobby-versioned-site/releases/0.10.0/Daemon-Fault-Tolerance.md
storm/branches/bobby-versioned-site/releases/0.10.0/Defining-a-non-jvm-language-dsl-for-storm.md
storm/branches/bobby-versioned-site/releases/0.10.0/Distributed-RPC.md
storm/branches/bobby-versioned-site/releases/0.10.0/FAQ.md
storm/branches/bobby-versioned-site/releases/0.10.0/Guaranteeing-message-processing.md
storm/branches/bobby-versioned-site/releases/0.10.0/Hooks.md
storm/branches/bobby-versioned-site/releases/0.10.0/Implementation-docs.md
storm/branches/bobby-versioned-site/releases/0.10.0/Installing-native-dependencies.md
storm/branches/bobby-versioned-site/releases/0.10.0/Kestrel-and-Storm.md
storm/branches/bobby-versioned-site/releases/0.10.0/Lifecycle-of-a-topology.md
storm/branches/bobby-versioned-site/releases/0.10.0/Local-mode.md
storm/branches/bobby-versioned-site/releases/0.10.0/Maven.md
storm/branches/bobby-versioned-site/releases/0.10.0/Message-passing-implementation.md
storm/branches/bobby-versioned-site/releases/0.10.0/Metrics.md
storm/branches/bobby-versioned-site/releases/0.10.0/Multilang-protocol.md
storm/branches/bobby-versioned-site/releases/0.10.0/Rationale.md
storm/branches/bobby-versioned-site/releases/0.10.0/Running-topologies-on-a-production-cluster.md
storm/branches/bobby-versioned-site/releases/0.10.0/SECURITY.md
storm/branches/bobby-versioned-site/releases/0.10.0/STORM-UI-REST-API.md
storm/branches/bobby-versioned-site/releases/0.10.0/Serialization.md
storm/branches/bobby-versioned-site/releases/0.10.0/Serializers.md
storm/branches/bobby-versioned-site/releases/0.10.0/Setting-up-a-Storm-cluster.md
storm/branches/bobby-versioned-site/releases/0.10.0/Setting-up-development-environment.md
storm/branches/bobby-versioned-site/releases/0.10.0/Spout-implementations.md
storm/branches/bobby-versioned-site/releases/0.10.0/Structure-of-the-codebase.md
storm/branches/bobby-versioned-site/releases/0.10.0/Support-for-non-java-languages.md
storm/branches/bobby-versioned-site/releases/0.10.0/Transactional-topologies.md
storm/branches/bobby-versioned-site/releases/0.10.0/Trident-API-Overview.md
storm/branches/bobby-versioned-site/releases/0.10.0/Trident-spouts.md
storm/branches/bobby-versioned-site/releases/0.10.0/Trident-state.md
storm/branches/bobby-versioned-site/releases/0.10.0/Trident-tutorial.md
storm/branches/bobby-versioned-site/releases/0.10.0/Troubleshooting.md
storm/branches/bobby-versioned-site/releases/0.10.0/Tutorial.md
storm/branches/bobby-versioned-site/releases/0.10.0/Understanding-the-parallelism-of-a-Storm-topology.md
storm/branches/bobby-versioned-site/releases/0.10.0/Using-non-JVM-languages-with-Storm.md
storm/branches/bobby-versioned-site/releases/0.10.0/flux.md
storm/branches/bobby-versioned-site/releases/0.10.0/index.md
storm/branches/bobby-versioned-site/releases/0.10.0/storm-eventhubs.md
storm/branches/bobby-versioned-site/releases/0.10.0/storm-hbase.md
storm/branches/bobby-versioned-site/releases/0.10.0/storm-hdfs.md
storm/branches/bobby-versioned-site/releases/0.10.0/storm-hive.md
storm/branches/bobby-versioned-site/releases/0.10.0/storm-jdbc.md
storm/branches/bobby-versioned-site/releases/0.10.0/storm-kafka.md
storm/branches/bobby-versioned-site/releases/0.10.0/storm-redis.md
storm/branches/bobby-versioned-site/releases/0.9.6/Acking-framework-implementation.md
storm/branches/bobby-versioned-site/releases/0.9.6/Clojure-DSL.md
storm/branches/bobby-versioned-site/releases/0.9.6/Command-line-client.md
storm/branches/bobby-versioned-site/releases/0.9.6/Common-patterns.md
storm/branches/bobby-versioned-site/releases/0.9.6/Concepts.md
storm/branches/bobby-versioned-site/releases/0.9.6/Configuration.md
storm/branches/bobby-versioned-site/releases/0.9.6/Contributing-to-Storm.md
storm/branches/bobby-versioned-site/releases/0.9.6/Creating-a-new-Storm-project.md
storm/branches/bobby-versioned-site/releases/0.9.6/DSLs-and-multilang-adapters.md
storm/branches/bobby-versioned-site/releases/0.9.6/Defining-a-non-jvm-language-dsl-for-storm.md
storm/branches/bobby-versioned-site/releases/0.9.6/Distributed-RPC.md
storm/branches/bobby-versioned-site/releases/0.9.6/Documentation.md
storm/branches/bobby-versioned-site/releases/0.9.6/FAQ.md
storm/branches/bobby-versioned-site/releases/0.9.6/Fault-tolerance.md
storm/branches/bobby-versioned-site/releases/0.9.6/Guaranteeing-message-processing.md
storm/branches/bobby-versioned-site/releases/0.9.6/Hooks.md
storm/branches/bobby-versioned-site/releases/0.9.6/Implementation-docs.md
storm/branches/bobby-versioned-site/releases/0.9.6/Installing-native-dependencies.md
storm/branches/bobby-versioned-site/releases/0.9.6/Kestrel-and-Storm.md
storm/branches/bobby-versioned-site/releases/0.9.6/Lifecycle-of-a-topology.md
storm/branches/bobby-versioned-site/releases/0.9.6/Local-mode.md
storm/branches/bobby-versioned-site/releases/0.9.6/Maven.md
storm/branches/bobby-versioned-site/releases/0.9.6/Message-passing-implementation.md
storm/branches/bobby-versioned-site/releases/0.9.6/Metrics.md
storm/branches/bobby-versioned-site/releases/0.9.6/Multilang-protocol.md
storm/branches/bobby-versioned-site/releases/0.9.6/Powered-By.md
storm/branches/bobby-versioned-site/releases/0.9.6/Project-ideas.md
storm/branches/bobby-versioned-site/releases/0.9.6/Rationale.md
storm/branches/bobby-versioned-site/releases/0.9.6/Running-topologies-on-a-production-cluster.md
storm/branches/bobby-versioned-site/releases/0.9.6/SECURITY.md
storm/branches/bobby-versioned-site/releases/0.9.6/STORM-UI-REST-API.md
storm/branches/bobby-versioned-site/releases/0.9.6/Serialization-(prior-to-0.6.0).md
storm/branches/bobby-versioned-site/releases/0.9.6/Serialization.md
storm/branches/bobby-versioned-site/releases/0.9.6/Serializers.md
storm/branches/bobby-versioned-site/releases/0.9.6/Setting-up-a-Storm-cluster.md
storm/branches/bobby-versioned-site/releases/0.9.6/Setting-up-development-environment.md
storm/branches/bobby-versioned-site/releases/0.9.6/Spout-implementations.md
storm/branches/bobby-versioned-site/releases/0.9.6/Storm-multi-language-protocol-(versions-0.7.0-and-below).md
storm/branches/bobby-versioned-site/releases/0.9.6/Structure-of-the-codebase.md
storm/branches/bobby-versioned-site/releases/0.9.6/Support-for-non-java-languages.md
storm/branches/bobby-versioned-site/releases/0.9.6/Transactional-topologies.md
storm/branches/bobby-versioned-site/releases/0.9.6/Trident-API-Overview.md
storm/branches/bobby-versioned-site/releases/0.9.6/Trident-spouts.md
storm/branches/bobby-versioned-site/releases/0.9.6/Trident-state.md
storm/branches/bobby-versioned-site/releases/0.9.6/Trident-tutorial.md
storm/branches/bobby-versioned-site/releases/0.9.6/Troubleshooting.md
storm/branches/bobby-versioned-site/releases/0.9.6/Tutorial.md
storm/branches/bobby-versioned-site/releases/0.9.6/Understanding-the-parallelism-of-a-Storm-topology.md
storm/branches/bobby-versioned-site/releases/0.9.6/Using-non-JVM-languages-with-Storm.md
storm/branches/bobby-versioned-site/releases/0.9.6/index.md
Added: storm/branches/bobby-versioned-site/_plugins/releases.rb
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/_plugins/releases.rb?rev=1735530&view=auto
==============================================================================
--- storm/branches/bobby-versioned-site/_plugins/releases.rb (added)
+++ storm/branches/bobby-versioned-site/_plugins/releases.rb Fri Mar 18
01:49:28 2016
@@ -0,0 +1,22 @@
+module Releases
+ class Generator < Jekyll::Generator
+ def generate(site)
+ for page in site.pages do
+ splitdir = page.dir.split("/").select{ |a| a != ""};
+ if (splitdir[0] == 'releases')
+ release_name = splitdir[1];
+ if (release_name == 'current')
+ release_name = File.readlink(splitdir.join("/")).split("/")[-1];
+ end
+ git_tree_base =
"http://github.com/apache/storm/tree/v#{release_name}/";
+ git_blob_base =
"http://github.com/apache/storm/blob/v#{release_name}/";
+
+ page.data['version'] = release_name;
+ page.data['git-tree-base'] = git_tree_base;
+ page.data['git-blob-base'] = git_blob_base;
+ #puts "#{page.name} #{page.data}";
+ end
+ end
+ end
+ end
+end
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Clojure-DSL.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Clojure-DSL.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Clojure-DSL.md
(original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Clojure-DSL.md Fri Mar
18 01:49:28 2016
@@ -2,9 +2,8 @@
title: Clojure DSL
layout: documentation
documentation: true
-version: v0.10.0
---
-Storm comes with a Clojure DSL for defining spouts, bolts, and topologies. The
Clojure DSL has access to everything the Java API exposes, so if you're a
Clojure user you can code Storm topologies without touching Java at all. The
Clojure DSL is defined in the source in the
[backtype.storm.clojure](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/clojure.clj)
namespace.
+Storm comes with a Clojure DSL for defining spouts, bolts, and topologies. The
Clojure DSL has access to everything the Java API exposes, so if you're a
Clojure user you can code Storm topologies without touching Java at all. The
Clojure DSL is defined in the source in the
[backtype.storm.clojure]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/clojure.clj)
namespace.
This page outlines all the pieces of the Clojure DSL, including:
@@ -18,7 +17,7 @@ This page outlines all the pieces of the
To define a topology, use the `topology` function. `topology` takes in two
arguments: a map of "spout specs" and a map of "bolt specs". Each spout and
bolt spec wires the code for the component into the topology by specifying
things like inputs and parallelism.
-Let's take a look at an example topology definition [from the storm-starter
project](https://github.com/apache/storm/blob/{{page.version}}/examples/storm-starter/src/clj/storm/starter/clj/word_count.clj):
+Let's take a look at an example topology definition [from the storm-starter
project]({{page.git-blob-base}}/examples/storm-starter/src/clj/storm/starter/clj/word_count.clj):
```clojure
(topology
@@ -204,7 +203,7 @@ The signature for `defspout` looks like
If you leave out the option map, it defaults to {:prepare true}. The output
declaration for `defspout` has the same syntax as `defbolt`.
-Here's an example `defspout` implementation from
[storm-starter](https://github.com/apache/storm/blob/{{page.version}}/examples/storm-starter/src/clj/storm/starter/clj/word_count.clj):
+Here's an example `defspout` implementation from
[storm-starter]({{page.git-blob-base}}/examples/storm-starter/src/clj/storm/starter/clj/word_count.clj):
```clojure
(defspout sentence-spout ["sentence"]
Modified:
storm/branches/bobby-versioned-site/releases/0.10.0/Command-line-client.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Command-line-client.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Command-line-client.md
(original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Command-line-client.md
Fri Mar 18 01:49:28 2016
@@ -2,7 +2,6 @@
title: Command Line Client
layout: documentation
documentation: true
-version: v0.10.0
---
This page describes all the commands that are possible with the "storm"
command line client. To learn how to set up your "storm" client to talk to a
remote cluster, follow the instructions in [Setting up development
environment](Setting-up-development-environment.html).
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Common-patterns.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Common-patterns.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Common-patterns.md
(original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Common-patterns.md Fri
Mar 18 01:49:28 2016
@@ -2,7 +2,6 @@
title: Common Topology Patterns
layout: documentation
documentation: true
-version: v0.10.0
---
This page lists a variety of common patterns in Storm topologies.
@@ -71,7 +70,7 @@ builder.setBolt("merge", new MergeObject
.globalGrouping("rank");
```
-This pattern works because of the fields grouping done by the first bolt which
gives the partitioning you need for this to be semantically correct. You can
see an example of this pattern in storm-starter
[here](https://github.com/apache/storm/blob/{{page.version}}/examples/storm-starter/src/jvm/storm/starter/RollingTopWords.java).
+This pattern works because of the fields grouping done by the first bolt which
gives the partitioning you need for this to be semantically correct. You can
see an example of this pattern in storm-starter
[here]({{page.git-blob-base}}/examples/storm-starter/src/jvm/storm/starter/RollingTopWords.java).
If however you have a known skew in the data being processed it can be
advantageous to use partialKeyGrouping instead of fieldsGrouping. This will
distribute the load for each key between two downstream bolts instead of a
single one.
@@ -84,7 +83,7 @@ builder.setBolt("merge", new MergeRanksO
.globalGrouping("rank");
```
-The topology needs an extra layer of processing to aggregate the partial
counts from the upstream bolts but this only processes aggregated values now so
the bolt it is not subject to the load caused by the skewed data. You can see
an example of this pattern in storm-starter
[here](https://github.com/apache/storm/blob/{{page.version}}/examples/storm-starter/src/jvm/storm/starter/SkewedRollingTopWords.java).
+The topology needs an extra layer of processing to aggregate the partial
counts from the upstream bolts but this only processes aggregated values now so
the bolt it is not subject to the load caused by the skewed data. You can see
an example of this pattern in storm-starter
[here]({{page.git-blob-base}}/examples/storm-starter/src/jvm/storm/starter/SkewedRollingTopWords.java).
### TimeCacheMap for efficiently keeping a cache of things that have been
recently updated
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Concepts.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Concepts.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Concepts.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Concepts.md Fri Mar 18
01:49:28 2016
@@ -2,7 +2,6 @@
title: Concepts
layout: documentation
documentation: true
-version: v0.10.0
---
This page lists the main concepts of Storm and links to resources where you
can find more information. The concepts discussed are:
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Configuration.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Configuration.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Configuration.md
(original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Configuration.md Fri
Mar 18 01:49:28 2016
@@ -2,11 +2,10 @@
title: Configuration
layout: documentation
documentation: true
-version: v0.10.0
---
Storm has a variety of configurations for tweaking the behavior of nimbus,
supervisors, and running topologies. Some configurations are system
configurations and cannot be modified on a topology by topology basis, whereas
other configurations can be modified per topology.
-Every configuration has a default value defined in
[defaults.yaml](https://github.com/apache/storm/blob/{{page.version}}/conf/defaults.yaml)
in the Storm codebase. You can override these configurations by defining a
storm.yaml in the classpath of Nimbus and the supervisors. Finally, you can
define a topology-specific configuration that you submit along with your
topology when using
[StormSubmitter](javadocs/backtype/storm/StormSubmitter.html). However, the
topology-specific configuration can only override configs prefixed with
"TOPOLOGY".
+Every configuration has a default value defined in
[defaults.yaml]({{page.git-blob-base}}/conf/defaults.yaml) in the Storm
codebase. You can override these configurations by defining a storm.yaml in the
classpath of Nimbus and the supervisors. Finally, you can define a
topology-specific configuration that you submit along with your topology when
using [StormSubmitter](javadocs/backtype/storm/StormSubmitter.html). However,
the topology-specific configuration can only override configs prefixed with
"TOPOLOGY".
Storm 0.7.0 and onwards lets you override configuration on a
per-bolt/per-spout basis. The only configurations that can be overriden this
way are:
@@ -26,7 +25,7 @@ The preference order for configuration v
**Resources:**
* [Config](javadocs/backtype/storm/Config.html): a listing of all
configurations as well as a helper class for creating topology specific
configurations
-*
[defaults.yaml](https://github.com/apache/storm/blob/{{page.version}}/conf/defaults.yaml):
the default values for all configurations
+* [defaults.yaml]({{page.git-blob-base}}/conf/defaults.yaml): the default
values for all configurations
* [Setting up a Storm cluster](Setting-up-a-Storm-cluster.html): explains how
to create and configure a Storm cluster
* [Running topologies on a production
cluster](Running-topologies-on-a-production-cluster.html): lists useful
configurations when running topologies on a cluster
* [Local mode](Local-mode.html): lists useful configurations when using local
mode
Modified:
storm/branches/bobby-versioned-site/releases/0.10.0/Contributing-to-Storm.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Contributing-to-Storm.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
---
storm/branches/bobby-versioned-site/releases/0.10.0/Contributing-to-Storm.md
(original)
+++
storm/branches/bobby-versioned-site/releases/0.10.0/Contributing-to-Storm.md
Fri Mar 18 01:49:28 2016
@@ -2,7 +2,6 @@
title: Contributing
layout: documentation
documentation: true
-version: v0.10.0
---
### Getting started with contributing
Modified:
storm/branches/bobby-versioned-site/releases/0.10.0/Creating-a-new-Storm-project.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Creating-a-new-Storm-project.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
---
storm/branches/bobby-versioned-site/releases/0.10.0/Creating-a-new-Storm-project.md
(original)
+++
storm/branches/bobby-versioned-site/releases/0.10.0/Creating-a-new-Storm-project.md
Fri Mar 18 01:49:28 2016
@@ -2,18 +2,17 @@
title: Creating a New Storm Project
layout: documentation
documentation: true
-version: v0.10.0
---
This page outlines how to set up a Storm project for development. The steps
are:
1. Add Storm jars to classpath
2. If using multilang, add multilang dir to classpath
-Follow along to see how to set up the
[storm-starter](https://github.com/apache/storm/blob/{{page.version}}/examples/storm-starter)
project in Eclipse.
+Follow along to see how to set up the
[storm-starter]({{page.git-blob-base}}/examples/storm-starter) project in
Eclipse.
### Add Storm jars to classpath
-You'll need the Storm jars on your classpath to develop Storm topologies.
Using [Maven](Maven.html) is highly recommended. [Here's an
example](https://github.com/apache/storm/blob/{{page.version}}/examples/storm-starter/pom.xml)
of how to setup your pom.xml for a Storm project. If you don't want to use
Maven, you can include the jars from the Storm release on your classpath.
+You'll need the Storm jars on your classpath to develop Storm topologies.
Using [Maven](Maven.html) is highly recommended. [Here's an
example]({{page.git-blob-base}}/examples/storm-starter/pom.xml) of how to setup
your pom.xml for a Storm project. If you don't want to use Maven, you can
include the jars from the Storm release on your classpath.
To set up the classpath in Eclipse, create a new Java project, include
`src/jvm/` as a source path, and make sure all the jars in `lib/` and
`lib/dev/` are in the `Referenced Libraries` section of the project.
Modified:
storm/branches/bobby-versioned-site/releases/0.10.0/DSLs-and-multilang-adapters.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/DSLs-and-multilang-adapters.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
---
storm/branches/bobby-versioned-site/releases/0.10.0/DSLs-and-multilang-adapters.md
(original)
+++
storm/branches/bobby-versioned-site/releases/0.10.0/DSLs-and-multilang-adapters.md
Fri Mar 18 01:49:28 2016
@@ -2,7 +2,6 @@
title: Storm DSLs and Multi-Lang Adapters
layout: documentation
documentation: true
-version: v0.10.0
---
* [Scala DSL](https://github.com/velvia/ScalaStorm)
* [JRuby DSL](https://github.com/colinsurprenant/redstorm)
Modified:
storm/branches/bobby-versioned-site/releases/0.10.0/Daemon-Fault-Tolerance.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Daemon-Fault-Tolerance.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
---
storm/branches/bobby-versioned-site/releases/0.10.0/Daemon-Fault-Tolerance.md
(original)
+++
storm/branches/bobby-versioned-site/releases/0.10.0/Daemon-Fault-Tolerance.md
Fri Mar 18 01:49:28 2016
@@ -2,7 +2,6 @@
title: Daemon Fault Tolerance
layout: documentation
documentation: true
-version: v0.10.0
---
Storm has several different daemon processes. Nimbus that schedules workers,
supervisors that launch and kill workers, the log viewer that gives access to
logs, and the UI that shows the status of a cluster.
Modified:
storm/branches/bobby-versioned-site/releases/0.10.0/Defining-a-non-jvm-language-dsl-for-storm.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Defining-a-non-jvm-language-dsl-for-storm.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
---
storm/branches/bobby-versioned-site/releases/0.10.0/Defining-a-non-jvm-language-dsl-for-storm.md
(original)
+++
storm/branches/bobby-versioned-site/releases/0.10.0/Defining-a-non-jvm-language-dsl-for-storm.md
Fri Mar 18 01:49:28 2016
@@ -2,9 +2,8 @@
title: Defining a Non-JVM DSL for Storm
layout: documentation
documentation: true
-version: v0.10.0
---
-The right place to start to learn how to make a non-JVM DSL for Storm is
[storm-core/src/storm.thrift](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/storm.thrift).
Since Storm topologies are just Thrift structures, and Nimbus is a Thrift
daemon, you can create and submit topologies in any language.
+The right place to start to learn how to make a non-JVM DSL for Storm is
[storm-core/src/storm.thrift]({{page.git-blob-base}}/storm-core/src/storm.thrift).
Since Storm topologies are just Thrift structures, and Nimbus is a Thrift
daemon, you can create and submit topologies in any language.
When you create the Thrift structs for spouts and bolts, the code for the
spout or bolt is specified in the ComponentObject struct:
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Distributed-RPC.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Distributed-RPC.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Distributed-RPC.md
(original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Distributed-RPC.md Fri
Mar 18 01:49:28 2016
@@ -2,7 +2,6 @@
title: Distributed RPC
layout: documentation
documentation: true
-version: v0.10.0
---
The idea behind distributed RPC (DRPC) is to parallelize the computation of
really intense functions on the fly using Storm. The Storm topology takes in as
input a stream of function arguments, and it emits an output stream of the
results for each of those function calls.
@@ -119,7 +118,7 @@ The reach of a URL is the number of uniq
A single reach computation can involve thousands of database calls and tens of
millions of follower records during the computation. It's a really, really
intense computation. As you're about to see, implementing this function on top
of Storm is dead simple. On a single machine, reach can take minutes to
compute; on a Storm cluster, you can compute reach for even the hardest URLs in
a couple seconds.
-A sample reach topology is defined in storm-starter
[here](https://github.com/apache/storm/blob/{{page.version}}/examples/storm-starter/src/jvm/storm/starter/ReachTopology.java).
Here's how you define the reach topology:
+A sample reach topology is defined in storm-starter
[here]({{page.git-blob-base}}/examples/storm-starter/src/jvm/storm/starter/ReachTopology.java).
Here's how you define the reach topology:
```java
LinearDRPCTopologyBuilder builder = new LinearDRPCTopologyBuilder("reach");
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/FAQ.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/FAQ.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/FAQ.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/FAQ.md Fri Mar 18
01:49:28 2016
@@ -2,7 +2,6 @@
title: FAQ
layout: documentation
documentation: true
-version: v0.10.0
---
## Best Practices
@@ -79,11 +78,11 @@ You should only store static data, and a
### How often is the 'emitPartitionBatchNew' function called?
-Since the MBC is the actual spout, all the tuples in a batch are just members
of its tupletree. That means storm's "max spout pending" config effectively
defines the number of concurrent batches trident runs. The MBC emits a new
batch if it has fewer than max-spending tuples pending and if at least one
[trident batch
interval](https://github.com/apache/storm/blob/{{page.version}}/conf/defaults.yaml#L115)'s
worth of seconds has passed since the last batch.
+Since the MBC is the actual spout, all the tuples in a batch are just members
of its tupletree. That means storm's "max spout pending" config effectively
defines the number of concurrent batches trident runs. The MBC emits a new
batch if it has fewer than max-spending tuples pending and if at least one
[trident batch interval]({{page.git-blob-base}}/conf/defaults.yaml#L115)'s
worth of seconds has passed since the last batch.
### If nothing was emitted does Trident slow down the calls?
-Yes, there's a pluggable "spout wait strategy"; the default is to sleep for a
[configurable amount of
time](https://github.com/apache/storm/blob/{{page.version}}/conf/defaults.yaml#L110)
+Yes, there's a pluggable "spout wait strategy"; the default is to sleep for a
[configurable amount of time]({{page.git-blob-base}}/conf/defaults.yaml#L110)
### OK, then what is the trident batch interval for?
Modified:
storm/branches/bobby-versioned-site/releases/0.10.0/Guaranteeing-message-processing.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Guaranteeing-message-processing.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
---
storm/branches/bobby-versioned-site/releases/0.10.0/Guaranteeing-message-processing.md
(original)
+++
storm/branches/bobby-versioned-site/releases/0.10.0/Guaranteeing-message-processing.md
Fri Mar 18 01:49:28 2016
@@ -2,7 +2,6 @@
title: Guaranteeing Message Processing
layout: documentation
documentation: true
-version: v0.10.0
---
Storm offers several different levels of guaranteed message processing,
includeing best effort, at least once, and exactly once through
[Trident](Trident-tutorial.html).
This page describes how Storm can guarantee at least once processing.
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Hooks.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Hooks.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Hooks.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Hooks.md Fri Mar 18
01:49:28 2016
@@ -2,7 +2,6 @@
title: Hooks
layout: documentation
documentation: true
-version: v0.10.0
---
Storm provides hooks with which you can insert custom code to run on any
number of events within Storm. You create a hook by extending the
[BaseTaskHook](javadocs/backtype/storm/hooks/BaseTaskHook.html) class and
overriding the appropriate method for the event you want to catch. There are
two ways to register your hook:
Modified:
storm/branches/bobby-versioned-site/releases/0.10.0/Implementation-docs.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Implementation-docs.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Implementation-docs.md
(original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Implementation-docs.md
Fri Mar 18 01:49:28 2016
@@ -2,7 +2,6 @@
title: Storm Internal Implementation
layout: documentation
documentation: true
-version: v0.10.0
---
This section of the wiki is dedicated to explaining how Storm is implemented.
You should have a good grasp of how to use Storm before reading these sections.
Modified:
storm/branches/bobby-versioned-site/releases/0.10.0/Installing-native-dependencies.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Installing-native-dependencies.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
---
storm/branches/bobby-versioned-site/releases/0.10.0/Installing-native-dependencies.md
(original)
+++
storm/branches/bobby-versioned-site/releases/0.10.0/Installing-native-dependencies.md
Fri Mar 18 01:49:28 2016
@@ -1,6 +1,5 @@
---
layout: documentation
-version: v0.10.0
---
The native dependencies are only needed on actual Storm clusters. When running
Storm in local mode, Storm uses a pure Java messaging system so that you don't
need to install native dependencies on your development machine.
Modified:
storm/branches/bobby-versioned-site/releases/0.10.0/Kestrel-and-Storm.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Kestrel-and-Storm.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Kestrel-and-Storm.md
(original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Kestrel-and-Storm.md
Fri Mar 18 01:49:28 2016
@@ -2,7 +2,6 @@
title: Storm and Kestrel
layout: documentation
documentation: true
-version: v0.10.0
---
This page explains how to use to Storm to consume items from a Kestrel cluster.
Modified:
storm/branches/bobby-versioned-site/releases/0.10.0/Lifecycle-of-a-topology.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Lifecycle-of-a-topology.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
---
storm/branches/bobby-versioned-site/releases/0.10.0/Lifecycle-of-a-topology.md
(original)
+++
storm/branches/bobby-versioned-site/releases/0.10.0/Lifecycle-of-a-topology.md
Fri Mar 18 01:49:28 2016
@@ -2,7 +2,6 @@
title: Lifecycle of a Storm Topology
layout: documentation
documentation: true
-version: v0.10.0
---
(**NOTE**: this page is based on the 0.7.1 code; many things have changed
since then, including a split between tasks and executors, and a reorganization
of the code under `storm-core/src` rather than `src/`.)
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Local-mode.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Local-mode.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Local-mode.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Local-mode.md Fri Mar
18 01:49:28 2016
@@ -2,7 +2,6 @@
title: Local Mode
layout: documentation
documentation: true
-version: v0.10.0
---
Local mode simulates a Storm cluster in process and is useful for developing
and testing topologies. Running topologies in local mode is similar to running
topologies [on a cluster](Running-topologies-on-a-production-cluster.html).
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Maven.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Maven.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Maven.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Maven.md Fri Mar 18
01:49:28 2016
@@ -2,8 +2,6 @@
title: Maven
layout: documentation
documentation: true
-version: v0.10.0
-raw-version: 0.10.0
---
To develop topologies, you'll need the Storm jars on your classpath. You
should either include the unpacked jars in the classpath for your project or
use Maven to include Storm as a development dependency. Storm is hosted on
Maven Central. To include Storm in your project as a development dependency,
add the following to your pom.xml:
@@ -12,13 +10,13 @@ To develop topologies, you'll need the S
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
- <version>{{page.raw-version}}</version>
+ <version>{{page.version}}</version>
<scope>provided</scope>
</dependency>
```
-[Here's an
example](https://github.com/apache/storm/blob/{{page.version}}/examples/storm-starter/pom.xml)
of a pom.xml for a Storm project.
+[Here's an example]({{page.git-blob-base}}/examples/storm-starter/pom.xml) of
a pom.xml for a Storm project.
### Developing Storm
-Please refer to
[DEVELOPER.md](https://github.com/apache/storm/blob/{{page.version}}/DEVELOPER.md)
for more details.
+Please refer to [DEVELOPER.md]({{page.git-blob-base}}/DEVELOPER.md) for more
details.
Modified:
storm/branches/bobby-versioned-site/releases/0.10.0/Message-passing-implementation.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Message-passing-implementation.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
---
storm/branches/bobby-versioned-site/releases/0.10.0/Message-passing-implementation.md
(original)
+++
storm/branches/bobby-versioned-site/releases/0.10.0/Message-passing-implementation.md
Fri Mar 18 01:49:28 2016
@@ -2,7 +2,6 @@
title: Message Passing Implementation
layout: documentation
documentation: true
-version: v0.10.0
---
(Note: this walkthrough is out of date as of 0.8.0. 0.8.0 revamped the message
passing infrastructure to be based on the Disruptor)
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Metrics.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Metrics.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Metrics.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Metrics.md Fri Mar 18
01:49:28 2016
@@ -2,7 +2,6 @@
title: Storm Metrics
layout: documentation
documentation: true
-version: v0.10.0
---
Storm exposes a metrics interface to report summary statistics across the full
topology.
It's used internally to track the numbers you see in the Nimbus UI console:
counts of executes and acks; average process latency per bolt; worker heap
usage; and so forth.
@@ -13,13 +12,13 @@ Metrics have to implement just one metho
Storm gives you these metric types:
-*
[AssignableMetric](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/jvm/backtype/storm/metric/api/AssignableMetric.java)
-- set the metric to the explicit value you supply. Useful if it's an external
value or in the case that you are already calculating the summary statistic
yourself.
-*
[CombinedMetric](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/jvm/backtype/storm/metric/api/CombinedMetric.java)
-- generic interface for metrics that can be updated associatively.
-*
[CountMetric](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/jvm/backtype/storm/metric/api/CountMetric.java)
-- a running total of the supplied values. Call `incr()` to increment by one,
`incrBy(n)` to add/subtract the given number.
- -
[MultiCountMetric](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/jvm/backtype/storm/metric/api/MultiCountMetric.java)
-- a hashmap of count metrics.
-*
[ReducedMetric](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/jvm/backtype/storm/metric/api/ReducedMetric.java)
- -
[MeanReducer](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/jvm/backtype/storm/metric/api/MeanReducer.java)
-- track a running average of values given to its `reduce()` method. (It
accepts `Double`, `Integer` or `Long` values, and maintains the internal
average as a `Double`.) Despite his reputation, the MeanReducer is actually a
pretty nice guy in person.
- -
[MultiReducedMetric](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/jvm/backtype/storm/metric/api/MultiReducedMetric.java)
-- a hashmap of reduced metrics.
+*
[AssignableMetric]({{page.git-blob-base}}/storm-core/src/jvm/backtype/storm/metric/api/AssignableMetric.java)
-- set the metric to the explicit value you supply. Useful if it's an external
value or in the case that you are already calculating the summary statistic
yourself.
+*
[CombinedMetric]({{page.git-blob-base}}/storm-core/src/jvm/backtype/storm/metric/api/CombinedMetric.java)
-- generic interface for metrics that can be updated associatively.
+*
[CountMetric]({{page.git-blob-base}}/storm-core/src/jvm/backtype/storm/metric/api/CountMetric.java)
-- a running total of the supplied values. Call `incr()` to increment by one,
`incrBy(n)` to add/subtract the given number.
+ -
[MultiCountMetric]({{page.git-blob-base}}/storm-core/src/jvm/backtype/storm/metric/api/MultiCountMetric.java)
-- a hashmap of count metrics.
+*
[ReducedMetric]({{page.git-blob-base}}/storm-core/src/jvm/backtype/storm/metric/api/ReducedMetric.java)
+ -
[MeanReducer]({{page.git-blob-base}}/storm-core/src/jvm/backtype/storm/metric/api/MeanReducer.java)
-- track a running average of values given to its `reduce()` method. (It
accepts `Double`, `Integer` or `Long` values, and maintains the internal
average as a `Double`.) Despite his reputation, the MeanReducer is actually a
pretty nice guy in person.
+ -
[MultiReducedMetric]({{page.git-blob-base}}/storm-core/src/jvm/backtype/storm/metric/api/MultiReducedMetric.java)
-- a hashmap of reduced metrics.
### Metric Consumer
@@ -31,7 +30,7 @@ Storm gives you these metric types:
### Builtin Metrics
-The [builtin
metrics](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/daemon/builtin_metrics.clj)
instrument Storm itself.
+The [builtin
metrics]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/daemon/builtin_metrics.clj)
instrument Storm itself.
-[builtin_metrics.clj](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/daemon/builtin_metrics.clj)
sets up data structures for the built-in metrics, and facade methods that the
other framework components can use to update them. The metrics themselves are
calculated in the calling code -- see for example
[`ack-spout-msg`](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/daemon/executor.clj#358)
in `clj/b/s/daemon/daemon/executor.clj`
+[builtin_metrics.clj]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/daemon/builtin_metrics.clj)
sets up data structures for the built-in metrics, and facade methods that the
other framework components can use to update them. The metrics themselves are
calculated in the calling code -- see for example
[`ack-spout-msg`]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/daemon/executor.clj#358)
in `clj/b/s/daemon/daemon/executor.clj`
Modified:
storm/branches/bobby-versioned-site/releases/0.10.0/Multilang-protocol.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Multilang-protocol.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Multilang-protocol.md
(original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Multilang-protocol.md
Fri Mar 18 01:49:28 2016
@@ -2,7 +2,6 @@
title: Multi-Lang Protocol
layout: documentation
documentation: true
-version: v0.10.0
---
This page explains the multilang protocol as of Storm 0.7.1. Versions prior to
0.7.1 used a somewhat different protocol, documented
[here](Storm-multi-language-protocol-(versions-0.7.0-and-below\).html).
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Rationale.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Rationale.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Rationale.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Rationale.md Fri Mar 18
01:49:28 2016
@@ -2,7 +2,6 @@
title: Rationale
layout: documentation
documentation: true
-version: v0.10.0
---
The past decade has seen a revolution in data processing. MapReduce, Hadoop,
and related technologies have made it possible to store and process data at
scales previously unthinkable. Unfortunately, these data processing
technologies are not realtime systems, nor are they meant to be. There's no
hack that will turn Hadoop into a realtime system; realtime data processing has
a fundamentally different set of requirements than batch processing.
Modified:
storm/branches/bobby-versioned-site/releases/0.10.0/Running-topologies-on-a-production-cluster.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Running-topologies-on-a-production-cluster.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
---
storm/branches/bobby-versioned-site/releases/0.10.0/Running-topologies-on-a-production-cluster.md
(original)
+++
storm/branches/bobby-versioned-site/releases/0.10.0/Running-topologies-on-a-production-cluster.md
Fri Mar 18 01:49:28 2016
@@ -2,7 +2,6 @@
title: Running Topologies on a Production Cluster
layout: documentation
documentation: true
-version: v0.10.0
---
Running topologies on a production cluster is similar to running in [Local
mode](Local-mode.html). Here are the steps:
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/SECURITY.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/SECURITY.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/SECURITY.md (original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/SECURITY.md Fri Mar 18
01:49:28 2016
@@ -2,7 +2,6 @@
title: Running Apache Storm Securely
layout: documentation
documentation: true
-version: v0.10.0
---
# Running Apache Storm Securely
Modified:
storm/branches/bobby-versioned-site/releases/0.10.0/STORM-UI-REST-API.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/STORM-UI-REST-API.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/STORM-UI-REST-API.md
(original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/STORM-UI-REST-API.md
Fri Mar 18 01:49:28 2016
@@ -2,7 +2,6 @@
title: Storm REST API
layout: documentation
documentation: true
-version: v0.10.0
---
# Storm UI REST API
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Serialization.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Serialization.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Serialization.md
(original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Serialization.md Fri
Mar 18 01:49:28 2016
@@ -2,7 +2,6 @@
title: Serialization
layout: documentation
documentation: true
-version: v0.10.0
---
This page is about how the serialization system in Storm works for versions
0.6.0 and onwards. Storm used a different serialization system prior to 0.6.0
which is documented on [Serialization (prior to
0.6.0)](Serialization-\(prior-to-0.6.0\).html).
Modified: storm/branches/bobby-versioned-site/releases/0.10.0/Serializers.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Serializers.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
--- storm/branches/bobby-versioned-site/releases/0.10.0/Serializers.md
(original)
+++ storm/branches/bobby-versioned-site/releases/0.10.0/Serializers.md Fri Mar
18 01:49:28 2016
@@ -1,5 +1,4 @@
---
layout: documentation
-version: v0.10.0
---
* [storm-json](https://github.com/rapportive-oss/storm-json): Simple JSON
serializer for Storm
Modified:
storm/branches/bobby-versioned-site/releases/0.10.0/Setting-up-a-Storm-cluster.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Setting-up-a-Storm-cluster.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
---
storm/branches/bobby-versioned-site/releases/0.10.0/Setting-up-a-Storm-cluster.md
(original)
+++
storm/branches/bobby-versioned-site/releases/0.10.0/Setting-up-a-Storm-cluster.md
Fri Mar 18 01:49:28 2016
@@ -2,7 +2,6 @@
title: Setting up a Storm Cluster
layout: documentation
documentation: true
-version: v0.10.0
---
This page outlines the steps for getting a Storm cluster up and running. If
you're on AWS, you should check out the
[storm-deploy](https://github.com/nathanmarz/storm-deploy/wiki) project.
[storm-deploy](https://github.com/nathanmarz/storm-deploy/wiki) completely
automates the provisioning, configuration, and installation of Storm clusters
on EC2. It also sets up Ganglia for you so you can monitor CPU, disk, and
network usage.
@@ -41,7 +40,7 @@ Next, download a Storm release and extra
### Fill in mandatory configurations into storm.yaml
-The Storm release contains a file at `conf/storm.yaml` that configures the
Storm daemons. You can see the default configuration values
[here](https://github.com/apache/storm/blob/{{page.version}}/conf/defaults.yaml).
storm.yaml overrides anything in defaults.yaml. There's a few configurations
that are mandatory to get a working cluster:
+The Storm release contains a file at `conf/storm.yaml` that configures the
Storm daemons. You can see the default configuration values
[here]({{page.git-blob-base}}/conf/defaults.yaml). storm.yaml overrides
anything in defaults.yaml. There's a few configurations that are mandatory to
get a working cluster:
1) **storm.zookeeper.servers**: This is a list of the hosts in the Zookeeper
cluster for your Storm cluster. It should look something like:
Modified:
storm/branches/bobby-versioned-site/releases/0.10.0/Setting-up-development-environment.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Setting-up-development-environment.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
---
storm/branches/bobby-versioned-site/releases/0.10.0/Setting-up-development-environment.md
(original)
+++
storm/branches/bobby-versioned-site/releases/0.10.0/Setting-up-development-environment.md
Fri Mar 18 01:49:28 2016
@@ -2,7 +2,6 @@
title: Setting Up a Development Environment
layout: documentation
documentation: true
-version: v0.10.0
---
This page outlines what you need to do to get a Storm development environment
set up. In summary, the steps are:
Modified:
storm/branches/bobby-versioned-site/releases/0.10.0/Spout-implementations.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Spout-implementations.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
---
storm/branches/bobby-versioned-site/releases/0.10.0/Spout-implementations.md
(original)
+++
storm/branches/bobby-versioned-site/releases/0.10.0/Spout-implementations.md
Fri Mar 18 01:49:28 2016
@@ -2,7 +2,6 @@
title: Spout Implementations
layout: documentation
documentation: true
-version: v0.10.0
---
* [storm-kestrel](https://github.com/nathanmarz/storm-kestrel): Adapter to use
Kestrel as a spout
* [storm-amqp-spout](https://github.com/rapportive-oss/storm-amqp-spout):
Adapter to use AMQP source as a spout
Modified:
storm/branches/bobby-versioned-site/releases/0.10.0/Structure-of-the-codebase.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Structure-of-the-codebase.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
---
storm/branches/bobby-versioned-site/releases/0.10.0/Structure-of-the-codebase.md
(original)
+++
storm/branches/bobby-versioned-site/releases/0.10.0/Structure-of-the-codebase.md
Fri Mar 18 01:49:28 2016
@@ -2,7 +2,6 @@
title: Structure of the Codebase
layout: documentation
documentation: true
-version: v0.10.0
---
There are three distinct layers to Storm's codebase.
@@ -16,18 +15,18 @@ The following sections explain each of t
### storm.thrift
-The first place to look to understand the structure of Storm's codebase is the
[storm.thrift](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/storm.thrift)
file.
+The first place to look to understand the structure of Storm's codebase is the
[storm.thrift]({{page.git-blob-base}}/storm-core/src/storm.thrift) file.
Storm uses [this fork](https://github.com/nathanmarz/thrift/tree/storm) of
Thrift (branch 'storm') to produce the generated code. This "fork" is actually
Thrift 7 with all the Java packages renamed to be `org.apache.thrift7`.
Otherwise, it's identical to Thrift 7. This fork was done because of the lack
of backwards compatibility in Thrift and the need for many people to use other
versions of Thrift in their Storm topologies.
-Every spout or bolt in a topology is given a user-specified identifier called
the "component id". The component id is used to specify subscriptions from a
bolt to the output streams of other spouts or bolts. A
[StormTopology](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/storm.thrift#L91)
structure contains a map from component id to component for each type of
component (spouts and bolts).
+Every spout or bolt in a topology is given a user-specified identifier called
the "component id". The component id is used to specify subscriptions from a
bolt to the output streams of other spouts or bolts. A
[StormTopology]({{page.git-blob-base}}/storm-core/src/storm.thrift#L91)
structure contains a map from component id to component for each type of
component (spouts and bolts).
-Spouts and bolts have the same Thrift definition, so let's just take a look at
the [Thrift definition for
bolts](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/storm.thrift#L79).
It contains a `ComponentObject` struct and a `ComponentCommon` struct.
+Spouts and bolts have the same Thrift definition, so let's just take a look at
the [Thrift definition for
bolts]({{page.git-blob-base}}/storm-core/src/storm.thrift#L79). It contains a
`ComponentObject` struct and a `ComponentCommon` struct.
The `ComponentObject` defines the implementation for the bolt. It can be one
of three types:
-1. A serialized java object (that implements
[IBolt](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/jvm/backtype/storm/task/IBolt.java))
-2. A `ShellComponent` object that indicates the implementation is in another
language. Specifying a bolt this way will cause Storm to instantiate a
[ShellBolt](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/jvm/backtype/storm/task/ShellBolt.java)
object to handle the communication between the JVM-based worker process and
the non-JVM-based implementation of the component.
+1. A serialized java object (that implements
[IBolt]({{page.git-blob-base}}/storm-core/src/jvm/backtype/storm/task/IBolt.java))
+2. A `ShellComponent` object that indicates the implementation is in another
language. Specifying a bolt this way will cause Storm to instantiate a
[ShellBolt]({{page.git-blob-base}}/storm-core/src/jvm/backtype/storm/task/ShellBolt.java)
object to handle the communication between the JVM-based worker process and
the non-JVM-based implementation of the component.
3. A `JavaObject` structure which tells Storm the classname and constructor
arguments to use to instantiate that bolt. This is useful if you want to define
a topology in a non-JVM language. This way, you can make use of JVM-based
spouts and bolts without having to create and serialize a Java object yourself.
`ComponentCommon` defines everything else for this component. This includes:
@@ -35,9 +34,9 @@ The `ComponentObject` defines the implem
1. What streams this component emits and the metadata for each stream (whether
it's a direct stream, the fields declaration)
2. What streams this component consumes (specified as a map from
component_id:stream_id to the stream grouping to use)
3. The parallelism for this component
-4. The component-specific
[configuration](https://github.com/apache/storm/wiki/Configuration) for this
component
+4. The component-specific [configuration](Configuration.html) for this
component
-Note that the structure spouts also have a `ComponentCommon` field, and so
spouts can also have declarations to consume other input streams. Yet the Storm
Java API does not provide a way for spouts to consume other streams, and if you
put any input declarations there for a spout you would get an error when you
tried to submit the topology. The reason that spouts have an input declarations
field is not for users to use, but for Storm itself to use. Storm adds implicit
streams and bolts to the topology to set up the [acking
framework](https://github.com/apache/storm/wiki/Guaranteeing-message-processing),
and two of these implicit streams are from the acker bolt to each spout in the
topology. The acker sends "ack" or "fail" messages along these streams whenever
a tuple tree is detected to be completed or failed. The code that transforms
the user's topology into the runtime topology is located
[here](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm
/daemon/common.clj#L279).
+Note that the structure spouts also have a `ComponentCommon` field, and so
spouts can also have declarations to consume other input streams. Yet the Storm
Java API does not provide a way for spouts to consume other streams, and if you
put any input declarations there for a spout you would get an error when you
tried to submit the topology. The reason that spouts have an input declarations
field is not for users to use, but for Storm itself to use. Storm adds implicit
streams and bolts to the topology to set up the [acking
framework](https://github.com/apache/storm/wiki/Guaranteeing-message-processing),
and two of these implicit streams are from the acker bolt to each spout in the
topology. The acker sends "ack" or "fail" messages along these streams whenever
a tuple tree is detected to be completed or failed. The code that transforms
the user's topology into the runtime topology is located
[here]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/daemon/common.clj#L279).
### Java interfaces
@@ -56,88 +55,88 @@ You can see this strategy at work with t
Spouts and bolts are serialized into the Thrift definition of the topology as
described above.
-One subtle aspect of the interfaces is the difference between `IBolt` and
`ISpout` vs. `IRichBolt` and `IRichSpout`. The main difference between them is
the addition of the `declareOutputFields` method in the "Rich" versions of the
interfaces. The reason for the split is that the output fields declaration for
each output stream needs to be part of the Thrift struct (so it can be
specified from any language), but as a user you want to be able to declare the
streams as part of your class. What `TopologyBuilder` does when constructing
the Thrift representation is call `declareOutputFields` to get the declaration
and convert it into the Thrift structure. The conversion happens [at this
portion](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/jvm/backtype/storm/topology/TopologyBuilder.java#L205)
of the `TopologyBuilder` code.
+One subtle aspect of the interfaces is the difference between `IBolt` and
`ISpout` vs. `IRichBolt` and `IRichSpout`. The main difference between them is
the addition of the `declareOutputFields` method in the "Rich" versions of the
interfaces. The reason for the split is that the output fields declaration for
each output stream needs to be part of the Thrift struct (so it can be
specified from any language), but as a user you want to be able to declare the
streams as part of your class. What `TopologyBuilder` does when constructing
the Thrift representation is call `declareOutputFields` to get the declaration
and convert it into the Thrift structure. The conversion happens [at this
portion]({{page.git-blob-base}}/storm-core/src/jvm/backtype/storm/topology/TopologyBuilder.java#L205)
of the `TopologyBuilder` code.
### Implementation
Specifying all the functionality via Java interfaces ensures that every
feature of Storm is available via Java. Moreso, the focus on Java interfaces
ensures that the user experience from Java-land is pleasant as well.
-The implementation of Storm, on the other hand, is primarily in Clojure. While
the codebase is about 50% Java and 50% Clojure in terms of LOC, most of the
implementation logic is in Clojure. There are two notable exceptions to this,
and that is the [DRPC](https://github.com/apache/storm/wiki/Distributed-RPC)
and [transactional
topologies](https://github.com/apache/storm/wiki/Transactional-topologies)
implementations. These are implemented purely in Java. This was done to serve
as an illustration for how to implement a higher level abstraction on Storm.
The DRPC and transactional topologies implementations are in the
[backtype.storm.coordination](https://github.com/apache/storm/tree/{{page.version}}/storm-core/src/jvm/backtype/storm/coordination),
[backtype.storm.drpc](https://github.com/apache/storm/tree/{{page.version}}/storm-core/src/jvm/backtype/storm/drpc),
and
[backtype.storm.transactional](https://github.com/apache/storm/tree/{{page.version}}/storm-core/src/jvm/backtype/storm/
transactional) packages.
+The implementation of Storm, on the other hand, is primarily in Clojure. While
the codebase is about 50% Java and 50% Clojure in terms of LOC, most of the
implementation logic is in Clojure. There are two notable exceptions to this,
and that is the [DRPC](https://github.com/apache/storm/wiki/Distributed-RPC)
and [transactional
topologies](https://github.com/apache/storm/wiki/Transactional-topologies)
implementations. These are implemented purely in Java. This was done to serve
as an illustration for how to implement a higher level abstraction on Storm.
The DRPC and transactional topologies implementations are in the
[backtype.storm.coordination]({{page.git-tree-base}}/storm-core/src/jvm/backtype/storm/coordination),
[backtype.storm.drpc]({{page.git-tree-base}}/storm-core/src/jvm/backtype/storm/drpc),
and
[backtype.storm.transactional]({{page.git-tree-base}}/storm-core/src/jvm/backtype/storm/transactional)
packages.
Here's a summary of the purpose of the main Java packages and Clojure
namespace:
#### Java packages
-[backtype.storm.coordination](https://github.com/apache/storm/tree/{{page.version}}/storm-core/src/jvm/backtype/storm/coordination):
Implements the pieces required to coordinate batch-processing on top of Storm,
which both DRPC and transactional topologies use. `CoordinatedBolt` is the most
important class here.
+[backtype.storm.coordination]({{page.git-tree-base}}/storm-core/src/jvm/backtype/storm/coordination):
Implements the pieces required to coordinate batch-processing on top of Storm,
which both DRPC and transactional topologies use. `CoordinatedBolt` is the most
important class here.
-[backtype.storm.drpc](https://github.com/apache/storm/tree/{{page.version}}/storm-core/src/jvm/backtype/storm/drpc):
Implementation of the DRPC higher level abstraction
+[backtype.storm.drpc]({{page.git-tree-base}}/storm-core/src/jvm/backtype/storm/drpc):
Implementation of the DRPC higher level abstraction
-[backtype.storm.generated](https://github.com/apache/storm/tree/{{page.version}}/storm-core/src/jvm/backtype/storm/generated):
The generated Thrift code for Storm (generated using [this
fork](https://github.com/nathanmarz/thrift) of Thrift, which simply renames the
packages to org.apache.thrift7 to avoid conflicts with other Thrift versions)
+[backtype.storm.generated]({{page.git-tree-base}}/storm-core/src/jvm/backtype/storm/generated):
The generated Thrift code for Storm (generated using [this
fork](https://github.com/nathanmarz/thrift) of Thrift, which simply renames the
packages to org.apache.thrift7 to avoid conflicts with other Thrift versions)
-[backtype.storm.grouping](https://github.com/apache/storm/tree/{{page.version}}/storm-core/src/jvm/backtype/storm/grouping):
Contains interface for making custom stream groupings
+[backtype.storm.grouping]({{page.git-tree-base}}/storm-core/src/jvm/backtype/storm/grouping):
Contains interface for making custom stream groupings
-[backtype.storm.hooks](https://github.com/apache/storm/tree/{{page.version}}/storm-core/src/jvm/backtype/storm/hooks):
Interfaces for hooking into various events in Storm, such as when tasks emit
tuples, when tuples are acked, etc. User guide for hooks is
[here](https://github.com/apache/storm/wiki/Hooks).
+[backtype.storm.hooks]({{page.git-tree-base}}/storm-core/src/jvm/backtype/storm/hooks):
Interfaces for hooking into various events in Storm, such as when tasks emit
tuples, when tuples are acked, etc. User guide for hooks is
[here](https://github.com/apache/storm/wiki/Hooks).
-[backtype.storm.serialization](https://github.com/apache/storm/tree/{{page.version}}/storm-core/src/jvm/backtype/storm/serialization):
Implementation of how Storm serializes/deserializes tuples. Built on top of
[Kryo](http://code.google.com/p/kryo/).
+[backtype.storm.serialization]({{page.git-tree-base}}/storm-core/src/jvm/backtype/storm/serialization):
Implementation of how Storm serializes/deserializes tuples. Built on top of
[Kryo](http://code.google.com/p/kryo/).
-[backtype.storm.spout](https://github.com/apache/storm/tree/{{page.version}}/storm-core/src/jvm/backtype/storm/spout):
Definition of spout and associated interfaces (like the
`SpoutOutputCollector`). Also contains `ShellSpout` which implements the
protocol for defining spouts in non-JVM languages.
+[backtype.storm.spout]({{page.git-tree-base}}/storm-core/src/jvm/backtype/storm/spout):
Definition of spout and associated interfaces (like the
`SpoutOutputCollector`). Also contains `ShellSpout` which implements the
protocol for defining spouts in non-JVM languages.
-[backtype.storm.task](https://github.com/apache/storm/tree/{{page.version}}/storm-core/src/jvm/backtype/storm/task):
Definition of bolt and associated interfaces (like `OutputCollector`). Also
contains `ShellBolt` which implements the protocol for defining bolts in
non-JVM languages. Finally, `TopologyContext` is defined here as well, which is
provided to spouts and bolts so they can get data about the topology and its
execution at runtime.
+[backtype.storm.task]({{page.git-tree-base}}/storm-core/src/jvm/backtype/storm/task):
Definition of bolt and associated interfaces (like `OutputCollector`). Also
contains `ShellBolt` which implements the protocol for defining bolts in
non-JVM languages. Finally, `TopologyContext` is defined here as well, which is
provided to spouts and bolts so they can get data about the topology and its
execution at runtime.
-[backtype.storm.testing](https://github.com/apache/storm/tree/{{page.version}}/storm-core/src/jvm/backtype/storm/testing):
Contains a variety of test bolts and utilities used in Storm's unit tests.
+[backtype.storm.testing]({{page.git-tree-base}}/storm-core/src/jvm/backtype/storm/testing):
Contains a variety of test bolts and utilities used in Storm's unit tests.
-[backtype.storm.topology](https://github.com/apache/storm/tree/{{page.version}}/storm-core/src/jvm/backtype/storm/topology):
Java layer over the underlying Thrift structure to provide a clean, pure-Java
API to Storm (users don't have to know about Thrift). `TopologyBuilder` is here
as well as the helpful base classes for the different spouts and bolts. The
slightly-higher level `IBasicBolt` interface is here, which is a simpler way to
write certain kinds of bolts.
+[backtype.storm.topology]({{page.git-tree-base}}/storm-core/src/jvm/backtype/storm/topology):
Java layer over the underlying Thrift structure to provide a clean, pure-Java
API to Storm (users don't have to know about Thrift). `TopologyBuilder` is here
as well as the helpful base classes for the different spouts and bolts. The
slightly-higher level `IBasicBolt` interface is here, which is a simpler way to
write certain kinds of bolts.
-[backtype.storm.transactional](https://github.com/apache/storm/tree/{{page.version}}/storm-core/src/jvm/backtype/storm/transactional):
Implementation of transactional topologies.
+[backtype.storm.transactional]({{page.git-tree-base}}/storm-core/src/jvm/backtype/storm/transactional):
Implementation of transactional topologies.
-[backtype.storm.tuple](https://github.com/apache/storm/tree/{{page.version}}/storm-core/src/jvm/backtype/storm/tuple):
Implementation of Storm's tuple data model.
+[backtype.storm.tuple]({{page.git-tree-base}}/storm-core/src/jvm/backtype/storm/tuple):
Implementation of Storm's tuple data model.
-[backtype.storm.utils](https://github.com/apache/storm/tree/{{page.version}}/storm-core/src/jvm/backtype/storm/tuple):
Data structures and miscellaneous utilities used throughout the codebase.
+[backtype.storm.utils]({{page.git-tree-base}}/storm-core/src/jvm/backtype/storm/tuple):
Data structures and miscellaneous utilities used throughout the codebase.
#### Clojure namespaces
-[backtype.storm.bootstrap](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/bootstrap.clj):
Contains a helpful macro to import all the classes and namespaces that are
used throughout the codebase.
+[backtype.storm.bootstrap]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/bootstrap.clj):
Contains a helpful macro to import all the classes and namespaces that are
used throughout the codebase.
-[backtype.storm.clojure](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/clojure.clj):
Implementation of the Clojure DSL for Storm.
+[backtype.storm.clojure]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/clojure.clj):
Implementation of the Clojure DSL for Storm.
-[backtype.storm.cluster](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/cluster.clj):
All Zookeeper logic used in Storm daemons is encapsulated in this file. This
code manages how cluster state (like what tasks are running where, what
spout/bolt each task runs as) is mapped to the Zookeeper "filesystem" API.
+[backtype.storm.cluster]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/cluster.clj):
All Zookeeper logic used in Storm daemons is encapsulated in this file. This
code manages how cluster state (like what tasks are running where, what
spout/bolt each task runs as) is mapped to the Zookeeper "filesystem" API.
-[backtype.storm.command.*](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/command):
These namespaces implement various commands for the `storm` command line
client. These implementations are very short.
+[backtype.storm.command.*]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/command):
These namespaces implement various commands for the `storm` command line
client. These implementations are very short.
-[backtype.storm.config](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/config.clj):
Implementation of config reading/parsing code for Clojure. Also has utility
functions for determining what local path nimbus/supervisor/daemons should be
using for various things. e.g. the `master-inbox` function will return the
local path that Nimbus should use when jars are uploaded to it.
+[backtype.storm.config]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/config.clj):
Implementation of config reading/parsing code for Clojure. Also has utility
functions for determining what local path nimbus/supervisor/daemons should be
using for various things. e.g. the `master-inbox` function will return the
local path that Nimbus should use when jars are uploaded to it.
-[backtype.storm.daemon.acker](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/daemon/acker.clj):
Implementation of the "acker" bolt, which is a key part of how Storm
guarantees data processing.
+[backtype.storm.daemon.acker]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/daemon/acker.clj):
Implementation of the "acker" bolt, which is a key part of how Storm
guarantees data processing.
-[backtype.storm.daemon.common](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/daemon/common.clj):
Implementation of common functions used in Storm daemons, like getting the id
for a topology based on the name, mapping a user's topology into the one that
actually executes (with implicit acking streams and acker bolt added - see
`system-topology!` function), and definitions for the various heartbeat and
other structures persisted by Storm.
+[backtype.storm.daemon.common]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/daemon/common.clj):
Implementation of common functions used in Storm daemons, like getting the id
for a topology based on the name, mapping a user's topology into the one that
actually executes (with implicit acking streams and acker bolt added - see
`system-topology!` function), and definitions for the various heartbeat and
other structures persisted by Storm.
-[backtype.storm.daemon.drpc](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/daemon/drpc.clj):
Implementation of the DRPC server for use with DRPC topologies.
+[backtype.storm.daemon.drpc]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/daemon/drpc.clj):
Implementation of the DRPC server for use with DRPC topologies.
-[backtype.storm.daemon.nimbus](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/daemon/nimbus.clj):
Implementation of Nimbus.
+[backtype.storm.daemon.nimbus]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/daemon/nimbus.clj):
Implementation of Nimbus.
-[backtype.storm.daemon.supervisor](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/daemon/supervisor.clj):
Implementation of Supervisor.
+[backtype.storm.daemon.supervisor]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/daemon/supervisor.clj):
Implementation of Supervisor.
-[backtype.storm.daemon.task](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/daemon/task.clj):
Implementation of an individual task for a spout or bolt. Handles message
routing, serialization, stats collection for the UI, as well as the
spout-specific and bolt-specific execution implementations.
+[backtype.storm.daemon.task]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/daemon/task.clj):
Implementation of an individual task for a spout or bolt. Handles message
routing, serialization, stats collection for the UI, as well as the
spout-specific and bolt-specific execution implementations.
-[backtype.storm.daemon.worker](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/daemon/worker.clj):
Implementation of a worker process (which will contain many tasks within).
Implements message transferring and task launching.
+[backtype.storm.daemon.worker]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/daemon/worker.clj):
Implementation of a worker process (which will contain many tasks within).
Implements message transferring and task launching.
-[backtype.storm.event](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/event.clj):
Implements a simple asynchronous function executor. Used in various places in
Nimbus and Supervisor to make functions execute in serial to avoid any race
conditions.
+[backtype.storm.event]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/event.clj):
Implements a simple asynchronous function executor. Used in various places in
Nimbus and Supervisor to make functions execute in serial to avoid any race
conditions.
-[backtype.storm.log](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/log.clj):
Defines the functions used to log messages to log4j.
+[backtype.storm.log]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/log.clj):
Defines the functions used to log messages to log4j.
-[backtype.storm.messaging.*](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/messaging):
Defines a higher level interface to implementing point to point messaging. In
local mode Storm uses in-memory Java queues to do this; on a cluster, it uses
ZeroMQ. The generic interface is defined in protocol.clj.
+[backtype.storm.messaging.*]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/messaging):
Defines a higher level interface to implementing point to point messaging. In
local mode Storm uses in-memory Java queues to do this; on a cluster, it uses
ZeroMQ. The generic interface is defined in protocol.clj.
-[backtype.storm.stats](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/stats.clj):
Implementation of stats rollup routines used when sending stats to ZK for use
by the UI. Does things like windowed and rolling aggregations at multiple
granularities.
+[backtype.storm.stats]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/stats.clj):
Implementation of stats rollup routines used when sending stats to ZK for use
by the UI. Does things like windowed and rolling aggregations at multiple
granularities.
-[backtype.storm.testing](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/testing.clj):
Implementation of facilities used to test Storm topologies. Includes time
simulation, `complete-topology` for running a fixed set of tuples through a
topology and capturing the output, tracker topologies for having fine grained
control over detecting when a cluster is "idle", and other utilities.
+[backtype.storm.testing]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/testing.clj):
Implementation of facilities used to test Storm topologies. Includes time
simulation, `complete-topology` for running a fixed set of tuples through a
topology and capturing the output, tracker topologies for having fine grained
control over detecting when a cluster is "idle", and other utilities.
-[backtype.storm.thrift](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/thrift.clj):
Clojure wrappers around the generated Thrift API to make working with Thrift
structures more pleasant.
+[backtype.storm.thrift]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/thrift.clj):
Clojure wrappers around the generated Thrift API to make working with Thrift
structures more pleasant.
-[backtype.storm.timer](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/timer.clj):
Implementation of a background timer to execute functions in the future or on
a recurring interval. Storm couldn't use the
[Timer](http://docs.oracle.com/javase/1.4.2/docs/api/java/util/Timer.html)
class because it needed integration with time simulation in order to be able to
unit test Nimbus and the Supervisor.
+[backtype.storm.timer]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/timer.clj):
Implementation of a background timer to execute functions in the future or on
a recurring interval. Storm couldn't use the
[Timer](http://docs.oracle.com/javase/1.4.2/docs/api/java/util/Timer.html)
class because it needed integration with time simulation in order to be able to
unit test Nimbus and the Supervisor.
-[backtype.storm.ui.*](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/ui):
Implementation of Storm UI. Completely independent from rest of code base and
uses the Nimbus Thrift API to get data.
+[backtype.storm.ui.*]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/ui):
Implementation of Storm UI. Completely independent from rest of code base and
uses the Nimbus Thrift API to get data.
-[backtype.storm.util](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/util.clj):
Contains generic utility functions used throughout the code base.
+[backtype.storm.util]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/util.clj):
Contains generic utility functions used throughout the code base.
-[backtype.storm.zookeeper](https://github.com/apache/storm/blob/{{page.version}}/storm-core/src/clj/backtype/storm/zookeeper.clj):
Clojure wrapper around the Zookeeper API and implements some "high-level"
stuff like "mkdirs" and "delete-recursive".
+[backtype.storm.zookeeper]({{page.git-blob-base}}/storm-core/src/clj/backtype/storm/zookeeper.clj):
Clojure wrapper around the Zookeeper API and implements some "high-level"
stuff like "mkdirs" and "delete-recursive".
Modified:
storm/branches/bobby-versioned-site/releases/0.10.0/Support-for-non-java-languages.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Support-for-non-java-languages.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
---
storm/branches/bobby-versioned-site/releases/0.10.0/Support-for-non-java-languages.md
(original)
+++
storm/branches/bobby-versioned-site/releases/0.10.0/Support-for-non-java-languages.md
Fri Mar 18 01:49:28 2016
@@ -2,7 +2,6 @@
title: Support for Non-Java Languages
layout: documentation
documentation: true
-version: v0.10.0
---
* [Scala DSL](https://github.com/velvia/ScalaStorm)
* [JRuby DSL](https://github.com/colinsurprenant/storm-jruby)
Modified:
storm/branches/bobby-versioned-site/releases/0.10.0/Transactional-topologies.md
URL:
http://svn.apache.org/viewvc/storm/branches/bobby-versioned-site/releases/0.10.0/Transactional-topologies.md?rev=1735530&r1=1735529&r2=1735530&view=diff
==============================================================================
---
storm/branches/bobby-versioned-site/releases/0.10.0/Transactional-topologies.md
(original)
+++
storm/branches/bobby-versioned-site/releases/0.10.0/Transactional-topologies.md
Fri Mar 18 01:49:28 2016
@@ -2,7 +2,6 @@
title: Transactional Topologies
layout: documentation
documentation: true
-version: v0.10.0
---
**NOTE**: Transactional topologies have been deprecated -- use the
[Trident](Trident-tutorial.html) framework instead.
@@ -82,7 +81,7 @@ Finally, another thing to note is that t
## The basics through example
-You build transactional topologies by using
[TransactionalTopologyBuilder](javadocs/backtype/storm/transactional/TransactionalTopologyBuilder.html).
Here's the transactional topology definition for a topology that computes the
global count of tuples from the input stream. This code comes from
[TransactionalGlobalCount](https://github.com/apache/storm/blob/{{page.version}}/examples/storm-starter/src/jvm/storm/starter/TransactionalGlobalCount.java)
in storm-starter.
+You build transactional topologies by using
[TransactionalTopologyBuilder](javadocs/backtype/storm/transactional/TransactionalTopologyBuilder.html).
Here's the transactional topology definition for a topology that computes the
global count of tuples from the input stream. This code comes from
[TransactionalGlobalCount]({{page.git-blob-base}}/examples/storm-starter/src/jvm/storm/starter/TransactionalGlobalCount.java)
in storm-starter.
```java
MemoryTransactionalSpout spout = new MemoryTransactionalSpout(DATA, new
Fields("word"), PARTITION_TAKE_PER_BATCH);
@@ -202,7 +201,7 @@ First, notice that this bolt implements
The code for `finishBatch` in `UpdateGlobalCount` gets the current value from
the database and compares its transaction id to the transaction id for this
batch. If they are the same, it does nothing. Otherwise, it increments the
value in the database by the partial count for this batch.
-A more involved transactional topology example that updates multiple databases
idempotently can be found in storm-starter in the
[TransactionalWords](https://github.com/apache/storm/blob/{{page.version}}/examples/storm-starter/src/jvm/storm/starter/TransactionalWords.java)
class.
+A more involved transactional topology example that updates multiple databases
idempotently can be found in storm-starter in the
[TransactionalWords]({{page.git-blob-base}}/examples/storm-starter/src/jvm/storm/starter/TransactionalWords.java)
class.
## Transactional Topology API
@@ -256,7 +255,7 @@ The details of implementing a `Transacti
#### Partitioned Transactional Spout
-A common kind of transactional spout is one that reads the batches from a set
of partitions across many queue brokers. For example, this is how
[TransactionalKafkaSpout](https://github.com/apache/storm/tree/{{page.version}}/external/storm-kafka/src/jvm/storm/kafka/TransactionalKafkaSpout.java)
works. An `IPartitionedTransactionalSpout` automates the bookkeeping work of
managing the state for each partition to ensure idempotent replayability. See
[the
Javadoc](javadocs/backtype/storm/transactional/partitioned/IPartitionedTransactionalSpout.html)
for more details.
+A common kind of transactional spout is one that reads the batches from a set
of partitions across many queue brokers. For example, this is how
[TransactionalKafkaSpout]({{page.git-tree-base}}/external/storm-kafka/src/jvm/storm/kafka/TransactionalKafkaSpout.java)
works. An `IPartitionedTransactionalSpout` automates the bookkeeping work of
managing the state for each partition to ensure idempotent replayability. See
[the
Javadoc](javadocs/backtype/storm/transactional/partitioned/IPartitionedTransactionalSpout.html)
for more details.
### Configuration
@@ -326,7 +325,7 @@ In this scenario, tuples 41-50 are skipp
By failing all subsequent transactions on failure, no tuples are skipped. This
also shows that a requirement of transactional spouts is that they always emit
where the last transaction left off.
-A non-idempotent transactional spout is more concisely referred to as an
"OpaqueTransactionalSpout" (opaque is the opposite of idempotent).
[IOpaquePartitionedTransactionalSpout](javadocs/backtype/storm/transactional/partitioned/IOpaquePartitionedTransactionalSpout.html)
is an interface for implementing opaque partitioned transactional spouts, of
which
[OpaqueTransactionalKafkaSpout](https://github.com/apache/storm/tree/{{page.version}}/external/storm-kafka/src/jvm/storm/kafka/OpaqueTransactionalKafkaSpout.java)
is an example. `OpaqueTransactionalKafkaSpout` can withstand losing individual
Kafka nodes without sacrificing accuracy as long as you use the update strategy
as explained in this section.
+A non-idempotent transactional spout is more concisely referred to as an
"OpaqueTransactionalSpout" (opaque is the opposite of idempotent).
[IOpaquePartitionedTransactionalSpout](javadocs/backtype/storm/transactional/partitioned/IOpaquePartitionedTransactionalSpout.html)
is an interface for implementing opaque partitioned transactional spouts, of
which
[OpaqueTransactionalKafkaSpout]({{page.git-tree-base}}/external/storm-kafka/src/jvm/storm/kafka/OpaqueTransactionalKafkaSpout.java)
is an example. `OpaqueTransactionalKafkaSpout` can withstand losing individual
Kafka nodes without sacrificing accuracy as long as you use the update strategy
as explained in this section.
## Implementation
