http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/1.0.3/Serialization-(prior-to-0.6.0).html ---------------------------------------------------------------------- diff --git a/content/releases/1.0.3/Serialization-(prior-to-0.6.0).html b/content/releases/1.0.3/Serialization-(prior-to-0.6.0).html index 815f0eb..6a68d68 100644 --- a/content/releases/1.0.3/Serialization-(prior-to-0.6.0).html +++ b/content/releases/1.0.3/Serialization-(prior-to-0.6.0).html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> @@ -182,9 +182,9 @@ <p>The interface looks like this:</p> <div class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">ISerialization</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="o">{</span> - <span class="kd">public</span> <span class="kt">boolean</span> <span class="n">accept</span><span class="o">(</span><span class="n">Class</span> <span class="n">c</span><span class="o">);</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">serialize</span><span class="o">(</span><span class="n">T</span> <span class="n">object</span><span class="o">,</span> <span class="n">DataOutputStream</span> <span class="n">stream</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span><span class="o">;</span> - <span class="kd">public</span> <span class="n">T</span> <span class="n">deserialize</span><span class="o">(</span><span class="n">DataInputStream</span> <span class="n">stream</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span><span class="o">;</span> + <span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">accept</span><span class="o">(</span><span class="n">Class</span> <span class="n">c</span><span class="o">);</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">serialize</span><span class="o">(</span><span class="n">T</span> <span class="n">object</span><span class="o">,</span> <span class="n">DataOutputStream</span> <span class="n">stream</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span><span class="o">;</span> + <span class="kd">public</span> <span class="n">T</span> <span class="nf">deserialize</span><span class="o">(</span><span class="n">DataInputStream</span> <span class="n">stream</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span><span class="o">;</span> <span class="o">}</span> </code></pre></div> <p>Storm uses the <code>accept</code> method to determine if a type can be serialized by this serializer. Remember, Storm's tuples are dynamically typed so Storm determines what serializer to use at runtime.</p>
http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/1.0.3/Serialization.html ---------------------------------------------------------------------- diff --git a/content/releases/1.0.3/Serialization.html b/content/releases/1.0.3/Serialization.html index 18bc709..817a923 100644 --- a/content/releases/1.0.3/Serialization.html +++ b/content/releases/1.0.3/Serialization.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/1.0.3/Serializers.html ---------------------------------------------------------------------- diff --git a/content/releases/1.0.3/Serializers.html b/content/releases/1.0.3/Serializers.html index 27ec8ac..4f3b677 100644 --- a/content/releases/1.0.3/Serializers.html +++ b/content/releases/1.0.3/Serializers.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/1.0.3/Setting-up-a-Storm-cluster.html ---------------------------------------------------------------------- diff --git a/content/releases/1.0.3/Setting-up-a-Storm-cluster.html b/content/releases/1.0.3/Setting-up-a-Storm-cluster.html index b40da00..9724e78 100644 --- a/content/releases/1.0.3/Setting-up-a-Storm-cluster.html +++ b/content/releases/1.0.3/Setting-up-a-Storm-cluster.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> @@ -252,7 +252,7 @@ The time to allow any given healthcheck script to run before it is marked failed <p>If you need support from external libraries or custom plugins, you can place such jars into the extlib/ and extlib-daemon/ directories. Note that the extlib-daemon/ directory stores jars used only by daemons (Nimbus, Supervisor, DRPC, UI, Logviewer), e.g., HDFS and customized scheduling libraries. Accordingly, two environmental variables STORM_EXT_CLASSPATH and STORM_EXT_CLASSPATH_DAEMON can be configured by users for including the external classpath and daemon-only external classpath.</p> -<h3 id="launch-daemons-under-supervision-using-quot-storm-quot-script-and-a-supervisor-of-your-choice">Launch daemons under supervision using "storm" script and a supervisor of your choice</h3> +<h3 id="launch-daemons-under-supervision-using-storm-script-and-a-supervisor-of-your-choice">Launch daemons under supervision using "storm" script and a supervisor of your choice</h3> <p>The last step is to launch all the Storm daemons. It is critical that you run each of these daemons under supervision. Storm is a <strong>fail-fast</strong> system which means the processes will halt whenever an unexpected error is encountered. Storm is designed so that it can safely halt at any point and recover correctly when the process is restarted. This is why Storm keeps no state in-process -- if Nimbus or the Supervisors restart, the running topologies are unaffected. Here's how to run the Storm daemons:</p> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/1.0.3/Setting-up-development-environment.html ---------------------------------------------------------------------- diff --git a/content/releases/1.0.3/Setting-up-development-environment.html b/content/releases/1.0.3/Setting-up-development-environment.html index 3ddd7d1..b63d768 100644 --- a/content/releases/1.0.3/Setting-up-development-environment.html +++ b/content/releases/1.0.3/Setting-up-development-environment.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/1.0.3/Spout-implementations.html ---------------------------------------------------------------------- diff --git a/content/releases/1.0.3/Spout-implementations.html b/content/releases/1.0.3/Spout-implementations.html index 2bc3d72..760e428 100644 --- a/content/releases/1.0.3/Spout-implementations.html +++ b/content/releases/1.0.3/Spout-implementations.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/1.0.3/State-checkpointing.html ---------------------------------------------------------------------- diff --git a/content/releases/1.0.3/State-checkpointing.html b/content/releases/1.0.3/State-checkpointing.html index f225745..596555b 100644 --- a/content/releases/1.0.3/State-checkpointing.html +++ b/content/releases/1.0.3/State-checkpointing.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> @@ -187,15 +187,15 @@ last committed by the framework during the previous run.</li> <span class="kd">private</span> <span class="n">OutputCollector</span> <span class="n">collector</span><span class="o">;</span> <span class="o">...</span> <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">prepare</span><span class="o">(</span><span class="n">Map</span> <span class="n">stormConf</span><span class="o">,</span> <span class="n">TopologyContext</span> <span class="n">context</span><span class="o">,</span> <span class="n">OutputCollector</span> <span class="n">collector</span><span class="o">)</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">prepare</span><span class="o">(</span><span class="n">Map</span> <span class="n">stormConf</span><span class="o">,</span> <span class="n">TopologyContext</span> <span class="n">context</span><span class="o">,</span> <span class="n">OutputCollector</span> <span class="n">collector</span><span class="o">)</span> <span class="o">{</span> <span class="k">this</span><span class="o">.</span><span class="na">collector</span> <span class="o">=</span> <span class="n">collector</span><span class="o">;</span> <span class="o">}</span> <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">initState</span><span class="o">(</span><span class="n">KeyValueState</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">></span> <span class="n">state</span><span class="o">)</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">initState</span><span class="o">(</span><span class="n">KeyValueState</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">></span> <span class="n">state</span><span class="o">)</span> <span class="o">{</span> <span class="n">wordCounts</span> <span class="o">=</span> <span class="n">state</span><span class="o">;</span> <span class="o">}</span> <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">execute</span><span class="o">(</span><span class="n">Tuple</span> <span class="n">tuple</span><span class="o">)</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">execute</span><span class="o">(</span><span class="n">Tuple</span> <span class="n">tuple</span><span class="o">)</span> <span class="o">{</span> <span class="n">String</span> <span class="n">word</span> <span class="o">=</span> <span class="n">tuple</span><span class="o">.</span><span class="na">getString</span><span class="o">(</span><span class="mi">0</span><span class="o">);</span> <span class="n">Integer</span> <span class="n">count</span> <span class="o">=</span> <span class="n">wordCounts</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">word</span><span class="o">,</span> <span class="mi">0</span><span class="o">);</span> <span class="n">count</span><span class="o">++;</span> @@ -216,20 +216,20 @@ in the extlib directory.</li> <li>The state provider properties can be overridden by setting <code>topology.state.provider.config</code>. For Redis state this is a json config with the following properties.</li> </ol> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="nt">"keyClass"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Optional fully qualified class name of the Key type."</span><span class="p">,</span><span class="w"> - </span><span class="nt">"valueClass"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Optional fully qualified class name of the Value type."</span><span class="p">,</span><span class="w"> - </span><span class="nt">"keySerializerClass"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Optional Key serializer implementation class."</span><span class="p">,</span><span class="w"> - </span><span class="nt">"valueSerializerClass"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Optional Value Serializer implementation class."</span><span class="p">,</span><span class="w"> - </span><span class="nt">"jedisPoolConfig"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="nt">"host"</span><span class="p">:</span><span class="w"> </span><span class="s2">"localhost"</span><span class="p">,</span><span class="w"> - </span><span class="nt">"port"</span><span class="p">:</span><span class="w"> </span><span class="mi">6379</span><span class="p">,</span><span class="w"> - </span><span class="nt">"timeout"</span><span class="p">:</span><span class="w"> </span><span class="mi">2000</span><span class="p">,</span><span class="w"> - </span><span class="nt">"database"</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> - </span><span class="nt">"password"</span><span class="p">:</span><span class="w"> </span><span class="s2">"xyz"</span><span class="w"> - </span><span class="p">}</span><span class="w"> - </span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang=""> { + "keyClass": "Optional fully qualified class name of the Key type.", + "valueClass": "Optional fully qualified class name of the Value type.", + "keySerializerClass": "Optional Key serializer implementation class.", + "valueSerializerClass": "Optional Value Serializer implementation class.", + "jedisPoolConfig": { + "host": "localhost", + "port": 6379, + "timeout": 2000, + "database": 0, + "password": "xyz" + } + } +</code></pre></div> <h2 id="checkpoint-mechanism">Checkpoint mechanism</h2> <p>Checkpoint is triggered by an internal checkpoint spout at the specified <code>topology.state.checkpoint.interval.ms</code>. If there is http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/1.0.3/Storm-Scheduler.html ---------------------------------------------------------------------- diff --git a/content/releases/1.0.3/Storm-Scheduler.html b/content/releases/1.0.3/Storm-Scheduler.html index 6773a4c..81ece2b 100644 --- a/content/releases/1.0.3/Storm-Scheduler.html +++ b/content/releases/1.0.3/Storm-Scheduler.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/1.0.3/Storm-multi-language-protocol-(versions-0.7.0-and-below).html ---------------------------------------------------------------------- diff --git a/content/releases/1.0.3/Storm-multi-language-protocol-(versions-0.7.0-and-below).html b/content/releases/1.0.3/Storm-multi-language-protocol-(versions-0.7.0-and-below).html index 19f0279..2871dcf 100644 --- a/content/releases/1.0.3/Storm-multi-language-protocol-(versions-0.7.0-and-below).html +++ b/content/releases/1.0.3/Storm-multi-language-protocol-(versions-0.7.0-and-below).html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> @@ -210,56 +210,56 @@ file lets the supervisor know the PID so it can shutdown the process later on.</ <li>The rest happens in a while(true) loop</li> <li>STDIN: A tuple! This is a JSON encoded structure like this:</li> </ul> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">tuple's</span><span class="w"> </span><span class="err">id</span><span class="w"> - </span><span class="nt">"id"</span><span class="p">:</span><span class="w"> </span><span class="mi">-6955786537413359385</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">component</span><span class="w"> </span><span class="err">that</span><span class="w"> </span><span class="err">created</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> - </span><span class="nt">"comp"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">stream</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> </span><span class="err">was</span><span class="w"> </span><span class="err">emitted</span><span class="w"> </span><span class="err">to</span><span class="w"> - </span><span class="nt">"stream"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">task</span><span class="w"> </span><span class="err">that</span><span class="w"> </span><span class="err">created</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> - </span><span class="nt">"task"</span><span class="p">:</span><span class="w"> </span><span class="mi">9</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">All</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">values</span><span class="w"> </span><span class="err">in</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> - </span><span class="nt">"tuple"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"snow white and the seven dwarfs"</span><span class="p">,</span><span class="w"> </span><span class="s2">"field2"</span><span class="p">,</span><span class="w"> </span><span class="mi">3</span><span class="p">]</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + // The tuple's id + "id": -6955786537413359385, + // The id of the component that created this tuple + "comp": 1, + // The id of the stream this tuple was emitted to + "stream": 1, + // The id of the task that created this tuple + "task": 9, + // All the values in this tuple + "tuple": ["snow white and the seven dwarfs", "field2", 3] +} +</code></pre></div> <ul> <li>STDOUT: The results of your bolt, JSON encoded. This can be a sequence of acks, fails, emits, and/or logs. Emits look like:</li> </ul> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"emit"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">ids</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">tuples</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">output</span><span class="w"> </span><span class="err">tuples</span><span class="w"> </span><span class="err">should</span><span class="w"> </span><span class="err">be</span><span class="w"> </span><span class="err">anchored</span><span class="w"> </span><span class="err">to</span><span class="w"> - </span><span class="nt">"anchors"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="mi">1231231</span><span class="p">,</span><span class="w"> </span><span class="mi">-234234234</span><span class="p">],</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">stream</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> </span><span class="err">was</span><span class="w"> </span><span class="err">emitted</span><span class="w"> </span><span class="err">to.</span><span class="w"> </span><span class="err">Leave</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">empty</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">emit</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">default</span><span class="w"> </span><span class="err">stream.</span><sp an class="w"> - </span><span class="nt">"stream"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">If</span><span class="w"> </span><span class="err">doing</span><span class="w"> </span><span class="err">an</span><span class="w"> </span><span class="err">emit</span><span class="w"> </span><span class="err">direct,</span><span class="w"> </span><span class="err">indicate</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">task</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">sent</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">tuple</span><span class="w"> </span><span class="err">to</span><span class="w"> - </span><span class="nt">"task"</span><span class="p">:</span><span class="w"> </span><span class="mi">9</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">All</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">values</span><span class="w"> </span><span class="err">in</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> - </span><span class="nt">"tuple"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"field1"</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w"> </span><span class="mi">3</span><span class="p">]</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + "command": "emit", + // The ids of the tuples this output tuples should be anchored to + "anchors": [1231231, -234234234], + // The id of the stream this tuple was emitted to. Leave this empty to emit to default stream. + "stream": 1, + // If doing an emit direct, indicate the task to sent the tuple to + "task": 9, + // All the values in this tuple + "tuple": ["field1", 2, 3] +} +</code></pre></div> <p>An ack looks like:</p> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ack"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">tuple</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">ack</span><span class="w"> - </span><span class="nt">"id"</span><span class="p">:</span><span class="w"> </span><span class="mi">123123</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + "command": "ack", + // the id of the tuple to ack + "id": 123123 +} +</code></pre></div> <p>A fail looks like:</p> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"fail"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">tuple</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">fail</span><span class="w"> - </span><span class="nt">"id"</span><span class="p">:</span><span class="w"> </span><span class="mi">123123</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + "command": "fail", + // the id of the tuple to fail + "id": 123123 +} +</code></pre></div> <p>A "log" will log a message in the worker log. It looks like:</p> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"log"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">message</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">log</span><span class="w"> - </span><span class="nt">"msg"</span><span class="p">:</span><span class="w"> </span><span class="s2">"hello world!"</span><span class="w"> +<div class="highlight"><pre><code class="language-" data-lang="">{ + "command": "log", + // the message to log + "msg": "hello world!" -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +} +</code></pre></div> <ul> <li>STDOUT: emit "sync" as a single line by itself when the bolt has finished emitting/acking/failing and is ready for the next input</li> </ul> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/1.0.3/Structure-of-the-codebase.html ---------------------------------------------------------------------- diff --git a/content/releases/1.0.3/Structure-of-the-codebase.html b/content/releases/1.0.3/Structure-of-the-codebase.html index 8ec35bb..280b7a4 100644 --- a/content/releases/1.0.3/Structure-of-the-codebase.html +++ b/content/releases/1.0.3/Structure-of-the-codebase.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/1.0.3/Support-for-non-java-languages.html ---------------------------------------------------------------------- diff --git a/content/releases/1.0.3/Support-for-non-java-languages.html b/content/releases/1.0.3/Support-for-non-java-languages.html index fa5648e..347b89a 100644 --- a/content/releases/1.0.3/Support-for-non-java-languages.html +++ b/content/releases/1.0.3/Support-for-non-java-languages.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/1.0.3/Transactional-topologies.html ---------------------------------------------------------------------- diff --git a/content/releases/1.0.3/Transactional-topologies.html b/content/releases/1.0.3/Transactional-topologies.html index 15394ac..613b30f 100644 --- a/content/releases/1.0.3/Transactional-topologies.html +++ b/content/releases/1.0.3/Transactional-topologies.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> @@ -211,7 +211,7 @@ <p>After bolt 1 finishes its portion of the processing, it will be idle until the rest of the bolts finish and the next batch can be emitted from the spout.</p> -<h3 id="design-3-storm-39-s-design">Design 3 (Storm's design)</h3> +<h3 id="design-3-storms-design">Design 3 (Storm's design)</h3> <p>A key realization is that not all the work for processing batches of tuples needs to be strongly ordered. For example, when computing a global count, there's two parts to the computation:</p> @@ -268,23 +268,23 @@ <span class="kt">int</span> <span class="n">_count</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">prepare</span><span class="o">(</span><span class="n">Map</span> <span class="n">conf</span><span class="o">,</span> <span class="n">TopologyContext</span> <span class="n">context</span><span class="o">,</span> <span class="n">BatchOutputCollector</span> <span class="n">collector</span><span class="o">,</span> <span class="n">Object</span> <span class="n">id</span><span class="o">)</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">prepare</span><span class="o">(</span><span class="n">Map</span> <span class="n">conf</span><span class="o">,</span> <span class="n">TopologyContext</span> <span class="n">context</span><span class="o">,</span> <span class="n">BatchOutputCollector</span> <span class="n">collector</span><span class="o">,</span> <span class="n">Object</span> <span class="n">id</span><span class="o">)</span> <span class="o">{</span> <span class="n">_collector</span> <span class="o">=</span> <span class="n">collector</span><span class="o">;</span> <span class="n">_id</span> <span class="o">=</span> <span class="n">id</span><span class="o">;</span> <span class="o">}</span> <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">execute</span><span class="o">(</span><span class="n">Tuple</span> <span class="n">tuple</span><span class="o">)</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">execute</span><span class="o">(</span><span class="n">Tuple</span> <span class="n">tuple</span><span class="o">)</span> <span class="o">{</span> <span class="n">_count</span><span class="o">++;</span> <span class="o">}</span> <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">finishBatch</span><span class="o">()</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">finishBatch</span><span class="o">()</span> <span class="o">{</span> <span class="n">_collector</span><span class="o">.</span><span class="na">emit</span><span class="o">(</span><span class="k">new</span> <span class="n">Values</span><span class="o">(</span><span class="n">_id</span><span class="o">,</span> <span class="n">_count</span><span class="o">));</span> <span class="o">}</span> <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">declareOutputFields</span><span class="o">(</span><span class="n">OutputFieldsDeclarer</span> <span class="n">declarer</span><span class="o">)</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">declareOutputFields</span><span class="o">(</span><span class="n">OutputFieldsDeclarer</span> <span class="n">declarer</span><span class="o">)</span> <span class="o">{</span> <span class="n">declarer</span><span class="o">.</span><span class="na">declare</span><span class="o">(</span><span class="k">new</span> <span class="n">Fields</span><span class="o">(</span><span class="s">"id"</span><span class="o">,</span> <span class="s">"count"</span><span class="o">));</span> <span class="o">}</span> <span class="o">}</span> @@ -313,18 +313,18 @@ <span class="kt">int</span> <span class="n">_sum</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">prepare</span><span class="o">(</span><span class="n">Map</span> <span class="n">conf</span><span class="o">,</span> <span class="n">TopologyContext</span> <span class="n">context</span><span class="o">,</span> <span class="n">BatchOutputCollector</span> <span class="n">collector</span><span class="o">,</span> <span class="n">TransactionAttempt</span> <span class="n">attempt</span><span class="o">)</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">prepare</span><span class="o">(</span><span class="n">Map</span> <span class="n">conf</span><span class="o">,</span> <span class="n">TopologyContext</span> <span class="n">context</span><span class="o">,</span> <span class="n">BatchOutputCollector</span> <span class="n">collector</span><span class="o">,</span> <span class="n">TransactionAttempt</span> <span class="n">attempt</span><span class="o">)</span> <span class="o">{</span> <span class="n">_collector</span> <span class="o">=</span> <span class="n">collector</span><span class="o">;</span> <span class="n">_attempt</span> <span class="o">=</span> <span class="n">attempt</span><span class="o">;</span> <span class="o">}</span> <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">execute</span><span class="o">(</span><span class="n">Tuple</span> <span class="n">tuple</span><span class="o">)</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">execute</span><span class="o">(</span><span class="n">Tuple</span> <span class="n">tuple</span><span class="o">)</span> <span class="o">{</span> <span class="n">_sum</span><span class="o">+=</span><span class="n">tuple</span><span class="o">.</span><span class="na">getInteger</span><span class="o">(</span><span class="mi">1</span><span class="o">);</span> <span class="o">}</span> <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">finishBatch</span><span class="o">()</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">finishBatch</span><span class="o">()</span> <span class="o">{</span> <span class="n">Value</span> <span class="n">val</span> <span class="o">=</span> <span class="n">DATABASE</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">GLOBAL_COUNT_KEY</span><span class="o">);</span> <span class="n">Value</span> <span class="n">newval</span><span class="o">;</span> <span class="k">if</span><span class="o">(</span><span class="n">val</span> <span class="o">==</span> <span class="kc">null</span> <span class="o">||</span> <span class="o">!</span><span class="n">val</span><span class="o">.</span><span class="na">txid</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="n">_attempt</span><span class="o">.</span><span class="na">getTransactionId</span><span class="o">()))</span> <span class="o">{</span> @@ -343,7 +343,7 @@ <span class="o">}</span> <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">declareOutputFields</span><span class="o">(</span><span class="n">OutputFieldsDeclarer</span> <span class="n">declarer</span><span class="o">)</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">declareOutputFields</span><span class="o">(</span><span class="n">OutputFieldsDeclarer</span> <span class="n">declarer</span><span class="o">)</span> <span class="o">{</span> <span class="n">declarer</span><span class="o">.</span><span class="na">declare</span><span class="o">(</span><span class="k">new</span> <span class="n">Fields</span><span class="o">(</span><span class="s">"id"</span><span class="o">,</span> <span class="s">"sum"</span><span class="o">));</span> <span class="o">}</span> <span class="o">}</span> @@ -421,7 +421,7 @@ <li><em>Number of active batches permissible at once:</em> You must set a limit to the number of batches that can be processed at once. You configure this using the "topology.max.spout.pending" config. If you don't set this config, it will default to 1.</li> </ol> -<h2 id="what-if-you-can-39-t-emit-the-same-batch-of-tuples-for-a-given-transaction-id">What if you can't emit the same batch of tuples for a given transaction id?</h2> +<h2 id="what-if-you-cant-emit-the-same-batch-of-tuples-for-a-given-transaction-id">What if you can't emit the same batch of tuples for a given transaction id?</h2> <p>So far the discussion around transactional topologies has assumed that you can always emit the exact same batch of tuples for the same transaction id. So what do you do if this is not possible?</p>
