http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/0.10.1/documentation/Multilang-protocol.html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/documentation/Multilang-protocol.html b/content/releases/0.10.1/documentation/Multilang-protocol.html index fe6f61c..8c3f8f6 100644 --- a/content/releases/0.10.1/documentation/Multilang-protocol.html +++ b/content/releases/0.10.1/documentation/Multilang-protocol.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> @@ -211,44 +211,44 @@ STDIN and STDOUT.</li> <ul> <li>STDIN: Setup info. This is a JSON object with the Storm configuration, a PID directory, and a topology context, like this:</li> </ul> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="nt">"conf"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="nt">"topology.message.timeout.secs"</span><span class="p">:</span><span class="w"> </span><span class="mi">3</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">etc</span><span class="w"> - </span><span class="err">}</span><span class="p">,</span><span class="w"> - </span><span class="nt">"pidDir"</span><span class="p">:</span><span class="w"> </span><span class="s2">"..."</span><span class="p">,</span><span class="w"> - </span><span class="nt">"context"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="nt">"task->component"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="nt">"1"</span><span class="p">:</span><span class="w"> </span><span class="s2">"example-spout"</span><span class="p">,</span><span class="w"> - </span><span class="nt">"2"</span><span class="p">:</span><span class="w"> </span><span class="s2">"__acker"</span><span class="p">,</span><span class="w"> - </span><span class="nt">"3"</span><span class="p">:</span><span class="w"> </span><span class="s2">"example-bolt1"</span><span class="p">,</span><span class="w"> - </span><span class="nt">"4"</span><span class="p">:</span><span class="w"> </span><span class="s2">"example-bolt2"</span><span class="w"> - </span><span class="p">},</span><span class="w"> - </span><span class="nt">"taskid"</span><span class="p">:</span><span class="w"> </span><span class="mi">3</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">Everything</span><span class="w"> </span><span class="err">below</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">line</span><span class="w"> </span><span class="err">is</span><span class="w"> </span><span class="err">only</span><span class="w"> </span><span class="err">available</span><span class="w"> </span><span class="err">in</span><span class="w"> </span><span class="err">Storm</span><span class="w"> </span><span class="err">0.10.0+</span><span class="w"> - </span><span class="nt">"componentid"</span><span class="p">:</span><span class="w"> </span><span class="s2">"example-bolt"</span><span class="w"> - </span><span class="s2">"stream->target->grouping"</span><span class="err">:</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="nt">"default"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="nt">"example-bolt2"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="nt">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"SHUFFLE"</span><span class="p">}}},</span><span class="w"> - </span><span class="nt">"streams"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"default"</span><span class="p">],</span><span class="w"> - </span><span class="nt">"stream->outputfields"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="nt">"default"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"word"</span><span class="p">]},</span><span class="w"> - </span><span class="nt">"source->stream->grouping"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="nt">"example-spout"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="nt">"default"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="nt">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"FIELDS"</span><span class="p">,</span><span class="w"> - </span><span class="nt">"fields"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"word"</span><span class="p">]</span><span class="w"> - </span><span class="p">}</span><span class="w"> - </span><span class="p">}</span><span class="w"> - </span><span class="p">}</span><span class="w"> - </span><span class="s2">"source->stream->fields"</span><span class="err">:</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="nt">"example-spout"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="nt">"default"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"word"</span><span class="p">]</span><span class="w"> - </span><span class="p">}</span><span class="w"> - </span><span class="p">}</span><span class="w"> - </span><span class="p">}</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + "conf": { + "topology.message.timeout.secs": 3, + // etc + }, + "pidDir": "...", + "context": { + "task->component": { + "1": "example-spout", + "2": "__acker", + "3": "example-bolt1", + "4": "example-bolt2" + }, + "taskid": 3, + // Everything below this line is only available in Storm 0.10.0+ + "componentid": "example-bolt" + "stream->target->grouping": { + "default": { + "example-bolt2": { + "type": "SHUFFLE"}}}, + "streams": ["default"], + "stream->outputfields": {"default": ["word"]}, + "source->stream->grouping": { + "example-spout": { + "default": { + "type": "FIELDS", + "fields": ["word"] + } + } + } + "source->stream->fields": { + "example-spout": { + "default": ["word"] + } + } + } +} +</code></pre></div> <p>Your script should create an empty file named with its PID in this directory. e.g. the PID is 1234, so an empty file named 1234 is created in the directory. This file lets the supervisor know the PID so it can shutdown the process later on.</p> @@ -277,47 +277,47 @@ to specify which fields are involved in a <code>FIELDS</code> grouping.</p> </ul> <p>"next" is the equivalent of ISpout's <code>nextTuple</code>. It looks like:</p> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"next"</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{"command": "next"} +</code></pre></div> <p>"ack" looks like:</p> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ack"</span><span class="p">,</span><span class="w"> </span><span class="nt">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1231231"</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{"command": "ack", "id": "1231231"} +</code></pre></div> <p>"fail" looks like:</p> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"fail"</span><span class="p">,</span><span class="w"> </span><span class="nt">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1231231"</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{"command": "fail", "id": "1231231"} +</code></pre></div> <ul> <li>STDOUT: The results of your spout for the previous command. This can be a sequence of emits and logs.</li> </ul> <p>An emit looks like:</p> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"emit"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">for</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">tuple.</span><span class="w"> </span><span class="err">Leave</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">out</span><span class="w"> </span><span class="err">for</span><span class="w"> </span><span class="err">an</span><span class="w"> </span><span class="err">unreliable</span><span class="w"> </span><span class="err">emit.</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">can</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">be</span><span class="w"> </span><span class="err">a</span><span class="w"> </span><span class="err">string</span><span class="w"> </span><span class="err">or</span><span class="w"> </span><span class="err">a</span><span class="w"> </span><span class="err">number.</span><span class="w"> - </span><span class="nt">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1231231"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">stream</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> </span><span class="err">was</span><span class="w"> </span><span class="err">emitted</span><span class="w"> </span><span class="err">to.</span><span class="w"> </span><span class="err">Leave</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">empty</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">emit</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">default</span><span class="w"> </span><span class="err">stream.</span><sp an class="w"> - </span><span class="nt">"stream"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">If</span><span class="w"> </span><span class="err">doing</span><span class="w"> </span><span class="err">an</span><span class="w"> </span><span class="err">emit</span><span class="w"> </span><span class="err">direct,</span><span class="w"> </span><span class="err">indicate</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">task</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">send</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">tuple</span><span class="w"> </span><span class="err">to</span><span class="w"> - </span><span class="nt">"task"</span><span class="p">:</span><span class="w"> </span><span class="mi">9</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">All</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">values</span><span class="w"> </span><span class="err">in</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> - </span><span class="nt">"tuple"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"field1"</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w"> </span><span class="mi">3</span><span class="p">]</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + "command": "emit", + // The id for the tuple. Leave this out for an unreliable emit. The id can + // be a string or a number. + "id": "1231231", + // The id of the stream this tuple was emitted to. Leave this empty to emit to default stream. + "stream": "1", + // If doing an emit direct, indicate the task to send the tuple to + "task": 9, + // All the values in this tuple + "tuple": ["field1", 2, 3] +} +</code></pre></div> <p>If not doing an emit direct, you will immediately receive the task ids to which the tuple was emitted on STDIN as a JSON array.</p> <p>A "log" will log a message in the worker log. It looks like:</p> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"log"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">message</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">log</span><span class="w"> - </span><span class="nt">"msg"</span><span class="p">:</span><span class="w"> </span><span class="s2">"hello world!"</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + "command": "log", + // the message to log + "msg": "hello world!" +} +</code></pre></div> <ul> <li>STDOUT: a "sync" command ends the sequence of emits and logs. It looks like:</li> </ul> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"sync"</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{"command": "sync"} +</code></pre></div> <p>After you sync, ShellSpout will not read your output until it sends another next, ack, or fail command.</p> <p>Note that, similarly to ISpout, all of the spouts in the worker will be locked up after a next, ack, or fail, until you sync. Also like ISpout, if you have no tuples to emit for a next, you should sleep for a small amount of time before syncing. ShellSpout will not automatically sleep for you.</p> @@ -329,34 +329,34 @@ be a sequence of emits and logs.</li> <ul> <li>STDIN: A tuple! This is a JSON encoded structure like this:</li> </ul> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">tuple's</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">-</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">is</span><span class="w"> </span><span class="err">a</span><span class="w"> </span><span class="err">string</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">support</span><span class="w"> </span><span class="err">languages</span><span class="w"> </span><span class="err">lacking</span><span class="w"> </span><span class="err">64-bit</span><span class="w"> </span><span class="err">precision</span><span class="w"> - </span><span class="nt">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"-6955786537413359385"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">component</span><span class="w"> </span><span class="err">that</span><span class="w"> </span><span class="err">created</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> - </span><span class="nt">"comp"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">stream</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> </span><span class="err">was</span><span class="w"> </span><span class="err">emitted</span><span class="w"> </span><span class="err">to</span><span class="w"> - </span><span class="nt">"stream"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">task</span><span class="w"> </span><span class="err">that</span><span class="w"> </span><span class="err">created</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> - </span><span class="nt">"task"</span><span class="p">:</span><span class="w"> </span><span class="mi">9</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">All</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">values</span><span class="w"> </span><span class="err">in</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> - </span><span class="nt">"tuple"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"snow white and the seven dwarfs"</span><span class="p">,</span><span class="w"> </span><span class="s2">"field2"</span><span class="p">,</span><span class="w"> </span><span class="mi">3</span><span class="p">]</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + // The tuple's id - this is a string to support languages lacking 64-bit precision + "id": "-6955786537413359385", + // The id of the component that created this tuple + "comp": "1", + // The id of the stream this tuple was emitted to + "stream": "1", + // The id of the task that created this tuple + "task": 9, + // All the values in this tuple + "tuple": ["snow white and the seven dwarfs", "field2", 3] +} +</code></pre></div> <ul> <li>STDOUT: An ack, fail, emit, or log. Emits look like:</li> </ul> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"emit"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">ids</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">tuples</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">output</span><span class="w"> </span><span class="err">tuples</span><span class="w"> </span><span class="err">should</span><span class="w"> </span><span class="err">be</span><span class="w"> </span><span class="err">anchored</span><span class="w"> </span><span class="err">to</span><span class="w"> - </span><span class="nt">"anchors"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"1231231"</span><span class="p">,</span><span class="w"> </span><span class="s2">"-234234234"</span><span class="p">],</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">stream</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> </span><span class="err">was</span><span class="w"> </span><span class="err">emitted</span><span class="w"> </span><span class="err">to.</span><span class="w"> </span><span class="err">Leave</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">empty</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">emit</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">default</span><span class="w"> </span><span class="err">stream.</span><sp an class="w"> - </span><span class="nt">"stream"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">If</span><span class="w"> </span><span class="err">doing</span><span class="w"> </span><span class="err">an</span><span class="w"> </span><span class="err">emit</span><span class="w"> </span><span class="err">direct,</span><span class="w"> </span><span class="err">indicate</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">task</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">send</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">tuple</span><span class="w"> </span><span class="err">to</span><span class="w"> - </span><span class="nt">"task"</span><span class="p">:</span><span class="w"> </span><span class="mi">9</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">All</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">values</span><span class="w"> </span><span class="err">in</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> - </span><span class="nt">"tuple"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"field1"</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w"> </span><span class="mi">3</span><span class="p">]</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + "command": "emit", + // The ids of the tuples this output tuples should be anchored to + "anchors": ["1231231", "-234234234"], + // The id of the stream this tuple was emitted to. Leave this empty to emit to default stream. + "stream": "1", + // If doing an emit direct, indicate the task to send the tuple to + "task": 9, + // All the values in this tuple + "tuple": ["field1", 2, 3] +} +</code></pre></div> <p>If not doing an emit direct, you will receive the task ids to which the tuple was emitted on STDIN as a JSON array. Note that, due to the asynchronous nature of the shell bolt protocol, when you read after @@ -366,26 +366,26 @@ receive the task id lists in the same order as their corresponding emits, however.</p> <p>An ack looks like:</p> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ack"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">tuple</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">ack</span><span class="w"> - </span><span class="nt">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"123123"</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + "command": "ack", + // the id of the tuple to ack + "id": "123123" +} +</code></pre></div> <p>A fail looks like:</p> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"fail"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">tuple</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">fail</span><span class="w"> - </span><span class="nt">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"123123"</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + "command": "fail", + // the id of the tuple to fail + "id": "123123" +} +</code></pre></div> <p>A "log" will log a message in the worker log. It looks like:</p> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"log"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">message</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">log</span><span class="w"> - </span><span class="nt">"msg"</span><span class="p">:</span><span class="w"> </span><span class="s2">"hello world!"</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + "command": "log", + // the message to log + "msg": "hello world!" +} +</code></pre></div> <ul> <li>Note that, as of version 0.7.1, there is no longer any need for a shell bolt to 'sync'.</li> @@ -409,15 +409,15 @@ timeout during <code>next()</code>.</p> <p>Shell bolts are asynchronous, so a ShellBolt will send heartbeat tuples to its subprocess periodically. Heartbeat tuple looks like:</p> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="nt">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"-6955786537413359385"</span><span class="p">,</span><span class="w"> - </span><span class="nt">"comp"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1"</span><span class="p">,</span><span class="w"> - </span><span class="nt">"stream"</span><span class="p">:</span><span class="w"> </span><span class="s2">"__heartbeat"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">shell</span><span class="w"> </span><span class="err">bolt's</span><span class="w"> </span><span class="err">system</span><span class="w"> </span><span class="err">task</span><span class="w"> </span><span class="err">id</span><span class="w"> - </span><span class="nt">"task"</span><span class="p">:</span><span class="w"> </span><span class="mi">-1</span><span class="p">,</span><span class="w"> - </span><span class="nt">"tuple"</span><span class="p">:</span><span class="w"> </span><span class="p">[]</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + "id": "-6955786537413359385", + "comp": "1", + "stream": "__heartbeat", + // this shell bolt's system task id + "task": -1, + "tuple": [] +} +</code></pre></div> <p>When subprocess receives heartbeat tuple, it must send a <code>sync</code> command back to ShellBolt.</p>
http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/0.10.1/documentation/Powered-By.html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/documentation/Powered-By.html b/content/releases/0.10.1/documentation/Powered-By.html index d02ae78..888fb59 100644 --- a/content/releases/0.10.1/documentation/Powered-By.html +++ b/content/releases/0.10.1/documentation/Powered-By.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/0.10.1/documentation/Project-ideas.html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/documentation/Project-ideas.html b/content/releases/0.10.1/documentation/Project-ideas.html index c343c36..80aee28 100644 --- a/content/releases/0.10.1/documentation/Project-ideas.html +++ b/content/releases/0.10.1/documentation/Project-ideas.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/0.10.1/documentation/Rationale.html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/documentation/Rationale.html b/content/releases/0.10.1/documentation/Rationale.html index 9bc2796..71abccf 100644 --- a/content/releases/0.10.1/documentation/Rationale.html +++ b/content/releases/0.10.1/documentation/Rationale.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/0.10.1/documentation/Running-topologies-on-a-production-cluster.html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/documentation/Running-topologies-on-a-production-cluster.html b/content/releases/0.10.1/documentation/Running-topologies-on-a-production-cluster.html index d0091e4..2d264d1 100644 --- a/content/releases/0.10.1/documentation/Running-topologies-on-a-production-cluster.html +++ b/content/releases/0.10.1/documentation/Running-topologies-on-a-production-cluster.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/0.10.1/documentation/Serialization-(prior-to-0.6.0).html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/documentation/Serialization-(prior-to-0.6.0).html b/content/releases/0.10.1/documentation/Serialization-(prior-to-0.6.0).html index 066fe49..4d1ef97 100644 --- a/content/releases/0.10.1/documentation/Serialization-(prior-to-0.6.0).html +++ b/content/releases/0.10.1/documentation/Serialization-(prior-to-0.6.0).html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> @@ -182,9 +182,9 @@ <p>The interface looks like this:</p> <div class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">ISerialization</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="o">{</span> - <span class="kd">public</span> <span class="kt">boolean</span> <span class="n">accept</span><span class="o">(</span><span class="n">Class</span> <span class="n">c</span><span class="o">);</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">serialize</span><span class="o">(</span><span class="n">T</span> <span class="n">object</span><span class="o">,</span> <span class="n">DataOutputStream</span> <span class="n">stream</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span><span class="o">;</span> - <span class="kd">public</span> <span class="n">T</span> <span class="n">deserialize</span><span class="o">(</span><span class="n">DataInputStream</span> <span class="n">stream</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span><span class="o">;</span> + <span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">accept</span><span class="o">(</span><span class="n">Class</span> <span class="n">c</span><span class="o">);</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">serialize</span><span class="o">(</span><span class="n">T</span> <span class="n">object</span><span class="o">,</span> <span class="n">DataOutputStream</span> <span class="n">stream</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span><span class="o">;</span> + <span class="kd">public</span> <span class="n">T</span> <span class="nf">deserialize</span><span class="o">(</span><span class="n">DataInputStream</span> <span class="n">stream</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span><span class="o">;</span> <span class="o">}</span> </code></pre></div> <p>Storm uses the <code>accept</code> method to determine if a type can be serialized by this serializer. Remember, Storm's tuples are dynamically typed so Storm determines what serializer to use at runtime.</p> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/0.10.1/documentation/Serialization.html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/documentation/Serialization.html b/content/releases/0.10.1/documentation/Serialization.html index 89366f8..0f6c7ba 100644 --- a/content/releases/0.10.1/documentation/Serialization.html +++ b/content/releases/0.10.1/documentation/Serialization.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/0.10.1/documentation/Serializers.html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/documentation/Serializers.html b/content/releases/0.10.1/documentation/Serializers.html index c3004ef..1dc692e 100644 --- a/content/releases/0.10.1/documentation/Serializers.html +++ b/content/releases/0.10.1/documentation/Serializers.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/0.10.1/documentation/Setting-up-a-Storm-cluster.html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/documentation/Setting-up-a-Storm-cluster.html b/content/releases/0.10.1/documentation/Setting-up-a-Storm-cluster.html index f34fb2d..07497e0 100644 --- a/content/releases/0.10.1/documentation/Setting-up-a-Storm-cluster.html +++ b/content/releases/0.10.1/documentation/Setting-up-a-Storm-cluster.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> @@ -235,7 +235,7 @@ You can leave it empty with default value <code>$STORM_HOME/storm-local</code> <p>If you need support from external libraries or custom plugins, you can place such jars into the extlib/ and extlib-daemon/ directories. Note that the extlib-daemon/ directory stores jars used only by daemons (Nimbus, Supervisor, DRPC, UI, Logviewer), e.g., HDFS and customized scheduling libraries. Accordingly, two environmental variables STORM_EXT_CLASSPATH and STORM_EXT_CLASSPATH_DAEMON can be configured by users for including the external classpath and daemon-only external classpath.</p> -<h3 id="launch-daemons-under-supervision-using-quot-storm-quot-script-and-a-supervisor-of-your-choice">Launch daemons under supervision using "storm" script and a supervisor of your choice</h3> +<h3 id="launch-daemons-under-supervision-using-storm-script-and-a-supervisor-of-your-choice">Launch daemons under supervision using "storm" script and a supervisor of your choice</h3> <p>The last step is to launch all the Storm daemons. It is critical that you run each of these daemons under supervision. Storm is a <strong>fail-fast</strong> system which means the processes will halt whenever an unexpected error is encountered. Storm is designed so that it can safely halt at any point and recover correctly when the process is restarted. This is why Storm keeps no state in-process -- if Nimbus or the Supervisors restart, the running topologies are unaffected. Here's how to run the Storm daemons:</p> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/0.10.1/documentation/Setting-up-development-environment.html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/documentation/Setting-up-development-environment.html b/content/releases/0.10.1/documentation/Setting-up-development-environment.html index 19d340f..f95e12d 100644 --- a/content/releases/0.10.1/documentation/Setting-up-development-environment.html +++ b/content/releases/0.10.1/documentation/Setting-up-development-environment.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/0.10.1/documentation/Spout-implementations.html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/documentation/Spout-implementations.html b/content/releases/0.10.1/documentation/Spout-implementations.html index d6f9c9a..672ebd7 100644 --- a/content/releases/0.10.1/documentation/Spout-implementations.html +++ b/content/releases/0.10.1/documentation/Spout-implementations.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/0.10.1/documentation/Storm-multi-language-protocol-(versions-0.7.0-and-below).html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/documentation/Storm-multi-language-protocol-(versions-0.7.0-and-below).html b/content/releases/0.10.1/documentation/Storm-multi-language-protocol-(versions-0.7.0-and-below).html index dd241ad..9b89b42 100644 --- a/content/releases/0.10.1/documentation/Storm-multi-language-protocol-(versions-0.7.0-and-below).html +++ b/content/releases/0.10.1/documentation/Storm-multi-language-protocol-(versions-0.7.0-and-below).html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> @@ -210,56 +210,56 @@ file lets the supervisor know the PID so it can shutdown the process later on.</ <li>The rest happens in a while(true) loop</li> <li>STDIN: A tuple! This is a JSON encoded structure like this:</li> </ul> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">tuple's</span><span class="w"> </span><span class="err">id</span><span class="w"> - </span><span class="nt">"id"</span><span class="p">:</span><span class="w"> </span><span class="mi">-6955786537413359385</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">component</span><span class="w"> </span><span class="err">that</span><span class="w"> </span><span class="err">created</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> - </span><span class="nt">"comp"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">stream</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> </span><span class="err">was</span><span class="w"> </span><span class="err">emitted</span><span class="w"> </span><span class="err">to</span><span class="w"> - </span><span class="nt">"stream"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">task</span><span class="w"> </span><span class="err">that</span><span class="w"> </span><span class="err">created</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> - </span><span class="nt">"task"</span><span class="p">:</span><span class="w"> </span><span class="mi">9</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">All</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">values</span><span class="w"> </span><span class="err">in</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> - </span><span class="nt">"tuple"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"snow white and the seven dwarfs"</span><span class="p">,</span><span class="w"> </span><span class="s2">"field2"</span><span class="p">,</span><span class="w"> </span><span class="mi">3</span><span class="p">]</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + // The tuple's id + "id": -6955786537413359385, + // The id of the component that created this tuple + "comp": 1, + // The id of the stream this tuple was emitted to + "stream": 1, + // The id of the task that created this tuple + "task": 9, + // All the values in this tuple + "tuple": ["snow white and the seven dwarfs", "field2", 3] +} +</code></pre></div> <ul> <li>STDOUT: The results of your bolt, JSON encoded. This can be a sequence of acks, fails, emits, and/or logs. Emits look like:</li> </ul> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"emit"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">ids</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">tuples</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">output</span><span class="w"> </span><span class="err">tuples</span><span class="w"> </span><span class="err">should</span><span class="w"> </span><span class="err">be</span><span class="w"> </span><span class="err">anchored</span><span class="w"> </span><span class="err">to</span><span class="w"> - </span><span class="nt">"anchors"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="mi">1231231</span><span class="p">,</span><span class="w"> </span><span class="mi">-234234234</span><span class="p">],</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">stream</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> </span><span class="err">was</span><span class="w"> </span><span class="err">emitted</span><span class="w"> </span><span class="err">to.</span><span class="w"> </span><span class="err">Leave</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">empty</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">emit</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">default</span><span class="w"> </span><span class="err">stream.</span><sp an class="w"> - </span><span class="nt">"stream"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">If</span><span class="w"> </span><span class="err">doing</span><span class="w"> </span><span class="err">an</span><span class="w"> </span><span class="err">emit</span><span class="w"> </span><span class="err">direct,</span><span class="w"> </span><span class="err">indicate</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">task</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">sent</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">tuple</span><span class="w"> </span><span class="err">to</span><span class="w"> - </span><span class="nt">"task"</span><span class="p">:</span><span class="w"> </span><span class="mi">9</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">All</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">values</span><span class="w"> </span><span class="err">in</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> - </span><span class="nt">"tuple"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"field1"</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w"> </span><span class="mi">3</span><span class="p">]</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + "command": "emit", + // The ids of the tuples this output tuples should be anchored to + "anchors": [1231231, -234234234], + // The id of the stream this tuple was emitted to. Leave this empty to emit to default stream. + "stream": 1, + // If doing an emit direct, indicate the task to sent the tuple to + "task": 9, + // All the values in this tuple + "tuple": ["field1", 2, 3] +} +</code></pre></div> <p>An ack looks like:</p> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ack"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">tuple</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">ack</span><span class="w"> - </span><span class="nt">"id"</span><span class="p">:</span><span class="w"> </span><span class="mi">123123</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + "command": "ack", + // the id of the tuple to ack + "id": 123123 +} +</code></pre></div> <p>A fail looks like:</p> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"fail"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">tuple</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">fail</span><span class="w"> - </span><span class="nt">"id"</span><span class="p">:</span><span class="w"> </span><span class="mi">123123</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + "command": "fail", + // the id of the tuple to fail + "id": 123123 +} +</code></pre></div> <p>A "log" will log a message in the worker log. It looks like:</p> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"log"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">message</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">log</span><span class="w"> - </span><span class="nt">"msg"</span><span class="p">:</span><span class="w"> </span><span class="s2">"hello world!"</span><span class="w"> +<div class="highlight"><pre><code class="language-" data-lang="">{ + "command": "log", + // the message to log + "msg": "hello world!" -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +} +</code></pre></div> <ul> <li>STDOUT: emit "sync" as a single line by itself when the bolt has finished emitting/acking/failing and is ready for the next input</li> </ul> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/0.10.1/documentation/Structure-of-the-codebase.html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/documentation/Structure-of-the-codebase.html b/content/releases/0.10.1/documentation/Structure-of-the-codebase.html index 86306a9..6b07c3e 100644 --- a/content/releases/0.10.1/documentation/Structure-of-the-codebase.html +++ b/content/releases/0.10.1/documentation/Structure-of-the-codebase.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/0.10.1/documentation/Support-for-non-java-languages.html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/documentation/Support-for-non-java-languages.html b/content/releases/0.10.1/documentation/Support-for-non-java-languages.html index dff8e1e..fa743b1 100644 --- a/content/releases/0.10.1/documentation/Support-for-non-java-languages.html +++ b/content/releases/0.10.1/documentation/Support-for-non-java-languages.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/0.10.1/documentation/Transactional-topologies.html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/documentation/Transactional-topologies.html b/content/releases/0.10.1/documentation/Transactional-topologies.html index b95381e..96d1241 100644 --- a/content/releases/0.10.1/documentation/Transactional-topologies.html +++ b/content/releases/0.10.1/documentation/Transactional-topologies.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> @@ -211,7 +211,7 @@ <p>After bolt 1 finishes its portion of the processing, it will be idle until the rest of the bolts finish and the next batch can be emitted from the spout.</p> -<h3 id="design-3-storm-39-s-design">Design 3 (Storm's design)</h3> +<h3 id="design-3-storms-design">Design 3 (Storm's design)</h3> <p>A key realization is that not all the work for processing batches of tuples needs to be strongly ordered. For example, when computing a global count, there's two parts to the computation:</p> @@ -268,23 +268,23 @@ <span class="kt">int</span> <span class="n">_count</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">prepare</span><span class="o">(</span><span class="n">Map</span> <span class="n">conf</span><span class="o">,</span> <span class="n">TopologyContext</span> <span class="n">context</span><span class="o">,</span> <span class="n">BatchOutputCollector</span> <span class="n">collector</span><span class="o">,</span> <span class="n">Object</span> <span class="n">id</span><span class="o">)</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">prepare</span><span class="o">(</span><span class="n">Map</span> <span class="n">conf</span><span class="o">,</span> <span class="n">TopologyContext</span> <span class="n">context</span><span class="o">,</span> <span class="n">BatchOutputCollector</span> <span class="n">collector</span><span class="o">,</span> <span class="n">Object</span> <span class="n">id</span><span class="o">)</span> <span class="o">{</span> <span class="n">_collector</span> <span class="o">=</span> <span class="n">collector</span><span class="o">;</span> <span class="n">_id</span> <span class="o">=</span> <span class="n">id</span><span class="o">;</span> <span class="o">}</span> <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">execute</span><span class="o">(</span><span class="n">Tuple</span> <span class="n">tuple</span><span class="o">)</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">execute</span><span class="o">(</span><span class="n">Tuple</span> <span class="n">tuple</span><span class="o">)</span> <span class="o">{</span> <span class="n">_count</span><span class="o">++;</span> <span class="o">}</span> <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">finishBatch</span><span class="o">()</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">finishBatch</span><span class="o">()</span> <span class="o">{</span> <span class="n">_collector</span><span class="o">.</span><span class="na">emit</span><span class="o">(</span><span class="k">new</span> <span class="n">Values</span><span class="o">(</span><span class="n">_id</span><span class="o">,</span> <span class="n">_count</span><span class="o">));</span> <span class="o">}</span> <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">declareOutputFields</span><span class="o">(</span><span class="n">OutputFieldsDeclarer</span> <span class="n">declarer</span><span class="o">)</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">declareOutputFields</span><span class="o">(</span><span class="n">OutputFieldsDeclarer</span> <span class="n">declarer</span><span class="o">)</span> <span class="o">{</span> <span class="n">declarer</span><span class="o">.</span><span class="na">declare</span><span class="o">(</span><span class="k">new</span> <span class="n">Fields</span><span class="o">(</span><span class="s">"id"</span><span class="o">,</span> <span class="s">"count"</span><span class="o">));</span> <span class="o">}</span> <span class="o">}</span> @@ -313,18 +313,18 @@ <span class="kt">int</span> <span class="n">_sum</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">prepare</span><span class="o">(</span><span class="n">Map</span> <span class="n">conf</span><span class="o">,</span> <span class="n">TopologyContext</span> <span class="n">context</span><span class="o">,</span> <span class="n">BatchOutputCollector</span> <span class="n">collector</span><span class="o">,</span> <span class="n">TransactionAttempt</span> <span class="n">attempt</span><span class="o">)</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">prepare</span><span class="o">(</span><span class="n">Map</span> <span class="n">conf</span><span class="o">,</span> <span class="n">TopologyContext</span> <span class="n">context</span><span class="o">,</span> <span class="n">BatchOutputCollector</span> <span class="n">collector</span><span class="o">,</span> <span class="n">TransactionAttempt</span> <span class="n">attempt</span><span class="o">)</span> <span class="o">{</span> <span class="n">_collector</span> <span class="o">=</span> <span class="n">collector</span><span class="o">;</span> <span class="n">_attempt</span> <span class="o">=</span> <span class="n">attempt</span><span class="o">;</span> <span class="o">}</span> <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">execute</span><span class="o">(</span><span class="n">Tuple</span> <span class="n">tuple</span><span class="o">)</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">execute</span><span class="o">(</span><span class="n">Tuple</span> <span class="n">tuple</span><span class="o">)</span> <span class="o">{</span> <span class="n">_sum</span><span class="o">+=</span><span class="n">tuple</span><span class="o">.</span><span class="na">getInteger</span><span class="o">(</span><span class="mi">1</span><span class="o">);</span> <span class="o">}</span> <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">finishBatch</span><span class="o">()</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">finishBatch</span><span class="o">()</span> <span class="o">{</span> <span class="n">Value</span> <span class="n">val</span> <span class="o">=</span> <span class="n">DATABASE</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">GLOBAL_COUNT_KEY</span><span class="o">);</span> <span class="n">Value</span> <span class="n">newval</span><span class="o">;</span> <span class="k">if</span><span class="o">(</span><span class="n">val</span> <span class="o">==</span> <span class="kc">null</span> <span class="o">||</span> <span class="o">!</span><span class="n">val</span><span class="o">.</span><span class="na">txid</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="n">_attempt</span><span class="o">.</span><span class="na">getTransactionId</span><span class="o">()))</span> <span class="o">{</span> @@ -343,7 +343,7 @@ <span class="o">}</span> <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">declareOutputFields</span><span class="o">(</span><span class="n">OutputFieldsDeclarer</span> <span class="n">declarer</span><span class="o">)</span> <span class="o">{</span> + <span class="kd">public</span> <span class="kt">void</span> <span class="nf">declareOutputFields</span><span class="o">(</span><span class="n">OutputFieldsDeclarer</span> <span class="n">declarer</span><span class="o">)</span> <span class="o">{</span> <span class="n">declarer</span><span class="o">.</span><span class="na">declare</span><span class="o">(</span><span class="k">new</span> <span class="n">Fields</span><span class="o">(</span><span class="s">"id"</span><span class="o">,</span> <span class="s">"sum"</span><span class="o">));</span> <span class="o">}</span> <span class="o">}</span> @@ -421,7 +421,7 @@ <li><em>Number of active batches permissible at once:</em> You must set a limit to the number of batches that can be processed at once. You configure this using the "topology.max.spout.pending" config. If you don't set this config, it will default to 1.</li> </ol> -<h2 id="what-if-you-can-39-t-emit-the-same-batch-of-tuples-for-a-given-transaction-id">What if you can't emit the same batch of tuples for a given transaction id?</h2> +<h2 id="what-if-you-cant-emit-the-same-batch-of-tuples-for-a-given-transaction-id">What if you can't emit the same batch of tuples for a given transaction id?</h2> <p>So far the discussion around transactional topologies has assumed that you can always emit the exact same batch of tuples for the same transaction id. So what do you do if this is not possible?</p>
