http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/0.10.1/Multilang-protocol.html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/Multilang-protocol.html b/content/releases/0.10.1/Multilang-protocol.html index fe6f61c..8c3f8f6 100644 --- a/content/releases/0.10.1/Multilang-protocol.html +++ b/content/releases/0.10.1/Multilang-protocol.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> @@ -211,44 +211,44 @@ STDIN and STDOUT.</li> <ul> <li>STDIN: Setup info. This is a JSON object with the Storm configuration, a PID directory, and a topology context, like this:</li> </ul> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="nt">"conf"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="nt">"topology.message.timeout.secs"</span><span class="p">:</span><span class="w"> </span><span class="mi">3</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">etc</span><span class="w"> - </span><span class="err">}</span><span class="p">,</span><span class="w"> - </span><span class="nt">"pidDir"</span><span class="p">:</span><span class="w"> </span><span class="s2">"..."</span><span class="p">,</span><span class="w"> - </span><span class="nt">"context"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="nt">"task->component"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="nt">"1"</span><span class="p">:</span><span class="w"> </span><span class="s2">"example-spout"</span><span class="p">,</span><span class="w"> - </span><span class="nt">"2"</span><span class="p">:</span><span class="w"> </span><span class="s2">"__acker"</span><span class="p">,</span><span class="w"> - </span><span class="nt">"3"</span><span class="p">:</span><span class="w"> </span><span class="s2">"example-bolt1"</span><span class="p">,</span><span class="w"> - </span><span class="nt">"4"</span><span class="p">:</span><span class="w"> </span><span class="s2">"example-bolt2"</span><span class="w"> - </span><span class="p">},</span><span class="w"> - </span><span class="nt">"taskid"</span><span class="p">:</span><span class="w"> </span><span class="mi">3</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">Everything</span><span class="w"> </span><span class="err">below</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">line</span><span class="w"> </span><span class="err">is</span><span class="w"> </span><span class="err">only</span><span class="w"> </span><span class="err">available</span><span class="w"> </span><span class="err">in</span><span class="w"> </span><span class="err">Storm</span><span class="w"> </span><span class="err">0.10.0+</span><span class="w"> - </span><span class="nt">"componentid"</span><span class="p">:</span><span class="w"> </span><span class="s2">"example-bolt"</span><span class="w"> - </span><span class="s2">"stream->target->grouping"</span><span class="err">:</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="nt">"default"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="nt">"example-bolt2"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="nt">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"SHUFFLE"</span><span class="p">}}},</span><span class="w"> - </span><span class="nt">"streams"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"default"</span><span class="p">],</span><span class="w"> - </span><span class="nt">"stream->outputfields"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="nt">"default"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"word"</span><span class="p">]},</span><span class="w"> - </span><span class="nt">"source->stream->grouping"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="nt">"example-spout"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="nt">"default"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="nt">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"FIELDS"</span><span class="p">,</span><span class="w"> - </span><span class="nt">"fields"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"word"</span><span class="p">]</span><span class="w"> - </span><span class="p">}</span><span class="w"> - </span><span class="p">}</span><span class="w"> - </span><span class="p">}</span><span class="w"> - </span><span class="s2">"source->stream->fields"</span><span class="err">:</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="nt">"example-spout"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="nt">"default"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"word"</span><span class="p">]</span><span class="w"> - </span><span class="p">}</span><span class="w"> - </span><span class="p">}</span><span class="w"> - </span><span class="p">}</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + "conf": { + "topology.message.timeout.secs": 3, + // etc + }, + "pidDir": "...", + "context": { + "task->component": { + "1": "example-spout", + "2": "__acker", + "3": "example-bolt1", + "4": "example-bolt2" + }, + "taskid": 3, + // Everything below this line is only available in Storm 0.10.0+ + "componentid": "example-bolt" + "stream->target->grouping": { + "default": { + "example-bolt2": { + "type": "SHUFFLE"}}}, + "streams": ["default"], + "stream->outputfields": {"default": ["word"]}, + "source->stream->grouping": { + "example-spout": { + "default": { + "type": "FIELDS", + "fields": ["word"] + } + } + } + "source->stream->fields": { + "example-spout": { + "default": ["word"] + } + } + } +} +</code></pre></div> <p>Your script should create an empty file named with its PID in this directory. e.g. the PID is 1234, so an empty file named 1234 is created in the directory. This file lets the supervisor know the PID so it can shutdown the process later on.</p> @@ -277,47 +277,47 @@ to specify which fields are involved in a <code>FIELDS</code> grouping.</p> </ul> <p>"next" is the equivalent of ISpout's <code>nextTuple</code>. It looks like:</p> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"next"</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{"command": "next"} +</code></pre></div> <p>"ack" looks like:</p> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ack"</span><span class="p">,</span><span class="w"> </span><span class="nt">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1231231"</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{"command": "ack", "id": "1231231"} +</code></pre></div> <p>"fail" looks like:</p> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"fail"</span><span class="p">,</span><span class="w"> </span><span class="nt">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1231231"</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{"command": "fail", "id": "1231231"} +</code></pre></div> <ul> <li>STDOUT: The results of your spout for the previous command. This can be a sequence of emits and logs.</li> </ul> <p>An emit looks like:</p> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"emit"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">for</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">tuple.</span><span class="w"> </span><span class="err">Leave</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">out</span><span class="w"> </span><span class="err">for</span><span class="w"> </span><span class="err">an</span><span class="w"> </span><span class="err">unreliable</span><span class="w"> </span><span class="err">emit.</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">can</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">be</span><span class="w"> </span><span class="err">a</span><span class="w"> </span><span class="err">string</span><span class="w"> </span><span class="err">or</span><span class="w"> </span><span class="err">a</span><span class="w"> </span><span class="err">number.</span><span class="w"> - </span><span class="nt">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1231231"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">stream</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> </span><span class="err">was</span><span class="w"> </span><span class="err">emitted</span><span class="w"> </span><span class="err">to.</span><span class="w"> </span><span class="err">Leave</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">empty</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">emit</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">default</span><span class="w"> </span><span class="err">stream.</span><sp an class="w"> - </span><span class="nt">"stream"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">If</span><span class="w"> </span><span class="err">doing</span><span class="w"> </span><span class="err">an</span><span class="w"> </span><span class="err">emit</span><span class="w"> </span><span class="err">direct,</span><span class="w"> </span><span class="err">indicate</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">task</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">send</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">tuple</span><span class="w"> </span><span class="err">to</span><span class="w"> - </span><span class="nt">"task"</span><span class="p">:</span><span class="w"> </span><span class="mi">9</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">All</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">values</span><span class="w"> </span><span class="err">in</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> - </span><span class="nt">"tuple"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"field1"</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w"> </span><span class="mi">3</span><span class="p">]</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + "command": "emit", + // The id for the tuple. Leave this out for an unreliable emit. The id can + // be a string or a number. + "id": "1231231", + // The id of the stream this tuple was emitted to. Leave this empty to emit to default stream. + "stream": "1", + // If doing an emit direct, indicate the task to send the tuple to + "task": 9, + // All the values in this tuple + "tuple": ["field1", 2, 3] +} +</code></pre></div> <p>If not doing an emit direct, you will immediately receive the task ids to which the tuple was emitted on STDIN as a JSON array.</p> <p>A "log" will log a message in the worker log. It looks like:</p> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"log"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">message</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">log</span><span class="w"> - </span><span class="nt">"msg"</span><span class="p">:</span><span class="w"> </span><span class="s2">"hello world!"</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + "command": "log", + // the message to log + "msg": "hello world!" +} +</code></pre></div> <ul> <li>STDOUT: a "sync" command ends the sequence of emits and logs. It looks like:</li> </ul> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"sync"</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{"command": "sync"} +</code></pre></div> <p>After you sync, ShellSpout will not read your output until it sends another next, ack, or fail command.</p> <p>Note that, similarly to ISpout, all of the spouts in the worker will be locked up after a next, ack, or fail, until you sync. Also like ISpout, if you have no tuples to emit for a next, you should sleep for a small amount of time before syncing. ShellSpout will not automatically sleep for you.</p> @@ -329,34 +329,34 @@ be a sequence of emits and logs.</li> <ul> <li>STDIN: A tuple! This is a JSON encoded structure like this:</li> </ul> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">tuple's</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">-</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">is</span><span class="w"> </span><span class="err">a</span><span class="w"> </span><span class="err">string</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">support</span><span class="w"> </span><span class="err">languages</span><span class="w"> </span><span class="err">lacking</span><span class="w"> </span><span class="err">64-bit</span><span class="w"> </span><span class="err">precision</span><span class="w"> - </span><span class="nt">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"-6955786537413359385"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">component</span><span class="w"> </span><span class="err">that</span><span class="w"> </span><span class="err">created</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> - </span><span class="nt">"comp"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">stream</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> </span><span class="err">was</span><span class="w"> </span><span class="err">emitted</span><span class="w"> </span><span class="err">to</span><span class="w"> - </span><span class="nt">"stream"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">task</span><span class="w"> </span><span class="err">that</span><span class="w"> </span><span class="err">created</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> - </span><span class="nt">"task"</span><span class="p">:</span><span class="w"> </span><span class="mi">9</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">All</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">values</span><span class="w"> </span><span class="err">in</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> - </span><span class="nt">"tuple"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"snow white and the seven dwarfs"</span><span class="p">,</span><span class="w"> </span><span class="s2">"field2"</span><span class="p">,</span><span class="w"> </span><span class="mi">3</span><span class="p">]</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + // The tuple's id - this is a string to support languages lacking 64-bit precision + "id": "-6955786537413359385", + // The id of the component that created this tuple + "comp": "1", + // The id of the stream this tuple was emitted to + "stream": "1", + // The id of the task that created this tuple + "task": 9, + // All the values in this tuple + "tuple": ["snow white and the seven dwarfs", "field2", 3] +} +</code></pre></div> <ul> <li>STDOUT: An ack, fail, emit, or log. Emits look like:</li> </ul> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"emit"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">ids</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">tuples</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">output</span><span class="w"> </span><span class="err">tuples</span><span class="w"> </span><span class="err">should</span><span class="w"> </span><span class="err">be</span><span class="w"> </span><span class="err">anchored</span><span class="w"> </span><span class="err">to</span><span class="w"> - </span><span class="nt">"anchors"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"1231231"</span><span class="p">,</span><span class="w"> </span><span class="s2">"-234234234"</span><span class="p">],</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">The</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">stream</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> </span><span class="err">was</span><span class="w"> </span><span class="err">emitted</span><span class="w"> </span><span class="err">to.</span><span class="w"> </span><span class="err">Leave</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">empty</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">emit</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">default</span><span class="w"> </span><span class="err">stream.</span><sp an class="w"> - </span><span class="nt">"stream"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">If</span><span class="w"> </span><span class="err">doing</span><span class="w"> </span><span class="err">an</span><span class="w"> </span><span class="err">emit</span><span class="w"> </span><span class="err">direct,</span><span class="w"> </span><span class="err">indicate</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">task</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">send</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">tuple</span><span class="w"> </span><span class="err">to</span><span class="w"> - </span><span class="nt">"task"</span><span class="p">:</span><span class="w"> </span><span class="mi">9</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">All</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">values</span><span class="w"> </span><span class="err">in</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">tuple</span><span class="w"> - </span><span class="nt">"tuple"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"field1"</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w"> </span><span class="mi">3</span><span class="p">]</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + "command": "emit", + // The ids of the tuples this output tuples should be anchored to + "anchors": ["1231231", "-234234234"], + // The id of the stream this tuple was emitted to. Leave this empty to emit to default stream. + "stream": "1", + // If doing an emit direct, indicate the task to send the tuple to + "task": 9, + // All the values in this tuple + "tuple": ["field1", 2, 3] +} +</code></pre></div> <p>If not doing an emit direct, you will receive the task ids to which the tuple was emitted on STDIN as a JSON array. Note that, due to the asynchronous nature of the shell bolt protocol, when you read after @@ -366,26 +366,26 @@ receive the task id lists in the same order as their corresponding emits, however.</p> <p>An ack looks like:</p> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ack"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">tuple</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">ack</span><span class="w"> - </span><span class="nt">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"123123"</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + "command": "ack", + // the id of the tuple to ack + "id": "123123" +} +</code></pre></div> <p>A fail looks like:</p> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"fail"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">tuple</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">fail</span><span class="w"> - </span><span class="nt">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"123123"</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + "command": "fail", + // the id of the tuple to fail + "id": "123123" +} +</code></pre></div> <p>A "log" will log a message in the worker log. It looks like:</p> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="nt">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"log"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">message</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="err">log</span><span class="w"> - </span><span class="nt">"msg"</span><span class="p">:</span><span class="w"> </span><span class="s2">"hello world!"</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + "command": "log", + // the message to log + "msg": "hello world!" +} +</code></pre></div> <ul> <li>Note that, as of version 0.7.1, there is no longer any need for a shell bolt to 'sync'.</li> @@ -409,15 +409,15 @@ timeout during <code>next()</code>.</p> <p>Shell bolts are asynchronous, so a ShellBolt will send heartbeat tuples to its subprocess periodically. Heartbeat tuple looks like:</p> -<div class="highlight"><pre><code class="language-" data-lang=""><span class="p">{</span><span class="w"> - </span><span class="nt">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"-6955786537413359385"</span><span class="p">,</span><span class="w"> - </span><span class="nt">"comp"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1"</span><span class="p">,</span><span class="w"> - </span><span class="nt">"stream"</span><span class="p">:</span><span class="w"> </span><span class="s2">"__heartbeat"</span><span class="p">,</span><span class="w"> - </span><span class="err">//</span><span class="w"> </span><span class="err">this</span><span class="w"> </span><span class="err">shell</span><span class="w"> </span><span class="err">bolt's</span><span class="w"> </span><span class="err">system</span><span class="w"> </span><span class="err">task</span><span class="w"> </span><span class="err">id</span><span class="w"> - </span><span class="nt">"task"</span><span class="p">:</span><span class="w"> </span><span class="mi">-1</span><span class="p">,</span><span class="w"> - </span><span class="nt">"tuple"</span><span class="p">:</span><span class="w"> </span><span class="p">[]</span><span class="w"> -</span><span class="p">}</span><span class="w"> -</span></code></pre></div> +<div class="highlight"><pre><code class="language-" data-lang="">{ + "id": "-6955786537413359385", + "comp": "1", + "stream": "__heartbeat", + // this shell bolt's system task id + "task": -1, + "tuple": [] +} +</code></pre></div> <p>When subprocess receives heartbeat tuple, it must send a <code>sync</code> command back to ShellBolt.</p>
http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/0.10.1/Powered-By.html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/Powered-By.html b/content/releases/0.10.1/Powered-By.html index 043560d..e591d52 100644 --- a/content/releases/0.10.1/Powered-By.html +++ b/content/releases/0.10.1/Powered-By.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/0.10.1/Project-ideas.html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/Project-ideas.html b/content/releases/0.10.1/Project-ideas.html index c343c36..80aee28 100644 --- a/content/releases/0.10.1/Project-ideas.html +++ b/content/releases/0.10.1/Project-ideas.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/0.10.1/Rationale.html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/Rationale.html b/content/releases/0.10.1/Rationale.html index 9bc2796..71abccf 100644 --- a/content/releases/0.10.1/Rationale.html +++ b/content/releases/0.10.1/Rationale.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/0.10.1/Running-topologies-on-a-production-cluster.html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/Running-topologies-on-a-production-cluster.html b/content/releases/0.10.1/Running-topologies-on-a-production-cluster.html index 7274bac..457ee52 100644 --- a/content/releases/0.10.1/Running-topologies-on-a-production-cluster.html +++ b/content/releases/0.10.1/Running-topologies-on-a-production-cluster.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> http://git-wip-us.apache.org/repos/asf/storm-site/blob/a7a19afb/content/releases/0.10.1/SECURITY.html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/SECURITY.html b/content/releases/0.10.1/SECURITY.html index 35a4e54..0063c8a 100644 --- a/content/releases/0.10.1/SECURITY.html +++ b/content/releases/0.10.1/SECURITY.html @@ -82,18 +82,16 @@ - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - + <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> + <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> @@ -104,6 +102,8 @@ + + <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> @@ -144,7 +144,7 @@ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> </ul> </li> - <li><a href="/2017/09/15/storm105-released.html" id="news">News</a></li> + <li><a href="/2018/02/14/storm106-released.html" id="news">News</a></li> </ul> </nav> </div> @@ -281,7 +281,7 @@ curl -i --negotiate -u:anyUser -b ~/cookiejar.txt -c ~/cookiejar.txt <a href= <ol> <li>Firefox: Goto about:config and search for network.negotiate-auth.trusted-uris double-click to add value "<a href="http://storm-ui-hostname:8080";>http://storm-ui-hostname:8080</a>"</li> -<li>Google-chrome: start from command line with: google-chrome --auth-server-whitelist="<em>storm-ui-hostname" --auth-negotiate-delegate-whitelist="</em>storm-ui-hostname"<br></li> +<li>Google-chrome: start from command line with: google-chrome --auth-server-whitelist="*storm-ui-hostname" --auth-negotiate-delegate-whitelist="*storm-ui-hostname"<br></li> <li>IE: Configure trusted websites to include "storm-ui-hostname" and allow negotiation for that website </li> </ol> @@ -349,14 +349,14 @@ The supervisors and UI also need a principal to run as, but because they are out The following is an example of how to setup kerberos principals, but the details may vary depending on your KDC and OS.</p> <div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># Zookeeper (Will need one of these for each box in teh Zk ensamble)</span> -sudo kadmin.local -q <span class="s1">'addprinc zookeeper/zk1.example....@storm.example.com'</span> -sudo kadmin.local -q <span class="s2">"ktadd -k /tmp/zk.keytab zookeeper/zk1.example....@storm.example.com"</span> +<span class="nb">sudo </span>kadmin.local <span class="nt">-q</span> <span class="s1">'addprinc zookeeper/zk1.example....@storm.example.com'</span> +<span class="nb">sudo </span>kadmin.local <span class="nt">-q</span> <span class="s2">"ktadd -k /tmp/zk.keytab zookeeper/zk1.example....@storm.example.com"</span> <span class="c"># Nimbus and DRPC</span> -sudo kadmin.local -q <span class="s1">'addprinc storm/storm.example....@storm.example.com'</span> -sudo kadmin.local -q <span class="s2">"ktadd -k /tmp/storm.keytab storm/storm.example....@storm.example.com"</span> +<span class="nb">sudo </span>kadmin.local <span class="nt">-q</span> <span class="s1">'addprinc storm/storm.example....@storm.example.com'</span> +<span class="nb">sudo </span>kadmin.local <span class="nt">-q</span> <span class="s2">"ktadd -k /tmp/storm.keytab storm/storm.example....@storm.example.com"</span> <span class="c"># All UI logviewer and Supervisors</span> -sudo kadmin.local -q <span class="s1">'addprinc st...@storm.example.com'</span> -sudo kadmin.local -q <span class="s2">"ktadd -k /tmp/storm.keytab st...@storm.example.com"</span> +<span class="nb">sudo </span>kadmin.local <span class="nt">-q</span> <span class="s1">'addprinc st...@storm.example.com'</span> +<span class="nb">sudo </span>kadmin.local <span class="nt">-q</span> <span class="s2">"ktadd -k /tmp/storm.keytab st...@storm.example.com"</span> </code></pre></div> <p>be sure to distribute the keytab(s) to the appropriate boxes and set the FS permissions so that only the headless user running ZK, or storm has access to them.</p> @@ -540,7 +540,7 @@ Be aware that many of the features of this scheduler rely on storm authenticatio <h3 id="run-worker-processes-as-user-who-submitted-the-topology">Run worker processes as user who submitted the topology</h3> <p>By default storm runs workers as the user that is running the supervisor. This is not ideal for security. To make storm run the topologies as the user that launched them set.</p> -<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="s">supervisor.run.worker.as.user</span><span class="pi">:</span> <span class="s">true</span> +<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="s">supervisor.run.worker.as.user</span><span class="pi">:</span> <span class="no">true</span> </code></pre></div> <p>There are several files that go along with this that are needed to be configured properly to make storm secure.</p> @@ -563,15 +563,15 @@ to get a nimbus client as some other user and perform any nimbus action(i.e. kil The <code>ImpersonationAuthorizer</code> uses <code>nimbus.impersonation.acl</code> as the acl to authorize users. Following is a sample nimbus config for supporting impersonation:</p> <div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="s">nimbus.impersonation.authorizer</span><span class="pi">:</span> <span class="s">backtype.storm.security.auth.authorizer.ImpersonationAuthorizer</span> <span class="s">nimbus.impersonation.acl</span><span class="pi">:</span> - <span class="s">impersonating_user1</span><span class="pi">:</span> - <span class="s">hosts</span><span class="pi">:</span> + <span class="na">impersonating_user1</span><span class="pi">:</span> + <span class="na">hosts</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">comma separated list of hosts from which impersonating_user1 is allowed to impersonate other users</span><span class="pi">]</span> - <span class="s">groups</span><span class="pi">:</span> + <span class="na">groups</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">comma separated list of groups whose users impersonating_user1 is allowed to impersonate</span><span class="pi">]</span> - <span class="s">impersonating_user2</span><span class="pi">:</span> - <span class="s">hosts</span><span class="pi">:</span> + <span class="na">impersonating_user2</span><span class="pi">:</span> + <span class="na">hosts</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">comma separated list of hosts from which impersonating_user2 is allowed to impersonate other users</span><span class="pi">]</span> - <span class="s">groups</span><span class="pi">:</span> + <span class="na">groups</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">comma separated list of groups whose users impersonating_user2 is allowed to impersonate</span><span class="pi">]</span> </code></pre></div> <p>To support the oozie use case following config can be supplied:
