[
https://issues.apache.org/jira/browse/STORM-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Basti Liu updated STORM-1358:
-----------------------------
Description:
There are two modes of spout, "single-thread" and "multi-thread" in JStorm. The
"single-thread" mode is simliar to Storm while the "multi-thread" mode
separates the processing of ack/fail and nextTuple to two threads. It means we
can stay in nextTuple for a long time without any side effect on ack/fail.
Let's think about an example of kafka spout. We can initiate a consumer thread
for kafka when initialization of spout. Then the comsumer starts to pull events
from kafka and pulish the retreived events into a local queue. At meantime,
nextTuple waits to read at this queue. If any available events, nextTuple will
get notification faster and flush them to downstream. This model could probably
introduce better performance compared with "single-thread" mode.
For this mode, the max pending configuration of spout might not be useful as
expectation. It depends on how long we stay in nextTuple. But backpressure is a
good choice to resolve flow control problem.
was:
There are two modes of spout, "single-thread" and "multi-thread" in JStorm. The
"single-thread" mode is simliar to Storm while the "multi-thread" mode
separates the processing of ack/fail and nextTuple to two threads. It means we
can stay in nextTuple for a long time without any impact on ack/fail.
Let's think about an example of kafka spout. We can initiate a consumer thread
for kafka when initialization of spout. Then the comsumer starts to pull events
from kafka and pulish the retreived events into a local queue. At meantime,
nextTuple waits to read at this queue. If any available events, nextTuple will
get notification faster and flush them to downstream. This model could probably
introduce better performance compared with "single-thread" mode.
For this mode, the max pending configuration of spout might not be useful as
expectation. It depends on how long we stay in nextTuple. But backpressure is a
good choice to resolve flow control problem.
> Porting JStorm multi-thread mode of spout to Storm
> --------------------------------------------------
>
> Key: STORM-1358
> URL: https://issues.apache.org/jira/browse/STORM-1358
> Project: Apache Storm
> Issue Type: New Feature
> Reporter: Basti Liu
> Assignee: Basti Liu
> Labels: jstorm-merger
>
> There are two modes of spout, "single-thread" and "multi-thread" in JStorm.
> The "single-thread" mode is simliar to Storm while the "multi-thread" mode
> separates the processing of ack/fail and nextTuple to two threads. It means
> we can stay in nextTuple for a long time without any side effect on ack/fail.
> Let's think about an example of kafka spout. We can initiate a consumer
> thread for kafka when initialization of spout. Then the comsumer starts to
> pull events from kafka and pulish the retreived events into a local queue. At
> meantime, nextTuple waits to read at this queue. If any available events,
> nextTuple will get notification faster and flush them to downstream. This
> model could probably introduce better performance compared with
> "single-thread" mode.
> For this mode, the max pending configuration of spout might not be useful as
> expectation. It depends on how long we stay in nextTuple. But backpressure is
> a good choice to resolve flow control problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
