[
https://issues.apache.org/jira/browse/STORM-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255795#comment-15255795
]
Basti Liu commented on STORM-1358:
----------------------------------
[~snisarg] Sorry for the late response. I am busy on the development of JStorm
recently. If you are interested in this issue, please feel free to assign it to
yourself. But just be noticed that it is a "phase 2" issue which might be
supposed to be merged after the migration of "phase 1". Besides it, some
current implementation shall be updated for thread safety guarantee when spout
is "multi-thread" mode, since ack/fail operation and nextTuple will be located
in two separated thread, e.g. MasterCoordinator of Trident, Kafka spout.....
> Porting JStorm multi-thread mode of spout to Storm
> --------------------------------------------------
>
> Key: STORM-1358
> URL: https://issues.apache.org/jira/browse/STORM-1358
> Project: Apache Storm
> Issue Type: New Feature
> Reporter: Basti Liu
> Assignee: Basti Liu
> Labels: jstorm-merger
>
> There are two modes of spout, "single-thread" and "multi-thread" in JStorm.
> The "single-thread" mode is simliar to Storm while the "multi-thread" mode
> separates the processing of ack/fail and nextTuple to two threads. It means
> we can stay in nextTuple for a long time without any side effect on ack/fail.
> Let's think about an example of kafka spout. We can initiate a consumer
> thread for kafka when initialization of spout. Then the comsumer starts to
> pull events from kafka and pulish the retreived events into a local queue. At
> meantime, nextTuple waits to read at this queue. If any available events,
> nextTuple will get notification faster and flush them to downstream. This
> model could probably introduce better performance compared with
> "single-thread" mode.
> For this mode, the max pending configuration of spout might not be useful as
> expectation. It depends on how long we stay in nextTuple. But backpressure is
> a good choice to resolve flow control problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
