Basti Liu created STORM-1358:
--------------------------------
Summary: Porting JStorm multi-thread mode of spout to Storm
Key: STORM-1358
URL: https://issues.apache.org/jira/browse/STORM-1358
Project: Apache Storm
Issue Type: New Feature
Reporter: Basti Liu
Assignee: Basti Liu
There are two modes of spout, "single-thread" and "multi-thread" in JStorm. The
"single-thread" mode is simliar to Storm while the "multi-thread" mode
separates the processing of ack/fail and nextTuple to two threads. It means we
can stay in nextTuple for a long time without any impact on ack/fail.
Let's think about an example of kafka spout. We can initiate a consumer thread
for kafka when initialization of spout. Then the comsumer starts to pull events
from kafka and pulish the retreived events into a local queue. At meantime,
nextTuple waits to read at this queue. If any available events, nextTuple will
get notification faster and flush them to downstream. This model could probably
introduce better performance compared with "single-thread" mode.
For this mode, the max pending configuration of spout might not be useful as
expectation. It depends on how long we stay in nextTuple. But backpressure is a
good choice to resolve flow control problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
