[
https://issues.apache.org/jira/browse/STORM-329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14316446#comment-14316446
]
ASF GitHub Bot commented on STORM-329:
--------------------------------------
Github user clockfly commented on the pull request:
https://github.com/apache/storm/pull/268#issuecomment-73909944
@miguno,
I am sure you are aware that you can still send data to this channel even
if channel.isWritable return false.
check http://netty.io/4.0/api/io/netty/channel/Channel.html#isWritable()
boolean isWritable()
Returns true if and only if the I/O thread will perform the requested write
operation **immediately**. Any write requests made when this method returns
false are **queued** until the I/O thread is ready to process the queued write
requests.
doing or not doing isWritable check have nothing to do with message loss,
the isWritable check is purely used to optimize the performance for small
messages.
case1: If we have a large enough batch, then we will just flush to netty
internal queue. Netty will flush pending data in queueto wire when the wire is
not that busy.
case2: If we don't have a large enough batch, and the wire is busy, then
the Storm netty Client will wait for a while, buffer more messages in the
batch, and set a timer to flush later.
case3: If we don't have a large enough batch, and the wire is free, then
Storm Client will flush immediately, so that we have better latency.
> Add Option to Config Message handling strategy when connection timeout
> ----------------------------------------------------------------------
>
> Key: STORM-329
> URL: https://issues.apache.org/jira/browse/STORM-329
> Project: Apache Storm
> Issue Type: Improvement
> Affects Versions: 0.9.2-incubating
> Reporter: Sean Zhong
> Priority: Minor
> Labels: Netty
> Attachments: storm-329.patch, worker-kill-recover3.jpg
>
>
> This is to address a [concern brought
> up|https://github.com/apache/incubator-storm/pull/103#issuecomment-43632986]
> during the work at STORM-297:
> {quote}
> [~revans2] wrote: Your logic makes since to me on why these calls are
> blocking. My biggest concern around the blocking is in the case of a worker
> crashing. If a single worker crashes this can block the entire topology from
> executing until that worker comes back up. In some cases I can see that being
> something that you would want. In other cases I can see speed being the
> primary concern and some users would like to get partial data fast, rather
> then accurate data later.
> Could we make it configurable on a follow up JIRA where we can have a max
> limit to the buffering that is allowed, before we block, or throw data away
> (which is what zeromq does)?
> {quote}
> If some worker crash suddenly, how to handle the message which was supposed
> to be delivered to the worker?
> 1. Should we buffer all message infinitely?
> 2. Should we block the message sending until the connection is resumed?
> 3. Should we config a buffer limit, try to buffer the message first, if the
> limit is met, then block?
> 4. Should we neither block, nor buffer too much, but choose to drop the
> messages, and use the built-in storm failover mechanism?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
