[
https://issues.apache.org/jira/browse/STORM-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14958909#comment-14958909
]
ASF GitHub Bot commented on STORM-350:
--------------------------------------
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/797#issuecomment-148388035
@HeartSaVioR
The original version of consumeBatch looked like this.
```
public void consumeBatch(EventHandler<Object> handler) {
consumeBatchToCursor(_barrier.getCursor(), handler);
}
```
where as consumeBatchWhenAvailable will wait for the cursor to be available.
```
_barrier.waitFor(nextSequence)
```
I am not sure what happened in the disruptor code between the old version
and the new version, but simply getting the cursor from the barrier does not
guarantee that it has been committed. It may still be being updated. Perhaps
this has always been a problem but we lose the race a lot more frequently with
the newer code.
When I originally wrote the disruptor upgrade code I copied/modified the
code so that we could do batching on the receiver side, not on the sender side.
This didn't help as much as I had hoped, but I also left in some code that
when the timeout happened it would try to do something similar to what
consumeBatch did, so we could read as much of the batch as possible. This also
resulted in the same issue, but it would only show up when a timeout happened
on read, meaning the queue was empty for 1 ms, and right at that moment
something was inserted into the queue. Surprisingly with word count on 16
workers it only takes about 10 mins to reproduce the issue. My tests were
pushing things as fast as possible so they never let any queue stay empty for
more then 1 ms.
The fix was to remove the special handling in the Timeout, and just say
there was nothing to process so loop around and try again.
> Update disruptor to latest version (3.2.1)
> ------------------------------------------
>
> Key: STORM-350
> URL: https://issues.apache.org/jira/browse/STORM-350
> Project: Apache Storm
> Issue Type: Dependency upgrade
> Components: storm-core
> Reporter: Boris Aksenov
> Assignee: Boris Aksenov
> Priority: Minor
> Fix For: 0.10.0
>
> Attachments:
> 20141117-0.9.3-rc1-3-worker-separate-1-spout-and-2-bolts-failing-tuples.png,
> 20141117-0.9.3-rc1-one-worker-failing-tuples.png,
> 20141117-0.9.3-rc1-three-workers-1-spout-3-bolts-failing-tuples.png,
> 20141118-0.9.3-branch-3-worker-separate-1-spout-and-2-bolts-ok.png,
> 20141118-0.9.3-branch-one-worker-ok.png,
> 20141118-0.9.3-branch-three-workers-1-spout-3-bolts-ok.png, Storm UI1.pdf,
> Storm UI2.pdf, storm-0.9.3-rc1-failing-tuples.png,
> storm-0_9_2-incubating-failing-tuples.png,
> storm-0_9_2-incubating-no-failing-tuples.png,
> storm-failed-tuples-multi-node.png, storm-multi-node-without-350.png
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
