[
https://issues.apache.org/jira/browse/STORM-855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716669#comment-14716669
]
Robert Joseph Evans commented on STORM-855:
-------------------------------------------
[~mjsax] and [~sriharsha],
Micro batching is happening for two different reasons and two very distinct
ways. In trident the microbatch is logical only. The batch is used to divide
the tuples into distinct logical groups that can be committed to a state store
as a whole. The tuples still flow through storm one at a time. This feature,
however, is intended to provide a knob that allows you to give up some latency
for increased throughput. When profiling storm one of the things I notice is
that a very large percentage of the CPU is used context switching, and waking
up threads. This is great for very low latency processing, but provides a huge
overhead per tuple. If we can batch the tuples together in a single data
structure we amortize the overhead of those context switches, and we can get
increased throughput but at the cost of waiting for a batch to fill.
After talking to Matthias, I have been thinking about this feature a lot, and I
honestly don't know the best way to implement it. Flink uses a different
mechanism where the input buffer to each bolt equivalent will wait to wake-up
the processing piece until a timeout occurs or a specific number of tuples is
reached. This is essentially micro batching the input to each bolt
dynamically. I am starting to lean more towards that side for a few reasons.
First it allows for a very fine grained tuning, on a per bolt basis. It makes
knowing how congested a bolt or queue is much simpler. And the batching
efficiency does not degrade with the depth of the topology. In this case each
batch as it flows through a grouping is likely to be split into smaller and
smaller batches to the point that the batch has a very small number of tuples
in it. All of these feel like they would make STORM-815 much more likely to
work well.
That said this has an implementation that will give users a benefit now. How
about this I will try to come up with a quick prototype for disruptor queue
microbatching as an alternative, and we can see through actual benchmarks how
close they are to each other performance wise, and how much impact there is on
the code base.
> Add tuple batching
> ------------------
>
> Key: STORM-855
> URL: https://issues.apache.org/jira/browse/STORM-855
> Project: Apache Storm
> Issue Type: New Feature
> Reporter: Matthias J. Sax
> Assignee: Matthias J. Sax
> Priority: Minor
>
> In order to increase Storm's throughput, multiple tuples can be grouped
> together in a batch of tuples (ie, fat-tuple) and transfered from producer to
> consumer at once.
> The initial idea is taken from https://github.com/mjsax/aeolus. However, we
> aim to integrate this feature deep into the system (in contrast to building
> it on top), what has multiple advantages:
> - batching can be even more transparent to the user (eg, no extra
> direct-streams needed to mimic Storm's data distribution patterns)
> - fault-tolerance (anchoring/acking) can be done on a tuple granularity
> (not on a batch granularity, what leads to much more replayed tuples -- and
> result duplicates -- in case of failure)
> The aim is to extend TopologyBuilder interface with an additional parameter
> 'batch_size' to expose this feature to the user. Per default, batching will
> be disabled.
> This batching feature has pure tuple transport purpose, ie, tuple-by-tuple
> processing semantics are preserved. An output batch is assembled at the
> producer and completely disassembled at the consumer. The consumer output can
> be batched again, however, independent of batched or non-batched input. Thus,
> batches can be of different size for each producer-consumer pair.
> Furthermore, consumers can receive batches of different size from different
> producers (including regular non batched input).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
