Hi Everyone,
As a follow on to my e-mail last week introducing the core broker
prototype that Hiram and I have been working on, I wanted to spin
up a
thread on the flow control model that we're using.
I'd be interested to hear in your thoughts on current shortcomings
associated with flow control / memory management in 5.3 so we can
make
sure that the use cases are covered. Beyond that any additional
input on
the design or implementation would be great ... are we on the right
track?
Cheers,
Colin
The text below is taken straight from the webgen in the project, my
apologies if it's a little verbose!
As a reminder the bits can be found at:
https://svn.apache.org/repos/asf/activemq/sandbox/activemq-flow
The activemq-flow package is meant to be a standalone module that
deals
generically with Resources and Flow's of elements that flow through
and
between them. The current implementation is designed with the
following
goals in mind:
* SIMPLE: Want a fairly simple and consistent model for controlling
flow of messages and other data in the system to control memory and
disk
space. The module must be able to handle fan-in/fan-out as well as
simpler 1 to 1 cases.
* PERFORMANT: The flow control mechanism must be performant and
should not introduce much overhead in cases where downstream
resources
are able to keep up.
* MODULARIZED: The module should be independent generic and
reusable.
* FAIRNESS: We should be able to provide better fairness. If I've
got several producers putting messages on a queue, the flow
controller
should not prefer one source over the other (unless configured to do
so)
* VISIBILITY: With a unified model in place we can instrument it to
provide visibility in the product (e.g. a visual graph of flows in
the
system). When a customer says that they are not using PERSISTENT
messages yet we see 1000msgs/sec flowing through the recovery log....
* ADMINISTRATION: We can explore the possibility of
administratively
limiting message flows. E.g. I've done my production stress testing
and
can successfully handle my anticipated load of 4000 msgs/sec on
topic1
... I'd prefer to avoid the case where publishers go berserk and
overload my backend with messages).
* POLICIES: We should be able to better instrument general flow
control policies. E.g. I want to tune for latency or throughput. If a
subscriber gets behind, I'd like the policy for messages on topic1
to be
that I drop the oldest messages instead of initiating flow control.
The Basics:
Each resource creates a FlowController for each of it's Flows which
is
assigned a corresponding FlowLimiter. As elements (e.g. messages)
pass
from one resource to another they are passed through the downstream
resource's FlowController which updates its Limiter. If propagation
of
an element from one resource to another causes the downstream
limiter to
become throttled the associated FlowController will block the source
of
the element. The flow module is used heavily by the rest of the core
for
memory and disk management.
* Memory Management: Memory is managed based on the resources in
play -- the usage is computed by summing of the space allocated to
each
of the resources' limiters. This strategy intentionally avoids a
centralized memory limit which leads to complicated logic to track
when
a centralized limiter needs to be decremented and avoids contention
between multiple resources/threads accessing the limiter and also
reduces the potential for memory limiter related deadlocks. However,
it
should be noted that this approach doesn't preclude implementing
centralized limiters in the future.
* Flow Control: As messages propagate from one resource A to
another
B, then if A overflows B's limit, B will block A and A can't release
it's limiter space until B unblocks it. This allowance for overflow
into
downstream resources is a key concept in flow control performance and
ease of use. Provided that the upstream resource has already
accounted
for the message's memory it can freely overflow any downstream
limiter
providing it reserves space from elements that caused overflow.
* Threading Model: Note that as a message propagates from A to B,
that the general contract is that A won't release it's memory if B
blocks it during the course of dispatch. This means that it is not
safe
to perform a thread handoff during dispatch between two resources
since
the thread dispatching A relies on the message making it to B (so
that B
can block it) prior to A completing dispatch.
* Management/Visibility: Another intended use of the activemq-flow
module is to assist in visibility e.g. provide an underlying map of
resources that can be exposed via tooling to see the relationships
between sources and sinks of messages and to find bottlenecks ...
this
aspect has been downplayed for now as we have been focusing more on
the
queueing/memory management model in the prototype, but eventually the
flow package itself will provide a handy way of providing visibility
in
the system particularly in terms of finding performance bottlenecks.
FlowResource (FlowSink and FlowSource): A container for
FlowControllers
providing some lifecycle related logic. The base resource class
handles
interaction/registration with the FlowManager (below).
FlowManager: Registry for Flow's and FlowResources. The manager will
provide some hooks into system visibility. As mentioned above this
aspect has been downplayed somewhat for the present time.
FlowController: Wraps a FlowLimiter and actually implements common
basic
block/resume logic between FlowControllers.
FlowLimiter: Defines the limits enforced by a FlowController.
Currently
the package has size based limiter implementations, but eventually
should also support other common limiter types such as rate based
limiters. The limiter's are also extended at other points in the
broker
(for example implementing a protocol based WindowLimiter). It is also
likely that we would want to introduce CompositeLimiters to combine
various limiter types.
Flow: The concept of a flow is not used very heavily right now. But a
Flow defines the stream of elements that can be blocked. In general
the
prototype creates a single flow per resource, but in the future a
source
may break it's elements down into more granular flows on which
downstream sinks may block it. One case where this is anticipated as
being useful is in networks of brokers where-in it may be desirable
to
partition messages into more granular flows (e.g based on producer or
destination) to avoid blocking the broker-broker connection
uncessarily).
