[ http://issues.apache.org/jira/browse/QPID-32?page=all ]
Marnie McCormack updated QPID-32:
---------------------------------
Description:
Currently all messages, including the headers and body, are held in memory in
the broker. This means that even with persistent queues, all messages must be
able to fit in the heap. Clearly that is a severe limitation.
This effectively means that there is an upper limit for the number of messages
a broker can handle, bound by the heap size available and defined by the size
of the messages in transit.
As we know that there are users who wish to combine high volume throughput with
large message size (100Mb) we can expect that this limitation will be
constraining fairly quickly, given finite available memory. We have had
requests for the broker to be able to handle 5 million messages at any one
time, with message size being frmo small (a few kb) to large (hundreds of Mb).
We need to introduce a paging algorithm into the broker such that it can page
messages in & out of memory as appropriate during transit. The broker needs to
be enhanced so that messages do not need to be held in memory at all times.
Ideally the redesign of this aspect will support flow-to-disk for
non-persistent messages when configurable thresholds are reached.
This may also affect transactional messages, persistent or not, as they are
also held in memory during transit and thus are affected by the same constraint.
The design of the solution for memory management is critical for the
performance of the broker and should be published and reviewed, as well as
having defined performance benchmarks to meet during testing. We need to be
very confident that we're putting in a solution which is quick and efficient.
We may also want to consider the configuration options available for memory
management - allowing our users to tune deployed brokers in a way appropriate
for their application profile i.e. if the app is sending large messages
infrequently then the fastest solution could potentially differ from the
appropriate model for an application sending mainly small messages and
occasional very large messages.
Also need to consider the impact of priority/qos settings on messages when
designing the solution.
The assignee for this work should expand the requirements in more detail once
analysis is complete, prior to completion of development.
Please add/amend the detail here if you have hands on experience of the
existing persistence code. Thanks.
was:
At present all persistent messages are held in memory when being acted upon by
the broker.
This effectively means that there is an upper limit for the number of messages
a broker can handle, bound by the heap size available and defined by the size
of the messages in transit.
As we know that there are users who wish to combine high volume throughput with
large message size (100Mb) we can expect that this limitation will be
constraining fairly quickly, given finite available memory. We have had
requests for the broker to be able to handle 5 million messages at any one
time, with message size being frmo small (a few kb) to large (hundreds of Mb).
We need to introduce a paging algorithm into the broker such that it can page
messages in & out of memory as appropriate during transit.
This may also affect transactional messages, persistent or not, as they are
also held in memory during transit and thus are affected by the same constraint.
The design of the solution for memory management is critical for the
performance of the broker and should be published and reviewed, as well as
having defined performance benchmarks to meet during testing. We need to be
very confident that we're putting in a solution which is quick and efficient.
We may also want to consider the configuration available for memory management
- allowing our users to tune deployed brokers in a way appropriate for their
application profile i.e. if the app is sending large messages infrequently then
the fastest solution could potentially differ from the appropriate model for an
application sending mainly small messages and occasional very large messages.
Also need to consider the impact of priority/qos settings on messages when
designing the solution.
The assignee for this work should expand the requirements in more detail once
analysis is complete, prior to completion of development.
Please add/amend the detail here if you have hands on experience of the
existing persistence code. Thanks.
Assignee: Robert Greig
MM 12th Oct 15:48 GMT - Added detail from QPID-17 and cancelled it. Assigned to
RG as he has work in progress.
> Amend persistence algorithm to introduce paging to physical memory
> ------------------------------------------------------------------
>
> Key: QPID-32
> URL: http://issues.apache.org/jira/browse/QPID-32
> Project: Qpid
> Issue Type: Improvement
> Components: Java Broker
> Reporter: Marnie McCormack
> Assigned To: Robert Greig
>
> Currently all messages, including the headers and body, are held in memory in
> the broker. This means that even with persistent queues, all messages must be
> able to fit in the heap. Clearly that is a severe limitation.
> This effectively means that there is an upper limit for the number of
> messages a broker can handle, bound by the heap size available and defined by
> the size of the messages in transit.
> As we know that there are users who wish to combine high volume throughput
> with large message size (100Mb) we can expect that this limitation will be
> constraining fairly quickly, given finite available memory. We have had
> requests for the broker to be able to handle 5 million messages at any one
> time, with message size being frmo small (a few kb) to large (hundreds of Mb).
> We need to introduce a paging algorithm into the broker such that it can page
> messages in & out of memory as appropriate during transit. The broker needs
> to be enhanced so that messages do not need to be held in memory at all
> times. Ideally the redesign of this aspect will support flow-to-disk for
> non-persistent messages when configurable thresholds are reached.
> This may also affect transactional messages, persistent or not, as they are
> also held in memory during transit and thus are affected by the same
> constraint.
> The design of the solution for memory management is critical for the
> performance of the broker and should be published and reviewed, as well as
> having defined performance benchmarks to meet during testing. We need to be
> very confident that we're putting in a solution which is quick and efficient.
> We may also want to consider the configuration options available for memory
> management - allowing our users to tune deployed brokers in a way appropriate
> for their application profile i.e. if the app is sending large messages
> infrequently then the fastest solution could potentially differ from the
> appropriate model for an application sending mainly small messages and
> occasional very large messages.
> Also need to consider the impact of priority/qos settings on messages when
> designing the solution.
> The assignee for this work should expand the requirements in more detail once
> analysis is complete, prior to completion of development.
> Please add/amend the detail here if you have hands on experience of the
> existing persistence code. Thanks.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
