Amend persistence algorithm to introduce paging to physical memory
-------------------------------------------------------------------
Key: QPID-32
URL: http://issues.apache.org/jira/browse/QPID-32
Project: Qpid
Issue Type: Improvement
Components: Java Broker
Reporter: Marnie McCormack
At present all persistent messages are held in memory when being acted upon by
the broker.
This effectively means that there is an upper limit for the number of messages
a broker can handle, bound by the heap size available and defined by the size
of the messages in transit.
As we know that there are users who wish to combine high volume throughput with
large message size (100Mb) we can expect that this limitation will be
constraining fairly quickly, given finite available memory.
We need to introduce a paging algorithm into the broker such that it can page
messages in & out of memory as appropriate during transit.
This may also affect transactional messages, persistent or not, as they are
also held in memory during transit and thus are affected by the same constraint.
The design of the solution for memory management is critical for the
performance of the broker and should be published and reviewed, as well as
having defined performance benchmarks to meet during testing. We need to be
very confident that we're putting in a solution which is quick and efficient.
We may also want to consider the configuration available for memory management
- allowing our users to tune deployed brokers in a way appropriate for their
application profile i.e. if the app is sending large messages infrequently then
the fastest solution could potentially differ from the appropriate model for an
application sending mainly small messages and occasional very large messages.
Also need to consider the impact of priority/qos settings on messages when
designing the solution.
The assignee for this work should expand the requirements in more detail once
analysis is complete, prior to completion of development.
Please add/amend the detail here if you have hands on experience of the
existing persistence code. Thanks.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
