Hi Ram,
I am not too familiar with how 0.28 handles things but I am sure
it will always keep some sort of representation of the message in
memory even if a message was flown to disk so it will always
incur somememory overhead.
A quick back-of-the-envelope calculation:
500,000 kB memory / 1,500,000 messages = 0.3 kB memory / message
That does not seem unreasonable.
Kind regards,
Lorenz
On 24/08/16 22:55, rammohan ganapavarapu wrote:
Rob,
In 0.28 broker i was using ( i know you encourage me to use latest version
but i dont have that flexibility for what ever the reason), i was doing
some load testing, i have configured to use 512MB for heap and i stopped
all consumers to see if my broker can flow messages to disk at some point
but when it reaches 1.5 mil of messages broker got killed with oom error,
shouldn't it flow them to disk when it close to heap max(heap)? what is the
expected behavior?
2016-08-24 14:31:41,942 INFO [IoReceiver - /1host1:37996] (channel.create)
- [con:3(ANONYMOUS@/1host1:37996/default)/ch:0] CHN-1001 : Create
2016-08-24 14:31:43,380 ERROR [IoReceiver - /host1:52762] (server.Main) -
Uncaught exception, shutting down.
java.lang.OutOfMemoryError: Java heap space
at
org.apache.qpid.server.store.AbstractJDBCMessageStore.addContent(AbstractJDBCMessageStore.java:1625)
at
org.apache.qpid.server.store.AbstractJDBCMessageStore.access$500(AbstractJDBCMessageStore.java:63)
at
org.apache.qpid.server.store.AbstractJDBCMessageStore$StoredJDBCMessage.store(AbstractJDBCMessageStore.java:1938)
at
org.apache.qpid.server.store.AbstractJDBCMessageStore$StoredJDBCMessage.access$000(AbstractJDBCMessageStore.java:1785)
at
org.apache.qpid.server.store.AbstractJDBCMessageStore$JDBCTransaction.enqueueMessage(AbstractJDBCMessageStore.java:1733)
at
org.apache.qpid.server.txn.AsyncAutoCommitTransaction.enqueue(AsyncAutoCommitTransaction.java:255)
at
org.apache.qpid.server.exchange.AbstractExchange.send(AbstractExchange.java:499)
at
org.apache.qpid.server.protocol.v0_10.ServerSession.enqueue(ServerSession.java:234)
at
org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.messageTransfer(ServerSessionDelegate.java:332)
at
org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.messageTransfer(ServerSessionDelegate.java:74)
Thanks,
Ram
On Wed, Aug 24, 2016 at 10:45 AM, Rob Godfrey <[email protected]>
wrote:
Hi Ram,
So, firstly 0.28 is pretty old now - it was released at the beginning of
June 2014... how the broker handles messages in memory has changed
considerably since then. Historically the broker simply used soft
references to manage message memory consumption - that is when the JVM GC
decided that there was insufficient heap memory to store all the message
data, it will evict messages from memory - and the broker will then need to
reload them from disk. As such I don't think there are any settings in
0.28 which will affect when messages are flowed to disk (note the queue
depth limit setting is merely a trigger for when warning messages are
sent). There are separate settings which activate producer flow control
(rather than flow to disk) but I don't think this is what you were asking
for.
Later versions of the broker (in particular post v6) take a more pro-active
approach to managing memory, in v6 and above you can set the context
variable "broker.flowToDiskThreshold" which will cause the broker to start
flowing messages to disk once the total memory required by enqueued
messages breaches this threshold. By default this value is set to 40% of
the available direct memory (note v6 and above store the message data in
direct rather than heap memory). Individual queues will be assigned
"target" sizes based on their current size so that queues flow to disk in a
somewhat fair manner.
Hope this helps,
Rob
On 24 August 2016 at 17:52, rammohan ganapavarapu <[email protected]
wrote:
Any idea? i also wanted to have those settings at global level of
virtualhost as i create queues dynamically using client and i dont know
the queue name a head of time.
Thanks,
Ram
On Tue, Aug 23, 2016 at 9:30 PM, rammohan ganapavarapu <
[email protected]> wrote:
Hi,
I am using java broker 0.28 and would like to know how to configure a
queue flow control in initial json config? for example i wanted to
store
100MB of messages in memory and when it exceeds flow to disk. From
docs i
see xml version of config but wanted to see if we have any JSON format.
Also if i don't define queue depth limit what is the default settings?
it
keep jvm heap size worth of messages in queue? what will happened if
heap
gets full does it flow those messages to disk and processes from disk?
Finally how do i decide queue depth settings, based on available RAM or
any other criteria?
Thanks,
RG
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
