Rob,
In 0.28 broker i was using ( i know you encourage me to use latest version
but i dont have that flexibility for what ever the reason), i was doing
some load testing, i have configured to use 512MB for heap and i stopped
all consumers to see if my broker can flow messages to disk at some point
but when it reaches 1.5 mil of messages broker got killed with oom error,
shouldn't it flow them to disk when it close to heap max(heap)? what is the
expected behavior?
2016-08-24 14:31:41,942 INFO [IoReceiver - /1host1:37996] (channel.create)
- [con:3(ANONYMOUS@/1host1:37996/default)/ch:0] CHN-1001 : Create
2016-08-24 14:31:43,380 ERROR [IoReceiver - /host1:52762] (server.Main) -
Uncaught exception, shutting down.
java.lang.OutOfMemoryError: Java heap space
at
org.apache.qpid.server.store.AbstractJDBCMessageStore.addContent(AbstractJDBCMessageStore.java:1625)
at
org.apache.qpid.server.store.AbstractJDBCMessageStore.access$500(AbstractJDBCMessageStore.java:63)
at
org.apache.qpid.server.store.AbstractJDBCMessageStore$StoredJDBCMessage.store(AbstractJDBCMessageStore.java:1938)
at
org.apache.qpid.server.store.AbstractJDBCMessageStore$StoredJDBCMessage.access$000(AbstractJDBCMessageStore.java:1785)
at
org.apache.qpid.server.store.AbstractJDBCMessageStore$JDBCTransaction.enqueueMessage(AbstractJDBCMessageStore.java:1733)
at
org.apache.qpid.server.txn.AsyncAutoCommitTransaction.enqueue(AsyncAutoCommitTransaction.java:255)
at
org.apache.qpid.server.exchange.AbstractExchange.send(AbstractExchange.java:499)
at
org.apache.qpid.server.protocol.v0_10.ServerSession.enqueue(ServerSession.java:234)
at
org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.messageTransfer(ServerSessionDelegate.java:332)
at
org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.messageTransfer(ServerSessionDelegate.java:74)
Thanks,
Ram
On Wed, Aug 24, 2016 at 10:45 AM, Rob Godfrey <[email protected]>
wrote:
> Hi Ram,
>
> So, firstly 0.28 is pretty old now - it was released at the beginning of
> June 2014... how the broker handles messages in memory has changed
> considerably since then. Historically the broker simply used soft
> references to manage message memory consumption - that is when the JVM GC
> decided that there was insufficient heap memory to store all the message
> data, it will evict messages from memory - and the broker will then need to
> reload them from disk. As such I don't think there are any settings in
> 0.28 which will affect when messages are flowed to disk (note the queue
> depth limit setting is merely a trigger for when warning messages are
> sent). There are separate settings which activate producer flow control
> (rather than flow to disk) but I don't think this is what you were asking
> for.
>
> Later versions of the broker (in particular post v6) take a more pro-active
> approach to managing memory, in v6 and above you can set the context
> variable "broker.flowToDiskThreshold" which will cause the broker to start
> flowing messages to disk once the total memory required by enqueued
> messages breaches this threshold. By default this value is set to 40% of
> the available direct memory (note v6 and above store the message data in
> direct rather than heap memory). Individual queues will be assigned
> "target" sizes based on their current size so that queues flow to disk in a
> somewhat fair manner.
>
> Hope this helps,
> Rob
>
> On 24 August 2016 at 17:52, rammohan ganapavarapu <[email protected]
> >
> wrote:
>
> > Any idea? i also wanted to have those settings at global level of
> > virtualhost as i create queues dynamically using client and i dont know
> > the queue name a head of time.
> >
> >
> > Thanks,
> > Ram
> >
> > On Tue, Aug 23, 2016 at 9:30 PM, rammohan ganapavarapu <
> > [email protected]> wrote:
> >
> > > Hi,
> > >
> > > I am using java broker 0.28 and would like to know how to configure a
> > > queue flow control in initial json config? for example i wanted to
> store
> > > 100MB of messages in memory and when it exceeds flow to disk. From
> docs i
> > > see xml version of config but wanted to see if we have any JSON format.
> > >
> > > Also if i don't define queue depth limit what is the default settings?
> it
> > > keep jvm heap size worth of messages in queue? what will happened if
> heap
> > > gets full does it flow those messages to disk and processes from disk?
> > >
> > > Finally how do i decide queue depth settings, based on available RAM or
> > > any other criteria?
> > >
> > > Thanks,
> > > RG
> > >
> > >
> >
>
