Re: java broker 6.0.2 OOM

Hi Ram, Hi Alex, The current RC for the Qpid for Java 6.0.5 release contains the fix for the issue you raised (QPID-7465 ). It would be appreciated if you had the time to test and verify that the RC in fact fixes your use case. See [1] for the

Re: java broker 6.0.2 OOM

Hi Ram, Hi Alex, I committed to fixes to the AMQP 0-10 code path under QPID-7465. This should reduce memory usage to a "normal" level and thus allow the heuristic that triggers flow to disk to work properly. This will undergo our automated testing in the coming days but maybe you want to alre

Re: java broker 6.0.2 OOM

Thank you!! On Wed, Oct 19, 2016 at 4:15 AM, Lorenz Quack wrote: > Hi Ram, Hi Alex, > > thanks for sending me your example code and logs. > I am now able to reproduce and I am investigating now. > It currently looks like there is a defect somewhere in the AMQP 0-10 code > path. > > I will keep y

Re: java broker 6.0.2 OOM

Hi Ram, Hi Alex, thanks for sending me your example code and logs. I am now able to reproduce and I am investigating now. It currently looks like there is a defect somewhere in the AMQP 0-10 code path. I will keep you posted. Kind regards, Lorenz On 18/10/16 22:13, rammohan ganapavarapu wr

Re: java broker 6.0.2 OOM

Java-Broker sustained longer with [Broker] BRK-1011 : Maximum Memory : Heap : 4,225,236,992 bytes Direct : 209,715,200 bytes -- View this message in context: http://qpid.2158936.n2.nabble.com/java-broker-6-0-2-OOM-tp7651831p7652140.html Sent from the Apache Qpid users mailing list archive at Na

Re: java broker 6.0.2 OOM

Lorenz, Alex and i work together, please find the config,logs for the tests he performed. Thanks, Ram On Tue, Oct 18, 2016 at 2:04 PM, alexk wrote: > Hi folks, > Here is the config config.json > > Here is the program qpid.java > <

Re: java broker 6.0.2 OOM

Hi folks, Here is the config config.json Here is the program qpid.java Here is the log of qpid java broker qpid.log

Re: java broker 6.0.2 OOM

Lorenz, Thanks for quick test, we also see its flowing to disk but direct mem is not leveling off, we will perform basic test with out application and share results. Do you have any recommendation on heap and direct mem settings, i was testing with heap: 768m and direct: 2304m. Ram On Tue, Oct

Re: java broker 6.0.2 OOM

Hello Ram, I just tried to reproduce your issue but was not successful. I ran a 6.0.2 broker (with default config) and trunk clients. I created 30 producers on their own connections and sent 10k persistent messages each in its own transaction. After hitting 634,583,040 B direct memory usage flow

Re: java broker 6.0.2 OOM

please let me know if u need any thing else. On Oct 17, 2016 11:02 AM, "rammohan ganapavarapu" wrote: > Lorenz, > > > Actually message size vary between ~ 1kb to 10k > > Thanks, > Ram > > On Mon, Oct 17, 2016 at 10:23 AM, rammohan ganapavarapu < > rammohanga...@gmail.com> wrote: > >> Lorenz, >>

Re: java broker 6.0.2 OOM

Lorenz, Actually message size vary between ~ 1kb to 10k Thanks, Ram On Mon, Oct 17, 2016 at 10:23 AM, rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Lorenz, > > Thanks for trying to help, Please find the below answers for your > questions. > > > Q:What is the type of your virtualhos

Re: java broker 6.0.2 OOM

Lorenz, Thanks for trying to help, Please find the below answers for your questions. Q:What is the type of your virtualhost (Derby, BDB, ...)? A: Derby ( i actually wanted to know your recomendation) Q: How large are your messages? Do they vary in size or all the same size? A: Message size is

Re: java broker 6.0.2 OOM

Hello Ram, This seems curious. Yes, the idea behind flow to disk is to prevent the broker from running out of direct memory. The broker does keep a certain representation of the message in memory but that should affect heap and not direct memory. I currently do not understand what is happenin

Re: java broker 6.0.2 OOM

Hi, I am confused with flow to disk context, when direct memory reaches flow to disk threshold, broker directly write to disk or it keep in both memory and disk? i was in the impression that flow to disk threshold to free up direct memory so that broker wont crash, isn't it? So i have 1.5gb direc

Re: java broker 6.0.2 OOM

So i ran the test few more times and it is happening every time, i was monitoring direct memory usage and looks like it ran out of direct memory. "maximumDirectMemorySize" : 2415919104, "usedDirectMemorySize" : 2414720896, Any thoughts guys? Ram On Thu, Oct 13, 2016 at 4:37 PM, rammohan g

Re: java broker 6.0.2 OOM

Guys, Not sure what i am doing wrong, i have set heap to 1gb and direct mem to 2gb after ~150k msgs queuedepth in the queue i am getting bellow error and broker is getting killed. Any suggestions? 2016-10-13 23:27:41,894 ERROR [IO-/10.16.1.34:46096] (o.a.q.s.Main) - Uncaught exception, shutting

Re: java broker 6.0.2 OOM

On 13 October 2016 at 19:54, rammohan ganapavarapu wrote: > Rob, > > Understood, we are doing negative testing like what will happened to broker > when all the consumers are down but producers are pumping messages, so i > was the in the impression that flow to disk threshold will avoid broker go

Re: java broker 6.0.2 OOM

Rob, Understood, we are doing negative testing like what will happened to broker when all the consumers are down but producers are pumping messages, so i was the in the impression that flow to disk threshold will avoid broker go bad because of OOM. So i have bumped up the heap and direct mem setti

Re: java broker 6.0.2 OOM

On 13 October 2016 at 17:36, rammohan ganapavarapu wrote: > Lorenz, > > Thank you for the link, so no matter how much heap you have you will hit > the hard limit at some point right?, i thought flow to disk will make > broker not to crash because of out of memory issue but looks like its not > th

Re: java broker 6.0.2 OOM

Lorenz, Thank you for the link, so no matter how much heap you have you will hit the hard limit at some point right?, i thought flow to disk will make broker not to crash because of out of memory issue but looks like its not the case. In my environment we will have dynamic number of producers and

Re: java broker 6.0.2 OOM

Here is the heap dump if any one interested. "HttpManagement-HTTP-121" prio=5 tid=121 RUNNABLE at java.lang.OutOfMemoryError.(OutOfMemoryError.java:48) at javax.security.auth.SubjectDomainCombiner.combine(SubjectDomainCombiner.java:241) Local Variable: java.security.ProtectionDomain

Re: java broker 6.0.2 OOM

Hello Ram, may I refer you to the relevant section of the documentation [1]. As explained there in more detail, the broker keeps a representation of each message in heap even when flowing the message to disk. Therefore the amount of JVM heap memory puts a hard limit on the number of message the