On Thu, Jun 9, 2011 at 10:39 PM, Tharindu Mathew <[email protected]> wrote:
> > > On Thu, Jun 9, 2011 at 6:51 PM, Amila Suriarachchi <[email protected]> wrote: > >> >> >> On Tue, Jun 7, 2011 at 3:02 PM, Tharindu Mathew <[email protected]>wrote: >> >>> For high load systems, we can always recommend to increase the event >>> threshold. >> >> >> The practical approach is to define a maximum queue length for executor. >> If the system hit this limit system drop the messages effectively increasing >> the threshold automatically. >> >> If events are published within a time interval, that will work with stat >> module since there is no accumulation of memory. But if the events has to >> store in memory (for activity monitoring) then the amount of memory need >> will grow with increase of message rate. Again the only solution is to limit >> the size and drop if goes after some point. >> >> Therefore first, the data saving path of the BAM publishers should be >> optimized so that a rate at which it can work goes to the possible upper >> limit. After that there should be a proper mathematical model which make it >> operate at the highest point in a high load and drop messages after that >> without going to OOM. >> > There is also a limitation of not being able to use a thread pool to > dequeue from the queue. Since we look at the cumulative count, we need to > preserve order, thus using a single thread to dequeue. This is a big hit > IMO. > > This is something we can improve and fix at the BAM server side for a > future release. To look at time stamp (sent by the publisher) and get the > latest message when displaying on the dashboard, regardless of the order > that it came in. This will allow us to use a pool. > +1. In distributed computing there is a concept called logical clock for this kind of scenarios. you can keep some sequence number at the message generation point to preserve the order. thanks, Amila. > Lots of learning we can apply as we are writing the new BAM :) > >> >> thanks, >> Amila. >> >> >> >> >> >> >>> >>> When bursts occur, the default value of 20, is definitely not enough. >>> >>> >>> On Tue, Jun 7, 2011 at 2:47 PM, Amila Suriarachchi <[email protected]>wrote: >>> >>>> In a separate note the BAM publisher memory leak is not related to Qpid >>>> memory leak which we experience in long running. >>>> >>>> BAM publisher problem is mainly due to speed mismatching. i.e BAM >>>> publisher publish the messages to its internal thread pool queue in a much >>>> higher rate than it submit the message to the Qpid queue. This is becuase >>>> it >>>> take some time to publish the message to Qpid and it uses only one thread >>>> for that. (even with multiple threads it failing) >>>> >>>> In other words this OOM issue is caused by the tasks accumilated in the >>>> BAM thread pool. >>>> >>>> The only solution for this is to improve the performace of the path >>>> which saves the BAM messages to the database. >>>> >>>> Directly saving messages to the data base using jdbc from the BAM >>>> publisher it self is the fastest and hence can withhold for higher speeds. >>>> >>>> Anyway even that has an upper limit after which there will be an OOM >>>> error due to speed mismatch. >>>> >>>> thanks, >>>> Amila. >>>> >>>> On Mon, Jun 6, 2011 at 1:28 PM, Sumedha Rubasinghe <[email protected]>wrote: >>>> >>>>> This is the conclusion from the discussion happened little while ago. >>>>> >>>>> As Amila mentioned, we can switch to in-memory storage from Qpid. But >>>>> this has not been fully test. We will test this set up (in-memory) using >>>>> BAM >>>>> & then make the change permanent for 3.2.0. >>>>> >>>>> This way, each of the products will have a minimum impact from a code >>>>> change perspective. >>>>> >>>>> /sumedha >>>>> >>>>> >>>>> On Mon, Jun 6, 2011 at 10:50 AM, Tharindu Mathew <[email protected]>wrote: >>>>> >>>>>> >>>>>> >>>>>> On Mon, Jun 6, 2011 at 8:36 AM, Sanjiva Weerawarana <[email protected] >>>>>> > wrote: >>>>>> >>>>>>> On Sun, Jun 5, 2011 at 5:11 PM, Tharindu Mathew >>>>>>> <[email protected]>wrote: >>>>>>> >>>>>>>> That can be achieved by using a simple non-blocking Axis2 sender >>>>>>>>> type thing - which I believe is what Hiranya wrote a while ago. >>>>>>>>> >>>>>>>>> The BAM server needs to be set up in a clustered manner with an LB >>>>>>>>> etc. to handle the large number of messages it will receive when >>>>>>>>> Stratos is >>>>>>>>> running at full tilt (and have additional load balancing things like >>>>>>>>> DNS >>>>>>>>> round-robin). We can make that work easily. >>>>>>>>> >>>>>>>>> +1, for this model. The publisher that is used in Stratos is >>>>>>>> already done in a non blocking way. I feel it's better to go for the >>>>>>>> jdbc >>>>>>>> model. >>>>>>>> >>>>>>> >>>>>>> Tharindu what is the "JDBC model"?? We don't need persistence for >>>>>>> these messages, so where does JDBC come into the picture? >>>>>>> >>>>>> >>>>>> In the publisher, Hiranya wrote, we directly inject the event into the >>>>>> 'BAM' DB. This is what I referred to as the jdbc model. Instead of >>>>>> sending >>>>>> an event to BAM, and then inserting to the BAM DB. This option is also >>>>>> provided, which is what we use by default. >>>>>> >>>>>>> >>>>>>> Sanjiva. >>>>>>> -- >>>>>>> Sanjiva Weerawarana, Ph.D. >>>>>>> Founder, Chairman & CEO; WSO2, Inc.; http://wso2.com/ >>>>>>> email: [email protected]; phone: +94 11 763 9614; cell: +94 77 787 >>>>>>> 6880 | +1 650 265 8311 >>>>>>> blog: http://sanjiva.weerawarana.org/ >>>>>>> >>>>>>> Lean . Enterprise . Middleware >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Carbon-dev mailing list >>>>>>> [email protected] >>>>>>> http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Regards, >>>>>> >>>>>> Tharindu >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Carbon-dev mailing list >>>>>> [email protected] >>>>>> http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> Carbon-dev mailing list >>>>> [email protected] >>>>> http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> Carbon-dev mailing list >>>> [email protected] >>>> http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev >>>> >>>> >>> >>> >>> -- >>> Regards, >>> >>> Tharindu >>> >>> >>> _______________________________________________ >>> Carbon-dev mailing list >>> [email protected] >>> http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev >>> >>> >> >> _______________________________________________ >> Carbon-dev mailing list >> [email protected] >> http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev >> >> > > > -- > Regards, > > Tharindu > > > _______________________________________________ > Carbon-dev mailing list > [email protected] > http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev > >
_______________________________________________ Carbon-dev mailing list [email protected] http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev
