On Sun, Jun 5, 2011 at 8:20 AM, Sanjiva Weerawarana <[email protected]>wrote:
> I'd like to come for this discussion too please. > > On Azeez's proposal- he's not suggesting removing the Qpid based event > broker from products that have event publishing as a feature. Those are > G-Reg, DSS and ESB AFAIK. Those products will continue to use the broker. > > However all products have the BAM publisher to push information out to the > BAM server. The suggestion is to change so that ALL products (including the > 3 listed above) use a different BAM publishing codebase and not use the > event broker to publish BAM data. BAM publishing has NOTHING to do with > eventing- all that's needed is for the publication to happen on a low > priority threadpool and for the initial recording of information to be > non-blocking to the max. Furthermore the server should NEVER be able to > crash while trying to publish data to BAM. Ideally it should never lose any > messages but if the choice is between having the publishing stuff grow to a > point of threatening the health of the server vs. losing a few messages > (e.g. because the delivery threads can't keep up) then the answer is clear. > +1. So this change has nothing to do with event implementation. Only BAM publishers have to directly push messages to BAM MessageReceiver instead of going through and event broker (which is obviously in efficient as you have pointed out). thanks, Amila. > > That can be achieved by using a simple non-blocking Axis2 sender type thing > - which I believe is what Hiranya wrote a while ago. > > The BAM server needs to be set up in a clustered manner with an LB etc. to > handle the large number of messages it will receive when Stratos is running > at full tilt (and have additional load balancing things like DNS > round-robin). We can make that work easily. > > Sanjiva. > > On Sun, Jun 5, 2011 at 7:53 AM, Amila Suriarachchi <[email protected]> wrote: > >> >> >> On Sun, Jun 5, 2011 at 7:41 AM, Srinath Perera <[email protected]> wrote: >> >>> Amila is it a option to ship current eventing impl, but with in-memory >>> or registry based delivery manager instad of Qpid based one? >>> >> >> yes. >> >> But we have not test this in memory implementation extensively with >> authorization, multi tenancy etc .. >> >> Lets meet on Monday and discuss what to do. >> >> thanks, >> Amila. >> >> >>> >>> --Srinath >>> >>> On Sat, Jun 4, 2011 at 5:35 PM, Amila Suriarachchi <[email protected]> >>> wrote: >>> > >>> > >>> > On Sat, Jun 4, 2011 at 4:13 PM, Senaka Fernando <[email protected]> >>> wrote: >>> >> >>> >> Hi all, >>> >> >>> >> Just a thought. >>> >> >>> >> On Sat, Jun 4, 2011 at 3:10 PM, Tharindu Mathew <[email protected]> >>> wrote: >>> >>> >>> >>> >>> >>> On Sat, Jun 4, 2011 at 11:50 AM, Afkham Azeez <[email protected]> >>> wrote: >>> >>>> >>> >>>> Sanjiva, Shankar & I had a discussion yesterday, and it seems that >>> >>>> having the event feature in most components is just overkill. Also, >>> in >>> >>>> Stratos, QPid causes the server to go out of memory when we do load >>> testing >>> >>>> because BAM tries to publish events, this will make all our services >>> >>>> unstable hence one of the most critical L1s. Sanjiva mentioned that >>> Hiranya >>> >>>> had developed a bit of lightweight code for publishing these BAM >>> events, and >>> >>>> we could use that. Apart from BAM, only deployment synchronizer uses >>> this >>> >>>> feature in the case of the services such as AS, ESB etc right? >>> >>>> >>> >>> G-Reg, DSS also use event components. IIRC, DSS for internal pub-sub >>> and >>> >>> G-Reg for resource subscriptions. >>> >> >>> >> We had a stable in-memory WS-Eventing pub/sub implementation which was >>> >> re-factored into this Qpid-based version. Can't we have an in-memory >>> version >>> >> (similar to what we had before), and use it in place of Qpid? Since >>> the API >>> >> changes wasn't so complicated, won't this be possible? >>> > >>> > hi Senaka, >>> > >>> > If you want to use the old event implementation you can use that. I is >>> up to >>> > you to decide. >>> > >>> > But if you want to use an inmemory registry based system (as in old >>> event >>> > implementaiton) only thing you need to do is to replace the delivery >>> manager >>> > with this in event-broker.xml >>> > >>> > <delivaryManager name="delivaryManager" >>> > >>> > >>> class="org.wso2.carbon.event.core.internal.delivary.inmemory.InMemoryDelivaryManagerFactory"> >>> > <minSpareThreads>25</minSpareThreads> >>> > <maxThreads>150</maxThreads> >>> > <maxQueuedRequests>100</maxQueuedRequests> >>> > <keepAliveTime>1000</keepAliveTime> >>> > <!--<matchingManager name="matchingManager"--> >>> > >>> > >>> <!--class="org.wso2.carbon.event.core.internal.delivary.inmemory.InMemoryMatchingManagerFactory"/>--> >>> > <matchingManager name="matchingManager" >>> > >>> > >>> class="org.wso2.carbon.event.core.internal.delivary.registry.RegistryMatchingManagerFactory"> >>> > <subscriptionStoragePath>event</subscriptionStoragePath> >>> > </matchingManager> >>> > </delivaryManager> >>> > >>> > But this is not properly tested since we do testing only with Qpid >>> based >>> > delivery manager and does not support hierachical subscriptions. >>> > >>> > The problem with the old event implementation was the problem you had >>> > mentioned here. If we want to have a jms based event model then whole >>> event >>> > broker implementation has to re wrote. This was the problem both >>> Srinath and >>> > me talked when we were asked to merge it with a jms based implemenation >>> and >>> > hence this new design. >>> > >>> > As you can see with the new model if some one want to publish event >>> registry >>> > notifications to jms based one (suppose we by default shift registry >>> based >>> > one) still it is a simple change of delivery manager. And we can have >>> > different jms based delivary managers if there are differences with >>> those >>> > brokers. >>> > >>> > thanks, >>> > Amila. >>> > >>> >> >>> >> Thanks, >>> >> Senaka. >>> >> >>> >>> >>> >>> We will lose some functionality of subscribing for topics and so >>> forth, >>> >>> but with some testing we should be able to avoid leaks OOM errors. >>> >>> >>> >>> >>> >>> The lightweight code was a tremendous improvement for high load >>> >>> scenarios. But for very high loads even this was failing. We had some >>> >>> improvements suggested and integrated(?), based on high and low >>> watermarks >>> >>> plus a persistent queue as well. IIRC, these improvements were never >>> >>> properly load tested and verified. >>> >>> Also, are we doing this for this release? This would involve quite a >>> bit >>> >>> of work, IMO. >>> >>>> >>> >>>> -- >>> >>>> Afkham Azeez >>> >>>> Director of Architecture; WSO2, Inc.; http://wso2.com >>> >>>> Member; Apache Software Foundation; http://www.apache.org/ >>> >>>> >>> >>>> email: [email protected] cell: +94 77 3320919 >>> >>>> blog: http://blog.afkham.org >>> >>>> twitter: http://twitter.com/afkham_azeez >>> >>>> linked-in: http://lk.linkedin.com/in/afkhamazeez >>> >>>> >>> >>>> Lean . Enterprise . Middleware >>> >>>> >>> >>>> _______________________________________________ >>> >>>> Carbon-dev mailing list >>> >>>> [email protected] >>> >>>> http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev >>> >>>> >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> Regards, >>> >>> >>> >>> Tharindu >>> >>> >>> >>> >>> >>> _______________________________________________ >>> >>> Carbon-dev mailing list >>> >>> [email protected] >>> >>> http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev >>> >>> >>> >> >>> >> >>> >> >>> >> -- >>> >> Senaka Fernando >>> >> Product Manager - WSO2 Governance Registry; >>> >> Associate Technical Lead; WSO2 Inc.; http://wso2.com >>> >> Member; Apache Software Foundation; http://apache.org >>> >> >>> >> E-mail: senaka AT wso2.com >>> >> P: +1 408 754 7388; ext: 51736; M: +94 77 322 1818 >>> >> Linked-In: http://linkedin.com/in/senakafernando >>> >> >>> >> Lean . Enterprise . Middleware >>> >> >>> >> >>> >> _______________________________________________ >>> >> Carbon-dev mailing list >>> >> [email protected] >>> >> http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev >>> >> >>> > >>> > >>> > _______________________________________________ >>> > Carbon-dev mailing list >>> > [email protected] >>> > http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev >>> > >>> > >>> >>> >>> >>> -- >>> ============================ >>> Srinath Perera, Ph.D. >>> Senior Software Architect, WSO2 Inc. >>> Visiting Faculty, University of Moratuwa >>> Member, Apache Software Foundation >>> Research Scientist, Lanka Software Foundation >>> Blog: http://srinathsview.blogspot.com/ >>> _______________________________________________ >>> Carbon-dev mailing list >>> [email protected] >>> http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev >>> >> >> >> _______________________________________________ >> Carbon-dev mailing list >> [email protected] >> http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev >> >> > > > -- > Sanjiva Weerawarana, Ph.D. > Founder, Chairman & CEO; WSO2, Inc.; http://wso2.com/ > email: [email protected]; phone: +94 11 763 9614; cell: +94 77 787 6880 | +1 > 650 265 8311 > blog: http://sanjiva.weerawarana.org/ > > Lean . Enterprise . Middleware > > _______________________________________________ > Carbon-dev mailing list > [email protected] > http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev > >
_______________________________________________ Carbon-dev mailing list [email protected] http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev
