On Sun, Jun 5, 2011 at 8:20 AM, Sanjiva Weerawarana <[email protected]>wrote:
> I'd like to come for this discussion too please. > > On Azeez's proposal- he's not suggesting removing the Qpid based event > broker from products that have event publishing as a feature. Those are > G-Reg, DSS and ESB AFAIK. Those products will continue to use the broker. > > However all products have the BAM publisher to push information out to the > BAM server. The suggestion is to change so that ALL products (including the > 3 listed above) use a different BAM publishing codebase and not use the > event broker to publish BAM data. BAM publishing has NOTHING to do with > eventing- all that's needed is for the publication to happen on a low > priority threadpool and for the initial recording of information to be > non-blocking to the max. Furthermore the server should NEVER be able to > crash while trying to publish data to BAM. Ideally it should never lose any > messages but if the choice is between having the publishing stuff grow to a > point of threatening the health of the server vs. losing a few messages > (e.g. because the delivery threads can't keep up) then the answer is clear. > > That can be achieved by using a simple non-blocking Axis2 sender type thing > - which I believe is what Hiranya wrote a while ago. > > The BAM server needs to be set up in a clustered manner with an LB etc. to > handle the large number of messages it will receive when Stratos is running > at full tilt (and have additional load balancing things like DNS > round-robin). We can make that work easily. > > +1, for this model. The publisher that is used in Stratos is already done in a non blocking way. I feel it's better to go for the jdbc model. > Sanjiva. > > On Sun, Jun 5, 2011 at 7:53 AM, Amila Suriarachchi <[email protected]> wrote: > >> >> >> On Sun, Jun 5, 2011 at 7:41 AM, Srinath Perera <[email protected]> wrote: >> >>> Amila is it a option to ship current eventing impl, but with in-memory >>> or registry based delivery manager instad of Qpid based one? >>> >> >> yes. >> >> But we have not test this in memory implementation extensively with >> authorization, multi tenancy etc .. >> >> Lets meet on Monday and discuss what to do. >> >> thanks, >> Amila. >> >> >>> >>> --Srinath >>> >>> On Sat, Jun 4, 2011 at 5:35 PM, Amila Suriarachchi <[email protected]> >>> wrote: >>> > >>> > >>> > On Sat, Jun 4, 2011 at 4:13 PM, Senaka Fernando <[email protected]> >>> wrote: >>> >> >>> >> Hi all, >>> >> >>> >> Just a thought. >>> >> >>> >> On Sat, Jun 4, 2011 at 3:10 PM, Tharindu Mathew <[email protected]> >>> wrote: >>> >>> >>> >>> >>> >>> On Sat, Jun 4, 2011 at 11:50 AM, Afkham Azeez <[email protected]> >>> wrote: >>> >>>> >>> >>>> Sanjiva, Shankar & I had a discussion yesterday, and it seems that >>> >>>> having the event feature in most components is just overkill. Also, >>> in >>> >>>> Stratos, QPid causes the server to go out of memory when we do load >>> testing >>> >>>> because BAM tries to publish events, this will make all our services >>> >>>> unstable hence one of the most critical L1s. Sanjiva mentioned that >>> Hiranya >>> >>>> had developed a bit of lightweight code for publishing these BAM >>> events, and >>> >>>> we could use that. Apart from BAM, only deployment synchronizer uses >>> this >>> >>>> feature in the case of the services such as AS, ESB etc right? >>> >>>> >>> >>> G-Reg, DSS also use event components. IIRC, DSS for internal pub-sub >>> and >>> >>> G-Reg for resource subscriptions. >>> >> >>> >> We had a stable in-memory WS-Eventing pub/sub implementation which was >>> >> re-factored into this Qpid-based version. Can't we have an in-memory >>> version >>> >> (similar to what we had before), and use it in place of Qpid? Since >>> the API >>> >> changes wasn't so complicated, won't this be possible? >>> > >>> > hi Senaka, >>> > >>> > If you want to use the old event implementation you can use that. I is >>> up to >>> > you to decide. >>> > >>> > But if you want to use an inmemory registry based system (as in old >>> event >>> > implementaiton) only thing you need to do is to replace the delivery >>> manager >>> > with this in event-broker.xml >>> > >>> > <delivaryManager name="delivaryManager" >>> > >>> > >>> class="org.wso2.carbon.event.core.internal.delivary.inmemory.InMemoryDelivaryManagerFactory"> >>> > <minSpareThreads>25</minSpareThreads> >>> > <maxThreads>150</maxThreads> >>> > <maxQueuedRequests>100</maxQueuedRequests> >>> > <keepAliveTime>1000</keepAliveTime> >>> > <!--<matchingManager name="matchingManager"--> >>> > >>> > >>> <!--class="org.wso2.carbon.event.core.internal.delivary.inmemory.InMemoryMatchingManagerFactory"/>--> >>> > <matchingManager name="matchingManager" >>> > >>> > >>> class="org.wso2.carbon.event.core.internal.delivary.registry.RegistryMatchingManagerFactory"> >>> > <subscriptionStoragePath>event</subscriptionStoragePath> >>> > </matchingManager> >>> > </delivaryManager> >>> > >>> > But this is not properly tested since we do testing only with Qpid >>> based >>> > delivery manager and does not support hierachical subscriptions. >>> > >>> > The problem with the old event implementation was the problem you had >>> > mentioned here. If we want to have a jms based event model then whole >>> event >>> > broker implementation has to re wrote. This was the problem both >>> Srinath and >>> > me talked when we were asked to merge it with a jms based implemenation >>> and >>> > hence this new design. >>> > >>> > As you can see with the new model if some one want to publish event >>> registry >>> > notifications to jms based one (suppose we by default shift registry >>> based >>> > one) still it is a simple change of delivery manager. And we can have >>> > different jms based delivary managers if there are differences with >>> those >>> > brokers. >>> > >>> > thanks, >>> > Amila. >>> > >>> >> >>> >> Thanks, >>> >> Senaka. >>> >> >>> >>> >>> >>> We will lose some functionality of subscribing for topics and so >>> forth, >>> >>> but with some testing we should be able to avoid leaks OOM errors. >>> >>> >>> >>> >>> >>> The lightweight code was a tremendous improvement for high load >>> >>> scenarios. But for very high loads even this was failing. We had some >>> >>> improvements suggested and integrated(?), based on high and low >>> watermarks >>> >>> plus a persistent queue as well. IIRC, these improvements were never >>> >>> properly load tested and verified. >>> >>> Also, are we doing this for this release? This would involve quite a >>> bit >>> >>> of work, IMO. >>> >>>> >>> >>>> -- >>> >>>> Afkham Azeez >>> >>>> Director of Architecture; WSO2, Inc.; http://wso2.com >>> >>>> Member; Apache Software Foundation; http://www.apache.org/ >>> >>>> >>> >>>> email: [email protected] cell: +94 77 3320919 >>> >>>> blog: http://blog.afkham.org >>> >>>> twitter: http://twitter.com/afkham_azeez >>> >>>> linked-in: http://lk.linkedin.com/in/afkhamazeez >>> >>>> >>> >>>> Lean . Enterprise . Middleware >>> >>>> >>> >>>> _______________________________________________ >>> >>>> Carbon-dev mailing list >>> >>>> [email protected] >>> >>>> http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev >>> >>>> >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> Regards, >>> >>> >>> >>> Tharindu >>> >>> >>> >>> >>> >>> _______________________________________________ >>> >>> Carbon-dev mailing list >>> >>> [email protected] >>> >>> http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev >>> >>> >>> >> >>> >> >>> >> >>> >> -- >>> >> Senaka Fernando >>> >> Product Manager - WSO2 Governance Registry; >>> >> Associate Technical Lead; WSO2 Inc.; http://wso2.com >>> >> Member; Apache Software Foundation; http://apache.org >>> >> >>> >> E-mail: senaka AT wso2.com >>> >> P: +1 408 754 7388; ext: 51736; M: +94 77 322 1818 >>> >> Linked-In: http://linkedin.com/in/senakafernando >>> >> >>> >> Lean . Enterprise . Middleware >>> >> >>> >> >>> >> _______________________________________________ >>> >> Carbon-dev mailing list >>> >> [email protected] >>> >> http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev >>> >> >>> > >>> > >>> > _______________________________________________ >>> > Carbon-dev mailing list >>> > [email protected] >>> > http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev >>> > >>> > >>> >>> >>> >>> -- >>> ============================ >>> Srinath Perera, Ph.D. >>> Senior Software Architect, WSO2 Inc. >>> Visiting Faculty, University of Moratuwa >>> Member, Apache Software Foundation >>> Research Scientist, Lanka Software Foundation >>> Blog: http://srinathsview.blogspot.com/ >>> _______________________________________________ >>> Carbon-dev mailing list >>> [email protected] >>> http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev >>> >> >> >> _______________________________________________ >> Carbon-dev mailing list >> [email protected] >> http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev >> >> > > > -- > Sanjiva Weerawarana, Ph.D. > Founder, Chairman & CEO; WSO2, Inc.; http://wso2.com/ > email: [email protected]; phone: +94 11 763 9614; cell: +94 77 787 6880 | +1 > 650 265 8311 > blog: http://sanjiva.weerawarana.org/ > > Lean . Enterprise . Middleware > > _______________________________________________ > Carbon-dev mailing list > [email protected] > http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev > > -- Regards, Tharindu
_______________________________________________ Carbon-dev mailing list [email protected] http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev
