Just for "fun", I wrote a performance test program that does exactly this. Here is the result I got (single-node system using TCP transport, no PBE, totally 18000 transactions):
OpenSAF 5.0: 971.5 transactions per second OpenSAF 5.1.FC: 29.5 transactions per second That is what I would call a performance regression! Note that due to the throttling we have introduced in OpenSAF 5.1.FC, the theoretical maximum is 33.3 transactions per second. No matter how much we optimize the code or how powerful the node is, we can never go beyond this limit. I knew already before I ran the test that the result would be close to this number... / Anders Widell On 09/05/2016 04:50 PM, [email protected] wrote: > Again, that's a test of a performance metric (throughput) that I claim is not > relevant for the IMM service. > Response time per transaction does have some relevance though. > > The IMM is not intended to be a high throughput database. > The throughput depends very much on the configuration PBE/not-PBE/2-PBE. > > During sync there is zero throughput, which is why the docs say that a sync > should never take longer than 60 seconds, > which is what defines a limit on IMM database size. > > (not to mention headless where there is zero throughput indefinitely and > there is no service availability guarantee indefinitely == disabled SA). > > The typical use case is either one or a very few operators executing single > commands. > Or an upgrade campaign that is more or less complex in steps. > > Neither has any realtime requirements. > > They may have some implicit application specific upper time requirements > expressed as timeouts. > > If single operator commanded CCB takes minutes to commit, then there is > likely a problem, because the operator may have > difficulty in performing a task at the human level. But CCB commit involves > callbacks to applications (implementers). There is a timeout > for that calback, but it is configurable... > > /AndersBj > > > > >> ----Ursprungligt meddelande---- >> Från : [email protected] >> Datum : 2016-09-05 - 15:49 (CEST) >> Till : [email protected], [email protected], >> [email protected], [email protected], >> [email protected], [email protected] >> Ämne : Re: [devel] Immnd: maximum Ccbs limit 10000 has been reached very >> quickly >> >> How about this use case: a performance test program that measures the >> number of CCBs per second that IMM can execute? >> >> / Anders Widell >> >> On 09/05/2016 03:17 PM, [email protected] wrote: >>> Hi, >>> >>> Neither of those examples (1) A shell script calling immcfg command from a >>> loop; (2) A buggy application; are real use cases. >>> For (2) it is obvious. >>> >>> For (1), it is at best a very sloppy implementation of some use case. >>> The intention of that shell script is to accomplish what amounts to one >>> configuration change, but it is doing so using thousands of separate ccbs! >>> Say that the script itself crashes after it has completed some immcfg >>> commands but not all the intended ones. >>> What does the user do then ? >>> So (1) would be either just a very sloppy (and error prone) implementation >>> or possibly a test case of some kind, but a test that is not a valid >>> functional/reliability test. Possibly a performance test of a metric where >>> the imm is not intended to have high performance (high transaction >>> throughput). >>> >>> /AndersBj >>> >>> >>> >>> >>> >>>> ----Ursprungligt meddelande---- >>>> Från : [email protected] >>>> Datum : 2016-09-05 - 14:00 (CEST) >>>> Till : [email protected], [email protected], >>>> [email protected], [email protected], >>>> [email protected] >>>> Ämne : Re: [devel] Immnd: maximum Ccbs limit 10000 has been reached very >>>> quickly >>>> >>>> I already mentioned one use case: a shell script calling the immcfg >>>> command from a loop. There is also another important use case: a buggy >>>> application. Imagine that an application by mistake creates a huge >>>> amount of CCBs. Since this new limit is global, the buggy application >>>> will now affect all other applications in the whole cluster. A variant >>>> of this use-case is an attacker carrying out a denial-of-service attack >>>> on the system by somehow (e.g. via NBI) creating many CCBs. >>>> >>>> regards, >>>> >>>> Anders Widell >>>> >>>> >>>> On 09/05/2016 12:56 PM, Zoran Milinkovic wrote: >>>>> Hi Hung, >>>>> >>>>> Agree with you and Anders Widell for the first item. >>>>> >>>>> For the second item, as I mentioned earlier, IMM is not developed to >>>>> handle a huge amount of CCBs. If we need to support that, then we need >>>>> to rewrite CCB part, and replace std::vector with some other >>>>> structure, possibly with std::map. Then the performance degradation >>>>> will be minimal. >>>>> >>>>> And when this is changed, probably CCB limitation would me more >>>>> related to memory usage instead of IMM performance. Then we can >>>>> consider all CCB. >>>>> >>>>> To repeat myself again, I don’t see a valid case where I’m not able to >>>>> open/use a CCB for minutes because some other application intensively >>>>> used CCBs and managed to close 10000 handles within a minute. >>>>> >>>>> BR, >>>>> >>>>> Zoran >>>>> >>>>> *From:*Hung Nguyen [mailto:[email protected]] >>>>> *Sent:* den 5 september 2016 12:29 >>>>> *To:* Anders Widell; Zoran Milinkovic; A V Mahesh; Neelakanta Reddy; >>>>> [email protected] >>>>> *Subject:* Re: [devel] Immnd: maximum Ccbs limit 10000 has been >>>>> reached very quickly >>>>> >>>>> Hi, >>>>> >>>>> I agree with Anders that we should let users enable the limit instead of >>>>> enabling it automatically. >>>>> Maybe we can set default value of those attributes to 0 and IMM will >>>>> interpret 0 as unlimited. >>>>> It's going to be a defect ticket for #195. >>>>> >>>>> About the second problem (i.e. the meaning of maxCcbs limit), I think the >>>>> limit should be applied on all CCBs as we store them in the same list. >>>>> This limit is for pure performance purpose. >>>>> >>>>> It should be well documented though. >>>>> The easiest way for users to check how much resource left is to use >>>>> 'immadm -O displayverbose'. >>>>> >>>>> BR, >>>>> Hung Nguyen - DEK Technologies >>>>> >>>>> -------------------------------------------------------------------------------- >>>>> From: Anders [email protected] >>>>> <mailto:[email protected]> >>>>> Sent: Friday, September 02, 2016 9:53PM >>>>> To: Zoran Milinkovic, Mahesh Valla, Neelakanta Reddy, Opensaf-devel >>>>> [email protected] >>>>> <mailto:[email protected]>,[email protected] >>>>> <mailto:[email protected]>,[email protected] >>>>> <mailto:[email protected]>,[email protected] >>>>> <mailto:[email protected]> >>>>> Cc: >>>>> >>>>> Subject: Re: [devel] Immnd: maximum Ccbs limit 10000 has been reached >>>>> very quickly >>>>> >>>>> >>>>> We have to consider the backwards compatibility aspect here. The general >>>>> principle is that if an application works fine with one version of >>>>> OpenSAF, then it should work fine also with a newer version of OpenSAF. >>>>> Mahesh has evidently an example application (a test program) that starts >>>>> to fail after upgrading from OpenSAF 5.0 to OpenSAF 5.1.FC. If it is a >>>>> pathological example program then you can argue that it is not relevant, >>>>> but I don't think it is the case here. You could imagine a real-life >>>>> case where someone has written a shell script that runs the immcfg >>>>> command in a loop 10000 times. >>>>> >>>>> The safest approach to introduce a new limit where we previously didn't >>>>> have any limit, is to set the default value of the new limit to infinite >>>>> (unlimited), and let the user take an active decision to configure a >>>>> lower value of the limit. This way, applications will not get unpleasant >>>>> surprises after upgrading OpenSAF. >>>>> >>>>> regards, >>>>> >>>>> Anders Widell >>>>> >>>>> On 09/02/2016 10:56 AM, Zoran Milinkovic wrote: >>>>> >>>>> Hi Mahesh, >>>>> >>>>> >>>>> >>>>> Ticket #195 introduced this limitation. >>>>> >>>>> >>>>> >>>>> After all, I'm not sure who is right... Neelakanta or me :) >>>>> >>>>> Earlier, we didn't have any limitation, and limitation is >>>>> introduced to have some reasonable amount of CCB handles in IMM. >>>>> >>>>> IMM keeps non-used and active CCB handles in the same place. If the >>>>> amount of CCB handles grows a lot, then we will see a degradation of IMM >>>>> performance. >>>>> >>>>> >>>>> >>>>> >From earlier discussion with Anders, it's not meant from the >>>>> beginning that CCB will be used so often. This might be very specific >>>>> case. That's also the reason why CCB handles are handled in a way that is >>>>> not good for huge amount of CCB handles. >>>>> >>>>> >From this point of view, Neelakanta is correct. >>>>> >>>>> >From my point of view, we should only consider CCBs that can be >>>>> used. I don't see acceptable that CCB operations cannot be done for >>>>> minutes if we have a huge amount of closed CCBs. >>>>> >>>>> >>>>> >>>>> We can improve CCB handling in the next OpenSAF release. Now it's >>>>> too late. >>>>> >>>>> >>>>> >>>>> For this particular problem, I would go with checking only CCBs >>>>> that can be used (CCB that have SA_AIS_OK on mVeto variable, or calling >>>>> ->isOk() method on CcbInfo struct). This solution may have impact on IMM >>>>> performance. >>>>> >>>>> Or we can revert the ticket and implement it in the next release >>>>> with better performance. >>>>> >>>>> >>>>> >>>>> I would also like to hear Neelakanta's and Hung's proposals. >>>>> >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Zoran >>>>> >>>>> >>>>> >>>>> -----Original Message----- >>>>> >>>>> From: A V Mahesh [mailto:[email protected]] >>>>> >>>>> Sent: den 2 september 2016 05:56 >>>>> >>>>> To: Zoran Milinkovic; Neelakanta >>>>> Reddy;[email protected] >>>>> <mailto:[email protected]> >>>>> >>>>> Subject: Re: [devel] Immnd: maximum Ccbs limit 10000 has been >>>>> reached very quickly >>>>> >>>>> >>>>> >>>>> Hi Zoran, >>>>> >>>>> >>>>> >>>>> Thanks for the update , >>>>> >>>>> >>>>> >>>>> you are right ERR_NO_RESOURCES only justifiable if 10000 IMM >>>>> application concurrently holding 10000 CCB`s ( active CCB), in current >>>>> case agent is already finalized. >>>>> >>>>> >>>>> >>>>> By the way by roll backing which change set I can continue my >>>>> testing ? >>>>> >>>>> >>>>> >>>>> -AVM >>>>> >>>>> >>>>> >>>>> On 9/1/2016 5:52 PM, Zoran Milinkovic wrote: >>>>> >>>>> Hi Neelakanta, >>>>> >>>>> >>>>> >>>>> This is definitely a bug. >>>>> >>>>> When CCB is finalized, it's still shown as an active CCB. >>>>> >>>>> If you check number of CCBs with resource display >>>>> functionality, you will see that number of CCBs is not decreasing. >>>>> >>>>> >>>>> >>>>> BR, >>>>> >>>>> Zoran >>>>> >>>>> >>>>> >>>>> -----Original Message----- >>>>> >>>>> From: Neelakanta Reddy [mailto:[email protected]] >>>>> >>>>> Sent: den 1 september 2016 11:09 >>>>> >>>>> To:[email protected] >>>>> <mailto:[email protected]> >>>>> >>>>> Subject: Re: [devel] Immnd: maximum Ccbs limit 10000 has been >>>>> reached >>>>> >>>>> very quickly >>>>> >>>>> >>>>> >>>>> Hi, >>>>> >>>>> >>>>> >>>>> I think this is not yet documented and will be documented. >>>>> >>>>> Try to increase the ccb limits and check. >>>>> >>>>> >>>>> >>>>> /Neel. >>>>> >>>>> >>>>> >>>>> On 2016/09/01 02:01 PM, Chani Srivastava wrote: >>>>> >>>>> Hi, >>>>> >>>>> >>>>> >>>>> With current 5.1 OpenSAF changeset 7997 the issue is >>>>> reproducible >>>>> >>>>> >>>>> >>>>> ============================================== >>>>> >>>>> Sep 1 12:16:52 OSAF-SC1 osafimmnd[27298]: NO Ccb 10002 >>>>> COMMITTED >>>>> >>>>> (chaniTestClass) >>>>> >>>>> Sep 1 12:16:52 OSAF-SC1 osafimmnd[27298]: NO Ccb 10003 >>>>> COMMITTED >>>>> >>>>> (chaniTestClass) >>>>> >>>>> Sep 1 12:16:52 OSAF-SC1 osafimmnd[27298]: NO Ccb 10004 >>>>> COMMITTED >>>>> >>>>> (chaniTestClass) >>>>> >>>>> Sep 1 12:16:52 OSAF-SC1 osafimmnd[27298]: NO Ccb 10005 >>>>> COMMITTED >>>>> >>>>> (chaniTestClass) >>>>> >>>>> Sep 1 12:16:52 OSAF-SC1 osafimmnd[27298]: NO >>>>> ERR_NO_RESOURCES: >>>>> >>>>> maximum Ccbs limit 10000 has been reached for the cluster >>>>> >>>>> =============================================== >>>>> >>>>> >>>>> >>>>> This looks like a candidate for ticket. Let me know i'll >>>>> raise it in >>>>> >>>>> community. >>>>> >>>>> >>>>> >>>>> -Chani >>>>> >>>>> >>>>> >>>>> On 9/1/2016 1:49 PM, Neelakanta Reddy wrote: >>>>> >>>>> Hi, >>>>> >>>>> >>>>> >>>>> can you verify the same with 5.1. >>>>> >>>>> >>>>> >>>>> /Neel. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On 2016/09/01 12:58 PM, Chani Srivastava wrote: >>>>> >>>>> Hi All, >>>>> >>>>> >>>>> >>>>> I verified this is a break in functionality. In >>>>> OpenSAF 5.0 I tried >>>>> >>>>> creating 15000 objects per CCB and it worked fine. >>>>> >>>>> >>>>> >>>>> ===================================== >>>>> >>>>> Nov 26 15:21:17 SCALE_SLOT-91 osafimmnd[24451]: NO >>>>> Ccb 237 >>>>> >>>>> COMMITTED >>>>> >>>>> (chaniTestClass) >>>>> >>>>> Nov 26 15:21:17 SCALE_SLOT-91 osafimmnd[24451]: NO >>>>> Ccb 238 >>>>> >>>>> COMMITTED >>>>> >>>>> (chaniTestClass) >>>>> >>>>> Nov 26 15:21:17 SCALE_SLOT-91 osafimmnd[24451]: NO >>>>> Ccb 239 >>>>> >>>>> COMMITTED >>>>> >>>>> (chaniTestClass) >>>>> >>>>> Nov 26 15:21:17 SCALE_SLOT-91 osafimmnd[24451]: NO >>>>> Ccb 240 >>>>> >>>>> COMMITTED >>>>> >>>>> (chaniTestClass) >>>>> >>>>> . >>>>> >>>>> . >>>>> >>>>> . >>>>> >>>>> Nov 26 15:23:05 SCALE_SLOT-91 osafimmnd[24451]: NO >>>>> Ccb 15230 >>>>> >>>>> COMMITTED >>>>> >>>>> (chaniTestClass) >>>>> >>>>> Nov 26 15:23:05 SCALE_SLOT-91 osafimmnd[24451]: NO >>>>> Ccb 15231 >>>>> >>>>> COMMITTED >>>>> >>>>> (chaniTestClass) >>>>> >>>>> Nov 26 15:23:05 SCALE_SLOT-91 osafimmnd[24451]: NO >>>>> Ccb 15232 >>>>> >>>>> COMMITTED >>>>> >>>>> (chaniTestClass) >>>>> >>>>> Nov 26 15:23:05 SCALE_SLOT-91 osafimmnd[24451]: NO >>>>> Ccb 15233 >>>>> >>>>> COMMITTED >>>>> >>>>> (chaniTestClass) >>>>> >>>>> Nov 26 15:23:05 SCALE_SLOT-91 osafimmnd[24451]: NO >>>>> Ccb 15234 >>>>> >>>>> COMMITTED >>>>> >>>>> (chaniTestClass) >>>>> >>>>> Nov 26 15:23:05 SCALE_SLOT-91 osafimmnd[24451]: NO >>>>> Ccb 15235 >>>>> >>>>> COMMITTED >>>>> >>>>> (chaniTestClass) >>>>> >>>>> Nov 26 15:23:05 SCALE_SLOT-91 osafimmnd[24451]: NO >>>>> Ccb 15236 >>>>> >>>>> COMMITTED >>>>> >>>>> (chaniTestClass) >>>>> >>>>> ====================================== >>>>> >>>>> >>>>> >>>>> The ticket #195 only makes the MAX parameters >>>>> configurable. >>>>> >>>>> If accepted, I'll raise the ticket in sourceforge. >>>>> >>>>> >>>>> >>>>> -Chani >>>>> >>>>> >>>>> >>>>> On 9/1/2016 10:59 AM, A V Mahesh wrote: >>>>> >>>>> Hi All, >>>>> >>>>> >>>>> >>>>> I was running `immcfg` in a loop to create >>>>> some object , once it >>>>> >>>>> reaches 10001objects creation Immnd is >>>>> returning `ERR_NO_RESOURCES: >>>>> >>>>> maximum Ccbs limit 10000 has been reached for >>>>> the cluster` error >>>>> >>>>> which is unexpected , once `immcfg` reruns , >>>>> it is expected that >>>>> >>>>> the `maximum Ccbs` will decremented ( this was >>>>> previous behavior) >>>>> >>>>> , >>>>> >>>>> >>>>> >>>>> can you please check. >>>>> >>>>> >>>>> >>>>> >>>>> ================================================================== >>>>> >>>>> = ====================================== >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> for (( i = 1 ; i <=300000; i++)) >>>>> >>>>> immcfg -c PinvId -a >>>>> pinvPhoneNumber=+46768 pinvRdn=$i >>>>> >>>>> >>>>> >>>>> Sep 1 10:43:33 SC-1 osafimmnd[4466]: NO Ccb >>>>> 9996 COMMITTED >>>>> >>>>> (immcfg_SC-1_2334) >>>>> >>>>> Sep 1 10:43:33 SC-1 osafimmnd[4466]: NO Ccb >>>>> 9997 COMMITTED >>>>> >>>>> (immcfg_SC-1_2337) >>>>> >>>>> Sep 1 10:43:33 SC-1 osafimmnd[4466]: NO Ccb >>>>> 9998 COMMITTED >>>>> >>>>> (immcfg_SC-1_2340) >>>>> >>>>> Sep 1 10:43:33 SC-1 osafimmnd[4466]: NO Ccb >>>>> 9999 COMMITTED >>>>> >>>>> (immcfg_SC-1_2343) >>>>> >>>>> Sep 1 10:43:33 SC-1 osafimmnd[4466]: NO Ccb >>>>> 10000 COMMITTED >>>>> >>>>> (immcfg_SC-1_2346) >>>>> >>>>> Sep 1 10:43:33 SC-1 osafimmnd[4466]: NO Ccb >>>>> 10001 COMMITTED >>>>> >>>>> (immcfg_SC-1_2349) >>>>> >>>>> Sep 1 10:43:33 SC-1 osafimmnd[4466]: NO >>>>> ERR_NO_RESOURCES: maximum >>>>> >>>>> Ccbs limit 10000 has been reached for the >>>>> cluster Sep 1 10:43:33 >>>>> >>>>> SC-1 osafimmnd[4466]: NO ERR_NO_RESOURCES: >>>>> maximum Ccbs limit >>>>> >>>>> 10000 has been reached for the cluster Sep 1 >>>>> 10:43:33 SC-1 >>>>> >>>>> osafimmnd[4466]: NO ERR_NO_RESOURCES: maximum >>>>> Ccbs limit 10000 has >>>>> >>>>> been reached for the cluster >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> ================================================================== >>>>> >>>>> = ====================================== >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -AVM >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> ------------------------------------------------------------------ >>>>> >>>>> - >>>>> >>>>> ----------- >>>>> _______________________________________________ >>>>> >>>>> Opensaf-devel mailing list >>>>> >>>>> [email protected] >>>>> <mailto:[email protected]> >>>>> >>>>> >>>>> https://lists.sourceforge.net/lists/listinfo/opensaf-devel >>>>> >>>>> >>>>> ------------------------------------------------------------------- >>>>> >>>>> - >>>>> >>>>> ---------- >>>>> _______________________________________________ >>>>> >>>>> Opensaf-devel mailing list >>>>> >>>>> [email protected] >>>>> <mailto:[email protected]> >>>>> >>>>> >>>>> https://lists.sourceforge.net/lists/listinfo/opensaf-devel >>>>> >>>>> >>>>> -------------------------------------------------------------------- >>>>> >>>>> - >>>>> >>>>> --------- >>>>> _______________________________________________ >>>>> >>>>> Opensaf-devel mailing list >>>>> >>>>> [email protected] >>>>> <mailto:[email protected]> >>>>> >>>>> >>>>> https://lists.sourceforge.net/lists/listinfo/opensaf-devel >>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> >>>>> - >>>>> >>>>> -------- _______________________________________________ >>>>> >>>>> Opensaf-devel mailing list >>>>> >>>>> [email protected] >>>>> <mailto:[email protected]> >>>>> >>>>> https://lists.sourceforge.net/lists/listinfo/opensaf-devel >>>>> >>>>> >>>>> ---------------------------------------------------------------------- >>>>> >>>>> -------- _______________________________________________ >>>>> >>>>> Opensaf-devel mailing list >>>>> >>>>> [email protected] >>>>> <mailto:[email protected]> >>>>> >>>>> https://lists.sourceforge.net/lists/listinfo/opensaf-devel >>>>> >>>>> >>>>> >>>>> >>>>> ---------------------------------------------------------------------- >>>>> >>>>> -------- _______________________________________________ >>>>> >>>>> Opensaf-devel mailing list >>>>> >>>>> [email protected] >>>>> <mailto:[email protected]> >>>>> >>>>> https://lists.sourceforge.net/lists/listinfo/opensaf-devel >>>>> >>>>> >>>>> ------------------------------------------------------------------------------ >>>>> >>>>> _______________________________________________ >>>>> >>>>> Opensaf-devel mailing list >>>>> >>>>> [email protected] >>>>> <mailto:[email protected]> >>>>> >>>>> https://lists.sourceforge.net/lists/listinfo/opensaf-devel >>>>> >>>>> >>>>> >>>>> >>>>> ------------------------------------------------------------------------------ >>>>> _______________________________________________ >>>>> Opensaf-devel mailing list >>>>> [email protected] >>>>> <mailto:[email protected]> >>>>> https://lists.sourceforge.net/lists/listinfo/opensaf-devel >>>>> >>>> ------------------------------------------------------------------------------ >>>> _______________________________________________ >>>> Opensaf-devel mailing list >>>> [email protected] >>>> https://lists.sourceforge.net/lists/listinfo/opensaf-devel >>>> >> >> ------------------------------------------------------------------------------ _______________________________________________ Opensaf-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/opensaf-devel
