Yes, limiting the number of active CCBs is much more sensible. The big problem with the currently implemented limit is that it also counts finished CCBs, and that this limit is global for the whole cluster. It opens up the possibility for denial-of-service - either because of a bug or because of a malicious attacker.
/ Anders Widell On 09/05/2016 03:22 PM, [email protected] wrote: > Having said that, a case can be made that the IMM should protect itself from > that kind of abuse. > > Such protection against overload should be associated with the imm-ccb-handle > that is operating. > The point here is that one faulty ccb-application should not be capable of > causing denial of imm-ccb service for others. > > A case caould also be made for limiting the total number of concurrently open > and active ccb handles. > Something like we have for iterators/accessors. > > /AndersBj > > >> ----Ursprungligt meddelande---- >> Från : [email protected] >> Datum : 2016-09-05 - 15:17 (CEST) >> Till : [email protected], [email protected], >> [email protected], [email protected], >> [email protected], [email protected] >> Ämne : Re: [devel] Immnd: maximum Ccbs limit 10000 has been reached very >> quickly >> >> Hi, >> >> Neither of those examples (1) A shell script calling immcfg command from a >> loop; (2) A buggy application; are real use cases. >> For (2) it is obvious. >> >> For (1), it is at best a very sloppy implementation of some use case. >> The intention of that shell script is to accomplish what amounts to one >> configuration change, but it is doing so using thousands of separate ccbs! >> Say that the script itself crashes after it has completed some immcfg >> commands but not all the intended ones. >> What does the user do then ? >> So (1) would be either just a very sloppy (and error prone) implementation >> or possibly a test case of some kind, but a test that is not a valid >> functional/reliability test. Possibly a performance test of a metric where >> the imm is not intended to have high performance (high transaction >> throughput). >> >> /AndersBj >> >> >> >> >> >>> ----Ursprungligt meddelande---- >>> Från : [email protected] >>> Datum : 2016-09-05 - 14:00 (CEST) >>> Till : [email protected], [email protected], >>> [email protected], [email protected], >>> [email protected] >>> Ämne : Re: [devel] Immnd: maximum Ccbs limit 10000 has been reached very >>> quickly >>> >>> I already mentioned one use case: a shell script calling the immcfg >>> command from a loop. There is also another important use case: a buggy >>> application. Imagine that an application by mistake creates a huge >>> amount of CCBs. Since this new limit is global, the buggy application >>> will now affect all other applications in the whole cluster. A variant >>> of this use-case is an attacker carrying out a denial-of-service attack >>> on the system by somehow (e.g. via NBI) creating many CCBs. >>> >>> regards, >>> >>> Anders Widell >>> >>> >>> On 09/05/2016 12:56 PM, Zoran Milinkovic wrote: >>>> Hi Hung, >>>> >>>> Agree with you and Anders Widell for the first item. >>>> >>>> For the second item, as I mentioned earlier, IMM is not developed to >>>> handle a huge amount of CCBs. If we need to support that, then we need >>>> to rewrite CCB part, and replace std::vector with some other >>>> structure, possibly with std::map. Then the performance degradation >>>> will be minimal. >>>> >>>> And when this is changed, probably CCB limitation would me more >>>> related to memory usage instead of IMM performance. Then we can >>>> consider all CCB. >>>> >>>> To repeat myself again, I don’t see a valid case where I’m not able to >>>> open/use a CCB for minutes because some other application intensively >>>> used CCBs and managed to close 10000 handles within a minute. >>>> >>>> BR, >>>> >>>> Zoran >>>> >>>> *From:*Hung Nguyen [mailto:[email protected]] >>>> *Sent:* den 5 september 2016 12:29 >>>> *To:* Anders Widell; Zoran Milinkovic; A V Mahesh; Neelakanta Reddy; >>>> [email protected] >>>> *Subject:* Re: [devel] Immnd: maximum Ccbs limit 10000 has been >>>> reached very quickly >>>> >>>> Hi, >>>> >>>> I agree with Anders that we should let users enable the limit instead of >>>> enabling it automatically. >>>> Maybe we can set default value of those attributes to 0 and IMM will >>>> interpret 0 as unlimited. >>>> It's going to be a defect ticket for #195. >>>> >>>> About the second problem (i.e. the meaning of maxCcbs limit), I think the >>>> limit should be applied on all CCBs as we store them in the same list. >>>> This limit is for pure performance purpose. >>>> >>>> It should be well documented though. >>>> The easiest way for users to check how much resource left is to use >>>> 'immadm -O displayverbose'. >>>> >>>> BR, >>>> Hung Nguyen - DEK Technologies >>>> >>>> -------------------------------------------------------------------------------- >>>> From: Anders [email protected] >>>> <mailto:[email protected]> >>>> Sent: Friday, September 02, 2016 9:53PM >>>> To: Zoran Milinkovic, Mahesh Valla, Neelakanta Reddy, Opensaf-devel >>>> [email protected] >>>> <mailto:[email protected]>,[email protected] >>>> <mailto:[email protected]>,[email protected] >>>> <mailto:[email protected]>,[email protected] >>>> <mailto:[email protected]> >>>> Cc: >>>> >>>> Subject: Re: [devel] Immnd: maximum Ccbs limit 10000 has been reached very >>>> quickly >>>> >>>> >>>> We have to consider the backwards compatibility aspect here. The general >>>> principle is that if an application works fine with one version of >>>> OpenSAF, then it should work fine also with a newer version of OpenSAF. >>>> Mahesh has evidently an example application (a test program) that starts >>>> to fail after upgrading from OpenSAF 5.0 to OpenSAF 5.1.FC. If it is a >>>> pathological example program then you can argue that it is not relevant, >>>> but I don't think it is the case here. You could imagine a real-life >>>> case where someone has written a shell script that runs the immcfg >>>> command in a loop 10000 times. >>>> >>>> The safest approach to introduce a new limit where we previously didn't >>>> have any limit, is to set the default value of the new limit to infinite >>>> (unlimited), and let the user take an active decision to configure a >>>> lower value of the limit. This way, applications will not get unpleasant >>>> surprises after upgrading OpenSAF. >>>> >>>> regards, >>>> >>>> Anders Widell >>>> >>>> On 09/02/2016 10:56 AM, Zoran Milinkovic wrote: >>>> >>>> Hi Mahesh, >>>> >>>> >>>> >>>> Ticket #195 introduced this limitation. >>>> >>>> >>>> >>>> After all, I'm not sure who is right... Neelakanta or me :) >>>> >>>> Earlier, we didn't have any limitation, and limitation is introduced >>>> to have some reasonable amount of CCB handles in IMM. >>>> >>>> IMM keeps non-used and active CCB handles in the same place. If the >>>> amount of CCB handles grows a lot, then we will see a degradation of IMM >>>> performance. >>>> >>>> >>>> >>>> >From earlier discussion with Anders, it's not meant from the >>>> beginning that CCB will be used so often. This might be very specific >>>> case. That's also the reason why CCB handles are handled in a way that is >>>> not good for huge amount of CCB handles. >>>> >>>> >From this point of view, Neelakanta is correct. >>>> >>>> >From my point of view, we should only consider CCBs that can be >>>> used. I don't see acceptable that CCB operations cannot be done for >>>> minutes if we have a huge amount of closed CCBs. >>>> >>>> >>>> >>>> We can improve CCB handling in the next OpenSAF release. Now it's too >>>> late. >>>> >>>> >>>> >>>> For this particular problem, I would go with checking only CCBs that >>>> can be used (CCB that have SA_AIS_OK on mVeto variable, or calling >>>> ->isOk() method on CcbInfo struct). This solution may have impact on IMM >>>> performance. >>>> >>>> Or we can revert the ticket and implement it in the next release with >>>> better performance. >>>> >>>> >>>> >>>> I would also like to hear Neelakanta's and Hung's proposals. >>>> >>>> >>>> >>>> Thanks, >>>> >>>> Zoran >>>> >>>> >>>> >>>> -----Original Message----- >>>> >>>> From: A V Mahesh [mailto:[email protected]] >>>> >>>> Sent: den 2 september 2016 05:56 >>>> >>>> To: Zoran Milinkovic; Neelakanta >>>> Reddy;[email protected] >>>> <mailto:[email protected]> >>>> >>>> Subject: Re: [devel] Immnd: maximum Ccbs limit 10000 has been reached >>>> very quickly >>>> >>>> >>>> >>>> Hi Zoran, >>>> >>>> >>>> >>>> Thanks for the update , >>>> >>>> >>>> >>>> you are right ERR_NO_RESOURCES only justifiable if 10000 IMM >>>> application concurrently holding 10000 CCB`s ( active CCB), in current >>>> case agent is already finalized. >>>> >>>> >>>> >>>> By the way by roll backing which change set I can continue my testing >>>> ? >>>> >>>> >>>> >>>> -AVM >>>> >>>> >>>> >>>> On 9/1/2016 5:52 PM, Zoran Milinkovic wrote: >>>> >>>> Hi Neelakanta, >>>> >>>> >>>> >>>> This is definitely a bug. >>>> >>>> When CCB is finalized, it's still shown as an active CCB. >>>> >>>> If you check number of CCBs with resource display functionality, >>>> you will see that number of CCBs is not decreasing. >>>> >>>> >>>> >>>> BR, >>>> >>>> Zoran >>>> >>>> >>>> >>>> -----Original Message----- >>>> >>>> From: Neelakanta Reddy [mailto:[email protected]] >>>> >>>> Sent: den 1 september 2016 11:09 >>>> >>>> To:[email protected] >>>> <mailto:[email protected]> >>>> >>>> Subject: Re: [devel] Immnd: maximum Ccbs limit 10000 has been >>>> reached >>>> >>>> very quickly >>>> >>>> >>>> >>>> Hi, >>>> >>>> >>>> >>>> I think this is not yet documented and will be documented. >>>> >>>> Try to increase the ccb limits and check. >>>> >>>> >>>> >>>> /Neel. >>>> >>>> >>>> >>>> On 2016/09/01 02:01 PM, Chani Srivastava wrote: >>>> >>>> Hi, >>>> >>>> >>>> >>>> With current 5.1 OpenSAF changeset 7997 the issue is >>>> reproducible >>>> >>>> >>>> >>>> ============================================== >>>> >>>> Sep 1 12:16:52 OSAF-SC1 osafimmnd[27298]: NO Ccb 10002 >>>> COMMITTED >>>> >>>> (chaniTestClass) >>>> >>>> Sep 1 12:16:52 OSAF-SC1 osafimmnd[27298]: NO Ccb 10003 >>>> COMMITTED >>>> >>>> (chaniTestClass) >>>> >>>> Sep 1 12:16:52 OSAF-SC1 osafimmnd[27298]: NO Ccb 10004 >>>> COMMITTED >>>> >>>> (chaniTestClass) >>>> >>>> Sep 1 12:16:52 OSAF-SC1 osafimmnd[27298]: NO Ccb 10005 >>>> COMMITTED >>>> >>>> (chaniTestClass) >>>> >>>> Sep 1 12:16:52 OSAF-SC1 osafimmnd[27298]: NO >>>> ERR_NO_RESOURCES: >>>> >>>> maximum Ccbs limit 10000 has been reached for the cluster >>>> >>>> =============================================== >>>> >>>> >>>> >>>> This looks like a candidate for ticket. Let me know i'll >>>> raise it in >>>> >>>> community. >>>> >>>> >>>> >>>> -Chani >>>> >>>> >>>> >>>> On 9/1/2016 1:49 PM, Neelakanta Reddy wrote: >>>> >>>> Hi, >>>> >>>> >>>> >>>> can you verify the same with 5.1. >>>> >>>> >>>> >>>> /Neel. >>>> >>>> >>>> >>>> >>>> >>>> On 2016/09/01 12:58 PM, Chani Srivastava wrote: >>>> >>>> Hi All, >>>> >>>> >>>> >>>> I verified this is a break in functionality. In >>>> OpenSAF 5.0 I tried >>>> >>>> creating 15000 objects per CCB and it worked fine. >>>> >>>> >>>> >>>> ===================================== >>>> >>>> Nov 26 15:21:17 SCALE_SLOT-91 osafimmnd[24451]: NO >>>> Ccb 237 >>>> >>>> COMMITTED >>>> >>>> (chaniTestClass) >>>> >>>> Nov 26 15:21:17 SCALE_SLOT-91 osafimmnd[24451]: NO >>>> Ccb 238 >>>> >>>> COMMITTED >>>> >>>> (chaniTestClass) >>>> >>>> Nov 26 15:21:17 SCALE_SLOT-91 osafimmnd[24451]: NO >>>> Ccb 239 >>>> >>>> COMMITTED >>>> >>>> (chaniTestClass) >>>> >>>> Nov 26 15:21:17 SCALE_SLOT-91 osafimmnd[24451]: NO >>>> Ccb 240 >>>> >>>> COMMITTED >>>> >>>> (chaniTestClass) >>>> >>>> . >>>> >>>> . >>>> >>>> . >>>> >>>> Nov 26 15:23:05 SCALE_SLOT-91 osafimmnd[24451]: NO >>>> Ccb 15230 >>>> >>>> COMMITTED >>>> >>>> (chaniTestClass) >>>> >>>> Nov 26 15:23:05 SCALE_SLOT-91 osafimmnd[24451]: NO >>>> Ccb 15231 >>>> >>>> COMMITTED >>>> >>>> (chaniTestClass) >>>> >>>> Nov 26 15:23:05 SCALE_SLOT-91 osafimmnd[24451]: NO >>>> Ccb 15232 >>>> >>>> COMMITTED >>>> >>>> (chaniTestClass) >>>> >>>> Nov 26 15:23:05 SCALE_SLOT-91 osafimmnd[24451]: NO >>>> Ccb 15233 >>>> >>>> COMMITTED >>>> >>>> (chaniTestClass) >>>> >>>> Nov 26 15:23:05 SCALE_SLOT-91 osafimmnd[24451]: NO >>>> Ccb 15234 >>>> >>>> COMMITTED >>>> >>>> (chaniTestClass) >>>> >>>> Nov 26 15:23:05 SCALE_SLOT-91 osafimmnd[24451]: NO >>>> Ccb 15235 >>>> >>>> COMMITTED >>>> >>>> (chaniTestClass) >>>> >>>> Nov 26 15:23:05 SCALE_SLOT-91 osafimmnd[24451]: NO >>>> Ccb 15236 >>>> >>>> COMMITTED >>>> >>>> (chaniTestClass) >>>> >>>> ====================================== >>>> >>>> >>>> >>>> The ticket #195 only makes the MAX parameters >>>> configurable. >>>> >>>> If accepted, I'll raise the ticket in sourceforge. >>>> >>>> >>>> >>>> -Chani >>>> >>>> >>>> >>>> On 9/1/2016 10:59 AM, A V Mahesh wrote: >>>> >>>> Hi All, >>>> >>>> >>>> >>>> I was running `immcfg` in a loop to create some >>>> object , once it >>>> >>>> reaches 10001objects creation Immnd is returning >>>> `ERR_NO_RESOURCES: >>>> >>>> maximum Ccbs limit 10000 has been reached for the >>>> cluster` error >>>> >>>> which is unexpected , once `immcfg` reruns , it >>>> is expected that >>>> >>>> the `maximum Ccbs` will decremented ( this was >>>> previous behavior) >>>> >>>> , >>>> >>>> >>>> >>>> can you please check. >>>> >>>> >>>> >>>> >>>> ================================================================== >>>> >>>> = ====================================== >>>> >>>> >>>> >>>> >>>> >>>> for (( i = 1 ; i <=300000; i++)) >>>> >>>> immcfg -c PinvId -a >>>> pinvPhoneNumber=+46768 pinvRdn=$i >>>> >>>> >>>> >>>> Sep 1 10:43:33 SC-1 osafimmnd[4466]: NO Ccb 9996 >>>> COMMITTED >>>> >>>> (immcfg_SC-1_2334) >>>> >>>> Sep 1 10:43:33 SC-1 osafimmnd[4466]: NO Ccb 9997 >>>> COMMITTED >>>> >>>> (immcfg_SC-1_2337) >>>> >>>> Sep 1 10:43:33 SC-1 osafimmnd[4466]: NO Ccb 9998 >>>> COMMITTED >>>> >>>> (immcfg_SC-1_2340) >>>> >>>> Sep 1 10:43:33 SC-1 osafimmnd[4466]: NO Ccb 9999 >>>> COMMITTED >>>> >>>> (immcfg_SC-1_2343) >>>> >>>> Sep 1 10:43:33 SC-1 osafimmnd[4466]: NO Ccb >>>> 10000 COMMITTED >>>> >>>> (immcfg_SC-1_2346) >>>> >>>> Sep 1 10:43:33 SC-1 osafimmnd[4466]: NO Ccb >>>> 10001 COMMITTED >>>> >>>> (immcfg_SC-1_2349) >>>> >>>> Sep 1 10:43:33 SC-1 osafimmnd[4466]: NO >>>> ERR_NO_RESOURCES: maximum >>>> >>>> Ccbs limit 10000 has been reached for the cluster >>>> Sep 1 10:43:33 >>>> >>>> SC-1 osafimmnd[4466]: NO ERR_NO_RESOURCES: >>>> maximum Ccbs limit >>>> >>>> 10000 has been reached for the cluster Sep 1 >>>> 10:43:33 SC-1 >>>> >>>> osafimmnd[4466]: NO ERR_NO_RESOURCES: maximum >>>> Ccbs limit 10000 has >>>> >>>> been reached for the cluster >>>> >>>> >>>> >>>> >>>> >>>> >>>> ================================================================== >>>> >>>> = ====================================== >>>> >>>> >>>> >>>> >>>> >>>> -AVM >>>> >>>> >>>> >>>> >>>> >>>> >>>> ------------------------------------------------------------------ >>>> >>>> - >>>> >>>> ----------- >>>> _______________________________________________ >>>> >>>> Opensaf-devel mailing list >>>> >>>> [email protected] >>>> <mailto:[email protected]> >>>> >>>> >>>> https://lists.sourceforge.net/lists/listinfo/opensaf-devel >>>> >>>> >>>> ------------------------------------------------------------------- >>>> >>>> - >>>> >>>> ---------- >>>> _______________________________________________ >>>> >>>> Opensaf-devel mailing list >>>> >>>> [email protected] >>>> <mailto:[email protected]> >>>> >>>> >>>> https://lists.sourceforge.net/lists/listinfo/opensaf-devel >>>> >>>> >>>> -------------------------------------------------------------------- >>>> >>>> - >>>> >>>> --------- _______________________________________________ >>>> >>>> Opensaf-devel mailing list >>>> >>>> [email protected] >>>> <mailto:[email protected]> >>>> >>>> https://lists.sourceforge.net/lists/listinfo/opensaf-devel >>>> >>>> >>>> --------------------------------------------------------------------- >>>> >>>> - >>>> >>>> -------- _______________________________________________ >>>> >>>> Opensaf-devel mailing list >>>> >>>> [email protected] >>>> <mailto:[email protected]> >>>> >>>> https://lists.sourceforge.net/lists/listinfo/opensaf-devel >>>> >>>> >>>> ---------------------------------------------------------------------- >>>> >>>> -------- _______________________________________________ >>>> >>>> Opensaf-devel mailing list >>>> >>>> [email protected] >>>> <mailto:[email protected]> >>>> >>>> https://lists.sourceforge.net/lists/listinfo/opensaf-devel >>>> >>>> >>>> >>>> >>>> ---------------------------------------------------------------------- >>>> >>>> -------- _______________________________________________ >>>> >>>> Opensaf-devel mailing list >>>> >>>> [email protected] >>>> <mailto:[email protected]> >>>> >>>> https://lists.sourceforge.net/lists/listinfo/opensaf-devel >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> >>>> _______________________________________________ >>>> >>>> Opensaf-devel mailing list >>>> >>>> [email protected] >>>> <mailto:[email protected]> >>>> >>>> https://lists.sourceforge.net/lists/listinfo/opensaf-devel >>>> >>>> >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> _______________________________________________ >>>> Opensaf-devel mailing list >>>> [email protected] >>>> <mailto:[email protected]> >>>> https://lists.sourceforge.net/lists/listinfo/opensaf-devel >>>> >>> ------------------------------------------------------------------------------ >>> _______________________________________________ >>> Opensaf-devel mailing list >>> [email protected] >>> https://lists.sourceforge.net/lists/listinfo/opensaf-devel >>> ------------------------------------------------------------------------------ _______________________________________________ Opensaf-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/opensaf-devel
