Hi Devs,
*Why ${subject} **? *
*I*mplement "Circuit Breaker" pattern in user store manager is becoming an
essential part when it comes to multi tenant and multi-user store manager
(USM) use case in IS. Here are the reasons.
a) IS connects heterogeneous user stores implemented in LDAP/AD, JDBC, AWS,
NoSQL, which has different timing characteristics.
b) Each user store may be hosted in locations outside the data center which
IS resides. The network delay, connection characteristics affects IO-Waits.
c) Having single User-Store which causes few seconds of IO wait can starve
all the HTTP processing thread pool (Tomcat pool) when there is average TPS
(e.g. 100TPS) hits to offending user store.
*How?*
Hence I propose adding a layer around user store manager calls. What it
does are,
1. Track delay in each call to user store manager.
2. Keep histogram of delay vs each call. History is kept for few minutes in
memory.
3. Throttle down calls to any user store manager if there is considerable
delay in a particular USM. Report the case in error log.
4. Throttle down is to throw a variant of IOException, so that the call
(authentication, get claim, etc) will fail fast.
5. This will help not to starve tomcat thread pool un-necessarily on
mis-behaving (slow) USM, so that the system is kept responsive.
*Algorithm*
1. Calculation of histogram
H = Number of request received(per USM)* IO Delay of each request(per USM)/
sum(Number of request received(per USM)* IO Delay of each request(per USM))
2. Activate throttling
Throttle activation, if Threads blocked in USM > pre-defined factor * total
tomcat threads
3. Throttling
IOException for each request when,
3.1 - H > 0.1 (say) and IO Wait > 50ms (say)( both factors are configurable)
3.2 - Every request in ratio of H will be thrown IOException.
With the above algorithm, the circuit breaker is kicked in when there is
significant IO Delay and the threads seem to starve due to that. There will
be no throttling when system behaves well, when no significant IO (network)
delay.
*Effort*
Adding a layer to do the "circuit-breaker" is not something hard to do. We
need to wrap all the calls to existing USM with "CircuitBreaker" (new
class) which keeps track of calls and throw necessary IOException.
Cheers,
Ruwan
--
*Ruwan Abeykoon*
*Associate Director/Architect**,*
*WSO2, Inc. http://wso2.com <https://wso2.com/signature> *
*lean.enterprise.middleware.*
_______________________________________________
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
