[
https://issues.apache.org/jira/browse/OAK-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15818093#comment-15818093
]
Stefan Eissing commented on OAK-5433:
-------------------------------------
I do not think there would be a performance issue and those could co-exist.
However each would not know about the other and, whatever they do, they might
interfere or, worst case, block each other.
So, I think it's definitely possible, but it might not be advisable. It can
always be left up to the user's/application's configuration, of course.
OTOH, it seems the CommitRateLimiter is already close to an implementation of
such a pacing service. Then one configures either the oak CommitRateLimiter as
Pacing Service, or another more application specific one. (Which could also
wrap that instance among other things it wants to do - the usual indirection
game.)
IMO, the pacing service should contain all the decision logic of when to limit
commit rates, e.g. when to delay and when to fail. Currently, this is all part
of the {{ChangeProcessor}} and it needs to be in a application specific
location.
> System Pacing Service
> ---------------------
>
> Key: OAK-5433
> URL: https://issues.apache.org/jira/browse/OAK-5433
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: core
> Reporter: Stefan Eissing
> Attachments: obs-pacing.diff
>
>
> h3. tl;dr
> By adding Pacing, suitable to the application {{oak}} is running in, a system
> will dynamically adapt the load to its own capabilities. This effectively, in
> tests, keeps the system stable and responsive under stress.
> h3. The Situation
> During experimental Lab tests on large clusters, it became clear that the a
> web system based using oak is challenged by fluctuating load in relation to
> its own capabilities.
> When the load increases "too much" it shows the following symptoms:
> * event observation queues grow
> * maintenance tasks (on master) take too long
> * async tasks, triggered by requests, (e.g. workflows) accumulate
> and eventually
> * login sessions complain about freshness
> * revisions diffs are old and no longer in caches
> and sometimes
> * database lease times out and oak-core shuts down
> This problem can arise when outside requests increased, or when local
> maintenance tasks occupy resources, or when available CPU diminishes due to
> other processes or page faults or, or, or.
> Unfortunately, whenever the system becomes overburdened, the secondary
> effects make the system even slower and, thus, more overburdened. This can
> end in a vicious circle, making the system total unresponsive. Eventual
> recovery is an option, not a guarantee.
> h3. Pacing
> By _Pacing_ I mean a system behaviour that tries to balance load in relation
> to capabilities. If the latter one drops, the load must be reduced until the
> system recovers. This is related to what the {{CommitRateLimiter}} wanted to
> achieve
> by monitoring observation queues.
> The design of the {{CommitRateLimiter}} could be very efficient, if it only
> know _which_ commits to delay. But it does not know the application that oak
> runs in. I propose replacing the Limiter by a {{PacingService}} that can be
> provided by the application using oak. The service will get the data about
> the current commit, queue length and limits. Whatever else it does remains
> opaque. It may raise a proper exception to indicate that the commit shall
> fail. But mostly, it is expected to delay those commits that would negatively
> affect system stability.
> h3. An Example
> In a proof of concept, an AEM system was blasted with endless uploads on
> multiple connections in order to eventually overwhelm queues. The a pacing
> was patched into oak-core that delayed commits from servlet requests and from
> certain workflows for some milliseconds until the queue length shrank again.
> The pacing had a maximum wait time that would make the commit fail.
> The pacer was configured to trigger at 75% of maximum queue length and the
> system was blasted with uploads again. In the tests:
> # the max queue length stayed under 80%
> # no upload did reach the maximum time, all succeeded
> The system adapted the external load to its capabilities successfully.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
