Timothée Maret created SLING-9078:
-------------------------------------
Summary: Keep skipped offsets in memory
Key: SLING-9078
URL: https://issues.apache.org/jira/browse/SLING-9078
Project: Sling
Issue Type: Improvement
Components: Content Distribution
Affects Versions: Content Distribution Journal Core 0.1.8
Reporter: Timothée Maret
Fix For: Content Distribution Journal Core 0.1.10
In SLING-8908 we introduced a mechanism to keep track of skipped packages such
that in case of journal availability flip flop, the agents would still
progress. The implementation was done by persisting the offset in the
repository. In environments where many skipped packages will occur (e.g. large
customer with multiple agents) then those commits will start to generate a non
negligible load on the repository, consuming resources and creating bottlenecks.
A first mitigation to this issue has been added in SLING-8965, however for very
large customers the mitigation will not be effective. Increasing the grouping
of skipped packages (e.g. 1/100 instead of 1/10) reduce the load on the
repository but may fail to ensure progress in case of flip/flop.
Instead of storing the skipped packages in the repository, we should keep them
in memory. A new service that does not reference the {{JournalAvailable}}
service could be added. The offsets would be consolidated on demand from the
stored offsets and those stored in memory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
