Juan Ramos created GEODE-8325:
---------------------------------
Summary: Remove Unused DRF Files More Frequently
Key: GEODE-8325
URL: https://issues.apache.org/jira/browse/GEODE-8325
Project: Geode
Issue Type: Improvement
Components: persistence
Reporter: Juan Ramos
Users have reported that, even with {{auto-compaction}} enabled, there are
certain scenarios for which there are too many orphaned {{drf}} files undeleted
on the {{disk-stores}}.
This causes slowness and memory spikes during the startup (we iterate over all
{{drf}} files and load records into memory as the first step when recovering
disk regions) and, prior to the fix for GEODE-8029, could also cause a member
to fail and shutdown.
We should implement a more long term fix, and make sure unused {{drf}} files
are removed more often (maybe have a dedicated {{OplogCompactor}} that
frequently removes the {{drf}} files even when the {{threshold}} is not hit?)
It would also be good to apply Darrel's suggestions:
{quote}
We should consider a more long term fix. We used to have code that would remove
drfs that were no longer needed. If we could do that more periodically it seems
like it would prevent a large number of drfs from building up.
Also it seems like the OplogEntryIdSet itself could be reimplemented to use
less memory. Since the ids for an diskstore always start at 1 and then simply
increase, and many use cases will end up with large sequences of ids that have
been deleted, we could consider using algorithms like we have on our
concurrency check code for tracking what version ids we have seen. Basically
you can just have a sequence (start of sequence and length) which can represent
a long sequence with just 2 32-bit values (or 2 64-bit values if longs).
Anything in the sequence range that is still alive you would note in another
data structure as an exception. If exceptions are rare then this can save lots
of memory.
{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
