>From Michael Blow <[email protected]>:
Attention is currently required from: Ali Alsuliman, Michael Blow, Murtadha
Hubail, Ritik Raj.
Hello Ali Alsuliman, Ian Maxon, Jenkins, Murtadha Hubail, Ritik Raj,
I'd like you to reexamine a change. Please visit
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21627?usp=email
to look at the new patch set (#3).
The following approvals got outdated and were removed:
Code-Review+1 by Michael Blow, Verified+1 by Jenkins
Change subject: [ASTERIXDB-3940][STO] Reclaim idle memory components when vbc
is full
......................................................................
[ASTERIXDB-3940][STO] Reclaim idle memory components when vbc is full
- user model changes: no
- storage format changes: no
- interface changes: no
Details: An index which has never been written still allocates its
current memory component when it is first entered, by a read or by a
scheduled flush as well as by a write, and the component pins a metadata
page and a root page as soon as its BTree is created. Only a write
activates it, so it stays INACTIVE and keeps those pages until the
dataset is closed: only a flush returns a component's pages, and the
flush of an empty primary index is skipped. Enough such indexes fill the
global virtual buffer cache with components selectFlushIndex() will
never pick, because it only picks components which are not empty.
Writers then fail threadEnter()'s !vbc.isFull() check and retry every
100ms forever, with nothing left which can make it false, and nothing is
logged. The deadlock watchdog sees no cycle- the writer is in a timed
wait, not blocked on a monitor.
Selecting such components for flush would not help. A selected index
stays in flushingIndexes until flushed() removes it, which a skipped
flush never reaches, so each one would hold a flush slot for good.
So when a scheduleFlush() finds the cache full, nothing flushable and no
flush in flight- so no completion is coming to free pages or to wake the
flush thread again- reclaim the pages held by components which are
allocated but INACTIVE, and warn if even that finds nothing. Reclaiming
costs the next writer to that index a re-allocation, which is the same
one it already pays after every flush. The components are not reset:
they are INACTIVE, so they have already been reset or were never used,
and resetting again would decrement the index's scheduled flush count a
second time.
The pass takes each index's operation tracker, which is what a writer
holds while it activates a component, and copies primaryIndexes under
the cache monitor first so that the cache-then-op-tracker order
selectFlushIndex() establishes is preserved rather than extended.
Ext-ref: MB-74383
Generated-by: Claude Opus 5 <[email protected]>
Generated-by: Claude Opus 5.5 <[email protected]>
Change-Id: I4cccd9857d277b1a32017f0afcbd644e33388f2f
---
M
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/GlobalVirtualBufferCacheTest.java
M
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/GlobalVirtualBufferCache.java
2 files changed, 184 insertions(+), 0 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb
refs/changes/27/21627/3
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21627?usp=email
To unsubscribe, or for help writing mail filters, visit
https://asterix-gerrit.ics.uci.edu/settings?usp=email
Gerrit-MessageType: newpatchset
Gerrit-Project: asterixdb
Gerrit-Branch: totoro
Gerrit-Change-Id: I4cccd9857d277b1a32017f0afcbd644e33388f2f
Gerrit-Change-Number: 21627
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Blow <[email protected]>
Gerrit-Reviewer: Ali Alsuliman <[email protected]>
Gerrit-Reviewer: Ian Maxon <[email protected]>
Gerrit-Reviewer: Jenkins <[email protected]>
Gerrit-Reviewer: Michael Blow <[email protected]>
Gerrit-Reviewer: Murtadha Hubail <[email protected]>
Gerrit-Reviewer: Ritik Raj <[email protected]>
Gerrit-Attention: Murtadha Hubail <[email protected]>
Gerrit-Attention: Ali Alsuliman <[email protected]>
Gerrit-Attention: Michael Blow <[email protected]>
Gerrit-Attention: Ritik Raj <[email protected]>
