>From Michael Blow <[email protected]>: Michael Blow has uploaded a new patch set (#2). ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21627?usp=email )
Change subject: [NO ISSUE][STO] Reclaim idle memory components when the vbc is full ...................................................................... [NO ISSUE][STO] Reclaim idle memory components when the vbc is full Above a few thousand concurrently ingesting datasets the global virtual buffer cache fills with memory components which hold no data, and ingestion stalls permanently with nothing logged by any logger. allocateMemoryComponents() allocates every memory component of an index on the first write to it, and each pins a metadata page and a root page as soon as its BTree is created. Only the component which is written is flushed, and only a flush cleans a component up and returns its pages; the siblings stay allocated and empty until the dataset is closed, which nothing does under memory pressure. A dataset written once therefore leaves pages behind for data which never arrived, and enough of them fill the cache with components selectFlushIndex() will never pick, because it only picks components which are not empty. Writers then fail threadEnter()'s !vbc.isFull() check and retry every 100ms forever, with nothing left which can make it false. The deadlock watchdog sees no cycle- the writer is in a timed wait, not blocked on a monitor. When a scheduleFlush() finds the cache full, nothing flushable and no flush in flight- so no completion is coming to free pages or to wake the flush thread again- reclaim the pages held by components which are allocated but INACTIVE, and warn if even that finds nothing. Reclaiming costs the next writer to that index a re-allocation, which is the same one it already pays after every flush. The components are not reset: they are INACTIVE, so they have already been reset or were never used, and resetting again would decrement the index's scheduled flush count a second time. The pass takes each index's operation tracker, which is what a writer holds while it activates a component, and copies primaryIndexes under the cache monitor first so that the cache-then-op-tracker order selectFlushIndex() establishes is preserved rather than extended. This bounds the damage rather than removing the residue; allocating the sibling components lazily, when the index switches to them, is the fix for that and is not done here. Co-Authored-By: Claude Opus 5 <[email protected]> Change-Id: I4cccd9857d277b1a32017f0afcbd644e33388f2f Ext-ref: MB-73593 --- M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/dataflow/GlobalVirtualBufferCacheTest.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/GlobalVirtualBufferCache.java 2 files changed, 178 insertions(+), 0 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/27/21627/2 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21627?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: newpatchset Gerrit-Project: asterixdb Gerrit-Branch: totoro Gerrit-Change-Id: I4cccd9857d277b1a32017f0afcbd644e33388f2f Gerrit-Change-Number: 21627 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Blow <[email protected]> Gerrit-CC: Jenkins <[email protected]>
