>From Ali Alsuliman <[email protected]>: Ali Alsuliman has uploaded this change for review. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21703?usp=email )
Change subject: [ASTERIXDB-XXXX][STO] Release pages held by exhausted sample cursors ...................................................................... [ASTERIXDB-XXXX][STO] Release pages held by exhausted sample cursors - user model changes: no - storage format changes: no - interface changes: no Details: LSMIndexSampleCursor opened a sample cursor per disk component and advanced through them without closing the exhausted one, so every visited component kept its pinned pages until the whole scan finished. For a columnar component that is page0 plus the entire column page set of the last mega-leaf it materialized -- up to storage.column.max.leaf.node.size / bufferCachePageSize frames, and ANALYZE projects every field, so effectively the whole leaf. Frames held therefore grew as numDiskBTrees x megaLeafPages x partitions and exhausted the buffer cache on datasets with many components and wide records, failing with "Unable to find free page in buffer cache after 1000 cycles". Row-format components leaked a single leaf page each, which kept this hidden. Close each per-component cursor before advancing. close() is idempotent, so closeCursors() may still close it again on the way out. Also release in ColumnBtreeSampleCursor.yieldNextFromPhase2 when the samples run out: Phase 2 released a mega-leaf only on a page-id change, so the final page's column set stayed pinned until doClose(). This additionally covers the early return in LSMIndexSampleCursor.doHasNext when the sample cardinality is reached, which never reaches the component cursor. Ext-ref: MB-73705 Co-Authored-By: Claude Opus 5 <[email protected]> Change-Id: I708ec383df2bd490c297d29bdbb31092825118d3 --- M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree-column/src/main/java/org/apache/hyracks/storage/am/lsm/btree/column/impls/btree/ColumnBtreeSampleCursor.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/impls/LSMIndexSampleCursor.java 2 files changed, 22 insertions(+), 1 deletion(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/03/21703/1 diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree-column/src/main/java/org/apache/hyracks/storage/am/lsm/btree/column/impls/btree/ColumnBtreeSampleCursor.java b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree-column/src/main/java/org/apache/hyracks/storage/am/lsm/btree/column/impls/btree/ColumnBtreeSampleCursor.java index a123882..67aab71 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree-column/src/main/java/org/apache/hyracks/storage/am/lsm/btree/column/impls/btree/ColumnBtreeSampleCursor.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree-column/src/main/java/org/apache/hyracks/storage/am/lsm/btree/column/impls/btree/ColumnBtreeSampleCursor.java @@ -423,8 +423,17 @@ // Phase 2: Sorted column collection — one mega-page load per page // ────────────────────────────────────────────────────────────────────── + @AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_CLI, contributionKind = AiProvenance.ContributionKind.ASSISTED, notes = "Release the final mega-leaf's pages when Phase 2 is exhausted instead of holding them " + + "until doClose()") private boolean yieldNextFromPhase2() throws HyracksDataException { if (yieldPos >= collectedCount) { + /* + * Phase 2 releases a mega-leaf's pages only when the page id changes, so when the samples run out + * the last page's column set is still pinned. Release it here rather than holding a full mega-leaf + * of frames until doClose(). releasePages() is idempotent: context.release/unpinAll and + * unpinColumnsPages clear their page lists, and unpinCurrentPage0 no-ops on a null page0. + */ + releasePages(); return false; } diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/impls/LSMIndexSampleCursor.java b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/impls/LSMIndexSampleCursor.java index aac5ba7..3fab1f6 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/impls/LSMIndexSampleCursor.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/impls/LSMIndexSampleCursor.java @@ -313,6 +313,8 @@ } @Override + @AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_CLI, contributionKind = AiProvenance.ContributionKind.ASSISTED, notes = "Close each exhausted per-component sample cursor before advancing, so its pinned pages are " + + "not retained for the rest of the scan") protected boolean doHasNext() throws HyracksDataException { // Need to find the next tuple in the sample. if (numDiskBTrees == 0 || sampledCount == sampleCardinality) { @@ -329,7 +331,17 @@ return true; } - // Current disk component cursor has no more tuples, move to the next + /* + * Current disk component cursor has no more tuples, move to the next. Close it first: an exhausted + * but still-open cursor holds its pinned pages until its own close(), and for a columnar component + * that is page0 plus the whole column page set of the last mega-leaf it materialized (up to + * storage.column.max.leaf.node.size / bufferCachePageSize frames, and ANALYZE projects every field + * so effectively the entire leaf). Leaving every visited component open therefore pins + * numDiskBTrees x megaLeafPages frames for the remainder of the scan, which exhausts the buffer + * cache on datasets with many disk components and wide records ("Unable to find free page in buffer + * cache after N cycles"). close() is idempotent, so closeCursors() may still close it again. + */ + btreeCursors[currentDiskComponentIndex].close(); currentDiskComponentIndex++; if (currentDiskComponentIndex >= numDiskBTrees) { return false; // No more disk components to sample from. -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21703?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I708ec383df2bd490c297d29bdbb31092825118d3 Gerrit-Change-Number: 21703 Gerrit-PatchSet: 1 Gerrit-Owner: Ali Alsuliman <[email protected]>
