[
https://issues.apache.org/jira/browse/IGNITE-29297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhang Zhenhao updated IGNITE-29297:
-----------------------------------
Labels: ise (was: )
> SQL onheap row cache shared by a cache group serves stale rows of one cache
> to index operations of another cache in the group (node restarts up to 2.17,
> silent wrong rows since 2.18)
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: IGNITE-29297
> URL: https://issues.apache.org/jira/browse/IGNITE-29297
> Project: Ignite
> Issue Type: Bug
> Affects Versions: 2.16, 2.17, 2.18
> Reporter: Zhang Zhenhao
> Assignee: Zhang Zhenhao
> Priority: Major
> Labels: ise
> Attachments: IndexRowCacheCrossTablePoisonTest.java
>
>
> Two SQL tables (caches) share one CACHE_GROUP, and any cache of the group has
> sqlOnheapCacheEnabled=true. Under these conditions the group-wide
> IndexRowCache can keep stale entries after pages are freed in bulk without
> per-row row-cache cleanup:
> * Partition destroy on topology change / rebalancing —
> GridCacheOffheapManager#destroyCacheDataStore0 →
> GridCacheDatabaseSharedManager#schedulePartitionDestroy →
> destroyPartitionStore invalidates and truncates the whole partition without
> touching IndexRowCache (per-row cleanup only happens in RowStore#removeRow /
> #updateRow; PageMemoryImpl#clearRowCache is only called on RAM page eviction).
> * IndexRowCacheRegistry#onCacheUnregistered early-returns for caches with
> sqlOnheapCacheEnabled=false, so clearForCache never removes their rows from
> the group row cache — although those rows DO enter the group row cache,
> because index definitions are wired to the group row cache regardless of the
> per-cache flag (SortedIndexDescriptorFactory passes
> rowCacheCleaner(cacheInfo.groupId())).
> After a freed (pageId, itemId) pair — a data row link — is reused by a row of
> another table of the same group (all caches of a group share one
> per-partition CacheDataRowStore/FreeList), index operations resolving that
> link hit the stale entry. InlineIndexTree#createIndexRow re-wraps it with the
> current index's row handler without any row-ownership validation:
> {code:java}
> // InlineIndexTree (2.16 - current master)
> IndexRowImpl cachedRow = idxRowCache == null ? null : idxRowCache.get(link);
> if (cachedRow != null) {
> return cachedRow.rowHandler() == rowHandler() ? cachedRow :
> new IndexRowImpl(rowHandler(), cachedRow.cacheDataRow()); // no
> cacheId / type check
> }
> {code}
> *Failure modes by version (same root cause):*
> * 2.16.0 / 2.17.0 — BinaryObjectException (see the production stack below),
> wrapped into CorruptedTreeException: CRITICAL_ERROR kills the node
> (RestartProcessFailureHandler), recurring "index is corrupted ... maintenance
> mode" warnings. The maintenance rebuild "fixes" a tree that was never
> corrupted: the poison is on-heap and dies with the JVM, but the cleaning gap
> re-creates it on the next bulk free, so the failure returns after every
> restart. QueryBinaryProperty#binaryField binds its BinaryField once in these
> versions, so the foreign object fails the type check.
> * 2.18.0 / current master — since IGNITE-26261
> QueryBinaryProperty#binaryField re-binds to the type of the passed object on
> typeId mismatch, which masks the exception: the index of one table is
> silently maintained with field values of another table's rows, and a range
> scan of one table returns rows of another table. No exception, no log.
> Verified on master @ 5c02023000 with the attached test: a range scan of cache
> A returned 50 rows of cache B out of 51 (the only correct row was inserted
> after the row cache was last warmed).
> *Production stack (2.17.0, shared cache group, rebalancing in progress;
> identifiers redacted):*
> {noformat}
> class org.apache.ignite.binary.BinaryObjectException: Failed to get field
> because type ID of passed
> object differs from type ID this BinaryField belongs to
> [expected=[typeId=..., typeName=SQL_PUBLIC_T_M3CPMOBL_<uuid>_KEY],
> actual=[typeId=..., typeName=SQL_PUBLIC_T_RL_ID_NEW_<uuid>_KEY], fieldId=...,
> fieldName=PID, fieldType=String]
> at
> org.apache.ignite.internal.binary.BinaryFieldImpl.fieldOrder(BinaryFieldImpl.java:302)
> at
> org.apache.ignite.internal.binary.BinaryFieldImpl.value(BinaryFieldImpl.java:110)
> at
> org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.fieldValue(QueryBinaryProperty.java:229)
> at
> org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.value(QueryBinaryProperty.java:121)
> at
> org.apache.ignite.internal.cache.query.index.sorted.QueryIndexRowHandler.indexKey(QueryIndexRowHandler.java:93)
> at
> org.apache.ignite.internal.cache.query.index.sorted.IndexRowImpl.key(IndexRowImpl.java:67)
> at
> org.apache.ignite.internal.cache.query.index.sorted.IndexRowComparatorImpl.compareRow(IndexRowComparatorImpl.java:74)
> at
> org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.compareFullRows(InlineIndexTree.java:335)
> at
> org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.compare(InlineIndexTree.java:310)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:5293)
> class
> org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
> B+Tree is corrupted
> [groupId=..., cacheId=..., cacheName=T_M3CPMOBL, indexName=IDX_T_M3CPMOBL,
> msg=Runtime failure on row: ...]
> {noformat}
> Both the put path (BPlusTree#doPut → findInsertionPoint → compare, index
> maintenance on INSERT) and the find path (BPlusTree#find → traverse →
> compare, index range scans) are affected.
> Topology note: on 2.8+ a group name must not collide with any registered
> cache name (ClusterCachesInfo#checkCacheConflict), so the exposed topology is
> a shared group with a neutral name.
> *Reproduction (test attached, verified on master @ 5c02023000, JDK 17; the
> same poisoned state on 2.17.0 produces the BinaryObjectException above):*
> # Two caches in one group: cache A with setSqlOnheapCacheEnabled(true)
> created first, cache B with false created second (its rows enter the group
> row cache anyway — wiring is group-level).
> # Both caches have a sorted index on a String field with inlineSize=0
> (forces every index comparison to resolve the full row through the row cache).
> # Warm the row cache with range scans of both caches: 100 entries = 50 rows
> of A + 50 rows of B.
> # Inject the stale state a node ends up in after a bulk free followed by
> link reuse: every cached entry under A's links is replaced with B's
> materialized row.
> # Range scan of cache A returns rows of cache B (50 of 51); INSERT into A
> proceeds with comparisons against foreign field values (2.18+) or fails with
> BinaryObjectException (2.17).
> {noformat}
> mvn -pl modules/indexing test -Dtest=IndexRowCacheCrossTablePoisonTest
> {noformat}
> The test asserts the defect version-agnostically (BinaryObjectException OR
> foreign rows in the scan result), so it is green while the defect is present
> and turns red after a fix.
> *Suggested fix:*
> # InlineIndexTree#createIndexRow: validate that the cached entry belongs to
> the current cache and evict + fall through to initFromLink on mismatch
> (self-healing). NOTE: IndexRowImpl#cacheId() cannot be used for this — cache
> IDs are stored in data pages only when the data region has page eviction
> enabled (CacheGroupContext: storeCacheId = affNode && evictionMode !=
> DISABLED), so with the default configuration the materialized
> CacheDataRowAdapter always reports cacheId=0 (this also makes
> IndexRowCache#clearForCache a no-op without page eviction — the per-row
> cleanup in RowStore is what actually cleans entries). The workable approach
> is to stamp the asking tree's cacheId onto the entry when it is put into the
> row cache, and compare that stamp on lookup.
> # IndexRowCacheRegistry#onCacheUnregistered: remove the
> isSqlOnheapCacheEnabled early-return; always run clearForCache for caches of
> groups that have a row cache (and make it effective for eviction-disabled
> groups, see above).
> # Partition destroy paths (GridCacheOffheapManager#destroyCacheDataStore0 →
> schedulePartitionDestroy → destroyPartitionStore) should sweep the group row
> cache for the destroyed partition's links (the way
> PageMemoryImpl#clearRowCache does for page eviction).
> *Workaround:* disable sqlOnheapCacheEnabled for the caches of the group, or
> move the tables into separate cache groups. A node restart clears the poison
> temporarily.
> *Related:* IGNITE-19495 (same-cache/multi-index predecessor of the re-wrap),
> IGNITE-16484 (same symptom family, single cache), IGNITE-26261 (masks the
> exception since 2.18 — the failure becomes silent), IGNITE-16661 (maintenance
> index rebuild invoked by the false-positive corruption marking).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)