Zhang Zhenhao created IGNITE-29297:
--------------------------------------

             Summary: SQL onheap row cache shared by a cache group serves stale 
rows of one cache to index operations of another cache in the group (node 
restarts up to 2.17, silent wrong rows since 2.18)
                 Key: IGNITE-29297
                 URL: https://issues.apache.org/jira/browse/IGNITE-29297
             Project: Ignite
          Issue Type: Bug
    Affects Versions: 2.18, 2.17, 2.16
            Reporter: Zhang Zhenhao
            Assignee: Zhang Zhenhao
         Attachments: IndexRowCacheCrossTablePoisonTest.java

```text
Two SQL tables (caches) share one CACHE_GROUP, and any cache of the group has 
sqlOnheapCacheEnabled=true. Under these conditions the group-wide IndexRowCache 
can keep stale entries after pages are freed in bulk without per-row row-cache 
cleanup:
* Partition destroy on topology change / rebalancing — 
GridCacheOffheapManager#destroyCacheDataStore0 → 
GridCacheDatabaseSharedManager#schedulePartitionDestroy → destroyPartitionStore 
invalidates and truncates the whole partition without touching IndexRowCache 
(per-row cleanup only happens in RowStore#removeRow / #updateRow; 
PageMemoryImpl#clearRowCache is only called on RAM page eviction).
* IndexRowCacheRegistry#onCacheUnregistered early-returns for caches with 
sqlOnheapCacheEnabled=false, so clearForCache never removes their rows from the 
group row cache — although those rows DO enter the group row cache, because 
index definitions are wired to the group row cache regardless of the per-cache 
flag (SortedIndexDescriptorFactory passes rowCacheCleaner(cacheInfo.groupId())).

After a freed (pageId, itemId) pair — a data row link — is reused by a row of 
another table of the same group (all caches of a group share one per-partition 
CacheDataRowStore/FreeList), index operations resolving that link hit the stale 
entry. InlineIndexTree#createIndexRow re-wraps it with the current index's row 
handler without any row-ownership validation:

{code:java}
// InlineIndexTree (2.16 - current master)
IndexRowImpl cachedRow = idxRowCache == null ? null : idxRowCache.get(link);
if (cachedRow != null) {
    return cachedRow.rowHandler() == rowHandler() ? cachedRow :
        new IndexRowImpl(rowHandler(), cachedRow.cacheDataRow()); // no cacheId 
/ type check
}
{code}

*Failure modes by version (same root cause):*
* 2.16.0 / 2.17.0 — BinaryObjectException (see the production stack below), 
wrapped into CorruptedTreeException: CRITICAL_ERROR kills the node 
(RestartProcessFailureHandler), recurring "index is corrupted ... maintenance 
mode" warnings. The maintenance rebuild "fixes" a tree that was never 
corrupted: the poison is on-heap and dies with the JVM, but the cleaning gap 
re-creates it on the next bulk free, so the failure returns after every 
restart. QueryBinaryProperty#binaryField binds its BinaryField once in these 
versions, so the foreign object fails the type check.
* 2.18.0 / current master — since IGNITE-26261 QueryBinaryProperty#binaryField 
re-binds to the type of the passed object on typeId mismatch, which masks the 
exception: the index of one table is silently maintained with field values of 
another table's rows, and a range scan of one table returns rows of another 
table. No exception, no log. Verified on master @ 5c02023000 with the attached 
test: a range scan of cache A returned 50 rows of cache B out of 51 (the only 
correct row was inserted after the row cache was last warmed).

*Production stack (2.17.0, shared cache group, rebalancing in progress; 
identifiers redacted):*
{noformat}
class org.apache.ignite.binary.BinaryObjectException: Failed to get field 
because type ID of passed
object differs from type ID this BinaryField belongs to [expected=[typeId=..., 
typeName=SQL_PUBLIC_T_M3CPMOBL_<uuid>_KEY],
actual=[typeId=..., typeName=SQL_PUBLIC_T_RL_ID_NEW_<uuid>_KEY], fieldId=..., 
fieldName=PID, fieldType=String]
    at 
org.apache.ignite.internal.binary.BinaryFieldImpl.fieldOrder(BinaryFieldImpl.java:302)
    at 
org.apache.ignite.internal.binary.BinaryFieldImpl.value(BinaryFieldImpl.java:110)
    at 
org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.fieldValue(QueryBinaryProperty.java:229)
    at 
org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.value(QueryBinaryProperty.java:121)
    at 
org.apache.ignite.internal.cache.query.index.sorted.QueryIndexRowHandler.indexKey(QueryIndexRowHandler.java:93)
    at 
org.apache.ignite.internal.cache.query.index.sorted.IndexRowImpl.key(IndexRowImpl.java:67)
    at 
org.apache.ignite.internal.cache.query.index.sorted.IndexRowComparatorImpl.compareRow(IndexRowComparatorImpl.java:74)
    at 
org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.compareFullRows(InlineIndexTree.java:335)
    at 
org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.compare(InlineIndexTree.java:310)
    at 
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:5293)
class 
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
 B+Tree is corrupted
[groupId=..., cacheId=..., cacheName=T_M3CPMOBL, indexName=IDX_T_M3CPMOBL, 
msg=Runtime failure on row: ...]
{noformat}
Both the put path (BPlusTree#doPut → findInsertionPoint → compare, index 
maintenance on INSERT) and the find path (BPlusTree#find → traverse → compare, 
index range scans) are affected.

Topology note: on 2.8+ a group name must not collide with any registered cache 
name (ClusterCachesInfo#checkCacheConflict), so the exposed topology is a 
shared group with a neutral name.

*Reproduction (test attached, verified on master @ 5c02023000, JDK 17; the same 
poisoned state on 2.17.0 produces the BinaryObjectException above):*
# Two caches in one group: cache A with setSqlOnheapCacheEnabled(true) created 
first, cache B with false created second (its rows enter the group row cache 
anyway — wiring is group-level).
# Both caches have a sorted index on a String field with inlineSize=0 (forces 
every index comparison to resolve the full row through the row cache).
# Warm the row cache with range scans of both caches: 100 entries = 50 rows of 
A + 50 rows of B.
# Inject the stale state a node ends up in after a bulk free followed by link 
reuse: every cached entry under A's links is replaced with B's materialized row.
# Range scan of cache A returns rows of cache B (50 of 51); INSERT into A 
proceeds with comparisons against foreign field values (2.18+) or fails with 
BinaryObjectException (2.17).

{noformat}
mvn -pl modules/indexing test -Dtest=IndexRowCacheCrossTablePoisonTest
{noformat}
The test asserts the defect version-agnostically (BinaryObjectException OR 
foreign rows in the scan result), so it is green while the defect is present 
and turns red after a fix.

*Suggested fix:*
# InlineIndexTree#createIndexRow: validate that the cached entry belongs to the 
current cache and evict + fall through to initFromLink on mismatch 
(self-healing). NOTE: IndexRowImpl#cacheId() cannot be used for this — cache 
IDs are stored in data pages only when the data region has page eviction 
enabled (CacheGroupContext: storeCacheId = affNode && evictionMode != 
DISABLED), so with the default configuration the materialized 
CacheDataRowAdapter always reports cacheId=0 (this also makes 
IndexRowCache#clearForCache a no-op without page eviction — the per-row cleanup 
in RowStore is what actually cleans entries). The workable approach is to stamp 
the asking tree's cacheId onto the entry when it is put into the row cache, and 
compare that stamp on lookup.
# IndexRowCacheRegistry#onCacheUnregistered: remove the isSqlOnheapCacheEnabled 
early-return; always run clearForCache for caches of groups that have a row 
cache (and make it effective for eviction-disabled groups, see above).
# Partition destroy paths (GridCacheOffheapManager#destroyCacheDataStore0 → 
schedulePartitionDestroy → destroyPartitionStore) should sweep the group row 
cache for the destroyed partition's links (the way PageMemoryImpl#clearRowCache 
does for page eviction).

*Workaround:* disable sqlOnheapCacheEnabled for the caches of the group, or 
move the tables into separate cache groups. A node restart clears the poison 
temporarily.

*Related:* IGNITE-19495 (same-cache/multi-index predecessor of the re-wrap), 
IGNITE-16484 (same symptom family, single cache), IGNITE-26261 (masks the 
exception since 2.18 — the failure becomes silent), IGNITE-16661 (maintenance 
index rebuild invoked by the false-positive corruption marking).
```



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to