[ 
https://issues.apache.org/jira/browse/IGNITE-29297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18144825#comment-18144825
 ] 

Zhang Zhenhao commented on IGNITE-29297:
----------------------------------------

[~alex_pl] Could you please review the PR: 
[https://github.com/apache/ignite/pull/13834] Thank you!

> SQL onheap row cache shared by a cache group serves stale rows of one cache 
> to index operations of another cache in the group (node restarts up to 2.17, 
> silent wrong rows since 2.18)
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: IGNITE-29297
>                 URL: https://issues.apache.org/jira/browse/IGNITE-29297
>             Project: Ignite
>          Issue Type: Bug
>          Components: sql
>    Affects Versions: 2.16, 2.17, 2.18
>            Reporter: Zhang Zhenhao
>            Assignee: Zhang Zhenhao
>            Priority: Major
>         Attachments: IndexRowCacheCrossTablePoisonTest.java
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Two SQL tables (caches) share one CACHE_GROUP, and any cache of the group has 
> sqlOnheapCacheEnabled=true. Under these conditions the group-wide 
> IndexRowCache can keep stale entries after pages are freed in bulk without 
> per-row row-cache cleanup:
>  * Partition destroy on topology change / rebalancing — 
> GridCacheOffheapManager#destroyCacheDataStore0 → 
> GridCacheDatabaseSharedManager#schedulePartitionDestroy → 
> destroyPartitionStore invalidates and truncates the whole partition without 
> touching IndexRowCache (per-row cleanup only happens in RowStore#removeRow / 
> #updateRow; PageMemoryImpl#clearRowCache is only called on RAM page eviction).
>  * IndexRowCacheRegistry#onCacheUnregistered early-returns for caches with 
> sqlOnheapCacheEnabled=false, so clearForCache never removes their rows from 
> the group row cache — although those rows DO enter the group row cache, 
> because index definitions are wired to the group row cache regardless of the 
> per-cache flag (SortedIndexDescriptorFactory passes 
> rowCacheCleaner(cacheInfo.groupId())).
> After a freed (pageId, itemId) pair — a data row link — is reused by a row of 
> another table of the same group (all caches of a group share one 
> per-partition CacheDataRowStore/FreeList), index operations resolving that 
> link hit the stale entry. InlineIndexTree#createIndexRow re-wraps it with the 
> current index's row handler without any row-ownership validation:
> {code:java}
> // InlineIndexTree (2.16 - current master)
> IndexRowImpl cachedRow = idxRowCache == null ? null : idxRowCache.get(link);
> if (cachedRow != null) {
>     return cachedRow.rowHandler() == rowHandler() ? cachedRow :
>         new IndexRowImpl(rowHandler(), cachedRow.cacheDataRow()); // no 
> cacheId / type check
> }
> {code}
> *Failure modes by version (same root cause):*
>  * 2.16.0 / 2.17.0 — BinaryObjectException (see the production stack below), 
> wrapped into CorruptedTreeException: CRITICAL_ERROR kills the node 
> (RestartProcessFailureHandler), recurring "index is corrupted ... maintenance 
> mode" warnings. The maintenance rebuild "fixes" a tree that was never 
> corrupted: the poison is on-heap and dies with the JVM, but the cleaning gap 
> re-creates it on the next bulk free, so the failure returns after every 
> restart. QueryBinaryProperty#binaryField binds its BinaryField once in these 
> versions, so the foreign object fails the type check.
>  * 2.18.0 / current master — since IGNITE-26261 
> QueryBinaryProperty#binaryField re-binds to the type of the passed object on 
> typeId mismatch, which masks the exception: the index of one table is 
> silently maintained with field values of another table's rows, and a range 
> scan of one table returns rows of another table. No exception, no log. 
> Verified on master @ 5c02023000 with the attached test: a range scan of cache 
> A returned 50 rows of cache B out of 51 (the only correct row was inserted 
> after the row cache was last warmed).
> *Production stack (2.17.0, shared cache group, rebalancing in progress; 
> identifiers redacted):*
> {noformat}
> class org.apache.ignite.binary.BinaryObjectException: Failed to get field 
> because type ID of passed
> object differs from type ID this BinaryField belongs to 
> [expected=[typeId=..., typeName=SQL_PUBLIC_T_M3CPMOBL_<uuid>_KEY],
> actual=[typeId=..., typeName=SQL_PUBLIC_T_RL_ID_NEW_<uuid>_KEY], fieldId=..., 
> fieldName=PID, fieldType=String]
>     at 
> org.apache.ignite.internal.binary.BinaryFieldImpl.fieldOrder(BinaryFieldImpl.java:302)
>     at 
> org.apache.ignite.internal.binary.BinaryFieldImpl.value(BinaryFieldImpl.java:110)
>     at 
> org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.fieldValue(QueryBinaryProperty.java:229)
>     at 
> org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.value(QueryBinaryProperty.java:121)
>     at 
> org.apache.ignite.internal.cache.query.index.sorted.QueryIndexRowHandler.indexKey(QueryIndexRowHandler.java:93)
>     at 
> org.apache.ignite.internal.cache.query.index.sorted.IndexRowImpl.key(IndexRowImpl.java:67)
>     at 
> org.apache.ignite.internal.cache.query.index.sorted.IndexRowComparatorImpl.compareRow(IndexRowComparatorImpl.java:74)
>     at 
> org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.compareFullRows(InlineIndexTree.java:335)
>     at 
> org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.compare(InlineIndexTree.java:310)
>     at 
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:5293)
> class 
> org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
>  B+Tree is corrupted
> [groupId=..., cacheId=..., cacheName=T_M3CPMOBL, indexName=IDX_T_M3CPMOBL, 
> msg=Runtime failure on row: ...]
> {noformat}
> Both the put path (BPlusTree#doPut → findInsertionPoint → compare, index 
> maintenance on INSERT) and the find path (BPlusTree#find → traverse → 
> compare, index range scans) are affected.
> Topology note: on 2.8+ a group name must not collide with any registered 
> cache name (ClusterCachesInfo#checkCacheConflict), so the exposed topology is 
> a shared group with a neutral name.
> *Reproduction (test attached, verified on master @ 5c02023000, JDK 17; the 
> same poisoned state on 2.17.0 produces the BinaryObjectException above):*
>  # Two caches in one group: cache A with setSqlOnheapCacheEnabled(true) 
> created first, cache B with false created second (its rows enter the group 
> row cache anyway — wiring is group-level).
>  # Both caches have a sorted index on a String field with inlineSize=0 
> (forces every index comparison to resolve the full row through the row cache).
>  # Warm the row cache with range scans of both caches: 100 entries = 50 rows 
> of A + 50 rows of B.
>  # Inject the stale state a node ends up in after a bulk free followed by 
> link reuse: every cached entry under A's links is replaced with B's 
> materialized row.
>  # Range scan of cache A returns rows of cache B (50 of 51); INSERT into A 
> proceeds with comparisons against foreign field values (2.18+) or fails with 
> BinaryObjectException (2.17).
> {noformat}
> mvn -pl modules/indexing test -Dtest=IndexRowCacheCrossTablePoisonTest
> {noformat}
> The test asserts the defect version-agnostically (BinaryObjectException OR 
> foreign rows in the scan result), so it is green while the defect is present 
> and turns red after a fix.
> *Suggested fix:*
>  # InlineIndexTree#createIndexRow: validate that the cached entry belongs to 
> the current cache and evict + fall through to initFromLink on mismatch 
> (self-healing). NOTE: IndexRowImpl#cacheId() cannot be used for this — cache 
> IDs are stored in data pages only when the data region has page eviction 
> enabled (CacheGroupContext: storeCacheId = affNode && evictionMode != 
> DISABLED), so with the default configuration the materialized 
> CacheDataRowAdapter always reports cacheId=0 (this also makes 
> IndexRowCache#clearForCache a no-op without page eviction — the per-row 
> cleanup in RowStore is what actually cleans entries). The workable approach 
> is to stamp the asking tree's cacheId onto the entry when it is put into the 
> row cache, and compare that stamp on lookup.
>  # IndexRowCacheRegistry#onCacheUnregistered: remove the 
> isSqlOnheapCacheEnabled early-return; always run clearForCache for caches of 
> groups that have a row cache (and make it effective for eviction-disabled 
> groups, see above).
>  # Partition destroy paths (GridCacheOffheapManager#destroyCacheDataStore0 → 
> schedulePartitionDestroy → destroyPartitionStore) should sweep the group row 
> cache for the destroyed partition's links (the way 
> PageMemoryImpl#clearRowCache does for page eviction).
> *Workaround:* disable sqlOnheapCacheEnabled for the caches of the group, or 
> move the tables into separate cache groups. A node restart clears the poison 
> temporarily.
> *Related:* IGNITE-19495 (same-cache/multi-index predecessor of the re-wrap), 
> IGNITE-16484 (same symptom family, single cache), IGNITE-26261 (masks the 
> exception since 2.18 — the failure becomes silent), IGNITE-16661 (maintenance 
> index rebuild invoked by the false-positive corruption marking).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to