[
https://issues.apache.org/jira/browse/IGNITE-29297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhang Zhenhao updated IGNITE-29297:
-----------------------------------
Description:
Two SQL tables (caches) share one CACHE_GROUP, and any cache of the group has
sqlOnheapCacheEnabled=true. Under these conditions the group-wide IndexRowCache
can keep stale entries after pages are freed in bulk without per-row row-cache
cleanup:
* Partition destroy on topology change / rebalancing —
GridCacheOffheapManager#destroyCacheDataStore0 →
GridCacheDatabaseSharedManager#schedulePartitionDestroy → destroyPartitionStore
invalidates and truncates the whole partition without touching IndexRowCache
(per-row cleanup only happens in RowStore#removeRow / #updateRow;
PageMemoryImpl#clearRowCache is only called on RAM page eviction).
* IndexRowCacheRegistry#onCacheUnregistered early-returns for caches with
sqlOnheapCacheEnabled=false, so clearForCache never removes their rows from the
group row cache — although those rows DO enter the group row cache, because
index definitions are wired to the group row cache regardless of the per-cache
flag (SortedIndexDescriptorFactory passes rowCacheCleaner(cacheInfo.groupId())).
After a freed (pageId, itemId) pair — a data row link — is reused by a row of
another table of the same group (all caches of a group share one per-partition
CacheDataRowStore/FreeList), index operations resolving that link hit the stale
entry. InlineIndexTree#createIndexRow re-wraps it with the current index's row
handler without any row-ownership validation:
{code:java}
// InlineIndexTree (2.16 - current master)
IndexRowImpl cachedRow = idxRowCache == null ? null : idxRowCache.get(link);
if (cachedRow != null) {
return cachedRow.rowHandler() == rowHandler() ? cachedRow :
new IndexRowImpl(rowHandler(), cachedRow.cacheDataRow()); // no cacheId
/ type check
}
{code}
*Failure modes by version (same root cause):*
* 2.16.0 / 2.17.0 — BinaryObjectException (see the production stack below),
wrapped into CorruptedTreeException: CRITICAL_ERROR kills the node
(RestartProcessFailureHandler), recurring "index is corrupted ... maintenance
mode" warnings. The maintenance rebuild "fixes" a tree that was never
corrupted: the poison is on-heap and dies with the JVM, but the cleaning gap
re-creates it on the next bulk free, so the failure returns after every
restart. QueryBinaryProperty#binaryField binds its BinaryField once in these
versions, so the foreign object fails the type check.
* 2.18.0 / current master — since IGNITE-26261 QueryBinaryProperty#binaryField
re-binds to the type of the passed object on typeId mismatch, which masks the
exception: the index of one table is silently maintained with field values of
another table's rows, and a range scan of one table returns rows of another
table. No exception, no log. Verified on master @ 5c02023000 with the attached
test: a range scan of cache A returned 50 rows of cache B out of 51 (the only
correct row was inserted after the row cache was last warmed).
*Production stack (2.17.0, shared cache group, rebalancing in progress;
identifiers redacted):*
{noformat}
class org.apache.ignite.binary.BinaryObjectException: Failed to get field
because type ID of passed
object differs from type ID this BinaryField belongs to [expected=[typeId=...,
typeName=SQL_PUBLIC_T_M3CPMOBL_<uuid>_KEY],
actual=[typeId=..., typeName=SQL_PUBLIC_T_RL_ID_NEW_<uuid>_KEY], fieldId=...,
fieldName=PID, fieldType=String]
at
org.apache.ignite.internal.binary.BinaryFieldImpl.fieldOrder(BinaryFieldImpl.java:302)
at
org.apache.ignite.internal.binary.BinaryFieldImpl.value(BinaryFieldImpl.java:110)
at
org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.fieldValue(QueryBinaryProperty.java:229)
at
org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.value(QueryBinaryProperty.java:121)
at
org.apache.ignite.internal.cache.query.index.sorted.QueryIndexRowHandler.indexKey(QueryIndexRowHandler.java:93)
at
org.apache.ignite.internal.cache.query.index.sorted.IndexRowImpl.key(IndexRowImpl.java:67)
at
org.apache.ignite.internal.cache.query.index.sorted.IndexRowComparatorImpl.compareRow(IndexRowComparatorImpl.java:74)
at
org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.compareFullRows(InlineIndexTree.java:335)
at
org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.compare(InlineIndexTree.java:310)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:5293)
class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
B+Tree is corrupted
[groupId=..., cacheId=..., cacheName=T_M3CPMOBL, indexName=IDX_T_M3CPMOBL,
msg=Runtime failure on row: ...]
{noformat}
Both the put path (BPlusTree#doPut → findInsertionPoint → compare, index
maintenance on INSERT) and the find path (BPlusTree#find → traverse → compare,
index range scans) are affected.
Topology note: on 2.8+ a group name must not collide with any registered cache
name (ClusterCachesInfo#checkCacheConflict), so the exposed topology is a
shared group with a neutral name.
*Reproduction (test attached, verified on master @ 5c02023000, JDK 17; the same
poisoned state on 2.17.0 produces the BinaryObjectException above):*
# Two caches in one group: cache A with setSqlOnheapCacheEnabled(true) created
first, cache B with false created second (its rows enter the group row cache
anyway — wiring is group-level).
# Both caches have a sorted index on a String field with inlineSize=0 (forces
every index comparison to resolve the full row through the row cache).
# Warm the row cache with range scans of both caches: 100 entries = 50 rows of
A + 50 rows of B.
# Inject the stale state a node ends up in after a bulk free followed by link
reuse: every cached entry under A's links is replaced with B's materialized row.
# Range scan of cache A returns rows of cache B (50 of 51); INSERT into A
proceeds with comparisons against foreign field values (2.18+) or fails with
BinaryObjectException (2.17).
{noformat}
mvn -pl modules/indexing test -Dtest=IndexRowCacheCrossTablePoisonTest
{noformat}
The test asserts the defect version-agnostically (BinaryObjectException OR
foreign rows in the scan result), so it is green while the defect is present
and turns red after a fix.
*Suggested fix:*
# InlineIndexTree#createIndexRow: validate that the cached entry belongs to
the current cache and evict + fall through to initFromLink on mismatch
(self-healing). NOTE: IndexRowImpl#cacheId() cannot be used for this — cache
IDs are stored in data pages only when the data region has page eviction
enabled (CacheGroupContext: storeCacheId = affNode && evictionMode !=
DISABLED), so with the default configuration the materialized
CacheDataRowAdapter always reports cacheId=0 (this also makes
IndexRowCache#clearForCache a no-op without page eviction — the per-row cleanup
in RowStore is what actually cleans entries). The workable approach is to stamp
the asking tree's cacheId onto the entry when it is put into the row cache, and
compare that stamp on lookup.
# IndexRowCacheRegistry#onCacheUnregistered: remove the
isSqlOnheapCacheEnabled early-return; always run clearForCache for caches of
groups that have a row cache (and make it effective for eviction-disabled
groups, see above).
# Partition destroy paths (GridCacheOffheapManager#destroyCacheDataStore0 →
schedulePartitionDestroy → destroyPartitionStore) should sweep the group row
cache for the destroyed partition's links (the way PageMemoryImpl#clearRowCache
does for page eviction).
*Workaround:* disable sqlOnheapCacheEnabled for the caches of the group, or
move the tables into separate cache groups. A node restart clears the poison
temporarily.
*Related:* IGNITE-19495 (same-cache/multi-index predecessor of the re-wrap),
IGNITE-16484 (same symptom family, single cache), IGNITE-26261 (masks the
exception since 2.18 — the failure becomes silent), IGNITE-16661 (maintenance
index rebuild invoked by the false-positive corruption marking).
was:
```text
Two SQL tables (caches) share one CACHE_GROUP, and any cache of the group has
sqlOnheapCacheEnabled=true. Under these conditions the group-wide IndexRowCache
can keep stale entries after pages are freed in bulk without per-row row-cache
cleanup:
* Partition destroy on topology change / rebalancing —
GridCacheOffheapManager#destroyCacheDataStore0 →
GridCacheDatabaseSharedManager#schedulePartitionDestroy → destroyPartitionStore
invalidates and truncates the whole partition without touching IndexRowCache
(per-row cleanup only happens in RowStore#removeRow / #updateRow;
PageMemoryImpl#clearRowCache is only called on RAM page eviction).
* IndexRowCacheRegistry#onCacheUnregistered early-returns for caches with
sqlOnheapCacheEnabled=false, so clearForCache never removes their rows from the
group row cache — although those rows DO enter the group row cache, because
index definitions are wired to the group row cache regardless of the per-cache
flag (SortedIndexDescriptorFactory passes rowCacheCleaner(cacheInfo.groupId())).
After a freed (pageId, itemId) pair — a data row link — is reused by a row of
another table of the same group (all caches of a group share one per-partition
CacheDataRowStore/FreeList), index operations resolving that link hit the stale
entry. InlineIndexTree#createIndexRow re-wraps it with the current index's row
handler without any row-ownership validation:
{code:java}
// InlineIndexTree (2.16 - current master)
IndexRowImpl cachedRow = idxRowCache == null ? null : idxRowCache.get(link);
if (cachedRow != null) {
return cachedRow.rowHandler() == rowHandler() ? cachedRow :
new IndexRowImpl(rowHandler(), cachedRow.cacheDataRow()); // no cacheId
/ type check
}
{code}
*Failure modes by version (same root cause):*
* 2.16.0 / 2.17.0 — BinaryObjectException (see the production stack below),
wrapped into CorruptedTreeException: CRITICAL_ERROR kills the node
(RestartProcessFailureHandler), recurring "index is corrupted ... maintenance
mode" warnings. The maintenance rebuild "fixes" a tree that was never
corrupted: the poison is on-heap and dies with the JVM, but the cleaning gap
re-creates it on the next bulk free, so the failure returns after every
restart. QueryBinaryProperty#binaryField binds its BinaryField once in these
versions, so the foreign object fails the type check.
* 2.18.0 / current master — since IGNITE-26261 QueryBinaryProperty#binaryField
re-binds to the type of the passed object on typeId mismatch, which masks the
exception: the index of one table is silently maintained with field values of
another table's rows, and a range scan of one table returns rows of another
table. No exception, no log. Verified on master @ 5c02023000 with the attached
test: a range scan of cache A returned 50 rows of cache B out of 51 (the only
correct row was inserted after the row cache was last warmed).
*Production stack (2.17.0, shared cache group, rebalancing in progress;
identifiers redacted):*
{noformat}
class org.apache.ignite.binary.BinaryObjectException: Failed to get field
because type ID of passed
object differs from type ID this BinaryField belongs to [expected=[typeId=...,
typeName=SQL_PUBLIC_T_M3CPMOBL_<uuid>_KEY],
actual=[typeId=..., typeName=SQL_PUBLIC_T_RL_ID_NEW_<uuid>_KEY], fieldId=...,
fieldName=PID, fieldType=String]
at
org.apache.ignite.internal.binary.BinaryFieldImpl.fieldOrder(BinaryFieldImpl.java:302)
at
org.apache.ignite.internal.binary.BinaryFieldImpl.value(BinaryFieldImpl.java:110)
at
org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.fieldValue(QueryBinaryProperty.java:229)
at
org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.value(QueryBinaryProperty.java:121)
at
org.apache.ignite.internal.cache.query.index.sorted.QueryIndexRowHandler.indexKey(QueryIndexRowHandler.java:93)
at
org.apache.ignite.internal.cache.query.index.sorted.IndexRowImpl.key(IndexRowImpl.java:67)
at
org.apache.ignite.internal.cache.query.index.sorted.IndexRowComparatorImpl.compareRow(IndexRowComparatorImpl.java:74)
at
org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.compareFullRows(InlineIndexTree.java:335)
at
org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.compare(InlineIndexTree.java:310)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:5293)
class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
B+Tree is corrupted
[groupId=..., cacheId=..., cacheName=T_M3CPMOBL, indexName=IDX_T_M3CPMOBL,
msg=Runtime failure on row: ...]
{noformat}
Both the put path (BPlusTree#doPut → findInsertionPoint → compare, index
maintenance on INSERT) and the find path (BPlusTree#find → traverse → compare,
index range scans) are affected.
Topology note: on 2.8+ a group name must not collide with any registered cache
name (ClusterCachesInfo#checkCacheConflict), so the exposed topology is a
shared group with a neutral name.
*Reproduction (test attached, verified on master @ 5c02023000, JDK 17; the same
poisoned state on 2.17.0 produces the BinaryObjectException above):*
# Two caches in one group: cache A with setSqlOnheapCacheEnabled(true) created
first, cache B with false created second (its rows enter the group row cache
anyway — wiring is group-level).
# Both caches have a sorted index on a String field with inlineSize=0 (forces
every index comparison to resolve the full row through the row cache).
# Warm the row cache with range scans of both caches: 100 entries = 50 rows of
A + 50 rows of B.
# Inject the stale state a node ends up in after a bulk free followed by link
reuse: every cached entry under A's links is replaced with B's materialized row.
# Range scan of cache A returns rows of cache B (50 of 51); INSERT into A
proceeds with comparisons against foreign field values (2.18+) or fails with
BinaryObjectException (2.17).
{noformat}
mvn -pl modules/indexing test -Dtest=IndexRowCacheCrossTablePoisonTest
{noformat}
The test asserts the defect version-agnostically (BinaryObjectException OR
foreign rows in the scan result), so it is green while the defect is present
and turns red after a fix.
*Suggested fix:*
# InlineIndexTree#createIndexRow: validate that the cached entry belongs to the
current cache and evict + fall through to initFromLink on mismatch
(self-healing). NOTE: IndexRowImpl#cacheId() cannot be used for this — cache
IDs are stored in data pages only when the data region has page eviction
enabled (CacheGroupContext: storeCacheId = affNode && evictionMode !=
DISABLED), so with the default configuration the materialized
CacheDataRowAdapter always reports cacheId=0 (this also makes
IndexRowCache#clearForCache a no-op without page eviction — the per-row cleanup
in RowStore is what actually cleans entries). The workable approach is to stamp
the asking tree's cacheId onto the entry when it is put into the row cache, and
compare that stamp on lookup.
# IndexRowCacheRegistry#onCacheUnregistered: remove the isSqlOnheapCacheEnabled
early-return; always run clearForCache for caches of groups that have a row
cache (and make it effective for eviction-disabled groups, see above).
# Partition destroy paths (GridCacheOffheapManager#destroyCacheDataStore0 →
schedulePartitionDestroy → destroyPartitionStore) should sweep the group row
cache for the destroyed partition's links (the way PageMemoryImpl#clearRowCache
does for page eviction).
*Workaround:* disable sqlOnheapCacheEnabled for the caches of the group, or
move the tables into separate cache groups. A node restart clears the poison
temporarily.
*Related:* IGNITE-19495 (same-cache/multi-index predecessor of the re-wrap),
IGNITE-16484 (same symptom family, single cache), IGNITE-26261 (masks the
exception since 2.18 — the failure becomes silent), IGNITE-16661 (maintenance
index rebuild invoked by the false-positive corruption marking).
```
> SQL onheap row cache shared by a cache group serves stale rows of one cache
> to index operations of another cache in the group (node restarts up to 2.17,
> silent wrong rows since 2.18)
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: IGNITE-29297
> URL: https://issues.apache.org/jira/browse/IGNITE-29297
> Project: Ignite
> Issue Type: Bug
> Affects Versions: 2.16, 2.17, 2.18
> Reporter: Zhang Zhenhao
> Assignee: Zhang Zhenhao
> Priority: Major
> Attachments: IndexRowCacheCrossTablePoisonTest.java
>
>
> Two SQL tables (caches) share one CACHE_GROUP, and any cache of the group has
> sqlOnheapCacheEnabled=true. Under these conditions the group-wide
> IndexRowCache can keep stale entries after pages are freed in bulk without
> per-row row-cache cleanup:
> * Partition destroy on topology change / rebalancing —
> GridCacheOffheapManager#destroyCacheDataStore0 →
> GridCacheDatabaseSharedManager#schedulePartitionDestroy →
> destroyPartitionStore invalidates and truncates the whole partition without
> touching IndexRowCache (per-row cleanup only happens in RowStore#removeRow /
> #updateRow; PageMemoryImpl#clearRowCache is only called on RAM page eviction).
> * IndexRowCacheRegistry#onCacheUnregistered early-returns for caches with
> sqlOnheapCacheEnabled=false, so clearForCache never removes their rows from
> the group row cache — although those rows DO enter the group row cache,
> because index definitions are wired to the group row cache regardless of the
> per-cache flag (SortedIndexDescriptorFactory passes
> rowCacheCleaner(cacheInfo.groupId())).
> After a freed (pageId, itemId) pair — a data row link — is reused by a row of
> another table of the same group (all caches of a group share one
> per-partition CacheDataRowStore/FreeList), index operations resolving that
> link hit the stale entry. InlineIndexTree#createIndexRow re-wraps it with the
> current index's row handler without any row-ownership validation:
> {code:java}
> // InlineIndexTree (2.16 - current master)
> IndexRowImpl cachedRow = idxRowCache == null ? null : idxRowCache.get(link);
> if (cachedRow != null) {
> return cachedRow.rowHandler() == rowHandler() ? cachedRow :
> new IndexRowImpl(rowHandler(), cachedRow.cacheDataRow()); // no
> cacheId / type check
> }
> {code}
> *Failure modes by version (same root cause):*
> * 2.16.0 / 2.17.0 — BinaryObjectException (see the production stack below),
> wrapped into CorruptedTreeException: CRITICAL_ERROR kills the node
> (RestartProcessFailureHandler), recurring "index is corrupted ... maintenance
> mode" warnings. The maintenance rebuild "fixes" a tree that was never
> corrupted: the poison is on-heap and dies with the JVM, but the cleaning gap
> re-creates it on the next bulk free, so the failure returns after every
> restart. QueryBinaryProperty#binaryField binds its BinaryField once in these
> versions, so the foreign object fails the type check.
> * 2.18.0 / current master — since IGNITE-26261
> QueryBinaryProperty#binaryField re-binds to the type of the passed object on
> typeId mismatch, which masks the exception: the index of one table is
> silently maintained with field values of another table's rows, and a range
> scan of one table returns rows of another table. No exception, no log.
> Verified on master @ 5c02023000 with the attached test: a range scan of cache
> A returned 50 rows of cache B out of 51 (the only correct row was inserted
> after the row cache was last warmed).
> *Production stack (2.17.0, shared cache group, rebalancing in progress;
> identifiers redacted):*
> {noformat}
> class org.apache.ignite.binary.BinaryObjectException: Failed to get field
> because type ID of passed
> object differs from type ID this BinaryField belongs to
> [expected=[typeId=..., typeName=SQL_PUBLIC_T_M3CPMOBL_<uuid>_KEY],
> actual=[typeId=..., typeName=SQL_PUBLIC_T_RL_ID_NEW_<uuid>_KEY], fieldId=...,
> fieldName=PID, fieldType=String]
> at
> org.apache.ignite.internal.binary.BinaryFieldImpl.fieldOrder(BinaryFieldImpl.java:302)
> at
> org.apache.ignite.internal.binary.BinaryFieldImpl.value(BinaryFieldImpl.java:110)
> at
> org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.fieldValue(QueryBinaryProperty.java:229)
> at
> org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.value(QueryBinaryProperty.java:121)
> at
> org.apache.ignite.internal.cache.query.index.sorted.QueryIndexRowHandler.indexKey(QueryIndexRowHandler.java:93)
> at
> org.apache.ignite.internal.cache.query.index.sorted.IndexRowImpl.key(IndexRowImpl.java:67)
> at
> org.apache.ignite.internal.cache.query.index.sorted.IndexRowComparatorImpl.compareRow(IndexRowComparatorImpl.java:74)
> at
> org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.compareFullRows(InlineIndexTree.java:335)
> at
> org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.compare(InlineIndexTree.java:310)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:5293)
> class
> org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
> B+Tree is corrupted
> [groupId=..., cacheId=..., cacheName=T_M3CPMOBL, indexName=IDX_T_M3CPMOBL,
> msg=Runtime failure on row: ...]
> {noformat}
> Both the put path (BPlusTree#doPut → findInsertionPoint → compare, index
> maintenance on INSERT) and the find path (BPlusTree#find → traverse →
> compare, index range scans) are affected.
> Topology note: on 2.8+ a group name must not collide with any registered
> cache name (ClusterCachesInfo#checkCacheConflict), so the exposed topology is
> a shared group with a neutral name.
> *Reproduction (test attached, verified on master @ 5c02023000, JDK 17; the
> same poisoned state on 2.17.0 produces the BinaryObjectException above):*
> # Two caches in one group: cache A with setSqlOnheapCacheEnabled(true)
> created first, cache B with false created second (its rows enter the group
> row cache anyway — wiring is group-level).
> # Both caches have a sorted index on a String field with inlineSize=0
> (forces every index comparison to resolve the full row through the row cache).
> # Warm the row cache with range scans of both caches: 100 entries = 50 rows
> of A + 50 rows of B.
> # Inject the stale state a node ends up in after a bulk free followed by
> link reuse: every cached entry under A's links is replaced with B's
> materialized row.
> # Range scan of cache A returns rows of cache B (50 of 51); INSERT into A
> proceeds with comparisons against foreign field values (2.18+) or fails with
> BinaryObjectException (2.17).
> {noformat}
> mvn -pl modules/indexing test -Dtest=IndexRowCacheCrossTablePoisonTest
> {noformat}
> The test asserts the defect version-agnostically (BinaryObjectException OR
> foreign rows in the scan result), so it is green while the defect is present
> and turns red after a fix.
> *Suggested fix:*
> # InlineIndexTree#createIndexRow: validate that the cached entry belongs to
> the current cache and evict + fall through to initFromLink on mismatch
> (self-healing). NOTE: IndexRowImpl#cacheId() cannot be used for this — cache
> IDs are stored in data pages only when the data region has page eviction
> enabled (CacheGroupContext: storeCacheId = affNode && evictionMode !=
> DISABLED), so with the default configuration the materialized
> CacheDataRowAdapter always reports cacheId=0 (this also makes
> IndexRowCache#clearForCache a no-op without page eviction — the per-row
> cleanup in RowStore is what actually cleans entries). The workable approach
> is to stamp the asking tree's cacheId onto the entry when it is put into the
> row cache, and compare that stamp on lookup.
> # IndexRowCacheRegistry#onCacheUnregistered: remove the
> isSqlOnheapCacheEnabled early-return; always run clearForCache for caches of
> groups that have a row cache (and make it effective for eviction-disabled
> groups, see above).
> # Partition destroy paths (GridCacheOffheapManager#destroyCacheDataStore0 →
> schedulePartitionDestroy → destroyPartitionStore) should sweep the group row
> cache for the destroyed partition's links (the way
> PageMemoryImpl#clearRowCache does for page eviction).
> *Workaround:* disable sqlOnheapCacheEnabled for the caches of the group, or
> move the tables into separate cache groups. A node restart clears the poison
> temporarily.
> *Related:* IGNITE-19495 (same-cache/multi-index predecessor of the re-wrap),
> IGNITE-16484 (same symptom family, single cache), IGNITE-26261 (masks the
> exception since 2.18 — the failure becomes silent), IGNITE-16661 (maintenance
> index rebuild invoked by the false-positive corruption marking).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)