Egor Baranov created IGNITE-29102:
-------------------------------------
Summary: Add a metric to track B+Tree cursor corruption failures
during SQL query execution
Key: IGNITE-29102
URL: https://issues.apache.org/jira/browse/IGNITE-29102
Project: Ignite
Issue Type: Task
Reporter: Egor Baranov
Assignee: Egor Baranov
{*}Problem{*}{*}{*}
When a B+Tree cursor encounters a broken invariant (typically an
AssertionError in AbstractForwardCursor.find() or
AbstractForwardCursor.fillFromBuffer()), the exception is wrapped
into a CorruptedTreeException and propagated up the call chain to
GridMapQueryExecutor. There, it is caught in a generic catch (Throwable e)
block and logged as "Failed to execute
local query" at ERROR/WARN level.
There is currently *no metric* that tracks how often such
corruption-related query failures occur. Operators can only detect this problem
by parsing log files, which is impractical for:
- Large clusters with high query throughput
- Intermittent corruption events that are hard to spot in log noise
- Automated alerting on infrastructure degradation
Example stack trace:
1 java.lang.AssertionError: null
2 at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.find(BPlusTree.java:5559)
3 at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.reinitialize(BPlusTree.java:5572)
4 at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.nextPage(BPlusTree.java:5627)
5 ...
6 at
org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0(GridMapQueryExecutor.java:538)
{*}Solution{*}{*}{*}
Add a dedicated long counter metric in the map query execution path that is
incremented whenever a query fails due to a CorruptedTreeException. This
follows the same pattern as
FailureProcessor metrics (IGNITE-29047) and the existing sql.queries.user
metrics.
{*}Changes{*}{*}{*}
1. *`GridMapQueryExecutor`* (modules/indexing):
- Register a new metric registry sql.queries.treeCorruption with a
single long metric count.
- Increment the counter in both error-handling paths (onQueryRequest0
and onNextPageRequest) when the caught throwable's cause chain contains a
CorruptedTreeException.
2. *Documentation* (docs/_docs/monitoring-metrics/new-metrics.adoc):
- Document the new registry and metric.
3. *Test* (TreeCorruptionMetricTest):
- Verify the metric is registered and initially zero.
- Simulate B+Tree corruption via BPlusTree.testHndWrapper, execute a
query, and assert the counter is incremented.
{*}Resulting Metric{*}{*}{*}
┌────────────────────────────┬───────┬──────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ *Registry* │ *Name* │ *Type* │ *Description*
│
├────────────────────────────┼───────┼──────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ sql.queries.treeCorruption │ count │ long │ Number of queries that failed
due to B+Tree cursor corruption (AssertionError during B+Tree page iteration). │
└────────────────────────────┴───────┴──────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
This metric enables:
- Automated alerting on tree corruption frequency
- Correlation of corruption events with specific operations (rebalance,
compaction, etc.)
- Quantification of data reliability without log scraping
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
