[
https://issues.apache.org/jira/browse/IGNITE-29102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Egor Baranov updated IGNITE-29102:
----------------------------------
Description:
*Problem:*
When a B+Tree cursor encounters a broken invariant (typically an AssertionError
in AbstractForwardCursor.find() or AbstractForwardCursor.fillFromBuffer()), the
exception is wrapped into a CorruptedTreeException and propagated up the call
chain to GridMapQueryExecutor. There, it is caught in a generic catch
(Throwable e) block and logged as "Failed to execute local query" at ERROR/WARN
level.
There is currently *no metric* that tracks how often such corruption-related
query failures occur. Operators can only detect this problem by parsing log
files, which is impractical for:
- Large clusters with high query throughput
- Intermittent corruption events that are hard to spot in log noise
- Automated alerting on infrastructure degradation
Example stack trace:
{code:text}
java.lang.AssertionError: null
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.find(BPlusTree.java:5559)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.reinitialize(BPlusTree.java:5572)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.nextPage(BPlusTree.java:5627)
...
at
org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0(GridMapQueryExecutor.java:538)
{code}
{*}Solution{*}{*}{{*}}
Add a dedicated long counter metric in the map query execution path that is
incremented whenever a query fails due to a CorruptedTreeException.
{*}Resulting Metric{*}{*}{{*}}
||*Registry*||*Name*||*Type*||*Description*||
|sql.queries.treeCorruption|count|long|Number of queries that failed due to
B+Tree cursor corruption (AssertionError during B+Tree page iteration)|
*This metric enables:*
- Automated alerting on tree corruption frequency
- Correlation of corruption events with specific operations (rebalance,
compaction, etc.)
- Quantification of data reliability without log scraping
was:
*Problem:*
When a B+Tree cursor encounters a broken invariant (typically an AssertionError
in AbstractForwardCursor.find() or AbstractForwardCursor.fillFromBuffer()), the
exception is wrapped into a CorruptedTreeException and propagated up the call
chain to GridMapQueryExecutor. There, it is caught in a generic catch
(Throwable e) block and logged as "Failed to execute local query" at ERROR/WARN
level.
There is currently *no metric* that tracks how often such corruption-related
query failures occur. Operators can only detect this problem by parsing log
files, which is impractical for:
- Large clusters with high query throughput
- Intermittent corruption events that are hard to spot in log noise
- Automated alerting on infrastructure degradation
Example stack trace:
{code:text}
java.lang.AssertionError: null
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.find(BPlusTree.java:5559)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.reinitialize(BPlusTree.java:5572)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.nextPage(BPlusTree.java:5627)
...
at
org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0(GridMapQueryExecutor.java:538)
{code}
{*}Solution{*}{*}{{*}}
Add a dedicated long counter metric in the map query execution path that is
incremented whenever a query fails due to a CorruptedTreeException.
{*}Resulting Metric{*}{*}{{*}}
||*Registry*||*Name*||*Type*||*Description*||
|sql.queries.treeCorruption|count|long|"Number of queries that failed due to
B+Tree cursor corruption (AssertionError during B+Tree page iteration)."|
*This metric enables:*
- Automated alerting on tree corruption frequency
- Correlation of corruption events with specific operations (rebalance,
compaction, etc.)
- Quantification of data reliability without log scraping
> Add a metric to track B+Tree cursor corruption failures during SQL query
> execution
> ----------------------------------------------------------------------------------
>
> Key: IGNITE-29102
> URL: https://issues.apache.org/jira/browse/IGNITE-29102
> Project: Ignite
> Issue Type: Task
> Reporter: Egor Baranov
> Assignee: Egor Baranov
> Priority: Major
> Labels: from-log-to-metric
>
> *Problem:*
> When a B+Tree cursor encounters a broken invariant (typically an
> AssertionError in AbstractForwardCursor.find() or
> AbstractForwardCursor.fillFromBuffer()), the exception is wrapped into a
> CorruptedTreeException and propagated up the call chain to
> GridMapQueryExecutor. There, it is caught in a generic catch (Throwable e)
> block and logged as "Failed to execute local query" at ERROR/WARN level.
> There is currently *no metric* that tracks how often such corruption-related
> query failures occur. Operators can only detect this problem by parsing log
> files, which is impractical for:
> - Large clusters with high query throughput
> - Intermittent corruption events that are hard to spot in log noise
> - Automated alerting on infrastructure degradation
> Example stack trace:
> {code:text}
> java.lang.AssertionError: null
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.find(BPlusTree.java:5559)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.reinitialize(BPlusTree.java:5572)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.nextPage(BPlusTree.java:5627)
> ...
> at
> org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0(GridMapQueryExecutor.java:538)
> {code}
> {*}Solution{*}{*}{{*}}
> Add a dedicated long counter metric in the map query execution path that is
> incremented whenever a query fails due to a CorruptedTreeException.
> {*}Resulting Metric{*}{*}{{*}}
> ||*Registry*||*Name*||*Type*||*Description*||
> |sql.queries.treeCorruption|count|long|Number of queries that failed due to
> B+Tree cursor corruption (AssertionError during B+Tree page iteration)|
> *This metric enables:*
> - Automated alerting on tree corruption frequency
> - Correlation of corruption events with specific operations (rebalance,
> compaction, etc.)
> - Quantification of data reliability without log scraping
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
