[
https://issues.apache.org/jira/browse/CASSANDRA-21485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18119809#comment-18119809
]
George commented on CASSANDRA-21485:
------------------------------------
Patch available for review: https://github.com/apache/cassandra/pull/5216
The plan now snapshots the memtable estimate once per table, shares it across
token ranges, and reuses those saved estimates when generating repair
assignments. Flushes or subsequent writes therefore do not change the estimate
between statistics collection and assignment generation. Fixed-split
assignments also preserve the plan's total, including integer remainders and
zero-byte assignments.
Regression coverage includes all three repair types, flushes and writes between
planning and assignment generation, multiple ranges and tables, both
fixed-split modes, rounding, empty tables, and SSTable/memtable coexistence.
The scheduler statistics test now starts repair only after its setup and
flushes finish.
Validation on trunk, commit e70566a758697ddb58cd3a33f3f34368a658c365 (macOS,
JDK 17.0.19, Ant 1.10.13):
* .build/check-code.sh -s: passed (build, RAT and Checkstyle).
* AutoRepairStateTest: 99 passed.
* RepairTokenRangeSplitterTest: 50 passed.
* FixedSplitTokenRangeSplitter tests: 162 passed.
* AutoRepairSchedulerStatsNoVNodesTest and AutoRepairSchedulerStatsVNodesTest:
1 passed each.
Total: 311 unit tests and 2 JVM distributed tests passed. Exact commands are in
the PR description.
Community pre-commit CI has not been run; I do not currently have access
configured. Could a reviewer or committer help run it and attach the ci_summary
and results_details artifacts here?
There are no configuration, wire-format or on-disk format changes. The metrics
remain estimates of repair work, rather than measured transfer sizes.
> AutoRepair metric BytesAlreadyRepaired can exceed TotalBytesToRepair due to
> memtable data not included in total estimation
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21485
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21485
> Project: Apache Cassandra
> Issue Type: Task
> Reporter: Paulo Motta
> Priority: Normal
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h2. Summary
> The {{TotalBytesToRepair}} metric does not account for memtable data, causing
> {{BytesAlreadyRepaired}} to exceed {{TotalBytesToRepair}} (progress > 100%)
> after a repair turn completes.
> h2. Root Cause
> {{TotalBytesToRepair}} is computed in
> {{AutoRepairState.updateRepairScheduleStatistics()}} by summing
> {{KeyspaceRepairPlan.getEstimatedBytes()}}, which only considers on-disk
> SSTable sizes via {{SizeEstimate.sizeForRepair}}.
> However, during repair assignment generation,
> {{RepairTokenRangeSplitter.getRepairAssignments()}} (line 522-527) discovers
> tables with no SSTables but active memtable data and creates assignments with
> {{estimatedBytes = memtableSize}}. These bytes are then accumulated in
> {{bytesAlreadyRepaired}} but were never counted in the total.
> h2. Code References
> * {{AutoRepairState.java:117-119}} - computes {{totalBytesToRepair}} from
> plan (SSTable-only)
> * {{RepairTokenRangeSplitter.java:522-527}} - discovers memtable data and
> assigns {{memtableSize}} as estimated bytes
> * {{AutoRepair.java:401}} - accumulates {{bytesAlreadyRepaired}} including
> memtable-based assignments
> * {{AutoRepairUtils.java:1391}} - {{sizeForRepair}} is derived from SSTable
> sizes only
> h2. Impact
> * The {{BytesAlreadyRepaired}} and {{TotalBytesToRepair}} JMX metrics report
> inconsistent values (repaired > total)
> * Progress monitoring dashboards show values above 100% which is confusing
> for operators
> * The repair itself is not affected - all data (including memtable) is
> correctly repaired
> h2. Suggested Fix
> Include memtable sizes in the total estimation. When computing
> {{totalBytesToRepair}}, also check for live memtable data on tables that have
> {{sizeForRepair == 0}}, mirroring the logic in the splitter. This could be
> done in {{calcTotalBytesToBeRepaired()}} or
> {{updateRepairScheduleStatistics()}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
