[
https://issues.apache.org/jira/browse/CASSANDRA-20418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18119929#comment-18119929
]
George commented on CASSANDRA-20418:
------------------------------------
I investigated this on trunk at f43a7b9265827cc1a2765ee8bebd53ffe6f261ae.
Before implementing production changes, I would appreciate feedback on the
scope and approach.
h4. Findings
* RepairAssignment currently represents one token range.
RepairTokenRangeSplitter generates assignments separately for each range; its
table batching combines tables for the same range, rather than combining vnode
ranges. FixedSplitTokenRangeSplitter also creates at least one assignment per
input range, even when number_of_subranges=1.
* AutoRepair.repairKeyspace already accumulates ranges according to
number_of_repair_threads, so there is some batching today. The default is 1,
and batch size is coupled to job concurrency rather than replica compatibility
or workload size.
* The repair core already supports this operation:
RepairCoordinator.addRangeToNeighbors groups ranges by identical endpoint and
transient-endpoint sets, and AbstractRepairTask creates a RepairSession per
CommonRange. Disjoint ranges can remain a collection; they do not need to be
widened into one enclosing range.
h4. Local validation
On OpenJDK 17.0.19 / Ant 1.10.13, four new JVM distributed characterization
tests passed using three nodes, eight tokens per node, and NETWORK/GOSSIP:
* At RF=3, node 1 has nine unwrapped primary ranges sharing one replica group.
For FULL, INCREMENTAL and PREVIEW_REPAIRED, submitting the empty-table ranges
individually produced nine commands/sessions; submitting them together produced
one command/session, successfully.
* Empty and small-table planning still produced nine assignments with
repair_by_keyspace=true, or eighteen for two tables with it disabled, with
either splitter (including fixed target=1).
* A grouped FULL repair of two disjoint ranges restored the requested subset of
256 rows missing from the other replicas, without repairing the rows outside
those ranges.
* At RF=2, seventeen local ranges with two replica groups produced two sessions
under one command, preserving the requested ranges.
These tests exercise the existing core directly, without Auto Repair's default
5s minimum task duration. They establish the grouping behavior, not a
production performance claim. Large-data, repeated performance,
failure-injection and mixed-version tests have not been run. The existing
FixedSplitTokenRangeSplitterVNodesTest (81 cases) and
RepairTokenRangeSplitterTest (50 cases) also passed.
h4. Proposed direction for discussion
* Preserve atomic assignments and introduce an internal bounded batch-planning
step, grouping only compatible keyspace/priority/repair-type/table sets and
replica roles. This could preserve the existing splitter interface; an
extension within the splitter is another option to consider.
* Retain exact range membership and enforce byte, partition, range-count and
table-count budgets. Range-count limits matter even for zero-byte estimates.
Partition estimates would need to be carried forward because RepairAssignment
currently exposes only estimated bytes.
* Separate batch size from job concurrency. Make batch boundaries, retries,
cancellation and accounting explicit, while reusing the existing
RepairCoordinator and wire formats.
* Reuse the same estimate snapshot for planning and progress (related to
CASSANDRA-21485), and account for relevant placement/schema changes before
submission. Avoid changing defaults, metric semantics or the effective keyspace
timeout budget accidentally.
h4. Separate correctness finding
With repair_by_keyspace=false and number_of_repair_threads=2, the scheduler can
lose a table from the submitted work. Given assignments a:(0,10] and b:(0,10],
the current loop accumulates the range but submits only the last assignment's
table list, so only b:(0,10] is requested. A regression probe invoking the real
repairKeyspace method and capturing its downstream coordinator arguments fails
for all three repair types. Six other scheduler probes passed (single-range
submission and existing same-table batching). This is a scheduler-level
reproduction, not an end-to-end data-loss test; the default thread count of 1
does not trigger it. I suggest tracking that bug separately and linking it here.
h4. Questions
# Would maintainers prefer a bounded planner between the splitter and executor,
or extending assignments/splitters to represent multiple ranges?
# Should batching get a separate opt-in limit initially, and how should it
relate to the existing per-assignment budgets and number_of_repair_threads?
# Is this appropriately scoped as an enhancement under CEP-37, or should the
design/configuration changes be taken to the dev list and documented as a
CEP-37 revision?
No production implementation patch or community CI run is available yet; the
above tests are local investigation tests. I would like to agree on the
direction before expanding the implementation.
> Ability to create single repair assignments for shared ranges with vnodes
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-20418
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20418
> Project: Apache Cassandra
> Issue Type: Task
> Components: Consistency/Repair
> Reporter: Jaydeepkumar Chovatia
> Priority: Normal
>
> From the comment:
> https://github.com/apache/cassandra/pull/3598#issuecomment-2425935493
> {code:java}
> Combine ranges instead of splitting - in ecChronos we saw huge improvements
> in some scenarios (when the data is low or empty tables) compared to
> repairing 1 vnode at a time. This improvement scaled with amount of vnodes,
> although it might've been related to overhead due to running repairs through
> JMX. Not sure but might be worth investigating.{code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
