[
https://issues.apache.org/jira/browse/CASSANDRA-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17514369#comment-17514369
]
David Capwell commented on CASSANDRA-15399:
-------------------------------------------
There are 2 types of tables: coordinator, participate; here are examples from
both sides
*Participate* (host involved in the repair, this may be the coordinator, but
would include all peers involved in the repair)
{code}
cqlsh> SELECT * FROM system_views.repair_participates WHERE tables CONTAINS
'tlp_stress.clusteringslicing' ALLOW FILTERING;
@ Row 1
-------------------------+----------------------------------------
id | 584eca30-afb6-11ec-9bff-cd38a4d988d3
duration_millis | 14773
failure_cause | null
global | True
incremental | False
last_updated_at | 2022-03-29 23:17:18.993000+0000
preview_kind | NONE
ranges | ['(-3074457345618258603,0]']
repaired_at | null
state_failure_timestamp | null
state_init_timestamp | 2022-03-29 23:17:04.220000+0000
state_skipped_timestamp | null
state_success_timestamp | 2022-03-29 23:17:18.993000+0000
success_message | Cleanup message recieved
tables | {'tlp_stress.clusteringslicing'}
validations | {0034e064-b54a-384f-99d0-af1bf42844a4}
(1 rows)
cqlsh> SELECT * FROM system_views.repair_validations WJERE
id=0034e064-b54a-384f-99d0-af1bf42844a4; -- id from validations field above
@ Row 1
-------------------------------+--------------------------------------
id | 0034e064-b54a-384f-99d0-af1bf42844a4
duration_millis | 14688
estimated_partitions | 897
estimated_total_bytes | 2321697131
failure_cause | null
initiator | /127.0.0.1:7000
keyspace_name | tlp_stress
last_updated_at | 2022-03-29 23:17:18.916000+0000
partitions_processed | 181
progress_percentage | 100
ranges | ['(-3074457345618258603,0]']
repair_id | 584eca30-afb6-11ec-9bff-cd38a4d988d3
session_id | 58509ef0-afb6-11ec-9bff-cd38a4d988d3
state_failure_timestamp | null
state_init_timestamp | 2022-03-29 23:17:04.228000+0000
state_sending_trees_timestamp | 2022-03-29 23:17:18.916000+0000
state_skipped_timestamp | null
state_start_timestamp | 2022-03-29 23:17:04.299000+0000
state_success_timestamp | 2022-03-29 23:17:18.916000+0000
status | success
success_message | null
table_name | clusteringslicing
(1 rows)
{code}
What is lacking? Linking with system_views.streaming, my plan is to update
repair/streaming to support deep linking after this patch is merged (need to
modify network messages for remote repair + streaming, so repair knows stream
ids, and stream knows repair id)
*Coordinator*
{code}
cqlsh> SELECT * FROM system_views.repairs WHERE table_names CONTAINS
'clusteringslicing' ALLOW FILTERING;
@ Row 1
---------------------------------------+--------------------------------------------------
id | 34424630-afb6-11ec-ab62-fb846f906c0a
command_id | 2
duration_millis | 7799
failure_cause | null
keyspace_name | tlp_stress
last_updated_at | 2022-03-29 23:16:11.530000+0000
options_column_families | {}
options_data_centers | {}
options_force_repair | False
options_hosts | {}
options_ignore_unreplicated_keyspaces | False
options_incremental | null
options_job_threads | 1
options_optimise_streams | False
options_parallelism | PARALLEL
options_preview_kind | NONE
options_primary_range | True
options_pull_repair | False
options_ranges |
{'(6148914691236517205,-9223372036854775808]'}
options_subrange_repair | False
options_trace | False
participants | ['/127.0.0.1:7000', '/127.0.0.2:7000']
ranges |
[['(6148914691236517205,-9223372036854775808]']]
sessions | {34441af0-afb6-11ec-ab62-fb846f906c0a}
state_failure_timestamp | null
state_init_timestamp | 2022-03-29 23:16:03.731000+0000
state_prepare_complete_timestamp | 2022-03-29 23:16:03.741000+0000
state_prepare_start_timestamp | 2022-03-29 23:16:03.737000+0000
state_repair_complete_timestamp | 2022-03-29 23:16:11.529000+0000
state_repair_start_timestamp | 2022-03-29 23:16:03.742000+0000
state_setup_timestamp | 2022-03-29 23:16:03.731000+0000
state_skipped_timestamp | null
state_start_timestamp | 2022-03-29 23:16:03.732000+0000
state_success_timestamp | 2022-03-29 23:16:11.530000+0000
status | success
success_message | Repair completed successfully
table_names | ['clusteringslicing']
type | full
unfiltered_ranges |
[['(6148914691236517205,-9223372036854775808]']]
(1 rows)
cqlsh> SELECT * FROM system_views.repair_sessions WHERE
id=34441af0-afb6-11ec-ab62-fb846f906c0a; -- from sessions column above
@ Row 1
----------------------------+------------------------------------------------
id | 34441af0-afb6-11ec-ab62-fb846f906c0a
duration_millis | 7785
failure_cause | null
jobs | {f08d1aa3-04d5-3439-93a4-7beaf6bcd393}
keyspace_name | tlp_stress
last_updated_at | 2022-03-29 23:16:11.528000+0000
ranges | ['(6148914691236517205,-9223372036854775808]']
repair_id | 34424630-afb6-11ec-ab62-fb846f906c0a
state_failure_timestamp | null
state_init_timestamp | 2022-03-29 23:16:03.743000+0000
state_jobs_start_timestamp | 2022-03-29 23:16:03.746000+0000
state_skipped_timestamp | null
state_start_timestamp | 2022-03-29 23:16:03.743000+0000
state_success_timestamp | 2022-03-29 23:16:11.528000+0000
status | success
success_message | null
table_names | ['clusteringslicing']
(1 rows)
cqlsh> SELECT * FROM system_views.repair_jobs WHERE
id=f08d1aa3-04d5-3439-93a4-7beaf6bcd393; -- from jobs column above
@ Row 1
-------------------------------------+------------------------------------------------
id | f08d1aa3-04d5-3439-93a4-7beaf6bcd393
duration_millis | 7779
failure_cause | null
keyspace_name | tlp_stress
last_updated_at | 2022-03-29 23:16:11.526000+0000
ranges |
['(6148914691236517205,-9223372036854775808]']
repair_id | 34424630-afb6-11ec-ab62-fb846f906c0a
session_id | 34441af0-afb6-11ec-ab62-fb846f906c0a
state_failure_timestamp | null
state_init_timestamp | 2022-03-29 23:16:03.747000+0000
state_skipped_timestamp | null
state_snapshot_complete_timestamp | null
state_snapshot_start_timestamp | null
state_start_timestamp | 2022-03-29 23:16:03.747000+0000
state_stream_start_timestamp | 2022-03-29 23:16:11.526000+0000
state_success_timestamp | 2022-03-29 23:16:11.526000+0000
state_validation_complete_timestamp | 2022-03-29 23:16:11.526000+0000
state_validation_start_timestamp | 2022-03-29 23:16:03.747000+0000
status | success
success_message | null
table_name | clusteringslicing
(1 rows)
{code}
Missing
* links to streaming
* repair_jobs/sessions doesn't show the participants, so requires a join with
repairs
> Add ability to track state in repair
> ------------------------------------
>
> Key: CASSANDRA-15399
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15399
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Repair
> Reporter: David Capwell
> Assignee: David Capwell
> Priority: Normal
> Labels: pull-request-available
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> To enhance the visibility in repair, we should expose internal state via
> virtual tables; the state should include coordinator as well as participant
> state (validation, sync, etc.)
> I propose the following tables:
> repairs - high level summary of the global state of repair; this should be
> called on the coordinator.
> {code:sql}
> CREATE TABLE repairs (
> id uuid,
> keyspace_name text,
> table_names frozen<list<text>>,
> ranges frozen<list<text>>,
> coordinator text,
> participants frozen<list<text>>,
> state text,
> progress_percentage float,
> last_updated_at_millis bigint,
> duration_micro bigint,
> failure_cause text,
> PRIMARY KEY ( (id) )
> )
> {code}
> repair_tasks - represents RepairJob and participants state. This will show
> if validations are running on participants and the progress they are making;
> this should be called on the coordinator.
> {code:sql}
> CREATE TABLE repair_tasks (
> id uuid,
> session_id uuid,
> keyspace_name text,
> table_name text,
> ranges frozen<list<text>>,
> coordinator text,
> participant text,
> state text,
> state_description text,
> progress_percentage float, -- between 0.0 and 100.0
> last_updated_at_millis bigint,
> duration_micro bigint,
> failure_cause text,
> PRIMARY KEY ( (id), session_id, table_name, participant )
> )
> {code}
> repair_validations - shows the state of the validation task and updated
> periodically while validation is running; this should be called on the
> participants.
> {code:sql}
> CREATE TABLE repair_validations (
> id uuid,
> session_id uuid,
> ranges frozen<list<text>>,
> keyspace_name text,
> table_name text,
> initiator text,
> state text,
> progress_percentage float,
> queue_duration_ms bigint,
> runtime_duration_ms bigint,
> total_duration_ms bigint,
> estimated_partitions bigint,
> partitions_processed bigint,
> estimated_total_bytes bigint,
> failure_cause text,
> PRIMARY KEY ( (id), session_id, table_name )
> )
> {code}
> The main reason for exposing virtual tables rather than exposing through
> durable tables is to make sure what is exposed is accurate. In cases of
> write failures or node failures, the durable tables could become in-accurate
> and could add edge cases where the repair is not running but the tables say
> it is; by relying on repair's internal in-memory bookkeeping, these problems
> go away.
> This jira does not try to solve the following:
> 1) repair resiliency - there are edge cases where repair hits an error and
> runs forever (at least from nodetool's perspective).
> 2) repair stream tracking - I have not learned the streaming side yet and
> what I see is multiple implementations exist, so seems like high scope. My
> hope is to punt from this jira and tackle separately.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
