[
https://issues.apache.org/jira/browse/CASSANDRA-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13931875#comment-13931875
]
Lyuben Todorov commented on CASSANDRA-5483:
-------------------------------------------
It's a good idea to add a verbose option to show tracing as it happens.
Currently to enable logging to the system_traces.events table the {{-tr}} flag
has to be supplied {{./nodetool repair -tr}}, and then to view the traces
{{SELECT * FROM system_traces.events}}
{noformat}
cqlsh> SELECT * FROM system_traces.events ;
session_id | event_id |
activity
| command | source | source_elapsed | thread
--------------------------------------+--------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+-----------+----------------+-----------------------
fedbe970-a9fa-11e3-ab61-2387bbb38302 | fedbe971-a9fa-11e3-ab61-2387bbb38302 |
Starting repair command #3, repairing 2 ranges for keyspace system_traces
(seq=true, full=true) | REPAIR | 127.0.0.1 | 11 |
Thread-16
fedbe970-a9fa-11e3-ab61-2387bbb38302 | fedc5ea0-a9fa-11e3-ab61-2387bbb38302 |
[repair #fedc3790-a9fa-11e3-ab61-2387bbb38302] new session: will sync
/127.0.0.1, /127.0.0.3 on range (-3074457345618258603,3074457345618258602] for
system_traces.[sessions, events] | REPAIR | 127.0.0.1 | 2493 |
AntiEntropySessions:7
fedbe970-a9fa-11e3-ab61-2387bbb38302 | fedef6b0-a9fa-11e3-ab61-2387bbb38302 |
[repair
#fedc3790-a9fa-11e3-ab61-2387bbb38302] requesting merkle trees for sessions (to
[/127.0.0.3, /127.0.0.1]) | REPAIR | 127.0.0.1 | 19687 |
RepairJobTask:2
fedbe970-a9fa-11e3-ab61-2387bbb38302 | fee167b0-a9fa-11e3-b120-2387bbb38302 |
[repair
#fedc3790-a9fa-11e3-ab61-2387bbb38302] Sending completed merkle tree to
/127.0.0.1 for system_traces/sessions | REPAIR | 127.0.0.3 | 31937 |
AntiEntropyStage:1
fedbe970-a9fa-11e3-ab61-2387bbb38302 | fee3b1a0-a9fa-11e3-ab61-2387bbb38302 |
[repair #fedc3790-a9fa-11e3-ab61-2387bbb38302] Received merkle tree for
sessions from /127.0.0.3 | REPAIR | 127.0.0.1 | 50996 |
AntiEntropyStage:1
fedbe970-a9fa-11e3-ab61-2387bbb38302 | fee7d050-a9fa-11e3-ab61-2387bbb38302 |
[repair #fedc3790-a9fa-11e3-ab61-2387bbb38302] Received merkle tree for
sessions from /127.0.0.1 | REPAIR | 127.0.0.1 | 77907 |
AntiEntropyStage:1
fedbe970-a9fa-11e3-ab61-2387bbb38302 | feea1a40-a9fa-11e3-ab61-2387bbb38302 |
[repair
#fedc3790-a9fa-11e3-ab61-2387bbb38302] requesting merkle trees for events (to
[/127.0.0.3, /127.0.0.1]) | REPAIR | 127.0.0.1 | 92358 |
RepairJobTask:2
fedbe970-a9fa-11e3-ab61-2387bbb38302 | feecd960-a9fa-11e3-b120-2387bbb38302 |
[repair
#fedc3790-a9fa-11e3-ab61-2387bbb38302] Sending completed merkle tree to
/127.0.0.1 for system_traces/events | REPAIR | 127.0.0.3 | 107062 |
AntiEntropyStage:1
fedbe970-a9fa-11e3-ab61-2387bbb38302 | feef2350-a9fa-11e3-ab61-2387bbb38302 |
[repair #fedc3790-a9fa-11e3-ab61-2387bbb38302] Received merkle tree for
events from /127.0.0.3 | REPAIR | 127.0.0.1 | 126230 |
AntiEntropyStage:1
fedbe970-a9fa-11e3-ab61-2387bbb38302 | fef3de40-a9fa-11e3-ab61-2387bbb38302 |
[repair #fedc3790-a9fa-11e3-ab61-2387bbb38302] Received merkle tree for
events from /127.0.0.1 | REPAIR | 127.0.0.1 | 157175 |
AntiEntropyStage:1
fedbe970-a9fa-11e3-ab61-2387bbb38302 | fef40551-a9fa-11e3-ab61-2387bbb38302 |
[repair #fef40550-a9fa-11e3-ab61-2387bbb38302] new session: will sync
/127.0.0.1, /127.0.0.2 on range (3074457345618258602,-9223372036854775808] for
system_traces.[sessions, events] | REPAIR | 127.0.0.1 | 158225 |
AntiEntropySessions:8
fedbe970-a9fa-11e3-ab61-2387bbb38302 | fef4c8a0-a9fa-11e3-ab61-2387bbb38302 |
[repair
#fef40550-a9fa-11e3-ab61-2387bbb38302] requesting merkle trees for sessions (to
[/127.0.0.2, /127.0.0.1]) | REPAIR | 127.0.0.1 | 162702 |
RepairJobTask:2
fedbe970-a9fa-11e3-ab61-2387bbb38302 | fef4efb0-a9fa-11e3-ab61-2387bbb38302 |
[repair #fedc3790-a9fa-11e3-ab61-2387bbb38302] session
completed successfully | REPAIR | 127.0.0.1 | 163376 |
AntiEntropySessions:7
fedbe970-a9fa-11e3-ab61-2387bbb38302 | fef60120-a9fa-11e3-b701-2387bbb38302 |
[repair
#fef40550-a9fa-11e3-ab61-2387bbb38302] Sending completed merkle tree to
/127.0.0.1 for system_traces/sessions | REPAIR | 127.0.0.2 | 11744 |
AntiEntropyStage:1
fedbe970-a9fa-11e3-ab61-2387bbb38302 | fefae320-a9fa-11e3-ab61-2387bbb38302 |
[repair #fef40550-a9fa-11e3-ab61-2387bbb38302] Received merkle tree for
sessions from /127.0.0.2 | REPAIR | 127.0.0.1 | 203106 |
AntiEntropyStage:1
fedbe970-a9fa-11e3-ab61-2387bbb38302 | feff9e10-a9fa-11e3-ab61-2387bbb38302 |
[repair #fef40550-a9fa-11e3-ab61-2387bbb38302] Received merkle tree for
sessions from /127.0.0.1 | REPAIR | 127.0.0.1 | 233383 |
AntiEntropyStage:1
fedbe970-a9fa-11e3-ab61-2387bbb38302 | ff0199e0-a9fa-11e3-ab61-2387bbb38302 |
[repair
#fef40550-a9fa-11e3-ab61-2387bbb38302] requesting merkle trees for events (to
[/127.0.0.2, /127.0.0.1]) | REPAIR | 127.0.0.1 | 247143 |
RepairJobTask:2
fedbe970-a9fa-11e3-ab61-2387bbb38302 | ff036ea0-a9fa-11e3-b701-2387bbb38302 |
[repair
#fef40550-a9fa-11e3-ab61-2387bbb38302] Sending completed merkle tree to
/127.0.0.1 for system_traces/events | REPAIR | 127.0.0.2 | 99303 |
AntiEntropyStage:1
fedbe970-a9fa-11e3-ab61-2387bbb38302 | ff054360-a9fa-11e3-ab61-2387bbb38302 |
[repair #fef40550-a9fa-11e3-ab61-2387bbb38302] Received merkle tree for
events from /127.0.0.2 | REPAIR | 127.0.0.1 | 270729 |
AntiEntropyStage:1
fedbe970-a9fa-11e3-ab61-2387bbb38302 | ff089ec0-a9fa-11e3-ab61-2387bbb38302 |
[repair #fef40550-a9fa-11e3-ab61-2387bbb38302] Received merkle tree for
events from /127.0.0.1 | REPAIR | 127.0.0.1 | 292795 |
AntiEntropyStage:1
fedbe970-a9fa-11e3-ab61-2387bbb38302 | ff096210-a9fa-11e3-ab61-2387bbb38302 |
[repair #fef40550-a9fa-11e3-ab61-2387bbb38302] session
completed successfully | REPAIR | 127.0.0.1 | 297948 |
AntiEntropySessions:8
fedbe970-a9fa-11e3-ab61-2387bbb38302 | ff098920-a9fa-11e3-ab61-2387bbb38302 |
Repair command #3 finished | REPAIR | 127.0.0.1 | 298314 |
Thread-16
feb74a70-a9fa-11e3-ab61-2387bbb38302 | feb74a71-a9fa-11e3-ab61-2387bbb38302 |
Starting repair command #2, repairing 2 ranges for keyspace test
(seq=true, full=true) | REPAIR | 127.0.0.1 | 8 |
Thread-15
feb74a70-a9fa-11e3-ab61-2387bbb38302 | feb79890-a9fa-11e3-ab61-2387bbb38302 |
[repair #feb77180-a9fa-11e3-ab61-2387bbb38302] new
session: will sync /127.0.0.1, /127.0.0.3 on range
(-3074457345618258603,3074457345618258602] for test.[jj] | REPAIR | 127.0.0.1
| 1909 | AntiEntropySessions:5
feb74a70-a9fa-11e3-ab61-2387bbb38302 | feb834d0-a9fa-11e3-ab61-2387bbb38302 |
[repair #feb77180-a9fa-11e3-ab61-2387bbb38302] requesting merkle trees for jj
(to [/127.0.0.3, /127.0.0.1]) | REPAIR | 127.0.0.1 | 6344 |
RepairJobTask:2
feb74a70-a9fa-11e3-ab61-2387bbb38302 | febcefc0-a9fa-11e3-b120-2387bbb38302 |
[repair #feb77180-a9fa-11e3-ab61-2387bbb38302] Sending completed merkle tree to
/127.0.0.1 for test/jj | REPAIR | 127.0.0.3 | 33924 |
AntiEntropyStage:1
feb74a70-a9fa-11e3-ab61-2387bbb38302 | febf87d0-a9fa-11e3-ab61-2387bbb38302 |
[repair #feb77180-a9fa-11e3-ab61-2387bbb38302] Received merkle tree
for jj from /127.0.0.3 | REPAIR | 127.0.0.1 | 54472 |
AntiEntropyStage:1
feb74a70-a9fa-11e3-ab61-2387bbb38302 | fec728f0-a9fa-11e3-ab61-2387bbb38302 |
[repair #feb77180-a9fa-11e3-ab61-2387bbb38302] Received merkle tree
for jj from /127.0.0.1 | REPAIR | 127.0.0.1 | 104179 |
AntiEntropyStage:1
feb74a70-a9fa-11e3-ab61-2387bbb38302 | fec75000-a9fa-11e3-ab61-2387bbb38302 |
[repair #fec728f1-a9fa-11e3-ab61-2387bbb38302] new
session: will sync /127.0.0.1, /127.0.0.2 on range
(3074457345618258602,-9223372036854775808] for test.[jj] | REPAIR | 127.0.0.1
| 105545 | AntiEntropySessions:6
feb74a70-a9fa-11e3-ab61-2387bbb38302 | fec81350-a9fa-11e3-ab61-2387bbb38302 |
[repair #feb77180-a9fa-11e3-ab61-2387bbb38302] session
completed successfully | REPAIR | 127.0.0.1 | 110357 |
AntiEntropySessions:5
feb74a70-a9fa-11e3-ab61-2387bbb38302 | fec86170-a9fa-11e3-ab61-2387bbb38302 |
[repair #fec728f1-a9fa-11e3-ab61-2387bbb38302] requesting merkle trees for jj
(to [/127.0.0.2, /127.0.0.1]) | REPAIR | 127.0.0.1 | 111725 |
RepairJobTask:2
feb74a70-a9fa-11e3-ab61-2387bbb38302 | fed30fd0-a9fa-11e3-b701-2387bbb38302 |
[repair #fec728f1-a9fa-11e3-ab61-2387bbb38302] Sending completed merkle tree to
/127.0.0.1 for test/jj | REPAIR | 127.0.0.2 | 74919 |
AntiEntropyStage:1
feb74a70-a9fa-11e3-ab61-2387bbb38302 | fed5a7e0-a9fa-11e3-ab61-2387bbb38302 |
[repair #fec728f1-a9fa-11e3-ab61-2387bbb38302] Received merkle tree
for jj from /127.0.0.2 | REPAIR | 127.0.0.1 | 199165 |
AntiEntropyStage:1
feb74a70-a9fa-11e3-ab61-2387bbb38302 | fed9c690-a9fa-11e3-ab61-2387bbb38302 |
[repair #fec728f1-a9fa-11e3-ab61-2387bbb38302] Received merkle tree
for jj from /127.0.0.1 | REPAIR | 127.0.0.1 | 225798 |
AntiEntropyStage:1
feb74a70-a9fa-11e3-ab61-2387bbb38302 | feda89e0-a9fa-11e3-ab61-2387bbb38302 |
[repair #fec728f1-a9fa-11e3-ab61-2387bbb38302] session
completed successfully | REPAIR | 127.0.0.1 | 230662 |
AntiEntropySessions:6
feb74a70-a9fa-11e3-ab61-2387bbb38302 | feda89e1-a9fa-11e3-ab61-2387bbb38302 |
Repair command #2 finished | REPAIR | 127.0.0.1 | 231159 |
Thread-15
(36 rows)
{noformat}
> Repair tracing
> --------------
>
> Key: CASSANDRA-5483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5483
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Reporter: Yuki Morishita
> Assignee: Ben Chan
> Priority: Minor
> Labels: repair
> Attachments: 5483-full-trunk.txt,
> 5483-v06-04-Allow-tracing-ttl-to-be-configured.patch,
> 5483-v06-05-Add-a-command-column-to-system_traces.events.patch,
> 5483-v06-06-Fix-interruption-in-tracestate-propagation.patch,
> 5483-v07-07-Better-constructor-parameters-for-DebuggableThreadPoolExecutor.patch,
> 5483-v07-08-Fix-brace-style.patch,
> 5483-v07-09-Add-trace-option-to-a-more-complete-set-of-repair-functions.patch,
> 5483-v07-10-Correct-name-of-boolean-repairedAt-to-fullRepair.patch,
> ccm-repair-test, test-5483-system_traces-events.txt,
> trunk@4620823-5483-v02-0001-Trace-filtering-and-tracestate-propagation.patch,
> trunk@4620823-5483-v02-0002-Put-a-few-traces-parallel-to-the-repair-logging.patch,
> tr...@8ebeee1-5483-v01-001-trace-filtering-and-tracestate-propagation.txt,
> [email protected],
> v02p02-5483-v03-0003-Make-repair-tracing-controllable-via-nodetool.patch,
> v02p02-5483-v04-0003-This-time-use-an-EnumSet-to-pass-boolean-repair-options.patch,
> v02p02-5483-v05-0003-Use-long-instead-of-EnumSet-to-work-with-JMX.patch
>
>
> I think it would be nice to log repair stats and results like query tracing
> stores traces to system keyspace. With it, you don't have to lookup each log
> file to see what was the status and how it performed the repair you invoked.
> Instead, you can query the repair log with session ID to see the state and
> stats of all nodes involved in that repair session.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
