[
https://issues.apache.org/jira/browse/CASSANDRA-11965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Deng updated CASSANDRA-11965:
---------------------------------
Description:
[~jbellis] mentioned this as a potential improvement in his 2013 committer
meeting notes
(http://grokbase.com/t/cassandra/dev/132s6sh415/notes-from-committers-meeting-streaming-and-repair):
"making the repair coordinator smarter to know when to avoid duplicate
streaming. E.g., if replicas A and B have row X, but C does not, currently both
A and B will stream to C."
I tested in C* 3.0.6 and looks like this is still happening. Basically on a
3-node cluster I inserted into a trivial table under a keyspace with RF=3 and
forced two flushes on all nodes so that I have two SSTables on each node, then
I shutdown the 1st node and removed one SSTable from its data directory and
restarted the node. I connected cqlsh to this node and verified that with
CL.ONE the data is indeed missing; I now moved onto the 2nd node running a
"nodetool repair <keyspace> <table>", here are what I observed from system.log
on the 2nd node (as repair coordinator):
{noformat}
INFO [Thread-47] 2016-06-06 23:19:54,173 RepairRunnable.java:125 - Starting
repair command #1, repairing keyspace weitest with repair options (parallelism:
parallel, primary range: false, incremental: true, job threads: 1,
ColumnFamilies: [songs], dataCenters: [], hosts: [], # of ranges: 3)
INFO [Thread-47] 2016-06-06 23:19:54,253 RepairSession.java:237 - [repair
#2d177cc0-2c3d-11e6-94d2-b35b6c93de57] new session: will sync /172.31.44.75,
/172.31.40.215, /172.31.36.148 on range
[(3074457345618258602,-9223372036854775808],
(-9223372036854775808,-3074457345618258603],
(-3074457345618258603,3074457345618258602]] for weitest.[songs]
INFO [Repair#1:1] 2016-06-06 23:19:54,268 RepairJob.java:172 - [repair
#2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Requesting merkle trees for songs (to
[/172.31.40.215, /172.31.36.148, /172.31.44.75])
INFO [AntiEntropyStage:1] 2016-06-06 23:19:54,335 RepairSession.java:181 -
[repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Received merkle tree for songs
from /172.31.40.215
INFO [AntiEntropyStage:1] 2016-06-06 23:19:54,427 RepairSession.java:181 -
[repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Received merkle tree for songs
from /172.31.44.75
INFO [AntiEntropyStage:1] 2016-06-06 23:19:54,460 RepairSession.java:181 -
[repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Received merkle tree for songs
from /172.31.36.148
INFO [RepairJobTask:1] 2016-06-06 23:19:54,466 SyncTask.java:73 - [repair
#2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Endpoints /172.31.40.215 and
/172.31.36.148 have 3 range(s) out of sync for songs
INFO [RepairJobTask:1] 2016-06-06 23:19:54,467 RemoteSyncTask.java:54 -
[repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Forwarding streaming repair of 3
ranges to /172.31.40.215 (to be streamed with /172.31.36.148)
INFO [RepairJobTask:1] 2016-06-06 23:19:54,472 SyncTask.java:66 - [repair
#2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Endpoints /172.31.36.148 and
/172.31.44.75 are consistent for songs
INFO [RepairJobTask:3] 2016-06-06 23:19:54,474 SyncTask.java:73 - [repair
#2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Endpoints /172.31.40.215 and
/172.31.44.75 have 3 range(s) out of sync for songs
INFO [RepairJobTask:3] 2016-06-06 23:19:54,529 LocalSyncTask.java:68 -
[repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Performing streaming repair of 3
ranges with /172.31.40.215
INFO [RepairJobTask:3] 2016-06-06 23:19:54,574 StreamResultFuture.java:86 -
[Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57] Executing streaming plan for
Repair
INFO [StreamConnectionEstablisher:1] 2016-06-06 23:19:54,576
StreamSession.java:238 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57]
Starting streaming to /172.31.40.215
INFO [StreamConnectionEstablisher:1] 2016-06-06 23:19:54,580
StreamCoordinator.java:213 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57,
ID#0] Beginning stream session with /172.31.40.215
INFO [STREAM-IN-/172.31.40.215] 2016-06-06 23:19:54,588
StreamResultFuture.java:168 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57
ID#0] Prepare completed. Receiving 0 files(0 bytes), sending 1 files(174 bytes)
INFO [STREAM-IN-/172.31.40.215] 2016-06-06 23:19:55,117
StreamResultFuture.java:182 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57]
Session with /172.31.40.215 is complete
INFO [STREAM-IN-/172.31.40.215] 2016-06-06 23:19:55,120
StreamResultFuture.java:214 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57]
All sessions completed
INFO [STREAM-IN-/172.31.40.215] 2016-06-06 23:19:55,123
LocalSyncTask.java:114 - [repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Sync
complete using session 2d177cc0-2c3d-11e6-94d2-b35b6c93de57 between
/172.31.40.215 and /172.31.44.75 on songs
INFO [RepairJobTask:3] 2016-06-06 23:19:55,123 RepairJob.java:143 - [repair
#2d177cc0-2c3d-11e6-94d2-b35b6c93de57] songs is fully synced
INFO [RepairJobTask:3] 2016-06-06 23:19:55,125 RepairSession.java:279 -
[repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Session completed successfully
INFO [RepairJobTask:3] 2016-06-06 23:19:55,126 RepairRunnable.java:240 -
Repair session 2d177cc0-2c3d-11e6-94d2-b35b6c93de57 for range
[(3074457345618258602,-9223372036854775808],
(-9223372036854775808,-3074457345618258603],
(-3074457345618258603,3074457345618258602]] finished
INFO [CompactionExecutor:991] 2016-06-06 23:19:55,131
CompactionManager.java:511 - Starting anticompaction for weitest.songs on
2/[BigTableReader(path='/mnt/ephemeral/cassandra/data/weitest/songs-b254f711134611e692c45f08f496518a/ma-2-big-Data.db'),
BigTableReader(path='/mnt/ephemeral/cassandra/data/weitest/songs-b254f711134611e692c45f08f496518a/ma-1-big-Data.db')]
sstables
INFO [CompactionExecutor:991] 2016-06-06 23:19:55,131
CompactionManager.java:540 - SSTable
BigTableReader(path='/mnt/ephemeral/cassandra/data/weitest/songs-b254f711134611e692c45f08f496518a/ma-2-big-Data.db')
fully contained in range (-9223372036854775808,-9223372036854775808], mutating
repairedAt instead of anticompacting
INFO [CompactionExecutor:991] 2016-06-06 23:19:55,135
CompactionManager.java:540 - SSTable
BigTableReader(path='/mnt/ephemeral/cassandra/data/weitest/songs-b254f711134611e692c45f08f496518a/ma-1-big-Data.db')
fully contained in range (-9223372036854775808,-9223372036854775808], mutating
repairedAt instead of anticompacting
INFO [CompactionExecutor:991] 2016-06-06 23:19:55,137
CompactionManager.java:578 - Completed anticompaction successfully
INFO [InternalResponseStage:8] 2016-06-06 23:19:55,145
RepairRunnable.java:322 - Repair command #1 finished in 0 seconds
{noformat}
This is the log entry from the 1st node where one SSTable was missing and
needed to be repaired, indeed confirming that two equivalent streaming happened
from two replica nodes:
{noformat}
INFO [AntiEntropyStage:1] 2016-06-06 23:19:54,307 Validator.java:274 -
[repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Sending completed merkle tree to
/172.31.44.75 for weitest.songs
INFO [AntiEntropyStage:1] 2016-06-06 23:19:54,470 StreamingRepairTask.java:58
- [streaming task #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Performing streaming
repair of 3 ranges with /172.31.36.148
INFO [AntiEntropyStage:1] 2016-06-06 23:19:54,497 StreamResultFuture.java:86
- [Stream #2d38e770-2c3d-11e6-80ed-e382fc580483] Executing streaming plan for
Repair
INFO [StreamConnectionEstablisher:1] 2016-06-06 23:19:54,498
StreamSession.java:238 - [Stream #2d38e770-2c3d-11e6-80ed-e382fc580483]
Starting streaming to /172.31.36.148
INFO [StreamConnectionEstablisher:1] 2016-06-06 23:19:54,512
StreamCoordinator.java:213 - [Stream #2d38e770-2c3d-11e6-80ed-e382fc580483,
ID#0] Beginning stream session with /172.31.36.148
INFO [STREAM-IN-/172.31.36.148] 2016-06-06 23:19:54,562
StreamResultFuture.java:168 - [Stream #2d38e770-2c3d-11e6-80ed-e382fc580483
ID#0] Prepare completed. Receiving 1 files(174 bytes), sending 0 files(0 bytes)
INFO [STREAM-INIT-/172.31.44.75:57066] 2016-06-06 23:19:54,579
StreamResultFuture.java:111 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57
ID#0] Creating new streaming plan for Repair
INFO [STREAM-INIT-/172.31.44.75:57066] 2016-06-06 23:19:54,580
StreamResultFuture.java:118 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57,
ID#0] Received streaming plan for Repair
INFO [STREAM-INIT-/172.31.44.75:47984] 2016-06-06 23:19:54,581
StreamResultFuture.java:118 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57,
ID#0] Received streaming plan for Repair
INFO [STREAM-IN-/172.31.44.75] 2016-06-06 23:19:54,584
StreamResultFuture.java:168 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57
ID#0] Prepare completed. Receiving 1 files(174 bytes), sending 0 files(0 bytes)
INFO [StreamReceiveTask:1] 2016-06-06 23:19:55,034
StreamResultFuture.java:182 - [Stream #2d38e770-2c3d-11e6-80ed-e382fc580483]
Session with /172.31.36.148 is complete
INFO [StreamReceiveTask:1] 2016-06-06 23:19:55,037
StreamResultFuture.java:214 - [Stream #2d38e770-2c3d-11e6-80ed-e382fc580483]
All sessions completed
INFO [StreamReceiveTask:1] 2016-06-06 23:19:55,040
StreamingRepairTask.java:85 - [repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57]
streaming task succeed, returning response to /172.31.44.75
INFO [StreamReceiveTask:2] 2016-06-06 23:19:55,114
StreamResultFuture.java:182 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57]
Session with /172.31.44.75 is complete
INFO [StreamReceiveTask:2] 2016-06-06 23:19:55,115
StreamResultFuture.java:214 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57]
All sessions completed
INFO [CompactionExecutor:3] 2016-06-06 23:19:55,130
CompactionManager.java:511 - Starting anticompaction for weitest.songs on
1/[BigTableReader(path='/mnt/ephemeral/cassandra/data/weitest/songs-b254f711134611e692c45f08f496518a/ma-4-big-Data.db'),
BigTableReader(path='/mnt/ephemeral/cassandra/data/weitest/songs-b254f711134611e692c45f08f496518a/ma-3-big-Data.db'),
BigTableReader(path='/mnt/ephemeral/cassandra/data/weitest/songs-b254f711134611e692c45f08f496518a/ma-2-big-Data.db')]
sstables
INFO [CompactionExecutor:3] 2016-06-06 23:19:55,131
CompactionManager.java:540 - SSTable
BigTableReader(path='/mnt/ephemeral/cassandra/data/weitest/songs-b254f711134611e692c45f08f496518a/ma-2-big-Data.db')
fully contained in range (-9223372036854775808,-9223372036854775808], mutating
repairedAt instead of anticompacting
INFO [CompactionExecutor:3] 2016-06-06 23:19:55,135
CompactionManager.java:578 - Completed anticompaction successfully
{noformat}
This will be a welcomed improvement for environments where compaction can
easily get behind by a lot of incoming small SSTables from repair streaming
(LCS and now-obsolete DTCS both suffer from this symptom a lot).
was:
[~jbellis] mentioned this as a potential improvement in his 2013 committer
meeting notes
(http://grokbase.com/t/cassandra/dev/132s6sh415/notes-from-committers-meeting-streaming-and-repair):
"making the repair coordinator smarter to know when to avoid duplicate
streaming. E.g., if replicas A and B have row X, but C does not, currently both
A and B will stream to C."
I tested in C* 3.0.6 and looks like this is still happening. Basically on a
3-node cluster I inserted into a trivial table under a keyspace with RF=3 and
forced two flushes on all nodes so that I have two SSTables on each node, then
I shutdown the 1st node and removed one SSTable from its data directory and
restarted the node. I connected cqlsh to this node and verified that with
CL.ONE the data is indeed missing; I now moved onto the 2nd node running a
"nodetool repair <keyspace> <table>", here are what I observed from system.log
on the 2nd node (as repair coordinator):
{noformat}
INFO [Thread-47] 2016-06-06 23:19:54,173 RepairRunnable.java:125 - Starting
repair command #1, repairing keyspace weitest with repair options (parallelism:
parallel, primary range: false, incremental: true, job threads: 1,
ColumnFamilies: [songs], dataCenters: [], hosts: [], # of ranges: 3)
INFO [Thread-47] 2016-06-06 23:19:54,253 RepairSession.java:237 - [repair
#2d177cc0-2c3d-11e6-94d2-b35b6c93de57] new session: will sync /172.31.44.75,
/172.31.40.215, /172.31.36.148 on range
[(3074457345618258602,-9223372036854775808],
(-9223372036854775808,-3074457345618258603],
(-3074457345618258603,3074457345618258602]] for weitest.[songs]
INFO [Repair#1:1] 2016-06-06 23:19:54,268 RepairJob.java:172 - [repair
#2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Requesting merkle trees for songs (to
[/172.31.40.215, /172.31.36.148, /172.31.44.75])
INFO [AntiEntropyStage:1] 2016-06-06 23:19:54,335 RepairSession.java:181 -
[repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Received merkle tree for songs
from /172.31.40.215
INFO [AntiEntropyStage:1] 2016-06-06 23:19:54,427 RepairSession.java:181 -
[repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Received merkle tree for songs
from /172.31.44.75
INFO [AntiEntropyStage:1] 2016-06-06 23:19:54,460 RepairSession.java:181 -
[repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Received merkle tree for songs
from /172.31.36.148
INFO [RepairJobTask:1] 2016-06-06 23:19:54,466 SyncTask.java:73 - [repair
#2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Endpoints /172.31.40.215 and
/172.31.36.148 have 3 range(s) out of sync for songs
INFO [RepairJobTask:1] 2016-06-06 23:19:54,467 RemoteSyncTask.java:54 -
[repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Forwarding streaming repair of 3
ranges to /172.31.40.215 (to be streamed with /172.31.36.148)
INFO [RepairJobTask:1] 2016-06-06 23:19:54,472 SyncTask.java:66 - [repair
#2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Endpoints /172.31.36.148 and
/172.31.44.75 are consistent for songs
INFO [RepairJobTask:3] 2016-06-06 23:19:54,474 SyncTask.java:73 - [repair
#2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Endpoints /172.31.40.215 and
/172.31.44.75 have 3 range(s) out of sync for songs
INFO [RepairJobTask:3] 2016-06-06 23:19:54,529 LocalSyncTask.java:68 -
[repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Performing streaming repair of 3
ranges with /172.31.40.215
INFO [RepairJobTask:3] 2016-06-06 23:19:54,574 StreamResultFuture.java:86 -
[Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57] Executing streaming plan for
Repair
INFO [StreamConnectionEstablisher:1] 2016-06-06 23:19:54,576
StreamSession.java:238 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57]
Starting streaming to /172.31.40.215
INFO [StreamConnectionEstablisher:1] 2016-06-06 23:19:54,580
StreamCoordinator.java:213 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57,
ID#0] Beginning stream session with /172.31.40.215
INFO [STREAM-IN-/172.31.40.215] 2016-06-06 23:19:54,588
StreamResultFuture.java:168 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57
ID#0] Prepare completed. Receiving 0 files(0 bytes), sending 1 files(174 bytes)
INFO [STREAM-IN-/172.31.40.215] 2016-06-06 23:19:55,117
StreamResultFuture.java:182 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57]
Session with /172.31.40.215 is complete
INFO [STREAM-IN-/172.31.40.215] 2016-06-06 23:19:55,120
StreamResultFuture.java:214 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57]
All sessions completed
INFO [STREAM-IN-/172.31.40.215] 2016-06-06 23:19:55,123
LocalSyncTask.java:114 - [repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Sync
complete using session 2d177cc0-2c3d-11e6-94d2-b35b6c93de57 between
/172.31.40.215 and /172.31.44.75 on songs
INFO [RepairJobTask:3] 2016-06-06 23:19:55,123 RepairJob.java:143 - [repair
#2d177cc0-2c3d-11e6-94d2-b35b6c93de57] songs is fully synced
INFO [RepairJobTask:3] 2016-06-06 23:19:55,125 RepairSession.java:279 -
[repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Session completed successfully
INFO [RepairJobTask:3] 2016-06-06 23:19:55,126 RepairRunnable.java:240 -
Repair session 2d177cc0-2c3d-11e6-94d2-b35b6c93de57 for range
[(3074457345618258602,-9223372036854775808],
(-9223372036854775808,-3074457345618258603],
(-3074457345618258603,3074457345618258602]] finished
INFO [CompactionExecutor:991] 2016-06-06 23:19:55,131
CompactionManager.java:511 - Starting anticompaction for weitest.songs on
2/[BigTableReader(path='/mnt/ephemeral/cassandra/data/weitest/songs-b254f711134611e692c45f08f496518a/ma-2-big-Data.db'),
BigTableReader(path='/mnt/ephemeral/cassandra/data/weitest/songs-b254f711134611e692c45f08f496518a/ma-1-big-Data.db')]
sstables
INFO [CompactionExecutor:991] 2016-06-06 23:19:55,131
CompactionManager.java:540 - SSTable
BigTableReader(path='/mnt/ephemeral/cassandra/data/weitest/songs-b254f711134611e692c45f08f496518a/ma-2-big-Data.db')
fully contained in range (-9223372036854775808,-9223372036854775808], mutating
repairedAt instead of anticompacting
INFO [CompactionExecutor:991] 2016-06-06 23:19:55,135
CompactionManager.java:540 - SSTable
BigTableReader(path='/mnt/ephemeral/cassandra/data/weitest/songs-b254f711134611e692c45f08f496518a/ma-1-big-Data.db')
fully contained in range (-9223372036854775808,-9223372036854775808], mutating
repairedAt instead of anticompacting
INFO [CompactionExecutor:991] 2016-06-06 23:19:55,137
CompactionManager.java:578 - Completed anticompaction successfully
INFO [InternalResponseStage:8] 2016-06-06 23:19:55,145
RepairRunnable.java:322 - Repair command #1 finished in 0 seconds
{noformat}
This is the log entry from the 1st node where one SSTable was missing and
needed to be repaired, indeed confirming that two equivalent streaming happened
from two replica nodes:
{noformat}
INFO [AntiEntropyStage:1] 2016-06-06 23:19:54,307 Validator.java:274 -
[repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Sending completed merkle tree to
/172.31.44.75 for weitest.songs
INFO [AntiEntropyStage:1] 2016-06-06 23:19:54,470 StreamingRepairTask.java:58
- [streaming task #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Performing streaming
repair of 3 ranges with /172.31.36.148
INFO [AntiEntropyStage:1] 2016-06-06 23:19:54,497 StreamResultFuture.java:86
- [Stream #2d38e770-2c3d-11e6-80ed-e382fc580483] Executing streaming plan for
Repair
INFO [StreamConnectionEstablisher:1] 2016-06-06 23:19:54,498
StreamSession.java:238 - [Stream #2d38e770-2c3d-11e6-80ed-e382fc580483]
Starting streaming to /172.31.36.148
INFO [StreamConnectionEstablisher:1] 2016-06-06 23:19:54,512
StreamCoordinator.java:213 - [Stream #2d38e770-2c3d-11e6-80ed-e382fc580483,
ID#0] Beginning stream session with /172.31.36.148
INFO [STREAM-IN-/172.31.36.148] 2016-06-06 23:19:54,562
StreamResultFuture.java:168 - [Stream #2d38e770-2c3d-11e6-80ed-e382fc580483
ID#0] Prepare completed. Receiving 1 files(174 bytes), sending 0 files(0 bytes)
INFO [STREAM-INIT-/172.31.44.75:57066] 2016-06-06 23:19:54,579
StreamResultFuture.java:111 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57
ID#0] Creating new streaming plan for Repair
INFO [STREAM-INIT-/172.31.44.75:57066] 2016-06-06 23:19:54,580
StreamResultFuture.java:118 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57,
ID#0] Received streaming plan for Repair
INFO [STREAM-INIT-/172.31.44.75:47984] 2016-06-06 23:19:54,581
StreamResultFuture.java:118 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57,
ID#0] Received streaming plan for Repair
INFO [STREAM-IN-/172.31.44.75] 2016-06-06 23:19:54,584
StreamResultFuture.java:168 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57
ID#0] Prepare completed. Receiving 1 files(174 bytes), sending 0 files(0 bytes)
INFO [StreamReceiveTask:1] 2016-06-06 23:19:55,034
StreamResultFuture.java:182 - [Stream #2d38e770-2c3d-11e6-80ed-e382fc580483]
Session with /172.31.36.148 is complete
INFO [StreamReceiveTask:1] 2016-06-06 23:19:55,037
StreamResultFuture.java:214 - [Stream #2d38e770-2c3d-11e6-80ed-e382fc580483]
All sessions completed
INFO [StreamReceiveTask:1] 2016-06-06 23:19:55,040
StreamingRepairTask.java:85 - [repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57]
streaming task succeed, returning response to /172.31.44.75
INFO [StreamReceiveTask:2] 2016-06-06 23:19:55,114
StreamResultFuture.java:182 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57]
Session with /172.31.44.75 is complete
INFO [StreamReceiveTask:2] 2016-06-06 23:19:55,115
StreamResultFuture.java:214 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57]
All sessions completed
INFO [CompactionExecutor:3] 2016-06-06 23:19:55,130
CompactionManager.java:511 - Starting anticompaction for weitest.songs on
1/[BigTableReader(path='/mnt/ephemeral/cassandra/data/weitest/songs-b254f711134611e692c45f08f496518a/ma-4-big-Data.db'),
BigTableReader(path='/mnt/ephemeral/cassandra/data/weitest/songs-b254f711134611e692c45f08f496518a/ma-3-big-Data.db'),
BigTableReader(path='/mnt/ephemeral/cassandra/data/weitest/songs-b254f711134611e692c45f08f496518a/ma-2-big-Data.db')]
sstables
INFO [CompactionExecutor:3] 2016-06-06 23:19:55,131
CompactionManager.java:540 - SSTable
BigTableReader(path='/mnt/ephemeral/cassandra/data/weitest/songs-b254f711134611e692c45f08f496518a/ma-2-big-Data.db')
fully contained in range (-9223372036854775808,-9223372036854775808], mutating
repairedAt instead of anticompacting
INFO [CompactionExecutor:3] 2016-06-06 23:19:55,135
CompactionManager.java:578 - Completed anticompaction successfully
{noformat}
> Duplicated effort in repair streaming
> -------------------------------------
>
> Key: CASSANDRA-11965
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11965
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
> Reporter: Wei Deng
>
> [~jbellis] mentioned this as a potential improvement in his 2013 committer
> meeting notes
> (http://grokbase.com/t/cassandra/dev/132s6sh415/notes-from-committers-meeting-streaming-and-repair):
> "making the repair coordinator smarter to know when to avoid duplicate
> streaming. E.g., if replicas A and B have row X, but C does not, currently
> both A and B will stream to C."
> I tested in C* 3.0.6 and looks like this is still happening. Basically on a
> 3-node cluster I inserted into a trivial table under a keyspace with RF=3 and
> forced two flushes on all nodes so that I have two SSTables on each node,
> then I shutdown the 1st node and removed one SSTable from its data directory
> and restarted the node. I connected cqlsh to this node and verified that with
> CL.ONE the data is indeed missing; I now moved onto the 2nd node running a
> "nodetool repair <keyspace> <table>", here are what I observed from
> system.log on the 2nd node (as repair coordinator):
> {noformat}
> INFO [Thread-47] 2016-06-06 23:19:54,173 RepairRunnable.java:125 - Starting
> repair command #1, repairing keyspace weitest with repair options
> (parallelism: parallel, primary range: false, incremental: true, job threads:
> 1, ColumnFamilies: [songs], dataCenters: [], hosts: [], # of ranges: 3)
> INFO [Thread-47] 2016-06-06 23:19:54,253 RepairSession.java:237 - [repair
> #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] new session: will sync /172.31.44.75,
> /172.31.40.215, /172.31.36.148 on range
> [(3074457345618258602,-9223372036854775808],
> (-9223372036854775808,-3074457345618258603],
> (-3074457345618258603,3074457345618258602]] for weitest.[songs]
> INFO [Repair#1:1] 2016-06-06 23:19:54,268 RepairJob.java:172 - [repair
> #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Requesting merkle trees for songs (to
> [/172.31.40.215, /172.31.36.148, /172.31.44.75])
> INFO [AntiEntropyStage:1] 2016-06-06 23:19:54,335 RepairSession.java:181 -
> [repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Received merkle tree for songs
> from /172.31.40.215
> INFO [AntiEntropyStage:1] 2016-06-06 23:19:54,427 RepairSession.java:181 -
> [repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Received merkle tree for songs
> from /172.31.44.75
> INFO [AntiEntropyStage:1] 2016-06-06 23:19:54,460 RepairSession.java:181 -
> [repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Received merkle tree for songs
> from /172.31.36.148
> INFO [RepairJobTask:1] 2016-06-06 23:19:54,466 SyncTask.java:73 - [repair
> #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Endpoints /172.31.40.215 and
> /172.31.36.148 have 3 range(s) out of sync for songs
> INFO [RepairJobTask:1] 2016-06-06 23:19:54,467 RemoteSyncTask.java:54 -
> [repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Forwarding streaming repair of
> 3 ranges to /172.31.40.215 (to be streamed with /172.31.36.148)
> INFO [RepairJobTask:1] 2016-06-06 23:19:54,472 SyncTask.java:66 - [repair
> #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Endpoints /172.31.36.148 and
> /172.31.44.75 are consistent for songs
> INFO [RepairJobTask:3] 2016-06-06 23:19:54,474 SyncTask.java:73 - [repair
> #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Endpoints /172.31.40.215 and
> /172.31.44.75 have 3 range(s) out of sync for songs
> INFO [RepairJobTask:3] 2016-06-06 23:19:54,529 LocalSyncTask.java:68 -
> [repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Performing streaming repair of
> 3 ranges with /172.31.40.215
> INFO [RepairJobTask:3] 2016-06-06 23:19:54,574 StreamResultFuture.java:86 -
> [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57] Executing streaming plan for
> Repair
> INFO [StreamConnectionEstablisher:1] 2016-06-06 23:19:54,576
> StreamSession.java:238 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57]
> Starting streaming to /172.31.40.215
> INFO [StreamConnectionEstablisher:1] 2016-06-06 23:19:54,580
> StreamCoordinator.java:213 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57,
> ID#0] Beginning stream session with /172.31.40.215
> INFO [STREAM-IN-/172.31.40.215] 2016-06-06 23:19:54,588
> StreamResultFuture.java:168 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57
> ID#0] Prepare completed. Receiving 0 files(0 bytes), sending 1 files(174
> bytes)
> INFO [STREAM-IN-/172.31.40.215] 2016-06-06 23:19:55,117
> StreamResultFuture.java:182 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57]
> Session with /172.31.40.215 is complete
> INFO [STREAM-IN-/172.31.40.215] 2016-06-06 23:19:55,120
> StreamResultFuture.java:214 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57]
> All sessions completed
> INFO [STREAM-IN-/172.31.40.215] 2016-06-06 23:19:55,123
> LocalSyncTask.java:114 - [repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Sync
> complete using session 2d177cc0-2c3d-11e6-94d2-b35b6c93de57 between
> /172.31.40.215 and /172.31.44.75 on songs
> INFO [RepairJobTask:3] 2016-06-06 23:19:55,123 RepairJob.java:143 - [repair
> #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] songs is fully synced
> INFO [RepairJobTask:3] 2016-06-06 23:19:55,125 RepairSession.java:279 -
> [repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Session completed successfully
> INFO [RepairJobTask:3] 2016-06-06 23:19:55,126 RepairRunnable.java:240 -
> Repair session 2d177cc0-2c3d-11e6-94d2-b35b6c93de57 for range
> [(3074457345618258602,-9223372036854775808],
> (-9223372036854775808,-3074457345618258603],
> (-3074457345618258603,3074457345618258602]] finished
> INFO [CompactionExecutor:991] 2016-06-06 23:19:55,131
> CompactionManager.java:511 - Starting anticompaction for weitest.songs on
> 2/[BigTableReader(path='/mnt/ephemeral/cassandra/data/weitest/songs-b254f711134611e692c45f08f496518a/ma-2-big-Data.db'),
>
> BigTableReader(path='/mnt/ephemeral/cassandra/data/weitest/songs-b254f711134611e692c45f08f496518a/ma-1-big-Data.db')]
> sstables
> INFO [CompactionExecutor:991] 2016-06-06 23:19:55,131
> CompactionManager.java:540 - SSTable
> BigTableReader(path='/mnt/ephemeral/cassandra/data/weitest/songs-b254f711134611e692c45f08f496518a/ma-2-big-Data.db')
> fully contained in range (-9223372036854775808,-9223372036854775808],
> mutating repairedAt instead of anticompacting
> INFO [CompactionExecutor:991] 2016-06-06 23:19:55,135
> CompactionManager.java:540 - SSTable
> BigTableReader(path='/mnt/ephemeral/cassandra/data/weitest/songs-b254f711134611e692c45f08f496518a/ma-1-big-Data.db')
> fully contained in range (-9223372036854775808,-9223372036854775808],
> mutating repairedAt instead of anticompacting
> INFO [CompactionExecutor:991] 2016-06-06 23:19:55,137
> CompactionManager.java:578 - Completed anticompaction successfully
> INFO [InternalResponseStage:8] 2016-06-06 23:19:55,145
> RepairRunnable.java:322 - Repair command #1 finished in 0 seconds
> {noformat}
> This is the log entry from the 1st node where one SSTable was missing and
> needed to be repaired, indeed confirming that two equivalent streaming
> happened from two replica nodes:
> {noformat}
> INFO [AntiEntropyStage:1] 2016-06-06 23:19:54,307 Validator.java:274 -
> [repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Sending completed merkle tree
> to /172.31.44.75 for weitest.songs
> INFO [AntiEntropyStage:1] 2016-06-06 23:19:54,470
> StreamingRepairTask.java:58 - [streaming task
> #2d177cc0-2c3d-11e6-94d2-b35b6c93de57] Performing streaming repair of 3
> ranges with /172.31.36.148
> INFO [AntiEntropyStage:1] 2016-06-06 23:19:54,497
> StreamResultFuture.java:86 - [Stream #2d38e770-2c3d-11e6-80ed-e382fc580483]
> Executing streaming plan for Repair
> INFO [StreamConnectionEstablisher:1] 2016-06-06 23:19:54,498
> StreamSession.java:238 - [Stream #2d38e770-2c3d-11e6-80ed-e382fc580483]
> Starting streaming to /172.31.36.148
> INFO [StreamConnectionEstablisher:1] 2016-06-06 23:19:54,512
> StreamCoordinator.java:213 - [Stream #2d38e770-2c3d-11e6-80ed-e382fc580483,
> ID#0] Beginning stream session with /172.31.36.148
> INFO [STREAM-IN-/172.31.36.148] 2016-06-06 23:19:54,562
> StreamResultFuture.java:168 - [Stream #2d38e770-2c3d-11e6-80ed-e382fc580483
> ID#0] Prepare completed. Receiving 1 files(174 bytes), sending 0 files(0
> bytes)
> INFO [STREAM-INIT-/172.31.44.75:57066] 2016-06-06 23:19:54,579
> StreamResultFuture.java:111 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57
> ID#0] Creating new streaming plan for Repair
> INFO [STREAM-INIT-/172.31.44.75:57066] 2016-06-06 23:19:54,580
> StreamResultFuture.java:118 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57,
> ID#0] Received streaming plan for Repair
> INFO [STREAM-INIT-/172.31.44.75:47984] 2016-06-06 23:19:54,581
> StreamResultFuture.java:118 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57,
> ID#0] Received streaming plan for Repair
> INFO [STREAM-IN-/172.31.44.75] 2016-06-06 23:19:54,584
> StreamResultFuture.java:168 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57
> ID#0] Prepare completed. Receiving 1 files(174 bytes), sending 0 files(0
> bytes)
> INFO [StreamReceiveTask:1] 2016-06-06 23:19:55,034
> StreamResultFuture.java:182 - [Stream #2d38e770-2c3d-11e6-80ed-e382fc580483]
> Session with /172.31.36.148 is complete
> INFO [StreamReceiveTask:1] 2016-06-06 23:19:55,037
> StreamResultFuture.java:214 - [Stream #2d38e770-2c3d-11e6-80ed-e382fc580483]
> All sessions completed
> INFO [StreamReceiveTask:1] 2016-06-06 23:19:55,040
> StreamingRepairTask.java:85 - [repair #2d177cc0-2c3d-11e6-94d2-b35b6c93de57]
> streaming task succeed, returning response to /172.31.44.75
> INFO [StreamReceiveTask:2] 2016-06-06 23:19:55,114
> StreamResultFuture.java:182 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57]
> Session with /172.31.44.75 is complete
> INFO [StreamReceiveTask:2] 2016-06-06 23:19:55,115
> StreamResultFuture.java:214 - [Stream #2d423640-2c3d-11e6-94d2-b35b6c93de57]
> All sessions completed
> INFO [CompactionExecutor:3] 2016-06-06 23:19:55,130
> CompactionManager.java:511 - Starting anticompaction for weitest.songs on
> 1/[BigTableReader(path='/mnt/ephemeral/cassandra/data/weitest/songs-b254f711134611e692c45f08f496518a/ma-4-big-Data.db'),
>
> BigTableReader(path='/mnt/ephemeral/cassandra/data/weitest/songs-b254f711134611e692c45f08f496518a/ma-3-big-Data.db'),
>
> BigTableReader(path='/mnt/ephemeral/cassandra/data/weitest/songs-b254f711134611e692c45f08f496518a/ma-2-big-Data.db')]
> sstables
> INFO [CompactionExecutor:3] 2016-06-06 23:19:55,131
> CompactionManager.java:540 - SSTable
> BigTableReader(path='/mnt/ephemeral/cassandra/data/weitest/songs-b254f711134611e692c45f08f496518a/ma-2-big-Data.db')
> fully contained in range (-9223372036854775808,-9223372036854775808],
> mutating repairedAt instead of anticompacting
> INFO [CompactionExecutor:3] 2016-06-06 23:19:55,135
> CompactionManager.java:578 - Completed anticompaction successfully
> {noformat}
> This will be a welcomed improvement for environments where compaction can
> easily get behind by a lot of incoming small SSTables from repair streaming
> (LCS and now-obsolete DTCS both suffer from this symptom a lot).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
