[
https://issues.apache.org/jira/browse/CASSANDRA-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16006733#comment-16006733
]
Julius Žaromskis commented on CASSANDRA-13441:
----------------------------------------------
Here's a apiece of debug log, maybe this will help to identify if it's the same
issue or not.
{noformat}
DEBUG [MemtableFlushWriter:107525] 2017-05-11 16:04:05,896 Memtable.java:401 -
Completed flushing
/mnt/storage/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/mc-22120-big-Data.db
(0.098KiB) for commitlog position ReplayPosition(segmentId=1483955071260,
position=8644066)
DEBUG [InternalResponseStage:589] 2017-05-11 16:04:05,941
MigrationManager.java:556 - Gossiping my schema version
77a40699-8e9e-35aa-834e-68c32e40a45a
DEBUG [InternalResponseStage:588] 2017-05-11 16:04:05,944
ColumnFamilyStore.java:850 - Enqueuing flush of keyspaces: 2079 (0%) on-heap, 0
(0%) off-heap
DEBUG [MemtableFlushWriter:107524] 2017-05-11 16:04:05,944 Memtable.java:368 -
Writing Memtable-keyspaces@1326973542(0.582KiB serialized bytes, 4 ops, 0%/0%
of on/off-heap limit)
DEBUG [MemtableFlushWriter:107524] 2017-05-11 16:04:05,951 Memtable.java:401 -
Completed flushing
/mnt/storage/cassandra/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/mc-22200-big-Data.db
(0.489KiB) for commitlog position ReplayPosition(segmentId=1483955071260,
position=8685297)
DEBUG [InternalResponseStage:588] 2017-05-11 16:04:05,971
ColumnFamilyStore.java:850 - Enqueuing flush of tables: 65895 (0%) on-heap, 0
(0%) off-heap
DEBUG [MemtableFlushWriter:107525] 2017-05-11 16:04:05,972 Memtable.java:368 -
Writing Memtable-tables@512792876(20.714KiB serialized bytes, 31 ops, 0%/0% of
on/off-heap limit)
DEBUG [MemtableFlushWriter:107525] 2017-05-11 16:04:05,980 Memtable.java:401 -
Completed flushing
/mnt/storage/cassandra/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mc-22197-big-Data.db
(13.019KiB) for commitlog position ReplayPosition(segmentId=1483955071260,
position=8700606)
DEBUG [InternalResponseStage:588] 2017-05-11 16:04:06,005
ColumnFamilyStore.java:850 - Enqueuing flush of columns: 204664 (0%) on-heap, 0
(0%) off-heap
DEBUG [MemtableFlushWriter:107524] 2017-05-11 16:04:06,006 Memtable.java:368 -
Writing Memtable-columns@643217662(43.015KiB serialized bytes, 286 ops, 0%/0%
of on/off-heap limit)
DEBUG [MemtableFlushWriter:107524] 2017-05-11 16:04:06,018 Memtable.java:401 -
Completed flushing
/mnt/storage/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/mc-22195-big-Data.db
(20.975KiB) for commitlog position ReplayPosition(segmentId=1483955071260,
position=8707212)
DEBUG [CompactionExecutor:47263] 2017-05-11 16:04:06,055
CompactionTask.java:146 - Compacting (75968370-3663-11e7-ab7b-d7e32ecfc62d)
[/mnt/storage/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/mc-22193-big-Data.db:level=0,
/mnt/storage/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/mc-22194-big-Data.db:level=0,
/mnt/storage/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/mc-22192-big-Data.db:level=0,
/mnt/storage/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/mc-22195-big-Data.db:level=0,
]
DEBUG [MemtablePostFlush:95264] 2017-05-11 16:04:06,057
ColumnFamilyStore.java:903 - forceFlush requested but everything is clean in
dropped_columns
DEBUG [MemtablePostFlush:95264] 2017-05-11 16:04:06,057
ColumnFamilyStore.java:903 - forceFlush requested but everything is clean in
triggers
DEBUG [MemtablePostFlush:95264] 2017-05-11 16:04:06,057
ColumnFamilyStore.java:903 - forceFlush requested but everything is clean in
views
DEBUG [MemtablePostFlush:95264] 2017-05-11 16:04:06,057
ColumnFamilyStore.java:903 - forceFlush requested but everything is clean in
types
DEBUG [MemtablePostFlush:95264] 2017-05-11 16:04:06,057
ColumnFamilyStore.java:903 - forceFlush requested but everything is clean in
functions
DEBUG [MemtablePostFlush:95264] 2017-05-11 16:04:06,057
ColumnFamilyStore.java:903 - forceFlush requested but everything is clean in
aggregates
DEBUG [InternalResponseStage:588] 2017-05-11 16:04:06,057
ColumnFamilyStore.java:850 - Enqueuing flush of indexes: 598 (0%) on-heap, 0
(0%) off-heap
DEBUG [MemtableFlushWriter:107525] 2017-05-11 16:04:06,057 Memtable.java:368 -
Writing Memtable-indexes@1653592334(0.127KiB serialized bytes, 1 ops, 0%/0% of
on/off-heap limit)
DEBUG [MemtableFlushWriter:107525] 2017-05-11 16:04:06,068 Memtable.java:401 -
Completed flushing
/mnt/storage/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/mc-22121-big-Data.db
(0.098KiB) for commitlog position ReplayPosition(segmentId=1483955071260,
position=8742156)
DEBUG [CompactionExecutor:47266] 2017-05-11 16:04:06,090
CompactionTask.java:146 - Compacting (759bdaa0-3663-11e7-ab7b-d7e32ecfc62d)
[/mnt/storage/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/mc-22120-big-Data.db:level=0,
/mnt/storage/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/mc-22121-big-Data.db:level=0,
/mnt/storage/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/mc-22119-big-Data.db:level=0,
/mnt/storage/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/mc-22118-big-Data.db:level=0,
]
DEBUG [InternalResponseStage:588] 2017-05-11 16:04:06,112
MigrationManager.java:556 - Gossiping my schema version
77a40699-8e9e-35aa-834e-68c32e40a45a
DEBUG [CompactionExecutor:47263] 2017-05-11 16:04:06,120
CompactionTask.java:218 - Compacted (75968370-3663-11e7-ab7b-d7e32ecfc62d) 4
sstables to
[/mnt/storage/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/mc-22196-big,]
to level=0. 29,348 bytes to 10,277 (~35% of original) in 64ms = 0.153139MB/s.
0 total partitions merged to 6. Partition merge counts were {1:2, 4:4, }
DEBUG [CompactionExecutor:47266] 2017-05-11 16:04:06,136
CompactionTask.java:218 - Compacted (759bdaa0-3663-11e7-ab7b-d7e32ecfc62d) 4
sstables to
[/mnt/storage/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/mc-22122-big,]
to level=0. 392 bytes to 98 (~25% of original) in 45ms = 0.002077MB/s. 0
total partitions merged to 1. Partition merge counts were {4:1, }
DEBUG [NonPeriodicTasks:1] 2017-05-11 16:04:06,782 MigrationManager.java:124 -
submitting migration task for /10.240.0.6
DEBUG [NonPeriodicTasks:1] 2017-05-11 16:04:06,782 MigrationManager.java:124 -
submitting migration task for /10.240.0.7
DEBUG [InternalResponseStage:586] 2017-05-11 16:04:06,792
ColumnFamilyStore.java:850 - Enqueuing flush of keyspaces: 2079 (0%) on-heap, 0
(0%) off-heap
DEBUG [MemtableFlushWriter:107524] 2017-05-11 16:04:06,793 Memtable.java:368 -
Writing Memtable-keyspaces@847538504(0.582KiB serialized bytes, 4 ops, 0%/0% of
on/off-heap limit)
DEBUG [MemtableFlushWriter:107524] 2017-05-11 16:04:06,799 Memtable.java:401 -
Completed flushing
/mnt/storage/cassandra/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/mc-22201-big-Data.db
(0.489KiB) for commitlog position ReplayPosition(segmentId=1483955071260,
position=8939092)
DEBUG [CompactionExecutor:47266] 2017-05-11 16:04:06,816
CompactionTask.java:146 - Compacting (760aa200-3663-11e7-ab7b-d7e32ecfc62d)
[/mnt/storage/cassandra/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/mc-22201-big-Data.db:level=0,
/mnt/storage/cassandra/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/mc-22198-big-Data.db:level=0,
/mnt/storage/cassandra/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/mc-22199-big-Data.db:level=0,
/mnt/storage/cassandra/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/mc-22200-big-Data.db:level=0,
]
DEBUG [InternalResponseStage:586] 2017-05-11 16:04:06,816
ColumnFamilyStore.java:850 - Enqueuing flush of tables: 65895 (0%) on-heap, 0
(0%) off-heap
DEBUG [MemtableFlushWriter:107525] 2017-05-11 16:04:06,817 Memtable.java:368 -
Writing Memtable-tables@1477384931(20.714KiB serialized bytes, 31 ops, 0%/0% of
on/off-heap limit)
{noformat}
This has continued for several hours until 've decided to continue with the
migration to 3.0.13, here's the result:
{noformat}
Cluster Information:
Name: cloud.zaromskis.lt cluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
600b7268-d42a-3b72-8706-093b6c8cfaff: [10.240.0.6, 10.240.0.7,
10.240.0.8]
{noformat}
Looks like cluster is consistent now
> Schema version changes for each upgraded node in a rolling upgrade, causing
> migration storms
> --------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-13441
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13441
> Project: Cassandra
> Issue Type: Bug
> Components: Schema
> Reporter: Jeff Jirsa
> Assignee: Jeff Jirsa
> Fix For: 3.0.14, 3.11.0, 4.0
>
>
> In versions < 3.0, during a rolling upgrade (say 2.0 -> 2.1), the first node
> to upgrade to 2.1 would add the new tables, setting the new 2.1 version ID,
> and subsequently upgraded hosts would settle on that version.
> When a 3.0 node upgrades and writes its own new-in-3.0 system tables, it'll
> write the same tables that exist in the schema with brand new timestamps. As
> written, this will cause all nodes in the cluster to change schema (to the
> version with the newest timestamp). On a sufficiently large cluster with a
> non-trivial schema, this could cause (literally) millions of migration tasks
> to needlessly bounce across the cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
