[
https://issues.apache.org/jira/browse/CASSANDRA-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099052#comment-16099052
]
Tania S Engel commented on CASSANDRA-13441:
-------------------------------------------
It is unclear to me if the migration task timing out caused none of our tables
to stream. I am 100% certain we have keyspaces on the seed node but none of it
was streamed. From reading the various related/duplicate cases, it is unclear
if the suggestion is to increase -Dcassandra.migration_task_wait_in_seconds or
upgrade to 3.11 or both. This problem is intermittent and it happens even when
we are bootstrapping the first node to form a cluster of 2. Is the idea that if
we "wait for schema agreement...", we will eventually get the tables? It seems
that is not the case for this scenario. Right now our code will wait for
>nodetool netstats MODE=NORMAL, but in this case, the CQL port is already open
and we have already seen "boostrap complete" so I am 99% sure the MODE=NORMAL
(no longer JOINING).
Also, I have seen a bootstrap scenario where just one "ERROR Migration task
failed to complete" logged and then the bootstrap succeeded and streamed all
our tables.
> Schema version changes for each upgraded node in a rolling upgrade, causing
> migration storms
> --------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-13441
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13441
> Project: Cassandra
> Issue Type: Bug
> Components: Distributed Metadata
> Reporter: Jeff Jirsa
> Assignee: Jeff Jirsa
> Fix For: 3.0.14, 3.11.0, 4.0
>
> Attachments: screenshot-1.png
>
>
> In versions < 3.0, during a rolling upgrade (say 2.0 -> 2.1), the first node
> to upgrade to 2.1 would add the new tables, setting the new 2.1 version ID,
> and subsequently upgraded hosts would settle on that version.
> When a 3.0 node upgrades and writes its own new-in-3.0 system tables, it'll
> write the same tables that exist in the schema with brand new timestamps. As
> written, this will cause all nodes in the cluster to change schema (to the
> version with the newest timestamp). On a sufficiently large cluster with a
> non-trivial schema, this could cause (literally) millions of migration tasks
> to needlessly bounce across the cluster.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
