[
https://issues.apache.org/jira/browse/CASSANDRA-17691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17552823#comment-17552823
]
BugFinder commented on CASSANDRA-17691:
---------------------------------------
Just out of curiosity here, I agree with you that it's necessary to disallow
partial state gossiping, question is, is the complexity necessary? My curiosity
comes from trying to understand how these issues are approached when detected
since I assume modifying getAddressReplicas will require a lot of refactoring.
Thx!
> Gossip/Decommission tasklock contention on large clusters
> ---------------------------------------------------------
>
> Key: CASSANDRA-17691
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17691
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Gossip, Cluster/Membership
> Reporter: BugFinder
> Priority: Normal
>
> Hi,
> I am a researcher working on finding scale issues in distributed systems. I
> have been analyzing Cassandra 4.0.0 and found a potential issue on the Gossip
> path. The method
> 'org.apache.cassandra.gms.Gossiper.addLocalApplicationStates' (line 1958)
> holds the tasklock that could end up in the invocation of
> getAddressRepplicas, like this (format is [method][lineNumber]):
> [org.apache.cassandra.gms.Gossiper.addLocalApplicationStates] [1958]
> {{*Type=EXPLICIT_LOCK, start=1960, end=1970 // Lock being held along these
> lines*}}
> {{
> [org.apache.cassandra.gms.Gossiper.addLocalApplicationStateInternal][1965]}}
> {{ [org.apache.cassandra.gms.Gossiper.doOnChangeNotifications][1950]}}
> {{
> [org.apache.cassandra.gms.IEndpointStateChangeSubscriber.onChange][1551]}}
> {{ [org.apache.cassandra.service.StorageService.onChange][1551]}}
> {{
> [org.apache.cassandra.service.StorageService.handleStateRemoving][2308]}}
> {{
> [org.apache.cassandra.service.StorageService.restoreReplicaCount][2921]}}
> {{
> [org.apache.cassandra.service.StorageService.getChangedReplicasForLeaving][3128]}}
> {{
> [org.apache.cassandra.locator.AbstractReplicationStrategy.getAddressReplicas][3203]}}
> {{
> [org.apache.cassandra.locator.AbstractReplicationStrategy.getAddressReplicas][284]}}
> {{ *[line=243, dimensions=[Peers * Tokens]] // Approx. Complexity of
> this loop*}}
>
> This seems to be affecting decommission path and the complexity is at least
> dependent on the number of tokens and peers in the cluster, thus when
> decommissioning a node with a large number of peers and tokens this path will
> end up holding the Gossiper's task lock for a long time, which could end up
> causing flapping.
> This is likely to be affecting other 4.x versions too.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
