[
https://issues.apache.org/jira/browse/CASSANDRA-11038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337209#comment-15337209
]
Joel Knighton commented on CASSANDRA-11038:
-------------------------------------------
LGTM.
Patches look good, testall runs look good. Looks like your dtests got branched
at a bad time when there was a brief problem on the dtest branch - I rebased
your dtest branch on master and pushed at
[jkni/cassandra-dtest/11038|https://github.com/jkni/cassandra-dtest/tree/11038].
Because ccm has changes for latest trunk after the CDC merge, I've rebased
your trunk branch and pushed it at
[jkni/cassandra/11038-trunk-rebase|https://github.com/jkni/cassandra/tree/11038-trunk-rebase].
The rebase was clean so you can just rebase your trunk branch on commit. Dtest
runs after this update are clean for
[2.2|http://cassci.datastax.com/view/Dev/view/jkni/job/jkni-11038-2.2-dtest/],
[3.0|http://cassci.datastax.com/view/Dev/view/jkni/job/jkni-11038-3.0-dtest/],
and
[trunk|http://cassci.datastax.com/view/Dev/view/jkni/job/jkni-11038-trunk-dtest/].
Stefania has a PR open for her [CASSANDRA-11731] fixes
[here|https://github.com/riptano/cassandra-dtest/pull/983]. I don't see a PR
for your added test. Do you want to OK the 11731 PR and PR your added test on
commit [~beobal], or do you want me to?
> Is node being restarted treated as node joining?
> ------------------------------------------------
>
> Key: CASSANDRA-11038
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11038
> Project: Cassandra
> Issue Type: Bug
> Components: Distributed Metadata
> Reporter: cheng ren
> Assignee: Sam Tunnicliffe
> Priority: Minor
> Fix For: 2.2.x, 3.0.x, 3.x
>
>
> Hi,
> What we found recently is that every time we restart a node, all other nodes
> in the cluster treat the restarted node as a new node joining and issue node
> joining notification to clients. We have traced the code path being hit when
> a peer node detected a restarted node:
> src/java/org/apache/cassandra/gms/Gossiper.java
> {code}
> private void handleMajorStateChange(InetAddress ep, EndpointState epState)
> {
> if (!isDeadState(epState))
> {
> if (endpointStateMap.get(ep) != null)
> logger.info("Node {} has restarted, now UP", ep);
> else
> logger.info("Node {} is now part of the cluster", ep);
> }
> if (logger.isTraceEnabled())
> logger.trace("Adding endpoint state for " + ep);
> endpointStateMap.put(ep, epState);
> // the node restarted: it is up to the subscriber to take whatever
> action is necessary
> for (IEndpointStateChangeSubscriber subscriber : subscribers)
> subscriber.onRestart(ep, epState);
> if (!isDeadState(epState))
> markAlive(ep, epState);
> else
> {
> logger.debug("Not marking " + ep + " alive due to dead state");
> markDead(ep, epState);
> }
> for (IEndpointStateChangeSubscriber subscriber : subscribers)
> subscriber.onJoin(ep, epState);
> }
> {code}
> subscriber.onJoin(ep, epState) ends up with calling onJoinCluster in
> Server.java
> {code}
> src/java/org/apache/cassandra/transport/Server.java
> public void onJoinCluster(InetAddress endpoint)
> {
> server.connectionTracker.send(Event.TopologyChange.newNode(getRpcAddress(endpoint),
> server.socket.getPort()));
> }
> {code}
> We have a full trace of code path and skip some intermedia function calls
> here for being brief.
> Upon receiving the node joining notification, clients would go and scan
> system peer table to fetch the latest topology information. Since we have
> tens of thousands of client connections, scans from all of them put an
> enormous load to our cluster.
> Although in the newer version of driver, client skips fetching peer table if
> the new node has already existed in local metadata, we are still curious why
> node being restarted is handled as node joining on server side? Did we hit a
> bug or this is the way supposed to be? Our old java driver version is 1.0.4
> and cassandra version is 2.0.12.
> Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
