Drain will take care of stopping gossip, and does a few tasks before
stopping gossip (stops batchlog, hints, auth, cache saver and a few other
things). I'm not sure why this causes a side effect when you restart the
node, but there should be no need to issue a disablegossip anyway, just
leave that to the drain. As Jeff said, we need to fix drain because this
chain of commands should be unnecessary.

On 12 February 2018 at 18:36, Mike Torra <mto...@salesforce.com> wrote:

> Interestingly, it seems that changing the order of steps I take during the
> node restart resolves the problem. Instead of:
> `nodetool disablebinary && nodetool disablethrift && *nodetool
> disablegossip* && nodetool drain && sudo service cassandra restart`,
> if I do:
> `nodetool disablebinary && nodetool disablethrift && nodetool drain && 
> *nodetool
> disablegossip* && sudo service cassandra restart`,
> I see no application errors, no latency, and no nodes marked as
> Down/Normal on the restarted node. Note the only thing I changed is that I
> moved `nodetool disablegossip` to after `nodetool drain`. This is pretty
> anecdotal, but is there any explanation for why this might happen? I'll be
> monitoring my cluster closely to see if this change does indeed fix the
> problem.
> On Mon, Feb 12, 2018 at 9:33 AM, Mike Torra <mto...@salesforce.com> wrote:
>> Any other ideas? If I simply stop the node, there is no latency problem,
>> but once I start the node the problem appears. This happens consistently
>> for all nodes in the cluster
>> On Wed, Feb 7, 2018 at 11:36 AM, Mike Torra <mto...@salesforce.com>
>> wrote:
>>> No, I am not
>>> On Wed, Feb 7, 2018 at 11:35 AM, Jeff Jirsa <jji...@gmail.com> wrote:
>>>> Are you using internode ssl?
>>>> --
>>>> Jeff Jirsa
>>>> On Feb 7, 2018, at 8:24 AM, Mike Torra <mto...@salesforce.com> wrote:
>>>> Thanks for the feedback guys. That example data model was indeed
>>>> abbreviated - the real queries have the partition key in them. I am using
>>>> RF 3 on the keyspace, so I don't think a node being down would mean the key
>>>> I'm looking for would be unavailable. The load balancing policy of the
>>>> driver seems correct (https://docs.datastax.com/en/
>>>> developer/nodejs-driver/3.4/features/tuning-policies/#load-b
>>>> alancing-policy, and I am using the default `TokenAware` policy with
>>>> `DCAwareRoundRobinPolicy` as a child), but I will look more closely at the
>>>> implementation.
>>>> It was an oversight of mine to not include `nodetool disablebinary`,
>>>> but I still experience the same issue with that.
>>>> One other thing I've noticed is that after restarting a node and seeing
>>>> application latency, I also see that the node I just restarted sees many
>>>> other nodes in the same DC as being down (ie status 'DN'). However,
>>>> checking `nodetool status` on those other nodes shows all nodes as
>>>> up/normal. To me this could kind of explain the problem - node comes back
>>>> online, thinks it is healthy but many others are not, so it gets traffic
>>>> from the client application. But then it gets requests for ranges that
>>>> belong to a node it thinks is down, so it responds with an error. The
>>>> latency issue seems to start roughly when the node goes down, but persists
>>>> long (ie 15-20 mins) after it is back online and accepting connections. It
>>>> seems to go away once the bounced node shows the other nodes in the same DC
>>>> as up again.
>>>> As for speculative retry, my CF is using the default of '99th
>>>> percentile'. I could try something different there, but nodes being seen as
>>>> down seems like an issue.
>>>> On Tue, Feb 6, 2018 at 6:26 PM, Jeff Jirsa <jji...@gmail.com> wrote:
>>>>> Unless you abbreviated, your data model is questionable (SELECT
>>>>> without any equality in the WHERE clause on the partition key will always
>>>>> cause a range scan, which is super inefficient). Since you're doing
>>>>> LOCAL_ONE and a range scan, timeouts sorta make sense - the owner of at
>>>>> least one range would be down for a bit.
>>>>> If you actually have a partition key in your where clause, then the
>>>>> next most likely guess is your clients aren't smart enough to route around
>>>>> the node as it restarts, or your key cache is getting cold during the
>>>>> bounce. Double check your driver's load balancing policy.
>>>>> It's also likely the case that speculative retry may help other nodes
>>>>> route around the bouncing instance better - if you're not using it, you
>>>>> probably should be (though with CL: LOCAL_ONE, it seems like it'd be less
>>>>> of an issue).
>>>>> We need to make bouncing nodes easier (or rather, we need to make
>>>>> drain do the right thing), but in this case, your data model looks like 
>>>>> the
>>>>> biggest culprit (unless it's an incomplete recreation).
>>>>> - Jeff
>>>>> On Tue, Feb 6, 2018 at 10:58 AM, Mike Torra <mto...@salesforce.com>
>>>>> wrote:
>>>>>> Hi -
>>>>>> I am running a 29 node cluster spread over 4 DC's in EC2, using C*
>>>>>> 3.11.1 on Ubuntu. Occasionally I have the need to restart nodes in the
>>>>>> cluster, but every time I do, I see errors and application (nodejs)
>>>>>> timeouts.
>>>>>> I restart a node like this:
>>>>>> nodetool disablethrift && nodetool disablegossip && nodetool drain
>>>>>> sudo service cassandra restart
>>>>>> When I do that, I very often get timeouts and errors like this in my
>>>>>> nodejs app:
>>>>>> Error: Cannot achieve consistency level LOCAL_ONE
>>>>>> My queries are all pretty much the same, things like: "select * from
>>>>>> history where ts > {current_time}"
>>>>>> The errors and timeouts seem to go away on their own after a while,
>>>>>> but it is frustrating because I can't track down what I am doing wrong!
>>>>>> I've tried waiting between steps of shutting down cassandra, and I've
>>>>>> tried stopping, waiting, then starting the node. One thing I've noticed 
>>>>>> is
>>>>>> that even after `nodetool drain`ing the node, there are open connections 
>>>>>> to
>>>>>> other nodes in the cluster (ie looking at the output of netstat) until I
>>>>>> stop cassandra. I don't see any errors or warnings in the logs.
>>>>>> What can I do to prevent this? Is there something else I should be
>>>>>> doing to gracefully restart the cluster? It could be something to do with
>>>>>> the nodejs driver, but I can't find anything there to try.
>>>>>> I appreciate any suggestions or advice.
>>>>>> - Mike

Reply via email to