Cameron Zemek created CASSANDRA-19473:
-----------------------------------------
Summary: Latency Spike on NTR startup
Key: CASSANDRA-19473
URL: https://issues.apache.org/jira/browse/CASSANDRA-19473
Project: Cassandra
Issue Type: Improvement
Reporter: Cameron Zemek
Firstly you need the patch from
https://issues.apache.org/jira/browse/CASSANDRA-18845 to solve consistency
query errors on startup. With that patch there is still a further issue we see
on some clusters where the latency spikes too high when initially starting. I
see pending compactions and hints metrics increased during this time.
I tried lowering the hint delivery threshold across the cluster thinking it was
overloading the node starting up, but this didn't resolve the issue. So at this
time I am not sure what the root cause (I still think its combination of the
compactions and hints).
As workaround I have this small code change:
{code:java}
int START_NATIVE_DELAY =
Integer.getInteger("cassandra.start_native_transport_delay_secs", 120);
if (START_NATIVE_DELAY > 0)
{
logger.info("Waiting an extra {} seconds before enabling NTR",
START_NATIVE_DELAY);
Uninterruptibles.sleepUninterruptibly(START_NATIVE_DELAY,
TimeUnit.SECONDS);
}
startNativeTransport();
{code}
Where wait an configurable time before starting native transport. Delaying NTR
startup resolved the issue.
A better solution would be to wait for hints/compactions or whatever is root
cause to complete.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
