It sounds like a bad policy, and you should push for that to be changed. Failing that, you have some options:

1. Use faster disks. This improves cold start performance, without relying on the caches.

2. Rely on row cache instead. It can be saved periodically and loaded at startup time.

3. Ensure read CL < RF, and rely on speculative retries. Note: you will need to avoid restarting two severs owning the same token range consecutively for this to work.

These are on top of my head, but I'm sure there's more ways to do it. You should decide based on your situation.

BTW, manually load the chunk cache is never going to work unless you know what the hot data is. Load a whole table into chunk cache makes no sense unless the table on each server can fit in 512 MB of memory, but then why do you even need Cassandra?

On 21/03/2023 17:15, Carlos Diaz wrote:
Hi Team,

We are heavy users of Cassandra at a pretty big bank. Security measures require us to constantly refresh our C* nodes every x number of days.  We normally do this in a rolling fashion, taking one node down at a time and then refreshing it with a new instance.  This process has been working for us great for the past few years.

However, we recently started having issues when a newly refreshed instance comes back online, our automation waits a few minutes for the node to become "ready (UN)" and then moves on to the next node.  The problem that we are facing is that when the node is ready, the chunk cache is still empty so when the node starts accepting new connections, queries that go to take much longer to respond and this causes errors for our apps.

I was thinking that it would be great if we had a nodetool command that would allow us to prefetch a certain table or a set of tables to preload the chunk cache.  Then we could simply add another check (nodetool info?), to ensure that the chunk cache has been preloaded enough to handle queries to this particular node.

Would love to hear others' feedback on the feasibility of this idea.


Reply via email to