Re: How to warm up a cold node
Shouldn't the dynamic snitch take into account response times and ask a slow node for less requests? It seems that at node startup, only a handfull of requests arrive to the node and it keeps up well, but there's moment where there's more than it can handle with a cold cache and starts droping messages like crazy. Could it be the transition from slow node to ok node is to steep? El vie, 15-04-2011 a las 16:19 +0200, Peter Schuller escribió: Hi everyone, is there any recommended procedure to warm up a node before bringing it up? Currently the only out-of-the-box support for warming up caches is that implied by the key cache and row cache, which will pre-heat on start-up. Indexes will be indirectly preheated by index sampling, to the extent that they operating system retains them in page cache. If you're wanting to pre-heat sstables there's currently no way to do that (but it's a useful feature to have). Pragmatically, you can script something that e.g. does cat path/to/keyspace/* /dev/null or similar. But that only works if the total database size fits reasonably well in page cache. Pre-heating sstables on a per-cf basis on start-up would be a nice feature to have.
Re: How to warm up a cold node
The dynamic snitch only reduces the chance that a node used in a read operation, it depends on the RF, the CL for the operation, the partitioner and possibly the network topology. Dropping read messages is ok, so long as your operation completes at the requested CL. Are you using either a key_cache or a row_cache ? If so have you enabled the background save for those as part of the column definition? If this is about getting the OS caches warmed up you should see the pending count on the READ stage backup up and the io stats start to show longer queues. In that case Peters suggestion below is prob the best option. Another hacky way to warm things would be to use get_count() on rows the app thiks it may need to use. This will cause all the columns to be read from disk, but not sent over the network. Aaron On 20 Apr 2011, at 00:41, Héctor Izquierdo Seliva wrote: Shouldn't the dynamic snitch take into account response times and ask a slow node for less requests? It seems that at node startup, only a handfull of requests arrive to the node and it keeps up well, but there's moment where there's more than it can handle with a cold cache and starts droping messages like crazy. Could it be the transition from slow node to ok node is to steep? El vie, 15-04-2011 a las 16:19 +0200, Peter Schuller escribió: Hi everyone, is there any recommended procedure to warm up a node before bringing it up? Currently the only out-of-the-box support for warming up caches is that implied by the key cache and row cache, which will pre-heat on start-up. Indexes will be indirectly preheated by index sampling, to the extent that they operating system retains them in page cache. If you're wanting to pre-heat sstables there's currently no way to do that (but it's a useful feature to have). Pragmatically, you can script something that e.g. does cat path/to/keyspace/* /dev/null or similar. But that only works if the total database size fits reasonably well in page cache. Pre-heating sstables on a per-cf basis on start-up would be a nice feature to have.
Re: How to warm up a cold node
El mié, 20-04-2011 a las 07:59 +1200, aaron morton escribió: The dynamic snitch only reduces the chance that a node used in a read operation, it depends on the RF, the CL for the operation, the partitioner and possibly the network topology. Dropping read messages is ok, so long as your operation completes at the requested CL. Are you using either a key_cache or a row_cache ? If so have you enabled the background save for those as part of the column definition? If this is about getting the OS caches warmed up you should see the pending count on the READ stage backup up and the io stats start to show longer queues. In that case Peters suggestion below is prob the best option. Another hacky way to warm things would be to use get_count() on rows the app thiks it may need to use. This will cause all the columns to be read from disk, but not sent over the network. Aaron The rows that need to be readed change over time and they are only read for short amounts of time, so it's a bit tricky. I'll try to figure out a way to use your suggestions. In another thread I asked if it would be feasible to somehow store what parts of the sstables are hot on shutdown and re read those parts of the files on startup. Could it be done in a similar way to the work that's being done on page migrations? What do you think? Thanks for your time! On 20 Apr 2011, at 00:41, Héctor Izquierdo Seliva wrote: Shouldn't the dynamic snitch take into account response times and ask a slow node for less requests? It seems that at node startup, only a handfull of requests arrive to the node and it keeps up well, but there's moment where there's more than it can handle with a cold cache and starts droping messages like crazy. Could it be the transition from slow node to ok node is to steep? El vie, 15-04-2011 a las 16:19 +0200, Peter Schuller escribió: Hi everyone, is there any recommended procedure to warm up a node before bringing it up? Currently the only out-of-the-box support for warming up caches is that implied by the key cache and row cache, which will pre-heat on start-up. Indexes will be indirectly preheated by index sampling, to the extent that they operating system retains them in page cache. If you're wanting to pre-heat sstables there's currently no way to do that (but it's a useful feature to have). Pragmatically, you can script something that e.g. does cat path/to/keyspace/* /dev/null or similar. But that only works if the total database size fits reasonably well in page cache. Pre-heating sstables on a per-cf basis on start-up would be a nice feature to have.
How to warm up a cold node
Hi everyone, is there any recommended procedure to warm up a node before bringing it up? Thanks!
Re: How to warm up a cold node
Hi everyone, is there any recommended procedure to warm up a node before bringing it up? Currently the only out-of-the-box support for warming up caches is that implied by the key cache and row cache, which will pre-heat on start-up. Indexes will be indirectly preheated by index sampling, to the extent that they operating system retains them in page cache. If you're wanting to pre-heat sstables there's currently no way to do that (but it's a useful feature to have). Pragmatically, you can script something that e.g. does cat path/to/keyspace/* /dev/null or similar. But that only works if the total database size fits reasonably well in page cache. Pre-heating sstables on a per-cf basis on start-up would be a nice feature to have. -- / Peter Schuller
Re: How to warm up a cold node
How difficult do you think this could be? I would be interested into developing this if it's feasible. El vie, 15-04-2011 a las 16:19 +0200, Peter Schuller escribió: Hi everyone, is there any recommended procedure to warm up a node before bringing it up? Currently the only out-of-the-box support for warming up caches is that implied by the key cache and row cache, which will pre-heat on start-up. Indexes will be indirectly preheated by index sampling, to the extent that they operating system retains them in page cache. If you're wanting to pre-heat sstables there's currently no way to do that (but it's a useful feature to have). Pragmatically, you can script something that e.g. does cat path/to/keyspace/* /dev/null or similar. But that only works if the total database size fits reasonably well in page cache. Pre-heating sstables on a per-cf basis on start-up would be a nice feature to have.
