Jacob, You are right in that increasing the timeout to 20,000ms (20 seconds) is a real concern as it just hides an underlying issue with your environment. Without additional information, I was suspecting that this could be due to the environment not being optimised.
These write timeouts can occur when the systems are under load or low on resources. My questioning around memory is leading to the fact that your system(s) may possibly be under load due to GC which points to JVM running out of memory. Have a look at the logs as they will give you clues as to what is happening, and possibly the cause of the issue. And keep us posted. Thanks! Cheers, Erick On Mon, Feb 17, 2014 at 1:41 PM, Jacob Rhoden <jacob.rho...@me.com> wrote: > Hi Erick, > > On 17 Feb 2014, at 1:19 pm, Erick Ramirez <er...@ramirez.com.au> wrote: > > Are you able to post log snippets around the time that the timeouts occur? > > I have a suspicion you may be running out of heap memory and might need to > tune your environment. The INFO entries in the log should indicate this. > > > Im kicking off the load and not watching it so I don't have a timestamp to > see where it occurred. After some mucking around I worked out that adding > an extra zero to the following parameter on both nodes makes the problem > has gone away: > > write_request_timeout_in_ms: 20000 > > Whatever that parameter exactly controls, Im pretty sure I don't want to > keep a 20s write timeout :D but it allows my bulk loads to run for the time > being. > > The nodes are running on some test VM's with xmx/xms set at 1Gb. So are > you assuming that bulk counter row adding/incrementing can cause memory > issues? How much memory do you need to allocate before this category of > problem would disappear? > > Thanks, > Jacob >
