I just upgraded to 1.4.10 from 1.4.5. In 1.4.5 we used a unix socket instead of tcp, and memcached would crash as soon as a single "Couldn't realloc input buffer" error ocurred.
The server has plenty of memory and is monitored by nagios which doesn't show any oddities at the time of the crashes. The only thing that worries me is that openvz might have something to do with it? Although with these generous limits and few connections I can't think of anything. Taking a quick look at the source code at the realloc error message shows that it's a buffer used to read from a connection to the memcached. It doubles this buffer in size whenever not enough memory is available to read data from the connection. It starts at 2K. I cannot imagine that any connection could have so much data going over it that this would ever be a problem?? Very strange.. maybe I should add extra debugging info there displaying the amount of memory it tried to allocate. For now I have just made sure that all the objects do have an expiry time set so that it should never hit full capacity. I'll update here if this at least solves the crashing / protocol / memory errors.
