Hi Ulf, Today Ulf Zimmermann wrote:
> If I am running into memory, I am close to the top even shortly > after restart: > > 29837 collectd 15 0 2990m 55m 772 S 8.0 0.3 1:44.38 rrdcached > > This is a 32-bit installation right now (I was going to go > 64-bit, but had issue with .. rrdtool, although now with > rrdcached I could get around that). > > 288 connected machines right now: > > log02 ulf /home/ulf $ netstat -an | grep 42217 | grep ESTA | wc -l > 288 I would suggest you try the snapshot ... your problem seems simple enough to reproduce, so you would see quickly if it helps ... cheers tobi > > > -----Original Message----- > > From: Tobias Oetiker [mailto:[email protected]] > > Sent: Sunday, October 31, 2010 12:10 AM > > To: Ulf Zimmermann > > Cc: '[email protected]' > > Subject: Re: [rrd-users] rrdcached issues with larger number of clients > > via network/pthread > > > > HI Ulf, > > > > Yesterday Ulf Zimmermann wrote: > > > > > I got close to 300 machines running collectd, configured to use > > unixsocks to rrdcached on a central server. We are running more and > > more into threads dieing (collectd then starts complaining and fills up > > /var/messages) and when we try to restart collectd, sometimes it works, > > sometimes we end up with: > > > > > > Oct 30 22:27:19 log02 rrdcached[16864]: listen_thread_main: > > pthread_create failed. > > > Oct 30 22:27:34 log02 rrdcached[16864]: listen_thread_main: > > pthread_create failed. > > > Oct 30 22:28:10 log02 rrdcached[16864]: listen_thread_main: > > pthread_create failed. > > > > > > And at this point we usual have to restart the rrdcached daemon, > > which then means having to restart collectd on close to 300 machines. > > > > > > How can this be debugged to find the issue (potential inside of > > pthreads). The central server is running RedHat EL5 Update 4, the > > rrdtool/rrdcached is 1.4.4 from rpmforge. > > > > > > Ulf, who is getting more grey hair by the minute with issues like > > this :-( > > > > try the latest stable snapshot ... there are already a number of > > fixes in the 1.4 branche ... for memory issues and such ... maybe > > this affects your problem too. > > > > http://oss.oetiker.ch/rrdtool/pub/beta/rrdtool-1.4-svn-snap.tar.gz > > > > cheers > > tobi > > > > > > > > _______________________________________________ > > > rrd-users mailing list > > > [email protected] > > > https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users > > > > > > > > > > -- > > Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland > > http://it.oetiker.ch [email protected] ++41 62 775 9902 / sb: -9900 > > -- Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland http://it.oetiker.ch [email protected] ++41 62 775 9902 / sb: -9900 _______________________________________________ rrd-users mailing list [email protected] https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
