Re: SpamAssassin/Perl eating enormous amounts of memory?
>> On Mon, 02 Mar 2009 17:21:53 -0800, >> Andrew Moran said: A> Thank you for your suggestion. I'll try compiling Perl and it's A> dependencies without using PERL_MALLOC. I've had similar memory problems using Hyperestraier to index collections exceeding 1,000,000 documents. The indexer would run without complaint under Solaris, but die periodically on FreeBSD with "out of memory" errors. Since I had around 6 Gb of RAM, I was pretty sure memory wasn't the problem, so I recompiled using a version of Doug Lea's malloc and the problem went away. The most recent malloc sources are here: ftp://gee.cs.oswego.edu/pub/misc/malloc-2.8.3.c ftp://gee.cs.oswego.edu/pub/misc/malloc-2.8.3.h Here's a Makefile suitable for building and installing the library with GCC. I'm sure the tabs have been mangled: CC= gcc CPLUS = g++ LIBS = libmalloc.a libcppmalloc.a DEST = /usr/local/lib INC = /usr/local/include all: $(LIBS) clean: rm -f $(LIBS) *.o cppmalloc.o: malloc.c $(CPLUS) -O -c -I. malloc.c -o cppmalloc.o install: $(LIBS) cp $(LIBS) $(DEST) cp -p malloc.h $(INC) ranlib $(DEST)/libcppmalloc.a ranlib $(DEST)/libmalloc.a libcppmalloc.a: cppmalloc.o rm -f libcppmalloc.a ar q libcppmalloc.a cppmalloc.o libmalloc.a: malloc.o rm -f libmalloc.a ar q libmalloc.a malloc.o malloc.o: malloc.c $(CC) -O -c -I. malloc.c To build something using configure and this library, change the configure commands to include these environment variables: LDFLAGS="-L/usr/local/lib -lmalloc" CPPFLAGS="-I/usr/local/include" -- Karl Vogel I don't speak for the USAF or my company Why are they called apartments, when they're all stuck together? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: SpamAssassin/Perl eating enormous amounts of memory?
On Mar 3, 2009, at 12:31 PM, Mel wrote: On Tuesday 03 March 2009 07:26:02 Andrew Moran wrote: Ok sadly that didn't seem to do much: celebrian# cat /boot/loader.conf kern.maxdsiz="8G" kern.defdsiz="4G" celebrian# Can you show limits -H -d? [r...@celebrian ~]# limits -H -d Resource limits (current): datasize 8388608 kB [r...@celebrian ~]# I rebooted, but still see my memory being chewed up. Almost immediately after booting, one of my spamassassin processes spun out of control. Here's the top with it eating 16 gigs of memory: PIDUIDTHR PRI NICE SIZERES STATE C TIME WCPU COMMAND 1611 1030 1 680 15062M 818M CPU2 2 0:44 20.65% perl5.8.9 At this point, have your cd to a partition large enough to hold a few 100 megs, and type: ktrace -p 1611 where 1611 is the PID of the perl process. You may want to be a bit earlier then this point. After a few seconds, type ktrace -C. Then kdump| less. There should be plenty of allocations there (*alloc* functions). I'll try to do this if I can catch it in the act. I've mitigated the problem by turning off swap completely, which means my system doesn't get bogged down but then it's less obvious when it's happening. I do seem to be seeing a lot of other processes dying as well: pid 53393 (perl5.8.9), uid 1030, was killed: out of swap space pid 53415 (sendmail), uid 0, was killed: out of swap space pid 53401 (imap-login), uid 143, was killed: out of swap space pid 53400 (imap-login), uid 143, was killed: out of swap space pid 53399 (imap-login), uid 143, was killed: out of swap space pid 53417 (procmail), uid 1030, was killed: out of swap space pid 47702 (sendmail), uid 0, was killed: out of swap space pid 53418 (perl5.8.9), uid 1030, was killed: out of swap space pid 53416 (procmail), uid 1030, was killed: out of swap space pid 971 (dovecot), uid 0: exited on signal 11 (core dumped) This morning i had to restart many of my services. The mystery continues. --Andy ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: SpamAssassin/Perl eating enormous amounts of memory?
On Tuesday 03 March 2009 07:26:02 Andrew Moran wrote: > On Mar 3, 2009, at 8:01 AM, Andrew Moran wrote: > > In Mar 3, 2009, at 7:40 AM, Mel wrote: > >> On Monday 02 March 2009 16:21:53 Andrew Moran wrote: > What's even weirder is that the process gets that far. Did you play > with > kern.maxdsiz loader tuneable? > If so, set it lower, so you can at least have the machine in a > usable state at > all times. 4G should be enough for any process and should give > enough time > for you to spot the leak and get a ktrace. > >>> > >>> Nope, I haven't tweaked any kernel settings, just using the generic > >>> DEFAULT amd64 kernel. I've been way about tweaking settings > >>> because > >>> I don't fully understand what the 'correct' values for my setup are. > >> > >> Could you show kenv kern.maxdsiz and if unset limits -H -d? Looks > >> like it's > >> 32G on my 6.x amd64, in which case setting it is a good idea. > >> echo 'kern.maxdsiz="8G"' >> /boot/loader.conf > >> echo 'kern.defdsiz="4G"' >> /boot/loader.conf > > > > Thank you for helping me. The earlier suggestion of toggling > > Perl Malloc didn't work, nor did syncing sources and installing new > > kernel and new world which is what I did lsat night. > > > > As for kernel settings, I don't have anything in my loader.conf, > > and I'm not entirely sure how to show things that aren't sysctls, as > > this one doesn't seem to be: > > > > celebrian# sysctl -a | grep kern.maxdsiz > > celebrian# sysctl -a | grep kern.defdsiz > > celebrian# > > Ok sadly that didn't seem to do much: > > celebrian# cat /boot/loader.conf > kern.maxdsiz="8G" > kern.defdsiz="4G" > celebrian# Can you show limits -H -d? > I rebooted, but still see my memory being chewed up. Almost > immediately after booting, one of my spamassassin processes spun out > of control. Here's the top with it eating 16 gigs of memory: > >PIDUIDTHR PRI NICE SIZERES STATE C TIME WCPU > COMMAND > 1611 1030 1 680 15062M 818M CPU2 2 0:44 20.65% > perl5.8.9 At this point, have your cd to a partition large enough to hold a few 100 megs, and type: ktrace -p 1611 where 1611 is the PID of the perl process. You may want to be a bit earlier then this point. After a few seconds, type ktrace -C. Then kdump|less. There should be plenty of allocations there (*alloc* functions). > > and from dmesg: > > > swap_pager_getswapspace(16): failed > swap_pager_getswapspace(16): failed > swap_pager_getswapspace(16): failed > swap_pager_getswapspace(16): failed > swap_pager_getswapspace(3): failed > swap_pager_getswapspace(16): failed > pid 1611 (perl5.8.9), uid 1030, was killed: out of swap space > > > The computer gets pretty unresponsive during this, unless swapoff my > swap, then the process eats the memory faster and gets killed > faster. Did I type in the wrong variables? Nope, I'm now reading up myself, I would think this works on amd64. You can see defaults in /boot/defaults/loader.conf. Maybe it only works on what top shows as RES memory (truely used memory, so overallocation is possible). In that case, set kern.defdsiz to 512M, perl should bug out earlier and probably doesn't affect normal operations (busy MySQL server might). I do hope this feature isn't obsolete on amd64, I kinda like knowing there's a guard against my programming errors ;) -- Mel Problem with today's modular software: they start with the modules and never get to the software part. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: SpamAssassin/Perl eating enormous amounts of memory?
On Mar 3, 2009, at 8:01 AM, Andrew Moran wrote: In Mar 3, 2009, at 7:40 AM, Mel wrote: On Monday 02 March 2009 16:21:53 Andrew Moran wrote: What's even weirder is that the process gets that far. Did you play with kern.maxdsiz loader tuneable? If so, set it lower, so you can at least have the machine in a usable state at all times. 4G should be enough for any process and should give enough time for you to spot the leak and get a ktrace. Nope, I haven't tweaked any kernel settings, just using the generic DEFAULT amd64 kernel. I've been way about tweaking settings because I don't fully understand what the 'correct' values for my setup are. Could you show kenv kern.maxdsiz and if unset limits -H -d? Looks like it's 32G on my 6.x amd64, in which case setting it is a good idea. echo 'kern.maxdsiz="8G"' >> /boot/loader.conf echo 'kern.defdsiz="4G"' >> /boot/loader.conf Thank you for helping me. The earlier suggestion of toggling Perl Malloc didn't work, nor did syncing sources and installing new kernel and new world which is what I did lsat night. As for kernel settings, I don't have anything in my loader.conf, and I'm not entirely sure how to show things that aren't sysctls, as this one doesn't seem to be: celebrian# sysctl -a | grep kern.maxdsiz celebrian# sysctl -a | grep kern.defdsiz celebrian# Ok sadly that didn't seem to do much: celebrian# cat /boot/loader.conf kern.maxdsiz="8G" kern.defdsiz="4G" celebrian# I rebooted, but still see my memory being chewed up. Almost immediately after booting, one of my spamassassin processes spun out of control. Here's the top with it eating 16 gigs of memory: PIDUIDTHR PRI NICE SIZERES STATE C TIME WCPU COMMAND 1611 1030 1 680 15062M 818M CPU2 2 0:44 20.65% perl5.8.9 and from dmesg: swap_pager_getswapspace(16): failed swap_pager_getswapspace(16): failed swap_pager_getswapspace(16): failed swap_pager_getswapspace(16): failed swap_pager_getswapspace(3): failed swap_pager_getswapspace(16): failed pid 1611 (perl5.8.9), uid 1030, was killed: out of swap space The computer gets pretty unresponsive during this, unless swapoff my swap, then the process eats the memory faster and gets killed faster. Did I type in the wrong variables? --Andy ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: SpamAssassin/Perl eating enormous amounts of memory?
In Mar 3, 2009, at 7:40 AM, Mel wrote: On Monday 02 March 2009 16:21:53 Andrew Moran wrote: What's even weirder is that the process gets that far. Did you play with kern.maxdsiz loader tuneable? If so, set it lower, so you can at least have the machine in a usable state at all times. 4G should be enough for any process and should give enough time for you to spot the leak and get a ktrace. Nope, I haven't tweaked any kernel settings, just using the generic DEFAULT amd64 kernel. I've been way about tweaking settings because I don't fully understand what the 'correct' values for my setup are. Could you show kenv kern.maxdsiz and if unset limits -H -d? Looks like it's 32G on my 6.x amd64, in which case setting it is a good idea. echo 'kern.maxdsiz="8G"' >> /boot/loader.conf echo 'kern.defdsiz="4G"' >> /boot/loader.conf Thank you for helping me. The earlier suggestion of toggling Perl Malloc didn't work, nor did syncing sources and installing new kernel and new world which is what I did lsat night. As for kernel settings, I don't have anything in my loader.conf, and I'm not entirely sure how to show things that aren't sysctls, as this one doesn't seem to be: celebrian# sysctl -a | grep kern.maxdsiz celebrian# sysctl -a | grep kern.defdsiz celebrian# (Btw, using Perl Malloc didn't work, nor did syncing sources and installing new kernel and new world.) I'm going to try your settings in loader.conf and see if they work for me. --Andy ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: SpamAssassin/Perl eating enormous amounts of memory?
On Monday 02 March 2009 16:21:53 Andrew Moran wrote: > > What's even weirder is that the process gets that far. Did you play > > with > > kern.maxdsiz loader tuneable? > > If so, set it lower, so you can at least have the machine in a > > usable state at > > all times. 4G should be enough for any process and should give > > enough time > > for you to spot the leak and get a ktrace. > > Nope, I haven't tweaked any kernel settings, just using the generic > DEFAULT amd64 kernel. I've been way about tweaking settings because > I don't fully understand what the 'correct' values for my setup are. Could you show kenv kern.maxdsiz and if unset limits -H -d? Looks like it's 32G on my 6.x amd64, in which case setting it is a good idea. echo 'kern.maxdsiz="8G"' >> /boot/loader.conf echo 'kern.defdsiz="4G"' >> /boot/loader.conf would set it to 4G soft limit, 8G hard limit. The difference between soft and hard is, that the limits(1) program can be used to run a process with more then 4G allocatable memory and nothing can run with more then 8G, until loader tunable is changed and a reboot is done. I really have no idea why on amd64 this default is so high, surely 32G for a process is an extreme circumstance, for which one would require 4 physical CPU's to begin with. -- Mel Problem with today's modular software: they start with the modules and never get to the software part. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: SpamAssassin/Perl eating enormous amounts of memory?
On Mar 2, 2009, at 3:39 PM, Mel wrote: This is hard to debug, but I'd first toggle the PERL_MALLOC option in the configuration dialog for the perl port. If this doesn't solve the problem, then you'd have to get a ktrace to get some indication of what is allocating the memory. Thank you for your suggestion. I'll try compiling Perl and it's dependencies without using PERL_MALLOC. What's even weirder is that the process gets that far. Did you play with kern.maxdsiz loader tuneable? If so, set it lower, so you can at least have the machine in a usable state at all times. 4G should be enough for any process and should give enough time for you to spot the leak and get a ktrace. Nope, I haven't tweaked any kernel settings, just using the generic DEFAULT amd64 kernel. I've been way about tweaking settings because I don't fully understand what the 'correct' values for my setup are. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: SpamAssassin/Perl eating enormous amounts of memory?
On Sunday 01 March 2009 19:23:33 Andrew Moran wrote: > Hey guys, > > After having lots of problems with memory and 7.1/ZFS, I first > switched everything to 64-bit (amd64), and then I had a new problem, > so I eventually gave up and switched back to UFS (saying on amd64 > distro/ports), but I'm still having memory issues. > > My current one is that SpamAssassin seems to be periodically eating > up all my memory, causing the server to slow to a crawl until the > kernel kills the process and then I have enormous amounts of free > memory. Rinse, repeat. I thought maybe it had something to do with > the fact that I was using multithreaded perl (which I wasn't before I > jumped into 7.1/ZFS), but rebuilding perl (and all its' dependencies) > without threads didn't help. > > I have 8GB of physical memory and 16GB of swap memory. Here is a > line from top showing perl taking 21 gigs of memory: > > 6035 0 1 760 21190M 791M pfault 0 1:20 4.69% > perl5.8.9 This is hard to debug, but I'd first toggle the PERL_MALLOC option in the configuration dialog for the perl port. If this doesn't solve the problem, then you'd have to get a ktrace to get some indication of what is allocating the memory. What's even weirder is that the process gets that far. Did you play with kern.maxdsiz loader tuneable? If so, set it lower, so you can at least have the machine in a usable state at all times. 4G should be enough for any process and should give enough time for you to spot the leak and get a ktrace. -- Mel Problem with today's modular software: they start with the modules and never get to the software part. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: SpamAssassin/Perl eating enormous amounts of memory?
Andrew Moran wrote: [...] > Has anyone heard of this? Or any pointers on what I can do to figure > out what is causing it? > > Your advice is much appreciated. As an alternative: You could try mail/mimedefang. It calls spamassassin to evaluate an e-mail but without having it running all the time. This can solve your problem of running out of memory. For instance, sendmail could be the first line of defence with its own rules (rDNS, noMX, RBLs and what not). Then mimedefang with a set of rules to further reject dodgy e-mails (helo/ehlo, spoofings, SPF and so on). After that, spamassassin comes into play (controlled by mimedefang), but only at the end as a last line of defence. By the time it gets to the end, a lion's share has already been rejected and that means less work for spamassassin. I hope this helps. Regards, Mikhail. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
SpamAssassin/Perl eating enormous amounts of memory?
Hey guys, After having lots of problems with memory and 7.1/ZFS, I first switched everything to 64-bit (amd64), and then I had a new problem, so I eventually gave up and switched back to UFS (saying on amd64 distro/ports), but I'm still having memory issues. My current one is that SpamAssassin seems to be periodically eating up all my memory, causing the server to slow to a crawl until the kernel kills the process and then I have enormous amounts of free memory. Rinse, repeat. I thought maybe it had something to do with the fact that I was using multithreaded perl (which I wasn't before I jumped into 7.1/ZFS), but rebuilding perl (and all its' dependencies) without threads didn't help. I have 8GB of physical memory and 16GB of swap memory. Here is a line from top showing perl taking 21 gigs of memory: 6035 0 1 760 21190M 791M pfault 0 1:20 4.69% perl5.8.9 The error messages I get when this is happening look like: swap_pager_getswapspace(16): failed swap_pager_getswapspace(16): failed swap_pager_getswapspace(16): failed swap_pager_getswapspace(16): failed pid 6035 (perl5.8.9), uid 1030, was killed: out of swap space Obviously I have a memory leak somewhere, but I can't tell where. I upgraded to the latest port (synced and compiled today). I spent the day doing the "portupgrade -f R r p5-Mail-SpamAssassin" to make sure everything was rebuilt. But alas, the problem persists. Has anyone heard of this? Or any pointers on what I can do to figure out what is causing it? Your advice is much appreciated. --Andy ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
