I've been having a similar problem and it's been present in all Squid version 2
releases, not just 2.5STABLE4. I've seen it in all releases of 2.4 and 2.3.
I have three identical squid servers, each with two 16GB cache directories. Right now
cachemgr reports ~265 MB of memory accounted for, but sbrk and mallinfo say ~700 MB of
memory is used. The sbrk and mallinfo reports continue to increase as the system
runs, but the "Memory accounted for:" hovers around 265 MB. See
ftp://ftp.sas.com/pub/nam/squid-mem.png for a graph that shows how our memory usage
increases. The graph is with a full cache and shows a week's worth of data following
a restart.
I've tried both the standard malloc library and compiling with --enable-dlmalloc and
have gotten the same results.
I've seen this problem on HP-UX 10.20, Red Hat 7.2, Red Hat 7.3, and now Red Hat ES
2.1.
I eventually wrote a script that checks if squid is too large. I then call that
script in the nightly cron job and restart squid if the script returns true. Here's
the script for Linux:
#!/usr/bin/perl
# returns 1 if the squid process consumes more than
# the passed-in percent of physical memory, returns 0
# otherwise. Typical use:
# squid_too_big 45
#
# Can also pass in the number of megabytes instead of
# percent:
# squid_too_big 300M
#
$Debug = 0;
$lim = 0;
$per = 0;
if ($ARGV[0] =~ /m/i) {
$lim = $ARGV[0];
$lim =~ s/[^\d]//;
$lim = int($lim) * 1024 * 1024;
} else {
$per = $ARGV[0];
$per =~ s/[^\d]//;
$per = int($per);
}
$per = 45 if ($per <= 0 || $per > 100);
# Get the physical memory in the machine
open(MEM, "</proc/meminfo");
while(<MEM>) {
next unless /^Mem:\s/;
($jnk, $tot, $used, $free) = split(' ');
}
close(MEM);
exit 0 if ($tot <= 0);
# calculate the maximum allowed
$lim = $tot * $per / 100 if ($lim <= 0 || $lim > $tot);
#
# Get the amount of memory the squid process is using
#
$tot = 0;
open(PS, "ps -efl |");
while(<PS>) {
next unless /\s\(squid\)$/;
($jnk, $jnk, $jnk, $jnk, $jnk, $jnk, $jnk, $jnk, $jnk, $tmp, $jnk)
= split(' ');
$tmp *= 4096; # convert memory pages to bytes
$tot = $tmp if ($tmp > $tot);
}
close(PS);
print "limit = $lim, squid = $tot\n" if ($Debug);
exit 1 if ($tot > $lim);
exit 0;
-----Original Message-----
From: Tay Teck Wee [mailto:[EMAIL PROTECTED]
Sent: Thursday, October 09, 2003 10:33 PM
To: Squid Users
Subject: Re: [squid-users] xmalloc and out of memory errors in messages log
Can anyone help? Thanks.
--
Wolf
--- Tay Teck Wee <[EMAIL PROTECTED]> wrote: >
--- Robert Collins <[EMAIL PROTECTED]> wrote:
> >
> On Tue, 2003-10-07 at 17:17, Tay Teck Wee wrote:
> >
> > > The Total Accounted Memory (seen from cachemgr)
> > always
> > > shows about 60-70% of the size of the squid
> > process,
> > > i.e. when squid process is at 500M, Total
> > Accounted
> > > Memory is about 350M. This ratio is maintained
> > > throughout the squid process lifetime.
> > >
> > > One thing peculiar is that under Memory usage
> for
> > > squid via mallinfo(): the total in use is always
> > 100%.
> > >
> > >
> > > From Memory Utilization->In Use->KB column, the
> > top 3
> > > are:
> > > StoreEntry 356131
> > > MD5 digest 118711
> > > mem_node 65677
> > >
> >
> > Well you've got 356MB of StoreEntries - cache
> index
> > elements. How much
> > mem cache and disk cache do you ahve in your
> > configuration?
>
> cache_mem 64 MB
> cache_dir aufs /cdata1 16000 36 256
> cache_dir aufs /cdata2 16000 36 256
> cache_dir aufs /cdata3 16000 36 256
>
> BTW the physical RAM is 2G.
>
> Thanks.
>
> >
> > Cheers,
> > Rob
> >
> >
> > --
> > GPG key available at:
> >
> <http://members.aardvark.net.au/lifeless/keys.txt>.
> >
>
> > ATTACHMENT part 2 application/pgp-signature
> name=signature.asc
>
>
> __________________________________________________
> Do You Yahoo!?
> A free party for the most "shiok" photo!
> http://sg.yahoo.com/shiok
__________________________________________________
Do You Yahoo!?
A free party for the most "shiok" photo!
http://sg.yahoo.com/shiok
