Re: 'periodic daily' memory usage
Dag-Erling Smørgrav <[EMAIL PROTECTED]> writes: > Ulrich Spoerlein <[EMAIL PROTECTED]> writes: > > Since $MP might be empty (think NFS_ROOT or CD9660_ROOT) you'd still > > need the [ -n "$MP" ] check, AFAICS. > Correct, and thanks for noticing. ...although I ended up borrowing a trick commonly used with grep, which is to add /dev/null to the argument list, so it doesn't matter in $MP is empty. DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: 'periodic daily' memory usage
Ulrich Spoerlein <[EMAIL PROTECTED]> writes: > Since $MP might be empty (think NFS_ROOT or CD9660_ROOT) you'd still > need the [ -n "$MP" ] check, AFAICS. Correct, and thanks for noticing. DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: 'periodic daily' memory usage
On Sun, 27.01.2008 at 13:55:39 +0100, Dag-Erling Smørgrav wrote:
> --- etc/periodic/security/100.chksetuid 23 Nov 2007 13:00:31 -
> 1.9
> +++ etc/periodic/security/100.chksetuid 27 Jan 2008 12:54:38 -
> @@ -43,22 +43,17 @@
> [Yy][Ee][Ss])
> echo ""
> echo 'Checking setuid files and devices:'
> - # XXX Note that there is the possibility of overrunning the args to ls
> - MP=`mount -t ufs,zfs | egrep -v " no(suid|exec)" | awk '{ print $3 }' |
> sort`
> - if [ -n "${MP}" ]
> - then
> - set ${MP}
> - while [ $# -ge 1 ]; do
> - mount=$1
> - shift
> - find $mount -xdev -type f \
> - \( -perm -u+x -or -perm -g+x -or -perm -o+x \) \
> - \( -perm -u+s -or -perm -g+s \) -print0
> - done | xargs -0 -n 20 ls -liTd | sed 's/^ *//' | sort -k 11 |
> - check_diff setuid - "${host} setuid diffs:"
> - rc=$?
> - fi;;
> -*) rc=0;;
> + MP=`mount -t ufs,zfs | awk '$0 !~ /no(suid|exec)/ { print $3 }'`
> + find -sx $MP -type f \
> + \( -perm -u+x -or -perm -g+x -or -perm -o+x \) \
> + \( -perm -u+s -or -perm -g+s \) -print0 |
> + xargs -0 ls -liTd |
> + check_diff setuid - "${host} setuid diffs:"
> + rc=$?
> + ;;
Since $MP might be empty (think NFS_ROOT or CD9660_ROOT) you'd still
need the [ -n "$MP" ] check, AFAICS.
Cheers,
Ulrich Spoerlein
--
It is better to remain silent and be thought a fool,
than to speak, and remove all doubt.
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: 'periodic daily' memory usage
Doug Barton <[EMAIL PROTECTED]> writes: > Doug Barton <[EMAIL PROTECTED]> writes: > > Why don't you post that diff? Sounds like a good idea to me. > Oh, duh. Never mind. :) :) Now that you've seen it, any objections to committing it? DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: 'periodic daily' memory usage
Doug Barton wrote: > Why don't you post that diff? Sounds like a good idea to me. Oh, duh. Never mind. :) -- This .signature sanitized for your protection ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: 'periodic daily' memory usage
Dag-Erling Smørgrav wrote: > The solution I found that did work was to eliminate the loop over > $MP and use 'find -s $MP ...' instead, which eliminates the need > for sort. This reduces the memory requirement for 100.checksetuid > by, oh, 80% or so, and greatly simplifies the logic. Why don't you post that diff? Sounds like a good idea to me. > Note that 'find -s' and find | sort may not produce the same > output, but this only means you'll get an ugly diff the first time > you run the new script - it won't cause any trouble later. > > An entirely different issue is why named uses so much memory... It's a feature. :) If you're using it to resolve stuff out in the wild, it's caching the answers to questions it has been asked (plus whatever additional, authoritative information that was sent by the remote name server). If you artificially reduce the size of the cache, it will result in more network traffic, assuming it gets asked the same question again. If you have a situation where you have applications that ask a lot of questions once, or just a few times in quick succession (mail farms often fit this profile) then reducing the cache size probably won't hurt too much. If you have an environment where your users ask a lot of different questions, but keep asking them, reducing the cache is just going to shift the load elsewhere. One thing you can do is reduce the cleaning interval, since that will flush out the data that is past its TTL that much faster. This works particularly well for the first scenario described above. > does anybody know of a way to specify how much memory named may use > for its cache? You'll want to use BIND 9.4.2 for that if you want to do it, and take a look at the ARM manual to get the right combination of options. I'd tell you if I knew, but I never limit the cache on my name servers. :) BTW, I think you mentioned that named had grown to 24M or so, FYI that is actually a very small process size. My very lightly loaded resolver for my home network is at 23M, with hardly any cache at all. If I thought there was a good alternative resolver I'd recommend one, but IMO BIND is it right now. There are a couple others in development right now, but they are not yet ready for prime time (again, IMO). hth, Doug -- This .signature sanitized for your protection ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: 'periodic daily' memory usage
Bert JW Regeer <[EMAIL PROTECTED]> writes: > Dag-Erling Smørgrav <[EMAIL PROTECTED]> writes: > > 'periodic daily' runs my router out of swap every night, usually > > killing named as a result. > From your email it sounds like you run bind in just a caching > situation No, it is also a master for a few private zones and a dozen public ones. DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: 'periodic daily' memory usage
Mike Bristow <[EMAIL PROTECTED]> writes:
> Dag-Erling Smørgrav <[EMAIL PROTECTED]> writes:
> > An entirely different issue is why named uses so much memory... does
> > anybody know of a way to specify how much memory named may use for its
> > cache?
>
> Something like :
>
> options {
>directory "/etc/namedb";
>pid-file"/var/run/named/pid";
>*max-cache-size 10485760;
> };
That made no difference. It looked like it might at first, but after
just a few minutes RSS was back to 24 MB.
DES
--
Dag-Erling Smørgrav - [EMAIL PROTECTED]
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: 'periodic daily' memory usage
On Jan 27, 2008, at 05:55 , Dag-Erling Smørgrav wrote: 'periodic daily' runs my router out of swap every night, usually killing named as a result. From your email it sounds like you run bind in just a caching situation, have you looked at alternatives yet? For example dnscache from DJB, you give it a set amount of memory to use, and that is all it uses. [...] DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] Bert JW Regeer
Re: 'periodic daily' memory usage
Dag-Erling Smørgrav wrote:
An entirely different issue is why named uses so much memory... does
anybody know of a way to specify how much memory named may use for its
cache?
Something like :
options {
directory "/etc/namedb";
pid-file"/var/run/named/pid";
*max-cache-size 10485760;
};
According to http://www.isc.org/sw/bind/arm94/Bv9ARM.ch06.html#options
*
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
'periodic daily' memory usage
'periodic daily' runs my router out of swap every night, usually killing
named as a result.
A little sleuthing uncovered that the culprit is the 'sort -k 11'
command in /etc/periodic/security/100.checksetuid. The easy solution
would be to disable that script, but for obvious reasons, I'd rather
not.
Most of the time, named has the largest RSS of all the processes running
on my router, by an order of magnitude. It's difficult to tell precisely
since ssh'ing in to run 'top -o res' skews the results (how are you
doing, mister Heisenberg?), but it's usually named followed by sshd and
zsh.
When 100.checksetuid is running, however, sort grows larger than even
named.
I tried modifying the script to feed considerably less data to sort,
(only fields 2 and 11 from each line), but it doesn't seem to affect
sort's memory usage. I'm starting to wonder if perhaps GNU sort uses a
fixed-size buffer for each line of input, so reducing the length of the
lines makes no difference.
The solution I found that did work was to eliminate the loop over $MP
and use 'find -s $MP ...' instead, which eliminates the need for sort.
This reduces the memory requirement for 100.checksetuid by, oh, 80% or
so, and greatly simplifies the logic.
Note that 'find -s' and find | sort may not produce the same output, but
this only means you'll get an ugly diff the first time you run the new
script - it won't cause any trouble later.
An entirely different issue is why named uses so much memory... does
anybody know of a way to specify how much memory named may use for its
cache?
DES
--
Dag-Erling Smørgrav - [EMAIL PROTECTED]
Index: etc/periodic/security/100.chksetuid
===
RCS file: /home/ncvs/src/etc/periodic/security/100.chksetuid,v
retrieving revision 1.9
diff -u -r1.9 100.chksetuid
--- etc/periodic/security/100.chksetuid 23 Nov 2007 13:00:31 - 1.9
+++ etc/periodic/security/100.chksetuid 27 Jan 2008 12:54:38 -
@@ -43,22 +43,17 @@
[Yy][Ee][Ss])
echo ""
echo 'Checking setuid files and devices:'
- # XXX Note that there is the possibility of overrunning the args to ls
- MP=`mount -t ufs,zfs | egrep -v " no(suid|exec)" | awk '{ print $3 }' | sort`
- if [ -n "${MP}" ]
- then
- set ${MP}
- while [ $# -ge 1 ]; do
- mount=$1
- shift
- find $mount -xdev -type f \
- \( -perm -u+x -or -perm -g+x -or -perm -o+x \) \
- \( -perm -u+s -or -perm -g+s \) -print0
- done | xargs -0 -n 20 ls -liTd | sed 's/^ *//' | sort -k 11 |
- check_diff setuid - "${host} setuid diffs:"
- rc=$?
- fi;;
-*) rc=0;;
+ MP=`mount -t ufs,zfs | awk '$0 !~ /no(suid|exec)/ { print $3 }'`
+ find -sx $MP -type f \
+ \( -perm -u+x -or -perm -g+x -or -perm -o+x \) \
+ \( -perm -u+s -or -perm -g+s \) -print0 |
+ xargs -0 ls -liTd |
+ check_diff setuid - "${host} setuid diffs:"
+ rc=$?
+ ;;
+*)
+ rc=0
+ ;;
esac
exit $rc
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
