getpwnam
I encountered a situation where sendmail was opening up what appeared to be
listening on random UDP ports. In the process of tracking this down I
discovered that the culprit is getpwnam. A ktrace of the following simple
program show whats happening:
#include stdio.h
#include stdlib.h
#include string.h
#include sys/types.h
#include pwd.h
int main (int argc, char *argv[])
{
struct passwd *pe;
pe = getpwnam (xxx);
}
Note, xxx is a valid user id in that system.
The ktrace output is way too large to include here, but below is the
interesting portion. Note that it creates a socket with address of 0.0.0.0:932
and then sends data to it. Somehow it gets a response although I can find
nothing other than this program using that port. I would think that it would
use a unix socket rather than UDP to access local NIS information. The
unknown address family error is also puzzling. I have traced this into
_nsdispatch but it gets a bit convoluted at that point with all the caching.
What is this doing?
87443 test CALL socket(PF_INET,SOCK_DGRAM,IPPROTO_UDP)
87443 test RET socket 4
87443 test CALL getsockname(0x4,0xbfbfe16c,0xbfbfe1f0)
87443 test STRU struct sockaddr { AF_INET, 0.0.0.0:0 }
87443 test RET getsockname 0
87443 test CALL getsockopt(0x4,0,0x13,0xbfbfe1f8,0xbfbfe1ec)
87443 test RET getsockopt 0
87443 test CALL setsockopt(0x4,0,0x13,0xbfbfe1f4,0x4)
87443 test RET setsockopt 0
87443 test CALL bind(0x4,0xbfbfe16c,0x10)
87443 test STRU struct sockaddr { AF_INET, 0.0.0.0:0 }
87443 test RET bind 0
87443 test CALL getsockname(0x4,0xbfbfe144,0xbfbfe1c8)
87443 test STRU struct sockaddr { AF_INET, 0.0.0.0:932 }
87443 test RET getsockname 0
87443 test CALL getsockopt(0x4,SOL_SOCKET,SO_TYPE,0xbfbfe1c4,0xbfbfe1c8)
87443 test RET getsockopt 0
87443 test CALL getrlimit(RLIMIT_NOFILE,0xbfbfe0f4)
87443 test RET getrlimit 0
87443 test CALL getsockname(0x4,0xbfbfe074,0xbfbfe0f8)
87443 test STRU struct sockaddr { AF_INET, 0.0.0.0:932 }
87443 test RET getsockname 0
87443 test CALL getsockopt(0x4,SOL_SOCKET,SO_TYPE,0xbfbfe0f4,0xbfbfe0f8)
87443 test RET getsockopt 0
87443 test CALL gettimeofday(0xbfbfe1c0,0)
87443 test RET gettimeofday 0
87443 test CALL getpid
87443 test RET getpid 87443/0x15593
87443 test CALL ioctl(0x4,FIONBIO,0xbfbfe1c8)
87443 test RET ioctl 0
87443 test CALL fcntl(0x4,F_SETFD,FD_CLOEXEC)
87443 test RET fcntl 0
87443 test CALL bind(0x4,0xbfbfe700,0x10)
87443 test STRU struct sockaddr { AF_UNSPEC, unknown address family }
87443 test RET bind -1 errno 22 Invalid argument
87443 test CALL getsockname(0x4,0xbfbfe700,0xbfbfe740)
87443 test STRU struct sockaddr { AF_INET, 0.0.0.0:932 }
87443 test RET getsockname 0
87443 test CALL gettimeofday(0xbfbfe740,0)
87443 test RET gettimeofday 0
87443 test CALL kqueue
87443 test RET kqueue 5
87443 test CALL sendto(0x4,0x282359f4,0x48,0,0x28235008,0x10)
87443 test GIO fd 4 wrote 72 bytes
0x 4b8e 2491 0002 0001 86a4 |K.$.|
0x0010 0002 000a ||
0x0020 0004 7465 7374 |test|
0x0030 0014 6d61 7374 6572 2e70 6173 7377 |master.passw|
0x0040 642e 6279 6e61 6d65 |d.byname|
87443 test RET sendto 72/0x48
87443 test CALL kevent(0x5,0x282350dc,0x1,0xbfbfe6fc,0x1,0xbfbfe730)
87443 test GIO fd 5 wrote 20 bytes
0x 0400 0100 ||
0x0010 ||
87443 test GIO fd 5 read 20 bytes
0x 0400 2000 | ...|
0x0010 ||
87443 test RET kevent 1
87443 test CALL recvfrom(0x4,0x282350f4,0x900,0,0,0)
87443 test GIO fd 4 read 32 bytes
0x 4b8e 2491 0001 |K.$.|
0x0010 0001 4a3f f709 |J?..|
87443 test STRU struct sockaddr { AF_INET, 206.117.18.7:876 }
87443 test RET recvfrom 32/0x20
87443 test CALL close(0x5)
87443 test RET close 0
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: New FreeBSD 5.3 e-mail server extremely slow - traced to getpwnam maybe ?
Quoting Kris Kennaway [EMAIL PROTECTED]:
On Tue, Jan 04, 2005 at 09:27:27PM -0500, Bruce Campbell wrote:
I wrote a small program:
#include sys/types.h
#include pwd.h
main( int argc, char *argv[] )
{
getpwuid( 13076 );
}
and ran it under truss on 5.x and it generated 178,711 lines of output.
(the bulk of which is those lseek/read calls as above)
...
Try tuning the pwd_mkdb parameters (see hash(3)) in
/usr/src/usr.sbin/pwd_mkdb/pwd_mkdb.c and recompile:
HASHINFO openinfo = {
4096, /* bsize */
32, /* ffactor */
256,/* nelem */
2048 * 1024,/* cachesize */
NULL, /* hash() */
0 /* lorder */
};
e.g. adjust nelem to 12000 to accomodate your
significantly-larger-than-average password database. If this helps,
please submit a PR requesting that someone make an option to pwd_mkdb
to tune this at runtime (or better yet, submit the patch to do this
yourself - it's straightforward to modify the source to do this).
Thanks. That had no effect on the large number of seeks/reads
to do a getpwuid of a specific uid. I tried boosting that
number further, still no change. I suspect the problem is related
to some change to the hash functions between 4.7 and 5.2.1 and I
hope to get to the bottom of it today.
I tried two getpwnam (as opposed to getpwuid) calls on 2 different userids, one
took 1000 seek/reads, the other 16,000, so it's all
pretty random, no doubt related to how stuff gets hashed. On
4.7 it takes just one or two reads/seeks.
As each login via ipop, imap, and each sendmail, and just about everything
will be doing getpwnam's I think this is our problem.
--
Bruce Campbell
Engineering Computing
CPH-2374B
University of Waterloo
(519)888-4567 ext 5889
This mail sent through www.mywaterloo.ca
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: New FreeBSD 5.3 e-mail server extremely slow - traced to getpwnam maybe ?
Quoting Bruce Campbell [EMAIL PROTECTED]:
On Tue, Jan 04, 2005 at 09:27:27PM -0500, Bruce Campbell wrote:
I wrote a small program:
#include sys/types.h
#include pwd.h
main( int argc, char *argv[] )
{
getpwuid( 13076 );
}
and ran it under truss on 5.x and it generated 178,711 lines of output.
(the bulk of which is those lseek/read calls as above)
It looks like the overhaul of getpwent Apr/2003 to make it thread safe:
http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libc/gen/getpwent.c
may be the problem.
I've tested the dbm_fetch function independently on a large
file, and it is fine.
I've opened a bug report, and plan to build a replacement 4.x
mail server, as the most deterministic path to restoring
adequate e-mail service to our users.
Can anyone suggest a workaround ?
--
Bruce Campbell
Engineering Computing
CPH-2374B
University of Waterloo
(519)888-4567 ext 5889
This mail sent through www.mywaterloo.ca
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: New FreeBSD 5.3 e-mail server extremely slow - traced to getpwnam maybe ?
Quoting Bruce Campbell [EMAIL PROTECTED]:
Quoting Bruce Campbell [EMAIL PROTECTED]:
On Tue, Jan 04, 2005 at 09:27:27PM -0500, Bruce Campbell wrote:
I wrote a small program:
#include sys/types.h
#include pwd.h
main( int argc, char *argv[] )
{
getpwuid( 13076 );
}
and ran it under truss on 5.x and it generated 178,711 lines of output.
(the bulk of which is those lseek/read calls as above)
It looks like the overhaul of getpwent Apr/2003 to make it thread safe:
http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libc/gen/getpwent.c
may be the problem.
I've tested the dbm_fetch function independently on a large
file, and it is fine.
I've opened a bug report, and plan to build a replacement 4.x
mail server, as the most deterministic path to restoring
adequate e-mail service to our users.
Can anyone suggest a workaround ?
Well, somewhat unbelievably, copying a getpwent.c from 4.7
and remaking libc on 5.3 with it worked. Load average
has gone from 70 to 2.
And, so that this qualifies as a question...
Am I crazy to pull an old getpwnam from 4.7 and blindly
build it on 5.3 ?
--
Bruce Campbell
Engineering Computing
CPH-2374B
University of Waterloo
(519)888-4567 ext 5889
This mail sent through www.mywaterloo.ca
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: New FreeBSD 5.3 e-mail server extremely slow - traced to getpwnam maybe ?
Quoting Bruce Campbell [EMAIL PROTECTED]: ... Well, somewhat unbelievably, copying a getpwent.c from 4.7 and remaking libc on 5.3 with it worked. Load average has gone from 70 to 2. One of my co-workers has found a less kludgey workaround for the high load problem we were seeing on 5.3 with large /etc/master.passwd, as follows: --- /etc/nsswitch.conf.old Wed Jan 5 19:23:24 2005 +++ /etc/nsswitch.conf Wed Jan 5 19:23:43 2005 @@ -1,7 +1,7 @@ -group: compat +group: files group_compat: nis hosts: files dns networks: files -passwd: compat +passwd: files passwd_compat: nis shells: files System is purring with load average under 1 now, 200,000 pop/imap sessions per day and 200,000 e-mails per day, all spamassassinated. For more details and ongoing followup, see: http://www.freebsd.org/cgi/query-pr.cgi?pr=75855 -- Bruce Campbell Engineering Computing CPH-2374B University of Waterloo (519)888-4567 ext 5889 This mail sent through www.mywaterloo.ca ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: New FreeBSD 5.3 e-mail server extremely slow - traced to getpwnam maybe ?
Quoting Kris Kennaway [EMAIL PROTECTED]:
Well, no, not quite.
old: imap-uw-2002_1,1
new: imap-uw-2004a,1
OK, that's where you should start, then. Go back to the software
configuration that you know is working and see if it still misbehaves.
Kris
Thanks. I shutdown imapd/ipop3d completely so I just had sendmail running, and
still
load av. was 20-30.
Anyways, I have just found something very odd with both 5.2.1 and 5.3
on multiple different systems here, including a brand new GENERIC install.
On 5.x, ls -l or ps waux is very slow with our
/etc/master.passwd which has 11320 entries. I truss'ed
those commands, and gave up after watching :
lseek(4,0x17d000,SEEK_SET) = 1560576 (0x17d000)
read(0x4,0x8074000,0x1000) = 4096 (0x1000)
lseek(4,0x17e000,SEEK_SET) = 1564672 (0x17e000)
read(0x4,0x8062000,0x1000) = 4096 (0x1000)
lseek(4,0x17f000,SEEK_SET) = 1568768 (0x17f000)
read(0x4,0x8066000,0x1000) = 4096 (0x1000)
lseek(4,0x18,SEEK_SET) = 1572864 (0x18)
scroll by for 10 minutes. (handle 4 = /etc/spwd.db)
I wrote a small program:
#include sys/types.h
#include pwd.h
main( int argc, char *argv[] )
{
getpwuid( 13076 );
}
and ran it under truss on 5.x and it generated 178,711 lines of output.
(the bulk of which is those lseek/read calls as above)
4.7 (with same master.passwd file) gave 59 lines of output, which seems
normal.
I'm speculating that imap and sendmail and just about everything use
getpwuid and getpwuid is misbehaving on 5.x especially with a large
master.passwd file.
I will report this through the proper mechanism once I do
just a bit more testing. And perhaps it is a known issue
already and I'll look into that also. Or perhaps I have messed
something up unwittingly, which I have been known to do.
We do have an extremely busy 5.2.1 system running here fine on
the same hardware, just it has a small /etc/master.passwd which may explain that
systems success to date.
Thank you to everyone who sent suggestions.
--
Bruce Campbell
Engineering Computing
CPH-2374B
University of Waterloo
(519)888-4567 ext 5889
This mail sent through www.mywaterloo.ca
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: New FreeBSD 5.3 e-mail server extremely slow - traced to getpwnam maybe ?
On Tue, Jan 04, 2005 at 09:27:27PM -0500, Bruce Campbell wrote:
I wrote a small program:
#include sys/types.h
#include pwd.h
main( int argc, char *argv[] )
{
getpwuid( 13076 );
}
and ran it under truss on 5.x and it generated 178,711 lines of output.
(the bulk of which is those lseek/read calls as above)
4.7 (with same master.passwd file) gave 59 lines of output, which seems
normal.
I'm speculating that imap and sendmail and just about everything use
getpwuid and getpwuid is misbehaving on 5.x especially with a large
master.passwd file.
Try tuning the pwd_mkdb parameters (see hash(3)) in
/usr/src/usr.sbin/pwd_mkdb/pwd_mkdb.c and recompile:
HASHINFO openinfo = {
4096, /* bsize */
32, /* ffactor */
256,/* nelem */
2048 * 1024,/* cachesize */
NULL, /* hash() */
0 /* lorder */
};
e.g. adjust nelem to 12000 to accomodate your
significantly-larger-than-average password database. If this helps,
please submit a PR requesting that someone make an option to pwd_mkdb
to tune this at runtime (or better yet, submit the patch to do this
yourself - it's straightforward to modify the source to do this).
Kris
pgpAjUrFD81hG.pgp
Description: PGP signature
