Both lines are "files dns". I'll look into the nscd process, but first I'm doing some dtrace analysis of the long ustacks to see which ones are taking a long time, just to make sure the mdb stack I provided wasn't an outlier.
Justin -----Original Message----- From: Brian Utterback [mailto:[EMAIL PROTECTED] Sent: Thursday, May 22, 2008 10:29 AM To: Justin Lloyd Cc: [email protected] Subject: Re: [dtrace-discuss] Slow pkginstalls due to long door_calls to nscd Okay, what does the host and ipnodes lines of your nsswitch.conf file look like? You can also try the same pstack trick on the nscd process to see where it is spending its time. Justin Lloyd wrote: > This is strange: > > bash-3.00# pstop `pgrep pkginstall` > bash-3.00# pstack `pgrep pkginstall` > 24319: pkginstall -S -C -N pkgadd -R /zones/daf98/root -a > /etc/lu/zones_pkgad > feec6104 door (4, ffbfcef0, 6400, fe9e0000, 0, 3) > bash-3.00# > > Based on the number of args, this appears to be a call to doorfs() with > a subcode of 3 (DOOR_CALL), leading to a tail call of door_call(4, > ffbfcef0). However, it occurred to me to try mdb (with which I'm > admittedly not an expert): > > bash-3.00# mdb -p `pgrep pkginstall` > Loading modules: [ ld.so.1 libc.so.1 libnvpair.so.1 libavl.so.1 > libuutil.so.1 libsysevent.so.1 libproc.so.1 ] >> ::stack > libc.so.1`_door_call+0xc(feef1668, ffbfd064, ffbfd060, ffbfd05c, > ffbfcf74, fe9e0000) > libc.so.1`_nsc_trydoorcall_ext+0x1b8(ffbfd064, ffbfd060, ffbfd05c, 0, 0, > deadbeed) > libc.so.1`_nsc_search+0xc0(ff2a6848, ff223c14, 4, ffbfd138, fe9e0000, > 7b1c8) > libc.so.1`nss_search+0x34(ff2a6848, ff223c14, 4, ffbfd138, 1, 0) > libnsl.so.1`_switch_getipnodebyname_r+0x60(ffbfd350, 78388, 7839c, 2120, > 2, 5) > libnsl.so.1`_get_hostserv_inetnetdir_byname+0x89c(6, ffbfd270, ffbfd268, > 0, ffbfd204, ffbfd274) > libnsl.so.1`getipnodebyname+0xf4(ffbfd350, 2, 5, ffbfd2ec, 1a, 2120) > is_local_host+0x10(ffbfd350, 7014c, 5, 0, 63766673, 64617067) > 0x293e4(763e8, 73, ffbfd354, 12, 1, 9) > 0x2a0c4(ffbfe02c, ffbfe02c, 4f400, 4f400, 0, 77570) > get_mntinfo+0x80(1, 0, 4f400, 0, ffbfe02c, 729e0) > main+0x97c(19, ffbff76c, 73c00, ffbffa3f, 0, 0) > _start+0x108(0, 0, 0, 0, 0, 0) >> _door_call::dis > libc.so.1`_door_call: mov 3, %o5 > libc.so.1`_door_call+4: mov 0xc9, %g1 > libc.so.1`_door_call+8: ta 8 > libc.so.1`_door_call+0xc: blu -0xa2ca8 > <libc.so.1`_cerror> > libc.so.1`_door_call+0x10: nop > libc.so.1`_door_call+0x14: retl > libc.so.1`_door_call+0x18: nop > > The call in pkginstall to is_local_host() seems to be what is sparking > the lookup. The beginning of my /etc/hosts has the following > > ::1 localhost > 127.0.0.1 localhost loghost > > Is something wrong with that? > > Justin > > > -----Original Message----- > From: Brian Utterback [mailto:[EMAIL PROTECTED] > Sent: Thursday, May 22, 2008 7:45 AM > To: Justin Lloyd > Cc: [email protected] > Subject: Re: [dtrace-discuss] Slow pkginstalls due to long door_calls to > nscd > > Well, the simple way to find out is to stop it and look. Or even to > not stop it. > > Since it is spending enough time in the door call for you to catch it > with a truss, instead of running truss, try running pstack on the same > processes. I bet it will be immediately obvious what is being looked > up by the stack. > > Justin Lloyd wrote: >> Hi all, >> >> I am installing a zone onto two different V445s running S10U4 and the >> zones are taking hours to install (about 1000 packages), that is, the >> problem is identical on both systems. A bit of trussing and dtracing > has >> shown that the pkginstalls being run by the zoneadm install are making > >> door_call calls to nscd that are taking very long, so far observed to > be >> 5 to 40 seconds, but always in times divisible by 5 seconds (5, 10, > 15, >> etc.), leading to the thought that it is one or more name service >> timeouts. Eventually, once the door_calls finish, the actual package >> install is very quick. >> >> There may be a simple misconfiguration on both systems, but I can't >> figure out what that might be. Additionally, I don't know why the >> pkginstalls are making door_calls to begin with, and I'm not sure how > to >> figure that out. So that's one my main question: >> >> *** How can I find out exactly what the pkginstalls are asking nscd > >> to resolve? *** >> >> Hopefully that will lead me (and the Sun engineer with whom I have a >> case open) to an understanding of what is leading to the presumed > timeouts. >> FWIW, here's a partial truss with timestamps of an existing pkginstall > >> process. Once it hit the ioctl(3,...) it flew through the package >> installation. Note the times around the door_calls: 13, 10 and 15 >> seconds (the first isn't a multiple of 5 since I started the truss >> during the call). >> >> Base time stamp: ***** 1210181159*.9172 [ Wed May 7 17:25:59 GMT > 2008 ] >> door_call(4, 0xFFBFD030) (sleeping...) >> >> *****1210472*.1302 door_call(4, 0xFFBFD030) > = 0 >> 1210472.1306 open("/dev/udp6", O_RDONLY) = 5 >> >> 1210472.1310 ioctl(5, SIOCGLIFNUM, 0xFFBFD2DC) = 0 >> >> 1210472.1312 close(5) = 0 >> >> 1210472.1312 getuid() = 0 > [0] >> 1210472.1313 getuid() = 0 > [0] >> *****1210472*.1314 door_info(4, 0xFFBFCF88) > = 0 >> door_call(4, 0xFFBFD030) (sleeping...) >> >> *****1210482*.1503 door_call(4, 0xFFBFD030) > = 0 >> 1210482.1505 getuid() = 0 > [0] >> 1210482.1505 getuid() = 0 > [0] >> *****1210482*.1506 door_info(4, 0xFFBFCF88) > = 0 >> door_call(4, 0xFFBFD030) (sleeping...) >> >> *****1210497*.1802 door_call(4, 0xFFBFD030) > = 0 >> 1210497.1804 ioctl(3, (('m'<<8)|7), 0xFFBFD8EC) = 0 >> >> Thanks, >> Justin >> >> >> > ------------------------------------------------------------------------ >> _______________________________________________ >> dtrace-discuss mailing list >> [email protected] > -- blu There are two rules in life: Rule 1- Don't tell people everything you know ---------------------------------------------------------------------- Brian Utterback - Solaris RPE, Sun Microsystems, Inc. Ph:877-259-7345, Em:brian.utterback-at-ess-you-enn-dot-kom _______________________________________________ dtrace-discuss mailing list [email protected]
