On Tue, 2011-07-12 at 08:33 -0700, Brickles, Stephen wrote:
> This bug seems to crop up from time to time. I’ve never had much luck
> in being able
>
> to fix this by running Redhat updates. It seems to be pretty random.
> Finally I found a
>
> machine on which this bug recurred over and over and so I was able to
> set up debugging on it.
>
> The mount point listed in these maps below is a fully functioning
> mount-point most of the time.
>
> There are other mount points in ‘auto.mntpnt’ which have the same
> random error – I just pruned
>
> these out to make the explanation simpler.
>
> I set up the debugging according to Jeff’s instructions at
> http://people.redhat.com/jmoyer
>
> and the output is below.
>
> It would appear from Google that “found negative cache entry for key”
> is an issue which seems
>
> to have come up before. Also from the debug log, it’s difficult to
> tell if this is a problem
That's probably a consequence of the mount failures but I'm not very
familiar with the caching system changes that Jeff did for autofs
version 4 in RHEL-4.
>
> with ‘autofs’ or ‘mount’.
The RPC error is coming from mount.
>
> Also this machine is part of a simulation server farm. It would
> appear that sending a sequence
>
> of simulation jobs in rapid succession to a host (ie. multiple rsh by
> a user in a short period of time)
>
> causing a whole bunch of mount requests at once, seems more likely
> trigger this problem.
Which is usually what causes reserved port exhaustion. Just how many
reserved ports are actually available has changed somewhat over time and
is configurable to a limit (obviously), not sure about RHEL-4.
One thing that compounds the problem is that mount.nfs probes mountd for
version information and NFS for version information leaving behind a
bunch of reserved ports in a wait state. Then the kernel uses another
port for the mount itself. Again I haven't looked at RHEL-4 for some
time so I can't say whether NFS in RHEL-4 can share the port used for
all mounts to a given server, it certainly does in RHEL-5 and later. If
NFS doesn't do that then each mount will take another port until it is
umounted. The wait state (I think 60 seconds) I mentioned is required by
the TCP protocol and so is unavoidable and those ports cannot be reused
until the timeout has expired. Basically, it's fairly easy to use a lot
of reserved ports really quickly when rapidly mounting mounts and end up
with not a great many mounts actually done.
There's not a lot you can do about the port exhaustion other than ensure
that the NFS server allows connections from higher numbered ports (ie.
not in reserved port range) and ensure that your version of mount.nfs
also supports it. I think the RHEL-4 you are using should be fine with
that for mount.nfs but I think it still uses a little more ports than it
really could get away with.
>
> Thanks,
>
> Stephen
>
> > uname -a
>
> Linux server3 2.6.9-100.ELsmp #1 SMP Tue Feb 1 12:04:42 EST 2011
> x86_64 x86_64 x86_64 GNU/Linux
>
> > cat /etc/redhat-release
>
> Red Hat Enterprise Linux WS release 4 (Nahant Update 9)
>
> > rpm -qa | grep autofs
>
> autofs-4.1.3-240
>
> > rpm -qa | grep nfs-utils
>
> nfs-utils-lib-devel-1.0.6-10.el4_8.1
>
> nfs-utils-1.0.6-94.EL4
>
> nfs-utils-lib-1.0.6-10.el4_8.1
>
> auto.master:
>
> /mntpnt auto.mntpnt -rw,intr,soft,vers=3
>
> auto.mntpnt:
>
> mail -rw,actimeo=0,vers=3 nis_server:/var/mail
>
> /var/log/messages:
>
> Jul 11 23:54:01 server3 automount[6778]: attempting to mount
> entry /mntpnt/mail
>
> Jul 11 23:54:01 server3 kernel: RPC: Can't bind to reserved port (98).
>
> Jul 11 23:54:01 server3 kernel: RPC: can't bind to reserved port.
>
> Jul 11 23:54:01 server3 kernel: RPC: error 5 connecting to server
> nis_server
>
> Jul 11 23:54:01 server3 kernel: RPC: Can't bind to reserved port (98).
>
> Jul 11 23:54:01 server3 kernel: RPC: can't bind to reserved port.
>
> Jul 11 23:54:01 server3 automount[3564]: >> mount:
> nis_server:/var/mail: can't read superblock
>
> Jul 11 23:54:01 server3 kernel: RPC: error 5 connecting to server
> nis_server
>
> Jul 11 23:54:01 server3 automount[3564]: mount(nfs): nfs: mount
> failure nis_server:/var/mail on /mntpnt/mail
>
> Jul 11 23:54:01 server3 automount[3564]: failed to mount /mntpnt/mail
>
> /var/log/debug:
>
> Jul 11 23:54:01 server3 automount[6778]: send_fail: token=247707
>
> Jul 11 23:54:01 server3 automount[6778]: handle_packet: type = 0
>
> Jul 11 23:54:01 server3 automount[6778]: handle_packet_missing: token
> 247708, name mail
>
> Jul 11 23:54:01 server3 automount[6778]: handle_packet_missing:
> expired negative cache entry for key.
>
> Jul 11 23:54:01 server3 automount[6778]: attempting to mount
> entry /mntpnt/mail
>
> Jul 11 23:54:01 server3 automount[3564]: lookup(yp): looking up mail
>
> Jul 11 23:54:01 server3 automount[6778]: mt->key set to mail
>
> Jul 11 23:54:01 server3 automount[3564]: lookup(yp): mail ->
> -rw,actimeo=0,vers=3 nis_server:/var/mail
>
> Jul 11 23:54:01 server3 automount[3564]: parse(sun): expanded entry:
> -rw,actimeo=0,vers=3 nis_server:/var/mail
>
> Jul 11 23:54:01 server3 automount[3564]: parse(sun):
> dequote("rw,actimeo=0,vers=3") -> rw,actimeo=0,vers=3
>
> Jul 11 23:54:01 server3 automount[3564]: parse(sun): gathered options:
> 7,rw,intr,soft,vers=3,rw,actimeo=0,vers=3
>
> Jul 11 23:54:01 server3 automount[3564]: parse(sun):
> dequote("nis_server:/var/mail") -> nis_server:/var/mail
>
> Jul 11 23:54:01 server3 automount[3564]: parse(sun): core of entry:
> options=7,rw,intr,soft,vers=3,rw,actimeo=0,vers=3,
> loc=nis_server:/var/mail
>
> Jul 11 23:54:01 server3 automount[3564]: parse(sun): mounting
> root /mntpnt, mountpoint mail, what nis_server:/var/mail, fstype nfs,
> options 7,rw,intr,soft,vers=3,rw,actimeo=0,vers=3
>
> Jul 11 23:54:01 server3 automount[3564]: mount(nfs): root=/mntpnt
> name=mail what=nis_server:/var/mail, fstype=nfs,
> options=7,rw,intr,soft,vers=3,rw,actimeo=0,vers=3
>
> Jul 11 23:54:01 server3 automount[3564]: mount(nfs): nfs
> options="7,rw,intr,soft,vers=3,rw,actimeo=0,vers=3", nosymlink=0 Jul
> 11 23:54:01 server3 automount[3564]: mount(nfs): calling
> mkdir_path /mntpnt/mail
>
> Jul 11 23:54:01 server3 automount[3564]: mount(nfs): calling mount -t
> nfs -s -o 7,rw,intr,soft,vers=3,rw,actimeo=0,vers=3
> nis_server:/var/mail /mntpnt/mail
>
> Jul 11 23:54:01 server3 automount[3564]: >> mount:
> nis_server:/var/mail: can't read superblock
>
> Jul 11 23:54:01 server3 automount[3564]: mount(nfs): nfs: mount
> failure nis_server:/var/mail on /mntpnt/mail
>
> Jul 11 23:54:01 server3 automount[3564]: failed to mount /mntpnt/mail
>
> Jul 11 23:54:01 server3 automount[3564]: umount_multi:
> path=/mntpnt/mail incl=1
>
> Jul 11 23:54:01 server3 automount[6778]: handle_child: got pid 3564,
> sig 0 (0), stat 2
>
> Jul 11 23:54:01 server3 automount[6778]: sig_child: found pending iop
> pid 3564: signalled 0 (sig 0), exit status 2
>
> Jul 11 23:54:01 server3 automount[6778]: update_negative_cache: key:
> mail
>
> Jul 11 23:54:01 server3 automount[6778]: Adding negative cache entry
> for key mail
>
> Jul 11 23:54:01 server3 automount[6778]: Key mail added to negative
> cache
>
> Jul 11 23:54:01 server3 automount[6778]: send_fail: token=247708
>
> Jul 11 23:54:01 server3 automount[6778]: handle_packet: type = 0
>
> Jul 11 23:54:01 server3 automount[6778]: handle_packet_missing: token
> 247709, name mail
>
> Jul 11 23:54:01 server3 automount[6778]: handle_packet_missing: found
> negative cache entry for key.
>
> Jul 11 23:54:01 server3 automount[6778]: send_fail: token=247709
>
> Jul 11 23:54:01 server3 automount[6778]: handle_packet: type = 0
>
> Jul 11 23:54:01 server3 automount[6778]: handle_packet_missing: token
> 247710, name mail
>
> Jul 11 23:54:01 server3 automount[6778]: handle_packet_missing: found
> negative cache entry for key.
>
> Jul 11 23:54:01 server3 automount[6778]: send_fail: token=247710
>
> Jul 11 23:54:01 server3 automount[6778]: handle_packet: type = 0
>
> Jul 11 23:54:01 server3 automount[6778]: handle_packet_missing: token
> 247711, name mail
>
> Jul 11 23:54:01 server3 automount[6778]: handle_packet_missing: found
> negative cache entry for key.
>
> Jul 11 23:54:01 server3 automount[6778]: send_fail: token=247711
>
> /etc/sysconfig/autofs:
>
> # Define custom options in /etc/sysconfig/autofs
>
> # Use LOCALOPTIONS for defining variables, e.g. OSREL
>
> # Use DAEMONOPTIONS to define the unmount timeout
>
> # Define UNDERSCORETODOT as 1 to convert
>
> # auto_home to auto.home and auto_mnt to auto.mnt
>
> # Mount options, e.g. rsize=8192, should go in auto.master or
>
> # the auto_* map entry for a specific mount point
>
> #
>
> LOCALOPTIONS=""
>
> DAEMONOPTIONS="--timeout=60 --debug --set-log-priority 7"
>
> LDAPAUTOMASTER=""
>
> # UNDERSCORETODOT changes auto_home to auto.home and auto_mnt to
> auto.mnt
>
> UNDERSCORETODOT=1
>
> DISABLE_DIRECT=1
>
> # Only source one master map if set to 1. This would mimic Sun
> behaviour.
>
> # The default is 0 to maintain backwards compatibility.
>
> ONE_AUTO_MASTER=0
>
> # List of directories to be ghosted, separated by white space.
>
> GHOSTDIRS=""
>
> # Base DN to use when searching for the master map BASEDN=
>
> # The ldap module was updated to only try the first schema that works
>
> # (instead of trying all 3 autofs schemas for each lookup). It is
>
> # possible, though unlikely, that this could cause problems. If you
> update
>
> # the automounter and find that your ldap schemas doesn't work as it
> did
>
> # previously, try setting this option to 1. Please also report the
> problem,
>
> # as it is likely that your schema is incompatible with autofs v5.
>
> OLD_LDAP_LOOKUP=0
>
> /etc/nsswitch.conf:
>
> #
>
> # /etc/nsswitch.conf
>
> #
>
> # An example Name Service Switch config file. This file should be #
> sorted with the most-used services at the beginning.
>
> #
>
> # The entry '[NOTFOUND=return]' means that the search for an # entry
> should stop if the search in the previous entry turned # up nothing.
> Note that if the search failed due to some other reason # (like no NIS
> server responding) then the search continues with the # next entry.
>
> #
>
> # Legal entries are:
>
> #
>
> # nis or yp Use NIS (NIS version 2), also called
> YP
>
> # dns Use DNS (Domain Name Service)
>
> # files Use the local files
>
> # db Use the local database (.db) files
>
> # compat Use NIS on compat mode
>
> # hesiod Use Hesiod for user lookups
>
> # ldap Use LDAP (only if nss_ldap is
> installed)
>
> # nisplus or nis+ Use NIS+ (NIS version 3), unsupported
>
> # [NOTFOUND=return] Stop searching if not found so far
>
> #
>
> # To use db, put the "db" in front of "files" for entries you want to
> be # looked up first in the databases # # Example:
>
> #passwd: db files ldap nis
>
> #shadow: db files ldap nis
>
> #group: db files ldap nis
>
> passwd: files nis
>
> shadow: files nis
>
> group: files nis
>
> #hosts: db files ldap nis dns
>
> hosts: files nis dns
>
> # Example - obey only what ldap tells us...
>
> #services: ldap [NOTFOUND=return] files
>
> #networks: ldap [NOTFOUND=return] files
>
> #protocols: ldap [NOTFOUND=return] files
>
> #rpc: ldap [NOTFOUND=return] files
>
> #ethers: ldap [NOTFOUND=return] files
>
> bootparams: files
>
> ethers: files
>
> netmasks: files
>
> networks: files
>
> protocols: files nis
>
> rpc: files
>
> services: files nis
>
> netgroup: files nis
>
> publickey: files
>
> automount: files nis
>
> aliases: files
>
> _______________________________________________
> autofs mailing list
> autofs@linux.kernel.org
> http://linux.kernel.org/mailman/listinfo/autofs
_______________________________________________
autofs mailing list
autofs@linux.kernel.org
http://linux.kernel.org/mailman/listinfo/autofs
