Ok, a server was already hung when I got to work today.
**********************************
x4500-04: NFS Server, Sol 10 5/08
Server IP (real) 172.20.12.226 netmask ffffff00
NFS IP (alias) 172.20.12.227 netmask ffffff00
x4500-04:~# netstat -in ; netstat -rn
Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis
Queue
lo0 8232 127.0.0.0 127.0.0.1 1411 0 1411 0 0
0
e1000g0 1500 172.20.12.0 172.20.12.226 2762497849 0 1789082372 0
0 0
e1000g1 1500 172.20.19.0 172.20.19.226 96059758 0 52485074 0
0 0
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ---------- ---------
default 172.20.12.1 UG 1 20456
172.20.12.0 172.20.12.226 U 1 45968 e1000g0
172.20.12.0 172.20.12.227 U 1 0 e1000g0:1
172.20.19.0 172.20.19.226 U 1 1662 e1000g1
224.0.0.0 172.20.12.226 U 1 0 e1000g0
127.0.0.1 127.0.0.1 UH 5 316 lo0
**********************************
NFS client: Sol 10 5/08
Client IP 172.20.12.6 netmask ffffff00
# netstat -in ; netstat -rn
Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis
Queue
lo0 8232 127.0.0.0 127.0.0.1 2175 0 2175 0 0
0
e1000g0 1500 172.20.12.0 172.20.12.6 43315618 0 41987515 0
0 0
e1000g1 1500 172.20.11.0 172.20.11.6 19673254 0 13928826 0
0 0
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ---------- ---------
default 172.20.11.4 UG 1 52386
10.0.0.0 172.20.12.1 UG 1 0
172.16.0.0 172.20.12.1 UG 1 193
172.20.11.0 172.20.11.6 U 1 2406 e1000g1
172.20.12.0 172.20.12.6 U 1 3163 e1000g0
192.168.0.0 172.20.12.1 UG 1 120
224.0.0.0 172.20.12.6 U 1 0 e1000g0
127.0.0.1 127.0.0.1 UH 4 2046 lo0
*********************************
Snoop running on NFS Client 172.20.12.6 attempting to (re)mount volume
with TCP:
# snoop -r host 172.20.12.227 or host 172.20.12.226 &
# mount /export/www
172.20.12.6 -> 172.20.12.227 PORTMAP C GETPORT prog=100005 (MOUNT)
vers=3 proto=UDP
172.20.12.226 -> 172.20.12.6 PORTMAP R GETPORT port=39049
172.20.12.6 -> 172.20.12.227 MOUNT3 C Null
172.20.12.226 -> 172.20.12.6 MOUNT3 R Null
172.20.12.6 -> 172.20.12.227 MOUNT3 C Mount /export/www
172.20.12.226 -> 172.20.12.6 MOUNT3 R Mount OK FH=D402 Auth=unix
172.20.12.6 -> 172.20.12.227 PORTMAP C GETPORT prog=100003 (NFS)
vers=3 proto=TCP
172.20.12.226 -> 172.20.12.6 PORTMAP R GETPORT port=2049
172.20.12.6 -> 172.20.12.227 TCP D=2049 S=63800 Syn Seq=788700586
Len=0 Win=49640 Options=<mss 1460,nop,wscale 0,nop,nop,sackOK>
172.20.12.227 -> 172.20.12.6 TCP D=63800 S=2049 Syn Ack=788700587
Seq=3596066619 Len=0 Win=49640 Options=<mss 1460,nop,wscale
0,nop,nop,sackOK>
172.20.12.6 -> 172.20.12.227 TCP D=2049 S=63800 Ack=3596066620
Seq=788700587 Len=0 Win=49640
172.20.12.6 -> 172.20.12.227 NFS C NULL3
172.20.12.227 -> 172.20.12.6 TCP D=63800 S=2049 Ack=788700707
Seq=3596066620 Len=0 Win=49520
172.20.12.227 -> 172.20.12.6 NFS R NULL3
172.20.12.6 -> 172.20.12.227 TCP D=2049 S=63800 Ack=3596066648
Seq=788700707 Len=0 Win=49640
172.20.12.6 -> 172.20.12.227 TCP D=2049 S=63800 Fin Ack=3596066648
Seq=788700707 Len=0 Win=49640
172.20.12.227 -> 172.20.12.6 TCP D=63800 S=2049 Ack=788700708
Seq=3596066648 Len=0 Win=49640
172.20.12.227 -> 172.20.12.6 TCP D=63800 S=2049 Fin Ack=788700708
Seq=3596066648 Len=0 Win=49640
172.20.12.6 -> 172.20.12.227 TCP D=2049 S=63800 Ack=3596066649
Seq=788700708 Len=0 Win=49640
172.20.12.6 -> 172.20.12.227 TCP D=2049 S=664 Syn Seq=2946510831 Len=0
Win=49640 Options=<mss 1460,nop,wscale 0,nop,nop,sackOK>
172.20.12.6 -> 172.20.12.227 TCP D=2049 S=664 Syn Seq=2946510831 Len=0
Win=49640 Options=<mss 1460,nop,wscale 0,nop,nop,sackOK>
172.20.12.6 -> 172.20.12.227 TCP D=2049 S=664 Syn Seq=2946510831 Len=0
Win=49640 Options=<mss 1460,nop,wscale 0,nop,nop,sackOK>
Interesting, looks like x4500-04 is replying with the wrong IP.
Packet capture on x4500-04:
# snoop -r host 172.20.12.6
Using device /dev/e1000g0 (promiscuous mode)
172.20.12.6 -> 172.20.12.227 TCP D=2049 S=664 Rst Ack=0 Seq=2924968134
Len=0 Win=49640
172.20.12.227 -> 172.20.12.6 TCP D=664 S=2049 Rst Win=49640
172.20.12.6 -> 172.20.12.227 PORTMAP C GETPORT prog=100005 (MOUNT)
vers=3 proto=UDP
172.20.12.226 -> 172.20.12.6 PORTMAP R GETPORT port=39049
172.20.12.6 -> 172.20.12.227 MOUNT3 C Null
172.20.12.226 -> 172.20.12.6 MOUNT3 R Null
172.20.12.6 -> 172.20.12.227 MOUNT3 C Mount /export/www
172.20.12.226 -> 172.20.12.6 MOUNT3 R Mount OK FH=D402 Auth=unix
172.20.12.6 -> 172.20.12.227 PORTMAP C GETPORT prog=100003 (NFS)
vers=3 proto=TCP
172.20.12.226 -> 172.20.12.6 PORTMAP R GETPORT port=2049
172.20.12.6 -> 172.20.12.227 TCP D=2049 S=63800 Syn Seq=788700586
Len=0 Win=49640 Options=<mss 1460,nop,wscale 0,nop,nop,sackOK>
172.20.12.227 -> 172.20.12.6 TCP D=63800 S=2049 Syn Ack=788700587
Seq=3596066619 Len=0 Win=49640 Options=<mss 1460,nop,wscale
0,nop,nop,sackOK>
172.20.12.6 -> 172.20.12.227 TCP D=2049 S=63800 Ack=3596066620
Seq=788700587 Len=0 Win=49640
172.20.12.6 -> 172.20.12.227 NFS C NULL3
172.20.12.227 -> 172.20.12.6 TCP D=63800 S=2049 Ack=788700707
Seq=3596066620 Len=0 Win=49520
172.20.12.227 -> 172.20.12.6 NFS R NULL3
172.20.12.6 -> 172.20.12.227 TCP D=2049 S=63800 Ack=3596066648
Seq=788700707 Len=0 Win=49640
172.20.12.6 -> 172.20.12.227 TCP D=2049 S=63800 Fin Ack=3596066648
Seq=788700707 Len=0 Win=49640
172.20.12.227 -> 172.20.12.6 TCP D=63800 S=2049 Ack=788700708
Seq=3596066648 Len=0 Win=49640
172.20.12.227 -> 172.20.12.6 TCP D=63800 S=2049 Fin Ack=788700708
Seq=3596066648 Len=0 Win=49640
172.20.12.6 -> 172.20.12.227 TCP D=2049 S=63800 Ack=3596066649
Seq=788700708 Len=0 Win=49640
172.20.12.6 -> 172.20.12.227 TCP D=2049 S=664 Syn Seq=2946510831 Len=0
Win=49640 Options=<mss 1460,nop,wscale 0,nop,nop,sackOK>
172.20.12.227 -> 172.20.12.6 TCP D=664 S=2049 Ack=2876021783
Seq=3544124023 Len=0 Win=49640
172.20.12.6 -> 172.20.12.227 TCP D=2049 S=664 Syn Seq=2946510831 Len=0
Win=49640 Options=<mss 1460,nop,wscale 0,nop,nop,sackOK>
172.20.12.227 -> 172.20.12.6 TCP D=664 S=2049 Ack=2876021783
Seq=3544124023 Len=0 Win=49640
172.20.12.6 -> 172.20.12.227 TCP D=2049 S=664 Syn Seq=2946510831 Len=0
Win=49640 Options=<mss 1460,nop,wscale 0,nop,nop,sackOK>
172.20.12.227 -> 172.20.12.6 TCP D=664 S=2049 Ack=2876021783
Seq=3544124023 Len=0 Win=49640
*** Attempting mount using the real IP instead of the alias:
# mount -o vers=3,hard,intr,quota 172.20.12.226:/export/www /export/www
ssl01:/# 172.20.12.6 -> 172.20.12.226 PORTMAP C GETPORT prog=100005
(MOUNT) vers=3 proto=UDP
172.20.12.226 -> 172.20.12.6 PORTMAP R GETPORT port=39049
172.20.12.6 -> 172.20.12.226 MOUNT3 C Null
172.20.12.226 -> 172.20.12.6 MOUNT3 R Null
172.20.12.6 -> 172.20.12.226 MOUNT3 C Mount /export/www
172.20.12.226 -> 172.20.12.6 MOUNT3 R Mount OK FH=D402 Auth=unix
172.20.12.6 -> 172.20.12.226 PORTMAP C GETPORT prog=100003 (NFS)
vers=3 proto=TCP
172.20.12.226 -> 172.20.12.6 PORTMAP R GETPORT port=2049
172.20.12.6 -> 172.20.12.226 TCP D=2049 S=63802 Syn Seq=88322761 Len=0
Win=49640 Options=<mss 1460,nop,wscale 0,nop,nop,sackOK>
172.20.12.226 -> 172.20.12.6 TCP D=63802 S=2049 Syn Ack=88322762
Seq=3700270536 Len=0 Win=49640 Options=<mss 1460,nop,wscale
0,nop,nop,sackOK>
172.20.12.6 -> 172.20.12.226 TCP D=2049 S=63802 Ack=3700270537
Seq=88322762 Len=0 Win=49640
172.20.12.6 -> 172.20.12.226 NFS C NULL3
172.20.12.226 -> 172.20.12.6 TCP D=63802 S=2049 Ack=88322882
Seq=3700270537 Len=0 Win=49520
172.20.12.226 -> 172.20.12.6 NFS R NULL3
172.20.12.6 -> 172.20.12.226 TCP D=2049 S=63802 Ack=3700270565
Seq=88322882 Len=0 Win=49640
172.20.12.6 -> 172.20.12.226 TCP D=2049 S=63802 Fin Ack=3700270565
Seq=88322882 Len=0 Win=49640
172.20.12.226 -> 172.20.12.6 TCP D=63802 S=2049 Ack=88322883
Seq=3700270565 Len=0 Win=49640
172.20.12.226 -> 172.20.12.6 TCP D=63802 S=2049 Fin Ack=88322883
Seq=3700270565 Len=0 Win=49640
172.20.12.6 -> 172.20.12.226 TCP D=2049 S=63802 Ack=3700270566
Seq=88322883 Len=0 Win=49640
172.20.12.6 -> 172.20.12.227 TCP D=2049 S=664 Rst Ack=0 Seq=3056789346
Len=0 Win=49640
172.20.12.6 -> 172.20.12.226 TCP D=2049 S=620 Syn Seq=1932893789 Len=0
Win=49640 Options=<mss 1460,nop,wscale 0,nop,nop,sackOK>
172.20.12.226 -> 172.20.12.6 TCP D=620 S=2049 Syn Ack=1932893790
Seq=3700480396 Len=0 Win=49640 Options=<mss 1460,nop,wscale
0,nop,nop,sackOK>
172.20.12.6 -> 172.20.12.226 TCP D=2049 S=620 Ack=3700480397
Seq=1932893790 Len=0 Win=49640
172.20.12.6 -> 172.20.12.226 NFS C FSINFO3 FH=D402
172.20.12.226 -> 172.20.12.6 TCP D=620 S=2049 Ack=1932893946
Seq=3700480397 Len=0 Win=49640
172.20.12.226 -> 172.20.12.6 NFS R FSINFO3 OK
172.20.12.6 -> 172.20.12.226 TCP D=2049 S=620 Ack=3700480565
Seq=1932893946 Len=0 Win=49640
172.20.12.6 -> 172.20.12.226 NFS C FSSTAT3 FH=D402
172.20.12.226 -> 172.20.12.6 TCP D=620 S=2049 Ack=1932894102
Seq=3700480565 Len=0 Win=49640
172.20.12.226 -> 172.20.12.6 NFS R FSSTAT3 OK
172.20.12.6 -> 172.20.12.226 TCP D=2049 S=620 Ack=3700480737
Seq=1932894102 Len=0 Win=49640
Which works without issue. So it is not an NFS problem, it seems to be
related to alias IPs.
Do you know a way around this? Or perhaps you can suggest a place where
I can go to ask. As a quick solution we will just forgo the Alias IP and
mount directly on the "real" IP. Why can I change protocol (TCP->UDP and
vv) to get around it, why can I reboot the NFS client as well. Did we
create the aliases wrong?
I apologise for the noise in NFS discussion list.
Lund
Dai Ngo wrote:
> The problem seems to be on the TCP connection between the client and the
> nfsd on
> the server. The portmap and mount requests used UDP and they went OK.
>
> There are a number TCP RST packets sent from both the client and server,
> this indicated
> there might be problem with packets lost causing both sides to be out of
> sync.
>
> Looks like the server has 2 NICs on the same subnet, 172.20.12.221 and
> 172.20.12.220.
> Have you tried disable 172.20.12.220 and just use 172.20.12.221 to see
> if it helps.
> What the output of the 'netstat -in' and 'netstat -rn' on the server and
> the client look like?
>
> By the way, where were the packets captured from? on the server or the
> client. It's more
> useful if you can capture the packets on both sides and attach the raw
> capture files so
> they can be compared and examined in more details.
--
Jorgen Lundman | <lundman at lundman.net>
Unix Administrator | +81 (0)3 -5456-2687 ext 1017 (work)
Shibuya-ku, Tokyo | +81 (0)90-5578-8500 (cell)
Japan | +81 (0)3 -3375-1767 (home)
