OK, I'll see about changing my 'tcp/udp' rules to separate
'tcp .... flags S ... keep state keep frags'
'udp ................ keep state keep frags'
rules.
The problem is the system where I showed these messages from is
production so I'll have to play with another system to show it works /
has no ill-effect / proof of concept and then I can apply to these
specific systems later as I have seen it on a few.
It makes sense based on what Darren has stated why it is occasionally
loosing track of the connection state. It seems that when these OOW
packets show up randomly as Darren states but one side or the other will
be able to pick up from these OOW packets and continues.
Perhaps, once we free up some equipment from a platform in migration I
can setup a system to test pushing/pulling files from/to the NetApp and
capture the output using tcpdump. How would I feed such file through
ipfilter to see if OOW packets show up? How long would I need to
capture for this to show up; as it may be random as you say. If I knew
how to pass it through (replay it through) IP Filter then I could see if
OOW packets show up.
My understanding is that the NetApp Filer DataOnTap OS is based on
NetBSD if that helps with understanding the TCP/IP stack. That is,
after reading through these threads about NFS (before posting
originally) I believe someone posted saying that NetBSD has static port
assignments for the service made available/ Hence, it would be easier
to use in a data centre where you may wish to filter on a host, firewall
or perhaps the NetApp filtering itself (using IP Filter perhaps behind
the scenes? ;)
Besides fixing up the rules with 'flags S' on the tcp/udp rules (below);
I would like to also reduce the number of rules that I have. My
understanding is that I really need outbound rules to the NetApp Filer
and then to allowing incoming fragments based on the rules that have
been hit.
For example, only the inbound rules are being hit on the NetApp
(10.206.6.80) trying to send fragmented packets, whereas outbound all
the rules are being hit.
I based my rules on what I saw via 'rpcinfo -p 10.206.6.80' to build my
rules and since there wasn't really any good info around about NFS
traffic flows I put in the 'extra' inbound rules which may not be
needed.
I haven't really seen what happens when the server side (NetApp) reboots
and if it initiates a connection back to the server to say that it is
alive again and to start using it again.
------------------------
# ipfstat -i -h -n | grep 10.206.6.80
0 @17 pass in quick on bge1 proto tcp/udp from 10.206.6.80/32 to
10.206.6.9/32 port = sunrpc keep state keep frags group 101
0 @18 pass in quick on bge1 proto tcp/udp from 10.206.6.80/32 to
10.206.6.9/32 port = nfsd keep state keep frags group 101
0 @19 pass in quick on bge1 proto tcp/udp from 10.206.6.80/32 to
10.206.6.9/32 port 4044 >< 4048 keep state keep frags group 101
0 @20 pass in quick on bge1 proto udp from 10.206.6.80/32 to
10.206.6.9/32 port = 4049 keep state keep frags group 101
4220 @21 pass in quick on bge1 proto udp from 10.206.6.80/32 to
10.206.6.9/32 with frag group 101
#
# ipfstat -o -h -n | grep 10.206.6.80
17 @20 pass out quick on bge1 proto tcp/udp from 10.206.6.9/32 to
10.206.6.80/32 port = sunrpc keep state keep frags group 102
6361 @21 pass out quick on bge1 proto tcp/udp from 10.206.6.9/32 to
10.206.6.80/32 port = nfsd keep state keep frags group 102
9 @22 pass out quick on bge1 proto tcp/udp from 10.206.6.9/32 to
10.206.6.80/32 port 4044 >< 4048 keep state keep frags group 102
5 @23 pass out quick on bge1 proto udp from 10.206.6.9/32 to
10.206.6.80/32 port = 4049 keep state keep frags group 102
603 @24 pass out quick on bge1 proto udp from 10.206.6.9/32 to
10.206.6.80/32 with frag group 102
#
------------------------
-----Original Message-----
From: Darren Reed [mailto:[EMAIL PROTECTED]
Sent: March 13, 2005 1:21 PM
To: Olmsted, Brian
Cc: Joseph Spenner; [email protected]
Subject: Re: rules for NFS with NetApp and OOW packets
>
> Anybody got some insight into this? I only see this once and a while
> and after running 'ipf -Fa -Fs -f ipf.conf' to reload the rules it
> doesn't show up for a while.
>
> I'm allowing in both directions for now but would like to trim it to
> just outbound (or perhaps the list can help me out with defining the
> proper rules for it) but this is what I have now. (below)
>
> I'm doing the 'keep state' and 'keep frags' on my rules but not doing
> the 'flags' option on my packets. Perhaps I should be changing to
> 'flags S' and adding a separate rule for tcp to do this as I likely
> can't do this for 'tcp/udp' rules if I remember correctly.
Right.
The problem with not using "flags S" is that you attempt to allow
tracking the state of a TCP connection without being able to learn
what the window scaling factor should be and thus you're doomed to
getting out-of-window packets if either the NetApp or Solaris has
agreed to use it in the beginning.
I'm not convinced that the OOW packets are an indication of problems
aside from IPFilter (in nearly all circumstances) not quite getting
its stateful tracking right. One day I'll get a tcpdump data file
of a connection that works (without ipfilter in the way) and when
played through ipfilter generates OOW messages - at that point I'll
be able to look closely and see what it's doing wrong and fix it.
But for now, the OOW packets seem to be mostly random and it's
hard to diagnose.
Darren
