Alex Still wrote:

We're using tcpdump to try to diagnose an NFS performance problem.
We're seeing a lot of these :
16:01:19.890794 IP nfs_server.nfs > client_46.1205729087: reply ERR 1448
16:01:19.890831 IP nfs_server.nfs > client_46.3664003007: reply ERR 1448

That seems to be an RPC error,

It seems to be one, but it isn't necessarily one.

The code to print that printed "reply OK {length}" if the field in the packet that, *IF* it contains the beginning of an ONC RPC reply, would be the reply code is 0, and "reply ERR {length}" if it's not zero.

However, not all packets from an ONC RPC server (such as an NFS server) necessarily contain the beginning of an ONC RPC reply, as a reply (or request) can require more than one link-layer packet - for example, an NFS READ reply that returns more bytes that fit in one Ethernet packet will, when sent over Ethernet, take more than one Ethernet packet.

If the request or reply is coming over UDP, that's handled by IP fragmentation; the first packet of the reply will be the first fragment, i.e. it will have a fragment offset of 0, and can be identified by looking at the IP header. tcpdump only attempts to dissect the first fragment as anything other than IP (for one thing, subsequent fragments don't have UDP or TCP port numbers, so, unless it keeps track of the fragments, it has no idea what port number any fragment other than the first one was sent from or to), so it won't try to dissect the subsequent fragments as if they contained the beginning of an ONC RPC request or reply.

If, however, it's coming over TCP, that's handled by the RPC implementation, as TCP merely provides a byte stream, and has no idea what the message boundaries are for the protocols that run atop it.

Therefore, the current tcpdump code to dissect RPC requests and replies doesn't identify TCP segments that are in the middle of an ONC RPC request or reply, and tries to dissect them as if they were at the beginning of the request or reply.

This can cause packets to be mis-dissected; the apparent RPC errors you're seeing are probably examples of that.

That part of the code
has changed in tcpdump-3.9.7, which we installed, and now gives :

16:01:19.890794 IP nfs_server.nfs > client_46.1205729087: reply Unknown rpc
response code=3128327487 1448
16:01:19.890831 IP nfs_server.nfs > client_46.3664003007: reply Unknown rpc
response code=910276799 1448

It now prints the value of the field that, if the packet really *is* an RPC error, would be the error code (the code is derived from from NetBSD's version of tcpdump). If the packet *isn't* an RPC error, the value at the location that's interpreted as a reply code has a very good chance of being bogus, as was the case here.

I don't understand what's happening here. Is it something wrong with our NFS


or I'm misunderstanding the tcpdump output ?


Wireshark/TShark should do a better job of this, for two reasons:

1) it does reassembly of ONC RPC requests and replies over TCP, as well as handling TCP segments with multiple requests or replies;

2) it does some heuristics to check whether a packet *is* an ONC RPC request or reply.

1) would probably be a major undertaking to do in tcpdump, but 2) could be done:

for requests, check whether the RPC version field (not the NFS version) has the value 2 (neither Sun nor anybody else have released any version of ONC RPC other than 2), and check whether the program number in the request is the program number for NFS (note that Sun also run another RPC protocol on port 2049 - a protocol to allow access to POSIX ACLs over NFS - which will be misdissected if interpreted as NFS);

for replies, check whether the transaction ID of the reply matches the transaction ID of a request we've already seen (tcpdump already does that matching for replies without errors, as it needs to know what type of request the reply is for).

I'll see whether that could be done.
This is the tcpdump-workers list.
Visit to unsubscribe.

Reply via email to