Erik, I'm giving a talk on 'network analysis techniques' at the LCA conference later this month (see http://linux.conf.au/). This will basically cover the techniques we have used in the Samba Team for dissecting protocols. My slides will be available after the conference if you happen not to be able to come to Perth :)
One of the most powerful techniques is the 'protocol filter' method. This involves writing a very simple proxy for the protocol that modifies the packets on the way through in a controlled way. This allows you to watch two 'native' implementations of the protocol (for example two win2000 boxes) talk with different protocol options than they would normally negotiate, or selectively kill off portions of packets to see what happens. One sample implementation of such a proxy is smbfilter in the Samba CVS tree, or the little socket proxy at http://samba.org/ftp/unpacked/junkcode/sockspy-spnego.c I wrote sockspy-spnego.c when I was trying to dissect SPNEGO and GSS-SPNEGO. If you look at it all it does it replace the strings GSS-SPNEGO with GSS-XXXXXX and NTLMSSP with NTLMXXX. This allows me to force win2k <-> win2k to talk alternative auth protocols and that allowed me to work out how the auth protocols are structured. Similarly, I have used smbfilter to watch win2k <-> win2k talk SMB but with ACSII instead of UCS2 strings, or to watch what happens when windows tries to negotiate a down-level protocol with itself. It is amazing how much info this can teach you about how the protocol works. Cheers, Tridge
