I was going to respond inline, but it just got too complex... so here goes...
The FAQ has been posted a few times, but I don't try to keep the hmtl version up to date with every little change. You can usually find something under the documentation link at http://www.ntopsupport.com, but the best reference is the actual text file. (You can always create your own html-ized version from the source via "make faq.html"). "So while libpcap may be unable to decipher the HDLC headers, it can still make a pretty accurate guess about what the layer-3 protocol headers contain" - no, libpcap doesn't interpret anything, it's just giving you an array of bytes. From the DLT_ type of the interface, you have enough information to begin the decoding process (i.e. DLT_EN10MB means you have an ethernet frame, which means that... "sniffers do correctly (?) perform inspection on what is happening on the TCP/IP level" - yeah, they're coded for it. "What is meaningful to me is the ability to determine src/dst for IP and TCP/UDP" - yes, IF it's encapsulated tcp/ip in that hdlc frame. (And just because it is, doesn't mean the previous frame was or the next one or that thye all are). See, the secret to handling protocols other than Ethernet is two fold. One is that the encapsulated (layer 3) packet is in fact tcp/ip. Sort of obviously if it's something else entirely, it's going to be silly to interpret it as tcp/ip. The second item is a little more subtle. Which is that there's no common format for layer 2 packets (frames). So handling them AT ALL, requires coding to pick out the encapsulated packet. tcpdump can do that, but in doing so, you lose the access to the raw frame data (-e switch). And it has to be coded for - everytime somebody comes out with some new wonderful protocol, the sniffers need to be updated (google for all the whining about 'tcpdump doesn't support 802.11b for example'). The code to look at in ntop is processPacket() in pbuf.c -- if you look, you'll see that we use the DLT type to figure out the packet framing: switch(myGlobals.device[deviceId].datalink) { case DLT_FDDI: But there's also this telling comment, just above: /* ethernet assumed */ If you look at the various cases (e.g. $ grep ' case DLT_' pbuf.c), you'll see that except for TokenRing, they all make that assumption that the ultimate packet is tcp/ip, so they call processIpPkt(). "Ntop just needs to know that the first few bytes of frames should be ignored" That's not quite right - see the prior discussion about libpcap and raw frames. As I said, if it's encapsulating tcp/ip, it should be doable. So it *might* be better to say that 'ntop needs to ignore the first few bytes of the raw frames when tcp/ip is encapsulated in HDLC'. And frankly, that shouldn't be that hard to do - if you can uncover the magic settings and offsets. The FDDI code is a good example - it analyzes enough of the FDDI frame to get at the underlying encapsulation protocol, figure out if it's tcp/ip and then forwards it to processIpPkt(): case DLT_FDDI: fddip = (struct fddi_header *)p; length -= FDDI_HDRLEN; p += FDDI_HDRLEN; caplen -= FDDI_HDRLEN; extract_fddi_addrs(fddip, (char *)ESRC(&ehdr), (char *)EDST(&ehdr)); ether_src = (u_char*)ESRC(&ehdr), ether_dst = (u_char*)EDST(&ehdr); case DLT_FDDI: fddip = (struct fddi_header *)p; length -= FDDI_HDRLEN; p += FDDI_HDRLEN; caplen -= FDDI_HDRLEN; extract_fddi_addrs(fddip, (char *)ESRC(&ehdr), (char *)EDST(&ehdr)); ether_src = (u_char*)ESRC(&ehdr), ether_dst = (u_char*)EDST(&ehdr); if((fddip->fc & CONST_FDDIFC_CLFF) == CONST_FDDIFC_CONST_LLC_ASYNC) { struct llc llc; /* Info on SNAP/LLC: http://www.erg.abdn.ac.uk/users/gorry/course/lan-pages/llc.html http://www.ece.wpi.edu/courses/ee535/hwk96/hwk3cd96/li/li.html http://www.ece.wpi.edu/courses/ee535/hwk96/hwk3cd96/li/li.html */ memcpy((char *)&llc, (char *)p, min(caplen, sizeof(llc))); if(llc.ssap == LLCSAP_SNAP && llc.dsap == LLCSAP_SNAP && llc.ctl.snap.snap_ui == CONST_LLC_UI) { if(caplen >= sizeof(llc)) { caplen -= sizeof(llc); length -= sizeof(llc); p += sizeof(llc); >> if(EXTRACT_16BITS(&llc.ctl.snap_ether.snap_ethertype[0]) == ETHERTYPE_IP) { /* encapsulated IP packet */ >> processIpPkt(p, h, length, ether_src, ether_dst, actualDeviceId, vlanId); /* Patch below courtesy of Fabrice Bellet <[EMAIL PROTECTED]> */ #ifdef CFG_MULTITHREADED releaseMutex(&myGlobals.hostsHashMutex); #endif return; } } } } break; "While I'm not against sponsoring development"... Well, unless somebody is willing to fund my time, I can't afford to play with ntop, vs. finding a paying job... it's that simple. "running 'ntop -f' with a traffic dump from running 'tcpdump -w'"... um... It may be that the ntop code isn't expecting an HDLC frame in from the -r interface. I'll bet that if you enable ntop's NOISY mode, the DLT type is EN10MB, i.e. Ethernet, and the pointers/offsets are accordingly totally bogus. If you want to look at that, the code is in initialize.c, initDeviceDatalink(). "Better yet, I will learn how to use Ntop's "probe" functionality and direct it towards Ntop running on a dedicated "display" machine." That sounds reasonable - using the netFlow module means that all the handling of packets is in ONE system, and only small amounts of data get sent recording the basics of the flows. If you can monitor INSIDE the HDLC encapsulation, (just treat it as a tunnel you don't look inside of), then you should see normal tcp/ip that ntop eats quite happily. -----Burton -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Adam Henry Sent: Tuesday, September 09, 2003 4:51 PM To: [EMAIL PROTECTED] Subject: Re: [Ntop] Warning: arptype 15 not supported by libpcap - falling back to cooked socket Mr. Strauss, Thank you for taking the time for responding to my original post. On Fri, Jun 06, 2003 at 12:19:34PM -0500, Burton M. Strauss III wrote: > 1. http://www.gmane.org - it's in the docs/FAQ for ntop. Couldn't find a link to the FAQ under the above URL, so I grabbed it from the tarball. Thank you for reminding me to read it--please forgive me for not doing so before posting here to begin with. > 2. 'Cooked' sockets are unique to Linux. They're not quite raw, but they're > not quite a real protocol level socket library, they're something in > between. Don't let the terminology scare you. > > It doesn't sound like an ntop problem, it's sounds like a problem with the > underlying packet capture library (libpcap) that doesn't support the > arptype. But since you've cut & pasted only part of the log message, it's > hard to tell. So while libpcap may be unable to decipher the HDLC headers, it can still make a pretty accurate guess about what the layer-3 protocol headers contain. While sniffers which rely on the libpcap library report the same error as Ntop does, these sniffers do correctly (?) perform inspection on what is happening on the TCP/IP level. What is meaningful to me is the ability to determine src/dst for IP and TCP/UDP. The ability to report traffic usage would also be needed. If this can all be done, using Ntop, then further discussions about ARP Type support and HDLC headers may not be necessary. Would you agree? > If you google for arptype "not supported by libpcap" -- you'll see some > entries referring to using an old libpcap. So updating might help. > > If you want to pursue this further, use the ntop problem report form - you > should be able to get ntop up long enough to generate the skeleton if you > don't point ntop at the problem interface. > > However - > > There's no support in ntop right now for HDLC. And you need to understand > how you're using 'HDLC'. HDLC covers a lot of ground. HDLC operates at the > same network layer as TCP/IP, it's a complete layer 2 communications > protocol. > > See, for example, http://www2.rad.com/networks/1994/hdlc/hdlc.htm and > http://members.tripod.com/~vkalra/hdlc.html > > While it's conceptually feasible to extend ntop to support another layer 2 > and higher protocol, it would be very difficult to coerce into ntop's > Ethernet/TCPIP model of MAC and IP addresses. Difficult, but not > impossible. Precisely what reports and such would be available would have > to be determined. > > HDLC can also be used to encapsulate IP traffic for point to point links. > If that's the situation, then (assuming you can update to a libpcap that > supports the interface), we could train ntop to ignore the HDLC headers and > just process the IP portion (like we do today with PPP and PPPoE). I think my previous response was strongly influenced by this comment of yours. Ntop just needs to know that the first few bytes of frames should be ignored (it seems that tethereal and tcpdump may do this already). > HDLC is a dying protocol - so if you're not willing to sponsor development, > I doubt anyone else will. I'll warn you, that either way, this will entail > a fair amount of work, just to get to the point of giving you a cost > quotation. If you're interested in sponsoring the development, contact me > off list. While I'm not against sponsoring development, I need to make sure we really need such ability. If it is possible to obtain the information I require using Ntop (exporting a flow from one unit to be analyzed on another dedicated machine), then Ntop would be the answer. My difficulty getting this to work may be due to a SIGSEGV received from running 'ntop -f' with a traffic dump from running 'tcpdump -w' on a HDLC interface of another machine. Running 'tcpdump -r' allows me to inspect packets. If what I am attempting to do isn't too outrageous, then I will start another thread with the results of a GDB backtrace. Better yet, I will learn how to use Ntop's "probe" functionality and direct it towards Ntop running on a dedicated "display" machine. I have a lot to learn and test. Thank you for allocating your time addressing my comments. Sincerely, hank _______________________________________________ Ntop mailing list [EMAIL PROTECTED] http://listgateway.unipi.it/mailman/listinfo/ntop _______________________________________________ Ntop mailing list [EMAIL PROTECTED] http://listgateway.unipi.it/mailman/listinfo/ntop
