What about ifconfig stats? That should show the # of packets arriving at the interface. If you look, the counts are consistent, but just 1/2 of what you expect. It could be that ntop isn't pulling them fast enough from the interface, but that's just a pretty simple select() recvfrom() call pair.
-----Burton -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of R.H.Hoek Sent: Monday, June 06, 2005 3:07 AM To: [email protected] Subject: Re: [Ntop-dev] netflow-plugin misses packets -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Burton Strauss wrote on 4-6-2005 14:25: > How about the stats from the plugin itself? The packetcounts from Ntop netflow are taken from the stats from the plugin itself: Flow Statistics Received Flows Flow Senders 130.89.244.12 [2,251,319 pkts] Number of Packets Received 2,251,319 Number of Packets with Bad Version 0 Number of Packets Processed 2,251,319 Number of Valid Flows Received 65,850,914 Average Number of Flows per Packet 29.2 Number of V1 Flows Received 0 Number of V5 Flows Received 65,850,914 Number of V7 Flows Received 0 Number of V9 Flows Received 0 Discarded Flows Number of Flows with Zero Packet Count 0 Number of Flows with Zero Byte Count 0 Number of Flows with Bad Data 0 Number of Flows with Unknown Template 0 Total Number of Flows Processed 65,850,914 Flowtools reports: - ------------------- Jun 6 09:41:00 localhost flow-capture[27759]: STAT: now=1118043660 startup=1117802440 src_ip=127.0.0.1 dst_ip=127.0.0.1 d_ver=5 pkts=4590376 flows=133781439 lost=0 reset=0 filter_drops=0 Jun 6 09:42:00 localhost flow-capture[27759]: STAT: now=1118043720 startup=1117802440 src_ip=127.0.0.1 dst_ip=127.0.0.1 d_ver=5 pkts=4592106 flows=133831723 lost=0 reset=0 filter_drops=0 Jun 6 09:43:00 localhost flow-capture[27759]: STAT: now=1118043780 startup=1117802440 src_ip=127.0.0.1 dst_ip=127.0.0.1 d_ver=5 pkts=4595919 flows=133942507 lost=0 reset=0 filter_drops=0 - -------------------- Cisco Netflow reports: - -------------------- UTWENTE-router>sh ip flow export Flow export is enabled Exporting flows to x.x.x.x (2055) Exporting using source interface Vlan102 Version 5 flow records *21104941* flows exported in *703498* udp datagrams 0 flows failed due to lack of export packet 0 export packets were sent up to process level 0 export packets were dropped due to no fib 0 export packets were dropped due to adjacency issues 0 export packets were dropped due to fragmentation failures 0 export packets were dropped due to encapsulation fixup failures 0 export packets were dropped enqueuing for the RP 0 export packets were dropped due to IPC rate limiting UTWENTE-router>sh mls nde Netflow Data Export enabled Exporting flows to x.x.x.x (2055) Exporting flows from y.y.y.y (49744) Version: 5 Include Filter not configured Exclude Filter not configured Total Netflow Data Export Packets are: *3892409* packets, 0 no packets, *112837207* records Total Netflow Data Export Send Errors: IPWRITE_NO_FIB = 0 IPWRITE_ADJ_FAILED = 0 IPWRITE_PROCESS = 0 IPWRITE_ENQUEUE_FAILED = 0 IPWRITE_IPC_FAILED = 0 IPWRITE_MTU_FAILED = 0 IPWRITE_ENCAPFIX_FAILED = 0 UTWENTE-router>sho clo 09:42:52.569 MET-DST Mon Jun 6 2005 - -------------------- This measurement runs from Fri 3jun Cisco and Flowtools reports both the same packets/flows: Cisco: 21104941+112837207= 133,942,148 flowtools: 133,942,507 (flows) Cisco: 703498+3892409= 4,595,907 flowtools: 4,595,919 (packets) Netflowplugin: 65,850,914 flows Netflowplugin: 2,251,319 packets N.B. counters are reset and read manualy on 'about' the same time. (within 30 sec) > -----Burton > > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On > Behalf Of R.H.Hoek > Sent: Friday, June 03, 2005 9:15 AM > To: [email protected] > Subject: [Ntop-dev] netflow-plugin misses packets > > Dear Ntoppers, > > Some weeks ago I already mentioned that I experience some problems > with Ntop in combination with Netflow from a Cisco6509 with supervisor2. > Some of you gave me some hints, but nothing couldn't solve my problem > (see below). > > The problem is that the Netflow-plugin misses about 40-60% of the > netflowpackets send by the Cisco6509. Some figures: > > Last night, in a period of about 17 hours the Cisco has sendout > 31,274,087 flows in 1,073,210 packets. > The Netflow-plugin reports it received 16,035,229 valid flows in > 548,345 packets. The plugin reports no discarded flow. > > My first though is that it is not the hardware. I (can) run Ntop on > two different systems, but experience on both the same problem: > System1: > Dual-Xeon 3.2GHz, 5GB RAM, with 4 disks in two hardware SCSI raidsets > (RAID1)(18GB,73GB) and 100M/1G Ethernet interface with a 100Mbps > networkconecction. On this system the OS is SuSE-UnitedLinux 8. (2.4 > kernel) System2(originally meant for probe): > Dual-Xeon 3.2GHz, 3GB RAM, with 2 disks in a hardware SCSI raidsets > (RAID1)(18GB) and 100M/1G Ethernet interface with a 100Mbps > networkconecction. On this system the OS is Debian Sarge 2.6 kernel. > > I have done testing with Ntop 3.1 and the latest CVS versions -> same > results. With tcpdump I did some counting. The conclusion is that the > packets send by Cisco6509 are received on the system Ntop is running > on, but are discarded/missed by the Netflow-plugin > > Answers on some questions: > 1) > Ntop is running with --interface-none, --track-local-host and the > Netflow pseudo-nic is selected. > 2) > Even, with RRD-plugin is set to low-detail, and data to dump is 'none' > does not solve the problem > 3) > for testing I have incremented MAX_SUBNET_HOSTS to 8192/16384 in > globals-defines.h. > Load avg 0.14 (98% idle) > The memory usage is about 1.8GB (3.3 GB free) after 1 hour Ntop is started. > 4) > The load graph and (traffic)protocol-distribution should display the > total network load of the Cisco6509 when the pseudo-nic is selected. I > have seen this on a Ntop system with low netflowtraffic. > 5) > In a test config the Cisco-Netflow is first send to system2 and > redistributed with flow-fanout to system1 running Ntop. On system2 I > do flow-capture and flow-stat. When I compare this output with > Ntop-output, the differences are very large as of 5min net workload > and protocol distribution. > 6) > The average netflowstream is about 20 packets/s. But I have seen > bursts of 900 packets/s > > > > I think that the last point is possibly the problem. With these bursts > the netflowplugin inputbuffer is overloaded ? > Is there a way to tune this buffer? Or does anybody else has some hints? > > > -- > > Groeten, > > Roel H.Hoek, SeniorNetworkmanager > Dienst Informatietechnologie, Bibliotheek en Educatie (ITBE) > Universiteit Twente, Postbus 217, 7500 AE Enschede kmr SP 422, > telefoon: 053 - 489 4598, fax: 053 - 489 2383 > e-mail: [EMAIL PROTECTED] http://www.utwente.nl/itbe > _______________________________________________ Ntop-dev mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-dev _______________________________________________ Ntop-dev mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-dev - -- Groeten, Roel H.Hoek, SeniorNetworkmanager Dienst Informatietechnologie, Bibliotheek en Educatie (ITBE) Universiteit Twente, Postbus 217, 7500 AE Enschede kmr SP 422, telefoon: 053 - 489 4598, fax: 053 - 489 2383 e-mail: [EMAIL PROTECTED] http://www.utwente.nl/itbe -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (MingW32) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCpAQJJwlRSGnYBcYRAva7AKC7obriggq0ebIyFhup65bKyh5lSACfZfJI Xv97uZooRktzyNcr1eVYPxg= =DSvF -----END PGP SIGNATURE----- _______________________________________________ Ntop-dev mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-dev _______________________________________________ Ntop-dev mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-dev
