RE: [Ntop-dev] Spreading ntop's RRD dumps to spread system load"So the hardware is still bottleneck here"
What's your hardware? Basically your #s suck. My configuration is about as slow as you can get - normal IDE - 5400RPM 60GB drive, running UDMA33 (Yeah, I need to fix that). # dmesg | grep hd hde: 117231408 sectors (60022 MB) w/2048KiB Cache, CHS=116301/16/63, UDMA(33) hdh: 19807200 sectors (10141 MB) w/466KiB Cache, CHS=19650/16/63, UDMA(33) hde: [PTBL] [7297/255/63] hde1 hde2 hde4 < hde5 hde6 hde7 > hdh: hdh1 hdd: ATAPI 40X DVD-ROM drive, 512kB Cache But even so, on my dual P3-1000, I'm seeing this: Jun 16 11:50:20 tigger ntop[9619]: RRD: 298 RRDs updated in 0.171831 seconds (0.000577 per) to Jun 16 15:51:20 tigger ntop[9619]: RRD: 330 RRDs updated in 0.046533 seconds (0.000141 per) You're about 1/6th to 1/7th of my speed. But even at my best #s, you're looking at close to a minute for 380K RRDs. The only thing I can suggest is to: (A) reduce the frequency of RRD updates. Instead of recording data every 5m, go for 30m. (B) Record less detail in the RRDs - switch to Low or Medium detail. (C) Ditch the 1980s hardware. Ditching (hardware) RAID1 probably won't save you anything - software RAID1 could be another thing altogether. The beauty of the patch is that under Linux with copy-on-write, you 1) don't lock packetProcess() and 2) are correctly recording the values as of the fork() call no matter how long it takes. -----Burton PS: Must be nice to have 11GB to 'spend' on RRDs. I know, I know, IDE disk space is cheap (says he with >1/2TB at home, mostly empty)... -----Original Message----- From: Kouprie, Robbert [mailto:[EMAIL PROTECTED] Sent: Wednesday, June 16, 2004 2:36 PM To: Burton M. Strauss III Cc: [EMAIL PROTECTED] Subject: RE: [Ntop-dev] Spreading ntop's RRD dumps to spread system load Hi Burton, Time for some results. (Note: I didn't put your last modification in as it was just cosmetic). Anyway, it looks like the patch is doing its job ok. Ntop indeed stays responsive while updating RRDs in contrary to the situation when there is no forking. However, in my situation, it is still not usable: Jun 16 20:33:34 ntop ntop[1886]: [MSGID0797160] RRD: 37183 updated in 84.789 seconds Jun 16 20:33:34 ntop ntop[1873]: [MSGID8475868] RRD: fork()ed child No child processes(10) Jun 16 20:47:52 ntop ntop[1933]: [MSGID0797160] RRD: 269383 updated in 642.530 seconds Jun 16 20:47:52 ntop ntop[1873]: [MSGID8475868] RRD: fork()ed child No child processes(10) Jun 16 21:11:31 ntop ntop[1962]: [MSGID0797160] RRD: 355828 updated in 1162.019 seconds Jun 16 21:11:32 ntop ntop[1873]: [MSGID8475868] RRD: fork()ed child No child processes(10) Jun 16 21:32:57 ntop ntop[2006]: [MSGID0797160] RRD: 321247 updated in 1247.692 seconds Jun 16 21:32:57 ntop ntop[1873]: [MSGID8475868] RRD: fork()ed child No child processes(10) As you can see, updating all RRDs in my situation stil takes about 20 minutes. So the hardware is still bottleneck here. (Although this is a setup with the /var directory on a RAID 1+0 array on 4 Ultra160 SCSI disks). I might do some more tweaking, or ditch the RAID1. But a *real* setup would send the data to a seperate database-server anyway. Still, I'd vote for inclusion of the patch since it makes ntop more responsive while doing RRD updates (and of course the forking can be disabled in the web interface). Regards, Robbert <snip/>
_______________________________________________ Ntop-dev mailing list [EMAIL PROTECTED] http://listgateway.unipi.it/mailman/listinfo/ntop-dev
