Hi Adrian, Thanks for your reply. It doesn't look like this is the problem in this case. The log indicates that the profiles are being updated in under 2 minutes:
Jul 30 15:55:15 ntbs nfsen[26412]: Run periodic at Thu Jul 30 15:55:00 2009 Jul 30 15:55:15 ntbs nfsen[26412]: Prepare profiling './site1' (... and so on for the rest of the profiles) Jul 30 15:55:15 ntbs nfsen[26412]: 270 channels/alerts to profile Jul 30 15:55:15 ntbs nfsen[14739]: profile opts: .#site3#6#in#site3 for profiler 0 Jul 30 15:55:15 ntbs nfsen[14743]: profile opts: .#site1#6#in#site1 for profiler 2 Jul 30 15:55:15 ntbs nfsen[14740]: profile opts: .#site1#6#out#site1 for profiler 1 Jul 30 15:55:15 ntbs nfsen[14744]: profile opts: .#site4#6#out#site4 for profiler 3 (... and so on for the rest of the shadow profile channels) Jul 30 15:55:15 ntbs nfsen[14744]: profiler 3 finished Jul 30 15:55:15 ntbs nfsen[14747]: profiler 4 finished Jul 30 15:55:17 ntbs nfsen[14743]: profiler 2 finished Jul 30 15:55:17 ntbs nfsen[14740]: profiler 1 finished Jul 30 15:55:17 ntbs nfsen[14739]: profiler 0 finished Jul 30 15:55:17 ntbs nfsen[14749]: profiler 5 finished Jul 30 15:55:17 ntbs nfsen[26412]: Update profile site1 in group . (... and so on for the rest of the profiles until live) Jul 30 15:55:57 ntbs nfsen[26412]: Update profile live in group . Jul 30 15:55:57 ntbs nfsen[26412]: Error reading channel stat information. Missing key 'first' Jul 30 15:55:58 ntbs last message repeated 137 times Jul 30 15:55:58 ntbs nfsen[26412]: Set new profile size: 0 Jul 30 15:55:59 ntbs nfsen[26412]: Add .:live:200907301550 for plugin processing Jul 30 15:56:01 ntbs nfsen[26412]: Update profile longmeadow in group . (... and so on for the remaining profiles after live) Jul 30 15:56:49 ntbs nfsen[26412]: Update profile ypos in group . Jul 30 15:56:49 ntbs nfsen[26412]: Run plugins for 200907301550 Jul 30 15:56:49 ntbs nfsen[26413]: connection on UNIX socket Jul 30 15:56:49 ntbs nfsen[26413]: comm server started: 14889 Jul 30 15:56:49 ntbs nfsen[14889]: Cmd Decode: run-plugins Jul 30 15:56:49 ntbs nfsen[14889]: Plugin Cycle: ., live, 200907301550 Jul 30 15:56:49 ntbs nfsen[14889]: Cmd Decode: quit Jul 30 15:56:49 ntbs nfsen[26412]: Run plugins done. Jul 30 15:56:49 ntbs nfsen[26412]: Check alerts for Thu Jul 30 15:50:00 2009 Jul 30 15:56:49 ntbs nfsen[26412]: Check alerts done. Jul 30 15:56:49 ntbs nfsen[26412]: Run expire at Thu Jul 30 15:55:00 2009 Jul 30 15:56:49 ntbs nfsen[26413]: comm child[14889] terminated with no exit value Jul 30 15:56:49 ntbs nfsen[26412]: End expire at Thu Jul 30 15:55:00 2009 Jul 30 16:01:26 ntbs nfsen[26412]: Check alerts for Thu Jul 30 15:55:00 2009 Thanks Chris. 2009/7/31 Adrian Popa <[email protected]>: > Hello Chris, > > If you have lots of flows it may take your server more than 5 minutes to > process your profiles - so it may die when the deadline is reached - and it > will not be able to update all rrds. > > You should be able to check this to see if the load on the server is high > and if you get all your profiles listed in the log: > > Jul 31 08:40:15 localhost nfsen[19265]: 45 channels/alerts to profile > Jul 31 08:40:37 localhost nfsen[19265]: Update profile Shadow1 in group . > Jul 31 08:40:39 localhost nfsen[19265]: Update profile Shadow2 in group . > Jul 31 08:40:41 localhost nfsen[19265]: Update profile live in group . > ... > > You can measure the time from the last Update profile message to the next > update cycle. > > Good luck. > > > On Thu, Jul 30, 2009 at 7:11 PM, Chris Wigglesworth <[email protected]> wrote: >> >> Hi, >> >> I installed nfsen 1.3.2 and nfdump 1.5.8 on Centos 5.3 x86_64 using >> the rrdtool 1.3.8 packages from RPMforge. The installation seems to >> have gone fine. I have added 138 sflow sources to my nfsen.conf and >> the rrd files for these channels are updating correctly. I have >> created a shadow profile for 135 of the sources using nfsen the >> command line tool, each with two channels. 90 of these shadow profiles >> are working correctly and updating the rrd files in the profiles-stat >> directory, 45 of these shadow profiles are not updating either of the >> channel rrd files in the profiles-stat directory. >> I cannot see any obvious difference between the shadow profiles that >> work and those that do not. I have tried wiping the installation and >> starting from scratch, but the same problems occur each time. If I >> change the names of the sources, then the same problems occur but with >> different sources affected. I cannot see anything obvious in the logs >> to explain what is going on. Are there any predefined limits that I >> need to change in the program or operating system? Any help would be >> appreciated. >> I have included the non commented parts of my nfsen.conf below (I have >> changed the site names). >> >> Cheers Chris. >> >> $BASEDIR = "/home/nfsen"; >> $BINDIR="${BASEDIR}/bin"; >> $LIBEXECDIR="${BASEDIR}/libexec"; >> $CONFDIR="${BASEDIR}/etc"; >> $HTMLDIR = "/home/nfsen/www/"; >> $DOCDIR="${HTMLDIR}/doc"; >> $VARDIR="${BASEDIR}/var"; >> $PROFILESTATDIR="${BASEDIR}/profiles-stat"; >> $PROFILEDATADIR="${BASEDIR}/profiles-data"; >> $BACKEND_PLUGINDIR="${BASEDIR}/plugins"; >> $FRONTEND_PLUGINDIR="${HTMLDIR}/plugins"; >> $PREFIX = '/usr/local/bin'; >> $USER = "nfsen"; >> $WWWUSER = "apache"; >> $WWWGROUP = "apache"; >> $BUFFLEN = 200000; >> $SUBDIRLAYOUT = 1; >> $ZIPcollected = 1; >> $ZIPprofiles = 1; >> $PROFILERS = 6; >> $DISKLIMIT = 98; >> $PROFILERS = 6; >> %sources = ( >> 'site9001' => { 'port' => '9001', 'col' => '#08088a', 'type' >> => 'sflow' }, >> 'site9002' => { 'port' => '9002', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9003' => { 'port' => '9003', 'col' => '#08088a', 'type' >> => 'sflow' }, >> 'site9004' => { 'port' => '9004', 'col' => '#08088a', 'type' >> => 'sflow' }, >> 'site9005' => { 'port' => '9005', 'col' => '#08088a', 'type' >> => 'sflow' }, >> 'site9006' => { 'port' => '9006', 'col' => '#08088a', 'type' >> => 'sflow' }, >> 'site9007' => { 'port' => '9007', 'col' => '#08088a', 'type' >> => 'sflow' }, >> 'site9008' => { 'port' => '9008', 'col' => '#08088a', 'type' >> => 'sflow' }, >> 'site9009' => { 'port' => '9009', 'col' => '#08088a', 'type' >> => 'sflow' }, >> 'site9010' => { 'port' => '9010', 'col' => '#08088a', 'type' >> => 'sflow' }, >> 'site9011' => { 'port' => '9011', 'col' => '#08088a', 'type' >> => 'sflow' }, >> 'site9012' => { 'port' => '9012', 'col' => '#08088a', 'type' >> => 'sflow' }, >> 'site9013' => { 'port' => '9013', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9014' => { 'port' => '9014', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9015' => { 'port' => '9015', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9016' => { 'port' => '9016', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9017' => { 'port' => '9017', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9018' => { 'port' => '9018', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9019' => { 'port' => '9019', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9020' => { 'port' => '9020', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9021' => { 'port' => '9021', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9022' => { 'port' => '9022', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9023' => { 'port' => '9023', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9024' => { 'port' => '9024', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9025' => { 'port' => '9025', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9026' => { 'port' => '9026', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9027' => { 'port' => '9027', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9028' => { 'port' => '9028', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9029' => { 'port' => '9029', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9030' => { 'port' => '9030', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9031' => { 'port' => '9031', 'col' => '#08088a', 'type' >> => 'sflow' }, >> 'site9032' => { 'port' => '9032', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9033' => { 'port' => '9033', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9034' => { 'port' => '9034', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9035' => { 'port' => '9035', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9036' => { 'port' => '9036', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9037' => { 'port' => '9037', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9038' => { 'port' => '9038', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9039' => { 'port' => '9039', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9040' => { 'port' => '9040', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9041' => { 'port' => '9041', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9042' => { 'port' => '9042', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9043' => { 'port' => '9043', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9044' => { 'port' => '9044', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9045' => { 'port' => '9045', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9046' => { 'port' => '9046', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9047' => { 'port' => '9047', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9048' => { 'port' => '9048', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9049' => { 'port' => '9049', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9050' => { 'port' => '9050', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9051' => { 'port' => '9051', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9052' => { 'port' => '9052', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9053' => { 'port' => '9053', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9054' => { 'port' => '9054', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9055' => { 'port' => '9055', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9056' => { 'port' => '9056', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9057' => { 'port' => '9057', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9058' => { 'port' => '9058', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9059' => { 'port' => '9059', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9060' => { 'port' => '9060', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9061' => { 'port' => '9061', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9062' => { 'port' => '9062', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9063' => { 'port' => '9063', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9064' => { 'port' => '9064', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9065' => { 'port' => '9065', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9066' => { 'port' => '9066', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9067' => { 'port' => '9067', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9068' => { 'port' => '9068', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9069' => { 'port' => '9069', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9070' => { 'port' => '9070', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9071' => { 'port' => '9071', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9072' => { 'port' => '9072', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9073' => { 'port' => '9073', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9074' => { 'port' => '9074', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9075' => { 'port' => '9075', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9076' => { 'port' => '9076', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9077' => { 'port' => '9077', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9078' => { 'port' => '9078', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9079' => { 'port' => '9079', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9080' => { 'port' => '9080', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9081' => { 'port' => '9081', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9082' => { 'port' => '9082', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9083' => { 'port' => '9083', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9084' => { 'port' => '9084', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9085' => { 'port' => '9085', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9086' => { 'port' => '9086', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9087' => { 'port' => '9087', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9088' => { 'port' => '9088', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9089' => { 'port' => '9089', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9090' => { 'port' => '9090', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9091' => { 'port' => '9091', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9092' => { 'port' => '9092', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9093' => { 'port' => '9093', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9094' => { 'port' => '9094', 'col' => '#b40404', 'type' >> => 'sflow' }, >> 'site9095' => { 'port' => '9095', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9096' => { 'port' => '9096', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9097' => { 'port' => '9097', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9098' => { 'port' => '9098', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9099' => { 'port' => '9099', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9100' => { 'port' => '9100', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9101' => { 'port' => '9101', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9102' => { 'port' => '9102', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9103' => { 'port' => '9103', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9104' => { 'port' => '9104', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9105' => { 'port' => '9105', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9106' => { 'port' => '9106', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9107' => { 'port' => '9107', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9108' => { 'port' => '9108', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9109' => { 'port' => '9109', 'col' => '#08088a', 'type' >> => 'sflow' }, >> 'site9110' => { 'port' => '9110', 'col' => '#08088a', 'type' >> => 'sflow' }, >> 'site9111' => { 'port' => '9111', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9112' => { 'port' => '9112', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9113' => { 'port' => '9113', 'col' => '#b40404', 'type' >> => 'sflow' }, >> 'site9114' => { 'port' => '9114', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9115' => { 'port' => '9115', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9116' => { 'port' => '9116', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9117' => { 'port' => '9117', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9118' => { 'port' => '9118', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9119' => { 'port' => '9119', 'col' => '#08088a', 'type' >> => 'sflow' }, >> 'site9120' => { 'port' => '9120', 'col' => '#08088a', 'type' >> => 'sflow' }, >> 'site9121' => { 'port' => '9121', 'col' => '#ffff00', 'type' >> => 'sflow' }, >> 'site9122' => { 'port' => '9122', 'col' => '#0080ff', 'type' >> => 'sflow' }, >> 'site9123' => { 'port' => '9123', 'col' => '#04b404', 'type' >> => 'sflow' }, >> 'site9124' => { 'port' => '9124', 'col' => '#04b404', 'type' >> => 'sflow' }, >> 'site9125' => { 'port' => '9125', 'col' => '#04b404', 'type' >> => 'sflow' }, >> 'site9126' => { 'port' => '9126', 'col' => '#04b404', 'type' >> => 'sflow' }, >> 'site9127' => { 'port' => '9127', 'col' => '#04b404', 'type' >> => 'sflow' }, >> 'site9128' => { 'port' => '9128', 'col' => '#04b404', 'type' >> => 'sflow' }, >> 'site9129' => { 'port' => '9129', 'col' => '#04b404', 'type' >> => 'sflow' }, >> 'site9130' => { 'port' => '9130', 'col' => '#04b404', 'type' >> => 'sflow' }, >> 'site9131' => { 'port' => '9131', 'col' => '#04b404', 'type' >> => 'sflow' }, >> 'site9132' => { 'port' => '9132', 'col' => '#b40404', 'type' >> => 'sflow' }, >> 'site9133' => { 'port' => '9133', 'col' => '#b40404', 'type' >> => 'sflow' }, >> 'site9134' => { 'port' => '9134', 'col' => '#b40404', 'type' >> => 'sflow' }, >> 'site9140' => { 'port' => '9140', 'col' => '#ffff00', 'type' >> => 'sflow' }, >> 'unknown' => { 'port' => '9000', 'col' => '#f5a9f2', 'type' => >> 'sflow' }, >> 'legacy' => { 'port' => '6343', 'col' => '#848484', 'type' => >> 'sflow' }, >> 'links' => { 'port' => '9999', 'col' => '#8a0886', 'type' => >> 'sflow' } >> ); >> $low_water = 90; >> $syslog_facility = 'local3'; >> @plugins = ( >> # profile # module >> # [ '*', 'demoplugin' ], >> ); >> %PluginConf = ( >> # For plugin demoplugin >> demoplugin => { >> # scalar >> param2 => 42, >> # hash >> param1 => { 'key' => 'value' }, >> }, >> # for plugin otherplugin >> otherplugin => [ >> # array >> 'mary had a little lamb' >> ], >> ); >> $MAIL_FROM = '[email protected]'; >> $SMTP_SERVER = 'localhost'; >> $MAIL_BODY = q{ >> Alert '@alert@' triggered at timeslot @timeslot@ >> }; >> 1; >> >> >> ------------------------------------------------------------------------------ >> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 >> 30-Day >> trial. Simplify your report design, integration and deployment - and focus >> on >> what you do best, core application coding. Discover what's new with >> Crystal Reports now. http://p.sf.net/sfu/bobj-july >> _______________________________________________ >> Nfsen-discuss mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/nfsen-discuss > > ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ Nfsen-discuss mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
