Hi Peter, I cannot see any error messages in the logs except "Error reading channel stat information. Missing key 'first'" and "Failed get stat info for requested time slot", both of which I believe are normal from reading the mailing list archives.
The problem does seem to be related to the number of shadow profiles rather than their individual characteristics. I deleted all of the shadow profiles and added back 46 of them, all with exactly the same parameters as before and including a number of which were failing to update the RRD files. All of them worked perfectly. I added another 45 and again all 91 worked correctly with the shadow profile RRD files being updated. I then added the remaining 43 shadow profiles bringing the total to 134. After this only 44 of the shadow profiles continued to update the RRD files with 90 of them now failing to do so. The working profiles were seemingly a random mix from each of the three batches added. If I remove profiles to reduce the number down to 89, then some RRD files start updating again and some do not.Deleting a further 30 profiles causes all of the RRD files to be updated correctly again. Cheers Chris. 2009/7/31 Peter Haag <[email protected]>: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Well - even if the system is too slow, *all* profile should be updated. > The only option I can think of, is that the names of the channels does not > fully conform to RRD. Check the log file to get some hints. > > - Peter > > > Adrian Popa wrote: >> Hello Chris, >> >> If you have lots of flows it may take your server more than 5 minutes to >> process your profiles - so it may die when the deadline is reached - and it >> will not be able to update all rrds. >> >> You should be able to check this to see if the load on the server is high >> and if you get all your profiles listed in the log: >> >> Jul 31 08:40:15 localhost nfsen[19265]: 45 channels/alerts to profile >> Jul 31 08:40:37 localhost nfsen[19265]: Update profile Shadow1 in group . >> Jul 31 08:40:39 localhost nfsen[19265]: Update profile Shadow2 in group . >> Jul 31 08:40:41 localhost nfsen[19265]: Update profile live in group . >> ... >> >> You can measure the time from the last Update profile message to the next >> update cycle. >> >> Good luck. >> >> >> On Thu, Jul 30, 2009 at 7:11 PM, Chris Wigglesworth <[email protected]> wrote: >> >>> Hi, >>> >>> I installed nfsen 1.3.2 and nfdump 1.5.8 on Centos 5.3 x86_64 using >>> the rrdtool 1.3.8 packages from RPMforge. The installation seems to >>> have gone fine. I have added 138 sflow sources to my nfsen.conf and >>> the rrd files for these channels are updating correctly. I have >>> created a shadow profile for 135 of the sources using nfsen the >>> command line tool, each with two channels. 90 of these shadow profiles >>> are working correctly and updating the rrd files in the profiles-stat >>> directory, 45 of these shadow profiles are not updating either of the >>> channel rrd files in the profiles-stat directory. >>> I cannot see any obvious difference between the shadow profiles that >>> work and those that do not. I have tried wiping the installation and >>> starting from scratch, but the same problems occur each time. If I >>> change the names of the sources, then the same problems occur but with >>> different sources affected. I cannot see anything obvious in the logs >>> to explain what is going on. Are there any predefined limits that I >>> need to change in the program or operating system? Any help would be >>> appreciated. >>> I have included the non commented parts of my nfsen.conf below (I have >>> changed the site names). >>> >>> Cheers Chris. >>> >>> $BASEDIR = "/home/nfsen"; >>> $BINDIR="${BASEDIR}/bin"; >>> $LIBEXECDIR="${BASEDIR}/libexec"; >>> $CONFDIR="${BASEDIR}/etc"; >>> $HTMLDIR = "/home/nfsen/www/"; >>> $DOCDIR="${HTMLDIR}/doc"; >>> $VARDIR="${BASEDIR}/var"; >>> $PROFILESTATDIR="${BASEDIR}/profiles-stat"; >>> $PROFILEDATADIR="${BASEDIR}/profiles-data"; >>> $BACKEND_PLUGINDIR="${BASEDIR}/plugins"; >>> $FRONTEND_PLUGINDIR="${HTMLDIR}/plugins"; >>> $PREFIX = '/usr/local/bin'; >>> $USER = "nfsen"; >>> $WWWUSER = "apache"; >>> $WWWGROUP = "apache"; >>> $BUFFLEN = 200000; >>> $SUBDIRLAYOUT = 1; >>> $ZIPcollected = 1; >>> $ZIPprofiles = 1; >>> $PROFILERS = 6; >>> $DISKLIMIT = 98; >>> $PROFILERS = 6; >>> %sources = ( >>> 'site9001' => { 'port' => '9001', 'col' => '#08088a', 'type' >>> => 'sflow' }, >>> 'site9002' => { 'port' => '9002', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9003' => { 'port' => '9003', 'col' => '#08088a', 'type' >>> => 'sflow' }, >>> 'site9004' => { 'port' => '9004', 'col' => '#08088a', 'type' >>> => 'sflow' }, >>> 'site9005' => { 'port' => '9005', 'col' => '#08088a', 'type' >>> => 'sflow' }, >>> 'site9006' => { 'port' => '9006', 'col' => '#08088a', 'type' >>> => 'sflow' }, >>> 'site9007' => { 'port' => '9007', 'col' => '#08088a', 'type' >>> => 'sflow' }, >>> 'site9008' => { 'port' => '9008', 'col' => '#08088a', 'type' >>> => 'sflow' }, >>> 'site9009' => { 'port' => '9009', 'col' => '#08088a', 'type' >>> => 'sflow' }, >>> 'site9010' => { 'port' => '9010', 'col' => '#08088a', 'type' >>> => 'sflow' }, >>> 'site9011' => { 'port' => '9011', 'col' => '#08088a', 'type' >>> => 'sflow' }, >>> 'site9012' => { 'port' => '9012', 'col' => '#08088a', 'type' >>> => 'sflow' }, >>> 'site9013' => { 'port' => '9013', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9014' => { 'port' => '9014', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9015' => { 'port' => '9015', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9016' => { 'port' => '9016', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9017' => { 'port' => '9017', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9018' => { 'port' => '9018', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9019' => { 'port' => '9019', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9020' => { 'port' => '9020', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9021' => { 'port' => '9021', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9022' => { 'port' => '9022', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9023' => { 'port' => '9023', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9024' => { 'port' => '9024', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9025' => { 'port' => '9025', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9026' => { 'port' => '9026', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9027' => { 'port' => '9027', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9028' => { 'port' => '9028', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9029' => { 'port' => '9029', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9030' => { 'port' => '9030', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9031' => { 'port' => '9031', 'col' => '#08088a', 'type' >>> => 'sflow' }, >>> 'site9032' => { 'port' => '9032', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9033' => { 'port' => '9033', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9034' => { 'port' => '9034', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9035' => { 'port' => '9035', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9036' => { 'port' => '9036', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9037' => { 'port' => '9037', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9038' => { 'port' => '9038', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9039' => { 'port' => '9039', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9040' => { 'port' => '9040', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9041' => { 'port' => '9041', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9042' => { 'port' => '9042', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9043' => { 'port' => '9043', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9044' => { 'port' => '9044', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9045' => { 'port' => '9045', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9046' => { 'port' => '9046', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9047' => { 'port' => '9047', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9048' => { 'port' => '9048', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9049' => { 'port' => '9049', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9050' => { 'port' => '9050', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9051' => { 'port' => '9051', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9052' => { 'port' => '9052', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9053' => { 'port' => '9053', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9054' => { 'port' => '9054', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9055' => { 'port' => '9055', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9056' => { 'port' => '9056', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9057' => { 'port' => '9057', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9058' => { 'port' => '9058', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9059' => { 'port' => '9059', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9060' => { 'port' => '9060', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9061' => { 'port' => '9061', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9062' => { 'port' => '9062', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9063' => { 'port' => '9063', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9064' => { 'port' => '9064', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9065' => { 'port' => '9065', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9066' => { 'port' => '9066', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9067' => { 'port' => '9067', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9068' => { 'port' => '9068', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9069' => { 'port' => '9069', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9070' => { 'port' => '9070', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9071' => { 'port' => '9071', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9072' => { 'port' => '9072', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9073' => { 'port' => '9073', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9074' => { 'port' => '9074', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9075' => { 'port' => '9075', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9076' => { 'port' => '9076', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9077' => { 'port' => '9077', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9078' => { 'port' => '9078', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9079' => { 'port' => '9079', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9080' => { 'port' => '9080', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9081' => { 'port' => '9081', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9082' => { 'port' => '9082', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9083' => { 'port' => '9083', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9084' => { 'port' => '9084', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9085' => { 'port' => '9085', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9086' => { 'port' => '9086', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9087' => { 'port' => '9087', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9088' => { 'port' => '9088', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9089' => { 'port' => '9089', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9090' => { 'port' => '9090', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9091' => { 'port' => '9091', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9092' => { 'port' => '9092', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9093' => { 'port' => '9093', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9094' => { 'port' => '9094', 'col' => '#b40404', 'type' >>> => 'sflow' }, >>> 'site9095' => { 'port' => '9095', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9096' => { 'port' => '9096', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9097' => { 'port' => '9097', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9098' => { 'port' => '9098', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9099' => { 'port' => '9099', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9100' => { 'port' => '9100', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9101' => { 'port' => '9101', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9102' => { 'port' => '9102', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9103' => { 'port' => '9103', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9104' => { 'port' => '9104', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9105' => { 'port' => '9105', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9106' => { 'port' => '9106', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9107' => { 'port' => '9107', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9108' => { 'port' => '9108', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9109' => { 'port' => '9109', 'col' => '#08088a', 'type' >>> => 'sflow' }, >>> 'site9110' => { 'port' => '9110', 'col' => '#08088a', 'type' >>> => 'sflow' }, >>> 'site9111' => { 'port' => '9111', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9112' => { 'port' => '9112', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9113' => { 'port' => '9113', 'col' => '#b40404', 'type' >>> => 'sflow' }, >>> 'site9114' => { 'port' => '9114', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9115' => { 'port' => '9115', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9116' => { 'port' => '9116', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9117' => { 'port' => '9117', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9118' => { 'port' => '9118', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9119' => { 'port' => '9119', 'col' => '#08088a', 'type' >>> => 'sflow' }, >>> 'site9120' => { 'port' => '9120', 'col' => '#08088a', 'type' >>> => 'sflow' }, >>> 'site9121' => { 'port' => '9121', 'col' => '#ffff00', 'type' >>> => 'sflow' }, >>> 'site9122' => { 'port' => '9122', 'col' => '#0080ff', 'type' >>> => 'sflow' }, >>> 'site9123' => { 'port' => '9123', 'col' => '#04b404', 'type' >>> => 'sflow' }, >>> 'site9124' => { 'port' => '9124', 'col' => '#04b404', 'type' >>> => 'sflow' }, >>> 'site9125' => { 'port' => '9125', 'col' => '#04b404', 'type' >>> => 'sflow' }, >>> 'site9126' => { 'port' => '9126', 'col' => '#04b404', 'type' >>> => 'sflow' }, >>> 'site9127' => { 'port' => '9127', 'col' => '#04b404', 'type' >>> => 'sflow' }, >>> 'site9128' => { 'port' => '9128', 'col' => '#04b404', 'type' >>> => 'sflow' }, >>> 'site9129' => { 'port' => '9129', 'col' => '#04b404', 'type' >>> => 'sflow' }, >>> 'site9130' => { 'port' => '9130', 'col' => '#04b404', 'type' >>> => 'sflow' }, >>> 'site9131' => { 'port' => '9131', 'col' => '#04b404', 'type' >>> => 'sflow' }, >>> 'site9132' => { 'port' => '9132', 'col' => '#b40404', 'type' >>> => 'sflow' }, >>> 'site9133' => { 'port' => '9133', 'col' => '#b40404', 'type' >>> => 'sflow' }, >>> 'site9134' => { 'port' => '9134', 'col' => '#b40404', 'type' >>> => 'sflow' }, >>> 'site9140' => { 'port' => '9140', 'col' => '#ffff00', 'type' >>> => 'sflow' }, >>> 'unknown' => { 'port' => '9000', 'col' => '#f5a9f2', 'type' => >>> 'sflow' }, >>> 'legacy' => { 'port' => '6343', 'col' => '#848484', 'type' => >>> 'sflow' }, >>> 'links' => { 'port' => '9999', 'col' => '#8a0886', 'type' => 'sflow' >>> } >>> ); >>> $low_water = 90; >>> $syslog_facility = 'local3'; >>> @plugins = ( >>> # profile # module >>> # [ '*', 'demoplugin' ], >>> ); >>> %PluginConf = ( >>> # For plugin demoplugin >>> demoplugin => { >>> # scalar >>> param2 => 42, >>> # hash >>> param1 => { 'key' => 'value' }, >>> }, >>> # for plugin otherplugin >>> otherplugin => [ >>> # array >>> 'mary had a little lamb' >>> ], >>> ); >>> $MAIL_FROM = '[email protected]'; >>> $SMTP_SERVER = 'localhost'; >>> $MAIL_BODY = q{ >>> Alert '@alert@' triggered at timeslot @timeslot@ >>> }; >>> 1; >>> >>> >>> ------------------------------------------------------------------------------ >>> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day >>> trial. Simplify your report design, integration and deployment - and focus >>> on >>> what you do best, core application coding. Discover what's new with >>> Crystal Reports now. http://p.sf.net/sfu/bobj-july >>> _______________________________________________ >>> Nfsen-discuss mailing list >>> [email protected] >>> https://lists.sourceforge.net/lists/listinfo/nfsen-discuss >>> >> >> >> ------------------------------------------------------------------------ >> >> ------------------------------------------------------------------------------ >> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day >> trial. Simplify your report design, integration and deployment - and focus on >> what you do best, core application coding. Discover what's new with >> Crystal Reports now. http://p.sf.net/sfu/bobj-july >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Nfsen-discuss mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/nfsen-discuss > > - -- > _______ SWITCH - The Swiss Education and Research Network ______ > Peter Haag, Security Engineer, Member of SWITCH CERT > PGP fingerprint: D9 31 D5 83 03 95 68 BA FB 84 CA 94 AB FC 5D D7 > SWITCH, Werdstrasse 2, P.O. Box, CH-8021 Zurich, Switzerland > E-mail: [email protected] Web: http://www.switch.ch/ > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.5 (Darwin) > > iQCVAwUBSnKUA/5AbZRALNr/AQLQuwQAlLkuKPKWLOBQHBf737M5UrwJtKVh0zC7 > KRb9NR00b7Oea6eZLV8pND+A7yjAGZ9nwMN0Z0GqS6jIadM2KuRVrZhwWLfEWmjq > Y9YeWu2n8L1rh1a5wTf5xa4my17Vg7oL+0+aAgoPGOk6HaGhjMzmi99zO0mbxDxi > WbBsXRumsVo= > =yZa2 > -----END PGP SIGNATURE----- > ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ Nfsen-discuss mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
