-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi, list!
I'm stuck with a rather peculiar problem here. Currently running nfsen
1.3 with nfdump-1.5.7 on a server which is not the collector. That means
I've got nfcapd running on a separate machine writing its data to a SAN
volume. The nfsen server is sharing this volume through a OCFS2 cluster
in read-only mode. I've set port = 0 for each flow source in nfsen.conf
so that it'll not act as collector. By creating symlinks in the live
directory mapping to the shared read-only volume, nfsen are now able to
read data from each source. This works perfectly!
The problem is profiles. When I create new profiles in this
configuration (no matter which way I try to set it up), the result is
always the same; no data and therefore empty graphs.
But when I add a "local" source to nfsen, a source which I allow nfsen
to be collector for (I simply installed fprobe on the same machine
that's running nfsen), then profiles for this source works just fine.
Here's my setup. Probe01-10 are sources delivering flows to the nfcapd
collector on another server, whilst test01 is the local source running
fprobe on the the same server as nfsen.
# pwd
/var/log/nfsen/profiles-data
# ls -la
total 4
drwxrwxr-x 5 www-data www-data 48 2008-05-02 10:14 .
drwxr-xr-x 4 root root 46 2008-04-22 15:05 ..
drwxrwxr-x 3 www-data www-data 18 2008-05-01 18:57 probe01-web
drwxrwxr-x 13 www-data www-data 4096 2008-05-01 18:40 live
drwxrwxr-x 3 www-data www-data 19 2008-05-01 18:51 test01-ssh
# pwd
/var/log/nfsen/profiles-data/test01-ssh/test01/2008/05/07
# ls -la | head
total 592
drwxr-xr-x 2 www-data www-data 8192 2008-05-07 12:05 .
drwxr-xr-x 9 www-data www-data 69 2008-05-07 00:05 ..
- -rw-r--r-- 1 www-data www-data 1432 2008-05-07 00:05 nfcapd.200805070000
- -rw-r--r-- 1 www-data www-data 1276 2008-05-07 00:10 nfcapd.200805070005
- -rw-r--r-- 1 www-data www-data 1484 2008-05-07 00:15 nfcapd.200805070010
- -rw-r--r-- 1 www-data www-data 1588 2008-05-07 00:20 nfcapd.200805070015
- -rw-r--r-- 1 www-data www-data 1224 2008-05-07 00:25 nfcapd.200805070020
- -rw-r--r-- 1 www-data www-data 1380 2008-05-07 00:30 nfcapd.200805070025
- -rw-r--r-- 1 www-data www-data 1640 2008-05-07 00:35 nfcapd.200805070030
# pwd
/var/log/nfsen/profiles-data/probe01-web/probe01/2008/05/07
# ls -la | head
total 588
drwxr-xr-x 2 www-data www-data 8192 2008-05-07 12:00 .
drwxrwxr-x 9 www-data www-data 69 2008-05-07 00:05 ..
- -rw-r--r-- 1 www-data www-data 276 2008-05-07 00:05 nfcapd.200805070000
- -rw-r--r-- 1 www-data www-data 276 2008-05-07 00:10 nfcapd.200805070005
- -rw-r--r-- 1 www-data www-data 276 2008-05-07 00:15 nfcapd.200805070010
- -rw-r--r-- 1 www-data www-data 276 2008-05-07 00:20 nfcapd.200805070015
- -rw-r--r-- 1 www-data www-data 276 2008-05-07 00:25 nfcapd.200805070020
- -rw-r--r-- 1 www-data www-data 276 2008-05-07 00:30 nfcapd.200805070025
- -rw-r--r-- 1 www-data www-data 276 2008-05-07 00:35 nfcapd.200805070030
This shows that data from test01 are successfully processed by the
test01-ssh-profile, but data files for the probe01-web-profile are empty
(276 bytes header only).
May 2 10:20:15 nfsen[7756]: Run periodic at Fri May 2 10:20:00 2008
May 2 10:20:15 nfsen[7756]: Prepare profiling './probe01-web'
May 2 10:20:15 nfsen[7756]: Prepare profiling './live'
May 2 10:20:15 nfsen[7756]: Prepare profiling './test01-ssh'
May 2 10:20:15 nfsen[7756]: 2 channels/alerts to profile
May 2 10:20:15 nfsen[7756]: profile opts: .#probe01-web#2#probe01#probe01
May 2 10:20:15 nfsen[7756]: profile opts: .#test01-ssh#2#test01#test01
May 2 10:20:17 nfsen[7756]: Update profile probe01-web in group .
May 2 10:20:17 nfsen[7756]: Add channel size 765952
May 2 10:20:17 nfsen[7756]: Set new profile size: 765952
May 2 10:20:17 nfsen[7756]: Add .:probe01-web:200805021015 for plugin
processing
May 2 10:20:17 nfsen[7756]: Update profile live in group .
May 2 10:20:17 nfsen[7756]: Add channel size 937164800
May 2 10:20:17 nfsen[7756]: Add channel size 47934865408
May 2 10:20:17 nfsen[7756]: Add channel size 2294874112
May 2 10:20:17 nfsen[7756]: Add channel size 8851030016
May 2 10:20:17 nfsen[7756]: Add channel size 213041807360
May 2 10:20:17 nfsen[7756]: Add channel size 200572272640
May 2 10:20:17 nfsen[7756]: Add channel size 94284611584
May 2 10:20:17 nfsen[7756]: Channel info file missing for channel
'test01' in './live'
May 2 10:20:17 nfsen[7756]: Add channel size 10805903360
May 2 10:20:17 nfsen[7756]: Add channel size 963772416
May 2 10:20:17 nfsen[7756]: Add channel size 2371158016
May 2 10:20:17 nfsen[7756]: Set new profile size: 582057459712
May 2 10:20:17 nfsen[7756]: Add .:live:200805021015 for plugin processing
May 2 10:20:18 nfsen[7756]: Update profile test01-ssh in group .
May 2 10:20:18 nfsen[7756]: Add channel size 761856
May 2 10:20:18 nfsen[7756]: Set new profile size: 761856
May 2 10:20:18 nfsen[7756]: Add .:test01-ssh:200805021015 for plugin
processing
May 2 10:20:18 nfsen[7756]: Run plugins for 200805021015
May 2 10:20:18 nfsen[7757]: connection on UNIX socket
May 2 10:20:18 nfsen[7757]: comm server started: 21222
May 2 10:20:18 nfsen[21222]: Cmd Decode: run-plugins
May 2 10:20:18 nfsen[21222]: Plugin Cycle: ., probe01-web, 200805021015
May 2 10:20:18 nfsen[21222]: Plugin Cycle: ., live, 200805021015
May 2 10:20:18 nfsen[21222]: Plugin Cycle: ., test01-ssh, 200805021015
May 2 10:20:18 nfsen[21222]: Cmd Decode: quit
May 2 10:20:18 nfsen[7756]: Run plugins done.
May 2 10:20:18 nfsen[7756]: Check alerts for Fri May 2 10:15:00 2008
May 2 10:20:18 nfsen[7756]: Check alerts done.
May 2 10:20:18 nfsen[7756]: Run expire at Fri May 2 10:20:00 2008
May 2 10:20:18 nfsen[7756]: End expire at Fri May 2 10:20:00 2008
May 2 10:20:18 nfsen[7757]: comm child[21222] terminated with no exit
value
May 2 10:20:20 snmpd[5748]: Connection from UDP: [10.20.0.29]:54231
And looking at syslog give me no clues as to what the problem might be.
You can see that both test01-ssh profile and probe01-web profile are
running. The "terminated with no exit value" and "info file missing for
channel" have I seen on other working systems as well.
Here is the symlinking of the directories I was talking about. This is
done for each probe:
# pwd
/var/log/nfsen/profiles-data/live/probe01
# ls -la
total 8
drwxrwxr-x 2 www-data www-data 31 2008-05-01 17:45 .
drwxrwxr-x 13 www-data www-data 4096 2008-05-01 18:40 ..
lrwxrwxrwx 1 root root 32 2008-04-22 16:35 2008 ->
/var/log/netflow/probe01/2008
- -rw-r--r-- 1 www-data www-data 108 2008-05-01 18:01 .nfstat
While the test probe (running locally on the nfsen server) has a "real"
directory structure.
# pwd
/var/log/nfsen/profiles-data/live/test01
# ls -la
total 68
drwxrwxr-x 3 www-data www-data 43 2008-05-07 11:40 .
drwxrwxr-x 13 www-data www-data 4096 2008-05-01 18:40 ..
drwxr-xr-x 3 www-data www-data 15 2008-05-01 18:45 2008
- -rw-r--r-- 1 www-data www-data 276 2008-05-07 11:40 nfcapd.current.7755
Mount here shows that the /var/log/netflow directory is actually an
ocfs2 cluster-volume in read-only mode.
# mount
/dev/mapper/3600508b4000685ab0000b000062f0000p2 on / type xfs (rw)
proc on /proc type proc (rw,noexec,nosuid,nodev)
/sys on /sys type sysfs (rw,noexec,nosuid,nodev)
varrun on /var/run type tmpfs (rw,noexec,nosuid,nodev,mode=0755)
varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777)
udev on /dev type tmpfs (rw,mode=0755)
devshm on /dev/shm type tmpfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/mapper/3600508b4000685ab0000b000062f0000p1 on /boot type ext2 (rw)
securityfs on /sys/kernel/security type securityfs (rw)
configfs on /sys/kernel/config type configfs (rw)
ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw)
/dev/mapper/netflowvg-netflowlv on /var/log/netflow type ocfs2
(ro,_netdev,heartbeat=local)
And finally here is my nfsen configuration file, which shows the port =
0 setup.
# cat /opt/nfsen/etc/nfsen.conf
$BASEDIR = "/opt/nfsen";
$BINDIR ="${BASEDIR}/bin";
$LIBEXECDIR ="${BASEDIR}/libexec";
$CONFDIR ="${BASEDIR}/etc";
$HTMLDIR = "/var/www/html/nfsen/";
$DOCDIR ="${HTMLDIR}/doc";
$VARDIR ="${BASEDIR}/var";
$PROFILESTATDIR ="/var/log/nfsen/profiles-stat";
$PROFILEDATADIR ="/var/log/nfsen/profiles-data";
$BACKEND_PLUGINDIR ="${BASEDIR}/plugins";
$FRONTEND_PLUGINDIR ="${HTMLDIR}/plugins";
$PREFIX = '/usr/local/bin';
$USER = "www-data";
$WWWUSER = "www-data";
$WWWGROUP = "www-data";
$BUFFLEN = 200000;
$SUBDIRLAYOUT = 1;
$ZIPcollected = 0;
$ZIPprofiles = 0;
$DISKLIMIT = 0;
%sources = (
'probe01' => { 'port' => '0', 'col' => '#ff0000' },
'probe02' => { 'port' => '0', 'col' => '#00ff00' },
'probe03' => { 'port' => '0', 'col' => '#0000ff' },
'probe04' => { 'port' => '0', 'col' => '#ffff00' },
'probe05' => { 'port' => '0', 'col' => '#ff00ff' },
'probe06' => { 'port' => '0', 'col' => '#00ffff' },
'probe07' => { 'port' => '0', 'col' => '#880000' },
'probe08' => { 'port' => '0', 'col' => '#008800' },
'probe09' => { 'port' => '0', 'col' => '#000088' },
'probe10' => { 'port' => '0', 'col' => '#880088' },
'test01' => { 'port' => '9999', 'col' => '#000000' },
);
$low_water = 90;
$syslog_facility = 'local3';
@plugins = (
# profile # module
# [ '*', 'demoplugin' ],
);
%PluginConf = (
demoplugin => {
param2 => 42,
param1 => { 'key' => 'value' },
},
otherplugin => [
'mary had a little lamb'
],
);
$MAIL_FROM = '[EMAIL PROTECTED]';
$SMTP_SERVER = 'apollo.example.net';
$MAIL_BODY = q{
Alert '@alert@' triggered at timeslot @timeslot@
};
1;
So, I'm puzzled as to what may cause the data files for the profiles to
be empty. Nfsen has write permissions to everything but
/var/log/nfsen/profiles-data/live/probeXX/2008/ so I don't see how this
can be a permissions problem (and if it were, shouldn't I have seen
complaints in syslog about that).
Sorry for the overly long mail, but if you've read this far, I would
very much appreciate any thoughts you may have.
Thanks.
Tor I. Skaar
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFII3HK6kzbtNj+3wMRArv+AJoC0xAsY3z9A6CPAytjjlDq0HEbegCgvCI/
ePo/f4FnmOsvQb2QBnwmmUk=
=OXiC
-----END PGP SIGNATURE-----
-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
_______________________________________________
Nfsen-discuss mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
