-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Paul, It's a bug in nfdump. The file selection required for the second file name the full sub directory hierarchy to work correctly. Find appended a fixed version of flist.c, which understands both versions of the file names. Recompile and reinstall nfdump and it should work afterwards.
- Peter
- --On April 24, 2007 16:54:55 +0000 Paul Vlaar <[EMAIL PROTECTED]> wrote:
| Hi all.
|
| In nfsen, I have $SUBDIRLAYOUT = 1;, and so I have a directory structure
| that consists of <profile>/<source>/<year>/<month>/<day>/nfcapd.<timestamp>
|
| I would like to get some netflow statistics that spans multiple days, so
| first of all I try this using the nfsen web interface, by selecting a
| time window and then running the Stat TopN function over it. The time
| window selected is 2007-04-21-15-05 to 2007-04-24-15-10 but what I see
| is that the result of the "process" only makes it to roughly 00:00 on
| 2007-04-21, no further:
|
| nfdump -R
|
/opt/netflow/nfsen/profiles/./live/myrouter/2007/04/21/nfcapd.200704211505:nfcapd.200704241510
-n
| 10 -s dstport/packets
|
| Top 10 Dst Port ordered by packets:
| Date first seen Duration Proto Dst Port Flows Packets
Bytes pps
| bps bpp 2007-04-21 15:04:48.819 32103.274 any 25 37326
39102 4.1 M
| 1 1067 109 2007-04-21 15:04:59.815 32081.786 any 53
3895 3977
| 284750 0 71 71 2007-04-21 17:20:26.063 23566.947 any
3102 17
| 224 304763 0 103 1360 2007-04-21 15:08:43.187 31285.996 any
113
| 150 153 6969 0 1 45 2007-04-21 15:36:10.204
27374.256 any
| 2378 13 149 209089 0 61 1403 2007-04-21
15:31:14.003 30067.599 any
| 2321 14 145 198457 0 52 1368 2007-04-21
15:19:01.806 26485.057 any
| 2444 10 143 197758 0 59 1382 2007-04-21
15:29:25.617 25955.168 any
| 1574 19 123 167863 0 51 1364 2007-04-21
15:17:21.832 30593.236 any
| 80 113 123 11232 0 2 91 2007-04-21 15:41:44.725
10058.798 any
| 49427 6 104 140648 0 111 1352
|
| Summary: total flows: 131499, total bytes: 96.2 M, total packets: 152557, avg
bps: 25100, avg
| pps: 4, avg bpp: 660 Time window: 2007-04-21 15:03:58 - 2007-04-21 23:59:57
| Total flows processed: 2461470, skipped: 0, Bytes read: 127999008
| Sys: 0.480s flows/second: 5127218.6 Wall: 8.955s flows/second: 274866.4
|
|
| I want more than just that limited time window, so I try this using nfdump
| directly, and I think I need to use a -M / -R combination according to the
| man page:
|
| $ nfdump -M /opt/netflow/nfsen/profiles/./live/myrouter/2007/04/21:22:23:24 -R
| nfcapd.200704211505:nfcapd.200704241510 -n 10 -s dstport/packets
|
| Top 10 Dst Port ordered by packets:
| Date first seen Duration Proto Dst Port Flows Packets
Bytes pps
| bps bpp 2007-04-21 15:04:00.499 259843.564 any 53 5.9 M
7.2 M 648.8 M
| 29 20946 89 2007-04-21 15:04:32.775 259810.220 any 123
4.4 M 4.6 M
| 345.8 M 18 11166 75 2007-04-21 15:04:08.707 259848.148 any
80 1.2
| M 1.8 M 159.6 M 7 5152 86 2007-04-21 15:04:25.447
259819.592 any
| 22 238196 1.1 M 266.9 M 4 8617 240 2007-04-21 15:04:32.007
259810.712 any
| 2048 670209 706080 31.9 M 2 1030 47 2007-04-21
15:04:44.947 259808.244 any
| 25 427251 476236 107.1 M 1 3458 235 2007-04-21 15:04:22.339
259820.848 any
| 32768 343978 418339 40.9 M 1 1322 102 2007-04-21
15:04:44.311 259798.812 any
| 2816 206620 228431 14.3 M 0 462 65 2007-04-21
15:04:24.715 259807.524 any
| 873 8918 191816 11.6 M 0 374 63 2007-04-21
15:07:00.917 259201.426 any
| 5432 2746 186060 9.6 M 0 309 53
|
| Summary: total flows: 22286340, total bytes: 9.9 G, total packets: 31.1 M,
avg bps: 326373, avg
| pps: 125, avg bpp: 325 Time window: 2007-04-21 15:03:58 - 2007-04-24 15:14:58
| Total flows processed: 22286340, skipped: 0, Bytes read: 1158912336
| Sys: 5.536s flows/second: 4025618.6 Wall: 46.991s flows/second: 474265.1
|
|
| The latter try works, so my guess is that nfsen is buggy in it's call to
| nfdump for when the directory layout is not flat. I am running version
| snapshot-20070208.
|
|
| Paul Vlaar
|
| --
| [EMAIL PROTECTED] - ISC Operations - PGP 0x294EC062
|
| -------------------------------------------------------------------------
| This SF.net email is sponsored by DB2 Express
| Download DB2 Express C - the FREE version of DB2 express and take
| control of your XML. No limits. Just data. Click to get it now.
| http://sourceforge.net/powerbar/db2/
| _______________________________________________
| Nfsen-discuss mailing list
| Nfsen-discuss@lists.sourceforge.net
| https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
- --
_______ SWITCH - The Swiss Education and Research Network ______
Peter Haag, Security Engineer, Member of SWITCH CERT
PGP fingerprint: D9 31 D5 83 03 95 68 BA FB 84 CA 94 AB FC 5D D7
SWITCH, Werdstrasse 2, P.O. Box, CH-8021 Zurich, Switzerland
E-mail: [EMAIL PROTECTED] Web: http://www.switch.ch/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)
iQCVAwUBRjDFkP5AbZRALNr/AQLhIAQAo8balbEiKheO5USU3LuWZOcvkcmAJJTz
ZROGVPqsozOgR9e2JV1YKS7gsfyCYsS+HEqwWJN0bvUeUk2y6FTUX+qXuWzPO83t
zJKOtFBZ9UwxJrfYptWDy4RXbQrrSyMemFJo2w9p2kRkjwSpSjOd4W5H809FX9yq
qaxGmD11/Yw=
=8gPp
-----END PGP SIGNATURE-----
/*
* This file is part of the nfdump project.
*
* Copyright (c) 2004, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
notice,
* this list of conditions and the following disclaimer in the
documentation
* and/or other materials provided with the distribution.
* * Neither the name of SWITCH nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: peter $
*
* $Id: flist.c 88 2007-03-06 08:49:26Z peter $
*
* $LastChangedRevision: 88 $
*
*/
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <time.h>
#include <string.h>
#include <errno.h>
#include <dirent.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/param.h>
#include <fcntl.h>
#include <ctype.h>
#include <netinet/in.h>
#include "config.h"
#ifdef HAVE_STDINT_H
# include <stdint.h>
#endif
#ifdef HAVE_FTS_H
# include <fts.h>
#else
# include "fts_compat.h"
#define fts_children fts_children_compat
#define fts_close fts_close_compat
#define fts_open fts_open_compat
#define fts_read fts_read_compat
#define fts_set fts_set_compat
#endif
#include "nffile.h"
#include "util.h"
#include "flist.h"
/* Global vars */
extern char *CurrentIdent;
/*
* Select a single file
* --------------------
* -r [/]path/to/single_file
* Select a single file: absolute or relativ path to a single file.
* Recursive: no
*
* Selecting a range of files
* --------------------------
* -R [/]path/to/first_file
* Select a range of files in directory specified by absolut or relative path
[/]path/to/
* Files are selected in alphabetical order starting with 'first_file' to the
end of
* the directory.
*
* -R [/]path/to/first_file:last_file
* Select a range of files in directory specified by absolut or relative path
[/]path/to/
* Files are selected in alphabetical order starting with 'first_file' and
ending with
* 'last_file'.
*
* -R [/]path/to/directory
* Select all files in alphabetical order in directory specified by
absolut or relative
* path [/]path/to/directory
*
* Selecting files over multiple sources
* -------------------------------------
* -M /path/to/multiple/source1:source2[:..:sourceN]
* It is assumed, that each source directory 'source1', 'source2' etc. exists
in directory
* /path/to/multiple. This will expand to multiple directories:
* /path/to/multiple/source1
* /path/to/multiple/source2
* ..
* /path/to/multiple/sourceN
* Each of these directories contain the same files.
* Used in combination with -r and -R to prepend file selections.
*
* Select a single file from multiple directories
* ----------------------------------------------
* -M /path/to/source1:source2 -r single_file
* Select the same file 'single_file' from each source directory: e.g.
* /path/to/source1/single_file
* /path/to/source2/single_file
*
*
* Select a range of files from multiple directories
* -------------------------------------------------
* -M /path/to/source1:source2[:..] -R first_file
* For each expanded directory specified by -M /path/to/source1:source2
* select a range of files as described above. Would be identical to
* -R /path/to/source1/first_file -R /path/to/source2/first_file
*
* -M /path/to/source1:source2[:..] -R first_file:last_file
* For each expanded directory specified by -M /path/to/source1:source2
* select a range of files as described above. Would be identical to
* -R /path/to/source1/first_file:last_file -R
/path/to/source2/first_file:last_file [-R .. ]
*
* -M /path/to/source1:source2[:..] -R .
* For each expanded directory specified by -M /path/to/source1:source2
* select all files of the directory as described above. Would be to
* -R /path/to/source1 -R /path/to/source2 [-R ...]
*
*
* Hierarchical file organinisation:
* For performance reasons, files may be store in various sub directries
instead of a
* single directory. These sub directories are assumed to be created in
alpabetical order.
* For example daily sub directories: 2006/04/01 .. 2006/04/30 as created by
nfcapd with
* option -S %y/%m/%d
*
* Single file selection is identical to the flat file layout:
* -r [/]path/to/sub1/sub2/sub3/single_file
*
* Selecting a range of files in a hierarchical file layout
* --------------------------------------------------------
* -R [/]path/to/sub1/sub2/first_file
* Select a range of files in directory specified by absolut or relative path
* [/]path/to/sub1/sub2/. Files are selected in alphabetical order starting
with
* 'first_file' to the end of the directory. The hierarchy has no impact here.
*
* -R [/]path/to/first_sub1/first_sub2/first_file:last_sub1/last_sub2/last_file
* Select a range of files over multiple sub directories starting at absolut
or
* relative path [/]path/to/first_sub1/first_sub2/first_file up to and
including
* [/]path/to/last_sub1/last_sub2/last_file. Files are selected in
alphabetical
* order by iterating over the required sub directory hierachy
* Example:
* -R
/path/to/2006/03/31/nfcapd.200603312300:2006/04/01/nfcapd.200604010600
*
* -R [/]path/to/directory
* Select all files in alphabetical order in directory specified by
absolut or relative
* path [/]path/to/directory, identical to flat layout
*
* The same methode applies for selecting a range of files over multiple sub
directories
* and multiple sources.
*
* Example:
* -M /path/to/source1:source2 -R
2006/03/31/nfcapd.200603312300:2006/04/01/nfcapd.200604010600
*
*/
/*
* syntax for possible sub dir definitions:
*
* %Y is replaced by the year with century as a decimal number.
* %y is replaced by the year without century as a decimal number (00-99).
* %m is replaced by the month as a decimal number (01-12).
* %d is replaced by the day of the month as a decimal number (01-31).
* %j is replaced by the day of the year as a decimal number (001-366).
* %H is replaced by the hour (24-hour clock) as a decimal number (00-23).
* %M is replaced by the minute as a decimal number (00-59).
* %s is replaced by the number of seconds since the Epoch, UTC
* %U is replaced by the week number of the year (Sunday as the first day
* of the week) as a decimal number (00-53).
* %W is replaced by the week number of the year (Monday as the first day
* of the week) as a decimal number (00-53).
* %w is replaced by the weekday (Sunday as the first day of the week) as
* a decimal number (0-6).
* %u is replaced by the weekday (Monday as the first day of the week) as
* a decimal number (1-7).
* %F is equivalent to ``%Y-%m-%d''.
*/
// predefined and accpeted formats
static const char *subdir_def[] = {
"", // default index 0 - no subdir hierarchy
"%Y/%m/%d",
"%Y/%m/%d/%H",
"%Y/%W/%u",
"%Y/%W/%u/%H",
"%Y/%j",
"%Y/%j/%H",
"%F",
"%F/%H",
NULL
};
// all accpeted char in a string
#define AcceptedFormatChar "YymdjHMsUWwuF"
static mode_t mode, dir_mode;
static const char *subdir_format;
static struct entry_filter_s {
char *first_entry;
char *last_entry;
int list_files;
} *dir_entry_filter;
#define NUM_PTR 16
static char *first_file, *last_file, current_file[255];
static uint32_t twin_first, twin_last;
static stringlist_t source_dirs, file_list;
/* Function prototypes */
static inline int CheckTimeWindow(uint32_t t_start, uint32_t t_end,
stat_record_t *stat_record);
static void GetFileList(char *path);
static void CleanPath(char *entry);
static void Getsource_dirs(char *dirs);
static int mkpath(char *path, mode_t mode, mode_t dir_mode, char *error, size_t
errlen);
static char *GuessSubDir(char *channeldir, char *filename);
static char *VerifyFileRange(char *path, char *last_file);
/* Functions */
static int compare(const FTSENT **f1, const FTSENT **f2) {
return strcmp( (*f1)->fts_name, (*f2)->fts_name);
} // End of compare
static void CleanPath(char *entry) {
char *p, *q;
size_t len;
// wash out any '//' in entry
while ( (p = strstr(entry, "//")) != NULL ) {
p++;
q = p+1; // q points to first char after '//'
while ( *p )
*p++ = *q++;
}
// remove trailing '/'
len = strlen(entry);
if ( entry[len-1] == '/' )
entry[len-1] = '\0';
// wash out any '/./' in entry
while ( (p = strstr(entry, "/./")) != NULL ) {
p++;
q = p+2; // q points to first char after '/./'
while ( *p )
*p++ = *q++;
}
// remove leading './' in entry
if ( strstr(entry, "./") == entry ) {
p = entry;
q = p + 2;
while ( *p )
*p++ = *q++;
}
} // End of CleanPath
static inline int CheckTimeWindow(uint32_t t_start, uint32_t t_end,
stat_record_t *stat_record) {
/*
printf("t start %u %s", t_start, ctime(&t_start));
printf("t end %u %s", t_end, ctime(&t_end));
printf("f start %u %s", NetflowStat.first_seen,
ctime(&NetflowStat.first_seen));
printf("f end %u %s", NetflowStat.last_seen,
ctime(&NetflowStat.last_seen));
*/
// if no time window is set, return true
if ( t_start == 0 )
return 1;
if ( stat_record->first_seen == 0 )
return 0;
if ( t_start >= stat_record->first_seen && t_start <=
stat_record->last_seen )
return 1;
if ( t_end >= stat_record->first_seen && t_end <=
stat_record->last_seen )
return 1;
if ( t_start < stat_record->first_seen && t_end >
stat_record->last_seen )
return 1;
return 0;
} // End of CheckTimeWindow
// file filter for scandir function
static int dirlevels(char *dir) {
int num;
if ( !dir )
return 0;
num = 0;
if ( dir[0] == '/' )
dir++;
while ( *dir ) {
if ( *dir == '/' )
num++;
dir++;
}
return num;
} // End of dirlevels
static void CreateDirListFilter(char *first_path, char *last_path, int
file_list_level) {
int i;
char *p, *q, *first_mark, *last_mark;
// printf("First Dir: '%s', first_path: '%s', last_path '%s', first_file
'%s', last_file '%s', list_level: %i\n",
// source_dirs.list[0], first_path, last_path, first_file,
last_file, file_list_level);
if ( file_list_level == 0 )
return;
if ( file_list_level < 0 ) {
fprintf(stderr, "software error in %s line %d: %s\n", __FILE__,
__LINE__, strerror(errno) );
exit(250);
}
dir_entry_filter = (struct entry_filter_s *)malloc((file_list_level+1)
* sizeof(struct entry_filter_s));
if ( !dir_entry_filter ) {
fprintf(stderr, "malloc() error in %s line %d: %s\n", __FILE__,
__LINE__, strerror(errno) );
exit(250);
}
// first default entry - the directory itself
dir_entry_filter[0].first_entry = NULL;
dir_entry_filter[0].last_entry = NULL;
dir_entry_filter[0].list_files = 0;
first_mark = first_path;
last_mark = last_path;
// intermediate directory level filters
for ( i=1; i<file_list_level; i++ ) {
if ( first_mark ) {
p = strchr(first_mark, '/');
if ( p ) {
*p = '\0';
dir_entry_filter[i].first_entry =
strdup(first_path);
*p++ = '/';
first_mark = p;
} else {
dir_entry_filter[i].first_entry =
strdup(first_path);
first_mark = NULL;
}
} else {
dir_entry_filter[i].first_entry = NULL;
}
dir_entry_filter[i].list_files = 0;
if ( last_mark ) {
q = strchr(last_mark, '/');
if ( q ) {
*q = '\0';
dir_entry_filter[i].last_entry =
strdup(last_path);
*q++ = '/';
last_mark = q;
} else {
dir_entry_filter[i].last_entry =
strdup(last_path);
last_mark = NULL;
}
} else {
dir_entry_filter[i].last_entry = NULL;
}
if ( dir_entry_filter[i].first_entry &&
dir_entry_filter[i].last_entry &&
strcmp(dir_entry_filter[i].first_entry,
dir_entry_filter[i].last_entry) > 0 )
fprintf(stderr, "WARNING: Entry '%s' > '%s'. Will not
match anything!\n",
dir_entry_filter[i].first_entry,
dir_entry_filter[i].last_entry);
// printf("%i first: '%s', last: '%s'\n",
// i, dir_entry_filter[i].first_entry,
dir_entry_filter[i].last_entry);
}
// the last level - files are listed here
dir_entry_filter[file_list_level].first_entry = first_file;
dir_entry_filter[file_list_level].last_entry = last_file;
dir_entry_filter[file_list_level].list_files = 1;
if ( dir_entry_filter[file_list_level].first_entry &&
dir_entry_filter[file_list_level].last_entry &&
strcmp(dir_entry_filter[file_list_level].first_entry,
dir_entry_filter[file_list_level].last_entry) > 0 )
fprintf(stderr, "WARNING: File '%s' > '%s'. Will not match
anything!\n",
dir_entry_filter[file_list_level].first_entry,
dir_entry_filter[file_list_level].last_entry);
// printf("%i first: '%s', last: '%s'\n",
// file_list_level, dir_entry_filter[file_list_level].first_entry,
dir_entry_filter[file_list_level].last_entry);
} // End of CreateDirListFilter
static void GetFileList(char *path) {
struct stat stat_buf;
char *last_file_ptr, *first_path, *last_path;
int levels_first_file, levels_last_file, file_list_level;
int sub_index;
FTS *fts;
FTSENT *ftsent;
CleanPath(path);
// Check for last_file option
last_file_ptr = strchr(path, ':');
first_path = last_path = NULL;
levels_first_file = levels_last_file = 0;
if ( last_file_ptr ) {
// make sure we have only a single ':' in path
if ( strrchr(path, ':') != last_file_ptr ) {
fprintf(stderr, "Multiple file separators ':' in path
not allowed!\n");
exit(250);
}
*last_file_ptr++ = '\0';
// last_file_ptr points to last_file
if ( strlen(last_file_ptr) == 0 ) {
fprintf(stderr, "Missing last file option after
':'!\n");
exit(250);
}
CleanPath(last_file_ptr);
// make sure last_file option is not a full path
if ( last_file_ptr[0] == '/') {
fprintf(stderr, "Last file name in -R list must not
start with '/'\n");
exit(250);
}
// how may sub dir levels has last_file option?
levels_last_file = dirlevels(last_file_ptr);
// if no subdirs are given for last_file, try to find out, if
the last_file
if ( levels_last_file == 0 ) {
char s[MAXPATHLEN];
char *r = VerifyFileRange(path, last_file_ptr);
if ( r != last_file_ptr ) {
snprintf(s, MAXPATHLEN-1, "%s/%s", r,
last_file_ptr);
s[MAXPATHLEN-1] = '\0';
last_file_ptr = strdup(s);
levels_last_file = dirlevels(last_file_ptr);
}
}
}
levels_first_file = dirlevels(path);
if ( source_dirs.num_strings == 0 ) {
// No multiple sources option -M
// path contains the path to a file/directory
// stat this entry
if ( stat(path, &stat_buf) ) {
fprintf(stderr, "stat() error '%s': %s\n", path,
strerror(errno));
exit(250);
}
if ( !S_ISDIR(stat_buf.st_mode) && !S_ISREG(stat_buf.st_mode) )
{
fprintf(stderr, "Not a file or directory: '%s'\n",
path);
exit(250);
}
// Check, how many levels of directory in path
levels_first_file = dirlevels(path);
if ( last_file_ptr ) {
// path is [/]path/to/any/dir|file:last_file_ptr
// make sure first_file is a file
if ( S_ISDIR(stat_buf.st_mode) ) {
fprintf(stderr, "Not a file: '%s'\n", path);
exit(250);
}
if ( levels_last_file ) {
// we have levels_last_file number of sub dirs
// sub dir levels of first_file mus have at
least the same number of levels as last_file
if ( levels_first_file < levels_last_file ) {
fprintf(stderr, "Number of sub dirs for
sub level hierarchy for file list -R do not match\n");
exit(250);
}
if ( levels_first_file == levels_last_file ) {
char *p, *q;
// path =
[/]sub1[/..]/first_file:sub1[/...]/last_file
if ( path[0] == '/' ) {
// this is rather strange, but
strctly spoken, valid anyway
InsertString(&source_dirs, "/");
path++;
} else {
InsertString(&source_dirs, ".");
}
// path =
sub_first[/..]/first_file:sub_last[/...]/last_file
p = strrchr(path, '/');
q = strrchr(last_file_ptr, '/');
if ( !p || !q ) {
// this should never happen
fprintf(stderr, "software error
in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
exit(250);
}
*p++ = '\0';
*q++ = '\0';
first_file = strdup(p);
last_file = strdup(q);
file_list_level = levels_last_file + 1;
first_path = path;
last_path = last_file_ptr;
} else {
// path =
[/]path/to/sub_first[/..]/first_file:sub_last[/...]/last_file
int i;
char *p, *r, *s;
p = strrchr(path, '/');
// levels_first_file > levels_last_file
// step back the number of sub dirs in
first_file
for ( i=0; i<levels_last_file; i++ ) {
do {
p--;
} while ( p >= path && *p !=
'/');
}
*p++ = '\0';
InsertString(&source_dirs, path);
r = strrchr(p, '/');
s = strrchr(last_file_ptr, '/');
if ( !r || !s ) {
// this must never happen
fprintf(stderr, "software error
in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
exit(250);
}
*r++ = '\0';
*s++ = '\0';
first_file = strdup(r);
last_file = strdup(s);
// files are listed at this sub dir
level
file_list_level = levels_last_file + 1;
first_path = p;
last_path = last_file_ptr;
}
} else {
// we have no sub dir levels given
// path is [/]path/to/any/file
char *p = strrchr(path, '/');
if ( p ) {
// path is
[/]path/to/any/first_file:last_file
*p++ = '\0';
// path is the direcory containing all
the files
InsertString(&source_dirs, path);
first_file = strdup(p);
} else {
// path is first_file:last_file
InsertString(&source_dirs, ".");
first_file = strdup(path);
}
// set last_file filter
last_file = strdup(last_file_ptr);
// in any case we list the files of directory
level 1
file_list_level = 1;
}
} else {
// path is [/]path/to/any/dir|file
if ( S_ISDIR(stat_buf.st_mode) ) {
// path is [/]path/to/any/dir
// list all files in this directory
InsertString(&source_dirs, path);
first_file = NULL;
file_list_level = 0;
} else {
// path is [/]path/to/any/file
char *p = strrchr(path, '/');
if ( p ) {
// path is [/]path/to/any/file
*p++ = '\0';
// path is the direcory containing all
the files
InsertString(&source_dirs, path);
first_file = strdup(p);
} else {
// path is file
InsertString(&source_dirs, ".");
first_file = strdup(path);
}
// in any case we list the files of directory
level 1
file_list_level = 1;
}
// in any case, no last_file filter
last_file = NULL;
}
} else {
char pathbuff[MAXPATHLEN];
// multiple sources option -M given
if ( path[0] == '/') {
fprintf(stderr, "File list -R must not start with '/'
when combined with a source list -M\n");
exit(250);
}
// special case for all files in directory
if ( strcmp(path, ".") == 0 ) {
first_file = NULL;
last_file = NULL;
file_list_level = 0;
} else {
// pathbuff contains the path to a file/directory,
compiled using the first entry
// in the source_dirs
snprintf(pathbuff, MAXPATHLEN-1, "%s/%s",
source_dirs.list[0], path);
pathbuff[MAXPATHLEN-1] = '\0';
// pathbuff must point to a file
if ( stat(pathbuff, &stat_buf) ) {
if ( errno == ENOENT ) {
// file not found - try to guess a
possible subdir
char *sub_dir =
GuessSubDir(source_dirs.list[0], path);
if ( sub_dir ) { // subdir found
snprintf(pathbuff,
MAXPATHLEN-1, "%s/%s", sub_dir, path);
pathbuff[MAXPATHLEN-1] = '\0';
// update path
path = strdup(pathbuff);
free(sub_dir);
// need guessing subdir with
last_file too
if ( last_file_ptr ) {
sub_dir =
GuessSubDir(source_dirs.list[0], last_file_ptr);
if ( sub_dir ) {
// subdir found
snprintf(pathbuff, MAXPATHLEN-1, "%s/%s", sub_dir, last_file_ptr);
pathbuff[MAXPATHLEN-1] = '\0';
last_file_ptr =
strdup(pathbuff);
free(sub_dir);
// update dir
levels of extended file path
levels_last_file = dirlevels(last_file_ptr);
} else {
fprintf(stderr,
"'%s': %s\n", last_file_ptr, "File not found!");
exit(250);
}
}
} else { // no file in any
possible subdir found
fprintf(stderr, "stat() error
'%s': %s\n", pathbuff, "File not found!");
exit(250);
}
} else { // Any other stat error
fprintf(stderr, "stat() error '%s':
%s\n", pathbuff, strerror(errno));
exit(250);
}
} else if ( !S_ISREG(stat_buf.st_mode) ) {
fprintf(stderr, "Not a file : '%s'\n",
pathbuff);
exit(250);
}
// Check, how many levels of directory in path
levels_first_file = dirlevels(path);
if ( last_file_ptr ) {
// path is path/to/any/first_file:last_file_ptr
char *p, *q;
// the number of sub dirs must be eqal for
first_file and last_file
if ( levels_first_file != levels_last_file ) {
fprintf(stderr, "Number of sub dirs
must agree in '%s' and '%s'\n", path, last_file_ptr);
exit(250);
}
p = strrchr(path, '/');
if ( p ) {
// path is fist_sub/to/any/first_file
// recursive all files in sub dirs
file_list_level = dirlevels(path) + 1;
*p++ = '\0';
first_file = strdup(p);
first_path = path;
} else {
// path is first_file
first_file = strdup(path);
file_list_level = 1;
}
q = strrchr(last_file_ptr, '/');
if ( q ) {
*q++ = '\0';
last_file = strdup(q);
last_path = last_file_ptr;
} else {
last_file = strdup(last_file_ptr);
}
} else {
// path is path/to/any/first_file
char *p = strrchr(path, '/');
if ( p ) {
// path is fist_sub/to/any/first_file
// recursive all files in sub dirs
file_list_level = dirlevels(path) + 1;
*p++ = '\0';
first_file = strdup(p);
first_path = path;
} else {
// path is first_file
first_file = strdup(path);
file_list_level = 1;
}
last_file = NULL;
}
}
}
/*
printf("first_file %s\n", first_file ? first_file : "<none>");
printf("last_file %s\n", last_file ? last_file : "<none>");
printf("first_path %s\n", first_path ? first_path : "<none>");
printf("last_path %s\n", last_path ? last_path : "<none>");
printf("file_list_level: %i\n", file_list_level);
*/
CreateDirListFilter(first_path, last_path, file_list_level );
// last entry must be NULL
InsertString(&source_dirs, NULL);
fts = fts_open(source_dirs.list, FTS_LOGICAL, compare);
sub_index = 0;
while ( (ftsent = fts_read(fts)) != NULL) {
int fts_level = ftsent->fts_level;
char *fts_path;
// printf("DBG: %u %i %s %s\n", ftsent->fts_info, ftsent->fts_level,
ftsent->fts_path, ftsent->fts_name);
if ( fts_level == 0 ) {
sub_index = ftsent->fts_pathlen + 1;
continue;
}
if ( ftsent->fts_pathlen < sub_index ) {
printf("ERROR: fts_pathlen error at %s line %d\n",
__FILE__, __LINE__);
exit(250);
}
fts_path = &ftsent->fts_path[sub_index];
/*
if ( file_list_level )
printf("DGB: short fts: '%s', filer_first: '%s', filter_last: '%s'\n",
fts_path,
dir_entry_filter[fts_level].first_entry ,
dir_entry_filter[fts_level].last_entry);
*/
switch (ftsent->fts_info) {
case FTS_D:
// dir entry pre descend
if ( file_list_level && file_list_level && (
(
dir_entry_filter[fts_level].first_entry &&
( strcmp(fts_path,
dir_entry_filter[fts_level].first_entry ) < 0 ) ) ||
(
dir_entry_filter[fts_level].last_entry &&
( strcmp(fts_path,
dir_entry_filter[fts_level].last_entry ) > 0 ) )
))
fts_set(fts, ftsent, FTS_SKIP );
break;
case FTS_DP:
break;
case FTS_F:
// file entry
// printf("==> Check: %s\n", ftsent->fts_name);
// skip stat file
if ( strcmp(ftsent->fts_name, ".nfstat") == 0 ||
strncmp(ftsent->fts_name,
"nfcapd.current", 14) == 0)
continue;
if ( strstr(ftsent->fts_name, ".stat") != NULL )
continue;
if ( file_list_level && (
( fts_level != file_list_level ) ||
(
dir_entry_filter[fts_level].first_entry &&
( strcmp(ftsent->fts_name,
dir_entry_filter[fts_level].first_entry) < 0 ) ) ||
(
dir_entry_filter[fts_level].last_entry &&
( strcmp(ftsent->fts_name,
dir_entry_filter[fts_level].last_entry) > 0 ) )
) )
continue;
// printf("==> Listed: %s\n", ftsent->fts_path);
InsertString(&file_list, ftsent->fts_path);
break;
}
}
fts_close(fts);
} // End of GetFileList
/*
* Get the list of directories
* dirs: user supplied parameter: /any/path/dir1:dir2:dir3:...
* source_dirs must result in
* /any/path/dir1
* /any/path/dir2
* /any/path/dir3
* /any/path is dir prefix, which may be NULL e.g. dir1:dir2:dir3:...
* dir1, dir2 etc entrys
*/
void Getsource_dirs(char *dirs) {
struct stat stat_buf;
char *p, *q, *dirprefix;
char path[MAXPATHLEN];
q = strchr(dirs, ':');
if ( q ) { // we have /path/to/firstdir:dir1:dir2:...
*q = 0;
p = strrchr(dirs, '/');
if ( p ) {
*p++ = 0; // p points now to the first name in
the dir list
dirprefix = dirs;
} else { // we have a source_dirs in current directory
p = dirs; // p points now to the first name in
the dir list
dirprefix = "."; // current directory
}
*q = ':'; // restore ':' in source_dirs
while ( p ) { // iterate over all elements in the dir list
q = strchr(p, ':');
if ( q )
*q = 0;
// p point to a dir name
snprintf(path, 1023, "%s/%s", dirprefix, p);
path[MAXPATHLEN-1] = 0;
if ( stat(dirs, &stat_buf) ) {
fprintf(stderr, "Can't stat '%s': %s\n", path,
strerror(errno));
return;
}
if ( !S_ISDIR(stat_buf.st_mode) ) {
fprintf(stderr, "Not a directory: '%s'\n",
path);
return;
}
// save path into source_dirs
InsertString(&source_dirs, path);
p = q ? q + 1 : NULL;
}
} else { // we have only one directory
if ( stat(dirs, &stat_buf) ) {
fprintf(stderr, "Can't stat '%s': %s\n", dirs,
strerror(errno));
return;
}
if ( !S_ISDIR(stat_buf.st_mode) ) {
fprintf(stderr, "Not a directory: '%s'\n", dirs);
return;
}
// save the path into source_dirs
InsertString(&source_dirs, dirs);
}
} // End of Getsource_dirs
void SetupInputFileSequence(char *multiple_dirs, char *single_file, char
*multiple_files) {
char *error;
int fd;
twin_first = 0;
twin_last = 0xffffffff;
first_file = NULL;
last_file = NULL;
InitStringlist(&source_dirs, NUM_PTR);
InitStringlist(&file_list, 64);
if ( multiple_dirs )
Getsource_dirs(multiple_dirs);
if ( multiple_files ) {
// use multiple files
GetFileList(multiple_files);
// get time window spanning all the files
if ( file_list.num_strings ) {
stat_record_t *stat_ptr;
fd = OpenFile(file_list.list[0], &stat_ptr, &error);
// read the stat record
if ( error != NULL ) {
fprintf(stderr, "%s\n", error);
exit(250);
}
close(fd);
twin_first = stat_ptr->first_seen;
fd = OpenFile(file_list.list[file_list.num_strings-1],
&stat_ptr, &error); // read the stat record of last file
if ( error != NULL ) {
fprintf(stderr, "%s\n", error);
exit(250);
}
close(fd);
twin_last = stat_ptr->last_seen;
}
} else if ( single_file ) {
CleanPath(single_file);
if ( source_dirs.num_strings == 0 ) {
InsertString(&file_list, single_file);
} else {
int i;
if ( single_file[0] == '/' ) {
fprintf(stderr, "File -r must not start with
'/', when combined with a source list -M\n");
exit(250);
}
for ( i=0; i<source_dirs.num_strings; i++ ) {
char s[MAXPATHLEN];
struct stat stat_buf;
snprintf(s, MAXPATHLEN-1, "%s/%s",
source_dirs.list[i], single_file);
s[MAXPATHLEN-1] = '\0';
if ( stat(s, &stat_buf) ) {
if ( errno == ENOENT ) {
// file not found - try to
guess subdir
char *sub_dir =
GuessSubDir(source_dirs.list[i], single_file);
if ( sub_dir ) { //
subdir found
snprintf(s,
MAXPATHLEN-1, "%s/%s/%s", source_dirs.list[i], sub_dir, single_file);
s[MAXPATHLEN-1] = '\0';
InsertString(&file_list, s);
} else { // no subdir
found
fprintf(stderr, "stat()
error '%s': %s\n", s, "File not found!");
}
} else { // Any other stat error
fprintf(stderr, "stat() error
'%s': %s\n", s, strerror(errno));
exit(250);
}
} else { // stat() successful
if ( !S_ISREG(stat_buf.st_mode) ) {
fprintf(stderr, "Skip non file
entry: '%s'\n", s);
} else {
InsertString(&file_list, s);
}
}
}
}
} else // else use stdin
InsertString(&file_list, NULL);
} // End of SetupInputFileSequence
char *GetCurrentFilename(void) {
return current_file;
} // End of GetCurrentFilename
int GetNextFile(int current, time_t twin_start, time_t twin_end, stat_record_t
**stat_record) {
stat_record_t *stat_ptr;
char *error;
int fd;
static int cnt;
// is it first time init ?
if ( current < 0 ) {
cnt = 0;
}
// close current file before open the next one
// stdin ( current = 0 ) is not closed
if ( current > 0 ) {
close(current);
current_file[0] = '\0';
}
// no or no more files available
if ( file_list.num_strings == cnt ) {
if ( stat_record )
*stat_record = NULL;
return EMPTY_LIST;
}
/*
while ( cnt < file_list.num_strings ) {
printf("Process: '%s'\n", file_list.list[cnt++]);
}
*/
while ( cnt < file_list.num_strings ) {
fd = OpenFile(file_list.list[cnt], &stat_ptr, &error); // Open
the file
cnt++;
// stdin
if ( fd == STDIN_FILENO ) {
if ( stat_record )
*stat_record = NULL;
CurrentIdent = "none";
return fd;
}
if ( fd > 0 && CheckTimeWindow(twin_start, twin_end, stat_ptr)
) {
// printf("Return file: %s\n", string);
if ( stat_record )
*stat_record = stat_ptr;
return fd;
}
close(fd);
if ( error != NULL )
fprintf(stderr, "%s\n", error);
}
if ( stat_record )
*stat_record = NULL;
return EMPTY_LIST;
} // End of GetNextFile
int InitHierPath(int num) {
int i;
subdir_format = NULL;
i=0;
while ( subdir_def[i] != NULL ) {
if ( i == num )
break;
i++;
}
if ( subdir_def[i] == NULL ) {
fprintf(stderr, "No such subdir level %i\n", num);
return 0;
}
subdir_format = subdir_def[i];
/*
* The default file mode is a=rwx (0777) with selected permissions
* removed in accordance with the file mode creation mask. For
* intermediate path name components, the mode is the default modified
* by u+wx so that the subdirectories can always be created.
*/
// get umask
mode = umask(0);
umask(mode);
mode = 0777 & ~mode;
dir_mode = mode | S_IWUSR | S_IXUSR;
return 1;
} // End of InitHierPath
static char *VerifyFileRange(char *path, char *last_file) {
char *p, *q, *r;
r = strdup(path);
p = strrchr(r, '/');
while ( p ) {
*p = '\0';
q = GuessSubDir(r, last_file);
if ( q ) {
free(r);
return q;
}
p = strrchr(r, '/');
}
free(r);
return last_file;
} // End of VerifyFileRange
static char *GuessSubDir(char *channeldir, char *filename) {
char s[MAXPATHLEN];
struct tm *t_tm;
int i;
if ( strlen(filename) == 19 && (strncmp(filename, "nfcapd.", 7) == 0) )
{
char *p = &filename[7];
time_t t = ISO2UNIX(p);
t_tm = localtime(&t);
} else
return NULL;
i = 0;
// if the file exists, it must be in any of the possible subdirs
// so try one after the next - one will match
while ( subdir_def[i] ) {
char const *sub_fmt = subdir_def[i];
char subpath[255];
struct stat stat_buf;
strftime(subpath, 254, sub_fmt, t_tm);
subpath[254] = '\0';
snprintf(s, MAXPATHLEN-1, "%s/%s/%s", channeldir, subpath,
filename);
if ( stat(s, &stat_buf) == 0 && S_ISREG(stat_buf.st_mode) ) {
// found file in subdir
return strdup(subpath);
}
i++;
}
return NULL;
} // End of GuessSubDir
char *SetupSubDir(struct tm *now, char *error, size_t errlen ) {
static char subpath[255];
struct stat stat_buf;
size_t len;
int err;
error[0] = '\0';
len = strftime(subpath, 254, subdir_format, now);
if ( len == 0 ) {
return NULL;
}
// our cwd is basedir ( -l ) so test if, dir exists
if ( stat(subpath, &stat_buf) == 0 ) {
if ( S_ISDIR(stat_buf.st_mode) ) {
// sub directory already exists
return subpath;
} else {
// an eentry with this name exists, but it's not a
directory
snprintf(error, errlen, "Path '%s': %s ", subpath,
strerror(ENOTDIR));
return NULL;
}
}
// no such entry exists - try to create the directory, assuming path
below exists
err = mkdir(subpath, dir_mode);
if ( err == 0 ) // success
return subpath;
// else errno is set
if ( errno == ENOENT ) { // we need to create intermediate directories
as well
err = mkpath(subpath, mode, dir_mode, error, errlen);
if ( err == 0 ) // creation was successful
return subpath;
} else {
snprintf(error, errlen, "mkdir() error for '%s': %s\n",
subpath, strerror(errno));
}
// anything else failed and error string is set
return NULL;
} // End of SetupSubDir
/*
* mkpath -- create directories.
* path - path
* mode - file mode of terminal directory
* dir_mode - file mode of intermediate directories
*/
static int mkpath(char *path, mode_t mode, mode_t dir_mode, char *error, size_t
errlen) {
struct stat sb;
char *slash;
int done = 0;
slash = path;
while (!done) {
slash += strspn(slash, "/");
slash += strcspn(slash, "/");
done = (*slash == '\0');
*slash = '\0';
if (stat(path, &sb)) {
if (errno != ENOENT || (mkdir(path, done ? mode : dir_mode) &&
errno != EEXIST)) {
snprintf(error, errlen, "mkdir() error for
'%s': %s\n", path, strerror(errno));
return (-1);
}
} else if (!S_ISDIR(sb.st_mode)) {
snprintf(error, errlen, "Path '%s': %s ", path,
strerror(ENOTDIR));
return (-1);
}
*slash = '/';
}
return (0);
} // End of mkpath
pgpMFeyvtvEjT.pgp
Description: PGP signature
------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
_______________________________________________ Nfsen-discuss mailing list Nfsen-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
