Re: [netsniff-ng] Synchronize file creation between netsniff-ng processes possible?

[Tobias Klauser]

On 2016-01-28 at 22:04:45 +0100, e.bengts...@gmail.com  
wrote:
> Sounds good. Thanks!

Patch now applied, thanks a lot Erik!

Tobias

-- 
You received this message because you are subscribed to the Google Groups 
"netsniff-ng" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to netsniff-ng+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [netsniff-ng] Synchronize file creation between netsniff-ng processes possible?

[Tobias Klauser]

On 2016-01-28 at 13:17:30 +0100, e.bengts...@gmail.com  
wrote:
> 
> 
> Den onsdag 27 januari 2016 kl. 18:02:23 UTC+1 skrev e.ben...@gmail.com:
> >
> >
> >
> > Den onsdag 27 januari 2016 kl. 16:17:46 UTC+1 skrev Tobias Klauser:
> >>
> >> Hi again 
> >>
> >> On 2016-01-27 at 16:10:30 +0100, Tobias Klauser  
> >> wrote: 
> >> > On 2016-01-27 at 14:40:55 +0100, Erik Bengtsson  
> >> wrote: 
> >> > > By "start time" I mean file creation time. It is totally possible to 
> >> post 
> >> > > process all log files and synchronize them using pcap timestamps, as 
> >> you 
> >> > > suggested, but I'm afraid that it will be less effective for us since 
> >> there 
> >> > > will be a huge amount of data to move around and process. 
> >> > 
> >> > Ah ok, I see. Thanks for clarifying. 
> >> > 
> >> > In that case, I'm afraid there currently is no method to synchronize 
> >> the file 
> >> > creation among multiple instances of netsniff-ng. As Daniel suggested, 
> >> > timerfd might be an option to implement a feature along these lines. If 
> >> > you want, feel free to have a look into it - patches are gladly 
> >> accepted 
> >> > :) 
> >>
> >> Something which I completely forgot about... 
> >>
> >> There might be an option (though a bit curde) to solve this using the 
> >> permature rotation caused by SIGHUP. You could set up a separate task 
> >> simultaneously sending a SIGHUP every minute to your netsniff-ng 
> >> processes. See commit 46289df6bc8f573bc01be4fb4aa93343ecc6d50a 
> >> ("netsniff-ng: Rotate pcap files prematurely on SIGHUP") for details. 
> >>
> >> Tobias 
> >>
> >
> > That sounds really interesting! :-)
> >
> > I ended up using signal USR2 since i didn't want to change how SIGHUP was 
> > used. When USR2 is received, the current time is saved and used when naming 
> > the next file (which is created when the next packet is received).
> >
> > A patch is included if you want to have a look.
> >
> > / Erik
> >  
> >
> 
> I've continued working on this a bit and have a solution that seems to be 
> working. The patch is included if someone wants to have a look.

Nice, thanks a lot. I think we don't even need to have a command line
option for this but could just make it the default behavior to record
the timestamp of the sighup in the filename. The file creation time will
still reflect the actual time of when the first packet arrived.

I'll adjust the patch accordingly and commit it with your Signed-off-by,
if that's ok with you.

Thanks
Tobias

-- 
You received this message because you are subscribed to the Google Groups 
"netsniff-ng" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to netsniff-ng+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [netsniff-ng] Synchronize file creation between netsniff-ng processes possible?

[e . bengtsson]

Den onsdag 27 januari 2016 kl. 18:02:23 UTC+1 skrev e.ben...@gmail.com:
>
>
>
> Den onsdag 27 januari 2016 kl. 16:17:46 UTC+1 skrev Tobias Klauser:
>>
>> Hi again 
>>
>> On 2016-01-27 at 16:10:30 +0100, Tobias Klauser  
>> wrote: 
>> > On 2016-01-27 at 14:40:55 +0100, Erik Bengtsson  
>> wrote: 
>> > > By "start time" I mean file creation time. It is totally possible to 
>> post 
>> > > process all log files and synchronize them using pcap timestamps, as 
>> you 
>> > > suggested, but I'm afraid that it will be less effective for us since 
>> there 
>> > > will be a huge amount of data to move around and process. 
>> > 
>> > Ah ok, I see. Thanks for clarifying. 
>> > 
>> > In that case, I'm afraid there currently is no method to synchronize 
>> the file 
>> > creation among multiple instances of netsniff-ng. As Daniel suggested, 
>> > timerfd might be an option to implement a feature along these lines. If 
>> > you want, feel free to have a look into it - patches are gladly 
>> accepted 
>> > :) 
>>
>> Something which I completely forgot about... 
>>
>> There might be an option (though a bit curde) to solve this using the 
>> permature rotation caused by SIGHUP. You could set up a separate task 
>> simultaneously sending a SIGHUP every minute to your netsniff-ng 
>> processes. See commit 46289df6bc8f573bc01be4fb4aa93343ecc6d50a 
>> ("netsniff-ng: Rotate pcap files prematurely on SIGHUP") for details. 
>>
>> Tobias 
>>
>
> That sounds really interesting! :-)
>
> I ended up using signal USR2 since i didn't want to change how SIGHUP was 
> used. When USR2 is received, the current time is saved and used when naming 
> the next file (which is created when the next packet is received).
>
> A patch is included if you want to have a look.
>
> / Erik
>  
>

I've continued working on this a bit and have a solution that seems to be 
working. The patch is included if someone wants to have a look.

Best regards, Erik



 

-- 
You received this message because you are subscribed to the Google Groups 
"netsniff-ng" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to netsniff-ng+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
>From 02ea1c96cce854f0691d18ba782113006cb61532 Mon Sep 17 00:00:00 2001
From: Erik Bengtsson 
Date: Thu, 28 Jan 2016 13:01:41 +0100
Subject: [PATCH] Added support for using SIGHUP's time when rotating files
 prematurely

---
 netsniff-ng.c | 26 +-
 1 file changed, 21 insertions(+), 5 deletions(-)

diff --git a/netsniff-ng.c b/netsniff-ng.c
index e5d1886..ccce27a 100644
--- a/netsniff-ng.c
+++ b/netsniff-ng.c
@@ -58,7 +58,7 @@ struct ctx {
 	int cpu, rfraw, dump, print_mode, dump_dir, packet_type;
 	unsigned long kpull, dump_interval, tx_bytes, tx_packets;
 	size_t reserve_size;
-	bool randomize, promiscuous, enforce, jumbo, dump_bpf, hwtimestamp, verbose;
+	bool randomize, promiscuous, enforce, jumbo, dump_bpf, hwtimestamp, verbose, pre_timestamp;
 	enum pcap_ops_groups pcap;
 	enum dump_mode dump_mode;
 	uid_t uid;
@@ -71,9 +71,10 @@ struct ctx {
 
 static volatile sig_atomic_t sigint = 0, sighup = 0;
 static volatile bool next_dump = false;
+static volatile sig_atomic_t sighup_time = 0;
 
 static const char *short_options =
-	"d:i:o:rf:MNJt:S:k:n:b:HQmcsqXlvhF:RGAP:Vu:g:T:DBUC:K:L:w";
+	"d:i:o:rf:MNJt:S:k:n:b:HQmcsqXlvhF:RGAP:pVu:g:T:DBUC:K:L:w";
 static const struct option long_options[] = {
 	{"dev",			required_argument,	NULL, 'd'},
 	{"in",			required_argument,	NULL, 'i'},
@@ -92,6 +93,7 @@ static const struct option long_options[] = {
 	{"fanout-group",	required_argument,	NULL, 'C'},
 	{"fanout-type",		required_argument,	NULL, 'K'},
 	{"fanout-opts",		required_argument,	NULL, 'L'},
+	{"pre-timestamp",	no_argument,		NULL, 'p'},
 	{"rand",		no_argument,		NULL, 'r'},
 	{"rfraw",		no_argument,		NULL, 'R'},
 	{"mmap",		no_argument,		NULL, 'm'},
@@ -129,6 +131,7 @@ static const char *copyright = "Please report bugs to pcap]
 
@@ -142,6 +145,7 @@ static void signal_handler(int number)
 		break;
 	case SIGHUP:
 		sighup = 1;
+		sighup_time = (sig_atomic_t)(time(NULL) - start_time);
 		break;
 	default:
 		break;
@@ -776,8 +780,14 @@ static int next_multi_pcap_file(struct ctx *ctx, int fd)
 
 	close(fd);
 
+	time_t ftime = time(NULL);
+	if (ctx->pre_timestamp && sighup_time > 0) {
+		ftime = (time_t)(start_time + sighup_time);
+		sighup_time = 0;
+	}
+
 	slprintf(fname, sizeof(fname), "%s/%s%lu.pcap", ctx->device_out,
-		 ctx->prefix ? : "dump-", time(NULL));
+		 ctx->prefix ? : "dump-", ftime);
 
 	fd = open_or_die_m(fname, O_RDWR | O_CREAT | O_TRUNC |
 			   O_LARGEFILE, DEFFILEMODE);
@@ -816,7 +826,7 @@ static int 

Re: [netsniff-ng] Synchronize file creation between netsniff-ng processes possible?

[e . bengtsson]

Den onsdag 27 januari 2016 kl. 16:17:46 UTC+1 skrev Tobias Klauser:
>
> Hi again 
>
> On 2016-01-27 at 16:10:30 +0100, Tobias Klauser  > wrote: 
> > On 2016-01-27 at 14:40:55 +0100, Erik Bengtsson  > wrote: 
> > > By "start time" I mean file creation time. It is totally possible to 
> post 
> > > process all log files and synchronize them using pcap timestamps, as 
> you 
> > > suggested, but I'm afraid that it will be less effective for us since 
> there 
> > > will be a huge amount of data to move around and process. 
> > 
> > Ah ok, I see. Thanks for clarifying. 
> > 
> > In that case, I'm afraid there currently is no method to synchronize the 
> file 
> > creation among multiple instances of netsniff-ng. As Daniel suggested, 
> > timerfd might be an option to implement a feature along these lines. If 
> > you want, feel free to have a look into it - patches are gladly accepted 
> > :) 
>
> Something which I completely forgot about... 
>
> There might be an option (though a bit curde) to solve this using the 
> permature rotation caused by SIGHUP. You could set up a separate task 
> simultaneously sending a SIGHUP every minute to your netsniff-ng 
> processes. See commit 46289df6bc8f573bc01be4fb4aa93343ecc6d50a 
> ("netsniff-ng: Rotate pcap files prematurely on SIGHUP") for details. 
>
> Tobias 
>

That sounds really interesting! :-)

I ended up using signal USR2 since i didn't want to change how SIGHUP was 
used. When USR2 is received, the current time is saved and used when naming 
the next file (which is created when the next packet is received).

A patch is included if you want to have a look.

/ Erik
 

-- 
You received this message because you are subscribed to the Google Groups 
"netsniff-ng" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to netsniff-ng+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
commit 20ffe5b2b3734d61da727fc6c982a83847df5b8f
Author: Erik Bengtsson 
Date:   Wed Jan 27 17:25:02 2016 +0100

Added support for rotating dump files by sending signal USR2 to netsniff-ng.

diff --git a/netsniff-ng.c b/netsniff-ng.c
index 9bc4985..2831d90 100644
--- a/netsniff-ng.c
+++ b/netsniff-ng.c
@@ -48,6 +48,8 @@
 #include "dissector.h"
 #include "xmalloc.h"
 
+#define SIGNEXTDUMP SIGUSR2
+
 enum dump_mode {
DUMP_INTERVAL_TIME,
DUMP_INTERVAL_SIZE,
@@ -69,7 +71,7 @@ struct ctx {
uint64_t pkts_recvd_last, pkts_drops_last;
 };
 
-static volatile sig_atomic_t sigint = 0, sighup = 0;
+static volatile sig_atomic_t sigint = 0, sighup = 0, signextdump = 0;
 static volatile bool next_dump = false;
 
 static const char *short_options =
@@ -129,11 +131,11 @@ static const char *copyright = "Please report bugs to 
pcap]
 
 static void signal_handler(int number)
-{
+{  
switch (number) {
case SIGINT:
case SIGQUIT:
@@ -143,6 +145,10 @@ static void signal_handler(int number)
case SIGHUP:
sighup = 1;
break;
+   case SIGNEXTDUMP:
+   signextdump = 1;
+   nextdump_time = time(NULL);
+   break;
default:
break;
}
@@ -461,6 +467,7 @@ static void receive_to_xmit(struct ctx *ctx)
fflush(stdout);
 
while (likely(sigint == 0)) {
+
while (user_may_pull_from_rx(rx_ring.frames[it_in].iov_base)) {
hdr_in = rx_ring.frames[it_in].iov_base;
in = ((uint8_t *) hdr_in) + hdr_in->tp_h.tp_mac;
@@ -776,8 +783,11 @@ static int next_multi_pcap_file(struct ctx *ctx, int fd)
 
close(fd);
 
+   printf("next multi pcap file %lu\n", nextdump_time);
+   time_t ftime = nextdump_time ? nextdump_time : time(NULL);
+   nextdump_time = 0;
slprintf(fname, sizeof(fname), "%s/%s%lu.pcap", ctx->device_out,
-ctx->prefix ? : "dump-", time(NULL));
+ctx->prefix ? : "dump-", ftime);
 
fd = open_or_die_m(fname, O_RDWR | O_CREAT | O_TRUNC |
   O_LARGEFILE, DEFFILEMODE);
@@ -895,10 +905,17 @@ static void update_pcap_next_dump(struct ctx *ctx, 
unsigned long snaplen,
}
}
 
-   if (sighup) {
-   if (ctx->verbose)
-   printf("SIGHUP received, prematurely rotating pcap\n");
-   sighup = 0;
+   if (sighup || signextdump) {
+   if (sighup) {
+   if (ctx->verbose)
+   printf("SIGHUP received, prematurely rotating 
pcap\n");
+   sighup = 0;
+   }
+   if (signextdump) {
+   

Re: [netsniff-ng] Synchronize file creation between netsniff-ng processes possible?

[Vadim Kochan]

Hi,

On Tue, Jan 26, 2016 at 12:12 PM,   wrote:
> Hi everyone!
>
> I've googled a bit but not found any good answer to my question. Maybe you 
> can help me out?
>
> Currently I'm running five netsniff processes capturing data from five 
> different interfaces. It works really well with the limiting factor SSD write 
> speed, not netsniff itself. To make post processing easier, it would be nice 
> if all log files were the same size and each set of files had the same start 
> time.
>
> It seems like netsniff only checks file size / time when a packet is received 
> so the start time within a set of files drift more or less depending on the 
> busload.
>
> Any ideas on how to synchronize the processes?
>
> Best regards
> Erik
>
> --
> You received this message because you are subscribed to the Google Groups 
> "netsniff-ng" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to netsniff-ng+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Really I use netsniff-ng just for simple dissecting so I might be not
aware about some usages of the tool, but
as developer may be I may help you if you provide more info with some
output's like:

1) How do you exactly run the tool ?

2) What do you see exactly with console output example ?

3) What you 'd like to expect exactly (probably with examples of output) ?

Regards,
Vadim Kochan

-- 
You received this message because you are subscribed to the Google Groups 
"netsniff-ng" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to netsniff-ng+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [netsniff-ng] Synchronize file creation between netsniff-ng processes possible?

[Daniel Borkmann]

On 01/27/2016 02:13 PM, e.bengts...@gmail.com wrote:
[...]

Is it possible to add some kind of signal interface for controlling start
of new files instead of relying on incoming events?
I have pulled the source and had a look but don't really know how to
proceed...


If you have some cycles, it would be worth to add timerfd [1] support to
netsniff-ng, so that the timerfd is set to the dump interval we already
pass via cmdline, and we pack that fd along with the pf_packet one into
the poll() syscall, so we get a poll notification from there. We might need
to move update_pcap_next_dump() out of the loop so we can generate a new
dump file. This should reduce the latency for low/mid traffic patterns
quite a bit.

Thanks,
Daniel

  [1] http://man7.org/linux/man-pages/man2/timerfd_create.2.html

--
You received this message because you are subscribed to the Google Groups 
"netsniff-ng" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to netsniff-ng+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [netsniff-ng] Synchronize file creation between netsniff-ng processes possible?

[Tobias Klauser]

Hi again

On 2016-01-27 at 16:10:30 +0100, Tobias Klauser  wrote:
> On 2016-01-27 at 14:40:55 +0100, Erik Bengtsson  wrote:
> > By "start time" I mean file creation time. It is totally possible to post
> > process all log files and synchronize them using pcap timestamps, as you
> > suggested, but I'm afraid that it will be less effective for us since there
> > will be a huge amount of data to move around and process.
> 
> Ah ok, I see. Thanks for clarifying.
> 
> In that case, I'm afraid there currently is no method to synchronize the file
> creation among multiple instances of netsniff-ng. As Daniel suggested,
> timerfd might be an option to implement a feature along these lines. If
> you want, feel free to have a look into it - patches are gladly accepted
> :)

Something which I completely forgot about...

There might be an option (though a bit curde) to solve this using the
permature rotation caused by SIGHUP. You could set up a separate task
simultaneously sending a SIGHUP every minute to your netsniff-ng
processes. See commit 46289df6bc8f573bc01be4fb4aa93343ecc6d50a
("netsniff-ng: Rotate pcap files prematurely on SIGHUP") for details.

Tobias

-- 
You received this message because you are subscribed to the Google Groups 
"netsniff-ng" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to netsniff-ng+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[netsniff-ng] Synchronize file creation between netsniff-ng processes possible?

[e . bengtsson]

Hi everyone!

I've googled a bit but not found any good answer to my question. Maybe you can 
help me out?

Currently I'm running five netsniff processes capturing data from five 
different interfaces. It works really well with the limiting factor SSD write 
speed, not netsniff itself. To make post processing easier, it would be nice if 
all log files were the same size and each set of files had the same start time.

It seems like netsniff only checks file size / time when a packet is received 
so the start time within a set of files drift more or less depending on the 
busload.

Any ideas on how to synchronize the processes?

Best regards 
Erik

-- 
You received this message because you are subscribed to the Google Groups 
"netsniff-ng" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to netsniff-ng+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.