Re: Continuous packet capture
Brad,
Great stuff. Thanks. I'm going on the assumption that $inst-[1] (line
94) is the raw packet, headers and all. I plan to test by pointing pcap at
a capture file and comparing output to input. My next plan is to add a sql
insert of $hexpacket at line 108. My goal is to keep up with an incoming
data stream of 20-30 k-bytes a second. After the hex data is safely stored
in the database I can go back and insert whatever fields (layer 3-4, packet
length, etc) I want. Then do a gui to extract traces and do other analysis
from the stored data. Here is my code;
1 use strict;
2 use warnings;
3
4 use Data::Dumper;
5 use Carp;
6
7 #
8 # POE Environment
9 # use POE qw(
10 # Component::Pcap
11 # Component::Daemon
12 # );
13
14 use POE qw(
15 Component::Pcap
16 );
17
18 #
19 # PCAP Options
20 my %pcap_opts = (
21 dev = 'eth0',
22 snaplen = 1514,
23 promisc = 1,
24 timeout = 100,
25 );
26
27 ##
28 # POE Environment Setup
29 ##
30 #POE::Component::Daemon-spawn( detach = 1, babysit = 600, max_children
= 5 );
31
32 POE::Session-create(
33 inline_states = {
34 _start = \start_processor,
35 _stop = \stop_processor,
36 handle_packet = \handle_packet,
37 },
38 );
39
40 $_[HEAP]-{i} = 0;
41
42 ##
43 # Run the POE Sessions
44 ##
45 POE::Kernel-run;
46
47 ##
48 exit 0;
49 ##
50
51 ##
52 # Start the Processor
53 sub start_processor {
54 my ($kernel, $heap) = @_[KERNEL, HEAP];
55
56 $kernel-alias_set('processor');
57
58 #
59 # Start Packet Capturing
60 POE::Component::Pcap-spawn(
61 Alias = 'pcap',
62 Device= $pcap_opts{dev},
63 Dispatch= 'handle_packet',
64 Session = 'processor',
65 );
66 $kernel-post( pcap = open_live = @pcap_opts{qw(dev snaplen promisc
timeout)} );
67
68 $kernel-post( pcap = 'run' );
69 }
70 ##
71
72 ##
73 # stop the processor
74 sub stop_processor {
75 my ($kernel,$heap) = @_[KERNEL,HEAP];
76
77 #
78 # Stop pcap
79 $kernel-post( 'pcap' = 'shutdown' );
80 }
81 ##
82
83 ##
84 sub handle_packet {
85 my $offset = 0;
86 my $linechars = 0;
87 my $hexpacket;
88
89 $_[HEAP]-{i}++;
90 print sub handle_packet called $_[HEAP]-{i} times\n;
91 my ($kernel,$heap,$packets) = @_[KERNEL,HEAP,ARG0];
92
93 foreach my $inst ( @{ $packets } ) {
94 foreach my $char (split(//, $inst-[1])) {
95 if ( $linechars == 0 ) {
96 $hexpacket .= sprintf( %04X , $offset );
97 }
98 $linechars++;
99 $hexpacket .= sprintf( %02X , ord($char) );
100 if ( $linechars == 16 ) {
101 $hexpacket .= \n;
102 $linechars = 0;
103 $offset += 16;
104 }
105 }
106 $hexpacket .= \n\n;
107 }
108 print $hexpacket;
109 undef($hexpacket);
110 }
111
112 ##
Re: Continuous packet capture
Nosing around in the mailing list archive I find;
Re: Prioritizing Event Loops
Martijn van Beers
Fri, 18 Jan 2008 11:37:22 -0800
snip
In POE, all your external data, whether from a socket or a file, or
something else entirely, enters your app through a POE::Wheel (you might
not be aware of having one, since it is mostly hidden inside
POE::Component::Server::TCP). How the wheel gets the data is supposed to be
abstracted away.
/snip
So POE::Component::Pcap is a wheel?
Jon
__
Brad,
Great stuff. Thanks. I'm going on the assumption that $inst-[1] (line
94) is the raw packet, headers and all. I plan to test by pointing pcap at
a capture file and comparing output to input. My next plan is to add a sql
insert of $hexpacket at line 108. My goal is to keep up with an incoming
data stream of 20-30 k-bytes a second. After the hex data is safely stored
in the database I can go back and insert whatever fields (layer 3-4, packet
length, etc) I want. Then do a gui to extract traces and do other analysis
from the stored data. Here is my code;
1 use strict;
2 use warnings;
3
4 use Data::Dumper;
5 use Carp;
6
7 #
8 # POE Environment
9 # use POE qw(
10 # Component::Pcap
11 # Component::Daemon
12 # );
13
14 use POE qw(
15 Component::Pcap
16 );
17
18 #
19 # PCAP Options
20 my %pcap_opts = (
21 dev = 'eth0',
22 snaplen = 1514,
23 promisc = 1,
24 timeout = 100,
25 );
26
27 ##
28 # POE Environment Setup
29 ##
30 #POE::Component::Daemon-spawn( detach = 1, babysit = 600, max_children
= 5 );
31
32 POE::Session-create(
33 inline_states = {
34 _start = \start_processor,
35 _stop = \stop_processor,
36 handle_packet = \handle_packet,
37 },
38 );
39
40 $_[HEAP]-{i} = 0;
41
42 ##
43 # Run the POE Sessions
44 ##
45 POE::Kernel-run;
46
47 ##
48 exit 0;
49 ##
50
51 ##
52 # Start the Processor
53 sub start_processor {
54 my ($kernel, $heap) = @_[KERNEL, HEAP];
55
56 $kernel-alias_set('processor');
57
58 #
59 # Start Packet Capturing
60 POE::Component::Pcap-spawn(
61 Alias = 'pcap',
62 Device= $pcap_opts{dev},
63 Dispatch= 'handle_packet',
64 Session = 'processor',
65 );
66 $kernel-post( pcap = open_live = @pcap_opts{qw(dev snaplen promisc
timeout)} );
67
68 $kernel-post( pcap = 'run' );
69 }
70 ##
71
72 ##
73 # stop the processor
74 sub stop_processor {
75 my ($kernel,$heap) = @_[KERNEL,HEAP];
76
77 #
78 # Stop pcap
79 $kernel-post( 'pcap' = 'shutdown' );
80 }
81 ##
82
83 ##
84 sub handle_packet {
85 my $offset = 0;
86 my $linechars = 0;
87 my $hexpacket;
88
89 $_[HEAP]-{i}++;
90 print sub handle_packet called $_[HEAP]-{i} times\n;
91 my ($kernel,$heap,$packets) = @_[KERNEL,HEAP,ARG0];
92
93 foreach my $inst ( @{ $packets } ) {
94 foreach my $char (split(//, $inst-[1])) {
95 if ( $linechars == 0 ) {
96 $hexpacket .= sprintf( %04X , $offset );
97 }
98 $linechars++;
99 $hexpacket .= sprintf( %02X , ord($char) );
100 if ( $linechars == 16 ) {
101 $hexpacket .= \n;
102 $linechars = 0;
103 $offset += 16;
104 }
105 }
106 $hexpacket .= \n\n;
107 }
108 print $hexpacket;
109 undef($hexpacket);
110 }
111
112 ##
Re: Continuous packet capture
Sorry, I thought I found a bug in the code, but it turns out my code was
correct. $inst-[0] is the Packet Header, $inst-[1] is the packet in
raw form.
Jonathan S. Polacheck wrote:
Brad,
Great stuff. Thanks. I'm going on the assumption that $inst-[1] (line
94) is the raw packet, headers and all. I plan to test by pointing pcap at
a capture file and comparing output to input. My next plan is to add a sql
insert of $hexpacket at line 108. My goal is to keep up with an incoming
data stream of 20-30 k-bytes a second. After the hex data is safely stored
in the database I can go back and insert whatever fields (layer 3-4, packet
length, etc) I want. Then do a gui to extract traces and do other analysis
from the stored data. Here is my code;
1 use strict;
2 use warnings;
3
4 use Data::Dumper;
5 use Carp;
6
7 #
8 # POE Environment
9 # use POE qw(
10 # Component::Pcap
11 # Component::Daemon
12 # );
13
14 use POE qw(
15 Component::Pcap
16 );
17
18 #
19 # PCAP Options
20 my %pcap_opts = (
21 dev = 'eth0',
22 snaplen = 1514,
23 promisc = 1,
24 timeout = 100,
25 );
26
27 ##
28 # POE Environment Setup
29 ##
30 #POE::Component::Daemon-spawn( detach = 1, babysit = 600, max_children
= 5 );
31
32 POE::Session-create(
33 inline_states = {
34 _start = \start_processor,
35 _stop = \stop_processor,
36 handle_packet = \handle_packet,
37 },
38 );
39
40 $_[HEAP]-{i} = 0;
41
42 ##
43 # Run the POE Sessions
44 ##
45 POE::Kernel-run;
46
47 ##
48 exit 0;
49 ##
50
51 ##
52 # Start the Processor
53 sub start_processor {
54 my ($kernel, $heap) = @_[KERNEL, HEAP];
55
56 $kernel-alias_set('processor');
57
58 #
59 # Start Packet Capturing
60 POE::Component::Pcap-spawn(
61 Alias = 'pcap',
62 Device= $pcap_opts{dev},
63 Dispatch= 'handle_packet',
64 Session = 'processor',
65 );
66 $kernel-post( pcap = open_live = @pcap_opts{qw(dev snaplen promisc
timeout)} );
67
68 $kernel-post( pcap = 'run' );
69 }
70 ##
71
72 ##
73 # stop the processor
74 sub stop_processor {
75 my ($kernel,$heap) = @_[KERNEL,HEAP];
76
77 #
78 # Stop pcap
79 $kernel-post( 'pcap' = 'shutdown' );
80 }
81 ##
82
83 ##
84 sub handle_packet {
85 my $offset = 0;
86 my $linechars = 0;
87 my $hexpacket;
88
89 $_[HEAP]-{i}++;
90 print sub handle_packet called $_[HEAP]-{i} times\n;
91 my ($kernel,$heap,$packets) = @_[KERNEL,HEAP,ARG0];
92
93 foreach my $inst ( @{ $packets } ) {
94 foreach my $char (split(//, $inst-[1])) {
95 if ( $linechars == 0 ) {
96 $hexpacket .= sprintf( %04X , $offset );
97 }
98 $linechars++;
99 $hexpacket .= sprintf( %02X , ord($char) );
100 if ( $linechars == 16 ) {
101 $hexpacket .= \n;
102 $linechars = 0;
103 $offset += 16;
104 }
105 }
106 $hexpacket .= \n\n;
107 }
108 print $hexpacket;
109 undef($hexpacket);
110 }
111
112 ##
--
Brad Lhotsky lhots...@mail.nih.gov
NCTS Computer Specialist -- 410.558.8006
.. WAR IS PEACE,
FREEDOM IS SLAVERY,
IGNORANCE IS STRENGTH ..
Re: Continuous packet capture
I'm not sure that POE::Component::Pcap works like the
POE::Component::Server::TCP. It appears to be implemented as a separate
POE::Session and uses select() with an internal file descriptor to poll
the device for new packets and fire the events specified by the dispath
argument.
Jonathan S. Polacheck wrote:
Nosing around in the mailing list archive I find;
Re: Prioritizing Event Loops
Martijn van Beers
Fri, 18 Jan 2008 11:37:22 -0800
snip
In POE, all your external data, whether from a socket or a file, or
something else entirely, enters your app through a POE::Wheel (you might
not be aware of having one, since it is mostly hidden inside
POE::Component::Server::TCP). How the wheel gets the data is supposed to be
abstracted away.
/snip
So POE::Component::Pcap is a wheel?
Jon
__
Brad,
Great stuff. Thanks. I'm going on the assumption that $inst-[1] (line
94) is the raw packet, headers and all. I plan to test by pointing pcap at
a capture file and comparing output to input. My next plan is to add a sql
insert of $hexpacket at line 108. My goal is to keep up with an incoming
data stream of 20-30 k-bytes a second. After the hex data is safely stored
in the database I can go back and insert whatever fields (layer 3-4, packet
length, etc) I want. Then do a gui to extract traces and do other analysis
from the stored data. Here is my code;
1 use strict;
2 use warnings;
3
4 use Data::Dumper;
5 use Carp;
6
7 #
8 # POE Environment
9 # use POE qw(
10 # Component::Pcap
11 # Component::Daemon
12 # );
13
14 use POE qw(
15 Component::Pcap
16 );
17
18 #
19 # PCAP Options
20 my %pcap_opts = (
21 dev = 'eth0',
22 snaplen = 1514,
23 promisc = 1,
24 timeout = 100,
25 );
26
27 ##
28 # POE Environment Setup
29 ##
30 #POE::Component::Daemon-spawn( detach = 1, babysit = 600, max_children
= 5 );
31
32 POE::Session-create(
33 inline_states = {
34 _start = \start_processor,
35 _stop = \stop_processor,
36 handle_packet = \handle_packet,
37 },
38 );
39
40 $_[HEAP]-{i} = 0;
41
42 ##
43 # Run the POE Sessions
44 ##
45 POE::Kernel-run;
46
47 ##
48 exit 0;
49 ##
50
51 ##
52 # Start the Processor
53 sub start_processor {
54 my ($kernel, $heap) = @_[KERNEL, HEAP];
55
56 $kernel-alias_set('processor');
57
58 #
59 # Start Packet Capturing
60 POE::Component::Pcap-spawn(
61 Alias = 'pcap',
62 Device= $pcap_opts{dev},
63 Dispatch= 'handle_packet',
64 Session = 'processor',
65 );
66 $kernel-post( pcap = open_live = @pcap_opts{qw(dev snaplen promisc
timeout)} );
67
68 $kernel-post( pcap = 'run' );
69 }
70 ##
71
72 ##
73 # stop the processor
74 sub stop_processor {
75 my ($kernel,$heap) = @_[KERNEL,HEAP];
76
77 #
78 # Stop pcap
79 $kernel-post( 'pcap' = 'shutdown' );
80 }
81 ##
82
83 ##
84 sub handle_packet {
85 my $offset = 0;
86 my $linechars = 0;
87 my $hexpacket;
88
89 $_[HEAP]-{i}++;
90 print sub handle_packet called $_[HEAP]-{i} times\n;
91 my ($kernel,$heap,$packets) = @_[KERNEL,HEAP,ARG0];
92
93 foreach my $inst ( @{ $packets } ) {
94 foreach my $char (split(//, $inst-[1])) {
95 if ( $linechars == 0 ) {
96 $hexpacket .= sprintf( %04X , $offset );
97 }
98 $linechars++;
99 $hexpacket .= sprintf( %02X , ord($char) );
100 if ( $linechars == 16 ) {
101 $hexpacket .= \n;
102 $linechars = 0;
103 $offset += 16;
104 }
105 }
106 $hexpacket .= \n\n;
107 }
108 print $hexpacket;
109 undef($hexpacket);
110 }
111
112 ##
--
Brad Lhotsky lhots...@mail.nih.gov
NCTS Computer Specialist -- 410.558.8006
.. WAR IS PEACE,
FREEDOM IS SLAVERY,
IGNORANCE IS STRENGTH ..
Re: Continuous packet capture
I don't know if this will help you or not, but here's a script I have
running to monitor and track recursive DNS Queries on my network. You
can ignore the database stuff, but it might help.
http://divisionbyzero.net/~brad/code/dns_snoop.pl.html
I process the packets as they come in, but I could be using
POE::Wheel::Run on them as well... The key part of the tutorial (http://poe.perl.org/?POE_Cookbook/Child_Processes_3
) seems to be the while() loop in the start_tasks routine.
It doesn't sound like you need the feedback from the child tasks
because you'll be feeding them via the heap. The Filter::Reference
stuff is being used to return status information from the child tasks
to the main processor.
Here's another script I've written that needs some reworking as well.
It listens on our egress link, and collects statistics inside the
heap. At intervals (POE::Component::Cron), the heap is processed
written to a database (or RRD) and then cleared. The processing
continues.
http://divisionbyzero.net/~brad/code/traffic_detection.pl.html
I hope there's something in there that helps.
On Mar 24, 2009, at 4:25 PM, Jonathan S. Polacheck wrote:
I have spent a couple of days casting about, looking at examples and
perldoc. I don't seem to be able to get the data from 'tcpdump' (or
POE::Component::PCAP) to the wheel that will process the data. For
my last
attempt, I removed Filter::Referece and tried with just the wheel.
Still
no luck. Any suggestions?
#!/usr/bin/perl
use warnings;
use strict;
use POE qw( Wheel::Run ); #Filter::Reference );
use Data::Dumper;
our $offset = 0;
our $linechars = '';
POE::Session-create
( inline_states =
{
_start = sub {
my ($heap) = $_[HEAP];
my $gp = POE::Wheel::Run-new
( Program = '/usr/sbin/tcpdump -i eth0 -w - '
# , Conduit = pipe
# , StdoutFilter = POE::Filter::Reference-
new()
# , StdinEvent = 'process_packet'
, StdinEvent = 'stdin'
# , StdoutEvent = 'print'
# , InputEvent = 'process_packet'
);
print gp compleate\n;
$heap-{gp} = $gp;
# my $pp = POE::Wheel::Run-new
# ( Program = process_packet
# , StdoutEvent = 'stdout'
# );
}
}
);
sub process_packet {
print process_packet called\n;
# my ($heap) = $_[HEAP];
# my $filter = POE::Filter::Reference-new();
# my $pdump = $filter-get( [ $heap-{gp} ] );
my $pdump = $_[ARG0];
foreach my $char (split(//, $pdump)) {
if($char !~ /\n/) {
dump_char($char) ;
} else {
print \n\n;
$offset = 0;
$linechars = '';
}
1;
}
dump_char( ' ', 1 ) while length($linechars) != 0;
}
sub dump_char {
my ( $char ) = shift;
if ( length( $linechars ) == 0 ) {
printf( %06X , $offset );
}
$linechars .= ( $char =~ m#[!-~ ]# ) ? $char : '.';
printf( %02X , ord($char) );
if ( length( $linechars ) == 16 ) {
print( \n );
$linechars = '';
$offset += 16;
}
}
$poe_kernel-run();
exit 0;
--
Brad Lhotsky lhots...@mail.nih.gov
Security Administrator / NIA Alt. ISSO
.. WAR IS PEACE
FREEDOM IS SLAVERY
IGNORANCE IS STRENGTH ..
Re: Continuous packet capture
I have spent a couple of days casting about, looking at examples and
perldoc. I don't seem to be able to get the data from 'tcpdump' (or
POE::Component::PCAP) to the wheel that will process the data. For my last
attempt, I removed Filter::Referece and tried with just the wheel. Still
no luck. Any suggestions?
#!/usr/bin/perl
use warnings;
use strict;
use POE qw( Wheel::Run ); #Filter::Reference );
use Data::Dumper;
our $offset = 0;
our $linechars = '';
POE::Session-create
( inline_states =
{
_start = sub {
my ($heap) = $_[HEAP];
my $gp = POE::Wheel::Run-new
( Program = '/usr/sbin/tcpdump -i eth0 -w - '
# , Conduit = pipe
# , StdoutFilter = POE::Filter::Reference-new()
# , StdinEvent = 'process_packet'
, StdinEvent = 'stdin'
# , StdoutEvent = 'print'
# , InputEvent = 'process_packet'
);
print gp compleate\n;
$heap-{gp} = $gp;
# my $pp = POE::Wheel::Run-new
# ( Program = process_packet
# , StdoutEvent = 'stdout'
# );
}
}
);
sub process_packet {
print process_packet called\n;
# my ($heap) = $_[HEAP];
# my $filter = POE::Filter::Reference-new();
# my $pdump = $filter-get( [ $heap-{gp} ] );
my $pdump = $_[ARG0];
foreach my $char (split(//, $pdump)) {
if($char !~ /\n/) {
dump_char($char) ;
} else {
print \n\n;
$offset = 0;
$linechars = '';
}
1;
}
dump_char( ' ', 1 ) while length($linechars) != 0;
}
sub dump_char {
my ( $char ) = shift;
if ( length( $linechars ) == 0 ) {
printf( %06X , $offset );
}
$linechars .= ( $char =~ m#[!-~ ]# ) ? $char : '.';
printf( %02X , ord($char) );
if ( length( $linechars ) == 16 ) {
print( \n );
$linechars = '';
$offset += 16;
}
}
$poe_kernel-run();
exit 0;
Re: Continuous packet capture
What I get out the the example;
Wheel::Run is used to create tasks (as opposed to processes or sessions)
Filter::Reference is used to create a communications channel between the
tasks
What I am still wondering is if the communications channel would buffer
or act as a fifo for the data coming out of the enque task and if
prioritizing would be needed to make sure that the enque task would not
drop any packets.
___
Alex _
Fri, 13 Mar 2009 15:49:37 -0700
heres an example of producer/consumer with wheel::run
http://poe.perl.org/?POE_Cookbook/Child_Processes_3
From: Jonathan S. Polacheck jpola...@texasmutual.com
To: poe@perl.org
Sent: Friday, March 13, 2009 1:13:55 PM
Subject: Re: Continuous packet capture
ch...@fedde.us wrote;
Your first step is going to be to make sure your code is 'use strict; use
warnings;' clean. I'm not saying that your code is not, but since I don't
see the strictures I'm making the knee jerk comment. Second, POE might be
a fine way to go. There are lots of components available that'll make
coding this up easier. Third, what's wrong with using one of the packages
you list?
and jpola...@texasmutual.com replied;
strict and warnings is definitely a good idea. But if I port the whole
thing to POE, I guess I'll start using them on that code. The two included
scripts are just a sort of proof of concept I did to get things going. I
ultimately may not be able to pull this off in Perl, but it's what I know
so I starting there.
As for the packages, if you are referring to Infinistrream, Gigastor,
etc, the problem in my view is that they are both proprietary and
expensive. I believe that continuous packet capture will become a standard
way of doing things, supplanting ad-hoc capture (pcap, wireshark, etc), at
least in any production environment large enough to require staff
knowledgeable to use such tools. That, I think, is the time when open
source solutions break into to market.
We have Infinistreams in the production environment I work at. We have the
lowest end devices at our remote sites. We paid in excess of $10k each for
them, plus ongoing support contract costs. They work find, but the vendor
(now Netscout) has dropped them from the product line (no replacements, no
further updates, end-of-life on the horizon). Netscout has a track record
of going for the high end of the market with product development and
pricing to match.
So I decided that wireshark should evolve to include cpc capabilities. I
sent my code to wireshark-dev and was roundly ignored (no time for Perl
programmers, perhaps). But no matter, communities are where you find them.
So I tried the POE list, and here we are.
I agree that there are lots of components. Here's where my POE solution
stands;
use POE;
use strict;
use warnings;
use Net::Pcap;
use POE::Component::Pcap;
use Data::Hexdumper qw( hexdump );
use Data::Dumper;
use lib 't';
my $dev = eth0;
my $i = 0;
POE::Session-create(
inline_states = {
_start = \start,
got_packet = \got_packet,
},
);
POE::Kernel-run;
sub start {
#diag [POE:start] spawning new Pcap session , $_[SESSION]-ID,
on device $dev;
POE::Component::Pcap-spawn(
Alias = 'pcap',
Device = $dev,
Dispatch = 'got_packet',
Session = $_[SESSION],
);
$_[KERNEL]-post(pcap = open_live = $dev, 1514, 1);
$_[KERNEL]-post(pcap = 'run');
}
# sub stop {
# #diag [POE:stop];
# $_[KERNEL]-post(pcap = 'shutdown');
# }
sub got_packet {
#diag [POE:got_packet];
# $i++;
# print got_packet run $i\n;
my $packets = $_[ARG0];
# process the first packet only
process_packet(@{ $packets-[0] });
# send a message to stop the capture
#$_[KERNEL]-post(pcap = 'shutdown');
}
sub process_packet {
# my ($pkt) = $_[1];
my $results = hexdump( data = $_[1]
, number_format = 'C',
);
# print Dumper($header);
print $results;
}
#start;
exit;
So I have the hexdump moved into process_packet and out of got_packet.
And it;'s easy enough to change print $results to a SQL insert statement.
But I think I need a que or fifo (HEAP?) to hold the packets and a priority
on get_packet to make sure it keeps up with POE::Component::Pcap and the
incoming traffic. Or is there a better way?
Thanks for your interest,
Re: Continuous packet capture
the tasks is the queue. Wheel::run pops of tasks and runs it in a separate
process
From: Jon Polacheck jon...@grandecom.net
To: poe@perl.org
Sent: Sunday, March 15, 2009 7:08:01 AM
Subject: Re: Continuous packet capture
What I get out the the example;
Wheel::Run is used to create tasks (as opposed to processes or sessions)
Filter::Reference is used to create a communications channel between the tasks
What I am still wondering is if the communications channel would buffer or act
as a fifo for the data coming out of the enque task and if prioritizing would
be needed to make sure that the enque task would not drop any packets.
___
Alex _
Fri, 13 Mar 2009 15:49:37 -0700
heres an example of producer/consumer with wheel::run
http://poe.perl.org/?POE_Cookbook/Child_Processes_3
From: Jonathan S. Polacheck jpola...@texasmutual.com
To: poe@perl.org
Sent: Friday, March 13, 2009 1:13:55 PM
Subject: Re: Continuous packet capture
ch...@fedde.us wrote;
Your first step is going to be to make sure your code is 'use strict; use
warnings;' clean. I'm not saying that your code is not, but since I don't
see the strictures I'm making the knee jerk comment. Second, POE might be
a fine way to go. There are lots of components available that'll make
coding this up easier. Third, what's wrong with using one of the packages
you list?
and jpola...@texasmutual.com replied;
strict and warnings is definitely a good idea. But if I port the whole
thing to POE, I guess I'll start using them on that code. The two included
scripts are just a sort of proof of concept I did to get things going. I
ultimately may not be able to pull this off in Perl, but it's what I know
so I starting there.
As for the packages, if you are referring to Infinistrream, Gigastor,
etc, the problem in my view is that they are both proprietary and
expensive. I believe that continuous packet capture will become a standard
way of doing things, supplanting ad-hoc capture (pcap, wireshark, etc), at
least in any production environment large enough to require staff
knowledgeable to use such tools. That, I think, is the time when open
source solutions break into to market.
We have Infinistreams in the production environment I work at. We have the
lowest end devices at our remote sites. We paid in excess of $10k each for
them, plus ongoing support contract costs. They work find, but the vendor
(now Netscout) has dropped them from the product line (no replacements, no
further updates, end-of-life on the horizon). Netscout has a track record
of going for the high end of the market with product development and
pricing to match.
So I decided that wireshark should evolve to include cpc capabilities. I
sent my code to wireshark-dev and was roundly ignored (no time for Perl
programmers, perhaps). But no matter, communities are where you find them.
So I tried the POE list, and here we are.
I agree that there are lots of components. Here's where my POE solution
stands;
use POE;
use strict;
use warnings;
use Net::Pcap;
use POE::Component::Pcap;
use Data::Hexdumper qw( hexdump );
use Data::Dumper;
use lib 't';
my $dev = eth0;
my $i = 0;
POE::Session-create(
inline_states = {
_start = \start,
got_packet = \got_packet,
},
);
POE::Kernel-run;
sub start {
#diag [POE:start] spawning new Pcap session , $_[SESSION]-ID,
on device $dev;
POE::Component::Pcap-spawn(
Alias = 'pcap',
Device = $dev,
Dispatch = 'got_packet',
Session = $_[SESSION],
);
$_[KERNEL]-post(pcap = open_live = $dev, 1514, 1);
$_[KERNEL]-post(pcap = 'run');
}
# sub stop {
# #diag [POE:stop];
# $_[KERNEL]-post(pcap = 'shutdown');
# }
sub got_packet {
#diag [POE:got_packet];
# $i++;
# print got_packet run $i\n;
my $packets = $_[ARG0];
# process the first packet only
process_packet(@{ $packets-[0] });
# send a message to stop the capture
#$_[KERNEL]-post(pcap = 'shutdown');
}
sub process_packet {
# my ($pkt) = $_[1];
my $results = hexdump( data = $_[1]
, number_format = 'C',
);
# print Dumper($header);
print $results;
}
#start;
exit;
So I have the hexdump moved into process_packet and out of got_packet.
And it;'s easy enough to change print $results to a SQL insert statement.
But I think I need a que or fifo (HEAP?) to hold the packets and a priority
on get_packet to make sure it keeps up with POE::Component::Pcap and the
incoming traffic. Or is there a better way?
Thanks for your interest,
Continuous packet capture
Perhaps this is a case of premature optimization. I am working toward a
continuous packet capture application (as in Infinistream, Gigastor,
NetVCR, etc). So far, I have two perl scripts. One takes Net::Pcap output
and Hexdumps it to a fifo file. The other reads the fifo and inserts the
hexdump into a database table.
This is all well and good, but when I tried to dump the pcap output
directly into the fifo and convert to hex as the fifo was read things got
ugly. So now I am considering rewriting the whole thing in POE.
I want some kind of buffer between pcap and the database inserts to reduce
the chance of dropped packets. It seems to me that the process that has
the pcap session should have a higher priority than the db insert
process.
Is POE the way to go? Are there existing examples of POE code that does
something similar?
Thanks for your interest,
Jon Polacheck
Used mkfifo to create the named pipe. Perl sees it as a disk file
(that I called qtfifo). ENQUE.pl dumps packet hexdumps to the fifo.
DEQUE.pl reads lines from the fifo. /^ / acts as the delimiter. mysql
compression worked with the standard OpenSuSE install, no recompiling or
other mucking about necessary.
Lines used for debugging marked as such.
ENQUE.pl
use Net::Pcap;
use Data::Hexdumper qw(hexdump);
$dev = eth0;
# used a 50 packet cap file to make sure what came out matched what went in
#$dump = ip.cap;
#$pcap = Net::Pcap::open_offline($dump, \$err) or die Can't read
'$dump': $err\n;
# live, real-time feed
$pcap = Net::Pcap::open_live($dev, 1514, 1, 0, \$err);
Net::Pcap::loop($pcap, -1, \process_pkt, ); # - subroutine call
sub process_pkt {
open(QT, qtfifo);
# $_[2] is the third element of the default array @_ which was created
# by the subroutine call process_pkt
my $pkt=$_[2];
$results = hexdump( data = $pkt
, number_format = 'C',
);
print QT $results;
close(QT);
$i++; # debug
stop_run if $i 100; #debug
}
# all debug below
sub stop_run {
print stop_run\n;
open(QT, qtfifo);
print QT \nx\n;
close(QT);
print enque ended\n;
exit;
}
DEQUE.pl
use Time::HiRes ( nanosleep );
use DBI;
$hostname=127.0.0.1;
$database=cpc;
$port=3306;
$dsn = DBI:mysql:database=$database;host=$hostname;port=$port;
$dbh = DBI-connect($dsn,
root,
,
{'RaiseError' = 1});
# call the Net::Packet collector script
system(q{perl ENQUE.pl});
open(EQT, qtfifo);
$i = 0; # debug
$pc = 0; # debug
while(1) {
$i++; # debug
$line = readline(EQT);
if ($line =~ /^ / ) {
$dbh-do(qq{INSERT INTO cpc VALUES ( compress($pkt))});
$pc++ if defined($pkt); # debug
print packet $pc:\n$pkt\n if defined($pkt); # debug
undef($pkt);
$pkt .= $line;
} else {
$pkt .= $line;
theend if $pkt =~ /x/; # debug
}
nanosleep(1); # would not work without this!
}
# all debug below
sub theend {
close(EQT);
print $i loops\ndeque ended\n;
exit;
}
This generated a cap file that looks just fine in Wireshark.
mysql -Br -D cpc -e select uncompress(packet) from cpc; | text2pcap -
m_cap.cap
Re: Continuous packet capture
heres an example of producer/consumer with wheel::run
http://poe.perl.org/?POE_Cookbook/Child_Processes_3
From: Jonathan S. Polacheck jpola...@texasmutual.com
To: poe@perl.org
Sent: Friday, March 13, 2009 1:13:55 PM
Subject: Re: Continuous packet capture
ch...@fedde.us wrote;
Your first step is going to be to make sure your code is 'use strict; use
warnings;' clean. I'm not saying that your code is not, but since I don't
see the strictures I'm making the knee jerk comment. Second, POE might be
a fine way to go. There are lots of components available that'll make
coding this up easier. Third, what's wrong with using one of the packages
you list?
and jpola...@texasmutual.com replied;
strict and warnings is definitely a good idea. But if I port the whole
thing to POE, I guess I'll start using them on that code. The two included
scripts are just a sort of proof of concept I did to get things going. I
ultimately may not be able to pull this off in Perl, but it's what I know
so I starting there.
As for the packages, if you are referring to Infinistrream, Gigastor,
etc, the problem in my view is that they are both proprietary and
expensive. I believe that continuous packet capture will become a standard
way of doing things, supplanting ad-hoc capture (pcap, wireshark, etc), at
least in any production environment large enough to require staff
knowledgeable to use such tools. That, I think, is the time when open
source solutions break into to market.
We have Infinistreams in the production environment I work at. We have the
lowest end devices at our remote sites. We paid in excess of $10k each for
them, plus ongoing support contract costs. They work find, but the vendor
(now Netscout) has dropped them from the product line (no replacements, no
further updates, end-of-life on the horizon). Netscout has a track record
of going for the high end of the market with product development and
pricing to match.
So I decided that wireshark should evolve to include cpc capabilities. I
sent my code to wireshark-dev and was roundly ignored (no time for Perl
programmers, perhaps). But no matter, communities are where you find them.
So I tried the POE list, and here we are.
I agree that there are lots of components. Here's where my POE solution
stands;
use POE;
use strict;
use warnings;
use Net::Pcap;
use POE::Component::Pcap;
use Data::Hexdumper qw( hexdump );
use Data::Dumper;
use lib 't';
my $dev = eth0;
my $i = 0;
POE::Session-create(
inline_states = {
_start = \start,
got_packet = \got_packet,
},
);
POE::Kernel-run;
sub start {
#diag [POE:start] spawning new Pcap session , $_[SESSION]-ID,
on device $dev;
POE::Component::Pcap-spawn(
Alias = 'pcap',
Device = $dev,
Dispatch = 'got_packet',
Session = $_[SESSION],
);
$_[KERNEL]-post(pcap = open_live = $dev, 1514, 1);
$_[KERNEL]-post(pcap = 'run');
}
# sub stop {
# #diag [POE:stop];
# $_[KERNEL]-post(pcap = 'shutdown');
# }
sub got_packet {
#diag [POE:got_packet];
# $i++;
# print got_packet run $i\n;
my $packets = $_[ARG0];
# process the first packet only
process_packet(@{ $packets-[0] });
# send a message to stop the capture
#$_[KERNEL]-post(pcap = 'shutdown');
}
sub process_packet {
# my ($pkt) = $_[1];
my $results = hexdump( data = $_[1]
, number_format = 'C',
);
# print Dumper($header);
print $results;
}
#start;
exit;
So I have the hexdump moved into process_packet and out of got_packet.
And it;'s easy enough to change print $results to a SQL insert statement.
But I think I need a que or fifo (HEAP?) to hold the packets and a priority
on get_packet to make sure it keeps up with POE::Component::Pcap and the
incoming traffic. Or is there a better way?
Thanks for your interest,
