Re: Continuous packet capture
I'm not sure that POE::Component::Pcap works like the
POE::Component::Server::TCP. It appears to be implemented as a separate
POE::Session and uses select() with an internal file descriptor to poll
the device for new packets and fire the events specified by the dispath
argument.
Jonathan S. Polacheck wrote:
Nosing around in the mailing list archive I find;
"Re: Prioritizing Event Loops
Martijn van Beers
Fri, 18 Jan 2008 11:37:22 -0800
In POE, all your external data, whether from a socket or a file, or
something else entirely, enters your app through a POE::Wheel (you might
not be aware of having one, since it is mostly hidden inside
POE::Component::Server::TCP). How the wheel gets the data is supposed to be
abstracted away.
"
So POE::Component::Pcap is a wheel?
Jon
__
Brad,
Great stuff. Thanks. I'm going on the assumption that $inst->[1] (line
94) is the raw packet, headers and all. I plan to test by pointing pcap at
a capture file and comparing output to input. My next plan is to add a sql
insert of $hexpacket at line 108. My goal is to keep up with an incoming
data stream of 20-30 k-bytes a second. After the hex data is safely stored
in the database I can go back and insert whatever fields (layer 3-4, packet
length, etc) I want. Then do a gui to extract traces and do other analysis
from the stored data. Here is my code;
1 use strict;
2 use warnings;
3
4 use Data::Dumper;
5 use Carp;
6
7 #
8 # POE Environment
9 # use POE qw(
10 # Component::Pcap
11 # Component::Daemon
12 # );
13
14 use POE qw(
15 Component::Pcap
16 );
17
18 #
19 # PCAP Options
20 my %pcap_opts = (
21 dev => 'eth0',
22 snaplen => 1514,
23 promisc => 1,
24 timeout => 100,
25 );
26
27 ##
28 # POE Environment Setup
29 ##
30 #POE::Component::Daemon->spawn( detach => 1, babysit => 600, max_children
=> 5 );
31
32 POE::Session->create(
33 inline_states => {
34 _start => \&start_processor,
35 _stop => \&stop_processor,
36 handle_packet => \&handle_packet,
37 },
38 );
39
40 $_[HEAP]->{i} = 0;
41
42 ##
43 # Run the POE Sessions
44 ##
45 POE::Kernel->run;
46
47 ##
48 exit 0;
49 ##
50
51 ##
52 # Start the Processor
53 sub start_processor {
54 my ($kernel, $heap) = @_[KERNEL, HEAP];
55
56 $kernel->alias_set('processor');
57
58 #
59 # Start Packet Capturing
60 POE::Component::Pcap->spawn(
61 Alias => 'pcap',
62 Device=> $pcap_opts{dev},
63 Dispatch=> 'handle_packet',
64 Session => 'processor',
65 );
66 $kernel->post( pcap => open_live => @pcap_opts{qw(dev snaplen promisc
timeout)} );
67
68 $kernel->post( pcap => 'run' );
69 }
70 ##
71
72 ##
73 # stop the processor
74 sub stop_processor {
75 my ($kernel,$heap) = @_[KERNEL,HEAP];
76
77 #
78 # Stop pcap
79 $kernel->post( 'pcap' => 'shutdown' );
80 }
81 ##
82
83 ##
84 sub handle_packet {
85 my $offset = 0;
86 my $linechars = 0;
87 my $hexpacket;
88
89 $_[HEAP]->{i}++;
90 print "sub handle_packet called $_[HEAP]->{i} times\n";
91 my ($kernel,$heap,$packets) = @_[KERNEL,HEAP,ARG0];
92
93 foreach my $inst ( @{ $packets } ) {
94 foreach my $char (split(//, $inst->[1])) {
95 if ( $linechars == 0 ) {
96 $hexpacket .= sprintf( "%04X ", $offset );
97 }
98 $linechars++;
99 $hexpacket .= sprintf( "%02X ", ord($char) );
100 if ( $linechars == 16 ) {
101 $hexpacket .= "\n";
102 $linechars = 0;
103 $offset += 16;
104 }
105 }
106 $hexpacket .= "\n\n";
107 }
108 print $hexpacket;
109 undef($hexpacket);
110 }
111
112 ##
--
Brad Lhotsky
NCTS Computer Specialist -- 410.558.8006
.. WAR IS PEACE,
FREEDOM IS SLAVERY,
IGNORANCE IS STRENGTH ..
Re: Continuous packet capture
Sorry, I thought I found a bug in the code, but it turns out my code was
correct. $inst->[0] is the Packet Header, $inst->[1] is the packet in
raw form.
Jonathan S. Polacheck wrote:
Brad,
Great stuff. Thanks. I'm going on the assumption that $inst->[1] (line
94) is the raw packet, headers and all. I plan to test by pointing pcap at
a capture file and comparing output to input. My next plan is to add a sql
insert of $hexpacket at line 108. My goal is to keep up with an incoming
data stream of 20-30 k-bytes a second. After the hex data is safely stored
in the database I can go back and insert whatever fields (layer 3-4, packet
length, etc) I want. Then do a gui to extract traces and do other analysis
from the stored data. Here is my code;
1 use strict;
2 use warnings;
3
4 use Data::Dumper;
5 use Carp;
6
7 #
8 # POE Environment
9 # use POE qw(
10 # Component::Pcap
11 # Component::Daemon
12 # );
13
14 use POE qw(
15 Component::Pcap
16 );
17
18 #
19 # PCAP Options
20 my %pcap_opts = (
21 dev => 'eth0',
22 snaplen => 1514,
23 promisc => 1,
24 timeout => 100,
25 );
26
27 ##
28 # POE Environment Setup
29 ##
30 #POE::Component::Daemon->spawn( detach => 1, babysit => 600, max_children
=> 5 );
31
32 POE::Session->create(
33 inline_states => {
34 _start => \&start_processor,
35 _stop => \&stop_processor,
36 handle_packet => \&handle_packet,
37 },
38 );
39
40 $_[HEAP]->{i} = 0;
41
42 ##
43 # Run the POE Sessions
44 ##
45 POE::Kernel->run;
46
47 ##
48 exit 0;
49 ##
50
51 ##
52 # Start the Processor
53 sub start_processor {
54 my ($kernel, $heap) = @_[KERNEL, HEAP];
55
56 $kernel->alias_set('processor');
57
58 #
59 # Start Packet Capturing
60 POE::Component::Pcap->spawn(
61 Alias => 'pcap',
62 Device=> $pcap_opts{dev},
63 Dispatch=> 'handle_packet',
64 Session => 'processor',
65 );
66 $kernel->post( pcap => open_live => @pcap_opts{qw(dev snaplen promisc
timeout)} );
67
68 $kernel->post( pcap => 'run' );
69 }
70 ##
71
72 ##
73 # stop the processor
74 sub stop_processor {
75 my ($kernel,$heap) = @_[KERNEL,HEAP];
76
77 #
78 # Stop pcap
79 $kernel->post( 'pcap' => 'shutdown' );
80 }
81 ##
82
83 ##
84 sub handle_packet {
85 my $offset = 0;
86 my $linechars = 0;
87 my $hexpacket;
88
89 $_[HEAP]->{i}++;
90 print "sub handle_packet called $_[HEAP]->{i} times\n";
91 my ($kernel,$heap,$packets) = @_[KERNEL,HEAP,ARG0];
92
93 foreach my $inst ( @{ $packets } ) {
94 foreach my $char (split(//, $inst->[1])) {
95 if ( $linechars == 0 ) {
96 $hexpacket .= sprintf( "%04X ", $offset );
97 }
98 $linechars++;
99 $hexpacket .= sprintf( "%02X ", ord($char) );
100 if ( $linechars == 16 ) {
101 $hexpacket .= "\n";
102 $linechars = 0;
103 $offset += 16;
104 }
105 }
106 $hexpacket .= "\n\n";
107 }
108 print $hexpacket;
109 undef($hexpacket);
110 }
111
112 ##
--
Brad Lhotsky
NCTS Computer Specialist -- 410.558.8006
.. WAR IS PEACE,
FREEDOM IS SLAVERY,
IGNORANCE IS STRENGTH ..
Re: Continuous packet capture
Nosing around in the mailing list archive I find;
"Re: Prioritizing Event Loops
Martijn van Beers
Fri, 18 Jan 2008 11:37:22 -0800
In POE, all your external data, whether from a socket or a file, or
something else entirely, enters your app through a POE::Wheel (you might
not be aware of having one, since it is mostly hidden inside
POE::Component::Server::TCP). How the wheel gets the data is supposed to be
abstracted away.
"
So POE::Component::Pcap is a wheel?
Jon
__
Brad,
Great stuff. Thanks. I'm going on the assumption that $inst->[1] (line
94) is the raw packet, headers and all. I plan to test by pointing pcap at
a capture file and comparing output to input. My next plan is to add a sql
insert of $hexpacket at line 108. My goal is to keep up with an incoming
data stream of 20-30 k-bytes a second. After the hex data is safely stored
in the database I can go back and insert whatever fields (layer 3-4, packet
length, etc) I want. Then do a gui to extract traces and do other analysis
from the stored data. Here is my code;
1 use strict;
2 use warnings;
3
4 use Data::Dumper;
5 use Carp;
6
7 #
8 # POE Environment
9 # use POE qw(
10 # Component::Pcap
11 # Component::Daemon
12 # );
13
14 use POE qw(
15 Component::Pcap
16 );
17
18 #
19 # PCAP Options
20 my %pcap_opts = (
21 dev => 'eth0',
22 snaplen => 1514,
23 promisc => 1,
24 timeout => 100,
25 );
26
27 ##
28 # POE Environment Setup
29 ##
30 #POE::Component::Daemon->spawn( detach => 1, babysit => 600, max_children
=> 5 );
31
32 POE::Session->create(
33 inline_states => {
34 _start => \&start_processor,
35 _stop => \&stop_processor,
36 handle_packet => \&handle_packet,
37 },
38 );
39
40 $_[HEAP]->{i} = 0;
41
42 ##
43 # Run the POE Sessions
44 ##
45 POE::Kernel->run;
46
47 ##
48 exit 0;
49 ##
50
51 ##
52 # Start the Processor
53 sub start_processor {
54 my ($kernel, $heap) = @_[KERNEL, HEAP];
55
56 $kernel->alias_set('processor');
57
58 #
59 # Start Packet Capturing
60 POE::Component::Pcap->spawn(
61 Alias => 'pcap',
62 Device=> $pcap_opts{dev},
63 Dispatch=> 'handle_packet',
64 Session => 'processor',
65 );
66 $kernel->post( pcap => open_live => @pcap_opts{qw(dev snaplen promisc
timeout)} );
67
68 $kernel->post( pcap => 'run' );
69 }
70 ##
71
72 ##
73 # stop the processor
74 sub stop_processor {
75 my ($kernel,$heap) = @_[KERNEL,HEAP];
76
77 #
78 # Stop pcap
79 $kernel->post( 'pcap' => 'shutdown' );
80 }
81 ##
82
83 ##
84 sub handle_packet {
85 my $offset = 0;
86 my $linechars = 0;
87 my $hexpacket;
88
89 $_[HEAP]->{i}++;
90 print "sub handle_packet called $_[HEAP]->{i} times\n";
91 my ($kernel,$heap,$packets) = @_[KERNEL,HEAP,ARG0];
92
93 foreach my $inst ( @{ $packets } ) {
94 foreach my $char (split(//, $inst->[1])) {
95 if ( $linechars == 0 ) {
96 $hexpacket .= sprintf( "%04X ", $offset );
97 }
98 $linechars++;
99 $hexpacket .= sprintf( "%02X ", ord($char) );
100 if ( $linechars == 16 ) {
101 $hexpacket .= "\n";
102 $linechars = 0;
103 $offset += 16;
104 }
105 }
106 $hexpacket .= "\n\n";
107 }
108 print $hexpacket;
109 undef($hexpacket);
110 }
111
112 ##
Re: Continuous packet capture
Brad,
Great stuff. Thanks. I'm going on the assumption that $inst->[1] (line
94) is the raw packet, headers and all. I plan to test by pointing pcap at
a capture file and comparing output to input. My next plan is to add a sql
insert of $hexpacket at line 108. My goal is to keep up with an incoming
data stream of 20-30 k-bytes a second. After the hex data is safely stored
in the database I can go back and insert whatever fields (layer 3-4, packet
length, etc) I want. Then do a gui to extract traces and do other analysis
from the stored data. Here is my code;
1 use strict;
2 use warnings;
3
4 use Data::Dumper;
5 use Carp;
6
7 #
8 # POE Environment
9 # use POE qw(
10 # Component::Pcap
11 # Component::Daemon
12 # );
13
14 use POE qw(
15 Component::Pcap
16 );
17
18 #
19 # PCAP Options
20 my %pcap_opts = (
21 dev => 'eth0',
22 snaplen => 1514,
23 promisc => 1,
24 timeout => 100,
25 );
26
27 ##
28 # POE Environment Setup
29 ##
30 #POE::Component::Daemon->spawn( detach => 1, babysit => 600, max_children
=> 5 );
31
32 POE::Session->create(
33 inline_states => {
34 _start => \&start_processor,
35 _stop => \&stop_processor,
36 handle_packet => \&handle_packet,
37 },
38 );
39
40 $_[HEAP]->{i} = 0;
41
42 ##
43 # Run the POE Sessions
44 ##
45 POE::Kernel->run;
46
47 ##
48 exit 0;
49 ##
50
51 ##
52 # Start the Processor
53 sub start_processor {
54 my ($kernel, $heap) = @_[KERNEL, HEAP];
55
56 $kernel->alias_set('processor');
57
58 #
59 # Start Packet Capturing
60 POE::Component::Pcap->spawn(
61 Alias => 'pcap',
62 Device=> $pcap_opts{dev},
63 Dispatch=> 'handle_packet',
64 Session => 'processor',
65 );
66 $kernel->post( pcap => open_live => @pcap_opts{qw(dev snaplen promisc
timeout)} );
67
68 $kernel->post( pcap => 'run' );
69 }
70 ##
71
72 ##
73 # stop the processor
74 sub stop_processor {
75 my ($kernel,$heap) = @_[KERNEL,HEAP];
76
77 #
78 # Stop pcap
79 $kernel->post( 'pcap' => 'shutdown' );
80 }
81 ##
82
83 ##
84 sub handle_packet {
85 my $offset = 0;
86 my $linechars = 0;
87 my $hexpacket;
88
89 $_[HEAP]->{i}++;
90 print "sub handle_packet called $_[HEAP]->{i} times\n";
91 my ($kernel,$heap,$packets) = @_[KERNEL,HEAP,ARG0];
92
93 foreach my $inst ( @{ $packets } ) {
94 foreach my $char (split(//, $inst->[1])) {
95 if ( $linechars == 0 ) {
96 $hexpacket .= sprintf( "%04X ", $offset );
97 }
98 $linechars++;
99 $hexpacket .= sprintf( "%02X ", ord($char) );
100 if ( $linechars == 16 ) {
101 $hexpacket .= "\n";
102 $linechars = 0;
103 $offset += 16;
104 }
105 }
106 $hexpacket .= "\n\n";
107 }
108 print $hexpacket;
109 undef($hexpacket);
110 }
111
112 ##
Re: Continuous packet capture
I don't know if this will help you or not, but here's a script I have
running to monitor and track recursive DNS Queries on my network. You
can ignore the database stuff, but it might help.
http://divisionbyzero.net/~brad/code/dns_snoop.pl.html
I process the packets as they come in, but I could be using
POE::Wheel::Run on them as well... The key part of the tutorial (http://poe.perl.org/?POE_Cookbook/Child_Processes_3
) seems to be the while() loop in the start_tasks routine.
It doesn't sound like you need the feedback from the child tasks
because you'll be feeding them via the heap. The Filter::Reference
stuff is being used to return status information from the child tasks
to the main processor.
Here's another script I've written that needs some reworking as well.
It listens on our egress link, and collects statistics inside the
heap. At intervals (POE::Component::Cron), the heap is processed
written to a database (or RRD) and then cleared. The processing
continues.
http://divisionbyzero.net/~brad/code/traffic_detection.pl.html
I hope there's something in there that helps.
On Mar 24, 2009, at 4:25 PM, Jonathan S. Polacheck wrote:
I have spent a couple of days casting about, looking at examples and
perldoc. I don't seem to be able to get the data from 'tcpdump' (or
POE::Component::PCAP) to the wheel that will process the data. For
my last
attempt, I removed Filter::Referece and tried with just the wheel.
Still
no luck. Any suggestions?
#!/usr/bin/perl
use warnings;
use strict;
use POE qw( Wheel::Run ); #Filter::Reference );
use Data::Dumper;
our $offset = 0;
our $linechars = '';
POE::Session->create
( inline_states =>
{
_start => sub {
my ($heap) = $_[HEAP];
my $gp = POE::Wheel::Run->new
( Program => '/usr/sbin/tcpdump -i eth0 -w - '
# , Conduit => "pipe"
# , StdoutFilter => POE::Filter::Reference-
>new()
# , StdinEvent => 'process_packet'
, StdinEvent => 'stdin'
# , StdoutEvent => 'print'
# , InputEvent => 'process_packet'
);
print "gp compleate\n";
$heap->{gp} = $gp;
# my $pp = POE::Wheel::Run->new
# ( Program => &process_packet
# , StdoutEvent => 'stdout'
# );
}
}
);
sub process_packet {
print "process_packet called\n";
# my ($heap) = $_[HEAP];
# my $filter = POE::Filter::Reference->new();
# my $pdump = $filter->get( [ $heap->{gp} ] );
my $pdump = $_[ARG0];
foreach my $char (split(//, $pdump)) {
if($char !~ /\n/) {
dump_char($char) ;
} else {
print "\n\n";
$offset = 0;
$linechars = '';
}
1;
}
dump_char( ' ', 1 ) while length($linechars) != 0;
}
sub dump_char {
my ( $char ) = shift;
if ( length( $linechars ) == 0 ) {
printf( "%06X ", $offset );
}
$linechars .= ( $char =~ m#[!-~ ]# ) ? $char : '.';
printf( "%02X ", ord($char) );
if ( length( $linechars ) == 16 ) {
print( "\n" );
$linechars = '';
$offset += 16;
}
}
$poe_kernel->run();
exit 0;
--
Brad Lhotsky
Security Administrator / NIA Alt. ISSO
.. WAR IS PEACE
FREEDOM IS SLAVERY
IGNORANCE IS STRENGTH ..
Re: Continuous packet capture
I have spent a couple of days casting about, looking at examples and
perldoc. I don't seem to be able to get the data from 'tcpdump' (or
POE::Component::PCAP) to the wheel that will process the data. For my last
attempt, I removed Filter::Referece and tried with just the wheel. Still
no luck. Any suggestions?
#!/usr/bin/perl
use warnings;
use strict;
use POE qw( Wheel::Run ); #Filter::Reference );
use Data::Dumper;
our $offset = 0;
our $linechars = '';
POE::Session->create
( inline_states =>
{
_start => sub {
my ($heap) = $_[HEAP];
my $gp = POE::Wheel::Run->new
( Program => '/usr/sbin/tcpdump -i eth0 -w - '
# , Conduit => "pipe"
# , StdoutFilter => POE::Filter::Reference->new()
# , StdinEvent => 'process_packet'
, StdinEvent => 'stdin'
# , StdoutEvent => 'print'
# , InputEvent => 'process_packet'
);
print "gp compleate\n";
$heap->{gp} = $gp;
# my $pp = POE::Wheel::Run->new
# ( Program => &process_packet
# , StdoutEvent => 'stdout'
# );
}
}
);
sub process_packet {
print "process_packet called\n";
# my ($heap) = $_[HEAP];
# my $filter = POE::Filter::Reference->new();
# my $pdump = $filter->get( [ $heap->{gp} ] );
my $pdump = $_[ARG0];
foreach my $char (split(//, $pdump)) {
if($char !~ /\n/) {
dump_char($char) ;
} else {
print "\n\n";
$offset = 0;
$linechars = '';
}
1;
}
dump_char( ' ', 1 ) while length($linechars) != 0;
}
sub dump_char {
my ( $char ) = shift;
if ( length( $linechars ) == 0 ) {
printf( "%06X ", $offset );
}
$linechars .= ( $char =~ m#[!-~ ]# ) ? $char : '.';
printf( "%02X ", ord($char) );
if ( length( $linechars ) == 16 ) {
print( "\n" );
$linechars = '';
$offset += 16;
}
}
$poe_kernel->run();
exit 0;
Re: Continuous packet capture
the tasks is the queue. Wheel::run pops of tasks and runs it in a separate process From: Jon Polacheck To: [email protected] Sent: Sunday, March 15, 2009 7:08:01 AM Subject: Re: Continuous packet capture What I get out the the example; Wheel::Run is used to create tasks (as opposed to processes or sessions) Filter::Reference is used to create a communications channel between the tasks What I am still wondering is if the communications channel would buffer or act as a fifo for the data coming out of the enque task and if prioritizing would be needed to make sure that the enque task would not drop any packets. ___ Alex _ Fri, 13 Mar 2009 15:49:37 -0700 heres an example of producer/consumer with wheel::run http://poe.perl.org/?POE_Cookbook/Child_Processes_3 From: Jonathan S. Polacheck To: [email protected] Sent: Friday, March 13, 2009 1:13:55 PM Subject: Re: Continuous packet capture [email protected] wrote; Your first step is going to be to make sure your code is 'use strict; use warnings;' clean. I'm not saying that your code is not, but since I don't see the strictures I'm making the knee jerk comment. Second, POE might be a fine way to go. There are lots of components available that'll make coding this up easier. Third, what's wrong with using one of the packages you list? and [email protected] replied; strict and warnings is definitely a good idea. But if I port the whole thing to POE, I guess I'll start using them on that code. The two included scripts are just a sort of "proof of concept" I did to get things going. I ultimately may not be able to pull this off in Perl, but it's what I know so I starting there. As for the "packages", if you are referring to Infinistrream, Gigastor, etc, the "problem" in my view is that they are both proprietary and expensive. I believe that continuous packet capture will become a standard way of doing things, supplanting ad-hoc capture (pcap, wireshark, etc), at least in any production environment large enough to require staff knowledgeable to use such tools. That, I think, is the time when open source solutions break into to market. We have Infinistreams in the production environment I work at. We have the lowest end devices at our remote sites. We paid in excess of $10k each for them, plus ongoing support contract costs. They work find, but the vendor (now Netscout) has dropped them from the product line (no replacements, no further updates, end-of-life on the horizon). Netscout has a track record of going for the high end of the market with product development and pricing to match. So I decided that wireshark should evolve to include cpc capabilities. I sent my code to wireshark-dev and was roundly ignored (no time for Perl programmers, perhaps). But no matter, communities are where you find them. So I tried the POE list, and here we are. I agree that there are lots of components. Here's where my POE solution stands; use POE; use strict; use warnings; use Net::Pcap; use POE::Component::Pcap; use Data::Hexdumper qw( hexdump ); use Data::Dumper; use lib 't'; my $dev = "eth0"; my $i = 0; POE::Session->create( inline_states => { _start => \&start, got_packet => \&got_packet, }, ); POE::Kernel->run; sub start { #diag "[POE:start] spawning new Pcap session ", $_[&SESSION]->ID, " on device $dev"; POE::Component::Pcap->spawn( Alias => 'pcap', Device => $dev, Dispatch => 'got_packet', Session => $_[&SESSION], ); $_[&KERNEL]->post(pcap => open_live => $dev, 1514, 1); $_[&KERNEL]->post(pcap => 'run'); } # sub stop { # #diag "[POE:stop]"; # $_[&KERNEL]->post(pcap => 'shutdown'); # } sub got_packet { #diag "[POE:got_packet]"; # $i++; # print "got_packet run $i\n"; my $packets = $_[&ARG0]; # process the first packet only process_packet(@{ $packets->[0] }); # send a message to stop the capture #$_[&KERNEL]->post(pcap => 'shutdown'); } sub process_packet { # my ($pkt) = $_[1]; my $results = hexdump( data => $_[1] , number_format => 'C', ); # print Dumper($header); print $results; } #&start; exit; So I have the hexdump moved into "process_packet" and out of "got_packet". And it;'s easy enough to change "print $results" to a SQL insert statement. But I think I need a que or fifo (HEAP?) to hold the packets and a priority on "get_packet" to make sure it keeps up with POE::Component::Pcap and the incoming traffic. Or is there a better way? Thanks for your interest,
Re: Continuous packet capture
What I get out the the example; Wheel::Run is used to create tasks (as opposed to processes or sessions) Filter::Reference is used to create a communications channel between the tasks What I am still wondering is if the communications channel would buffer or act as a fifo for the data coming out of the enque task and if prioritizing would be needed to make sure that the enque task would not drop any packets. ___ Alex _ Fri, 13 Mar 2009 15:49:37 -0700 heres an example of producer/consumer with wheel::run http://poe.perl.org/?POE_Cookbook/Child_Processes_3 From: Jonathan S. Polacheck To: [email protected] Sent: Friday, March 13, 2009 1:13:55 PM Subject: Re: Continuous packet capture [email protected] wrote; Your first step is going to be to make sure your code is 'use strict; use warnings;' clean. I'm not saying that your code is not, but since I don't see the strictures I'm making the knee jerk comment. Second, POE might be a fine way to go. There are lots of components available that'll make coding this up easier. Third, what's wrong with using one of the packages you list? and [email protected] replied; strict and warnings is definitely a good idea. But if I port the whole thing to POE, I guess I'll start using them on that code. The two included scripts are just a sort of "proof of concept" I did to get things going. I ultimately may not be able to pull this off in Perl, but it's what I know so I starting there. As for the "packages", if you are referring to Infinistrream, Gigastor, etc, the "problem" in my view is that they are both proprietary and expensive. I believe that continuous packet capture will become a standard way of doing things, supplanting ad-hoc capture (pcap, wireshark, etc), at least in any production environment large enough to require staff knowledgeable to use such tools. That, I think, is the time when open source solutions break into to market. We have Infinistreams in the production environment I work at. We have the lowest end devices at our remote sites. We paid in excess of $10k each for them, plus ongoing support contract costs. They work find, but the vendor (now Netscout) has dropped them from the product line (no replacements, no further updates, end-of-life on the horizon). Netscout has a track record of going for the high end of the market with product development and pricing to match. So I decided that wireshark should evolve to include cpc capabilities. I sent my code to wireshark-dev and was roundly ignored (no time for Perl programmers, perhaps). But no matter, communities are where you find them. So I tried the POE list, and here we are. I agree that there are lots of components. Here's where my POE solution stands; use POE; use strict; use warnings; use Net::Pcap; use POE::Component::Pcap; use Data::Hexdumper qw( hexdump ); use Data::Dumper; use lib 't'; my $dev = "eth0"; my $i = 0; POE::Session->create( inline_states => { _start => \&start, got_packet => \&got_packet, }, ); POE::Kernel->run; sub start { #diag "[POE:start] spawning new Pcap session ", $_[&SESSION]->ID, " on device $dev"; POE::Component::Pcap->spawn( Alias => 'pcap', Device => $dev, Dispatch => 'got_packet', Session => $_[&SESSION], ); $_[&KERNEL]->post(pcap => open_live => $dev, 1514, 1); $_[&KERNEL]->post(pcap => 'run'); } # sub stop { # #diag "[POE:stop]"; # $_[&KERNEL]->post(pcap => 'shutdown'); # } sub got_packet { #diag "[POE:got_packet]"; # $i++; # print "got_packet run $i\n"; my $packets = $_[&ARG0]; # process the first packet only process_packet(@{ $packets->[0] }); # send a message to stop the capture #$_[&KERNEL]->post(pcap => 'shutdown'); } sub process_packet { # my ($pkt) = $_[1]; my $results = hexdump( data => $_[1] , number_format => 'C', ); # print Dumper($header); print $results; } #&start; exit; So I have the hexdump moved into "process_packet" and out of "got_packet". And it;'s easy enough to change "print $results" to a SQL insert statement. But I think I need a que or fifo (HEAP?) to hold the packets and a priority on "get_packet" to make sure it keeps up with POE::Component::Pcap and the incoming traffic. Or is there a better way? Thanks for your interest,
Re: Continuous packet capture
heres an example of producer/consumer with wheel::run http://poe.perl.org/?POE_Cookbook/Child_Processes_3 From: Jonathan S. Polacheck To: [email protected] Sent: Friday, March 13, 2009 1:13:55 PM Subject: Re: Continuous packet capture [email protected] wrote; Your first step is going to be to make sure your code is 'use strict; use warnings;' clean. I'm not saying that your code is not, but since I don't see the strictures I'm making the knee jerk comment. Second, POE might be a fine way to go. There are lots of components available that'll make coding this up easier. Third, what's wrong with using one of the packages you list? and [email protected] replied; strict and warnings is definitely a good idea. But if I port the whole thing to POE, I guess I'll start using them on that code. The two included scripts are just a sort of "proof of concept" I did to get things going. I ultimately may not be able to pull this off in Perl, but it's what I know so I starting there. As for the "packages", if you are referring to Infinistrream, Gigastor, etc, the "problem" in my view is that they are both proprietary and expensive. I believe that continuous packet capture will become a standard way of doing things, supplanting ad-hoc capture (pcap, wireshark, etc), at least in any production environment large enough to require staff knowledgeable to use such tools. That, I think, is the time when open source solutions break into to market. We have Infinistreams in the production environment I work at. We have the lowest end devices at our remote sites. We paid in excess of $10k each for them, plus ongoing support contract costs. They work find, but the vendor (now Netscout) has dropped them from the product line (no replacements, no further updates, end-of-life on the horizon). Netscout has a track record of going for the high end of the market with product development and pricing to match. So I decided that wireshark should evolve to include cpc capabilities. I sent my code to wireshark-dev and was roundly ignored (no time for Perl programmers, perhaps). But no matter, communities are where you find them. So I tried the POE list, and here we are. I agree that there are lots of components. Here's where my POE solution stands; use POE; use strict; use warnings; use Net::Pcap; use POE::Component::Pcap; use Data::Hexdumper qw( hexdump ); use Data::Dumper; use lib 't'; my $dev = "eth0"; my $i = 0; POE::Session->create( inline_states => { _start => \&start, got_packet => \&got_packet, }, ); POE::Kernel->run; sub start { #diag "[POE:start] spawning new Pcap session ", $_[&SESSION]->ID, " on device $dev"; POE::Component::Pcap->spawn( Alias => 'pcap', Device => $dev, Dispatch => 'got_packet', Session => $_[&SESSION], ); $_[&KERNEL]->post(pcap => open_live => $dev, 1514, 1); $_[&KERNEL]->post(pcap => 'run'); } # sub stop { # #diag "[POE:stop]"; # $_[&KERNEL]->post(pcap => 'shutdown'); # } sub got_packet { #diag "[POE:got_packet]"; # $i++; # print "got_packet run $i\n"; my $packets = $_[&ARG0]; # process the first packet only process_packet(@{ $packets->[0] }); # send a message to stop the capture #$_[&KERNEL]->post(pcap => 'shutdown'); } sub process_packet { # my ($pkt) = $_[1]; my $results = hexdump( data => $_[1] , number_format => 'C', ); # print Dumper($header); print $results; } #&start; exit; So I have the hexdump moved into "process_packet" and out of "got_packet". And it;'s easy enough to change "print $results" to a SQL insert statement. But I think I need a que or fifo (HEAP?) to hold the packets and a priority on "get_packet" to make sure it keeps up with POE::Component::Pcap and the incoming traffic. Or is there a better way? Thanks for your interest,
Re: Continuous packet capture
[email protected] wrote; Your first step is going to be to make sure your code is 'use strict; use warnings;' clean. I'm not saying that your code is not, but since I don't see the strictures I'm making the knee jerk comment. Second, POE might be a fine way to go. There are lots of components available that'll make coding this up easier. Third, what's wrong with using one of the packages you list? and [email protected] replied; strict and warnings is definitely a good idea. But if I port the whole thing to POE, I guess I'll start using them on that code. The two included scripts are just a sort of "proof of concept" I did to get things going. I ultimately may not be able to pull this off in Perl, but it's what I know so I starting there. As for the "packages", if you are referring to Infinistrream, Gigastor, etc, the "problem" in my view is that they are both proprietary and expensive. I believe that continuous packet capture will become a standard way of doing things, supplanting ad-hoc capture (pcap, wireshark, etc), at least in any production environment large enough to require staff knowledgeable to use such tools. That, I think, is the time when open source solutions break into to market. We have Infinistreams in the production environment I work at. We have the lowest end devices at our remote sites. We paid in excess of $10k each for them, plus ongoing support contract costs. They work find, but the vendor (now Netscout) has dropped them from the product line (no replacements, no further updates, end-of-life on the horizon). Netscout has a track record of going for the high end of the market with product development and pricing to match. So I decided that wireshark should evolve to include cpc capabilities. I sent my code to wireshark-dev and was roundly ignored (no time for Perl programmers, perhaps). But no matter, communities are where you find them. So I tried the POE list, and here we are. I agree that there are lots of components. Here's where my POE solution stands; use POE; use strict; use warnings; use Net::Pcap; use POE::Component::Pcap; use Data::Hexdumper qw( hexdump ); use Data::Dumper; use lib 't'; my $dev = "eth0"; my $i = 0; POE::Session->create( inline_states => { _start => \&start, got_packet => \&got_packet, }, ); POE::Kernel->run; sub start { #diag "[POE:start] spawning new Pcap session ", $_[&SESSION]->ID, " on device $dev"; POE::Component::Pcap->spawn( Alias => 'pcap', Device => $dev, Dispatch => 'got_packet', Session => $_[&SESSION], ); $_[&KERNEL]->post(pcap => open_live => $dev, 1514, 1); $_[&KERNEL]->post(pcap => 'run'); } # sub stop { # #diag "[POE:stop]"; # $_[&KERNEL]->post(pcap => 'shutdown'); # } sub got_packet { #diag "[POE:got_packet]"; # $i++; # print "got_packet run $i\n"; my $packets = $_[&ARG0]; # process the first packet only process_packet(@{ $packets->[0] }); # send a message to stop the capture #$_[&KERNEL]->post(pcap => 'shutdown'); } sub process_packet { # my ($pkt) = $_[1]; my $results = hexdump( data => $_[1] , number_format => 'C', ); # print Dumper($header); print $results; } #&start; exit; So I have the hexdump moved into "process_packet" and out of "got_packet". And it;'s easy enough to change "print $results" to a SQL insert statement. But I think I need a que or fifo (HEAP?) to hold the packets and a priority on "get_packet" to make sure it keeps up with POE::Component::Pcap and the incoming traffic. Or is there a better way? Thanks for your interest,
