Re: [tcpdump-workers] When will a packet filter be ignored/unused?

[Guy Harris]

On Mar 18, 2010, at 8:20 AM, Eloy Paris wrote:

> "pcap_create() and pcap_activate() were not available in versions of
> libpcap prior to 1.0; if you are writing an application that must work on 
> versions of libpcap prior to 1.0, either use pcap_open_live() to get a handle 
> for a live capture or, if you want to be able to use the additional 
> capabilities offered by using pcap_create() and pcap_activate(), use an 
> autoconf(1) script or some other configuration script to check whether the 
> libpcap 1.0 APIs are available and use them only if they are."

That pretty much sums it up.  If you want to be able to control open-time-only 
properties *not* supported by pcap_open_live(), such as monitor mode and 
capture buffer size (for *some* packet capture mechanism the buffer size could 
be set after the device is opened, but BPF is not such a mechanism, so we 
needed something that could set it before the open finishes), you would have to 
use pcap_create() and pcap_activate().

> Guess that's what happens when you read the documentation once and never go 
> back to it after new library versions are released.

Note that the man pages were reorganized for libpcap 1.0.0 - instead of a 
single big pcap(3) man page that documents everything, there's a pcap(3PCAP) 
man page that discusses libpcap as a whole, and individual man pages for 
particular routines.


-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] When will a packet filter be ignored/unused?

[Guy Harris]

On Mar 18, 2010, at 8:02 AM, Jim Lloyd wrote:

> Perhaps someone can clarify this point for me. When is filtering done?

If the packet capture mechanism supports BPF packet filtering in the kernel 
(and the filter isn't too complicated to fit in the kernel or otherwise 
incapable of being handled by the kernel - "ip6 protochain {proto}" requires 
that the BPF program loop, which is *NOT* supported by kernel BPF interpreters, 
so that you can't hand the kernel a BPF program that loops infinitely), the 
filtering is done when the packet is handed to the packet capture mechanism.

If the packet capture mechanism doesn't support BPF packet filtering in the 
kernel (or the filter can't be handled by the kernel), it's done when 
pcap_loop()/pcap_dispatch()/pcap_next()/pcap_next_ex() first looks at the 
packet.

-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] When will a packet filter be ignored/unused?

[Eloy Paris]

On 03/18/10 11:02, Jim Lloyd wrote:


See 'man pcap' and 'man pcap_create'. You don't need pcap_activate() if you
use pcap_open_live().


Ah, I see. I found this gem at the end of the pcap man page (for libpcap 
1.0.0):


"pcap_create() and pcap_activate() were not available in versions of
libpcap prior to 1.0; if you are writing an application that must work 
on versions of libpcap prior to 1.0, either use pcap_open_live() to get 
a handle for a live capture or, if you want to be able to use the 
additional capabilities offered by using pcap_create() and 
pcap_activate(), use an autoconf(1) script or some other configuration 
script to check whether the libpcap 1.0 APIs are available and use them 
only if they are."


Guess that's what happens when you read the documentation once and never 
go back to it after new library versions are released.


Cheers,

Eloy Paris.-
netexpect.org
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] When will a packet filter be ignored/unused?

[Jim Lloyd]

On Thu, Mar 18, 2010 at 7:33 AM, Eloy Paris  wrote:

> On 03/17/10 18:45, Guy Harris wrote:
>
>  On Mar 17, 2010, at 10:54 AM, Jim Lloyd wrote:
>>
>>  I've done some experimentation and determined that apparently I
>>> must call pcap_activate before calling pcap_setfilter.
>>>
>>
>> Yes.
>>
>>  That is counter intuitive, so I wonder if it is by design or not.
>>>
>>
>> It may change at some point.  It is, however, currently a requirement
>> (on all platforms).  It is also a requirement that the pcap_t be
>> activated before you call pcap_compile(); lifting *that* restriction
>> would require that we be able to get the link-layer type for the
>> device before activating it, which may or may not be easy to do.
>>
>> I will update the documentation to indicate that the operations in
>> question (pcap_datalink(), pcap_compile(), pcap_setfilter()) require
>> that the pcap_t be activated *and* make what code changes are
>> necessary to arrange that they fail on un-activated pcap_t's.-
>>
>
> Is this new in libpcap 1.1? If not then I guess my uses of libpcap through
> the years, on a handful of different platforms, have been working by magic
> since I've never called pcap_activate() in my life. I do see a man page for
> pcap_activate() in libpcap 1.0.0, though.
>
> If applications using libpcap services have always been required to call
> pcap_activate() then I'd be curious to know why things have worked without
> doing so (at least for me). To be honest, the first time I heard about
> pcap_activate() was when Jim and Guy brought it up in this discussion.
>

See 'man pcap' and 'man pcap_create'. You don't need pcap_activate() if you
use pcap_open_live().

I think the main reason I switched from using pcap_open_live to using
pcap_create/pcap_activate was due to speculation on my part that the former
could start delivering packets before the filter was set. Now I discover
that the filter can't be installed until after pcap_activate, so the point
is moot.

Perhaps someone can clarify this point for me. When is filtering done? If it
is only done as one of the last steps to deliver a packet as a result of the
application calling pcap_next_ex (or pcap_loop/dispatch) then I can see how
you can guarantee that the first packets will be filtered as intended. But
this would imply that unfiltered packets must be buffered, only to be
discarded later.

Thanks,
Jim Lloyd
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] When will a packet filter be ignored/unused?

[Eloy Paris]

On 03/17/10 18:45, Guy Harris wrote:


On Mar 17, 2010, at 10:54 AM, Jim Lloyd wrote:


I've done some experimentation and determined that apparently I
must call pcap_activate before calling pcap_setfilter.


Yes.


That is counter intuitive, so I wonder if it is by design or not.


It may change at some point.  It is, however, currently a requirement
(on all platforms).  It is also a requirement that the pcap_t be
activated before you call pcap_compile(); lifting *that* restriction
would require that we be able to get the link-layer type for the
device before activating it, which may or may not be easy to do.

I will update the documentation to indicate that the operations in
question (pcap_datalink(), pcap_compile(), pcap_setfilter()) require
that the pcap_t be activated *and* make what code changes are
necessary to arrange that they fail on un-activated pcap_t's.-


Is this new in libpcap 1.1? If not then I guess my uses of libpcap 
through the years, on a handful of different platforms, have been 
working by magic since I've never called pcap_activate() in my life. I 
do see a man page for pcap_activate() in libpcap 1.0.0, though.


If applications using libpcap services have always been required to call 
pcap_activate() then I'd be curious to know why things have worked 
without doing so (at least for me). To be honest, the first time I heard 
about pcap_activate() was when Jim and Guy brought it up in this discussion.


Cheers,

Eloy Paris.-
netexpect.org
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] When will a packet filter be ignored/unused?

[Guy Harris]

On Mar 17, 2010, at 10:54 AM, Jim Lloyd wrote:

> So, what does an error code of -3 indicate?

#define PCAP_ERROR_NOT_ACTIVATED-3  /* the capture needs to 
be activated */

> I've done some experimentation and determined that apparently I must call
> pcap_activate before calling pcap_setfilter.

Yes.

> That is counter intuitive, so I wonder if it is by design or not.

It may change at some point.  It is, however, currently a requirement (on all 
platforms).  It is also a requirement that the pcap_t be activated before you 
call pcap_compile(); lifting *that* restriction would require that we be able 
to get the link-layer type for the device before activating it, which may or 
may not be easy to do.

I will update the documentation to indicate that the operations in question 
(pcap_datalink(), pcap_compile(), pcap_setfilter()) require that the pcap_t be 
activated *and* make what code changes are necessary to arrange that they fail 
on un-activated pcap_t's.-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] When will a packet filter be ignored/unused?

[Jim Lloyd]

On Tue, Mar 16, 2010 at 4:40 PM, Jim Lloyd wrote:

> I have a working application using libpcap that doesn't always filter as I
> expect. The application is designed to sniff http traffic, so the filter can
> be as simple as "tcp port 80". However, we allow sniffing multiple http
> servers running on different ports, so it is common to use a filter like
> "tcp and (port 80 or port 8080)". Because of this, when sniffing a single
> port, the filter we use will look like "tcp and (port 80)".
>
> This filter seems to work correctly "most" of the time, but in varying
> situations we start to see packets where neither the source port or the
> destination port matches the specified port. Until today, I only noticed
> this for relatively high volume packet sniffing (say 250mbps). But now I am
> seeing it on one of our machines with a relatively modest volume of traffic
> (about 18mbps).
>
> I imagine it might be driver specific, and now I see that the ethernet
> interfaces on the servers I have available for testing have a more diverse
> set of drivers than I was aware. Using ethtool and checking a few different
> interfaces, I see drivers bnx2, tg3, and forcedeth. I seem to be having
> problems with bnx2. Is this a known issue?
>
> No error code is returned when we compile and install the filter. Is there
> any way to determine if a filter is being ignored?
>
>
It turns out I was mistaken. An error code of -3 is being returned by
pcap_setfilter. Instead of checking for a function result of 0, and assuming
anything else was an error, my code was checking for a function result of
-1, and assuming anything else was success. My bad.

So, what does an error code of -3 indicate? I've skimmed the source code and
haven't found a path where -3 would be returned.

I've done some experimentation and determined that apparently I must call
pcap_activate before calling pcap_setfilter. That is counter intuitive, so I
wonder if it is by design or not.

FYI this is all on linux, using kernel 2.6.18, and using the recently
release libpcap 1.1.

Thanks,
Jim Lloyd
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] When will a packet filter be ignored/unused?

[Michael Richardson]

> "Jim" == Jim Lloyd  writes:
Jim> I imagine it might be driver specific, and now I see that the
Jim> ethernet interfaces on the servers I have available for testing
Jim> have a more diverse set of drivers than I was aware. Using
Jim> ethtool and checking a few different interfaces, I see drivers
Jim> bnx2, tg3, and forcedeth. I seem to be having problems with
Jim> bnx2. Is this a known issue?

If you take the resulting pcap file and run it again through the same
filter, does it drop the unwanted packets?

I ask because that might rule out something high-level that was
indeterminate.

I don't know what a bnx2 driver is.  Based upon the other things you
mention, I'm guessing this is Linux. (You should say. Tcpdump runs on
dozens of Unixes)

Does bnx2 do any hardware offload?
Is there any 802.3 encoding occuring (VLAN, QOS, etc) on that link?

-- 
]   He who is tired of Weird Al is tired of life!   |  firewalls  [
]   Michael Richardson, Sandelman Software Works, Ottawa, ON|net architect[
] m...@sandelman.ottawa.on.ca http://www.sandelman.ottawa.on.ca/ |device driver[
   Kyoto Plus: watch the video 
   then sign the petition. 
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

Re: [tcpdump-workers] When will a packet filter be ignored/unused?

[Darren Reed]

On 16/03/10 04:40 PM, Jim Lloyd wrote:

I have a working application using libpcap that doesn't always filter as I
expect. The application is designed to sniff http traffic, so the filter can
be as simple as "tcp port 80". However, we allow sniffing multiple http
servers running on different ports, so it is common to use a filter like
"tcp and (port 80 or port 8080)". Because of this, when sniffing a single
port, the filter we use will look like "tcp and (port 80)".

This filter seems to work correctly "most" of the time, but in varying
situations we start to see packets where neither the source port or the
destination port matches the specified port. Until today, I only noticed
this for relatively high volume packet sniffing (say 250mbps). But now I am
seeing it on one of our machines with a relatively modest volume of traffic
(about 18mbps).

I imagine it might be driver specific, and now I see that the ethernet
interfaces on the servers I have available for testing have a more diverse
set of drivers than I was aware. Using ethtool and checking a few different
interfaces, I see drivers bnx2, tg3, and forcedeth. I seem to be having
problems with bnx2. Is this a known issue?

No error code is returned when we compile and install the filter. Is there
any way to determine if a filter is being ignored?


You might want to verify that libpcap is generating the correct byte 
codes with tcpdump with something like this:


tcpdump -d 'tcp and (port 80)'
(000) ldh  [12]
(001) jeq  #0x86dd  jt 2jf 8  # ipv6?
(002) ldb  [20]   # yes...
(003) jeq  #0x6 jt 4jf 19 # tcp?
(004) ldh  [54]   # yes...
(005) jeq  #0x50jt 18   jf 6  # port 80?
(006) ldh  [56]   # no...
(007) jeq  #0x50jt 18   jf 19 # port 80?
(008) jeq  #0x800   jt 9jf 19 # ipv4?
(009) ldb  [23]   # yes...
(010) jeq  #0x6 jt 11   jf 19 # tcp?
(011) ldh  [20]   # yes...
(012) jset #0x1fff  jt 19   jf 13 # compute start of tcp
(013) ldxb 4*([14]&0xf)
(014) ldh  [x + 14]
(015) jeq  #0x50jt 18   jf 16 # port 80?
(016) ldh  [x + 16]   # no...
(017) jeq  #0x50jt 18   jf 19 # port 80?
(018) ret  #96# yes..
(019) ret  #0

If the byte code verifies correctly then you might want to do a full 
packet header dump of those that get to you and aren't meant to - maybe 
there is something different about them...


-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

[tcpdump-workers] When will a packet filter be ignored/unused?

[Jim Lloyd]

I have a working application using libpcap that doesn't always filter as I
expect. The application is designed to sniff http traffic, so the filter can
be as simple as "tcp port 80". However, we allow sniffing multiple http
servers running on different ports, so it is common to use a filter like
"tcp and (port 80 or port 8080)". Because of this, when sniffing a single
port, the filter we use will look like "tcp and (port 80)".

This filter seems to work correctly "most" of the time, but in varying
situations we start to see packets where neither the source port or the
destination port matches the specified port. Until today, I only noticed
this for relatively high volume packet sniffing (say 250mbps). But now I am
seeing it on one of our machines with a relatively modest volume of traffic
(about 18mbps).

I imagine it might be driver specific, and now I see that the ethernet
interfaces on the servers I have available for testing have a more diverse
set of drivers than I was aware. Using ethtool and checking a few different
interfaces, I see drivers bnx2, tg3, and forcedeth. I seem to be having
problems with bnx2. Is this a known issue?

No error code is returned when we compile and install the filter. Is there
any way to determine if a filter is being ignored?

Thanks,
Jim Lloyd
Silver Tail Systems
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.