Re: DPI for pf(4)

[Franco Fichtner]

Hi Stuart,

On May 1, 2013, at 1:11 AM, Stuart Henderson st...@openbsd.org wrote:

 On 2013/05/01 00:16, Franco Fichtner wrote:
 
 Yes, I am proposing a lightweight approach: hard-wired regex-like
 code, no allocations, no reassembly or state machines.  I've seen
 far worse things being put into Kernels and I assure you that I do
 refrain from putting in anything that could cause segmentation
 faults, sleeps, or other non-suitable behaviour.
 
 Would it be fair to describe it as a bit more complex than osfp,
 but not hugely so?

Not sure if that's a fitting comparison; and I know too little OSPF
to answer.  Let me try another route.  The logic consists of an array
of application detection functions, which can be invoked via their
respective IP types.  There's 32 bits of external state for the
table and a single hook into the application detection.  And the
detection for TLS/SSL3.0 follows.  I have really tried to condense
it down to the bare minimum.

LI_DESCRIBE_APP(tls)
{
struct tls {
uint8_t record_type;
uint16_t version;
uint16_t data_length;
} __packed *ptr = (void *)packet-app.raw;
uint16_t decoded;

if (packet-app_len  sizeof(struct tls)) {
return (0);
}

decoded = be16dec(ptr-data_length);

if (!decoded || decoded  0x4000) {
/* no empty records possible, also = 2^14 */
return (0);
}

switch (ptr-record_type) {
case 20:/* change_cipher_spec */
case 21:/* alert */
case 22:/* handshake */
case 23:/* application_data */
break;
default:
return (0);
}

switch (be16dec(ptr-version)) {
case 0x0300:/* SSL 3.0 */
case 0x0301:/* TLS 1.0 */
case 0x0302:/* TLS 1.1 */
case 0x0303:/* TLS 1.2 */
break;
default:
return (0);
}

return (1);
}

 Would a protocol like BGP have a bright future in relayd(8)?
 I don't know enough, maybe Reyk can clear this up?
 
 L7 filtering is cute, but ipfw-classifyd isn't maintained, DPI in
 Linux netfilter is not hitting it off, and there really is no
 BSD DPI.  Franky, I don't care which way to go, but I believe
 that pf(4) is a suitable candidate.  I especially like the one-
 rule-to-rule-them-all approach.  Adding a keyword app to
 pf.conf(5) seems like the simplest solution -- much like proto
 does deal with IP types.
 
 And talking about complexity: 1000 LOC for 25 protocols.  I'm afraid
 it can't be simplified any more than this.
 
 What sort of protocols do you think could be reasonably handled by
 this approach, and what would be too complicated?

Good question!  Text protocols are easy, RFCs and open implementations
are generally easy.  Anything too commercial/proprietary, especially
in binary, is more guessing than anything else and may not be worth
the effort.  I don't see world of warcraft happening as a supported
application.  This is what I have done so far (by no means free of
errors, though):

-- BitTorrent
-- Gnutella
-- Network Basic Input Output System
-- Telecommunication Network
-- Hypertext Transfer Protocol
-- Post Office Protocol (Version 3)
-- Internet Message Access Protocol
-- Simple Mail Transfer Protocol
-- Session Traversal Utilities for NAT
-- Dynamic Host Configuration Protocol
-- Point-to-Point Tunneling Protocol
-- Lightweight Directory Access Protocol
-- Simple Network Management Protocol
-- Secure Shell
-- File Transfer Protocol
-- Session Initiation Protocol
-- Domain Name System
-- Real-time Transport Control Protocol
-- Real-time Transport Protocol
-- Routing Information Protocol
-- Boarder Gateway Protocol
-- Internet Key Exchange
-- Datagram Transport Layer Security
-- Transport Layer Security
-- Concurrent Versions System

 There is definitely something appealing about being able to say, for
 example, 'block proto tcp on port 443; pass proto tcp on port 443 app tls',
 or 'block app ssh; pass proto tcp from somehosts to port 22 app ssh'
 without a bunch more complexity involved in passing across to a separate
 proxy (which would then need to implement its own completely separate
 filtering and would, I think, not really be able to integrate with
 things like PF tags and queue assignment)...

Yes, that would be one scenario.  I like to think of lightweight packet
inspection as application tagging.  That's the first stage.  Second
stage is a real parser/proxy/endpoint.  It's not a security functionality
per se, but it can help to break down the workload.  It doesn't care
aboute IP versions, ports (mostly ;) ), different flavours (netbios
could be session, datagram, and name service as one for example), and so
forth.

 Basically what I'm wondering if it's possible to go far enough to be
 useful whilst keeping the complexity down to a level which is sane
 and 

Re: DPI for pf(4)

[Franco Fichtner]

Hi Ted,

On May 1, 2013, at 1:14 AM, Ted Unangst t...@tedunangst.com wrote:

 On Wed, May 01, 2013 at 00:16, Franco Fichtner wrote:
 Yes, I am proposing a lightweight approach: hard-wired regex-like
 code, no allocations, no reassembly or state machines.  I've seen
 far worse things being put into Kernels and I assure you that I do
 refrain from putting in anything that could cause segmentation
 faults, sleeps, or other non-suitable behaviour.
 
 And talking about complexity: 1000 LOC for 25 protocols.  I'm afraid
 it can't be simplified any more than this.
 
 Well, it's really hard to comment on code we can't see.

I understand.  The code is hooked up to a library feeding off of
recorded network traces at the moment.  The idea doesn't feel mature
enough to me at this time, not knowing where to put it.  So there's
no point in releasing a half-done code blob that does nothing on its
own, but I'm willing to share it off-list with OpenBSD developers.

 My thoughts on the matter have always been that it would be cool to
 integrate bpf into pf (though other developers surely have other
 opinions). Then you get filtering for as many protocols as you care to
 write bpf matchers for.

You mean externalising the DPI?  People[1] have tried to work on such
ideas, but the general drift is that there are not enough interested
individuals in the field to drive second tier development for
application detections.  I find C to be quite flexible and empowering
if one doesn't overcomplicate[2].


Franco

[1] https://code.google.com/p/appid/source/browse/trunk/apps/aim
[2] https://github.com/fichtner/OpenDPI/blob/master/src/lib/protocols/ssl.c

Re: DPI for pf(4)

[Stuart Henderson]

On 2013/05/01 09:01, Franco Fichtner wrote:
 Hi Stuart,
 
 On May 1, 2013, at 1:11 AM, Stuart Henderson st...@openbsd.org wrote:
 
  On 2013/05/01 00:16, Franco Fichtner wrote:
  
  Yes, I am proposing a lightweight approach: hard-wired regex-like
  code, no allocations, no reassembly or state machines.  I've seen
  far worse things being put into Kernels and I assure you that I do
  refrain from putting in anything that could cause segmentation
  faults, sleeps, or other non-suitable behaviour.
  
  Would it be fair to describe it as a bit more complex than osfp,
  but not hugely so?
 
 Not sure if that's a fitting comparison; and I know too little OSPF
 to answer.

I should have expanded the acronum to make it clear - osfp i.e. the
OS fingerprinting code (pf_osfp.c).

 Let me try another route.  The logic consists of an array
 of application detection functions, which can be invoked via their
 respective IP types.  There's 32 bits of external state for the
 table and a single hook into the application detection.  And the
 detection for TLS/SSL3.0 follows.  I have really tried to condense
 it down to the bare minimum.
 
 LI_DESCRIBE_APP(tls)
 {
 struct tls {
 uint8_t record_type;
 uint16_t version;
 uint16_t data_length;
 } __packed *ptr = (void *)packet-app.raw;
 uint16_t decoded;
 
 if (packet-app_len  sizeof(struct tls)) {
 return (0);
 }
 
 decoded = be16dec(ptr-data_length);
 
 if (!decoded || decoded  0x4000) {
 /* no empty records possible, also = 2^14 */
 return (0);
 }
 
 switch (ptr-record_type) {
 case 20:/* change_cipher_spec */
 case 21:/* alert */
 case 22:/* handshake */
 case 23:/* application_data */
 break;
 default:
 return (0);
 }
 
 switch (be16dec(ptr-version)) {
 case 0x0300:/* SSL 3.0 */
 case 0x0301:/* TLS 1.0 */
 case 0x0302:/* TLS 1.1 */
 case 0x0303:/* TLS 1.2 */
 break;
 default:
 return (0);
 }
 
 return (1);
 }

This type of thing looks sane to me, but others will want to comment. (I'll
point others at your posts at http://lastsummer.de/category/technology/
too :-)

  Would a protocol like BGP have a bright future in relayd(8)?
  I don't know enough, maybe Reyk can clear this up?
  
  L7 filtering is cute, but ipfw-classifyd isn't maintained, DPI in
  Linux netfilter is not hitting it off, and there really is no
  BSD DPI.  Franky, I don't care which way to go, but I believe
  that pf(4) is a suitable candidate.  I especially like the one-
  rule-to-rule-them-all approach.  Adding a keyword app to
  pf.conf(5) seems like the simplest solution -- much like proto
  does deal with IP types.
  
  And talking about complexity: 1000 LOC for 25 protocols.  I'm afraid
  it can't be simplified any more than this.
  
  What sort of protocols do you think could be reasonably handled by
  this approach, and what would be too complicated?
 
 Good question!  Text protocols are easy, RFCs and open implementations
 are generally easy.  Anything too commercial/proprietary, especially
 in binary, is more guessing than anything else and may not be worth
 the effort.  I don't see world of warcraft happening as a supported
 application.  This is what I have done so far (by no means free of
 errors, though):
 
 -- BitTorrent
 -- Gnutella
 -- Network Basic Input Output System
 -- Telecommunication Network
 -- Hypertext Transfer Protocol
 -- Post Office Protocol (Version 3)
 -- Internet Message Access Protocol
 -- Simple Mail Transfer Protocol
 -- Session Traversal Utilities for NAT
 -- Dynamic Host Configuration Protocol
 -- Point-to-Point Tunneling Protocol
 -- Lightweight Directory Access Protocol
 -- Simple Network Management Protocol
 -- Secure Shell
 -- File Transfer Protocol
 -- Session Initiation Protocol
 -- Domain Name System
 -- Real-time Transport Control Protocol
 -- Real-time Transport Protocol
 -- Routing Information Protocol
 -- Boarder Gateway Protocol
 -- Internet Key Exchange
 -- Datagram Transport Layer Security
 -- Transport Layer Security
 -- Concurrent Versions System
 
  There is definitely something appealing about being able to say, for
  example, 'block proto tcp on port 443; pass proto tcp on port 443 app tls',
  or 'block app ssh; pass proto tcp from somehosts to port 22 app ssh'
  without a bunch more complexity involved in passing across to a separate
  proxy (which would then need to implement its own completely separate
  filtering and would, I think, not really be able to integrate with
  things like PF tags and queue assignment)...
 
 Yes, that would be one scenario.  I like to think of lightweight packet
 inspection as application tagging.  That's the first stage.  Second
 

Re: DPI for pf(4)

[Jiri B]

On Tue, Apr 30, 2013 at 07:14:50PM -0400, Ted Unangst wrote:
 On Wed, May 01, 2013 at 00:16, Franco Fichtner wrote:
  Yes, I am proposing a lightweight approach: hard-wired regex-like
  code, no allocations, no reassembly or state machines.  I've seen
  far worse things being put into Kernels and I assure you that I do
  refrain from putting in anything that could cause segmentation
  faults, sleeps, or other non-suitable behaviour.
 
  And talking about complexity: 1000 LOC for 25 protocols.  I'm afraid
  it can't be simplified any more than this.
 
 Well, it's really hard to comment on code we can't see.
 
 My thoughts on the matter have always been that it would be cool to
 integrate bpf into pf (though other developers surely have other
 opinions). Then you get filtering for as many protocols as you care to
 write bpf matchers for.

My first thought was why not to have something like squid does (ICAP)
you can forward some inspection to other app and it would return
you some agreed data (tag) and then you could work with then in
pf rules... ???

Re: DPI for pf(4)

[Franco Fichtner]

On May 1, 2013, at 9:41 AM, Stuart Henderson st...@openbsd.org wrote:

 I should have expanded the acronum to make it clear - osfp i.e. the
 OS fingerprinting code (pf_osfp.c).

oh, sorry, my mistake.  This I can comment on. :)

The idea is the same.  I'd say at this stage osfp has more complexity
due to parsing the TCP header, splitting fields, pulling in external
descriptions, etc.  Looking beyond the headers is far less structured,
because applications do the structuring on their own, which in turn
makes external descriptions hard to, er, describe -- hence the hard-
wired C approach.  The only complexity is the growing amount of
application descriptions, but each application function is completely
isolated.

Here's the DPI hook function (a bit simplified for the context of this
discussion):

li_get(const struct li_packet *packet, const struct li_flow *flow)
{
unsigned int i;

if (!packet-app_len) {
return (LI_UNKNOWN);
}

for (i = 0; i  lengthof(apps); ++i) {
if ((apps[i].p1 == flow-type) ||
(apps[i].p2 == flow-type)) {
if (apps[i].function(packet, flow)) {
return (apps[i].number);
}
}
}

/*
 * Set 'undefined' right away. Only one chance for
 * each side of the flow. This makes it easier for
 * a rules engine to do negation of policies.
 */
return (LI_UNDEFINED);
}

apps is an array of all of the available application functions. It looks
something like this:

static const struct li_apps apps[] = {
LI_LIST_APP(LI_PPTP, pptp, IPPROTO_TCP, IPPROTO_GRE),
LI_LIST_APP(LI_HTTP, http, IPPROTO_TCP, IPPROTO_MAX),
/* more stuff here */
};

Really, that's all there is to it.

 So another example might be: pass proto tcp app $someapp divert-packet
 $someproxy, with $someproxy handling the second stage?

Yes, that looks reasonable.  proto tcp may be zapped as well.
If we are talking use cases the biggest ones would be traffic shaping
and policy enforcement in general (no SMTP to the outside, blocking
non-TLS stuff on port 443, etc.)

 Yes, this is clearly a less messy approaach than opendpi ;)

I probably shouldn't say I worked for these guys a few years ago.
Nobody would believe me I never touched the DPI code, but it's the
truth!


Franco

Re: Test needed: ehci(4) suspend/resume rework

[Ted Unangst]

On Sun, Apr 28, 2013 at 15:44, Martin Pieuchot wrote:
 Diff below is a rework of the suspend/resume logic in ehci(4).
 

 In case this diff doesn't help or if you have a problem when resuming,
 I left an #ifdef 0 block in the DVACT_RESUME. Try enabling it and tell
 me if it changes something.

Got around to testing this. Now everything works. It still prints
echi_idone about 100 times after resume, but it doesn't print it
forever.

I'd say the diff works, but only with the reset in the resume case as
well.

Re: faster fast grep

[Ted Unangst]

On Wed, May 01, 2013 at 04:28, J?r?mie Courr?ges-Anglas wrote:
 Ted Unangst t...@tedunangst.com writes:
 
 For simple patterns, grep has an optimization to avoid regex and run
 about 50% faster. The problem is its idea of simple patterns is too
 simple.
 
 IIUC the idea is to optimize for a lazy user that didn't want to
 type grep -F or fgrep:

More or less. Note that it does support . wildcards unlike -F. I also
have scripts that run grep where it would be annoying to get -F to
grep when necessary.

 The following diff also tests for ']', ')' and '}' so that unbalanced
 use of those can be catched by regcomp if the latter happens to become
 stricter.
 
 Regress passes; does it seem OK?

I think that's a good improvement.