Re: Need stateless NAT
On Apr 8, 2008, at 11:59 PM, Adam Richards wrote: You're looking at creating 1:1 mappings from internal IPs to 150-500k public IPs. No. Sorry, I should've been clearer: 1:1 mappings between, say, a / 18 worth of public IP space to something like a /13 worth of possible private IP space. Okay. It's not 1:1 over the entire mathematical set, obviously, but rather 1:1 between (public, private) IP pairs for an arbitrarily defined duration. Maybe a requirement is that I'll never have more concurrent connections than I have public IPs? Yes. Given a /18 as your smallest address pool, you can have a max of ~~16k 1:1 mappings. Maybe this will change in the future to where I may need 1:n mapping? Note that 1:N requires flow tracking, even if it's only as granular as remote:internal/remote:external address pairs. However it is clear that in stateful NATing with your typical non- TE'd network, return traffic will always come back through the same NAT device irrespective of the internal server's network locality to the Internet. That is, the internal server (or VIP) may live closer, topologically, to Internet Feed B, whereas the NAT device may live closer to Internet Feed A. Thus depending on where internal services live, standard stateful NATing may inhibit return traffic from egressing the *nearest* exit. This is the opposite of what I want! But it's a requirement inherent in NAT itself; the traffic must go through the same mapping in both directions. It's not a matter of stateful vs stateless, it's a matter of where you do NAT. (I think we're on the same page but differing in use of state, see below.) * You want to distribute NAT by implementing it on or near each border. In order for anything bidirectional to function, the mappings must be consistent Yup. so that implies synchronizing state between them. Well, it depends on what we mean by synchronizing state. I'm using state in a general sense here, not in pf-specific terms. Static NAT is the kind of thing you'd set up on a simple rule-based engine, like a couple binat lines to map one block to another of the same size. I'd call this stateless. Dynamic NAT is what you get when you've got more addresses on one side than the other, and need to apply policy decisions to allocate and remove mappings. Since you have to keep track of the NAT mappings in use, I consider this stateful. Besides internal-external address mappings, pf's state engine also tracks remote address, protocol, TCP/UDP port numbers, and TCP connection state and sequence numbers. The TCP connection state and sequence checks are firewall operations, but the rest apply to NAT. This is just a matter of how much state is tracked, and determines how large N can be for 1:N scenarios. I want to have the same mapping information-base exist everywhere, but NAT state (ie, flow state matching) is NOT required nor needed. Just a simple unidirectional L3 translation and then move on our merry way. I say unidirectional to mean that the NAT device doesn't drop a packet if it results in a flow-matching miss. This is just the old style dumb nating from yesteryear... as a matter of fact, the same NAT which used to exist on the linux side of the house (and incidentally is being re-introduced; again, another part of the thread mentioned above http://kerneltrap.org/mailarchive/linux-netdev/2007/9/27/323772 ) As Ryan mentioned, pf isn't equipped for this. pf is designed as a firewall first, and happens to have NAT abilities as an easy bonus. As such, the rule-driven part of it expects to have a relatively small ruleset compared to the number of states to be tracked. The ruleset requires a linear scan and simply won't scale to very large numbers of rules, but does support partial matching, while the state engine is built to handle large numbers of exact matches. (That tends to be true of a lot of systems, including the iptables engine, hence the patch above to use a completely different method of specifying mappings. iptables can do it, it just can't scale.) In order to get pf doing what you want, you'll essentially have to cripple the state engine to first remove the firewalling logic, then scale back the amount of state tracked until it's as dumb as you want. Then add an interface to add entries to the state table itself, instead of using the ruleset. Or build your own engine based off of pf's state organization. Honestly, if that netfilter patch does what you want, I'd seriously consider using it. It'd be far easier than ripping apart pf. (It won't do 1:N, but I suspect you'll need a different approach for that anyway.) I would like preface my inline replies with stating what my original goal was, and still is, in starting this thread: I want to pursuade the pf community that stateless NAT is a desired feature and should be part of the core
Re: Need stateless NAT
On Mon, Apr 14, 2008 at 06:50:24PM -0700, Adam Richards wrote: And there's another nuance as well: on ingress I need dest re-mapped while preserving src, Yes, that's how binat works. and on egress I need src re-mapped while passing on the [preserved] src as the egress dest. I'm not sure I understand this bit. You want to reflect packets back to the source? Or by src here you actually mean the dst? To be clear, lets talk about this from the point of view of the ip header fields in the packet, not the source of the connection. You want this to be stateless, so we have to consider each packet independantly. You're right, it should be relatively easy to give binat a 'no state' option... But not for a /18 of arbitrary mappings with a high rate of change. I'm not so concerned with arbitrary mappings as I'll be statically configuring the mappings, maintaining them from outside pf. By arbitrary, I meant essentially random ie, You're not talking about mapping 10.29.4.0/18 to 192.168.64.0/18, you're talking about 10.29.4.192 to 192.168.67.12 10.29.5.1 to 192.168.66.250 10.29.4.9 to 192.168.64.99 [ cut 2^14 examples ] To your last sentence, what if they're static mappings, ie - not arbitrary, with a high rate of change? ;) With the current code, this will be super slow. Either you're reloading entire rulesets, or managing 2^14 anchors. Lots of linked-list walking involved. What happens in pf when a table has changed and pf need to re-read it? Will pkts get dropped? Adding/removing single entries from a table is a /relatively/ inexpensive and atomic operation, but it involves some cost to cross from userland to kernel. I imagine it can be pretty fast using a custom tool rather than calling pfctl. With the current translation code this would require a rule for every mapping, This is how I plan on using pf -- 1:1 statically configured translations. IOW I don't care about free-floating addr pools. Right. Although the free-floating pools might actually be easier to implement. and every packet is going to require a linear search of this ruleset. Fixing this is going to require fairly major changes to how binat works. Major changes, *if* we want binat to work with pools, right? Well, tables. I have plans to remove the 'pools' bits from the pf translation and routing code. It's basically a duplication of the tables functionality. If it the core binat code remains unchanged, I'd guess modifying the search algorithm to be something as simple as a bubble sort, or maybe a radix tree implementation (as is common in networking code I believe), would be fairly easy to someone familiar with pf's inner-workings. I could be over-simplifying it. radix tree implementation: that's what the tables are. There are already some other optimizations in pf and pfctl that deal with the ruleset, but they won't help too much case (dealing with a long list of nearly-identical rules). There may be some ways to optimize for this case, but I'm not sure it's worth the trouble. I'll have to think about this more. BTW: What kind of packet forwarding rate are you hoping to get with this solution? To be on par with my Linux colleagues running the latest netfilter/iptables code I'd need to get = 1Mpps on 10G links. Even though pkt sizes will obviously influence pps, most flows I deal with are rather short lived and contain, on avg, pkt sizes = 512B. I expect to find predictible inflection points. You will most certainly NOT get this with the current code, nor the diff I've posted. -Ryan
Re: Need stateless NAT
Trevor: I mostly agree with your analysis, and without more information about the actual problem Adam is trying to solve I'm chalking it all up to horrendous network design. That being said, part of PFs usefulness is it's ability to make some horrendous network situations manageable. So I don't think the ugliness of the problem precludes us solving it... especially since some of the actual issues with this (cost of table/ruleset updates, rule matching performance) are issues that affect many other pf users. A couple of inline comments: On Mon, Apr 14, 2008 at 07:49:43PM -0700, Trevor Talbot wrote: Maybe this will change in the future to where I may need 1:n mapping? Note that 1:N requires flow tracking, even if it's only as granular as remote:internal/remote:external address pairs. Somewere low down on my todo list is to use the src-nodes code in pf to handle M:N binat mappings, where M can be smaller than N, so long as no more than M nodes are active in a given period. In fact, something like this could possibly solve Adam's problem, given a tool to add/remove src-node entries. In order to get pf doing what you want, you'll essentially have to cripple the state engine to first remove the firewalling logic, then scale back the amount of state tracked until it's as dumb as you want. Then add an interface to add entries to the state table itself, instead of using the ruleset. Or build your own engine based off of pf's state organization. I don't think it's necessary to cripple any of pf's current behaviour to support what Adam wants, but it's certainly not going to be a trivial change to do it cleanly. That being said, a change which result in a simplification of this code is definately something we'd consider, and there's definately room for that in this code. And to Adam's comment: I would like preface my inline replies with stating what my original goal was, and still is, in starting this thread: I want to pursuade the pf community that stateless NAT is a desired feature and should be part of the core code. :) My use-case is just one example of how this can be a useful feature. I'm not convinced of this yet, but I agree that it's an interesting problem. I'm looking forward to seeing your reply to Trevor's analysis. -Ryan
Re: Need stateless NAT
Adam Richards wrote: I need to be able to create *stateless* nat rules for at least 150,000 entries, potentially to grow to 1/2million entries. The reason has to do with being able to work in an asymetric routing environment -- stateless nat must be used because traffic might not egress at the same location it ingressed. In other words I want to do a unidirectional translation and then fahgettaboutit. [...] there are some unique requirements this network brings which make public IP consumption unworkable. [...] table-management operations -- perhaps 10's of operations per second on a table of 500,000 entries/mappings. (operations, like inserts or deletes) Maybe for resilliency, or for ISP cost reasons, you want to take advantage of asymetric routing, or nearest-exit routing, in a unified multi-site network? That is, routing where no Traffic Engineering priciples are used. Clearly stateful tracking inhibits nearest-exit routing by nature. (Side note: to ensure TCP connections are maintained in a stateless enviornment you'd obviously need to enforce 1:1 binat mappings). I'm seeing mixed messages from the above quotes. It's your project, but consider this a friendly thought-check :) You're looking at creating 1:1 mappings from internal IPs to 150-500k public IPs. You talk about a high rate of mapping changes, so clearly you're managing entries dynamically, which is just another way of keeping state. You mention NAT being an impediment to asymmetric routing, but NAT is unrelated to routing, so I can infer some possibilities here: * You want to distribute NAT by implementing it on or near each border. In order for anything bidirectional to function, the mappings must be consistent, so that implies synchronizing state between them. You consider pf's stateful behavior to be ill-suited to this use case, so you're investigating stateless options. * You are implementing routing on the same device that runs pf, and you believe pf's state-keeping will prevent asymmetric routing from taking place. Are either of these correct? If you are trying to do distributed NAT, I would ask if that is actually a design requirement. It may be easier to place NAT near the internal devices instead. In a dynamic environment, it may reduce complexity and failure modes simply by avoiding the synchronization needed for distributed mappings.
Re: Need stateless NAT
On Tue, Apr 08, 2008 at 11:59:11PM -0700, Adam Richards wrote: Maybe a pf.conf knob that allows me to turn off stateful tracking for a particular nat on iface ... rule? Ah, you keep mentioning 'nat' and 'rdr', which confused me before, but I guess what you're actually talking about is called 'binat' in pf: binat A binat rule specifies a bidirectional mapping between an external IP netblock and an internal IP netblock. You're right, it should be relatively easy to give binat a 'no state' option... But not for a /18 of arbitrary mappings with a high rate of change. With the current translation code this would require a rule for every mapping, and every packet is going to require a linear search of this ruleset. Fixing this is going to require fairly major changes to how binat works. Are you willing to pay someone to make this happen? BTW: What kind of packet forwarding rate are you hoping to get with this solution? Much of pf's performance comes from the fact that packets matching state entries are not evaluated against the ruleset. -Ryan
Re: Need stateless NAT
On Wed, Apr 09, 2008 at 05:36:57PM +0900, Ryan McBride wrote:
You're right, it should be relatively easy to give binat a 'no state'
option...
Try the attached diff, eg:
binat on egress from 192.168.100.1 to any - 10.99.99.99 no state
Index: sys/net/pf.c
===
RCS file: /cvs/src/sys/net/pf.c,v
retrieving revision 1.567
diff -u -r1.567 pf.c
--- sys/net/pf.c20 Feb 2008 23:40:13 - 1.567
+++ sys/net/pf.c9 Apr 2008 11:41:02 -
@@ -3321,7 +3321,8 @@
return (PF_DROP);
}
- if (!state_icmp (r-keep_state || nr != NULL ||
+ if (!state_icmp (r-keep_state ||
+ (nr != NULL nr-keep_state) ||
(pd-flags PFDESC_TCP_NORM))) {
/* create new state */
struct pf_state *s = NULL;
Index: sbin/pfctl/parse.y
===
RCS file: /cvs/src/sbin/pfctl/parse.y,v
retrieving revision 1.536
diff -u -r1.536 parse.y
--- sbin/pfctl/parse.y 1 Feb 2008 06:58:45 - 1.536
+++ sbin/pfctl/parse.y 9 Apr 2008 11:41:02 -
@@ -439,7 +439,7 @@
%type v.number number icmptype icmp6type uid gid
%type v.number tos not yesno
%type v.probability probability
-%type v.i no dir af fragcache optimizer
+%type v.i no dir af fragcache optimizer binatkeep
%type v.i sourcetrack flush unaryop statelock
%type v.b action nataction natpasslog scrubaction
%type v.b flags flag blockspec
@@ -3741,6 +3741,7 @@
memset(r, 0, sizeof(r));
r.action = $1.b1;
+ r.keep_state = 1;
r.natpass = $1.b2;
r.log = $1.w;
r.logif = $1.w2;
@@ -3889,8 +3890,12 @@
}
;
+binatkeep : /* empty */ { $$ = 1; }
+ | NO STATE { $$ = 0; }
+ ;
+
binatrule : no BINAT natpasslog interface af proto FROM host TO ipspec tag
- tagged rtable redirection
+ tagged rtable redirection binatkeep
{
struct pf_rule binat;
struct pf_pooladdr *pa;
@@ -3915,6 +3920,7 @@
binat.log = $3.b2;
binat.logif = $3.w2;
binat.af = $5;
+ binat.keep_state = $15;
if (!binat.af $8 != NULL $8-af)
binat.af = $8-af;
if (!binat.af $10 != NULL $10-af)
Index: sbin/pfctl/pfctl_parser.c
===
RCS file: /cvs/src/sbin/pfctl/pfctl_parser.c,v
retrieving revision 1.235
diff -u -r1.235 pfctl_parser.c
--- sbin/pfctl/pfctl_parser.c 15 Oct 2007 02:16:35 - 1.235
+++ sbin/pfctl/pfctl_parser.c 9 Apr 2008 11:41:02 -
@@ -986,6 +986,8 @@
printf( - );
print_pool(r-rpool, r-rpool.proxy_port[0],
r-rpool.proxy_port[1], r-af, r-action);
+ if (!r-keep_state r-action == PF_BINAT)
+ printf( no state);
}
}
Re: Need stateless NAT
Do the machines on the inside of the firewall have private
addresses?
Yes.
Perhaps a transparent firewall will work if the internal
machines also have public addresses.
I'd love to be able to do this! :) However there are some unique
requirements this network brings which make public IP consumption
unworkable.
Can you could make use of the filter directive route-to to
route packets to a different network?
nat on $ExtIf from !($ExtIf) to any - ($ExtIf:0)
rdr on $ExtIf from any to ($ExtIf) port www - 10.10.10.100 \
port www
pass in on $IntIf route-to { ($ExtIf_1 $ExtGw_1) } proto tcp \
from $IntNet to any port www
As I understand it, I don't think this will work for me as it
breaks the advantages of running a dynamic routing protocol to
select the nearest-exit, but I'll need to think a little more on
this.
I'm also wondering if there's a creative way to use tags...
It also sounds like you could make use of anchors in pf. With
an anchor you can add remove nat, rdr or filter rules on the
fly.
Ah! Yes indeed. Great idea, and could be quite handy. Elegant
too. But I'll certainly need to study performance
characteristics of using anchors during a heavy load of
table-management operations -- perhaps 10's of operations per
second on a table of 500,000 entries/mappings. (operations, like
inserts or deletes)
OpenBSD Pf Firewall how to ( pf.conf )
http://calomel.org/pf_config.html
Fantastic site, btw! I've already bookmarked it. :)
-Adam
Re: Need stateless NAT
On Sun, Apr 06, 2008 at 07:53:39PM +0900, Ryan McBride wrote: On Wed, Apr 02, 2008 at 04:27:17PM -0700, Adam Richards wrote: Is there a no state directive for nat rules, similar to the no-state directive for filter rules? Or another clever way to use nat/rdr/filter statements? Even though I wasn't able to find any affirmative evidence in pf.conf(5) manpage I thought I'd ask anyway. While I'd prefer a yes pf can do this answer, I will accept a no...but here are the code sections you'll want to look at to start your patch work answer. ;) No, PF does not do this. Keeping state is required for NAT to work, because you need to keep track of the mapping so that the return packets can be translated back the other way; Required? Technically, no (although it's a good idea for many reasons in most situations). It's possible to contrive situations where NAT is perhaps more desireable to provide better control knobs for presenting services to the Internet than your network's dynamic routing protocol provides (perhaps because of the lack of stability with really large IGP databases, or to simplify management of complex provisioning systems). Maybe for resilliency, or for ISP cost reasons, you want to take advantage of asymetric routing, or nearest-exit routing, in a unified multi-site network? That is, routing where no Traffic Engineering priciples are used. Clearly stateful tracking inhibits nearest-exit routing by nature. (Side note: to ensure TCP connections are maintained in a stateless enviornment you'd obviously need to enforce 1:1 binat mappings). Anyway, if you google around you can find others that are interested in stateless NAT. I cite one here: Quoting from http://lists.openwall.net/netdev/2007/09/27/55 Stateless NAT is useful in controlled environments where restrictions are placed on through traffic such that we don't need connection tracking to correctly NAT protocol-specific data. In particular, this is of interest when the number of flows or the number of addresses being NATed is large, or if connection tracking information has to be replicated and where it is not practical to do so. So what would it take modify pf code to allow for a pf.conf option to disable NAT stateful tracking where it's desired? Am I the only one interested in this potential feature? :) Thanks! -Adam
Re: Need stateless NAT
On Wed, Apr 02, 2008 at 04:27:17PM -0700, Adam Richards wrote: While I'd prefer a yes pf can do this answer, I will accept a no...but here are the code sections you'll want to look at to start your patch work answer. ;) No, pf can't do it. Not because it's technically impossible or unreasonable, it's just not a typical use case. For most users, routable address space is a scarcer resource than RAM for state table entries (they have much less external IP addresses than internal ones). Take a look at pf.c, you probably have to add some short-circuit to pf_test_state_*(). Instead of looking up the state entry, simply do the address mapping. Or maybe fiddle with the state lookup, so it returns a static state entry with the fields filled out correctly for the mapping. It's probably not trivial. Good luck ;) Daniel
Re: Need stateless NAT
On Tue, Apr 08, 2008 at 10:08:56AM -0500, Karl O. Pinc wrote: Stateless NAT is useful in controlled environments where restrictions are placed on through traffic such that we don't need connection tracking to correctly NAT protocol-specific data. What restrictions would those be? The only one I can think of is when the NATting is 1-to-1 with respect to internal and external IPs. Yes, 1:1 mapping is the primary restriction I'm thinking of, especially in asymetric routing environments I mentioned earlier. -Adam
Re: Need stateless NAT
On Wed, Apr 02, 2008 at 04:27:17PM -0700, Adam Richards wrote: Is there a no state directive for nat rules, similar to the no-state directive for filter rules? Or another clever way to use nat/rdr/filter statements? Even though I wasn't able to find any affirmative evidence in pf.conf(5) manpage I thought I'd ask anyway. While I'd prefer a yes pf can do this answer, I will accept a no...but here are the code sections you'll want to look at to start your patch work answer. ;) No, PF does not do this. Keeping state is required for NAT to work, because you need to keep track of the mapping so that the return packets can be translated back the other way; I'm guessing that your actual problem is not the basic state mechanism, but the TCP sequence number tracking. Look for the commend Sequence tracking algorithm from Guido van Rooij's paper in sys/net/pf.c for this code; it should be relatively trivial to add a bypass option for this in pfctl, and skip these checks in pf.c if that option is present. None of this is really a good idea though, and it shouldn't be seen as an appropriate response to a nasty network design. -Ryan
Re: Need stateless NAT
Adam,
Do the machines on the inside of the firewall have private addresses?
Perhaps a transparent firewall will work if the internal machines also have
public addresses.
Can you could make use of the filter directive route-to to route packets
to a different network?
nat on $ExtIf from !($ExtIf) to any - ($ExtIf:0)
rdr on $ExtIf from any to ($ExtIf) port www - 10.10.10.100 port www
pass in on $IntIf route-to { ($ExtIf_1 $ExtGw_1) } proto tcp from $IntNet to
any port www
It also sounds like you could make use of anchors in pf. With an anchor you
can add remove nat, rdr or filter rules on the fly.
Just guessing here.
OpenBSD Pf Firewall how to ( pf.conf )
http://calomel.org/pf_config.html
--
Calomel @ http://calomel.org
Open Source Research and Reference
On Wed, Apr 02, 2008 at 04:27:17PM -0700, Adam Richards wrote:
Hi. Let me just say pf is absolutely fantastic! It is actually
a joy to work with.
Ok, the problem:
I need to be able to create *stateless* nat rules for at least
150,000 entries, potentially to grow to 1/2million entries. The
reason has to do with being able to work in an asymetric routing
environment -- stateless nat must be used because traffic might
not egress at the same location it ingressed. In other words I
want to do a unidirectional translation and then fahgettaboutit.
For the moment let's ignore the cpu/mem/pcie/bus or other
hardware requirements to maintain a large translation table and
perform hdr rewrites fast enough. We can also ignore any
performance requirements of nats/sec. And lastly we can ignore
how translation table insertion or removal operations may affect
pf's realtime stability/reliability. I will likely touch on
these subtopics (perhaps in a new thread?) once I find out if
stateless nat is even possible.
Is there a no state directive for nat rules, similar to the
no-state directive for filter rules? Or another clever way to
use nat/rdr/filter statements? Even though I wasn't able to find
any affirmative evidence in pf.conf(5) manpage I thought I'd ask
anyway.
While I'd prefer a yes pf can do this answer, I will accept a
no...but here are the code sections you'll want to look at to
start your patch work answer. ;)
Thanks.
--
Adam Richards
e:[EMAIL PROTECTED] | o:[EMAIL PROTECTED] | k:0x0BA2643B |
Need stateless NAT
Hi. Let me just say pf is absolutely fantastic! It is actually a joy to work with. Ok, the problem: I need to be able to create *stateless* nat rules for at least 150,000 entries, potentially to grow to 1/2million entries. The reason has to do with being able to work in an asymetric routing environment -- stateless nat must be used because traffic might not egress at the same location it ingressed. In other words I want to do a unidirectional translation and then fahgettaboutit. For the moment let's ignore the cpu/mem/pcie/bus or other hardware requirements to maintain a large translation table and perform hdr rewrites fast enough. We can also ignore any performance requirements of nats/sec. And lastly we can ignore how translation table insertion or removal operations may affect pf's realtime stability/reliability. I will likely touch on these subtopics (perhaps in a new thread?) once I find out if stateless nat is even possible. Is there a no state directive for nat rules, similar to the no-state directive for filter rules? Or another clever way to use nat/rdr/filter statements? Even though I wasn't able to find any affirmative evidence in pf.conf(5) manpage I thought I'd ask anyway. While I'd prefer a yes pf can do this answer, I will accept a no...but here are the code sections you'll want to look at to start your patch work answer. ;) Thanks. -- Adam Richards e:[EMAIL PROTECTED] | o:[EMAIL PROTECTED] | k:0x0BA2643B |
