Some feedback on this, and a bit of a performance indicator which might be of use to someone else.
I got the new box put in our live network on Sunday, so it's now running with real traffic. At the moment we're running with around 10Mbps in and out of general hosting traffic (not large media/video files). The traffic is accounted for by perhaps a dozen or two hosts, with a fair mix of stuff going on - mostly web based access to business systems, and public web servers. We sell on some of our bandwidth to other tenants in the building, and also to another nearby building via a wireless link. Some of these get dedicated bandwidth, others share bandwidth. Also, some of our hosting customers get dedicated bandwidth. In all, at the moment we have 6 groups, and each group has four priority levels - so it's very much like the "wondershaper" setup, but multiplied by 6 and working on both inbound (as best it's possible) and outbound traffic. The System is an Alix 2D2 LX800 board. 500MHz AMD Geode processor, 256M RAM, Debian Wheezy OS & 3.2.0 kernel. The system is routing (no NAT) at the network boundary, and doing traffic shaping and accounting. It runs from a 4G Compact Flash card, logging (syslog) is sent to another server, and RRD updates are sent to another server using RRDCached. Overall, excluding my collection & RRD update scripts, the system runs at around 40-50% idle and 50-60% system interrupt. When collection scripts are running, idle can drop to 10-20%, so not a huge amount of headroom, but enough for what we need now - and it takes a lot less power than the old 1.1GHz Pentium 3 rack mount server it replaces ! I also costs a lot less than a replacement for the old server which had died. If we significantly increase out bandwidth, then I'll need to look for an alternative board with a little more horsepower - the Alix boards are attractive as they have multiple real NICs on them. I've used some includes, and the intention is that some of these files will be script generated once I've written the script. Eg, the traffic shaping classes in particular will have a text file listing the groups and the rates allowed - the script will then do some checks (make sure the committed rates don't add up to more than the parents and so on) before building the include file and sending it to both routers - as well as updating the graphing we have. The accounting comprises : ># Outside global stats >outside:COUNT - ethext - >outside:COUNT - - ethext >DONE outside > ># Do acocunting by IP address >account-ip - - - >total-ip:COUNT account-ip ethext - >total-ip:COUNT account-ip - ethext >DONE total-ip > >acc-serv >total-serv:COUNT acc-serv ethext - >total-serv:COUNT acc-serv - ethext >DONE total-serv > > >INCLUDE accounting.ip >INCLUDE accounting.service And accounting.ip contains : >acc1:COUNT account-ip ethext a.b.c.1 >acc1:COUNT account-ip a.b.c.1 ethext >DONE acc1 > >acc2:COUNT account-ip ethext a.b.c.2 >acc2:COUNT account-ip a.b.c.2 ethext >DONE acc2 > >... > >acc253:COUNT account-ip ethext a.b.c.253 >acc253:COUNT account-ip a.b.c.253 ethext >DONE acc253 > >acc254:COUNT account-ip ethext a.b.c.254 >acc254:COUNT account-ip a.b.c.254 ethext >DONE acc254 So that's counting the traffic, by IP, in and out, for 254 addresses. On the traffic shaping side, I have : >#INTERFACE MARK RATE CEIL PRIORITY OPTIONS > ># Allow traffic from firewall to internal network to be effectively unlimited ># Class for firewall internal traffic >ethint:1:2 - 10mbit 95mbit 2 ># Class for normal traffic >ethint:1:3 - xxxxxkbit xxxxxkbit 1 > ># Main group (default if not otherwise classified) >ethext:1:10 - xxxxkbit xxxxxkbit 1 >ethext:10:11 - xxxxkbit xxxxxkbit 1 >ethext:10:12 - xxxxkbit xxxxxkbit 2 default >ethext:10:13 - xxxkbit xxxxxkbit 3 >ethext:10:14 - xxxkbit xxxxkbit 4 > >ethint:3:10 - xxxxkbit xxxxxkbit 1 >ethint:10:11 - xxxxkbit xxxxxkbit 1 >ethint:10:12 - xxxxkbit xxxxxkbit 2 default >ethint:10:13 - xxxkbit xxxxxkbit 3 >ethint:10:14 - xxxkbit xxxxkbit 4 > >INCLUDE tcclasses-extra tcclasses-extra is intended to be built from a script when I get that bit done, at the moment it contains : ># Customer group A >ethext:1:15 - xxxxkbit 10240kbit 1 >ethext:15:16 - xxxkbit 10240kbit 1 >ethext:15:17 - xxxkbit 10240kbit 2 >ethext:15:18 - xxkbit 10240kbit 3 >ethext:15:19 - xxkbit 10240kbit 4 > >ethint:3:15 - xxxxkbit 10240kbit 1 >ethint:15:16 - xxxkbit 10240kbit 1 >ethint:15:17 - xxxkbit 10240kbit 2 >ethint:15:18 - xxkbit 10240kbit 3 >ethint:15:19 - xxkbit 10240kbit 4 > ># Customer group B >ethext:1:20 - xxxxkbit 8192kbit 1 >ethext:20:21 - xxxkbit 8192kbit 1 >ethext:20:22 - xxxkbit 8192kbit 2 >ethext:20:23 - xxxkbit 8192kbit 3 >ethext:20:24 - xxxkbit 8192kbit 4 > >ethint:3:20 - xxxxkbit 8192kbit 1 >ethint:20:21 - xxxkbit 8192kbit 1 >ethint:20:22 - xxxkbit 8192kbit 2 >ethint:20:23 - xxxkbit 8192kbit 3 >ethint:20:24 - xxxkbit 8192kbit 4 And another 3 groups for a total of 6. And then the TC rules : >#ACTION SOURCE DEST PROTO PORT(S) CLIENT > USER TEST LENGTH TOS ># PORT(S) ># If the connection already has a mark, use it and skip the rest of the rules >RESTORE 0.0.0.0/0 0.0.0.0/0 all - - > - 0 >CONTINUE 0.0.0.0/0 0.0.0.0/0 all - - > - !0 > >INCLUDE tcrules-extra-1 > ># Traffic originating in the firewall >2 $FW ethint all - - > - 0 > ># General (Anything not already classified) >12 0.0.0.0/0 0.0.0.0/0 all - - > - 0 >12 0.0.0.0/0 0.0.0.0/0 all - - > - 0 >11 0.0.0.0/0 0.0.0.0/0 icmp echo-request - > - 12 >11 0.0.0.0/0 0.0.0.0/0 icmp echo-reply - > - 12 ># DNS >COMMENT DNS >11 0.0.0.0/0 0.0.0.0/0 tcp 53 - > - 12 >11 0.0.0.0/0 0.0.0.0/0 tcp - 53 > - 12 ># Mail (SMTP 25 & 465, Submission 587, POP3 110 & 995, IMAP 143 & 993) >COMMENT MAIL >13 0.0.0.0/0 0.0.0.0/0 tcp 110,995,143,993 - > - 12 >13 0.0.0.0/0 0.0.0.0/0 tcp - >110,995,143,993 - 12 >14 0.0.0.0/0 0.0.0.0/0 tcp 25,465,587 - > - 12 >14 0.0.0.0/0 0.0.0.0/0 tcp - 25,465,587 > - 12 ># RSync traffic (873) >COMMENT RSync >14 0.0.0.0/0 0.0.0.0/0 tcp 873 - > - 12 >14 0.0.0.0/0 0.0.0.0/0 tcp - 873 > - 12 > >COMMENT > ># Assuming we've marked the packet, now save the mark to the connection >SAVE 0.0.0.0/0 0.0.0.0/0 all - - > - !0 > ># Convert marks to classes > >1:11:T 0.0.0.0/0 0.0.0.0/0 all - - > - 11 >2:11:T 0.0.0.0/0 0.0.0.0/0 all - - > - 11 >CONTINUE:T 0.0.0.0/0 0.0.0.0/0 all - - > - 11 >1:12:T 0.0.0.0/0 0.0.0.0/0 all - - > - 12 >2:12:T 0.0.0.0/0 0.0.0.0/0 all - - > - 12 >CONTINUE:T 0.0.0.0/0 0.0.0.0/0 all - - > - 12 >1:13:T 0.0.0.0/0 0.0.0.0/0 all - - > - 13 >2:13:T 0.0.0.0/0 0.0.0.0/0 all - - > - 13 >CONTINUE:T 0.0.0.0/0 0.0.0.0/0 all - - > - 13 >1:14:T 0.0.0.0/0 0.0.0.0/0 all - - > - 14 >2:14:T 0.0.0.0/0 0.0.0.0/0 all - - > - 14 >CONTINUE:T 0.0.0.0/0 0.0.0.0/0 all - - > - 14 > >INCLUDE tcrules-extra-2 > >2:2:T $FW ethint all - - > - 2 tcrules-extra-1 contains blocks like this : ># Customer group A >COMMENT Customer 1 >17 a.b.c.123/32 0.0.0.0/0 all - - >- 0 >17 0.0.0.0/0 a.b.c.123/32 all - - >- 0 >COMMENT Customer 2 >17 a.b.c.127/32 0.0.0.0/0 all - - >- 0 >17 0.0.0.0/0 a.b.c.127/32 all - - >- 0 >COMMENT Customer 3 >17 a.b.c.136/32 0.0.0.0/0 all - - >- 0 >17 0.0.0.0/0 a.b.c.136/32 all - - >- 0 >COMMENT Customer 4 >17 a.b.c.103/32 0.0.0.0/0 all - - >- 0 >17 0.0.0.0/0 a.b.c.103/32 all - - >- 0 >COMMENT Customer 5 >17 a.b.c.213/32 0.0.0.0/0 all - - >- 0 >17 0.0.0.0/0 a.b.c.213/32 all - - >- 0 >17 a.b.c.214/32 0.0.0.0/0 all - - >- 0 >17 0.0.0.0/0 a.b.c.214/32 all - - >- 0 >COMMENT >16 0.0.0.0/0 0.0.0.0/0 icmp echo-request - > - 17 >16 0.0.0.0/0 0.0.0.0/0 icmp echo-reply - > - 17 >18 0.0.0.0/0 0.0.0.0/0 tcp 110,995,143,993 - > - 17 >18 0.0.0.0/0 0.0.0.0/0 tcp - >110,995,143,993 - 17 >19 0.0.0.0/0 0.0.0.0/0 tcp 25,465,587 - > - 17 >19 0.0.0.0/0 0.0.0.0/0 tcp - >25,465,587 - 17 The rules vary by customer - so for example, those with a PBX get prioritisation for their SIP/IAX traffic. Note the use of SAVE, RESTORE, and CONTINUE. TC filters are applied to every packet, so that could mean applying all the filters to every packet. By saving the packet mark to the connection mark, and then restoring it as the first rule, most of the filters can be skipped for all but the first packet in each connection. And tcrules-extra-2 contains corresponding blocks like this : ># Customer group A >1:16:T 0.0.0.0/0 0.0.0.0/0 all - - - > 16 >2:16:T 0.0.0.0/0 0.0.0.0/0 all - - - > 16 >CONTINUE:T 0.0.0.0/0 0.0.0.0/0 all - - - > 16 >1:17:T 0.0.0.0/0 0.0.0.0/0 all - - - > 17 >2:17:T 0.0.0.0/0 0.0.0.0/0 all - - - > 17 >CONTINUE:T 0.0.0.0/0 0.0.0.0/0 all - - - > 17 >1:18:T 0.0.0.0/0 0.0.0.0/0 all - - - > 18 >2:18:T 0.0.0.0/0 0.0.0.0/0 all - - - > 18 >CONTINUE:T 0.0.0.0/0 0.0.0.0/0 all - - - > 18 >1:19:T 0.0.0.0/0 0.0.0.0/0 all - - - > 19 >2:19:T 0.0.0.0/0 0.0.0.0/0 all - - - > 19 >CONTINUE:T 0.0.0.0/0 0.0.0.0/0 all - - - > 19 tcdevices just contains : >ethext - xxxxxkbit >ethint - 100mbit Note that ethint is set at 100mbit as we don't restrict the internal interface here - we need to allow full wire speed for the router to communicate to other servers internally. -- Simon Hobson Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed author Gladys Hobson. Novels - poetry - short stories - ideal as Christmas stocking fillers. Some available as e-books. ------------------------------------------------------------------------------ Monitor your physical, virtual and cloud infrastructure from a single web console. Get in-depth insight into apps, servers, databases, vmware, SAP, cloud infrastructure, etc. Download 30-day Free Trial. Pricing starts from $795 for 25 servers or applications! http://p.sf.net/sfu/zoho_dev2dev_nov _______________________________________________ Shorewall-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/shorewall-users
