Hi Thomas, Thanks for this great detailed information. Unfortunately it is still not very clear for me whast to write instead of your 999.999.999.999 example. My wan interfaces are ppp0 and ppp1 . They have dynamic IP addresses and their gateways are same because they connect to the same ISP.
What should I write for them ? Also, I didn't still install lsm yet but checked fpor availability of the explained files on my system. There are no "[interface].status" files under vARDIR (/var/lib/shorewall) How will they be populated? Thanks. -----Original Message----- From: Thomas Harold [mailto:[email protected]] Sent: Thursday, September 12, 2013 10:22 AM To: Shorewall Users Cc: İlker Aktuna Subject: Re: [Shorewall-users] Fallback in a multi-isp configuration Note: Our setup is a CentOS 6 box, running Shorewall 4.5.4 (from the Prereqs: First off, you should print out the MultiISP document from the Shorewall website and start marking it up with notes: http://shorewall.net/MultiISP.html About 98% of the answers are in the MultiISP file or one of the other shorewall documents. The other 2% are found on the mailing list. A) I suggest renaming your ethernet devices to have more meaningful names. In a CentOS environment, we do this in /etc/sysconfig/network-scripts by: - Renaming ifcfg-eth0 to ifcfg-newname - Editing ifcfg-newname and using DEVICE=newname We used "wancbl" (our cable modem, 25-30Mbps inbound, 2-3Mbps outbound, DHCP address), "want1" (our T1 line, static IP), "lan1" and "lan2" for the internal ethernet ports. This made it much easier for us to see traffic when looking using "atop" or other tools, rather then trying to remember eth0..eth3 and which was what. B) Turn off the NetworkManager service if you have it installed. Simply doing "chkconfig NetworkManager off" takes care of it in CentOS/RHEL. - The number one bugaboo when you have NetworkManager installed is that it likes to overwrite your /etc/resolv.conf file (and does other things not suitable for a firewall server). C) We bonded our two LAN ethernet ports (lan1 and lan2) into a simple active-backup (mode 1) bond called "bond0". So wherever we refer to "bond0", that is our internal LAN address. Setting up bonding is not hard, but beyond the scope of this. D) We are still using IPv4 for everything. While our T1 provider is talking about IPv6 rollout soon, our cable modem provider has no plans at all. E) Use a version control system for /etc, /usr/local and any other directories where configuration files might be stored. My recommendation has been FSVS (stores data into a SVN repository) for a number of years. But you could also use git, hg, cvs, etc. - It allows you to easily revert changes, without having to reach for the backup tapes. - You can leave notes to yourself in the commit messages (in addition to leaving comments inside the configuration file). With FSVS, I can look back at the entire history of the server and see when I installed and configured a particular bit of software, what files I touched, and what I changed. - Doing a diff between the configuration right after you installed something and after you mucked it up makes it easy to undo changes (or see how you did something 2-3 years from now). ... Files that we changed from the Shorewall 4.5.4 defaults: A) Zones file (self-explanatory, we don't yet use a DMZ zone or anything fancy like VLANs): /etc/shorewall/zones fw firewall loc ipv4 net ipv4 B) Interfaces (options may not be 100% correct, but seems to work): /etc/shorewall/interfaces loc bond0 dhcp,logmartians,nosmurfs,required,tcpflags net wancbl dhcp,logmartians,nosmurfs,optional,tcpflags net want1 logmartians,nosmurfs,optional,tcpflags In your case, since both WAN interfaces are DHCP, you'll need to use the dhcp option on both. For us, only "wancbl" interface used a DHCP address. Since DHCP is in use on the internal LAN, the bond0 interface also needed the 'dhcp' option. C) Policies (you should customize this based on how secure you want things, letting the $FW talk to everything is probably not best): /etc/shorewall/policies loc net ACCEPT net all DROP info $FW loc ACCEPT $FW net ACCEPT all all REJECT info D) Rules (you will be doing very custom things to this based on your needs, we run SSH on multiple ports, only allow SSH from outside via port 2222/tcp, plus we have NTP and BIND, plus allowing ICMP): /etc/shorewall/rules ACCEPT loc $FW tcp 22 # SSHD port ACCEPT loc $FW tcp 2222 # 2nd SSHD port ACCEPT net $FW tcp 2222 # only allow SSH over tcp/2222 from outside ACCEPT $FW net udp ntp ACCEPT loc $FW udp ntp ACCEPT $FW loc udp ntp REDIRECT loc ntp udp ntp # redirect NTP traffic to firewall NTP server DNS/ACCEPT $FW net DNS/ACCEPT loc $FW DNS/ACCEPT $FW loc DNS/ACCEPT loc net ACCEPT loc $FW icmp ACCEPT $FW loc icmp ACCEPT $FW net icmp ACCEPT net $FW icmp E) Masq file (NAT for the LAN clients to the internet). The example here uses 192.168.0.0/24 as our internal LAN address range (in reality we use a 172.x.y.z range). The important thing is that you need to list both WAN ethernet devices, with the same internal LAN address range for both lines.: /etc/shorewall/masq wancbl 192.168.0.0/24 want1 192.168.0.0/24 F) Providers (I am still a bit fuzzy about things here). Note that we use TRACK_PROVIDERS=Yes and USE_DEFAULT_RT=Yes in the shorewall.conf file. This affects how you define your two providers: /etc/shorewall/providers provcbl 1 2 - wancbl detect balance=100 provt1 2 2 - want1 999.999.999.999 balance=1 - Replace 999.999.999.999 with "detect" if you are using DHCP, or enter the static external IP address assigned to that interface. - USE_DEFAULT_RT=Yes means that the DUPLICATE column should be left as '-' in the providers file. - TRACK_PROVIDERS=Yes automatically adds "track" to the OPTIONS column, which is why I don't specify it as "track,balance=###". - The 100:1 weight means that 99% of all traffic goes out over the "wancbl" interface. Only if the cable modem is down, does it start to route traffic over our T1 line. - The provider names (1st column) will also show up in your routing tables (see "shorewall show routing"). So we prefix our provider names with "prov" to make it clearer. - If you need specific traffic to go out over the secondary provider, then you need to play with "rtrules" or "tcrules" (we don't use this yet, so that is not a certainty). Get it working first without touching those files. - I have been unable to get "fallback" option working properly in Shorewall 4.5.4. But the above works well enough for our purposes. G) shorewall.conf (the main config file). I am only listing lines that changed from the base install. You should *definitely* read the main Shorewall documentation and make notes in the configuration file for why you changed a particular value. /etc/shorewall/shorewall.conf VERBOSITY=2 NULL_ROUTE_RFC1918=Yes RESTORE_DEFAULT_ROUTE=No TRACK_PROVIDERS=Yes USE_DEFAULT_RT=Yes H) The "routestopped" file (for Shorewall v4.5.4), it has a different name in later versions of Shorewall (stoppedrules). These are the iptables rules that get generate whenever Shorewall shuts down. bond0 - - icmp bond0 - - tcp 22 bond0 - - udp 123 bond0 - - tcp 2222 want1 - source icmp want1 - source tcp 2222 wancbl - source icmp wancbl - source tcp 2222 ... That's the end of the *simple* stuff. Other then the providers file and defining multiple WAN interfaces, it looks very similar to a single-ISP setup. ... LSM configuration notes: - We used lsm-0.163-1.foo6.src.rpm for our LSM install on CentOS 6. - The default LSM configuration is not quite correct for working with Shorewall. ... A) You need to change a few defaults in the lsm.conf file. These lines were all added or changed inside the "defaults{}" block. eventscript=/etc/lsm/script notifyscript= status=1 B) Before the "#EOF" line at the end of the /etc/lsm/lsm.conf file, you need to add the following line: include /var/lib/shorewall/lsm.conf C) A sample /var/lib/shorewall/lsm.conf connection { name=T1 checkip=999.999.999.999 device=want1 ttl=2 } connection { name=Cablevision checkip=999.999.999.999 device=wancbl ttl=2 } - You should replace "999.999.999.999" with the current DHCP IP address - This file actually gets written out by Shorewall, but for testing LSM, you will want to initially create it by hand. D) You do *not* need to run LSM as a service. On CentOS 6, if "chkconfig --list lsm" shows that LSM is running in runlevels 3-5, then you should "chkconfig lsm off" to disable the service. E) /etc/lsm/script - This is the script that LSM will run whenever the status of a link changes (eventscript= argument in /etc/lsm/lsm.conf). See attachment to this email for a copy of our LSM script (which is slightly different then the Shorewall MultiISP example script). It needs to perform the following tasks when a link goes "up": - Remove the [interface].status file - Execute the Shorewall "firewall enable {device}" script When the link goes down, it needs to: - Execute the Shorewall "firewall disable {device}" script Note: The above only works for Shorewall 4.5.4 and later. I have added some other logging statements to the script that shoves the details and events into /var/log/lsm.log. ... Now that LSM is setup, we can go back to setting up Shorewall to interact with LSM. ... A) /etc/shorewall/isusable This *should* be already setup by default. It looks for "[interface].status" files in the VARDIR, where LSM will write those status files. B) /etc/shorewall/params You *could* define various things here instead of hardcoding them in the other files. But we did not do so in our setup. C) /etc/shorewall/started This is the shell script commands that get run whenever shorewall starts up. We decided to use the single solitary line in our "started" file rather then check whether this was a "restart" command. The "start_lsm" is a function defined in /etc/shorewall/lib.private # [Re]Start LSM start_lsm D) /etc/shorewall/restored This is covered in the multi-ISP document. It checks to see if LSM is already running, otherwise it starts it. # Start lsm if it isn't running if [ -z "$(ps ax | grep 'lsm ' | grep -v 'grep ' )" ]; then start_lsm fi E) /etc/shorewall/lib.private The meat-and-potatoes for interacting with LSM. It defines the "start_lsm()" shell function. See attached file. - The checkip line can be written two ways. You can specify a static IP address that is the "next hop out" for that particular interface. Or you can use the "SW_(interfacename)_GATEWAY" environment variable for DHCP setups and also specify a fallback address. Static version: checkip=999.999.999.999 DHCP version: checkip=${SW_WANCBL_GATEWAY:-999.999.999.999} - 999.999.999.999 in this file needs to be your "next-hop" gateway IP address for that WAN interface, not the external IP address of your firewall. - If you need to go two+ hops out, then you will *also* need to add a static "ip route". I'm not showing an example of that, and for cable modem setups it is probably not needed. - Our lib.private is somewhat custom, with some additional logging added that is not in the MultiISP example document. - start_lsm() is responsible for writing out the /var/shorewall/lsm.conf file. - We should be using more of Shorewall variables (such as VARDIR) instead of hardcoding /var/lib/shorewall in places. F) /etc/shorewall/findgw This file may not exist by default, or may not be compatible with your distro. Sample versions of the "findgw" script can be found at: http://www.shorewall.net/pub/shorewall/contrib/findgw/ Without the proper "findgw" file, you will see that even though the interface is up, there is no default route being added to that provider's routing table (see "shorewall show routing"). ... I think that's everything that we changed when setting up our MultiISP configuration. There are no warranties, but this should serve as a trail of breadcrumbs to help you figure out what needs to be touched. ------------------------------------------------------------------------------ How ServiceNow helps IT people transform IT departments: 1. Consolidate legacy IT systems to a single system of record for IT 2. Standardize and globalize service processes across IT 3. Implement zero-touch automation to replace manual, redundant tasks http://pubads.g.doubleclick.net/gampad/clk?id=51271111&iu=/4140/ostg.clktrk _______________________________________________ Shorewall-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/shorewall-users
