-----Original Message-----
From: Tom Eastep
Sent: Monday, April 15, 2019 12:46 AM
To: shorewall-users@lists.sourceforge.net
Subject: Re: [Shorewall-users] Switching between multi-ISP
On 4/14/19 1:30 PM, Andrey Andreev wrote:
Dear Tom,
I have been using Shorewall on all company routers, thanks for the great
work!
Recently installed GPRS routers as a backup to cable connections and
after an epic fight (like a pig with a pumpkin!) resorted to asking for
help here.
The installation where I make initial testing is: Fedora 26
4.11.9-300.fc26.x86_64 & Shorewall 5.1.10.2-1, GPRS router HUAWEI
B310s for ISP2, cable ISP1 with DHCP.
Below is my ISP 1&2 structure. My plan is:
(a) to route all traffic through the main ISP1 when it is available. No
load ballancing.
(b) if ISP1 goes down, all traffic is routed through the backup ISP2.
(c) when ISP1 goes up, all traffic is routed again through ISP1 though
ISP2 is still up.
What I get successfully is a) and b), but c) does not happen. Is it
possible to achieve this functionality, where do I make mistakes?
image
I have installed foolsm as described in Multi-ISP shorewall tutorial.
Other settings:
*interfaces* contains:
net enp3s0 detect routeback,optional,dhcp,wait=20
net enp1s0 detect routeback,optional,dhcp,wait=20
*masq* contains:
enp3s0 0.0.0.0/0 WAN1
enp1s0 0.0.0.0/0 192.168.42.254
*providers* contains:
N3 1 1 - enp3s0 WAN1 track,primary -
A1 2 2 - enp1s0 192.168.42.1 track -
*shorewall.conf* contains:
USE_DEFAULT_RT=Yes
TRACK PROVIDERS=Yes
BALLANCE_PROVIDERS=No
In ifcfg files for enp1s0&enp3s0 DEFROUTE=no
*lib.private* contains the following ISP1&2 description:
name=N3
eventscript=/usr/libexec/foolsm/shorewall_script
checkip=GW1
sourceip=WAN1
device=enp3s0
ttl=20
name=A1
eventscript=/usr/libexec/foolsm/shorewall_script
checkip=192.168.42.1 # checkip=WAN2 – ???
sourceip=192.168.42.254
device=enp1s0
ttl=20
The output of systemctl status shorewall and foolsm log look normal.
Checking ISP1&2 routing tables shows:
*ip route ls table N3 (when connected):*
default via GW1 dev enp3s0 src WAN1
GW1 dev enp3s0 scope link src WAN1
*ip route ls table A1 (when connected):*
default via 192.168.42.1 dev enp1s0 src 192.168.42.254
192.168.42.1 dev enp1s0 scope link src 192.168.42.254
*ip route ls table 253* – empty
Now I fiddle with the cables to the LAN cards of the FW disconnecting
and connecting them in turn and check routing table 254:
*1. N3 up, A1 up restart FW CPU OK, traffic goes through ISP1 main
as expected*
*ip route ls table 254:*
GW1.0/22 dev enp3s0 proto kernel scope link src WAN1 metric 100
GW1 dev enp3s0 scope link src WAN1
*2. N3 up --> down, A1 up OK, traffic goes through ISP2 as
expected*
*ip route ls table 254:*
default via 192.168.42.1 dev enp1s0 proto static metric 100
*3. N3 down --> up, A1 up BAD, traffic goes through ISP2 but ISP1
is expected*
*ip route ls table 254:*
default via 192.168.42.1 dev enp1s0 proto static metric 100
default via GW1 dev enp3s0 proto static metric 101
GW1.0/22 dev enp3s0 proto kernel scope link src WAN1 metric 100
GW1 dev enp3s0 scope link src WAN1
*4. nothing changed (N3 up, A1 up), restart shorewall OK, traffic
goes through ISP1 *
*ip route ls table 254:*
GW1.0/22 dev enp3s0 proto kernel scope link src WAN1 metric 100
GW1 dev enp3s0 scope link src WAN1
*5. . . . . after 1-2 minutes BAD, traffic goes through ISP2 but
ISP1 is expected*
*ip route ls table 254:*
default via 192.168.42.1 dev enp1s0 proto static metric 100
GW1.0/22 dev enp3s0 proto kernel scope link src WAN1 metric 100
GW1 dev enp3s0 scope link src WAN1
*6. . . . disconnect A1:*
*ip route ls table 254 OK, traffic goes through ISP1 main as
expected*
GW1.0/22 dev enp3s0 proto kernel scope link src WAN1 metric 100
GW1 dev enp3s0 scope link src WAN1
It looks to me that things mess up in 3. as the priority of the default
routes are wrong, ideally there should be no default route through ISP2.
I am not sure what IP should be in lib.private for ISP2:
checkip=192.168.42.1 (LAN IP of the GPRS router) or checkip=WAN2
(WAN IP of the GPRS router. It has no GW IP).
I will supply any additional info if needed. Thanks for Your advice!
How are you simulating interface failure and restoration? Are you
downing and upping the devices? The reason that I ask is that FooLSM's
default configuration uses 'shorewall disable' and 'shorewall enable' to
react to an interface going down and up respectively. Those operations
should *never* add default routes to the main routing table.
-Tom
--
Tom Eastep \ Q: What do you get when you cross a mobster with
Shoreline, \ an international standard?
Washington, USA \ A: Someone who makes you an offer you can't
http://shorewall.org \ understand
\_______________________________________________
To simulate ISP1 (main, cable) going down I pull the net cable from LAN
card. Then insert it back.
To simulate ISP2 (backup, GPRS) going down I disconnect the net cable
between FW computer and GPRS router.
I think that the same events sequence happens when ISP1 is simply out of
service for a minute (cable still duely connected): then automatic switching
to ISP2 takes place, connection is a bit slower but noone pays attention as
speed is decent, after the traffic limit of GPRS connection is exhausted the
traffic speed decreases dramatically and someone should pull the cable to
GPRS router to trigger switching to ISP1 which hopefully is up again.
Same story happened 3-4 times since ISP2 was installed in 01.2019 and
shorewall&foolsm were configured. During initial testing I did not pay
attention, first such case thought that there is something extraordinary in
ISP going up/down, but after doing extensive testing (on several Sundays
when noone is at work) I got the data above and started scratching my head.
Fiddled with configuration parameters with no success. I am rather a user,
not a dveloper.
By the way, ttl=1 in lib.private is fine for connections to router
installed next to the FW computer (as in shorewall multi ISP example), but
if the cable is long (in our case it is optics from ISP to office +
converter), then ttl should be increased or response from this GW never
comes. You can mention it for greenheads like me who copy-paste config files
without understanding the meaning of some parameters.
Here is file shorewall_script , a bit modified:
#! /bin/bash
[[ -n $BASH_VERSION ]] && shopt -s extglob
STATE=${1}
NAME=${2}
CHECKIP=${3}
DEVICE=${4}
WARN_EMAIL=${5}
REPLIED=${6}
WAITING=${7}
TIMEOUT=${8}
REPLY_LATE=${9}
CONS_RCVD=${10}
CONS_WAIT=${11}
CONS_MISS=${12}
AVG_RTT=${13}
SRCIP=${14}
PREVSTATE=${15}
TIMESTAMP=${16}
DATE=$(date --date=${TIMESTAMP})
LOGFILE=/var/log/foolsm.log
VARDIR=$(/sbin/shorewall show vardir)
# show connection information
function connection_info () {
cat <<ENDINFO
Connection ${NAME} is now ${STATE}.
Following parameters were passed:
newstate = ${STATE}
prevstate = ${PREVSTATE}
time_stamp = ${TIMESTAMP}
name = ${NAME}
checkip = ${CHECKIP}
sourceip = ${SRCIP}
device = ${DEVICE}
warn_email = ${WARN_EMAIL}
Packet counters:
replied = ${REPLIED} packets replied
waiting = ${WAITING} packets waiting for reply
timeout = ${TIMEOUT} packets that have timed out (= packet loss)
reply_late = ${REPLY_LATE} packets that received a reply after timeout
cons_rcvd = ${CONS_RCVD} consecutively received replies in sequence
cons_wait = ${CONS_WAIT} consecutive packets waiting for reply
cons_miss = ${CONS_MISS} consecutive packets that have timed out
avg_rtt = ${AVG_RTT} average rtt, notice that waiting and timed out
packets have rtt = 0 when calculating this
ENDINFO
} # connection_info
# show routing information
function routing_info () {
/sbin/shorewall status
/sbin/shorewall show routing
}
# main work to do
function do_main () {
echo
"***************************************************************************"
date +"*** %Y-%m-%d %H:%M:%S LSM Status change
***"
echo
"***************************************************************************"
connection_info
if [ "${STATE}" = "up" ]; then
state=0
action=enable
else
state=1
action=disable
fi
# inform shorewall isusabel extention script
echo $state > ${VARDIR:-/var/lib/shorewall}/${DEVICE}.status
# firt case - disable provider interface
sh ${VARDIR:-/var/lib/shorewall}/firewall ${action} ${DEVICE} 2>&1
routing_info
send_mail_notification
}
do_main >> $LOGFILE
exit 0;
#EOF
_______________________________________________
Shorewall-users mailing list
Shorewall-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/shorewall-users
_______________________________________________
Shorewall-users mailing list
Shorewall-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/shorewall-users
