Just wanted to add some observations of my own. Like Lonnie I have been
testing WAN failover using the Netgear LB1121 and it is working well.
I did run into one quirk though which is when using bridge mode in the
LB1121 I was getting unreliable connections when switching to test
failover. On investigation it appeared that the IP address issued by AT&T
was changing and the interface/route was getting into a weird state. I
have switched my LB1121 back over to router mode with static IP address and
it is working fine. This may be specific to AT&T, I think Lonnie is on
T-Mobile, but is something to look out for. As I am tunneling through a
VPN to an external endpoint whether bridge or router makes no difference.
Lonnie and I both set up an Astlinux on Linode and wireguard VPN into it.
I tried to get Astlinux onto other cloud providers (AWS, Google) with no
success... their environment appears to require the GRUB loader. However I
did try firing up a plain old Ubuntu on those cloud providers, installed
wireguard, setup a few iptables rules to implement simple NAT/Forwarding
and lo it all works. So that is an alternate path.
You can also set an inbound NAT rule on your failover host to forward a
specific port to your AstLinux web interface so you can access that when in
failover situations (note, it will not work if your Astlinux box is using
the primary WAN as the default route will cause replies to go through the
primary interface and not the failover interface). If you do this, add the
URL for your failover host to your SSL certificate so that you don't get
warnings.
Finally... the wireless plan I am using for my failover is not unlimited.
I therefore need to strictly limit traffic over the link and specifically
limit video streaming and online backups which could drive high usage. I
implemented a number of rules to enforce this which I will post to this
list in a separate note.
David
On Tue, Jun 5, 2018 at 12:02 PM, Lonnie Abelbeck <li...@lonnie.abelbeck.com>
wrote:
> I have now more completely tested my WAN Failover using 4G/LTE over
> WireGuard ...and now works perfectly with Asterisk with a configuration
> change.
>
> My 4G/LTE over WireGuard tunnel endpoint is with AstLinux on a Linode KVM
> (as documented previously). The AstLinux on the Linode KVM instance is not
> running Asterisk.
>
> Previously my local Asterisk was configured with nat=no and since I have a
> static IPv4 address my SIP provider has an option to authenticate using the
> static address (no registration). If I added the static IPv4 of my Linode
> KVM instance it mostly worked for outbound calls during failover but would
> require failover call configuration at the SIP provided end to handle
> inbound calls.
>
> Instead, I switched my SIP provider configuration to require registration
> and removed my static IPv4 authentication. In order to force a quick
> registration update on the switch I use the wan-failover.script to reload
> sip:
>
> -- /mnt/kd/wan-failover.script snippet --
> SECONDARY)
> ## Switched to Failover using secondary WAN link
> asterisk -rx "sip reload" >/dev/null
> ;;
>
> PRIMARY)
> ## Switched back to normal using primary WAN link
> asterisk -rx "sip reload" >/dev/null
> ;;
> --
>
> Almost there, but when failover occurs the registration is NAT'ed at the
> AstLinux on Linode KVM end, so for my SIP provider all it took was to
> configure my SIP account to support NAT. Even though my Asterisk is still
> configured with nat=no, the registration via the failover tunnel now works.
>
> Inbound and outbound SIP calls work perfectly over the 4G/LTE failover.
>
> Most interestingly, the calls keep working during the failover switch and
> back again ... I was not expecting that, and I am still scratching my head
> over that. The active calls do not miss a beat as I see data over my
> 4G/LTE and then no data over 4G/LTE, the call continues perfectly. I even
> have directmedia=no in my provider peer's sip.conf.
>
> Lonnie
>
>
>
> > On May 28, 2018, at 8:30 PM, Michael Knill <michael.knill@ipcsolutions.
> com.au> wrote:
> >
> > I decided today the architecture I want to test.
> > I will be setting up a DR Astlinux box running Asterisk and having a SIP
> Trunk with a 100 Block number range on it for indial.
> > All Astklinux boxes configured for failover will establish a VPN to this
> box and a SIP Trunk to the DR server will be second choice for outgoing
> calls. I can set the Caller ID to be anything I want for DR outgoing so no
> issues there. Billing should be fine also as long as the Originating Caller
> ID is correct.
> > All my providers allow the setting of a call forward on busy (and
> unreachable) to forward to one of the numbers in the 100 block which is
> redirected to the associated Astlinux box. So incoming calls are sorted
> also. If it's the same provider, then all forwarded calls are free.
> >
> > I think it should work with no manual intervention required.
> >
> > Regards
> > Michael Knill
> >
> > On 29/5/18, 10:07 am, "Lonnie Abelbeck" <li...@lonnie.abelbeck.com>
> wrote:
> >
> > Hey Michael,
> >
> > Currently my "cloud" AstLinux Linode KVM (4G/LTE over VPN endpoint)
> has Asterisk disabled, ASTERISK_DAHDI_DISABLE="yes" so when my main
> AstLinux box goes to failover the SIP packets originate from my "home"
> WireGuard private 10.0.0.0/24 address and are NAT'ed at the Linode end.
> Normally there is no NAT and SIP packets originate with my public IP.
> >
> > Surprisingly outbound calls work without any changes, though a remote
> hangup BYE is not received. Inbound failover could be handled on my SIP
> provider's platform.
> >
> > I could enable Asterisk at the Linode end and use it as a proxy of
> sorts.
> >
> > Possibly better, per what you suggested, is to implement an outbound
> call failover in the "main" Asterisk regardless of the WAN Failover status
> ... if the standard outbound call fails it tries the call over the 4G/LTE
> VPN path.
> >
> > I'm still pondering my options. The exact solution is somewhat
> dependent on the user's SIP provider.
> >
> > Lonnie
> >
> >
> >
> >
> >> On May 28, 2018, at 5:08 PM, Michael Knill <michael.knill@ipcsolutions.
> com.au> wrote:
> >>
> >> Hi Lonnie
> >>
> >> So what are you trying to solve with Asterisk failover?
> >>
> >> Regards
> >> Michael Knill
> >>
> >> On 28/5/18, 10:00 pm, "Lonnie Abelbeck" <li...@lonnie.abelbeck.com>
> wrote:
> >>
> >> Hi Michael,
> >>
> >> Yes, you can use OpenVPN and WireGuard at the same time, no problem.
> I do.
> >>
> >> WireGuard is much faster / more efficient than OpenVPN, mostly since
> it resides in the kernel and can use multiple cores. Here are some
> benchmarks I posted to the WireGuard mailing list:
> >>
> >> https://lists.zx2c4.com/pipermail/wireguard/2017-December/002204.html
> >>
> >> There are user-space implementations of WireGuard, written in Golang,
> starting to appear for testing, but for non-Linux endpoints I would stick
> with OpenVPN for now.
> >>
> >> BTW, I currently have WAN Failover on my production AstLinux box
> using the Netgear LB1121 4G/LTE over WireGuard VPN to a Linode KVM running
> AstLinux. Working is dual stack IPv4/IPv6 failover for the AstLinux box
> itself and any internal network of my choosing. I have outbound Asterisk
> failover working, but that is still a work in progress, not sure the best
> method yet.
> >>
> >> Lonnie
> >>
> >>
> >>
> >>
> >>> On May 28, 2018, at 5:03 AM, Michael Knill <
> michael.kn...@ipcsolutions.com.au> wrote:
> >>>
> >>> Hi group
> >>>
> >>> Im ready to do some testing.
> >>> I have a number of sites that are set up as OpenVPN Servers. Should
> there be any issues using Wireguard as well?
> >>> PS I just looked up Wireguard and I cant believe the difference in
> benchmarks to Open VPN. That's crazy!
> >>>
> >>> Regards
> >>> Michael Knill
> >>>
> >>> On 24/5/18, 9:23 am, "Michael Knill" <michael.knill@ipcsolutions.
> com.au> wrote:
> >>>
> >>> Thanks Lonnie. I don't have a specific scenario yet but handy to know
> its possible.
> >>>
> >>> Regards
> >>> Michael Knill
> >>>
> >>> On 24/5/18, 8:54 am, "Lonnie Abelbeck" <li...@lonnie.abelbeck.com>
> wrote:
> >>>
> >>> Michael,
> >>>
> >>>> So are you saying that you can configure a second external interface
> and the associated routing to it with the Failover Tab but just leave
> Failover disabled?
> >>>
> >>> Yes, "External Failover Destination Routes:" automatically
> defines static routes, automatically removed and added for DHCP changes.
> >>>
> >>>
> >>>> If so, I assume it uses the same EXT firewall rules?
> >>>
> >>> Yes. There is a way to treat EXTIF and EXT2IF firewall rules
> differently, but the same is usually OK.
> >>>
> >>> Lonnie
> >>>
> >>>
> >>>
> >>>> On May 23, 2018, at 5:17 PM, Michael Knill <
> michael.kn...@ipcsolutions.com.au> wrote:
> >>>>
> >>>> Hi Lonnie
> >>>>
> >>>> So are you saying that you can configure a second external interface
> and the associated routing to it with the Failover Tab but just leave
> Failover disabled?
> >>>> If so, I assume it uses the same EXT firewall rules?
> >>>>
> >>>> Regards
> >>>> Michael Knill
> >>>>
> >>>> On 22/5/18, 8:59 am, "Lonnie Abelbeck" <li...@lonnie.abelbeck.com>
> wrote:
> >>>>
> >>>> Hi Michael,
> >>>>
> >>>>> I noticed you also pass the VPN traffic to the site LAN
> >>>>
> >>>> Yes, I tried to implement the general case, easy to remove stuff.
> >>>>
> >>>>> the VPN would normally just be used for voice traffic and management
> only.
> >>>>
> >>>> In that case "External Failover Destination Routes: IPv4 Routes:"
> could define all the destination routes you need without "Failover" enabled
> ... and let Asterisk dynamically choose the SIP route. Handling inbound
> calls over the 4G/LTE VPN would also be possible.
> >>>>
> >>>>
> >>>> All seems to work well, the only fundamental issue may be the latency
> of 4G/LTE for SIP traffic ... though clearly much better than no traffic.
> >>>>
> >>>> Lonnie
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>> On May 21, 2018, at 5:36 PM, Michael Knill <
> michael.kn...@ipcsolutions.com.au> wrote:
> >>>>>
> >>>>> Thanks Lonnie you beat me to it.
> >>>>> Interestingly one of my partners is using Asterisk as their
> Softswitch and they were thinking of setting up a single VPN Tunnel to the
> SoftSwitch for voice traffic and so everything still works on both the
> primary and failover links. There should be no failover scripts required!
> >>>>>
> >>>>> I noticed you also pass the VPN traffic to the site LAN but this
> would not actually be required in practice as the VPN would normally just
> be used for voice traffic and management only. On all VPN connections that
> run voice traffic I set directmedia=no in sip.conf. PS I actually now use a
> directmedia ACL on the VPN subnet so I don't need to configure anything.
> E.g.
> >>>>>
> >>>>> directmedia=yes
> >>>>> directmediapermit=0.0.0.0/0
> >>>>> directmediadeny=<VPN Subnet>
> >>>>>
> >>>>> Thanks again Lonnie for testing. Im looking forward to implementing
> it.
> >>>>>
> >>>>> Regards
> >>>>> Michael Knill
> >>>>>
> >>>>> On 22/5/18, 6:59 am, "Lonnie Abelbeck" <li...@lonnie.abelbeck.com>
> wrote:
> >>>>>
> >>>>> Followup, Enabling Failover using a Netgear LB1121-100NAS (review
> below):
> >>>>>
> >>>>> The basic failover configuration is documented here:
> >>>>>
> >>>>> WAN Failover
> >>>>> https://doc.astlinux-project.org/userdoc:tt_wan_failover
> >>>>>
> >>>>> Since most 4G/LTE providers only support outbound-only (NAT'ed),
> IPv4-only, dynamic IPv4 address networks, any basic failover configuration
> over 4G/LTE must deal with those constraints.
> >>>>>
> >>>>> But, there is another way ...
> >>>>>
> >>>>> Enhanced WAN Failover using WireGuard:
> >>>>>
> >>>>> If you are able to run a second AstLinux instance (or most any
> distro with WireGuard) on a static IPv4 address you can establish an
> always-up WireGuard VPN over the 4G/LTE connection. When idle the VPN
> consumes less than 0.5 MB/day of data.
> >>>>>
> >>>>> With this setup, both IPv4 and IPv6 can be supported as well as
> allowing inbound traffic to the failover. When failover occurs, all the
> IPv4/IPv6 traffic is sent over the WireGuard VPN to the "Static" WireGuard
> endpoint.
> >>>>>
> >>>>> To be clear, while the WireGuard VPN is established over IPv4-only,
> the tunnel can simultaneously transport IPv4 and IPv6.
> >>>>>
> >>>>> Example:
> >>>>>
> >>>>> AstLinux "4G/LTE": Cable/DSL Modem on external interface and 4G/LTE
> Modem on failover interface.
> >>>>> --
> >>>>> Internal 1st LAN IPv4: 192.168.101.1/255.255.255.0
> >>>>> Internal 1st LAN IPv6: fda6:a6:a6:d2::1/64
> >>>>> WireGuard IPv4: 10.4.1.10/255.255.255.0
> >>>>> WireGuard IPv6: fda6:a6:a6:ff::10/64
> >>>>> IPv6 ULA/NPTv6: fda6:a6:a6::/56
> >>>>>
> >>>>> AstLinux "Static": Static IPv4 (or IPv4/IPv6) on external interface.
> >>>>> --
> >>>>> Routable Public IPv4: 1.2.3.4
> >>>>> WireGuard IPv4: 10.4.1.1/255.255.255.0
> >>>>> WireGuard IPv6: fda6:a6:a6:ff::1/64
> >>>>> IPv6 ULA/NPTv6: fda6:a6:a6::/56
> >>>>>
> >>>>>
> >>>>> == AstLinux "4G/LTE" Endpoint Configuration
> >>>>>
> >>>>> Network tab -> WireGuard Configuration:
> >>>>> Tunnel Options:
> >>>>> IPv4 Address: 10.4.1.10
> >>>>> IPv4 NetMask: 255.255.255.0
> >>>>> IPv6/nn Address: fda6:a6:a6:ff::10/64
> >>>>>
> >>>>> -- /mnt/kd/wireguard/peer/wg0.peer snippet --
> >>>>> [Peer]
> >>>>> ## 4G/LTE Endpoint
> >>>>> PublicKey = <For Static Endpoint>
> >>>>> Endpoint = 1.2.3.4:51820
> >>>>> AllowedIPs = 0.0.0.0/0, ::/0
> >>>>> PersistentKeepalive = 25
> >>>>> --
> >>>>>
> >>>>> Network tab -> WAN Failover Configuration:
> >>>>> WAN Failover:
> >>>>> Failover: [enabled]
> >>>>> Secondary Gateway IPv4: 10.4.1.1
> >>>>> Secondary Gateway IPv6: fda6:a6:a6:ff::1
> >>>>>
> >>>>> External Failover Interface:
> >>>>> Connection Type: [DHCP]
> >>>>>
> >>>>> External Failover Destination Routes:
> >>>>> IPv4 Routes: 192.168.5.0/24 1.2.3.4
> >>>>>
> >>>>>
> >>>>> Network tab -> Firewall Configuration:
> >>>>> Firewall Options:
> >>>>> _x_ Allow WireGuard VPN tunnel to the [1st] LAN Interface(s)
> >>>>>
> >>>>>
> >>>>> == AstLinux "Static" Endpoint Configuration
> >>>>>
> >>>>> Network tab -> WireGuard Configuration:
> >>>>> Tunnel Options:
> >>>>> IPv4 Address: 10.4.1.1
> >>>>> IPv4 NetMask: 255.255.255.0
> >>>>> IPv6/nn Address: fda6:a6:a6:ff::1/64
> >>>>>
> >>>>>
> >>>>> -- /mnt/kd/wireguard/peer/wg0.peer snippet --
> >>>>> [Peer]
> >>>>> ## Static Endpoint
> >>>>> PublicKey = <For 4G/LTE Endpoint>
> >>>>> AllowedIPs = 10.4.1.10/32, 192.168.101.0/24, fda6:a6:a6:ff::10/128,
> fda6:a6:a6:d2::/64
> >>>>> --
> >>>>>
> >>>>> -- /mnt/kd/rc.conf.d/user.conf snippet --
> >>>>> NAT_FOREIGN_NETWORK="192.168.101.0/24"
> >>>>> --
> >>>>>
> >>>>> ==
> >>>>>
> >>>>> I personally tested this scenario and it worked as expected.
> >>>>>
> >>>>> Note that one AstLinux "Static" server can support many remote
> failover AstLinux "4G/LTE" boxes.
> >>>>>
> >>>>> Tip: if you have shell access to AstLinux "Static", 'ssh
> root@10.4.1.10' will access AstLinux "4G/LTE" over the VPN connection,
> regardless if failover is active.
> >>>>>
> >>>>> Lonnie
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> ==================================
> >>>>> Per a post by Michael Knill "4G backup" I purchased a Netgear
> LB1121-100NAS (North America) supporting PoE and includes a power adapter.
> >>>>>
> >>>>> LTE Modem LB1120 and LB1121 User Manual
> >>>>> https://www.downloads.netgear.com/files/GDC/LB1120/LB112x_UM_EN.pdf
> >>>>>
> >>>>> Overall, I'm pleased with the LB1121, the PoE is good to have, makes
> easy positioning for good reception.
> >>>>>
> >>>>> I also tested the Netgear 6000450 MIMO Antenna, it can add 1-bar,
> but with no antenna and 4 out of 5 bars sitting on the lab bench I was able
> to get 90/20 Mbps (down/up) on a speed test.
> >>>>>
> >>>>> If a person were to mount the modem on a wall next to a window, the
> antenna would be useful to reach over and place on the glass.
> >>>>>
> >>>>> I tested with "Ting" a MVNO (Mobile Virtual Network Operator) for
> T-Mobile's GSM network. I ordered a GSM SIM card from Ting, the Netgear
> LB1121 comes with an empty SIM slot.
> >>>>>
> >>>>> I connected the Netgear LB1121 to a spare ethernet interface,
> Network tab -> Failover Interface: [eth2] and also ...
> >>>>> -- Network tab -> WAN Failover Configuration: --
> >>>>> External Failover Interface:
> >>>>> Connection Type: [DHCP]
> >>>>>
> >>>>> External Failover Destination Routes:
> >>>>> IPv4 Routes: 192.168.5.0/24
> >>>>> --
> >>>>> If you change the LB1121's IPv4 address, also change the above IPv4
> Routes: as this is required when the LB1121 is set to "Bridge Mode".
> >>>>> Note: WAN Failover is disabled at this point in time. We are now
> simply defining a 2nd external interface.
> >>>>>
> >>>>> With Ting I needed to edit the APN ...
> >>>>> --
> >>>>> Ting (GSM) T-Mobile
> >>>>> APN: wholesale
> >>>>> --
> >>>>> and the LB1121 easily allows for that via the web interface, which
> defaults to http://192.168.5.1
> >>>>>
> >>>>> Firmware updates are via the web interface, but you must have a SIM
> card activated and installed to perform an upgrade over the GSM network.
> >>>>>
> >>>>> Web interface password changes don't ask for a match, so a typo
> requires a reset to factory defaults to fix it. But overall, the web
> interface is nicely done.
> >>>>>
> >>>>> After I got the LB1121 configured as desired, working, and firmware
> upgraded, I then switched to "Bridge Mode", depending on your 4G/LTE
> carrier your DHCP will acquire a publicly routable IPv4 address or an
> address that looks public but is actually behind NAT.
> >>>>> BTW: Ting/T-Mobile uses odd "private" address ranges like 25.0.0.0/8
> (UK Ministry of Defense) and 100.128.0.0/9 (T-Mobile), they look publicly
> routable, but they are NAT'ed to a different public address :-(
> >>>>>
> >>>>> On a PoE 802.3af switch, the LB1121 draws 1.1 Watts, cool to the
> touch.
> >>>>>
> >>>>> The main issues are the 4G/LTE networks, the Ting MVNO for T-Mobile
> is IPv4 only, and NAT'ed even when in bridge mode. So a true failover is
> difficult to do, but by limiting your failover requirements this can still
> be useful. Below is one such technique using WireGuard VPN.
> >>>>>
> >>>>> I have a test AstLinux box talking to my main AstLinux box over
> WireGuard over 4G/LTE ... works nicely. Though "PersistentKeepalive = 25"
> is required to deal with the NAT and dynamic addressing.
> >>>>>
> >>>>> FYI: Interestingly, the WireGuard overhead even with a keepalive
> every 25 seconds results in 454 KB/day of data, which at $10/GB is only
> 0.00454 $/day.
> >>>>>
> >>>>> == Dynamic 4G/LTE Modem Endpoint
> >>>>>
> >>>>> -- WireGuard IPv4 10.4.1.10/255.255.255.0 --
> >>>>> [Peer]
> >>>>> ## 4G/LTE Endpoint
> >>>>> PublicKey = <For Static Endpoint>
> >>>>> Endpoint = 1.2.3.4:51820
> >>>>> AllowedIPs = 10.4.1.1/32
> >>>>> PersistentKeepalive = 25
> >>>>> --
> >>>>>
> >>>>> -- Network tab -> WAN Failover Configuration: --
> >>>>> External Failover Interface:
> >>>>> Connection Type: [DHCP]
> >>>>>
> >>>>> External Failover Destination Routes:
> >>>>> IPv4 Routes: 192.168.5.0/24 1.2.3.4
> >>>>> --
> >>>>>
> >>>>> == Static IPv4 1.2.3.4 Endpoint
> >>>>>
> >>>>> -- WireGuard IPv4 10.4.1.1/255.255.255.0 --
> >>>>> [Peer]
> >>>>> ## Static Endpoint
> >>>>> PublicKey = <For 4G/LTE Endpoint>
> >>>>> AllowedIPs = 10.4.1.10/32
> >>>>> --
> >>>>>
> >>>>> iperf3 test across the VPN ...
> >>>>>
> >>>>> 4G/LTE ~ # iperf3 -s
> >>>>>
> >>>>> Static ~ # iperf3 -c 10.4.1.10 -u
> >>>>> Connecting to host 10.4.1.10, port 5201
> >>>>> [ 5] local 10.4.1.1 port 37415 connected to 10.4.1.10 port 5201
> >>>>> [ ID] Interval Transfer Bitrate Total Datagrams
> >>>>> [ 5] 0.00-1.00 sec 128 KBytes 1.05 Mbits/sec 96
> >>>>> ...
> >>>>> [ 5] 9.00-10.00 sec 128 KBytes 1.05 Mbits/sec 96
> >>>>> - - - - - - - - - - - - - - - - - - - - - - - - -
> >>>>> [ ID] Interval Transfer Bitrate Jitter
> Lost/Total Datagrams
> >>>>> [ 5] 0.00-10.00 sec 1.25 MBytes 1.05 Mbits/sec 0.000 ms
> 0/959 (0%) sender
> >>>>> [ 5] 0.00-10.16 sec 1.25 MBytes 1.03 Mbits/sec 2.543 ms
> 0/959 (0%) receiver
> >>>>>
> >>>>>
> >>>>> Typical ping times: 100-400 ms
> >>>>>
> >>>>> Note that without the VPN there would be no way to reach "4G/LTE"
> from "Static" with the network NAT issues described above.
> >>>>>
> >>>>> So with a Netgear LB1121 4G/LTE Modem, by using this WireGuard VPN
> technique on the "Failover Interface" (2nd External) your public server on
> 1.2.3.4 will be able to access a remote AstLinux box via 4G/LTE.
> >>>>>
> >>>>>
> >>>>> Lonnie
> >>>>>
> >>>>>
> >>>>> ------------------------------------------------------------
> ------------------
> >>>>> Check out the vibrant tech community on one of the world's most
> >>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> >>>>> _______________________________________________
> >>>>> Astlinux-users mailing list
> >>>>> Astlinux-users@lists.sourceforge.net
> >>>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
> >>>>>
> >>>>> Donations to support AstLinux are graciously accepted via PayPal to
> pay...@krisk.org.
> >>>>>
> >>>>>
> >>>>> ------------------------------------------------------------
> ------------------
> >>>>> Check out the vibrant tech community on one of the world's most
> >>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> >>>>> _______________________________________________
> >>>>> Astlinux-users mailing list
> >>>>> Astlinux-users@lists.sourceforge.net
> >>>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
> >>>>>
> >>>>> Donations to support AstLinux are graciously accepted via PayPal to
> pay...@krisk.org.
> >>>>
> >>>>
> >>>> ------------------------------------------------------------
> ------------------
> >>>> Check out the vibrant tech community on one of the world's most
> >>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> >>>> _______________________________________________
> >>>> Astlinux-users mailing list
> >>>> Astlinux-users@lists.sourceforge.net
> >>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
> >>>>
> >>>> Donations to support AstLinux are graciously accepted via PayPal to
> pay...@krisk.org.
> >>>>
> >>>> ------------------------------------------------------------
> ------------------
> >>>> Check out the vibrant tech community on one of the world's most
> >>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> >>>> _______________________________________________
> >>>> Astlinux-users mailing list
> >>>> Astlinux-users@lists.sourceforge.net
> >>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
> >>>>
> >>>> Donations to support AstLinux are graciously accepted via PayPal to
> pay...@krisk.org.
> >>>
> >>>
> >>> ------------------------------------------------------------
> ------------------
> >>> Check out the vibrant tech community on one of the world's most
> >>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> >>> _______________________________________________
> >>> Astlinux-users mailing list
> >>> Astlinux-users@lists.sourceforge.net
> >>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
> >>>
> >>> Donations to support AstLinux are graciously accepted via PayPal
> to pay...@krisk.org.
> >>>
> >>> ------------------------------------------------------------
> ------------------
> >>> Check out the vibrant tech community on one of the world's most
> >>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> >>> _______________________________________________
> >>> Astlinux-users mailing list
> >>> Astlinux-users@lists.sourceforge.net
> >>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
> >>>
> >>> Donations to support AstLinux are graciously accepted via PayPal to
> pay...@krisk.org.
> >>>
> >>> ------------------------------------------------------------
> ------------------
> >>> Check out the vibrant tech community on one of the world's most
> >>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> >>> _______________________________________________
> >>> Astlinux-users mailing list
> >>> Astlinux-users@lists.sourceforge.net
> >>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
> >>>
> >>> Donations to support AstLinux are graciously accepted via PayPal to
> pay...@krisk.org.
> >>
> >>
> >> ------------------------------------------------------------
> ------------------
> >> Check out the vibrant tech community on one of the world's most
> >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> >> _______________________________________________
> >> Astlinux-users mailing list
> >> Astlinux-users@lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/astlinux-users
> >>
> >> Donations to support AstLinux are graciously accepted via PayPal to
> pay...@krisk.org.
> >>
> >> ------------------------------------------------------------
> ------------------
> >> Check out the vibrant tech community on one of the world's most
> >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> >> _______________________________________________
> >> Astlinux-users mailing list
> >> Astlinux-users@lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/astlinux-users
> >>
> >> Donations to support AstLinux are graciously accepted via PayPal to
> pay...@krisk.org.
> >
> >
> > ------------------------------------------------------------
> ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Astlinux-users mailing list
> > Astlinux-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/astlinux-users
> >
> > Donations to support AstLinux are graciously accepted via PayPal to
> pay...@krisk.org.
> >
> > ------------------------------------------------------------
> ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Astlinux-users mailing list
> > Astlinux-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/astlinux-users
> >
> > Donations to support AstLinux are graciously accepted via PayPal to
> pay...@krisk.org.
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Astlinux-users mailing list
> Astlinux-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>
> Donations to support AstLinux are graciously accepted via PayPal to
> pay...@krisk.org.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Astlinux-users mailing list
Astlinux-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/astlinux-users
Donations to support AstLinux are graciously accepted via PayPal to
pay...@krisk.org.
