I have now more completely tested my WAN Failover using 4G/LTE over WireGuard
...and now works perfectly with Asterisk with a configuration change.
My 4G/LTE over WireGuard tunnel endpoint is with AstLinux on a Linode KVM (as
documented previously). The AstLinux on the Linode KVM instance is not running
Asterisk.
Previously my local Asterisk was configured with nat=no and since I have a
static IPv4 address my SIP provider has an option to authenticate using the
static address (no registration). If I added the static IPv4 of my Linode KVM
instance it mostly worked for outbound calls during failover but would require
failover call configuration at the SIP provided end to handle inbound calls.
Instead, I switched my SIP provider configuration to require registration and
removed my static IPv4 authentication. In order to force a quick registration
update on the switch I use the wan-failover.script to reload sip:
-- /mnt/kd/wan-failover.script snippet --
SECONDARY)
## Switched to Failover using secondary WAN link
asterisk -rx "sip reload" >/dev/null
;;
PRIMARY)
## Switched back to normal using primary WAN link
asterisk -rx "sip reload" >/dev/null
;;
--
Almost there, but when failover occurs the registration is NAT'ed at the
AstLinux on Linode KVM end, so for my SIP provider all it took was to configure
my SIP account to support NAT. Even though my Asterisk is still configured
with nat=no, the registration via the failover tunnel now works.
Inbound and outbound SIP calls work perfectly over the 4G/LTE failover.
Most interestingly, the calls keep working during the failover switch and back
again ... I was not expecting that, and I am still scratching my head over
that. The active calls do not miss a beat as I see data over my 4G/LTE and
then no data over 4G/LTE, the call continues perfectly. I even have
directmedia=no in my provider peer's sip.conf.
Lonnie
> On May 28, 2018, at 8:30 PM, Michael Knill
> <michael.kn...@ipcsolutions.com.au> wrote:
>
> I decided today the architecture I want to test.
> I will be setting up a DR Astlinux box running Asterisk and having a SIP
> Trunk with a 100 Block number range on it for indial.
> All Astklinux boxes configured for failover will establish a VPN to this box
> and a SIP Trunk to the DR server will be second choice for outgoing calls. I
> can set the Caller ID to be anything I want for DR outgoing so no issues
> there. Billing should be fine also as long as the Originating Caller ID is
> correct.
> All my providers allow the setting of a call forward on busy (and
> unreachable) to forward to one of the numbers in the 100 block which is
> redirected to the associated Astlinux box. So incoming calls are sorted also.
> If it's the same provider, then all forwarded calls are free.
>
> I think it should work with no manual intervention required.
>
> Regards
> Michael Knill
>
> On 29/5/18, 10:07 am, "Lonnie Abelbeck" <li...@lonnie.abelbeck.com> wrote:
>
> Hey Michael,
>
> Currently my "cloud" AstLinux Linode KVM (4G/LTE over VPN endpoint) has
> Asterisk disabled, ASTERISK_DAHDI_DISABLE="yes" so when my main AstLinux box
> goes to failover the SIP packets originate from my "home" WireGuard private
> 10.0.0.0/24 address and are NAT'ed at the Linode end. Normally there is no
> NAT and SIP packets originate with my public IP.
>
> Surprisingly outbound calls work without any changes, though a remote
> hangup BYE is not received. Inbound failover could be handled on my SIP
> provider's platform.
>
> I could enable Asterisk at the Linode end and use it as a proxy of sorts.
>
> Possibly better, per what you suggested, is to implement an outbound call
> failover in the "main" Asterisk regardless of the WAN Failover status ... if
> the standard outbound call fails it tries the call over the 4G/LTE VPN path.
>
> I'm still pondering my options. The exact solution is somewhat dependent
> on the user's SIP provider.
>
> Lonnie
>
>
>
>
>> On May 28, 2018, at 5:08 PM, Michael Knill
>> <michael.kn...@ipcsolutions.com.au> wrote:
>>
>> Hi Lonnie
>>
>> So what are you trying to solve with Asterisk failover?
>>
>> Regards
>> Michael Knill
>>
>> On 28/5/18, 10:00 pm, "Lonnie Abelbeck" <li...@lonnie.abelbeck.com> wrote:
>>
>> Hi Michael,
>>
>> Yes, you can use OpenVPN and WireGuard at the same time, no problem. I do.
>>
>> WireGuard is much faster / more efficient than OpenVPN, mostly since it
>> resides in the kernel and can use multiple cores. Here are some benchmarks
>> I posted to the WireGuard mailing list:
>>
>> https://lists.zx2c4.com/pipermail/wireguard/2017-December/002204.html
>>
>> There are user-space implementations of WireGuard, written in Golang,
>> starting to appear for testing, but for non-Linux endpoints I would stick
>> with OpenVPN for now.
>>
>> BTW, I currently have WAN Failover on my production AstLinux box using the
>> Netgear LB1121 4G/LTE over WireGuard VPN to a Linode KVM running AstLinux.
>> Working is dual stack IPv4/IPv6 failover for the AstLinux box itself and any
>> internal network of my choosing. I have outbound Asterisk failover working,
>> but that is still a work in progress, not sure the best method yet.
>>
>> Lonnie
>>
>>
>>
>>
>>> On May 28, 2018, at 5:03 AM, Michael Knill
>>> <michael.kn...@ipcsolutions.com.au> wrote:
>>>
>>> Hi group
>>>
>>> Im ready to do some testing.
>>> I have a number of sites that are set up as OpenVPN Servers. Should there
>>> be any issues using Wireguard as well?
>>> PS I just looked up Wireguard and I cant believe the difference in
>>> benchmarks to Open VPN. That's crazy!
>>>
>>> Regards
>>> Michael Knill
>>>
>>> On 24/5/18, 9:23 am, "Michael Knill" <michael.kn...@ipcsolutions.com.au>
>>> wrote:
>>>
>>> Thanks Lonnie. I don't have a specific scenario yet but handy to know its
>>> possible.
>>>
>>> Regards
>>> Michael Knill
>>>
>>> On 24/5/18, 8:54 am, "Lonnie Abelbeck" <li...@lonnie.abelbeck.com> wrote:
>>>
>>> Michael,
>>>
>>>> So are you saying that you can configure a second external interface and
>>>> the associated routing to it with the Failover Tab but just leave Failover
>>>> disabled?
>>>
>>> Yes, "External Failover Destination Routes:" automatically defines
>>> static routes, automatically removed and added for DHCP changes.
>>>
>>>
>>>> If so, I assume it uses the same EXT firewall rules?
>>>
>>> Yes. There is a way to treat EXTIF and EXT2IF firewall rules
>>> differently, but the same is usually OK.
>>>
>>> Lonnie
>>>
>>>
>>>
>>>> On May 23, 2018, at 5:17 PM, Michael Knill
>>>> <michael.kn...@ipcsolutions.com.au> wrote:
>>>>
>>>> Hi Lonnie
>>>>
>>>> So are you saying that you can configure a second external interface and
>>>> the associated routing to it with the Failover Tab but just leave Failover
>>>> disabled?
>>>> If so, I assume it uses the same EXT firewall rules?
>>>>
>>>> Regards
>>>> Michael Knill
>>>>
>>>> On 22/5/18, 8:59 am, "Lonnie Abelbeck" <li...@lonnie.abelbeck.com> wrote:
>>>>
>>>> Hi Michael,
>>>>
>>>>> I noticed you also pass the VPN traffic to the site LAN
>>>>
>>>> Yes, I tried to implement the general case, easy to remove stuff.
>>>>
>>>>> the VPN would normally just be used for voice traffic and management only.
>>>>
>>>> In that case "External Failover Destination Routes: IPv4 Routes:" could
>>>> define all the destination routes you need without "Failover" enabled ...
>>>> and let Asterisk dynamically choose the SIP route. Handling inbound calls
>>>> over the 4G/LTE VPN would also be possible.
>>>>
>>>>
>>>> All seems to work well, the only fundamental issue may be the latency of
>>>> 4G/LTE for SIP traffic ... though clearly much better than no traffic.
>>>>
>>>> Lonnie
>>>>
>>>>
>>>>
>>>>
>>>>> On May 21, 2018, at 5:36 PM, Michael Knill
>>>>> <michael.kn...@ipcsolutions.com.au> wrote:
>>>>>
>>>>> Thanks Lonnie you beat me to it.
>>>>> Interestingly one of my partners is using Asterisk as their Softswitch
>>>>> and they were thinking of setting up a single VPN Tunnel to the
>>>>> SoftSwitch for voice traffic and so everything still works on both the
>>>>> primary and failover links. There should be no failover scripts required!
>>>>>
>>>>> I noticed you also pass the VPN traffic to the site LAN but this would
>>>>> not actually be required in practice as the VPN would normally just be
>>>>> used for voice traffic and management only. On all VPN connections that
>>>>> run voice traffic I set directmedia=no in sip.conf. PS I actually now use
>>>>> a directmedia ACL on the VPN subnet so I don't need to configure
>>>>> anything. E.g.
>>>>>
>>>>> directmedia=yes
>>>>> directmediapermit=0.0.0.0/0
>>>>> directmediadeny=<VPN Subnet>
>>>>>
>>>>> Thanks again Lonnie for testing. Im looking forward to implementing it.
>>>>>
>>>>> Regards
>>>>> Michael Knill
>>>>>
>>>>> On 22/5/18, 6:59 am, "Lonnie Abelbeck" <li...@lonnie.abelbeck.com> wrote:
>>>>>
>>>>> Followup, Enabling Failover using a Netgear LB1121-100NAS (review below):
>>>>>
>>>>> The basic failover configuration is documented here:
>>>>>
>>>>> WAN Failover
>>>>> https://doc.astlinux-project.org/userdoc:tt_wan_failover
>>>>>
>>>>> Since most 4G/LTE providers only support outbound-only (NAT'ed),
>>>>> IPv4-only, dynamic IPv4 address networks, any basic failover
>>>>> configuration over 4G/LTE must deal with those constraints.
>>>>>
>>>>> But, there is another way ...
>>>>>
>>>>> Enhanced WAN Failover using WireGuard:
>>>>>
>>>>> If you are able to run a second AstLinux instance (or most any distro
>>>>> with WireGuard) on a static IPv4 address you can establish an always-up
>>>>> WireGuard VPN over the 4G/LTE connection. When idle the VPN consumes
>>>>> less than 0.5 MB/day of data.
>>>>>
>>>>> With this setup, both IPv4 and IPv6 can be supported as well as allowing
>>>>> inbound traffic to the failover. When failover occurs, all the IPv4/IPv6
>>>>> traffic is sent over the WireGuard VPN to the "Static" WireGuard endpoint.
>>>>>
>>>>> To be clear, while the WireGuard VPN is established over IPv4-only, the
>>>>> tunnel can simultaneously transport IPv4 and IPv6.
>>>>>
>>>>> Example:
>>>>>
>>>>> AstLinux "4G/LTE": Cable/DSL Modem on external interface and 4G/LTE Modem
>>>>> on failover interface.
>>>>> --
>>>>> Internal 1st LAN IPv4: 192.168.101.1/255.255.255.0
>>>>> Internal 1st LAN IPv6: fda6:a6:a6:d2::1/64
>>>>> WireGuard IPv4: 10.4.1.10/255.255.255.0
>>>>> WireGuard IPv6: fda6:a6:a6:ff::10/64
>>>>> IPv6 ULA/NPTv6: fda6:a6:a6::/56
>>>>>
>>>>> AstLinux "Static": Static IPv4 (or IPv4/IPv6) on external interface.
>>>>> --
>>>>> Routable Public IPv4: 1.2.3.4
>>>>> WireGuard IPv4: 10.4.1.1/255.255.255.0
>>>>> WireGuard IPv6: fda6:a6:a6:ff::1/64
>>>>> IPv6 ULA/NPTv6: fda6:a6:a6::/56
>>>>>
>>>>>
>>>>> == AstLinux "4G/LTE" Endpoint Configuration
>>>>>
>>>>> Network tab -> WireGuard Configuration:
>>>>> Tunnel Options:
>>>>> IPv4 Address: 10.4.1.10
>>>>> IPv4 NetMask: 255.255.255.0
>>>>> IPv6/nn Address: fda6:a6:a6:ff::10/64
>>>>>
>>>>> -- /mnt/kd/wireguard/peer/wg0.peer snippet --
>>>>> [Peer]
>>>>> ## 4G/LTE Endpoint
>>>>> PublicKey = <For Static Endpoint>
>>>>> Endpoint = 1.2.3.4:51820
>>>>> AllowedIPs = 0.0.0.0/0, ::/0
>>>>> PersistentKeepalive = 25
>>>>> --
>>>>>
>>>>> Network tab -> WAN Failover Configuration:
>>>>> WAN Failover:
>>>>> Failover: [enabled]
>>>>> Secondary Gateway IPv4: 10.4.1.1
>>>>> Secondary Gateway IPv6: fda6:a6:a6:ff::1
>>>>>
>>>>> External Failover Interface:
>>>>> Connection Type: [DHCP]
>>>>>
>>>>> External Failover Destination Routes:
>>>>> IPv4 Routes: 192.168.5.0/24 1.2.3.4
>>>>>
>>>>>
>>>>> Network tab -> Firewall Configuration:
>>>>> Firewall Options:
>>>>> _x_ Allow WireGuard VPN tunnel to the [1st] LAN Interface(s)
>>>>>
>>>>>
>>>>> == AstLinux "Static" Endpoint Configuration
>>>>>
>>>>> Network tab -> WireGuard Configuration:
>>>>> Tunnel Options:
>>>>> IPv4 Address: 10.4.1.1
>>>>> IPv4 NetMask: 255.255.255.0
>>>>> IPv6/nn Address: fda6:a6:a6:ff::1/64
>>>>>
>>>>>
>>>>> -- /mnt/kd/wireguard/peer/wg0.peer snippet --
>>>>> [Peer]
>>>>> ## Static Endpoint
>>>>> PublicKey = <For 4G/LTE Endpoint>
>>>>> AllowedIPs = 10.4.1.10/32, 192.168.101.0/24, fda6:a6:a6:ff::10/128,
>>>>> fda6:a6:a6:d2::/64
>>>>> --
>>>>>
>>>>> -- /mnt/kd/rc.conf.d/user.conf snippet --
>>>>> NAT_FOREIGN_NETWORK="192.168.101.0/24"
>>>>> --
>>>>>
>>>>> ==
>>>>>
>>>>> I personally tested this scenario and it worked as expected.
>>>>>
>>>>> Note that one AstLinux "Static" server can support many remote failover
>>>>> AstLinux "4G/LTE" boxes.
>>>>>
>>>>> Tip: if you have shell access to AstLinux "Static", 'ssh root@10.4.1.10'
>>>>> will access AstLinux "4G/LTE" over the VPN connection, regardless if
>>>>> failover is active.
>>>>>
>>>>> Lonnie
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ==================================
>>>>> Per a post by Michael Knill "4G backup" I purchased a Netgear
>>>>> LB1121-100NAS (North America) supporting PoE and includes a power adapter.
>>>>>
>>>>> LTE Modem LB1120 and LB1121 User Manual
>>>>> https://www.downloads.netgear.com/files/GDC/LB1120/LB112x_UM_EN.pdf
>>>>>
>>>>> Overall, I'm pleased with the LB1121, the PoE is good to have, makes easy
>>>>> positioning for good reception.
>>>>>
>>>>> I also tested the Netgear 6000450 MIMO Antenna, it can add 1-bar, but
>>>>> with no antenna and 4 out of 5 bars sitting on the lab bench I was able
>>>>> to get 90/20 Mbps (down/up) on a speed test.
>>>>>
>>>>> If a person were to mount the modem on a wall next to a window, the
>>>>> antenna would be useful to reach over and place on the glass.
>>>>>
>>>>> I tested with "Ting" a MVNO (Mobile Virtual Network Operator) for
>>>>> T-Mobile's GSM network. I ordered a GSM SIM card from Ting, the Netgear
>>>>> LB1121 comes with an empty SIM slot.
>>>>>
>>>>> I connected the Netgear LB1121 to a spare ethernet interface, Network tab
>>>>> -> Failover Interface: [eth2] and also ...
>>>>> -- Network tab -> WAN Failover Configuration: --
>>>>> External Failover Interface:
>>>>> Connection Type: [DHCP]
>>>>>
>>>>> External Failover Destination Routes:
>>>>> IPv4 Routes: 192.168.5.0/24
>>>>> --
>>>>> If you change the LB1121's IPv4 address, also change the above IPv4
>>>>> Routes: as this is required when the LB1121 is set to "Bridge Mode".
>>>>> Note: WAN Failover is disabled at this point in time. We are now simply
>>>>> defining a 2nd external interface.
>>>>>
>>>>> With Ting I needed to edit the APN ...
>>>>> --
>>>>> Ting (GSM) T-Mobile
>>>>> APN: wholesale
>>>>> --
>>>>> and the LB1121 easily allows for that via the web interface, which
>>>>> defaults to http://192.168.5.1
>>>>>
>>>>> Firmware updates are via the web interface, but you must have a SIM card
>>>>> activated and installed to perform an upgrade over the GSM network.
>>>>>
>>>>> Web interface password changes don't ask for a match, so a typo requires
>>>>> a reset to factory defaults to fix it. But overall, the web interface is
>>>>> nicely done.
>>>>>
>>>>> After I got the LB1121 configured as desired, working, and firmware
>>>>> upgraded, I then switched to "Bridge Mode", depending on your 4G/LTE
>>>>> carrier your DHCP will acquire a publicly routable IPv4 address or an
>>>>> address that looks public but is actually behind NAT.
>>>>> BTW: Ting/T-Mobile uses odd "private" address ranges like 25.0.0.0/8 (UK
>>>>> Ministry of Defense) and 100.128.0.0/9 (T-Mobile), they look publicly
>>>>> routable, but they are NAT'ed to a different public address :-(
>>>>>
>>>>> On a PoE 802.3af switch, the LB1121 draws 1.1 Watts, cool to the touch.
>>>>>
>>>>> The main issues are the 4G/LTE networks, the Ting MVNO for T-Mobile is
>>>>> IPv4 only, and NAT'ed even when in bridge mode. So a true failover is
>>>>> difficult to do, but by limiting your failover requirements this can
>>>>> still be useful. Below is one such technique using WireGuard VPN.
>>>>>
>>>>> I have a test AstLinux box talking to my main AstLinux box over WireGuard
>>>>> over 4G/LTE ... works nicely. Though "PersistentKeepalive = 25" is
>>>>> required to deal with the NAT and dynamic addressing.
>>>>>
>>>>> FYI: Interestingly, the WireGuard overhead even with a keepalive every 25
>>>>> seconds results in 454 KB/day of data, which at $10/GB is only 0.00454
>>>>> $/day.
>>>>>
>>>>> == Dynamic 4G/LTE Modem Endpoint
>>>>>
>>>>> -- WireGuard IPv4 10.4.1.10/255.255.255.0 --
>>>>> [Peer]
>>>>> ## 4G/LTE Endpoint
>>>>> PublicKey = <For Static Endpoint>
>>>>> Endpoint = 1.2.3.4:51820
>>>>> AllowedIPs = 10.4.1.1/32
>>>>> PersistentKeepalive = 25
>>>>> --
>>>>>
>>>>> -- Network tab -> WAN Failover Configuration: --
>>>>> External Failover Interface:
>>>>> Connection Type: [DHCP]
>>>>>
>>>>> External Failover Destination Routes:
>>>>> IPv4 Routes: 192.168.5.0/24 1.2.3.4
>>>>> --
>>>>>
>>>>> == Static IPv4 1.2.3.4 Endpoint
>>>>>
>>>>> -- WireGuard IPv4 10.4.1.1/255.255.255.0 --
>>>>> [Peer]
>>>>> ## Static Endpoint
>>>>> PublicKey = <For 4G/LTE Endpoint>
>>>>> AllowedIPs = 10.4.1.10/32
>>>>> --
>>>>>
>>>>> iperf3 test across the VPN ...
>>>>>
>>>>> 4G/LTE ~ # iperf3 -s
>>>>>
>>>>> Static ~ # iperf3 -c 10.4.1.10 -u
>>>>> Connecting to host 10.4.1.10, port 5201
>>>>> [ 5] local 10.4.1.1 port 37415 connected to 10.4.1.10 port 5201
>>>>> [ ID] Interval Transfer Bitrate Total Datagrams
>>>>> [ 5] 0.00-1.00 sec 128 KBytes 1.05 Mbits/sec 96
>>>>> ...
>>>>> [ 5] 9.00-10.00 sec 128 KBytes 1.05 Mbits/sec 96
>>>>> - - - - - - - - - - - - - - - - - - - - - - - - -
>>>>> [ ID] Interval Transfer Bitrate Jitter
>>>>> Lost/Total Datagrams
>>>>> [ 5] 0.00-10.00 sec 1.25 MBytes 1.05 Mbits/sec 0.000 ms 0/959
>>>>> (0%) sender
>>>>> [ 5] 0.00-10.16 sec 1.25 MBytes 1.03 Mbits/sec 2.543 ms 0/959
>>>>> (0%) receiver
>>>>>
>>>>>
>>>>> Typical ping times: 100-400 ms
>>>>>
>>>>> Note that without the VPN there would be no way to reach "4G/LTE" from
>>>>> "Static" with the network NAT issues described above.
>>>>>
>>>>> So with a Netgear LB1121 4G/LTE Modem, by using this WireGuard VPN
>>>>> technique on the "Failover Interface" (2nd External) your public server
>>>>> on 1.2.3.4 will be able to access a remote AstLinux box via 4G/LTE.
>>>>>
>>>>>
>>>>> Lonnie
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Check out the vibrant tech community on one of the world's most
>>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>>> _______________________________________________
>>>>> Astlinux-users mailing list
>>>>> Astlinux-users@lists.sourceforge.net
>>>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>>>>>
>>>>> Donations to support AstLinux are graciously accepted via PayPal to
>>>>> pay...@krisk.org.
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Check out the vibrant tech community on one of the world's most
>>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>>> _______________________________________________
>>>>> Astlinux-users mailing list
>>>>> Astlinux-users@lists.sourceforge.net
>>>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>>>>>
>>>>> Donations to support AstLinux are graciously accepted via PayPal to
>>>>> pay...@krisk.org.
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Check out the vibrant tech community on one of the world's most
>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>> _______________________________________________
>>>> Astlinux-users mailing list
>>>> Astlinux-users@lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>>>>
>>>> Donations to support AstLinux are graciously accepted via PayPal to
>>>> pay...@krisk.org.
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Check out the vibrant tech community on one of the world's most
>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>> _______________________________________________
>>>> Astlinux-users mailing list
>>>> Astlinux-users@lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>>>>
>>>> Donations to support AstLinux are graciously accepted via PayPal to
>>>> pay...@krisk.org.
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>> _______________________________________________
>>> Astlinux-users mailing list
>>> Astlinux-users@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>>>
>>> Donations to support AstLinux are graciously accepted via PayPal to
>>> pay...@krisk.org.
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>> _______________________________________________
>>> Astlinux-users mailing list
>>> Astlinux-users@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>>>
>>> Donations to support AstLinux are graciously accepted via PayPal to
>>> pay...@krisk.org.
>>>
>>> ------------------------------------------------------------------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>> _______________________________________________
>>> Astlinux-users mailing list
>>> Astlinux-users@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>>>
>>> Donations to support AstLinux are graciously accepted via PayPal to
>>> pay...@krisk.org.
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Astlinux-users mailing list
>> Astlinux-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>>
>> Donations to support AstLinux are graciously accepted via PayPal to
>> pay...@krisk.org.
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Astlinux-users mailing list
>> Astlinux-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>>
>> Donations to support AstLinux are graciously accepted via PayPal to
>> pay...@krisk.org.
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Astlinux-users mailing list
> Astlinux-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>
> Donations to support AstLinux are graciously accepted via PayPal to
> pay...@krisk.org.
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Astlinux-users mailing list
> Astlinux-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>
> Donations to support AstLinux are graciously accepted via PayPal to
> pay...@krisk.org.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Astlinux-users mailing list
Astlinux-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/astlinux-users
Donations to support AstLinux are graciously accepted via PayPal to
pay...@krisk.org.
