Hi.
I've had a chain of a sudden unexplained problems which crippled my
tinc network.
It looks as if UDP communication between hosts in different parts of
the globe all of a sudden completely broke.
While waiting for my list subscription to be accepted, I've managed to
solve, (or perhaps mask) the problem by re-enabling PMTUDiscovery for
all hosts involved.
I still have no understanding why this was of any help, as discovery
settled on higher PMTU than the one I've previously used.
I'll attach an original email I've meant to send with all the
details, and if anyone has any ideas about causes, I'll be glad to hear
them out.
Sergey.
--- Begin Message ---
Hi. Sorry about this email being quite long. The situation is bizarre and can't
be briefly described. I've also used this email to reconfirm that I'm seeing
what
I'm seeing and it isn't just a bad dream I can wake up from next morning.
I've been running tinc for awhile without significant troubles up until
yesterday.
Then things went completely sideways in a way, I'm struggling to understand.
First, I'd like to share my network configuration.
Hosts are:
vault13: amd64 VPS on Debian Jessie, tinc version 1.0.24, has public ipv4
address
cassiopeia: amd64 VPS on Debian Wheezy, tinc version 1.0.19, has public ipv4
address
thinkpad: amd64 notebook on Debian Jessie, tinc version 1.0.28, behind router's
NAT
vaio: i386 notebook on Debian Jessie, tinc version 1.0.28, behind router's NAT
All have static addresses inside tinc vpn network:
vault13: 169.254.200.1
cassiopeia: 169.254.200.2
thinkpad: 169.254.200.10
vaio: 169.254.200.11
vault13, cassiopeia and my soho router are all in different countries.
Both thinkpad and vaio are behind the same Wi-Fi router, which has real ipv4
address.
All hosts have identical tinc.conf (except for ConnectTo directives)
ConnectTo = cassiopeia
ConnectTo = vault13
Name = $HOST
LocalDiscovery = yes
PingInterval = 60
PingTimeout = 8
PMTU = 1360
PMTUDiscovery = no
ReplayWindow = 64
ProcessPriority = high
MaxTimeout = 100
Mode = switch
=================
Current situation
Despite this stable configuration remained unchanged in any way for a long
time, it all fell apart around 48 hours ago.
E.g. pings, and Avahi multicasts no longer receive replies.
All machines indicate they successfully established all possible direct
connections.
I'll focus on the exchange from thinkpad with both vault13 and cassiopeia first.
There are no error messages from tincd anywhere.
Just for the reference, here is what get's repeated over and over after sending
SIGINT on thinkpad:
Jun 2 00:40:28 thinkpad tinc.vaultnet[20886]: Read packet of 42 bytes from
Linux tun/tap device (tap mode)
Jun 2 00:40:28 thinkpad tinc.vaultnet[20886]: Broadcasting packet of 42 bytes
from thinkpad (MYSELF)
Jun 2 00:40:28 thinkpad tinc.vaultnet[20886]: Sending packet of 42 bytes to
cassiopeia (x.x.x.x port 655)
Jun 2 00:40:28 thinkpad tinc.vaultnet[20886]: Sending packet of 42 bytes to
vault13 (y.y.y.y port 655)
Let's try to ping both hosts I'm directly connected to, according to tinc:
Cassipeia's tcpdump on the vaultnet (tinc network interface) is flooded with:
00:54:31.073719 ARP, Request who-has cassiopeia.local tell thinkpad.local,
length 28
00:54:31.073737 ARP, Reply cassiopeia.local is-at aa:1e:07:c7:04:70 (oui
Unknown), length 28
Vault13 tcpdump on the vaultnet interface is similarly flooded with:
17:52:57.726623 ARP, Request who-has vault13.local tell thinkpad.local, length
28
17:52:57.726634 ARP, Reply vault13.local is-at 66:5a:29:ba:3c:67 (oui Unknown),
length 28
Thinkpad (ping sender) tcpdump of vaultnet is full of unanswered cries:
0:58:54.693544 ARP, Request who-has 169.254.200.2 tell thinkpad.local, length 28
Thinkpad tcpdump 'port 655' confirms absence of incoming packets
01:00:19.693630 IP thinkpad.local.tinc > cassiopeia.tinc: UDP, length 52
01:00:19.693701 IP thinkpad.local.tinc > vault13.tinc: UDP, length 52
Both vault13 and cassiopeia do receive and do send a reply on physical
interfaces
as confirmed by tcpdump 'port 655'
18:03:40.279285 IP thinkpad.ip.address.tinc >
cassiopeia/vault13.ip.address.tinc: UDP, length 52
18:03:40.279541 IP cassiopeia/vault13.ip.address.tinc >
thinkpad.ip.address.tinc: UDP, length 52
Cassiopeia pings vault13. Vaio is off for this test. Tcpdump on vaultnet:
01:30:33.292052 ARP, Request who-has 169.254.200.1 tell cassiopeia.local,
length 28
Tcpdump on cassiopeia physical interface:
01:24:16.450134 IP cassiopeia.tinc > thinkpad.ip.address.tinc: UDP, length 52
01:24:16.450179 IP cassiopeia.tinc > thinkpad.ip.address.600: UDP, length 52
Thinkpad receives nothing, there are no attempts to send anything to vault13.
Situation is completely identical in reverse, when vault13 pings cassiopeia.
Now let's power on Vaio.. surprisingly it has a stable link to the cassiopeia
and cassiopeia alone.
But when I'm trying to ping vault13 it sends packages not to vault13, but to
the cassiopeia
11:49:31.015805 IP 192.168.0.101.tinc > cassiopeia.tinc: UDP, length 52
11:49:32.015536 IP 192.168.0.101.tinc > cassiopeia.tinc: UDP, length 52
===============================
My interpretation of data above
1. Thinkpad pings from behind the NAT
Both hosts with real ips hear 'behind-the-NAT' host loud and clear.
Both reply but none of those replies are heard by origin.
Possible causes:
Packets are filtered by origin firewall
Packets are filtered by wi-fi router
Packets are filtered by ISP
Mitigation:
I've lowered all firewalls and tried direct cable connection to no avail
Questions:
Why connection was sucessfully established in the first place?
How can I confirm or rule out ISP tampering with my data?
2. Hosts with real ips talking to each other
There are no attempts whatsoever to send packets to each other.
Ping sender sends ARP requests to 'behind-the-NAT' host instead.
It's well established above that said host does not hear a thing.
Possible causes:
Outgoing packets to another host are filtered by own firewall
Mitigation:
Firewalls were all lowered down. Nothing changed.
Questions:
I'm lost, this routing approach doesn't make any sense whatsoever.
3. Vaio talks to cassiopeia
For some reason there are zero issues here. Everything works like it did before.
Questions:
This is the most baffling datum of them all. If there is filtering somewhere
between my wi-fi router and cassiopeia it should've affected vaio too.
What conclusion can I draw from this?
Why vaio has no problems whatsoever in talking to the cassiopeia when everyone
else do?
4. Vaio talks to vault13
There are no attempts to reach vault13, it talks to cassiopeia instead,
and as established above, cassiopeia <=> vault13 exchange not happening either
Possible causes:
No ideas. Physical network situation doesn't even matter here, because nothing
is being sent in the first place.
Question:
Why there are no attempts to send packages to vault13, to which a direct
connection is established?
================
Closing thoughts
I've tried:
- lowering all firewalls
- powering off all hosts one by one
- reboots (implied by above)
- syncing clocks
None of the above had any effect whatsoever.
All of the failures appear to have happened simultaneously, which lands credit
to a single cause, but I
don't have a clue what it could possibly be.
I'll take all the help I can get.
========================================
Addendum: tinc debug data from all hosts
vault13:
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: Statistics for Linux tun/tap
device (tap mode) /dev/net/tun:
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: total bytes in: 1854087
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: total bytes out: 948571
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: Nodes:
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: cassiopeia at
cassiopeia.ip.address port 655 cipher 91 digest 64 maclength 4 compression 0
options 8 status 001a nexthop cassiopeia via cassiopeia pmtu 1360 (min 0 max
1518)
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: thinkpad at wi-fi.ip.address port
655 cipher 91 digest 64 maclength 4 compression 0 options 9 status 003a nexthop
thinkpad via vault13 pmtu 1360 (min 0 max 1518)
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: vaio at wi-fi.ip.address port 655
cipher 91 digest 64 maclength 4 compression 0 options 9 status 0038 nexthop
vaio via vault13 pmtu 1360 (min 0 max 1518)
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: vault13 at MYSELF cipher 0 digest
0 maclength 0 compression 0 options 8 status 0018 nexthop vault13 via vault13
pmtu 1518 (min 0 max 1518)
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: End of nodes.
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: Edges:
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: cassiopeia to thinkpad at
wi-fi.ip.address port 655 options 9 weight 109
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: cassiopeia to vaio at
wi-fi.ip.address port 655 options 9 weight 171
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: cassiopeia to vault13 at
vault13.ip.address port 655 options 8 weight 408
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: thinkpad to cassiopeia at
cassiopeia.ip.address port 655 options 9 weight 109
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: thinkpad to vault13 at
vault13.ip.address port 655 options 9 weight 404
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: vaio to cassiopeia at
cassiopeia.ip.address port 655 options 9 weight 171
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: vaio to vault13 at
vault13.ip.address port 655 options 9 weight 468
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: vault13 to cassiopeia at
cassiopeia.ip.address port 655 options 8 weight 408
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: vault13 to thinkpad at
wi-fi.ip.address port 655 options 9 weight 404
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: vault13 to vaio at
wi-fi.ip.address port 655 options 9 weight 468
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: End of edges.
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: Subnet list:
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: 46:ec:a:4f:e7:f5#10 owner
cassiopeia
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: 66:5a:29:ba:3c:67#10 owner vault13
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: 82:2:30:b1:30:95#10 owner vaio
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: 82:10:f2:65:1a:8d#10 owner
thinkpad
Jun 2 05:09:42 vault13 tinc.vaultnet[2227]: End of subnet list.
cassiopeia:
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: Statistics for Linux tun/tap
device (tap mode) /dev/net/tun:
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: total bytes in: 224083
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: total bytes out: 483499
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: Nodes:
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: cassiopeia at MYSELF cipher 0
digest 0 maclength 0 compression 0 options 8 status 0018 nexthop cassiopeia via
cassiopeia pmtu 1518 (min 0 max 1518)
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: thinkpad at wi-fi.ip.address
port 655 cipher 91 digest 64 maclength 4 compression 0 options 9 status 003a
nexthop thinkpad via cassiopeia pmtu 1360 (min 0 max 1518)
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: vaio at wi-fi.ip.address port
600 cipher 91 digest 64 maclength 4 compression 0 options 9 status 003a nexthop
vaio via cassiopeia pmtu 1360 (min 0 max 1518)
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: vault13 at vault13.ip.address
port 655 cipher 91 digest 64 maclength 4 compression 0 options 8 status 001a
nexthop vault13 via vault13 pmtu 1360 (min 0 max 1518)
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: End of nodes.
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: Edges:
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: cassiopeia to thinkpad at
wi-fi.ip.address port 655 options 9 weight 109
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: cassiopeia to vaio at
wi-fi.ip.address port 655 options 9 weight 171
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: cassiopeia to vault13 at
vault13.ip.address port 655 options 8 weight 408
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: thinkpad to cassiopeia at
cassiopeia.ip.address port 655 options 9 weight 109
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: thinkpad to vault13 at
vault13.ip.address port 655 options 9 weight 404
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: vaio to cassiopeia at
cassiopeia.ip.address port 655 options 9 weight 171
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: vaio to vault13 at
vault13.ip.address port 655 options 9 weight 468
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: vault13 to cassiopeia at
cassiopeia.ip.address port 655 options 8 weight 408
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: vault13 to thinkpad at
wi-fi.ip.address port 655 options 9 weight 404
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: vault13 to vaio at
wi-fi.ip.address port 655 options 9 weight 468
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: End of edges.
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: Subnet list:
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: 46:ec:a:4f:e7:f5#10 owner
cassiopeia
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: 66:5a:29:ba:3c:67#10 owner
vault13
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: 82:2:30:b1:30:95#10 owner vaio
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: 82:10:f2:65:1a:8d#10 owner
thinkpad
Jun 2 12:10:49 cassiopeia tinc.vaultnet[14639]: End of subnet list.
thinkpad:
un 2 12:11:51 thinkpad tinc.vaultnet[20886]: Statistics for Linux tun/tap
device (tap mode) /dev/net/tun:
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: total bytes in: 985208
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: total bytes out: 280
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: Nodes:
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: cassiopeia at
cassiopeia.ip.address port 655 cipher 91 digest 64 maclength 4 compression 0
options 9 status 003a nexthop cassiopeia via thinkpad pmtu 1360 (min 0 max 1518)
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: thinkpad at MYSELF cipher 0
digest 0 maclength 0 compression 0 options 9 status 0018 nexthop thinkpad via
thinkpad pmtu 1518 (min 0 max 1518)
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: vaio at wi-fi.ip.address port
655 cipher 0 digest 0 maclength 0 compression 0 options 9 status 0038 nexthop
cassiopeia via thinkpad pmtu 1518 (min 0 max 1518)
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: vault13 at vault13.ip.address
port 655 cipher 91 digest 64 maclength 4 compression 0 options 9 status 003a
nexthop vault13 via thinkpad pmtu 1360 (min 0 max 1518)
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: End of nodes.
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: Edges:
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: cassiopeia to thinkpad at
wi-fi.ip.address port 655 options 9 weight 109
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: cassiopeia to vaio at
wi-fi.ip.address port 655 options 9 weight 171
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: cassiopeia to vault13 at
vault13.ip.address port 655 options 8 weight 408
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: thinkpad to cassiopeia at
cassiopeia.ip.address port 655 options 9 weight 109
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: thinkpad to vault13 at
vault13.ip.address port 655 options 9 weight 404
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: vaio to cassiopeia at
cassiopeia.ip.address port 655 options 9 weight 171
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: vaio to vault13 at
vault13.ip.address port 655 options 9 weight 468
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: vault13 to cassiopeia at
cassiopeia.ip.address port 655 options 8 weight 408
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: vault13 to thinkpad at
wi-fi.ip.address port 655 options 9 weight 404
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: vault13 to vaio at
wi-fi.ip.address port 655 options 9 weight 468
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: End of edges.
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: Subnet list:
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: 46:ec:a:4f:e7:f5#10 owner
cassiopeia
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: 66:5a:29:ba:3c:67#10 owner
vault13
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: 82:2:30:b1:30:95#10 owner vaio
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: 82:10:f2:65:1a:8d#10 owner
thinkpad
Jun 2 12:11:51 thinkpad tinc.vaultnet[20886]: End of subnet list.
vaio:
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: Statistics for Linux tun/tap device
(tap mode) /dev/net/tun:
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: total bytes in: 213481
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: total bytes out: 602047
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: Nodes:
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: cassiopeia at cassiopeia.ip.address
port 655 cipher 91 digest 64 maclength 4 compression 0 options 9 status 003a
nexthop cassiopeia via vaio pmtu 1360 (min 0 max 1518)
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: thinkpad at wi-fi.ip.address port
655 cipher 0 digest 0 maclength 0 compression 0 options 9 status 0038 nexthop
cassiopeia via vaio pmtu 1518 (min 0 max 1518)
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: vaio at MYSELF cipher 0 digest 0
maclength 0 compression 0 options 9 status 0018 nexthop vaio via vaio pmtu 1518
(min 0 max 1518)
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: vault13 at vault13.ip.address port
655 cipher 0 digest 0 maclength 0 compression 0 options 9 status 0038 nexthop
vault13 via vaio pmtu 1360 (min 0 max 1518)
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: End of nodes.
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: Edges:
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: cassiopeia to thinkpad at
wi-fi.ip.address port 655 options 9 weight 109
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: cassiopeia to vaio at
wi-fi.ip.address port 655 options 9 weight 171
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: cassiopeia to vault13 at
vault13.ip.address port 655 options 8 weight 408
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: thinkpad to cassiopeia at
cassiopeia.ip.address port 655 options 9 weight 109
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: thinkpad to vault13 at
vault13.ip.address port 655 options 9 weight 404
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: vaio to cassiopeia at
cassiopeia.ip.address port 655 options 9 weight 171
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: vaio to vault13 at
vault13.ip.address port 655 options 9 weight 468
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: vault13 to cassiopeia at
cassiopeia.ip.address port 655 options 8 weight 408
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: vault13 to thinkpad at
wi-fi.ip.address port 655 options 9 weight 404
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: vault13 to vaio at wi-fi.ip.address
port 655 options 9 weight 468
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: End of edges.
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: Subnet list:
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: 46:ec:a:4f:e7:f5#10 owner cassiopeia
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: 66:5a:29:ba:3c:67#10 owner vault13
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: 82:2:30:b1:30:95#10 owner vaio
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: 82:10:f2:65:1a:8d#10 owner thinkpad
Jun 2 12:12:24 vaio tinc.vaultnet[3182]: End of subnet list.
--- End Message ---
_______________________________________________
tinc mailing list
[email protected]
https://www.tinc-vpn.org/cgi-bin/mailman/listinfo/tinc
