Bug#815793: IPv6 code ignores unsolicited router advertisements
I would like to suggest closing this issue. I have never seen it again. Greetings Marc -- - Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany| lose things."Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
Bug#815793: IPv6 code ignores unsolicited router advertisements
Hi Martin, On Fri, Jul 15, 2016 at 09:04:59PM +0200, Marc Haber wrote: > On Wed, Jul 06, 2016 at 08:57:43AM +0200, Martin Pitt wrote: > > while we reverted the change in 229, we don't want to carry the > > reversion forever. Also, some problems were fixed already in 230, like > > [1]. I forwarded this upstream to > > https://github.com/systemd/systemd/issues/3663, and will now play > > relay :-) > > > > If this is hardware specific, can you please try this with 230 with > > the patch reverted? I build packages for amd64 here: > > https://people.debian.org/~mpitt/tmp/systemd-userspace-ndisc/ > > (there's also a Packages.gz so this can be used as an apt deb line) > > There were a lot of changes to the RA handling in 230. > > Unfortunately the system I have been seeing this on is a Banana Pi, > thus armhf, and I cannot run your test packages on this system. This > bug does not, however, show itself on a reference system running amd64 > with 230-5pitti1. I regret to inform you that we have a heisenbug here. When reconfiguring and restarting radvd, systemd sometimes decides to act on the new announcement, and sometimes doesn't. For example, after receiving this announcement: 13:40:23.075889 IP6 (flowlabel 0x4717d, hlim 255, next-header ICMPv6 (58) payload length: 240) fe80::1 > ff02::1: [icmp6 sum ok] ICMP6, hop limit 64, Flags [none], pref medium, router lifetime 1800s, reachable time 0s, retrans time 0s prefix info option (3), length 32 (4): 2a01:db8:0::3282::/64, Flags [onlink, auto], valid time 86400s, pref. time 14400s route info option (24), length 24 (3): ::/0, pref=high, lifetime=1800s route info option (24), length 24 (3): 2000::/3, pref=high, lifetime=1800s route info option (24), length 24 (3): 2a01:db8:0::3280::/60, pref=high, lifetime=1800s route info option (24), length 24 (3): 2a01:db8:0::32b0::/60, pref=high, lifetime=1800s rdnss option (25), length 40 (5): lifetime 600s, addr: 2a01:db8:0::3281::35:100 addr: 2a01:db8:0::328e::35:100 dnssl option (31), length 48 (6): lifetime 600s, domain(s): zugschlus.de. ka51.zugschlus.de. source link-address option (1), length 8 (1): 7e:79:61:31:55:28 systemd didn't configure an SLAAC IPv6 address on the interface. I guess that this depends on whether and how many other IP addresses are on the interface. It seems to work more reliably if the interface doesn't already have an address from this prefix. If this is the case, it is another proof of the gross misunderstanding of IPv6 that the person writing this piece of the code has. Greetings Marc -- - Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany| lose things."Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
Bug#815793: IPv6 code ignores unsolicited router advertisements
Hi Martin, On Wed, Jul 06, 2016 at 08:57:43AM +0200, Martin Pitt wrote: > while we reverted the change in 229, we don't want to carry the > reversion forever. Also, some problems were fixed already in 230, like > [1]. I forwarded this upstream to > https://github.com/systemd/systemd/issues/3663, and will now play > relay :-) > > If this is hardware specific, can you please try this with 230 with > the patch reverted? I build packages for amd64 here: > https://people.debian.org/~mpitt/tmp/systemd-userspace-ndisc/ > (there's also a Packages.gz so this can be used as an apt deb line) > There were a lot of changes to the RA handling in 230. Unfortunately the system I have been seeing this on is a Banana Pi, thus armhf, and I cannot run your test packages on this system. This bug does not, however, show itself on a reference system running amd64 with 230-5pitti1. I have found another showstopper IPv6 bug in 230-5pitti1 and have duly reported it. Greetings Marc -- - Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany| lose things."Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
Bug#815793: IPv6 code ignores unsolicited router advertisements
On Wed, Jul 06, 2016 at 11:55:19AM +0200, Martin Pitt wrote: > Marc Haber [2016-07-06 11:52 +0200]: > > On Wed, Jul 06, 2016 at 11:49:17AM +0200, Martin Pitt wrote: > > > This looks the same for 229, 230/upstream and 230/sid (with the > > > reverted patch). Thus I cannot confirm that networkd ignores these > > > unsolicited router advertisements. > > > > Can you, just for reference, try with systemd 229-1 > > I did (that was the "229" above). Ah, I thought you meant 229 with the ipv6 code disabled. I will try to reproduce things. Greetings Marc -- - Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany| lose things."Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
Bug#815793: IPv6 code ignores unsolicited router advertisements
Hello Marc, Marc Haber [2016-07-06 11:52 +0200]: > On Wed, Jul 06, 2016 at 11:49:17AM +0200, Martin Pitt wrote: > > This looks the same for 229, 230/upstream and 230/sid (with the > > reverted patch). Thus I cannot confirm that networkd ignores these > > unsolicited router advertisements. > > Can you, just for reference, try with systemd 229-1 I did (that was the "229" above). Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
Bug#815793: IPv6 code ignores unsolicited router advertisements
On Wed, Jul 06, 2016 at 11:49:17AM +0200, Martin Pitt wrote: > This looks the same for 229, 230/upstream and 230/sid (with the > reverted patch). Thus I cannot confirm that networkd ignores these > unsolicited router advertisements. Can you, just for reference, try with systemd 229-1, just to see wheteher your setup triggers the bug in the same software version that I reported it with? > So re-trying with my test packages and getting a networkd debug log > would be appreciated. Thanks! Will try, but probably only later this or even next week. Greetings Marc -- - Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany| lose things."Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
Bug#815793: IPv6 code ignores unsolicited router advertisements
Hello Marc,
Marc Haber [2016-07-06 9:42 +0200]:
> I think this particular issue can be triggered by:
>
> - set up a router running radvd
> - start up a client with networkd and systemd IPv6 implementation
> - manually remove any IPv6 address and IPv6 route from the client
> - restart the radvd on the router
>
> The client should pick up the prefix and route that the newly started
> radvd announces. At least this would be my first test.
I did that now:
| ip link add name enc type veth peer name ens
| ip a add 2600::1/64 dev ens
| ip link set ens up
|
| cat < /tmp/ra.conf
| interface ens
| {
|AdvSendAdvert on;
|prefix 2600::1/64
|{};
| };
| EOF
|
| radvd -C /tmp/ra.conf -n -m stderr -d2
Other terminal:
| mkdir -p /run/systemd/network
| cat < /run/systemd/network/enc.network
| [Match]
| Name=enc
|
| [Network]
| IPv6AcceptRouterAdvertisements=yes
| EOF
|
| SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd
I also ran "tcpdump -i ens" in parallel, as before. This does the
solicitation and picks up radvd's RA and I get some random 2600:*
address:
| Disc CLIENT: Received Router Advertisement: flags none preference medium
lifetime 1800 sec
| NDisc CLIENT: Update prefix 2600:::::::0001/64
lifetime 86400 expires in 1d
| ens: Updating address: 2600::7852:45ff:fed7:9ab4/64 (valid for 1d)
I now killed radvd, then "ip a del 2600:[..] dev enc", and restart
radvd. And once again networkd picks this up and adds an IPv6 address.
This looks the same for 229, 230/upstream and 230/sid (with the
reverted patch). Thus I cannot confirm that networkd ignores these
unsolicited router advertisements.
So re-trying with my test packages and getting a networkd debug log
would be appreciated. Thanks!
Martin
--
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
Bug#815793: IPv6 code ignores unsolicited router advertisements
On Wed, Jul 06, 2016 at 09:29:35AM +0200, Martin Pitt wrote: > Marc Haber [2016-07-06 9:03 +0200]: > > > Do you have some instructions how this can be reproduced? It sounds > > > like it should be possible to test this using a veth pair, possibly a > > > dnsmasq on one end, and some ip commands. Then I'd like to write a > > > test case for [2]. > > > > > > If this is hardware specific, can you please try this with 230 with > > > the patch reverted? I build packages for amd64 here: > > > https://people.debian.org/~mpitt/tmp/systemd-userspace-ndisc/ > > > (there's also a Packages.gz so this can be used as an apt deb line) > > > There were a lot of changes to the RA handling in 230. > > > > I will try this, but since my current project is in a critical phase I > > do not have too much time. I am confident that I can produce results > > during july, but most probably not next weekend. I will keep you posted. > > Thanks. Testing the above packages will indeed take some time, but I > wondered if you could give me a hint whether this can be reproduced > with something like "set up a normal RA server/IPv6 networkd client" > (I know how to do that of course) and then change foo/run blah to > trigger this". I think this particular issue can be triggered by: - set up a router running radvd - start up a client with networkd and systemd IPv6 implementation - manually remove any IPv6 address and IPv6 route from the client - restart the radvd on the router The client should pick up the prefix and route that the newly started radvd announces. At least this would be my first test. I _think_ that I stumbled upon this issue because the systemd IPv6 code sent out its _single_ router solicitation so early that the network link was not yet up (thus, no IPv6 on the client), and then proceeded to ignore the regular route updates sent out by the radvd because it didn't ask for them. Ignoring unsolicited announcements is state of the art for ARP and DHCP to protect a system's caches against poisioning attacks, which is probably the reason why this was implemented in systemd IPv6 code as well. But, IPv6 works differently and it is necessary for network functionality to listen and to act even on unsolicited router announcements. At least this is how I explained to myself why the code was written this way. Too bad they didn't test it in reality but instead released it. Greetings Marc -- - Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany| lose things."Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
Bug#815793: IPv6 code ignores unsolicited router advertisements
Hey Marc, Marc Haber [2016-07-06 9:03 +0200]: > > Do you have some instructions how this can be reproduced? It sounds > > like it should be possible to test this using a veth pair, possibly a > > dnsmasq on one end, and some ip commands. Then I'd like to write a > > test case for [2]. > > > > If this is hardware specific, can you please try this with 230 with > > the patch reverted? I build packages for amd64 here: > > https://people.debian.org/~mpitt/tmp/systemd-userspace-ndisc/ > > (there's also a Packages.gz so this can be used as an apt deb line) > > There were a lot of changes to the RA handling in 230. > > I will try this, but since my current project is in a critical phase I > do not have too much time. I am confident that I can produce results > during july, but most probably not next weekend. I will keep you posted. Thanks. Testing the above packages will indeed take some time, but I wondered if you could give me a hint whether this can be reproduced with something like "set up a normal RA server/IPv6 networkd client" (I know how to do that of course) and then change foo/run blah to trigger this". Thanks, Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
Bug#815793: IPv6 code ignores unsolicited router advertisements
Hi Martin, On Wed, Jul 06, 2016 at 08:57:43AM +0200, Martin Pitt wrote: > while we reverted the change in 229, we don't want to carry the > reversion forever. Also, some problems were fixed already in 230, like > [1]. I forwarded this upstream to > https://github.com/systemd/systemd/issues/3663, and will now play > relay :-) Thanks for your consideration. > Do you have some instructions how this can be reproduced? It sounds > like it should be possible to test this using a veth pair, possibly a > dnsmasq on one end, and some ip commands. Then I'd like to write a > test case for [2]. > > If this is hardware specific, can you please try this with 230 with > the patch reverted? I build packages for amd64 here: > https://people.debian.org/~mpitt/tmp/systemd-userspace-ndisc/ > (there's also a Packages.gz so this can be used as an apt deb line) > There were a lot of changes to the RA handling in 230. I will try this, but since my current project is in a critical phase I do not have too much time. I am confident that I can produce results during july, but most probably not next weekend. I will keep you posted. I recently bought new hardware for virtualization and will use building the new system as test lab, so your request comes just at the right time since I can now experiment without breaking productive systems. Greetings Marc -- - Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany| lose things."Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
Bug#815793: IPv6 code ignores unsolicited router advertisements
Control: reopen -1 Control: tag -1 moreinfo Hello Marc, while we reverted the change in 229, we don't want to carry the reversion forever. Also, some problems were fixed already in 230, like [1]. I forwarded this upstream to https://github.com/systemd/systemd/issues/3663, and will now play relay :-) Marc Haber [2016-02-24 15:01 +0100]: > systemd 229 seems to ignore unsolicited router advertisements. This > breaks, for example, prefix changes. And it prevents the self-healing > characteristics of lost router solicitations. > > tcpdump logs this: > 14:41:40.062484 7e:79:61:31:55:28 > 33:33:00:00:00:01, ethertype IPv6 > (0x86dd), length 294: fe80::1 > ff02::1: ICMP6, router advertisement, length > 240 Do you have some instructions how this can be reproduced? It sounds like it should be possible to test this using a veth pair, possibly a dnsmasq on one end, and some ip commands. Then I'd like to write a test case for [2]. If this is hardware specific, can you please try this with 230 with the patch reverted? I build packages for amd64 here: https://people.debian.org/~mpitt/tmp/systemd-userspace-ndisc/ (there's also a Packages.gz so this can be used as an apt deb line) There were a lot of changes to the RA handling in 230. If it still happens, can you please run it in debugging mode with systemd-analyze set-log-level debug systemctl restart systemd-networkd then reproduce the bug, and finally systemd-analyze set-log-level info journalctl -u systemd-networkd.service > /tmp/networkd.log and append the log file here? Thanks! [1] https://github.com/systemd/systemd/issues/2572 [2] https://github.com/systemd/systemd/blob/master/test/networkd-test.py -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
Bug#815793: IPv6 code ignores unsolicited router advertisements
Hi Martin, On Sun, Feb 28, 2016 at 10:20:22PM +0100, Martin Pitt wrote: > Marc Haber [2016-02-24 15:01 +0100]: > > systemd 229 seems to ignore unsolicited router advertisements. This > > breaks, for example, prefix changes. And it prevents the self-healing > > characteristics of lost router solicitations. > > Can you please report this and also #815884 upstream? I don't know > much about IPv6 routing I'm afraid, so I think you are in a better > position to explain/discuss this with Tom. Some regressions are > already covered by https://github.com/systemd/systemd/issues/2572 > but not these two. I think that it would not be productive if I tried communicating with systemd upstream. We both consider each other toxic, and I'd rather not mess with my sanity trying to be nice to a . I lost my temper on systemd-devel in december and cannot write there any more due to policy decision of the list owner. I doubt that a bug report filed by me would get any helpful upstream attention. So, I regret that in this case my answer is thanks, but no thanks. Greetings Marc -- - Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany| lose things."Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
Bug#815793: IPv6 code ignores unsolicited router advertisements
Hello Marc, Marc Haber [2016-02-24 15:01 +0100]: > systemd 229 seems to ignore unsolicited router advertisements. This > breaks, for example, prefix changes. And it prevents the self-healing > characteristics of lost router solicitations. Can you please report this and also #815884 upstream? I don't know much about IPv6 routing I'm afraid, so I think you are in a better position to explain/discuss this with Tom. Some regressions are already covered by https://github.com/systemd/systemd/issues/2572 but not these two. In the meantime, I reverted the networkd handling of RA to go back to the pre-229 behaviour. Many thanks! Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
