Bug#923249: [Pkg-libvirt-maintainers] Bug#923249: libvirt0: libvirt sets disable_ipv6 on bridge, entirely breaking internal IPv6 networking

[Ralf Jung]

Hi,

> https://bugzilla.redhat.com/show_bug.cgi?id=1680681 indicates it's a
> kernel issue. Can you add a link to the debian bug you filed so I can
> close this issue but mark the other one as affecting libvirt?
> Cheers
>  -- Guido

That Debian kernel bug is at
.

; Ralf

Bug#923249: [Pkg-libvirt-maintainers] Bug#923249: libvirt0: libvirt sets disable_ipv6 on bridge, entirely breaking internal IPv6 networking

[Guido Günther]

Hi,
On Thu, Feb 28, 2019 at 07:49:47AM +0100, Guido Günther wrote:
> Hi,
> On Wed, Feb 27, 2019 at 10:08:24PM +0100, Ralf Jung wrote:
> > Btw, I reported this upstream at
> > , and made some headway
> > debugging things there.
> 
> Cool, thanks!

https://bugzilla.redhat.com/show_bug.cgi?id=1680681 indicates it's a
kernel issue. Can you add a link to the debian bug you filed so I can
close this issue but mark the other one as affecting libvirt?
Cheers
 -- Guido

>  -- Guido

Bug#923249: [Pkg-libvirt-maintainers] Bug#923249: libvirt0: libvirt sets disable_ipv6 on bridge, entirely breaking internal IPv6 networking

[Guido Günther]

Hi,
On Wed, Feb 27, 2019 at 10:08:24PM +0100, Ralf Jung wrote:
> Btw, I reported this upstream at
> , and made some headway
> debugging things there.

Cool, thanks!
 -- Guido

Bug#923249: [Pkg-libvirt-maintainers] Bug#923249: libvirt0: libvirt sets disable_ipv6 on bridge, entirely breaking internal IPv6 networking

[Ralf Jung]

Btw, I reported this upstream at
, and made some headway
debugging things there.

; Ralf

Bug#923249: [Pkg-libvirt-maintainers] Bug#923249: libvirt0: libvirt sets disable_ipv6 on bridge, entirely breaking internal IPv6 networking

[Ralf Jung]

Hi,

>> It turns out that setting disable_ipv6=0 is not enough to regain working 
>> IPv6:
>> Now the RAs from the router arrive at the client, but when the client sends a
>> neighbor discovery, it never makes its way to the router, so the client 
>> cannot
>> actually communicate via IPv6. I can see the ND packages on vnet2 on the 
>> host,
>> but not on virbr1 (which vnet2 is attached to). Again this is something that
>> worked just fine before the update.
>>
>> I have no idea what is preventing the ND packages to be forwarded by the 
>> bridge.
>> I always thought bridges are just switches operating on layer 2, but it seems
>> that's as from the truth...
> 
> Can you check if downgrading the kernel helps?

I downgraded back to 4.18, which definitely used to work. No change in behavior.
Seems like it was a libvirt/firewalld upgrade that broke it.

; Ralf

Bug#923249: [Pkg-libvirt-maintainers] Bug#923249: libvirt0: libvirt sets disable_ipv6 on bridge, entirely breaking internal IPv6 networking

[Guido Günther]

Hi,
On Mon, Feb 25, 2019 at 04:53:27PM +0100, Ralf Jung wrote:
> Hi again,
> 
> It turns out that setting disable_ipv6=0 is not enough to regain working IPv6:
> Now the RAs from the router arrive at the client, but when the client sends a
> neighbor discovery, it never makes its way to the router, so the client cannot
> actually communicate via IPv6. I can see the ND packages on vnet2 on the host,
> but not on virbr1 (which vnet2 is attached to). Again this is something that
> worked just fine before the update.
> 
> I have no idea what is preventing the ND packages to be forwarded by the 
> bridge.
> I always thought bridges are just switches operating on layer 2, but it seems
> that's as from the truth...

Can you check if downgrading the kernel helps?
 -- Guido

Bug#923249: [Pkg-libvirt-maintainers] Bug#923249: libvirt0: libvirt sets disable_ipv6 on bridge, entirely breaking internal IPv6 networking

[Ralf Jung]

Hi again,

It turns out that setting disable_ipv6=0 is not enough to regain working IPv6:
Now the RAs from the router arrive at the client, but when the client sends a
neighbor discovery, it never makes its way to the router, so the client cannot
actually communicate via IPv6. I can see the ND packages on vnet2 on the host,
but not on virbr1 (which vnet2 is attached to). Again this is something that
worked just fine before the update.

I have no idea what is preventing the ND packages to be forwarded by the bridge.
I always thought bridges are just switches operating on layer 2, but it seems
that's as from the truth...

Kind regards,
Ralf

Bug#923249: [Pkg-libvirt-maintainers] Bug#923249: libvirt0: libvirt sets disable_ipv6 on bridge, entirely breaking internal IPv6 networking

[Ralf Jung]

Hi Guido,

thanks for the quick reply!

>> After a recent upgrade, IPv6 communication between a virtual router and 
>> another
>> virtual client over an isolated network stopped working. I am seeing the 
>> rotuer
>> advertisments sent by the router on vnet0, which is attached to the bridge
>> virbr1, but when I capture packages on the bridge, the IPv6 traffic is gone. 
>>  It
>> just took me several hours of debugging to realize that the reason for this 
>> is
>> that /proc/sys/net/ipv6/conf/virbr1/disable_ipv6 is set to 1.  After setting 
>> it
>> to 0, IPv6 is working as expected now.
>>
>> This is a regression, IPv6 used to work between virtual clients just fine
>> without having to manually fiddle with the network configuration.
> 
> I'm not near a ipv6 setup atm but according to the git logs nothing
> changed in that area for quite some time. Please indicate which version
> you updated from so it's easier to check for related changes and also
> provide details about your setup (preferably network XML and domain XML).

I updated from 4.10.0-2 to 5.0.0-1.

Looking at the code in bridge_driver.c, I also came to the conclusion that
nothing changed, and that setting disable_ipv6 like this is intended behavior --
it happens whenever the network has no host IPv6 address.  The docs say that
guest-to-guest IPv6 communication can be enabled with the `ipv6` attribute, but
that attribute has no bearing on whether `disable_ipv6` gets set. It only
controls some firwall stuff. Maybe disable_ipv6 was always set but it somehow
used to not kill the entire IPv6 traffic on the bridge? A kernel update happened
together with all the other updates (from 4.19.12-1 to 4.19.16-1).

The network config now is (after adding the `ipv6` attribute, which however made
no difference):

> 
>   ffnet
>   cfd2c92a-db77-4b27-ad78-a8a81ace32b6
>   
>   
>   
> 

The part where the virtual router gets attached is

> 
>   
>   
>   
>queues='5'>
>  mrg_rxbuf='off'/>
> 
>   
>function='0x0'/>
> 

And for the virtual client

> 
>   
>   
>   
>queues='5'>
>  mrg_rxbuf='off'/>
> 
>   
>function='0x0'/>
> 


> There were some ipv6 related changes with firewalld though which might be 
> worth
> investigating.

firewalld got updated from 0.6.3-4 to 0.6.3-5 at the same time.
I have set `FirewallBackend=iptables` some time ago because the default
(`nftables`) broke libvirt.

; Ralf

Bug#923249: [Pkg-libvirt-maintainers] Bug#923249: libvirt0: libvirt sets disable_ipv6 on bridge, entirely breaking internal IPv6 networking

[Guido Günther]

Hi,
On Mon, Feb 25, 2019 at 02:25:57PM +0100, Ralf Jung wrote:
> Package: libvirt0
> Version: 5.0.0-1
> Severity: grave
> Justification: renders package unusable
> 
> Dear Maintainer,
> 
> After a recent upgrade, IPv6 communication between a virtual router and 
> another
> virtual client over an isolated network stopped working. I am seeing the 
> rotuer
> advertisments sent by the router on vnet0, which is attached to the bridge
> virbr1, but when I capture packages on the bridge, the IPv6 traffic is gone.  
> It
> just took me several hours of debugging to realize that the reason for this is
> that /proc/sys/net/ipv6/conf/virbr1/disable_ipv6 is set to 1.  After setting 
> it
> to 0, IPv6 is working as expected now.
> 
> This is a regression, IPv6 used to work between virtual clients just fine
> without having to manually fiddle with the network configuration.

I'm not near a ipv6 setup atm but according to the git logs nothing
changed in that area for quite some time. Please indicate which version
you updated from so it's easier to check for related changes and also
provide details about your setup (preferably network XML and domain XML).

There were some ipv6 related changes with firewalld though which might be worth
investigating.
 -- Guido