Re: [Lxc-users] lxcbr0 MAC addr issue
On 06/05/2013 02:40 PM, Serge Hallyn wrote: Now my question, could not lxc (at boot) setup a fixed MAC addr for the host port? Yeah, given how bad this was for libvirt/qemu I'm surprised I've not seen this happen in lxc - but I haven't, and noone else has reported it. Since you mention lxcbr0, I assume you're using ubuntu? Until we Yes 12.04 and 13.04 (somehow - netcf?) implement a distro-agnostic lxc bridge upstream, this is not an upstream issue, but an ubuntu package issue. Have you actually seen a hang as a result of this? If so, then please Yes but not a permanent one. Just until the arp cache times out. But long enough to fail my test case... For testing purposes I have a set of lxc systems that communicate with each other as well as the host. The problem occurs when I reboot an lxc system and another lxc system tries to communicate with the host. The host might have changed mac addr and ... open a bug at https://bugs.launchpad.net/ubuntu/+source/lxc/+filebug as it should be trivially fixable in /etc/init/lxc-net.conf. OK will do. /Hans -serge -- How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views of enterprise services 3. A single system of record for all IT processes http://p.sf.net/sfu/servicenow-d2d-j ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] lxcbr0 MAC addr issue
On 06/05/2013 06:26 PM, Serge Hallyn wrote: BUT I just did some testing, and even as I watch lxcbr0's addr go down when starting a new container, my ssh to the container which had the higher macaddr doesn't hiccough. the problem appears in the other direction, container to host /Hans -- How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views of enterprise services 3. A single system of record for all IT processes http://p.sf.net/sfu/servicenow-d2d-j ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] lxcbr0 MAC addr issue
Quoting Hans Feldt (hans.fe...@ericsson.com): open a bug at https://bugs.launchpad.net/ubuntu/+source/lxc/+filebug as it should be trivially fixable in /etc/init/lxc-net.conf. OK will do. Cool, thanks, I'll track it there. -- How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views of enterprise services 3. A single system of record for all IT processes http://p.sf.net/sfu/servicenow-d2d-j ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] lxcbr0 MAC addr issue
Now my question, could not lxc (at boot) setup a fixed MAC addr for the host port? Yeah, given how bad this was for libvirt/qemu I'm surprised I've not seen this happen in lxc - but I haven't, and noone else has reported it. Since you mention lxcbr0, I assume you're using ubuntu? Until we (somehow - netcf?) implement a distro-agnostic lxc bridge upstream, this is not an upstream issue, but an ubuntu package issue. Have you actually seen a hang as a result of this? If so, then please open a bug at https://bugs.launchpad.net/ubuntu/+source/lxc/+filebug as it should be trivially fixable in /etc/init/lxc-net.conf. -serge -- How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views of enterprise services 3. A single system of record for all IT processes http://p.sf.net/sfu/servicenow-d2d-j ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] lxcbr0 MAC addr issue
On Wed, 2013-06-05 at 07:40 -0500, Serge Hallyn wrote: Now my question, could not lxc (at boot) setup a fixed MAC addr for the host port? Yeah, given how bad this was for libvirt/qemu I'm surprised I've not seen this happen in lxc - but I haven't, and noone else has reported it. Actually, this has come up on both the -devel list and here, the last time about a year and a half ago: On the -devel list... Subject: Set high byte of mac addresses for host veth devices to 0xfe It had a patch and referenced an open bug associated with it: random veth device MAC addresses cause bridge problems - ID: 3411497 https://sourceforge.net/tracker/index.php?func=detailaid=3411497group_id=163076atid=826303 Christian Seiler included a proposed patch with his original posting to the -devel list back in November of 2011 where he set the high order byte to FE for a private locally administered MAC and some discussion ensued. After a couple of bug fixes, it was Acked it on 01/03/2012. It looks like it was applied. Right around line 3109 of src/lxc/conf.c: static int setup_private_host_hw_addr(char *veth1) ... ifr.ifr_hwaddr.sa_data[0] = 0xfe; err = ioctl(sockfd, SIOCSIFHWADDR, ifr); close(sockfd); The bug is still open. Is this not working? What version did it go into? Someone should probably close that bug with an annotation. Looks to have been fixed in the 0.8.0 release. Seems to be working for me: ip addr ls ... 7: vethNoBHvN: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc pfifo_fast master lxcbr0 state UP qlen 1000 link/ether fe:57:fa:34:4c:35 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc57:faff:fe34:4c35/64 scope link valid_lft forever preferred_lft forever 9: vethFmslRq: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc pfifo_fast master lxcbr0 state UP qlen 1000 link/ether fe:a8:39:d6:c2:ee brd ff:ff:ff:ff:ff:ff inet6 fe80::fca8:39ff:fed6:c2ee/64 scope link valid_lft forever preferred_lft forever 11: vethhw7OZw: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc pfifo_fast master lxcbr0 state UP qlen 1000 link/ether fe:6b:8e:9d:f2:4f brd ff:ff:ff:ff:ff:ff inet6 fe80::fc6b:8eff:fe9d:f24f/64 scope link valid_lft forever preferred_lft forever So this shouldn't be happening if all of the host side MAC addresses are beginning with fe:. What version of the lxc tools is in play here? 0.7.5 dates back to August of 2011 and would have predated that fix. There's still a lot of distros with 0.7.5 still running around out there. It was also discussed here under these threads here in this forum from around the time time frame (about a year and a half ago): Subject: New LXC Creation Script: lxc-ubuntu-x Subject: seeing a network pause when starting and stopping LXCs - how do I stop this ? Yes, I had seen the bug show up. It causes the network to hang, but only for a half a minute or less as the ARP caches settle. If you have frequent starting and stopping of containers, it can be a PITA. My solution was to assign a high vendor field (top 3 bytes) using some safe vendor value that was above my host card vendor value. I seem to recall some discussion a few years ago about also trying to get an official MAC vendor code assigned to this project in one of the earlier discussions. I don't think that went anywhere and I'm having a problem finding it now. I do remember participating in that discussion as well. I seem to recall that it was decided not to do this even though some other projects such as VMware and, maybe, VirtualBox did chose this route. Since you mention lxcbr0, I assume you're using ubuntu? Until we (somehow - netcf?) implement a distro-agnostic lxc bridge upstream, this is not an upstream issue, but an ubuntu package issue. Have you actually seen a hang as a result of this? If so, then please open a bug at https://bugs.launchpad.net/ubuntu/+source/lxc/+filebug as it should be trivially fixable in /etc/init/lxc-net.conf. See above. Opened on 09/19/2011 and still open. You'll find I commented in the bug the same day. There's been other discussions in several forum, including including these. I believe it has also come up on the OpenVZ forum (and is where a lot of my earlier information originated) and has been mentioned in some of the kernel lists, probably the net list. There is a reason it works that way, mostly because the bridge has to have an address and it has to be an address on the bridge so the kernel can know what's on the bridge and what's foreign to the bridge in order to manage local routing. So, it has to pick something. But, if you delete an interface from the bridge and the bridge is using that address, it has to pick another. The rule the followed was to always choose the lowest. The only downside to assigning a static bridge address to the bridge is the chance of that associated interface ever being deleted from the bridge. I think you also loose that static state if certain bridge
Re: [Lxc-users] lxcbr0 MAC addr issue
Quoting Michael H. Warfield (m...@wittsend.com): On Wed, 2013-06-05 at 07:40 -0500, Serge Hallyn wrote: Now my question, could not lxc (at boot) setup a fixed MAC addr for the host port? Yeah, given how bad this was for libvirt/qemu I'm surprised I've not seen this happen in lxc - but I haven't, and noone else has reported it. Actually, this has come up on both the -devel list and here, the last time about a year and a half ago: On the -devel list... Subject: Set high byte of mac addresses for host veth devices to 0xfe It had a patch and referenced an open bug associated with it: random veth device MAC addresses cause bridge problems - ID: 3411497 https://sourceforge.net/tracker/index.php?func=detailaid=3411497group_id=163076atid=826303 Christian Seiler included a proposed patch with his original posting to the -devel list back in November of 2011 where he set the high order byte to FE for a private locally administered MAC and some discussion ensued. After a couple of bug fixes, it was Acked it on 01/03/2012. It looks like it was applied. Right around line 3109 of src/lxc/conf.c: static int setup_private_host_hw_addr(char *veth1) ... ifr.ifr_hwaddr.sa_data[0] = 0xfe; err = ioctl(sockfd, SIOCSIFHWADDR, ifr); close(sockfd); yes and it does this. The point is that lxcbr0 is not tied to any physical nic. So the first container you start, however high the macaddr is, lxcbr0 takes its mac. If the next container gets a lower macaddr, lxcbr0's macaddr drops. Right now I have: lxcbr0Link encap:Ethernet HWaddr fe:02:72:77:79:ff vethtdjU5K Link encap:Ethernet HWaddr fe:02:72:77:79:ff -serge -- How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views of enterprise services 3. A single system of record for all IT processes http://p.sf.net/sfu/servicenow-d2d-j ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] lxcbr0 MAC addr issue
On Wed, 2013-06-05 at 09:30 -0500, Serge Hallyn wrote: Quoting Michael H. Warfield (m...@wittsend.com): On Wed, 2013-06-05 at 07:40 -0500, Serge Hallyn wrote: Now my question, could not lxc (at boot) setup a fixed MAC addr for the host port? Yeah, given how bad this was for libvirt/qemu I'm surprised I've not seen this happen in lxc - but I haven't, and noone else has reported it. Actually, this has come up on both the -devel list and here, the last time about a year and a half ago: On the -devel list... Subject: Set high byte of mac addresses for host veth devices to 0xfe It had a patch and referenced an open bug associated with it: random veth device MAC addresses cause bridge problems - ID: 3411497 https://sourceforge.net/tracker/index.php?func=detailaid=3411497group_id=163076atid=826303 Christian Seiler included a proposed patch with his original posting to the -devel list back in November of 2011 where he set the high order byte to FE for a private locally administered MAC and some discussion ensued. After a couple of bug fixes, it was Acked it on 01/03/2012. It looks like it was applied. Right around line 3109 of src/lxc/conf.c: static int setup_private_host_hw_addr(char *veth1) ... ifr.ifr_hwaddr.sa_data[0] = 0xfe; err = ioctl(sockfd, SIOCSIFHWADDR, ifr); close(sockfd); yes and it does this. The point is that lxcbr0 is not tied to any physical nic. So the first container you start, however high the macaddr is, lxcbr0 takes its mac. If the next container gets a lower macaddr, lxcbr0's macaddr drops. Right now I have: lxcbr0Link encap:Ethernet HWaddr fe:02:72:77:79:ff vethtdjU5K Link encap:Ethernet HWaddr fe:02:72:77:79:ff Oh, yeah... I'm always using bridged mode with the host ethernet address on the bridge, so that's never a problem. You're right. The NAT mode is problematical there because it's not anchored to a physical interface. I think someone, at one point in those earlier threads, was suggesting that they were using a dummy interface or a dummy container as a place holder to lock the interface address for that reason. I just happen to have enough IPv4 addresses, personally, that I bridge everything and never really need to NAT anything where I can avoid it. :-P Of course, in a server environment, where you are hosting a farm of virtual servers, you would almost always want global public addresses on the servers, I would imagine. You're still going to have the problem that, if you shut down the container that the bridge is using for the address, the address is going to shift, static or not. The address of the bridge must be an address of an interface on the bridge or you are going to have routing problems. That was made clear in some discussion on some of the kernel mailing lists. How do you deal with that then? Do you designate a container that must never be shut down or the bridge hangs? You could load the dummy module and bridge a dummy interface to the bridge. That would guarantee a MAC address lower than the fe: private addresses of the bridge and would be cheaper than a dummy container. I've done that before a long time ago. (We should still close that old bug, however.) -serge -- How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views of enterprise services 3. A single system of record for all IT processes http://p.sf.net/sfu/servicenow-d2d-j Regards, Mike -- Michael H. Warfield (AI4NB) | (770) 985-6132 | m...@wittsend.com /\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/ NIC whois: MHW9 | An optimist believes we live in the best of all PGP Key: 0x674627FF| possible worlds. A pessimist is sure of it! signature.asc Description: This is a digitally signed message part -- How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views of enterprise services 3. A single system of record for all IT processes http://p.sf.net/sfu/servicenow-d2d-j___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] lxcbr0 MAC addr issue
On Wed, 2013-06-05 at 15:17 +, Jäkel, Guido wrote: yes and it does this. The point is that lxcbr0 is not tied to any physical nic. So the first container you start, however high the macaddr is, lxcbr0 takes its mac. If the next container gets a lower macaddr, lxcbr0's macaddr drops. This lxcbr0 is special to Ubuntu, right? And if not to a physical NIC, to what is this bridge connected to on the host? Not to the best of my knowledge. It should be a simple bridge. What do you get for this command? brctl show A bridge doesn -- How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views of enterprise services 3. A single system of record for all IT processes http://p.sf.net/sfu/servicenow-d2d-j ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users -- Michael H. Warfield (AI4NB) | (770) 985-6132 | m...@wittsend.com /\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/ NIC whois: MHW9 | An optimist believes we live in the best of all PGP Key: 0x674627FF| possible worlds. A pessimist is sure of it! signature.asc Description: This is a digitally signed message part -- How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views of enterprise services 3. A single system of record for all IT processes http://p.sf.net/sfu/servicenow-d2d-j___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] lxcbr0 MAC addr issue
Crap... Bumped the keyboard and this one got away from me prematurely. On Wed, 2013-06-05 at 11:23 -0400, Michael H. Warfield wrote: On Wed, 2013-06-05 at 15:17 +, Jäkel, Guido wrote: yes and it does this. The point is that lxcbr0 is not tied to any physical nic. So the first container you start, however high the macaddr is, lxcbr0 takes its mac. If the next container gets a lower macaddr, lxcbr0's macaddr drops. This lxcbr0 is special to Ubuntu, right? And if not to a physical NIC, to what is this bridge connected to on the host? Not to the best of my knowledge. It should be a simple bridge. What do you get for this command? brctl show A bridge doesn A bridge doesn't have to be attach to a device. A bridge is its own logical entity in the kernel to which you may attach devices. You can not attach a bridge to something else. You can only attach something else to the bridge. There's a difference. In the case of a NATing configuration, you set up a bridge (name it whatever you want) and attach the containers to it. Then you use the NAT modules to route between the bridge and the external interface while NATing the addresses. I use lxcbr0 on my Fedora hosts. It's just a bridge. I could see where Ubuntu might have some preconfigured setups for this purpose where I have to set them up by hand in Fedora. That's just a matter of the distro specific support scripts. Regards, Mike -- Michael H. Warfield (AI4NB) | (770) 985-6132 | m...@wittsend.com /\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/ NIC whois: MHW9 | An optimist believes we live in the best of all PGP Key: 0x674627FF| possible worlds. A pessimist is sure of it! signature.asc Description: This is a digitally signed message part -- How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views of enterprise services 3. A single system of record for all IT processes http://p.sf.net/sfu/servicenow-d2d-j___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] lxcbr0 MAC addr issue
On Wed, 2013-06-05 at 06:23 +, Hans Feldt wrote: It is a fact that the bridge takes the lowest MAC address from the attached ports for the host port. See for example http://backreference.org/2010/07/28/linux-bridge-mac-addresses-and-dynamic-ports/ Thus if a container is restarted, the host port can potentially change its MAC address and containers will have a stale ARP cache. This of course causes problem for communication container-host. Tested the workaround mentioned in the link but then I got problem with network manager on a later Ubuntu version. Then I tried using a dummy container and reusing its MAC addr for the host port. Works but... Now my question, could not lxc (at boot) setup a fixed MAC addr for the host port? There's a gotcha in there. You can not set an arbitrary MAC address on a bridge. It can only be the MAC address of an attached interface. It has to do with how packets are routed down in the kernel and determining if a packet is to be handled locally on the bridge or not. It also may have some ties in to the spanning tree algorithm protocol logic (whether you are using STP or have it enabled or not). If you set it to a fixed MAC address of a container, you can't stop or reboot that container without losing that static assignment on the bridge. A dummy container is one option, if you don't have a host hardware interface connected to the bridge. But you need one with a MAC address lower than any of the others. Another alternative is to use a dummy interface... modprobe dummy brctl addif lxcbr0 dummy0 The dummy0 interface doesn't even need to be up or have an IP address assigned to it. Since the container host-local addresses will all be private (fe:...) and the dummy0 interface will have something lower, you should be good to go. Thanks, Hans Regards, Mike -- Michael H. Warfield (AI4NB) | (770) 985-6132 | m...@wittsend.com /\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/ NIC whois: MHW9 | An optimist believes we live in the best of all PGP Key: 0x674627FF| possible worlds. A pessimist is sure of it! signature.asc Description: This is a digitally signed message part -- How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views of enterprise services 3. A single system of record for all IT processes http://p.sf.net/sfu/servicenow-d2d-j___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] lxcbr0 MAC addr issue
Quoting Michael H. Warfield (m...@wittsend.com): Crap... Bumped the keyboard and this one got away from me prematurely. On Wed, 2013-06-05 at 11:23 -0400, Michael H. Warfield wrote: On Wed, 2013-06-05 at 15:17 +, Jäkel, Guido wrote: yes and it does this. The point is that lxcbr0 is not tied to any physical nic. So the first container you start, however high the macaddr is, lxcbr0 takes its mac. If the next container gets a lower macaddr, lxcbr0's macaddr drops. This lxcbr0 is special to Ubuntu, right? And if not to a physical NIC, to what is this bridge connected to on the host? Not to the best of my knowledge. It should be a simple bridge. What do you get for this command? brctl show A bridge doesn A bridge doesn't have to be attach to a device. A bridge is its own logical entity in the kernel to which you may attach devices. You can not attach a bridge to something else. You can only attach something else to the bridge. There's a difference. In the case of a NATing configuration, you set up a bridge (name it whatever you want) and attach the containers to it. Then you use the NAT modules to route between the bridge and the external interface while NATing the addresses. I use lxcbr0 on my Fedora hosts. It's just a bridge. I could see where Ubuntu might have some preconfigured setups for this purpose where I have to set them up by hand in Fedora. That's just a matter of the distro specific support scripts. Right. And we *could* attach a dummy device with mac starting with something lower. BUT I just did some testing, and even as I watch lxcbr0's addr go down when starting a new container, my ssh to the container which had the higher macaddr doesn't hiccough. Perhaps it'll be a problem when connected from an outside host (through port forwarding). In that case I'll happily do the dummy if hack. But it seems possible that it just isn't needed. (And since the iptables rule is --to-destination an ip address, I'm thinking it won't be) -serge -- How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views of enterprise services 3. A single system of record for all IT processes http://p.sf.net/sfu/servicenow-d2d-j ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] lxcbr0 MAC addr issue
On Wed, 2013-06-05 at 11:26 -0500, Serge Hallyn wrote: Quoting Michael H. Warfield (m...@wittsend.com): Crap... Bumped the keyboard and this one got away from me prematurely. On Wed, 2013-06-05 at 11:23 -0400, Michael H. Warfield wrote: On Wed, 2013-06-05 at 15:17 +, Jäkel, Guido wrote: yes and it does this. The point is that lxcbr0 is not tied to any physical nic. So the first container you start, however high the macaddr is, lxcbr0 takes its mac. If the next container gets a lower macaddr, lxcbr0's macaddr drops. This lxcbr0 is special to Ubuntu, right? And if not to a physical NIC, to what is this bridge connected to on the host? Not to the best of my knowledge. It should be a simple bridge. What do you get for this command? brctl show A bridge doesn A bridge doesn't have to be attach to a device. A bridge is its own logical entity in the kernel to which you may attach devices. You can not attach a bridge to something else. You can only attach something else to the bridge. There's a difference. In the case of a NATing configuration, you set up a bridge (name it whatever you want) and attach the containers to it. Then you use the NAT modules to route between the bridge and the external interface while NATing the addresses. I use lxcbr0 on my Fedora hosts. It's just a bridge. I could see where Ubuntu might have some preconfigured setups for this purpose where I have to set them up by hand in Fedora. That's just a matter of the distro specific support scripts. Right. And we *could* attach a dummy device with mac starting with something lower. BUT I just did some testing, and even as I watch lxcbr0's addr go down when starting a new container, my ssh to the container which had the higher macaddr doesn't hiccough. Hmmm... It would be interesting to see what you get from arp -a on the host and the container before and after that. It would also be interesting to see what happens if you ssh to a container with the higher address first and then bring up a container with a lower mac address and see if it impacts the existing connection. It really all depends on how the host is managing the arp table and, if the MAC address changes, how does the bridge change impact the arp table. In the case of ssh'ing to a container from the host, the host would still have the correct arp entry for the container which would facilitate the delivery of the initial SYN. The container would have an incorrect entry but that should correct itself as soon as that new packet arrives connecting from the host, invalidating the containers arp table entry, I would think. It should also correct the MAC table entries for the bridge (used by the STP) if it originates from the interface that changed. It works the same way over an eitherswitch externally. The moment a packet is sent with a new from MAC address, the switch (bridge) remembers what port (attachment) that MAC address was last seen on and enters it into it's mac switch table. What would be interesting to know is how the container sees it, since his arp table entry for the host is then wrong if he initiates a packet first, after the change. It could be that the tap/bridge connection from container to host bridge would clear that up quickly. Perhaps it'll be a problem when connected from an outside host (through port forwarding). In that case I'll happily do the dummy if hack. But it seems possible that it just isn't needed. (And since the iptables rule is --to-destination an ip address, I'm thinking it won't be) Yeah, since the host exposed IP and MAC are not changing, it shouldn't be, in the external case with NAT as you describe. The external systems will only be referencing the external IP address for which the MAC doesn't change and the ARP tables are perfectly coherent, externally. So NAT w/ port forwarding to a container should be covered. It's where the MAC address of the host interface changes (because it's attached to a bridge) where you get into trouble in the external case. But, we have that case covered now with the fixes that are in-place since 0.8.0 (assuming the OP is on a recent enough version of the lxc tools). So both of those should be covered. That really just leaves the host - container case (container - container should be a trivial non-issue). Then I think it's an issue of which side of the bridge is issuing the first packet after the MAC address changes and what does the bridge logic do about it. Seems like a corner case to me. I also don't know what role STP has in this game. On my fedora system, my lxcbr0 bridge has STP disabled but the virbr0 bridge (used by libvirt and setup by default) has it enabled and I have no idea why. It's possible that these symptoms could vary depending on that setting. I've definitely heard it mentioned before. Oh, and this is likely to get really ugly with IPv6. With the host
Re: [Lxc-users] lxcbr0 MAC addr issue
Quoting Michael H. Warfield (m...@wittsend.com): On Wed, 2013-06-05 at 11:26 -0500, Serge Hallyn wrote: Quoting Michael H. Warfield (m...@wittsend.com): Crap... Bumped the keyboard and this one got away from me prematurely. On Wed, 2013-06-05 at 11:23 -0400, Michael H. Warfield wrote: On Wed, 2013-06-05 at 15:17 +, Jäkel, Guido wrote: yes and it does this. The point is that lxcbr0 is not tied to any physical nic. So the first container you start, however high the macaddr is, lxcbr0 takes its mac. If the next container gets a lower macaddr, lxcbr0's macaddr drops. This lxcbr0 is special to Ubuntu, right? And if not to a physical NIC, to what is this bridge connected to on the host? Not to the best of my knowledge. It should be a simple bridge. What do you get for this command? brctl show A bridge doesn A bridge doesn't have to be attach to a device. A bridge is its own logical entity in the kernel to which you may attach devices. You can not attach a bridge to something else. You can only attach something else to the bridge. There's a difference. In the case of a NATing configuration, you set up a bridge (name it whatever you want) and attach the containers to it. Then you use the NAT modules to route between the bridge and the external interface while NATing the addresses. I use lxcbr0 on my Fedora hosts. It's just a bridge. I could see where Ubuntu might have some preconfigured setups for this purpose where I have to set them up by hand in Fedora. That's just a matter of the distro specific support scripts. Right. And we *could* attach a dummy device with mac starting with something lower. BUT I just did some testing, and even as I watch lxcbr0's addr go down when starting a new container, my ssh to the container which had the higher macaddr doesn't hiccough. Hmmm... It would be interesting to see what you get from arp -a on Uh, well - with one container started, I get ? (10.0.3.182) at 00:16:3e:1a:33:6c [ether] on lxcbr0 static.65.5.9.176.clients.your-server.de (176.9.5.65) at 00:26:88:76:1c:05 [ether] on eth0 Note that lxcbr0 and the container's veth are HWaddr fe:ab:34:b2:2f:cc. So I guess the reason it doesn't matter is that lxcbr0 is not an endpoint for anything. The endpoint is always the other veth endpoint. the host and the container before and after that. It would also be interesting to see what happens if you ssh to a container with the higher address first and then bring up a container with a lower mac address and see if it impacts the existing connection. (I did both.) -serge -- How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views of enterprise services 3. A single system of record for all IT processes http://p.sf.net/sfu/servicenow-d2d-j ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
