[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
SRU proposal for Wily (extended with fix to bug #1532722) ** Patch added: "wily_ifupdown_0.7.54ubuntu3.debdiff" https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/1337873/+attachment/4548423/+files/wily_ifupdown_0.7.54ubuntu3.debdiff -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in ifenslave package in Ubuntu: Fix Released Status in ifupdown package in Ubuntu: Fix Released Status in ifenslave source package in Precise: Won't Fix Status in ifupdown source package in Precise: Won't Fix Status in ifenslave source package in Trusty: Fix Committed Status in ifupdown source package in Trusty: Fix Committed Status in ifenslave source package in Vivid: Fix Released Status in ifupdown source package in Vivid: Won't Fix Status in ifenslave source package in Wily: Fix Released Status in ifupdown source package in Wily: Fix Committed Status in ifupdown package in Debian: New Bug description: [Impact] * Lack of proper synchronization in ifupdown causes a race condition resulting in occasional incorrect network interface initialization (e.g. in bonding case - wrong bonding settings, network unavailable because slave<->master interfaces initialization order was wrong * This is very annoying in case of large deployments (e.g. when bringing up 1000 machines it is almost guaranteed that at least a few of them will end up with network down). * It has been fixed by introducing hierarchical and per-interface locking mechanism ensuring the right order (along with the correct order in the /e/n/interfaces file) of initialization [Test Case] 1. Create a VM with bonding configured with at least 2 slave interfaces. 2. Reboot. 3. If all interfaces are up - go to 2. [Regression Potential] * This change has been introduced upstream in Debian. * It does not require any config changes to existing installations. [Other Info] Original bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only "networking" script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script "network-interface" enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running network-interface (eth0)
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
SRU proposal for Trusty (extended with fix to bug #1532722) ** Patch added: "trusty_ifupdown_0.7.47.2ubuntu4.3.debdiff" https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/1337873/+attachment/4548422/+files/trusty_ifupdown_0.7.47.2ubuntu4.3.debdiff -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in ifenslave package in Ubuntu: Fix Released Status in ifupdown package in Ubuntu: Fix Released Status in ifenslave source package in Precise: Won't Fix Status in ifupdown source package in Precise: Won't Fix Status in ifenslave source package in Trusty: Fix Committed Status in ifupdown source package in Trusty: Fix Committed Status in ifenslave source package in Vivid: Fix Released Status in ifupdown source package in Vivid: Won't Fix Status in ifenslave source package in Wily: Fix Released Status in ifupdown source package in Wily: Fix Committed Status in ifupdown package in Debian: New Bug description: [Impact] * Lack of proper synchronization in ifupdown causes a race condition resulting in occasional incorrect network interface initialization (e.g. in bonding case - wrong bonding settings, network unavailable because slave<->master interfaces initialization order was wrong * This is very annoying in case of large deployments (e.g. when bringing up 1000 machines it is almost guaranteed that at least a few of them will end up with network down). * It has been fixed by introducing hierarchical and per-interface locking mechanism ensuring the right order (along with the correct order in the /e/n/interfaces file) of initialization [Test Case] 1. Create a VM with bonding configured with at least 2 slave interfaces. 2. Reboot. 3. If all interfaces are up - go to 2. [Regression Potential] * This change has been introduced upstream in Debian. * It does not require any config changes to existing installations. [Other Info] Original bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only "networking" script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script "network-interface" enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
Since per-interface locking landed in Xenial, we've been getting crashes, see bug 1532722. Until this is fixed, I'm marking this as v-failed. We'll then need to update the SRU with this fix as well. ** Tags removed: verification-done ** Tags added: verification-failed -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in ifenslave package in Ubuntu: Fix Released Status in ifupdown package in Ubuntu: Fix Released Status in ifenslave source package in Precise: Won't Fix Status in ifupdown source package in Precise: Won't Fix Status in ifenslave source package in Trusty: Fix Committed Status in ifupdown source package in Trusty: Fix Committed Status in ifenslave source package in Vivid: Fix Released Status in ifupdown source package in Vivid: Won't Fix Status in ifenslave source package in Wily: Fix Released Status in ifupdown source package in Wily: Fix Committed Status in ifupdown package in Debian: New Bug description: [Impact] * Lack of proper synchronization in ifupdown causes a race condition resulting in occasional incorrect network interface initialization (e.g. in bonding case - wrong bonding settings, network unavailable because slave<->master interfaces initialization order was wrong * This is very annoying in case of large deployments (e.g. when bringing up 1000 machines it is almost guaranteed that at least a few of them will end up with network down). * It has been fixed by introducing hierarchical and per-interface locking mechanism ensuring the right order (along with the correct order in the /e/n/interfaces file) of initialization [Test Case] 1. Create a VM with bonding configured with at least 2 slave interfaces. 2. Reboot. 3. If all interfaces are up - go to 2. [Regression Potential] * This change has been introduced upstream in Debian. * It does not require any config changes to existing installations. [Other Info] Original bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only "networking" script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script "network-interface" enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
I have verified both Trusty and Wily. The verification was automated cyclic rebooting of a VM containing 3 NICs - 2 of them were used in bonding in active-backup. Before the fix has been implemented this test failed with some interfaces uninitialized or the bonding mode being wrong (the default round-robin was set). This time, with the -proposed versions, after over 48 hours of the test none of the symptoms occurred. Thus, tagging as verified. ** Tags removed: cts verification-needed ** Tags added: sts verification-done -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in ifenslave package in Ubuntu: Fix Released Status in ifupdown package in Ubuntu: Fix Released Status in ifenslave source package in Precise: Won't Fix Status in ifupdown source package in Precise: Won't Fix Status in ifenslave source package in Trusty: Fix Committed Status in ifupdown source package in Trusty: Fix Committed Status in ifenslave source package in Vivid: Fix Released Status in ifupdown source package in Vivid: Won't Fix Status in ifenslave source package in Wily: Fix Released Status in ifupdown source package in Wily: Fix Committed Status in ifupdown package in Debian: New Bug description: [Impact] * Lack of proper synchronization in ifupdown causes a race condition resulting in occasional incorrect network interface initialization (e.g. in bonding case - wrong bonding settings, network unavailable because slave<->master interfaces initialization order was wrong * This is very annoying in case of large deployments (e.g. when bringing up 1000 machines it is almost guaranteed that at least a few of them will end up with network down). * It has been fixed by introducing hierarchical and per-interface locking mechanism ensuring the right order (along with the correct order in the /e/n/interfaces file) of initialization [Test Case] 1. Create a VM with bonding configured with at least 2 slave interfaces. 2. Reboot. 3. If all interfaces are up - go to 2. [Regression Potential] * This change has been introduced upstream in Debian. * It does not require any config changes to existing installations. [Other Info] Original bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only "networking" script enabled: --- root@provisioned:~# initctl list | grep
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
** Changed in: ifupdown (Ubuntu Wily) Importance: Undecided => Medium ** Changed in: ifenslave (Ubuntu) Importance: Undecided => Medium ** Changed in: ifenslave (Ubuntu Precise) Importance: Undecided => Medium ** Changed in: ifenslave (Ubuntu Trusty) Importance: Undecided => Medium ** Changed in: ifenslave (Ubuntu Vivid) Importance: Undecided => Medium ** Changed in: ifenslave (Ubuntu Wily) Importance: Undecided => Medium -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in ifenslave package in Ubuntu: Fix Released Status in ifupdown package in Ubuntu: Fix Released Status in ifenslave source package in Precise: Won't Fix Status in ifupdown source package in Precise: Won't Fix Status in ifenslave source package in Trusty: Fix Committed Status in ifupdown source package in Trusty: Fix Committed Status in ifenslave source package in Vivid: Fix Released Status in ifupdown source package in Vivid: Won't Fix Status in ifenslave source package in Wily: Fix Released Status in ifupdown source package in Wily: Fix Committed Status in ifupdown package in Debian: New Bug description: [Impact] * Lack of proper synchronization in ifupdown causes a race condition resulting in occasional incorrect network interface initialization (e.g. in bonding case - wrong bonding settings, network unavailable because slave<->master interfaces initialization order was wrong * This is very annoying in case of large deployments (e.g. when bringing up 1000 machines it is almost guaranteed that at least a few of them will end up with network down). * It has been fixed by introducing hierarchical and per-interface locking mechanism ensuring the right order (along with the correct order in the /e/n/interfaces file) of initialization [Test Case] 1. Create a VM with bonding configured with at least 2 slave interfaces. 2. Reboot. 3. If all interfaces are up - go to 2. [Regression Potential] * This change has been introduced upstream in Debian. * It does not require any config changes to existing installations. [Other Info] Original bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only "networking" script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
Hello Rafael, or anyone else affected, Accepted ifupdown into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ifupdown/0.7.54ubuntu1.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: ifupdown (Ubuntu Wily) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in ifenslave package in Ubuntu: Fix Released Status in ifupdown package in Ubuntu: Fix Released Status in ifenslave source package in Precise: Won't Fix Status in ifupdown source package in Precise: Won't Fix Status in ifenslave source package in Trusty: Fix Committed Status in ifupdown source package in Trusty: Fix Committed Status in ifenslave source package in Vivid: Fix Released Status in ifupdown source package in Vivid: Won't Fix Status in ifenslave source package in Wily: Fix Released Status in ifupdown source package in Wily: Fix Committed Status in ifupdown package in Debian: New Bug description: [Impact] * Lack of proper synchronization in ifupdown causes a race condition resulting in occasional incorrect network interface initialization (e.g. in bonding case - wrong bonding settings, network unavailable because slave<->master interfaces initialization order was wrong * This is very annoying in case of large deployments (e.g. when bringing up 1000 machines it is almost guaranteed that at least a few of them will end up with network down). * It has been fixed by introducing hierarchical and per-interface locking mechanism ensuring the right order (along with the correct order in the /e/n/interfaces file) of initialization [Test Case] 1. Create a VM with bonding configured with at least 2 slave interfaces. 2. Reboot. 3. If all interfaces are up - go to 2. [Regression Potential] * This change has been introduced upstream in Debian. * It does not require any config changes to existing installations. [Other Info] Original bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
Hello Rafael, or anyone else affected, Accepted ifenslave into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ifenslave/2.4ubuntu1.2 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: ifenslave (Ubuntu Trusty) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in ifenslave package in Ubuntu: Fix Released Status in ifupdown package in Ubuntu: Fix Released Status in ifenslave source package in Precise: Won't Fix Status in ifupdown source package in Precise: Won't Fix Status in ifenslave source package in Trusty: Fix Committed Status in ifupdown source package in Trusty: Fix Committed Status in ifenslave source package in Vivid: Fix Released Status in ifupdown source package in Vivid: Won't Fix Status in ifenslave source package in Wily: Fix Released Status in ifupdown source package in Wily: In Progress Status in ifupdown package in Debian: New Bug description: [Impact] * Lack of proper synchronization in ifupdown causes a race condition resulting in occasional incorrect network interface initialization (e.g. in bonding case - wrong bonding settings, network unavailable because slave<->master interfaces initialization order was wrong * This is very annoying in case of large deployments (e.g. when bringing up 1000 machines it is almost guaranteed that at least a few of them will end up with network down). * It has been fixed by introducing hierarchical and per-interface locking mechanism ensuring the right order (along with the correct order in the /e/n/interfaces file) of initialization [Test Case] 1. Create a VM with bonding configured with at least 2 slave interfaces. 2. Reboot. 3. If all interfaces are up - go to 2. [Regression Potential] * This change has been introduced upstream in Debian. * It does not require any config changes to existing installations. [Other Info] Original bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~#
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
Hello Rafael, or anyone else affected, Accepted ifupdown into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ifupdown/0.7.47.2ubuntu4.2 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: ifupdown (Ubuntu Trusty) Status: In Progress => Fix Committed ** Tags added: verification-needed -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in ifenslave package in Ubuntu: Fix Released Status in ifupdown package in Ubuntu: Fix Released Status in ifenslave source package in Precise: Won't Fix Status in ifupdown source package in Precise: Won't Fix Status in ifenslave source package in Trusty: Fix Committed Status in ifupdown source package in Trusty: Fix Committed Status in ifenslave source package in Vivid: Fix Released Status in ifupdown source package in Vivid: Won't Fix Status in ifenslave source package in Wily: Fix Released Status in ifupdown source package in Wily: In Progress Status in ifupdown package in Debian: New Bug description: [Impact] * Lack of proper synchronization in ifupdown causes a race condition resulting in occasional incorrect network interface initialization (e.g. in bonding case - wrong bonding settings, network unavailable because slave<->master interfaces initialization order was wrong * This is very annoying in case of large deployments (e.g. when bringing up 1000 machines it is almost guaranteed that at least a few of them will end up with network down). * It has been fixed by introducing hierarchical and per-interface locking mechanism ensuring the right order (along with the correct order in the /e/n/interfaces file) of initialization [Test Case] 1. Create a VM with bonding configured with at least 2 slave interfaces. 2. Reboot. 3. If all interfaces are up - go to 2. [Regression Potential] * This change has been introduced upstream in Debian. * It does not require any config changes to existing installations. [Other Info] Original bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! #
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
I sponsored the trusty and wily patches. ** Also affects: ifupdown (Ubuntu Wily) Importance: Undecided Status: New ** Changed in: ifupdown (Ubuntu Vivid) Status: Confirmed => Won't Fix ** Changed in: ifupdown (Ubuntu Wily) Status: New => In Progress ** Changed in: ifupdown (Ubuntu Trusty) Status: Confirmed => In Progress ** Also affects: ifenslave (Ubuntu) Importance: Undecided Status: New ** Changed in: ifenslave (Ubuntu Wily) Status: New => Fix Released ** Changed in: ifenslave (Ubuntu Vivid) Status: New => Fix Released ** Changed in: ifenslave (Ubuntu Trusty) Status: New => In Progress -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in ifenslave package in Ubuntu: Fix Released Status in ifupdown package in Ubuntu: Fix Released Status in ifenslave source package in Precise: Won't Fix Status in ifupdown source package in Precise: Won't Fix Status in ifenslave source package in Trusty: In Progress Status in ifupdown source package in Trusty: In Progress Status in ifenslave source package in Vivid: Fix Released Status in ifupdown source package in Vivid: Won't Fix Status in ifenslave source package in Wily: Fix Released Status in ifupdown source package in Wily: In Progress Status in ifupdown package in Debian: New Bug description: [Impact] * Lack of proper synchronization in ifupdown causes a race condition resulting in occasional incorrect network interface initialization (e.g. in bonding case - wrong bonding settings, network unavailable because slave<->master interfaces initialization order was wrong * This is very annoying in case of large deployments (e.g. when bringing up 1000 machines it is almost guaranteed that at least a few of them will end up with network down). * It has been fixed by introducing hierarchical and per-interface locking mechanism ensuring the right order (along with the correct order in the /e/n/interfaces file) of initialization [Test Case] 1. Create a VM with bonding configured with at least 2 slave interfaces. 2. Reboot. 3. If all interfaces are up - go to 2. [Regression Potential] * This change has been introduced upstream in Debian. * It does not require any config changes to existing installations. [Other Info] Original bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
Setting precise tasks to "wontfix", this is too complex to backport and the bug is not nearly important enough to risk regressions due to too invasive backports. ** Changed in: ifenslave (Ubuntu Precise) Status: New => Won't Fix ** Changed in: ifupdown (Ubuntu Precise) Status: Confirmed => Won't Fix ** Changed in: ifenslave (Ubuntu) Status: New => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in ifenslave package in Ubuntu: Fix Released Status in ifupdown package in Ubuntu: Fix Released Status in ifenslave source package in Precise: Won't Fix Status in ifupdown source package in Precise: Won't Fix Status in ifenslave source package in Trusty: In Progress Status in ifupdown source package in Trusty: In Progress Status in ifenslave source package in Vivid: Fix Released Status in ifupdown source package in Vivid: Won't Fix Status in ifenslave source package in Wily: Fix Released Status in ifupdown source package in Wily: In Progress Status in ifupdown package in Debian: New Bug description: [Impact] * Lack of proper synchronization in ifupdown causes a race condition resulting in occasional incorrect network interface initialization (e.g. in bonding case - wrong bonding settings, network unavailable because slave<->master interfaces initialization order was wrong * This is very annoying in case of large deployments (e.g. when bringing up 1000 machines it is almost guaranteed that at least a few of them will end up with network down). * It has been fixed by introducing hierarchical and per-interface locking mechanism ensuring the right order (along with the correct order in the /e/n/interfaces file) of initialization [Test Case] 1. Create a VM with bonding configured with at least 2 slave interfaces. 2. Reboot. 3. If all interfaces are up - go to 2. [Regression Potential] * This change has been introduced upstream in Debian. * It does not require any config changes to existing installations. [Other Info] Original bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only "networking" script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script "network-interface" enabled: --- root@provisioned:~#
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
Sponsored the patch for xenial. Let's give this some maturing there first. ** Changed in: ifupdown (Ubuntu) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in ifupdown package in Ubuntu: Fix Committed Status in ifupdown source package in Precise: Confirmed Status in ifupdown source package in Trusty: Confirmed Status in ifupdown source package in Vivid: Confirmed Status in ifupdown package in Debian: New Bug description: [Impact] * Lack of proper synchronization in ifupdown causes a race condition resulting in occasional incorrect network interface initialization (e.g. in bonding case - wrong bonding settings, network unavailable because slave<->master interfaces initialization order was wrong * This is very annoying in case of large deployments (e.g. when bringing up 1000 machines it is almost guaranteed that at least a few of them will end up with network down). * It has been fixed by introducing hierarchical and per-interface locking mechanism ensuring the right order (along with the correct order in the /e/n/interfaces file) of initialization [Test Case] 1. Create a VM with bonding configured with at least 2 slave interfaces. 2. Reboot. 3. If all interfaces are up - go to 2. [Regression Potential] * This change has been introduced upstream in Debian. * It does not require any config changes to existing installations. [Other Info] Original bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only "networking" script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script "network-interface" enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running network-interface (eth0) start/running network-interface (eth1) start/running --- Enabling bonding: Following ifenslave configuration example (/usr/share/doc/ifenslave/ examples/two_hotplug_ethernet), my /etc/network/interfaces has to look like this: --- auto eth1 iface eth1 inet manual bond-master bond0 auto eth2 iface eth2 inet manual bond-master bond0 auto bond0 iface bond0 inet static bond-mode 1
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
(unsubscribing sponsors for now then, please subscribe them back after getting some feedback from the xenial update) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in ifupdown package in Ubuntu: Fix Committed Status in ifupdown source package in Precise: Confirmed Status in ifupdown source package in Trusty: Confirmed Status in ifupdown source package in Vivid: Confirmed Status in ifupdown package in Debian: New Bug description: [Impact] * Lack of proper synchronization in ifupdown causes a race condition resulting in occasional incorrect network interface initialization (e.g. in bonding case - wrong bonding settings, network unavailable because slave<->master interfaces initialization order was wrong * This is very annoying in case of large deployments (e.g. when bringing up 1000 machines it is almost guaranteed that at least a few of them will end up with network down). * It has been fixed by introducing hierarchical and per-interface locking mechanism ensuring the right order (along with the correct order in the /e/n/interfaces file) of initialization [Test Case] 1. Create a VM with bonding configured with at least 2 slave interfaces. 2. Reboot. 3. If all interfaces are up - go to 2. [Regression Potential] * This change has been introduced upstream in Debian. * It does not require any config changes to existing installations. [Other Info] Original bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only "networking" script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script "network-interface" enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running network-interface (eth0) start/running network-interface (eth1) start/running --- Enabling bonding: Following ifenslave configuration example (/usr/share/doc/ifenslave/ examples/two_hotplug_ethernet), my /etc/network/interfaces has to look like this: --- auto eth1 iface eth1 inet manual bond-master bond0 auto eth2 iface eth2 inet manual bond-master bond0 auto bond0 iface bond0 inet static bond-mode 1 bond-miimon 100 bond-primary eth1 eth2
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
This bug was fixed in the package ifupdown - 0.7.54ubuntu2 --- ifupdown (0.7.54ubuntu2) xenial; urgency=medium * Per-interface hierarchical locking. Backported from Debian git head. (LP: #1337873, Closes: #753755) -- Dariusz GadomskiThu, 10 Nov 2015 11:30:14 +0200 ** Changed in: ifupdown (Ubuntu) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in ifupdown package in Ubuntu: Fix Released Status in ifupdown source package in Precise: Confirmed Status in ifupdown source package in Trusty: Confirmed Status in ifupdown source package in Vivid: Confirmed Status in ifupdown package in Debian: New Bug description: [Impact] * Lack of proper synchronization in ifupdown causes a race condition resulting in occasional incorrect network interface initialization (e.g. in bonding case - wrong bonding settings, network unavailable because slave<->master interfaces initialization order was wrong * This is very annoying in case of large deployments (e.g. when bringing up 1000 machines it is almost guaranteed that at least a few of them will end up with network down). * It has been fixed by introducing hierarchical and per-interface locking mechanism ensuring the right order (along with the correct order in the /e/n/interfaces file) of initialization [Test Case] 1. Create a VM with bonding configured with at least 2 slave interfaces. 2. Reboot. 3. If all interfaces are up - go to 2. [Regression Potential] * This change has been introduced upstream in Debian. * It does not require any config changes to existing installations. [Other Info] Original bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only "networking" script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script "network-interface" enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running network-interface (eth0) start/running network-interface (eth1) start/running --- Enabling bonding: Following ifenslave configuration example (/usr/share/doc/ifenslave/ examples/two_hotplug_ethernet), my
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
Adding SRU proposal for Xenial. ** Patch added: "xenial_ifupdown_0.7.54ubuntu2.debdiff" https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/1337873/+attachment/4515989/+files/xenial_ifupdown_0.7.54ubuntu2.debdiff -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in ifupdown package in Ubuntu: In Progress Status in ifupdown source package in Precise: Confirmed Status in ifupdown source package in Trusty: Confirmed Status in ifupdown source package in Vivid: Confirmed Status in ifupdown package in Debian: New Bug description: [Impact] * Lack of proper synchronization in ifupdown causes a race condition resulting in occasional incorrect network interface initialization (e.g. in bonding case - wrong bonding settings, network unavailable because slave<->master interfaces initialization order was wrong * This is very annoying in case of large deployments (e.g. when bringing up 1000 machines it is almost guaranteed that at least a few of them will end up with network down). * It has been fixed by introducing hierarchical and per-interface locking mechanism ensuring the right order (along with the correct order in the /e/n/interfaces file) of initialization [Test Case] 1. Create a VM with bonding configured with at least 2 slave interfaces. 2. Reboot. 3. If all interfaces are up - go to 2. [Regression Potential] * This change has been introduced upstream in Debian. * It does not require any config changes to existing installations. [Other Info] Original bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only "networking" script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script "network-interface" enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running network-interface (eth0) start/running network-interface (eth1) start/running --- Enabling bonding: Following ifenslave configuration example (/usr/share/doc/ifenslave/ examples/two_hotplug_ethernet), my /etc/network/interfaces has to look like this: --- auto eth1 iface eth1 inet manual bond-master bond0 auto eth2 iface eth2 inet manual bond-master bond0 auto
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
** Changed in: ifupdown (Ubuntu) Importance: Undecided => Medium ** Changed in: ifupdown (Ubuntu Precise) Importance: Undecided => Medium ** Changed in: ifupdown (Ubuntu Trusty) Importance: Undecided => Medium ** Changed in: ifupdown (Ubuntu Vivid) Importance: Undecided => Medium -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in ifupdown package in Ubuntu: In Progress Status in ifupdown source package in Precise: Confirmed Status in ifupdown source package in Trusty: Confirmed Status in ifupdown source package in Vivid: Confirmed Status in ifupdown package in Debian: New Bug description: [Impact] * Lack of proper synchronization in ifupdown causes a race condition resulting in occasional incorrect network interface initialization (e.g. in bonding case - wrong bonding settings, network unavailable because slave<->master interfaces initialization order was wrong * This is very annoying in case of large deployments (e.g. when bringing up 1000 machines it is almost guaranteed that at least a few of them will end up with network down). * It has been fixed by introducing hierarchical and per-interface locking mechanism ensuring the right order (along with the correct order in the /e/n/interfaces file) of initialization [Test Case] 1. Create a VM with bonding configured with at least 2 slave interfaces. 2. Reboot. 3. If all interfaces are up - go to 2. [Regression Potential] * This change has been introduced upstream in Debian. * It does not require any config changes to existing installations. [Other Info] Original bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only "networking" script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script "network-interface" enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running network-interface (eth0) start/running network-interface (eth1) start/running --- Enabling bonding: Following ifenslave configuration example (/usr/share/doc/ifenslave/ examples/two_hotplug_ethernet), my /etc/network/interfaces has to look like this: --- auto eth1 iface eth1 inet manual bond-master
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
Adding SRU proposal for wily. ** Description changed: + [Impact] + + * A lack of proper synchronization in ifupdown causes a race condition + resulting in occasional incorrect network interface initialization (e.g. + in bonding case - wrong bonding settings, network unavailable because + slave<->master interfaces initialization order was wrong + + * This is very annoying in case of large deployments (e.g. when + bringing up 1000 machines it is almost guaranteed that at least a few of + them will end up with network down). + + * It has been fixed by introducing hierarchical and per-interface + locking mechanism ensuring the right order (along with the correct order + in the /e/n/interfaces file) of initialization + + [Test Case] + + 1. Create a VM with bonding configured with at least 2 slave interfaces. + 2. Reboot. + 3. If all interfaces are up - go to 2. + + [Regression Potential] + + * This change has been introduced upstream in Debian. + * It does not require any config changes to existing installations. + + [Other Info] + + Original bug description: + * please consider my bonding examples are using eth1 and eth2 as slave - interfaces. + interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. - 1-1. Wait for locking ifstate.lock - file. + 1-1. Wait for locking ifstate.lock + file. 1-2. Read ifstate file to check - the target NIC. + the target NIC. 1-3. close(=release) ifstate.lock - file. + file. 1-4. Judge that the target NIC - isn't processed. - 1-2. Read ifstate file to check -the target NIC. - 1-3. close(=release) ifstate.lock -file. - 1-4. Judge that the target NIC -isn't processed. + isn't processed. + 1-2. Read ifstate file to check + the target NIC. + 1-3. close(=release) ifstate.lock + file. + 1-4. Judge that the target NIC + isn't processed. 2. Lock and update ifstate file. -Release the lock. - 2. Lock and update ifstate file. - Release the lock. + Release the lock. + 2. Lock and update ifstate file. + Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to -/sys/class/net/bond0/bonding -/slaves then NIC gets up - 4. Link down the target NIC. - 5. Fails to write NIC id to - /sys/class/net/bond0/bonding/ - slaves it is already written. + /sys/class/net/bond0/bonding + /slaves then NIC gets up + 4. Link down the target NIC. + 5. Fails to write NIC id to + /sys/class/net/bond0/bonding/ + slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface - enabled) with no problem. I can also boot with only "networking" + enabled) with no problem. I can also boot with only "networking" script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script "network-interface" enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
Adding SRU proposal for Vivid. ** Patch added: "vivid_ifupdown_0.7.48.1ubuntu11.debdiff" https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/1337873/+attachment/4501798/+files/vivid_ifupdown_0.7.48.1ubuntu11.debdiff -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in ifupdown package in Ubuntu: In Progress Status in ifupdown source package in Precise: Confirmed Status in ifupdown source package in Trusty: Confirmed Status in ifupdown source package in Vivid: Confirmed Status in ifupdown package in Debian: New Bug description: [Impact] * A lack of proper synchronization in ifupdown causes a race condition resulting in occasional incorrect network interface initialization (e.g. in bonding case - wrong bonding settings, network unavailable because slave<->master interfaces initialization order was wrong * This is very annoying in case of large deployments (e.g. when bringing up 1000 machines it is almost guaranteed that at least a few of them will end up with network down). * It has been fixed by introducing hierarchical and per-interface locking mechanism ensuring the right order (along with the correct order in the /e/n/interfaces file) of initialization [Test Case] 1. Create a VM with bonding configured with at least 2 slave interfaces. 2. Reboot. 3. If all interfaces are up - go to 2. [Regression Potential] * This change has been introduced upstream in Debian. * It does not require any config changes to existing installations. [Other Info] Original bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only "networking" script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script "network-interface" enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running network-interface (eth0) start/running network-interface (eth1) start/running --- Enabling bonding: Following ifenslave configuration example (/usr/share/doc/ifenslave/ examples/two_hotplug_ethernet), my /etc/network/interfaces has to look like this: --- auto eth1 iface eth1 inet manual bond-master bond0 auto eth2 iface eth2 inet manual bond-master
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
Adding SRU proposal for Trusty (to make ifenslave compatible with ifupdown changes). ** Patch added: "trusty_ifenslave_2.4ubuntu1.2.debdiff" https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/1337873/+attachment/4501800/+files/trusty_ifenslave_2.4ubuntu1.2.debdiff -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in ifupdown package in Ubuntu: In Progress Status in ifupdown source package in Precise: Confirmed Status in ifupdown source package in Trusty: Confirmed Status in ifupdown source package in Vivid: Confirmed Status in ifupdown package in Debian: New Bug description: [Impact] * A lack of proper synchronization in ifupdown causes a race condition resulting in occasional incorrect network interface initialization (e.g. in bonding case - wrong bonding settings, network unavailable because slave<->master interfaces initialization order was wrong * This is very annoying in case of large deployments (e.g. when bringing up 1000 machines it is almost guaranteed that at least a few of them will end up with network down). * It has been fixed by introducing hierarchical and per-interface locking mechanism ensuring the right order (along with the correct order in the /e/n/interfaces file) of initialization [Test Case] 1. Create a VM with bonding configured with at least 2 slave interfaces. 2. Reboot. 3. If all interfaces are up - go to 2. [Regression Potential] * This change has been introduced upstream in Debian. * It does not require any config changes to existing installations. [Other Info] Original bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only "networking" script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script "network-interface" enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running network-interface (eth0) start/running network-interface (eth1) start/running --- Enabling bonding: Following ifenslave configuration example (/usr/share/doc/ifenslave/ examples/two_hotplug_ethernet), my /etc/network/interfaces has to look like this: --- auto eth1 iface eth1 inet manual bond-master bond0 auto
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
Adding SRU proposal for Trusty. ** Patch added: "trusty_ifupdown_0.7.47.2ubuntu4.2.debdiff" https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/1337873/+attachment/4501799/+files/trusty_ifupdown_0.7.47.2ubuntu4.2.debdiff -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in ifupdown package in Ubuntu: In Progress Status in ifupdown source package in Precise: Confirmed Status in ifupdown source package in Trusty: Confirmed Status in ifupdown source package in Vivid: Confirmed Status in ifupdown package in Debian: New Bug description: [Impact] * A lack of proper synchronization in ifupdown causes a race condition resulting in occasional incorrect network interface initialization (e.g. in bonding case - wrong bonding settings, network unavailable because slave<->master interfaces initialization order was wrong * This is very annoying in case of large deployments (e.g. when bringing up 1000 machines it is almost guaranteed that at least a few of them will end up with network down). * It has been fixed by introducing hierarchical and per-interface locking mechanism ensuring the right order (along with the correct order in the /e/n/interfaces file) of initialization [Test Case] 1. Create a VM with bonding configured with at least 2 slave interfaces. 2. Reboot. 3. If all interfaces are up - go to 2. [Regression Potential] * This change has been introduced upstream in Debian. * It does not require any config changes to existing installations. [Other Info] Original bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only "networking" script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script "network-interface" enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running network-interface (eth0) start/running network-interface (eth1) start/running --- Enabling bonding: Following ifenslave configuration example (/usr/share/doc/ifenslave/ examples/two_hotplug_ethernet), my /etc/network/interfaces has to look like this: --- auto eth1 iface eth1 inet manual bond-master bond0 auto eth2 iface eth2 inet manual
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
** Description changed: [Impact] - * A lack of proper synchronization in ifupdown causes a race condition + * Lack of proper synchronization in ifupdown causes a race condition resulting in occasional incorrect network interface initialization (e.g. in bonding case - wrong bonding settings, network unavailable because slave<->master interfaces initialization order was wrong - * This is very annoying in case of large deployments (e.g. when + * This is very annoying in case of large deployments (e.g. when bringing up 1000 machines it is almost guaranteed that at least a few of them will end up with network down). - * It has been fixed by introducing hierarchical and per-interface + * It has been fixed by introducing hierarchical and per-interface locking mechanism ensuring the right order (along with the correct order in the /e/n/interfaces file) of initialization [Test Case] - 1. Create a VM with bonding configured with at least 2 slave interfaces. - 2. Reboot. - 3. If all interfaces are up - go to 2. + 1. Create a VM with bonding configured with at least 2 slave interfaces. + 2. Reboot. + 3. If all interfaces are up - go to 2. [Regression Potential] - * This change has been introduced upstream in Debian. - * It does not require any config changes to existing installations. + * This change has been introduced upstream in Debian. + * It does not require any config changes to existing installations. [Other Info] - + Original bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only "networking" script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script "network-interface" enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running network-interface (eth0) start/running network-interface (eth1) start/running --- Enabling bonding: Following ifenslave configuration example (/usr/share/doc/ifenslave/ examples/two_hotplug_ethernet), my /etc/network/interfaces has to look like this: --- auto eth1 iface eth1 inet manual bond-master bond0 auto eth2 iface eth2 inet manual bond-master bond0 auto bond0 iface bond0 inet static bond-mode 1 bond-miimon 100 bond-primary eth1 eth2 address 192.168.169.1 netmask 255.255.255.0 broadcast 192.168.169.255 --- Having both scripts running does
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: ifupdown (Ubuntu Precise) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in ifupdown package in Ubuntu: In Progress Status in ifupdown source package in Precise: Confirmed Status in ifupdown source package in Trusty: Confirmed Status in ifupdown source package in Vivid: Confirmed Status in ifupdown package in Debian: New Bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only "networking" script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script "network-interface" enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running network-interface (eth0) start/running network-interface (eth1) start/running --- Enabling bonding: Following ifenslave configuration example (/usr/share/doc/ifenslave/ examples/two_hotplug_ethernet), my /etc/network/interfaces has to look like this: --- auto eth1 iface eth1 inet manual bond-master bond0 auto eth2 iface eth2 inet manual bond-master bond0 auto bond0 iface bond0 inet static bond-mode 1 bond-miimon 100 bond-primary eth1 eth2 address 192.168.169.1 netmask 255.255.255.0 broadcast 192.168.169.255 --- Having both scripts running does not make any difference since we are missing "bond-slaves" keyword on slave interfaces, for ifenslave to work, and they are set to "manual". Ifenslave code: """ for slave in $BOND_SLAVES ; do ... # Ensure $slave is down. ip link set "$slave" down 2>/dev/null if ! sysfs_add slaves "$slave" 2>/dev/null ; then echo "Failed to enslave $slave to $BOND_MASTER. Is $BOND_MASTER ready and a bonding interface ?" >&2 else # Bring up slave if it is the target of an allow-bondX stanza. # This is usefull to bring up slaves that need extra setup. if [ -z "$(which ifquery)" ] || ifquery --allow \"$BOND_MASTER\" --list | grep -q $slave; then ifup $v --allow "$BOND_MASTER" "$slave" fi """ Without the keyword "bond-slaves" on the master
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: ifupdown (Ubuntu Vivid) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in ifupdown package in Ubuntu: In Progress Status in ifupdown source package in Precise: Confirmed Status in ifupdown source package in Trusty: Confirmed Status in ifupdown source package in Vivid: Confirmed Status in ifupdown package in Debian: New Bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only "networking" script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script "network-interface" enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running network-interface (eth0) start/running network-interface (eth1) start/running --- Enabling bonding: Following ifenslave configuration example (/usr/share/doc/ifenslave/ examples/two_hotplug_ethernet), my /etc/network/interfaces has to look like this: --- auto eth1 iface eth1 inet manual bond-master bond0 auto eth2 iface eth2 inet manual bond-master bond0 auto bond0 iface bond0 inet static bond-mode 1 bond-miimon 100 bond-primary eth1 eth2 address 192.168.169.1 netmask 255.255.255.0 broadcast 192.168.169.255 --- Having both scripts running does not make any difference since we are missing "bond-slaves" keyword on slave interfaces, for ifenslave to work, and they are set to "manual". Ifenslave code: """ for slave in $BOND_SLAVES ; do ... # Ensure $slave is down. ip link set "$slave" down 2>/dev/null if ! sysfs_add slaves "$slave" 2>/dev/null ; then echo "Failed to enslave $slave to $BOND_MASTER. Is $BOND_MASTER ready and a bonding interface ?" >&2 else # Bring up slave if it is the target of an allow-bondX stanza. # This is usefull to bring up slaves that need extra setup. if [ -z "$(which ifquery)" ] || ifquery --allow \"$BOND_MASTER\" --list | grep -q $slave; then ifup $v --allow "$BOND_MASTER" "$slave" fi """ Without the keyword "bond-slaves" on the master
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: ifupdown (Ubuntu Trusty) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in ifupdown package in Ubuntu: In Progress Status in ifupdown source package in Precise: Confirmed Status in ifupdown source package in Trusty: Confirmed Status in ifupdown source package in Vivid: Confirmed Status in ifupdown package in Debian: New Bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only "networking" script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script "network-interface" enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running network-interface (eth0) start/running network-interface (eth1) start/running --- Enabling bonding: Following ifenslave configuration example (/usr/share/doc/ifenslave/ examples/two_hotplug_ethernet), my /etc/network/interfaces has to look like this: --- auto eth1 iface eth1 inet manual bond-master bond0 auto eth2 iface eth2 inet manual bond-master bond0 auto bond0 iface bond0 inet static bond-mode 1 bond-miimon 100 bond-primary eth1 eth2 address 192.168.169.1 netmask 255.255.255.0 broadcast 192.168.169.255 --- Having both scripts running does not make any difference since we are missing "bond-slaves" keyword on slave interfaces, for ifenslave to work, and they are set to "manual". Ifenslave code: """ for slave in $BOND_SLAVES ; do ... # Ensure $slave is down. ip link set "$slave" down 2>/dev/null if ! sysfs_add slaves "$slave" 2>/dev/null ; then echo "Failed to enslave $slave to $BOND_MASTER. Is $BOND_MASTER ready and a bonding interface ?" >&2 else # Bring up slave if it is the target of an allow-bondX stanza. # This is usefull to bring up slaves that need extra setup. if [ -z "$(which ifquery)" ] || ifquery --allow \"$BOND_MASTER\" --list | grep -q $slave; then ifup $v --allow "$BOND_MASTER" "$slave" fi """ Without the keyword "bond-slaves" on the master
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
** Also affects: ifupdown (Ubuntu Vivid) Importance: Undecided Status: New ** Also affects: ifupdown (Ubuntu Trusty) Importance: Undecided Status: New ** Also affects: ifupdown (Ubuntu Precise) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in ifupdown package in Ubuntu: In Progress Status in ifupdown source package in Precise: New Status in ifupdown source package in Trusty: New Status in ifupdown source package in Vivid: New Status in ifupdown package in Debian: New Bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only "networking" script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script "network-interface" enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running network-interface (eth0) start/running network-interface (eth1) start/running --- Enabling bonding: Following ifenslave configuration example (/usr/share/doc/ifenslave/ examples/two_hotplug_ethernet), my /etc/network/interfaces has to look like this: --- auto eth1 iface eth1 inet manual bond-master bond0 auto eth2 iface eth2 inet manual bond-master bond0 auto bond0 iface bond0 inet static bond-mode 1 bond-miimon 100 bond-primary eth1 eth2 address 192.168.169.1 netmask 255.255.255.0 broadcast 192.168.169.255 --- Having both scripts running does not make any difference since we are missing "bond-slaves" keyword on slave interfaces, for ifenslave to work, and they are set to "manual". Ifenslave code: """ for slave in $BOND_SLAVES ; do ... # Ensure $slave is down. ip link set "$slave" down 2>/dev/null if ! sysfs_add slaves "$slave" 2>/dev/null ; then echo "Failed to enslave $slave to $BOND_MASTER. Is $BOND_MASTER ready and a bonding interface ?" >&2 else # Bring up slave if it is the target of an allow-bondX stanza. # This is usefull to bring up slaves that need extra setup. if [ -z "$(which ifquery)" ] || ifquery --allow \"$BOND_MASTER\" --list | grep -q $slave; then ifup $v
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
** Changed in: ifupdown (Ubuntu) Assignee: (unassigned) => Dariusz Gadomski (dgadomski) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in ifupdown package in Ubuntu: In Progress Status in ifupdown package in Debian: New Bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only "networking" script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script "network-interface" enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running network-interface (eth0) start/running network-interface (eth1) start/running --- Enabling bonding: Following ifenslave configuration example (/usr/share/doc/ifenslave/ examples/two_hotplug_ethernet), my /etc/network/interfaces has to look like this: --- auto eth1 iface eth1 inet manual bond-master bond0 auto eth2 iface eth2 inet manual bond-master bond0 auto bond0 iface bond0 inet static bond-mode 1 bond-miimon 100 bond-primary eth1 eth2 address 192.168.169.1 netmask 255.255.255.0 broadcast 192.168.169.255 --- Having both scripts running does not make any difference since we are missing "bond-slaves" keyword on slave interfaces, for ifenslave to work, and they are set to "manual". Ifenslave code: """ for slave in $BOND_SLAVES ; do ... # Ensure $slave is down. ip link set "$slave" down 2>/dev/null if ! sysfs_add slaves "$slave" 2>/dev/null ; then echo "Failed to enslave $slave to $BOND_MASTER. Is $BOND_MASTER ready and a bonding interface ?" >&2 else # Bring up slave if it is the target of an allow-bondX stanza. # This is usefull to bring up slaves that need extra setup. if [ -z "$(which ifquery)" ] || ifquery --allow \"$BOND_MASTER\" --list | grep -q $slave; then ifup $v --allow "$BOND_MASTER" "$slave" fi """ Without the keyword "bond-slaves" on the master interface declaration, ifenslave will NOT bring any slave interface up on the "master" interface ifup invocation. *** Part 1 So, having networking sysv init script AND upstart network-interface script
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
** Changed in: ifupdown (Ubuntu) Assignee: Rafael David Tinoco (inaddy) = (unassigned) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in ifupdown package in Ubuntu: In Progress Status in ifupdown package in Debian: New Bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only networking script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script network-interface enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running network-interface (eth0) start/running network-interface (eth1) start/running --- Enabling bonding: Following ifenslave configuration example (/usr/share/doc/ifenslave/ examples/two_hotplug_ethernet), my /etc/network/interfaces has to look like this: --- auto eth1 iface eth1 inet manual bond-master bond0 auto eth2 iface eth2 inet manual bond-master bond0 auto bond0 iface bond0 inet static bond-mode 1 bond-miimon 100 bond-primary eth1 eth2 address 192.168.169.1 netmask 255.255.255.0 broadcast 192.168.169.255 --- Having both scripts running does not make any difference since we are missing bond-slaves keyword on slave interfaces, for ifenslave to work, and they are set to manual. Ifenslave code: for slave in $BOND_SLAVES ; do ... # Ensure $slave is down. ip link set $slave down 2/dev/null if ! sysfs_add slaves $slave 2/dev/null ; then echo Failed to enslave $slave to $BOND_MASTER. Is $BOND_MASTER ready and a bonding interface ? 2 else # Bring up slave if it is the target of an allow-bondX stanza. # This is usefull to bring up slaves that need extra setup. if [ -z $(which ifquery) ] || ifquery --allow \$BOND_MASTER\ --list | grep -q $slave; then ifup $v --allow $BOND_MASTER $slave fi Without the keyword bond-slaves on the master interface declaration, ifenslave will NOT bring any slave interface up on the master interface ifup invocation. *** Part 1 So, having networking sysv init script AND upstart network-interface script running together... the following
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
Im getting back to this after sometime. After the discussion was brought to upstream we did not get feedback regarding proposed changes but investigating further it is clear that ifupdown is suffering from race conditions that cannot be solved simply by creating: 1) big lock - since its ifup/ifdown/ifquery are reentrant* 2) big lock - does not attend to interface order/priority for parallel executions** 3) fine-grained lock - does not attend interface order/priority for parallel executions** * could be solved by ENV variable being set not to lock childs) by up/down scripts. ** group of interfaces such as bridges and all interfaces connected to it, interfaces and all vlans connected to it Final approach here will be to guarantee: 1) interfaces should be locked independently on executions 2) locks have to respect interface hierarchy (locking group of inter-connected interfaces such as bridges/interfaces, interfaces/vlans) 3) all up/down scripts have to be reviewed after any locking mechanism change (deadlock by reentrant calls) IMO 1) stanzas should be created to group interfaces to be locked (for parallel executions) respecting hierarchy/order between them 2) locking/state have to be together and independent FINALLY The change to guarantee all that will involve code AND interfaces file change (for adding special stanzas to make sure appropriate order and locking is done during interfaces activation). It is not clear if this change will be smooth enough for a stable release update. If not I'll try to provide a PPA to address any needed code-change for those who suffer from this issue. BY NOW The only way to guarantee interfaces activation ordering (without suffering from intermittent race conditions like the one explained on this bug) would be to activate interface one by one outside sysv/upstart scripts OR to use pre/post commands with reentrant calls to ifupdown based on the desired order. Any comments here are much appreciated. Thank you Rafael Tinoco -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in “ifupdown” package in Ubuntu: In Progress Status in “ifupdown” package in Debian: New Bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only networking script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
** Tags added: cts -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in “ifupdown” package in Ubuntu: In Progress Status in “ifupdown” package in Debian: New Bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only networking script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script network-interface enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running network-interface (eth0) start/running network-interface (eth1) start/running --- Enabling bonding: Following ifenslave configuration example (/usr/share/doc/ifenslave/ examples/two_hotplug_ethernet), my /etc/network/interfaces has to look like this: --- auto eth1 iface eth1 inet manual bond-master bond0 auto eth2 iface eth2 inet manual bond-master bond0 auto bond0 iface bond0 inet static bond-mode 1 bond-miimon 100 bond-primary eth1 eth2 address 192.168.169.1 netmask 255.255.255.0 broadcast 192.168.169.255 --- Having both scripts running does not make any difference since we are missing bond-slaves keyword on slave interfaces, for ifenslave to work, and they are set to manual. Ifenslave code: for slave in $BOND_SLAVES ; do ... # Ensure $slave is down. ip link set $slave down 2/dev/null if ! sysfs_add slaves $slave 2/dev/null ; then echo Failed to enslave $slave to $BOND_MASTER. Is $BOND_MASTER ready and a bonding interface ? 2 else # Bring up slave if it is the target of an allow-bondX stanza. # This is usefull to bring up slaves that need extra setup. if [ -z $(which ifquery) ] || ifquery --allow \$BOND_MASTER\ --list | grep -q $slave; then ifup $v --allow $BOND_MASTER $slave fi Without the keyword bond-slaves on the master interface declaration, ifenslave will NOT bring any slave interface up on the master interface ifup invocation. *** Part 1 So, having networking sysv init script AND upstart network-interface script running together... the following example works: --- root@provisioned:~# cat /etc/network/interfaces
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
(unsubscribing ~ubuntu-sponsors per comment #13, please re-subscribe when patches are ready) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in “ifupdown” package in Ubuntu: In Progress Status in “ifupdown” package in Debian: New Bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only networking script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script network-interface enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running network-interface (eth0) start/running network-interface (eth1) start/running --- Enabling bonding: Following ifenslave configuration example (/usr/share/doc/ifenslave/ examples/two_hotplug_ethernet), my /etc/network/interfaces has to look like this: --- auto eth1 iface eth1 inet manual bond-master bond0 auto eth2 iface eth2 inet manual bond-master bond0 auto bond0 iface bond0 inet static bond-mode 1 bond-miimon 100 bond-primary eth1 eth2 address 192.168.169.1 netmask 255.255.255.0 broadcast 192.168.169.255 --- Having both scripts running does not make any difference since we are missing bond-slaves keyword on slave interfaces, for ifenslave to work, and they are set to manual. Ifenslave code: for slave in $BOND_SLAVES ; do ... # Ensure $slave is down. ip link set $slave down 2/dev/null if ! sysfs_add slaves $slave 2/dev/null ; then echo Failed to enslave $slave to $BOND_MASTER. Is $BOND_MASTER ready and a bonding interface ? 2 else # Bring up slave if it is the target of an allow-bondX stanza. # This is usefull to bring up slaves that need extra setup. if [ -z $(which ifquery) ] || ifquery --allow \$BOND_MASTER\ --list | grep -q $slave; then ifup $v --allow $BOND_MASTER $slave fi Without the keyword bond-slaves on the master interface declaration, ifenslave will NOT bring any slave interface up on the master interface ifup invocation. *** Part 1 So, having networking sysv init script AND upstart network-interface script running together... the
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
I have introduced one big lock for ifupdown. The ifup, ifdown or ifquery commands cannot be run simultaneously. Since SEVERAL ifupdown pre/post scripts do need to make reentrant calls do these commands I created on environment variable that disabled the locking when reentrant calls are made to these scripts. This way sysv and upstart networking scripts will never step into other's feet. Attaching fix for ifupdown. PS: This breaks even more ifenslave buggy behavior.. wait for next comments. ** Patch added: trusty_ifupdown_0.7.47.2ubuntu4.2.debdiff https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/1337873/+attachment/4200562/+files/trusty_ifupdown_0.7.47.2ubuntu4.2.debdiff -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in “ifupdown” package in Ubuntu: In Progress Status in “ifupdown” package in Debian: New Bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only networking script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script network-interface enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running network-interface (eth0) start/running network-interface (eth1) start/running --- Enabling bonding: Following ifenslave configuration example (/usr/share/doc/ifenslave/ examples/two_hotplug_ethernet), my /etc/network/interfaces has to look like this: --- auto eth1 iface eth1 inet manual bond-master bond0 auto eth2 iface eth2 inet manual bond-master bond0 auto bond0 iface bond0 inet static bond-mode 1 bond-miimon 100 bond-primary eth1 eth2 address 192.168.169.1 netmask 255.255.255.0 broadcast 192.168.169.255 --- Having both scripts running does not make any difference since we are missing bond-slaves keyword on slave interfaces, for ifenslave to work, and they are set to manual. Ifenslave code: for slave in $BOND_SLAVES ; do ... # Ensure $slave is down. ip link set $slave down 2/dev/null if ! sysfs_add slaves $slave 2/dev/null ; then echo Failed to enslave $slave to $BOND_MASTER. Is $BOND_MASTER ready and a bonding interface ? 2 else # Bring up
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
Let's try everything again from the beggining but now with a fixed ifupdown version (no race conditions between upstart and sysv scripts ). My interfaces file will be exactly the same as the one proposed for ifenslave examples: --- auto eth1 iface eth1 inet manual bond-master bond0 auto eth2 iface eth2 inet manual bond-master bond0 auto bond0 iface bond0 inet static bond-mode 1 bond-miimon 100 bond-primary eth1 eth2 address 192.168.169.1 netmask 255.255.255.0 broadcast 192.168.169.255 --- We do have bond0 created but still no bonding configured: --- root@provisioned:~# ifconfig bond0 bond0 Link encap:Ethernet HWaddr 62:64:29:45:df:ef BROADCAST MASTER MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) root@provisioned:~# cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: load balancing (round-robin) MII Status: down MII Polling Interval (ms): 0 Up Delay (ms): 0 Down Delay (ms): 0 --- -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in “ifupdown” package in Ubuntu: In Progress Status in “ifupdown” package in Debian: New Bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only networking script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script network-interface enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running network-interface (eth0) start/running network-interface (eth1) start/running --- Enabling bonding: Following ifenslave configuration example (/usr/share/doc/ifenslave/ examples/two_hotplug_ethernet), my /etc/network/interfaces has to look like this: --- auto eth1 iface eth1 inet manual bond-master bond0 auto eth2 iface eth2 inet manual bond-master bond0 auto bond0 iface bond0 inet static bond-mode 1 bond-miimon 100 bond-primary eth1 eth2 address 192.168.169.1 netmask 255.255.255.0 broadcast 192.168.169.255 --- Having both scripts
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
And you can check that upstart got deadlocked: --- root@provisioned:~# ps -ef | grep ifup root 618 1 0 10:21 ?00:00:00 ifup --allow auto eth2 root 619 1 0 10:21 ?00:00:00 ifup --allow auto eth1 root 620 1 0 10:21 ?00:00:00 ifup --allow auto lo root 621 1 0 10:21 ?00:00:00 ifup --allow auto eth0 root 726 1 0 10:21 ?00:00:00 ifup --allow auto bond0 root 739 733 0 10:21 ?00:00:00 ifup -a root@provisioned:~# for i in `ps -ef | grep ifup | grep -v grep | awk '{print $2}'`; do echo $i; cat /proc/$i/environ; done 618 ...UPSTART_INSTANCE=eth2... ...UPSTART_INSTANCE=eth1... ...UPSTART_INSTANCE=lo... ...UPSTART_INSTANCE=eth0... ...INSTANCE=UPSTART_JOB=networking... --- As I said before, sysv scripts and upstart scripts were depending on each other to run in parallel (unfortunately with race conditions) to configure bonding. We can see here that one of upstart networking processes (networking or network-instance) got the lock and is on an infinite loop waiting for other instance.. who is waiting for the lock. --- root@provisioned:~# ps -ef | grep ifenslave root 647 641 0 10:21 ?00:00:00 /bin/sh /etc/network/if-pre-up.d/ifenslave --- -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in “ifupdown” package in Ubuntu: In Progress Status in “ifupdown” package in Debian: New Bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only networking script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script network-interface enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running network-interface (eth0) start/running network-interface (eth1) start/running --- Enabling bonding: Following ifenslave configuration example (/usr/share/doc/ifenslave/ examples/two_hotplug_ethernet), my /etc/network/interfaces has to look like this: --- auto eth1 iface eth1 inet manual bond-master bond0 auto eth2 iface eth2 inet manual bond-master bond0 auto bond0 iface bond0 inet static bond-mode 1 bond-miimon 100 bond-primary eth1 eth2 address
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
YES! --- root@provisioned:~# pstree -a ... ├─ifup --allow auto eth2 │ └─sh -c run-parts /etc/network/if-pre-up.d │ └─run-parts /etc/network/if-pre-up.d │ └─ifenslave /etc/network/if-pre-up.d/ifenslave │ └─sleep 0.1 --- One slave interface, eth2 in this case, got the ifupdown lock and is running an infite loop waiting for the master bonding interface which will never run without the lock. Resuming: So bonding had to have both networking scripts running (network- interface and networking) to work AND having both scripts running would case race conditions sometime. Disabling one of the scripts would also cause race condition if right triggers are set (like i showed in this example). Fixing ifupdown race conditions led me to realize ifenslave is taking wrong decisions and can cause deadlocks. Ifenslave must be fixed together... * wait for next comments. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in “ifupdown” package in Ubuntu: In Progress Status in “ifupdown” package in Debian: New Bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only networking script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script network-interface enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running network-interface (eth0) start/running network-interface (eth1) start/running --- Enabling bonding: Following ifenslave configuration example (/usr/share/doc/ifenslave/ examples/two_hotplug_ethernet), my /etc/network/interfaces has to look like this: --- auto eth1 iface eth1 inet manual bond-master bond0 auto eth2 iface eth2 inet manual bond-master bond0 auto bond0 iface bond0 inet static bond-mode 1 bond-miimon 100 bond-primary eth1 eth2 address 192.168.169.1 netmask 255.255.255.0 broadcast 192.168.169.255 --- Having both scripts running does not make any difference since we are missing bond-slaves keyword on slave interfaces, for ifenslave to work, and they are set to manual. Ifenslave code: for slave in $BOND_SLAVES ; do ... # Ensure $slave is down. ip link set $slave down
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
Checking Ubuntu bzr tree... --- inaddy@trusty.00064811:/bugs/00064811/sources/bazaar/ubuntu/$ git clone bzr::lp:ubuntu/ifenslave Cloning into 'ifenslave'... Most recent Ubuntu version: 3 Packaging branch version: 2.5ubuntu1 Packaging branch status: OUT-OF-DATE Most recent Ubuntu version: 3 Packaging branch version: 2.5ubuntu1 Packaging branch status: OUT-OF-DATE Most recent Ubuntu version: 3 Packaging branch version: 2.5ubuntu1 Packaging branch status: OUT-OF-DATE Most recent Ubuntu version: 3 Packaging branch version: 2.5ubuntu1 Packaging branch status: OUT-OF-DATE Checking connectivity... done. --- --- inaddy@trusty.00064811:/bugs/00064811/sources/bazaar/ubuntu/ifenslave$ git tag | grep -e 2.4 -e 2.5 2.4 2.4ubuntu1 2.5 2.5ubuntu1 --- --- inaddy@trusty.00064811:/bugs/00064811/sources/bazaar/ubuntu/ifenslave$ git checkout 2.4 Previous HEAD position was 64392a5... Re-apply Ubuntu delta to new source. HEAD is now at 1d22c9b... Added ifenslave-2.6.prerm to remove dangling alternatives (Closes: #736668). Thanks to Andreas Beckmann. inaddy@trusty.00064811:/bugs/00064811/sources/bazaar/ubuntu/ifenslave$ git checkout 2.4ubuntu1 Previous HEAD position was 1d22c9b... Added ifenslave-2.6.prerm to remove dangling alternatives (Closes: #736668). Thanks to Andreas Beckmann. HEAD is now at 64392a5... Re-apply Ubuntu delta to new source. --- --- inaddy@trusty.00064811:/bugs/00064811/sources/bazaar/ubuntu/ifenslave$ git checkout 2.5 Previous HEAD position was 64392a5... Re-apply Ubuntu delta to new source. HEAD is now at 1701e16... * ifupdown (= 0.7.46) compatibility update (Closes: #742410). Thanks to Andrew Shadura. * Added versioned Depends on ifupdown (= 0.7.46). inaddy@trusty.00064811:/bugs/00064811/sources/bazaar/ubuntu/ifenslave$ git checkout 2.5ubuntu1 Previous HEAD position was 1701e16... * ifupdown (= 0.7.46) compatibility update (Closes: #742410). Thanks to Andrew Shadura. * Added versioned Depends on ifupdown (= 0.7.46). HEAD is now at e47d568... * Merge from Debian unstable. Remaining changes: - Upstart event based bond bringup: + Drop ethernet+wifi example + Drop two_ethernet example + Update ethernet+hotplug_wifi example + Update two_hotplug_ethernet example + Update pre-up and post-down scripts for event bringup + Update README.Debian examples - Update scripts to use /run/network/ifstate instead of /etc/network/run/ifstate --- I could see that we diverged from upstream code (Debian's) in favor of some other modifications. Ubuntu fix would be different then a possible upstream fix. We have nowadays: inaddy@trusty.00064811:/bugs/00064811/sources/trusty/ifenslave$ rmadison ifenslave ifenslave | 2.4ubuntu1 | trusty | source, all ifenslave | 2.5ubuntu1 | utopic | source, all Both need fixes for this particular case. * wait for next comments. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in “ifupdown” package in Ubuntu: In Progress Status in “ifupdown” package in Debian: New Bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
After talking to Stéphane Graber, from Ubuntu Core Foundations Team, we decided that I should implement independent locking for every interface (like I have already proposed to Debian upstream project) and to implement locking mechanisms for dependent interfaces inside the hooks. So: 1) ifupdown would lock every given interface (or all if -a is given). 2) locking for child interfaces (slaves for bonding, attached to bridges, ...) is going to be done inside hooks. Today most important hooks for ifupdown are: bridging, vlan and bonding. I have to guarantee those 3 are ok with any change made to ifupdown tool. * wait for next comments/patches/actions. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in “ifupdown” package in Ubuntu: In Progress Status in “ifupdown” package in Debian: New Bug description: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only networking script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting networking start/running --- OR only the script network-interface enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running network-interface (eth0) start/running network-interface (eth1) start/running --- Enabling bonding: Following ifenslave configuration example (/usr/share/doc/ifenslave/ examples/two_hotplug_ethernet), my /etc/network/interfaces has to look like this: --- auto eth1 iface eth1 inet manual bond-master bond0 auto eth2 iface eth2 inet manual bond-master bond0 auto bond0 iface bond0 inet static bond-mode 1 bond-miimon 100 bond-primary eth1 eth2 address 192.168.169.1 netmask 255.255.255.0 broadcast 192.168.169.255 --- Having both scripts running does not make any difference since we are missing bond-slaves keyword on slave interfaces, for ifenslave to work, and they are set to manual. Ifenslave code: for slave in $BOND_SLAVES ; do ... # Ensure $slave is down. ip link set $slave down 2/dev/null if ! sysfs_add slaves $slave 2/dev/null ; then echo Failed to enslave $slave to $BOND_MASTER. Is $BOND_MASTER ready and a bonding interface ? 2 else # Bring up slave if it is the target of an
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
** Description changed: - It was brought to my attention (by others) that ifupdown runs into race - conditions on some specific cases. - - [Impact] - - When trying to deploy many servers at once (higher chances of happening) - or from time-to-time, like any other intermittent race-condition. - Interfaces are not brought up like they should and this has a big impact - for servers that cannot rely on network start scripts. - - The problem is caused by a race condition when init(upstart) starts up - network interfaces in parallel. - - [Test Case] - - Use attached script to reproduce the error (it might take some hours, in - a single virtual machine, for the error to occur). - * please consider my bonding examples are using eth1 and eth2 as slave - interfaces. - - ifupdown some race conditions explained bellow: + interfaces. + + ifupdown some race conditions explained bellow. ifenslave does not + behave well with sysv networking and upstart network-interface scripts + running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. - 1-1. Wait for locking ifstate.lock - file. + 1-1. Wait for locking ifstate.lock + file. 1-2. Read ifstate file to check - the target NIC. + the target NIC. 1-3. close(=release) ifstate.lock - file. + file. 1-4. Judge that the target NIC - isn't processed. - 1-2. Read ifstate file to check - the target NIC. - 1-3. close(=release) ifstate.lock - file. - 1-4. Judge that the target NIC - isn't processed. + isn't processed. + 1-2. Read ifstate file to check +the target NIC. + 1-3. close(=release) ifstate.lock +file. + 1-4. Judge that the target NIC +isn't processed. 2. Lock and update ifstate file. - Release the lock. - 2. Lock and update ifstate file. - Release the lock. +Release the lock. + 2. Lock and update ifstate file. + Release the lock. !!! + + to be explained !!! case 2) - (a) ifenslave of eth0 (b) ifenslave of eth0 + (a) ifenslave of eth0 (b) ifenslave of eth0 -- - 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. + 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to - /sys/class/net/bond0/bonding - /slaves then NIC gets up - 4. Link down the target NIC. - 5. Fails to write NIC id to - /sys/class/net/bond0/bonding/ - slaves it is already written. +/sys/class/net/bond0/bonding +/slaves then NIC gets up + 4. Link down the target NIC. + 5. Fails to write NIC id to + /sys/class/net/bond0/bonding/ + slaves it is already written. !!! + + # + + My setup: + + root@provisioned:~# cat /etc/modprobe.d/bonding.conf + alias bond0 bonding options bonding mode=1 arp_interval=2000 + + Both, /etc/init.d/networking and upstart network-interface begin + enabled. + + Beginning: + + root@provisioned:~# cat /etc/network/interfaces + # /etc/network/interfaces + + auto lo + iface lo inet loopback + + auto eth0 + iface eth0 inet dhcp + + I'm able to boot with both scripts (networking and network-interface + enabled) with no problem. I can also boot with only networking + script enabled: + + --- + root@provisioned:~# initctl list | grep network + network-interface stop/waiting + network-interface-security (networking) start/running + networking start/running + network-interface-container stop/waiting + --- + + OR only the script network-interface enabled: + + --- + root@provisioned:~# initctl list | grep network + network-interface (eth2) start/running + network-interface (lo) start/running + network-interface (eth0) start/running + network-interface (eth1) start/running + networking start/running + network-interface-container stop/waiting + --- + + Enabling bonding: +
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
CORRECT WAY OF SETTING INTERFACES FILE FOR BONDING: 1) This model has race conditions. 2) YOU HAVE to have both scripts running (networking and network-interfaces) # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp auto eth1 iface eth1 inet manual bond-master bond0 auto eth2 iface eth2 inet manual bond-master bond0 auto bond0 iface bond0 inet static bond-mode 1 bond-miimon 100 bond-primary eth1 bond-slaves eth1 eth2 address 192.168.169.1 netmask 255.255.255.0 broadcast 192.168.169.255 You can expect this to fail from time to time but it works... working on a fix for this. ** Description changed: * please consider my bonding examples are using eth1 and eth2 as slave interfaces. ifupdown some race conditions explained bellow. ifenslave does not behave well with sysv networking and upstart network-interface scripts running together. case 1) (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. !!! to be explained !!! case 2) (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. !!! # My setup: root@provisioned:~# cat /etc/modprobe.d/bonding.conf alias bond0 bonding options bonding mode=1 arp_interval=2000 Both, /etc/init.d/networking and upstart network-interface begin enabled. Beginning: root@provisioned:~# cat /etc/network/interfaces # /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp I'm able to boot with both scripts (networking and network-interface enabled) with no problem. I can also boot with only networking script enabled: --- root@provisioned:~# initctl list | grep network network-interface stop/waiting - network-interface-security (networking) start/running networking start/running - network-interface-container stop/waiting --- OR only the script network-interface enabled: --- root@provisioned:~# initctl list | grep network network-interface (eth2) start/running network-interface (lo) start/running network-interface (eth0) start/running network-interface (eth1) start/running - networking start/running - network-interface-container stop/waiting --- Enabling bonding: Following ifenslave configuration example (/usr/share/doc/ifenslave/ examples/two_hotplug_ethernet), my /etc/network/interfaces has to look like this: --- auto eth1 iface eth1 inet manual bond-master bond0 auto eth2 iface eth2 inet manual bond-master bond0 auto bond0 iface bond0 inet static bond-mode 1 bond-miimon 100 bond-primary eth1 eth2 address 192.168.169.1 netmask 255.255.255.0 broadcast 192.168.169.255 --- Having both scripts running does not make any difference since we are missing bond-slaves keyword on slave interfaces, for ifenslave to work, and they are set to manual. Ifenslave code: for slave in $BOND_SLAVES ; do ... # Ensure $slave is down. ip link set $slave down 2/dev/null if ! sysfs_add slaves $slave 2/dev/null ; then echo Failed to enslave $slave to $BOND_MASTER. Is $BOND_MASTER ready and a bonding interface ? 2 else # Bring up slave if it is the target of an allow-bondX stanza. # This is usefull to bring up slaves that need extra setup. if [ -z $(which ifquery) ] || ifquery --allow \$BOND_MASTER\
[Touch-packages] [Bug 1337873] Re: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition
This bug is being fixed upstream together with the developer. I'll provide the fix here as soon as it gets accepted upstream: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=753755 Thank you. ** Patch removed: ifupdown_0.7.47.2ubuntu4.2~lp1337873.diff https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/1337873/+attachment/4145680/+files/ifupdown_0.7.47.2ubuntu4.2%7Elp1337873.diff -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1337873 Title: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition Status in “ifupdown” package in Ubuntu: In Progress Status in “ifupdown” package in Debian: New Bug description: It was brought to my attention (by others) that ifupdown runs into race conditions on some specific cases. [Impact] When trying to deploy many servers at once (higher chances of happening) or from time-to-time, like any other intermittent race- condition. Interfaces are not brought up like they should and this has a big impact for servers that cannot rely on network start scripts. The problem is caused by a race condition when init(upstart) starts up network interfaces in parallel. [Test Case] Use attached script to reproduce the error (it might take some hours, in a single virtual machine, for the error to occur). (example 1) *** sequence to trigger race-condition *** (a) ifup eth0 (b) ifup -a for eth0 - 1-1. Lock ifstate.lock file. 1-1. Wait for locking ifstate.lock file. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 1-2. Read ifstate file to check the target NIC. 1-3. close(=release) ifstate.lock file. 1-4. Judge that the target NIC isn't processed. 2. Lock and update ifstate file. Release the lock. 2. Lock and update ifstate file. Release the lock. (example 2) Bonding device using eth0. ifenslave for eth0 is also executed in parallel, eth0 remains down. *** sequence to trigger race-condition *** (a) ifenslave of eth0 (b) ifenslave of eth0 -- 3. Execute ifenslave of eth0. 3. Execute ifenslave of eth0. 4. Link down the target NIC. 5. Write NIC id to /sys/class/net/bond0/bonding /slaves then NIC gets up 4. Link down the target NIC. 5. Fails to write NIC id to /sys/class/net/bond0/bonding/ slaves it is already written. (example 3) bonding is not set to active-backup as defined in config file: When the init(upstart) executes if-pre-up.d/ifenslave script and if-pre- up.d/vlan script for bond0 device in parallel, the if-pre- up.d/ifenslave script fails to change the bonding mode with a error message, bonding: unable to update mode of bond0 because interface is up.. *** sequence to trigger race-condition *** (a)ifup bond0 (b)ifup -a --- 1. Update statefile about bond0. 1. Does nothing about bond0 because statefile is already updated about it. 2. ifenslave::setup_master() sysfs_change_down mode 1 and link down bond0. 2. Link up bond0 by the vlan script on the processing for linking up bond0.201(*1). 3. echo 1 .../mode fails. [ /etc/network/if-pre-up.d/vlan ] 46 if [ -n $IF_VLAN_RAW_DEVICE ] [ ! -d /sys/class/net/$IFACE ]; then 47 if [ ! -x /sbin/vconfig ]; then 48 exit 0 49 fi 50 if ! ip link show dev $IF_VLAN_RAW_DEVICE /dev/null; then 51 echo $IF_VLAN_RAW_DEVICE does not exist, unable to create $IFACE 52 exit 1 53 fi 54 ip link set up dev $IF_VLAN_RAW_DEVICE -- (*1). 55 vconfig add $IF_VLAN_RAW_DEVICE $VLANID 56 fi [Regression Potential] * Attaching proposed patch (for upstream as well) and describing potential later on today. [Other Info] Example: [ /etc/network/interfaces ] auto lo iface lo inet