[Bug 1071910] Re: lxc stop will hang forever
*** This bug is a duplicate of bug 1073238 *** https://bugs.launchpad.net/bugs/1073238 The title of this bug is 'lxc stop will hang forever' - are you still seeing that as well? In both this and 1073238, you claimed that 3.7 fixed it for you. You're saying that lvremove hanging happened in 3.5, was fixed in 3.7, and seems to have broken again in 3.8, is that right? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1071910 Title: lxc stop will hang forever To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1071910/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1071910] Re: lxc stop will hang forever
*** This bug is a duplicate of bug 1073238 *** https://bugs.launchpad.net/bugs/1073238 I don't believe that this is a duplicate of Bug #1073238. I'm able to replicate this relatively often on 3.5 and (I believe 3.8 from 13.04). lvremove remains unkillalble. I'm not using cgroups so I don't think this is related. I'm going to see if I can trigger a recovery using dmsetup. I also see the "/sbin/dmeventd: stat failed: No such file or directory", but not every time. It is intermittent that it reports that to me via the command line. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1071910 Title: lxc stop will hang forever To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1071910/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1071910] Re: lxc stop will hang forever
*** This bug is a duplicate of bug 1073238 *** https://bugs.launchpad.net/bugs/1073238 I hope this helps someone some day: >From what I observed: lvremove was failing to remove the COW device because udev was occasionally doing a vgscan at the same time. This then cause certain components to remain in a suspended state blocking lvm commands against them. The solution was to detect the error "Unable to deactivate" from stderr of lvremove, then call dmsetup resume on all related suspended devices, in the above post it is only "vmg1-vm" that needed to be resumed. Then wait a little bit for the other lvm commands to finish, then call lvremove again. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1071910 Title: lxc stop will hang forever To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1071910/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1071910] Re: lxc stop will hang forever
*** This bug is a duplicate of bug 1073238 *** https://bugs.launchpad.net/bugs/1073238 I might try 3.7 later and see if I can get the memory cgroup working again, but I'm not sure if I really need it. 3.6.3 is working with the memory cgroup disabled, I tested with over 400 iterations, I have a kernel bug post as well and posted this info there. I'm going to try and fix the lvremove issue now, I believe it is caused by this: udevd[1562]: timeout: killing 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' [1574] udevd[1562]: 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' [1574] terminated by signal 9 (Killed) I tried adding --noudevsync as one user suggested to: /lib/udev/rules.d/85-lvm2.rules but I still get the issue, and then LVM and lvremove hang udevd[6757]: timeout: killing 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange --noudevsync -a y'' [7383] udevd[6757]: 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange --noudevsync -a y'' [7383] terminated by signal 9 (Killed) lvremove first outputs: Using logical volume(s) on command line Archiving volume group "vmg1" metadata (seqno 594). Removing snapshot snap Found volume group "vmg1" Found volume group "vmg1" Loading vmg1-vm table (252:0) Loading vmg1-snap table (252:1) /sbin/dmeventd: stat failed: No such file or directory vmg1/snapshot0 already not monitored. Suspending vmg1-vm (252:0) with device flush Suspending vmg1-snap (252:1) with device flush Suspending vmg1-vm-real (252:2) with device flush Suspending vmg1-snap-cow (252:3) with device flush Found volume group "vmg1" Resuming vmg1-snap-cow (252:3) Resuming vmg1-vm-real (252:2) Resuming vmg1-snap (252:1) Unable to deactivate open vmg1-snap-cow (252:3) Failed to resume snap. libdevmapper exiting with 1 device(s) still suspended. And then my script calls it a second time which is when it hangs -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1071910 Title: lxc stop will hang forever To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1071910/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 1071910] Re: lxc stop will hang forever
*** This bug is a duplicate of bug 1073238 *** https://bugs.launchpad.net/bugs/1073238 Quoting Tim (ice...@gmail.com): > *** This bug is a duplicate of bug 1073238 *** > https://bugs.launchpad.net/bugs/1073238 > > :o the memory cgroup dissapeared when I switched to 3.6.3, is it also > gone in 3.7, is that why you asked me to comment out the memory setting? No, in the kernel I tested it was there, but seemed broken. I couldn't write into the memory swap limit file. But I did vaguely remember there being some restrictions on what could be written there so wanted to make sure you had the same problem. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1071910 Title: lxc stop will hang forever To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1071910/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1071910] Re: lxc stop will hang forever
*** This bug is a duplicate of bug 1073238 *** https://bugs.launchpad.net/bugs/1073238 :o the memory cgroup dissapeared when I switched to 3.6.3, is it also gone in 3.7, is that why you asked me to comment out the memory setting? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1071910 Title: lxc stop will hang forever To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1071910/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1071910] Re: lxc stop will hang forever
*** This bug is a duplicate of bug 1073238 *** https://bugs.launchpad.net/bugs/1073238 I'll test that soon too :) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1071910 Title: lxc stop will hang forever To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1071910/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1071910] Re: lxc stop will hang forever
*** This bug is a duplicate of bug 1073238 *** https://bugs.launchpad.net/bugs/1073238 D'oh, I do believe I tested on a 3.7 kernel, so that would make sense :) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1071910 Title: lxc stop will hang forever To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1071910/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1071910] Re: lxc stop will hang forever
Thanks for your suggestion Serge, I believe I just solved the issue, except for the part I was seeing with lvremove which appears to have some relation to: udevd[7497]: timeout: killing 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' [11645] udevd[7497]: 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' [11645] terminated by signal 9 (Killed) I solved the lxc hanging issue by upgrading to kernel 3.5.7 from 3.5.0 - I tested over 120 iterations without issue, but I'll try your suggestion and re-open this bug if my testing was insufficient ** Changed in: lxc (Ubuntu) Status: Incomplete => Fix Released -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1071910 Title: lxc stop will hang forever To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1071910/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1071910] Re: lxc stop will hang forever
Can you confirm whether commenting out the line: lxc.cgroup.memory.memsw.limit_in_bytes = 1024M lets the container start? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1071910 Title: lxc stop will hang forever To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1071910/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1071910] Re: lxc stop will hang forever
** Description changed: Background: This is issue occurs during an automated process and occurs with a 1/20 chance per iteration I have one lxc-container on the machine It is backed with an lvm2 snapshot Running on ubuntu 12.10 on ec2 small instance - upgraded from 12.04 fresh instance This is a new issue that has occurred after migrating my code from 11.10 Process: create snapshot "lvcreate" mount snapshot "mount" lxc-start do actions in container lxc-stop unmount snapshot "umount" remove snapshot "lvremove" -repeat The issue can occur at either lxc-stop or lvremove. when it occurs with lxc-stop: ps -A reveals that lxc-start is still running along with kdmflush, kjournald, and init that appears to be the init process for the container kdmflush, kjournald, init or it's sub-processes cannot be killed with "kill -9 pid" but lxc-start can when it occurs with lvremove it occurs after lvremove is called again after failing the first time with stderr: Using logical volume(s) on command line - Archiving volume group "vmg1" metadata (seqno 272). - Removing snapshot snap - Found volume group "vmg1" - Found volume group "vmg1" - Loading vmg1-vm table (252:0) - Loading vmg1-snap table (252:1) - /sbin/dmeventd: stat failed: No such file or directory - vmg1/snapshot0 already not monitored. - Suspending vmg1-vm (252:0) with device flush - Suspending vmg1-snap (252:1) with device flush - Suspending vmg1-vm-real (252:2) with device flush - Suspending vmg1-snap-cow (252:3) with device flush - Found volume group "vmg1" - Resuming vmg1-snap-cow (252:3) - Resuming vmg1-vm-real (252:2) - Resuming vmg1-snap (252:1) - Removing vmg1-snap-cow (252:3) - device-mapper: remove ioctl on failed: Device or resource busy - Unable to deactivate vmg1-snap-cow (252:3) - Failed to resume snap. - libdevmapper exiting with 1 device(s) still suspended. + Archiving volume group "vmg1" metadata (seqno 272). + Removing snapshot snap + Found volume group "vmg1" + Found volume group "vmg1" + Loading vmg1-vm table (252:0) + Loading vmg1-snap table (252:1) + /sbin/dmeventd: stat failed: No such file or directory + vmg1/snapshot0 already not monitored. + Suspending vmg1-vm (252:0) with device flush + Suspending vmg1-snap (252:1) with device flush + Suspending vmg1-vm-real (252:2) with device flush + Suspending vmg1-snap-cow (252:3) with device flush + Found volume group "vmg1" + Resuming vmg1-snap-cow (252:3) + Resuming vmg1-vm-real (252:2) + Resuming vmg1-snap (252:1) + Removing vmg1-snap-cow (252:3) + device-mapper: remove ioctl on failed: Device or resource busy + Unable to deactivate vmg1-snap-cow (252:3) + Failed to resume snap. + libdevmapper exiting with 1 device(s) still suspended. lvremove spawns the lvm process and neither can be killed with "kill -9 pid" which indicates to me that they are waiting for something from the kernel, and I am guessing this happens because of the same reason why lxc-stop also hangs, and the containers processes can not be killed. - This is all I can report for now, but I'll try getting some log info - from lxc next Friday, let me know if you have any suggestions in the - meantime. + Here is an excerpt from the syslog - lxc-stop hangs because of this + kernel error, and the error doesn't always occur on the cat command, it + can happen on others as well. + + --START + Oct 30 14:43:12 domU-12-31-39-14-64-79 kernel: [ 1094.406366] kjournald starting. Commit interval 5 seconds + Oct 30 14:43:12 domU-12-31-39-14-64-79 kernel: [ 1094.406929] EXT3-fs (dm-1): using internal journal + Oct 30 14:43:12 domU-12-31-39-14-64-79 kernel: [ 1094.406931] EXT3-fs (dm-1): mounted filesystem with ordered data mode + Oct 30 14:43:12 domU-12-31-39-14-64-79 kernel: [ 1094.435218] device vethyurrCc entered promiscuous mode + Oct 30 14:43:12 domU-12-31-39-14-64-79 kernel: [ 1094.435613] IPv6: ADDRCONF(NETDEV_UP): vethyurrCc: link is not ready + Oct 30 14:43:12 domU-12-31-39-14-64-79 kernel: [ 1094.534518] IPv6: ADDRCONF(NETDEV_CHANGE): vethyurrCc: link becomes ready + Oct 30 14:43:12 domU-12-31-39-14-64-79 kernel: [ 1094.534543] br0: port 1(vethyurrCc) entered forwarding state + Oct 30 14:43:12 domU-12-31-39-14-64-79 kernel: [ 1094.534547] br0: port 1(vethyurrCc) entered forwarding state + + --STOP + Oct 30 14:43:17 domU-12-31-39-14-64-79 kernel: [ 1099.112881] br0: port 1(vethyurrCc) entered disabled state + Oct 30 14:43:17 domU-12-31-39-14-64-79 kernel: [ 1099.115187] device vethyurrCc left promiscuous mode + Oct 30 14:43:17 domU-12-31-39-14-64-79 kernel: [ 1099.115190] br0: port 1(vethyurrCc) entered disabled state + + --START + Oct 30 14:43:18 domU-12-31-39-14-64-79 kernel: [ 1100.188337] kjournald starting. Commit interval 5 seconds + Oct 30 14:43:18 domU-12-31-39-14-64-79 kernel: [ 1100.188852] EXT
[Bug 1071910] Re: lxc stop will hang forever
typo in container config, actually using: lxc.mount = /vms/vm1/fstab instead of lxc.mount = /vm1/vm1/fstab -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1071910 Title: lxc stop will hang forever To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1071910/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1071910] Re: lxc stop will hang forever
- Container config:
lxc.utsname = vm1
lxc.tty = 4
lxc.pts = 1024
lxc.rootfs = /vms/vm1/rootfs
lxc.mount = /vm1/vm1/fstab
lxc.network.type = veth
lxc.network.flags = up
lxc.network.name = eth0
lxc.network.link = br0
lxc.network.ipv4 = 192.168.3.2
# Set system resource limitations
lxc.cgroup.cpuset.cpus = 0
lxc.cgroup.cpu.shares = 1
lxc.cgroup.memory.limit_in_bytes = 512M
lxc.cgroup.memory.memsw.limit_in_bytes = 1024M
lxc.cgroup.devices.deny = a
# /dev/null and zero
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
# consoles
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 4:0 rwm
lxc.cgroup.devices.allow = c 4:1 rwm
# /dev/{,u}random
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 5:2 rwm
# rtc
lxc.cgroup.devices.allow = c 254:0 rwm
- /vms/vm1/fstab:
proc/vms/vm1/rootfs/proc procnodev,noexec,nosuid 0 0
devpts /vms/vm1/rootfs/dev/pts devpts defaults 0 0
sysfs /vms/vm1/rootfs/sys sysfs defaults 0 0
- Container creation command:
lxc-create -n vm1 -f /vms/vm1/config
- Stop Script:
lxc-stop -n vm1
umount /vms/vm1/rootfs
lvremove -f /dev/vmg1/snap
- Start Script:
lvcreate -L512M -s -n snap /dev/vmg1/vm
mount /dev/vmg1/snap /vms/vm1/rootfs
lxc-start -n vm1 -d
I needed to have a clean file-system for the container that had a different set
of software available than the host OS.
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to lxc in Ubuntu.
https://bugs.launchpad.net/bugs/1071910
Title:
lxc stop will hang forever
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1071910/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1071910] Re: lxc stop will hang forever
Could you please post the script which triggers this? Note that you shouldn't need to manually mount and umount the lvm partitions. If you've create the container with either 'lxc-create -t ubuntu -B lvm -n q1' or 'lxc-clone -s -o q1 -n q2', then you can simply 'lxc-start -n q1' (or q2) to start the container. (I use lxc with lvm quite a bit and have not seen this) ** Changed in: lxc (Ubuntu) Importance: Undecided => Medium ** Changed in: lxc (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1071910 Title: lxc stop will hang forever To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1071910/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
