Bug#598149: [Pkg-libvirt-maintainers] Bug#598149: libvirt-bin: autostart fails on reboot
Hi Jean, On Sun, Sep 26, 2010 at 11:25:48PM +0200, Jean-Louis Dupond wrote: Subject: libvirt-bin: autostart fails on reboot Package: libvirt-bin Version: 0.8.3-1 Severity: normal I have a routed setup of kvm (with libvirt). My tun devices are created in /etc/network/interfaces. auto tap0 iface tap0 inet static address xxx.xxx.xxx.xxx netmask 255.255.255.255 tunctl_user libvirt-qemu up route add xxx.xxx.xxx.xxx/32 dev tap0 But when I reboot my host server, and libvirt starts, it can't start the guests. I get the following error: Sep 21 21:28:51 cloud libvirtd: 21:28:51.046: error : qemudReadLogOutput:2118 : internal error Process exited while reading console log output: char device redirected to /dev/pts/1#012could not open /dev/net/tun: Permission denied#012 Sep 21 21:28:51 cloud libvirtd: 21:28:51.047: error : qemuAutostartDomain:827 : Failed to autostart VM 'dupondje': internal error Process exited while reading console log output: char device redirected to /dev/pts/1#012could not open /dev/net/tun: Permission denied#012 It seems to fail because the tap0 doesn't exist yet, and libvirt tries to make it. If I restart libvirt-bin after like 30 seconds, the devices get started without any issue. So it seems like there is some race condition, that makes libvirt-bin start before the network is initialized. Are you using dependency based booting? Please check if the network is fully up when libvirt-bin starts (e.g. by dumping the output of ifconfing in the init script). It should be since we have a dependency on $network in the init script. Cheers, -- Guido -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#598149: [Pkg-libvirt-maintainers] Bug#598149: libvirt-bin: autostart fails on reboot
On Mon, 27 Sep 2010 09:06:22 +0200, Guido Günther a...@sigxcpu.org wrote: Hi Jean, Are you using dependency based booting? Please check if the network is fully up when libvirt-bin starts (e.g. by dumping the output of ifconfing in the init script). It should be since we have a dependency on $network in the init script. Cheers, -- Guido Hi Guido, I am using squeeze with dependency based booting. So I modified the libvirt-bin in init.d: start) if check_start_libvirtd_option; then log_daemon_msg Starting $DESC $NAME if running ; then log_progress_msg already running log_end_msg 0 exit 0 fi rm -f /var/run/libvirtd.pid /sbin/ifconfig /root/if-boot start-stop-daemon --start --quiet --pidfile $PIDFILE \ --exec $DAEMON -- $libvirtd_opts if running; then log_end_msg 0 else log_end_msg 1 fi fi ;; After a reboot I checked /root/if-boot. All the tap devices were created successfully, but still I got the following error: Sep 27 10:21:22 cloud libvirtd: 10:21:22.407: error : qemudReadLogOutput:2118 : internal error Process exited while reading console log output: char device redirected to /dev/pts/1#012could not open /dev/net/tun: Permission denied#012 Sep 27 10:21:22 cloud libvirtd: 10:21:22.407: error : qemuAutostartDomain:827 : Failed to autostart VM 'dupondje': internal error Process exited while reading console log output: char device redirected to /dev/pts/1#012could not open /dev/net/tun: Permission denied#012 What I also see in the libvirt logs: On reboot it starts with: LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin HOME=/ QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 1024 -smp 2,sockets=2,cores=1,threads=1 -name dupondje -uuid 28e9972d-c3ef-dfff-792e-5f80cd51a68a -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/dupondje.monitor,server,nowait -mon chardev=monitor,mode=readline -rtc base=utc -boot dc -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/var/lib/libvirt/images/dupondje,if=none,id=drive-virtio-disk0,boot=on,format=raw -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -device virtio-net-pci,vlan=0,id=net0,mac=52:54:00:xxx:xx:xx,bus=pci.0,addr=0x6 -net tap,ifname=tap2,script=no,vlan=0,name=hostnet0 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -vnc 127.0.0.1:0 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 char device redirected to /dev/pts/1 could not open /dev/net/tun: Permission denied When I manually do /etc/init.d/libvirt-bin restart it does: (see the HOME=/root USER=root LOGNAME=root). But that should make any difference? LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin HOME=/root USER=root LOGNAME=root QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 1024 -smp 2,sockets=2,cores=1,threads=1 -name dupondje -uuid 28e9972d-c3ef-dfff-792e-5f80cd51a68a -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/dupondje.monitor,server,nowait -mon chardev=monitor,mode=readline -rtc base=utc -boot dc -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/var/lib/libvirt/images/dupondje,if=none,id=drive-virtio-disk0,boot=on,format=raw -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -device virtio-net-pci,vlan=0,id=net0,mac=52:54:00:xx:xx:xx,bus=pci.0,addr=0x6 -net tap,ifname=tap2,script=no,vlan=0,name=hostnet0 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -vnc 127.0.0.1:0 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 Thanks for the follow up! -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#598149: [Pkg-libvirt-maintainers] Bug#598149: libvirt-bin: autostart fails on reboot
On Mon, Sep 27, 2010 at 10:26:43AM +0200, Jean-Louis Dupond wrote: On Mon, 27 Sep 2010 09:06:22 +0200, Guido Günther a...@sigxcpu.org wrote: Hi Jean, Are you using dependency based booting? Please check if the network is fully up when libvirt-bin starts (e.g. by dumping the output of ifconfing in the init script). It should be since we have a dependency on $network in the init script. Cheers, -- Guido Hi Guido, I am using squeeze with dependency based booting. So I modified the libvirt-bin in init.d: start) if check_start_libvirtd_option; then log_daemon_msg Starting $DESC $NAME if running ; then log_progress_msg already running log_end_msg 0 exit 0 fi rm -f /var/run/libvirtd.pid /sbin/ifconfig /root/if-boot start-stop-daemon --start --quiet --pidfile $PIDFILE \ --exec $DAEMON -- $libvirtd_opts if running; then log_end_msg 0 else log_end_msg 1 fi fi ;; After a reboot I checked /root/if-boot. All the tap devices were created successfully, but still I got the following error: Sep 27 10:21:22 cloud libvirtd: 10:21:22.407: error : qemudReadLogOutput:2118 : internal error Process exited while reading console log output: char device redirected to /dev/pts/1#012could not open /dev/net/tun: Permission denied#012 Sep 27 10:21:22 cloud libvirtd: 10:21:22.407: error : qemuAutostartDomain:827 : Failed to autostart VM 'dupondje': internal error Process exited while reading console log output: char device redirected to /dev/pts/1#012could not open /dev/net/tun: Permission denied#012 Could you check the permissions of /dev/net/tun the same way please? Also please check if setting user=root/group=root in /etc/libvirt/qemu.conf changes anything. Please attache the domain xml of the domain that fails to start. Cheers, -- Guido -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#598149: [Pkg-libvirt-maintainers] Bug#598149: libvirt-bin: autostart fails on reboot
On Mon, 27 Sep 2010 11:16:35 +0200, Guido Günther a...@sigxcpu.org wrote: Could you check the permissions of /dev/net/tun the same way please? Also please check if setting user=root/group=root in /etc/libvirt/qemu.conf changes anything. Please attache the domain xml of the domain that fails to start. Cheers, -- Guido Permissions of /dev/net/tun: crw-rw 1 root uml-net 10, 200 Sep 27 10:21 /dev/net/tun The guests are running under the user: libvirt-qemu Thats why I make the tap devices in /etc/network/interfaces with the correct user. Changing the group/user to root should fix it, but thats not what I want. It should be able to use the created tap's. The XML: domain type='kvm' namedupondje/name uuid28e9972d-c3ef-dfff-792e-5f80cd51a68a/uuid memory1048576/memory currentMemory524288/currentMemory vcpu2/vcpu os type arch='x86_64' machine='pc-0.12'hvm/type boot dev='cdrom'/ boot dev='hd'/ /os features acpi/ apic/ pae/ /features clock offset='utc'/ on_poweroffdestroy/on_poweroff on_rebootrestart/on_reboot on_crashrestart/on_crash devices emulator/usr/bin/kvm/emulator disk type='file' device='cdrom' target dev='hdc' bus='ide'/ readonly/ address type='drive' controller='0' bus='1' unit='0'/ /disk disk type='file' device='disk' source file='/var/lib/libvirt/images/dupondje'/ target dev='vda' bus='virtio'/ address type='pci' domain='0x' bus='0x00' slot='0x04' function='0x0'/ /disk controller type='ide' index='0' address type='pci' domain='0x' bus='0x00' slot='0x01' function='0x1'/ /controller interface type='ethernet' mac address='52:54:00:xx:xx:xx'/ script path='no'/ target dev='tap2'/ model type='virtio'/ address type='pci' domain='0x' bus='0x00' slot='0x06' function='0x0'/ /interface serial type='pty' target port='0'/ /serial console type='pty' target port='0'/ /console input type='mouse' bus='ps2'/ graphics type='vnc' port='-1' autoport='yes'/ video model type='cirrus' vram='9216' heads='1'/ address type='pci' domain='0x' bus='0x00' slot='0x02' function='0x0'/ /video /devices /domain In the /etc/network/interfaces: auto tap2 iface tap2 inet static address xxx.xx.xx.xx netmask 255.255.255.255 tunctl_user libvirt-qemu up route add xx.xx.xx.xx/32 dev tap2 -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#598149: [Pkg-libvirt-maintainers] Bug#598149: libvirt-bin: autostart fails on reboot
On Mon, Sep 27, 2010 at 11:22:10AM +0200, Jean-Louis Dupond wrote: On Mon, 27 Sep 2010 11:16:35 +0200, Guido Günther a...@sigxcpu.org wrote: Could you check the permissions of /dev/net/tun the same way please? Also please check if setting user=root/group=root in /etc/libvirt/qemu.conf changes anything. Please attache the domain xml of the domain that fails to start. Cheers, -- Guido Permissions of /dev/net/tun: crw-rw 1 root uml-net 10, 200 Sep 27 10:21 /dev/net/tun So the permissions stay the same when checked from within the init script and after the boot completed? Does setting /dev/net/tun to 0666 in the init script before libvirt starts change anything? Cheers, -- Guido -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#598149: [Pkg-libvirt-maintainers] Bug#598149: libvirt-bin: autostart fails on reboot
On Mon, 27 Sep 2010 11:56:15 +0200, Guido Günther a...@sigxcpu.org wrote: So the permissions stay the same when checked from within the init script and after the boot completed? Does setting /dev/net/tun to 0666 in the init script before libvirt starts change anything? Cheers, -- Guido Did some more testing now: /bin/ls -la /dev/net/tun /root/boot /sbin/ifconfig /root/boot /bin/chmod 0666 /dev/net/tun /bin/ls -la /dev/net/tun /root/boot start-stop-daemon --start --quiet --pidfile $PIDFILE \ --exec $DAEMON -- $libvirtd_opts On reboot, the permissions: crw-rw 1 root uml-net 10, 200 Sep 27 12:48 /dev/net/tun Afterwards it lists all the tap devices correctly. They are up. Then I chmod /dev/net/tun: crw-rw-rw- 1 root uml-net 10, 200 Sep 27 12:48 /dev/net/tun And what do I see, the guests started without problems :) -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#598149: [Pkg-libvirt-maintainers] Bug#598149: libvirt-bin: autostart fails on reboot
On Mon, Sep 27, 2010 at 12:55:18PM +0200, Jean-Louis Dupond wrote: On Mon, 27 Sep 2010 11:56:15 +0200, Guido Günther a...@sigxcpu.org wrote: So the permissions stay the same when checked from within the init script and after the boot completed? Does setting /dev/net/tun to 0666 in the init script before libvirt starts change anything? Cheers, -- Guido Did some more testing now: /bin/ls -la /dev/net/tun /root/boot /sbin/ifconfig /root/boot /bin/chmod 0666 /dev/net/tun /bin/ls -la /dev/net/tun /root/boot start-stop-daemon --start --quiet --pidfile $PIDFILE \ --exec $DAEMON -- $libvirtd_opts Thanks a lot for your testing! On reboot, the permissions: crw-rw 1 root uml-net 10, 200 Sep 27 12:48 /dev/net/tun Afterwards it lists all the tap devices correctly. They are up. Then I chmod /dev/net/tun: crw-rw-rw- 1 root uml-net 10, 200 Sep 27 12:48 /dev/net/tun And what do I see, the guests started without problems :) So the issue isn't in libvirtd-bin - it's a permission problem when using uml-utilties. There's one open question though (as stated in my previous mail): How do the permissions on your system look like when the system is fully booted (and you revert your changes in /etc/init.d/libvirt-bin)? (since you said you can then start your guests without problems too). Cheers, -- Guido -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#598149: [Pkg-libvirt-maintainers] Bug#598149: libvirt-bin: autostart fails on reboot
On Mon, 27 Sep 2010 13:20:34 +0200, Guido Günther a...@sigxcpu.org wrote: Thanks a lot for your testing! So the issue isn't in libvirtd-bin - it's a permission problem when using uml-utilties. There's one open question though (as stated in my previous mail): How do the permissions on your system look like when the system is fully booted (and you revert your changes in /etc/init.d/libvirt-bin)? (since you said you can then start your guests without problems too). Cheers, -- Guido Ok I removed the changes again in the init.d. Permission @ boot: crw-rw 1 root uml-net 10, 200 Sep 27 13:24 /dev/net/tun The tap devices again created without issues. But again: Sep 27 13:25:03 cloud libvirtd: 13:25:03.027: error : qemudReadLogOutput:2118 : internal error Process exited while reading console log output: char device redirected to /dev/pts/1#012could not open /dev/net/tun: Permission denied#012 Sep 27 13:25:03 cloud libvirtd: 13:25:03.027: error : qemuAutostartDomain:827 : Failed to autostart VM 'dupondje': internal error Process exited while reading console log output: char device redirected to /dev/pts/1#012could not open /dev/net/tun: Permission denied#012 Without any change I do: cloud:~# /etc/init.d/libvirt-bin restart Restarting libvirt management daemon: /usr/sbin/libvirtd. And the guest devices are started without any permission error... -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#598149: [Pkg-libvirt-maintainers] Bug#598149: libvirt-bin: autostart fails on reboot
tag 598149 +unreproducible thanks On Mon, Sep 27, 2010 at 01:28:37PM +0200, Jean-Louis Dupond wrote: On Mon, 27 Sep 2010 13:20:34 +0200, Guido Günther a...@sigxcpu.org wrote: Thanks a lot for your testing! So the issue isn't in libvirtd-bin - it's a permission problem when using uml-utilties. There's one open question though (as stated in my previous mail): How do the permissions on your system look like when the system is fully booted (and you revert your changes in /etc/init.d/libvirt-bin)? (since you said you can then start your guests without problems too). Cheers, -- Guido Ok I removed the changes again in the init.d. Permission @ boot: crw-rw 1 root uml-net 10, 200 Sep 27 13:24 /dev/net/tun The tap devices again created without issues. But again: Sep 27 13:25:03 cloud libvirtd: 13:25:03.027: error : qemudReadLogOutput:2118 : internal error Process exited while reading console log output: char device redirected to /dev/pts/1#012could not open /dev/net/tun: Permission denied#012 Sep 27 13:25:03 cloud libvirtd: 13:25:03.027: error : qemuAutostartDomain:827 : Failed to autostart VM 'dupondje': internal error Process exited while reading console log output: char device redirected to /dev/pts/1#012could not open /dev/net/tun: Permission denied#012 Without any change I do: cloud:~# /etc/init.d/libvirt-bin restart Restarting libvirt management daemon: /usr/sbin/libvirtd. And the guest devices are started without any permission error... So your vm autostarts either when - fixing the /dev/net/tun permissions to 0666 - everything fine here - I understand that. or - when restarting libvirtd after the boot - I don't understand that one. * Are you using cgroups (though this shouldn't change anything about the device permissions)? * Any chance there's another self compiled libvirtd around that could get picked up? When I try to reproduce your problem here it's 100% consistent: the device comes up with 0640 and the vm won't start. After restarting libvirtd the device is still 0640 and the vm still won't start (as expected). When I change the permissions of /dev/net/tun to 0666 everything works fine and the vm gets autostarted on daemon restart. Could you doublecheck the permissions of /dev/net/tun when the vm runs? Cheers, -- Guido -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#598149: [Pkg-libvirt-maintainers] Bug#598149: libvirt-bin: autostart fails on reboot
On Mon, 27 Sep 2010 15:07:02 +0200, Guido Günther a...@sigxcpu.org wrote: tag 598149 +unreproducible thanks So your vm autostarts either when - fixing the /dev/net/tun permissions to 0666 - everything fine here - I understand that. If I chmod /dev/net/tun to 0666 before libvirt starts, it works. or - when restarting libvirtd after the boot - I don't understand that one. So when the node is booted, I restart libvirt-bin. (cloud:~# /etc/init.d/libvirt-bin restart Restarting libvirt management daemon: /usr/sbin/libvirtd.) Then the guests are started without any issue. Still I didn't change anything in the config/permissions between the boot the restart of libvirt. * Are you using cgroups (though this shouldn't change anything about the device permissions)? I'm using no cgroups. * Any chance there's another self compiled libvirtd around that could get picked up? Did never compile it myself. So there is no manual libvirt installation. When I try to reproduce your problem here it's 100% consistent: the device comes up with 0640 and the vm won't start. After restarting libvirtd the device is still 0640 and the vm still won't start (as expected). When I change the permissions of /dev/net/tun to 0666 everything works fine and the vm gets autostarted on daemon restart. Can you create a guest device that uses tap0 for example. Then create tap0 with the owner of libvirt. And then try to start the guest. This is my current setup. /etc/network/interfaces creates the tap devices with the libvirt owner, so that libvirt shouldn't have access to /dev/net/tun. Could you doublecheck the permissions of /dev/net/tun when the vm runs? The permissions are always: crw-rw 1 root uml-net 10, 200 Sep 27 13:54 /dev/net/tun Cheers, -- Guido -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#598149: [Pkg-libvirt-maintainers] Bug#598149: libvirt-bin: autostart fails on reboot
Hi Jean, On Mon, Sep 27, 2010 at 03:14:46PM +0200, Jean-Louis Dupond wrote: Then the guests are started without any issue. Still I didn't change anything in the config/permissions between the boot the restart of libvirt. Maybe you didn't but one of the startup scripts might have done so. [..snip..] Can you create a guest device that uses tap0 for example. Then create tap0 with the owner of libvirt. And then try to start the guest. This is my current setup. /etc/network/interfaces creates the tap devices with the libvirt owner, so that libvirt shouldn't have access to /dev/net/tun. That's exactly what I have done. Note that the permission denied isn't an error message from libvirt but from kvm/qemu itself trying to access /dev/net/tun. That's why it shows up in /var/log/libvirt/vm.log. Libvirt won't mess with /dev/net/tun if the interface type is ethernet. I wonder if kvm should access /dev/net/tun at all though since tap0 is already there? I assume libvirt-kvm is a regular user from /etc/passwd? No LDAP or similar involved? I'm really running out of ideas here. Cheers, -- Guido -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
