Indeed, we've applied that code change, compiled and installed the patched source, and things work great!
That was a fun code dive :D On Thu, Apr 17, 2014 at 2:44 PM, Filip Maj <f...@saucelabs.com> wrote: > Me again! > > Think we've found it. > > By diving into the LXC logs for the specific container, we found this: > > 2014-04-17 21:07:06.066+0000: 2861: debug : virCgroupSetValueStr:678 : Set > value '/sys/fs/cgroup/devices/machine/oshi32134.libvirt-lxc/devices.allow' > to 'c 189:130 rw' > > Looks like libvirt the permission to 'rw', not 'rmw' [1], so no surprise > that when it goes to call mknod [2], it can't do so. > > We are going to patch this so it gives 'rwm' access in the cgroup > whitelist, try to compile and replace libvirt to see if that works out for > us. Will report back :) > > [1] > http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_cgroup.c;h=39d955cd8cc2e34b6157e126fbb455d8f71e0647;hb=e8684eb541f01df9b45e87e0a8ce446c7bc90a17#l334 > [2] > http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_controller.c;h=5ca960f13e53501315b28f9086eaa389475b8feb;hb=e8684eb541f01df9b45e87e0a8ce446c7bc90a17#l1387 > > > On Thu, Apr 17, 2014 at 11:26 AM, Filip Maj <f...@saucelabs.com> wrote: > >> Ah, nevermind, think my last post may not have been entirely correct. >> We've spent some more time correlating the log output from our failed LXC >> startup via libvirt [1] with libvirt code from the 1.2.2 tag. >> Interestingly, the error we get back is different, and digging through the >> multitudes of logs, nothing comes up about 'operating not permitted' >> regarding the USB bus that I saw earlier. If we turn off the verbose >> logging, we get the original error back. The more-verbose error is a bit >> more cryptic: >> >> libvirt LXC Driver error: internal error: guest failed to start: 2014- >> >> Anyways, here's a rough overview of the logs: >> >> 1. it starts to check/detect for cgroup stuff [2] >> 2. starts to 'prepare host devices' and logs out that the appropriate usb >> address is initialized (002/003) [3] >> 3. sets the appropriate usb bus to 'active host devs' [4] >> 4. seems to then switch to doing network-related things, and stop talking >> about any hostdev/usb stuff [5] >> 5. eventually invokes `libvirt_lxc` [6] >> 6. does the cgroup song-n-dance again for some reason? [7]. not sure if >> this is critical or not. >> 7. eventually some kind of socket read wire error comes up [8] >> 8. and this leads to a 'guest failed to start error' [9], which it >> probably truncates to return the error i mentioned above. >> >> What is also confusing is, looking through the libvirt lxc code, >> shouldn't we be seeing a series of "Allowing any <blah>" log debug outputs >> in our logs? This is part of the virLXCCgroupSetupDeviceACL part of the >> lxc_cgroup.c file [10]. I imagine that we'd eventually want the >> virCgroupAllowDeviceMajor >> method [11] invoked, no? >> >> I realize that you are the author of most of this code, Daniel, so >> appreciate your patience with me and hope you are willing to continue >> trying to get to the bottom of this with us! Thanks a lot in advance! >> >> Cheers, >> Fil >> >> [1] https://gist.github.com/filmaj/207ab3c09f5b881d83b1 >> [2] >> https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L226-L253 >> [3] >> https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L257-L266 >> [4] >> https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L296 >> [5] >> https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L302 >> [6] >> https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L685 >> [7] >> https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L752-L799 >> [8] >> https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L1075 >> [9] >> https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L1090 >> [10] >> http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_cgroup.c;h=39d955cd8cc2e34b6157e126fbb455d8f71e0647;hb=e8684eb541f01df9b45e87e0a8ce446c7bc90a17#l342 >> [11] >> http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_cgroup.c;h=39d955cd8cc2e34b6157e126fbb455d8f71e0647;hb=e8684eb541f01df9b45e87e0a8ce446c7bc90a17#l446 >> >> >> On Wed, Apr 16, 2014 at 5:11 PM, Filip Maj <f...@saucelabs.com> wrote: >> >>> Further followups! >>> >>> We are correlating DEBUG-level output from libvirt with the libvirt >>> 1.2.2 code to try to figure out what libvirt is doing under the hood. >>> >>> Even though we have the log level set to 1 (info) in our libvirtd.conf, >>> we are not seeing the VIR_DEBUG() [1] statements being printed out. There >>> are tons of other presumably-debug lines of output showing up in our log. >>> We are sort of expecting to see the output from [1] in our logs somewhere, >>> given the type of usb mounting we are trying to do? >>> >>> [1] >>> http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_controller.c;h=c05dfec6428cad927cd5751004a4f3afc67899de;hb=HEAD#l1396 >>> >>> >>> On Wed, Apr 16, 2014 at 3:58 PM, Filip Maj <f...@saucelabs.com> wrote: >>> >>>> To follow up on this a little bit, tail'ing kern.log while trying to >>>> get our little container up doesn't yield anything with apparmor >>>> complaining, so, unless I'm looking in the wrong spots for apparmor logs >>>> (which I don't think so, as I see other apparmor-related log entries in >>>> kern.log), I am not entirely sure this is an apparmor issue at this point. >>>> >>>> >>>> On Wed, Apr 16, 2014 at 3:25 PM, Filip Maj <f...@saucelabs.com> wrote: >>>> >>>>> Yeah, AppArmor is enabled, but I put everything (that I could find) >>>>> into complain mode: >>>>> >>>>> $ sudo apparmor_status >>>>> apparmor module is loaded. >>>>> 12 profiles are loaded. >>>>> 3 profiles are in enforce mode. >>>>> lxc-container-default >>>>> lxc-container-default-with-mounting >>>>> lxc-container-default-with-nesting >>>>> 9 profiles are in complain mode. >>>>> /sbin/dhclient >>>>> /usr/bin/lxc-start >>>>> /usr/lib/NetworkManager/nm-dhcp-client.action >>>>> /usr/lib/connman/scripts/dhclient-script >>>>> /usr/lib/libvirt/virt-aa-helper >>>>> /usr/sbin/libvirtd >>>>> /usr/sbin/ntpd >>>>> /usr/sbin/rsyslogd >>>>> /usr/sbin/tcpdump >>>>> 3 processes have profiles defined. >>>>> 0 processes are in enforce mode. >>>>> 2 processes are in complain mode. >>>>> /usr/sbin/libvirtd (30419) >>>>> /usr/sbin/ntpd (3418) >>>>> 1 processes are unconfined but have a profile defined. >>>>> /usr/sbin/rsyslogd (626) >>>>> >>>>> And still get issues. From libvirtd.log: >>>>> >>>>> 2014-04-16 22:19:10.855+0000: 30419: info : libvirt version: 1.2.2 >>>>> 2014-04-16 22:19:10.855+0000: 30419: error : virNetSocketReadWire:1446 >>>>> : Cannot recv data: Connection reset by peer >>>>> 2014-04-16 22:19:10.940+0000: 30420: error : virLXCProcessStart:1299 : >>>>> internal error: guest failed to start: Unable to create device >>>>> //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not >>>>> permitted >>>>> >>>>> 2014-04-16 22:19:10.964+0000: 30420: warning : >>>>> virLXCDomainReAttachHostUsbDevices:388 : Unable to find device 000.000 in >>>>> list of active USB devices >>>>> >>>>> Thanks in advance for any help, Daniel! >>>>> >>>>> Cheers, >>>>> Fil >>>>> >>>>> >>>>> On Tue, Apr 15, 2014 at 1:33 AM, Daniel P. Berrange < >>>>> berra...@redhat.com> wrote: >>>>> >>>>>> On Fri, Apr 11, 2014 at 05:32:28PM -0700, Filip Maj wrote: >>>>>> > Hi! >>>>>> > >>>>>> > First post, kind of a noobie. I've been working with LXC and >>>>>> libvirt for a >>>>>> > few months now. Trying to do some interesting things with >>>>>> containers and >>>>>> > Android devices :D >>>>>> > Here's my entire domain definition: >>>>>> > >>>>>> > <domain type='lxc'> >>>>>> > <name>oshi32134</name> >>>>>> > <uuid>xxxxx</uuid> >>>>>> > <memory unit='KiB'>3145728</memory> >>>>>> > <currentMemory unit='KiB'>3145728</currentMemory> >>>>>> > <vcpu placement='static'>1</vcpu> >>>>>> > <resource> >>>>>> > <partition>/machine</partition> >>>>>> > </resource> >>>>>> > <os> >>>>>> > <type arch='i686'>exe</type> >>>>>> > <init>/sbin/init</init> >>>>>> > </os> >>>>>> > <clock offset='utc'/> >>>>>> > <on_poweroff>destroy</on_poweroff> >>>>>> > <on_reboot>restart</on_reboot> >>>>>> > <on_crash>destroy</on_crash> >>>>>> > <devices> >>>>>> > <emulator>/usr/lib/libvirt/libvirt_lxc</emulator> >>>>>> > <filesystem type='mount' accessmode='passthrough'> >>>>>> > <source dir='/some/valid/filesystem/location'/> >>>>>> > <target dir='/'/> >>>>>> > </filesystem> >>>>>> > <filesystem type='mount' accessmode='passthrough'> >>>>>> > <source dir='/another/valid/filesystem/location'/> >>>>>> > <target dir='/mnt/android'/> >>>>>> > </filesystem> >>>>>> > <interface type='bridge'> >>>>>> > <mac address='xx:xx:xx:xx:xx:xx'/> >>>>>> > <source bridge='br1'/> >>>>>> > </interface> >>>>>> > <console type='pty'> >>>>>> > <target type='lxc' port='0'/> >>>>>> > </console> >>>>>> > <hostdev mode='capabilities' type='misc'> >>>>>> > <source> >>>>>> > <char>/dev/kvm</char> >>>>>> > </source> >>>>>> > </hostdev> >>>>>> > <hostdev mode='subsystem' type='usb' managed='yes'> >>>>>> > <source> >>>>>> > <vendor id='0x04e8'/> >>>>>> > <product id='0x6860'/> >>>>>> > </source> >>>>>> > </hostdev> >>>>>> > </devices> >>>>>> > </domain> >>>>>> >>>>>> Your config looks fine here. >>>>>> >>>>>> > >>>>>> > Everything worked fine until I added the USB <hostdev> element. I'm >>>>>> > essentially trying to get access to a physical Android device >>>>>> connected to >>>>>> > the host from inside a container. When I go to start the container, >>>>>> I get >>>>>> > an error about Operation not permitted. Here's the relevant bits >>>>>> from >>>>>> > /var/log/libvirt/lxc/machine.log: >>>>>> > >>>>>> > 2014-04-11 22:46:40.491+0000: starting up >>>>>> > PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin >>>>>> > LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr >>>>>> /usr/lib/libvirt/libvirt_lxc >>>>>> > --name oshi32134 --console 24 --security=none --handshake 27 >>>>>> --background >>>>>> > --veth vnet1 >>>>>> > 2014-04-11 22:46:40.597+0000: 685: info : libvirt version: 1.2.2 >>>>>> > 2014-04-11 22:46:40.597+0000: 685: error : >>>>>> > virLXCControllerSetupHostdevSubsysUSB:1390 : Unable to create device >>>>>> > //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not >>>>>> > permitted >>>>>> > Unable to create device >>>>>> > //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not >>>>>> > permitted >>>>>> >>>>>> Do you have AppArmour enabled on the machine. That seems like the >>>>>> most likely thing that would result in libvirt getting that permission >>>>>> error. >>>>>> >>>>>> Regards, >>>>>> Daniel >>>>>> -- >>>>>> |: http://berrange.com -o- >>>>>> http://www.flickr.com/photos/dberrange/ :| >>>>>> |: http://libvirt.org -o- >>>>>> http://virt-manager.org :| >>>>>> |: http://autobuild.org -o- >>>>>> http://search.cpan.org/~danberr/ :| >>>>>> |: http://entangle-photo.org -o- >>>>>> http://live.gnome.org/gtk-vnc :| >>>>>> >>>>> >>>>> >>>> >>> >> >
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
