On Mon, Dec 08, 2014 at 01:20:19PM +0000, Dave Walker wrote: > On 8 December 2014 at 10:33, Daniel P. Berrange <[email protected]> wrote: > > On Sat, Dec 06, 2014 at 04:38:52PM +1100, Tony Breeds wrote: > >> Hi All, > >> In the most recent team meeting we briefly discussed: [1] where the > >> console.log grows indefinitely, eventually causing guest stalls. I > >> mentioned > >> that I was working on a spec to fix this issue. > >> > >> My original plan was fairly similar to [2] In that we'd switch > >> libvirt/qemu to > >> using a unix domain socket and write a simple helper to read from that > >> socket > >> and write to disk. That helper would close and reopen the on disk file > >> upon > >> receiving a HUP (so logrotate just works). Life would be good. and we > >> could > >> all move on. > >> > >> However I was encouraged to investigate fixing this in qemu, such that qemu > >> could process the HUP and make life better for all. This is certainly > >> doable > >> and I'm happy[3] to do this work. I've floated the idea past qemu-devel > >> and > >> they seem okay with the idea. My main concern is in lag and supporting > >> qemu/libvirt that can't handle this option. > > > > As mentioned in my reply on qemu-devel, I think the right long term solution > > for this is to fix it in libvirt. We have a general security goal to remove > > QEMU's ability to open any files whatsoever, instead having it receive all > > host resources as pre-opened file descriptors from libvirt. So what we > > anticipate is a new libvirt daemon for processing logs, virtlogd. Anywhere > > where QEMU currently gets a file to log to (<serial> devices, and its > > stdout/stderr), it would instead be given a FD that's connected to virtlogd. > > virtlogd would simply write the data out to file & would be able to close > > & re-open files to integrate with logrotate. > > > >> For the sake of discussion I'll lay out my best guess right now on fixing > >> this > >> in qemu. > >> > >> qemu 2.2.0 /should/ release this year the ETA is 2014-12-09[4] so the fix > >> I'm > >> proposing would be available in qemu 2.3.0 which I think will be available > >> in > >> June/July 2015. So we'd be into 'L' development before this fix is > >> available > >> and possibly 'M' before the community distros (Fedora and Ubuntu)[5] > >> include > >> and almost certainly longer for Enterprise distros. Along with the qemu > >> development I expect there to be some libvirt development as well but > >> right now > >> I don't think that's critical to the feature or this discussion. > >> > >> So if that timeline is approximately correct: > >> > >> - Can we wait this long to fix the bug? As opposed to having it squashed > >> in Kilo. > >> - What do we do in nova for the next ~12 months while know there isn't a > >> qemu to fix this? > >> - Then once there is a qemu that fixes the issue, do we just say 'thou > >> must use > >> qemu 2.3.0' or would nova still need to support old and new qemu's ? > > > > FWIW, by comparison libvirt is on a monthly release schedule, so a fix done > > in > > libvirt has potential to be available sooner, though obviously there's > > bigger > > dev work to be done in libvirt for this. > > > > Regards, > > Daniel > > Hey, > > This thread started by suggesting having a scheduled task to read from > a unix socket. I don't think this can really be considered an > acceptable fix, as the guest does indeed lock up when the buffer is > full. > > Initially, I proposed a quick fix for this back in 2011 which provided > a config option to enable a kernel level ring buffer via a > non-mainline module called emlog. This was not merged for > understandable reasons. (pre gerrit) - > http://bazaar.launchpad.net/~davewalker/nova/832507_with_emlog/revision/1509/nova/virt/libvirt/connection.py > > Later that same year, Robie Basak presented a change which introduced > similar logic ringbuffer support in the nova code itself making use of > eventlet. This seems quite a reasonable fix, but there was concern it > might lock-up guests.. https://review.openstack.org/#/c/706/ > > I think shortly after this, it was pretty widely agreed that fixing > this in Nova is not the correct layer. Personally, I struggle > thinking qemu or libvirt is right layer either. I can't think that > treating a console as a flat log file is the best default behavior. > > I still quite like the emlog approach, as having a ringbuffer device > type in the kernel provides exactly what we need and is pretty simple > to implement. > > Does anyone know if this generic ringbuffer kernel support was > proposed to mainline kernel?
The emlog approach means the data would only ever be stored in RAM on the host, so in the event of a host reboot/crash you loose all guest logs. While that might be ok for some people, I think we need to support the persistent store of the logs on disk for historical / auditing record purposes. We don't need kernel support to provide a ring buffer. An more or less identical solution can be done in userspace with just a pair of fixed size files. eg write to one file, when it hits a limit, switch to the second file, then back to the original, etc. We can easily do this in a libvirt based solution. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| _______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
