Re: [openstack-dev] [nova][libvirt] Logging interactions of libvirt + QEMU in Nova
On Wed, Feb 04, 2015 at 11:15:18AM -0500, Davanum Srinivas wrote: > Daniel, Kashyap, > > One question that came up on IRC was, how/where to configure say a > directory where core dumps from qemu would end up. Sean was seeing a > scenario where he noticed a core dump from qemu in dmesg/syslog If you're using something similar to ABRT on Ubuntu (assuming you're using that), you should be able to find the stack traces. Seems like on Ubuntu, this is what they use https://wiki.ubuntu.com/Apport > and > was wondering how to specify a directory to capture a core dump > if/when it occurs. Incidentally, last week I was debugging a QEMU seg fault with upstream folks[1] which involves qemu-nbd. To capture coredumps with QEMU, this is what I had to do on my Fedora 21 system. Ensure abrt-ccpp (Automatic bug reporting tool is running: $ systemctl status abrt-ccpp ● abrt-ccpp.service - Install ABRT coredump hook Loaded: loaded (/usr/lib/systemd/system/abrt-ccpp.service; enabled) Active: active (exited) since Tue 2015-02-03 16:16:14 CET; 1 day 3h ago Main PID: 1113 (code=exited, status=0/SUCCESS) CGroup: /system.slice/abrt-ccpp.service Then the coredump file should land in: /var/tmp/abrt/ccpp-2015-01-30-00\:01\:05-3145/ You can actually see that the specific coredump is for QEMU, by running (note, in this example it was `qemu-img` that was seg faulting - and there are patches accepted for this upstream): $ cd /var/tmp/abrt/ccpp-2015-01-30-00\:01\:05-3145/ $ file coredump coredump: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/home/kashyapc/build/qemu/qemu-img Assuming you have all the QEMU debug info packages installed, you can then invoke GDB to capture the trace backs. An example[2] $ gdb /var/tmp/abrt/ccpp-2015-01-30-00\:01\:05-3145/coredump [. . .] (gdb) bt full [. . .] [1] http://lists.nongnu.org/archive/html/qemu-devel/2015-01/msg04397.html [2] https://kashyapc.fedorapeople.org/virt/qemu-nbd-test/stack-traces-from-coredump.txt > On Wed, Feb 4, 2015 at 8:48 AM, Kashyap Chamarthy wrote: > > On Wed, Feb 04, 2015 at 10:27:34AM +, Daniel P. Berrange wrote: > >> On Wed, Feb 04, 2015 at 11:23:34AM +0100, Kashyap Chamarthy wrote: > >> > Heya, > >> > > >> > I noticed a ping (but couldn't respond in time) on #openstack-nova IRC > >> > about turning on logging in libvirt to capture Nova failures. > >> > > >> > This was discussed on this list previously by Daniel Berrange, just > >> > spelling it out here for reference and completness' sake. > >> > > >> > > >> > (1) To see the interactions between libvirt and QEMU, in > >> > /etc/libvirt/libvirtd.conf, have these two config attributes: > >> > > >> > . . . > >> > log_filters="1:libvirt 1:qemu 1:conf 1:security 3:event 3:json > >> > 3:file 1:util" > >> > >> You really want "3:object" in there /before/ 1:util too otherwise you'll > >> be spammed with object ref/unref messages > > > > Thanks for this detail. > > > >> > >> > log_outputs="1:file:/var/tmp/libvirtd.log" > >> > >> Use /var/log/libvirt/libvirtd.log instead of /var/tmp > > > > Ah, yeah, it was an incorrect copy/paste. > > > > -- > > /kashyap > > -- /kashyap __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][libvirt] Logging interactions of libvirt + QEMU in Nova
Aha, thanks Daniel. On Wed, Feb 4, 2015 at 12:01 PM, Daniel P. Berrange wrote: > On Wed, Feb 04, 2015 at 11:57:56AM -0500, Davanum Srinivas wrote: >> Daniel, >> >> The last tip on this page possibly? >> http://wiki.stoney-cloud.org/wiki/Debugging_Qemu > > Note that "tip" is not merely affecting QEMU processes - the recommended > change is affecting core dumps for everything on the entire OS. This is > really what a tool like abrt is aiming todo already :-) > > Regards, > Daniel > -- > |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| > |: http://libvirt.org -o- http://virt-manager.org :| > |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| > |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| -- Davanum Srinivas :: https://twitter.com/dims __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][libvirt] Logging interactions of libvirt + QEMU in Nova
On Wed, Feb 04, 2015 at 11:57:56AM -0500, Davanum Srinivas wrote: > Daniel, > > The last tip on this page possibly? > http://wiki.stoney-cloud.org/wiki/Debugging_Qemu Note that "tip" is not merely affecting QEMU processes - the recommended change is affecting core dumps for everything on the entire OS. This is really what a tool like abrt is aiming todo already :-) Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][libvirt] Logging interactions of libvirt + QEMU in Nova
Daniel, The last tip on this page possibly? http://wiki.stoney-cloud.org/wiki/Debugging_Qemu -- dims On Wed, Feb 4, 2015 at 11:18 AM, Daniel P. Berrange wrote: > On Wed, Feb 04, 2015 at 11:15:18AM -0500, Davanum Srinivas wrote: >> Daniel, Kashyap, >> >> One question that came up on IRC was, how/where to configure say a >> directory where core dumps from qemu would end up. Sean was seeing a >> scenario where he noticed a core dump from qemu in dmesg/syslog and >> was wondering how to specify a directory to capture a core dump >> if/when it occurs. > > That's really outside the scope of libvirt. On Fedora/RHEL there is > the abrt program, which captures core dumps from any process in the > entire OS and has configurable actions for where to save them. IIUC > Ubuntu has some general purpose core dump handler too but I don't know > much about it myself. They all work by hooking into the kernel's > core_pattern sysctl knob to redirect core dumps to their own binary > instead of using $CWD > > Regards, > Daniel > -- > |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| > |: http://libvirt.org -o- http://virt-manager.org :| > |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| > |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| -- Davanum Srinivas :: https://twitter.com/dims __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][libvirt] Logging interactions of libvirt + QEMU in Nova
On Wed, Feb 04, 2015 at 11:15:18AM -0500, Davanum Srinivas wrote: > Daniel, Kashyap, > > One question that came up on IRC was, how/where to configure say a > directory where core dumps from qemu would end up. Sean was seeing a > scenario where he noticed a core dump from qemu in dmesg/syslog and > was wondering how to specify a directory to capture a core dump > if/when it occurs. That's really outside the scope of libvirt. On Fedora/RHEL there is the abrt program, which captures core dumps from any process in the entire OS and has configurable actions for where to save them. IIUC Ubuntu has some general purpose core dump handler too but I don't know much about it myself. They all work by hooking into the kernel's core_pattern sysctl knob to redirect core dumps to their own binary instead of using $CWD Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][libvirt] Logging interactions of libvirt + QEMU in Nova
Daniel, Kashyap, One question that came up on IRC was, how/where to configure say a directory where core dumps from qemu would end up. Sean was seeing a scenario where he noticed a core dump from qemu in dmesg/syslog and was wondering how to specify a directory to capture a core dump if/when it occurs. thanks, dims On Wed, Feb 4, 2015 at 8:48 AM, Kashyap Chamarthy wrote: > On Wed, Feb 04, 2015 at 10:27:34AM +, Daniel P. Berrange wrote: >> On Wed, Feb 04, 2015 at 11:23:34AM +0100, Kashyap Chamarthy wrote: >> > Heya, >> > >> > I noticed a ping (but couldn't respond in time) on #openstack-nova IRC >> > about turning on logging in libvirt to capture Nova failures. >> > >> > This was discussed on this list previously by Daniel Berrange, just >> > spelling it out here for reference and completness' sake. >> > >> > >> > (1) To see the interactions between libvirt and QEMU, in >> > /etc/libvirt/libvirtd.conf, have these two config attributes: >> > >> > . . . >> > log_filters="1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file >> > 1:util" >> >> You really want "3:object" in there /before/ 1:util too otherwise you'll >> be spammed with object ref/unref messages > > Thanks for this detail. > >> >> > log_outputs="1:file:/var/tmp/libvirtd.log" >> >> Use /var/log/libvirt/libvirtd.log instead of /var/tmp > > Ah, yeah, it was an incorrect copy/paste. > > -- > /kashyap > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Davanum Srinivas :: https://twitter.com/dims __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][libvirt] Logging interactions of libvirt + QEMU in Nova
On Wed, Feb 04, 2015 at 10:27:34AM +, Daniel P. Berrange wrote: > On Wed, Feb 04, 2015 at 11:23:34AM +0100, Kashyap Chamarthy wrote: > > Heya, > > > > I noticed a ping (but couldn't respond in time) on #openstack-nova IRC > > about turning on logging in libvirt to capture Nova failures. > > > > This was discussed on this list previously by Daniel Berrange, just > > spelling it out here for reference and completness' sake. > > > > > > (1) To see the interactions between libvirt and QEMU, in > > /etc/libvirt/libvirtd.conf, have these two config attributes: > > > > . . . > > log_filters="1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file > > 1:util" > > You really want "3:object" in there /before/ 1:util too otherwise you'll > be spammed with object ref/unref messages Thanks for this detail. > > > log_outputs="1:file:/var/tmp/libvirtd.log" > > Use /var/log/libvirt/libvirtd.log instead of /var/tmp Ah, yeah, it was an incorrect copy/paste. -- /kashyap __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][libvirt] Logging interactions of libvirt + QEMU in Nova
On Wed, Feb 04, 2015 at 11:23:34AM +0100, Kashyap Chamarthy wrote: > Heya, > > I noticed a ping (but couldn't respond in time) on #openstack-nova IRC > about turning on logging in libvirt to capture Nova failures. > > This was discussed on this list previously by Daniel Berrange, just > spelling it out here for reference and completness' sake. > > > (1) To see the interactions between libvirt and QEMU, in > /etc/libvirt/libvirtd.conf, have these two config attributes: > > . . . > log_filters="1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file > 1:util" You really want "3:object" in there /before/ 1:util too otherwise you'll be spammed with object ref/unref messages > log_outputs="1:file:/var/tmp/libvirtd.log" Use /var/log/libvirt/libvirtd.log instead of /var/tmp Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova][libvirt] Logging interactions of libvirt + QEMU in Nova
Heya, I noticed a ping (but couldn't respond in time) on #openstack-nova IRC about turning on logging in libvirt to capture Nova failures. This was discussed on this list previously by Daniel Berrange, just spelling it out here for reference and completness' sake. (1) To see the interactions between libvirt and QEMU, in /etc/libvirt/libvirtd.conf, have these two config attributes: . . . log_filters="1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 1:util" log_outputs="1:file:/var/tmp/libvirtd.log" . . . (2) Restart libvirtd: $ systemctl restart libvirtd (3) Enable Nova debug logs in /etc/nova/nova.conf: . . . verbose = True debug = True . . . (4) Restart the Nova service: $ openstack-service restart nova (5) Repeat the offending test. The libvirtd.log file should have the relevant details. Other resources --- - http://wiki.libvirt.org/page/DebugLogs - https://fedoraproject.org/wiki/How_to_debug_Virtualization_problems -- /kashyap __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
