Re: [Qemu-devel] [PATCH] Python-lang gdb script to extract x86_64 guest vmcore from qemu coredump
On Thu, Sep 12, 2013 at 09:46:33PM +0200, Laszlo Ersek wrote:
When qemu dies unexpectedly, for example in response to an explicit
abort() call, or (more importantly) when an external signal is delivered
to it that results in a coredump, sometimes it is useful to extract the
guest vmcore from the qemu process' memory image. The guest vmcore might
help understand an emulation problem in qemu, or help debug the guest.
This script reimplements (and cuts many features of) the
qmp_dump_guest_memory() command in gdb/Python,
https://sourceware.org/gdb/current/onlinedocs/gdb/Python-API.html
working off the saved memory image of the qemu process. The docstring in
the patch (serving as gdb help text) describes the limitations relative to
the QMP command.
Dependencies of qmp_dump_guest_memory() have been reimplemented as needed.
I sought to follow the general structure, sticking to original function
names where possible. However, keeping it simple prevailed in some places.
The patch has been tested with a 4 VCPU, 768 MB, RHEL-6.4
(2.6.32-358.el6.x86_64) guest:
- The script printed
guest RAM blocks:
target_start target_end host_addrmessage count
--- -
000a 7f95d000 added 1
000a 000b 7f960ac0 added 2
000c 000ca000 7f95d00c added 3
000ca000 000cd000 7f95d00ca000 joined 3
000cd000 000d 7f95d00cd000 joined 3
000d 000f 7f95d00d joined 3
000f 0010 7f95d00f joined 3
0010 3000 7f95d010 joined 3
fc00 fc80 7f960ac0 added 4
fffe 0001 7f961880 added 5
dumping range at 7f95d000 for length 000a
dumping range at 7f960ac0 for length 0001
dumping range at 7f95d00c for length 2ff4
dumping range at 7f960ac0 for length 0080
dumping range at 7f961880 for length 0002
- The vmcore was checked with readelf, comparing the results against a
vmcore written by qmp_dump_guest_memory():
--- theirs 2013-09-12 17:38:59.797289404 +0200
+++ mine2013-09-12 17:39:03.820289404 +0200
@@ -27,16 +27,16 @@
Type Offset VirtAddr PhysAddr
FileSizMemSiz Flags Align
NOTE 0x0190 0x 0x
- 0x0ca0 0x0ca0 0
- LOAD 0x0e30 0x 0x
+ 0x001c 0x001c 0
+ LOAD 0x01ac 0x 0x
0x000a 0x000a 0
- LOAD 0x000a0e30 0x 0x000a
+ LOAD 0x000a01ac 0x 0x000a
0x0001 0x0001 0
- LOAD 0x000b0e30 0x 0x000c
+ LOAD 0x000b01ac 0x 0x000c
0x2ff4 0x2ff4 0
- LOAD 0x2fff0e30 0x 0xfc00
+ LOAD 0x2fff01ac 0x 0xfc00
0x0080 0x0080 0
- LOAD 0x307f0e30 0x 0xfffe
+ LOAD 0x307f01ac 0x 0xfffe
0x0002 0x0002 0
There is no dynamic section in this file.
@@ -47,13 +47,6 @@
No version information found in this file.
-Notes at offset 0x0190 with length 0x0ca0:
+Notes at offset 0x0190 with length 0x001c:
OwnerData size Description
- CORE 0x0150 NT_PRSTATUS (prstatus structure)
- CORE 0x0150 NT_PRSTATUS (prstatus structure)
- CORE 0x0150 NT_PRSTATUS (prstatus structure)
- CORE 0x0150 NT_PRSTATUS (prstatus structure)
- QEMU 0x01b0 Unknown note type: (0x)
- QEMU 0x01b0 Unknown note type: (0x)
- QEMU 0x01b0 Unknown note type: (0x)
- QEMU 0x01b0 Unknown note type: (0x)
+ NONE 0x0005 Unknown note type: (0x)
- The vmcore was checked with crash too, again comparing the results
against a vmcore written by qmp_dump_guest_memory():
---
Re: [Qemu-devel] [PATCH] Python-lang gdb script to extract x86_64 guest vmcore from qemu coredump
Laszlo Ersek ler...@redhat.com writes:
When qemu dies unexpectedly, for example in response to an explicit
abort() call, or (more importantly) when an external signal is delivered
to it that results in a coredump, sometimes it is useful to extract the
guest vmcore from the qemu process' memory image. The guest vmcore might
help understand an emulation problem in qemu, or help debug the guest.
This script reimplements (and cuts many features of) the
qmp_dump_guest_memory() command in gdb/Python,
https://sourceware.org/gdb/current/onlinedocs/gdb/Python-API.html
working off the saved memory image of the qemu process. The docstring in
the patch (serving as gdb help text) describes the limitations relative to
the QMP command.
Dependencies of qmp_dump_guest_memory() have been reimplemented as needed.
I sought to follow the general structure, sticking to original function
names where possible. However, keeping it simple prevailed in some places.
The patch has been tested with a 4 VCPU, 768 MB, RHEL-6.4
(2.6.32-358.el6.x86_64) guest:
- The script printed
guest RAM blocks:
target_start target_end host_addrmessage count
--- -
000a 7f95d000 added 1
000a 000b 7f960ac0 added 2
000c 000ca000 7f95d00c added 3
000ca000 000cd000 7f95d00ca000 joined 3
000cd000 000d 7f95d00cd000 joined 3
000d 000f 7f95d00d joined 3
000f 0010 7f95d00f joined 3
0010 3000 7f95d010 joined 3
fc00 fc80 7f960ac0 added 4
fffe 0001 7f961880 added 5
dumping range at 7f95d000 for length 000a
dumping range at 7f960ac0 for length 0001
dumping range at 7f95d00c for length 2ff4
dumping range at 7f960ac0 for length 0080
dumping range at 7f961880 for length 0002
- The vmcore was checked with readelf, comparing the results against a
vmcore written by qmp_dump_guest_memory():
--- theirs 2013-09-12 17:38:59.797289404 +0200
+++ mine2013-09-12 17:39:03.820289404 +0200
@@ -27,16 +27,16 @@
Type Offset VirtAddr PhysAddr
FileSizMemSiz Flags Align
NOTE 0x0190 0x 0x
- 0x0ca0 0x0ca0 0
- LOAD 0x0e30 0x 0x
+ 0x001c 0x001c 0
+ LOAD 0x01ac 0x 0x
0x000a 0x000a 0
- LOAD 0x000a0e30 0x 0x000a
+ LOAD 0x000a01ac 0x 0x000a
0x0001 0x0001 0
- LOAD 0x000b0e30 0x 0x000c
+ LOAD 0x000b01ac 0x 0x000c
0x2ff4 0x2ff4 0
- LOAD 0x2fff0e30 0x 0xfc00
+ LOAD 0x2fff01ac 0x 0xfc00
0x0080 0x0080 0
- LOAD 0x307f0e30 0x 0xfffe
+ LOAD 0x307f01ac 0x 0xfffe
0x0002 0x0002 0
There is no dynamic section in this file.
@@ -47,13 +47,6 @@
No version information found in this file.
-Notes at offset 0x0190 with length 0x0ca0:
+Notes at offset 0x0190 with length 0x001c:
OwnerData size Description
- CORE 0x0150 NT_PRSTATUS (prstatus structure)
- CORE 0x0150 NT_PRSTATUS (prstatus structure)
- CORE 0x0150 NT_PRSTATUS (prstatus structure)
- CORE 0x0150 NT_PRSTATUS (prstatus structure)
- QEMU 0x01b0 Unknown note type: (0x)
- QEMU 0x01b0 Unknown note type: (0x)
- QEMU 0x01b0 Unknown note type: (0x)
- QEMU 0x01b0 Unknown note type: (0x)
+ NONE 0x0005 Unknown note type: (0x)
- The vmcore was checked with crash too, again comparing the results
against a vmcore written by qmp_dump_guest_memory():
--- guest.vmcore.log2 2013-09-12 17:52:27.074289201 +0200
+++ example.dump.log2 2013-09-12
Re: [Qemu-devel] [PATCH] Python-lang gdb script to extract x86_64 guest vmcore from qemu coredump
On 10/11/13 19:54, Stefan Hajnoczi wrote: On Thu, Sep 12, 2013 at 9:46 PM, Laszlo Ersek ler...@redhat.com wrote: When qemu dies unexpectedly, for example in response to an explicit abort() call, or (more importantly) when an external signal is delivered to it that results in a coredump, sometimes it is useful to extract the guest vmcore from the qemu process' memory image. The guest vmcore might help understand an emulation problem in qemu, or help debug the guest. This script reimplements (and cuts many features of) the qmp_dump_guest_memory() command in gdb/Python, https://sourceware.org/gdb/current/onlinedocs/gdb/Python-API.html working off the saved memory image of the qemu process. The docstring in the patch (serving as gdb help text) describes the limitations relative to the QMP command. Dependencies of qmp_dump_guest_memory() have been reimplemented as needed. I sought to follow the general structure, sticking to original function names where possible. However, keeping it simple prevailed in some places. The patch has been tested with a 4 VCPU, 768 MB, RHEL-6.4 (2.6.32-358.el6.x86_64) guest: I tried this out with qemu-kvm-1.1.2 and it worked after a few minor tweaks due to memory data structure changes. I did hit a problem with crash since the vmlinux was 32-bit and the guest dump was 64-bit. But from what I can tell dump-guest-memory works as advertised. Reviewed-by: Stefan Hajnoczi stefa...@redhat.com Could someone please pick up the patch? Thanks! Laszlo
Re: [Qemu-devel] [PATCH] Python-lang gdb script to extract x86_64 guest vmcore from qemu coredump
On Thu, Sep 12, 2013 at 9:46 PM, Laszlo Ersek ler...@redhat.com wrote: +For simplicity, the paging, begin and end parameters of the QMP +command are not supported -- no attempt is made to get the guest's +internal paging structures (ie. paging=false is hard-wired), and guest +memory is always fully dumped. + +Only x86_64 guests are supported. + +The CORE/NT_PRSTATUS and QEMU notes (that is, the VCPUs' statuses) are +not written to the vmcore. Preparing these would require context that is +only present in the KVM host kernel module when the guest is alive. A +fake ELF note is written instead, only to keep the ELF parser of crash +happy. I found the lack of register state and paging information is easy to work around for guest kernel debugging: The Linux 32-bit PAE guest kernel I tried identity maps physical memory to 0xc000. Set program header p_vaddr fields to the physical address + 0xc000 in order to produce an ELF suitable for gdb kernel debugging. Also, you can get a snapshot of CPU registers from QEMU's first_cpu list. They are probably outdated since the KVM register state isn't synced on every exit but the cr registers contain useful information on paging. In other words, it may still be possible to reconstruct page tables by traversing cr3. Stefan
Re: [Qemu-devel] [PATCH] Python-lang gdb script to extract x86_64 guest vmcore from qemu coredump
On Thu, Sep 12, 2013 at 9:46 PM, Laszlo Ersek ler...@redhat.com wrote: When qemu dies unexpectedly, for example in response to an explicit abort() call, or (more importantly) when an external signal is delivered to it that results in a coredump, sometimes it is useful to extract the guest vmcore from the qemu process' memory image. The guest vmcore might help understand an emulation problem in qemu, or help debug the guest. This script reimplements (and cuts many features of) the qmp_dump_guest_memory() command in gdb/Python, https://sourceware.org/gdb/current/onlinedocs/gdb/Python-API.html working off the saved memory image of the qemu process. The docstring in the patch (serving as gdb help text) describes the limitations relative to the QMP command. Dependencies of qmp_dump_guest_memory() have been reimplemented as needed. I sought to follow the general structure, sticking to original function names where possible. However, keeping it simple prevailed in some places. The patch has been tested with a 4 VCPU, 768 MB, RHEL-6.4 (2.6.32-358.el6.x86_64) guest: I tried this out with qemu-kvm-1.1.2 and it worked after a few minor tweaks due to memory data structure changes. I did hit a problem with crash since the vmlinux was 32-bit and the guest dump was 64-bit. But from what I can tell dump-guest-memory works as advertised. Reviewed-by: Stefan Hajnoczi stefa...@redhat.com
Re: [Qemu-devel] [PATCH] Python-lang gdb script to extract x86_64 guest vmcore from qemu coredump
On 09/12/13 21:46, Laszlo Ersek wrote: When qemu dies unexpectedly, for example in response to an explicit abort() call, or (more importantly) when an external signal is delivered to it that results in a coredump, sometimes it is useful to extract the guest vmcore from the qemu process' memory image. The guest vmcore might help understand an emulation problem in qemu, or help debug the guest. This script reimplements (and cuts many features of) the qmp_dump_guest_memory() command in gdb/Python, https://sourceware.org/gdb/current/onlinedocs/gdb/Python-API.html working off the saved memory image of the qemu process. The docstring in the patch (serving as gdb help text) describes the limitations relative to the QMP command. Dependencies of qmp_dump_guest_memory() have been reimplemented as needed. I sought to follow the general structure, sticking to original function names where possible. However, keeping it simple prevailed in some places. Ping. I'd like to get this upstream because that would help with keeping it in sync with the memory API / internals. Boring as hell to review (you'd have to follow through most of the dump-guest-memory implementation, down to the memory internals; and probably read the gdb python API docs). OTOH it can't hurt anything (separate python file in the scripts directory), so maybe it could go in just with an Acked-by from someone. Thanks Laszlo
[Qemu-devel] [PATCH] Python-lang gdb script to extract x86_64 guest vmcore from qemu coredump
When qemu dies unexpectedly, for example in response to an explicit
abort() call, or (more importantly) when an external signal is delivered
to it that results in a coredump, sometimes it is useful to extract the
guest vmcore from the qemu process' memory image. The guest vmcore might
help understand an emulation problem in qemu, or help debug the guest.
This script reimplements (and cuts many features of) the
qmp_dump_guest_memory() command in gdb/Python,
https://sourceware.org/gdb/current/onlinedocs/gdb/Python-API.html
working off the saved memory image of the qemu process. The docstring in
the patch (serving as gdb help text) describes the limitations relative to
the QMP command.
Dependencies of qmp_dump_guest_memory() have been reimplemented as needed.
I sought to follow the general structure, sticking to original function
names where possible. However, keeping it simple prevailed in some places.
The patch has been tested with a 4 VCPU, 768 MB, RHEL-6.4
(2.6.32-358.el6.x86_64) guest:
- The script printed
guest RAM blocks:
target_start target_end host_addrmessage count
--- -
000a 7f95d000 added 1
000a 000b 7f960ac0 added 2
000c 000ca000 7f95d00c added 3
000ca000 000cd000 7f95d00ca000 joined 3
000cd000 000d 7f95d00cd000 joined 3
000d 000f 7f95d00d joined 3
000f 0010 7f95d00f joined 3
0010 3000 7f95d010 joined 3
fc00 fc80 7f960ac0 added 4
fffe 0001 7f961880 added 5
dumping range at 7f95d000 for length 000a
dumping range at 7f960ac0 for length 0001
dumping range at 7f95d00c for length 2ff4
dumping range at 7f960ac0 for length 0080
dumping range at 7f961880 for length 0002
- The vmcore was checked with readelf, comparing the results against a
vmcore written by qmp_dump_guest_memory():
--- theirs 2013-09-12 17:38:59.797289404 +0200
+++ mine2013-09-12 17:39:03.820289404 +0200
@@ -27,16 +27,16 @@
Type Offset VirtAddr PhysAddr
FileSizMemSiz Flags Align
NOTE 0x0190 0x 0x
- 0x0ca0 0x0ca0 0
- LOAD 0x0e30 0x 0x
+ 0x001c 0x001c 0
+ LOAD 0x01ac 0x 0x
0x000a 0x000a 0
- LOAD 0x000a0e30 0x 0x000a
+ LOAD 0x000a01ac 0x 0x000a
0x0001 0x0001 0
- LOAD 0x000b0e30 0x 0x000c
+ LOAD 0x000b01ac 0x 0x000c
0x2ff4 0x2ff4 0
- LOAD 0x2fff0e30 0x 0xfc00
+ LOAD 0x2fff01ac 0x 0xfc00
0x0080 0x0080 0
- LOAD 0x307f0e30 0x 0xfffe
+ LOAD 0x307f01ac 0x 0xfffe
0x0002 0x0002 0
There is no dynamic section in this file.
@@ -47,13 +47,6 @@
No version information found in this file.
-Notes at offset 0x0190 with length 0x0ca0:
+Notes at offset 0x0190 with length 0x001c:
OwnerData size Description
- CORE 0x0150 NT_PRSTATUS (prstatus structure)
- CORE 0x0150 NT_PRSTATUS (prstatus structure)
- CORE 0x0150 NT_PRSTATUS (prstatus structure)
- CORE 0x0150 NT_PRSTATUS (prstatus structure)
- QEMU 0x01b0 Unknown note type: (0x)
- QEMU 0x01b0 Unknown note type: (0x)
- QEMU 0x01b0 Unknown note type: (0x)
- QEMU 0x01b0 Unknown note type: (0x)
+ NONE 0x0005 Unknown note type: (0x)
- The vmcore was checked with crash too, again comparing the results
against a vmcore written by qmp_dump_guest_memory():
--- guest.vmcore.log2 2013-09-12 17:52:27.074289201 +0200
+++ example.dump.log2 2013-09-12 17:52:15.904289203 +0200
@@ -22,11 +22,11 @@
This GDB was
