Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-25 03:37, Wen Congyang wrote: At 10/24/2011 11:58 PM, Dave Anderson Write: - Original Message - No, an ELF image of the guest's physical memory. Well then that should be pretty straight forward to support. Depending upon how similar it would be to the standard kdump ELF format, the only other issue is how to determine the physical base address at which the kernel is loaded, in order to be able to translate the mapped kernel-text/static-data virtual region of the x86_64 arch (the __START_KERNEL_map region). I guess an elf note would work for that? Right -- here is an example of a RHEL6 ELF kdump header: Hi, Jan Is gdb standard core file like the following format? Does gdb support this format? If yes, I think the dump command can output the guest physical memory in the following format, and we can use both gdb and crash to analyze it. That would be ideal. But given lacking crash-like add-ons for vanilla gdb, it makes sense to focus on crash compatibility for now. I suspect we will need some changes to gdb anyway to make use of all information the kdump file contains. Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
At 10/24/2011 11:30 PM, Avi Kivity Write: On 10/24/2011 05:25 PM, Dave Anderson wrote: - Original Message - On 10/24/2011 04:25 PM, Dave Anderson wrote: The question is that: 'virsh dump' can not be used when host pci device is used by guest. We are discussing how to fix the problem. We have determined that introduce a new monitor command dump. Jan suggested that the core file's format is gdb standard core format. Does crash support such format? If no, is it possible to support such format? If you are talking about an ELF core dump of the user-space qemu-kvm process running on the host, then it's certainly not supported. No, an ELF image of the guest's physical memory. Well then that should be pretty straight forward to support. Depending upon how similar it would be to the standard kdump ELF format, the only other issue is how to determine the physical base address at which the kernel is loaded, in order to be able to translate the mapped kernel-text/static-data virtual region of the x86_64 arch (the __START_KERNEL_map region). I guess an elf note would work for that? Hi, Avi Kivity, Dave Anderson I have two questions about it: 1. How to know the guest's physical base address in qemu? 2. Does kdump format(not ELF) include such information in its head? Thanks Wen Congyang
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 10/25/2011 10:06 AM, Wen Congyang wrote: Hi, Avi Kivity, Dave Anderson I have two questions about it: 1. How to know the guest's physical base address in qemu? In fact, it's impossible. Perhaps crash can scan through the core looking for a signature. Need to be careful since multiple signatures may be detected (a kdump kernel, a kernel image in pagecache, or a nested guest). -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 10/25/2011 10:23 AM, Avi Kivity wrote: On 10/25/2011 10:06 AM, Wen Congyang wrote: Hi, Avi Kivity, Dave Anderson I have two questions about it: 1. How to know the guest's physical base address in qemu? In fact, it's impossible. Perhaps crash can scan through the core looking for a signature. Need to be careful since multiple signatures may be detected (a kdump kernel, a kernel image in pagecache, or a nested guest). Crash uses a heuristic based on the IDT base to identify the base from migrate-to-file output. It also needs CR3 in order to read from virtual address. Paolo
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 10/25/2011 10:35 AM, Paolo Bonzini wrote: On 10/25/2011 10:23 AM, Avi Kivity wrote: On 10/25/2011 10:06 AM, Wen Congyang wrote: Hi, Avi Kivity, Dave Anderson I have two questions about it: 1. How to know the guest's physical base address in qemu? In fact, it's impossible. Perhaps crash can scan through the core looking for a signature. Need to be careful since multiple signatures may be detected (a kdump kernel, a kernel image in pagecache, or a nested guest). Crash uses a heuristic based on the IDT base to identify the base from migrate-to-file output. It also needs CR3 in order to read from virtual address. Does elf not store cr3? Perhaps a note with the contents of the privileged cpu registers is needed. How will gdb cope with module sections? They are virtually contiguous but physically scattered. AFAICT gdb doesn't do translations itself. The kernel proper can be handled with a simple offset, but modules cannot. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 10/25/2011 10:52 AM, Avi Kivity wrote: Crash uses a heuristic based on the IDT base to identify the base from migrate-to-file output. It also needs CR3 in order to read from virtual address. Does elf not store cr3? Perhaps a note with the contents of the privileged cpu registers is needed. Yes. It would be nice if this note could be somewhat standardized. Jan, is there a code sample somewhere for the x86-gdb summer of code project? Paolo
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-25 10:56, Paolo Bonzini wrote: On 10/25/2011 10:52 AM, Avi Kivity wrote: Crash uses a heuristic based on the IDT base to identify the base from migrate-to-file output. It also needs CR3 in order to read from virtual address. Does elf not store cr3? Perhaps a note with the contents of the privileged cpu registers is needed. Yes. It would be nice if this note could be somewhat standardized. Jan, is there a code sample somewhere for the x86-gdb summer of code project? I'm not aware of a public repos, CC'ing Anthony and Tom. We definitely need all x86 system registers handled by gdb long-term, already for live debugging and for segmented guests / proper real mode support. But if crash depends on those values already today, we should head for its current format in a first step, then enhance gdb, and finally check how to integrate that information best with vanilla gdb. For me it is important that we agree on consolidating over (enhanced) gdb on the long run, both /wrt features and supported core formats. Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
- Original Message - On 10/25/2011 10:23 AM, Avi Kivity wrote: On 10/25/2011 10:06 AM, Wen Congyang wrote: Hi, Avi Kivity, Dave Anderson I have two questions about it: 1. How to know the guest's physical base address in qemu? In fact, it's impossible. Perhaps crash can scan through the core looking for a signature. Need to be careful since multiple signatures may be detected (a kdump kernel, a kernel image in pagecache, or a nested guest). Crash uses a heuristic based on the IDT base to identify the base from migrate-to-file output. It also needs CR3 in order to read from virtual address. Just to clarify -- the need for the IDT base and cr3 is *only* true for KVM dumps using the savevm format. With ELF-format kdumps, the phys_base is simply determined from the PT_LOAD segment that describes the kernel's __START_KERNEL_map region. I don't know why that couldn't be done in this proposed case. Dave
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 10/25/2011 03:18 PM, Dave Anderson wrote: Crash uses a heuristic based on the IDT base to identify the base from migrate-to-file output. It also needs CR3 in order to read from virtual address. Just to clarify -- the need for the IDT base and cr3 is *only* true for KVM dumps using the savevm format. With ELF-format kdumps, the phys_base is simply determined from the PT_LOAD segment that describes the kernel's __START_KERNEL_map region. I don't know why that couldn't be done in this proposed case. kvm has no idea where the kernel (if any) is running. The dump is taken without any guest cooperation. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
- Original Message - The question is that: 'virsh dump' can not be used when host pci device is used by guest. We are discussing how to fix the problem. We have determined that introduce a new monitor command dump. Jan suggested that the core file's format is gdb standard core format. Does crash support such format? If no, is it possible to support such format? If you are talking about an ELF core dump of the user-space qemu-kvm process running on the host, then it's certainly not supported. As to whether it's possible, I suppose it could be done if a phyical-memory-read function could be created for it, similar to what I asked about for live analysis of a guest kernel run on/from the KVM host. Dave
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 10/24/2011 04:25 PM, Dave Anderson wrote: The question is that: 'virsh dump' can not be used when host pci device is used by guest. We are discussing how to fix the problem. We have determined that introduce a new monitor command dump. Jan suggested that the core file's format is gdb standard core format. Does crash support such format? If no, is it possible to support such format? If you are talking about an ELF core dump of the user-space qemu-kvm process running on the host, then it's certainly not supported. No, an ELF image of the guest's physical memory. As to whether it's possible, I suppose it could be done if a phyical-memory-read function could be created for it, similar to what I asked about for live analysis of a guest kernel run on/from the KVM host. We could use the gdb protocol for this. Currently it reads/writes virtual memory, but we could extend the gdbserver command to make these commands apply to physical memory. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
- Original Message - On 10/24/2011 04:25 PM, Dave Anderson wrote: The question is that: 'virsh dump' can not be used when host pci device is used by guest. We are discussing how to fix the problem. We have determined that introduce a new monitor command dump. Jan suggested that the core file's format is gdb standard core format. Does crash support such format? If no, is it possible to support such format? If you are talking about an ELF core dump of the user-space qemu-kvm process running on the host, then it's certainly not supported. No, an ELF image of the guest's physical memory. Well then that should be pretty straight forward to support. Depending upon how similar it would be to the standard kdump ELF format, the only other issue is how to determine the physical base address at which the kernel is loaded, in order to be able to translate the mapped kernel-text/static-data virtual region of the x86_64 arch (the __START_KERNEL_map region). As to whether it's possible, I suppose it could be done if a phyical-memory-read function could be created for it, similar to what I asked about for live analysis of a guest kernel run on/from the KVM host. We could use the gdb protocol for this. Currently it reads/writes virtual memory, but we could extend the gdbserver command to make these commands apply to physical memory.
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 10/24/2011 05:25 PM, Dave Anderson wrote: - Original Message - On 10/24/2011 04:25 PM, Dave Anderson wrote: The question is that: 'virsh dump' can not be used when host pci device is used by guest. We are discussing how to fix the problem. We have determined that introduce a new monitor command dump. Jan suggested that the core file's format is gdb standard core format. Does crash support such format? If no, is it possible to support such format? If you are talking about an ELF core dump of the user-space qemu-kvm process running on the host, then it's certainly not supported. No, an ELF image of the guest's physical memory. Well then that should be pretty straight forward to support. Depending upon how similar it would be to the standard kdump ELF format, the only other issue is how to determine the physical base address at which the kernel is loaded, in order to be able to translate the mapped kernel-text/static-data virtual region of the x86_64 arch (the __START_KERNEL_map region). I guess an elf note would work for that? -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
- Original Message - No, an ELF image of the guest's physical memory. Well then that should be pretty straight forward to support. Depending upon how similar it would be to the standard kdump ELF format, the only other issue is how to determine the physical base address at which the kernel is loaded, in order to be able to translate the mapped kernel-text/static-data virtual region of the x86_64 arch (the __START_KERNEL_map region). I guess an elf note would work for that? Right -- here is an example of a RHEL6 ELF kdump header: $ readelf -a vmcore ELF Header: Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 Class: ELF64 Data: 2's complement, little endian Version: 1 (current) OS/ABI:UNIX - System V ABI Version: 0 Type: CORE (Core file) Machine: Advanced Micro Devices X86-64 Version: 0x1 Entry point address: 0x0 Start of program headers: 64 (bytes into file) Start of section headers: 0 (bytes into file) Flags: 0x0 Size of this header: 64 (bytes) Size of program headers: 56 (bytes) Number of program headers: 6 Size of section headers: 0 (bytes) Number of section headers: 0 Section header string table index: 0 There are no sections in this file. There are no sections in this file. Program Headers: Type Offset VirtAddr PhysAddr FileSizMemSiz Flags Align NOTE 0x0190 0x 0x 0x083c 0x083c 0 LOAD 0x09cc 0x8100 0x0100 0x00ba3000 0x00ba3000 RWE0 LOAD 0x00ba39cc 0x8100 0x 0x000a 0x000a RWE0 LOAD 0x00c439cc 0x8110 0x0010 0x01f0 0x01f0 RWE0 LOAD 0x02b439cc 0x81000a00 0x0a00 0xc5fc2840 0xc5fc2840 RWE0 LOAD 0xc8b0620c 0x8101 0x0001 0x3000 0x3000 RWE0 There is no dynamic section in this file. There are no relocations in this file. There are no unwind sections in this file. No version information found in this file. Notes at offset 0x0190 with length 0x083c: Owner Data size Description CORE 0x0150 NT_PRSTATUS (prstatus structure) CORE 0x0150 NT_PRSTATUS (prstatus structure) VMCOREINFO0x055b Unknown note type: (0x) $ In this example, the phys_base (of zero) can be determined by looking at the first PT_LOAD segment, and comparing the PhysAddr and the VirtAddr values -- given that __START_KERNEL_map region is based at 8. The remaining physical memory chunks are described by the subsequent unity-mapped segments. The NT_PRSTATUS notes are register dumps of each cpu, where this vmcore was from a 2-cpu system. But the crash utility is capable of surviving without them. It can also get by without the VMCOREINFO note, which is primarily there for use by the makedumpfile utility, which is used to compress ELF kdumps and filter out unwanted pages, and then make a different dumpfile format entirely. This may be another stupid question -- but does the guest failure mode render it incapable of using kdump? Dave
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
At 10/24/2011 11:58 PM, Dave Anderson Write: - Original Message - No, an ELF image of the guest's physical memory. Well then that should be pretty straight forward to support. Depending upon how similar it would be to the standard kdump ELF format, the only other issue is how to determine the physical base address at which the kernel is loaded, in order to be able to translate the mapped kernel-text/static-data virtual region of the x86_64 arch (the __START_KERNEL_map region). I guess an elf note would work for that? Right -- here is an example of a RHEL6 ELF kdump header: Hi, Jan Is gdb standard core file like the following format? Does gdb support this format? If yes, I think the dump command can output the guest physical memory in the following format, and we can use both gdb and crash to analyze it. $ readelf -a vmcore ELF Header: Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 Class: ELF64 Data: 2's complement, little endian Version: 1 (current) OS/ABI:UNIX - System V ABI Version: 0 Type: CORE (Core file) Machine: Advanced Micro Devices X86-64 Version: 0x1 Entry point address: 0x0 Start of program headers: 64 (bytes into file) Start of section headers: 0 (bytes into file) Flags: 0x0 Size of this header: 64 (bytes) Size of program headers: 56 (bytes) Number of program headers: 6 Size of section headers: 0 (bytes) Number of section headers: 0 Section header string table index: 0 There are no sections in this file. There are no sections in this file. Program Headers: Type Offset VirtAddr PhysAddr FileSizMemSiz Flags Align NOTE 0x0190 0x 0x 0x083c 0x083c 0 LOAD 0x09cc 0x8100 0x0100 0x00ba3000 0x00ba3000 RWE0 LOAD 0x00ba39cc 0x8100 0x 0x000a 0x000a RWE0 LOAD 0x00c439cc 0x8110 0x0010 0x01f0 0x01f0 RWE0 LOAD 0x02b439cc 0x81000a00 0x0a00 0xc5fc2840 0xc5fc2840 RWE0 LOAD 0xc8b0620c 0x8101 0x0001 0x3000 0x3000 RWE0 There is no dynamic section in this file. There are no relocations in this file. There are no unwind sections in this file. No version information found in this file. Notes at offset 0x0190 with length 0x083c: Owner Data size Description CORE 0x0150 NT_PRSTATUS (prstatus structure) CORE 0x0150 NT_PRSTATUS (prstatus structure) VMCOREINFO0x055b Unknown note type: (0x) $ In this example, the phys_base (of zero) can be determined by looking at the first PT_LOAD segment, and comparing the PhysAddr and the VirtAddr values -- given that __START_KERNEL_map region is based at 8. The remaining physical memory chunks are described by the subsequent unity-mapped segments. The NT_PRSTATUS notes are register dumps of each cpu, where this vmcore was from a 2-cpu system. But the crash utility is capable of surviving without them. It can also get by without the VMCOREINFO note, which is primarily there for use by the makedumpfile utility, which is used to compress ELF kdumps and filter out unwanted pages, and then make a different dumpfile format entirely. This may be another stupid question -- but does the guest failure mode render it incapable of using kdump? Guest failure mode? I do not what does it mean. But 'virsh dump' can be used when kdump fails or kdump service is not started. Thanks Wen Congyang Dave
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
At 10/21/2011 09:02 PM, Dave Anderson Write: - Original Message - At 10/21/2011 03:11 PM, Jan Kiszka Write: On 2011-10-20 12:03, Wen Congyang wrote: At 10/20/2011 05:41 PM, Jan Kiszka Write: On 2011-10-20 03:22, Wen Congyang wrote: I didn't read full story but 'crash' is used for investigating kernel core generated by kdump for several years. Considering support service guys, virsh dump should support a format for crash because they can't work well at investigating vmcore by gdb. crash has several functionality useful for them as 'show kerne log', 'focus on a cpu' 'for-each-task', 'for-each-vma', 'extract ftrace log' etc. Anyway, if a man, who is not developper of qemu/kvm, should learn 2 tools for investigating kernel dump, it sounds harmful. Right, that's why everything (live debugging crash analysis) should be consolidated on the long run over gdb. crash is architecturally obsolete today - not saying it is useless! I do not know why crash is obsoleted today. Is there a new better tool to instead crash? I'm not aware of equally powerful (python) scripts for gdb as replacement, but I think it's worth starting a porting effort at some point. At least, I always use crash to live debugging crash analysis. Then you may answer some questions to me: - Can you attach to a remote target (kgdb, qemu, etc.) and how? No. crash's live debugging only can work the kernel is live. I can use it get some var's value, or some other information from kernel. If kernel panics, we can use gdb to attach to a remote target as you said. But on end user machine, we can not do it, we should dump the memory into a file and analyze it in another machine while the end user's guest can be restart. - Can you use it with latest gdb versions or is the gdb functionality hard-wired due to an embedded gdb core in crash (that's how I understood Christoph's reply to this topic) If I use crash, I can not use latest gdb versions. Do we always need to use the latest gdb versions? Currently, gdb-7.0 is embedded into crash, and it is enough to me. If the gdb embedded into crash cannot anaylze the vmcore, I think we can update it and rebuild crash. crash is simply designed the wrong way around (from today's perspective): it should augment upstream gdb instead of forking it. Cc Dave Anderson. He knows how crash uses gdb. I think that crash does not fork a task to execute gdb, and gdb is a part of crash. I'm not sure what the question is, but you can consider crash as a huge The question is that: 'virsh dump' can not be used when host pci device is used by guest. We are discussing how to fix the problem. We have determined that introduce a new monitor command dump. Jan suggested that the core file's format is gdb standard core format. Does crash support such format? If no, is it possible to support such format? Thanks Wen Congyang wrapper around its embedded gdb, which it invokes as gdb vmlinux, and then takes over the user interface. It doesn't have a clue as to what the memory source is, i.e., whether it's one of the almost 20 different dumpfile formats that it supports (including virsh dump), or if it's running against a live system. It has its own command set, although you can enter some gdb commands, write gdb scripts, etc. But the main purpose of the embedded gdb is for the crash-level sources to be able to gather data structure information, disassemble text, add-symbol-file kernel modules, and so on. There is no kgdb remote linkage. It's currently embedding gdb-7.0, although as we speak I'm updating it to gdb-7.3.1 because the compiler guys have decided that dwarf4 should be used by default. It would be kind of cool if there was a /dev/mem-like interface to a KVM guest's physical memory, so that you could sit on a KVM host and enter crash vmlinux-of-guest /dev/mem-of-guest in order to run live analysis of a guest. Anyway, sorry if it doesn't meet your needs... Dave
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-20 12:03, Wen Congyang wrote: At 10/20/2011 05:41 PM, Jan Kiszka Write: On 2011-10-20 03:22, Wen Congyang wrote: I didn't read full story but 'crash' is used for investigating kernel core generated by kdump for several years. Considering support service guys, virsh dump should support a format for crash because they can't work well at investigating vmcore by gdb. crash has several functionality useful for them as 'show kerne log', 'focus on a cpu' 'for-each-task', 'for-each-vma', 'extract ftrace log' etc. Anyway, if a man, who is not developper of qemu/kvm, should learn 2 tools for investigating kernel dump, it sounds harmful. Right, that's why everything (live debugging crash analysis) should be consolidated on the long run over gdb. crash is architecturally obsolete today - not saying it is useless! I do not know why crash is obsoleted today. Is there a new better tool to instead crash? I'm not aware of equally powerful (python) scripts for gdb as replacement, but I think it's worth starting a porting effort at some point. At least, I always use crash to live debugging crash analysis. Then you may answer some questions to me: - Can you attach to a remote target (kgdb, qemu, etc.) and how? No. crash's live debugging only can work the kernel is live. I can use it get some var's value, or some other information from kernel. If kernel panics, we can use gdb to attach to a remote target as you said. But on end user machine, we can not do it, we should dump the memory into a file and analyze it in another machine while the end user's guest can be restart. - Can you use it with latest gdb versions or is the gdb functionality hard-wired due to an embedded gdb core in crash (that's how I understood Christoph's reply to this topic) If I use crash, I can not use latest gdb versions. Do we always need to use the latest gdb versions? Currently, gdb-7.0 is embedded into crash, and it is enough to me. If the gdb embedded into crash cannot anaylze the vmcore, I think we can update it and rebuild crash. crash is simply designed the wrong way around (from today's perspective): it should augment upstream gdb instead of forking it. Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
At 10/21/2011 03:11 PM, Jan Kiszka Write: On 2011-10-20 12:03, Wen Congyang wrote: At 10/20/2011 05:41 PM, Jan Kiszka Write: On 2011-10-20 03:22, Wen Congyang wrote: I didn't read full story but 'crash' is used for investigating kernel core generated by kdump for several years. Considering support service guys, virsh dump should support a format for crash because they can't work well at investigating vmcore by gdb. crash has several functionality useful for them as 'show kerne log', 'focus on a cpu' 'for-each-task', 'for-each-vma', 'extract ftrace log' etc. Anyway, if a man, who is not developper of qemu/kvm, should learn 2 tools for investigating kernel dump, it sounds harmful. Right, that's why everything (live debugging crash analysis) should be consolidated on the long run over gdb. crash is architecturally obsolete today - not saying it is useless! I do not know why crash is obsoleted today. Is there a new better tool to instead crash? I'm not aware of equally powerful (python) scripts for gdb as replacement, but I think it's worth starting a porting effort at some point. At least, I always use crash to live debugging crash analysis. Then you may answer some questions to me: - Can you attach to a remote target (kgdb, qemu, etc.) and how? No. crash's live debugging only can work the kernel is live. I can use it get some var's value, or some other information from kernel. If kernel panics, we can use gdb to attach to a remote target as you said. But on end user machine, we can not do it, we should dump the memory into a file and analyze it in another machine while the end user's guest can be restart. - Can you use it with latest gdb versions or is the gdb functionality hard-wired due to an embedded gdb core in crash (that's how I understood Christoph's reply to this topic) If I use crash, I can not use latest gdb versions. Do we always need to use the latest gdb versions? Currently, gdb-7.0 is embedded into crash, and it is enough to me. If the gdb embedded into crash cannot anaylze the vmcore, I think we can update it and rebuild crash. crash is simply designed the wrong way around (from today's perspective): it should augment upstream gdb instead of forking it. Cc Dave Anderson. He knows how crash uses gdb. I think that crash does not fork a task to execute gdb, and gdb is a part of crash. Thanks Wen Congyang Jan
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-21 15:02, Dave Anderson wrote: - Original Message - At 10/21/2011 03:11 PM, Jan Kiszka Write: On 2011-10-20 12:03, Wen Congyang wrote: At 10/20/2011 05:41 PM, Jan Kiszka Write: On 2011-10-20 03:22, Wen Congyang wrote: I didn't read full story but 'crash' is used for investigating kernel core generated by kdump for several years. Considering support service guys, virsh dump should support a format for crash because they can't work well at investigating vmcore by gdb. crash has several functionality useful for them as 'show kerne log', 'focus on a cpu' 'for-each-task', 'for-each-vma', 'extract ftrace log' etc. Anyway, if a man, who is not developper of qemu/kvm, should learn 2 tools for investigating kernel dump, it sounds harmful. Right, that's why everything (live debugging crash analysis) should be consolidated on the long run over gdb. crash is architecturally obsolete today - not saying it is useless! I do not know why crash is obsoleted today. Is there a new better tool to instead crash? I'm not aware of equally powerful (python) scripts for gdb as replacement, but I think it's worth starting a porting effort at some point. At least, I always use crash to live debugging crash analysis. Then you may answer some questions to me: - Can you attach to a remote target (kgdb, qemu, etc.) and how? No. crash's live debugging only can work the kernel is live. I can use it get some var's value, or some other information from kernel. If kernel panics, we can use gdb to attach to a remote target as you said. But on end user machine, we can not do it, we should dump the memory into a file and analyze it in another machine while the end user's guest can be restart. - Can you use it with latest gdb versions or is the gdb functionality hard-wired due to an embedded gdb core in crash (that's how I understood Christoph's reply to this topic) If I use crash, I can not use latest gdb versions. Do we always need to use the latest gdb versions? Currently, gdb-7.0 is embedded into crash, and it is enough to me. If the gdb embedded into crash cannot anaylze the vmcore, I think we can update it and rebuild crash. crash is simply designed the wrong way around (from today's perspective): it should augment upstream gdb instead of forking it. Cc Dave Anderson. He knows how crash uses gdb. I think that crash does not fork a task to execute gdb, and gdb is a part of crash. I'm not sure what the question is, but you can consider crash as a huge wrapper around its embedded gdb, which it invokes as gdb vmlinux, and then takes over the user interface. It doesn't have a clue as to what the memory source is, i.e., whether it's one of the almost 20 different dumpfile formats that it supports (including virsh dump), or if it's running against a live system. It has its own command set, although you can enter some gdb commands, write gdb scripts, etc. But the main purpose of the embedded gdb is for the crash-level sources to be able to gather data structure information, disassemble text, add-symbol-file kernel modules, and so on. There is no kgdb remote linkage. It's currently embedding gdb-7.0, although as we speak I'm updating it to gdb-7.3.1 because the compiler guys have decided that dwarf4 should be used by default. It would be kind of cool if there was a /dev/mem-like interface to a KVM guest's physical memory, so that you could sit on a KVM host and enter crash vmlinux-of-guest /dev/mem-of-guest in order to run live analysis of a guest. Anyway, sorry if it doesn't meet your needs... Yes, I'd prefer to have the added value of crash available with standard gdb, specifically to reuse it for remote target debugging which is fairly common in embedded scenarios and for guest debugging (via qemu/kvm). I do not yet see that there is anything preventing this except that it needs to be done - or collected from previous work: there should be, e.g., dozens of gdb scripts circling around that automate kernel module symbol loading with gdb. We do have a proper remote debugging interface already, no need to invent a new one for crash-on-qemu. We do lack complete x86 system-level debugging support, but that's an absolutely generic gdb problem, independent of Linux as a debugging target. And, AFAIK, we do not have such issues with common non-x86 targets. Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On Fri, Oct 21, 2011 at 09:02:37AM -0400, Dave Anderson wrote: It would be kind of cool if there was a /dev/mem-like interface to a KVM guest's physical memory, so that you could sit on a KVM host and enter crash vmlinux-of-guest /dev/mem-of-guest in order to run live analysis of a guest. OT for this thread, but this sort of thing does exist, kind of. You can send monitor commands to qemu to read physical and virtual memory (pmemsave and memsave respectively). At least one, and possibly now both of these are bound through libvirt APIs: http://libvirt.org/html/libvirt-libvirt.html#virDomainMemoryPeek Here was your previous response about 4 years ago: https://www.redhat.com/archives/crash-utility/2008-August/msg00032.html Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-20 03:22, Wen Congyang wrote: I didn't read full story but 'crash' is used for investigating kernel core generated by kdump for several years. Considering support service guys, virsh dump should support a format for crash because they can't work well at investigating vmcore by gdb. crash has several functionality useful for them as 'show kerne log', 'focus on a cpu' 'for-each-task', 'for-each-vma', 'extract ftrace log' etc. Anyway, if a man, who is not developper of qemu/kvm, should learn 2 tools for investigating kernel dump, it sounds harmful. Right, that's why everything (live debugging crash analysis) should be consolidated on the long run over gdb. crash is architecturally obsolete today - not saying it is useless! I do not know why crash is obsoleted today. Is there a new better tool to instead crash? I'm not aware of equally powerful (python) scripts for gdb as replacement, but I think it's worth starting a porting effort at some point. At least, I always use crash to live debugging crash analysis. Then you may answer some questions to me: - Can you attach to a remote target (kgdb, qemu, etc.) and how? - Can you use it with latest gdb versions or is the gdb functionality hard-wired due to an embedded gdb core in crash (that's how I understood Christoph's reply to this topic) Thanks, Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
At 10/20/2011 05:41 PM, Jan Kiszka Write: On 2011-10-20 03:22, Wen Congyang wrote: I didn't read full story but 'crash' is used for investigating kernel core generated by kdump for several years. Considering support service guys, virsh dump should support a format for crash because they can't work well at investigating vmcore by gdb. crash has several functionality useful for them as 'show kerne log', 'focus on a cpu' 'for-each-task', 'for-each-vma', 'extract ftrace log' etc. Anyway, if a man, who is not developper of qemu/kvm, should learn 2 tools for investigating kernel dump, it sounds harmful. Right, that's why everything (live debugging crash analysis) should be consolidated on the long run over gdb. crash is architecturally obsolete today - not saying it is useless! I do not know why crash is obsoleted today. Is there a new better tool to instead crash? I'm not aware of equally powerful (python) scripts for gdb as replacement, but I think it's worth starting a porting effort at some point. At least, I always use crash to live debugging crash analysis. Then you may answer some questions to me: - Can you attach to a remote target (kgdb, qemu, etc.) and how? No. crash's live debugging only can work the kernel is live. I can use it get some var's value, or some other information from kernel. If kernel panics, we can use gdb to attach to a remote target as you said. But on end user machine, we can not do it, we should dump the memory into a file and analyze it in another machine while the end user's guest can be restart. - Can you use it with latest gdb versions or is the gdb functionality hard-wired due to an embedded gdb core in crash (that's how I understood Christoph's reply to this topic) If I use crash, I can not use latest gdb versions. Do we always need to use the latest gdb versions? Currently, gdb-7.0 is embedded into crash, and it is enough to me. If the gdb embedded into crash cannot anaylze the vmcore, I think we can update it and rebuild crash. Thanks Wen Congyang. Thanks, Jan
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-19 04:04, KAMEZAWA Hiroyuki wrote: On Tue, 18 Oct 2011 10:31:10 +0200 Jan Kiszka jan.kis...@siemens.com wrote: On 2011-10-18 10:31, Wen Congyang wrote: At 10/18/2011 04:26 PM, Jan Kiszka Write: On 2011-10-18 10:25, Wen Congyang wrote: At 10/18/2011 04:19 PM, Jan Kiszka Write: On 2011-10-18 09:58, Wen Congyang wrote: At 10/18/2011 03:52 PM, Jan Kiszka Write: On 2011-10-18 09:15, Wen Congyang wrote: Hi, Jan Kiszka At 10/10/2011 05:34 PM, Jan Kiszka Write: On 2011-10-10 11:02, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote: Run gdb with set debug remote 1 and watch the communication, it is not that complex. But a dump command is probably simpler for those scenarios, I agree. I have implemented the command dump and reuse migration's code. But I meet a problem when I test it. Using migration code for dump is most probably the wrong approach as you saw through that conflict. All you need are the register states and the RAM. Reuse gdbstub services for this. Hmm, if the migration code can not be reused, I think we should define a new qemu's vmcore format, and add some codes into crash to support such format. Please try to avoid defining something new. Unless there is a striking reason, standard gdb core files should be generated so that you can load the dump directly into gdb for analysis. I am not sure whehter the standard gdb core files can not be analyzed by crash. If not, I think we should define something new because it's easier to use crash than gdb to analyze the core files. gdb allows you to walk up the frame and print variables (globals local) etc. Crash uses gdb to provide common function, and you can also use all the gdb commands in crash. That what's the added value here when I can use gdb directly? I didn't read full story but 'crash' is used for investigating kernel core generated by kdump for several years. Considering support service guys, virsh dump should support a format for crash because they can't work well at investigating vmcore by gdb. crash has several functionality useful for them as 'show kerne log', 'focus on a cpu' 'for-each-task', 'for-each-vma', 'extract ftrace log' etc. Anyway, if a man, who is not developper of qemu/kvm, should learn 2 tools for investigating kernel dump, it sounds harmful. Right, that's why everything (live debugging crash analysis) should be consolidated on the long run over gdb. crash is architecturally obsolete today - not saying it is useless! Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
At 10/19/2011 07:40 PM, Jan Kiszka Write: On 2011-10-19 04:04, KAMEZAWA Hiroyuki wrote: On Tue, 18 Oct 2011 10:31:10 +0200 Jan Kiszka jan.kis...@siemens.com wrote: On 2011-10-18 10:31, Wen Congyang wrote: At 10/18/2011 04:26 PM, Jan Kiszka Write: On 2011-10-18 10:25, Wen Congyang wrote: At 10/18/2011 04:19 PM, Jan Kiszka Write: On 2011-10-18 09:58, Wen Congyang wrote: At 10/18/2011 03:52 PM, Jan Kiszka Write: On 2011-10-18 09:15, Wen Congyang wrote: Hi, Jan Kiszka At 10/10/2011 05:34 PM, Jan Kiszka Write: On 2011-10-10 11:02, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote: Run gdb with set debug remote 1 and watch the communication, it is not that complex. But a dump command is probably simpler for those scenarios, I agree. I have implemented the command dump and reuse migration's code. But I meet a problem when I test it. Using migration code for dump is most probably the wrong approach as you saw through that conflict. All you need are the register states and the RAM. Reuse gdbstub services for this. Hmm, if the migration code can not be reused, I think we should define a new qemu's vmcore format, and add some codes into crash to support such format. Please try to avoid defining something new. Unless there is a striking reason, standard gdb core files should be generated so that you can load the dump directly into gdb for analysis. I am not sure whehter the standard gdb core files can not be analyzed by crash. If not, I think we should define something new because it's easier to use crash than gdb to analyze the core files. gdb allows you to walk up the frame and print variables (globals local) etc. Crash uses gdb to provide common function, and you can also use all the gdb commands in crash. That what's the added value here when I can use gdb directly? I didn't read full story but 'crash' is used for investigating kernel core generated by kdump for several years. Considering support service guys, virsh dump should support a format for crash because they can't work well at investigating vmcore by gdb. crash has several functionality useful for them as 'show kerne log', 'focus on a cpu' 'for-each-task', 'for-each-vma', 'extract ftrace log' etc. Anyway, if a man, who is not developper of qemu/kvm, should learn 2 tools for investigating kernel dump, it sounds harmful. Right, that's why everything (live debugging crash analysis) should be consolidated on the long run over gdb. crash is architecturally obsolete today - not saying it is useless! I do not know why crash is obsoleted today. Is there a new better tool to instead crash? At least, I always use crash to live debugging crash analysis. Thanks Wen Congyang Jan
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
Hi, Jan Kiszka
At 10/10/2011 05:34 PM, Jan Kiszka Write:
On 2011-10-10 11:02, Daniel P. Berrange wrote:
On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
Run gdb with set debug remote 1 and watch the communication, it is not
that complex. But a dump command is probably simpler for those
scenarios, I agree.
I have implemented the command dump and reuse migration's code. But I meet a
problem
when I test it.
My qemu-kvm's tree is not updated about 2 months ago, because kernel.org is
down, and
I forgot to pull from github.
After I pull it from github, I find the following changes:
@@ -1523,9 +1523,7 @@ static void
assigned_dev_unregister_msix_mmio(AssignedDevice *dev)
static const VMStateDescription vmstate_assigned_device = {
.name = pci-assign,
-.fields = (VMStateField []) {
-VMSTATE_END_OF_LIST()
-}
+.unmigratable = 1,
};
static void reset_assigned_device(DeviceState *dev)
Why do you remove fields from vmstate_assigned_device?
It is useful for dump because it does not check unmigratable. If
vmstate_assigned_device
does not contain .fields, qemu will crash in vmstate_save_state().
Thanks
Wen Congyang
Jan
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-18 09:15, Wen Congyang wrote: Hi, Jan Kiszka At 10/10/2011 05:34 PM, Jan Kiszka Write: On 2011-10-10 11:02, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote: Run gdb with set debug remote 1 and watch the communication, it is not that complex. But a dump command is probably simpler for those scenarios, I agree. I have implemented the command dump and reuse migration's code. But I meet a problem when I test it. Using migration code for dump is most probably the wrong approach as you saw through that conflict. All you need are the register states and the RAM. Reuse gdbstub services for this. Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
At 10/18/2011 03:52 PM, Jan Kiszka Write: On 2011-10-18 09:15, Wen Congyang wrote: Hi, Jan Kiszka At 10/10/2011 05:34 PM, Jan Kiszka Write: On 2011-10-10 11:02, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote: Run gdb with set debug remote 1 and watch the communication, it is not that complex. But a dump command is probably simpler for those scenarios, I agree. I have implemented the command dump and reuse migration's code. But I meet a problem when I test it. Using migration code for dump is most probably the wrong approach as you saw through that conflict. All you need are the register states and the RAM. Reuse gdbstub services for this. Hmm, if the migration code can not be reused, I think we should define a new qemu's vmcore format, and add some codes into crash to support such format. I will read gdbstub services's code now. Thanks Wen Congyang Jan
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On Tue, Oct 18, 2011 at 03:15:29PM +0800, Wen Congyang wrote:
Hi, Jan Kiszka
At 10/10/2011 05:34 PM, Jan Kiszka Write:
On 2011-10-10 11:02, Daniel P. Berrange wrote:
On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
Run gdb with set debug remote 1 and watch the communication, it is not
that complex. But a dump command is probably simpler for those
scenarios, I agree.
I have implemented the command dump and reuse migration's code. But I meet a
problem
when I test it.
My qemu-kvm's tree is not updated about 2 months ago, because kernel.org is
down, and
I forgot to pull from github.
After I pull it from github, I find the following changes:
@@ -1523,9 +1523,7 @@ static void
assigned_dev_unregister_msix_mmio(AssignedDevice *dev)
static const VMStateDescription vmstate_assigned_device = {
.name = pci-assign,
-.fields = (VMStateField []) {
-VMSTATE_END_OF_LIST()
-}
+.unmigratable = 1,
};
static void reset_assigned_device(DeviceState *dev)
Why do you remove fields from vmstate_assigned_device?
It is useful for dump because it does not check unmigratable. If
vmstate_assigned_device
does not contain .fields, qemu will crash in vmstate_save_state().
Given that '.fields' is allowed to be NULL for some devices, I'd say
even for normal migration, QEMU should be checking for NULL in the
vmstate_save_state() code. This would prevent QEMU crashes in the case
where someone removed the .unmigratable member, but forgot to add back
a .fields member.
Daniel
--
|: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-18 09:58, Daniel P. Berrange wrote:
On Tue, Oct 18, 2011 at 03:15:29PM +0800, Wen Congyang wrote:
Hi, Jan Kiszka
At 10/10/2011 05:34 PM, Jan Kiszka Write:
On 2011-10-10 11:02, Daniel P. Berrange wrote:
On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
Run gdb with set debug remote 1 and watch the communication, it is not
that complex. But a dump command is probably simpler for those
scenarios, I agree.
I have implemented the command dump and reuse migration's code. But I meet a
problem
when I test it.
My qemu-kvm's tree is not updated about 2 months ago, because kernel.org is
down, and
I forgot to pull from github.
After I pull it from github, I find the following changes:
@@ -1523,9 +1523,7 @@ static void
assigned_dev_unregister_msix_mmio(AssignedDevice *dev)
static const VMStateDescription vmstate_assigned_device = {
.name = pci-assign,
-.fields = (VMStateField []) {
-VMSTATE_END_OF_LIST()
-}
+.unmigratable = 1,
};
static void reset_assigned_device(DeviceState *dev)
Why do you remove fields from vmstate_assigned_device?
It is useful for dump because it does not check unmigratable. If
vmstate_assigned_device
does not contain .fields, qemu will crash in vmstate_save_state().
Given that '.fields' is allowed to be NULL for some devices, I'd say
even for normal migration, QEMU should be checking for NULL in the
vmstate_save_state() code. This would prevent QEMU crashes in the case
where someone removed the .unmigratable member, but forgot to add back
a .fields member.
That crash wouldn't be bad because removinb unmigratable without adding
proper fields is almost always a bug.
Jan
--
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-18 09:58, Wen Congyang wrote: At 10/18/2011 03:52 PM, Jan Kiszka Write: On 2011-10-18 09:15, Wen Congyang wrote: Hi, Jan Kiszka At 10/10/2011 05:34 PM, Jan Kiszka Write: On 2011-10-10 11:02, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote: Run gdb with set debug remote 1 and watch the communication, it is not that complex. But a dump command is probably simpler for those scenarios, I agree. I have implemented the command dump and reuse migration's code. But I meet a problem when I test it. Using migration code for dump is most probably the wrong approach as you saw through that conflict. All you need are the register states and the RAM. Reuse gdbstub services for this. Hmm, if the migration code can not be reused, I think we should define a new qemu's vmcore format, and add some codes into crash to support such format. Please try to avoid defining something new. Unless there is a striking reason, standard gdb core files should be generated so that you can load the dump directly into gdb for analysis. Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
At 10/18/2011 04:19 PM, Jan Kiszka Write: On 2011-10-18 09:58, Wen Congyang wrote: At 10/18/2011 03:52 PM, Jan Kiszka Write: On 2011-10-18 09:15, Wen Congyang wrote: Hi, Jan Kiszka At 10/10/2011 05:34 PM, Jan Kiszka Write: On 2011-10-10 11:02, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote: Run gdb with set debug remote 1 and watch the communication, it is not that complex. But a dump command is probably simpler for those scenarios, I agree. I have implemented the command dump and reuse migration's code. But I meet a problem when I test it. Using migration code for dump is most probably the wrong approach as you saw through that conflict. All you need are the register states and the RAM. Reuse gdbstub services for this. Hmm, if the migration code can not be reused, I think we should define a new qemu's vmcore format, and add some codes into crash to support such format. Please try to avoid defining something new. Unless there is a striking reason, standard gdb core files should be generated so that you can load the dump directly into gdb for analysis. I am not sure whehter the standard gdb core files can not be analyzed by crash. If not, I think we should define something new because it's easier to use crash than gdb to analyze the core files. Thanks Wen Congyang Jan
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-18 10:17, Jan Kiszka wrote:
On 2011-10-18 09:58, Daniel P. Berrange wrote:
On Tue, Oct 18, 2011 at 03:15:29PM +0800, Wen Congyang wrote:
Hi, Jan Kiszka
At 10/10/2011 05:34 PM, Jan Kiszka Write:
On 2011-10-10 11:02, Daniel P. Berrange wrote:
On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
Run gdb with set debug remote 1 and watch the communication, it is not
that complex. But a dump command is probably simpler for those
scenarios, I agree.
I have implemented the command dump and reuse migration's code. But I meet
a problem
when I test it.
My qemu-kvm's tree is not updated about 2 months ago, because kernel.org is
down, and
I forgot to pull from github.
After I pull it from github, I find the following changes:
@@ -1523,9 +1523,7 @@ static void
assigned_dev_unregister_msix_mmio(AssignedDevice *dev)
static const VMStateDescription vmstate_assigned_device = {
.name = pci-assign,
-.fields = (VMStateField []) {
-VMSTATE_END_OF_LIST()
-}
+.unmigratable = 1,
};
static void reset_assigned_device(DeviceState *dev)
Why do you remove fields from vmstate_assigned_device?
It is useful for dump because it does not check unmigratable. If
vmstate_assigned_device
does not contain .fields, qemu will crash in vmstate_save_state().
Given that '.fields' is allowed to be NULL for some devices, I'd say
even for normal migration, QEMU should be checking for NULL in the
vmstate_save_state() code. This would prevent QEMU crashes in the case
where someone removed the .unmigratable member, but forgot to add back
a .fields member.
That crash wouldn't be bad because removinb unmigratable without adding
proper fields is almost always a bug.
Err, s/almost//.
Jan
--
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On Tue, Oct 18, 2011 at 10:17:27AM +0200, Jan Kiszka wrote:
On 2011-10-18 09:58, Daniel P. Berrange wrote:
On Tue, Oct 18, 2011 at 03:15:29PM +0800, Wen Congyang wrote:
Hi, Jan Kiszka
At 10/10/2011 05:34 PM, Jan Kiszka Write:
On 2011-10-10 11:02, Daniel P. Berrange wrote:
On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
Run gdb with set debug remote 1 and watch the communication, it is not
that complex. But a dump command is probably simpler for those
scenarios, I agree.
I have implemented the command dump and reuse migration's code. But I meet
a problem
when I test it.
My qemu-kvm's tree is not updated about 2 months ago, because kernel.org
is down, and
I forgot to pull from github.
After I pull it from github, I find the following changes:
@@ -1523,9 +1523,7 @@ static void
assigned_dev_unregister_msix_mmio(AssignedDevice *dev)
static const VMStateDescription vmstate_assigned_device = {
.name = pci-assign,
-.fields = (VMStateField []) {
-VMSTATE_END_OF_LIST()
-}
+.unmigratable = 1,
};
static void reset_assigned_device(DeviceState *dev)
Why do you remove fields from vmstate_assigned_device?
It is useful for dump because it does not check unmigratable. If
vmstate_assigned_device
does not contain .fields, qemu will crash in vmstate_save_state().
Given that '.fields' is allowed to be NULL for some devices, I'd say
even for normal migration, QEMU should be checking for NULL in the
vmstate_save_state() code. This would prevent QEMU crashes in the case
where someone removed the .unmigratable member, but forgot to add back
a .fields member.
That crash wouldn't be bad because removinb unmigratable without adding
proper fields is almost always a bug.
This is the kind of example of why QEMU is too crashy in general. We
should be defensive against such bugs, by checking that preconditions
are valid in the code, even if we don't expect them to happen often.
Daniel
--
|: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-18 10:25, Wen Congyang wrote: At 10/18/2011 04:19 PM, Jan Kiszka Write: On 2011-10-18 09:58, Wen Congyang wrote: At 10/18/2011 03:52 PM, Jan Kiszka Write: On 2011-10-18 09:15, Wen Congyang wrote: Hi, Jan Kiszka At 10/10/2011 05:34 PM, Jan Kiszka Write: On 2011-10-10 11:02, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote: Run gdb with set debug remote 1 and watch the communication, it is not that complex. But a dump command is probably simpler for those scenarios, I agree. I have implemented the command dump and reuse migration's code. But I meet a problem when I test it. Using migration code for dump is most probably the wrong approach as you saw through that conflict. All you need are the register states and the RAM. Reuse gdbstub services for this. Hmm, if the migration code can not be reused, I think we should define a new qemu's vmcore format, and add some codes into crash to support such format. Please try to avoid defining something new. Unless there is a striking reason, standard gdb core files should be generated so that you can load the dump directly into gdb for analysis. I am not sure whehter the standard gdb core files can not be analyzed by crash. If not, I think we should define something new because it's easier to use crash than gdb to analyze the core files. gdb allows you to walk up the frame and print variables (globals local) etc. Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
At 10/18/2011 04:26 PM, Jan Kiszka Write: On 2011-10-18 10:25, Wen Congyang wrote: At 10/18/2011 04:19 PM, Jan Kiszka Write: On 2011-10-18 09:58, Wen Congyang wrote: At 10/18/2011 03:52 PM, Jan Kiszka Write: On 2011-10-18 09:15, Wen Congyang wrote: Hi, Jan Kiszka At 10/10/2011 05:34 PM, Jan Kiszka Write: On 2011-10-10 11:02, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote: Run gdb with set debug remote 1 and watch the communication, it is not that complex. But a dump command is probably simpler for those scenarios, I agree. I have implemented the command dump and reuse migration's code. But I meet a problem when I test it. Using migration code for dump is most probably the wrong approach as you saw through that conflict. All you need are the register states and the RAM. Reuse gdbstub services for this. Hmm, if the migration code can not be reused, I think we should define a new qemu's vmcore format, and add some codes into crash to support such format. Please try to avoid defining something new. Unless there is a striking reason, standard gdb core files should be generated so that you can load the dump directly into gdb for analysis. I am not sure whehter the standard gdb core files can not be analyzed by crash. If not, I think we should define something new because it's easier to use crash than gdb to analyze the core files. gdb allows you to walk up the frame and print variables (globals local) etc. Crash uses gdb to provide common function, and you can also use all the gdb commands in crash. Thanks Wen Congyang Jan
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-18 10:25, Daniel P. Berrange wrote:
On Tue, Oct 18, 2011 at 10:17:27AM +0200, Jan Kiszka wrote:
On 2011-10-18 09:58, Daniel P. Berrange wrote:
On Tue, Oct 18, 2011 at 03:15:29PM +0800, Wen Congyang wrote:
Hi, Jan Kiszka
At 10/10/2011 05:34 PM, Jan Kiszka Write:
On 2011-10-10 11:02, Daniel P. Berrange wrote:
On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
Run gdb with set debug remote 1 and watch the communication, it is not
that complex. But a dump command is probably simpler for those
scenarios, I agree.
I have implemented the command dump and reuse migration's code. But I meet
a problem
when I test it.
My qemu-kvm's tree is not updated about 2 months ago, because kernel.org
is down, and
I forgot to pull from github.
After I pull it from github, I find the following changes:
@@ -1523,9 +1523,7 @@ static void
assigned_dev_unregister_msix_mmio(AssignedDevice *dev)
static const VMStateDescription vmstate_assigned_device = {
.name = pci-assign,
-.fields = (VMStateField []) {
-VMSTATE_END_OF_LIST()
-}
+.unmigratable = 1,
};
static void reset_assigned_device(DeviceState *dev)
Why do you remove fields from vmstate_assigned_device?
It is useful for dump because it does not check unmigratable. If
vmstate_assigned_device
does not contain .fields, qemu will crash in vmstate_save_state().
Given that '.fields' is allowed to be NULL for some devices, I'd say
even for normal migration, QEMU should be checking for NULL in the
vmstate_save_state() code. This would prevent QEMU crashes in the case
where someone removed the .unmigratable member, but forgot to add back
a .fields member.
That crash wouldn't be bad because removinb unmigratable without adding
proper fields is almost always a bug.
This is the kind of example of why QEMU is too crashy in general. We
should be defensive against such bugs, by checking that preconditions
are valid in the code, even if we don't expect them to happen often.
A serious bug in a device model is tricky to handle gracefully. Given
that this case is fairly clear (specifically for the code reviewer), I'm
not sure what defensive measures you want to apply and what benefit they
would provide.
Jan
--
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-18 10:31, Wen Congyang wrote: At 10/18/2011 04:26 PM, Jan Kiszka Write: On 2011-10-18 10:25, Wen Congyang wrote: At 10/18/2011 04:19 PM, Jan Kiszka Write: On 2011-10-18 09:58, Wen Congyang wrote: At 10/18/2011 03:52 PM, Jan Kiszka Write: On 2011-10-18 09:15, Wen Congyang wrote: Hi, Jan Kiszka At 10/10/2011 05:34 PM, Jan Kiszka Write: On 2011-10-10 11:02, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote: Run gdb with set debug remote 1 and watch the communication, it is not that complex. But a dump command is probably simpler for those scenarios, I agree. I have implemented the command dump and reuse migration's code. But I meet a problem when I test it. Using migration code for dump is most probably the wrong approach as you saw through that conflict. All you need are the register states and the RAM. Reuse gdbstub services for this. Hmm, if the migration code can not be reused, I think we should define a new qemu's vmcore format, and add some codes into crash to support such format. Please try to avoid defining something new. Unless there is a striking reason, standard gdb core files should be generated so that you can load the dump directly into gdb for analysis. I am not sure whehter the standard gdb core files can not be analyzed by crash. If not, I think we should define something new because it's easier to use crash than gdb to analyze the core files. gdb allows you to walk up the frame and print variables (globals local) etc. Crash uses gdb to provide common function, and you can also use all the gdb commands in crash. That what's the added value here when I can use gdb directly? Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On Tue, Oct 18, 2011 at 10:31:10AM +0200, Jan Kiszka wrote: On 2011-10-18 10:31, Wen Congyang wrote: At 10/18/2011 04:26 PM, Jan Kiszka Write: On 2011-10-18 10:25, Wen Congyang wrote: At 10/18/2011 04:19 PM, Jan Kiszka Write: On 2011-10-18 09:58, Wen Congyang wrote: At 10/18/2011 03:52 PM, Jan Kiszka Write: On 2011-10-18 09:15, Wen Congyang wrote: Hi, Jan Kiszka At 10/10/2011 05:34 PM, Jan Kiszka Write: On 2011-10-10 11:02, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote: Run gdb with set debug remote 1 and watch the communication, it is not that complex. But a dump command is probably simpler for those scenarios, I agree. I have implemented the command dump and reuse migration's code. But I meet a problem when I test it. Using migration code for dump is most probably the wrong approach as you saw through that conflict. All you need are the register states and the RAM. Reuse gdbstub services for this. Hmm, if the migration code can not be reused, I think we should define a new qemu's vmcore format, and add some codes into crash to support such format. Please try to avoid defining something new. Unless there is a striking reason, standard gdb core files should be generated so that you can load the dump directly into gdb for analysis. I am not sure whehter the standard gdb core files can not be analyzed by crash. If not, I think we should define something new because it's easier to use crash than gdb to analyze the core files. gdb allows you to walk up the frame and print variables (globals local) etc. Crash uses gdb to provide common function, and you can also use all the gdb commands in crash. That what's the added value here when I can use gdb directly? Crash can decode kernel structures, providing for example process listings and debugging into userspace processes. Crash actually reuses gdb's code too, so it's kind of a superset. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-18 10:34, Richard W.M. Jones wrote: On Tue, Oct 18, 2011 at 10:31:10AM +0200, Jan Kiszka wrote: On 2011-10-18 10:31, Wen Congyang wrote: At 10/18/2011 04:26 PM, Jan Kiszka Write: On 2011-10-18 10:25, Wen Congyang wrote: At 10/18/2011 04:19 PM, Jan Kiszka Write: On 2011-10-18 09:58, Wen Congyang wrote: At 10/18/2011 03:52 PM, Jan Kiszka Write: On 2011-10-18 09:15, Wen Congyang wrote: Hi, Jan Kiszka At 10/10/2011 05:34 PM, Jan Kiszka Write: On 2011-10-10 11:02, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote: Run gdb with set debug remote 1 and watch the communication, it is not that complex. But a dump command is probably simpler for those scenarios, I agree. I have implemented the command dump and reuse migration's code. But I meet a problem when I test it. Using migration code for dump is most probably the wrong approach as you saw through that conflict. All you need are the register states and the RAM. Reuse gdbstub services for this. Hmm, if the migration code can not be reused, I think we should define a new qemu's vmcore format, and add some codes into crash to support such format. Please try to avoid defining something new. Unless there is a striking reason, standard gdb core files should be generated so that you can load the dump directly into gdb for analysis. I am not sure whehter the standard gdb core files can not be analyzed by crash. If not, I think we should define something new because it's easier to use crash than gdb to analyze the core files. gdb allows you to walk up the frame and print variables (globals local) etc. Crash uses gdb to provide common function, and you can also use all the gdb commands in crash. That what's the added value here when I can use gdb directly? Crash can decode kernel structures, providing for example process listings and debugging into userspace processes. Crash actually reuses gdb's code too, so it's kind of a superset. Yeah, I see. Could also be solved via gdb scripts, but crash is already there. But let's see if the formats actually differ. In the end, crash is just parsing the same information that also gdb sees. Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-18 10:36, Jan Kiszka wrote: Crash can decode kernel structures, providing for example process listings and debugging into userspace processes. Crash actually reuses gdb's code too, so it's kind of a superset. Yeah, I see. Could also be solved via gdb scripts, but crash is already there. [ BTW, crash is for the dead. But having those features in form of gdb scripts would also allow live system analysis. Looks like it's worth obsoleting crash on the long run. ] Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
At 10/18/2011 04:36 PM, Jan Kiszka Write:
On 2011-10-18 10:34, Richard W.M. Jones wrote:
Yeah, I see. Could also be solved via gdb scripts, but crash is already
there.
But let's see if the formats actually differ. In the end, crash is just
parsing the same information that also gdb sees.
I think the format can be similar with diskdump/kdump/netdump:
dump_header: 1 block
sub header: n blocks(n is stored in dump_header)
bitmap: m blocks(2m is stored in dump_header)
dumpable bitmap: m blocks
memory data(We can know whether a page is stored in the core by bitmap and
dumpable bitmap)
The format of dump header(It's like kdump/diskdump):
struct disk_dump_header {
charsignature[SIG_LEN]; /* = QEMU */
int header_version; /* Dump header version */
struct new_utsname utsname;/* copy of system_utsname */
struct timeval timestamp; /* Time stamp */
unsigned intstatus;
int block_size; /* Size of a block in byte */
int sub_hdr_size; /* Size of arch dependent
header in blocks */
unsigned intbitmap_blocks; /* Size of Memory bitmap in
block */
unsigned intmax_mapnr; /* = max_mapnr */
unsigned inttotal_ram_blocks;/* Number of blocks should be
written */
unsigned intdevice_blocks; /* Number of total blocks in
* the dump device */
unsigned intwritten_blocks; /* Number of written blocks */
unsigned intcurrent_cpu;/* CPU# which handles dump */
int nr_cpus;/* Number of CPUs */
};
The sub header can contains all registers's value on each vcpu, and other
information,
for example:
struct qemu_sub_header {
unsigned long start_pfn;
unsigned long end_pfn;
off_t offset_note;
unsigned long size_note;
};
Thanks
Wen Congyang
Jan
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-18 11:43, Wen Congyang wrote:
At 10/18/2011 04:36 PM, Jan Kiszka Write:
On 2011-10-18 10:34, Richard W.M. Jones wrote:
Yeah, I see. Could also be solved via gdb scripts, but crash is already
there.
But let's see if the formats actually differ. In the end, crash is just
parsing the same information that also gdb sees.
I think the format can be similar with diskdump/kdump/netdump:
dump_header: 1 block
sub header: n blocks(n is stored in dump_header)
bitmap: m blocks(2m is stored in dump_header)
dumpable bitmap: m blocks
memory data(We can know whether a page is stored in the core by bitmap and
dumpable bitmap)
The format of dump header(It's like kdump/diskdump):
struct disk_dump_header {
charsignature[SIG_LEN]; /* = QEMU */
int header_version; /* Dump header version */
struct new_utsname utsname;/* copy of system_utsname */
struct timeval timestamp; /* Time stamp */
unsigned intstatus;
int block_size; /* Size of a block in byte */
int sub_hdr_size; /* Size of arch dependent
header in blocks */
unsigned intbitmap_blocks; /* Size of Memory bitmap in
block */
unsigned intmax_mapnr; /* = max_mapnr */
unsigned inttotal_ram_blocks;/* Number of blocks should be
written */
unsigned intdevice_blocks; /* Number of total blocks in
* the dump device */
unsigned intwritten_blocks; /* Number of written blocks */
unsigned intcurrent_cpu;/* CPU# which handles dump */
int nr_cpus;/* Number of CPUs */
};
The sub header can contains all registers's value on each vcpu, and other
information,
for example:
struct qemu_sub_header {
unsigned long start_pfn;
unsigned long end_pfn;
off_t offset_note;
unsigned long size_note;
};
So is this a standard format or only similar to something? Which tools
will understand it out-of-the-box? If it's not standard, why?
Jan
--
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 10/18/2011 10:39 AM, Jan Kiszka wrote: Yeah, I see. Could also be solved via gdb scripts, but crash is already there. [ BTW, crash is for the dead. But having those features in form of gdb scripts would also allow live system analysis. Looks like it's worth obsoleting crash on the long run. ] Crash can also do live system analysis. :) Paolo
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On Tue, Oct 18, 2011 at 12:41:10PM +0200, Paolo Bonzini wrote: On 10/18/2011 10:39 AM, Jan Kiszka wrote: Yeah, I see. Could also be solved via gdb scripts, but crash is already there. [ BTW, crash is for the dead. But having those features in form of gdb scripts would also allow live system analysis. Looks like it's worth obsoleting crash on the long run. ] Crash can also do live system analysis. :) crash in fact is gdb - with a lot of additional commands.
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-18 12:41, Paolo Bonzini wrote: On 10/18/2011 10:39 AM, Jan Kiszka wrote: Yeah, I see. Could also be solved via gdb scripts, but crash is already there. [ BTW, crash is for the dead. But having those features in form of gdb scripts would also allow live system analysis. Looks like it's worth obsoleting crash on the long run. ] Crash can also do live system analysis. :) Does it work via remote interface? And it's still the wrong way around: You have to append gdb to the majority of command when doing debugging instead of issuing additional analysis commands from the gdb shell. That's understandable given the limited scripting power of old gdbs. But that's now history thanks to its python interface. Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-18 12:42, Christoph Hellwig wrote: On Tue, Oct 18, 2011 at 12:41:10PM +0200, Paolo Bonzini wrote: On 10/18/2011 10:39 AM, Jan Kiszka wrote: Yeah, I see. Could also be solved via gdb scripts, but crash is already there. [ BTW, crash is for the dead. But having those features in form of gdb scripts would also allow live system analysis. Looks like it's worth obsoleting crash on the long run. ] Crash can also do live system analysis. :) crash in fact is gdb - with a lot of additional commands. Is there a trick to make it work against a gdbserver? Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On Tue, Oct 18, 2011 at 02:47:39PM +0200, Jan Kiszka wrote: Crash can also do live system analysis. :) crash in fact is gdb - with a lot of additional commands. Is there a trick to make it work against a gdbserver? No idea. I just noticed that crash was built around a gdb source tree and additional files when I had to hack it for a new kernel version a while ago. The optimum would be if all the crash changes were merged back into upstream gdb, but I'm not sure the gdb community would like to see all that very linux kernel specific code (especially given that it's very version dependent)
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-18 15:08, Christoph Hellwig wrote: On Tue, Oct 18, 2011 at 02:47:39PM +0200, Jan Kiszka wrote: Crash can also do live system analysis. :) crash in fact is gdb - with a lot of additional commands. Is there a trick to make it work against a gdbserver? No idea. I just noticed that crash was built around a gdb source tree and additional files when I had to hack it for a new kernel version a while ago. The optimum would be if all the crash changes were merged back into upstream gdb, but I'm not sure the gdb community would like to see all that very linux kernel specific code (especially given that it's very version dependent) That's why I'm advocating gdb scripts, maybe maintained inside the kernel tree. But I need to look closer as well to see if vanilla gdb is already able to provide all registers used by crash so far (surely not all required for perfect debugging, but that's a long-known issue). Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On Tue, Oct 18, 2011 at 12:47:23PM +0200, Jan Kiszka wrote: On 2011-10-18 12:41, Paolo Bonzini wrote: On 10/18/2011 10:39 AM, Jan Kiszka wrote: Yeah, I see. Could also be solved via gdb scripts, but crash is already there. [ BTW, crash is for the dead. But having those features in form of gdb scripts would also allow live system analysis. Looks like it's worth obsoleting crash on the long run. ] Crash can also do live system analysis. :) Does it work via remote interface? And it's still the wrong way around: You have to append gdb to the majority of command when doing debugging instead of issuing additional analysis commands from the gdb shell. That's understandable given the limited scripting power of old gdbs. But that's now history thanks to its python interface. Before this conversation heads any further into the realm of absurdity, let's get back to the key requirements for production systems: (A) No development tools can be installed. (B) The core file must be removed from the production system quickly and analyzed by on another machine (indeed, often by experts from another company entirely). Anything that involves live debugging using GDB is simply not acceptable for production systems, whether you think that's right or not. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-18 15:30, Richard W.M. Jones wrote: On Tue, Oct 18, 2011 at 12:47:23PM +0200, Jan Kiszka wrote: On 2011-10-18 12:41, Paolo Bonzini wrote: On 10/18/2011 10:39 AM, Jan Kiszka wrote: Yeah, I see. Could also be solved via gdb scripts, but crash is already there. [ BTW, crash is for the dead. But having those features in form of gdb scripts would also allow live system analysis. Looks like it's worth obsoleting crash on the long run. ] Crash can also do live system analysis. :) Does it work via remote interface? And it's still the wrong way around: You have to append gdb to the majority of command when doing debugging instead of issuing additional analysis commands from the gdb shell. That's understandable given the limited scripting power of old gdbs. But that's now history thanks to its python interface. Before this conversation heads any further into the realm of absurdity, let's get back to the key requirements for production systems: (A) No development tools can be installed. (B) The core file must be removed from the production system quickly and analyzed by on another machine (indeed, often by experts from another company entirely). Anything that involves live debugging using GDB is simply not acceptable for production systems, whether you think that's right or not. My memory is usually that bad, but not in this case. :) This discussion should also work out a bigger picture than just your (valid) scenario. We should look at meeting that requirements but also see what would be a useful foundation for future extensions like having post-crash analysis and live debugging at a similar feature level with very similar or (much better) identical tools. Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 10/18/2011 06:28 PM, Jan Kiszka wrote:
On 2011-10-18 11:43, Wen Congyang wrote:
At 10/18/2011 04:36 PM, Jan Kiszka Write:
On 2011-10-18 10:34, Richard W.M. Jones wrote:
Yeah, I see. Could also be solved via gdb scripts, but crash is already
there.
But let's see if the formats actually differ. In the end, crash is just
parsing the same information that also gdb sees.
I think the format can be similar with diskdump/kdump/netdump:
dump_header: 1 block
sub header: n blocks(n is stored in dump_header)
bitmap: m blocks(2m is stored in dump_header)
dumpable bitmap: m blocks
memory data(We can know whether a page is stored in the core by bitmap and
dumpable bitmap)
The format of dump header(It's like kdump/diskdump):
struct disk_dump_header {
charsignature[SIG_LEN]; /* = QEMU */
int header_version; /* Dump header version */
struct new_utsname utsname;/* copy of system_utsname */
struct timeval timestamp; /* Time stamp */
unsigned intstatus;
int block_size; /* Size of a block in byte */
int sub_hdr_size; /* Size of arch dependent
header in blocks */
unsigned intbitmap_blocks; /* Size of Memory bitmap in
block */
unsigned intmax_mapnr; /* = max_mapnr */
unsigned inttotal_ram_blocks;/* Number of blocks should be
written */
unsigned intdevice_blocks; /* Number of total blocks in
* the dump device */
unsigned intwritten_blocks; /* Number of written blocks */
unsigned intcurrent_cpu;/* CPU# which handles dump */
int nr_cpus;/* Number of CPUs */
};
The sub header can contains all registers's value on each vcpu, and other
information,
for example:
struct qemu_sub_header {
unsigned long start_pfn;
unsigned long end_pfn;
off_t offset_note;
unsigned long size_note;
};
So is this a standard format or only similar to something? Which tools
will understand it out-of-the-box? If it's not standard, why?
Only similar to something, and we can add a little codes into crash to
support such format.
If you have a better format, please tell me.
Thanks
Wen Congyang
Jan
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-18 15:51, Wen Congyang wrote:
On 10/18/2011 06:28 PM, Jan Kiszka wrote:
On 2011-10-18 11:43, Wen Congyang wrote:
At 10/18/2011 04:36 PM, Jan Kiszka Write:
On 2011-10-18 10:34, Richard W.M. Jones wrote:
Yeah, I see. Could also be solved via gdb scripts, but crash is already
there.
But let's see if the formats actually differ. In the end, crash is just
parsing the same information that also gdb sees.
I think the format can be similar with diskdump/kdump/netdump:
dump_header: 1 block
sub header: n blocks(n is stored in dump_header)
bitmap: m blocks(2m is stored in dump_header)
dumpable bitmap: m blocks
memory data(We can know whether a page is stored in the core by bitmap and
dumpable bitmap)
The format of dump header(It's like kdump/diskdump):
struct disk_dump_header {
charsignature[SIG_LEN]; /* = QEMU */
int header_version; /* Dump header version */
struct new_utsname utsname;/* copy of system_utsname */
struct timeval timestamp; /* Time stamp */
unsigned intstatus;
int block_size; /* Size of a block in byte */
int sub_hdr_size; /* Size of arch dependent
header in blocks */
unsigned intbitmap_blocks; /* Size of Memory bitmap in
block */
unsigned intmax_mapnr; /* = max_mapnr */
unsigned inttotal_ram_blocks;/* Number of blocks should be
written */
unsigned intdevice_blocks; /* Number of total blocks in
* the dump device */
unsigned intwritten_blocks; /* Number of written blocks */
unsigned intcurrent_cpu;/* CPU# which handles dump */
int nr_cpus;/* Number of CPUs */
};
The sub header can contains all registers's value on each vcpu, and other
information,
for example:
struct qemu_sub_header {
unsigned long start_pfn;
unsigned long end_pfn;
off_t offset_note;
unsigned long size_note;
};
So is this a standard format or only similar to something? Which tools
will understand it out-of-the-box? If it's not standard, why?
Only similar to something, and we can add a little codes into crash to
support such format.
If you have a better format, please tell me.
The format crash already processes? What do you need in addition? That
should be discussed first, not the format details.
Jan
--
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 10/18/2011 09:55 PM, Jan Kiszka wrote:
On 2011-10-18 15:51, Wen Congyang wrote:
On 10/18/2011 06:28 PM, Jan Kiszka wrote:
On 2011-10-18 11:43, Wen Congyang wrote:
At 10/18/2011 04:36 PM, Jan Kiszka Write:
On 2011-10-18 10:34, Richard W.M. Jones wrote:
Yeah, I see. Could also be solved via gdb scripts, but crash is already
there.
But let's see if the formats actually differ. In the end, crash is just
parsing the same information that also gdb sees.
I think the format can be similar with diskdump/kdump/netdump:
dump_header: 1 block
sub header: n blocks(n is stored in dump_header)
bitmap: m blocks(2m is stored in dump_header)
dumpable bitmap: m blocks
memory data(We can know whether a page is stored in the core by bitmap and
dumpable bitmap)
The format of dump header(It's like kdump/diskdump):
struct disk_dump_header {
charsignature[SIG_LEN]; /* = QEMU */
int header_version; /* Dump header version */
struct new_utsname utsname;/* copy of system_utsname */
struct timeval timestamp; /* Time stamp */
unsigned intstatus;
int block_size; /* Size of a block in byte */
int sub_hdr_size; /* Size of arch dependent
header in blocks */
unsigned intbitmap_blocks; /* Size of Memory bitmap in
block */
unsigned intmax_mapnr; /* = max_mapnr */
unsigned inttotal_ram_blocks;/* Number of blocks should be
written */
unsigned intdevice_blocks; /* Number of total blocks in
* the dump device */
unsigned intwritten_blocks; /* Number of written blocks */
unsigned intcurrent_cpu;/* CPU# which handles dump */
int nr_cpus;/* Number of CPUs */
};
The sub header can contains all registers's value on each vcpu, and other
information,
for example:
struct qemu_sub_header {
unsigned long start_pfn;
unsigned long end_pfn;
off_t offset_note;
unsigned long size_note;
};
So is this a standard format or only similar to something? Which tools
will understand it out-of-the-box? If it's not standard, why?
Only similar to something, and we can add a little codes into crash to
support such format.
If you have a better format, please tell me.
The format crash already processes? What do you need in addition? That
should be discussed first, not the format details.
A way to dump the memory to a file, and crash can analyze it. We always
use crash
to analyze such file because it is easier than gdb to use.
We can use 'virsh dump' to dump the memory to a file, but 'virsh dump'
can not
work while host pci device is used by guest(The reason is that 'virsh
dump' uses
monitor command migrate to dump the memory to a file).
gdbserver is not the way need, because the guest should be restarted
quickly.
I think introducing a new monitor command(named dump) to do such thing,
and you
have agreed it in another mail.
I first to try to reuse migration's code, but I meet some problem(I have
pointed it
out in an early mail today).
You advice me to reuse gdbstub's code, but I think crash can not support
such format.
And I want to implement it without reusing any other codes in qemu(If
so, we should determine
the core's format).
I am happy if you have some other ideas.
Thanks
Wen Congyang
Jan
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-18 16:17, Wen Congyang wrote:
On 10/18/2011 09:55 PM, Jan Kiszka wrote:
On 2011-10-18 15:51, Wen Congyang wrote:
On 10/18/2011 06:28 PM, Jan Kiszka wrote:
On 2011-10-18 11:43, Wen Congyang wrote:
At 10/18/2011 04:36 PM, Jan Kiszka Write:
On 2011-10-18 10:34, Richard W.M. Jones wrote:
Yeah, I see. Could also be solved via gdb scripts, but crash is already
there.
But let's see if the formats actually differ. In the end, crash is just
parsing the same information that also gdb sees.
I think the format can be similar with diskdump/kdump/netdump:
dump_header: 1 block
sub header: n blocks(n is stored in dump_header)
bitmap: m blocks(2m is stored in dump_header)
dumpable bitmap: m blocks
memory data(We can know whether a page is stored in the core by bitmap
and dumpable bitmap)
The format of dump header(It's like kdump/diskdump):
struct disk_dump_header {
charsignature[SIG_LEN]; /* = QEMU */
int header_version; /* Dump header version */
struct new_utsname utsname;/* copy of system_utsname */
struct timeval timestamp; /* Time stamp */
unsigned intstatus;
int block_size; /* Size of a block in byte */
int sub_hdr_size; /* Size of arch dependent
header in blocks */
unsigned intbitmap_blocks; /* Size of Memory bitmap in
block */
unsigned intmax_mapnr; /* = max_mapnr */
unsigned inttotal_ram_blocks;/* Number of blocks should be
written */
unsigned intdevice_blocks; /* Number of total blocks in
* the dump device */
unsigned intwritten_blocks; /* Number of written blocks */
unsigned intcurrent_cpu;/* CPU# which handles dump */
int nr_cpus;/* Number of CPUs */
};
The sub header can contains all registers's value on each vcpu, and other
information,
for example:
struct qemu_sub_header {
unsigned long start_pfn;
unsigned long end_pfn;
off_t offset_note;
unsigned long size_note;
};
So is this a standard format or only similar to something? Which tools
will understand it out-of-the-box? If it's not standard, why?
Only similar to something, and we can add a little codes into crash to
support such format.
If you have a better format, please tell me.
The format crash already processes? What do you need in addition? That
should be discussed first, not the format details.
A way to dump the memory to a file, and crash can analyze it.
I was talking about information you need in the crash dump file that is
not contained in those files crash processes by default. If there is no
such information, then it looks strange to me defining a new format.
Jan
--
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 10/18/2011 10:21 PM, Jan Kiszka wrote: On 2011-10-18 16:17, Wen Congyang wrote: I was talking about information you need in the crash dump file that is not contained in those files crash processes by default. If there is no such information, then it looks strange to me defining a new format. Sorry for my bad english, I can not understant what information you are talking about. Do you mean if the core file does not contains such information and crash can not process it? Thanks Wen Congyang Jan
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-18 17:04, Wen Congyang wrote: On 10/18/2011 10:21 PM, Jan Kiszka wrote: On 2011-10-18 16:17, Wen Congyang wrote: I was talking about information you need in the crash dump file that is not contained in those files crash processes by default. If there is no such information, then it looks strange to me defining a new format. Sorry for my bad english, I can not understant what information you are talking about. Do you mean if the core file does not contains such information and crash can not process it? OK, I guess we are just talking past each other: Are you proposing the kernel dump format (as crash takes for its dumpfile parameter)? Should QEMU write out dumps in that file format? Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
At 10/18/2011 11:27 PM, Jan Kiszka Write: On 2011-10-18 17:04, Wen Congyang wrote: On 10/18/2011 10:21 PM, Jan Kiszka wrote: On 2011-10-18 16:17, Wen Congyang wrote: I was talking about information you need in the crash dump file that is not contained in those files crash processes by default. If there is no such information, then it looks strange to me defining a new format. Sorry for my bad english, I can not understant what information you are talking about. Do you mean if the core file does not contains such information and crash can not process it? OK, I guess we are just talking past each other: Are you proposing the kernel dump format (as crash takes for its dumpfile parameter)? Should QEMU write out dumps in that file format? Yes. We can reuse gdbstub's code, and use gdb scripts to instead of crash. But I do not like this way, because we should know the guest kernel version and it's config file. We should do too many things that crash has done for us. So I think qemu should write out dumps in that file format. Thanks Wen Congyang Jan
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On Tue, 18 Oct 2011 10:31:10 +0200 Jan Kiszka jan.kis...@siemens.com wrote: On 2011-10-18 10:31, Wen Congyang wrote: At 10/18/2011 04:26 PM, Jan Kiszka Write: On 2011-10-18 10:25, Wen Congyang wrote: At 10/18/2011 04:19 PM, Jan Kiszka Write: On 2011-10-18 09:58, Wen Congyang wrote: At 10/18/2011 03:52 PM, Jan Kiszka Write: On 2011-10-18 09:15, Wen Congyang wrote: Hi, Jan Kiszka At 10/10/2011 05:34 PM, Jan Kiszka Write: On 2011-10-10 11:02, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote: Run gdb with set debug remote 1 and watch the communication, it is not that complex. But a dump command is probably simpler for those scenarios, I agree. I have implemented the command dump and reuse migration's code. But I meet a problem when I test it. Using migration code for dump is most probably the wrong approach as you saw through that conflict. All you need are the register states and the RAM. Reuse gdbstub services for this. Hmm, if the migration code can not be reused, I think we should define a new qemu's vmcore format, and add some codes into crash to support such format. Please try to avoid defining something new. Unless there is a striking reason, standard gdb core files should be generated so that you can load the dump directly into gdb for analysis. I am not sure whehter the standard gdb core files can not be analyzed by crash. If not, I think we should define something new because it's easier to use crash than gdb to analyze the core files. gdb allows you to walk up the frame and print variables (globals local) etc. Crash uses gdb to provide common function, and you can also use all the gdb commands in crash. That what's the added value here when I can use gdb directly? I didn't read full story but 'crash' is used for investigating kernel core generated by kdump for several years. Considering support service guys, virsh dump should support a format for crash because they can't work well at investigating vmcore by gdb. crash has several functionality useful for them as 'show kerne log', 'focus on a cpu' 'for-each-task', 'for-each-vma', 'extract ftrace log' etc. Anyway, if a man, who is not developper of qemu/kvm, should learn 2 tools for investigating kernel dump, it sounds harmful. Thanks, -Kame
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-11 04:20, Wen Congyang wrote: The other reason why it would be good, is that we would then have a clearly defined standard QEMU dump format, instead of libvirt dump format for QEMU A core file would be that format - for direct gdb processing. No proprietary re-inventions please. I have no personal attachment to any particular format, so if the standard core file format is possible, then we should definitely try to use it in QEMU. I do not know whether there is such standard format. But IIRC, the format of kdump, netdump, and diskdump is very similar. If we want to use such format in qemu, I think we can not do live dump in libvirt. You said you wanted to perform crash dumps - so there hardly anything alive at this point. Jan signature.asc Description: OpenPGP digital signature
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
At 10/11/2011 02:58 PM, Jan Kiszka Write: On 2011-10-11 04:20, Wen Congyang wrote: The other reason why it would be good, is that we would then have a clearly defined standard QEMU dump format, instead of libvirt dump format for QEMU A core file would be that format - for direct gdb processing. No proprietary re-inventions please. I have no personal attachment to any particular format, so if the standard core file format is possible, then we should definitely try to use it in QEMU. I do not know whether there is such standard format. But IIRC, the format of kdump, netdump, and diskdump is very similar. If we want to use such format in qemu, I think we can not do live dump in libvirt. You said you wanted to perform crash dumps - so there hardly anything alive at this point. The purpose of 'virsh dump' is for crash dumps, but we can not prevent the user doing it when the kernel is alive. Thanks Wen Congyang Jan
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-10 04:21, Wen Congyang wrote: At 10/09/2011 06:23 PM, Richard W.M. Jones Write: On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote: As explained in the other replies: It is way more future-proof to use an interface for this which was designed for it (remote gdb) instead of artificially relaxing reasonable constraints of the migration mechanism plus having to follow that format with the post-processing tool. Any interface that isn't get this information off my production server *now* so that I can get the server restarted, and send it to an expert to analyse -- is a poor interface, whether it was designed like that or not. Perhaps we don't have the right interface at all, but remote gdb is not it. What about the following idea? Introduce a new monitor command named dump, and this command accepts a filename. We can use almost all migration's code. We use this command to dump guest's memory, so there is no need to check whether the guest has a unmigratable device. I do not want to reject this proposal categorically, but I would like to see the gdb path fail /wrt essential requirements first. So far I don't see it would. Jan signature.asc Description: OpenPGP digital signature
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
At 10/10/2011 02:52 PM, Jan Kiszka Write: On 2011-10-10 04:21, Wen Congyang wrote: At 10/09/2011 06:23 PM, Richard W.M. Jones Write: On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote: As explained in the other replies: It is way more future-proof to use an interface for this which was designed for it (remote gdb) instead of artificially relaxing reasonable constraints of the migration mechanism plus having to follow that format with the post-processing tool. Any interface that isn't get this information off my production server *now* so that I can get the server restarted, and send it to an expert to analyse -- is a poor interface, whether it was designed like that or not. Perhaps we don't have the right interface at all, but remote gdb is not it. What about the following idea? Introduce a new monitor command named dump, and this command accepts a filename. We can use almost all migration's code. We use this command to dump guest's memory, so there is no need to check whether the guest has a unmigratable device. I do not want to reject this proposal categorically, but I would like to see the gdb path fail /wrt essential requirements first. So far I don't see it would. ‘gdb path fail /wrt essential requirements’ what does it mean? Thanks Wen Congyang Jan
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-10 08:59, Wen Congyang wrote: At 10/10/2011 02:52 PM, Jan Kiszka Write: On 2011-10-10 04:21, Wen Congyang wrote: At 10/09/2011 06:23 PM, Richard W.M. Jones Write: On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote: As explained in the other replies: It is way more future-proof to use an interface for this which was designed for it (remote gdb) instead of artificially relaxing reasonable constraints of the migration mechanism plus having to follow that format with the post-processing tool. Any interface that isn't get this information off my production server *now* so that I can get the server restarted, and send it to an expert to analyse -- is a poor interface, whether it was designed like that or not. Perhaps we don't have the right interface at all, but remote gdb is not it. What about the following idea? Introduce a new monitor command named dump, and this command accepts a filename. We can use almost all migration's code. We use this command to dump guest's memory, so there is no need to check whether the guest has a unmigratable device. I do not want to reject this proposal categorically, but I would like to see the gdb path fail /wrt essential requirements first. So far I don't see it would. ‘gdb path fail /wrt essential requirements’ what does it mean? That you explain why reading reading memory and processor states via the remote gdb interface and dumping it into a proper core file cannot be made working for you. Jan signature.asc Description: OpenPGP digital signature
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
Am 10.10.2011 um 08:52 schrieb Jan Kiszka jan.kis...@web.de: On 2011-10-10 04:21, Wen Congyang wrote: At 10/09/2011 06:23 PM, Richard W.M. Jones Write: On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote: As explained in the other replies: It is way more future-proof to use an interface for this which was designed for it (remote gdb) instead of artificially relaxing reasonable constraints of the migration mechanism plus having to follow that format with the post-processing tool. Any interface that isn't get this information off my production server *now* so that I can get the server restarted, and send it to an expert to analyse -- is a poor interface, whether it was designed like that or not. Perhaps we don't have the right interface at all, but remote gdb is not it. What about the following idea? Introduce a new monitor command named dump, and this command accepts a filename. We can use almost all migration's code. We use this command to dump guest's memory, so there is no need to check whether the guest has a unmigratable device. I do not want to reject this proposal categorically, but I would like to see the gdb path fail /wrt essential requirements first. So far I don't see it would. Through gdb you access virtual address space, whilein this case you really want physical. All you need to create a core file is a memory dump and a register dump. For registers, gdb sounds like a good interface. For memory, why not just use pmemsave? Maybe add a monitor/qmp command that tells you all ram regions, so pmemsave can be invoked intelligently. The alternative to all this is obviously an in-qemu 'core' command that creates core files inside of qemu. That's where I think it really belongs to. Libvirt is a management tool stack, not a I don't like to work on qemu code so I push code to the layer on top mesh of things. Or at least that's how it's really supposed to be :). Alex
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-10 09:08, Alexander Graf wrote: Am 10.10.2011 um 08:52 schrieb Jan Kiszka jan.kis...@web.de: On 2011-10-10 04:21, Wen Congyang wrote: At 10/09/2011 06:23 PM, Richard W.M. Jones Write: On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote: As explained in the other replies: It is way more future-proof to use an interface for this which was designed for it (remote gdb) instead of artificially relaxing reasonable constraints of the migration mechanism plus having to follow that format with the post-processing tool. Any interface that isn't get this information off my production server *now* so that I can get the server restarted, and send it to an expert to analyse -- is a poor interface, whether it was designed like that or not. Perhaps we don't have the right interface at all, but remote gdb is not it. What about the following idea? Introduce a new monitor command named dump, and this command accepts a filename. We can use almost all migration's code. We use this command to dump guest's memory, so there is no need to check whether the guest has a unmigratable device. I do not want to reject this proposal categorically, but I would like to see the gdb path fail /wrt essential requirements first. So far I don't see it would. Through gdb you access virtual address space, whilein this case you really want physical. You want to attach a debugger afterwards as well. So this is just about offlining the gdb session the end user should not have to handle himself. Jan signature.asc Description: OpenPGP digital signature
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
At 10/10/2011 03:01 PM, Jan Kiszka Write: On 2011-10-10 08:59, Wen Congyang wrote: At 10/10/2011 02:52 PM, Jan Kiszka Write: On 2011-10-10 04:21, Wen Congyang wrote: At 10/09/2011 06:23 PM, Richard W.M. Jones Write: On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote: As explained in the other replies: It is way more future-proof to use an interface for this which was designed for it (remote gdb) instead of artificially relaxing reasonable constraints of the migration mechanism plus having to follow that format with the post-processing tool. Any interface that isn't get this information off my production server *now* so that I can get the server restarted, and send it to an expert to analyse -- is a poor interface, whether it was designed like that or not. Perhaps we don't have the right interface at all, but remote gdb is not it. What about the following idea? Introduce a new monitor command named dump, and this command accepts a filename. We can use almost all migration's code. We use this command to dump guest's memory, so there is no need to check whether the guest has a unmigratable device. I do not want to reject this proposal categorically, but I would like to see the gdb path fail /wrt essential requirements first. So far I don't see it would. ‘gdb path fail /wrt essential requirements’ what does it mean? That you explain why reading reading memory and processor states via the remote gdb interface and dumping it into a proper core file cannot be made working for you. First, I think crash can not analyze such core file. But it is not very important. What is remote gdb interface? Do you mean that: the supporter uses gdb from another machine to connect to customer's machine and get the data? If so, this way can not be used when the customer needs to dump the guest's memory automatically when watchdog timeouts. Thanks Wen Congyang Jan
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-10 09:17, Wen Congyang wrote: At 10/10/2011 03:01 PM, Jan Kiszka Write: On 2011-10-10 08:59, Wen Congyang wrote: At 10/10/2011 02:52 PM, Jan Kiszka Write: On 2011-10-10 04:21, Wen Congyang wrote: At 10/09/2011 06:23 PM, Richard W.M. Jones Write: On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote: As explained in the other replies: It is way more future-proof to use an interface for this which was designed for it (remote gdb) instead of artificially relaxing reasonable constraints of the migration mechanism plus having to follow that format with the post-processing tool. Any interface that isn't get this information off my production server *now* so that I can get the server restarted, and send it to an expert to analyse -- is a poor interface, whether it was designed like that or not. Perhaps we don't have the right interface at all, but remote gdb is not it. What about the following idea? Introduce a new monitor command named dump, and this command accepts a filename. We can use almost all migration's code. We use this command to dump guest's memory, so there is no need to check whether the guest has a unmigratable device. I do not want to reject this proposal categorically, but I would like to see the gdb path fail /wrt essential requirements first. So far I don't see it would. ‘gdb path fail /wrt essential requirements’ what does it mean? That you explain why reading reading memory and processor states via the remote gdb interface and dumping it into a proper core file cannot be made working for you. First, I think crash can not analyze such core file. But it is not very important. What is remote gdb interface? man qemu - gdb. Do you mean that: the supporter uses gdb from another machine Or locally. There are various transports possible. to connect to customer's machine and get the data? If so, this way can not be used when the customer needs to dump the guest's memory automatically when watchdog timeouts. It is just another channel that can conceptually be used like the monitor, by a management app like libvirt, directly or indirectly via a scripted gdb frontend, or also by a human who wants to save some ongoing gdb session for later analysis. This dual use make such an approach the preferred one. Jan signature.asc Description: OpenPGP digital signature
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
At 10/10/2011 03:22 PM, Jan Kiszka Write: On 2011-10-10 09:17, Wen Congyang wrote: At 10/10/2011 03:01 PM, Jan Kiszka Write: On 2011-10-10 08:59, Wen Congyang wrote: At 10/10/2011 02:52 PM, Jan Kiszka Write: On 2011-10-10 04:21, Wen Congyang wrote: At 10/09/2011 06:23 PM, Richard W.M. Jones Write: On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote: As explained in the other replies: It is way more future-proof to use an interface for this which was designed for it (remote gdb) instead of artificially relaxing reasonable constraints of the migration mechanism plus having to follow that format with the post-processing tool. Any interface that isn't get this information off my production server *now* so that I can get the server restarted, and send it to an expert to analyse -- is a poor interface, whether it was designed like that or not. Perhaps we don't have the right interface at all, but remote gdb is not it. What about the following idea? Introduce a new monitor command named dump, and this command accepts a filename. We can use almost all migration's code. We use this command to dump guest's memory, so there is no need to check whether the guest has a unmigratable device. I do not want to reject this proposal categorically, but I would like to see the gdb path fail /wrt essential requirements first. So far I don't see it would. ‘gdb path fail /wrt essential requirements’ what does it mean? That you explain why reading reading memory and processor states via the remote gdb interface and dumping it into a proper core file cannot be made working for you. First, I think crash can not analyze such core file. But it is not very important. What is remote gdb interface? man qemu - gdb. Do you mean that: the supporter uses gdb from another machine Or locally. There are various transports possible. to connect to customer's machine and get the data? If so, this way can not be used when the customer needs to dump the guest's memory automatically when watchdog timeouts. It is just another channel that can conceptually be used like the monitor, by a management app like libvirt, directly or indirectly via a scripted gdb frontend, or also by a human who wants to save some ongoing gdb session for later analysis. This dual use make such an approach the preferred one. Is the following is right? 1. execute the monitor command 'gdbserver' 2. run gdb and then 'target remote :1234' But, unfortunately, the monitor command gdbserver does not exit when we use json to connect to monitor. Thanks Wen Congyang Jan
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-10 09:47, Wen Congyang wrote: At 10/10/2011 03:22 PM, Jan Kiszka Write: On 2011-10-10 09:17, Wen Congyang wrote: At 10/10/2011 03:01 PM, Jan Kiszka Write: On 2011-10-10 08:59, Wen Congyang wrote: At 10/10/2011 02:52 PM, Jan Kiszka Write: On 2011-10-10 04:21, Wen Congyang wrote: At 10/09/2011 06:23 PM, Richard W.M. Jones Write: On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote: As explained in the other replies: It is way more future-proof to use an interface for this which was designed for it (remote gdb) instead of artificially relaxing reasonable constraints of the migration mechanism plus having to follow that format with the post-processing tool. Any interface that isn't get this information off my production server *now* so that I can get the server restarted, and send it to an expert to analyse -- is a poor interface, whether it was designed like that or not. Perhaps we don't have the right interface at all, but remote gdb is not it. What about the following idea? Introduce a new monitor command named dump, and this command accepts a filename. We can use almost all migration's code. We use this command to dump guest's memory, so there is no need to check whether the guest has a unmigratable device. I do not want to reject this proposal categorically, but I would like to see the gdb path fail /wrt essential requirements first. So far I don't see it would. ‘gdb path fail /wrt essential requirements’ what does it mean? That you explain why reading reading memory and processor states via the remote gdb interface and dumping it into a proper core file cannot be made working for you. First, I think crash can not analyze such core file. But it is not very important. What is remote gdb interface? man qemu - gdb. Do you mean that: the supporter uses gdb from another machine Or locally. There are various transports possible. to connect to customer's machine and get the data? If so, this way can not be used when the customer needs to dump the guest's memory automatically when watchdog timeouts. It is just another channel that can conceptually be used like the monitor, by a management app like libvirt, directly or indirectly via a scripted gdb frontend, or also by a human who wants to save some ongoing gdb session for later analysis. This dual use make such an approach the preferred one. Is the following is right? 1. execute the monitor command 'gdbserver' 2. run gdb and then 'target remote :1234' Yep. But, unfortunately, the monitor command gdbserver does not exit when we use json to connect to monitor. You mean via human-monitor-command (gdbserver is not a qmp command)? Would be a bug. Jan signature.asc Description: OpenPGP digital signature
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
At 10/10/2011 03:48 PM, Jan Kiszka Write: On 2011-10-10 09:47, Wen Congyang wrote: At 10/10/2011 03:22 PM, Jan Kiszka Write: On 2011-10-10 09:17, Wen Congyang wrote: At 10/10/2011 03:01 PM, Jan Kiszka Write: On 2011-10-10 08:59, Wen Congyang wrote: At 10/10/2011 02:52 PM, Jan Kiszka Write: On 2011-10-10 04:21, Wen Congyang wrote: At 10/09/2011 06:23 PM, Richard W.M. Jones Write: On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote: As explained in the other replies: It is way more future-proof to use an interface for this which was designed for it (remote gdb) instead of artificially relaxing reasonable constraints of the migration mechanism plus having to follow that format with the post-processing tool. Any interface that isn't get this information off my production server *now* so that I can get the server restarted, and send it to an expert to analyse -- is a poor interface, whether it was designed like that or not. Perhaps we don't have the right interface at all, but remote gdb is not it. What about the following idea? Introduce a new monitor command named dump, and this command accepts a filename. We can use almost all migration's code. We use this command to dump guest's memory, so there is no need to check whether the guest has a unmigratable device. I do not want to reject this proposal categorically, but I would like to see the gdb path fail /wrt essential requirements first. So far I don't see it would. ‘gdb path fail /wrt essential requirements’ what does it mean? That you explain why reading reading memory and processor states via the remote gdb interface and dumping it into a proper core file cannot be made working for you. First, I think crash can not analyze such core file. But it is not very important. What is remote gdb interface? man qemu - gdb. Do you mean that: the supporter uses gdb from another machine Or locally. There are various transports possible. to connect to customer's machine and get the data? If so, this way can not be used when the customer needs to dump the guest's memory automatically when watchdog timeouts. It is just another channel that can conceptually be used like the monitor, by a management app like libvirt, directly or indirectly via a scripted gdb frontend, or also by a human who wants to save some ongoing gdb session for later analysis. This dual use make such an approach the preferred one. Is the following is right? 1. execute the monitor command 'gdbserver' 2. run gdb and then 'target remote :1234' Yep. Can two guest use such port(1234) at the same time? If not, I think qemu should support to select a port automatically. Another problem is that: we should not require dev and debug tools(here is gdb) to be installed on the production host. But, unfortunately, the monitor command gdbserver does not exit when we use json to connect to monitor. You mean via human-monitor-command (gdbserver is not a qmp command)? Yes. Thanks Wen Congyang Would be a bug. Jan
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On Fri, Oct 07, 2011 at 02:56:56PM +0200, Jan Kiszka wrote: On 2011-10-07 14:25, Wen Congyang wrote: 于 2011/10/7 18:16, Jan Kiszka 写道: On 2011-10-07 11:46, Wen Congyang wrote: Currently, virsh dump uses monitor command migrate to dump guest's memory to file, and we can use crash to analyze the file. Unfortunately, virsh dump can not work if guest uses host pci device. The reason is that the device's status is also needed to migrate to remote machine, and the host pci device's status is not stored in qemu. So it is unmigratable. I think we can we can add a option to qmp command migrate(eg: skip) to allow the user to skip the check, and this option should be used only when dumping the guest's memory. Why not simply attach gdb? That works independently of migration. If qemu has some problem, we can use gdb to debug it. But if guest os has problem (eg:kernel panic and kdump does not work), we should dump guest's memory and use crash to analyze. qemu-system-xxx -s (or gdbserver via monitor if qemu is already running), gdb vmlinux, then target remote :1234. That is already possible, but that is not what we need for 'virsh dump'. The goal of that API is to provide a coredump of the guest, which can then be analysed off-node/site. While in theory you could attach GDB to the QEMU process and use GDB commands to then write out a coredump this isn't really satisfactory because too many large companies have security/audit compliance rules which forbid installation of developer tools (compilers, debuggers, etc) on production servers. So we cannot assume GDB is available. We need to be able to create a coredump natively from either QEMU or libvirt, with minimal of external tools. Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote: On 2011-10-10 04:21, Wen Congyang wrote: At 10/09/2011 06:23 PM, Richard W.M. Jones Write: On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote: As explained in the other replies: It is way more future-proof to use an interface for this which was designed for it (remote gdb) instead of artificially relaxing reasonable constraints of the migration mechanism plus having to follow that format with the post-processing tool. Any interface that isn't get this information off my production server *now* so that I can get the server restarted, and send it to an expert to analyse -- is a poor interface, whether it was designed like that or not. Perhaps we don't have the right interface at all, but remote gdb is not it. What about the following idea? Introduce a new monitor command named dump, and this command accepts a filename. We can use almost all migration's code. We use this command to dump guest's memory, so there is no need to check whether the guest has a unmigratable device. I do not want to reject this proposal categorically, but I would like to see the gdb path fail /wrt essential requirements first. So far I don't see it would. GDB is often forbidden on production servers, so that path is clearly not an option, unless we want libvirt to implement the GDB remote RPC protocol itself which just sounds like a world of hurt. Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On Mon, Oct 10, 2011 at 10:21:02AM +0800, Wen Congyang wrote: At 10/09/2011 06:23 PM, Richard W.M. Jones Write: On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote: As explained in the other replies: It is way more future-proof to use an interface for this which was designed for it (remote gdb) instead of artificially relaxing reasonable constraints of the migration mechanism plus having to follow that format with the post-processing tool. Any interface that isn't get this information off my production server *now* so that I can get the server restarted, and send it to an expert to analyse -- is a poor interface, whether it was designed like that or not. Perhaps we don't have the right interface at all, but remote gdb is not it. What about the following idea? Introduce a new monitor command named dump, and this command accepts a filename. We can use almost all migration's code. We use this command to dump guest's memory, so there is no need to check whether the guest has a unmigratable device. I think it would be a good idea of QEMU had a dedicated 'dump' command for this purpose, even if it was just an alias for 'migrate' initially. I have never really liked the fact that we abuse the 'migrate' command to generate a core dump. The resulting data file from this is more complex than it really needs to be, causing complexity for post-processing it. The needs of migration, are not entirely aligned with the needs of core dumping in the long term, so we should allow the possibility of their impls diverging without impacting apps using them. So adding a 'dump' command which wrote out data in a format that was optimized for offline processing by tools like 'crash' (or the windows equivalent) would be a good improvement, even if it just reuses the migrate code for now. Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On Mon, Oct 10, 2011 at 10:08:26AM +0100, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 10:21:02AM +0800, Wen Congyang wrote: At 10/09/2011 06:23 PM, Richard W.M. Jones Write: On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote: As explained in the other replies: It is way more future-proof to use an interface for this which was designed for it (remote gdb) instead of artificially relaxing reasonable constraints of the migration mechanism plus having to follow that format with the post-processing tool. Any interface that isn't get this information off my production server *now* so that I can get the server restarted, and send it to an expert to analyse -- is a poor interface, whether it was designed like that or not. Perhaps we don't have the right interface at all, but remote gdb is not it. What about the following idea? Introduce a new monitor command named dump, and this command accepts a filename. We can use almost all migration's code. We use this command to dump guest's memory, so there is no need to check whether the guest has a unmigratable device. I think it would be a good idea of QEMU had a dedicated 'dump' command for this purpose, even if it was just an alias for 'migrate' initially. I have never really liked the fact that we abuse the 'migrate' command to generate a core dump. The resulting data file from this is more complex than it really needs to be, causing complexity for post-processing it. The needs of migration, are not entirely aligned with the needs of core dumping in the long term, so we should allow the possibility of their impls diverging without impacting apps using them. So adding a 'dump' command which wrote out data in a format that was optimized for offline processing by tools like 'crash' (or the windows equivalent) would be a good improvement, even if it just reuses the migrate code for now. The other reason why it would be good, is that we would then have a clearly defined standard QEMU dump format, instead of libvirt dump format for QEMU Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-10 11:02, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote: On 2011-10-10 04:21, Wen Congyang wrote: At 10/09/2011 06:23 PM, Richard W.M. Jones Write: On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote: As explained in the other replies: It is way more future-proof to use an interface for this which was designed for it (remote gdb) instead of artificially relaxing reasonable constraints of the migration mechanism plus having to follow that format with the post-processing tool. Any interface that isn't get this information off my production server *now* so that I can get the server restarted, and send it to an expert to analyse -- is a poor interface, whether it was designed like that or not. Perhaps we don't have the right interface at all, but remote gdb is not it. What about the following idea? Introduce a new monitor command named dump, and this command accepts a filename. We can use almost all migration's code. We use this command to dump guest's memory, so there is no need to check whether the guest has a unmigratable device. I do not want to reject this proposal categorically, but I would like to see the gdb path fail /wrt essential requirements first. So far I don't see it would. GDB is often forbidden on production servers, so that path is clearly not an option, unless we want libvirt to implement the GDB remote RPC protocol itself which just sounds like a world of hurt. Run gdb with set debug remote 1 and watch the communication, it is not that complex. But a dump command is probably simpler for those scenarios, I agree. Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On Mon, Oct 10, 2011 at 10:10:21AM +0100, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 10:08:26AM +0100, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 10:21:02AM +0800, Wen Congyang wrote: At 10/09/2011 06:23 PM, Richard W.M. Jones Write: On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote: As explained in the other replies: It is way more future-proof to use an interface for this which was designed for it (remote gdb) instead of artificially relaxing reasonable constraints of the migration mechanism plus having to follow that format with the post-processing tool. Any interface that isn't get this information off my production server *now* so that I can get the server restarted, and send it to an expert to analyse -- is a poor interface, whether it was designed like that or not. Perhaps we don't have the right interface at all, but remote gdb is not it. What about the following idea? Introduce a new monitor command named dump, and this command accepts a filename. We can use almost all migration's code. We use this command to dump guest's memory, so there is no need to check whether the guest has a unmigratable device. I think it would be a good idea of QEMU had a dedicated 'dump' command for this purpose, even if it was just an alias for 'migrate' initially. I have never really liked the fact that we abuse the 'migrate' command to generate a core dump. The resulting data file from this is more complex than it really needs to be, causing complexity for post-processing it. The needs of migration, are not entirely aligned with the needs of core dumping in the long term, so we should allow the possibility of their impls diverging without impacting apps using them. So adding a 'dump' command which wrote out data in a format that was optimized for offline processing by tools like 'crash' (or the windows equivalent) would be a good improvement, even if it just reuses the migrate code for now. The other reason why it would be good, is that we would then have a clearly defined standard QEMU dump format, instead of libvirt dump format for QEMU Other reasons why this is a good idea: - so you don't need 3 processes doing a dance in order to capture a core dump - so that qemu can initiate a core dump itself (from watchdog events) - so that two qemu processes can do a core dump at the same time without trying to use conflicting ports Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones New in Fedora 11: Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 70 libraries supprt'd http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-10 11:10, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 10:08:26AM +0100, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 10:21:02AM +0800, Wen Congyang wrote: At 10/09/2011 06:23 PM, Richard W.M. Jones Write: On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote: As explained in the other replies: It is way more future-proof to use an interface for this which was designed for it (remote gdb) instead of artificially relaxing reasonable constraints of the migration mechanism plus having to follow that format with the post-processing tool. Any interface that isn't get this information off my production server *now* so that I can get the server restarted, and send it to an expert to analyse -- is a poor interface, whether it was designed like that or not. Perhaps we don't have the right interface at all, but remote gdb is not it. What about the following idea? Introduce a new monitor command named dump, and this command accepts a filename. We can use almost all migration's code. We use this command to dump guest's memory, so there is no need to check whether the guest has a unmigratable device. I think it would be a good idea of QEMU had a dedicated 'dump' command for this purpose, even if it was just an alias for 'migrate' initially. I have never really liked the fact that we abuse the 'migrate' command to generate a core dump. The resulting data file from this is more complex than it really needs to be, causing complexity for post-processing it. The needs of migration, are not entirely aligned with the needs of core dumping in the long term, so we should allow the possibility of their impls diverging without impacting apps using them. So adding a 'dump' command which wrote out data in a format that was optimized for offline processing by tools like 'crash' (or the windows equivalent) would be a good improvement, even if it just reuses the migrate code for now. The other reason why it would be good, is that we would then have a clearly defined standard QEMU dump format, instead of libvirt dump format for QEMU A core file would be that format - for direct gdb processing. No proprietary re-inventions please. Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-10 11:34, Richard W.M. Jones wrote: On Mon, Oct 10, 2011 at 10:10:21AM +0100, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 10:08:26AM +0100, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 10:21:02AM +0800, Wen Congyang wrote: At 10/09/2011 06:23 PM, Richard W.M. Jones Write: On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote: As explained in the other replies: It is way more future-proof to use an interface for this which was designed for it (remote gdb) instead of artificially relaxing reasonable constraints of the migration mechanism plus having to follow that format with the post-processing tool. Any interface that isn't get this information off my production server *now* so that I can get the server restarted, and send it to an expert to analyse -- is a poor interface, whether it was designed like that or not. Perhaps we don't have the right interface at all, but remote gdb is not it. What about the following idea? Introduce a new monitor command named dump, and this command accepts a filename. We can use almost all migration's code. We use this command to dump guest's memory, so there is no need to check whether the guest has a unmigratable device. I think it would be a good idea of QEMU had a dedicated 'dump' command for this purpose, even if it was just an alias for 'migrate' initially. I have never really liked the fact that we abuse the 'migrate' command to generate a core dump. The resulting data file from this is more complex than it really needs to be, causing complexity for post-processing it. The needs of migration, are not entirely aligned with the needs of core dumping in the long term, so we should allow the possibility of their impls diverging without impacting apps using them. So adding a 'dump' command which wrote out data in a format that was optimized for offline processing by tools like 'crash' (or the windows equivalent) would be a good improvement, even if it just reuses the migrate code for now. The other reason why it would be good, is that we would then have a clearly defined standard QEMU dump format, instead of libvirt dump format for QEMU Other reasons why this is a good idea: - so you don't need 3 processes doing a dance in order to capture a core dump If it made the overall implementation simpler, that would have been no valid reason. The key is only that banning of gdb on certain systems (though I wonder if you cannot configure it to only support remote targets, thus making it useless for potential attackers). - so that qemu can initiate a core dump itself (from watchdog events) This is a management tool decision. A dump command would not have any pre-condition or auto-trigger. - so that two qemu processes can do a core dump at the same time without trying to use conflicting ports That is a non-issue (just like you can have multiple monitors of multiple VMs listening on different TCP ports). Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On Mon, Oct 10, 2011 at 11:34:44AM +0200, Jan Kiszka wrote: On 2011-10-10 11:10, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 10:08:26AM +0100, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 10:21:02AM +0800, Wen Congyang wrote: At 10/09/2011 06:23 PM, Richard W.M. Jones Write: On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote: As explained in the other replies: It is way more future-proof to use an interface for this which was designed for it (remote gdb) instead of artificially relaxing reasonable constraints of the migration mechanism plus having to follow that format with the post-processing tool. Any interface that isn't get this information off my production server *now* so that I can get the server restarted, and send it to an expert to analyse -- is a poor interface, whether it was designed like that or not. Perhaps we don't have the right interface at all, but remote gdb is not it. What about the following idea? Introduce a new monitor command named dump, and this command accepts a filename. We can use almost all migration's code. We use this command to dump guest's memory, so there is no need to check whether the guest has a unmigratable device. I think it would be a good idea of QEMU had a dedicated 'dump' command for this purpose, even if it was just an alias for 'migrate' initially. I have never really liked the fact that we abuse the 'migrate' command to generate a core dump. The resulting data file from this is more complex than it really needs to be, causing complexity for post-processing it. The needs of migration, are not entirely aligned with the needs of core dumping in the long term, so we should allow the possibility of their impls diverging without impacting apps using them. So adding a 'dump' command which wrote out data in a format that was optimized for offline processing by tools like 'crash' (or the windows equivalent) would be a good improvement, even if it just reuses the migrate code for now. The other reason why it would be good, is that we would then have a clearly defined standard QEMU dump format, instead of libvirt dump format for QEMU A core file would be that format - for direct gdb processing. No proprietary re-inventions please. I have no personal attachment to any particular format, so if the standard core file format is possible, then we should definitely try to use it in QEMU. Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On Mon, Oct 10, 2011 at 11:34:44AM +0200, Jan Kiszka wrote: On 2011-10-10 11:10, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 10:08:26AM +0100, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 10:21:02AM +0800, Wen Congyang wrote: At 10/09/2011 06:23 PM, Richard W.M. Jones Write: On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote: As explained in the other replies: It is way more future-proof to use an interface for this which was designed for it (remote gdb) instead of artificially relaxing reasonable constraints of the migration mechanism plus having to follow that format with the post-processing tool. Any interface that isn't get this information off my production server *now* so that I can get the server restarted, and send it to an expert to analyse -- is a poor interface, whether it was designed like that or not. Perhaps we don't have the right interface at all, but remote gdb is not it. What about the following idea? Introduce a new monitor command named dump, and this command accepts a filename. We can use almost all migration's code. We use this command to dump guest's memory, so there is no need to check whether the guest has a unmigratable device. I think it would be a good idea of QEMU had a dedicated 'dump' command for this purpose, even if it was just an alias for 'migrate' initially. I have never really liked the fact that we abuse the 'migrate' command to generate a core dump. The resulting data file from this is more complex than it really needs to be, causing complexity for post-processing it. The needs of migration, are not entirely aligned with the needs of core dumping in the long term, so we should allow the possibility of their impls diverging without impacting apps using them. So adding a 'dump' command which wrote out data in a format that was optimized for offline processing by tools like 'crash' (or the windows equivalent) would be a good improvement, even if it just reuses the migrate code for now. The other reason why it would be good, is that we would then have a clearly defined standard QEMU dump format, instead of libvirt dump format for QEMU A core file would be that format - for direct gdb processing. No proprietary re-inventions please. Just a note: A core file to windows core dump file would be nice for windows guest crashes. Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 10/10/2011 12:21 PM, Alon Levy wrote: A core file would be that format - for direct gdb processing. No proprietary re-inventions please. Just a note: A core file to windows core dump file would be nice for windows guest crashes. That requires cooperation from a kernel driver in the Windows guest. The driver must call KeInitializeCrashDumpHeader and write somewhere the physical address of the buffer. Paolo
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On Mon, Oct 10, 2011 at 12:48:52PM +0200, Paolo Bonzini wrote: On 10/10/2011 12:21 PM, Alon Levy wrote: A core file would be that format - for direct gdb processing. No proprietary re-inventions please. Just a note: A core file to windows core dump file would be nice for windows guest crashes. That requires cooperation from a kernel driver in the Windows guest. The driver must call KeInitializeCrashDumpHeader and write somewhere the physical address of the buffer. That won't be suitable for 'virsh dump' then, because we need this to work when the guest OS is in a crashed/non-responsive state, and so we cna't rely on talking to it. Daniel. -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 10/10/2011 01:04 PM, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 12:48:52PM +0200, Paolo Bonzini wrote: On 10/10/2011 12:21 PM, Alon Levy wrote: A core file would be that format - for direct gdb processing. No proprietary re-inventions please. Just a note: A core file to windows core dump file would be nice for windows guest crashes. That requires cooperation from a kernel driver in the Windows guest. The driver must call KeInitializeCrashDumpHeader and write somewhere the physical address of the buffer. That won't be suitable for 'virsh dump' then, because we need this to work when the guest OS is in a crashed/non-responsive state, and so we cna't rely on talking to it. Note that the guest can generate the buffer before it crashes. Paolo
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
At 10/10/2011 06:19 PM, Daniel P. Berrange Write: On Mon, Oct 10, 2011 at 11:34:44AM +0200, Jan Kiszka wrote: On 2011-10-10 11:10, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 10:08:26AM +0100, Daniel P. Berrange wrote: On Mon, Oct 10, 2011 at 10:21:02AM +0800, Wen Congyang wrote: At 10/09/2011 06:23 PM, Richard W.M. Jones Write: On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote: As explained in the other replies: It is way more future-proof to use an interface for this which was designed for it (remote gdb) instead of artificially relaxing reasonable constraints of the migration mechanism plus having to follow that format with the post-processing tool. Any interface that isn't get this information off my production server *now* so that I can get the server restarted, and send it to an expert to analyse -- is a poor interface, whether it was designed like that or not. Perhaps we don't have the right interface at all, but remote gdb is not it. What about the following idea? Introduce a new monitor command named dump, and this command accepts a filename. We can use almost all migration's code. We use this command to dump guest's memory, so there is no need to check whether the guest has a unmigratable device. I think it would be a good idea of QEMU had a dedicated 'dump' command for this purpose, even if it was just an alias for 'migrate' initially. I have never really liked the fact that we abuse the 'migrate' command to generate a core dump. The resulting data file from this is more complex than it really needs to be, causing complexity for post-processing it. The needs of migration, are not entirely aligned with the needs of core dumping in the long term, so we should allow the possibility of their impls diverging without impacting apps using them. So adding a 'dump' command which wrote out data in a format that was optimized for offline processing by tools like 'crash' (or the windows equivalent) would be a good improvement, even if it just reuses the migrate code for now. The other reason why it would be good, is that we would then have a clearly defined standard QEMU dump format, instead of libvirt dump format for QEMU A core file would be that format - for direct gdb processing. No proprietary re-inventions please. I have no personal attachment to any particular format, so if the standard core file format is possible, then we should definitely try to use it in QEMU. I do not know whether there is such standard format. But IIRC, the format of kdump, netdump, and diskdump is very similar. If we want to use such format in qemu, I think we can not do live dump in libvirt. Thanks Wen Congyang Daniel
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
At 10/10/2011 05:34 PM, Jan Kiszka Write:
On 2011-10-10 11:02, Daniel P. Berrange wrote:
Run gdb with set debug remote 1 and watch the communication, it is not
that complex. But a dump command is probably simpler for those
scenarios, I agree.
We have determined to introduce a new command dump for such scenarios.
Now, let's to discuss the arguments for this command:
Here is the arguments for command migrate:
{
.name = migrate,
.args_type = detach:-d,blk:-b,inc:-i,uri:s,
.params = [-d] [-b] [-i] uri,
...
},
I think the arguments -b and -i are not required for dump, and -d is useful
for dump.
We can pass the file's path or fd to qemu instead of uri.
So here is my image for command dump(reuse migration's code, not generate a
standard
format core):
{
.name = dump,
.args_type = detach:-d,file:s,
.params = [-d] file,
...
},
The argument can be 'file:file's path' or 'fd:fd's name'
Any advice is welcome.
Thanks
Wen Congyang
Jan
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-08 17:16, Richard W.M. Jones wrote: On Fri, Oct 07, 2011 at 12:16:07PM +0200, Jan Kiszka wrote: On 2011-10-07 11:46, Wen Congyang wrote: Currently, virsh dump uses monitor command migrate to dump guest's memory to file, and we can use crash to analyze the file. Unfortunately, virsh dump can not work if guest uses host pci device. The reason is that the device's status is also needed to migrate to remote machine, and the host pci device's status is not stored in qemu. So it is unmigratable. I think we can we can add a option to qmp command migrate(eg: skip) to allow the user to skip the check, and this option should be used only when dumping the guest's memory. Why not simply attach gdb? That works independently of migration. It's a lot more hairy and requires dev tools to be installed on the production host. The core file is technically simple, has no dependencies, can be moved to another machine for analysis, and contains all the data (not just a backtrace). As explained in the other replies: It is way more future-proof to use an interface for this which was designed for it (remote gdb) instead of artificially relaxing reasonable constraints of the migration mechanism plus having to follow that format with the post-processing tool. Jan signature.asc Description: OpenPGP digital signature
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote: As explained in the other replies: It is way more future-proof to use an interface for this which was designed for it (remote gdb) instead of artificially relaxing reasonable constraints of the migration mechanism plus having to follow that format with the post-processing tool. Any interface that isn't get this information off my production server *now* so that I can get the server restarted, and send it to an expert to analyse -- is a poor interface, whether it was designed like that or not. Perhaps we don't have the right interface at all, but remote gdb is not it. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-09 12:23, Richard W.M. Jones wrote: On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote: As explained in the other replies: It is way more future-proof to use an interface for this which was designed for it (remote gdb) instead of artificially relaxing reasonable constraints of the migration mechanism plus having to follow that format with the post-processing tool. Any interface that isn't get this information off my production server *now* so that I can get the server restarted, and send it to an expert to analyse -- is a poor interface, whether it was designed like that or not. Perhaps we don't have the right interface at all, but remote gdb is not it. Why is it not (for implementing core file write-out - I'm not proposing to drop that part of libvirt)? Jan signature.asc Description: OpenPGP digital signature
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
At 10/09/2011 06:23 PM, Richard W.M. Jones Write: On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote: As explained in the other replies: It is way more future-proof to use an interface for this which was designed for it (remote gdb) instead of artificially relaxing reasonable constraints of the migration mechanism plus having to follow that format with the post-processing tool. Any interface that isn't get this information off my production server *now* so that I can get the server restarted, and send it to an expert to analyse -- is a poor interface, whether it was designed like that or not. Perhaps we don't have the right interface at all, but remote gdb is not it. What about the following idea? Introduce a new monitor command named dump, and this command accepts a filename. We can use almost all migration's code. We use this command to dump guest's memory, so there is no need to check whether the guest has a unmigratable device. Thanks Wen Congyang Rich.
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On Fri, Oct 07, 2011 at 12:16:07PM +0200, Jan Kiszka wrote: On 2011-10-07 11:46, Wen Congyang wrote: Currently, virsh dump uses monitor command migrate to dump guest's memory to file, and we can use crash to analyze the file. Unfortunately, virsh dump can not work if guest uses host pci device. The reason is that the device's status is also needed to migrate to remote machine, and the host pci device's status is not stored in qemu. So it is unmigratable. I think we can we can add a option to qmp command migrate(eg: skip) to allow the user to skip the check, and this option should be used only when dumping the guest's memory. Why not simply attach gdb? That works independently of migration. It's a lot more hairy and requires dev tools to be installed on the production host. The core file is technically simple, has no dependencies, can be moved to another machine for analysis, and contains all the data (not just a backtrace). Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://et.redhat.com/~rjones/virt-df/
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-07 11:46, Wen Congyang wrote: Currently, virsh dump uses monitor command migrate to dump guest's memory to file, and we can use crash to analyze the file. Unfortunately, virsh dump can not work if guest uses host pci device. The reason is that the device's status is also needed to migrate to remote machine, and the host pci device's status is not stored in qemu. So it is unmigratable. I think we can we can add a option to qmp command migrate(eg: skip) to allow the user to skip the check, and this option should be used only when dumping the guest's memory. Why not simply attach gdb? That works independently of migration. Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
于 2011/10/7 18:16, Jan Kiszka 写道: On 2011-10-07 11:46, Wen Congyang wrote: Currently, virsh dump uses monitor command migrate to dump guest's memory to file, and we can use crash to analyze the file. Unfortunately, virsh dump can not work if guest uses host pci device. The reason is that the device's status is also needed to migrate to remote machine, and the host pci device's status is not stored in qemu. So it is unmigratable. I think we can we can add a option to qmp command migrate(eg: skip) to allow the user to skip the check, and this option should be used only when dumping the guest's memory. Why not simply attach gdb? That works independently of migration. If qemu has some problem, we can use gdb to debug it. But if guest os has problem (eg:kernel panic and kdump does not work), we should dump guest's memory and use crash to analyze. Thanks Wen Congyang Jan
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-07 14:25, Wen Congyang wrote: 于 2011/10/7 18:16, Jan Kiszka 写道: On 2011-10-07 11:46, Wen Congyang wrote: Currently, virsh dump uses monitor command migrate to dump guest's memory to file, and we can use crash to analyze the file. Unfortunately, virsh dump can not work if guest uses host pci device. The reason is that the device's status is also needed to migrate to remote machine, and the host pci device's status is not stored in qemu. So it is unmigratable. I think we can we can add a option to qmp command migrate(eg: skip) to allow the user to skip the check, and this option should be used only when dumping the guest's memory. Why not simply attach gdb? That works independently of migration. If qemu has some problem, we can use gdb to debug it. But if guest os has problem (eg:kernel panic and kdump does not work), we should dump guest's memory and use crash to analyze. qemu-system-xxx -s (or gdbserver via monitor if qemu is already running), gdb vmlinux, then target remote :1234. Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
于 2011/10/7 20:56, Jan Kiszka 写道: On 2011-10-07 14:25, Wen Congyang wrote: 于 2011/10/7 18:16, Jan Kiszka 写道: On 2011-10-07 11:46, Wen Congyang wrote: Currently, virsh dump uses monitor command migrate to dump guest's memory to file, and we can use crash to analyze the file. Unfortunately, virsh dump can not work if guest uses host pci device. The reason is that the device's status is also needed to migrate to remote machine, and the host pci device's status is not stored in qemu. So it is unmigratable. I think we can we can add a option to qmp command migrate(eg: skip) to allow the user to skip the check, and this option should be used only when dumping the guest's memory. Why not simply attach gdb? That works independently of migration. If qemu has some problem, we can use gdb to debug it. But if guest os has problem (eg:kernel panic and kdump does not work), we should dump guest's memory and use crash to analyze. qemu-system-xxx -s (or gdbserver via monitor if qemu is already running), gdb vmlinux, then target remote :1234. Hmm, if i use qemu, i can do it as the above. But i can not hope our customer do it because it is difficult for them to debug kernel. So the customer can use 'virsh dump'(the guest is managed by libvirt) or autodump(if the guest has a watchdog) to dump the memory. The supporter can debug kernel in another machine. I still think that supporting to dump memory when guest uses host pci device is necessay. Thanks Wen Congyang Jan
Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest
On 2011-10-07 16:05, Wen Congyang wrote: 于 2011/10/7 20:56, Jan Kiszka 写道: On 2011-10-07 14:25, Wen Congyang wrote: 于 2011/10/7 18:16, Jan Kiszka 写道: On 2011-10-07 11:46, Wen Congyang wrote: Currently, virsh dump uses monitor command migrate to dump guest's memory to file, and we can use crash to analyze the file. Unfortunately, virsh dump can not work if guest uses host pci device. The reason is that the device's status is also needed to migrate to remote machine, and the host pci device's status is not stored in qemu. So it is unmigratable. I think we can we can add a option to qmp command migrate(eg: skip) to allow the user to skip the check, and this option should be used only when dumping the guest's memory. Why not simply attach gdb? That works independently of migration. If qemu has some problem, we can use gdb to debug it. But if guest os has problem (eg:kernel panic and kdump does not work), we should dump guest's memory and use crash to analyze. qemu-system-xxx -s (or gdbserver via monitor if qemu is already running), gdb vmlinux, then target remote :1234. Hmm, if i use qemu, i can do it as the above. But i can not hope our customer do it because it is difficult for them to debug kernel. So the customer can use 'virsh dump'(the guest is managed by libvirt) or autodump(if the guest has a watchdog) to dump the memory. The supporter can debug kernel in another machine. gdb is surely scriptable. It's just a bit more complex than running generate-core-file - gdb does not yet support this for remote sessions. Ideally, that feature should be added, and you are done, independent of QEMU migration format X.Y.whatever or limitations due to unmigratable devices. The current approach is, well, pragmatic. Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux
