Re: [Qemu-devel] [PATCH v4 0/9] Make 'dump-guest-memory' dump in kdump-compressed format

2013-07-04 Thread Stefan Hajnoczi
On Wed, Jul 03, 2013 at 03:39:51PM +0800, Qiao Nuohan wrote:
 On 07/01/2013 07:45 PM, Stefan Hajnoczi wrote:
 In flatten format, data will be write to dumpfile block by block, and uses the
 following structure to indicate the offset and size of a data block.
 
 struct makedumpfile_data_header {
 int64_t offset;
 int64_t buf_size;
 };
 
 For more information, please refer to makedumpfile
 
 
 http://sourceforge.net/projects/makedumpfile/

I see.  From the QEMU code perspective this will be simpler.

Stefan



Re: [Qemu-devel] [PATCH v4 0/9] Make 'dump-guest-memory' dump in kdump-compressed format

2013-07-03 Thread Qiao Nuohan

On 07/01/2013 07:45 PM, Stefan Hajnoczi wrote:

I'm now convinced that kdump is worthwhile, thanks for providing data.

It would be nice to see the flattened kdump approach.


Hi Stefan and Luiz,

Thanks for Stefan's review again!

I am starting to implement the function using flatten format. And the idea is:

the process will use seek to find the place to write if it possible,
otherwise flatten format will be used to avoid seek operation.

I send the mail just to make sure if the idea will be accepted by qemu people
and I am spending time on useful thing. Additionally, since the introspection
is not ready yet, the 5th version of my patches will be sent later just to
see if the process of creating kdump-compressed format will be convinced by
qemu people.

P.S.

In flatten format, data will be write to dumpfile block by block, and uses the
following structure to indicate the offset and size of a data block.

struct makedumpfile_data_header {
int64_t offset;
int64_t buf_size;
};

For more information, please refer to makedumpfile


http://sourceforge.net/projects/makedumpfile/

--
Regards
Qiao Nuohan



Re: [Qemu-devel] [PATCH v4 0/9] Make 'dump-guest-memory' dump in kdump-compressed format

2013-07-01 Thread Stefan Hajnoczi
On Fri, Jun 28, 2013 at 10:57:28AM +0800, Qiao Nuohan wrote:
 On 06/27/2013 04:54 PM, Stefan Hajnoczi wrote:
 
 The interesting question is how effective this approach is.  If it's
 good enough then it would be a fairly simple modification to dump.c.
 
 I see, if excluding zero page in ELF can make a lot of size reduce, it's 
 better
 to choose this method. But think over the situation that kernel is on for a
 long time, then few zero pages will be in memory, compression will do more 
 work
 to reduce size not excluding zero pages. So the approach is not always
 effective.
 
 A test on a 1GB memory, and the machine is just on:
 
 size   format method for reducing memory
 
 1.1GB  ELFno
 1.1GB  kdump  no
 227MB  kdump  with all zero pages excluded
  96MB  kdump  compressed with zero pages remained
  88MB  kdump  compressed with zero pages excluded
 
 excluding zero pages does some work, but compression seems to be more 
 effective.
[...]
 If it makes the code simpler and smaller it would be nice.
 
 That's the point. I will make the code simpler.

I'm now convinced that kdump is worthwhile, thanks for providing data.

It would be nice to see the flattened kdump approach.

Stefan



Re: [Qemu-devel] [PATCH v4 0/9] Make 'dump-guest-memory' dump in kdump-compressed format

2013-06-27 Thread Qiao Nuohan

Sorry for replying late.

On 06/20/2013 04:57 PM, Stefan Hajnoczi wrote:


Please link to the code that writes DISKDUMP kdump files on a physical
machine.  I only see the crash utility code to read the DISKDUMP code
but I haven't found anything in the Linux kernel, the crash utility, or
the kexec-utils code to actually write a DISKDUMP file.


I think you can refer to the following URL, and the kdump-compressed format is
described in the file called IMPLEMENTATION.

http://sourceforge.net/projects/makedumpfile/.



I understand why you need temporary files, but my questions stand:

Have you looked at using ELF more efficiently instead of duplicating
kdump code into QEMU?  kdump is not a great format for the problem
you're trying to solve - you're not filling in the Linux-specific
metadata and it's a pain to write due to its layout.

Why can't you simply omit zero pages from the ELF?

Why can't you compress the entire ELF file and add straightforward
decompression to the crash utility?



As I have said, the main purpose of this work is *reducing* the *size* of dump
file to make delivering dump file more conveniently.

Compared with migration, memory only dump has a feature regression without
compression and excluding zero pages. So the regression makes me feel it is
necessary to make these patches.


You asked about using ELF more efficiently. For implementing *excluding zero*
pages, *PT_LOAD* can be made use of. p_memsz and p_filesz fields of PT_LOAD
entry are used to describe memory size and the size of corresponding data in
dump file respectively. Blocks only get zero pages in it will have *p_filesz*
set to 0.

However, such implementation faces a problem: the number of PT_LOAD may
*exceed* the range. As zero pages occur *arbitrarily*, the number of PT_LOAD,
at the worst case, may reach the number of the total physical page frames. For
example, 256MB have 2^16 page frames may exceed the range of e_phnum. Although
sh_info is extended to store PT_LOAD when e_phnum is not enough, sh_info that
has 2^32 range may also exceed if the guest machine has more-than-16TB physical
memory (it won't occur soon, but it will happen one day).

OTOH, the reason why I chose kdump-compressed format is ELF doesn't support
compression and filtering yet. To implement compression and filtering on ELF,
we need to consider specific ABI among qemu, crash utility and makedumpfile.
After that, another work needs to be done to port them in specific
distributions.

Compared kdump-compressed format with ELF, compression and filtering is
supported and we don't need to modify tools like crash and makedumpfile.
Considering these reasons, kdump-compressed format is better than ELF. It get
more merit.


According to your comments, I think your objection first comes from the
*temporary files*. What if temporary files won't be used? Flatten kdump-
compressed format is supported by crash and makedumpfile which offers a
mechanism to avoid seek in the case of sending data through *pipe*, then I
don't need to cache pages' data in temporary files.

And about the metadata(do you mean VMCOREINFO here?) you pointed out, it
contains debugging information related to kernel memory. The debugging
information is useful if present, because we can use it to filter more kinds of
memory. But now we only need to exclude zero pages and there's formal mechanism
between qemu and linux kernel, so without metadata can also satisfy our need.

Think over all of the above, I still chose kdump-compressed format. What's your 
opinion about this?


--
Regards
Qiao Nuohan



Re: [Qemu-devel] [PATCH v4 0/9] Make 'dump-guest-memory' dump in kdump-compressed format

2013-06-27 Thread Stefan Hajnoczi
On Thu, Jun 27, 2013 at 03:11:09PM +0800, Qiao Nuohan wrote:
 You asked about using ELF more efficiently. For implementing *excluding zero*
 pages, *PT_LOAD* can be made use of. p_memsz and p_filesz fields of PT_LOAD
 entry are used to describe memory size and the size of corresponding data in
 dump file respectively. Blocks only get zero pages in it will have *p_filesz*
 set to 0.
 
 However, such implementation faces a problem: the number of PT_LOAD may
 *exceed* the range. As zero pages occur *arbitrarily*, the number of PT_LOAD,
 at the worst case, may reach the number of the total physical page frames. For
 example, 256MB have 2^16 page frames may exceed the range of e_phnum. Although
 sh_info is extended to store PT_LOAD when e_phnum is not enough, sh_info that
 has 2^32 range may also exceed if the guest machine has more-than-16TB 
 physical
 memory (it won't occur soon, but it will happen one day).

A simple patch to do this would scan memory for zero regions
write_elf_loads().  It could avoid a huge number of PT_LOAD entries by
setting a minimum threshold like 64 KB or 128 KB.  This would eliminate
the big zero regions from the ELF file.

Of course it doesn't compress the memory contents so it's still going to
be bigger than a compressed kdump FILEDUMP.

The interesting question is how effective this approach is.  If it's
good enough then it would be a fairly simple modification to dump.c.

 OTOH, the reason why I chose kdump-compressed format is ELF doesn't support
 compression and filtering yet. To implement compression and filtering on ELF,
 we need to consider specific ABI among qemu, crash utility and makedumpfile.
 After that, another work needs to be done to port them in specific
 distributions.
 
 Compared kdump-compressed format with ELF, compression and filtering is
 supported and we don't need to modify tools like crash and makedumpfile.
 Considering these reasons, kdump-compressed format is better than ELF. It get
 more merit.

I meant simply compressing the ELF output during creation.  But I guess
the tools need random access to the ELF file, which is inefficient with
zlib and friends.

 According to your comments, I think your objection first comes from the
 *temporary files*.

I'm not happy with duplicating the kdump FILEDUMP format code into QEMU
because it is relatively big and ugly (temporary files make up a large
part of that).

That said, compressed kdump FILEDUMP might really be the only option
that meets your needs.  I'm just surprised at all the extra work
necessary to make the dump compact.

 What if temporary files won't be used? Flatten kdump-
 compressed format is supported by crash and makedumpfile which offers a
 mechanism to avoid seek in the case of sending data through *pipe*, then I
 don't need to cache pages' data in temporary files.

If it makes the code simpler and smaller it would be nice.

Stefan



Re: [Qemu-devel] [PATCH v4 0/9] Make 'dump-guest-memory' dump in kdump-compressed format

2013-06-27 Thread Qiao Nuohan

On 06/27/2013 04:54 PM, Stefan Hajnoczi wrote:


The interesting question is how effective this approach is.  If it's
good enough then it would be a fairly simple modification to dump.c.


I see, if excluding zero page in ELF can make a lot of size reduce, it's better
to choose this method. But think over the situation that kernel is on for a
long time, then few zero pages will be in memory, compression will do more work
to reduce size not excluding zero pages. So the approach is not always
effective.

A test on a 1GB memory, and the machine is just on:

size   format method for reducing memory

1.1GB  ELFno
1.1GB  kdump  no
227MB  kdump  with all zero pages excluded
 96MB  kdump  compressed with zero pages remained
 88MB  kdump  compressed with zero pages excluded

excluding zero pages does some work, but compression seems to be more effective.



I meant simply compressing the ELF output during creation.  But I guess
the tools need random access to the ELF file, which is inefficient with
zlib and friends.


It is the reason of why I don't like entirely compression. It will make crash 
work in terrific bad performance.




I'm not happy with duplicating the kdump FILEDUMP format code into QEMU
because it is relatively big and ugly (temporary files make up a large
part of that).




If it makes the code simpler and smaller it would be nice.


That's the point. I will make the code simpler.

Thanks for your comments!

--
Regards
Qiao Nuohan



Re: [Qemu-devel] [PATCH v4 0/9] Make 'dump-guest-memory' dump in kdump-compressed format

2013-06-20 Thread Stefan Hajnoczi
On Thu, Jun 20, 2013 at 10:18:35AM +0800, Qiao Nuohan wrote:
 On 06/19/2013 09:49 PM, Stefan Hajnoczi wrote:
 Where does that code live that writes DISKDUMP files?  I can see the
 diskdump.[ch] code.
 
 Sorry, I cannot catch what do you mean here.

Please link to the code that writes DISKDUMP kdump files on a physical
machine.  I only see the crash utility code to read the DISKDUMP code
but I haven't found anything in the Linux kernel, the crash utility, or
the kexec-utils code to actually write a DISKDUMP file.

 
 The file format is pretty bad: we need 4 temporary files and a lot of
 data copying to write it out.
 
 Why not just compress an ELF file and teach the crash utility how to
 decompress while reading the ELF?
 
 Also, did you look into simply outputting the ELF file without zero
 pages?
 
 What I want is a dump file with smaller size. And compressed format and with
 zero pages excluded can make it. I choose kdump-compressed format because it 
 is
 a standard format and it can realize what I want.
 
 Why 4 temporary files are need? dump-guest-memory may be called with a fd 
 which
 is supposed to send data of dump to. If fd is opened on a pipe or etc which is
 unable to seek, then I need to cache the data.

I understand why you need temporary files, but my questions stand:

Have you looked at using ELF more efficiently instead of duplicating
kdump code into QEMU?  kdump is not a great format for the problem
you're trying to solve - you're not filling in the Linux-specific
metadata and it's a pain to write due to its layout.

Why can't you simply omit zero pages from the ELF?

Why can't you compress the entire ELF file and add straightforward
decompression to the crash utility?

Stefan



Re: [Qemu-devel] [PATCH v4 0/9] Make 'dump-guest-memory' dump in kdump-compressed format

2013-06-19 Thread Qiao Nuohan

On 06/14/2013 02:12 AM, Luiz Capitulino wrote:

I've started reviewing this, but I'm out of cycles for this week.


Hi Luiz,

No comments yet?

--
Regards
Qiao Nuohan



Re: [Qemu-devel] [PATCH v4 0/9] Make 'dump-guest-memory' dump in kdump-compressed format

2013-06-19 Thread Stefan Hajnoczi
On Tue, May 28, 2013 at 10:50:28AM +0800, qiaonuo...@cn.fujitsu.com wrote:
 The kdump-compressed format is *linux specific* *linux standard* crash dump
 format used in kdump framework. The kdump-compressed format is readable only
 with the crash utility, and it can be smaller than the ELF format because of
 the compression support.
 
 Note, similar to 'dump-guest-memory':
 1. The guest should be x86 or x86_64. The other arch is not supported now.
 2. If the OS is in the second kernel, gdb may not work well, and crash can
work by specifying '--machdep phys_addr=xxx' in the command line. The
reason is that the second kernel will update the page table, and we can
not get the page table for the first kernel.
 3. The cpu's state is stored in QEMU note.
 4. The vmcore are able to be compressed with zlib, lzo or snappy. zlib is
available by default, and option '--enable-lzo' or '--enable-snappy'
should be used with configure to make lzo or snappy available.

Please see the coding style docs and run scripts/checkpatch.pl on your
patches:
http://git.qemu.org/?p=qemu.git;a=blob;f=CODING_STYLE;hb=HEAD
http://git.qemu.org/?p=qemu.git;a=blob;f=HACKING;hb=HEAD

Where does that code live that writes DISKDUMP files?  I can see the
diskdump.[ch] code.

The file format is pretty bad: we need 4 temporary files and a lot of
data copying to write it out.

Why not just compress an ELF file and teach the crash utility how to
decompress while reading the ELF?

Also, did you look into simply outputting the ELF file without zero
pages?

Stefan



Re: [Qemu-devel] [PATCH v4 0/9] Make 'dump-guest-memory' dump in kdump-compressed format

2013-06-19 Thread Qiao Nuohan

On 06/19/2013 09:49 PM, Stefan Hajnoczi wrote:

Where does that code live that writes DISKDUMP files?  I can see the
diskdump.[ch] code.


Sorry, I cannot catch what do you mean here.



The file format is pretty bad: we need 4 temporary files and a lot of
data copying to write it out.

Why not just compress an ELF file and teach the crash utility how to
decompress while reading the ELF?

Also, did you look into simply outputting the ELF file without zero
pages?


What I want is a dump file with smaller size. And compressed format and with
zero pages excluded can make it. I choose kdump-compressed format because it is
a standard format and it can realize what I want.

Why 4 temporary files are need? dump-guest-memory may be called with a fd which
is supposed to send data of dump to. If fd is opened on a pipe or etc which is
unable to seek, then I need to cache the data.

--
Regards
Qiao Nuohan



Re: [Qemu-devel] [PATCH v4 0/9] Make 'dump-guest-memory' dump in kdump-compressed format

2013-06-13 Thread Luiz Capitulino
On Tue, 11 Jun 2013 09:48:40 +0800
Qiao Nuohan qiaonuo...@cn.fujitsu.com wrote:

 introspection. I just want these patches reviewed first, then I may get these
 patches ready for merging soon after introspection support.

I've started reviewing this, but I'm out of cycles for this week.

Can anyone CC'ed help with this?



Re: [Qemu-devel] [PATCH v4 0/9] Make 'dump-guest-memory' dump in kdump-compressed format

2013-06-10 Thread Luiz Capitulino
On Mon, 10 Jun 2013 10:15:35 +0800
Qiao Nuohan qiaonuo...@cn.fujitsu.com wrote:

 On 06/05/2013 10:15 AM, Luiz Capitulino wrote:
  I can review it until the end of this week. If this series is adding a new
  argument (which I believe is what it does) then there's only two ways
  to get this merged: either we wait for full introspection or you add this
  feature as a new command.
 
  I'd prefer to wait for full introspection, but it depends how long it's
  going to take to get it merged and how much time you're willing to wait.
 
  Amos, can you give us an update on that work?
 
 Hi Luiz,
 
 I would like to get these patches reviewed first. If introspection won't take
 too much time, I will choose to wait. However, I cannot figure out how long it
 will take, why not get the parts not related to introspection settled first?

What do you mean by settled? We can keep the review cycle going, but to
merge this we have two options: we wait for the full introspection or we
make this a different command.

 Seems Amos's draft will not change qapi-schema.json.

The point of Amos series is discovery, not conflicts. If we merge your
series w/o introspection support, then it's impossible for a mngt app like
libvirt to know whether or not a given QEMU version supports your new
argument.



Re: [Qemu-devel] [PATCH v4 0/9] Make 'dump-guest-memory' dump in kdump-compressed format

2013-06-10 Thread Qiao Nuohan

On 06/10/2013 08:54 PM, Luiz Capitulino wrote:

On Mon, 10 Jun 2013 10:15:35 +0800
Qiao Nuohanqiaonuo...@cn.fujitsu.com  wrote:


On 06/05/2013 10:15 AM, Luiz Capitulino wrote:

I can review it until the end of this week. If this series is adding a new
argument (which I believe is what it does) then there's only two ways
to get this merged: either we wait for full introspection or you add this
feature as a new command.

I'd prefer to wait for full introspection, but it depends how long it's
going to take to get it merged and how much time you're willing to wait.

Amos, can you give us an update on that work?


Hi Luiz,

I would like to get these patches reviewed first. If introspection won't take
too much time, I will choose to wait. However, I cannot figure out how long it
will take, why not get the parts not related to introspection settled first?


What do you mean by settled? We can keep the review cycle going, but to
merge this we have two options: we wait for the full introspection or we
make this a different command.


Sorry for confusing you. I mean reviewed enough here.

If it won't take several months, I have no doubt about waiting for
introspection. I just want these patches reviewed first, then I may get these
patches ready for merging soon after introspection support.




Seems Amos's draft will not change qapi-schema.json.


The point of Amos series is discovery, not conflicts. If we merge your
series w/o introspection support, then it's impossible for a mngt app like
libvirt to know whether or not a given QEMU version supports your new
argument.



Thanks for your explanation.






--
Regards
Qiao Nuohan



Re: [Qemu-devel] [PATCH v4 0/9] Make 'dump-guest-memory' dump in kdump-compressed format

2013-06-09 Thread Qiao Nuohan

On 06/05/2013 10:15 AM, Luiz Capitulino wrote:

I can review it until the end of this week. If this series is adding a new
argument (which I believe is what it does) then there's only two ways
to get this merged: either we wait for full introspection or you add this
feature as a new command.

I'd prefer to wait for full introspection, but it depends how long it's
going to take to get it merged and how much time you're willing to wait.

Amos, can you give us an update on that work?


Hi Luiz,

I would like to get these patches reviewed first. If introspection won't take
too much time, I will choose to wait. However, I cannot figure out how long it
will take, why not get the parts not related to introspection settled first?

Seems Amos's draft will not change qapi-schema.json.

--
Regards
Qiao Nuohan



Re: [Qemu-devel] [PATCH v4 0/9] Make 'dump-guest-memory' dump in kdump-compressed format

2013-06-05 Thread Amos Kong
On Tue, Jun 04, 2013 at 10:15:41PM -0400, Luiz Capitulino wrote:
 
 [CC'ing Amos this time]
 
 On Wed, 05 Jun 2013 09:29:19 +0800
 Qiao Nuohan qiaonuo...@cn.fujitsu.com wrote:
 
I haven't reviewed it yet, but we need introspection support before 
  merging
this.
  
  Hello Luiz,
  
  Is it possible to get this reviewed, or I am supposed to wait until
  introspection support being settled?
 
 I can review it until the end of this week. If this series is adding a new
 argument (which I believe is what it does) then there's only two ways
 to get this merged: either we wait for full introspection or you add this
 feature as a new command.
 
 I'd prefer to wait for full introspection, but it depends how long it's
 going to take to get it merged and how much time you're willing to wait.
 
 Amos, can you give us an update on that work?


Summary of upstream discussion: 

 1) move events to schema.json, then it can also be introspected
(future work) (Luiz)
 2) need to support to return everyting in one shot (Eric)
 3) support filter by cmd/tyep/event name
 4) processe qapi-schema.json for a more explicit wire format
with metadata (Eric)
 5) dynamic schema (only for compiled/loaded modules) (Kevin)
 6) the real motivation behind full introspection is to allow
commands/enum/etc to be extended

I'm tring to implement my original throught, it's a little bit slow
because of the effect of other tasks. I will send the draft patch 
next week.

Amos.



Re: [Qemu-devel] [PATCH v4 0/9] Make 'dump-guest-memory' dump in kdump-compressed format

2013-06-04 Thread Qiao Nuohan

 I haven't reviewed it yet, but we need introspection support before merging
 this.

Hello Luiz,

Is it possible to get this reviewed, or I am supposed to wait until
introspection support being settled?

On 05/28/2013 10:50 AM, qiaonuo...@cn.fujitsu.com wrote:

From: Qiao Nuohanqiaonuo...@cn.fujitsu.com

Hi, all

The last version is here:
http://lists.gnu.org/archive/html/qemu-devel/2013-05/msg02280.html

Command 'dump-guest-memory' was introduced to dump guest's memory. But the
vmcore's format is only elf32 or elf64. The message is here:
http://lists.gnu.org/archive/html/qemu-devel/2012-04/msg03379.html

For migration, 'dump-guest-memory' is supposed to support compression feature.
Because of the regression, the missing of compression feature, we post these
patches to make 'dump-guest-memory' be able to dump guest's in kdump-compressed
format. Then vmcore can be much smaller, and easily be delivered.

The kdump-compressed format is *linux specific* *linux standard* crash dump
format used in kdump framework. The kdump-compressed format is readable only
with the crash utility, and it can be smaller than the ELF format because of
the compression support.

Note, similar to 'dump-guest-memory':
1. The guest should be x86 or x86_64. The other arch is not supported now.
2. If the OS is in the second kernel, gdb may not work well, and crash can
work by specifying '--machdep phys_addr=xxx' in the command line. The
reason is that the second kernel will update the page table, and we can
not get the page table for the first kernel.
3. The cpu's state is stored in QEMU note.
4. The vmcore are able to be compressed with zlib, lzo or snappy. zlib is
available by default, and option '--enable-lzo' or '--enable-snappy'
should be used with configure to make lzo or snappy available.

Changelog:
Changes from v3 to v4:
1. change to avoid conflict with Andreas's patches
2. rebase

Changes from v2 to v3:
1. Address Eric's comment

Changes from v1 to v2:
1. Address Eric  Daniel's comment: fix manner of string copy.
2. Address Eric's comment: replace reinventing new constants by using the
ready-made ones accoring.
3. Address Andreas's comment: remove useless include.


Qiao Nuohan (9):
   dump: Add API to manipulate dump_bitmap
   dump: Add API to manipulate cache_data
   dump: Move struct definition into dump_memroy.h
   dump: Add API to create header of vmcore
   dump: Add API to create data of dump bitmap
   dump: Add API to create page
   dump: Add API to free memory used by creating header, bitmap and page
   dump: Add API to write header, bitmap and page into vmcore
   dump: Make kdump-compressed format available for 'dump-guest-memory'

  Makefile.target  |1 +
  cache_data.c |  121 ++
  configure|   50 +++
  dump.c   |  866 --
  dump_bitmap.c|  171 +
  hmp-commands.hx  |   12 +-
  hmp.c|   23 +-
  include/cache_data.h |   56 +++
  include/dump_bitmap.h|   60 +++
  include/sysemu/dump_memory.h |  178 +
  qapi-schema.json |   22 +-
  qmp-commands.hx  |6 +-
  12 files changed, 1526 insertions(+), 40 deletions(-)
  create mode 100644 cache_data.c
  create mode 100644 dump_bitmap.c
  create mode 100644 include/cache_data.h
  create mode 100644 include/dump_bitmap.h
  create mode 100644 include/sysemu/dump_memory.h





--
Regards
Qiao Nuohan



Re: [Qemu-devel] [PATCH v4 0/9] Make 'dump-guest-memory' dump in kdump-compressed format

2013-06-04 Thread Luiz Capitulino
On Wed, 05 Jun 2013 09:29:19 +0800
Qiao Nuohan qiaonuo...@cn.fujitsu.com wrote:

   I haven't reviewed it yet, but we need introspection support before merging
   this.
 
 Hello Luiz,
 
 Is it possible to get this reviewed, or I am supposed to wait until
 introspection support being settled?

I can review it until the end of this week. If this series is adding a new
argument (which I believe is what it does) then there's only two ways
to get this merged: either we wait for full introspection or you add this
feature as a new command.

I'd prefer to wait for full introspection, but it depends how long it's
going to take to get it merged and how much time you're willing to wait.

Amos, can you give us an update on that work?



Re: [Qemu-devel] [PATCH v4 0/9] Make 'dump-guest-memory' dump in kdump-compressed format

2013-06-04 Thread Luiz Capitulino

[CC'ing Amos this time]

On Wed, 05 Jun 2013 09:29:19 +0800
Qiao Nuohan qiaonuo...@cn.fujitsu.com wrote:

   I haven't reviewed it yet, but we need introspection support before merging
   this.
 
 Hello Luiz,
 
 Is it possible to get this reviewed, or I am supposed to wait until
 introspection support being settled?

I can review it until the end of this week. If this series is adding a new
argument (which I believe is what it does) then there's only two ways
to get this merged: either we wait for full introspection or you add this
feature as a new command.

I'd prefer to wait for full introspection, but it depends how long it's
going to take to get it merged and how much time you're willing to wait.

Amos, can you give us an update on that work?



[Qemu-devel] [PATCH v4 0/9] Make 'dump-guest-memory' dump in kdump-compressed format

2013-05-27 Thread qiaonuohan
From: Qiao Nuohan qiaonuo...@cn.fujitsu.com

Hi, all

The last version is here:
http://lists.gnu.org/archive/html/qemu-devel/2013-05/msg02280.html

Command 'dump-guest-memory' was introduced to dump guest's memory. But the
vmcore's format is only elf32 or elf64. The message is here:
http://lists.gnu.org/archive/html/qemu-devel/2012-04/msg03379.html

For migration, 'dump-guest-memory' is supposed to support compression feature.
Because of the regression, the missing of compression feature, we post these
patches to make 'dump-guest-memory' be able to dump guest's in kdump-compressed
format. Then vmcore can be much smaller, and easily be delivered.

The kdump-compressed format is *linux specific* *linux standard* crash dump
format used in kdump framework. The kdump-compressed format is readable only
with the crash utility, and it can be smaller than the ELF format because of
the compression support.

Note, similar to 'dump-guest-memory':
1. The guest should be x86 or x86_64. The other arch is not supported now.
2. If the OS is in the second kernel, gdb may not work well, and crash can
   work by specifying '--machdep phys_addr=xxx' in the command line. The
   reason is that the second kernel will update the page table, and we can
   not get the page table for the first kernel.
3. The cpu's state is stored in QEMU note.
4. The vmcore are able to be compressed with zlib, lzo or snappy. zlib is
   available by default, and option '--enable-lzo' or '--enable-snappy'
   should be used with configure to make lzo or snappy available.

Changelog:
Changes from v3 to v4:
1. change to avoid conflict with Andreas's patches
2. rebase

Changes from v2 to v3:
1. Address Eric's comment

Changes from v1 to v2:
1. Address Eric  Daniel's comment: fix manner of string copy.
2. Address Eric's comment: replace reinventing new constants by using the
   ready-made ones accoring.
3. Address Andreas's comment: remove useless include.


Qiao Nuohan (9):
  dump: Add API to manipulate dump_bitmap
  dump: Add API to manipulate cache_data
  dump: Move struct definition into dump_memroy.h
  dump: Add API to create header of vmcore
  dump: Add API to create data of dump bitmap
  dump: Add API to create page
  dump: Add API to free memory used by creating header, bitmap and page
  dump: Add API to write header, bitmap and page into vmcore
  dump: Make kdump-compressed format available for 'dump-guest-memory'

 Makefile.target  |1 +
 cache_data.c |  121 ++
 configure|   50 +++
 dump.c   |  866 --
 dump_bitmap.c|  171 +
 hmp-commands.hx  |   12 +-
 hmp.c|   23 +-
 include/cache_data.h |   56 +++
 include/dump_bitmap.h|   60 +++
 include/sysemu/dump_memory.h |  178 +
 qapi-schema.json |   22 +-
 qmp-commands.hx  |6 +-
 12 files changed, 1526 insertions(+), 40 deletions(-)
 create mode 100644 cache_data.c
 create mode 100644 dump_bitmap.c
 create mode 100644 include/cache_data.h
 create mode 100644 include/dump_bitmap.h
 create mode 100644 include/sysemu/dump_memory.h