Re: [Qemu-block] [Libguestfs] [Qemu-devel] [qemu-img] support for XVA

[Gandalf Corvotempesta]

https://stacklet.com/downloads/XenServer-XVA-Template-Debian-7.8-Lightweight-x86

Some XVAs

Il 15 nov 2017 10:42 PM, "Gandalf Corvotempesta" <
gandalf.corvotempe...@gmail.com> ha scritto:

> I'm thinking on how to prove you a sample XVA
> I have to create (and populate) a VM because an empty image will result in
> an empty XVA
> And a VM is 300-400Mb as minimum
>
> Il 15 nov 2017 10:30 PM, "Max Reitz" <mre...@redhat.com> ha scritto:
>
>> On 2017-11-15 21:41, Gandalf Corvotempesta wrote:
>> > 2017-11-15 21:29 GMT+01:00 Richard W.M. Jones <rjo...@redhat.com>:
>> >> Gandalf, is there an XVA file publically available (pref. not
>> >> too big) that we can look at?
>> >
>> > I can try to provide one, but it's simple:
>> >
>> > # tar tvf 20160630_124823_aa72_.xva.gz | head -n 50
>> > -- 0/0   42353 1970-01-01 01:00 ova.xml
>> > -- 0/0 1048576 1970-01-01 01:00 Ref:175/
>> > -- 0/0  40 1970-01-01 01:00
>> Ref:175/.checksum
>> > -- 0/0 1048576 1970-01-01 01:00 Ref:175/0001
>> > -- 0/0  40 1970-01-01 01:00
>> Ref:175/0001.checksum
>> > -- 0/0 1048576 1970-01-01 01:00 Ref:175/0003
>> > -- 0/0  40 1970-01-01 01:00
>> Ref:175/0003.checksum
>> > -- 0/0 1048576 1970-01-01 01:00 Ref:175/0004
>> > -- 0/0  40 1970-01-01 01:00
>> Ref:175/0004.checksum
>> > -- 0/0 1048576 1970-01-01 01:00 Ref:175/0005
>> > -- 0/0  40 1970-01-01 01:00
>> Ref:175/0005.checksum
>> > -- 0/0 1048576 1970-01-01 01:00 Ref:175/0006
>> > -- 0/0  40 1970-01-01 01:00
>> Ref:175/0006.checksum
>> > -- 0/0 1048576 1970-01-01 01:00 Ref:175/0007
>> > -- 0/0  40 1970-01-01 01:00
>> Ref:175/0007.checksum
>> > -- 0/0 1048576 1970-01-01 01:00 Ref:175/0009
>> > -- 0/0  40 1970-01-01 01:00
>> Ref:175/0009.checksum
>> > -- 0/0 1048576 1970-01-01 01:00 Ref:175/0010
>> > -- 0/0  40 1970-01-01 01:00
>> Ref:175/0010.checksum
>> >
>> >
>> > You can ignore the ova.xml and just use the "Ref:175" directory.
>> > Inside the XVA you'll fine one "Ref" directory for each virtual disk
>> > (ref number is different for each disk)
>> > Inside each Ref directory, you'll get tons of 1MB file, corrisponding
>> > to the RAW image.
>> > You have to merge these files in a single raw file with just an
>> > exception: numbers are not sequential.
>> > as you can see above, we have: , 0001, 0003
>> >
>> > The 0002 is missing, because it's totally blank. XenServer doesn't
>> > export any empty block, thus it will skip the corrisponding 1MB file.
>> > When building the raw image, you have to replace empty blocks with 1MB
>> > full of zeros.
>> >
>> > This is (in addition to the tar extract) the most time-consuming part.
>> > Currently I'm rebuilding a 250GB image, with tons of files to be
>> > merge.
>> > If qemu-img can be patched to automatically convert this kind of
>> > format, I can save about 3 hours (30 minutes for extracting the
>> > tarball, and about 2 hours to merge 250-300GB image)
>>
>> Hmmm...  OK, so as Rich said, you can use the raw driver with an offset
>> and a size.  And you can use the null-co driver to emulate holes.  And
>> qemu-img convert supports concatenation.
>>
>> Taking all of that together, it should be possible to create an input
>> "file" per 1 MB block and concatenate all of those with qemu-img
>> convert.  The only issue is that it'll get you a *really* long command
>> line.
>>
>> (And you'll still have to gunzip the archive before, of course.)
>>
>> I'll try to write a script, but of course it's hart without an example...
>>
>> Max
>>
>>

Re: [Qemu-block] [Libguestfs] [Qemu-devel] [qemu-img] support for XVA

[Gandalf Corvotempesta]

Il 15 nov 2017 11:28 PM, "Max Reitz"  ha scritto:

Hmmm, that gives me a timeout when downloading.  And another image I got
gives me an "argument list too long" -- which I had feared already...
It has a size of 3 MB and Linux "only" allows 2...


https://cloud.vates.fr/index.php/s/jynIvCjhso47nR8/download

Re: [Qemu-block] [Qemu-devel] [qemu-img] support for XVA

[Gandalf Corvotempesta]

2017-11-15 20:59 GMT+01:00 Max Reitz :
> Well, you can't just add support to qemu-img alone either.  Every image
> format supported by qemu-img is one supported by qemu as a whole and
> thus needs a proper block driver that needs to support random accesses
> as well.

I was talking about qemu-img convert, just to convert an XVA image to
something different, in a single pass, without having to extract the
tar.

Re: [Qemu-block] [Libguestfs] [Qemu-devel] [qemu-img] support for XVA

[Gandalf Corvotempesta]

I'm thinking on how to prove you a sample XVA
I have to create (and populate) a VM because an empty image will result in
an empty XVA
And a VM is 300-400Mb as minimum

Il 15 nov 2017 10:30 PM, "Max Reitz" <mre...@redhat.com> ha scritto:

> On 2017-11-15 21:41, Gandalf Corvotempesta wrote:
> > 2017-11-15 21:29 GMT+01:00 Richard W.M. Jones <rjo...@redhat.com>:
> >> Gandalf, is there an XVA file publically available (pref. not
> >> too big) that we can look at?
> >
> > I can try to provide one, but it's simple:
> >
> > # tar tvf 20160630_124823_aa72_.xva.gz | head -n 50
> > -- 0/0   42353 1970-01-01 01:00 ova.xml
> > -- 0/0 1048576 1970-01-01 01:00 Ref:175/
> > -- 0/0  40 1970-01-01 01:00 Ref:175/.checksum
> > -- 0/0 1048576 1970-01-01 01:00 Ref:175/0001
> > -- 0/0  40 1970-01-01 01:00 Ref:175/0001.checksum
> > -- 0/0 1048576 1970-01-01 01:00 Ref:175/0003
> > -- 0/0  40 1970-01-01 01:00 Ref:175/0003.checksum
> > -- 0/0 1048576 1970-01-01 01:00 Ref:175/0004
> > -- 0/0  40 1970-01-01 01:00 Ref:175/0004.checksum
> > -- 0/0 1048576 1970-01-01 01:00 Ref:175/0005
> > -- 0/0  40 1970-01-01 01:00 Ref:175/0005.checksum
> > -- 0/0 1048576 1970-01-01 01:00 Ref:175/0006
> > -- 0/0  40 1970-01-01 01:00 Ref:175/0006.checksum
> > -- 0/0 1048576 1970-01-01 01:00 Ref:175/0007
> > -- 0/0  40 1970-01-01 01:00 Ref:175/0007.checksum
> > -- 0/0 1048576 1970-01-01 01:00 Ref:175/0009
> > -- 0/0  40 1970-01-01 01:00 Ref:175/0009.checksum
> > -- 0/0 1048576 1970-01-01 01:00 Ref:175/0010
> > -- 0/0  40 1970-01-01 01:00 Ref:175/0010.checksum
> >
> >
> > You can ignore the ova.xml and just use the "Ref:175" directory.
> > Inside the XVA you'll fine one "Ref" directory for each virtual disk
> > (ref number is different for each disk)
> > Inside each Ref directory, you'll get tons of 1MB file, corrisponding
> > to the RAW image.
> > You have to merge these files in a single raw file with just an
> > exception: numbers are not sequential.
> > as you can see above, we have: , 0001, 0003
> >
> > The 0002 is missing, because it's totally blank. XenServer doesn't
> > export any empty block, thus it will skip the corrisponding 1MB file.
> > When building the raw image, you have to replace empty blocks with 1MB
> > full of zeros.
> >
> > This is (in addition to the tar extract) the most time-consuming part.
> > Currently I'm rebuilding a 250GB image, with tons of files to be
> > merge.
> > If qemu-img can be patched to automatically convert this kind of
> > format, I can save about 3 hours (30 minutes for extracting the
> > tarball, and about 2 hours to merge 250-300GB image)
>
> Hmmm...  OK, so as Rich said, you can use the raw driver with an offset
> and a size.  And you can use the null-co driver to emulate holes.  And
> qemu-img convert supports concatenation.
>
> Taking all of that together, it should be possible to create an input
> "file" per 1 MB block and concatenate all of those with qemu-img
> convert.  The only issue is that it'll get you a *really* long command
> line.
>
> (And you'll still have to gunzip the archive before, of course.)
>
> I'll try to write a script, but of course it's hart without an example...
>
> Max
>
>

Re: [Qemu-block] [Libguestfs] [Qemu-devel] [qemu-img] support for XVA

[Gandalf Corvotempesta]

2017-11-15 21:29 GMT+01:00 Richard W.M. Jones :
> Gandalf, is there an XVA file publically available (pref. not
> too big) that we can look at?

I can try to provide one, but it's simple:

# tar tvf 20160630_124823_aa72_.xva.gz | head -n 50
-- 0/0   42353 1970-01-01 01:00 ova.xml
-- 0/0 1048576 1970-01-01 01:00 Ref:175/
-- 0/0  40 1970-01-01 01:00 Ref:175/.checksum
-- 0/0 1048576 1970-01-01 01:00 Ref:175/0001
-- 0/0  40 1970-01-01 01:00 Ref:175/0001.checksum
-- 0/0 1048576 1970-01-01 01:00 Ref:175/0003
-- 0/0  40 1970-01-01 01:00 Ref:175/0003.checksum
-- 0/0 1048576 1970-01-01 01:00 Ref:175/0004
-- 0/0  40 1970-01-01 01:00 Ref:175/0004.checksum
-- 0/0 1048576 1970-01-01 01:00 Ref:175/0005
-- 0/0  40 1970-01-01 01:00 Ref:175/0005.checksum
-- 0/0 1048576 1970-01-01 01:00 Ref:175/0006
-- 0/0  40 1970-01-01 01:00 Ref:175/0006.checksum
-- 0/0 1048576 1970-01-01 01:00 Ref:175/0007
-- 0/0  40 1970-01-01 01:00 Ref:175/0007.checksum
-- 0/0 1048576 1970-01-01 01:00 Ref:175/0009
-- 0/0  40 1970-01-01 01:00 Ref:175/0009.checksum
-- 0/0 1048576 1970-01-01 01:00 Ref:175/0010
-- 0/0  40 1970-01-01 01:00 Ref:175/0010.checksum


You can ignore the ova.xml and just use the "Ref:175" directory.
Inside the XVA you'll fine one "Ref" directory for each virtual disk
(ref number is different for each disk)
Inside each Ref directory, you'll get tons of 1MB file, corrisponding
to the RAW image.
You have to merge these files in a single raw file with just an
exception: numbers are not sequential.
as you can see above, we have: , 0001, 0003

The 0002 is missing, because it's totally blank. XenServer doesn't
export any empty block, thus it will skip the corrisponding 1MB file.
When building the raw image, you have to replace empty blocks with 1MB
full of zeros.

This is (in addition to the tar extract) the most time-consuming part.
Currently I'm rebuilding a 250GB image, with tons of files to be
merge.
If qemu-img can be patched to automatically convert this kind of
format, I can save about 3 hours (30 minutes for extracting the
tarball, and about 2 hours to merge 250-300GB image)

Re: [Qemu-block] [Libguestfs] [Qemu-devel] [qemu-img] support for XVA

[Gandalf Corvotempesta]

I did something different, that will build a raw image directly from a
xenserver export, on the fly.
Compared the resulting file (via MD5) with xenmygrate.py and there is a match.

Currently, this is the faster way to convert a XenServer image to a
raw file. Don't need to wait
for export, tar extract and conversion. It does all of that, at the
same time during the VM export in a single pass.

If someone interested in testing it, it would be apreciated.

2017-11-16 15:02 GMT+01:00 Max Reitz <mre...@redhat.com>:
> On 2017-11-16 11:08, Gandalf Corvotempesta wrote:
>> 2017-11-15 23:55 GMT+01:00 Max Reitz <mre...@redhat.com>:
>>> https://xanclic.moe/convert-xva.rb -- does this work?
>>> (It seems to works on the two example images I found...)
>>>
>>> An example is in the code, you use it like this:
>>>
>>> $ ./convert-xva.rb ~/Downloads/stats-appliance-2.36.020502.xva Ref:73
>>
>>
>> It doesn't work on huge images:
>>
>> # time convert-xva.rb 20171115_193814_efff_.xva Ref:10 -O qcow2 disk1.qcow2
>> /usr/bin/convert-xva.rb:119:in `exec': Argument list too long -
>> qemu-img (Errno::E2BIG)
>> from /usr/bin/convert-xva.rb:119:in `'
>>
>> real 9m41.414s
>> user 0m19.280s
>> sys 0m3.260s
>
> Well, there is this:
>
> https://github.com/XanClic/qemu.rb/blob/master/examples/convert-xva-online.rb
>
> That should get around that restriction, but currently it is stupidly
> slow (because it just issues all 1 MB copy operations simultaneously),
> and it doesn't use qemu-img convert.  Instead, it uses a real qemu with
> block jobs.  Ideally, you'd use it like this:
>
> $ ./convert-xva-online.rb ~/Downloads/stats-appliance-2.36.020502.xva
> Ref:73  81936 M
> $ qemu-img create -f qcow2 [whatever options you need] disk1.qcow2 \
> 81936M
> Formatting 'disk1.qcow2', fmt=qcow2 size=85916123136 encryption=off
> cluster_size=65536 lazy_refcounts=off refcount_bits=16
> $ ./convert-xva-online.rb ~/Downloads/stats-appliance-2.36.020502.xva \
> Ref:73 \
> '{"driver":"qcow2",
>   "detect-zeroes":"unmap",
>   "discard":"unmap",
>   "file":{"driver":"file","filename":"disk1.qcow2"}}'
> Adding block devices...
> Starting block jobs...
> 100.00 % of jobs completed
>
> (Maybe I can get it faster.)
>
> Max
>


xva_conv.sh
Description: Bourne shell script

Re: [Qemu-block] [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Gandalf Corvotempesta]

2017-11-16 11:01 GMT+01:00 Richard W.M. Jones :
> As mentioned before you can use this to do a qemu-img convert using
> captive nbdkit:
>
>   $ nbdkit -U - \
>   perl script=./xva-reader.pl file=./debian8cloud.xva size=4294967296 \
>   --run 'qemu-img convert -f raw $nbd -O qcow2 /var/tmp/output.qcow2 -p'


What if XVA is hosting 2 or more partitions ? Which partition are you
converting with this command ?

Re: [Qemu-block] [Libguestfs] [Qemu-devel] [qemu-img] support for XVA

[Gandalf Corvotempesta]

2017-11-15 23:55 GMT+01:00 Max Reitz :
> https://xanclic.moe/convert-xva.rb -- does this work?
> (It seems to works on the two example images I found...)
>
> An example is in the code, you use it like this:
>
> $ ./convert-xva.rb ~/Downloads/stats-appliance-2.36.020502.xva Ref:73


It doesn't work on huge images:

# time convert-xva.rb 20171115_193814_efff_.xva Ref:10 -O qcow2 disk1.qcow2
/usr/bin/convert-xva.rb:119:in `exec': Argument list too long -
qemu-img (Errno::E2BIG)
from /usr/bin/convert-xva.rb:119:in `'

real 9m41.414s
user 0m19.280s
sys 0m3.260s