On Wed, May 30, 2018 at 12:35 AM Eric Blake <ebl...@redhat.com> wrote:
> On 05/29/2018 04:11 PM, Nir Soffer wrote:
>
> >> I think real streaming is unlikely to happen because most image formats
> >> that QEMU supports aren't made that way. If there is a compelling
> >> reason, we can consider it, but it would work only with very few target
> >> formats and as such would have to be separate from existing commands.
> >>
> >
> > Real streaming is exactly what we want, and we need it only for qcow2
> > format,
> > because it is our preferred way to pack images in ova.
> >
> > We have 2 possible use cases:
> >
> > Exporting images or ova files:
> >
> > image in any format -> qemu-img -> [qcow2 byte stream] -> imageio http
> > server -> http client
>
> image in any format -> qemu-img measure (to learn how large to size qcow2)
> then create destination qcow2 file that large
Isn't this a temporary qcow2 file we want to avoid?
> and serve it over NBD as
> raw (perhaps using an nbdkit plugin for this part)
> then qemu-img convert to destination format qcow2 as NBD client
>
> So, as long as your NBD server (via nbdkit plugin) can talk to imageio
> http server -> http client, and sized things properly according to
> qemu-img measure, then qemu-img can write qcow2 (rather than it's more
> usual raw) over the NBD connection, and when the process is complete,
> the http client will have a fully-populated qcow2 file with no temporary
> files created in the meantime.
>
Ok, it may work. What I need is:
qemu-img converting image to qcow2 format to a nbd server
nbd server writing qcow2 stream to stdout
imageio running both, sending data from nbd server stdout to client
But this means the nbd server cannot handle flow like:
- zero entire disk
- write data block 1 at offset x
- write data block 2 at offset y
We have seen this in virt-v2v nbdkit plugin on the server side using
raw output format, maybe this is not an issue with qcow2 output format?
qemu-img must know that the transport cannot seek.
When I tried in the past to do something like this it did not work, but
maybe
I was missing some options.
I think this should be implemented as a transport driver, like curl/ driver,
instead of creating these complex pipelines.
>
> > Importing images or ova files:
> >
> > http client -> imageio http server -> [qcow2 byte stream] -> qemu-img ->
> > image in any format
>
> Same sort of thing - as long as the NBD server is serving a qcow2 file
> as raw data, then the NBD client is interpreting that data as qcow2,
> then qemu-img convert should be able to convert that qcow2 stream into
> any format.
>
> Or, put another way, usually you do the conversion from qcow2 to raw at
> the server, then the client sees raw bytes:
>
> qemu-nbd -f qcow2 file.qcow2 # expose only the guest-visible bytes...
qemu-img convert -f raw nbd://host output # and write those bytes
> but in this case, you'd be serving raw bytes at the server, and letting
> the client do qcow2 conversion:
>
> qemu-nbd -f raw file.qcow2 # expose the full qcow2 file...
> qemu-img convert -f qcow2 nbd://host output # and extract the guest view
>
> where using nbdkit instead of qemu-nbd as your point of contact with the
> imageio http server may make more sense.
>
> >
> > If we will have this, we don't need to create temporary storage space
> and we
> > can avoid several image copies in the process.
> >
> > This will also improve the user experience, avoiding the wait until
> > a ova is created before the user can start downloading it.
> >
> > As for OVA files, I think it might be useful to have a tar block driver
> >> instead which would allow you to open a file inside a tar archive (you
> >> could then also directly run an OVA without extracting it first). We
> >> probably wouldn't be able to support resizing images there, but that
> >> should be okay.
> >>
> >> If you can create a tar file that reserves space for the image file
> >> without actually writing it, a possible workaround today would be using
> >> the offset/size runtime options of the raw driver to convert directly
> >> into a region inside the tar archive.
> >>
> >
> > What are the offset/size runtime options? I cannot find anything about
> > them in man qemu-img.
>
> ##
> # @BlockdevOptionsRaw:
> #
> # Driver specific block device options for the raw driver.
> #
> # @offset: position where the block device starts
> # @size: the assumed size of the device
> #
> # Since: 2.9
> ##
> { 'struct': 'BlockdevOptionsRaw',
> 'base': 'BlockdevOptionsGenericFormat',
> 'data': { '*offset': 'int', '*size': 'int' } }
>
>
> Yeah, it's a pity that "qemu-img create -o help -f raw" has forgotten to
> document them, the way "qemu-img create -o help -f qcow2" does for its
> options, so we should fix that.
>
Thanks for the hint, but I still don't understand for which command these
options can be used, and how.
Can you show an example of using the options?
Nir
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/3RS4CPNUAMSGVMI376ZLKOGYQZQC55QA/
