Re: [Libguestfs] [PATCH nbdkit 0/2] Fix a couple of problems found by Coverity.
On 06/14/2018 08:36 AM, Richard W.M. Jones wrote: There are a few other issues that Coverity found, but I believe all can be ignored ... except one: We don't set umask anywhere inside nbdkit. Coverity complains that this is a problem where we create temporary files, since the result of mkstemp depends implicitly on the umask value. I think we might consider setting umask anyway (eg. to 022) just to make plugin behaviour more predictable. What do you think? Setting umask() is not threadsafe - it must be done up front before any threads can be created (and is therefore unsafe to do in a library that might be linked into a larger multithreaded program). But setting a sane umask up front seems reasonable to me. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org ___ Libguestfs mailing list Libguestfs@redhat.com https://www.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH nbdkit 1/2] plugins: nbd: Free h (handle) along error paths.
On 06/14/2018 08:36 AM, Richard W.M. Jones wrote:
Found by Coverity.
---
plugins/nbd/nbd.c | 2 ++
1 file changed, 2 insertions(+)
ACK.
diff --git a/plugins/nbd/nbd.c b/plugins/nbd/nbd.c
index b9e72bc..2b5569b 100644
--- a/plugins/nbd/nbd.c
+++ b/plugins/nbd/nbd.c
@@ -465,6 +465,7 @@ nbd_open (int readonly)
h->fd = socket (AF_UNIX, SOCK_STREAM, 0);
if (h->fd < 0) {
nbdkit_error ("socket: %m");
+free (h);
return NULL;
}
/* We already validated length during nbd_config_complete */
@@ -559,6 +560,7 @@ nbd_open (int readonly)
err:
close (h->fd);
+ free (h);
return NULL;
}
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
___
Libguestfs mailing list
Libguestfs@redhat.com
https://www.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH] v2v: -o rhv-upload: Optimize http request sending
On Mon, Jun 18, 2018 at 09:58:23PM +0300, Nir Soffer wrote:
> On Mon, Jun 18, 2018 at 9:23 PM Richard W.M. Jones
> wrote:
>
> > On Mon, Jun 18, 2018 at 08:55:13PM +0300, Nir Soffer wrote:
> > > On Mon, Jun 18, 2018 at 1:37 PM Richard W.M. Jones
> > > wrote:
> > >
> > > > On Thu, Jun 14, 2018 at 09:24:48PM +0300, Nir Soffer wrote:
> > > > > On Thu, Jun 14, 2018 at 9:16 PM Nir Soffer wrote:
> > > > > > +headers = {"Content-Type": "application/json",
> > > > > > + "Content-Length", str(len(buf))}
> > > >
> > > > There were a few Python syntax errors such as this one. They
> > > > can be found by running:
> > > >
> > > > make -C v2v check TESTS=test-v2v-python-syntax.sh
> > > >
> > >
> > > Cool. Why not include this in "make check"?
> >
> > It is! That command is just if you want to run the single test.
> >
>
> Strange, I did run "make check" but it did not fail on checking
> the rhv plugin. It was failing because of unralted c-api test.
>
> Maybe make aborted before running the v2v tests?
Right, it runs the tests in subdirectories in order and aborts at the
first subdirectory which fails (this is just how make / automake
works, it's nothing special to libguestfs):
$ make print-subdirs | fold -72 -s
common/mlstdutils generator tests/qemu test-data gnulib/lib
gnulib/tests common/errnostring common/protocol common/qemuopts
common/utils common/structs lib docs examples po common/mlutils
common/mlaugeas common/mlpcre daemon tests/daemon appliance tests/c-api
tests/tmpdirs tests/protocol tests/events tests/parallel tests/create
tests/disks tests/discard tests/mountable tests/network tests/lvm
tests/luks tests/md tests/selinux tests/relabel tests/ntfs tests/btrfs
tests/xfs tests/charsets tests/xml tests/mount-local tests/9p
tests/rsync tests/bigdirs tests/disk-labels tests/hotplug tests/nbd
tests/http tests/syslinux tests/journal tests/relative-paths
tests/gdisk tests/regressions tests/tsk tests/yara common/edit
common/options common/parallel common/progress common/visit
common/windows test-tool fish align cat diff df edit format inspector
make-fs rescue common/miniexpect p2v bash perl perl/examples ocaml
ocaml/examples python python/examples ruby ruby/examples java
java/examples php erlang erlang/examples lua lua/examples gobject
csharp common/mlgettext common/mlprogress common/mlvisit common/mlxml
common/mltools customize builder builder/templates get-kernel resize
sparsify sysprep v2v v2v/test-harness dib tools fuse
utils/boot-analysis utils/boot-benchmark utils/max-disks
utils/qemu-boot utils/qemu-speed-test po-docs
If you want it to continue after the first failure (this also applies
to every automake-using program) then do:
make check -k
less `find -name test-suite.log | xargs grep -l ^FAIL:`
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
___
Libguestfs mailing list
Libguestfs@redhat.com
https://www.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH] v2v: -o rhv-upload: Optimize http request sending
On Mon, Jun 18, 2018 at 9:23 PM Richard W.M. Jones
wrote:
> On Mon, Jun 18, 2018 at 08:55:13PM +0300, Nir Soffer wrote:
> > On Mon, Jun 18, 2018 at 1:37 PM Richard W.M. Jones
> > wrote:
> >
> > > On Thu, Jun 14, 2018 at 09:24:48PM +0300, Nir Soffer wrote:
> > > > On Thu, Jun 14, 2018 at 9:16 PM Nir Soffer wrote:
> > > > > +headers = {"Content-Type": "application/json",
> > > > > + "Content-Length", str(len(buf))}
> > >
> > > There were a few Python syntax errors such as this one. They
> > > can be found by running:
> > >
> > > make -C v2v check TESTS=test-v2v-python-syntax.sh
> > >
> >
> > Cool. Why not include this in "make check"?
>
> It is! That command is just if you want to run the single test.
>
Strange, I did run "make check" but it did not fail on checking
the rhv plugin. It was failing because of unralted c-api test.
Maybe make aborted before running the v2v tests?
>
> > > Not really. In fact we don't have any unit tests for -o rhv-upload
> > > functionality because there's no way to simulate the imageio server.
> > >
> >
> > Actually it is easy to use real imageio server for testing.
> >
> > 1. install ovirt-imageio-daemon
>
> Yup, problem is this part.
>
> However I'll look to see if we can make the tests run optionally _if_
> someone has installed ovirt-imageio-daemon or has a local copy of the
> sources, so the rest of the instructions are useful. I think this
> method looks most promising, but as you say I'd have to see about
> mocking the other engine methods:
>
There is also:
https://github.com/kevin1024/vcrpy
I don't like these mocking libraries very much, but this may be useful.
After you run the tests once with a real system it can record all the
interactions, and the next time you run the tests it will use recorded data.
See https://il.pycon.org/2018/schedule/presentation/37/
>
> > Another option is to start the server from your tests like this. This
> > is how we run our tests.
> >
> > from ovirt_imageio_daemon import server
> > from ovirt_imageio_daemon import config
> >
> > config.daemon.pki_dir = test/pki
> > config.daemon.poll_interval = 0.1
> > config.images.port = 9876
> > config.tickets.socket = "/tmp/ovirt-imageio-daemon.sock"
> >
> > server.start(config)
> >
> > # run your test here...
> >
> > server.stop()
>
> [...]
> > However you can run virt-v2v locally against an oVirt instance without
> > > needing VMware. The command is rather lengthy, but here it is:
> > >
> > > $ virt-builder fedora-27
> > > $ ./run virt-v2v -i disk /var/tmp/fedora-27.img \
> > > -o rhv-upload \
> > > -oc https://ovirt-engine.example.com/ovirt-engine/api \
> > > -os ovirt-data \
> > > -op /tmp/password \
> > > -of raw \
> > > -oo rhv-cafile=/tmp/ca.pem \
> > > -oo rhv-direct
> > >
> > > /tmp/password should contain the oVirt admin password.
> > > /tmp/ca.pem should contain the oVirt CA cert.
> > >
> > > This will create a guest called ‘fedora-27’ which you'll need to
> > > delete (on oVirt) afterwards. You can add ‘-on name’ to name it
> > > something else.
> > >
> >
> > Should we document this?
>
> It's just the normal way to run virt-v2v, eg as documented here:
>
> http://libguestfs.org/virt-v2v.1.html#convert-from-vmware-to-rhv-ovirt
>
> ... except that I've changed the input side to use a local disk.
>
> We did consider having an ‘-i builder’ input method (just for testing).
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-builder quickly builds VMs from scratch
> http://libguestfs.org/virt-builder.1.html
>
___
Libguestfs mailing list
Libguestfs@redhat.com
https://www.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH] v2v: -o rhv-upload: Optimize http request sending
On Mon, Jun 18, 2018 at 08:55:13PM +0300, Nir Soffer wrote:
> On Mon, Jun 18, 2018 at 1:37 PM Richard W.M. Jones
> wrote:
>
> > On Thu, Jun 14, 2018 at 09:24:48PM +0300, Nir Soffer wrote:
> > > On Thu, Jun 14, 2018 at 9:16 PM Nir Soffer wrote:
> > > > +headers = {"Content-Type": "application/json",
> > > > + "Content-Length", str(len(buf))}
> >
> > There were a few Python syntax errors such as this one. They
> > can be found by running:
> >
> > make -C v2v check TESTS=test-v2v-python-syntax.sh
> >
>
> Cool. Why not include this in "make check"?
It is! That command is just if you want to run the single test.
> > Not really. In fact we don't have any unit tests for -o rhv-upload
> > functionality because there's no way to simulate the imageio server.
> >
>
> Actually it is easy to use real imageio server for testing.
>
> 1. install ovirt-imageio-daemon
Yup, problem is this part.
However I'll look to see if we can make the tests run optionally _if_
someone has installed ovirt-imageio-daemon or has a local copy of the
sources, so the rest of the instructions are useful. I think this
method looks most promising, but as you say I'd have to see about
mocking the other engine methods:
> Another option is to start the server from your tests like this. This
> is how we run our tests.
>
> from ovirt_imageio_daemon import server
> from ovirt_imageio_daemon import config
>
> config.daemon.pki_dir = test/pki
> config.daemon.poll_interval = 0.1
> config.images.port = 9876
> config.tickets.socket = "/tmp/ovirt-imageio-daemon.sock"
>
> server.start(config)
>
> # run your test here...
>
> server.stop()
[...]
> However you can run virt-v2v locally against an oVirt instance without
> > needing VMware. The command is rather lengthy, but here it is:
> >
> > $ virt-builder fedora-27
> > $ ./run virt-v2v -i disk /var/tmp/fedora-27.img \
> > -o rhv-upload \
> > -oc https://ovirt-engine.example.com/ovirt-engine/api \
> > -os ovirt-data \
> > -op /tmp/password \
> > -of raw \
> > -oo rhv-cafile=/tmp/ca.pem \
> > -oo rhv-direct
> >
> > /tmp/password should contain the oVirt admin password.
> > /tmp/ca.pem should contain the oVirt CA cert.
> >
> > This will create a guest called ‘fedora-27’ which you'll need to
> > delete (on oVirt) afterwards. You can add ‘-on name’ to name it
> > something else.
> >
>
> Should we document this?
It's just the normal way to run virt-v2v, eg as documented here:
http://libguestfs.org/virt-v2v.1.html#convert-from-vmware-to-rhv-ovirt
... except that I've changed the input side to use a local disk.
We did consider having an ‘-i builder’ input method (just for testing).
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
___
Libguestfs mailing list
Libguestfs@redhat.com
https://www.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH] v2v: -o rhv-upload: Optimize http request sending
On Mon, Jun 18, 2018 at 1:37 PM Richard W.M. Jones
wrote:
> On Thu, Jun 14, 2018 at 09:24:48PM +0300, Nir Soffer wrote:
> > On Thu, Jun 14, 2018 at 9:16 PM Nir Soffer wrote:
> > > +headers = {"Content-Type": "application/json",
> > > + "Content-Length", str(len(buf))}
>
> There were a few Python syntax errors such as this one. They
> can be found by running:
>
> make -C v2v check TESTS=test-v2v-python-syntax.sh
>
Cool. Why not include this in "make check"?
>
> However I have now fixed them.
>
Thanks!
>
> > Do we have an easy way to test the plugin without running the entire
> > virt-v2v pipeline?
>
> Not really. In fact we don't have any unit tests for -o rhv-upload
> functionality because there's no way to simulate the imageio server.
>
Actually it is easy to use real imageio server for testing.
1. install ovirt-imageio-daemon
2. configure it for testing
# /etc/ovirt-imageio-daemon/daemon.conf
[daemon]
# directory with this structure
# certs/
# vdsmcert.pem
# keys
# vdsmkey.pem
pki_dir = test/pki
poll_interval = 0.1
[images]
port = 9876
[tickets]
socket = /tmp/ovirt-imageio-daemon.sock
3. Now you can run it
/usr/bin/ovirt-imageio-daemon
To upload, without real engine, you can install your own ticket like this:
1. Create a ticket json:
$ cat ticket.json
{
"uuid": "test",
"size": 1073741824,
"url": "file:///var/tmp/sd-uukd/vol-uuid",
"timeout": 3000,
"ops": ["read", "write"]
}
2. Install the ticket
$ curl --unix-socket /tmp/vdsm/ovirt-imageio-daemon.sock \
-X PUT \
--upload-file ticket.json \
http://localhost/tickets/test
3. Create the image:
file:
touch /var/tmp/sd-uuuid/vol-uuid
(note that the file system must support direct I/O)
block:
lvcreate -n vol-uuid -L 1g sd-uuid
At this point you can upload or download using this transfer_url:
https://localhost:9876/images/test
We can make it even easier by supporting command line options so
you don't need to change /etc/ovirt-imageio-daemon, or even install
the package. For example:
git clone https://github.com/oVirt/ovirt-imageio.git
src/ovirt-image/daemon/ovirt-imageio-daemon -f my-test.conf
Another option is to start the server from your tests like this. This
is how we run our tests.
from ovirt_imageio_daemon import server
from ovirt_imageio_daemon import config
config.daemon.pki_dir = test/pki
config.daemon.poll_interval = 0.1
config.images.port = 9876
config.tickets.socket = "/tmp/ovirt-imageio-daemon.sock"
server.start(config)
# run your test here...
server.stop()
Note that we did not complete the port to python 3 yest, so upload tests
will work only with python 2.
The missing part is how to test without real engine. I think the best way
would
be to monkeypatch the ovirtsdk module so it does not send any requests,
and instead return fake responses. This add risk of having incorrect mocked
response, or not detecting wrong calls, but will make testing very easy,
which
make it easy for random contributors.
However you can run virt-v2v locally against an oVirt instance without
> needing VMware. The command is rather lengthy, but here it is:
>
> $ virt-builder fedora-27
> $ ./run virt-v2v -i disk /var/tmp/fedora-27.img \
> -o rhv-upload \
> -oc https://ovirt-engine.example.com/ovirt-engine/api \
> -os ovirt-data \
> -op /tmp/password \
> -of raw \
> -oo rhv-cafile=/tmp/ca.pem \
> -oo rhv-direct
>
> /tmp/password should contain the oVirt admin password.
> /tmp/ca.pem should contain the oVirt CA cert.
>
> This will create a guest called ‘fedora-27’ which you'll need to
> delete (on oVirt) afterwards. You can add ‘-on name’ to name it
> something else.
>
Should we document this?
>
> In any case I have fixed and verified this patch and will push it
> soon, thanks.
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-builder quickly builds VMs from scratch
> http://libguestfs.org/virt-builder.1.html
>
___
Libguestfs mailing list
Libguestfs@redhat.com
https://www.redhat.com/mailman/listinfo/libguestfs
[Libguestfs] [PATCH] v2v: changed to the actual size (if known).
Note that this attribute is optional.
Thanks: Arik Hadas
---
v2v/create_ovf.ml| 11 ---
v2v/test-v2v-o-rhv.ovf.expected | 2 +-
v2v/test-v2v-o-rhv.sh| 1 +
v2v/test-v2v-o-vdsm-options.ovf.expected | 2 +-
v2v/test-v2v-o-vdsm-options.sh | 1 +
5 files changed, 12 insertions(+), 5 deletions(-)
diff --git a/v2v/create_ovf.ml b/v2v/create_ovf.ml
index e0f81e0a2..9e0c772fd 100644
--- a/v2v/create_ovf.ml
+++ b/v2v/create_ovf.ml
@@ -819,12 +819,17 @@ and add_disks targets guestcaps output_alloc sd_uuid
image_uuids vol_uuids
(* Add disk to node. *)
let disk =
-e "File" [
+let attrs = ref [
"ovf:href", fileref;
"ovf:id", vol_uuid;
- "ovf:size", Int64.to_string ov.ov_virtual_size; (* NB: in bytes *)
"ovf:description", generated_by;
-] [] in
+] in
+(match t.target_actual_size with
+ | None -> ()
+ | Some actual_size ->
+List.push_back attrs ("ovf:size", Int64.to_string actual_size)
+);
+e "File" !attrs [] in
append_child disk references;
(* Add disk to DiskSection. *)
diff --git a/v2v/test-v2v-o-rhv.ovf.expected b/v2v/test-v2v-o-rhv.ovf.expected
index 1deec9c9d..7bcc456c5 100644
--- a/v2v/test-v2v-o-rhv.ovf.expected
+++ b/v2v/test-v2v-o-rhv.ovf.expected
@@ -2,7 +2,7 @@
-
+
List of networks
diff --git a/v2v/test-v2v-o-rhv.sh b/v2v/test-v2v-o-rhv.sh
index c9ec0dbce..eb22ed95b 100755
--- a/v2v/test-v2v-o-rhv.sh
+++ b/v2v/test-v2v-o-rhv.sh
@@ -74,6 +74,7 @@ sed -i \
-e "s/$DISK_ID/#DISK_ID#/g" \
-e "s/$VM_ID/#VM_ID#/g" \
-e "s/$VOL_ID/#VOL_ID#/g" \
+ -e "s/\('"$RE_UUID"'#UUID#[^<]*#DATE#http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData'
xmlns:vssd='http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_VirtualSystemSettingData'
xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'
xmlns:ovf='http://schemas.dmtf.org/ovf/envelope/1/'
xmlns:ovirt='http://www.ovirt.org/ovf' ovf:version='0.9'>
-
+
List of networks
diff --git a/v2v/test-v2v-o-vdsm-options.sh b/v2v/test-v2v-o-vdsm-options.sh
index 2c9de5eb7..c70446aa1 100755
--- a/v2v/test-v2v-o-vdsm-options.sh
+++ b/v2v/test-v2v-o-vdsm-options.sh
@@ -82,6 +82,7 @@
RE_UUID='\<[0-9a-fA-F]\{8\}-[0-9a-fA-F]\{4\}-[0-9a-fA-F]\{4\}-[0-9a-fA-F]\{4\}-[
# Filter variable strings
sed -i \
+ -e "s/\('"$RE_UUID"'#UUID#[^<]*#DATE#https://www.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH] v2v: -o rhv-upload: Optimize http request sending
On Thu, Jun 14, 2018 at 09:24:48PM +0300, Nir Soffer wrote:
> On Thu, Jun 14, 2018 at 9:16 PM Nir Soffer wrote:
> > +headers = {"Content-Type": "application/json",
> > + "Content-Length", str(len(buf))}
There were a few Python syntax errors such as this one. They
can be found by running:
make -C v2v check TESTS=test-v2v-python-syntax.sh
However I have now fixed them.
> Do we have an easy way to test the plugin without running the entire
> virt-v2v pipeline?
Not really. In fact we don't have any unit tests for -o rhv-upload
functionality because there's no way to simulate the imageio server.
However you can run virt-v2v locally against an oVirt instance without
needing VMware. The command is rather lengthy, but here it is:
$ virt-builder fedora-27
$ ./run virt-v2v -i disk /var/tmp/fedora-27.img \
-o rhv-upload \
-oc https://ovirt-engine.example.com/ovirt-engine/api \
-os ovirt-data \
-op /tmp/password \
-of raw \
-oo rhv-cafile=/tmp/ca.pem \
-oo rhv-direct
/tmp/password should contain the oVirt admin password.
/tmp/ca.pem should contain the oVirt CA cert.
This will create a guest called ‘fedora-27’ which you'll need to
delete (on oVirt) afterwards. You can add ‘-on name’ to name it
something else.
In any case I have fixed and verified this patch and will push it
soon, thanks.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
___
Libguestfs mailing list
Libguestfs@redhat.com
https://www.redhat.com/mailman/listinfo/libguestfs
[Libguestfs] [PATCH] v2v: Add attribute containing disk virtual size.
Required ever since this change was made to oVirt: https://gerrit.ovirt.org/#/c/91902/ Thanks: Arik Hadas --- v2v/create_ovf.ml| 1 + v2v/test-v2v-o-rhv.ovf.expected | 2 +- v2v/test-v2v-o-vdsm-options.ovf.expected | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/v2v/create_ovf.ml b/v2v/create_ovf.ml index ac3c61b13..e0f81e0a2 100644 --- a/v2v/create_ovf.ml +++ b/v2v/create_ovf.ml @@ -835,6 +835,7 @@ and add_disks targets guestcaps output_alloc sd_uuid image_uuids vol_uuids | OVirt -> image_uuid | RHVExportStorageDomain -> vol_uuid); "ovf:size", Int64.to_string size_gb; + "ovf:capacity", Int64.to_string ov.ov_virtual_size; "ovf:fileRef", fileref; "ovf:parentRef", ""; "ovf:vm_snapshot_id", uuidgen (); diff --git a/v2v/test-v2v-o-rhv.ovf.expected b/v2v/test-v2v-o-rhv.ovf.expected index 342eb99d3..1deec9c9d 100644 --- a/v2v/test-v2v-o-rhv.ovf.expected +++ b/v2v/test-v2v-o-rhv.ovf.expected @@ -10,7 +10,7 @@ List of Virtual Disks - + windows diff --git a/v2v/test-v2v-o-vdsm-options.ovf.expected b/v2v/test-v2v-o-vdsm-options.ovf.expected index af1ca01db..c2a3e336c 100644 --- a/v2v/test-v2v-o-vdsm-options.ovf.expected +++ b/v2v/test-v2v-o-vdsm-options.ovf.expected @@ -10,7 +10,7 @@ List of Virtual Disks - + windows -- 2.16.2 ___ Libguestfs mailing list Libguestfs@redhat.com https://www.redhat.com/mailman/listinfo/libguestfs
