Hi, few small comments below, but generally LGTM.
On Thu, 5 Apr 2018 14:26:07 +0100 "Richard W.M. Jones" <[email protected]> wrote: > This adds a new output mode to virt-v2v. virt-v2v -o rhv-upload > streams images directly to an oVirt or RHV >= 4 Data Domain using the > oVirt SDK v4. It is more efficient than -o rhv because it does not > need to go via the Export Storage Domain, and is possible for humans > to use unlike -o vdsm. > > The implementation uses the Python SDK (‘ovirtsdk4’ module). An > nbdkit Python 3 plugin translates NBD calls from qemu into HTTPS > requests to oVirt via the SDK. > --- > .gitignore | 3 + > TODO | 27 ++ > v2v/Makefile.am | 30 +- > v2v/cmdline.ml | 41 +++ > v2v/embed.sh | 45 +++ > v2v/output_rhv_upload.ml | 390 ++++++++++++++++++++++++++ > v2v/output_rhv_upload.mli | 33 +++ > v2v/output_rhv_upload_createvm_source.mli | 19 ++ > v2v/output_rhv_upload_plugin_source.mli | 19 ++ > v2v/output_rhv_upload_precheck_source.mli | 19 ++ > v2v/rhv-upload-createvm.py | 86 ++++++ > v2v/rhv-upload-plugin.py | 438 > ++++++++++++++++++++++++++++++ > v2v/rhv-upload-precheck.py | 73 +++++ > v2v/test-v2v-o-rhv-upload-oo-query.sh | 38 +++ > v2v/test-v2v-python-syntax.sh | 45 +++ > v2v/virt-v2v.pod | 138 ++++++++-- > 16 files changed, 1426 insertions(+), 18 deletions(-) > > diff --git a/.gitignore b/.gitignore > index d72447d1d..211376eef 100644 > --- a/.gitignore > +++ b/.gitignore > @@ -654,6 +654,9 @@ Makefile.in > /utils/qemu-speed-test/qemu-speed-test > /v2v/.depend > /v2v/oUnit-* > +/v2v/output_rhv_upload_createvm_source.ml > +/v2v/output_rhv_upload_plugin_source.ml > +/v2v/output_rhv_upload_precheck_source.ml > /v2v/real-*.d/ > /v2v/real-*.img > /v2v/real-*.xml > diff --git a/TODO b/TODO > index 2e37ce67c..d196a3f6b 100644 > --- a/TODO > +++ b/TODO > @@ -570,3 +570,30 @@ Subsecond handling in virt-diff, virt-ls > > Handle nanoseconds properly. You should be able to specify them on > the command line and display them. > + > +virt-v2v -o rhv-upload > +---------------------- > + > +* Set or disable the ticket timeout. The default is going to be > + increased (from current 60 seconds), so maybe we won't have to > + set it. See also: > + https://bugzilla.redhat.com/show_bug.cgi?id=1563278 > + > +* qcow2 cannot be supported yet because there is not yet any > + concept in imageio of read+write handles. > + https://bugzilla.redhat.com/show_bug.cgi?id=1563299 > + > +* preallocated cannot be supported yet because imageio doesn't > + know how to zero the image efficiently, instead it runs an > + fallocate process which writes to every block and that takes > + many minutes. > + > +* Really check what insecure/rhv_cafile do and implement it correctly. > + > +* Measure and resolve performance problems. > + > +* Allocated image size is unknown for v2v uploads, but imageio needs > + to know it. We pass initial_size == provisioned_size == virtual size. > + That can't be fixed from the v2v side. > + > +* There are unresolved issues about how to clean up disks on failure. > diff --git a/v2v/Makefile.am b/v2v/Makefile.am > index 6e71cae3c..f36731750 100644 > --- a/v2v/Makefile.am > +++ b/v2v/Makefile.am > @@ -22,12 +22,19 @@ generator_built = \ > uefi.mli > > BUILT_SOURCES = \ > - $(generator_built) > + $(generator_built) \ > + output_rhv_upload_createvm_source.ml \ > + output_rhv_upload_plugin_source.ml \ > + output_rhv_upload_precheck_source.ml > > EXTRA_DIST = \ > $(SOURCES_MLI) $(SOURCES_ML) $(SOURCES_C) \ > copy_to_local.ml \ > copy_to_local.mli \ > + embed-code.sh \ > + rhv-upload-createvm.py \ > + rhv-upload-plugin.py \ > + rhv-upload-precheck.py \ > v2v_slow_unit_tests.ml \ > v2v-slow-unit-tests.sh \ > v2v_unit_tests.ml \ > @@ -64,6 +71,10 @@ SOURCES_MLI = \ > output_null.mli \ > output_qemu.mli \ > output_rhv.mli \ > + output_rhv_upload.mli \ > + output_rhv_upload_createvm_source.mli \ > + output_rhv_upload_plugin_source.mli \ > + output_rhv_upload_precheck_source.mli \ > output_vdsm.mli \ > parse_ovf_from_ova.mli \ > parse_libvirt_xml.mli \ > @@ -116,6 +127,10 @@ SOURCES_ML = \ > output_local.ml \ > output_qemu.ml \ > output_rhv.ml \ > + output_rhv_upload_createvm_source.ml \ > + output_rhv_upload_plugin_source.ml \ > + output_rhv_upload_precheck_source.ml \ > + output_rhv_upload.ml \ > output_vdsm.ml \ > inspect_source.ml \ > target_bus_assignment.ml \ > @@ -126,6 +141,15 @@ SOURCES_C = \ > libvirt_utils-c.c \ > qemuopts-c.c > > +# These files are generated and contain rhv-upload-*.py embedded as an > +# OCaml string. > +output_rhv_upload_createvm_source.ml: rhv-upload-createvm.py > + ./embed.sh code $^ $@ > +output_rhv_upload_plugin_source.ml: rhv-upload-plugin.py > + ./embed.sh code $^ $@ > +output_rhv_upload_precheck_source.ml: rhv-upload-precheck.py > + ./embed.sh code $^ $@ > + > if HAVE_OCAML > > bin_PROGRAMS = virt-v2v virt-v2v-copy-to-local > @@ -295,6 +319,7 @@ TESTS_ENVIRONMENT = $(top_builddir)/run --test > > TESTS = \ > test-v2v-docs.sh \ > + test-v2v-python-syntax.sh \ > test-v2v-i-ova-bad-sha1.sh \ > test-v2v-i-ova-bad-sha256.sh \ > test-v2v-i-ova-formats.sh \ > @@ -307,6 +332,7 @@ TESTS = \ > test-v2v-i-ova-two-disks.sh \ > test-v2v-i-vmx.sh \ > test-v2v-it-vddk-io-query.sh \ > + test-v2v-o-rhv-upload-oo-query.sh \ > test-v2v-o-vdsm-oo-query.sh \ > test-v2v-bad-networks-and-bridges.sh > > @@ -485,6 +511,7 @@ EXTRA_DIST += \ > test-v2v-o-qemu.sh \ > test-v2v-o-rhv.ovf.expected \ > test-v2v-o-rhv.sh \ > + test-v2v-o-rhv-upload-oo-query.sh \ > test-v2v-o-vdsm-oo-query.sh \ > test-v2v-o-vdsm-options.ovf.expected \ > test-v2v-o-vdsm-options.sh \ > @@ -494,6 +521,7 @@ EXTRA_DIST += \ > test-v2v-print-source.expected \ > test-v2v-print-source.sh \ > test-v2v-print-source.xml \ > + test-v2v-python-syntax.sh \ > test-v2v-conversion-of.sh \ > test-v2v-sound.sh \ > test-v2v-sound.xml \ > diff --git a/v2v/cmdline.ml b/v2v/cmdline.ml > index b36ec9c5e..785e91ef9 100644 > --- a/v2v/cmdline.ml > +++ b/v2v/cmdline.ml > @@ -135,6 +135,8 @@ let parse_cmdline () = > | "disk" | "local" -> output_mode := `Local > | "null" -> output_mode := `Null > | "ovirt" | "rhv" | "rhev" -> output_mode := `RHV > + | "ovirt-upload" | "ovirt_upload" | "rhv-upload" | "rhv_upload" -> > + output_mode := `RHV_Upload > | "qemu" -> output_mode := `QEmu > | "vdsm" -> output_mode := `VDSM > | s -> > @@ -397,6 +399,16 @@ read the man page virt-v2v(1). > | `Null -> no_options (); `Null > | `RHV -> no_options (); `RHV > | `QEmu -> no_options (); `QEmu > + | `RHV_Upload -> > + if is_query then ( > + Output_rhv_upload.print_output_options (); > + exit 0 > + ) > + else ( > + let rhv_options = > + Output_rhv_upload.parse_output_options output_options in > + `RHV_Upload rhv_options > + ) > | `VDSM -> > if is_query then ( > Output_vdsm.print_output_options (); > @@ -579,6 +591,35 @@ read the man page virt-v2v(1). > Output_rhv.output_rhv os output_alloc, > output_format, output_alloc > > + | `RHV_Upload rhv_options -> > + let output_conn = > + match output_conn with > + | None -> > + error (f_"-o rhv-upload: use ‘-oc’ to point to the oVirt or RHV > server REST API URL, which is usually https://servername/ovirt-engine/api";) > + | Some oc -> oc in > + (* Output format / sparse must currently be raw+sparse. We can > + * change this in future. See TODO file for details. XXX > + *) > + if output_alloc <> Sparse || output_format <> Some "raw" then > + error (f_"-o rhv-upload: currently you must use ‘-of raw’ and you > cannot use ‘-oa preallocated’ with this output mode. These restrictions will > be loosened in a future version."); > + (* In theory we could make the password optional in future. *) > + let output_password = > + match output_password with > + | None -> > + error (f_"-o rhv-upload: output password file was not specified, > use ‘-op’ to point to a file which contains the password used to connect to > the oVirt or RHV server") > + | Some op -> op in > + let os = > + match output_storage with > + | None -> > + error (f_"-o rhv-upload: output storage was not specified, use > ‘-os’"); > + | Some os -> os in > + if qemu_boot then > + error_option_cannot_be_used_in_output_mode "rhv-upload" > "--qemu-boot"; > + Output_rhv_upload.output_rhv_upload output_alloc output_conn > + output_password os > + rhv_options, > + output_format, output_alloc > + > | `VDSM vdsm_options -> > if output_password <> None then > error_option_cannot_be_used_in_output_mode "vdsm" "-op"; > diff --git a/v2v/embed.sh b/v2v/embed.sh > new file mode 100755 > index 000000000..0a65cd428 > --- /dev/null > +++ b/v2v/embed.sh > @@ -0,0 +1,45 @@ > +#!/bin/bash - > +# Embed code or other content into an OCaml file. > +# Copyright (C) 2018 Red Hat Inc. > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation; either version 2 of the License, or > +# (at your option) any later version. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License > +# along with this program; if not, write to the Free Software > +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 > USA. > + > +# Embed code or other content into an OCaml file. > +# > +# It is embedded into a string. As OCaml string literals have virtually > +# no restrictions on length or content we only have to escape double > +# quotes for backslash characters. > + > +if [ $# -ne 3 ]; then > + echo "embed.sh identifier input output" > + exit 1 > +fi > + > +ident="$1" > +input="$2" > +output="$3" > + > +rm -f "$output" "$output"-t > + > +exec >"$output"-t > + > +echo "(* Generated by embed.sh from $input *)" > +echo > +echo let "$ident" = '"' > +sed -e 's/\(["\]\)/\\\1/g' < "$input" > +echo '"' > + > +chmod -w "$output"-t > +mv "$output"-t "$output" > diff --git a/v2v/output_rhv_upload.ml b/v2v/output_rhv_upload.ml > new file mode 100644 > index 000000000..1be755a25 > --- /dev/null > +++ b/v2v/output_rhv_upload.ml > @@ -0,0 +1,390 @@ > +(* virt-v2v > + * Copyright (C) 2009-2018 Red Hat Inc. > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License as published by > + * the Free Software Foundation; either version 2 of the License, or > + * (at your option) any later version. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License along > + * with this program; if not, write to the Free Software Foundation, Inc., > + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. > + *) > + > +open Printf > +open Unix > + > +open Std_utils > +open Tools_utils > +open Unix_utils > +open Common_gettext.Gettext > + > +open Types > +open Utils > + > +type rhv_options = { > + rhv_cafile : string; > + rhv_cluster : string option; > + rhv_direct : bool; > + rhv_verifypeer : bool; > +} > + > +let print_output_options () = > + printf (f_"Output options (-oo) which can be used with -o rhv-upload: > + > + -oo rhv-cafile=CA.PEM Set ‘ca.pem’ certificate bundle filename. > + -oo rhv-cluster=CLUSTERNAME Set RHV cluster name. > + -oo rhv-direct[=true|false] Use direct transfer mode (default: false). > + -oo rhv-verifypeer[=true|false] Verify server identity (default: false). > +") > + > +let parse_output_options options = > + let rhv_cafile = ref None in > + let rhv_cluster = ref None in > + let rhv_direct = ref false in > + let rhv_verifypeer = ref false in > + > + List.iter ( > + function > + | "rhv-cafile", v -> > + if !rhv_cafile <> None then > + error (f_"-o rhv-upload: -oo rhv-cafile set twice"); > + rhv_cafile := Some v > + | "rhv-cluster", v -> > + if !rhv_cluster <> None then > + error (f_"-o rhv-upload: -oo rhv-cluster set twice"); > + rhv_cluster := Some v > + | "rhv-direct", "" -> rhv_direct := true > + | "rhv-direct", v -> rhv_direct := bool_of_string v > + | "rhv-verifypeer", "" -> rhv_verifypeer := true > + | "rhv-verifypeer", v -> rhv_verifypeer := bool_of_string v > + | k, _ -> > + error (f_"-o rhv-upload: unknown output option ‘-oo %s’") k > + ) options; > + > + let rhv_cafile = > + match !rhv_cafile with > + | Some s -> s > + | None -> > + error (f_"-o rhv-upload: must use ‘-oo rhv-cafile’ to supply the path > to the oVirt or RHV user’s ‘ca.pem’ file") in > + let rhv_cluster = !rhv_cluster in > + let rhv_direct = !rhv_direct in > + let rhv_verifypeer = !rhv_verifypeer in > + > + { rhv_cafile; rhv_cluster; rhv_direct; rhv_verifypeer } > + > +let python3 = "python3" (* Defined by PEP 394 *) > +let pidfile_timeout = 30 > +let finalization_timeout = 5*60 > + > +class output_rhv_upload output_alloc output_conn > + output_password output_storage > + rhv_options = > + (* Create a temporary directory which will be deleted on exit. *) > + let tmpdir = > + let base_dir = (open_guestfs ())#get_cachedir () in > + let t = Mkdtemp.temp_dir ~base_dir "rhvupload." in > + rmdir_on_exit t; > + t in > + > + let diskid_file_of_id id = tmpdir // sprintf "diskid.%d" id in > + > + (* Write the Python precheck, plugin and create VM to a temporary file. *) > + let precheck = > + let precheck = tmpdir // "rhv-upload-precheck.py" in > + with_open_out > + precheck > + (fun chan -> output_string chan > Output_rhv_upload_precheck_source.code); > + precheck in > + let plugin = > + let plugin = tmpdir // "rhv-upload-plugin.py" in > + with_open_out > + plugin > + (fun chan -> output_string chan Output_rhv_upload_plugin_source.code); > + plugin in > + let createvm = > + let createvm = tmpdir // "rhv-upload-createvm.py" in > + with_open_out > + createvm > + (fun chan -> output_string chan > Output_rhv_upload_createvm_source.code); > + createvm in > + > + (* Is SELinux enabled and enforcing on the host? *) > + let have_selinux = > + 0 = Sys.command "getenforce 2>/dev/null | grep -isq Enforcing" in > + > + (* Check that nbdkit is available and new enough. *) > + let error_unless_nbdkit_working () = > + if 0 <> Sys.command "nbdkit --version >/dev/null" then > + error (f_"nbdkit is not installed or not working. It is required to > use ‘-o rhv-upload’. See \"OUTPUT TO RHV\" in the virt-v2v(1) manual."); > + > + (* Check it's a new enough version. The latest features we > + * require are ‘--exit-with-parent’ and ‘--selinux-label’, both > + * added in 1.1.14. (We use 1.1.16 as the minimum here because > + * it also adds the selinux=yes|no flag in --dump-config). > + *) > + let lines = external_command "nbdkit --help" in > + let lines = String.concat " " lines in > + if String.find lines "exit-with-parent" == -1 || > + String.find lines "selinux-label" == -1 then > + error (f_"nbdkit is not new enough, you need to upgrade to nbdkit ≥ > 1.1.16") > + in > + > + (* Check that the python3 plugin is installed and working > + * and can load the plugin script. > + *) > + let error_unless_nbdkit_python3_working () = > + let cmd = sprintf "nbdkit %s %s --dump-plugin >/dev/null" > + python3 (quote plugin) in > + if Sys.command cmd <> 0 then > + error (f_"nbdkit Python 3 plugin is not installed or not working. It > is required if you want to use ‘-o rhv-upload’. > + > +See also \"OUTPUT TO RHV\" in the virt-v2v(1) manual.") > + in > + > + (* Check that nbdkit was compiled with SELinux support (for the > + * --selinux-label option). > + *) > + let error_unless_nbdkit_compiled_with_selinux () = > + let lines = external_command "nbdkit --dump-config" in > + (* In nbdkit <= 1.1.15 the selinux attribute was not present > + * at all in --dump-config output so there was no way to tell. > + * Ignore this case because there will be an error later when > + * we try to use the --selinux-label parameter. > + *) > + if List.mem "selinux=no" (List.map String.trim lines) then > + error (f_"nbdkit was compiled without SELinux support. You will have > to recompile nbdkit with libselinux-devel installed, or else set SELinux to > Permissive mode while doing the conversion.") > + in > + > + (* JSON parameters which are invariant between disks. *) > + let json_params = [ > + "output_conn", JSON.String output_conn; > + "output_password", JSON.String output_password; > + "output_storage", JSON.String output_storage; > + "output_sparse", JSON.Bool (match output_alloc with > + | Sparse -> true > + | Preallocated -> false); > + "rhv_cafile", JSON.String rhv_options.rhv_cafile; > + "rhv_cluster", > + JSON.String (Option.default "Default" rhv_options.rhv_cluster); > + "rhv_direct", JSON.Bool rhv_options.rhv_direct; > + > + (* The 'Insecure' flag seems to be a number with various possible > + * meanings, however we just set it to True/False. > + * > + * > https://github.com/oVirt/ovirt-engine-sdk/blob/19aa7070b80e60a4cfd910448287aecf9083acbe/sdk/lib/ovirtsdk4/__init__.py#L395 > + *) > + "insecure", JSON.Bool (not rhv_options.rhv_verifypeer); > + ] in > + > + (* nbdkit command line args which are invariant between disks. *) > + let nbdkit_args = > + let args = [ > + "nbdkit"; > + > + "--foreground"; (* run in foreground *) > + "--exit-with-parent"; (* exit when virt-v2v exits *) > + "--newstyle"; (* use newstyle NBD protocol *) > + "--exportname"; "/"; > + > + "python3"; (* use the nbdkit Python 3 plugin *) > + plugin; (* Python plugin script *) > + ] in > + let args = if verbose () then args @ ["--verbose"] else args in > + let args = > + (* label the socket so qemu can open it *) > + if have_selinux then > + args @ ["--selinux-label"; "system_u:object_r:svirt_t:s0"] > + else args in > + args in > + > +object > + inherit output > + > + method precheck () = > + error_unless_nbdkit_working (); > + error_unless_nbdkit_python3_working (); > + if have_selinux then > + error_unless_nbdkit_compiled_with_selinux () > + > + method as_options = > + "-o rhv-upload" ^ > + (match output_alloc with > + | Sparse -> "" (* default, don't need to print it *) > + | Preallocated -> " -oa preallocated") ^ > + sprintf " -oc %s -op %s -os %s" > + output_conn output_password output_storage > + > + method supported_firmware = [ TargetBIOS ] > + > + method prepare_targets source targets = > + let output_name = source.s_name in > + let json_params = > + ("output_name", JSON.String output_name) :: json_params in > + > + (* Python code prechecks. These can't run in #precheck because > + * we need to know the name of the virtual machine. > + *) > + let json_param_file = tmpdir // "params.json" in > + with_open_out > + json_param_file > + (fun chan -> output_string chan (JSON.string_of_doc json_params)); > + if run_command [ python3; precheck; json_param_file ] <> 0 then > + error (f_"failed server prechecks, see earlier errors"); > + > + (* Create an nbdkit instance for each disk and set the > + * target URI to point to the NBD socket. > + *) > + List.map ( > + fun t -> > + let id = t.target_overlay.ov_source.s_disk_id in > + let disk_name = sprintf "%s-%03d" output_name id in > + let json_params = > + ("disk_name", JSON.String disk_name) :: json_params in > + > + let disk_format = > + match t.target_format with > + | ("raw" | "qcow2") as fmt -> fmt > + | _ -> > + error (f_"rhv-upload: -of %s: Only output format ‘raw’ or > ‘qcow2’ is supported. If the input is in a different format then force one > of these output formats by adding either ‘-of raw’ or ‘-of qcow2’ on the > command line.") > + t.target_format in > + let json_params = > + ("disk_format", JSON.String disk_format) :: json_params in > + > + let disk_size = t.target_overlay.ov_virtual_size in > + let json_params = > + ("disk_size", JSON.Int64 disk_size) :: json_params in > + > + (* Ask the plugin to write the disk ID to a special file. *) > + let diskid_file = diskid_file_of_id id in > + let json_params = > + ("diskid_file", JSON.String diskid_file) :: json_params in > + > + (* Write the JSON parameters to a file. *) > + let json_param_file = tmpdir // sprintf "params%d.json" id in > + with_open_out > + json_param_file > + (fun chan -> output_string chan (JSON.string_of_doc json_params)); > + > + let sock = tmpdir // sprintf "nbdkit%d.sock" id in > + let pidfile = tmpdir // sprintf "nbdkit%d.pid" id in > + > + (* Add common arguments to per-target arguments. *) > + let args = > + nbdkit_args @ [ "--pidfile"; pidfile; > + "--unix"; sock; > + sprintf "params=%s" json_param_file ] in > + > + (* Print the full command we are about to run when debugging. *) > + if verbose () then ( > + eprintf "running nbdkit:\n"; > + List.iter (fun arg -> eprintf " %s" (quote arg)) args; > + prerr_newline () > + ); > + > + (* Start an nbdkit instance in the background. By using > + * --exit-with-parent we don't have to worry about clean-up. > + *) > + let args = Array.of_list args in > + let pid = fork () in > + if pid = 0 then ( > + (* Child process (nbdkit). *) > + execvp "nbdkit" args > + ); > + > + (* Wait for the pidfile to appear so we know that nbdkit > + * is listening for requests. > + *) > + if not (wait_for_file pidfile pidfile_timeout) then ( > + if verbose () then > + error (f_"nbdkit did not start up. See previous debugging > messages for problems.") > + else > + error (f_"nbdkit did not start up. There may be errors printed > by nbdkit above. > + > +If the messages above are not sufficient to diagnose the problem then add > the ‘virt-v2v -v -x’ options and examine the debugging output carefully.") > + ); > + > + if have_selinux then ( > + (* Note that Unix domain sockets have both a file label and > + * a socket/process label. Using --selinux-label above > + * only set the socket label, but we must also set the file > + * label. > + *) > + ignore ( > + run_command ["chcon"; "system_u:object_r:svirt_image_t:s0"; > + sock] > + ); > + ); > + (* ... and the regular Unix permissions, in case qemu is > + * running as another user. > + *) > + chmod sock 0o777; > + > + (* Tell ‘qemu-img convert’ to write to the nbd socket which is > + * connected to nbdkit. > + *) > + let json_params = [ > + "file.driver", JSON.String "nbd"; > + "file.path", JSON.String sock; > + "file.export", JSON.String "/"; > + ] in > + let target_file = > + TargetURI ("json:" ^ JSON.string_of_doc json_params) in > + { t with target_file } > + ) targets > + > + method create_metadata source targets _ guestcaps inspect target_firmware = > + (* Get the UUIDs of each disk image. These files are written > + * out by the nbdkit plugins on successful finalization of the > + * transfer. > + *) > + let nr_disks = List.length targets in > + let image_uuids = > + List.map ( > + fun t -> > + let id = t.target_overlay.ov_source.s_disk_id in > + let diskid_file = diskid_file_of_id id in > + if not (wait_for_file diskid_file finalization_timeout) then > + error (f_"transfer of disk %d/%d failed, see earlier error > messages") > + (id+1) nr_disks; > + let diskid = read_whole_file diskid_file in > + diskid > + ) targets in > + > + (* We don't have the storage domain UUID, but instead we write > + * in a magic value which the Python code (which can get it) > + * will substitute. > + *) > + let sd_uuid = "@SD_UUID@" in > + > + (* The volume and VM UUIDs are made up. *) > + let vol_uuids = List.map (fun _ -> uuidgen ()) targets > + and vm_uuid = uuidgen () in > + > + (* Create the metadata. *) > + let ovf = > + Create_ovf.create_ovf source targets guestcaps inspect > + output_alloc > + sd_uuid image_uuids vol_uuids vm_uuid > + OVirt in > + let ovf = DOM.doc_to_string ovf in > + > + let json_param_file = tmpdir // "params.json" in > + with_open_out > + json_param_file > + (fun chan -> output_string chan (JSON.string_of_doc json_params)); > + > + let ovf_file = tmpdir // "vm.ovf" in > + with_open_out ovf_file (fun chan -> output_string chan ovf); > + if run_command [ python3; createvm; json_param_file; ovf_file ] <> 0 then > + error (f_"failed to create virtual machine, see earlier errors") > + > +end > + > +let output_rhv_upload = new output_rhv_upload > +let () = Modules_list.register_output_module "rhv-upload" > diff --git a/v2v/output_rhv_upload.mli b/v2v/output_rhv_upload.mli > new file mode 100644 > index 000000000..f6cd69a61 > --- /dev/null > +++ b/v2v/output_rhv_upload.mli > @@ -0,0 +1,33 @@ > +(* virt-v2v > + * Copyright (C) 2009-2018 Red Hat Inc. > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License as published by > + * the Free Software Foundation; either version 2 of the License, or > + * (at your option) any later version. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License along > + * with this program; if not, write to the Free Software Foundation, Inc., > + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. > + *) > + > +(** [-o rhv-upload] target. *) > + > +type rhv_options > +(** Miscellaneous extra command line parameters used by rhv-upload. *) > + > +val print_output_options : unit -> unit > +val parse_output_options : (string * string) list -> rhv_options > +(** Print and parse rhv-upload -oo options. *) > + > +val output_rhv_upload : Types.output_allocation -> string -> string -> > + string -> rhv_options -> Types.output > +(** [output_rhv_upload output_alloc output_conn output_password > output_storage > + rhv_options] > + creates and returns a new {!Types.output} object specialized for writing > + output to oVirt or RHV directly via RHV APIs. *) > diff --git a/v2v/output_rhv_upload_createvm_source.mli > b/v2v/output_rhv_upload_createvm_source.mli > new file mode 100644 > index 000000000..c1bafa15b > --- /dev/null > +++ b/v2v/output_rhv_upload_createvm_source.mli > @@ -0,0 +1,19 @@ > +(* virt-v2v > + * Copyright (C) 2018 Red Hat Inc. > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License as published by > + * the Free Software Foundation; either version 2 of the License, or > + * (at your option) any later version. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License along > + * with this program; if not, write to the Free Software Foundation, Inc., > + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. > + *) > + > +val code : string > diff --git a/v2v/output_rhv_upload_plugin_source.mli > b/v2v/output_rhv_upload_plugin_source.mli > new file mode 100644 > index 000000000..c1bafa15b > --- /dev/null > +++ b/v2v/output_rhv_upload_plugin_source.mli > @@ -0,0 +1,19 @@ > +(* virt-v2v > + * Copyright (C) 2018 Red Hat Inc. > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License as published by > + * the Free Software Foundation; either version 2 of the License, or > + * (at your option) any later version. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License along > + * with this program; if not, write to the Free Software Foundation, Inc., > + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. > + *) > + > +val code : string > diff --git a/v2v/output_rhv_upload_precheck_source.mli > b/v2v/output_rhv_upload_precheck_source.mli > new file mode 100644 > index 000000000..c1bafa15b > --- /dev/null > +++ b/v2v/output_rhv_upload_precheck_source.mli > @@ -0,0 +1,19 @@ > +(* virt-v2v > + * Copyright (C) 2018 Red Hat Inc. > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License as published by > + * the Free Software Foundation; either version 2 of the License, or > + * (at your option) any later version. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License along > + * with this program; if not, write to the Free Software Foundation, Inc., > + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. > + *) > + > +val code : string > diff --git a/v2v/rhv-upload-createvm.py b/v2v/rhv-upload-createvm.py > new file mode 100644 > index 000000000..a34627ec8 > --- /dev/null > +++ b/v2v/rhv-upload-createvm.py > @@ -0,0 +1,86 @@ > +# -*- python -*- > +# oVirt or RHV upload create VM used by ‘virt-v2v -o rhv-upload’ > +# Copyright (C) 2018 Red Hat Inc. > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation; either version 2 of the License, or > +# (at your option) any later version. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License along > +# with this program; if not, write to the Free Software Foundation, Inc., > +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. > + > +import json > +import logging > +import sys > +import time > + > +from http.client import HTTPSConnection > +from urllib.parse import urlparse > + > +import ovirtsdk4 as sdk > +import ovirtsdk4.types as types > + > +# Parameters are passed in via a JSON doc from the OCaml code. > +# Because this Python code ships embedded inside virt-v2v there > +# is no formal API here. > +params = None > +ovf = None # OVF file > + > +if len(sys.argv) != 3: > + raise RuntimeError("incorrect number of parameters") > + > +# Parameters are passed in via a JSON document. > +with open(sys.argv[1], 'r') as fp: > + params = json.load(fp) > + > +# What is passed in is a password file, read the actual password. > +with open(params['output_password'], 'r') as fp: > + output_password = fp.read() > +output_password = output_password.rstrip() > + > +# Read the OVF document. > +with open(sys.argv[2], 'r') as fp: > + ovf = fp.read() > + > +# Parse out the username from the output_conn URL. > +parsed = urlparse(params['output_conn']) > +username = parsed.username or "admin@internal" > + > +# Connect to the server. > +connection = sdk.Connection( > + url = params['output_conn'], > + username = username, > + password = output_password, > + ca_file = params['rhv_cafile'], > + log = logging.getLogger(), > + insecure = params['insecure'], > +) > + > +system_service = connection.system_service() > + > +# Get the storage domain UUID and substitute it into the OVF doc. > +sds_service = system_service.storage_domains_service() > +sd = sds_service.list(search=("name=%s" % params['output_storage']))[0] > +sd_uuid = sd.id > + > +ovf.replace("@SD_UUID@", sd_uuid) > + > +vms_service = system_service.vms_service() > +vm = vms_service.add( > + types.Vm( > + cluster=types.Cluster(name = params['rhv_cluster']), > + initialization=types.Initialization( > + configuration = types.Configuration( > + type = types.ConfigurationType.OVA, > + data = ovf, > + ) > + ) > + ) > +) > diff --git a/v2v/rhv-upload-plugin.py b/v2v/rhv-upload-plugin.py > new file mode 100644 > index 000000000..badfe63d8 > --- /dev/null > +++ b/v2v/rhv-upload-plugin.py > @@ -0,0 +1,438 @@ > +# -*- python -*- > +# oVirt or RHV upload nbdkit plugin used by ‘virt-v2v -o rhv-upload’ > +# Copyright (C) 2018 Red Hat Inc. > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation; either version 2 of the License, or > +# (at your option) any later version. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License along > +# with this program; if not, write to the Free Software Foundation, Inc., > +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. > + > +import builtins > +import json > +import logging > +import ssl > +import sys > +import time > + > +from http.client import HTTPSConnection > +from urllib.parse import urlparse > + > +import ovirtsdk4 as sdk > +import ovirtsdk4.types as types > + > +# Timeout to wait for oVirt disks to change status, or the transfer > +# object to finish initializing [seconds]. > +timeout = 5*60 > + > +# Parameters are passed in via a JSON doc from the OCaml code. > +# Because this Python code ships embedded inside virt-v2v there > +# is no formal API here. > +params = None > + > +def config(key, value): > + global params > + > + if key == "params": > + with builtins.open(value, 'r') as fp: > + params = json.load(fp) > + else: > + raise RuntimeError("unknown configuration key '%s'" % key) > + > +def config_complete(): > + if params is None: > + raise RuntimeError("missing configuration parameters") > + > +def open(readonly): > + # Parse out the username from the output_conn URL. > + parsed = urlparse(params['output_conn']) > + username = parsed.username or "admin@internal" > + > + # Read the password from file. > + with builtins.open(params['output_password'], 'r') as fp: > + password = fp.read() > + password = password.rstrip() > + > + # Connect to the server. > + connection = sdk.Connection( > + url = params['output_conn'], > + username = username, > + password = password, > + ca_file = params['rhv_cafile'], > + log = logging.getLogger(), > + insecure = params['insecure'], > + ) > + > + system_service = connection.system_service() > + > + # Create the disk. > + disks_service = system_service.disks_service() > + if params['disk_format'] == "raw": > + disk_format = types.DiskFormat.RAW > + else: > + disk_format = types.DiskFormat.COW > + disk = disks_service.add( > + disk = types.Disk( > + name = params['disk_name'], > + description = "Uploaded by virt-v2v", > + format = disk_format, > + initial_size = params['disk_size'], > + provisioned_size = params['disk_size'], > + # XXX Ignores params['output_sparse']. > + # Handling this properly will be complex, see: > + # > https://www.redhat.com/archives/libguestfs/2018-March/msg00177.html > + sparse = True, > + storage_domains = [ > + types.StorageDomain( > + name = params['output_storage'], > + ) > + ], > + ) > + ) > + > + # Wait till the disk is up, as the transfer can't start if the > + # disk is locked: > + disk_service = disks_service.disk_service(disk.id) > + > + endt = time.time() + timeout > + while True: > + time.sleep(5) > + disk = disk_service.get() > + if disk.status == types.DiskStatus.OK: > + break > + if time.time() > endt: > + raise RuntimeError("timed out waiting for disk to become > unlocked") > + > + # Get a reference to the transfer service. > + transfers_service = system_service.image_transfers_service() > + > + # Create a new image transfer. > + transfer = transfers_service.add( > + types.ImageTransfer( > + image = types.Image( > + id = disk.id > + ) > + ) > + ) I'm pretty sure I already mentioned it before. It would help debugging if we could dump the `disk.id` to the output here so it shows in virt-v2v debug logs. > + # Get a reference to the created transfer service. > + transfer_service = transfers_service.image_transfer_service(transfer.id) Dumping `transfer.id` could also be usefull... maybe? Don't know. If this shows in imageio dameon logs or not... > + > + # After adding a new transfer for the disk, the transfer's status > + # will be INITIALIZING. Wait until the init phase is over. The > + # actual transfer can start when its status is "Transferring". > + endt = time.time() + timeout > + while True: > + time.sleep(5) > + transfer = transfer_service.get() > + if transfer.phase != types.ImageTransferPhase.INITIALIZING: > + break > + if time.time() > endt: > + raise RuntimeError("timed out waiting for transfer status " + > + "!= INITIALIZING") > + > + # Now we have permission to start the transfer. > + if params['rhv_direct']: > + if transfer.transfer_url is None: > + raise RuntimeError("direct upload to host not supported, " + > + "requires ovirt-engine >= 4.2 and only works > " + > + "when virt-v2v is run within the oVirt/RHV " + > + "environment, eg. on an ovirt node.") spelling issue: oVirt instead of ovirt Tomas > + destination_url = urlparse(transfer.transfer_url) > + else: > + destination_url = urlparse(transfer.proxy_url) > + > + context = ssl.create_default_context() > + context.load_verify_locations(cafile = params['rhv_cafile']) > + > + http = HTTPSConnection( > + destination_url.hostname, > + destination_url.port, > + context = context > + ) > + > + # Save everything we need to make requests in the handle. > + return { > + 'can_flush': False, > + 'can_trim': False, > + 'can_zero': False, > + 'connection': connection, > + 'disk': disk, > + 'disk_service': disk_service, > + 'failed': False, > + 'got_options': False, > + 'highestwrite': 0, > + 'http': http, > + 'needs_auth': not params['rhv_direct'], > + 'path': destination_url.path, > + 'transfer': transfer, > + 'transfer_service': transfer_service, > + } > + > +# Can we issue zero, trim or flush requests? > +def get_options(h): > + if h['got_options']: > + return > + h['got_options'] = True > + > + http = h['http'] > + transfer = h['transfer'] > + > + http.putrequest("OPTIONS", h['path']) > + http.putheader("Authorization", transfer.signed_ticket) > + http.endheaders() > + > + r = http.getresponse() > + if r.status == 200: > + # New imageio never needs authentication. > + h['needs_auth'] = False > + > + j = json.loads(r.read()) > + h['can_zero'] = "zero" in j['features'] > + h['can_trim'] = "trim" in j['features'] > + h['can_flush'] = "flush" in j['features'] > + > + # Old imageio servers returned either 405 Method Not Allowed or > + # 204 No Content (with an empty body). If we see that we leave > + # all the features as False and they will be emulated. > + elif r.status == 405 or r.status == 204: > + pass > + > + else: > + raise RuntimeError("could not use OPTIONS request: %d: %s" % > + (r.status, r.reason)) > + > +def can_trim(h): > + get_options(h) > + return h['can_trim'] > + > +def can_flush(h): > + get_options(h) > + return h['can_flush'] > + > +def get_size(h): > + return params['disk_size'] > + > +# For documentation see: > +# https://github.com/oVirt/ovirt-imageio/blob/master/docs/random-io.md > +# For examples of working code to read/write from the server, see: > +# > https://github.com/oVirt/ovirt-imageio/blob/master/daemon/test/server_test.py > + > +def pread(h, count, offset): > + http = h['http'] > + transfer = h['transfer'] > + transfer_service = h['transfer_service'] > + > + http.putrequest("GET", h['path']) > + # Authorization is only needed for old imageio. > + if h['needs_auth']: > + http.putheader("Authorization", transfer.signed_ticket) > + http.putheader("Range", "bytes=%d-%d" % (offset, offset+count-1)) > + http.endheaders() > + > + r = http.getresponse() > + # 206 = HTTP Partial Content. > + if r.status != 206: > + h['transfer_service'].pause() > + h['failed'] = True > + raise RuntimeError("could not read sector (%d, %d): %d: %s" % > + (offset, count, r.status, r.reason)) > + return r.read() > + > +def pwrite(h, buf, offset): > + http = h['http'] > + transfer = h['transfer'] > + transfer_service = h['transfer_service'] > + > + count = len(buf) > + h['highestwrite'] = max(h['highestwrite'], offset+count) > + > + http.putrequest("PUT", h['path'] + "?flush=n") > + # Authorization is only needed for old imageio. > + if h['needs_auth']: > + http.putheader("Authorization", transfer.signed_ticket) > + # The oVirt server only uses the first part of the range, and the > + # content-length. > + http.putheader("Content-Range", "bytes %d-%d/*" % (offset, > offset+count-1)) > + http.putheader("Content-Length", str(count)) > + http.endheaders() > + http.send(buf) > + > + r = http.getresponse() > + if r.status != 200: > + transfer_service.pause() > + h['failed'] = True > + raise RuntimeError("could not write sector (%d, %d): %d: %s" % > + (offset, count, r.status, r.reason)) > + > +def zero(h, count, offset, may_trim): > + http = h['http'] > + transfer = h['transfer'] > + transfer_service = h['transfer_service'] > + > + # Unlike the trim and flush calls, there is no 'can_zero' method > + # so nbdkit could call this even if the server doesn't support > + # zeroing. If this is the case we must emulate. > + if not h['can_zero']: > + emulate_zero(h, count, offset) > + return > + > + # Construct the JSON request for zeroing. > + buf = json.dumps({'op': "zero", > + 'offset': offset, > + 'size': count, > + 'flush': False}).encode() > + > + http.putrequest("PATCH", h['path']) > + http.putheader("Content-Type", "application/json") > + http.putheader("Content-Length", len(buf)) > + http.endheaders() > + http.send(buf) > + > + r = http.getresponse() > + if r.status != 200: > + transfer_service.pause() > + h['failed'] = True > + raise RuntimeError("could not zero sector (%d, %d): %d: %s" % > + (offset, count, r.status, r.reason)) > + > +def emulate_zero(h, count, offset): > + # qemu-img convert starts by trying to zero/trim the whole device. > + # Since we've just created a new disk it's safe to ignore these > + # requests as long as they are smaller than the highest write seen. > + # After that we must emulate them with writes. > + if offset+count < h['highestwrite']: > + http.putrequest("PUT", h['path']) > + # Authorization is only needed for old imageio. > + if h['needs_auth']: > + http.putheader("Authorization", transfer.signed_ticket) > + http.putheader("Content-Range", > + "bytes %d-%d/*" % (offset, offset+count-1)) > + http.putheader("Content-Length", str(count)) > + http.endheaders() > + > + buf = bytearray(128*1024) > + while count > len(buf): > + http.send(buf) > + count -= len(buf) > + http.send(buffer(buf, 0, count)) > + > + r = http.getresponse() > + if r.status != 200: > + transfer_service.pause() > + h['failed'] = True > + raise RuntimeError("could not write zeroes (%d, %d): %d: %s" % > + (offset, count, r.status, r.reason)) > + > +def trim(h, count, offset): > + http = h['http'] > + transfer = h['transfer'] > + transfer_service = h['transfer_service'] > + > + # Construct the JSON request for trimming. > + buf = json.dumps({'op': "trim", > + 'offset': offset, > + 'size': count, > + 'flush': False}).encode() > + > + http.putrequest("PATCH", h['path']) > + http.putheader("Content-Type", "application/json") > + http.putheader("Content-Length", len(buf)) > + http.endheaders() > + http.send(buf) > + > + r = http.getresponse() > + if r.status != 200: > + transfer_service.pause() > + h['failed'] = True > + raise RuntimeError("could not trim sector (%d, %d): %d: %s" % > + (offset, count, r.status, r.reason)) > + > +def flush(h): > + http = h['http'] > + transfer = h['transfer'] > + transfer_service = h['transfer_service'] > + > + # Construct the JSON request for flushing. > + buf = json.dumps({'op': "flush"}).encode() > + > + http.putrequest("PATCH", h['path']) > + http.putheader("Content-Type", "application/json") > + http.putheader("Content-Length", len(buf)) > + http.endheaders() > + http.send(buf) > + > + r = http.getresponse() > + if r.status != 200: > + transfer_service.pause() > + h['failed'] = True > + raise RuntimeError("could not flush: %d: %s" % (r.status, r.reason)) > + > +def delete_disk_on_failure(h): > + disk_service = h['disk_service'] > + disk_service.remove() > + > +def close(h): > + http = h['http'] > + connection = h['connection'] > + > + # This is sometimes necessary because python doesn't set up > + # sys.stderr to be line buffered and so debug, errors or > + # exceptions printed previously might not be emitted before the > + # plugin exits. > + sys.stderr.flush() > + > + # If the connection failed earlier ensure we clean up the disk. > + if h['failed']: > + delete_disk_on_failure(h) > + connection.close() > + return > + > + try: > + # Issue a flush request on close so that the data is written to > + # persistent store before we create the VM. > + if h['can_flush']: > + flush(h) > + > + http.close() > + > + disk = h['disk'] > + transfer_service = h['transfer_service'] > + > + transfer_service.finalize() > + > + # Wait until the transfer disk job is completed since > + # only then we can be sure the disk is unlocked. As this > + # code is not very clear, what's happening is that we are > + # waiting for the transfer object to cease to exist, which > + # falls through to the exception case and then we can > + # continue. > + endt = time.time() + timeout > + try: > + while True: > + time.sleep(1) > + tmp = transfer_service.get() > + if time.time() > endt: > + raise RuntimeError("timed out waiting for transfer " + > + "to finalize") > + except sdk.NotFoundError: > + pass > + > + # Write the disk ID file. Only do this on successful completion. > + with builtins.open(params['diskid_file'], 'w') as fp: > + fp.write(disk.id) > + > + except: > + # Otherwise on any failure we must clean up the disk. > + delete_disk_on_failure(h) > + raise > + > + connection.close() > diff --git a/v2v/rhv-upload-precheck.py b/v2v/rhv-upload-precheck.py > new file mode 100644 > index 000000000..2798a29dd > --- /dev/null > +++ b/v2v/rhv-upload-precheck.py > @@ -0,0 +1,73 @@ > +# -*- python -*- > +# oVirt or RHV pre-upload checks used by ‘virt-v2v -o rhv-upload’ > +# Copyright (C) 2018 Red Hat Inc. > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation; either version 2 of the License, or > +# (at your option) any later version. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License along > +# with this program; if not, write to the Free Software Foundation, Inc., > +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. > + > +import json > +import logging > +import sys > +import time > + > +from http.client import HTTPSConnection > +from urllib.parse import urlparse > + > +import ovirtsdk4 as sdk > +import ovirtsdk4.types as types > + > +# Parameters are passed in via a JSON doc from the OCaml code. > +# Because this Python code ships embedded inside virt-v2v there > +# is no formal API here. > +params = None > + > +if len(sys.argv) != 2: > + raise RuntimeError("incorrect number of parameters") > + > +# Parameters are passed in via a JSON document. > +with open(sys.argv[1], 'r') as fp: > + params = json.load(fp) > + > +# What is passed in is a password file, read the actual password. > +with open(params['output_password'], 'r') as fp: > + output_password = fp.read() > +output_password = output_password.rstrip() > + > +# Parse out the username from the output_conn URL. > +parsed = urlparse(params['output_conn']) > +username = parsed.username or "admin@internal" > + > +# Connect to the server. > +connection = sdk.Connection( > + url = params['output_conn'], > + username = username, > + password = output_password, > + ca_file = params['rhv_cafile'], > + log = logging.getLogger(), > + insecure = params['insecure'], > +) > + > +system_service = connection.system_service() > + > +# Find if a virtual machine already exists with that name. > +vms_service = system_service.vms_service() > +vms = vms_service.list( > + search = ("name=%s" % params['output_name']), > +) > +if len(vms) > 0: > + vm = vms[0] > + raise RuntimeError("VM already exists with name ‘%s’, id ‘%s’" % > + (params['output_name'], vm.id)) > + > +# Otherwise everything is OK, exit with no error. > diff --git a/v2v/test-v2v-o-rhv-upload-oo-query.sh > b/v2v/test-v2v-o-rhv-upload-oo-query.sh > new file mode 100755 > index 000000000..29d69e1c1 > --- /dev/null > +++ b/v2v/test-v2v-o-rhv-upload-oo-query.sh > @@ -0,0 +1,38 @@ > +#!/bin/bash - > +# libguestfs virt-v2v test script > +# Copyright (C) 2018 Red Hat Inc. > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation; either version 2 of the License, or > +# (at your option) any later version. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License > +# along with this program; if not, write to the Free Software > +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 > USA. > + > +# Test -oo "?" option. > + > +set -e > + > +$TEST_FUNCTIONS > +skip_if_skipped > + > +export VIRT_TOOLS_DATA_DIR="$top_srcdir/test-data/fake-virt-tools" > +export VIRTIO_WIN="$top_srcdir/test-data/fake-virtio-win" > + > +f=test-v2v-o-rhv-upload-oo-query.actual > +rm -f $f > + > +$VG virt-v2v --debug-gc \ > + -o rhv-upload -oo "?" > $f > + > +grep -- "-oo rhv-cafile" $f > +grep -- "-oo rhv-verifypeer" $f > + > +rm $f > diff --git a/v2v/test-v2v-python-syntax.sh b/v2v/test-v2v-python-syntax.sh > new file mode 100755 > index 000000000..b167f4610 > --- /dev/null > +++ b/v2v/test-v2v-python-syntax.sh > @@ -0,0 +1,45 @@ > +#!/bin/bash - > +# libguestfs > +# Copyright (C) 2018 Red Hat Inc. > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation; either version 2 of the License, or > +# (at your option) any later version. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License > +# along with this program; if not, write to the Free Software > +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 > USA. > + > +set -e > + > +$TEST_FUNCTIONS > +skip_if_skipped > + > +# Files to check. > +files="rhv-upload-createvm.py rhv-upload-plugin.py rhv-upload-precheck.py" > + > +# Base version of Python. > +python=python3 > + > +# Checks the files are syntactically correct, but not very much else. > +for f in $files; do > + $python -m py_compile $f > +done > + > +# Checks the files correspond to PEP8 coding style. > +# https://www.python.org/dev/peps/pep-0008/ > +if $python-pep8 --version >/dev/null 2>&1; then > + for f in $files; do > + # Ignore: > + # E226 missing whitespace around arithmetic operator > + # E251 unexpected spaces around keyword / parameter equals > + # E302 expected 2 blank lines, found 1 > + $python-pep8 --ignore=E226,E251,E302 $f > + done > +fi > diff --git a/v2v/virt-v2v.pod b/v2v/virt-v2v.pod > index 5102e6841..4eab5a55d 100644 > --- a/v2v/virt-v2v.pod > +++ b/v2v/virt-v2v.pod > @@ -6,15 +6,18 @@ virt-v2v - Convert a guest to use KVM > > virt-v2v -ic vpx://vcenter.example.com/Datacenter/esxi vmware_guest > > - virt-v2v -ic vpx://vcenter.example.com/Datacenter/esxi vmware_guest \ > - -o rhv -os rhv.nfs:/export_domain --bridge ovirtmgmt > - > virt-v2v -i libvirtxml guest-domain.xml -o local -os /var/tmp > > virt-v2v -i disk disk.img -o local -os /var/tmp > > virt-v2v -i disk disk.img -o glance > > + virt-v2v -ic vpx://vcenter.example.com/Datacenter/esxi vmware_guest \ > + -o rhv-upload -oc https://ovirt-engine.example.com/ovirt-engine/api \ > + -os ovirt-data -op /tmp/ovirt-admin-password -of raw \ > + -oo rhv-cafile=/tmp/ca.pem -oo rhv-direct \ > + --bridge ovirtmgmt > + > virt-v2v -ic qemu:///system qemu_guest --in-place > > =head1 DESCRIPTION > @@ -52,20 +55,18 @@ For more information see L</INPUT FROM VMWARE VCENTER > SERVER> below. > =head2 Convert from VMware to RHV/oVirt > > This is the same as the previous example, except you want to send the > -guest to a RHV-M Export Storage Domain which is located remotely > -(over NFS) at C<rhv.nfs:/export_domain>. If you are unclear about > -the location of the Export Storage Domain you should check the > -settings on your RHV-M management console. Guest network > +guest to a RHV Data Domain using the RHV REST API. Guest network > interface(s) are connected to the target network called C<ovirtmgmt>. > > virt-v2v -ic vpx://vcenter.example.com/Datacenter/esxi vmware_guest \ > - -o rhv -os rhv.nfs:/export_domain --bridge ovirtmgmt > + -o rhv-upload -oc https://ovirt-engine.example.com/ovirt-engine/api \ > + -os ovirt-data -op /tmp/ovirt-admin-password -of raw \ > + -oo rhv-cafile=/tmp/ca.pem -oo rhv-direct \ > + --bridge ovirtmgmt > > In this case the host running virt-v2v acts as a B<conversion server>. > > -Note that after conversion, the guest will appear in the RHV-M Export > -Storage Domain, from where you will need to import it using the RHV-M > -user interface. (See L</OUTPUT TO RHV>). > +For more information see L</OUTPUT TO RHV> below. > > =head2 Convert from ESXi hypervisor over SSH to local libvirt > > @@ -129,9 +130,9 @@ qemu, do: > Xen ───▶│ -i libvirt ──▶ │ │ │ (default) │ > ... ───▶│ (default) │ │ │ ──┐ └────────────┘ > └────────────┘ │ │ ─┐└──────▶ -o glance > - -i libvirtxml ─────────▶ │ │ ┐└─────────▶ -o rhv > - -i vmx ────────────────▶ │ │ └──────────▶ -o vdsm > - └────────────┘ > + -i libvirtxml ─────────▶ │ │ ┐├─────────▶ -o rhv > + -i vmx ────────────────▶ │ │ │└─────────▶ -o vdsm > + └────────────┘ └──────────▶ -o rhv-upload > > Virt-v2v has a number of possible input and output modes, selected > using the I<-i> and I<-o> options. Only one input and output mode can > @@ -164,8 +165,9 @@ libvirt configuration file (mainly for testing). > I<-o qemu> writes to a local disk image with a shell script for > booting the guest directly in qemu (mainly for testing). > > -I<-o rhv> is used to write to a RHV / oVirt target. I<-o vdsm> > -is only used when virt-v2v runs under VDSM control. > +I<-o rhv-upload> is used to write to a RHV / oVirt target. I<-o rhv> > +is a legacy method to write to RHV / oVirt E<lt> 4.2. I<-o vdsm> is > +only used when virt-v2v runs under VDSM control. > > I<--in-place> instructs virt-v2v to customize the guest OS in the input > virtual machine, instead of creating a new VM in the target hypervisor. > @@ -550,6 +552,10 @@ written. > > This is the same as I<-o rhv>. > > +=item B<-o> B<ovirt-upload> > + > +This is the same as I<-o rhv-upload>. > + > =item B<-o> B<qemu> > > Set the output method to I<qemu>. > @@ -574,6 +580,16 @@ I<-os> parameter must also be used to specify the > location of the > Export Storage Domain. Note this does not actually import the guest > into RHV. You have to do that manually later using the UI. > > +See L</OUTPUT TO RHV (OLD METHOD)> below. > + > +=item B<-o> B<rhv-upload> > + > +Set the output method to I<rhv-upload>. > + > +The converted guest is written directly to a RHV Data Domain. > +This is a faster method than I<-o rhv>, but requires oVirt > +or RHV E<ge> 4.2. > + > See L</OUTPUT TO RHV> below. > > =item B<-o> B<vdsm> > @@ -615,7 +631,33 @@ the output name is the same as the input name. > Set output option(s) related to the current output mode. > To display short help on what options are available you can use: > > - virt-v2v -o vdsm -oo "?" > + virt-v2v -o rhv-upload -oo "?" > + > +=item B<-oo rhv-cafile=>F<ca.pem> > + > +For I<-o rhv-upload> (L<OUTPUT TO RHV>) only, the F<ca.pem> file > +(Certificate Authority), copied from F</etc/pki/ovirt-engine/ca.pem> > +on the oVirt engine. > + > +=item B<-oo rhv-cluster=>C<CLUSTERNAME> > + > +For I<-o rhv-upload> (L<OUTPUT TO RHV>) only, set the RHV Cluster > +Name. If not given it uses C<Default>. > + > +=item B<-oo rhv-direct> > + > +For I<-o rhv-upload> (L<OUTPUT TO RHV>) only, if this option is given > +then virt-v2v will attempt to directly upload the disk to the oVirt > +node, otherwise it will proxy the upload through the oVirt engine. > +Direct upload requires that you have network access to the oVirt > +nodes. Non-direct upload is slightly slower but should work in all > +situations. > + > +=item B<-oo rhv-verifypeer> > + > +For I<-o rhv-upload> (L<OUTPUT TO RHV>) only, verify the oVirt/RHV > +server’s identity by checking the server‘s certificate against the > +Certificate Authority. > > =item B<-oo vdsm-compat=0.10> > > @@ -1884,6 +1926,68 @@ Define the final guest in libvirt: > > =head1 OUTPUT TO RHV > > +This new method to upload guests to oVirt or RHV directly via the REST > +API requires oVirt/RHV E<ge> 4.2. > + > +You need to specify I<-o rhv-upload> as well as the following extra > +parameters: > + > +=over 4 > + > +=item I<-oc> C<https://ovirt-engine.example.com/ovirt-engine/api> > + > +The URL of the REST API which is usually the server name with > +C</ovirt-engine/api> appended, but might be different if you installed > +oVirt Engine on a different path. > + > +You can optionally add a username and port number to the URL. If the > +username is not specified then virt-v2v defaults to using > +C<admin@internal> which is the typical superuser account for oVirt > +instances. > + > +=item I<-of raw> > + > +Currently you must use I<-of raw> and you cannot use I<-oa preallocated>. > + > +These restrictions will be loosened in a future version. > + > +=item I<-op> F<password-file> > + > +A file containing a password to be used when connecting to the oVirt > +engine. Note the file should contain the whole password, B<without > +any trailing newline>, and for security the file should have mode > +C<0600> so that others cannot read it. > + > +=item I<-os> C<ovirt-data> > + > +The storage domain. > + > +=item I<-oo rhv-cafile=>F<ca.pem> > + > +The F<ca.pem> file (Certificate Authority), copied from > +F</etc/pki/ovirt-engine/ca.pem> on the oVirt engine. > + > +=item I<-oo rhv-cluster=>C<CLUSTERNAME> > + > +Set the RHV Cluster Name. If not given it uses C<Default>. > + > +=item I<-oo rhv-direct> > + > +If this option is given then virt-v2v will attempt to directly upload > +the disk to the oVirt node, otherwise it will proxy the upload through > +the oVirt engine. Direct upload requires that you have network access > +to the oVirt nodes. Non-direct upload is slightly slower but should > +work in all situations. > + > +=item I<-oo rhv-verifypeer> > + > +Verify the oVirt/RHV server’s identity by checking the server‘s > +certificate against the Certificate Authority. > + > +=back > + > +=head1 OUTPUT TO RHV (OLD METHOD) > + > This section only applies to the I<-o rhv> output mode. If you use > virt-v2v from the RHV-M user interface, then behind the scenes the > import is managed by VDSM using the I<-o vdsm> output mode (which end > -- > 2.16.2 > -- Tomáš Golembiovský <[email protected]> _______________________________________________ Libguestfs mailing list [email protected] https://www.redhat.com/mailman/listinfo/libguestfs
