Hi,

few small comments below, but generally LGTM.

On Thu,  5 Apr 2018 14:26:07 +0100
"Richard W.M. Jones" <rjo...@redhat.com> wrote:

> This adds a new output mode to virt-v2v.  virt-v2v -o rhv-upload
> streams images directly to an oVirt or RHV >= 4 Data Domain using the
> oVirt SDK v4.  It is more efficient than -o rhv because it does not
> need to go via the Export Storage Domain, and is possible for humans
> to use unlike -o vdsm.
> 
> The implementation uses the Python SDK (‘ovirtsdk4’ module).  An
> nbdkit Python 3 plugin translates NBD calls from qemu into HTTPS
> requests to oVirt via the SDK.
> ---
>  .gitignore                                |   3 +
>  TODO                                      |  27 ++
>  v2v/Makefile.am                           |  30 +-
>  v2v/cmdline.ml                            |  41 +++
>  v2v/embed.sh                              |  45 +++
>  v2v/output_rhv_upload.ml                  | 390 ++++++++++++++++++++++++++
>  v2v/output_rhv_upload.mli                 |  33 +++
>  v2v/output_rhv_upload_createvm_source.mli |  19 ++
>  v2v/output_rhv_upload_plugin_source.mli   |  19 ++
>  v2v/output_rhv_upload_precheck_source.mli |  19 ++
>  v2v/rhv-upload-createvm.py                |  86 ++++++
>  v2v/rhv-upload-plugin.py                  | 438 
> ++++++++++++++++++++++++++++++
>  v2v/rhv-upload-precheck.py                |  73 +++++
>  v2v/test-v2v-o-rhv-upload-oo-query.sh     |  38 +++
>  v2v/test-v2v-python-syntax.sh             |  45 +++
>  v2v/virt-v2v.pod                          | 138 ++++++++--
>  16 files changed, 1426 insertions(+), 18 deletions(-)
> 
> diff --git a/.gitignore b/.gitignore
> index d72447d1d..211376eef 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -654,6 +654,9 @@ Makefile.in
>  /utils/qemu-speed-test/qemu-speed-test
>  /v2v/.depend
>  /v2v/oUnit-*
> +/v2v/output_rhv_upload_createvm_source.ml
> +/v2v/output_rhv_upload_plugin_source.ml
> +/v2v/output_rhv_upload_precheck_source.ml
>  /v2v/real-*.d/
>  /v2v/real-*.img
>  /v2v/real-*.xml
> diff --git a/TODO b/TODO
> index 2e37ce67c..d196a3f6b 100644
> --- a/TODO
> +++ b/TODO
> @@ -570,3 +570,30 @@ Subsecond handling in virt-diff, virt-ls
>  
>  Handle nanoseconds properly.  You should be able to specify them on
>  the command line and display them.
> +
> +virt-v2v -o rhv-upload
> +----------------------
> +
> +* Set or disable the ticket timeout.  The default is going to be
> +  increased (from current 60 seconds), so maybe we won't have to
> +  set it.  See also:
> +  https://bugzilla.redhat.com/show_bug.cgi?id=1563278
> +
> +* qcow2 cannot be supported yet because there is not yet any
> +  concept in imageio of read+write handles.
> +  https://bugzilla.redhat.com/show_bug.cgi?id=1563299
> +
> +* preallocated cannot be supported yet because imageio doesn't
> +  know how to zero the image efficiently, instead it runs an
> +  fallocate process which writes to every block and that takes
> +  many minutes.
> +
> +* Really check what insecure/rhv_cafile do and implement it correctly.
> +
> +* Measure and resolve performance problems.
> +
> +* Allocated image size is unknown for v2v uploads, but imageio needs
> +  to know it.  We pass initial_size == provisioned_size == virtual size.
> +  That can't be fixed from the v2v side.
> +
> +* There are unresolved issues about how to clean up disks on failure.
> diff --git a/v2v/Makefile.am b/v2v/Makefile.am
> index 6e71cae3c..f36731750 100644
> --- a/v2v/Makefile.am
> +++ b/v2v/Makefile.am
> @@ -22,12 +22,19 @@ generator_built = \
>       uefi.mli
>  
>  BUILT_SOURCES = \
> -     $(generator_built)
> +     $(generator_built) \
> +     output_rhv_upload_createvm_source.ml \
> +     output_rhv_upload_plugin_source.ml \
> +     output_rhv_upload_precheck_source.ml
>  
>  EXTRA_DIST = \
>       $(SOURCES_MLI) $(SOURCES_ML) $(SOURCES_C) \
>       copy_to_local.ml \
>       copy_to_local.mli \
> +     embed-code.sh \
> +     rhv-upload-createvm.py \
> +     rhv-upload-plugin.py \
> +     rhv-upload-precheck.py \
>       v2v_slow_unit_tests.ml \
>       v2v-slow-unit-tests.sh \
>       v2v_unit_tests.ml \
> @@ -64,6 +71,10 @@ SOURCES_MLI = \
>       output_null.mli \
>       output_qemu.mli \
>       output_rhv.mli \
> +     output_rhv_upload.mli \
> +     output_rhv_upload_createvm_source.mli \
> +     output_rhv_upload_plugin_source.mli \
> +     output_rhv_upload_precheck_source.mli \
>       output_vdsm.mli \
>       parse_ovf_from_ova.mli \
>       parse_libvirt_xml.mli \
> @@ -116,6 +127,10 @@ SOURCES_ML = \
>       output_local.ml \
>       output_qemu.ml \
>       output_rhv.ml \
> +     output_rhv_upload_createvm_source.ml \
> +     output_rhv_upload_plugin_source.ml \
> +     output_rhv_upload_precheck_source.ml \
> +     output_rhv_upload.ml \
>       output_vdsm.ml \
>       inspect_source.ml \
>       target_bus_assignment.ml \
> @@ -126,6 +141,15 @@ SOURCES_C = \
>       libvirt_utils-c.c \
>       qemuopts-c.c
>  
> +# These files are generated and contain rhv-upload-*.py embedded as an
> +# OCaml string.
> +output_rhv_upload_createvm_source.ml: rhv-upload-createvm.py
> +     ./embed.sh code $^ $@
> +output_rhv_upload_plugin_source.ml: rhv-upload-plugin.py
> +     ./embed.sh code $^ $@
> +output_rhv_upload_precheck_source.ml: rhv-upload-precheck.py
> +     ./embed.sh code $^ $@
> +
>  if HAVE_OCAML
>  
>  bin_PROGRAMS = virt-v2v virt-v2v-copy-to-local
> @@ -295,6 +319,7 @@ TESTS_ENVIRONMENT = $(top_builddir)/run --test
>  
>  TESTS = \
>       test-v2v-docs.sh \
> +     test-v2v-python-syntax.sh \
>       test-v2v-i-ova-bad-sha1.sh \
>       test-v2v-i-ova-bad-sha256.sh \
>       test-v2v-i-ova-formats.sh \
> @@ -307,6 +332,7 @@ TESTS = \
>       test-v2v-i-ova-two-disks.sh \
>       test-v2v-i-vmx.sh \
>       test-v2v-it-vddk-io-query.sh \
> +     test-v2v-o-rhv-upload-oo-query.sh \
>       test-v2v-o-vdsm-oo-query.sh \
>       test-v2v-bad-networks-and-bridges.sh
>  
> @@ -485,6 +511,7 @@ EXTRA_DIST += \
>       test-v2v-o-qemu.sh \
>       test-v2v-o-rhv.ovf.expected \
>       test-v2v-o-rhv.sh \
> +     test-v2v-o-rhv-upload-oo-query.sh \
>       test-v2v-o-vdsm-oo-query.sh \
>       test-v2v-o-vdsm-options.ovf.expected \
>       test-v2v-o-vdsm-options.sh \
> @@ -494,6 +521,7 @@ EXTRA_DIST += \
>       test-v2v-print-source.expected \
>       test-v2v-print-source.sh \
>       test-v2v-print-source.xml \
> +     test-v2v-python-syntax.sh \
>       test-v2v-conversion-of.sh \
>       test-v2v-sound.sh \
>       test-v2v-sound.xml \
> diff --git a/v2v/cmdline.ml b/v2v/cmdline.ml
> index b36ec9c5e..785e91ef9 100644
> --- a/v2v/cmdline.ml
> +++ b/v2v/cmdline.ml
> @@ -135,6 +135,8 @@ let parse_cmdline () =
>      | "disk" | "local" -> output_mode := `Local
>      | "null" -> output_mode := `Null
>      | "ovirt" | "rhv" | "rhev" -> output_mode := `RHV
> +    | "ovirt-upload" | "ovirt_upload" | "rhv-upload" | "rhv_upload" ->
> +       output_mode := `RHV_Upload
>      | "qemu" -> output_mode := `QEmu
>      | "vdsm" -> output_mode := `VDSM
>      | s ->
> @@ -397,6 +399,16 @@ read the man page virt-v2v(1).
>      | `Null -> no_options (); `Null
>      | `RHV -> no_options (); `RHV
>      | `QEmu -> no_options (); `QEmu
> +    | `RHV_Upload ->
> +       if is_query then (
> +         Output_rhv_upload.print_output_options ();
> +         exit 0
> +       )
> +       else (
> +         let rhv_options =
> +           Output_rhv_upload.parse_output_options output_options in
> +         `RHV_Upload rhv_options
> +       )
>      | `VDSM ->
>         if is_query then (
>           Output_vdsm.print_output_options ();
> @@ -579,6 +591,35 @@ read the man page virt-v2v(1).
>        Output_rhv.output_rhv os output_alloc,
>        output_format, output_alloc
>  
> +    | `RHV_Upload rhv_options ->
> +      let output_conn =
> +        match output_conn with
> +        | None ->
> +           error (f_"-o rhv-upload: use ‘-oc’ to point to the oVirt or RHV 
> server REST API URL, which is usually https://servername/ovirt-engine/api";)
> +        | Some oc -> oc in
> +      (* Output format / sparse must currently be raw+sparse.  We can
> +       * change this in future.  See TODO file for details. XXX
> +       *)
> +      if output_alloc <> Sparse || output_format <> Some "raw" then
> +        error (f_"-o rhv-upload: currently you must use ‘-of raw’ and you 
> cannot use ‘-oa preallocated’ with this output mode.  These restrictions will 
> be loosened in a future version.");
> +      (* In theory we could make the password optional in future. *)
> +      let output_password =
> +        match output_password with
> +        | None ->
> +           error (f_"-o rhv-upload: output password file was not specified, 
> use ‘-op’ to point to a file which contains the password used to connect to 
> the oVirt or RHV server")
> +        | Some op -> op in
> +      let os =
> +        match output_storage with
> +        | None ->
> +           error (f_"-o rhv-upload: output storage was not specified, use 
> ‘-os’");
> +        | Some os -> os in
> +      if qemu_boot then
> +        error_option_cannot_be_used_in_output_mode "rhv-upload" 
> "--qemu-boot";
> +      Output_rhv_upload.output_rhv_upload output_alloc output_conn
> +                                          output_password os
> +                                          rhv_options,
> +      output_format, output_alloc
> +
>      | `VDSM vdsm_options ->
>        if output_password <> None then
>          error_option_cannot_be_used_in_output_mode "vdsm" "-op";
> diff --git a/v2v/embed.sh b/v2v/embed.sh
> new file mode 100755
> index 000000000..0a65cd428
> --- /dev/null
> +++ b/v2v/embed.sh
> @@ -0,0 +1,45 @@
> +#!/bin/bash -
> +# Embed code or other content into an OCaml file.
> +# Copyright (C) 2018 Red Hat Inc.
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write to the Free Software
> +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 
> USA.
> +
> +# Embed code or other content into an OCaml file.
> +#
> +# It is embedded into a string.  As OCaml string literals have virtually
> +# no restrictions on length or content we only have to escape double
> +# quotes for backslash characters.
> +
> +if [ $# -ne 3 ]; then
> +    echo "embed.sh identifier input output"
> +    exit 1
> +fi
> +
> +ident="$1"
> +input="$2"
> +output="$3"
> +
> +rm -f "$output" "$output"-t
> +
> +exec >"$output"-t
> +
> +echo "(* Generated by embed.sh from $input *)"
> +echo
> +echo let "$ident" = '"'
> +sed -e 's/\(["\]\)/\\\1/g' < "$input"
> +echo '"'
> +
> +chmod -w "$output"-t
> +mv "$output"-t "$output"
> diff --git a/v2v/output_rhv_upload.ml b/v2v/output_rhv_upload.ml
> new file mode 100644
> index 000000000..1be755a25
> --- /dev/null
> +++ b/v2v/output_rhv_upload.ml
> @@ -0,0 +1,390 @@
> +(* virt-v2v
> + * Copyright (C) 2009-2018 Red Hat Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License along
> + * with this program; if not, write to the Free Software Foundation, Inc.,
> + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> + *)
> +
> +open Printf
> +open Unix
> +
> +open Std_utils
> +open Tools_utils
> +open Unix_utils
> +open Common_gettext.Gettext
> +
> +open Types
> +open Utils
> +
> +type rhv_options = {
> +  rhv_cafile : string;
> +  rhv_cluster : string option;
> +  rhv_direct : bool;
> +  rhv_verifypeer : bool;
> +}
> +
> +let print_output_options () =
> +  printf (f_"Output options (-oo) which can be used with -o rhv-upload:
> +
> +  -oo rhv-cafile=CA.PEM           Set ‘ca.pem’ certificate bundle filename.
> +  -oo rhv-cluster=CLUSTERNAME     Set RHV cluster name.
> +  -oo rhv-direct[=true|false]     Use direct transfer mode (default: false).
> +  -oo rhv-verifypeer[=true|false] Verify server identity (default: false).
> +")
> +
> +let parse_output_options options =
> +  let rhv_cafile = ref None in
> +  let rhv_cluster = ref None in
> +  let rhv_direct = ref false in
> +  let rhv_verifypeer = ref false in
> +
> +  List.iter (
> +    function
> +    | "rhv-cafile", v ->
> +       if !rhv_cafile <> None then
> +         error (f_"-o rhv-upload: -oo rhv-cafile set twice");
> +       rhv_cafile := Some v
> +    | "rhv-cluster", v ->
> +       if !rhv_cluster <> None then
> +         error (f_"-o rhv-upload: -oo rhv-cluster set twice");
> +       rhv_cluster := Some v
> +    | "rhv-direct", "" -> rhv_direct := true
> +    | "rhv-direct", v -> rhv_direct := bool_of_string v
> +    | "rhv-verifypeer", "" -> rhv_verifypeer := true
> +    | "rhv-verifypeer", v -> rhv_verifypeer := bool_of_string v
> +    | k, _ ->
> +       error (f_"-o rhv-upload: unknown output option ‘-oo %s’") k
> +  ) options;
> +
> +  let rhv_cafile =
> +    match !rhv_cafile with
> +    | Some s -> s
> +    | None ->
> +       error (f_"-o rhv-upload: must use ‘-oo rhv-cafile’ to supply the path 
> to the oVirt or RHV user’s ‘ca.pem’ file") in
> +  let rhv_cluster = !rhv_cluster in
> +  let rhv_direct = !rhv_direct in
> +  let rhv_verifypeer = !rhv_verifypeer in
> +
> +  { rhv_cafile; rhv_cluster; rhv_direct; rhv_verifypeer }
> +
> +let python3 = "python3" (* Defined by PEP 394 *)
> +let pidfile_timeout = 30
> +let finalization_timeout = 5*60
> +
> +class output_rhv_upload output_alloc output_conn
> +                        output_password output_storage
> +                        rhv_options =
> +  (* Create a temporary directory which will be deleted on exit. *)
> +  let tmpdir =
> +    let base_dir = (open_guestfs ())#get_cachedir () in
> +    let t = Mkdtemp.temp_dir ~base_dir "rhvupload." in
> +    rmdir_on_exit t;
> +    t in
> +
> +  let diskid_file_of_id id = tmpdir // sprintf "diskid.%d" id in
> +
> +  (* Write the Python precheck, plugin and create VM to a temporary file. *)
> +  let precheck =
> +    let precheck = tmpdir // "rhv-upload-precheck.py" in
> +    with_open_out
> +      precheck
> +      (fun chan -> output_string chan 
> Output_rhv_upload_precheck_source.code);
> +    precheck in
> +  let plugin =
> +    let plugin = tmpdir // "rhv-upload-plugin.py" in
> +    with_open_out
> +      plugin
> +      (fun chan -> output_string chan Output_rhv_upload_plugin_source.code);
> +    plugin in
> +  let createvm =
> +    let createvm = tmpdir // "rhv-upload-createvm.py" in
> +    with_open_out
> +      createvm
> +      (fun chan -> output_string chan 
> Output_rhv_upload_createvm_source.code);
> +    createvm in
> +
> +  (* Is SELinux enabled and enforcing on the host? *)
> +  let have_selinux =
> +    0 = Sys.command "getenforce 2>/dev/null | grep -isq Enforcing" in
> +
> +  (* Check that nbdkit is available and new enough. *)
> +  let error_unless_nbdkit_working () =
> +    if 0 <> Sys.command "nbdkit --version >/dev/null" then
> +      error (f_"nbdkit is not installed or not working.  It is required to 
> use ‘-o rhv-upload’.  See \"OUTPUT TO RHV\" in the virt-v2v(1) manual.");
> +
> +    (* Check it's a new enough version.  The latest features we
> +     * require are ‘--exit-with-parent’ and ‘--selinux-label’, both
> +     * added in 1.1.14.  (We use 1.1.16 as the minimum here because
> +     * it also adds the selinux=yes|no flag in --dump-config).
> +     *)
> +    let lines = external_command "nbdkit --help" in
> +    let lines = String.concat " " lines in
> +    if String.find lines "exit-with-parent" == -1 ||
> +       String.find lines "selinux-label" == -1 then
> +      error (f_"nbdkit is not new enough, you need to upgrade to nbdkit ≥ 
> 1.1.16")
> +  in
> +
> +  (* Check that the python3 plugin is installed and working
> +   * and can load the plugin script.
> +   *)
> +  let error_unless_nbdkit_python3_working () =
> +    let cmd = sprintf "nbdkit %s %s --dump-plugin >/dev/null"
> +                      python3 (quote plugin) in
> +    if Sys.command cmd <> 0 then
> +      error (f_"nbdkit Python 3 plugin is not installed or not working.  It 
> is required if you want to use ‘-o rhv-upload’.
> +
> +See also \"OUTPUT TO RHV\" in the virt-v2v(1) manual.")
> +  in
> +
> +  (* Check that nbdkit was compiled with SELinux support (for the
> +   * --selinux-label option).
> +   *)
> +  let error_unless_nbdkit_compiled_with_selinux () =
> +    let lines = external_command "nbdkit --dump-config" in
> +    (* In nbdkit <= 1.1.15 the selinux attribute was not present
> +     * at all in --dump-config output so there was no way to tell.
> +     * Ignore this case because there will be an error later when
> +     * we try to use the --selinux-label parameter.
> +     *)
> +    if List.mem "selinux=no" (List.map String.trim lines) then
> +      error (f_"nbdkit was compiled without SELinux support.  You will have 
> to recompile nbdkit with libselinux-devel installed, or else set SELinux to 
> Permissive mode while doing the conversion.")
> +  in
> +
> +  (* JSON parameters which are invariant between disks. *)
> +  let json_params = [
> +    "output_conn", JSON.String output_conn;
> +    "output_password", JSON.String output_password;
> +    "output_storage", JSON.String output_storage;
> +    "output_sparse", JSON.Bool (match output_alloc with
> +                                | Sparse -> true
> +                                | Preallocated -> false);
> +    "rhv_cafile", JSON.String rhv_options.rhv_cafile;
> +    "rhv_cluster",
> +      JSON.String (Option.default "Default" rhv_options.rhv_cluster);
> +    "rhv_direct", JSON.Bool rhv_options.rhv_direct;
> +
> +    (* The 'Insecure' flag seems to be a number with various possible
> +     * meanings, however we just set it to True/False.
> +     *
> +     * 
> https://github.com/oVirt/ovirt-engine-sdk/blob/19aa7070b80e60a4cfd910448287aecf9083acbe/sdk/lib/ovirtsdk4/__init__.py#L395
> +     *)
> +    "insecure", JSON.Bool (not rhv_options.rhv_verifypeer);
> +  ] in
> +
> +  (* nbdkit command line args which are invariant between disks. *)
> +  let nbdkit_args =
> +    let args = [
> +      "nbdkit";
> +
> +      "--foreground";           (* run in foreground *)
> +      "--exit-with-parent";     (* exit when virt-v2v exits *)
> +      "--newstyle";             (* use newstyle NBD protocol *)
> +      "--exportname"; "/";
> +
> +      "python3";                (* use the nbdkit Python 3 plugin *)
> +      plugin;                   (* Python plugin script *)
> +    ] in
> +    let args = if verbose () then args @ ["--verbose"] else args in
> +    let args =
> +      (* label the socket so qemu can open it *)
> +      if have_selinux then
> +        args @ ["--selinux-label"; "system_u:object_r:svirt_t:s0"]
> +      else args in
> +    args in
> +
> +object
> +  inherit output
> +
> +  method precheck () =
> +    error_unless_nbdkit_working ();
> +    error_unless_nbdkit_python3_working ();
> +    if have_selinux then
> +      error_unless_nbdkit_compiled_with_selinux ()
> +
> +  method as_options =
> +    "-o rhv-upload" ^
> +    (match output_alloc with
> +     | Sparse -> "" (* default, don't need to print it *)
> +     | Preallocated -> " -oa preallocated") ^
> +    sprintf " -oc %s -op %s -os %s"
> +            output_conn output_password output_storage
> +
> +  method supported_firmware = [ TargetBIOS ]
> +
> +  method prepare_targets source targets =
> +    let output_name = source.s_name in
> +    let json_params =
> +      ("output_name", JSON.String output_name) :: json_params in
> +
> +    (* Python code prechecks.  These can't run in #precheck because
> +     * we need to know the name of the virtual machine.
> +     *)
> +    let json_param_file = tmpdir // "params.json" in
> +    with_open_out
> +      json_param_file
> +      (fun chan -> output_string chan (JSON.string_of_doc json_params));
> +    if run_command [ python3; precheck; json_param_file ] <> 0 then
> +      error (f_"failed server prechecks, see earlier errors");
> +
> +    (* Create an nbdkit instance for each disk and set the
> +     * target URI to point to the NBD socket.
> +     *)
> +    List.map (
> +      fun t ->
> +        let id = t.target_overlay.ov_source.s_disk_id in
> +        let disk_name = sprintf "%s-%03d" output_name id in
> +        let json_params =
> +          ("disk_name", JSON.String disk_name) :: json_params in
> +
> +        let disk_format =
> +          match t.target_format with
> +          | ("raw" | "qcow2") as fmt -> fmt
> +          | _ ->
> +             error (f_"rhv-upload: -of %s: Only output format ‘raw’ or 
> ‘qcow2’ is supported.  If the input is in a different format then force one 
> of these output formats by adding either ‘-of raw’ or ‘-of qcow2’ on the 
> command line.")
> +                   t.target_format in
> +        let json_params =
> +          ("disk_format", JSON.String disk_format) :: json_params in
> +
> +        let disk_size = t.target_overlay.ov_virtual_size in
> +        let json_params =
> +          ("disk_size", JSON.Int64 disk_size) :: json_params in
> +
> +        (* Ask the plugin to write the disk ID to a special file. *)
> +        let diskid_file = diskid_file_of_id id in
> +        let json_params =
> +          ("diskid_file", JSON.String diskid_file) :: json_params in
> +
> +        (* Write the JSON parameters to a file. *)
> +        let json_param_file = tmpdir // sprintf "params%d.json" id in
> +        with_open_out
> +          json_param_file
> +          (fun chan -> output_string chan (JSON.string_of_doc json_params));
> +
> +        let sock = tmpdir // sprintf "nbdkit%d.sock" id in
> +        let pidfile = tmpdir // sprintf "nbdkit%d.pid" id in
> +
> +        (* Add common arguments to per-target arguments. *)
> +        let args =
> +          nbdkit_args @ [ "--pidfile"; pidfile;
> +                          "--unix"; sock;
> +                          sprintf "params=%s" json_param_file ] in
> +
> +        (* Print the full command we are about to run when debugging. *)
> +        if verbose () then (
> +          eprintf "running nbdkit:\n";
> +          List.iter (fun arg -> eprintf " %s" (quote arg)) args;
> +          prerr_newline ()
> +        );
> +
> +        (* Start an nbdkit instance in the background.  By using
> +         * --exit-with-parent we don't have to worry about clean-up.
> +         *)
> +        let args = Array.of_list args in
> +        let pid = fork () in
> +        if pid = 0 then (
> +          (* Child process (nbdkit). *)
> +          execvp "nbdkit" args
> +        );
> +
> +        (* Wait for the pidfile to appear so we know that nbdkit
> +         * is listening for requests.
> +         *)
> +        if not (wait_for_file pidfile pidfile_timeout) then (
> +          if verbose () then
> +            error (f_"nbdkit did not start up.  See previous debugging 
> messages for problems.")
> +          else
> +            error (f_"nbdkit did not start up.  There may be errors printed 
> by nbdkit above.
> +
> +If the messages above are not sufficient to diagnose the problem then add 
> the ‘virt-v2v -v -x’ options and examine the debugging output carefully.")
> +        );
> +
> +        if have_selinux then (
> +          (* Note that Unix domain sockets have both a file label and
> +           * a socket/process label.  Using --selinux-label above
> +           * only set the socket label, but we must also set the file
> +           * label.
> +           *)
> +          ignore (
> +              run_command ["chcon"; "system_u:object_r:svirt_image_t:s0";
> +                           sock]
> +          );
> +        );
> +        (* ... and the regular Unix permissions, in case qemu is
> +         * running as another user.
> +         *)
> +        chmod sock 0o777;
> +
> +        (* Tell ‘qemu-img convert’ to write to the nbd socket which is
> +         * connected to nbdkit.
> +         *)
> +        let json_params = [
> +          "file.driver", JSON.String "nbd";
> +          "file.path", JSON.String sock;
> +          "file.export", JSON.String "/";
> +        ] in
> +        let target_file =
> +          TargetURI ("json:" ^ JSON.string_of_doc json_params) in
> +        { t with target_file }
> +    ) targets
> +
> +  method create_metadata source targets _ guestcaps inspect target_firmware =
> +    (* Get the UUIDs of each disk image.  These files are written
> +     * out by the nbdkit plugins on successful finalization of the
> +     * transfer.
> +     *)
> +    let nr_disks = List.length targets in
> +    let image_uuids =
> +      List.map (
> +        fun t ->
> +          let id = t.target_overlay.ov_source.s_disk_id in
> +          let diskid_file = diskid_file_of_id id in
> +          if not (wait_for_file diskid_file finalization_timeout) then
> +            error (f_"transfer of disk %d/%d failed, see earlier error 
> messages")
> +                  (id+1) nr_disks;
> +          let diskid = read_whole_file diskid_file in
> +          diskid
> +      ) targets in
> +
> +    (* We don't have the storage domain UUID, but instead we write
> +     * in a magic value which the Python code (which can get it)
> +     * will substitute.
> +     *)
> +    let sd_uuid = "@SD_UUID@" in
> +
> +    (* The volume and VM UUIDs are made up. *)
> +    let vol_uuids = List.map (fun _ -> uuidgen ()) targets
> +    and vm_uuid = uuidgen () in
> +
> +    (* Create the metadata. *)
> +    let ovf =
> +      Create_ovf.create_ovf source targets guestcaps inspect
> +                            output_alloc
> +                            sd_uuid image_uuids vol_uuids vm_uuid
> +                            OVirt in
> +    let ovf = DOM.doc_to_string ovf in
> +
> +    let json_param_file = tmpdir // "params.json" in
> +    with_open_out
> +      json_param_file
> +      (fun chan -> output_string chan (JSON.string_of_doc json_params));
> +
> +    let ovf_file = tmpdir // "vm.ovf" in
> +    with_open_out ovf_file (fun chan -> output_string chan ovf);
> +    if run_command [ python3; createvm; json_param_file; ovf_file ] <> 0 then
> +      error (f_"failed to create virtual machine, see earlier errors")
> +
> +end
> +
> +let output_rhv_upload = new output_rhv_upload
> +let () = Modules_list.register_output_module "rhv-upload"
> diff --git a/v2v/output_rhv_upload.mli b/v2v/output_rhv_upload.mli
> new file mode 100644
> index 000000000..f6cd69a61
> --- /dev/null
> +++ b/v2v/output_rhv_upload.mli
> @@ -0,0 +1,33 @@
> +(* virt-v2v
> + * Copyright (C) 2009-2018 Red Hat Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License along
> + * with this program; if not, write to the Free Software Foundation, Inc.,
> + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> + *)
> +
> +(** [-o rhv-upload] target. *)
> +
> +type rhv_options
> +(** Miscellaneous extra command line parameters used by rhv-upload. *)
> +
> +val print_output_options : unit -> unit
> +val parse_output_options : (string * string) list -> rhv_options
> +(** Print and parse rhv-upload -oo options. *)
> +
> +val output_rhv_upload : Types.output_allocation -> string -> string ->
> +                        string -> rhv_options -> Types.output
> +(** [output_rhv_upload output_alloc output_conn output_password 
> output_storage
> +     rhv_options]
> +    creates and returns a new {!Types.output} object specialized for writing
> +    output to oVirt or RHV directly via RHV APIs. *)
> diff --git a/v2v/output_rhv_upload_createvm_source.mli 
> b/v2v/output_rhv_upload_createvm_source.mli
> new file mode 100644
> index 000000000..c1bafa15b
> --- /dev/null
> +++ b/v2v/output_rhv_upload_createvm_source.mli
> @@ -0,0 +1,19 @@
> +(* virt-v2v
> + * Copyright (C) 2018 Red Hat Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License along
> + * with this program; if not, write to the Free Software Foundation, Inc.,
> + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> + *)
> +
> +val code : string
> diff --git a/v2v/output_rhv_upload_plugin_source.mli 
> b/v2v/output_rhv_upload_plugin_source.mli
> new file mode 100644
> index 000000000..c1bafa15b
> --- /dev/null
> +++ b/v2v/output_rhv_upload_plugin_source.mli
> @@ -0,0 +1,19 @@
> +(* virt-v2v
> + * Copyright (C) 2018 Red Hat Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License along
> + * with this program; if not, write to the Free Software Foundation, Inc.,
> + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> + *)
> +
> +val code : string
> diff --git a/v2v/output_rhv_upload_precheck_source.mli 
> b/v2v/output_rhv_upload_precheck_source.mli
> new file mode 100644
> index 000000000..c1bafa15b
> --- /dev/null
> +++ b/v2v/output_rhv_upload_precheck_source.mli
> @@ -0,0 +1,19 @@
> +(* virt-v2v
> + * Copyright (C) 2018 Red Hat Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License along
> + * with this program; if not, write to the Free Software Foundation, Inc.,
> + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> + *)
> +
> +val code : string
> diff --git a/v2v/rhv-upload-createvm.py b/v2v/rhv-upload-createvm.py
> new file mode 100644
> index 000000000..a34627ec8
> --- /dev/null
> +++ b/v2v/rhv-upload-createvm.py
> @@ -0,0 +1,86 @@
> +# -*- python -*-
> +# oVirt or RHV upload create VM used by ‘virt-v2v -o rhv-upload’
> +# Copyright (C) 2018 Red Hat Inc.
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License along
> +# with this program; if not, write to the Free Software Foundation, Inc.,
> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +
> +import json
> +import logging
> +import sys
> +import time
> +
> +from http.client import HTTPSConnection
> +from urllib.parse import urlparse
> +
> +import ovirtsdk4 as sdk
> +import ovirtsdk4.types as types
> +
> +# Parameters are passed in via a JSON doc from the OCaml code.
> +# Because this Python code ships embedded inside virt-v2v there
> +# is no formal API here.
> +params = None
> +ovf = None                      # OVF file
> +
> +if len(sys.argv) != 3:
> +    raise RuntimeError("incorrect number of parameters")
> +
> +# Parameters are passed in via a JSON document.
> +with open(sys.argv[1], 'r') as fp:
> +    params = json.load(fp)
> +
> +# What is passed in is a password file, read the actual password.
> +with open(params['output_password'], 'r') as fp:
> +    output_password = fp.read()
> +output_password = output_password.rstrip()
> +
> +# Read the OVF document.
> +with open(sys.argv[2], 'r') as fp:
> +    ovf = fp.read()
> +
> +# Parse out the username from the output_conn URL.
> +parsed = urlparse(params['output_conn'])
> +username = parsed.username or "admin@internal"
> +
> +# Connect to the server.
> +connection = sdk.Connection(
> +    url = params['output_conn'],
> +    username = username,
> +    password = output_password,
> +    ca_file = params['rhv_cafile'],
> +    log = logging.getLogger(),
> +    insecure = params['insecure'],
> +)
> +
> +system_service = connection.system_service()
> +
> +# Get the storage domain UUID and substitute it into the OVF doc.
> +sds_service = system_service.storage_domains_service()
> +sd = sds_service.list(search=("name=%s" % params['output_storage']))[0]
> +sd_uuid = sd.id
> +
> +ovf.replace("@SD_UUID@", sd_uuid)
> +
> +vms_service = system_service.vms_service()
> +vm = vms_service.add(
> +    types.Vm(
> +        cluster=types.Cluster(name = params['rhv_cluster']),
> +        initialization=types.Initialization(
> +            configuration = types.Configuration(
> +                type = types.ConfigurationType.OVA,
> +                data = ovf,
> +            )
> +        )
> +    )
> +)
> diff --git a/v2v/rhv-upload-plugin.py b/v2v/rhv-upload-plugin.py
> new file mode 100644
> index 000000000..badfe63d8
> --- /dev/null
> +++ b/v2v/rhv-upload-plugin.py
> @@ -0,0 +1,438 @@
> +# -*- python -*-
> +# oVirt or RHV upload nbdkit plugin used by ‘virt-v2v -o rhv-upload’
> +# Copyright (C) 2018 Red Hat Inc.
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License along
> +# with this program; if not, write to the Free Software Foundation, Inc.,
> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +
> +import builtins
> +import json
> +import logging
> +import ssl
> +import sys
> +import time
> +
> +from http.client import HTTPSConnection
> +from urllib.parse import urlparse
> +
> +import ovirtsdk4 as sdk
> +import ovirtsdk4.types as types
> +
> +# Timeout to wait for oVirt disks to change status, or the transfer
> +# object to finish initializing [seconds].
> +timeout = 5*60
> +
> +# Parameters are passed in via a JSON doc from the OCaml code.
> +# Because this Python code ships embedded inside virt-v2v there
> +# is no formal API here.
> +params = None
> +
> +def config(key, value):
> +    global params
> +
> +    if key == "params":
> +        with builtins.open(value, 'r') as fp:
> +            params = json.load(fp)
> +    else:
> +        raise RuntimeError("unknown configuration key '%s'" % key)
> +
> +def config_complete():
> +    if params is None:
> +        raise RuntimeError("missing configuration parameters")
> +
> +def open(readonly):
> +    # Parse out the username from the output_conn URL.
> +    parsed = urlparse(params['output_conn'])
> +    username = parsed.username or "admin@internal"
> +
> +    # Read the password from file.
> +    with builtins.open(params['output_password'], 'r') as fp:
> +        password = fp.read()
> +    password = password.rstrip()
> +
> +    # Connect to the server.
> +    connection = sdk.Connection(
> +        url = params['output_conn'],
> +        username = username,
> +        password = password,
> +        ca_file = params['rhv_cafile'],
> +        log = logging.getLogger(),
> +        insecure = params['insecure'],
> +    )
> +
> +    system_service = connection.system_service()
> +
> +    # Create the disk.
> +    disks_service = system_service.disks_service()
> +    if params['disk_format'] == "raw":
> +        disk_format = types.DiskFormat.RAW
> +    else:
> +        disk_format = types.DiskFormat.COW
> +    disk = disks_service.add(
> +        disk = types.Disk(
> +            name = params['disk_name'],
> +            description = "Uploaded by virt-v2v",
> +            format = disk_format,
> +            initial_size = params['disk_size'],
> +            provisioned_size = params['disk_size'],
> +            # XXX Ignores params['output_sparse'].
> +            # Handling this properly will be complex, see:
> +            # 
> https://www.redhat.com/archives/libguestfs/2018-March/msg00177.html
> +            sparse = True,
> +            storage_domains = [
> +                types.StorageDomain(
> +                    name = params['output_storage'],
> +                )
> +            ],
> +        )
> +    )
> +
> +    # Wait till the disk is up, as the transfer can't start if the
> +    # disk is locked:
> +    disk_service = disks_service.disk_service(disk.id)
> +
> +    endt = time.time() + timeout
> +    while True:
> +        time.sleep(5)
> +        disk = disk_service.get()
> +        if disk.status == types.DiskStatus.OK:
> +            break
> +        if time.time() > endt:
> +            raise RuntimeError("timed out waiting for disk to become 
> unlocked")
> +
> +    # Get a reference to the transfer service.
> +    transfers_service = system_service.image_transfers_service()
> +
> +    # Create a new image transfer.
> +    transfer = transfers_service.add(
> +        types.ImageTransfer(
> +            image = types.Image(
> +                id = disk.id
> +            )
> +        )
> +    )

I'm pretty sure I already mentioned it before. It would help debugging
if we could dump the `disk.id` to the output here so it shows in
virt-v2v debug logs.


> +    # Get a reference to the created transfer service.
> +    transfer_service = transfers_service.image_transfer_service(transfer.id)

Dumping `transfer.id` could also be usefull... maybe? Don't know. If
this shows in imageio dameon logs or not...


> +
> +    # After adding a new transfer for the disk, the transfer's status
> +    # will be INITIALIZING.  Wait until the init phase is over. The
> +    # actual transfer can start when its status is "Transferring".
> +    endt = time.time() + timeout
> +    while True:
> +        time.sleep(5)
> +        transfer = transfer_service.get()
> +        if transfer.phase != types.ImageTransferPhase.INITIALIZING:
> +            break
> +        if time.time() > endt:
> +            raise RuntimeError("timed out waiting for transfer status " +
> +                               "!= INITIALIZING")
> +
> +    # Now we have permission to start the transfer.
> +    if params['rhv_direct']:
> +        if transfer.transfer_url is None:
> +            raise RuntimeError("direct upload to host not supported, " +
> +                               "requires ovirt-engine >= 4.2 and only works 
> " +
> +                               "when virt-v2v is run within the oVirt/RHV " +
> +                               "environment, eg. on an ovirt node.")

spelling issue: oVirt instead of ovirt


    Tomas

> +        destination_url = urlparse(transfer.transfer_url)
> +    else:
> +        destination_url = urlparse(transfer.proxy_url)
> +
> +    context = ssl.create_default_context()
> +    context.load_verify_locations(cafile = params['rhv_cafile'])
> +
> +    http = HTTPSConnection(
> +        destination_url.hostname,
> +        destination_url.port,
> +        context = context
> +    )
> +
> +    # Save everything we need to make requests in the handle.
> +    return {
> +        'can_flush': False,
> +        'can_trim': False,
> +        'can_zero': False,
> +        'connection': connection,
> +        'disk': disk,
> +        'disk_service': disk_service,
> +        'failed': False,
> +        'got_options': False,
> +        'highestwrite': 0,
> +        'http': http,
> +        'needs_auth': not params['rhv_direct'],
> +        'path': destination_url.path,
> +        'transfer': transfer,
> +        'transfer_service': transfer_service,
> +    }
> +
> +# Can we issue zero, trim or flush requests?
> +def get_options(h):
> +    if h['got_options']:
> +        return
> +    h['got_options'] = True
> +
> +    http = h['http']
> +    transfer = h['transfer']
> +
> +    http.putrequest("OPTIONS", h['path'])
> +    http.putheader("Authorization", transfer.signed_ticket)
> +    http.endheaders()
> +
> +    r = http.getresponse()
> +    if r.status == 200:
> +        # New imageio never needs authentication.
> +        h['needs_auth'] = False
> +
> +        j = json.loads(r.read())
> +        h['can_zero'] = "zero" in j['features']
> +        h['can_trim'] = "trim" in j['features']
> +        h['can_flush'] = "flush" in j['features']
> +
> +    # Old imageio servers returned either 405 Method Not Allowed or
> +    # 204 No Content (with an empty body).  If we see that we leave
> +    # all the features as False and they will be emulated.
> +    elif r.status == 405 or r.status == 204:
> +        pass
> +
> +    else:
> +        raise RuntimeError("could not use OPTIONS request: %d: %s" %
> +                           (r.status, r.reason))
> +
> +def can_trim(h):
> +    get_options(h)
> +    return h['can_trim']
> +
> +def can_flush(h):
> +    get_options(h)
> +    return h['can_flush']
> +
> +def get_size(h):
> +    return params['disk_size']
> +
> +# For documentation see:
> +# https://github.com/oVirt/ovirt-imageio/blob/master/docs/random-io.md
> +# For examples of working code to read/write from the server, see:
> +# 
> https://github.com/oVirt/ovirt-imageio/blob/master/daemon/test/server_test.py
> +
> +def pread(h, count, offset):
> +    http = h['http']
> +    transfer = h['transfer']
> +    transfer_service = h['transfer_service']
> +
> +    http.putrequest("GET", h['path'])
> +    # Authorization is only needed for old imageio.
> +    if h['needs_auth']:
> +        http.putheader("Authorization", transfer.signed_ticket)
> +    http.putheader("Range", "bytes=%d-%d" % (offset, offset+count-1))
> +    http.endheaders()
> +
> +    r = http.getresponse()
> +    # 206 = HTTP Partial Content.
> +    if r.status != 206:
> +        h['transfer_service'].pause()
> +        h['failed'] = True
> +        raise RuntimeError("could not read sector (%d, %d): %d: %s" %
> +                           (offset, count, r.status, r.reason))
> +    return r.read()
> +
> +def pwrite(h, buf, offset):
> +    http = h['http']
> +    transfer = h['transfer']
> +    transfer_service = h['transfer_service']
> +
> +    count = len(buf)
> +    h['highestwrite'] = max(h['highestwrite'], offset+count)
> +
> +    http.putrequest("PUT", h['path'] + "?flush=n")
> +    # Authorization is only needed for old imageio.
> +    if h['needs_auth']:
> +        http.putheader("Authorization", transfer.signed_ticket)
> +    # The oVirt server only uses the first part of the range, and the
> +    # content-length.
> +    http.putheader("Content-Range", "bytes %d-%d/*" % (offset, 
> offset+count-1))
> +    http.putheader("Content-Length", str(count))
> +    http.endheaders()
> +    http.send(buf)
> +
> +    r = http.getresponse()
> +    if r.status != 200:
> +        transfer_service.pause()
> +        h['failed'] = True
> +        raise RuntimeError("could not write sector (%d, %d): %d: %s" %
> +                           (offset, count, r.status, r.reason))
> +
> +def zero(h, count, offset, may_trim):
> +    http = h['http']
> +    transfer = h['transfer']
> +    transfer_service = h['transfer_service']
> +
> +    # Unlike the trim and flush calls, there is no 'can_zero' method
> +    # so nbdkit could call this even if the server doesn't support
> +    # zeroing.  If this is the case we must emulate.
> +    if not h['can_zero']:
> +        emulate_zero(h, count, offset)
> +        return
> +
> +    # Construct the JSON request for zeroing.
> +    buf = json.dumps({'op': "zero",
> +                      'offset': offset,
> +                      'size': count,
> +                      'flush': False}).encode()
> +
> +    http.putrequest("PATCH", h['path'])
> +    http.putheader("Content-Type", "application/json")
> +    http.putheader("Content-Length", len(buf))
> +    http.endheaders()
> +    http.send(buf)
> +
> +    r = http.getresponse()
> +    if r.status != 200:
> +        transfer_service.pause()
> +        h['failed'] = True
> +        raise RuntimeError("could not zero sector (%d, %d): %d: %s" %
> +                           (offset, count, r.status, r.reason))
> +
> +def emulate_zero(h, count, offset):
> +    # qemu-img convert starts by trying to zero/trim the whole device.
> +    # Since we've just created a new disk it's safe to ignore these
> +    # requests as long as they are smaller than the highest write seen.
> +    # After that we must emulate them with writes.
> +    if offset+count < h['highestwrite']:
> +        http.putrequest("PUT", h['path'])
> +        # Authorization is only needed for old imageio.
> +        if h['needs_auth']:
> +            http.putheader("Authorization", transfer.signed_ticket)
> +        http.putheader("Content-Range",
> +                       "bytes %d-%d/*" % (offset, offset+count-1))
> +        http.putheader("Content-Length", str(count))
> +        http.endheaders()
> +
> +        buf = bytearray(128*1024)
> +        while count > len(buf):
> +            http.send(buf)
> +            count -= len(buf)
> +        http.send(buffer(buf, 0, count))
> +
> +        r = http.getresponse()
> +        if r.status != 200:
> +            transfer_service.pause()
> +            h['failed'] = True
> +            raise RuntimeError("could not write zeroes (%d, %d): %d: %s" %
> +                               (offset, count, r.status, r.reason))
> +
> +def trim(h, count, offset):
> +    http = h['http']
> +    transfer = h['transfer']
> +    transfer_service = h['transfer_service']
> +
> +    # Construct the JSON request for trimming.
> +    buf = json.dumps({'op': "trim",
> +                      'offset': offset,
> +                      'size': count,
> +                      'flush': False}).encode()
> +
> +    http.putrequest("PATCH", h['path'])
> +    http.putheader("Content-Type", "application/json")
> +    http.putheader("Content-Length", len(buf))
> +    http.endheaders()
> +    http.send(buf)
> +
> +    r = http.getresponse()
> +    if r.status != 200:
> +        transfer_service.pause()
> +        h['failed'] = True
> +        raise RuntimeError("could not trim sector (%d, %d): %d: %s" %
> +                           (offset, count, r.status, r.reason))
> +
> +def flush(h):
> +    http = h['http']
> +    transfer = h['transfer']
> +    transfer_service = h['transfer_service']
> +
> +    # Construct the JSON request for flushing.
> +    buf = json.dumps({'op': "flush"}).encode()
> +
> +    http.putrequest("PATCH", h['path'])
> +    http.putheader("Content-Type", "application/json")
> +    http.putheader("Content-Length", len(buf))
> +    http.endheaders()
> +    http.send(buf)
> +
> +    r = http.getresponse()
> +    if r.status != 200:
> +        transfer_service.pause()
> +        h['failed'] = True
> +        raise RuntimeError("could not flush: %d: %s" % (r.status, r.reason))
> +
> +def delete_disk_on_failure(h):
> +    disk_service = h['disk_service']
> +    disk_service.remove()
> +
> +def close(h):
> +    http = h['http']
> +    connection = h['connection']
> +
> +    # This is sometimes necessary because python doesn't set up
> +    # sys.stderr to be line buffered and so debug, errors or
> +    # exceptions printed previously might not be emitted before the
> +    # plugin exits.
> +    sys.stderr.flush()
> +
> +    # If the connection failed earlier ensure we clean up the disk.
> +    if h['failed']:
> +        delete_disk_on_failure(h)
> +        connection.close()
> +        return
> +
> +    try:
> +        # Issue a flush request on close so that the data is written to
> +        # persistent store before we create the VM.
> +        if h['can_flush']:
> +            flush(h)
> +
> +        http.close()
> +
> +        disk = h['disk']
> +        transfer_service = h['transfer_service']
> +
> +        transfer_service.finalize()
> +
> +        # Wait until the transfer disk job is completed since
> +        # only then we can be sure the disk is unlocked.  As this
> +        # code is not very clear, what's happening is that we are
> +        # waiting for the transfer object to cease to exist, which
> +        # falls through to the exception case and then we can
> +        # continue.
> +        endt = time.time() + timeout
> +        try:
> +            while True:
> +                time.sleep(1)
> +                tmp = transfer_service.get()
> +                if time.time() > endt:
> +                    raise RuntimeError("timed out waiting for transfer " +
> +                                       "to finalize")
> +        except sdk.NotFoundError:
> +            pass
> +
> +        # Write the disk ID file.  Only do this on successful completion.
> +        with builtins.open(params['diskid_file'], 'w') as fp:
> +            fp.write(disk.id)
> +
> +    except:
> +        # Otherwise on any failure we must clean up the disk.
> +        delete_disk_on_failure(h)
> +        raise
> +
> +    connection.close()
> diff --git a/v2v/rhv-upload-precheck.py b/v2v/rhv-upload-precheck.py
> new file mode 100644
> index 000000000..2798a29dd
> --- /dev/null
> +++ b/v2v/rhv-upload-precheck.py
> @@ -0,0 +1,73 @@
> +# -*- python -*-
> +# oVirt or RHV pre-upload checks used by ‘virt-v2v -o rhv-upload’
> +# Copyright (C) 2018 Red Hat Inc.
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License along
> +# with this program; if not, write to the Free Software Foundation, Inc.,
> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +
> +import json
> +import logging
> +import sys
> +import time
> +
> +from http.client import HTTPSConnection
> +from urllib.parse import urlparse
> +
> +import ovirtsdk4 as sdk
> +import ovirtsdk4.types as types
> +
> +# Parameters are passed in via a JSON doc from the OCaml code.
> +# Because this Python code ships embedded inside virt-v2v there
> +# is no formal API here.
> +params = None
> +
> +if len(sys.argv) != 2:
> +    raise RuntimeError("incorrect number of parameters")
> +
> +# Parameters are passed in via a JSON document.
> +with open(sys.argv[1], 'r') as fp:
> +    params = json.load(fp)
> +
> +# What is passed in is a password file, read the actual password.
> +with open(params['output_password'], 'r') as fp:
> +    output_password = fp.read()
> +output_password = output_password.rstrip()
> +
> +# Parse out the username from the output_conn URL.
> +parsed = urlparse(params['output_conn'])
> +username = parsed.username or "admin@internal"
> +
> +# Connect to the server.
> +connection = sdk.Connection(
> +    url = params['output_conn'],
> +    username = username,
> +    password = output_password,
> +    ca_file = params['rhv_cafile'],
> +    log = logging.getLogger(),
> +    insecure = params['insecure'],
> +)
> +
> +system_service = connection.system_service()
> +
> +# Find if a virtual machine already exists with that name.
> +vms_service = system_service.vms_service()
> +vms = vms_service.list(
> +    search = ("name=%s" % params['output_name']),
> +)
> +if len(vms) > 0:
> +    vm = vms[0]
> +    raise RuntimeError("VM already exists with name ‘%s’, id ‘%s’" %
> +                       (params['output_name'], vm.id))
> +
> +# Otherwise everything is OK, exit with no error.
> diff --git a/v2v/test-v2v-o-rhv-upload-oo-query.sh 
> b/v2v/test-v2v-o-rhv-upload-oo-query.sh
> new file mode 100755
> index 000000000..29d69e1c1
> --- /dev/null
> +++ b/v2v/test-v2v-o-rhv-upload-oo-query.sh
> @@ -0,0 +1,38 @@
> +#!/bin/bash -
> +# libguestfs virt-v2v test script
> +# Copyright (C) 2018 Red Hat Inc.
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write to the Free Software
> +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 
> USA.
> +
> +# Test -oo "?" option.
> +
> +set -e
> +
> +$TEST_FUNCTIONS
> +skip_if_skipped
> +
> +export VIRT_TOOLS_DATA_DIR="$top_srcdir/test-data/fake-virt-tools"
> +export VIRTIO_WIN="$top_srcdir/test-data/fake-virtio-win"
> +
> +f=test-v2v-o-rhv-upload-oo-query.actual
> +rm -f $f
> +
> +$VG virt-v2v --debug-gc \
> +    -o rhv-upload -oo "?" > $f
> +
> +grep -- "-oo rhv-cafile" $f
> +grep -- "-oo rhv-verifypeer" $f
> +
> +rm $f
> diff --git a/v2v/test-v2v-python-syntax.sh b/v2v/test-v2v-python-syntax.sh
> new file mode 100755
> index 000000000..b167f4610
> --- /dev/null
> +++ b/v2v/test-v2v-python-syntax.sh
> @@ -0,0 +1,45 @@
> +#!/bin/bash -
> +# libguestfs
> +# Copyright (C) 2018 Red Hat Inc.
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write to the Free Software
> +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 
> USA.
> +
> +set -e
> +
> +$TEST_FUNCTIONS
> +skip_if_skipped
> +
> +# Files to check.
> +files="rhv-upload-createvm.py rhv-upload-plugin.py rhv-upload-precheck.py"
> +
> +# Base version of Python.
> +python=python3
> +
> +# Checks the files are syntactically correct, but not very much else.
> +for f in $files; do
> +    $python -m py_compile $f
> +done
> +
> +# Checks the files correspond to PEP8 coding style.
> +# https://www.python.org/dev/peps/pep-0008/
> +if $python-pep8 --version >/dev/null 2>&1; then
> +    for f in $files; do
> +        # Ignore:
> +        # E226 missing whitespace around arithmetic operator
> +        # E251 unexpected spaces around keyword / parameter equals
> +        # E302 expected 2 blank lines, found 1
> +        $python-pep8 --ignore=E226,E251,E302 $f
> +    done
> +fi
> diff --git a/v2v/virt-v2v.pod b/v2v/virt-v2v.pod
> index 5102e6841..4eab5a55d 100644
> --- a/v2v/virt-v2v.pod
> +++ b/v2v/virt-v2v.pod
> @@ -6,15 +6,18 @@ virt-v2v - Convert a guest to use KVM
>  
>   virt-v2v -ic vpx://vcenter.example.com/Datacenter/esxi vmware_guest
>  
> - virt-v2v -ic vpx://vcenter.example.com/Datacenter/esxi vmware_guest \
> -   -o rhv -os rhv.nfs:/export_domain --bridge ovirtmgmt
> -
>   virt-v2v -i libvirtxml guest-domain.xml -o local -os /var/tmp
>  
>   virt-v2v -i disk disk.img -o local -os /var/tmp
>  
>   virt-v2v -i disk disk.img -o glance
>  
> + virt-v2v -ic vpx://vcenter.example.com/Datacenter/esxi vmware_guest \
> +   -o rhv-upload -oc https://ovirt-engine.example.com/ovirt-engine/api \
> +   -os ovirt-data -op /tmp/ovirt-admin-password -of raw \
> +   -oo rhv-cafile=/tmp/ca.pem -oo rhv-direct \
> +   --bridge ovirtmgmt
> +
>   virt-v2v -ic qemu:///system qemu_guest --in-place
>  
>  =head1 DESCRIPTION
> @@ -52,20 +55,18 @@ For more information see L</INPUT FROM VMWARE VCENTER 
> SERVER> below.
>  =head2 Convert from VMware to RHV/oVirt
>  
>  This is the same as the previous example, except you want to send the
> -guest to a RHV-M Export Storage Domain which is located remotely
> -(over NFS) at C<rhv.nfs:/export_domain>.  If you are unclear about
> -the location of the Export Storage Domain you should check the
> -settings on your RHV-M management console.  Guest network
> +guest to a RHV Data Domain using the RHV REST API.  Guest network
>  interface(s) are connected to the target network called C<ovirtmgmt>.
>  
>   virt-v2v -ic vpx://vcenter.example.com/Datacenter/esxi vmware_guest \
> -   -o rhv -os rhv.nfs:/export_domain --bridge ovirtmgmt
> +   -o rhv-upload -oc https://ovirt-engine.example.com/ovirt-engine/api \
> +   -os ovirt-data -op /tmp/ovirt-admin-password -of raw \
> +   -oo rhv-cafile=/tmp/ca.pem -oo rhv-direct \
> +   --bridge ovirtmgmt
>  
>  In this case the host running virt-v2v acts as a B<conversion server>.
>  
> -Note that after conversion, the guest will appear in the RHV-M Export
> -Storage Domain, from where you will need to import it using the RHV-M
> -user interface.  (See L</OUTPUT TO RHV>).
> +For more information see L</OUTPUT TO RHV> below.
>  
>  =head2 Convert from ESXi hypervisor over SSH to local libvirt
>  
> @@ -129,9 +130,9 @@ qemu, do:
>   Xen ───▶│ -i libvirt ──▶ │            │     │  (default) │
>   ... ───▶│  (default) │   │            │ ──┐ └────────────┘
>           └────────────┘   │            │ ─┐└──────▶ -o glance
> - -i libvirtxml ─────────▶ │            │ ┐└─────────▶ -o rhv
> - -i vmx ────────────────▶ │            │ └──────────▶ -o vdsm
> -                          └────────────┘
> + -i libvirtxml ─────────▶ │            │ ┐├─────────▶ -o rhv
> + -i vmx ────────────────▶ │            │ │└─────────▶ -o vdsm
> +                          └────────────┘ └──────────▶ -o rhv-upload
>  
>  Virt-v2v has a number of possible input and output modes, selected
>  using the I<-i> and I<-o> options.  Only one input and output mode can
> @@ -164,8 +165,9 @@ libvirt configuration file (mainly for testing).
>  I<-o qemu> writes to a local disk image with a shell script for
>  booting the guest directly in qemu (mainly for testing).
>  
> -I<-o rhv> is used to write to a RHV / oVirt target.  I<-o vdsm>
> -is only used when virt-v2v runs under VDSM control.
> +I<-o rhv-upload> is used to write to a RHV / oVirt target.  I<-o rhv>
> +is a legacy method to write to RHV / oVirt E<lt> 4.2.  I<-o vdsm> is
> +only used when virt-v2v runs under VDSM control.
>  
>  I<--in-place> instructs virt-v2v to customize the guest OS in the input
>  virtual machine, instead of creating a new VM in the target hypervisor.
> @@ -550,6 +552,10 @@ written.
>  
>  This is the same as I<-o rhv>.
>  
> +=item B<-o> B<ovirt-upload>
> +
> +This is the same as I<-o rhv-upload>.
> +
>  =item B<-o> B<qemu>
>  
>  Set the output method to I<qemu>.
> @@ -574,6 +580,16 @@ I<-os> parameter must also be used to specify the 
> location of the
>  Export Storage Domain.  Note this does not actually import the guest
>  into RHV.  You have to do that manually later using the UI.
>  
> +See L</OUTPUT TO RHV (OLD METHOD)> below.
> +
> +=item B<-o> B<rhv-upload>
> +
> +Set the output method to I<rhv-upload>.
> +
> +The converted guest is written directly to a RHV Data Domain.
> +This is a faster method than I<-o rhv>, but requires oVirt
> +or RHV E<ge> 4.2.
> +
>  See L</OUTPUT TO RHV> below.
>  
>  =item B<-o> B<vdsm>
> @@ -615,7 +631,33 @@ the output name is the same as the input name.
>  Set output option(s) related to the current output mode.
>  To display short help on what options are available you can use:
>  
> - virt-v2v -o vdsm -oo "?"
> + virt-v2v -o rhv-upload -oo "?"
> +
> +=item B<-oo rhv-cafile=>F<ca.pem>
> +
> +For I<-o rhv-upload> (L<OUTPUT TO RHV>) only, the F<ca.pem> file
> +(Certificate Authority), copied from F</etc/pki/ovirt-engine/ca.pem>
> +on the oVirt engine.
> +
> +=item B<-oo rhv-cluster=>C<CLUSTERNAME>
> +
> +For I<-o rhv-upload> (L<OUTPUT TO RHV>) only, set the RHV Cluster
> +Name.  If not given it uses C<Default>.
> +
> +=item B<-oo rhv-direct>
> +
> +For I<-o rhv-upload> (L<OUTPUT TO RHV>) only, if this option is given
> +then virt-v2v will attempt to directly upload the disk to the oVirt
> +node, otherwise it will proxy the upload through the oVirt engine.
> +Direct upload requires that you have network access to the oVirt
> +nodes.  Non-direct upload is slightly slower but should work in all
> +situations.
> +
> +=item B<-oo rhv-verifypeer>
> +
> +For I<-o rhv-upload> (L<OUTPUT TO RHV>) only, verify the oVirt/RHV
> +server’s identity by checking the server‘s certificate against the
> +Certificate Authority.
>  
>  =item B<-oo vdsm-compat=0.10>
>  
> @@ -1884,6 +1926,68 @@ Define the final guest in libvirt:
>  
>  =head1 OUTPUT TO RHV
>  
> +This new method to upload guests to oVirt or RHV directly via the REST
> +API requires oVirt/RHV E<ge> 4.2.
> +
> +You need to specify I<-o rhv-upload> as well as the following extra
> +parameters:
> +
> +=over 4
> +
> +=item I<-oc> C<https://ovirt-engine.example.com/ovirt-engine/api>
> +
> +The URL of the REST API which is usually the server name with
> +C</ovirt-engine/api> appended, but might be different if you installed
> +oVirt Engine on a different path.
> +
> +You can optionally add a username and port number to the URL.  If the
> +username is not specified then virt-v2v defaults to using
> +C<admin@internal> which is the typical superuser account for oVirt
> +instances.
> +
> +=item I<-of raw>
> +
> +Currently you must use I<-of raw> and you cannot use I<-oa preallocated>.
> +
> +These restrictions will be loosened in a future version.
> +
> +=item I<-op> F<password-file>
> +
> +A file containing a password to be used when connecting to the oVirt
> +engine.  Note the file should contain the whole password, B<without
> +any trailing newline>, and for security the file should have mode
> +C<0600> so that others cannot read it.
> +
> +=item I<-os> C<ovirt-data>
> +
> +The storage domain.
> +
> +=item I<-oo rhv-cafile=>F<ca.pem>
> +
> +The F<ca.pem> file (Certificate Authority), copied from
> +F</etc/pki/ovirt-engine/ca.pem> on the oVirt engine.
> +
> +=item I<-oo rhv-cluster=>C<CLUSTERNAME>
> +
> +Set the RHV Cluster Name.  If not given it uses C<Default>.
> +
> +=item I<-oo rhv-direct>
> +
> +If this option is given then virt-v2v will attempt to directly upload
> +the disk to the oVirt node, otherwise it will proxy the upload through
> +the oVirt engine.  Direct upload requires that you have network access
> +to the oVirt nodes.  Non-direct upload is slightly slower but should
> +work in all situations.
> +
> +=item I<-oo rhv-verifypeer>
> +
> +Verify the oVirt/RHV server’s identity by checking the server‘s
> +certificate against the Certificate Authority.
> +
> +=back
> +
> +=head1 OUTPUT TO RHV (OLD METHOD)
> +
>  This section only applies to the I<-o rhv> output mode.  If you use
>  virt-v2v from the RHV-M user interface, then behind the scenes the
>  import is managed by VDSM using the I<-o vdsm> output mode (which end
> -- 
> 2.16.2
> 


-- 
Tomáš Golembiovský <tgole...@redhat.com>

_______________________________________________
Libguestfs mailing list
Libguestfs@redhat.com
https://www.redhat.com/mailman/listinfo/libguestfs

Reply via email to