[Qemu-block] [PATCH v8 4/4] blockdev: Modularize nfs block driver

[Colin Lord]

Modularizes the nfs block driver so that it gets dynamically loaded.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefa...@redhat.com>
---
 block/Makefile.objs | 1 +
 configure   | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 595f366..fa4d8b8 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -28,6 +28,7 @@ block-obj-y += crypto.o
 common-obj-y += stream.o
 common-obj-y += backup.o
 
+nfs.o-libs := $(LIBNFS_LIBS)
 iscsi.o-cflags := $(LIBISCSI_CFLAGS)
 iscsi.o-libs   := $(LIBISCSI_LIBS)
 curl.o-cflags  := $(CURL_CFLAGS)
diff --git a/configure b/configure
index 8d84919..976cff7 100755
--- a/configure
+++ b/configure
@@ -4561,7 +4561,6 @@ if test "$libnfs" != "no" ; then
   if $pkg_config --atleast-version=1.9.3 libnfs; then
 libnfs="yes"
 libnfs_libs=$($pkg_config --libs libnfs)
-LIBS="$LIBS $libnfs_libs"
   else
 if test "$libnfs" = "yes" ; then
   feature_not_found "libnfs" "Install libnfs devel >= 1.9.3"
@@ -5320,7 +5319,8 @@ if test "$libiscsi" = "yes" ; then
 fi
 
 if test "$libnfs" = "yes" ; then
-  echo "CONFIG_LIBNFS=y" >> $config_host_mak
+  echo "CONFIG_LIBNFS=m" >> $config_host_mak
+  echo "LIBNFS_LIBS=$libnfs_libs" >> $config_host_mak
 fi
 
 if test "$seccomp" = "yes"; then
-- 
2.5.5

[Qemu-block] [PATCH v8 0/4] Dynamic module loading for block drivers

[Colin Lord]

v8:
- Add note to commit message about dmg not being modularized

v7:
- Add ifdef around qemu_iscsi_opts in vl.c (first patch)

v6:
- Fix bug so that users can specify a modularized driver on the cli
  without qemu exiting
- Remove extra lines from Makefile
- Add patch to modularize NFS

v5:
- No format drivers are modularized, therefore the probe functions are
  all being left completely untouched.
- Remove dmg from block-obj-m since it is not a target of the
  modularization effort.
- Modify module_block.py to only include the library name and protocol
  name fields in the generated struct. The other fields are no longer
  necessary for the drivers that are being modularized.

v4:
- Fix indentation of the generated header file module_block.h
- Drivers and probe functions are now all located in the block/
  directory, rather than being split between block/ and block/probe/. In
  addition the header files for each probe/driver pair are in the block/
  directory, not the include/block/driver/ directory (which no longer
  exists).
- Since the probe files are in block/ now, they follow the naming
  pattern of format-probe.c
- Renamed crypto probe file to be crypto-probe.c, luks is no longer in
  the filename
- Fixed formatting of parallels_probe() function header
- Enforced consistent naming convention for the probe functions. They
  now follow the pattern bdrv_format_probe().

Colin Lord (2):
  blockdev: prepare iSCSI block driver for dynamic loading
  blockdev: Modularize nfs block driver

Marc Mari (2):
  blockdev: Add dynamic generation of module_block.h
  blockdev: Add dynamic module loading for block drivers

 Makefile|  10 ++--
 block.c |  62 ---
 block/Makefile.objs |   4 +-
 block/iscsi.c   |  36 --
 configure   |   4 +-
 include/qemu/module.h   |   3 ++
 scripts/modules/module_block.py | 108 
 util/module.c   |  38 --
 vl.c|  40 +++
 9 files changed, 228 insertions(+), 77 deletions(-)
 create mode 100644 scripts/modules/module_block.py

-- 
2.5.5

Re: [Qemu-block] [Qemu-devel] [PATCH v7 3/4] blockdev: Add dynamic module loading for block drivers

[Colin Lord]

On 08/11/2016 09:29 PM, Fam Zheng wrote:
> On Thu, 08/11 12:03, Colin Lord wrote:
>> On 08/10/2016 11:23 PM, Fam Zheng wrote:
>>> On Wed, 08/10 21:06, Max Reitz wrote:
>>>> On 10.08.2016 21:04, Colin Lord wrote:
>>>>> On 08/10/2016 02:37 PM, Max Reitz wrote:
>>>>>> On 08.08.2016 20:07, Colin Lord wrote:
>>>>>>> From: Marc Mari <mar...@redhat.com>
>>>>>>>
>>>>>>> Extend the current module interface to allow for block drivers to be
>>>>>>> loaded dynamically on request. The only block drivers that can be
>>>>>>> converted into modules are the drivers that don't perform any init
>>>>>>> operation except for registering themselves.
>>>>>>>
>>>>>>> In addition, only the protocol drivers are being modularized, as they
>>>>>>> are the only ones which see significant performance benefits. The format
>>>>>>> drivers do not generally link to external libraries, so modularizing
>>>>>>> them is of no benefit from a performance perspective.
>>>>>>>
>>>>>>> All the necessary module information is located in a new structure found
>>>>>>> in module_block.h
>>>>>>>
>>>>>>> Signed-off-by: Marc Marí <mar...@redhat.com>
>>>>>>> Signed-off-by: Colin Lord <cl...@redhat.com>
>>>>>>> Reviewed-by: Stefan Hajnoczi <stefa...@redhat.com>
>>>>>>> ---
>>>>>>>  Makefile  |  3 ---
>>>>>>>  block.c   | 62 
>>>>>>> +--
>>>>>>>  block/Makefile.objs   |  3 +--
>>>>>>>  include/qemu/module.h |  3 +++
>>>>>>>  util/module.c | 38 +--
>>>>>>>  5 files changed, 70 insertions(+), 39 deletions(-)
>>>>>>>
>>>>>>
>>>>>> [...]
>>>>>>
>>>>>>> diff --git a/block.c b/block.c
>>>>>>> index 30d64e6..6c5e249 100644
>>>>>>> --- a/block.c
>>>>>>> +++ b/block.c
>>>>>>> @@ -26,6 +26,7 @@
>>>>>>>  #include "block/block_int.h"
>>>>>>>  #include "block/blockjob.h"
>>>>>>>  #include "qemu/error-report.h"
>>>>>>> +#include "module_block.h"
>>>>>>>  #include "qemu/module.h"
>>>>>>>  #include "qapi/qmp/qerror.h"
>>>>>>>  #include "qapi/qmp/qbool.h"
>>>>>>> @@ -241,17 +242,40 @@ BlockDriverState *bdrv_new(void)
>>>>>>>  return bs;
>>>>>>>  }
>>>>>>>  
>>>>>>> -BlockDriver *bdrv_find_format(const char *format_name)
>>>>>>> +static BlockDriver *bdrv_do_find_format(const char *format_name)
>>>>>>>  {
>>>>>>>  BlockDriver *drv1;
>>>>>>> +
>>>>>>>  QLIST_FOREACH(drv1, _drivers, list) {
>>>>>>>  if (!strcmp(drv1->format_name, format_name)) {
>>>>>>>  return drv1;
>>>>>>>  }
>>>>>>>  }
>>>>>>> +
>>>>>>>  return NULL;
>>>>>>>  }
>>>>>>>  
>>>>>>> +BlockDriver *bdrv_find_format(const char *format_name)
>>>>>>> +{
>>>>>>> +BlockDriver *drv1;
>>>>>>> +size_t i;
>>>>>>> +
>>>>>>> +drv1 = bdrv_do_find_format(format_name);
>>>>>>> +if (drv1) {
>>>>>>> +return drv1;
>>>>>>> +}
>>>>>>> +
>>>>>>> +/* The driver isn't registered, maybe we need to load a module */
>>>>>>> +for (i = 0; i < ARRAY_SIZE(block_driver_modules); ++i) {
>>>>>>> +if (!strcmp(block_driver_modules[i].format_name, format_name)) 
>>>>>>> {
>>>>>>> +
>>>>>>> block_module_load_one(block_driver_modules[i].library_name);
>>>>>>> +break;
>>>>>

[Qemu-block] [PATCH v8 3/4] blockdev: Add dynamic module loading for block drivers

[Colin Lord]

From: Marc Mari <mar...@redhat.com>

Extend the current module interface to allow for block drivers to be
loaded dynamically on request. The only block drivers that can be
converted into modules are the drivers that don't perform any init
operation except for registering themselves.

In addition, only the protocol drivers are being modularized, as they
are the only ones which see significant performance benefits. The format
drivers do not generally link to external libraries, so modularizing
them is of no benefit from a performance perspective.

All the necessary module information is located in a new structure found
in module_block.h

This spoils the purpose of 5505e8b76f (block/dmg: make it modular).

Before this patch, if module build is enabled, block-dmg.so is linked to
libbz2, whereas the main binary is not. In downstream, theoretically, it
means only the qemu-block-extra package depends on libbz2, while the
main QEMU package needn't to. With this patch, we (temporarily) change
the case so that the main QEMU depends on libbz2 again.

Signed-off-by: Marc Marí <mar...@redhat.com>
Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefa...@redhat.com>
---
 Makefile  |  3 ---
 block.c   | 62 +--
 block/Makefile.objs   |  3 +--
 include/qemu/module.h |  3 +++
 util/module.c | 38 +--
 5 files changed, 70 insertions(+), 39 deletions(-)

diff --git a/Makefile b/Makefile
index 3d4c97e..6c9881f 100644
--- a/Makefile
+++ b/Makefile
@@ -248,9 +248,6 @@ Makefile: $(version-obj-y) $(version-lobj-y)
 libqemustub.a: $(stub-obj-y)
 libqemuutil.a: $(util-obj-y)
 
-block-modules = $(foreach o,$(block-obj-m),"$(basename $(subst /,-,$o))",) NULL
-util/module.o-cflags = -D'CONFIG_BLOCK_MODULES=$(block-modules)'
-
 ##
 
 qemu-img.o: qemu-img-cmds.h
diff --git a/block.c b/block.c
index 30d64e6..6c5e249 100644
--- a/block.c
+++ b/block.c
@@ -26,6 +26,7 @@
 #include "block/block_int.h"
 #include "block/blockjob.h"
 #include "qemu/error-report.h"
+#include "module_block.h"
 #include "qemu/module.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/qmp/qbool.h"
@@ -241,17 +242,40 @@ BlockDriverState *bdrv_new(void)
 return bs;
 }
 
-BlockDriver *bdrv_find_format(const char *format_name)
+static BlockDriver *bdrv_do_find_format(const char *format_name)
 {
 BlockDriver *drv1;
+
 QLIST_FOREACH(drv1, _drivers, list) {
 if (!strcmp(drv1->format_name, format_name)) {
 return drv1;
 }
 }
+
 return NULL;
 }
 
+BlockDriver *bdrv_find_format(const char *format_name)
+{
+BlockDriver *drv1;
+size_t i;
+
+drv1 = bdrv_do_find_format(format_name);
+if (drv1) {
+return drv1;
+}
+
+/* The driver isn't registered, maybe we need to load a module */
+for (i = 0; i < ARRAY_SIZE(block_driver_modules); ++i) {
+if (!strcmp(block_driver_modules[i].format_name, format_name)) {
+block_module_load_one(block_driver_modules[i].library_name);
+break;
+}
+}
+
+return bdrv_do_find_format(format_name);
+}
+
 static int bdrv_is_whitelisted(BlockDriver *drv, bool read_only)
 {
 static const char *whitelist_rw[] = {
@@ -460,6 +484,19 @@ static BlockDriver *find_hdev_driver(const char *filename)
 return drv;
 }
 
+static BlockDriver *bdrv_do_find_protocol(const char *protocol)
+{
+BlockDriver *drv1;
+
+QLIST_FOREACH(drv1, _drivers, list) {
+if (drv1->protocol_name && !strcmp(drv1->protocol_name, protocol)) {
+return drv1;
+}
+}
+
+return NULL;
+}
+
 BlockDriver *bdrv_find_protocol(const char *filename,
 bool allow_protocol_prefix,
 Error **errp)
@@ -468,6 +505,7 @@ BlockDriver *bdrv_find_protocol(const char *filename,
 char protocol[128];
 int len;
 const char *p;
+size_t i;
 
 /* TODO Drivers without bdrv_file_open must be specified explicitly */
 
@@ -494,15 +532,25 @@ BlockDriver *bdrv_find_protocol(const char *filename,
 len = sizeof(protocol) - 1;
 memcpy(protocol, filename, len);
 protocol[len] = '\0';
-QLIST_FOREACH(drv1, _drivers, list) {
-if (drv1->protocol_name &&
-!strcmp(drv1->protocol_name, protocol)) {
-return drv1;
+
+drv1 = bdrv_do_find_protocol(protocol);
+if (drv1) {
+return drv1;
+}
+
+for (i = 0; i < ARRAY_SIZE(block_driver_modules); ++i) {
+if (block_driver_modules[i].protocol_name &&
+!strcmp(block_driver_modules[i].protocol_name, protocol)) {
+block_module_load_one(block_driver_modules[i].library_name);
+break;
  

[Qemu-block] [PATCH v8 2/4] blockdev: Add dynamic generation of module_block.h

[Colin Lord]

From: Marc Mari <mar...@redhat.com>

To simplify the addition of new block modules, add a script that generates
module_block.h automatically from the modules' source code.

This script assumes that the QEMU coding style rules are followed.

Signed-off-by: Marc Marí <mar...@redhat.com>
Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefa...@redhat.com>
---
 Makefile|   7 +++
 scripts/modules/module_block.py | 108 
 2 files changed, 115 insertions(+)
 create mode 100644 scripts/modules/module_block.py

diff --git a/Makefile b/Makefile
index 50b4b3a..3d4c97e 100644
--- a/Makefile
+++ b/Makefile
@@ -76,6 +76,8 @@ GENERATED_HEADERS += trace/generated-ust-provider.h
 GENERATED_SOURCES += trace/generated-ust.c
 endif
 
+GENERATED_HEADERS += module_block.h
+
 # Don't try to regenerate Makefile or configure
 # We don't generate any of them
 Makefile: ;
@@ -353,6 +355,11 @@ ivshmem-client$(EXESUF): $(ivshmem-client-obj-y) 
libqemuutil.a libqemustub.a
 ivshmem-server$(EXESUF): $(ivshmem-server-obj-y) libqemuutil.a libqemustub.a
$(call LINK, $^)
 
+module_block.h: $(SRC_PATH)/scripts/modules/module_block.py config-host.mak
+   $(call quiet-command,$(PYTHON) $< $@ \
+   $(addprefix $(SRC_PATH)/,$(patsubst %.mo,%.c,$(block-obj-m))), \
+   "  GEN   $@")
+
 clean:
 # avoid old build problems by removing potentially incorrect old files
rm -f config.mak op-i386.h opc-i386.h gen-op-i386.h op-arm.h opc-arm.h 
gen-op-arm.h
diff --git a/scripts/modules/module_block.py b/scripts/modules/module_block.py
new file mode 100644
index 000..db4fb54
--- /dev/null
+++ b/scripts/modules/module_block.py
@@ -0,0 +1,108 @@
+#!/usr/bin/python
+#
+# Module information generator
+#
+# Copyright Red Hat, Inc. 2015 - 2016
+#
+# Authors:
+#  Marc Mari <mar...@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2.
+# See the COPYING file in the top-level directory.
+
+from __future__ import print_function
+import sys
+import os
+
+def get_string_struct(line):
+data = line.split()
+
+# data[0] -> struct element name
+# data[1] -> =
+# data[2] -> value
+
+return data[2].replace('"', '')[:-1]
+
+def add_module(fheader, library, format_name, protocol_name):
+lines = []
+lines.append('.library_name = "' + library + '",')
+if format_name != "":
+lines.append('.format_name = "' + format_name + '",')
+if protocol_name != "":
+lines.append('.protocol_name = "' + protocol_name + '",')
+
+text = '\n'.join(lines)
+fheader.write('\n{\n' + text + '\n},')
+
+def process_file(fheader, filename):
+# This parser assumes the coding style rules are being followed
+with open(filename, "r") as cfile:
+found_something = False
+found_start = False
+library, _ = os.path.splitext(os.path.basename(filename))
+for line in cfile:
+if found_start:
+line = line.replace('\n', '')
+if line.find(".format_name") != -1:
+format_name = get_string_struct(line)
+elif line.find(".protocol_name") != -1:
+protocol_name = get_string_struct(line)
+elif line == "};":
+add_module(fheader, library, format_name, protocol_name)
+found_start = False
+elif line.find("static BlockDriver") != -1:
+found_something = True
+found_start = True
+format_name = ""
+protocol_name = ""
+
+if not found_something:
+print("No BlockDriver struct found in " + filename + ". \
+Is this really a module?", file=sys.stderr)
+sys.exit(1)
+
+def print_top(fheader):
+fheader.write('''/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
+/*
+ * QEMU Block Module Infrastructure
+ *
+ * Authors:
+ *  Marc Mari   <mar...@redhat.com>
+ */
+
+''')
+
+fheader.write('''#ifndef QEMU_MODULE_BLOCK_H
+#define QEMU_MODULE_BLOCK_H
+
+#include "qemu-common.h"
+
+static const struct {
+const char *format_name;
+const char *protocol_name;
+const char *library_name;
+} block_driver_modules[] = {''')
+
+def print_bottom(fheader):
+fheader.write('''
+};
+
+#endif
+''')
+
+# First argument: output file
+# All other arguments: modules source files (.c)
+output_file = sys.argv[1]
+with open(output_file, 'w') as fheader:
+print_top(fheader)
+
+for filename in sys.argv[2:]:
+if os.path.isfile(filename):
+process_file(fheader, filename)
+else:
+print("File " + filename + " does not exist.", file=sys.stderr)
+sys.exit(1)
+
+print_bottom(fheader)
+
+sys.exit(0)
-- 
2.5.5

Re: [Qemu-block] [Qemu-devel] [PATCH v7 3/4] blockdev: Add dynamic module loading for block drivers

[Colin Lord]

On 08/10/2016 11:23 PM, Fam Zheng wrote:
> On Wed, 08/10 21:06, Max Reitz wrote:
>> On 10.08.2016 21:04, Colin Lord wrote:
>>> On 08/10/2016 02:37 PM, Max Reitz wrote:
>>>> On 08.08.2016 20:07, Colin Lord wrote:
>>>>> From: Marc Mari <mar...@redhat.com>
>>>>>
>>>>> Extend the current module interface to allow for block drivers to be
>>>>> loaded dynamically on request. The only block drivers that can be
>>>>> converted into modules are the drivers that don't perform any init
>>>>> operation except for registering themselves.
>>>>>
>>>>> In addition, only the protocol drivers are being modularized, as they
>>>>> are the only ones which see significant performance benefits. The format
>>>>> drivers do not generally link to external libraries, so modularizing
>>>>> them is of no benefit from a performance perspective.
>>>>>
>>>>> All the necessary module information is located in a new structure found
>>>>> in module_block.h
>>>>>
>>>>> Signed-off-by: Marc Marí <mar...@redhat.com>
>>>>> Signed-off-by: Colin Lord <cl...@redhat.com>
>>>>> Reviewed-by: Stefan Hajnoczi <stefa...@redhat.com>
>>>>> ---
>>>>>  Makefile  |  3 ---
>>>>>  block.c   | 62 
>>>>> +--
>>>>>  block/Makefile.objs   |  3 +--
>>>>>  include/qemu/module.h |  3 +++
>>>>>  util/module.c | 38 +--
>>>>>  5 files changed, 70 insertions(+), 39 deletions(-)
>>>>>
>>>>
>>>> [...]
>>>>
>>>>> diff --git a/block.c b/block.c
>>>>> index 30d64e6..6c5e249 100644
>>>>> --- a/block.c
>>>>> +++ b/block.c
>>>>> @@ -26,6 +26,7 @@
>>>>>  #include "block/block_int.h"
>>>>>  #include "block/blockjob.h"
>>>>>  #include "qemu/error-report.h"
>>>>> +#include "module_block.h"
>>>>>  #include "qemu/module.h"
>>>>>  #include "qapi/qmp/qerror.h"
>>>>>  #include "qapi/qmp/qbool.h"
>>>>> @@ -241,17 +242,40 @@ BlockDriverState *bdrv_new(void)
>>>>>  return bs;
>>>>>  }
>>>>>  
>>>>> -BlockDriver *bdrv_find_format(const char *format_name)
>>>>> +static BlockDriver *bdrv_do_find_format(const char *format_name)
>>>>>  {
>>>>>  BlockDriver *drv1;
>>>>> +
>>>>>  QLIST_FOREACH(drv1, _drivers, list) {
>>>>>  if (!strcmp(drv1->format_name, format_name)) {
>>>>>  return drv1;
>>>>>  }
>>>>>  }
>>>>> +
>>>>>  return NULL;
>>>>>  }
>>>>>  
>>>>> +BlockDriver *bdrv_find_format(const char *format_name)
>>>>> +{
>>>>> +BlockDriver *drv1;
>>>>> +size_t i;
>>>>> +
>>>>> +drv1 = bdrv_do_find_format(format_name);
>>>>> +if (drv1) {
>>>>> +return drv1;
>>>>> +}
>>>>> +
>>>>> +/* The driver isn't registered, maybe we need to load a module */
>>>>> +for (i = 0; i < ARRAY_SIZE(block_driver_modules); ++i) {
>>>>> +if (!strcmp(block_driver_modules[i].format_name, format_name)) {
>>>>> +block_module_load_one(block_driver_modules[i].library_name);
>>>>> +break;
>>>>> +}
>>>>> +}
>>>>> +
>>>>> +return bdrv_do_find_format(format_name);
>>>>> +}
>>>>> +
>>>>
>>>> Did you reintroduce this function for dmg? I thought Fam is taking care
>>>> of that? I'm confused as to how Fam's patch for dmg and this series are
>>>> supposed to interact; the fact that the script added in patch 2 breaks
>>>> down with Fam's patch isn't exactly helping...
>>>>
>>>> Hm, so is this series now supposed to be applied without Fam's patch
>>>> with the idea of sorting dmg out later on?
>>>>
>>>> Max
>>>>
>>>>>  static int bdr

Re: [Qemu-block] [Qemu-devel] [PATCH v7 3/4] blockdev: Add dynamic module loading for block drivers

[Colin Lord]

On 08/10/2016 03:24 PM, Colin Lord wrote:
> On 08/10/2016 03:04 PM, Colin Lord wrote:
>> On 08/10/2016 02:37 PM, Max Reitz wrote:
>>> On 08.08.2016 20:07, Colin Lord wrote:
>>>> From: Marc Mari <mar...@redhat.com>
>>>>
>>>> Extend the current module interface to allow for block drivers to be
>>>> loaded dynamically on request. The only block drivers that can be
>>>> converted into modules are the drivers that don't perform any init
>>>> operation except for registering themselves.
>>>>
>>>> In addition, only the protocol drivers are being modularized, as they
>>>> are the only ones which see significant performance benefits. The format
>>>> drivers do not generally link to external libraries, so modularizing
>>>> them is of no benefit from a performance perspective.
>>>>
>>>> All the necessary module information is located in a new structure found
>>>> in module_block.h
>>>>
>>>> Signed-off-by: Marc Marí <mar...@redhat.com>
>>>> Signed-off-by: Colin Lord <cl...@redhat.com>
>>>> Reviewed-by: Stefan Hajnoczi <stefa...@redhat.com>
>>>> ---
>>>>  Makefile  |  3 ---
>>>>  block.c   | 62 
>>>> +--
>>>>  block/Makefile.objs   |  3 +--
>>>>  include/qemu/module.h |  3 +++
>>>>  util/module.c | 38 +--
>>>>  5 files changed, 70 insertions(+), 39 deletions(-)
>>>>
>>>
>>> [...]
>>>
>>>> diff --git a/block.c b/block.c
>>>> index 30d64e6..6c5e249 100644
>>>> --- a/block.c
>>>> +++ b/block.c
>>>> @@ -26,6 +26,7 @@
>>>>  #include "block/block_int.h"
>>>>  #include "block/blockjob.h"
>>>>  #include "qemu/error-report.h"
>>>> +#include "module_block.h"
>>>>  #include "qemu/module.h"
>>>>  #include "qapi/qmp/qerror.h"
>>>>  #include "qapi/qmp/qbool.h"
>>>> @@ -241,17 +242,40 @@ BlockDriverState *bdrv_new(void)
>>>>  return bs;
>>>>  }
>>>>  
>>>> -BlockDriver *bdrv_find_format(const char *format_name)
>>>> +static BlockDriver *bdrv_do_find_format(const char *format_name)
>>>>  {
>>>>  BlockDriver *drv1;
>>>> +
>>>>  QLIST_FOREACH(drv1, _drivers, list) {
>>>>  if (!strcmp(drv1->format_name, format_name)) {
>>>>  return drv1;
>>>>  }
>>>>  }
>>>> +
>>>>  return NULL;
>>>>  }
>>>>  
>>>> +BlockDriver *bdrv_find_format(const char *format_name)
>>>> +{
>>>> +BlockDriver *drv1;
>>>> +size_t i;
>>>> +
>>>> +drv1 = bdrv_do_find_format(format_name);
>>>> +if (drv1) {
>>>> +return drv1;
>>>> +}
>>>> +
>>>> +/* The driver isn't registered, maybe we need to load a module */
>>>> +for (i = 0; i < ARRAY_SIZE(block_driver_modules); ++i) {
>>>> +if (!strcmp(block_driver_modules[i].format_name, format_name)) {
>>>> +block_module_load_one(block_driver_modules[i].library_name);
>>>> +break;
>>>> +}
>>>> +}
>>>> +
>>>> +return bdrv_do_find_format(format_name);
>>>> +}
>>>> +
>>>
>>> Did you reintroduce this function for dmg? I thought Fam is taking care
>>> of that? I'm confused as to how Fam's patch for dmg and this series are
>>> supposed to interact; the fact that the script added in patch 2 breaks
>>> down with Fam's patch isn't exactly helping...
>>>
>>> Hm, so is this series now supposed to be applied without Fam's patch
>>> with the idea of sorting dmg out later on?
>>>
>>> Max
>>>
>>>>  static int bdrv_is_whitelisted(BlockDriver *drv, bool read_only)
>>>>  {
>>>>  static const char *whitelist_rw[] = {
>>>
>> I'm not completely sure how Fam's patch is supposed to interact with
>> this series actually. I'm kind of hoping it can be done on top of my
>> patches at some future point, but in either case this revision was not
>> done with the dmg patch in mind. The change in find_format was actually
>> due to a bug I discovered in my patch series (I fixed it in v6, but you
>> may have missed that).
>>
>> Essentially, if a user specifies the driver explicitly as part of their
>> call to qemu, eg driver=gluster, there was a bug in v5 where if the
>> driver was modularized, it would not be found/loaded. So since gluster
>> was modularized, if you said driver=gluster on the command line, the
>> gluster module would not be found. The modules could be found by probing
>> perfectly fine, this only happened when the driver was specified
>> manually. The reason is because the drivers get searched based on the
>> format field if they're specified manually, which means bdrv_find_format
>> gets called when the driver is specified on the command line. This makes
>> it necessary for bdrv_find_format to take into account modularized
>> drivers even though the format drivers are not being modularized. That's
>> also why the format field was added to the module_block header file again.
>>
>> Colin
>>
> Oops, this was meant to be a reply to your response of patch 4/4, in
> case anyone gets confused.
> 
> Colin
> 
Nevermind, looks like I was confused. Seems like my eyes are playing
tricks on me, sorry for the spam.

Re: [Qemu-block] [Qemu-devel] [PATCH v7 3/4] blockdev: Add dynamic module loading for block drivers

[Colin Lord]

On 08/10/2016 03:04 PM, Colin Lord wrote:
> On 08/10/2016 02:37 PM, Max Reitz wrote:
>> On 08.08.2016 20:07, Colin Lord wrote:
>>> From: Marc Mari <mar...@redhat.com>
>>>
>>> Extend the current module interface to allow for block drivers to be
>>> loaded dynamically on request. The only block drivers that can be
>>> converted into modules are the drivers that don't perform any init
>>> operation except for registering themselves.
>>>
>>> In addition, only the protocol drivers are being modularized, as they
>>> are the only ones which see significant performance benefits. The format
>>> drivers do not generally link to external libraries, so modularizing
>>> them is of no benefit from a performance perspective.
>>>
>>> All the necessary module information is located in a new structure found
>>> in module_block.h
>>>
>>> Signed-off-by: Marc Marí <mar...@redhat.com>
>>> Signed-off-by: Colin Lord <cl...@redhat.com>
>>> Reviewed-by: Stefan Hajnoczi <stefa...@redhat.com>
>>> ---
>>>  Makefile  |  3 ---
>>>  block.c   | 62 
>>> +--
>>>  block/Makefile.objs   |  3 +--
>>>  include/qemu/module.h |  3 +++
>>>  util/module.c | 38 +--
>>>  5 files changed, 70 insertions(+), 39 deletions(-)
>>>
>>
>> [...]
>>
>>> diff --git a/block.c b/block.c
>>> index 30d64e6..6c5e249 100644
>>> --- a/block.c
>>> +++ b/block.c
>>> @@ -26,6 +26,7 @@
>>>  #include "block/block_int.h"
>>>  #include "block/blockjob.h"
>>>  #include "qemu/error-report.h"
>>> +#include "module_block.h"
>>>  #include "qemu/module.h"
>>>  #include "qapi/qmp/qerror.h"
>>>  #include "qapi/qmp/qbool.h"
>>> @@ -241,17 +242,40 @@ BlockDriverState *bdrv_new(void)
>>>  return bs;
>>>  }
>>>  
>>> -BlockDriver *bdrv_find_format(const char *format_name)
>>> +static BlockDriver *bdrv_do_find_format(const char *format_name)
>>>  {
>>>  BlockDriver *drv1;
>>> +
>>>  QLIST_FOREACH(drv1, _drivers, list) {
>>>  if (!strcmp(drv1->format_name, format_name)) {
>>>  return drv1;
>>>  }
>>>  }
>>> +
>>>  return NULL;
>>>  }
>>>  
>>> +BlockDriver *bdrv_find_format(const char *format_name)
>>> +{
>>> +BlockDriver *drv1;
>>> +size_t i;
>>> +
>>> +drv1 = bdrv_do_find_format(format_name);
>>> +if (drv1) {
>>> +return drv1;
>>> +}
>>> +
>>> +/* The driver isn't registered, maybe we need to load a module */
>>> +for (i = 0; i < ARRAY_SIZE(block_driver_modules); ++i) {
>>> +if (!strcmp(block_driver_modules[i].format_name, format_name)) {
>>> +block_module_load_one(block_driver_modules[i].library_name);
>>> +break;
>>> +}
>>> +}
>>> +
>>> +return bdrv_do_find_format(format_name);
>>> +}
>>> +
>>
>> Did you reintroduce this function for dmg? I thought Fam is taking care
>> of that? I'm confused as to how Fam's patch for dmg and this series are
>> supposed to interact; the fact that the script added in patch 2 breaks
>> down with Fam's patch isn't exactly helping...
>>
>> Hm, so is this series now supposed to be applied without Fam's patch
>> with the idea of sorting dmg out later on?
>>
>> Max
>>
>>>  static int bdrv_is_whitelisted(BlockDriver *drv, bool read_only)
>>>  {
>>>  static const char *whitelist_rw[] = {
>>
> I'm not completely sure how Fam's patch is supposed to interact with
> this series actually. I'm kind of hoping it can be done on top of my
> patches at some future point, but in either case this revision was not
> done with the dmg patch in mind. The change in find_format was actually
> due to a bug I discovered in my patch series (I fixed it in v6, but you
> may have missed that).
> 
> Essentially, if a user specifies the driver explicitly as part of their
> call to qemu, eg driver=gluster, there was a bug in v5 where if the
> driver was modularized, it would not be found/loaded. So since gluster
> was modularized, if you said driver=gluster on the command line, the
> gluster module would not be found. The modules could be found by probing
> perfectly fine, this only happened when the driver was specified
> manually. The reason is because the drivers get searched based on the
> format field if they're specified manually, which means bdrv_find_format
> gets called when the driver is specified on the command line. This makes
> it necessary for bdrv_find_format to take into account modularized
> drivers even though the format drivers are not being modularized. That's
> also why the format field was added to the module_block header file again.
> 
> Colin
> 
Oops, this was meant to be a reply to your response of patch 4/4, in
case anyone gets confused.

Colin

Re: [Qemu-block] [PATCH v7 4/4] blockdev: Modularize nfs block driver

[Colin Lord]

On 08/10/2016 03:04 PM, Max Reitz wrote:
> On 08.08.2016 20:07, Colin Lord wrote:
>> Modularizes the nfs block driver so that it gets dynamically loaded.
>>
>> Signed-off-by: Colin Lord <cl...@redhat.com>
>> Reviewed-by: Stefan Hajnoczi <stefa...@redhat.com>
>> ---
>>  block/Makefile.objs | 1 +
>>  configure   | 4 ++--
>>  2 files changed, 3 insertions(+), 2 deletions(-)
> 
> I'm not quite sure what this achieves. From what I can see, the NFS
> block driver is still linked hard into qemu and it is unconditionally
> registered at qemu startup.
> 
ldd seems to be telling me that libnfs is not linked to the main binary
after this patch.
> (The output from a printf() in nfs_block_init() is visible even when
> just starting qemu-img or qemu-io without any arguments; most notably
> without bdrv_find_protocol() having been invoked at all.)
> 
> Max
> 
I can't seem to reproduce this. Is it possible you applied this patch
without applying the first 3 before it? Or maybe didn't have modules
enabled in the configuration? As far as I can tell NFS seems to building
as a module and doesn't seem to be hard linked, so I'm not really sure
what's going on.

Colin

Re: [Qemu-block] [PATCH v7 3/4] blockdev: Add dynamic module loading for block drivers

[Colin Lord]

On 08/10/2016 02:37 PM, Max Reitz wrote:
> On 08.08.2016 20:07, Colin Lord wrote:
>> From: Marc Mari <mar...@redhat.com>
>>
>> Extend the current module interface to allow for block drivers to be
>> loaded dynamically on request. The only block drivers that can be
>> converted into modules are the drivers that don't perform any init
>> operation except for registering themselves.
>>
>> In addition, only the protocol drivers are being modularized, as they
>> are the only ones which see significant performance benefits. The format
>> drivers do not generally link to external libraries, so modularizing
>> them is of no benefit from a performance perspective.
>>
>> All the necessary module information is located in a new structure found
>> in module_block.h
>>
>> Signed-off-by: Marc Marí <mar...@redhat.com>
>> Signed-off-by: Colin Lord <cl...@redhat.com>
>> Reviewed-by: Stefan Hajnoczi <stefa...@redhat.com>
>> ---
>>  Makefile  |  3 ---
>>  block.c   | 62 
>> +--
>>  block/Makefile.objs   |  3 +--
>>  include/qemu/module.h |  3 +++
>>  util/module.c | 38 +--
>>  5 files changed, 70 insertions(+), 39 deletions(-)
>>
> 
> [...]
> 
>> diff --git a/block.c b/block.c
>> index 30d64e6..6c5e249 100644
>> --- a/block.c
>> +++ b/block.c
>> @@ -26,6 +26,7 @@
>>  #include "block/block_int.h"
>>  #include "block/blockjob.h"
>>  #include "qemu/error-report.h"
>> +#include "module_block.h"
>>  #include "qemu/module.h"
>>  #include "qapi/qmp/qerror.h"
>>  #include "qapi/qmp/qbool.h"
>> @@ -241,17 +242,40 @@ BlockDriverState *bdrv_new(void)
>>  return bs;
>>  }
>>  
>> -BlockDriver *bdrv_find_format(const char *format_name)
>> +static BlockDriver *bdrv_do_find_format(const char *format_name)
>>  {
>>  BlockDriver *drv1;
>> +
>>  QLIST_FOREACH(drv1, _drivers, list) {
>>  if (!strcmp(drv1->format_name, format_name)) {
>>  return drv1;
>>  }
>>  }
>> +
>>  return NULL;
>>  }
>>  
>> +BlockDriver *bdrv_find_format(const char *format_name)
>> +{
>> +BlockDriver *drv1;
>> +size_t i;
>> +
>> +drv1 = bdrv_do_find_format(format_name);
>> +if (drv1) {
>> +return drv1;
>> +}
>> +
>> +/* The driver isn't registered, maybe we need to load a module */
>> +for (i = 0; i < ARRAY_SIZE(block_driver_modules); ++i) {
>> +if (!strcmp(block_driver_modules[i].format_name, format_name)) {
>> +block_module_load_one(block_driver_modules[i].library_name);
>> +break;
>> +}
>> +}
>> +
>> +return bdrv_do_find_format(format_name);
>> +}
>> +
> 
> Did you reintroduce this function for dmg? I thought Fam is taking care
> of that? I'm confused as to how Fam's patch for dmg and this series are
> supposed to interact; the fact that the script added in patch 2 breaks
> down with Fam's patch isn't exactly helping...
> 
> Hm, so is this series now supposed to be applied without Fam's patch
> with the idea of sorting dmg out later on?
> 
> Max
> 
>>  static int bdrv_is_whitelisted(BlockDriver *drv, bool read_only)
>>  {
>>  static const char *whitelist_rw[] = {
> 
I'm not completely sure how Fam's patch is supposed to interact with
this series actually. I'm kind of hoping it can be done on top of my
patches at some future point, but in either case this revision was not
done with the dmg patch in mind. The change in find_format was actually
due to a bug I discovered in my patch series (I fixed it in v6, but you
may have missed that).

Essentially, if a user specifies the driver explicitly as part of their
call to qemu, eg driver=gluster, there was a bug in v5 where if the
driver was modularized, it would not be found/loaded. So since gluster
was modularized, if you said driver=gluster on the command line, the
gluster module would not be found. The modules could be found by probing
perfectly fine, this only happened when the driver was specified
manually. The reason is because the drivers get searched based on the
format field if they're specified manually, which means bdrv_find_format
gets called when the driver is specified on the command line. This makes
it necessary for bdrv_find_format to take into account modularized
drivers even though the format drivers are not being modularized. That's
also why the format field was added to the module_block header file again.

Colin

[Qemu-block] [PATCH v7 3/4] blockdev: Add dynamic module loading for block drivers

[Colin Lord]

From: Marc Mari <mar...@redhat.com>

Extend the current module interface to allow for block drivers to be
loaded dynamically on request. The only block drivers that can be
converted into modules are the drivers that don't perform any init
operation except for registering themselves.

In addition, only the protocol drivers are being modularized, as they
are the only ones which see significant performance benefits. The format
drivers do not generally link to external libraries, so modularizing
them is of no benefit from a performance perspective.

All the necessary module information is located in a new structure found
in module_block.h

Signed-off-by: Marc Marí <mar...@redhat.com>
Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefa...@redhat.com>
---
 Makefile  |  3 ---
 block.c   | 62 +--
 block/Makefile.objs   |  3 +--
 include/qemu/module.h |  3 +++
 util/module.c | 38 +--
 5 files changed, 70 insertions(+), 39 deletions(-)

diff --git a/Makefile b/Makefile
index 4b9384e..c7aa8cd 100644
--- a/Makefile
+++ b/Makefile
@@ -247,9 +247,6 @@ Makefile: $(version-obj-y) $(version-lobj-y)
 libqemustub.a: $(stub-obj-y)
 libqemuutil.a: $(util-obj-y)
 
-block-modules = $(foreach o,$(block-obj-m),"$(basename $(subst /,-,$o))",) NULL
-util/module.o-cflags = -D'CONFIG_BLOCK_MODULES=$(block-modules)'
-
 ##
 
 qemu-img.o: qemu-img-cmds.h
diff --git a/block.c b/block.c
index 30d64e6..6c5e249 100644
--- a/block.c
+++ b/block.c
@@ -26,6 +26,7 @@
 #include "block/block_int.h"
 #include "block/blockjob.h"
 #include "qemu/error-report.h"
+#include "module_block.h"
 #include "qemu/module.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/qmp/qbool.h"
@@ -241,17 +242,40 @@ BlockDriverState *bdrv_new(void)
 return bs;
 }
 
-BlockDriver *bdrv_find_format(const char *format_name)
+static BlockDriver *bdrv_do_find_format(const char *format_name)
 {
 BlockDriver *drv1;
+
 QLIST_FOREACH(drv1, _drivers, list) {
 if (!strcmp(drv1->format_name, format_name)) {
 return drv1;
 }
 }
+
 return NULL;
 }
 
+BlockDriver *bdrv_find_format(const char *format_name)
+{
+BlockDriver *drv1;
+size_t i;
+
+drv1 = bdrv_do_find_format(format_name);
+if (drv1) {
+return drv1;
+}
+
+/* The driver isn't registered, maybe we need to load a module */
+for (i = 0; i < ARRAY_SIZE(block_driver_modules); ++i) {
+if (!strcmp(block_driver_modules[i].format_name, format_name)) {
+block_module_load_one(block_driver_modules[i].library_name);
+break;
+}
+}
+
+return bdrv_do_find_format(format_name);
+}
+
 static int bdrv_is_whitelisted(BlockDriver *drv, bool read_only)
 {
 static const char *whitelist_rw[] = {
@@ -460,6 +484,19 @@ static BlockDriver *find_hdev_driver(const char *filename)
 return drv;
 }
 
+static BlockDriver *bdrv_do_find_protocol(const char *protocol)
+{
+BlockDriver *drv1;
+
+QLIST_FOREACH(drv1, _drivers, list) {
+if (drv1->protocol_name && !strcmp(drv1->protocol_name, protocol)) {
+return drv1;
+}
+}
+
+return NULL;
+}
+
 BlockDriver *bdrv_find_protocol(const char *filename,
 bool allow_protocol_prefix,
 Error **errp)
@@ -468,6 +505,7 @@ BlockDriver *bdrv_find_protocol(const char *filename,
 char protocol[128];
 int len;
 const char *p;
+size_t i;
 
 /* TODO Drivers without bdrv_file_open must be specified explicitly */
 
@@ -494,15 +532,25 @@ BlockDriver *bdrv_find_protocol(const char *filename,
 len = sizeof(protocol) - 1;
 memcpy(protocol, filename, len);
 protocol[len] = '\0';
-QLIST_FOREACH(drv1, _drivers, list) {
-if (drv1->protocol_name &&
-!strcmp(drv1->protocol_name, protocol)) {
-return drv1;
+
+drv1 = bdrv_do_find_protocol(protocol);
+if (drv1) {
+return drv1;
+}
+
+for (i = 0; i < ARRAY_SIZE(block_driver_modules); ++i) {
+if (block_driver_modules[i].protocol_name &&
+!strcmp(block_driver_modules[i].protocol_name, protocol)) {
+block_module_load_one(block_driver_modules[i].library_name);
+break;
 }
 }
 
-error_setg(errp, "Unknown protocol '%s'", protocol);
-return NULL;
+drv1 = bdrv_do_find_protocol(protocol);
+if (!drv1) {
+error_setg(errp, "Unknown protocol '%s'", protocol);
+}
+return drv1;
 }
 
 /*
diff --git a/block/Makefile.objs b/block/Makefile.objs
index 2593a2f..595f366 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -1

[Qemu-block] [PATCH v7 2/4] blockdev: Add dynamic generation of module_block.h

[Colin Lord]

From: Marc Mari <mar...@redhat.com>

To simplify the addition of new block modules, add a script that generates
module_block.h automatically from the modules' source code.

This script assumes that the QEMU coding style rules are followed.

Signed-off-by: Marc Marí <mar...@redhat.com>
Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefa...@redhat.com>
---
 Makefile|   7 +++
 scripts/modules/module_block.py | 108 
 2 files changed, 115 insertions(+)
 create mode 100644 scripts/modules/module_block.py

diff --git a/Makefile b/Makefile
index 0d7647f..4b9384e 100644
--- a/Makefile
+++ b/Makefile
@@ -76,6 +76,8 @@ GENERATED_HEADERS += trace/generated-ust-provider.h
 GENERATED_SOURCES += trace/generated-ust.c
 endif
 
+GENERATED_HEADERS += module_block.h
+
 # Don't try to regenerate Makefile or configure
 # We don't generate any of them
 Makefile: ;
@@ -352,6 +354,11 @@ ivshmem-client$(EXESUF): $(ivshmem-client-obj-y) 
libqemuutil.a libqemustub.a
 ivshmem-server$(EXESUF): $(ivshmem-server-obj-y) libqemuutil.a libqemustub.a
$(call LINK, $^)
 
+module_block.h: $(SRC_PATH)/scripts/modules/module_block.py config-host.mak
+   $(call quiet-command,$(PYTHON) $< $@ \
+   $(addprefix $(SRC_PATH)/,$(patsubst %.mo,%.c,$(block-obj-m))), \
+   "  GEN   $@")
+
 clean:
 # avoid old build problems by removing potentially incorrect old files
rm -f config.mak op-i386.h opc-i386.h gen-op-i386.h op-arm.h opc-arm.h 
gen-op-arm.h
diff --git a/scripts/modules/module_block.py b/scripts/modules/module_block.py
new file mode 100644
index 000..db4fb54
--- /dev/null
+++ b/scripts/modules/module_block.py
@@ -0,0 +1,108 @@
+#!/usr/bin/python
+#
+# Module information generator
+#
+# Copyright Red Hat, Inc. 2015 - 2016
+#
+# Authors:
+#  Marc Mari <mar...@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2.
+# See the COPYING file in the top-level directory.
+
+from __future__ import print_function
+import sys
+import os
+
+def get_string_struct(line):
+data = line.split()
+
+# data[0] -> struct element name
+# data[1] -> =
+# data[2] -> value
+
+return data[2].replace('"', '')[:-1]
+
+def add_module(fheader, library, format_name, protocol_name):
+lines = []
+lines.append('.library_name = "' + library + '",')
+if format_name != "":
+lines.append('.format_name = "' + format_name + '",')
+if protocol_name != "":
+lines.append('.protocol_name = "' + protocol_name + '",')
+
+text = '\n'.join(lines)
+fheader.write('\n{\n' + text + '\n},')
+
+def process_file(fheader, filename):
+# This parser assumes the coding style rules are being followed
+with open(filename, "r") as cfile:
+found_something = False
+found_start = False
+library, _ = os.path.splitext(os.path.basename(filename))
+for line in cfile:
+if found_start:
+line = line.replace('\n', '')
+if line.find(".format_name") != -1:
+format_name = get_string_struct(line)
+elif line.find(".protocol_name") != -1:
+protocol_name = get_string_struct(line)
+elif line == "};":
+add_module(fheader, library, format_name, protocol_name)
+found_start = False
+elif line.find("static BlockDriver") != -1:
+found_something = True
+found_start = True
+format_name = ""
+protocol_name = ""
+
+if not found_something:
+print("No BlockDriver struct found in " + filename + ". \
+Is this really a module?", file=sys.stderr)
+sys.exit(1)
+
+def print_top(fheader):
+fheader.write('''/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
+/*
+ * QEMU Block Module Infrastructure
+ *
+ * Authors:
+ *  Marc Mari   <mar...@redhat.com>
+ */
+
+''')
+
+fheader.write('''#ifndef QEMU_MODULE_BLOCK_H
+#define QEMU_MODULE_BLOCK_H
+
+#include "qemu-common.h"
+
+static const struct {
+const char *format_name;
+const char *protocol_name;
+const char *library_name;
+} block_driver_modules[] = {''')
+
+def print_bottom(fheader):
+fheader.write('''
+};
+
+#endif
+''')
+
+# First argument: output file
+# All other arguments: modules source files (.c)
+output_file = sys.argv[1]
+with open(output_file, 'w') as fheader:
+print_top(fheader)
+
+for filename in sys.argv[2:]:
+if os.path.isfile(filename):
+process_file(fheader, filename)
+else:
+print("File " + filename + " does not exist.", file=sys.stderr)
+sys.exit(1)
+
+print_bottom(fheader)
+
+sys.exit(0)
-- 
2.5.5

[Qemu-block] [PATCH v7 4/4] blockdev: Modularize nfs block driver

[Colin Lord]

Modularizes the nfs block driver so that it gets dynamically loaded.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefa...@redhat.com>
---
 block/Makefile.objs | 1 +
 configure   | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 595f366..fa4d8b8 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -28,6 +28,7 @@ block-obj-y += crypto.o
 common-obj-y += stream.o
 common-obj-y += backup.o
 
+nfs.o-libs := $(LIBNFS_LIBS)
 iscsi.o-cflags := $(LIBISCSI_CFLAGS)
 iscsi.o-libs   := $(LIBISCSI_LIBS)
 curl.o-cflags  := $(CURL_CFLAGS)
diff --git a/configure b/configure
index f57fcc6..f1e7d14 100755
--- a/configure
+++ b/configure
@@ -4561,7 +4561,6 @@ if test "$libnfs" != "no" ; then
   if $pkg_config --atleast-version=1.9.3 libnfs; then
 libnfs="yes"
 libnfs_libs=$($pkg_config --libs libnfs)
-LIBS="$LIBS $libnfs_libs"
   else
 if test "$libnfs" = "yes" ; then
   feature_not_found "libnfs" "Install libnfs devel >= 1.9.3"
@@ -5320,7 +5319,8 @@ if test "$libiscsi" = "yes" ; then
 fi
 
 if test "$libnfs" = "yes" ; then
-  echo "CONFIG_LIBNFS=y" >> $config_host_mak
+  echo "CONFIG_LIBNFS=m" >> $config_host_mak
+  echo "LIBNFS_LIBS=$libnfs_libs" >> $config_host_mak
 fi
 
 if test "$seccomp" = "yes"; then
-- 
2.5.5

[Qemu-block] [PATCH v7 1/4] blockdev: prepare iSCSI block driver for dynamic loading

[Colin Lord]

This commit moves the initialization of the QemuOptsList qemu_iscsi_opts
struct out of block/iscsi.c in order to allow the iscsi module to be
dynamically loaded.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Fam Zheng <f...@redhat.com>
---
 block/iscsi.c | 36 
 vl.c  | 40 
 2 files changed, 40 insertions(+), 36 deletions(-)

diff --git a/block/iscsi.c b/block/iscsi.c
index 95ce9e1..c4a0937 100644
--- a/block/iscsi.c
+++ b/block/iscsi.c
@@ -2010,45 +2010,9 @@ static BlockDriver bdrv_iscsi = {
 .bdrv_attach_aio_context = iscsi_attach_aio_context,
 };
 
-static QemuOptsList qemu_iscsi_opts = {
-.name = "iscsi",
-.head = QTAILQ_HEAD_INITIALIZER(qemu_iscsi_opts.head),
-.desc = {
-{
-.name = "user",
-.type = QEMU_OPT_STRING,
-.help = "username for CHAP authentication to target",
-},{
-.name = "password",
-.type = QEMU_OPT_STRING,
-.help = "password for CHAP authentication to target",
-},{
-.name = "password-secret",
-.type = QEMU_OPT_STRING,
-.help = "ID of the secret providing password for CHAP "
-"authentication to target",
-},{
-.name = "header-digest",
-.type = QEMU_OPT_STRING,
-.help = "HeaderDigest setting. "
-"{CRC32C|CRC32C-NONE|NONE-CRC32C|NONE}",
-},{
-.name = "initiator-name",
-.type = QEMU_OPT_STRING,
-.help = "Initiator iqn name to use when connecting",
-},{
-.name = "timeout",
-.type = QEMU_OPT_NUMBER,
-.help = "Request timeout in seconds (default 0 = no timeout)",
-},
-{ /* end of list */ }
-},
-};
-
 static void iscsi_block_init(void)
 {
 bdrv_register(_iscsi);
-qemu_add_opts(_iscsi_opts);
 }
 
 block_init(iscsi_block_init);
diff --git a/vl.c b/vl.c
index c4eeaff..8b562d3 100644
--- a/vl.c
+++ b/vl.c
@@ -506,6 +506,43 @@ static QemuOptsList qemu_fw_cfg_opts = {
 },
 };
 
+#ifdef CONFIG_LIBISCSI
+static QemuOptsList qemu_iscsi_opts = {
+.name = "iscsi",
+.head = QTAILQ_HEAD_INITIALIZER(qemu_iscsi_opts.head),
+.desc = {
+{
+.name = "user",
+.type = QEMU_OPT_STRING,
+.help = "username for CHAP authentication to target",
+},{
+.name = "password",
+.type = QEMU_OPT_STRING,
+.help = "password for CHAP authentication to target",
+},{
+.name = "password-secret",
+.type = QEMU_OPT_STRING,
+.help = "ID of the secret providing password for CHAP "
+"authentication to target",
+},{
+.name = "header-digest",
+.type = QEMU_OPT_STRING,
+.help = "HeaderDigest setting. "
+"{CRC32C|CRC32C-NONE|NONE-CRC32C|NONE}",
+},{
+.name = "initiator-name",
+.type = QEMU_OPT_STRING,
+.help = "Initiator iqn name to use when connecting",
+},{
+.name = "timeout",
+.type = QEMU_OPT_NUMBER,
+.help = "Request timeout in seconds (default 0 = no timeout)",
+},
+{ /* end of list */ }
+},
+};
+#endif
+
 /**
  * Get machine options
  *
@@ -3001,6 +3038,9 @@ int main(int argc, char **argv, char **envp)
 qemu_add_opts(_icount_opts);
 qemu_add_opts(_semihosting_config_opts);
 qemu_add_opts(_fw_cfg_opts);
+#ifdef CONFIG_LIBISCSI
+qemu_add_opts(_iscsi_opts);
+#endif
 module_call_init(MODULE_INIT_OPTS);
 
 runstate_init();
-- 
2.5.5

[Qemu-block] [PATCH v7 0/4] Dynamic module loading for block drivers

[Colin Lord]

One more minor revision from v6, no big changes.

v7:
- Add ifdef around qemu_iscsi_opts in vl.c (first patch)

v6:
- Fix bug so that users can specify a modularized driver on the cli
  without qemu exiting
- Remove extra lines from Makefile
- Add patch to modularize NFS

v5:
- No format drivers are modularized, therefore the probe functions are
  all being left completely untouched.
- Remove dmg from block-obj-m since it is not a target of the
  modularization effort.
- Modify module_block.py to only include the library name and protocol
  name fields in the generated struct. The other fields are no longer
  necessary for the drivers that are being modularized.

v4:
- Fix indentation of the generated header file module_block.h
- Drivers and probe functions are now all located in the block/
  directory, rather than being split between block/ and block/probe/. In
  addition the header files for each probe/driver pair are in the block/
  directory, not the include/block/driver/ directory (which no longer
  exists).
- Since the probe files are in block/ now, they follow the naming
  pattern of format-probe.c
- Renamed crypto probe file to be crypto-probe.c, luks is no longer in
  the filename
- Fixed formatting of parallels_probe() function header
- Enforced consistent naming convention for the probe functions. They
  now follow the pattern bdrv_format_probe().

Colin Lord (2):
  blockdev: prepare iSCSI block driver for dynamic loading
  blockdev: Modularize nfs block driver

Marc Mari (2):
  blockdev: Add dynamic generation of module_block.h
  blockdev: Add dynamic module loading for block drivers

 Makefile|  10 ++--
 block.c |  62 ---
 block/Makefile.objs |   4 +-
 block/iscsi.c   |  36 --
 configure   |   4 +-
 include/qemu/module.h   |   3 ++
 scripts/modules/module_block.py | 108 
 util/module.c   |  38 --
 vl.c|  40 +++
 9 files changed, 228 insertions(+), 77 deletions(-)
 create mode 100644 scripts/modules/module_block.py

-- 
2.5.5

Re: [Qemu-block] [PATCH v6 1/4] blockdev: prepare iSCSI block driver for dynamic loading

[Colin Lord]

On 08/08/2016 11:18 AM, Stefan Hajnoczi wrote:
> On Tue, Aug 02, 2016 at 10:12:14AM -0400, Colin Lord wrote:
>> diff --git a/vl.c b/vl.c
>> index e7c2c62..1a5f807 100644
>> --- a/vl.c
>> +++ b/vl.c
>> @@ -506,6 +506,41 @@ static QemuOptsList qemu_fw_cfg_opts = {
>>  },
>>  };
>>  
>> +static QemuOptsList qemu_iscsi_opts = {
>> +.name = "iscsi",
>> +.head = QTAILQ_HEAD_INITIALIZER(qemu_iscsi_opts.head),
>> +.desc = {
>> +{
>> +.name = "user",
>> +.type = QEMU_OPT_STRING,
>> +.help = "username for CHAP authentication to target",
>> +},{
>> +.name = "password",
>> +.type = QEMU_OPT_STRING,
>> +.help = "password for CHAP authentication to target",
>> +},{
>> +.name = "password-secret",
>> +.type = QEMU_OPT_STRING,
>> +.help = "ID of the secret providing password for CHAP "
>> +"authentication to target",
>> +},{
>> +.name = "header-digest",
>> +.type = QEMU_OPT_STRING,
>> +.help = "HeaderDigest setting. "
>> +"{CRC32C|CRC32C-NONE|NONE-CRC32C|NONE}",
>> +},{
>> +.name = "initiator-name",
>> +.type = QEMU_OPT_STRING,
>> +.help = "Initiator iqn name to use when connecting",
>> +},{
>> +.name = "timeout",
>> +.type = QEMU_OPT_NUMBER,
>> +.help = "Request timeout in seconds (default 0 = no timeout)",
>> +},
>> +{ /* end of list */ }
>> +},
>> +};
>> +
>>  /**
>>   * Get machine options
>>   *
>> @@ -3001,6 +3036,9 @@ int main(int argc, char **argv, char **envp)
>>  qemu_add_opts(_icount_opts);
>>  qemu_add_opts(_semihosting_config_opts);
>>  qemu_add_opts(_fw_cfg_opts);
>> +#ifdef CONFIG_LIBISCSI
>> +qemu_add_opts(_iscsi_opts);
>> +#endif
> 
> This will probably produce "unused variable" warnings and cause the
> build to fail when CONFIG_LIBISCSI is not defined.
> 
> In this case I'd add an ifdef around qemu_iscsi_opts.
> 
Oddly enough I don't actually get any warnings when iscsi is disabled,
but in any case the ifdef should probably be there anyway so I'll fix
and resubmit.

Re: [Qemu-block] [PATCH v6 0/4] Dynamic module loading for block drivers

[Colin Lord]

On 08/02/2016 10:12 AM, Colin Lord wrote:
> I'm sending one more version of this series. v5 didn't seem to need any
> huge makeovers, but I found a small bug in it so I'm hoping it hasn't
> been committed yet. In v5, if a user explicitly specified the driver to
> use, eg driver=gluster, and the driver was modularized, it was not
> correctly detected and qemu would exit since it could not find the
> driver. This series fixes that.
> 
> In addition, the two patches I sent to remove unneeded lines in the
> Makefile and to modularize nfs are included in this series.
> 
> v6:
> - Fix bug so that users can specify a modularized driver on the cli
>   without qemu exiting
> - Remove extra lines from Makefile
> - Add patch to modularize NFS
> 
> v5:
> - No format drivers are modularized, therefore the probe functions are
>   all being left completely untouched.
> - Remove dmg from block-obj-m since it is not a target of the
>   modularization effort.
> - Modify module_block.py to only include the library name and protocol
>   name fields in the generated struct. The other fields are no longer
>   necessary for the drivers that are being modularized.
> 
> v4:
> - Fix indentation of the generated header file module_block.h
> - Drivers and probe functions are now all located in the block/
>   directory, rather than being split between block/ and block/probe/. In
>   addition the header files for each probe/driver pair are in the block/
>   directory, not the include/block/driver/ directory (which no longer
>   exists).
> - Since the probe files are in block/ now, they follow the naming
>   pattern of format-probe.c
> - Renamed crypto probe file to be crypto-probe.c, luks is no longer in
>   the filename
> - Fixed formatting of parallels_probe() function header
> - Enforced consistent naming convention for the probe functions. They
>   now follow the pattern bdrv_format_probe().
> 
> Colin Lord (3):
>   blockdev: prepare iSCSI block driver for dynamic loading
>   blockdev: Add dynamic module loading for block drivers
>   blockdev: Modularize nfs block driver
> 
> Marc Mari (1):
>   blockdev: Add dynamic generation of module_block.h
> 
>  Makefile|  10 ++--
>  block.c |  62 ---
>  block/Makefile.objs |   4 +-
>  block/iscsi.c   |  36 --
>  configure   |   4 +-
>  include/qemu/module.h   |   3 ++
>  scripts/modules/module_block.py | 108 
> 
>  util/module.c   |  38 --
>  vl.c|  38 ++
>  9 files changed, 226 insertions(+), 77 deletions(-)
>  create mode 100644 scripts/modules/module_block.py
> 
I did want to address one of the items Fam brought up in v5 that I never
replied to, so I'm adding Fam to CC here.

It sounds like since this series removes dmg from the list of
modularized drivers, there is a libbz2 dependency that may get pulled in
from the dmg driver. I believe Fam was suggesting moving that part of
the dmg driver into its own module (although not modularizing the entire
dmg driver). Does that sound like a correct interpretation Fam? I do
like the idea, but I'm a little hesitant to put that into this series
because I'd like to keep this series fairly simple so that hopefully
there doesn't need to be too much more discussion about it. My
internship will be ending soon so I'd prefer this series be finished
sooner rather than later. Also I don't think anything in this series
excludes this idea from being added later if we want. Of course there is
always room for discussion, so if there are other thoughts I'm happy to
listen.

Colin

[Qemu-block] [PATCH v6 4/4] blockdev: Modularize nfs block driver

[Colin Lord]

Modularizes the nfs block driver so that it gets dynamically loaded.
---
 block/Makefile.objs | 1 +
 configure   | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 595f366..fa4d8b8 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -28,6 +28,7 @@ block-obj-y += crypto.o
 common-obj-y += stream.o
 common-obj-y += backup.o
 
+nfs.o-libs := $(LIBNFS_LIBS)
 iscsi.o-cflags := $(LIBISCSI_CFLAGS)
 iscsi.o-libs   := $(LIBISCSI_LIBS)
 curl.o-cflags  := $(CURL_CFLAGS)
diff --git a/configure b/configure
index f57fcc6..f1e7d14 100755
--- a/configure
+++ b/configure
@@ -4561,7 +4561,6 @@ if test "$libnfs" != "no" ; then
   if $pkg_config --atleast-version=1.9.3 libnfs; then
 libnfs="yes"
 libnfs_libs=$($pkg_config --libs libnfs)
-LIBS="$LIBS $libnfs_libs"
   else
 if test "$libnfs" = "yes" ; then
   feature_not_found "libnfs" "Install libnfs devel >= 1.9.3"
@@ -5320,7 +5319,8 @@ if test "$libiscsi" = "yes" ; then
 fi
 
 if test "$libnfs" = "yes" ; then
-  echo "CONFIG_LIBNFS=y" >> $config_host_mak
+  echo "CONFIG_LIBNFS=m" >> $config_host_mak
+  echo "LIBNFS_LIBS=$libnfs_libs" >> $config_host_mak
 fi
 
 if test "$seccomp" = "yes"; then
-- 
2.5.5

[Qemu-block] [PATCH v6 0/4] Dynamic module loading for block drivers

[Colin Lord]

I'm sending one more version of this series. v5 didn't seem to need any
huge makeovers, but I found a small bug in it so I'm hoping it hasn't
been committed yet. In v5, if a user explicitly specified the driver to
use, eg driver=gluster, and the driver was modularized, it was not
correctly detected and qemu would exit since it could not find the
driver. This series fixes that.

In addition, the two patches I sent to remove unneeded lines in the
Makefile and to modularize nfs are included in this series.

v6:
- Fix bug so that users can specify a modularized driver on the cli
  without qemu exiting
- Remove extra lines from Makefile
- Add patch to modularize NFS

v5:
- No format drivers are modularized, therefore the probe functions are
  all being left completely untouched.
- Remove dmg from block-obj-m since it is not a target of the
  modularization effort.
- Modify module_block.py to only include the library name and protocol
  name fields in the generated struct. The other fields are no longer
  necessary for the drivers that are being modularized.

v4:
- Fix indentation of the generated header file module_block.h
- Drivers and probe functions are now all located in the block/
  directory, rather than being split between block/ and block/probe/. In
  addition the header files for each probe/driver pair are in the block/
  directory, not the include/block/driver/ directory (which no longer
  exists).
- Since the probe files are in block/ now, they follow the naming
  pattern of format-probe.c
- Renamed crypto probe file to be crypto-probe.c, luks is no longer in
  the filename
- Fixed formatting of parallels_probe() function header
- Enforced consistent naming convention for the probe functions. They
  now follow the pattern bdrv_format_probe().

Colin Lord (3):
  blockdev: prepare iSCSI block driver for dynamic loading
  blockdev: Add dynamic module loading for block drivers
  blockdev: Modularize nfs block driver

Marc Mari (1):
  blockdev: Add dynamic generation of module_block.h

 Makefile|  10 ++--
 block.c |  62 ---
 block/Makefile.objs |   4 +-
 block/iscsi.c   |  36 --
 configure   |   4 +-
 include/qemu/module.h   |   3 ++
 scripts/modules/module_block.py | 108 
 util/module.c   |  38 --
 vl.c|  38 ++
 9 files changed, 226 insertions(+), 77 deletions(-)
 create mode 100644 scripts/modules/module_block.py

-- 
2.5.5

[Qemu-block] [PATCH v6 2/4] blockdev: Add dynamic generation of module_block.h

[Colin Lord]

From: Marc Mari <mar...@redhat.com>

To simplify the addition of new block modules, add a script that generates
module_block.h automatically from the modules' source code.

This script assumes that the QEMU coding style rules are followed.

Signed-off-by: Marc Marí <mar...@redhat.com>
Signed-off-by: Colin Lord <cl...@redhat.com>
---
 Makefile|   7 +++
 scripts/modules/module_block.py | 108 
 2 files changed, 115 insertions(+)
 create mode 100644 scripts/modules/module_block.py

diff --git a/Makefile b/Makefile
index 0d7647f..4b9384e 100644
--- a/Makefile
+++ b/Makefile
@@ -76,6 +76,8 @@ GENERATED_HEADERS += trace/generated-ust-provider.h
 GENERATED_SOURCES += trace/generated-ust.c
 endif
 
+GENERATED_HEADERS += module_block.h
+
 # Don't try to regenerate Makefile or configure
 # We don't generate any of them
 Makefile: ;
@@ -352,6 +354,11 @@ ivshmem-client$(EXESUF): $(ivshmem-client-obj-y) 
libqemuutil.a libqemustub.a
 ivshmem-server$(EXESUF): $(ivshmem-server-obj-y) libqemuutil.a libqemustub.a
$(call LINK, $^)
 
+module_block.h: $(SRC_PATH)/scripts/modules/module_block.py config-host.mak
+   $(call quiet-command,$(PYTHON) $< $@ \
+   $(addprefix $(SRC_PATH)/,$(patsubst %.mo,%.c,$(block-obj-m))), \
+   "  GEN   $@")
+
 clean:
 # avoid old build problems by removing potentially incorrect old files
rm -f config.mak op-i386.h opc-i386.h gen-op-i386.h op-arm.h opc-arm.h 
gen-op-arm.h
diff --git a/scripts/modules/module_block.py b/scripts/modules/module_block.py
new file mode 100644
index 000..db4fb54
--- /dev/null
+++ b/scripts/modules/module_block.py
@@ -0,0 +1,108 @@
+#!/usr/bin/python
+#
+# Module information generator
+#
+# Copyright Red Hat, Inc. 2015 - 2016
+#
+# Authors:
+#  Marc Mari <mar...@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2.
+# See the COPYING file in the top-level directory.
+
+from __future__ import print_function
+import sys
+import os
+
+def get_string_struct(line):
+data = line.split()
+
+# data[0] -> struct element name
+# data[1] -> =
+# data[2] -> value
+
+return data[2].replace('"', '')[:-1]
+
+def add_module(fheader, library, format_name, protocol_name):
+lines = []
+lines.append('.library_name = "' + library + '",')
+if format_name != "":
+lines.append('.format_name = "' + format_name + '",')
+if protocol_name != "":
+lines.append('.protocol_name = "' + protocol_name + '",')
+
+text = '\n'.join(lines)
+fheader.write('\n{\n' + text + '\n},')
+
+def process_file(fheader, filename):
+# This parser assumes the coding style rules are being followed
+with open(filename, "r") as cfile:
+found_something = False
+found_start = False
+library, _ = os.path.splitext(os.path.basename(filename))
+for line in cfile:
+if found_start:
+line = line.replace('\n', '')
+if line.find(".format_name") != -1:
+format_name = get_string_struct(line)
+elif line.find(".protocol_name") != -1:
+protocol_name = get_string_struct(line)
+elif line == "};":
+add_module(fheader, library, format_name, protocol_name)
+found_start = False
+elif line.find("static BlockDriver") != -1:
+found_something = True
+found_start = True
+format_name = ""
+protocol_name = ""
+
+if not found_something:
+print("No BlockDriver struct found in " + filename + ". \
+Is this really a module?", file=sys.stderr)
+sys.exit(1)
+
+def print_top(fheader):
+fheader.write('''/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
+/*
+ * QEMU Block Module Infrastructure
+ *
+ * Authors:
+ *  Marc Mari   <mar...@redhat.com>
+ */
+
+''')
+
+fheader.write('''#ifndef QEMU_MODULE_BLOCK_H
+#define QEMU_MODULE_BLOCK_H
+
+#include "qemu-common.h"
+
+static const struct {
+const char *format_name;
+const char *protocol_name;
+const char *library_name;
+} block_driver_modules[] = {''')
+
+def print_bottom(fheader):
+fheader.write('''
+};
+
+#endif
+''')
+
+# First argument: output file
+# All other arguments: modules source files (.c)
+output_file = sys.argv[1]
+with open(output_file, 'w') as fheader:
+print_top(fheader)
+
+for filename in sys.argv[2:]:
+if os.path.isfile(filename):
+process_file(fheader, filename)
+else:
+print("File " + filename + " does not exist.", file=sys.stderr)
+sys.exit(1)
+
+print_bottom(fheader)
+
+sys.exit(0)
-- 
2.5.5

[Qemu-block] [PATCH v6 1/4] blockdev: prepare iSCSI block driver for dynamic loading

[Colin Lord]

This commit moves the initialization of the QemuOptsList qemu_iscsi_opts
struct out of block/iscsi.c in order to allow the iscsi module to be
dynamically loaded.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Fam Zheng <f...@redhat.com>
---
 block/iscsi.c | 36 
 vl.c  | 38 ++
 2 files changed, 38 insertions(+), 36 deletions(-)

diff --git a/block/iscsi.c b/block/iscsi.c
index 95ce9e1..c4a0937 100644
--- a/block/iscsi.c
+++ b/block/iscsi.c
@@ -2010,45 +2010,9 @@ static BlockDriver bdrv_iscsi = {
 .bdrv_attach_aio_context = iscsi_attach_aio_context,
 };
 
-static QemuOptsList qemu_iscsi_opts = {
-.name = "iscsi",
-.head = QTAILQ_HEAD_INITIALIZER(qemu_iscsi_opts.head),
-.desc = {
-{
-.name = "user",
-.type = QEMU_OPT_STRING,
-.help = "username for CHAP authentication to target",
-},{
-.name = "password",
-.type = QEMU_OPT_STRING,
-.help = "password for CHAP authentication to target",
-},{
-.name = "password-secret",
-.type = QEMU_OPT_STRING,
-.help = "ID of the secret providing password for CHAP "
-"authentication to target",
-},{
-.name = "header-digest",
-.type = QEMU_OPT_STRING,
-.help = "HeaderDigest setting. "
-"{CRC32C|CRC32C-NONE|NONE-CRC32C|NONE}",
-},{
-.name = "initiator-name",
-.type = QEMU_OPT_STRING,
-.help = "Initiator iqn name to use when connecting",
-},{
-.name = "timeout",
-.type = QEMU_OPT_NUMBER,
-.help = "Request timeout in seconds (default 0 = no timeout)",
-},
-{ /* end of list */ }
-},
-};
-
 static void iscsi_block_init(void)
 {
 bdrv_register(_iscsi);
-qemu_add_opts(_iscsi_opts);
 }
 
 block_init(iscsi_block_init);
diff --git a/vl.c b/vl.c
index e7c2c62..1a5f807 100644
--- a/vl.c
+++ b/vl.c
@@ -506,6 +506,41 @@ static QemuOptsList qemu_fw_cfg_opts = {
 },
 };
 
+static QemuOptsList qemu_iscsi_opts = {
+.name = "iscsi",
+.head = QTAILQ_HEAD_INITIALIZER(qemu_iscsi_opts.head),
+.desc = {
+{
+.name = "user",
+.type = QEMU_OPT_STRING,
+.help = "username for CHAP authentication to target",
+},{
+.name = "password",
+.type = QEMU_OPT_STRING,
+.help = "password for CHAP authentication to target",
+},{
+.name = "password-secret",
+.type = QEMU_OPT_STRING,
+.help = "ID of the secret providing password for CHAP "
+"authentication to target",
+},{
+.name = "header-digest",
+.type = QEMU_OPT_STRING,
+.help = "HeaderDigest setting. "
+"{CRC32C|CRC32C-NONE|NONE-CRC32C|NONE}",
+},{
+.name = "initiator-name",
+.type = QEMU_OPT_STRING,
+.help = "Initiator iqn name to use when connecting",
+},{
+.name = "timeout",
+.type = QEMU_OPT_NUMBER,
+.help = "Request timeout in seconds (default 0 = no timeout)",
+},
+{ /* end of list */ }
+},
+};
+
 /**
  * Get machine options
  *
@@ -3001,6 +3036,9 @@ int main(int argc, char **argv, char **envp)
 qemu_add_opts(_icount_opts);
 qemu_add_opts(_semihosting_config_opts);
 qemu_add_opts(_fw_cfg_opts);
+#ifdef CONFIG_LIBISCSI
+qemu_add_opts(_iscsi_opts);
+#endif
 module_call_init(MODULE_INIT_OPTS);
 
 runstate_init();
-- 
2.5.5

[Qemu-block] [PATCH v6 3/4] blockdev: Add dynamic module loading for block drivers

[Colin Lord]

Extend the current module interface to allow for block drivers to be
loaded dynamically on request. The only block drivers that can be
converted into modules are the drivers that don't perform any init
operation except for registering themselves.

In addition, only the protocol drivers are being modularized, as they
are the only ones which see significant performance benefits. The format
drivers do not generally link to external libraries, so modularizing
them is of no benefit from a performance perspective.

All the necessary module information is located in a new structure found
in module_block.h

Signed-off-by: Marc Marí <mar...@redhat.com>
Signed-off-by: Colin Lord <cl...@redhat.com>
---
 Makefile  |  3 ---
 block.c   | 62 +--
 block/Makefile.objs   |  3 +--
 include/qemu/module.h |  3 +++
 util/module.c | 38 +--
 5 files changed, 70 insertions(+), 39 deletions(-)

diff --git a/Makefile b/Makefile
index 4b9384e..c7aa8cd 100644
--- a/Makefile
+++ b/Makefile
@@ -247,9 +247,6 @@ Makefile: $(version-obj-y) $(version-lobj-y)
 libqemustub.a: $(stub-obj-y)
 libqemuutil.a: $(util-obj-y)
 
-block-modules = $(foreach o,$(block-obj-m),"$(basename $(subst /,-,$o))",) NULL
-util/module.o-cflags = -D'CONFIG_BLOCK_MODULES=$(block-modules)'
-
 ##
 
 qemu-img.o: qemu-img-cmds.h
diff --git a/block.c b/block.c
index 30d64e6..6c5e249 100644
--- a/block.c
+++ b/block.c
@@ -26,6 +26,7 @@
 #include "block/block_int.h"
 #include "block/blockjob.h"
 #include "qemu/error-report.h"
+#include "module_block.h"
 #include "qemu/module.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/qmp/qbool.h"
@@ -241,17 +242,40 @@ BlockDriverState *bdrv_new(void)
 return bs;
 }
 
-BlockDriver *bdrv_find_format(const char *format_name)
+static BlockDriver *bdrv_do_find_format(const char *format_name)
 {
 BlockDriver *drv1;
+
 QLIST_FOREACH(drv1, _drivers, list) {
 if (!strcmp(drv1->format_name, format_name)) {
 return drv1;
 }
 }
+
 return NULL;
 }
 
+BlockDriver *bdrv_find_format(const char *format_name)
+{
+BlockDriver *drv1;
+size_t i;
+
+drv1 = bdrv_do_find_format(format_name);
+if (drv1) {
+return drv1;
+}
+
+/* The driver isn't registered, maybe we need to load a module */
+for (i = 0; i < ARRAY_SIZE(block_driver_modules); ++i) {
+if (!strcmp(block_driver_modules[i].format_name, format_name)) {
+block_module_load_one(block_driver_modules[i].library_name);
+break;
+}
+}
+
+return bdrv_do_find_format(format_name);
+}
+
 static int bdrv_is_whitelisted(BlockDriver *drv, bool read_only)
 {
 static const char *whitelist_rw[] = {
@@ -460,6 +484,19 @@ static BlockDriver *find_hdev_driver(const char *filename)
 return drv;
 }
 
+static BlockDriver *bdrv_do_find_protocol(const char *protocol)
+{
+BlockDriver *drv1;
+
+QLIST_FOREACH(drv1, _drivers, list) {
+if (drv1->protocol_name && !strcmp(drv1->protocol_name, protocol)) {
+return drv1;
+}
+}
+
+return NULL;
+}
+
 BlockDriver *bdrv_find_protocol(const char *filename,
 bool allow_protocol_prefix,
 Error **errp)
@@ -468,6 +505,7 @@ BlockDriver *bdrv_find_protocol(const char *filename,
 char protocol[128];
 int len;
 const char *p;
+size_t i;
 
 /* TODO Drivers without bdrv_file_open must be specified explicitly */
 
@@ -494,15 +532,25 @@ BlockDriver *bdrv_find_protocol(const char *filename,
 len = sizeof(protocol) - 1;
 memcpy(protocol, filename, len);
 protocol[len] = '\0';
-QLIST_FOREACH(drv1, _drivers, list) {
-if (drv1->protocol_name &&
-!strcmp(drv1->protocol_name, protocol)) {
-return drv1;
+
+drv1 = bdrv_do_find_protocol(protocol);
+if (drv1) {
+return drv1;
+}
+
+for (i = 0; i < ARRAY_SIZE(block_driver_modules); ++i) {
+if (block_driver_modules[i].protocol_name &&
+!strcmp(block_driver_modules[i].protocol_name, protocol)) {
+block_module_load_one(block_driver_modules[i].library_name);
+break;
 }
 }
 
-error_setg(errp, "Unknown protocol '%s'", protocol);
-return NULL;
+drv1 = bdrv_do_find_protocol(protocol);
+if (!drv1) {
+error_setg(errp, "Unknown protocol '%s'", protocol);
+}
+return drv1;
 }
 
 /*
diff --git a/block/Makefile.objs b/block/Makefile.objs
index 2593a2f..595f366 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -1,4 +1,4 @@
-block-obj-y += raw_bsd.o qcow.o vdi.o vmdk.o cloop.o bochs.o vpc.o vvfat.o
+block-obj-y +

[Qemu-block] [PATCH 1/2] blockdev: Remove unnecessary lines from Makefile

[Colin Lord]

Removes some superfluous lines from the Makefile which should have been
removed as part of the patch series to add dynamic loading for block
driver modules.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 Makefile | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/Makefile b/Makefile
index 4b9384e..c7aa8cd 100644
--- a/Makefile
+++ b/Makefile
@@ -247,9 +247,6 @@ Makefile: $(version-obj-y) $(version-lobj-y)
 libqemustub.a: $(stub-obj-y)
 libqemuutil.a: $(util-obj-y)
 
-block-modules = $(foreach o,$(block-obj-m),"$(basename $(subst /,-,$o))",) NULL
-util/module.o-cflags = -D'CONFIG_BLOCK_MODULES=$(block-modules)'
-
 ##
 
 qemu-img.o: qemu-img-cmds.h
-- 
2.5.5

[Qemu-block] [PATCH 2/2] blockdev: Modularize nfs block driver

[Colin Lord]

Modularizes the nfs block driver so that it gets dynamically loaded.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block/Makefile.objs | 1 +
 configure   | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 595f366..fa4d8b8 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -28,6 +28,7 @@ block-obj-y += crypto.o
 common-obj-y += stream.o
 common-obj-y += backup.o
 
+nfs.o-libs := $(LIBNFS_LIBS)
 iscsi.o-cflags := $(LIBISCSI_CFLAGS)
 iscsi.o-libs   := $(LIBISCSI_LIBS)
 curl.o-cflags  := $(CURL_CFLAGS)
diff --git a/configure b/configure
index 879324b..6086d2e 100755
--- a/configure
+++ b/configure
@@ -4558,7 +4558,6 @@ if test "$libnfs" != "no" ; then
   if $pkg_config --atleast-version=1.9.3 libnfs; then
 libnfs="yes"
 libnfs_libs=$($pkg_config --libs libnfs)
-LIBS="$LIBS $libnfs_libs"
   else
 if test "$libnfs" = "yes" ; then
   feature_not_found "libnfs" "Install libnfs devel >= 1.9.3"
@@ -5317,7 +5316,8 @@ if test "$libiscsi" = "yes" ; then
 fi
 
 if test "$libnfs" = "yes" ; then
-  echo "CONFIG_LIBNFS=y" >> $config_host_mak
+  echo "CONFIG_LIBNFS=m" >> $config_host_mak
+  echo "LIBNFS_LIBS=$libnfs_libs" >> $config_host_mak
 fi
 
 if test "$seccomp" = "yes"; then
-- 
2.5.5

Re: [Qemu-block] [Qemu-devel] [PATCH v5 0/3] Dynamic module loading for block drivers

[Colin Lord]

On 07/23/2016 02:21 PM, Max Reitz wrote:
> On 20.07.2016 16:30, Colin Lord wrote:
>> Here's v5 of the modularization series. Since it seems the concensus is
>> that modularizing the format drivers is unnecessary, this series no
>> longer modularizes those and is thus much shorter than before.
>>
>> v5:
>> - No format drivers are modularized, therefore the probe functions are
>>   all being left completely untouched. The bdrv_find_format function is
>>   also left untouched as a result.
> 
> You also left the (host) device probing functions untouched in this
> revision. However, those are actually only used by protocol drivers
> (raw-posix and raw-win32, to be specific).
> 
> Probably fine since I think it's impossible to build raw-posix or
> raw-win32 as a module anyway (because bdrv_file is used as a "static"
> reference in block.c).
> 
>> - Remove dmg from block-obj-m since it is not a target of the
>>   modularization effort.
> 
> Hm, I'm afraid I don't quite understand the reasoning behind this.
> Intuitively, I'd say "Doesn't matter, it was already modular, so what
> prevents it from staying that way?"
> 
> Is it because the changes to util/module.c in patch 3 break how the dmg
> module worked, e.g. that it was always implicitly fully loaded on qemu
> startup if it was available, but now modules are only loaded on request?
> 
Yes, that's pretty much it. Previously all the modules would get loaded
during initialization, but since the third patch adds dynamic loading
that no longer happens. As we aren't loading format drivers on demand,
dmg.o should be added to block-obj-y instead of block-obj-m so that it
doesn't get modularized.

Also, I should mention that the third patch of this series is not quite
right. I was looking at some stuff with John on Friday and he found a
couple lines I somehow didn't delete from qemu/Makefile (around line 250):

block-modules = $(foreach o,$(block-obj-m),"$(basename $(subst
/,-,$o))",) NULL
util/module.o-cflags = -D'CONFIG_BLOCK_MODULES=$(block-modules)'

I was pretty sure I had taken care of these but I guess not. It doesn't
actually affect anything as all it's doing is setting
CONFIG_BLOCK_MODULES (which is no longer used anywhere once patch 3 gets
applied), but it's obviously not good to have unused code sitting
around. Should I submit another version with this fixed?

Colin

> Max
> 
>> - Modify module_block.py to only include the library name and protocol
>>   name fields in the generated struct. The other fields are no longer
>>   necessary for the drivers that are being modularized.
>>
>> v4:
>> - Fix indentation of the generated header file module_block.h
>> - Drivers and probe functions are now all located in the block/
>>   directory, rather than being split between block/ and block/probe/. In
>>   addition the header files for each probe/driver pair are in the block/
>>   directory, not the include/block/driver/ directory (which no longer
>>   exists).
>> - Since the probe files are in block/ now, they follow the naming
>>   pattern of format-probe.c
>> - Renamed crypto probe file to be crypto-probe.c, luks is no longer in
>>   the filename
>> - Fixed formatting of parallels_probe() function header
>> - Enforced consistent naming convention for the probe functions. They
>>   now follow the pattern bdrv_format_probe().
>>
>> Colin Lord (1):
>>   blockdev: prepare iSCSI block driver for dynamic loading
>>
>> Marc Mari (2):
>>   blockdev: Add dynamic generation of module_block.h
>>   blockdev: Add dynamic module loading for block drivers
>>
>>  Makefile|   7 +++
>>  block.c |  37 ---
>>  block/Makefile.objs |   3 +-
>>  block/iscsi.c   |  36 --
>>  include/qemu/module.h   |   3 ++
>>  scripts/modules/module_block.py | 102 
>> 
>>  util/module.c   |  38 +--
>>  vl.c|  38 +++
>>  8 files changed, 193 insertions(+), 71 deletions(-)
>>  create mode 100644 scripts/modules/module_block.py
>>
> 
> 

[Qemu-block] [PATCH v5 0/3] Dynamic module loading for block drivers

[Colin Lord]

Here's v5 of the modularization series. Since it seems the concensus is
that modularizing the format drivers is unnecessary, this series no
longer modularizes those and is thus much shorter than before.

v5:
- No format drivers are modularized, therefore the probe functions are
  all being left completely untouched. The bdrv_find_format function is
  also left untouched as a result.
- Remove dmg from block-obj-m since it is not a target of the
  modularization effort.
- Modify module_block.py to only include the library name and protocol
  name fields in the generated struct. The other fields are no longer
  necessary for the drivers that are being modularized.

v4:
- Fix indentation of the generated header file module_block.h
- Drivers and probe functions are now all located in the block/
  directory, rather than being split between block/ and block/probe/. In
  addition the header files for each probe/driver pair are in the block/
  directory, not the include/block/driver/ directory (which no longer
  exists).
- Since the probe files are in block/ now, they follow the naming
  pattern of format-probe.c
- Renamed crypto probe file to be crypto-probe.c, luks is no longer in
  the filename
- Fixed formatting of parallels_probe() function header
- Enforced consistent naming convention for the probe functions. They
  now follow the pattern bdrv_format_probe().

Colin Lord (1):
  blockdev: prepare iSCSI block driver for dynamic loading

Marc Mari (2):
  blockdev: Add dynamic generation of module_block.h
  blockdev: Add dynamic module loading for block drivers

 Makefile|   7 +++
 block.c |  37 ---
 block/Makefile.objs |   3 +-
 block/iscsi.c   |  36 --
 include/qemu/module.h   |   3 ++
 scripts/modules/module_block.py | 102 
 util/module.c   |  38 +--
 vl.c|  38 +++
 8 files changed, 193 insertions(+), 71 deletions(-)
 create mode 100644 scripts/modules/module_block.py

-- 
2.5.5

[Qemu-block] [PATCH v5 1/3] blockdev: prepare iSCSI block driver for dynamic loading

[Colin Lord]

This commit moves the initialization of the QemuOptsList qemu_iscsi_opts
struct out of block/iscsi.c in order to allow the iscsi module to be
dynamically loaded.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Fam Zheng <f...@redhat.com>
---
 block/iscsi.c | 36 
 vl.c  | 38 ++
 2 files changed, 38 insertions(+), 36 deletions(-)

diff --git a/block/iscsi.c b/block/iscsi.c
index 129c3af..afaf5ee 100644
--- a/block/iscsi.c
+++ b/block/iscsi.c
@@ -2018,45 +2018,9 @@ static BlockDriver bdrv_iscsi = {
 .bdrv_attach_aio_context = iscsi_attach_aio_context,
 };
 
-static QemuOptsList qemu_iscsi_opts = {
-.name = "iscsi",
-.head = QTAILQ_HEAD_INITIALIZER(qemu_iscsi_opts.head),
-.desc = {
-{
-.name = "user",
-.type = QEMU_OPT_STRING,
-.help = "username for CHAP authentication to target",
-},{
-.name = "password",
-.type = QEMU_OPT_STRING,
-.help = "password for CHAP authentication to target",
-},{
-.name = "password-secret",
-.type = QEMU_OPT_STRING,
-.help = "ID of the secret providing password for CHAP "
-"authentication to target",
-},{
-.name = "header-digest",
-.type = QEMU_OPT_STRING,
-.help = "HeaderDigest setting. "
-"{CRC32C|CRC32C-NONE|NONE-CRC32C|NONE}",
-},{
-.name = "initiator-name",
-.type = QEMU_OPT_STRING,
-.help = "Initiator iqn name to use when connecting",
-},{
-.name = "timeout",
-.type = QEMU_OPT_NUMBER,
-.help = "Request timeout in seconds (default 0 = no timeout)",
-},
-{ /* end of list */ }
-},
-};
-
 static void iscsi_block_init(void)
 {
 bdrv_register(_iscsi);
-qemu_add_opts(_iscsi_opts);
 }
 
 block_init(iscsi_block_init);
diff --git a/vl.c b/vl.c
index a455947..1f2d9c8 100644
--- a/vl.c
+++ b/vl.c
@@ -506,6 +506,41 @@ static QemuOptsList qemu_fw_cfg_opts = {
 },
 };
 
+static QemuOptsList qemu_iscsi_opts = {
+.name = "iscsi",
+.head = QTAILQ_HEAD_INITIALIZER(qemu_iscsi_opts.head),
+.desc = {
+{
+.name = "user",
+.type = QEMU_OPT_STRING,
+.help = "username for CHAP authentication to target",
+},{
+.name = "password",
+.type = QEMU_OPT_STRING,
+.help = "password for CHAP authentication to target",
+},{
+.name = "password-secret",
+.type = QEMU_OPT_STRING,
+.help = "ID of the secret providing password for CHAP "
+"authentication to target",
+},{
+.name = "header-digest",
+.type = QEMU_OPT_STRING,
+.help = "HeaderDigest setting. "
+"{CRC32C|CRC32C-NONE|NONE-CRC32C|NONE}",
+},{
+.name = "initiator-name",
+.type = QEMU_OPT_STRING,
+.help = "Initiator iqn name to use when connecting",
+},{
+.name = "timeout",
+.type = QEMU_OPT_NUMBER,
+.help = "Request timeout in seconds (default 0 = no timeout)",
+},
+{ /* end of list */ }
+},
+};
+
 /**
  * Get machine options
  *
@@ -3000,6 +3035,9 @@ int main(int argc, char **argv, char **envp)
 qemu_add_opts(_icount_opts);
 qemu_add_opts(_semihosting_config_opts);
 qemu_add_opts(_fw_cfg_opts);
+#ifdef CONFIG_LIBISCSI
+qemu_add_opts(_iscsi_opts);
+#endif
 module_call_init(MODULE_INIT_OPTS);
 
 runstate_init();
-- 
2.5.5

[Qemu-block] [PATCH v5 2/3] blockdev: Add dynamic generation of module_block.h

[Colin Lord]

From: Marc Mari <mar...@redhat.com>

To simplify the addition of new block modules, add a script that generates
module_block.h automatically from the modules' source code.

This script assumes that the QEMU coding style rules are followed.

Signed-off-by: Marc Marí <mar...@redhat.com>
Signed-off-by: Colin Lord <cl...@redhat.com>
---
 Makefile|   7 +++
 scripts/modules/module_block.py | 102 
 2 files changed, 109 insertions(+)
 create mode 100644 scripts/modules/module_block.py

diff --git a/Makefile b/Makefile
index 0d7647f..4b9384e 100644
--- a/Makefile
+++ b/Makefile
@@ -76,6 +76,8 @@ GENERATED_HEADERS += trace/generated-ust-provider.h
 GENERATED_SOURCES += trace/generated-ust.c
 endif
 
+GENERATED_HEADERS += module_block.h
+
 # Don't try to regenerate Makefile or configure
 # We don't generate any of them
 Makefile: ;
@@ -352,6 +354,11 @@ ivshmem-client$(EXESUF): $(ivshmem-client-obj-y) 
libqemuutil.a libqemustub.a
 ivshmem-server$(EXESUF): $(ivshmem-server-obj-y) libqemuutil.a libqemustub.a
$(call LINK, $^)
 
+module_block.h: $(SRC_PATH)/scripts/modules/module_block.py config-host.mak
+   $(call quiet-command,$(PYTHON) $< $@ \
+   $(addprefix $(SRC_PATH)/,$(patsubst %.mo,%.c,$(block-obj-m))), \
+   "  GEN   $@")
+
 clean:
 # avoid old build problems by removing potentially incorrect old files
rm -f config.mak op-i386.h opc-i386.h gen-op-i386.h op-arm.h opc-arm.h 
gen-op-arm.h
diff --git a/scripts/modules/module_block.py b/scripts/modules/module_block.py
new file mode 100644
index 000..8b3fcb9
--- /dev/null
+++ b/scripts/modules/module_block.py
@@ -0,0 +1,102 @@
+#!/usr/bin/python
+#
+# Module information generator
+#
+# Copyright Red Hat, Inc. 2015 - 2016
+#
+# Authors:
+#  Marc Mari <mar...@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2.
+# See the COPYING file in the top-level directory.
+
+from __future__ import print_function
+import sys
+import os
+
+def get_string_struct(line):
+data = line.split()
+
+# data[0] -> struct element name
+# data[1] -> =
+# data[2] -> value
+
+return data[2].replace('"', '')[:-1]
+
+def add_module(fheader, library, protocol_name):
+lines = []
+lines.append('.library_name = "' + library + '",')
+if protocol_name != "":
+lines.append('.protocol_name = "' + protocol_name + '",')
+
+text = '\n'.join(lines)
+fheader.write('\n{\n' + text + '\n},')
+
+def process_file(fheader, filename):
+# This parser assumes the coding style rules are being followed
+with open(filename, "r") as cfile:
+found_something = False
+found_start = False
+library, _ = os.path.splitext(os.path.basename(filename))
+for line in cfile:
+if found_start:
+line = line.replace('\n', '')
+if line.find(".protocol_name") != -1:
+protocol_name = get_string_struct(line)
+elif line == "};":
+add_module(fheader, library, protocol_name)
+found_start = False
+elif line.find("static BlockDriver") != -1:
+found_something = True
+found_start = True
+protocol_name = ""
+
+if not found_something:
+print("No BlockDriver struct found in " + filename + ". \
+Is this really a module?", file=sys.stderr)
+sys.exit(1)
+
+def print_top(fheader):
+fheader.write('''/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
+/*
+ * QEMU Block Module Infrastructure
+ *
+ * Authors:
+ *  Marc Mari   <mar...@redhat.com>
+ */
+
+''')
+
+fheader.write('''#ifndef QEMU_MODULE_BLOCK_H
+#define QEMU_MODULE_BLOCK_H
+
+#include "qemu-common.h"
+
+static const struct {
+const char *protocol_name;
+const char *library_name;
+} block_driver_modules[] = {''')
+
+def print_bottom(fheader):
+fheader.write('''
+};
+
+#endif
+''')
+
+# First argument: output file
+# All other arguments: modules source files (.c)
+output_file = sys.argv[1]
+with open(output_file, 'w') as fheader:
+print_top(fheader)
+
+for filename in sys.argv[2:]:
+if os.path.isfile(filename):
+process_file(fheader, filename)
+else:
+print("File " + filename + " does not exist.", file=sys.stderr)
+sys.exit(1)
+
+print_bottom(fheader)
+
+sys.exit(0)
-- 
2.5.5

Re: [Qemu-block] [Qemu-devel] [PATCH v4 00/32] Dynamic module loading for block drivers

[Colin Lord]

On 07/19/2016 06:54 AM, Paolo Bonzini wrote:
> 
> 
> On 14/07/2016 21:02, Colin Lord wrote:
>> Here's v4 of the modularization series. Things that have changed since
>> v3 include:
>>
>> - Fix indentation of the generated header file module_block.h
>> - Drivers and probe functions are now all located in the block/
>>   directory, rather than being split between block/ and block/probe/. In
>>   addition the header files for each probe/driver pair are in the block/
>>   directory, not the include/block/driver/ directory (which no longer
>>   exists).
>> - Since the probe files are in block/ now, they follow the naming
>>   pattern of format-probe.c
>> - Renamed crypto probe file to be crypto-probe.c, luks is no longer in
>>   the filename
>> - Fixed formatting of parallels_probe() function header
>> - Enforced consistent naming convention for the probe functions. They
>>   now follow the pattern bdrv_format_probe().
> 
> It's still not clear to me why probes need to be separate for drivers
> that (presumably) will never be modularized.
> 
I was hoping someone more experienced with this project would respond to
you the last time you asked this, so apologies if it felt like I was
ignoring it. I'll tell you what I'm seeing from my perspective though
and hopefully that can get some discussion going.

As far as performance benefits go, you are correct that modularizing the
drivers that these probes are being separated from will have essentially
no effect. At this point it's more of a question of how the project
should be organized. I can see why having modules seems like a nice
organization of things, but on the other hand it does move code around
without accomplishing anything other than a different project structure.

However, the protocol drivers do tend to see noticeable performance
benefits. Most of them get modularized in the first three patches of
this series. This includes the iscsi, glusterfs, ssh, curl, and rbd
drivers. A minimal configuration excluding all of these drivers takes
around 3 ms to reach main(), and each of these drivers respectively add
times of approximately 0.2, 0.9, 1.2, 2.5, and a massive time of 41.6 ms
to the time to main. So some of these aren't even a huge benefit by
themselves but they do add up, and also it makes sense to modularize all
of the protocol drivers if we're already modularizing one.

As for the format drivers, as I said that's a question of project
structure. The dmg driver is in this patch series simply because it was
there in Marc's patches and I didn't remove it, and only recently did I
realize that it wouldn't actually affect performance. I think the whole
deal of probes being separated just fell out from having the dmg driver
modularized. There is maybe an argument to be made that if the protocol
drivers are getting modularized, the format drivers should as well just
for consistency, but at the end of the day it's mostly just personal
preferences I think, and I will add to that that although this series
only modularizes dmg, if this series goes through I'll be working on
modularizing the rest of them as well. Most of them should be trivial to
modularize, although there may be a tricky one or two that I'm not aware
of yet.

Hopefully this has helped make the situation more clear.

Colin

> Paolo
> 
>> Colin Lord (30):
>>   blockdev: prepare iSCSI block driver for dynamic loading
>>   blockdev: Move bochs probe into separate file
>>   blockdev: Move cloop probe to its own file
>>   blockdev: Move luks probe to its own file
>>   blockdev: Move dmg probe to its own file
>>   blockdev: Move parallels probe to its own file
>>   blockdev: Move qcow probe to its own file
>>   blockdev: Move qcow2 probe to its own file
>>   blockdev: Move qed probe to its own file
>>   blockdev: Move raw probe to its own file
>>   blockdev: Move vdi probe to its own file
>>   blockdev: Move vhdx probe to its own file
>>   blockdev: Move vmdk probe to its own file
>>   blockdev: Move vpc probe to its own file
>>   blockdev: Separate bochs probe from its driver
>>   blockdev: Separate cloop probe from its driver
>>   blockdev: Separate luks probe from its driver
>>   blockdev: Separate dmg probe from its driver
>>   blockdev: Separate parallels probe from its driver
>>   blockdev: Separate qcow probe from its driver
>>   blockdev: Separate qcow2 probe from its driver
>>   blockdev: Separate qed probe from its driver
>>   blockdev: Separate raw probe from its driver
>>   blockdev: Separate vdi probe from its driver
>>   blockdev: Separate vhdx probe from its driver
>>   blockdev: Separate vmdk probe from its driver
>>   blockdev: Separate vpc probe from its driver
>>   blockdev: Remov

Re: [Qemu-block] [PATCH v4 17/32] blockdev: Separate bochs probe from its driver

[Colin Lord]

On 07/18/2016 12:28 PM, Max Reitz wrote:
> On 14.07.2016 21:03, Colin Lord wrote:
>> Modifies the bochs probe to return the format name as well as the
>> score as the final step of separating the probe function from the
>> driver. This keeps the probe completely independent of the driver,
>> making future modularization easier to accomplish. Returning the format
>> name as well as the score allows the score to be correlated to the
>> driver without the probe function needing to be part of the driver.
>>
>> Signed-off-by: Colin Lord <cl...@redhat.com>
>> ---
>>  block.c   | 20 
>>  block/bochs-probe.c   | 25 -
>>  block/bochs.c |  1 -
>>  include/block/probe.h |  3 ++-
>>  4 files changed, 38 insertions(+), 11 deletions(-)
>>
>> diff --git a/block.c b/block.c
>> index 3a0dc19..b16f8cf 100644
>> --- a/block.c
>> +++ b/block.c
>> @@ -25,6 +25,7 @@
>>  #include "trace.h"
>>  #include "block/block_int.h"
>>  #include "block/blockjob.h"
>> +#include "block/probe.h"
>>  #include "qemu/error-report.h"
>>  #include "module_block.h"
>>  #include "qemu/module.h"
>> @@ -56,6 +57,13 @@
>>  
>>  #define NOT_DONE 0x7fff /* used while emulated sync operation in 
>> progress */
>>  
>> +typedef const char *BdrvProbeFunc(const uint8_t *buf, int buf_size,
>> +  const char *filename, int *score);
>> +
>> +static BdrvProbeFunc *format_probes[] = {
>> +bdrv_bochs_probe,
>> +};
>> +
> 
> I really can't convince you of my "struct concept", can I? :-)
> 
> Unless maybe I can:
> 
> Reviewed-by: Max Reitz <mre...@redhat.com>
> 
If you really want me to I could go back and change it. I do have to
agree that it's a little odd to be returning the same string every time,
but it also strikes me as pretty weird if the probing functions
themselves can't even be associated with their drivers without the help
of another struct. It doesn't seem quite right to me if the functions
can't be used independently of the struct containing the mappings.
Technically of course they could still be used, but without the mappings
they are practically indistinguishable from each other so they're
basically useless.

Since both options seem to have their own quirks, I'm not too excited to
go back and edit all the patches only to end up with something I see as
equally weird. But again, if you (or others) really want me to change
it, or show me why having a bunch of indistinguishable probing functions
isn't as weird as I think, I could certainly do so.

Colin

[Qemu-block] [PATCH v4 30/32] blockdev: Remove the .bdrv_probe field from BlockDrivers

[Colin Lord]

This commit finalizes the separation of the block driver and probe
function by removing the .bdrv_probe field from all BlockDrivers.
Probing is now accomplished solely by iterating over the array of probe
function pointers in the format_probes array.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Max Reitz <mre...@redhat.com>
---
 block.c | 21 +
 block/raw-posix.c   |  1 -
 include/block/block_int.h   |  1 -
 scripts/modules/module_block.py | 10 ++
 4 files changed, 3 insertions(+), 30 deletions(-)

diff --git a/block.c b/block.c
index bc550e4..20b93cf 100644
--- a/block.c
+++ b/block.c
@@ -599,35 +599,16 @@ BlockDriver *bdrv_probe_all(const uint8_t *buf, int 
buf_size,
 const char *format_max = NULL;
 const char *format;
 size_t i;
-BlockDriver *drv = NULL, *d;
-
-for (i = 0; i < ARRAY_SIZE(block_driver_modules); ++i) {
-if (block_driver_modules[i].has_probe) {
-block_module_load_one(block_driver_modules[i].library_name);
-}
-}
-
-QLIST_FOREACH(d, _drivers, list) {
-if (d->bdrv_probe) {
-score = d->bdrv_probe(buf, buf_size, filename);
-if (score > score_max) {
-score_max = score;
-drv = d;
-}
-}
-}
 
 for (i = 0; i < ARRAY_SIZE(format_probes); i++) {
 format = format_probes[i](buf, buf_size, filename, );
 if (score > score_max) {
 score_max = score;
 format_max = format;
-/* TODO: move call to find_format outside this loop */
-drv = bdrv_find_format(format_max);
 }
 }
 
-return drv;
+return bdrv_find_format(format_max);
 }
 
 static int find_image_format(BdrvChild *file, const char *filename,
diff --git a/block/raw-posix.c b/block/raw-posix.c
index c979ac3..e321bcd 100644
--- a/block/raw-posix.c
+++ b/block/raw-posix.c
@@ -1936,7 +1936,6 @@ BlockDriver bdrv_file = {
 .protocol_name = "file",
 .instance_size = sizeof(BDRVRawState),
 .bdrv_needs_filename = true,
-.bdrv_probe = NULL, /* no probe for protocols */
 .bdrv_parse_filename = raw_parse_filename,
 .bdrv_file_open = raw_open,
 .bdrv_reopen_prepare = raw_reopen_prepare,
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 47b9aac..23f229d 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -99,7 +99,6 @@ struct BlockDriver {
 bool (*bdrv_recurse_is_first_non_filter)(BlockDriverState *bs,
  BlockDriverState *candidate);
 
-int (*bdrv_probe)(const uint8_t *buf, int buf_size, const char *filename);
 int (*bdrv_probe_device)(const char *filename);
 
 /* Any driver implementing this callback is expected to be able to handle
diff --git a/scripts/modules/module_block.py b/scripts/modules/module_block.py
index e5040f3..dba1717 100644
--- a/scripts/modules/module_block.py
+++ b/scripts/modules/module_block.py
@@ -24,15 +24,13 @@ def get_string_struct(line):
 return data[2].replace('"', '')[:-1]
 
 def add_module(fheader, library, format_name, protocol_name,
-probe, probe_device):
+   probe_device):
 lines = []
 lines.append('.library_name = "' + library + '",')
 if format_name != "":
 lines.append('.format_name = "' + format_name + '",')
 if protocol_name != "":
 lines.append('.protocol_name = "' + protocol_name + '",')
-if probe:
-lines.append('.has_probe = true,')
 if probe_device:
 lines.append('.has_probe_device = true,')
 
@@ -52,20 +50,17 @@ def process_file(fheader, filename):
 format_name = get_string_struct(line)
 elif line.find(".protocol_name") != -1:
 protocol_name = get_string_struct(line)
-elif line.find(".bdrv_probe") != -1:
-probe = True
 elif line.find(".bdrv_probe_device") != -1:
 probe_device = True
 elif line == "};":
 add_module(fheader, library, format_name, protocol_name,
-probe, probe_device)
+   probe_device)
 found_start = False
 elif line.find("static BlockDriver") != -1:
 found_something = True
 found_start = True
 format_name = ""
 protocol_name = ""
-probe = False
 probe_device = False
 
 if not found_something:
@@ -93,7 +88,6 @@ static const struct {
 const char *format_name;
 const char *protocol_name;
 const char *library_name;
-bool has_probe;
 bool has_probe_device;
 } block_driver_modules[] = {''')
 
-- 
2.5.5

[Qemu-block] [PATCH v4 31/32] blockdev: Separate out bdrv_probe_device functions

[Colin Lord]

This puts the bdrv_probe_device functions into their own files to
facilitate the modularization of the block drivers.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Max Reitz <mre...@redhat.com>
---
 block/Makefile.objs   |  1 +
 block/host_cdrom-probe.c  | 40 +
 block/host_device-probe.c | 30 
 block/raw-posix.c | 51 +--
 block/raw-win32.c | 10 +-
 include/block/probe.h |  2 ++
 6 files changed, 75 insertions(+), 59 deletions(-)
 create mode 100644 block/host_cdrom-probe.c
 create mode 100644 block/host_device-probe.c

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 6512073..fbc57f9 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -27,6 +27,7 @@ block-obj-y += crypto.o
 block-obj-y += bochs-probe.o cloop-probe.o crypto-probe.o dmg-probe.o
 block-obj-y += parallels-probe.o qcow-probe.o qcow2-probe.o qed-probe.o
 block-obj-y += raw-probe.o vdi-probe.o vhdx-probe.o vmdk-probe.o vpc-probe.o
+block-obj-y += host_device-probe.o host_cdrom-probe.o
 
 common-obj-y += stream.o
 common-obj-y += backup.o
diff --git a/block/host_cdrom-probe.c b/block/host_cdrom-probe.c
new file mode 100644
index 000..1886cad
--- /dev/null
+++ b/block/host_cdrom-probe.c
@@ -0,0 +1,40 @@
+#include "qemu/osdep.h"
+#include "block/probe.h"
+
+#if defined(__FreeBSD__) || defined(__FreeBSD_kernel__)
+int cdrom_probe_device(const char *filename)
+{
+if (strstart(filename, "/dev/cd", NULL) ||
+strstart(filename, "/dev/acd", NULL))
+return 100;
+return 0;
+}
+#elif defined(__linux__)
+#include 
+#include 
+int cdrom_probe_device(const char *filename)
+{
+int fd, ret;
+int prio = 0;
+struct stat st;
+
+fd = qemu_open(filename, O_RDONLY | O_NONBLOCK);
+if (fd < 0) {
+goto out;
+}
+ret = fstat(fd, );
+if (ret == -1 || !S_ISBLK(st.st_mode)) {
+goto outc;
+}
+
+/* Attempt to detect via a CDROM specific ioctl */
+ret = ioctl(fd, CDROM_DRIVE_STATUS, CDSL_CURRENT);
+if (ret >= 0)
+prio = 100;
+
+outc:
+qemu_close(fd);
+out:
+return prio;
+}
+#endif
diff --git a/block/host_device-probe.c b/block/host_device-probe.c
new file mode 100644
index 000..ebd969b
--- /dev/null
+++ b/block/host_device-probe.c
@@ -0,0 +1,30 @@
+#include "qemu/osdep.h"
+#include "block/probe.h"
+#include "qemu/cutils.h"
+
+#ifdef _WIN32
+int hdev_probe_device(const char *filename)
+{
+if (strstart(filename, "/dev/cdrom", NULL))
+return 100;
+if (is_windows_drive(filename))
+return 100;
+return 0;
+}
+#else
+int hdev_probe_device(const char *filename)
+{
+struct stat st;
+
+/* allow a dedicated CD-ROM driver to match with a higher priority */
+if (strstart(filename, "/dev/cdrom", NULL))
+return 50;
+
+if (stat(filename, ) >= 0 &&
+(S_ISCHR(st.st_mode) || S_ISBLK(st.st_mode))) {
+return 100;
+}
+
+return 0;
+}
+#endif
diff --git a/block/raw-posix.c b/block/raw-posix.c
index e321bcd..f42b74a 100644
--- a/block/raw-posix.c
+++ b/block/raw-posix.c
@@ -28,6 +28,7 @@
 #include "qemu/timer.h"
 #include "qemu/log.h"
 #include "block/block_int.h"
+#include "block/probe.h"
 #include "qemu/module.h"
 #include "trace.h"
 #include "block/thread-pool.h"
@@ -2084,22 +2085,6 @@ static void print_unmounting_directions(const char 
*file_name)
 
 #endif /* defined(__APPLE__) && defined(__MACH__) */
 
-static int hdev_probe_device(const char *filename)
-{
-struct stat st;
-
-/* allow a dedicated CD-ROM driver to match with a higher priority */
-if (strstart(filename, "/dev/cdrom", NULL))
-return 50;
-
-if (stat(filename, ) >= 0 &&
-(S_ISCHR(st.st_mode) || S_ISBLK(st.st_mode))) {
-return 100;
-}
-
-return 0;
-}
-
 static int check_hdev_writable(BDRVRawState *s)
 {
 #if defined(BLKROGET)
@@ -2441,32 +2426,6 @@ static int cdrom_open(BlockDriverState *bs, QDict 
*options, int flags,
 return raw_open_common(bs, options, flags, O_NONBLOCK, errp);
 }
 
-static int cdrom_probe_device(const char *filename)
-{
-int fd, ret;
-int prio = 0;
-struct stat st;
-
-fd = qemu_open(filename, O_RDONLY | O_NONBLOCK);
-if (fd < 0) {
-goto out;
-}
-ret = fstat(fd, );
-if (ret == -1 || !S_ISBLK(st.st_mode)) {
-goto outc;
-}
-
-/* Attempt to detect via a CDROM specific ioctl */
-ret = ioctl(fd, CDROM_DRIVE_STATUS, CDSL_CURRENT);
-if (ret >= 0)
-prio = 100;
-
-outc:
-qemu_close(fd);
-out:
-return prio;
-}
-
 static bool cdrom_is_inserted(BlockDriverState *bs)
 {
 BDRVRawState *s = bs

[Qemu-block] [PATCH v4 32/32] blockdev: Remove bdrv_probe_device field from BlockDriver

[Colin Lord]

This commit finalizes the separation of the BlockDriver from its
device probing function. Now the accesses to these functions in block.c
occur through the protocol_probes array, and each function returns a
score and protocol name with which to find the corresponding driver.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Max Reitz <mre...@redhat.com>
---
 block.c | 46 ++---
 block/host_cdrom-probe.c| 23 ++---
 block/host_device-probe.c   | 34 --
 block/raw-posix.c   |  3 ---
 block/raw-win32.c   |  1 -
 include/block/block_int.h   |  2 --
 include/block/probe.h   |  4 ++--
 scripts/modules/module_block.py | 12 ++-
 8 files changed, 76 insertions(+), 49 deletions(-)

diff --git a/block.c b/block.c
index 20b93cf..a129376 100644
--- a/block.c
+++ b/block.c
@@ -59,6 +59,7 @@
 
 typedef const char *BdrvProbeFunc(const uint8_t *buf, int buf_size,
   const char *filename, int *score);
+typedef const char *BdrvProbeDevFunc(const char *filename, int *score);
 
 static BdrvProbeFunc *format_probes[] = {
 bdrv_bochs_probe,
@@ -76,6 +77,13 @@ static BdrvProbeFunc *format_probes[] = {
 bdrv_vpc_probe
 };
 
+static BdrvProbeDevFunc *protocol_probes[] = {
+hdev_probe_device,
+#if defined(__FreeBSD__) || defined(__FreeBSD_kernel__) || defined(__linux__)
+cdrom_probe_device
+#endif
+};
+
 static QTAILQ_HEAD(, BlockDriverState) graph_bdrv_states =
 QTAILQ_HEAD_INITIALIZER(graph_bdrv_states);
 
@@ -95,6 +103,8 @@ static BlockDriverState *bdrv_open_inherit(const char 
*filename,
 /* If non-zero, use only whitelisted block drivers */
 static int use_bdrv_whitelist;
 
+static BlockDriver *bdrv_do_find_protocol(const char *protocol);
+
 #ifdef _WIN32
 static int is_windows_drive_prefix(const char *filename)
 {
@@ -487,25 +497,37 @@ int get_tmp_filename(char *filename, int size)
 static BlockDriver *find_hdev_driver(const char *filename)
 {
 int score_max = 0, score;
+const char *protocol_max = NULL;
+const char *protocol;
+BlockDriver *drv;
 size_t i;
-BlockDriver *drv = NULL, *d;
+
+for (i = 0; i < ARRAY_SIZE(protocol_probes); i++) {
+protocol = protocol_probes[i](filename, );
+if (score > score_max) {
+protocol_max = protocol;
+score_max = score;
+}
+}
+
+if (!protocol_max) {
+return NULL;
+}
+
+drv = bdrv_do_find_protocol(protocol_max);
+if (drv) {
+return drv;
+}
 
 for (i = 0; i < ARRAY_SIZE(block_driver_modules); ++i) {
-if (block_driver_modules[i].has_probe_device) {
+if (block_driver_modules[i].protocol_name &&
+!strcmp(block_driver_modules[i].protocol_name, protocol_max)) {
 block_module_load_one(block_driver_modules[i].library_name);
+break;
 }
 }
 
-QLIST_FOREACH(d, _drivers, list) {
-if (d->bdrv_probe_device) {
-score = d->bdrv_probe_device(filename);
-if (score > score_max) {
-score_max = score;
-drv = d;
-}
-}
-}
-
+drv = bdrv_do_find_protocol(protocol);
 return drv;
 }
 
diff --git a/block/host_cdrom-probe.c b/block/host_cdrom-probe.c
index 1886cad..3f7d863 100644
--- a/block/host_cdrom-probe.c
+++ b/block/host_cdrom-probe.c
@@ -1,22 +1,28 @@
 #include "qemu/osdep.h"
 #include "block/probe.h"
 
+static const char *protocol = "host_cdrom";
+
 #if defined(__FreeBSD__) || defined(__FreeBSD_kernel__)
-int cdrom_probe_device(const char *filename)
+const char *cdrom_probe_device(const char *filename, int *score)
 {
+assert(score);
 if (strstart(filename, "/dev/cd", NULL) ||
-strstart(filename, "/dev/acd", NULL))
-return 100;
+strstart(filename, "/dev/acd", NULL)) {
+*score = 100;
+return protocol;
+}
 return 0;
 }
 #elif defined(__linux__)
 #include 
 #include 
-int cdrom_probe_device(const char *filename)
+const char *cdrom_probe_device(const char *filename, int *score)
 {
 int fd, ret;
-int prio = 0;
 struct stat st;
+assert(score);
+*score = 0;
 
 fd = qemu_open(filename, O_RDONLY | O_NONBLOCK);
 if (fd < 0) {
@@ -29,12 +35,13 @@ int cdrom_probe_device(const char *filename)
 
 /* Attempt to detect via a CDROM specific ioctl */
 ret = ioctl(fd, CDROM_DRIVE_STATUS, CDSL_CURRENT);
-if (ret >= 0)
-prio = 100;
+if (ret >= 0) {
+*score = 100;
+}
 
 outc:
 qemu_close(fd);
 out:
-return prio;
+return protocol;
 }
 #endif
diff --git a/block/host_device-probe.c b/block/host_device-probe.c
index ebd969b..b4e4d20 100644
--- a/block/host_device-probe.c
+++ b/block/host_device-probe.c
@@ -2,29

[Qemu-block] [PATCH v4 28/32] blockdev: Separate vmdk probe from its driver

[Colin Lord]

Completes the separation of the vmdk probe from the vmdk driver. The
vmdk probe now returns the format in addition to the score, allowing
correlation of the score and driver without the probe function being
part of the driver itself.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Max Reitz <mre...@redhat.com>
---
 block.c   |  1 +
 block/vmdk-probe.c| 23 +++
 block/vmdk.c  |  1 -
 include/block/probe.h |  3 ++-
 4 files changed, 18 insertions(+), 10 deletions(-)

diff --git a/block.c b/block.c
index 941e208..bb2faf8 100644
--- a/block.c
+++ b/block.c
@@ -72,6 +72,7 @@ static BdrvProbeFunc *format_probes[] = {
 bdrv_raw_probe,
 bdrv_vdi_probe,
 bdrv_vhdx_probe,
+bdrv_vmdk_probe,
 };
 
 static QTAILQ_HEAD(, BlockDriverState) graph_bdrv_states =
diff --git a/block/vmdk-probe.c b/block/vmdk-probe.c
index 77b06af..86d6f46 100644
--- a/block/vmdk-probe.c
+++ b/block/vmdk-probe.c
@@ -3,17 +3,22 @@
 #include "block/probe.h"
 #include "vmdk.h"
 
-int vmdk_probe(const uint8_t *buf, int buf_size, const char *filename)
+const char *bdrv_vmdk_probe(const uint8_t *buf, int buf_size,
+const char *filename, int *score)
 {
+const char *format = "vmdk";
 uint32_t magic;
+assert(score);
+*score = 0;
 
 if (buf_size < 4) {
-return 0;
+return format;
 }
 magic = be32_to_cpu(*(uint32_t *)buf);
 if (magic == VMDK3_MAGIC ||
 magic == VMDK4_MAGIC) {
-return 100;
+*score = 100;
+return format;
 } else {
 const char *p = (const char *)buf;
 const char *end = p + buf_size;
@@ -36,7 +41,7 @@ int vmdk_probe(const uint8_t *buf, int buf_size, const char 
*filename)
 }
 /* only accept blank lines before 'version=' line */
 if (p == end || *p != '\n') {
-return 0;
+return format;
 }
 p++;
 continue;
@@ -44,17 +49,19 @@ int vmdk_probe(const uint8_t *buf, int buf_size, const char 
*filename)
 if (end - p >= strlen("version=X\n")) {
 if (strncmp("version=1\n", p, strlen("version=1\n")) == 0 ||
 strncmp("version=2\n", p, strlen("version=2\n")) == 0) {
-return 100;
+*score = 100;
+return format;
 }
 }
 if (end - p >= strlen("version=X\r\n")) {
 if (strncmp("version=1\r\n", p, strlen("version=1\r\n")) == 0 
||
 strncmp("version=2\r\n", p, strlen("version=2\r\n")) == 0) 
{
-return 100;
+*score = 100;
+return format;
 }
 }
-return 0;
+return format;
 }
-return 0;
+return format;
 }
 }
diff --git a/block/vmdk.c b/block/vmdk.c
index d2699b0..fb33362 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -2331,7 +2331,6 @@ static QemuOptsList vmdk_create_opts = {
 static BlockDriver bdrv_vmdk = {
 .format_name  = "vmdk",
 .instance_size= sizeof(BDRVVmdkState),
-.bdrv_probe   = vmdk_probe,
 .bdrv_open= vmdk_open,
 .bdrv_check   = vmdk_check,
 .bdrv_reopen_prepare  = vmdk_reopen_prepare,
diff --git a/include/block/probe.h b/include/block/probe.h
index b7eb00e..8f2c26c 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -1,7 +1,6 @@
 #ifndef PROBE_H
 #define PROBE_H
 
-int vmdk_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vpc_probe(const uint8_t *buf, int buf_size, const char *filename);
 const char *bdrv_bochs_probe(const uint8_t *buf, int buf_size,
  const char *filename, int *score);
@@ -25,5 +24,7 @@ const char *bdrv_vdi_probe(const uint8_t *buf, int buf_size,
const char *filename, int *score);
 const char *bdrv_vhdx_probe(const uint8_t *buf, int buf_size,
 const char *filename, int *score);
+const char *bdrv_vmdk_probe(const uint8_t *buf, int buf_size,
+const char *filename, int *score);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v4 23/32] blockdev: Separate qcow2 probe from its driver

[Colin Lord]

Completes the separation of the qcow2 probe from the qcow2 driver. The
qcow2 probe now returns the format in addition to the score, allowing
correlation of the score and driver without the probe function being
part of the driver itself.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Max Reitz <mre...@redhat.com>
---
 block.c   |  1 +
 block/qcow2-probe.c   | 16 +++-
 block/qcow2.c |  1 -
 include/block/probe.h |  3 ++-
 4 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/block.c b/block.c
index f531ad0..145be9b 100644
--- a/block.c
+++ b/block.c
@@ -67,6 +67,7 @@ static BdrvProbeFunc *format_probes[] = {
 bdrv_dmg_probe,
 bdrv_parallels_probe,
 bdrv_qcow_probe,
+bdrv_qcow2_probe,
 };
 
 static QTAILQ_HEAD(, BlockDriverState) graph_bdrv_states =
diff --git a/block/qcow2-probe.c b/block/qcow2-probe.c
index e5cd92c..b7a73c5 100644
--- a/block/qcow2-probe.c
+++ b/block/qcow2-probe.c
@@ -3,14 +3,20 @@
 #include "block/probe.h"
 #include "qcow2.h"
 
-int qcow2_probe(const uint8_t *buf, int buf_size, const char *filename)
+const char *bdrv_qcow2_probe(const uint8_t *buf, int buf_size,
+ const char *filename, int *score)
 {
+const char *format = "qcow2";
 const QCowHeader *cow_header = (const void *)buf;
+assert(score);
 
 if (buf_size >= sizeof(QCowHeader) &&
 be32_to_cpu(cow_header->magic) == QCOW_MAGIC &&
-be32_to_cpu(cow_header->version) >= 2)
-return 100;
-else
-return 0;
+be32_to_cpu(cow_header->version) >= 2) {
+*score = 100;
+return format;
+}
+
+*score = 0;
+return format;
 }
diff --git a/block/qcow2.c b/block/qcow2.c
index 17521f7..2f575d1 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -3382,7 +3382,6 @@ static QemuOptsList qcow2_create_opts = {
 BlockDriver bdrv_qcow2 = {
 .format_name= "qcow2",
 .instance_size  = sizeof(BDRVQcow2State),
-.bdrv_probe = qcow2_probe,
 .bdrv_open  = qcow2_open,
 .bdrv_close = qcow2_close,
 .bdrv_reopen_prepare  = qcow2_reopen_prepare,
diff --git a/include/block/probe.h b/include/block/probe.h
index e670341..1a68cef 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -1,7 +1,6 @@
 #ifndef PROBE_H
 #define PROBE_H
 
-int qcow2_probe(const uint8_t *buf, int buf_size, const char *filename);
 int bdrv_qed_probe(const uint8_t *buf, int buf_size, const char *filename);
 int raw_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vdi_probe(const uint8_t *buf, int buf_size, const char *filename);
@@ -20,5 +19,7 @@ const char *bdrv_parallels_probe(const uint8_t *buf, int 
buf_size,
  const char *filename, int *score);
 const char *bdrv_qcow_probe(const uint8_t *buf, int buf_size,
 const char *filename, int *score);
+const char *bdrv_qcow2_probe(const uint8_t *buf, int buf_size,
+ const char *filename, int *score);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v4 16/32] blockdev: Move vpc probe to its own file

[Colin Lord]

Isolates vpc probe as part of the modularization process.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Max Reitz <mre...@redhat.com>
---
 block/Makefile.objs   | 2 +-
 block/vpc-probe.c | 9 +
 block/vpc.c   | 8 +---
 include/block/probe.h | 1 +
 4 files changed, 12 insertions(+), 8 deletions(-)
 create mode 100644 block/vpc-probe.c

diff --git a/block/Makefile.objs b/block/Makefile.objs
index f86ab63..6512073 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -26,7 +26,7 @@ block-obj-y += write-threshold.o
 block-obj-y += crypto.o
 block-obj-y += bochs-probe.o cloop-probe.o crypto-probe.o dmg-probe.o
 block-obj-y += parallels-probe.o qcow-probe.o qcow2-probe.o qed-probe.o
-block-obj-y += raw-probe.o vdi-probe.o vhdx-probe.o vmdk-probe.o
+block-obj-y += raw-probe.o vdi-probe.o vhdx-probe.o vmdk-probe.o vpc-probe.o
 
 common-obj-y += stream.o
 common-obj-y += backup.o
diff --git a/block/vpc-probe.c b/block/vpc-probe.c
new file mode 100644
index 000..afe8271
--- /dev/null
+++ b/block/vpc-probe.c
@@ -0,0 +1,9 @@
+#include "qemu/osdep.h"
+#include "block/probe.h"
+
+int vpc_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+if (buf_size >= 8 && !strncmp((char *)buf, "conectix", 8))
+   return 100;
+return 0;
+}
diff --git a/block/vpc.c b/block/vpc.c
index 43707ed..7ab3e7f 100644
--- a/block/vpc.c
+++ b/block/vpc.c
@@ -26,6 +26,7 @@
 #include "qapi/error.h"
 #include "qemu-common.h"
 #include "block/block_int.h"
+#include "block/probe.h"
 #include "sysemu/block-backend.h"
 #include "qemu/module.h"
 #include "migration/migration.h"
@@ -179,13 +180,6 @@ static uint32_t vpc_checksum(uint8_t* buf, size_t size)
 }
 
 
-static int vpc_probe(const uint8_t *buf, int buf_size, const char *filename)
-{
-if (buf_size >= 8 && !strncmp((char *)buf, "conectix", 8))
-   return 100;
-return 0;
-}
-
 static void vpc_parse_options(BlockDriverState *bs, QemuOpts *opts,
   Error **errp)
 {
diff --git a/include/block/probe.h b/include/block/probe.h
index 392515d..6cf878b 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -14,5 +14,6 @@ int raw_probe(const uint8_t *buf, int buf_size, const char 
*filename);
 int vdi_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vhdx_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vmdk_probe(const uint8_t *buf, int buf_size, const char *filename);
+int vpc_probe(const uint8_t *buf, int buf_size, const char *filename);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v4 19/32] blockdev: Separate luks probe from its driver

[Colin Lord]

Completes the separation of the luks probe from the crypto driver. The
luks probe now returns the format in addition to the score, allowing
correlation of the score and driver without the probe function being
part of the driver itself.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Max Reitz <mre...@redhat.com>
---
 block.c   |  1 +
 block/crypto-probe.c  | 13 -
 block/crypto.c|  1 -
 include/block/probe.h |  4 ++--
 4 files changed, 11 insertions(+), 8 deletions(-)

diff --git a/block.c b/block.c
index 8d21f87..6a144cc 100644
--- a/block.c
+++ b/block.c
@@ -63,6 +63,7 @@ typedef const char *BdrvProbeFunc(const uint8_t *buf, int 
buf_size,
 static BdrvProbeFunc *format_probes[] = {
 bdrv_bochs_probe,
 bdrv_cloop_probe,
+bdrv_crypto_probe_luks,
 };
 
 static QTAILQ_HEAD(, BlockDriverState) graph_bdrv_states =
diff --git a/block/crypto-probe.c b/block/crypto-probe.c
index 5c6427a..b372613 100644
--- a/block/crypto-probe.c
+++ b/block/crypto-probe.c
@@ -15,9 +15,12 @@ static int block_crypto_probe_generic(QCryptoBlockFormat 
format,
 }
 }
 
-int block_crypto_probe_luks(const uint8_t *buf,
-   int buf_size,
-   const char *filename) {
-return block_crypto_probe_generic(Q_CRYPTO_BLOCK_FORMAT_LUKS,
-  buf, buf_size, filename);
+const char *bdrv_crypto_probe_luks(const uint8_t *buf, int buf_size,
+   const char *filename, int *score)
+{
+const char *format = "luks";
+assert(score);
+*score = block_crypto_probe_generic(Q_CRYPTO_BLOCK_FORMAT_LUKS,
+buf, buf_size, filename);
+return format;
 }
diff --git a/block/crypto.c b/block/crypto.c
index 5706ba6..bba350e 100644
--- a/block/crypto.c
+++ b/block/crypto.c
@@ -547,7 +547,6 @@ static int block_crypto_create_luks(const char *filename,
 BlockDriver bdrv_crypto_luks = {
 .format_name= "luks",
 .instance_size  = sizeof(BlockCrypto),
-.bdrv_probe = block_crypto_probe_luks,
 .bdrv_open  = block_crypto_open_luks,
 .bdrv_close = block_crypto_close,
 .bdrv_create= block_crypto_create_luks,
diff --git a/include/block/probe.h b/include/block/probe.h
index e434af5..b15dd91 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -1,8 +1,6 @@
 #ifndef PROBE_H
 #define PROBE_H
 
-int block_crypto_probe_luks(const uint8_t *buf, int buf_size,
-const char *filename);
 int dmg_probe(const uint8_t *buf, int buf_size, const char *filename);
 int parallels_probe(const uint8_t *buf, int buf_size, const char *filename);
 int qcow_probe(const uint8_t *buf, int buf_size, const char *filename);
@@ -17,5 +15,7 @@ const char *bdrv_bochs_probe(const uint8_t *buf, int buf_size,
  const char *filename, int *score);
 const char *bdrv_cloop_probe(const uint8_t *buf, int buf_size,
  const char *filename, int *score);
+const char *bdrv_crypto_probe_luks(const uint8_t *buf, int buf_size,
+   const char *filename, int *score);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v4 21/32] blockdev: Separate parallels probe from its driver

[Colin Lord]

Completes the separation of the parallels probe from the parallels
driver. The parallels probe now returns the format in addition to the
score, allowing correlation of the score and driver without the probe
function being part of the driver itself.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Max Reitz <mre...@redhat.com>
---
 block.c |  1 +
 block/parallels-probe.c | 17 +++--
 block/parallels.c   |  1 -
 include/block/probe.h   |  3 ++-
 4 files changed, 14 insertions(+), 8 deletions(-)

diff --git a/block.c b/block.c
index f7dc4a4..18974c6 100644
--- a/block.c
+++ b/block.c
@@ -65,6 +65,7 @@ static BdrvProbeFunc *format_probes[] = {
 bdrv_cloop_probe,
 bdrv_crypto_probe_luks,
 bdrv_dmg_probe,
+bdrv_parallels_probe,
 };
 
 static QTAILQ_HEAD(, BlockDriverState) graph_bdrv_states =
diff --git a/block/parallels-probe.c b/block/parallels-probe.c
index ff2f6ba..659a09b 100644
--- a/block/parallels-probe.c
+++ b/block/parallels-probe.c
@@ -3,19 +3,24 @@
 #include "block/probe.h"
 #include "parallels.h"
 
-int parallels_probe(const uint8_t *buf, int buf_size, const char *filename)
+const char *bdrv_parallels_probe(const uint8_t *buf, int buf_size,
+ const char *filename, int *score)
 {
+const char *format = "parallels";
 const ParallelsHeader *ph = (const void *)buf;
+assert(score);
+*score = 0;
 
 if (buf_size < sizeof(ParallelsHeader)) {
-return 0;
+return format;
 }
 
 if ((!memcmp(ph->magic, HEADER_MAGIC, 16) ||
-   !memcmp(ph->magic, HEADER_MAGIC2, 16)) &&
-   (le32_to_cpu(ph->version) == HEADER_VERSION)) {
-return 100;
+!memcmp(ph->magic, HEADER_MAGIC2, 16)) &&
+(le32_to_cpu(ph->version) == HEADER_VERSION)) {
+*score = 100;
+return format;
 }
 
-return 0;
+return format;
 }
diff --git a/block/parallels.c b/block/parallels.c
index 70e9439..88294f7 100644
--- a/block/parallels.c
+++ b/block/parallels.c
@@ -709,7 +709,6 @@ static QemuOptsList parallels_create_opts = {
 static BlockDriver bdrv_parallels = {
 .format_name   = "parallels",
 .instance_size = sizeof(BDRVParallelsState),
-.bdrv_probe= parallels_probe,
 .bdrv_open = parallels_open,
 .bdrv_close= parallels_close,
 .bdrv_co_get_block_status = parallels_co_get_block_status,
diff --git a/include/block/probe.h b/include/block/probe.h
index bd5a532..090e0ae 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -1,7 +1,6 @@
 #ifndef PROBE_H
 #define PROBE_H
 
-int parallels_probe(const uint8_t *buf, int buf_size, const char *filename);
 int qcow_probe(const uint8_t *buf, int buf_size, const char *filename);
 int qcow2_probe(const uint8_t *buf, int buf_size, const char *filename);
 int bdrv_qed_probe(const uint8_t *buf, int buf_size, const char *filename);
@@ -18,5 +17,7 @@ const char *bdrv_crypto_probe_luks(const uint8_t *buf, int 
buf_size,
const char *filename, int *score);
 const char *bdrv_dmg_probe(const uint8_t *buf, int buf_size,
const char *filename, int *score);
+const char *bdrv_parallels_probe(const uint8_t *buf, int buf_size,
+ const char *filename, int *score);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v4 18/32] blockdev: Separate cloop probe from its driver

[Colin Lord]

Completes the separation of the cloop probe from the cloop driver. The
cloop probe now returns the format in addition to the score, allowing
correlation of the score and driver without the probe function being
part of the driver itself.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Max Reitz <mre...@redhat.com>
---
 block.c   |  1 +
 block/cloop-probe.c   | 11 ---
 block/cloop.c |  1 -
 include/block/probe.h |  3 ++-
 4 files changed, 11 insertions(+), 5 deletions(-)

diff --git a/block.c b/block.c
index b16f8cf..8d21f87 100644
--- a/block.c
+++ b/block.c
@@ -62,6 +62,7 @@ typedef const char *BdrvProbeFunc(const uint8_t *buf, int 
buf_size,
 
 static BdrvProbeFunc *format_probes[] = {
 bdrv_bochs_probe,
+bdrv_cloop_probe,
 };
 
 static QTAILQ_HEAD(, BlockDriverState) graph_bdrv_states =
diff --git a/block/cloop-probe.c b/block/cloop-probe.c
index 955c29c..779b377 100644
--- a/block/cloop-probe.c
+++ b/block/cloop-probe.c
@@ -1,17 +1,22 @@
 #include "qemu/osdep.h"
 #include "block/probe.h"
 
-int cloop_probe(const uint8_t *buf, int buf_size, const char *filename)
+const char *bdrv_cloop_probe(const uint8_t *buf, int buf_size,
+ const char *filename, int *score)
 {
+const char *format = "cloop";
 const char *magic_version_2_0 = "#!/bin/sh\n"
 "#V2.0 Format\n"
 "modprobe cloop file=$0 && mount -r -t iso9660 /dev/cloop $1\n";
 int length = strlen(magic_version_2_0);
+assert(score);
 if (length > buf_size) {
 length = buf_size;
 }
 if (!memcmp(magic_version_2_0, buf, length)) {
-return 2;
+*score = 2;
+return format;
 }
-return 0;
+*score = 0;
+return format;
 }
diff --git a/block/cloop.c b/block/cloop.c
index c897e58..a2555d3 100644
--- a/block/cloop.c
+++ b/block/cloop.c
@@ -268,7 +268,6 @@ static void cloop_close(BlockDriverState *bs)
 static BlockDriver bdrv_cloop = {
 .format_name= "cloop",
 .instance_size  = sizeof(BDRVCloopState),
-.bdrv_probe = cloop_probe,
 .bdrv_open  = cloop_open,
 .bdrv_refresh_limits = cloop_refresh_limits,
 .bdrv_co_preadv = cloop_co_preadv,
diff --git a/include/block/probe.h b/include/block/probe.h
index aac1fca..e434af5 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -1,7 +1,6 @@
 #ifndef PROBE_H
 #define PROBE_H
 
-int cloop_probe(const uint8_t *buf, int buf_size, const char *filename);
 int block_crypto_probe_luks(const uint8_t *buf, int buf_size,
 const char *filename);
 int dmg_probe(const uint8_t *buf, int buf_size, const char *filename);
@@ -16,5 +15,7 @@ int vmdk_probe(const uint8_t *buf, int buf_size, const char 
*filename);
 int vpc_probe(const uint8_t *buf, int buf_size, const char *filename);
 const char *bdrv_bochs_probe(const uint8_t *buf, int buf_size,
  const char *filename, int *score);
+const char *bdrv_cloop_probe(const uint8_t *buf, int buf_size,
+ const char *filename, int *score);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v4 12/32] blockdev: Move raw probe to its own file

[Colin Lord]

Isolate raw probe as part of the modularization process.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Max Reitz <mre...@redhat.com>
---
 block/Makefile.objs   |  1 +
 block/raw-probe.c | 10 ++
 block/raw_bsd.c   |  9 +
 include/block/probe.h |  1 +
 4 files changed, 13 insertions(+), 8 deletions(-)
 create mode 100644 block/raw-probe.c

diff --git a/block/Makefile.objs b/block/Makefile.objs
index db9faef..07b0a2d 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -26,6 +26,7 @@ block-obj-y += write-threshold.o
 block-obj-y += crypto.o
 block-obj-y += bochs-probe.o cloop-probe.o crypto-probe.o dmg-probe.o
 block-obj-y += parallels-probe.o qcow-probe.o qcow2-probe.o qed-probe.o
+block-obj-y += raw-probe.o
 
 common-obj-y += stream.o
 common-obj-y += backup.o
diff --git a/block/raw-probe.c b/block/raw-probe.c
new file mode 100644
index 000..22c6bcb
--- /dev/null
+++ b/block/raw-probe.c
@@ -0,0 +1,10 @@
+#include "qemu/osdep.h"
+#include "block/probe.h"
+
+int raw_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+/* smallest possible positive score so that raw is used if and only if no
+ * other block driver works
+ */
+return 1;
+}
diff --git a/block/raw_bsd.c b/block/raw_bsd.c
index 5f9dd29..25d5185 100644
--- a/block/raw_bsd.c
+++ b/block/raw_bsd.c
@@ -28,6 +28,7 @@
 
 #include "qemu/osdep.h"
 #include "block/block_int.h"
+#include "block/probe.h"
 #include "qapi/error.h"
 #include "qemu/option.h"
 
@@ -213,14 +214,6 @@ static void raw_close(BlockDriverState *bs)
 {
 }
 
-static int raw_probe(const uint8_t *buf, int buf_size, const char *filename)
-{
-/* smallest possible positive score so that raw is used if and only if no
- * other block driver works
- */
-return 1;
-}
-
 static int raw_probe_blocksizes(BlockDriverState *bs, BlockSizes *bsz)
 {
 return bdrv_probe_blocksizes(bs->file->bs, bsz);
diff --git a/include/block/probe.h b/include/block/probe.h
index e3e9934..053f961 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -10,5 +10,6 @@ int parallels_probe(const uint8_t *buf, int buf_size, const 
char *filename);
 int qcow_probe(const uint8_t *buf, int buf_size, const char *filename);
 int qcow2_probe(const uint8_t *buf, int buf_size, const char *filename);
 int bdrv_qed_probe(const uint8_t *buf, int buf_size, const char *filename);
+int raw_probe(const uint8_t *buf, int buf_size, const char *filename);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v4 22/32] blockdev: Separate qcow probe from its driver

[Colin Lord]

Completes the separation of the qcow probe from the qcow driver. The
qcow probe now returns the format in addition to the score, allowing
correlation of the score and driver without the probe function being
part of the driver itself.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Max Reitz <mre...@redhat.com>
---
 block.c   |  1 +
 block/qcow-probe.c| 16 +++-
 block/qcow.c  |  1 -
 include/block/probe.h |  3 ++-
 4 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/block.c b/block.c
index 18974c6..f531ad0 100644
--- a/block.c
+++ b/block.c
@@ -66,6 +66,7 @@ static BdrvProbeFunc *format_probes[] = {
 bdrv_crypto_probe_luks,
 bdrv_dmg_probe,
 bdrv_parallels_probe,
+bdrv_qcow_probe,
 };
 
 static QTAILQ_HEAD(, BlockDriverState) graph_bdrv_states =
diff --git a/block/qcow-probe.c b/block/qcow-probe.c
index c5fd214..36bbd05 100644
--- a/block/qcow-probe.c
+++ b/block/qcow-probe.c
@@ -3,14 +3,20 @@
 #include "block/probe.h"
 #include "qcow.h"
 
-int qcow_probe(const uint8_t *buf, int buf_size, const char *filename)
+const char *bdrv_qcow_probe(const uint8_t *buf, int buf_size,
+const char *filename, int *score)
 {
+const char *format = "qcow";
 const QCowHeader *cow_header = (const void *)buf;
+assert(score);
 
 if (buf_size >= sizeof(QCowHeader) &&
 be32_to_cpu(cow_header->magic) == QCOW_MAGIC &&
-be32_to_cpu(cow_header->version) == QCOW_VERSION)
-return 100;
-else
-return 0;
+be32_to_cpu(cow_header->version) == QCOW_VERSION) {
+*score = 100;
+return format;
+}
+
+*score = 0;
+return format;
 }
diff --git a/block/qcow.c b/block/qcow.c
index 6ea1164..5144c8d 100644
--- a/block/qcow.c
+++ b/block/qcow.c
@@ -1044,7 +1044,6 @@ static QemuOptsList qcow_create_opts = {
 static BlockDriver bdrv_qcow = {
 .format_name   = "qcow",
 .instance_size = sizeof(BDRVQcowState),
-.bdrv_probe= qcow_probe,
 .bdrv_open = qcow_open,
 .bdrv_close= qcow_close,
 .bdrv_reopen_prepare= qcow_reopen_prepare,
diff --git a/include/block/probe.h b/include/block/probe.h
index 090e0ae..e670341 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -1,7 +1,6 @@
 #ifndef PROBE_H
 #define PROBE_H
 
-int qcow_probe(const uint8_t *buf, int buf_size, const char *filename);
 int qcow2_probe(const uint8_t *buf, int buf_size, const char *filename);
 int bdrv_qed_probe(const uint8_t *buf, int buf_size, const char *filename);
 int raw_probe(const uint8_t *buf, int buf_size, const char *filename);
@@ -19,5 +18,7 @@ const char *bdrv_dmg_probe(const uint8_t *buf, int buf_size,
const char *filename, int *score);
 const char *bdrv_parallels_probe(const uint8_t *buf, int buf_size,
  const char *filename, int *score);
+const char *bdrv_qcow_probe(const uint8_t *buf, int buf_size,
+const char *filename, int *score);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v4 14/32] blockdev: Move vhdx probe to its own file

[Colin Lord]

Isolates vhdx probe as part of the modularization process.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Max Reitz <mre...@redhat.com>
---
 block/Makefile.objs   |  2 +-
 block/vhdx-probe.c| 21 +
 block/vhdx.c  | 20 +---
 include/block/probe.h |  1 +
 4 files changed, 24 insertions(+), 20 deletions(-)
 create mode 100644 block/vhdx-probe.c

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 9f9e83f..4363667 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -26,7 +26,7 @@ block-obj-y += write-threshold.o
 block-obj-y += crypto.o
 block-obj-y += bochs-probe.o cloop-probe.o crypto-probe.o dmg-probe.o
 block-obj-y += parallels-probe.o qcow-probe.o qcow2-probe.o qed-probe.o
-block-obj-y += raw-probe.o vdi-probe.o
+block-obj-y += raw-probe.o vdi-probe.o vhdx-probe.o
 
 common-obj-y += stream.o
 common-obj-y += backup.o
diff --git a/block/vhdx-probe.c b/block/vhdx-probe.c
new file mode 100644
index 000..6c38aac
--- /dev/null
+++ b/block/vhdx-probe.c
@@ -0,0 +1,21 @@
+#include "qemu/osdep.h"
+#include "block/probe.h"
+
+/*
+ * Per the MS VHDX Specification, for every VHDX file:
+ *  - The header section is fixed size - 1 MB
+ *  - The header section is always the first "object"
+ *  - The first 64KB of the header is the File Identifier
+ *  - The first uint64 (8 bytes) is the VHDX Signature ("vhdxfile")
+ *  - The following 512 bytes constitute a UTF-16 string identifiying the
+ *software that created the file, and is optional and diagnostic only.
+ *
+ *  Therefore, we probe by looking for the vhdxfile signature "vhdxfile"
+ */
+int vhdx_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+if (buf_size >= 8 && !memcmp(buf, "vhdxfile", 8)) {
+return 100;
+}
+return 0;
+}
diff --git a/block/vhdx.c b/block/vhdx.c
index 75ef2b1..2f13c61 100644
--- a/block/vhdx.c
+++ b/block/vhdx.c
@@ -19,6 +19,7 @@
 #include "qapi/error.h"
 #include "qemu-common.h"
 #include "block/block_int.h"
+#include "block/probe.h"
 #include "sysemu/block-backend.h"
 #include "qemu/module.h"
 #include "qemu/crc32c.h"
@@ -273,25 +274,6 @@ static void vhdx_set_shift_bits(BDRVVHDXState *s)
 }
 
 /*
- * Per the MS VHDX Specification, for every VHDX file:
- *  - The header section is fixed size - 1 MB
- *  - The header section is always the first "object"
- *  - The first 64KB of the header is the File Identifier
- *  - The first uint64 (8 bytes) is the VHDX Signature ("vhdxfile")
- *  - The following 512 bytes constitute a UTF-16 string identifiying the
- *software that created the file, and is optional and diagnostic only.
- *
- *  Therefore, we probe by looking for the vhdxfile signature "vhdxfile"
- */
-static int vhdx_probe(const uint8_t *buf, int buf_size, const char *filename)
-{
-if (buf_size >= 8 && !memcmp(buf, "vhdxfile", 8)) {
-return 100;
-}
-return 0;
-}
-
-/*
  * Writes the header to the specified offset.
  *
  * This will optionally read in buffer data from disk (otherwise zero-fill),
diff --git a/include/block/probe.h b/include/block/probe.h
index f85c178..e901d8f 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -12,5 +12,6 @@ int qcow2_probe(const uint8_t *buf, int buf_size, const char 
*filename);
 int bdrv_qed_probe(const uint8_t *buf, int buf_size, const char *filename);
 int raw_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vdi_probe(const uint8_t *buf, int buf_size, const char *filename);
+int vhdx_probe(const uint8_t *buf, int buf_size, const char *filename);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v4 17/32] blockdev: Separate bochs probe from its driver

[Colin Lord]

Modifies the bochs probe to return the format name as well as the
score as the final step of separating the probe function from the
driver. This keeps the probe completely independent of the driver,
making future modularization easier to accomplish. Returning the format
name as well as the score allows the score to be correlated to the
driver without the probe function needing to be part of the driver.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block.c   | 20 
 block/bochs-probe.c   | 25 -
 block/bochs.c |  1 -
 include/block/probe.h |  3 ++-
 4 files changed, 38 insertions(+), 11 deletions(-)

diff --git a/block.c b/block.c
index 3a0dc19..b16f8cf 100644
--- a/block.c
+++ b/block.c
@@ -25,6 +25,7 @@
 #include "trace.h"
 #include "block/block_int.h"
 #include "block/blockjob.h"
+#include "block/probe.h"
 #include "qemu/error-report.h"
 #include "module_block.h"
 #include "qemu/module.h"
@@ -56,6 +57,13 @@
 
 #define NOT_DONE 0x7fff /* used while emulated sync operation in progress 
*/
 
+typedef const char *BdrvProbeFunc(const uint8_t *buf, int buf_size,
+  const char *filename, int *score);
+
+static BdrvProbeFunc *format_probes[] = {
+bdrv_bochs_probe,
+};
+
 static QTAILQ_HEAD(, BlockDriverState) graph_bdrv_states =
 QTAILQ_HEAD_INITIALIZER(graph_bdrv_states);
 
@@ -576,6 +584,8 @@ BlockDriver *bdrv_probe_all(const uint8_t *buf, int 
buf_size,
 const char *filename)
 {
 int score_max = 0, score;
+const char *format_max = NULL;
+const char *format;
 size_t i;
 BlockDriver *drv = NULL, *d;
 
@@ -595,6 +605,16 @@ BlockDriver *bdrv_probe_all(const uint8_t *buf, int 
buf_size,
 }
 }
 
+for (i = 0; i < ARRAY_SIZE(format_probes); i++) {
+format = format_probes[i](buf, buf_size, filename, );
+if (score > score_max) {
+score_max = score;
+format_max = format;
+/* TODO: move call to find_format outside this loop */
+drv = bdrv_find_format(format_max);
+}
+}
+
 return drv;
 }
 
diff --git a/block/bochs-probe.c b/block/bochs-probe.c
index befe2cf..57161dc 100644
--- a/block/bochs-probe.c
+++ b/block/bochs-probe.c
@@ -3,19 +3,26 @@
 #include "block/probe.h"
 #include "bochs.h"
 
-int bochs_probe(const uint8_t *buf, int buf_size, const char *filename)
+const char *bdrv_bochs_probe(const uint8_t *buf, int buf_size,
+ const char *filename, int *score)
 {
+const char *format = "bochs";
 const struct bochs_header *bochs = (const void *)buf;
+assert(score);
+*score = 0;
 
-if (buf_size < HEADER_SIZE)
-   return 0;
+if (buf_size < HEADER_SIZE) {
+return format;
+}
 
 if (!strcmp(bochs->magic, HEADER_MAGIC) &&
-   !strcmp(bochs->type, REDOLOG_TYPE) &&
-   !strcmp(bochs->subtype, GROWING_TYPE) &&
-   ((le32_to_cpu(bochs->version) == HEADER_VERSION) ||
-   (le32_to_cpu(bochs->version) == HEADER_V1)))
-   return 100;
+!strcmp(bochs->type, REDOLOG_TYPE) &&
+!strcmp(bochs->subtype, GROWING_TYPE) &&
+((le32_to_cpu(bochs->version) == HEADER_VERSION) ||
+(le32_to_cpu(bochs->version) == HEADER_V1))) {
+*score = 100;
+return format;
+}
 
-return 0;
+return format;
 }
diff --git a/block/bochs.c b/block/bochs.c
index 5c9a696..7051ffc 100644
--- a/block/bochs.c
+++ b/block/bochs.c
@@ -234,7 +234,6 @@ static void bochs_close(BlockDriverState *bs)
 static BlockDriver bdrv_bochs = {
 .format_name   = "bochs",
 .instance_size = sizeof(BDRVBochsState),
-.bdrv_probe= bochs_probe,
 .bdrv_open = bochs_open,
 .bdrv_refresh_limits = bochs_refresh_limits,
 .bdrv_co_preadv = bochs_co_preadv,
diff --git a/include/block/probe.h b/include/block/probe.h
index 6cf878b..aac1fca 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -1,7 +1,6 @@
 #ifndef PROBE_H
 #define PROBE_H
 
-int bochs_probe(const uint8_t *buf, int buf_size, const char *filename);
 int cloop_probe(const uint8_t *buf, int buf_size, const char *filename);
 int block_crypto_probe_luks(const uint8_t *buf, int buf_size,
 const char *filename);
@@ -15,5 +14,7 @@ int vdi_probe(const uint8_t *buf, int buf_size, const char 
*filename);
 int vhdx_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vmdk_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vpc_probe(const uint8_t *buf, int buf_size, const char *filename);
+const char *bdrv_bochs_probe(const uint8_t *buf, int buf_size,
+ const char *filename, int *score);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v4 25/32] blockdev: Separate raw probe from its driver

[Colin Lord]

Completes the separation of the raw probe from the raw driver. The
raw probe now returns the format in addition to the score, allowing
correlation of the score and driver without the probe function being
part of the driver itself.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Max Reitz <mre...@redhat.com>
---
 block.c   | 1 +
 block/raw-probe.c | 8 ++--
 block/raw_bsd.c   | 1 -
 include/block/probe.h | 3 ++-
 4 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/block.c b/block.c
index 3dc7644..f37161d 100644
--- a/block.c
+++ b/block.c
@@ -69,6 +69,7 @@ static BdrvProbeFunc *format_probes[] = {
 bdrv_qcow_probe,
 bdrv_qcow2_probe,
 bdrv_qed_probe,
+bdrv_raw_probe,
 };
 
 static QTAILQ_HEAD(, BlockDriverState) graph_bdrv_states =
diff --git a/block/raw-probe.c b/block/raw-probe.c
index 22c6bcb..ac45e13 100644
--- a/block/raw-probe.c
+++ b/block/raw-probe.c
@@ -1,10 +1,14 @@
 #include "qemu/osdep.h"
 #include "block/probe.h"
 
-int raw_probe(const uint8_t *buf, int buf_size, const char *filename)
+const char *bdrv_raw_probe(const uint8_t *buf, int buf_size,
+   const char *filename, int *score)
 {
+const char *format = "raw";
+assert(score);
 /* smallest possible positive score so that raw is used if and only if no
  * other block driver works
  */
-return 1;
+*score = 1;
+return format;
 }
diff --git a/block/raw_bsd.c b/block/raw_bsd.c
index 25d5185..908cb59 100644
--- a/block/raw_bsd.c
+++ b/block/raw_bsd.c
@@ -226,7 +226,6 @@ static int raw_probe_geometry(BlockDriverState *bs, 
HDGeometry *geo)
 
 BlockDriver bdrv_raw = {
 .format_name  = "raw",
-.bdrv_probe   = _probe,
 .bdrv_reopen_prepare  = _reopen_prepare,
 .bdrv_open= _open,
 .bdrv_close   = _close,
diff --git a/include/block/probe.h b/include/block/probe.h
index cbb9c12..662d5d7 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -1,7 +1,6 @@
 #ifndef PROBE_H
 #define PROBE_H
 
-int raw_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vdi_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vhdx_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vmdk_probe(const uint8_t *buf, int buf_size, const char *filename);
@@ -22,5 +21,7 @@ const char *bdrv_qcow2_probe(const uint8_t *buf, int buf_size,
  const char *filename, int *score);
 const char *bdrv_qed_probe(const uint8_t *buf, int buf_size,
const char *filename, int *score);
+const char *bdrv_raw_probe(const uint8_t *buf, int buf_size,
+   const char *filename, int *score);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v4 10/32] blockdev: Move qcow2 probe to its own file

[Colin Lord]

Isolates qcow2 probe as part of the modularization process.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Max Reitz <mre...@redhat.com>
---
 block/Makefile.objs   |  2 +-
 block/qcow2-probe.c   | 16 
 block/qcow2.c | 13 +
 include/block/probe.h |  1 +
 4 files changed, 19 insertions(+), 13 deletions(-)
 create mode 100644 block/qcow2-probe.c

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 01b2e4e..4a27bde 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -25,7 +25,7 @@ block-obj-y += write-threshold.o
 
 block-obj-y += crypto.o
 block-obj-y += bochs-probe.o cloop-probe.o crypto-probe.o dmg-probe.o
-block-obj-y += parallels-probe.o qcow-probe.o
+block-obj-y += parallels-probe.o qcow-probe.o qcow2-probe.o
 
 common-obj-y += stream.o
 common-obj-y += backup.o
diff --git a/block/qcow2-probe.c b/block/qcow2-probe.c
new file mode 100644
index 000..e5cd92c
--- /dev/null
+++ b/block/qcow2-probe.c
@@ -0,0 +1,16 @@
+#include "qemu/osdep.h"
+#include "block/block_int.h"
+#include "block/probe.h"
+#include "qcow2.h"
+
+int qcow2_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+const QCowHeader *cow_header = (const void *)buf;
+
+if (buf_size >= sizeof(QCowHeader) &&
+be32_to_cpu(cow_header->magic) == QCOW_MAGIC &&
+be32_to_cpu(cow_header->version) >= 2)
+return 100;
+else
+return 0;
+}
diff --git a/block/qcow2.c b/block/qcow2.c
index a5ea19b..17521f7 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -27,6 +27,7 @@
 #include "qemu/module.h"
 #include 
 #include "block/qcow2.h"
+#include "block/probe.h"
 #include "qemu/error-report.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/qmp/qbool.h"
@@ -64,18 +65,6 @@ typedef struct {
 #define  QCOW2_EXT_MAGIC_BACKING_FORMAT 0xE2792ACA
 #define  QCOW2_EXT_MAGIC_FEATURE_TABLE 0x6803f857
 
-static int qcow2_probe(const uint8_t *buf, int buf_size, const char *filename)
-{
-const QCowHeader *cow_header = (const void *)buf;
-
-if (buf_size >= sizeof(QCowHeader) &&
-be32_to_cpu(cow_header->magic) == QCOW_MAGIC &&
-be32_to_cpu(cow_header->version) >= 2)
-return 100;
-else
-return 0;
-}
-
 
 /* 
  * read qcow2 extension and fill bs
diff --git a/include/block/probe.h b/include/block/probe.h
index 5230da4..f9dd36e 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -8,5 +8,6 @@ int block_crypto_probe_luks(const uint8_t *buf, int buf_size,
 int dmg_probe(const uint8_t *buf, int buf_size, const char *filename);
 int parallels_probe(const uint8_t *buf, int buf_size, const char *filename);
 int qcow_probe(const uint8_t *buf, int buf_size, const char *filename);
+int qcow2_probe(const uint8_t *buf, int buf_size, const char *filename);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v4 04/32] blockdev: Move bochs probe into separate file

[Colin Lord]

This puts the bochs probe function into its own separate file as part of
the process of modularizing block drivers. Having the probe functions
separate from the rest of the driver allows us to probe without having
to potentially unnecessarily load the driver.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Max Reitz <mre...@redhat.com>
---
 block/Makefile.objs   |  1 +
 block/bochs-probe.c   | 21 
 block/bochs.c | 55 ++-
 block/bochs.h | 40 +
 include/block/probe.h |  6 ++
 5 files changed, 70 insertions(+), 53 deletions(-)
 create mode 100644 block/bochs-probe.c
 create mode 100644 block/bochs.h
 create mode 100644 include/block/probe.h

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 2593a2f..9d4be3b 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -24,6 +24,7 @@ block-obj-y += accounting.o dirty-bitmap.o
 block-obj-y += write-threshold.o
 
 block-obj-y += crypto.o
+block-obj-y += bochs-probe.o
 
 common-obj-y += stream.o
 common-obj-y += backup.o
diff --git a/block/bochs-probe.c b/block/bochs-probe.c
new file mode 100644
index 000..befe2cf
--- /dev/null
+++ b/block/bochs-probe.c
@@ -0,0 +1,21 @@
+#include "qemu/osdep.h"
+#include "block/block_int.h"
+#include "block/probe.h"
+#include "bochs.h"
+
+int bochs_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+const struct bochs_header *bochs = (const void *)buf;
+
+if (buf_size < HEADER_SIZE)
+   return 0;
+
+if (!strcmp(bochs->magic, HEADER_MAGIC) &&
+   !strcmp(bochs->type, REDOLOG_TYPE) &&
+   !strcmp(bochs->subtype, GROWING_TYPE) &&
+   ((le32_to_cpu(bochs->version) == HEADER_VERSION) ||
+   (le32_to_cpu(bochs->version) == HEADER_V1)))
+   return 100;
+
+return 0;
+}
diff --git a/block/bochs.c b/block/bochs.c
index 8c9652e..5c9a696 100644
--- a/block/bochs.c
+++ b/block/bochs.c
@@ -28,45 +28,11 @@
 #include "block/block_int.h"
 #include "qemu/module.h"
 #include "qemu/bswap.h"
+#include "bochs.h"
+#include "block/probe.h"
 
 /**/
 
-#define HEADER_MAGIC "Bochs Virtual HD Image"
-#define HEADER_VERSION 0x0002
-#define HEADER_V1 0x0001
-#define HEADER_SIZE 512
-
-#define REDOLOG_TYPE "Redolog"
-#define GROWING_TYPE "Growing"
-
-// not allocated: 0x
-
-// always little-endian
-struct bochs_header {
-char magic[32]; /* "Bochs Virtual HD Image" */
-char type[16];  /* "Redolog" */
-char subtype[16];   /* "Undoable" / "Volatile" / "Growing" */
-uint32_t version;
-uint32_t header;/* size of header */
-
-uint32_t catalog;   /* num of entries */
-uint32_t bitmap;/* bitmap size */
-uint32_t extent;/* extent size */
-
-union {
-struct {
-uint32_t reserved;  /* for ??? */
-uint64_t disk;  /* disk size */
-char padding[HEADER_SIZE - 64 - 20 - 12];
-} QEMU_PACKED redolog;
-struct {
-uint64_t disk;  /* disk size */
-char padding[HEADER_SIZE - 64 - 20 - 8];
-} QEMU_PACKED redolog_v1;
-char padding[HEADER_SIZE - 64 - 20];
-} extra;
-} QEMU_PACKED;
-
 typedef struct BDRVBochsState {
 CoMutex lock;
 uint32_t *catalog_bitmap;
@@ -79,23 +45,6 @@ typedef struct BDRVBochsState {
 uint32_t extent_size;
 } BDRVBochsState;
 
-static int bochs_probe(const uint8_t *buf, int buf_size, const char *filename)
-{
-const struct bochs_header *bochs = (const void *)buf;
-
-if (buf_size < HEADER_SIZE)
-   return 0;
-
-if (!strcmp(bochs->magic, HEADER_MAGIC) &&
-   !strcmp(bochs->type, REDOLOG_TYPE) &&
-   !strcmp(bochs->subtype, GROWING_TYPE) &&
-   ((le32_to_cpu(bochs->version) == HEADER_VERSION) ||
-   (le32_to_cpu(bochs->version) == HEADER_V1)))
-   return 100;
-
-return 0;
-}
-
 static int bochs_open(BlockDriverState *bs, QDict *options, int flags,
   Error **errp)
 {
diff --git a/block/bochs.h b/block/bochs.h
new file mode 100644
index 000..2883f7d
--- /dev/null
+++ b/block/bochs.h
@@ -0,0 +1,40 @@
+#ifndef BLOCK_BOCHS_H
+#define BLOCK_BOCHS_H
+
+#define HEADER_MAGIC "Bochs Virtual HD Image"
+#define HEADER_VERSION 0x0002
+#define HEADER_V1 0x0001
+#define HEADER_SIZE 512
+
+#define REDOLOG_TYPE "Redolog"
+#define GROWING_TYPE "Growing"
+
+// not allocated: 0x
+
+// always little-endian
+struct bochs_header {
+char magic[32]; /* "Bochs Virtual HD Image" */
+char type[16];  /* "Redolog" */
+char subtype

[Qemu-block] [PATCH v4 02/32] blockdev: Add dynamic generation of module_block.h

[Colin Lord]

From: Marc Mari <mar...@redhat.com>

To simplify the addition of new block modules, add a script that generates
module_block.h automatically from the modules' source code.

This script assumes that the QEMU coding style rules are followed.

Signed-off-by: Marc Marí <mar...@redhat.com>
Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Max Reitz <mre...@redhat.com>
---
 Makefile|   7 +++
 scripts/modules/module_block.py | 122 
 2 files changed, 129 insertions(+)
 create mode 100644 scripts/modules/module_block.py

diff --git a/Makefile b/Makefile
index c054bc6..29035ac 100644
--- a/Makefile
+++ b/Makefile
@@ -76,6 +76,8 @@ GENERATED_HEADERS += trace/generated-ust-provider.h
 GENERATED_SOURCES += trace/generated-ust.c
 endif
 
+GENERATED_HEADERS += module_block.h
+
 # Don't try to regenerate Makefile or configure
 # We don't generate any of them
 Makefile: ;
@@ -352,6 +354,11 @@ ivshmem-client$(EXESUF): $(ivshmem-client-obj-y) 
libqemuutil.a libqemustub.a
 ivshmem-server$(EXESUF): $(ivshmem-server-obj-y) libqemuutil.a libqemustub.a
$(call LINK, $^)
 
+module_block.h: $(SRC_PATH)/scripts/modules/module_block.py config-host.mak
+   $(call quiet-command,$(PYTHON) $< $@ \
+   $(addprefix $(SRC_PATH)/,$(patsubst %.mo,%.c,$(block-obj-m))), \
+   "  GEN   $@")
+
 clean:
 # avoid old build problems by removing potentially incorrect old files
rm -f config.mak op-i386.h opc-i386.h gen-op-i386.h op-arm.h opc-arm.h 
gen-op-arm.h
diff --git a/scripts/modules/module_block.py b/scripts/modules/module_block.py
new file mode 100644
index 000..e5040f3
--- /dev/null
+++ b/scripts/modules/module_block.py
@@ -0,0 +1,122 @@
+#!/usr/bin/python
+#
+# Module information generator
+#
+# Copyright Red Hat, Inc. 2015 - 2016
+#
+# Authors:
+#  Marc Mari <mar...@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2.
+# See the COPYING file in the top-level directory.
+
+from __future__ import print_function
+import sys
+import os
+
+def get_string_struct(line):
+data = line.split()
+
+# data[0] -> struct element name
+# data[1] -> =
+# data[2] -> value
+
+return data[2].replace('"', '')[:-1]
+
+def add_module(fheader, library, format_name, protocol_name,
+probe, probe_device):
+lines = []
+lines.append('.library_name = "' + library + '",')
+if format_name != "":
+lines.append('.format_name = "' + format_name + '",')
+if protocol_name != "":
+lines.append('.protocol_name = "' + protocol_name + '",')
+if probe:
+lines.append('.has_probe = true,')
+if probe_device:
+lines.append('.has_probe_device = true,')
+
+text = '\n'.join(lines)
+fheader.write('\n{\n' + text + '\n},')
+
+def process_file(fheader, filename):
+# This parser assumes the coding style rules are being followed
+with open(filename, "r") as cfile:
+found_something = False
+found_start = False
+library, _ = os.path.splitext(os.path.basename(filename))
+for line in cfile:
+if found_start:
+line = line.replace('\n', '')
+if line.find(".format_name") != -1:
+format_name = get_string_struct(line)
+elif line.find(".protocol_name") != -1:
+protocol_name = get_string_struct(line)
+elif line.find(".bdrv_probe") != -1:
+probe = True
+elif line.find(".bdrv_probe_device") != -1:
+probe_device = True
+elif line == "};":
+add_module(fheader, library, format_name, protocol_name,
+probe, probe_device)
+found_start = False
+elif line.find("static BlockDriver") != -1:
+found_something = True
+found_start = True
+format_name = ""
+protocol_name = ""
+probe = False
+probe_device = False
+
+if not found_something:
+print("No BlockDriver struct found in " + filename + ". \
+Is this really a module?", file=sys.stderr)
+sys.exit(1)
+
+def print_top(fheader):
+fheader.write('''/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
+/*
+ * QEMU Block Module Infrastructure
+ *
+ * Authors:
+ *  Marc Mari   <mar...@redhat.com>
+ */
+
+''')
+
+fheader.write('''#ifndef QEMU_MODULE_BLOCK_H
+#define QEMU_MODULE_BLOCK_H
+
+#include "qemu-common.h"
+
+static const struct {
+const char *format_name;
+const char *protocol_name;
+const char *library_name;
+   

[Qemu-block] [PATCH v4 11/32] blockdev: Move qed probe to its own file

[Colin Lord]

Isolate qed probe as part of the modularization process.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Max Reitz <mre...@redhat.com>
---
 block/Makefile.objs   |  2 +-
 block/qed-probe.c | 18 ++
 block/qed.c   | 15 +--
 include/block/probe.h |  1 +
 4 files changed, 21 insertions(+), 15 deletions(-)
 create mode 100644 block/qed-probe.c

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 4a27bde..db9faef 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -25,7 +25,7 @@ block-obj-y += write-threshold.o
 
 block-obj-y += crypto.o
 block-obj-y += bochs-probe.o cloop-probe.o crypto-probe.o dmg-probe.o
-block-obj-y += parallels-probe.o qcow-probe.o qcow2-probe.o
+block-obj-y += parallels-probe.o qcow-probe.o qcow2-probe.o qed-probe.o
 
 common-obj-y += stream.o
 common-obj-y += backup.o
diff --git a/block/qed-probe.c b/block/qed-probe.c
new file mode 100644
index 000..40261f0
--- /dev/null
+++ b/block/qed-probe.c
@@ -0,0 +1,18 @@
+#include "qemu/osdep.h"
+#include "block/block_int.h"
+#include "block/probe.h"
+#include "qed.h"
+
+int bdrv_qed_probe(const uint8_t *buf, int buf_size,
+  const char *filename)
+{
+const QEDHeader *header = (const QEDHeader *)buf;
+
+if (buf_size < sizeof(*header)) {
+return 0;
+}
+if (le32_to_cpu(header->magic) != QED_MAGIC) {
+return 0;
+}
+return 100;
+}
diff --git a/block/qed.c b/block/qed.c
index f619d82..544ed0d 100644
--- a/block/qed.c
+++ b/block/qed.c
@@ -16,6 +16,7 @@
 #include "qapi/error.h"
 #include "qemu/timer.h"
 #include "qemu/bswap.h"
+#include "block/probe.h"
 #include "trace.h"
 #include "qed.h"
 #include "qapi/qmp/qerror.h"
@@ -26,20 +27,6 @@ static const AIOCBInfo qed_aiocb_info = {
 .aiocb_size = sizeof(QEDAIOCB),
 };
 
-static int bdrv_qed_probe(const uint8_t *buf, int buf_size,
-  const char *filename)
-{
-const QEDHeader *header = (const QEDHeader *)buf;
-
-if (buf_size < sizeof(*header)) {
-return 0;
-}
-if (le32_to_cpu(header->magic) != QED_MAGIC) {
-return 0;
-}
-return 100;
-}
-
 /**
  * Check whether an image format is raw
  *
diff --git a/include/block/probe.h b/include/block/probe.h
index f9dd36e..e3e9934 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -9,5 +9,6 @@ int dmg_probe(const uint8_t *buf, int buf_size, const char 
*filename);
 int parallels_probe(const uint8_t *buf, int buf_size, const char *filename);
 int qcow_probe(const uint8_t *buf, int buf_size, const char *filename);
 int qcow2_probe(const uint8_t *buf, int buf_size, const char *filename);
+int bdrv_qed_probe(const uint8_t *buf, int buf_size, const char *filename);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v4 05/32] blockdev: Move cloop probe to its own file

[Colin Lord]

Isolates cloop probing function as part of the modularization process.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Max Reitz <mre...@redhat.com>
---
 block/Makefile.objs   |  2 +-
 block/cloop-probe.c   | 17 +
 block/cloop.c | 16 +---
 include/block/probe.h |  1 +
 4 files changed, 20 insertions(+), 16 deletions(-)
 create mode 100644 block/cloop-probe.c

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 9d4be3b..1a8d7c0 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -24,7 +24,7 @@ block-obj-y += accounting.o dirty-bitmap.o
 block-obj-y += write-threshold.o
 
 block-obj-y += crypto.o
-block-obj-y += bochs-probe.o
+block-obj-y += bochs-probe.o cloop-probe.o
 
 common-obj-y += stream.o
 common-obj-y += backup.o
diff --git a/block/cloop-probe.c b/block/cloop-probe.c
new file mode 100644
index 000..955c29c
--- /dev/null
+++ b/block/cloop-probe.c
@@ -0,0 +1,17 @@
+#include "qemu/osdep.h"
+#include "block/probe.h"
+
+int cloop_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+const char *magic_version_2_0 = "#!/bin/sh\n"
+"#V2.0 Format\n"
+"modprobe cloop file=$0 && mount -r -t iso9660 /dev/cloop $1\n";
+int length = strlen(magic_version_2_0);
+if (length > buf_size) {
+length = buf_size;
+}
+if (!memcmp(magic_version_2_0, buf, length)) {
+return 2;
+}
+return 0;
+}
diff --git a/block/cloop.c b/block/cloop.c
index 7b75f7e..c897e58 100644
--- a/block/cloop.c
+++ b/block/cloop.c
@@ -25,6 +25,7 @@
 #include "qapi/error.h"
 #include "qemu-common.h"
 #include "block/block_int.h"
+#include "block/probe.h"
 #include "qemu/module.h"
 #include "qemu/bswap.h"
 #include 
@@ -44,21 +45,6 @@ typedef struct BDRVCloopState {
 z_stream zstream;
 } BDRVCloopState;
 
-static int cloop_probe(const uint8_t *buf, int buf_size, const char *filename)
-{
-const char *magic_version_2_0 = "#!/bin/sh\n"
-"#V2.0 Format\n"
-"modprobe cloop file=$0 && mount -r -t iso9660 /dev/cloop $1\n";
-int length = strlen(magic_version_2_0);
-if (length > buf_size) {
-length = buf_size;
-}
-if (!memcmp(magic_version_2_0, buf, length)) {
-return 2;
-}
-return 0;
-}
-
 static int cloop_open(BlockDriverState *bs, QDict *options, int flags,
   Error **errp)
 {
diff --git a/include/block/probe.h b/include/block/probe.h
index 6450ca1..ed1a60b 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -2,5 +2,6 @@
 #define PROBE_H
 
 int bochs_probe(const uint8_t *buf, int buf_size, const char *filename);
+int cloop_probe(const uint8_t *buf, int buf_size, const char *filename);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v4 09/32] blockdev: Move qcow probe to its own file

[Colin Lord]

Isolates qcow probe as part of the modularization process.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Max Reitz <mre...@redhat.com>
---
 block/Makefile.objs   |  2 +-
 block/qcow-probe.c| 16 
 block/qcow.c  | 31 ++-
 block/qcow.h  | 21 +
 include/block/probe.h |  1 +
 5 files changed, 41 insertions(+), 30 deletions(-)
 create mode 100644 block/qcow-probe.c
 create mode 100644 block/qcow.h

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 43724ec..01b2e4e 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -25,7 +25,7 @@ block-obj-y += write-threshold.o
 
 block-obj-y += crypto.o
 block-obj-y += bochs-probe.o cloop-probe.o crypto-probe.o dmg-probe.o
-block-obj-y += parallels-probe.o
+block-obj-y += parallels-probe.o qcow-probe.o
 
 common-obj-y += stream.o
 common-obj-y += backup.o
diff --git a/block/qcow-probe.c b/block/qcow-probe.c
new file mode 100644
index 000..c5fd214
--- /dev/null
+++ b/block/qcow-probe.c
@@ -0,0 +1,16 @@
+#include "qemu/osdep.h"
+#include "block/block_int.h"
+#include "block/probe.h"
+#include "qcow.h"
+
+int qcow_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+const QCowHeader *cow_header = (const void *)buf;
+
+if (buf_size >= sizeof(QCowHeader) &&
+be32_to_cpu(cow_header->magic) == QCOW_MAGIC &&
+be32_to_cpu(cow_header->version) == QCOW_VERSION)
+return 100;
+else
+return 0;
+}
diff --git a/block/qcow.c b/block/qcow.c
index ac849bd..6ea1164 100644
--- a/block/qcow.c
+++ b/block/qcow.c
@@ -26,6 +26,8 @@
 #include "qemu-common.h"
 #include "qemu/error-report.h"
 #include "block/block_int.h"
+#include "block/probe.h"
+#include "qcow.h"
 #include "sysemu/block-backend.h"
 #include "qemu/module.h"
 #include "qemu/bswap.h"
@@ -37,28 +39,11 @@
 /**/
 /* QEMU COW block driver with compression and encryption support */
 
-#define QCOW_MAGIC (('Q' << 24) | ('F' << 16) | ('I' << 8) | 0xfb)
-#define QCOW_VERSION 1
-
 #define QCOW_CRYPT_NONE 0
 #define QCOW_CRYPT_AES  1
 
 #define QCOW_OFLAG_COMPRESSED (1LL << 63)
 
-typedef struct QCowHeader {
-uint32_t magic;
-uint32_t version;
-uint64_t backing_file_offset;
-uint32_t backing_file_size;
-uint32_t mtime;
-uint64_t size; /* in bytes */
-uint8_t cluster_bits;
-uint8_t l2_bits;
-uint16_t padding;
-uint32_t crypt_method;
-uint64_t l1_table_offset;
-} QEMU_PACKED QCowHeader;
-
 #define L2_CACHE_SIZE 16
 
 typedef struct BDRVQcowState {
@@ -85,18 +70,6 @@ typedef struct BDRVQcowState {
 
 static int decompress_cluster(BlockDriverState *bs, uint64_t cluster_offset);
 
-static int qcow_probe(const uint8_t *buf, int buf_size, const char *filename)
-{
-const QCowHeader *cow_header = (const void *)buf;
-
-if (buf_size >= sizeof(QCowHeader) &&
-be32_to_cpu(cow_header->magic) == QCOW_MAGIC &&
-be32_to_cpu(cow_header->version) == QCOW_VERSION)
-return 100;
-else
-return 0;
-}
-
 static int qcow_open(BlockDriverState *bs, QDict *options, int flags,
  Error **errp)
 {
diff --git a/block/qcow.h b/block/qcow.h
new file mode 100644
index 000..c96ea24
--- /dev/null
+++ b/block/qcow.h
@@ -0,0 +1,21 @@
+#ifndef QCOW_H
+#define QCOW_H
+
+#define QCOW_MAGIC (('Q' << 24) | ('F' << 16) | ('I' << 8) | 0xfb)
+#define QCOW_VERSION 1
+
+typedef struct QCowHeader {
+uint32_t magic;
+uint32_t version;
+uint64_t backing_file_offset;
+uint32_t backing_file_size;
+uint32_t mtime;
+uint64_t size; /* in bytes */
+uint8_t cluster_bits;
+uint8_t l2_bits;
+uint16_t padding;
+uint32_t crypt_method;
+uint64_t l1_table_offset;
+} QEMU_PACKED QCowHeader;
+
+#endif
diff --git a/include/block/probe.h b/include/block/probe.h
index f8b0984..5230da4 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -7,5 +7,6 @@ int block_crypto_probe_luks(const uint8_t *buf, int buf_size,
 const char *filename);
 int dmg_probe(const uint8_t *buf, int buf_size, const char *filename);
 int parallels_probe(const uint8_t *buf, int buf_size, const char *filename);
+int qcow_probe(const uint8_t *buf, int buf_size, const char *filename);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v4 03/32] blockdev: Add dynamic module loading for block drivers

[Colin Lord]

From: Marc Mari <mar...@redhat.com>

Extend the current module interface to allow for block drivers to be loaded
dynamically on request.

The only block drivers that can be converted into modules are the drivers
that don't perform any init operation except for registering themselves.

All the necessary module information is located in a new structure found in
module_block.h

Signed-off-by: Marc Marí <mar...@redhat.com>
Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Max Reitz <mre...@redhat.com>
---
 block.c   | 110 ++
 include/qemu/module.h |   3 ++
 util/module.c |  38 +
 3 files changed, 108 insertions(+), 43 deletions(-)

diff --git a/block.c b/block.c
index 823ff1d..3a0dc19 100644
--- a/block.c
+++ b/block.c
@@ -26,6 +26,7 @@
 #include "block/block_int.h"
 #include "block/blockjob.h"
 #include "qemu/error-report.h"
+#include "module_block.h"
 #include "qemu/module.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/qmp/qbool.h"
@@ -239,17 +240,40 @@ BlockDriverState *bdrv_new(void)
 return bs;
 }
 
-BlockDriver *bdrv_find_format(const char *format_name)
+static BlockDriver *bdrv_do_find_format(const char *format_name)
 {
 BlockDriver *drv1;
+
 QLIST_FOREACH(drv1, _drivers, list) {
 if (!strcmp(drv1->format_name, format_name)) {
 return drv1;
 }
 }
+
 return NULL;
 }
 
+BlockDriver *bdrv_find_format(const char *format_name)
+{
+BlockDriver *drv1;
+size_t i;
+
+drv1 = bdrv_do_find_format(format_name);
+if (drv1) {
+return drv1;
+}
+
+/* The driver isn't registered, maybe we need to load a module */
+for (i = 0; i < ARRAY_SIZE(block_driver_modules); ++i) {
+if (!strcmp(block_driver_modules[i].format_name, format_name)) {
+block_module_load_one(block_driver_modules[i].library_name);
+break;
+}
+}
+
+return bdrv_do_find_format(format_name);
+}
+
 static int bdrv_is_whitelisted(BlockDriver *drv, bool read_only)
 {
 static const char *whitelist_rw[] = {
@@ -443,8 +467,15 @@ int get_tmp_filename(char *filename, int size)
 static BlockDriver *find_hdev_driver(const char *filename)
 {
 int score_max = 0, score;
+size_t i;
 BlockDriver *drv = NULL, *d;
 
+for (i = 0; i < ARRAY_SIZE(block_driver_modules); ++i) {
+if (block_driver_modules[i].has_probe_device) {
+block_module_load_one(block_driver_modules[i].library_name);
+}
+}
+
 QLIST_FOREACH(d, _drivers, list) {
 if (d->bdrv_probe_device) {
 score = d->bdrv_probe_device(filename);
@@ -458,6 +489,19 @@ static BlockDriver *find_hdev_driver(const char *filename)
 return drv;
 }
 
+static BlockDriver *bdrv_do_find_protocol(const char *protocol)
+{
+BlockDriver *drv1;
+
+QLIST_FOREACH(drv1, _drivers, list) {
+if (drv1->protocol_name && !strcmp(drv1->protocol_name, protocol)) {
+return drv1;
+}
+}
+
+return NULL;
+}
+
 BlockDriver *bdrv_find_protocol(const char *filename,
 bool allow_protocol_prefix,
 Error **errp)
@@ -466,6 +510,7 @@ BlockDriver *bdrv_find_protocol(const char *filename,
 char protocol[128];
 int len;
 const char *p;
+size_t i;
 
 /* TODO Drivers without bdrv_file_open must be specified explicitly */
 
@@ -492,15 +537,25 @@ BlockDriver *bdrv_find_protocol(const char *filename,
 len = sizeof(protocol) - 1;
 memcpy(protocol, filename, len);
 protocol[len] = '\0';
-QLIST_FOREACH(drv1, _drivers, list) {
-if (drv1->protocol_name &&
-!strcmp(drv1->protocol_name, protocol)) {
-return drv1;
+
+drv1 = bdrv_do_find_protocol(protocol);
+if (drv1) {
+return drv1;
+}
+
+for (i = 0; i < ARRAY_SIZE(block_driver_modules); ++i) {
+if (block_driver_modules[i].protocol_name &&
+!strcmp(block_driver_modules[i].protocol_name, protocol)) {
+block_module_load_one(block_driver_modules[i].library_name);
+break;
 }
 }
 
-error_setg(errp, "Unknown protocol '%s'", protocol);
-return NULL;
+drv1 = bdrv_do_find_protocol(protocol);
+if (!drv1) {
+error_setg(errp, "Unknown protocol '%s'", protocol);
+}
+return drv1;
 }
 
 /*
@@ -521,8 +576,15 @@ BlockDriver *bdrv_probe_all(const uint8_t *buf, int 
buf_size,
 const char *filename)
 {
 int score_max = 0, score;
+size_t i;
 BlockDriver *drv = NULL, *d;
 
+for (i = 0; i < ARRAY_SIZE(block_driver_modules); ++i) {
+if (block_driver_modules[i].has_probe) {
+block_module_load_one(block_driver_modules[i].library_name);

[Qemu-block] [PATCH v4 15/32] blockdev: Move vmdk probe to its own file

[Colin Lord]

Isolates vmdk probe as part of the modularization process.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Max Reitz <mre...@redhat.com>
---
 block/Makefile.objs   |  2 +-
 block/vmdk-probe.c| 60 +++
 block/vmdk.c  | 60 ++-
 block/vmdk.h  |  7 ++
 include/block/probe.h |  1 +
 5 files changed, 71 insertions(+), 59 deletions(-)
 create mode 100644 block/vmdk-probe.c
 create mode 100644 block/vmdk.h

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 4363667..f86ab63 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -26,7 +26,7 @@ block-obj-y += write-threshold.o
 block-obj-y += crypto.o
 block-obj-y += bochs-probe.o cloop-probe.o crypto-probe.o dmg-probe.o
 block-obj-y += parallels-probe.o qcow-probe.o qcow2-probe.o qed-probe.o
-block-obj-y += raw-probe.o vdi-probe.o vhdx-probe.o
+block-obj-y += raw-probe.o vdi-probe.o vhdx-probe.o vmdk-probe.o
 
 common-obj-y += stream.o
 common-obj-y += backup.o
diff --git a/block/vmdk-probe.c b/block/vmdk-probe.c
new file mode 100644
index 000..77b06af
--- /dev/null
+++ b/block/vmdk-probe.c
@@ -0,0 +1,60 @@
+#include "qemu/osdep.h"
+#include "block/block_int.h"
+#include "block/probe.h"
+#include "vmdk.h"
+
+int vmdk_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+uint32_t magic;
+
+if (buf_size < 4) {
+return 0;
+}
+magic = be32_to_cpu(*(uint32_t *)buf);
+if (magic == VMDK3_MAGIC ||
+magic == VMDK4_MAGIC) {
+return 100;
+} else {
+const char *p = (const char *)buf;
+const char *end = p + buf_size;
+while (p < end) {
+if (*p == '#') {
+/* skip comment line */
+while (p < end && *p != '\n') {
+p++;
+}
+p++;
+continue;
+}
+if (*p == ' ') {
+while (p < end && *p == ' ') {
+p++;
+}
+/* skip '\r' if windows line endings used. */
+if (p < end && *p == '\r') {
+p++;
+}
+/* only accept blank lines before 'version=' line */
+if (p == end || *p != '\n') {
+return 0;
+}
+p++;
+continue;
+}
+if (end - p >= strlen("version=X\n")) {
+if (strncmp("version=1\n", p, strlen("version=1\n")) == 0 ||
+strncmp("version=2\n", p, strlen("version=2\n")) == 0) {
+return 100;
+}
+}
+if (end - p >= strlen("version=X\r\n")) {
+if (strncmp("version=1\r\n", p, strlen("version=1\r\n")) == 0 
||
+strncmp("version=2\r\n", p, strlen("version=2\r\n")) == 0) 
{
+return 100;
+}
+}
+return 0;
+}
+return 0;
+}
+}
diff --git a/block/vmdk.c b/block/vmdk.c
index d73f431..d2699b0 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -26,6 +26,8 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
 #include "block/block_int.h"
+#include "block/probe.h"
+#include "vmdk.h"
 #include "sysemu/block-backend.h"
 #include "qapi/qmp/qerror.h"
 #include "qemu/error-report.h"
@@ -35,8 +37,6 @@
 #include "qemu/cutils.h"
 #include 
 
-#define VMDK3_MAGIC (('C' << 24) | ('O' << 16) | ('W' << 8) | 'D')
-#define VMDK4_MAGIC (('K' << 24) | ('D' << 16) | ('M' << 8) | 'V')
 #define VMDK4_COMPRESSION_DEFLATE 1
 #define VMDK4_FLAG_NL_DETECT (1 << 0)
 #define VMDK4_FLAG_RGD (1 << 1)
@@ -151,62 +151,6 @@ enum {
 MARKER_FOOTER   = 3,
 };
 
-static int vmdk_probe(const uint8_t *buf, int buf_size, const char *filename)
-{
-uint32_t magic;
-
-if (buf_size < 4) {
-return 0;
-}
-magic = be32_to_cpu(*(uint32_t *)buf);
-if (magic == VMDK3_MAGIC ||
-magic == VMDK4_MAGIC) {
-return 100;
-} else {
-const char *p = (const char *)buf;
-const char *end = p + buf_size;
-while (p < end) {
-if (*p == '#') {
-/* skip comment line */
-while (p < end && *p != '\n') {
-p++;
-}
-p++;
-continue;
-}
-if (*p == ' ') {
-while (p < end && *p == ' ') {
-p++;
-}
-/* skip '\r' if windows line endings us

[Qemu-block] [PATCH v4 01/32] blockdev: prepare iSCSI block driver for dynamic loading

[Colin Lord]

This commit moves the initialization of the QemuOptsList qemu_iscsi_opts
struct out of block/iscsi.c in order to allow the iscsi module to be
dynamically loaded.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Fam Zheng <f...@redhat.com>
---
 block/iscsi.c | 36 
 vl.c  | 38 ++
 2 files changed, 38 insertions(+), 36 deletions(-)

diff --git a/block/iscsi.c b/block/iscsi.c
index 434cb37..b52cd37 100644
--- a/block/iscsi.c
+++ b/block/iscsi.c
@@ -1882,45 +1882,9 @@ static BlockDriver bdrv_iscsi = {
 .bdrv_attach_aio_context = iscsi_attach_aio_context,
 };
 
-static QemuOptsList qemu_iscsi_opts = {
-.name = "iscsi",
-.head = QTAILQ_HEAD_INITIALIZER(qemu_iscsi_opts.head),
-.desc = {
-{
-.name = "user",
-.type = QEMU_OPT_STRING,
-.help = "username for CHAP authentication to target",
-},{
-.name = "password",
-.type = QEMU_OPT_STRING,
-.help = "password for CHAP authentication to target",
-},{
-.name = "password-secret",
-.type = QEMU_OPT_STRING,
-.help = "ID of the secret providing password for CHAP "
-"authentication to target",
-},{
-.name = "header-digest",
-.type = QEMU_OPT_STRING,
-.help = "HeaderDigest setting. "
-"{CRC32C|CRC32C-NONE|NONE-CRC32C|NONE}",
-},{
-.name = "initiator-name",
-.type = QEMU_OPT_STRING,
-.help = "Initiator iqn name to use when connecting",
-},{
-.name = "timeout",
-.type = QEMU_OPT_NUMBER,
-.help = "Request timeout in seconds (default 0 = no timeout)",
-},
-{ /* end of list */ }
-},
-};
-
 static void iscsi_block_init(void)
 {
 bdrv_register(_iscsi);
-qemu_add_opts(_iscsi_opts);
 }
 
 block_init(iscsi_block_init);
diff --git a/vl.c b/vl.c
index 356713e..90ed255 100644
--- a/vl.c
+++ b/vl.c
@@ -506,6 +506,41 @@ static QemuOptsList qemu_fw_cfg_opts = {
 },
 };
 
+static QemuOptsList qemu_iscsi_opts = {
+.name = "iscsi",
+.head = QTAILQ_HEAD_INITIALIZER(qemu_iscsi_opts.head),
+.desc = {
+{
+.name = "user",
+.type = QEMU_OPT_STRING,
+.help = "username for CHAP authentication to target",
+},{
+.name = "password",
+.type = QEMU_OPT_STRING,
+.help = "password for CHAP authentication to target",
+},{
+.name = "password-secret",
+.type = QEMU_OPT_STRING,
+.help = "ID of the secret providing password for CHAP "
+"authentication to target",
+},{
+.name = "header-digest",
+.type = QEMU_OPT_STRING,
+.help = "HeaderDigest setting. "
+"{CRC32C|CRC32C-NONE|NONE-CRC32C|NONE}",
+},{
+.name = "initiator-name",
+.type = QEMU_OPT_STRING,
+.help = "Initiator iqn name to use when connecting",
+},{
+.name = "timeout",
+.type = QEMU_OPT_NUMBER,
+.help = "Request timeout in seconds (default 0 = no timeout)",
+},
+{ /* end of list */ }
+},
+};
+
 /**
  * Get machine options
  *
@@ -3000,6 +3035,9 @@ int main(int argc, char **argv, char **envp)
 qemu_add_opts(_icount_opts);
 qemu_add_opts(_semihosting_config_opts);
 qemu_add_opts(_fw_cfg_opts);
+#ifdef CONFIG_LIBISCSI
+qemu_add_opts(_iscsi_opts);
+#endif
 module_call_init(MODULE_INIT_OPTS);
 
 runstate_init();
-- 
2.5.5

[Qemu-block] [PATCH v4 06/32] blockdev: Move luks probe to its own file

[Colin Lord]

Isolates the luks probe function as part of the modularization process.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Max Reitz <mre...@redhat.com>
---
 block/Makefile.objs   |  2 +-
 block/crypto-probe.c  | 23 +++
 block/crypto.c| 21 +
 include/block/probe.h |  2 ++
 4 files changed, 27 insertions(+), 21 deletions(-)
 create mode 100644 block/crypto-probe.c

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 1a8d7c0..aca547f 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -24,7 +24,7 @@ block-obj-y += accounting.o dirty-bitmap.o
 block-obj-y += write-threshold.o
 
 block-obj-y += crypto.o
-block-obj-y += bochs-probe.o cloop-probe.o
+block-obj-y += bochs-probe.o cloop-probe.o crypto-probe.o
 
 common-obj-y += stream.o
 common-obj-y += backup.o
diff --git a/block/crypto-probe.c b/block/crypto-probe.c
new file mode 100644
index 000..5c6427a
--- /dev/null
+++ b/block/crypto-probe.c
@@ -0,0 +1,23 @@
+#include "qemu/osdep.h"
+#include "qapi-types.h"
+#include "block/probe.h"
+#include "crypto/block.h"
+
+static int block_crypto_probe_generic(QCryptoBlockFormat format,
+  const uint8_t *buf,
+  int buf_size,
+  const char *filename)
+{
+if (qcrypto_block_has_format(format, buf, buf_size)) {
+return 100;
+} else {
+return 0;
+}
+}
+
+int block_crypto_probe_luks(const uint8_t *buf,
+   int buf_size,
+   const char *filename) {
+return block_crypto_probe_generic(Q_CRYPTO_BLOCK_FORMAT_LUKS,
+  buf, buf_size, filename);
+}
diff --git a/block/crypto.c b/block/crypto.c
index 7eaa057..5706ba6 100644
--- a/block/crypto.c
+++ b/block/crypto.c
@@ -21,6 +21,7 @@
 #include "qemu/osdep.h"
 
 #include "block/block_int.h"
+#include "block/probe.h"
 #include "sysemu/block-backend.h"
 #include "crypto/block.h"
 #include "qapi/opts-visitor.h"
@@ -41,19 +42,6 @@ struct BlockCrypto {
 };
 
 
-static int block_crypto_probe_generic(QCryptoBlockFormat format,
-  const uint8_t *buf,
-  int buf_size,
-  const char *filename)
-{
-if (qcrypto_block_has_format(format, buf, buf_size)) {
-return 100;
-} else {
-return 0;
-}
-}
-
-
 static ssize_t block_crypto_read_func(QCryptoBlock *block,
   size_t offset,
   uint8_t *buf,
@@ -538,13 +526,6 @@ static int64_t block_crypto_getlength(BlockDriverState *bs)
 }
 
 
-static int block_crypto_probe_luks(const uint8_t *buf,
-   int buf_size,
-   const char *filename) {
-return block_crypto_probe_generic(Q_CRYPTO_BLOCK_FORMAT_LUKS,
-  buf, buf_size, filename);
-}
-
 static int block_crypto_open_luks(BlockDriverState *bs,
   QDict *options,
   int flags,
diff --git a/include/block/probe.h b/include/block/probe.h
index ed1a60b..35a8d00 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -3,5 +3,7 @@
 
 int bochs_probe(const uint8_t *buf, int buf_size, const char *filename);
 int cloop_probe(const uint8_t *buf, int buf_size, const char *filename);
+int block_crypto_probe_luks(const uint8_t *buf, int buf_size,
+const char *filename);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v4 08/32] blockdev: Move parallels probe to its own file

[Colin Lord]

Isolate parallels probe as part of the modularization process.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Max Reitz <mre...@redhat.com>
---
 block/Makefile.objs |  1 +
 block/parallels-probe.c | 21 +
 block/parallels.c   | 43 ++-
 block/parallels.h   | 26 ++
 include/block/probe.h   |  1 +
 5 files changed, 51 insertions(+), 41 deletions(-)
 create mode 100644 block/parallels-probe.c
 create mode 100644 block/parallels.h

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 09cc64c..43724ec 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -25,6 +25,7 @@ block-obj-y += write-threshold.o
 
 block-obj-y += crypto.o
 block-obj-y += bochs-probe.o cloop-probe.o crypto-probe.o dmg-probe.o
+block-obj-y += parallels-probe.o
 
 common-obj-y += stream.o
 common-obj-y += backup.o
diff --git a/block/parallels-probe.c b/block/parallels-probe.c
new file mode 100644
index 000..ff2f6ba
--- /dev/null
+++ b/block/parallels-probe.c
@@ -0,0 +1,21 @@
+#include "qemu/osdep.h"
+#include "block/block_int.h"
+#include "block/probe.h"
+#include "parallels.h"
+
+int parallels_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+const ParallelsHeader *ph = (const void *)buf;
+
+if (buf_size < sizeof(ParallelsHeader)) {
+return 0;
+}
+
+if ((!memcmp(ph->magic, HEADER_MAGIC, 16) ||
+   !memcmp(ph->magic, HEADER_MAGIC2, 16)) &&
+   (le32_to_cpu(ph->version) == HEADER_VERSION)) {
+return 100;
+}
+
+return 0;
+}
diff --git a/block/parallels.c b/block/parallels.c
index 807a801..70e9439 100644
--- a/block/parallels.c
+++ b/block/parallels.c
@@ -31,6 +31,8 @@
 #include "qapi/error.h"
 #include "qemu-common.h"
 #include "block/block_int.h"
+#include "block/probe.h"
+#include "parallels.h"
 #include "sysemu/block-backend.h"
 #include "qemu/module.h"
 #include "qemu/bswap.h"
@@ -39,29 +41,6 @@
 
 /**/
 
-#define HEADER_MAGIC "WithoutFreeSpace"
-#define HEADER_MAGIC2 "WithouFreSpacExt"
-#define HEADER_VERSION 2
-#define HEADER_INUSE_MAGIC  (0x746F6E59)
-
-#define DEFAULT_CLUSTER_SIZE 1048576/* 1 MiB */
-
-
-// always little-endian
-typedef struct ParallelsHeader {
-char magic[16]; // "WithoutFreeSpace"
-uint32_t version;
-uint32_t heads;
-uint32_t cylinders;
-uint32_t tracks;
-uint32_t bat_entries;
-uint64_t nb_sectors;
-uint32_t inuse;
-uint32_t data_off;
-char padding[12];
-} QEMU_PACKED ParallelsHeader;
-
-
 typedef enum ParallelsPreallocMode {
 PRL_PREALLOC_MODE_FALLOCATE = 0,
 PRL_PREALLOC_MODE_TRUNCATE = 1,
@@ -536,24 +515,6 @@ exit:
 }
 
 
-static int parallels_probe(const uint8_t *buf, int buf_size,
-   const char *filename)
-{
-const ParallelsHeader *ph = (const void *)buf;
-
-if (buf_size < sizeof(ParallelsHeader)) {
-return 0;
-}
-
-if ((!memcmp(ph->magic, HEADER_MAGIC, 16) ||
-   !memcmp(ph->magic, HEADER_MAGIC2, 16)) &&
-   (le32_to_cpu(ph->version) == HEADER_VERSION)) {
-return 100;
-}
-
-return 0;
-}
-
 static int parallels_update_header(BlockDriverState *bs)
 {
 BDRVParallelsState *s = bs->opaque;
diff --git a/block/parallels.h b/block/parallels.h
new file mode 100644
index 000..512ef5f
--- /dev/null
+++ b/block/parallels.h
@@ -0,0 +1,26 @@
+#ifndef PARALLELS_H
+#define PARALLELS_H
+
+#define HEADER_MAGIC "WithoutFreeSpace"
+#define HEADER_MAGIC2 "WithouFreSpacExt"
+#define HEADER_VERSION 2
+#define HEADER_INUSE_MAGIC  (0x746F6E59)
+
+#define DEFAULT_CLUSTER_SIZE 1048576/* 1 MiB */
+
+
+// always little-endian
+typedef struct ParallelsHeader {
+char magic[16]; // "WithoutFreeSpace"
+uint32_t version;
+uint32_t heads;
+uint32_t cylinders;
+uint32_t tracks;
+uint32_t bat_entries;
+uint64_t nb_sectors;
+uint32_t inuse;
+uint32_t data_off;
+char padding[12];
+} QEMU_PACKED ParallelsHeader;
+
+#endif
diff --git a/include/block/probe.h b/include/block/probe.h
index 267431d..f8b0984 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -6,5 +6,6 @@ int cloop_probe(const uint8_t *buf, int buf_size, const char 
*filename);
 int block_crypto_probe_luks(const uint8_t *buf, int buf_size,
 const char *filename);
 int dmg_probe(const uint8_t *buf, int buf_size, const char *filename);
+int parallels_probe(const uint8_t *buf, int buf_size, const char *filename);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v4 00/32] Dynamic module loading for block drivers

[Colin Lord]

Here's v4 of the modularization series. Things that have changed since
v3 include:

- Fix indentation of the generated header file module_block.h
- Drivers and probe functions are now all located in the block/
  directory, rather than being split between block/ and block/probe/. In
  addition the header files for each probe/driver pair are in the block/
  directory, not the include/block/driver/ directory (which no longer
  exists).
- Since the probe files are in block/ now, they follow the naming
  pattern of format-probe.c
- Renamed crypto probe file to be crypto-probe.c, luks is no longer in
  the filename
- Fixed formatting of parallels_probe() function header
- Enforced consistent naming convention for the probe functions. They
  now follow the pattern bdrv_format_probe().

Colin Lord (30):
  blockdev: prepare iSCSI block driver for dynamic loading
  blockdev: Move bochs probe into separate file
  blockdev: Move cloop probe to its own file
  blockdev: Move luks probe to its own file
  blockdev: Move dmg probe to its own file
  blockdev: Move parallels probe to its own file
  blockdev: Move qcow probe to its own file
  blockdev: Move qcow2 probe to its own file
  blockdev: Move qed probe to its own file
  blockdev: Move raw probe to its own file
  blockdev: Move vdi probe to its own file
  blockdev: Move vhdx probe to its own file
  blockdev: Move vmdk probe to its own file
  blockdev: Move vpc probe to its own file
  blockdev: Separate bochs probe from its driver
  blockdev: Separate cloop probe from its driver
  blockdev: Separate luks probe from its driver
  blockdev: Separate dmg probe from its driver
  blockdev: Separate parallels probe from its driver
  blockdev: Separate qcow probe from its driver
  blockdev: Separate qcow2 probe from its driver
  blockdev: Separate qed probe from its driver
  blockdev: Separate raw probe from its driver
  blockdev: Separate vdi probe from its driver
  blockdev: Separate vhdx probe from its driver
  blockdev: Separate vmdk probe from its driver
  blockdev: Separate vpc probe from its driver
  blockdev: Remove the .bdrv_probe field from BlockDrivers
  blockdev: Separate out bdrv_probe_device functions
  blockdev: Remove bdrv_probe_device field from BlockDriver

Marc Mari (2):
  blockdev: Add dynamic generation of module_block.h
  blockdev: Add dynamic module loading for block drivers

 Makefile|   7 ++
 block.c | 181 
 block/Makefile.objs |   4 +
 block/bochs-probe.c |  28 +++
 block/bochs.c   |  56 +
 block/bochs.h   |  40 +
 block/cloop-probe.c |  22 +
 block/cloop.c   |  17 +---
 block/crypto-probe.c|  26 ++
 block/crypto.c  |  22 +
 block/dmg-probe.c   |  22 +
 block/dmg.c |  17 +---
 block/host_cdrom-probe.c|  47 +++
 block/host_device-probe.c   |  42 ++
 block/iscsi.c   |  36 
 block/parallels-probe.c |  26 ++
 block/parallels.c   |  44 +-
 block/parallels.h   |  26 ++
 block/qcow-probe.c  |  22 +
 block/qcow.c|  32 +--
 block/qcow.h|  21 +
 block/qcow2-probe.c |  22 +
 block/qcow2.c   |  14 +---
 block/qed-probe.c   |  24 ++
 block/qed.c |  16 +---
 block/raw-posix.c   |  55 +---
 block/raw-probe.c   |  14 
 block/raw-win32.c   |  11 +--
 block/raw_bsd.c |  10 +--
 block/vdi-probe.c   |  29 +++
 block/vdi.c |  70 +---
 block/vdi.h |  49 +++
 block/vhdx-probe.c  |  27 ++
 block/vhdx.c|  21 +
 block/vmdk-probe.c  |  67 +++
 block/vmdk.c|  61 +-
 block/vmdk.h|   7 ++
 block/vpc-probe.c   |  16 
 block/vpc.c |   9 +-
 include/block/block_int.h   |   3 -
 include/block/probe.h   |  33 
 include/qemu/module.h   |   3 +
 scripts/modules/module_block.py | 108 
 util/module.c   |  38 +++--
 vl.c|  38 +
 45 files changed, 948 insertions(+), 535 deletions(-)
 create mode 100644 block/bochs-probe.c
 create mode 100644 block/bochs.h
 create mode 100644 block/cloop-probe.c
 create mode 100644 block/crypto-probe.c
 create mode 100644 block/dmg-probe.c
 create mode 100644 block/host_cdrom-probe.c
 create mode 100644 block/host_device-probe.c
 create mode 100644 block/parallels-probe.c
 create mode 100644 block/parallels.h
 create mode 100644 block/qcow-probe.c
 create mode 100644

[Qemu-block] [PATCH v4 07/32] blockdev: Move dmg probe to its own file

[Colin Lord]

Isolate dmg probe as part of the modularization process.

Signed-off-by: Colin Lord <cl...@redhat.com>
Reviewed-by: Max Reitz <mre...@redhat.com>
---
 block/Makefile.objs   |  2 +-
 block/dmg-probe.c | 17 +
 block/dmg.c   | 16 +---
 include/block/probe.h |  1 +
 4 files changed, 20 insertions(+), 16 deletions(-)
 create mode 100644 block/dmg-probe.c

diff --git a/block/Makefile.objs b/block/Makefile.objs
index aca547f..09cc64c 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -24,7 +24,7 @@ block-obj-y += accounting.o dirty-bitmap.o
 block-obj-y += write-threshold.o
 
 block-obj-y += crypto.o
-block-obj-y += bochs-probe.o cloop-probe.o crypto-probe.o
+block-obj-y += bochs-probe.o cloop-probe.o crypto-probe.o dmg-probe.o
 
 common-obj-y += stream.o
 common-obj-y += backup.o
diff --git a/block/dmg-probe.c b/block/dmg-probe.c
new file mode 100644
index 000..a40281b
--- /dev/null
+++ b/block/dmg-probe.c
@@ -0,0 +1,17 @@
+#include "qemu/osdep.h"
+#include "block/probe.h"
+
+int dmg_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+int len;
+
+if (!filename) {
+return 0;
+}
+
+len = strlen(filename);
+if (len > 4 && !strcmp(filename + len - 4, ".dmg")) {
+return 2;
+}
+return 0;
+}
diff --git a/block/dmg.c b/block/dmg.c
index b0ed89b..4bbb079 100644
--- a/block/dmg.c
+++ b/block/dmg.c
@@ -25,6 +25,7 @@
 #include "qapi/error.h"
 #include "qemu-common.h"
 #include "block/block_int.h"
+#include "block/probe.h"
 #include "qemu/bswap.h"
 #include "qemu/error-report.h"
 #include "qemu/module.h"
@@ -66,21 +67,6 @@ typedef struct BDRVDMGState {
 #endif
 } BDRVDMGState;
 
-static int dmg_probe(const uint8_t *buf, int buf_size, const char *filename)
-{
-int len;
-
-if (!filename) {
-return 0;
-}
-
-len = strlen(filename);
-if (len > 4 && !strcmp(filename + len - 4, ".dmg")) {
-return 2;
-}
-return 0;
-}
-
 static int read_uint64(BlockDriverState *bs, int64_t offset, uint64_t *result)
 {
 uint64_t buffer;
diff --git a/include/block/probe.h b/include/block/probe.h
index 35a8d00..267431d 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -5,5 +5,6 @@ int bochs_probe(const uint8_t *buf, int buf_size, const char 
*filename);
 int cloop_probe(const uint8_t *buf, int buf_size, const char *filename);
 int block_crypto_probe_luks(const uint8_t *buf, int buf_size,
 const char *filename);
+int dmg_probe(const uint8_t *buf, int buf_size, const char *filename);
 
 #endif
-- 
2.5.5

Re: [Qemu-block] [PATCH 3/3] iotests: Test format probes

[Colin Lord]

On 07/12/2016 05:17 PM, John Snow wrote:
> 
> 
> On 07/11/2016 03:50 PM, Colin Lord wrote:
>> Adds a new iotest for testing that the format probing functions work as
>> expected. This is done by booting up a vm with a disk image without
>> specifying the image format. Then the format is checked using a call to
>> query-block.
>>
>> For any format specified, at least one check is done with an image of
>> the given format, and one check is done against a raw image. This is to
>> ensure that a probe correctly returns a high score when the format is
>> matched, and does not return a high score when the format should not be
>> matched, as would be the case with raw images.
>>
>> Signed-off-by: Colin Lord <cl...@redhat.com>
>> ---
>>  tests/qemu-iotests/158| 53 
>> +++
>>  tests/qemu-iotests/158.out|  5 
>>  tests/qemu-iotests/group  |  1 +
>>  tests/qemu-iotests/iotests.py |  5 
>>  4 files changed, 64 insertions(+)
>>  create mode 100644 tests/qemu-iotests/158
>>  create mode 100644 tests/qemu-iotests/158.out
>>
>> diff --git a/tests/qemu-iotests/158 b/tests/qemu-iotests/158
>> new file mode 100644
>> index 000..70ad298
>> --- /dev/null
>> +++ b/tests/qemu-iotests/158
>> @@ -0,0 +1,53 @@
>> +#!/usr/bin/env python
>> +
> 
>  No license? It will default to GPL2, but it may be better to
> explicitly state the license for a new file. 
> 
>> +import iotests
>> +
>> +imgs = {'raw': ['grub_mbr.raw'], 'bochs': ['empty.bochs'],
>> +'cloop': ['simple-pattern.cloop'],
>> +'parallels': ['parallels-v1', 'parallels-v2'],
>> +'qcow': ['empty.qcow'], 'qcow2': ['empty.qcow2'],
>> +'qed': ['empty.qed'], 'vdi': ['empty.vdi'],
>> +'vpc': ['virtualpc-dynamic.vhd'], 'vhdx': 
>> ['iotest-dynamic-1G.vhdx'],
>> +'vmdk': ['iotest-version3.vmdk'], 'luks': ['empty.luks'],
>> +'dmg': ['empty.dmg']}
>> +
> 
> Might be nicer to do one format->file mapping per line.
> 
>> +class TestProbingFormats(iotests.QMPTestCase):
>> +def setUp(self):
>> +self.vm = iotests.VM()
>> +self.img_paths = []
>> +luks_opts = ''
>> +if iotests.imgfmt == 'luks':
>> +luks_opts = (',key-secret=sec0')
>> +for img_name in imgs[iotests.imgfmt]:
>> +self.img_paths.append(iotests.use_sample_image(img_name))
>> +self.vm.add_drive_raw('file=%s,id=drive%s,media=cdrom%s' %
>> +  (self.img_paths[-1], len(self.img_paths) 
>> - 1,
>> +  luks_opts))
> 
> Perhaps id=drive%s could be id=drive%d, but I'm no pythonista.
> 
>> +if iotests.imgfmt == 'luks':
>> +self.vm.use_luks()
>> +if iotests.imgfmt != 'raw':
>> +self.img_paths.append(iotests.use_sample_image(imgs['raw'][0]))
>> +self.vm.add_drive_raw('file=%s,id=drive%s,media=cdrom' %
>> +  (self.img_paths[-1], len(self.img_paths) 
>> - 1))
>> +self.vm.launch()
>> +
> 
> Oh, everything as a CDROM? I guess there's no reason that doesn't work,
> but it strikes me as a bit unconventional. I guess you'll be testing
> some quite esoteric configurations this way.
> 
Yeah it looks a little odd. I actually did it this way because some of
the image formats are read-only and so they throw errors if you try to
mount them as writable. I wasn't finding an option anywhere to mount
drives as read only so I just took the quick way out and went with
cdrom. Is there a better way to mount drives as read only?

>> +def tearDown(self):
>> +self.vm.shutdown()
>> +for img in self.img_paths:
>> +iotests.rm_test_image(img)
>> +
>> +def test_probe_detects_format(self):
>> +result = self.vm.qmp('query-block')
>> +for i in range(len(self.img_paths) - 1):
>> +self.assert_qmp(result, 'return[%s]/inserted/image/format' %
>> +i, iotests.imgfmt)
>> +
> 
> Seems sane.
> 
>> +def test_probe_detects_raw(self):
>> +result = self.vm.qmp('query-block')
>> +self.assert_qmp(result, 'return[%s]/inserted/image/format' %
>> +(len(self.img_paths) - 1), 'raw')
>> +
> 
> Why do we need to test raw against every format, exactly? The absence or
> presence of other drives shouldn't affect the probing, so allowing us to
> test the raw pro

[Qemu-block] [PATCH 0/3] Add tests for block driver format probes

[Colin Lord]

This series adds a couple simple tests for the block driver format
probes just to make sure that they're getting called correctly and
returning the expected values. This should help verify the correctness
of the other series I've been working on. The tests aren't super
complicated but are a good sanity check that things are still working as
they should be.

The commit message for the third patch pretty much explains how it
works, but basically the tests just boot up a vm with a sample image
file, check what format qemu detected the image was, and make sure that
the detected format is correct. Some of the sample images already exist
in the repository, but I also had to find/make some of my own. Being
binaries, I'm not sure how these should get checked in, so for now I'm
not including them. However, they can be accessed at my github if
desired:
https://github.com/dramborleg/qemu/tree/probe-iotests/tests/qemu-iotests/sample_images

Colin Lord (3):
  iotests: Add dmg format option
  iotests: Add python functions for using sample images
  iotests: Test format probes

 tests/qemu-iotests/158| 53 +++
 tests/qemu-iotests/158.out|  5 
 tests/qemu-iotests/common |  7 ++
 tests/qemu-iotests/group  |  1 +
 tests/qemu-iotests/iotests.py | 20 
 5 files changed, 86 insertions(+)
 create mode 100644 tests/qemu-iotests/158
 create mode 100644 tests/qemu-iotests/158.out

-- 
2.5.5

[Qemu-block] [PATCH 1/3] iotests: Add dmg format option

[Colin Lord]

Adds option to test the dmg format.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 tests/qemu-iotests/common | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/tests/qemu-iotests/common b/tests/qemu-iotests/common
index d60ea2c..dc937c7 100644
--- a/tests/qemu-iotests/common
+++ b/tests/qemu-iotests/common
@@ -140,6 +140,7 @@ common options
 
 check options
 -rawtest raw (default)
+-dmgtest dmg
 -bochs  test bochs
 -cloop  test cloop
 -parallels  test parallels
@@ -181,6 +182,12 @@ testlist options
 xpand=false
 ;;
 
+-dmg)
+IMGFMT=dmg
+IMGFMT_GENERIC=false
+xpand=false
+;;
+
 -bochs)
 IMGFMT=bochs
 IMGFMT_GENERIC=false
-- 
2.5.5

[Qemu-block] [PATCH 2/3] iotests: Add python functions for using sample images

[Colin Lord]

This adds a python equivalent of the _use_sample_img and _rm_sample_img
testing functions.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 tests/qemu-iotests/iotests.py | 15 +++
 1 file changed, 15 insertions(+)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 1687c33..3ba0de2 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -29,6 +29,7 @@ import qmp
 import qtest
 import struct
 import json
+import bz2
 
 
 # This will not work if arguments contain spaces but is necessary if we
@@ -53,6 +54,20 @@ cachemode = os.environ.get('CACHEMODE')
 qemu_default_machine = os.environ.get('QEMU_DEFAULT_MACHINE')
 
 socket_scm_helper = os.environ.get('SOCKET_SCM_HELPER', 'socket_scm_helper')
+sample_img_dir = os.environ.get('SAMPLE_IMG_DIR')
+
+def use_sample_image(sample_image_file):
+sample_img_path = os.path.join(sample_img_dir, sample_image_file + '.bz2')
+sample_img_file = bz2.BZ2File(sample_img_path)
+output_img_path = os.path.join(test_dir, sample_image_file)
+output_img = open(output_img_path, 'wb')
+output_img.write(sample_img_file.read())
+sample_img_file.close()
+output_img.close()
+return output_img_path
+
+def rm_test_image(test_image):
+os.remove(test_image)
 
 def qemu_img(*args):
 '''Run qemu-img and return the exit code'''
-- 
2.5.5

[Qemu-block] [PATCH 3/3] iotests: Test format probes

[Colin Lord]

Adds a new iotest for testing that the format probing functions work as
expected. This is done by booting up a vm with a disk image without
specifying the image format. Then the format is checked using a call to
query-block.

For any format specified, at least one check is done with an image of
the given format, and one check is done against a raw image. This is to
ensure that a probe correctly returns a high score when the format is
matched, and does not return a high score when the format should not be
matched, as would be the case with raw images.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 tests/qemu-iotests/158| 53 +++
 tests/qemu-iotests/158.out|  5 
 tests/qemu-iotests/group  |  1 +
 tests/qemu-iotests/iotests.py |  5 
 4 files changed, 64 insertions(+)
 create mode 100644 tests/qemu-iotests/158
 create mode 100644 tests/qemu-iotests/158.out

diff --git a/tests/qemu-iotests/158 b/tests/qemu-iotests/158
new file mode 100644
index 000..70ad298
--- /dev/null
+++ b/tests/qemu-iotests/158
@@ -0,0 +1,53 @@
+#!/usr/bin/env python
+
+import iotests
+
+imgs = {'raw': ['grub_mbr.raw'], 'bochs': ['empty.bochs'],
+'cloop': ['simple-pattern.cloop'],
+'parallels': ['parallels-v1', 'parallels-v2'],
+'qcow': ['empty.qcow'], 'qcow2': ['empty.qcow2'],
+'qed': ['empty.qed'], 'vdi': ['empty.vdi'],
+'vpc': ['virtualpc-dynamic.vhd'], 'vhdx': ['iotest-dynamic-1G.vhdx'],
+'vmdk': ['iotest-version3.vmdk'], 'luks': ['empty.luks'],
+'dmg': ['empty.dmg']}
+
+class TestProbingFormats(iotests.QMPTestCase):
+def setUp(self):
+self.vm = iotests.VM()
+self.img_paths = []
+luks_opts = ''
+if iotests.imgfmt == 'luks':
+luks_opts = (',key-secret=sec0')
+for img_name in imgs[iotests.imgfmt]:
+self.img_paths.append(iotests.use_sample_image(img_name))
+self.vm.add_drive_raw('file=%s,id=drive%s,media=cdrom%s' %
+  (self.img_paths[-1], len(self.img_paths) - 1,
+  luks_opts))
+if iotests.imgfmt == 'luks':
+self.vm.use_luks()
+if iotests.imgfmt != 'raw':
+self.img_paths.append(iotests.use_sample_image(imgs['raw'][0]))
+self.vm.add_drive_raw('file=%s,id=drive%s,media=cdrom' %
+  (self.img_paths[-1], len(self.img_paths) - 
1))
+self.vm.launch()
+
+def tearDown(self):
+self.vm.shutdown()
+for img in self.img_paths:
+iotests.rm_test_image(img)
+
+def test_probe_detects_format(self):
+result = self.vm.qmp('query-block')
+for i in range(len(self.img_paths) - 1):
+self.assert_qmp(result, 'return[%s]/inserted/image/format' %
+i, iotests.imgfmt)
+
+def test_probe_detects_raw(self):
+result = self.vm.qmp('query-block')
+self.assert_qmp(result, 'return[%s]/inserted/image/format' %
+(len(self.img_paths) - 1), 'raw')
+
+if __name__ == '__main__':
+iotests.main(supported_fmts=['raw', 'bochs', 'cloop', 'parallels', 'qcow',
+ 'qcow2', 'qed', 'vdi', 'vpc', 'vhdx', 'vmdk',
+ 'luks', 'dmg'])
diff --git a/tests/qemu-iotests/158.out b/tests/qemu-iotests/158.out
new file mode 100644
index 000..fbc63e6
--- /dev/null
+++ b/tests/qemu-iotests/158.out
@@ -0,0 +1,5 @@
+..
+--
+Ran 2 tests
+
+OK
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 1c6fcb6..4563e81 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -156,3 +156,4 @@
 154 rw auto backing quick
 155 rw auto
 156 rw auto quick
+158 quick
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 3ba0de2..101ae91 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -49,6 +49,7 @@ if os.environ.get('QEMU_OPTIONS'):
 imgfmt = os.environ.get('IMGFMT', 'raw')
 imgproto = os.environ.get('IMGPROTO', 'file')
 test_dir = os.environ.get('TEST_DIR')
+imgkeysecret = os.environ.get('IMGKEYSECRET')
 output_dir = os.environ.get('OUTPUT_DIR', '.')
 cachemode = os.environ.get('CACHEMODE')
 qemu_default_machine = os.environ.get('QEMU_DEFAULT_MACHINE')
@@ -208,6 +209,10 @@ class VM(object):
 self._num_drives += 1
 return self
 
+def use_luks(self):
+self._args.append('-object')
+self._args.append('secret,id=sec0,data=%s' % imgkeysecret)
+
 def pause_drive(self, drive, event=None):
 '''Pause drive r/w operations'''
 if not event:
-- 
2.5.5

Re: [Qemu-block] [Qemu-devel] [PATCH v3 04/32] blockdev: Move bochs probe into separate file

[Colin Lord]

On 07/06/2016 04:24 AM, Kevin Wolf wrote:
> Am 05.07.2016 um 22:50 hat John Snow geschrieben:
>>
>>
>> On 07/05/2016 11:49 AM, Daniel P. Berrange wrote:
>>> On Tue, Jul 05, 2016 at 11:24:04AM -0400, Colin Lord wrote:
>>>> This puts the bochs probe function into its own separate file as part of
>>>> the process of modularizing block drivers. Having the probe functions
>>>> separate from the rest of the driver allows us to probe without having
>>>> to potentially unnecessarily load the driver.
>>>>
>>>> Signed-off-by: Colin Lord <cl...@redhat.com>
>>>> ---
>>>>  block/Makefile.objs  |  1 +
>>>>  block/bochs.c| 55 
>>>> ++--
>>>>  block/probe/bochs.c  | 21 +
>>>
>>> Do we really need a sub-dir for this ?  If we were going to
>>> have sub-dirs under block/, I'd suggest we have one subdir
>>> per block driver, not spread code for one block driver
>>> across multiple dirs.
>>>
>>
>> Admittedly I have been nudging Colin to shoot from the hip a bit on
>> filename organization because I was short of ideas.
>>
>> Some ideas:
>>
>> (1) A combined probe.c file. This keeps the existing organization and
>> localizes everything to just one new file.
>>
>> Downside: many formats rely on at least some minimal amount of structure
>> and constant definitions, and some of those overlap with each other.
>> qcow and qcow2 both using "QcowHeader" would be a prominent example.
>>
>> They could all be disentangled, but it's less clear on where all the
>> common definitions go. A common probe.h is a bad idea since the modular
>> portion of the driver has no business gaining access to other formats'
>> definitions. We could create a probe.c and matching
>> include/block/bdrv/fmt.h includes, but we lost our zeal for this method.
>>
>> (2) Separate probe files for each driver.
>>
>> What we went with. Keeps refactoring to a minimum. Adds a bunch of
>> little files, but it's minimal and fairly noninvasive.
>>
>> Like #1 though, we still have to figure out what to do with the common
>> includes.
>>
>>> IMHO a block/bochs-probe.c would be better, unless we did
>>> move block/bochs.c into a block/bochs/driver.c dir.
>>>
>>> Either way, you should update MAINTAINERS file to record
>>> this newly added filename, against the bochs entry. The
>>> same applies to most other patches in this series adding
>>> new files.
>>>
>>>
>>> Regards,
>>> Daniel
>>>
>>
>> So, something like:
>>
>> block/drivers/bochs/
> 
> block/bochs/ if anything. We don't have to nest deeply just because we
> can. I don't really like new subdirectories, but when all drivers start
> to have multiple files, it might be unavoidable.
> 
>> bochs.c
>> probe.c (or bochs-probe.c)
>>
>> and
>>
>> include/block/drivers/bochs/
>>
>> common.h (or internal.h)
> 
> block/bochs/internal.h (or bochs.h)
> 
> Just like we already have some header files directly in block/ (e.g.
> qcow2.h). They are internal to the block driver, so no reason to move
> them to the global include directory.
> 
> Kevin
> 
So it sounds like what I should do is that for each driver I remove a
probe from, I'll create a new directory underneath block/ and put the
driver file, the probe file, and any new headers in it. eg for bochs,
there will be a new directory block/bochs/ under which there will be
block/bochs/bochs.c, block/bochs/bochs.h, and block/bochs/probe.c. Thus
no new headers will end up going in the include/ directory.

Also if that's what I'm doing, I assume I should leave alone any headers
already in the block/ directory that are used by other files that I'm
not touching? eg block/qed.h already exists and is used by files other
than the driver and probe, so it seems like I should leave that alone.

Colin

Re: [Qemu-block] [PATCH v3 17/32] blockdev: Separate bochs probe from its driver

[Colin Lord]

On 07/06/2016 11:59 AM, Max Reitz wrote:
> On 05.07.2016 17:24, Colin Lord wrote:
>> Modifies the bochs probe to return the format name as well as the
>> score as the final step of separating the probe function from the
>> driver. This keeps the probe completely independent of the driver,
>> making future modularization easier to accomplish. Returning the format
>> name as well as the score allows the score to be correlated to the
>> driver without the probe function needing to be part of the driver.
>>
>> Signed-off-by: Colin Lord <cl...@redhat.com>
>> ---
>>  block.c   | 19 +++
>>  block/bochs.c |  1 -
>>  block/probe/bochs.c   | 25 -
>>  include/block/probe.h |  3 ++-
>>  4 files changed, 37 insertions(+), 11 deletions(-)
> 
> Oh, and another comment: I think it would be better to split this patch
> into a part that modifies block.c (introducing the format_probes array
> and using it in bdrv_probe_all()) and another one that actually does the
> bochs-specific stuff.
> 
> Max
> 
I could do that, but if I leave the variable i as an unsigned type, I
get compiler warnings/errors that the for loop condition of
i < ARRAY_SIZE(format_probes)
must be false since the size of format_probes will be 0 (and unsigned is
always positive). Is it okay to change the type to int? Not sure how
strictly this aspect of the coding style gets enforced.

Colin

Re: [Qemu-block] [PATCH v3 02/32] blockdev: Add dynamic generation of module_block.h

[Colin Lord]

On 07/06/2016 09:17 AM, Max Reitz wrote:
> On 05.07.2016 17:24, Colin Lord wrote:
>> From: Marc Mari <mar...@redhat.com>
>>
>> To simplify the addition of new block modules, add a script that generates
>> module_block.h automatically from the modules' source code.
>>
>> This script assumes that the QEMU coding style rules are followed.
> 
> But the script itself doesn't follow it. :-P
> ("ERROR: code indent should never use tabs", to quote checkpatch.pl)
> 
I guess that is the case.

>>
>> Signed-off-by: Marc Marí <mar...@redhat.com>
>> Signed-off-by: Colin Lord <cl...@redhat.com>
>> ---
>>  Makefile|   7 +++
>>  scripts/modules/module_block.py | 122 
>> 
>>  2 files changed, 129 insertions(+)
>>  create mode 100644 scripts/modules/module_block.py
>>
> 
> [...]
> 
>> diff --git a/scripts/modules/module_block.py 
>> b/scripts/modules/module_block.py
>> new file mode 100644
>> index 000..4075574
>> --- /dev/null
>> +++ b/scripts/modules/module_block.py
>> @@ -0,0 +1,122 @@
> 
> [...]
> 
>> +def add_module(fheader, library, format_name, protocol_name,
>> +probe, probe_device):
>> +lines = []
>> +lines.append('.library_name = "' + library + '",')
>> +if format_name != "":
>> +lines.append('.format_name = "' + format_name + '",')
>> +if protocol_name != "":
>> +lines.append('.protocol_name = "' + protocol_name + '",')
>> +if probe:
>> +lines.append('.has_probe = true,')
>> +if probe_device:
>> +lines.append('.has_probe_device = true,')
>> +
>> +text = '\n\t'.join(lines)
>> +fheader.write('\n\t{\n\t' + text + '\n\t},')
> 
> I think our normal coding style would be met with:
> 
> text = '\n'.join(lines)
> fheader.write('\n{\n' + text + '\n},')
> 
> Rest looks good, so with that fixed (or maybe we don't need to fix it
> because it's just a generated header, I don't know):
> 
Shouldn't be too big a deal to fix.

> Reviewed-by: Max Reitz <mre...@redhat.com>
> 

[Qemu-block] [PATCH v3 23/32] blockdev: Separate qcow2 probe from its driver

[Colin Lord]

Completes the separation of the qcow2 probe from the qcow2 driver. The
qcow2 probe now returns the format in addition to the score, allowing
correlation of the score and driver without the probe function being
part of the driver itself.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block.c   |  1 +
 block/probe/qcow2.c   | 16 +++-
 block/qcow2.c |  1 -
 include/block/probe.h |  3 ++-
 4 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/block.c b/block.c
index 236e562..b4e347a 100644
--- a/block.c
+++ b/block.c
@@ -67,6 +67,7 @@ static BdrvProbeFunc *format_probes[] = {
 dmg_probe,
 parallels_probe,
 qcow_probe,
+qcow2_probe,
 };
 
 static QTAILQ_HEAD(, BlockDriverState) graph_bdrv_states =
diff --git a/block/probe/qcow2.c b/block/probe/qcow2.c
index 56f4e82..bd351e0 100644
--- a/block/probe/qcow2.c
+++ b/block/probe/qcow2.c
@@ -3,14 +3,20 @@
 #include "block/probe.h"
 #include "block/qcow2.h"
 
-int qcow2_probe(const uint8_t *buf, int buf_size, const char *filename)
+const char *qcow2_probe(const uint8_t *buf, int buf_size, const char *filename,
+int *score)
 {
+const char *format = "qcow2";
 const QCowHeader *cow_header = (const void *)buf;
+assert(score);
 
 if (buf_size >= sizeof(QCowHeader) &&
 be32_to_cpu(cow_header->magic) == QCOW_MAGIC &&
-be32_to_cpu(cow_header->version) >= 2)
-return 100;
-else
-return 0;
+be32_to_cpu(cow_header->version) >= 2) {
+*score = 100;
+return format;
+}
+
+*score = 0;
+return format;
 }
diff --git a/block/qcow2.c b/block/qcow2.c
index 55639eb..bc2da36 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -3336,7 +3336,6 @@ static QemuOptsList qcow2_create_opts = {
 BlockDriver bdrv_qcow2 = {
 .format_name= "qcow2",
 .instance_size  = sizeof(BDRVQcow2State),
-.bdrv_probe = qcow2_probe,
 .bdrv_open  = qcow2_open,
 .bdrv_close = qcow2_close,
 .bdrv_reopen_prepare  = qcow2_reopen_prepare,
diff --git a/include/block/probe.h b/include/block/probe.h
index 3aeab2d..222185b 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -1,7 +1,6 @@
 #ifndef PROBE_H
 #define PROBE_H
 
-int qcow2_probe(const uint8_t *buf, int buf_size, const char *filename);
 int bdrv_qed_probe(const uint8_t *buf, int buf_size, const char *filename);
 int raw_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vdi_probe(const uint8_t *buf, int buf_size, const char *filename);
@@ -20,5 +19,7 @@ const char *parallels_probe(const uint8_t *buf, int buf_size,
 const char *filename, int *score);
 const char *qcow_probe(const uint8_t *buf, int buf_size, const char *filename,
int *score);
+const char *qcow2_probe(const uint8_t *buf, int buf_size, const char *filename,
+int *score);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v3 32/32] blockdev: Remove bdrv_probe_device field from BlockDriver

[Colin Lord]

This commit finalizes the separation of the BlockDriver from its
device probing function. Now the accesses to these functions in block.c
occur through the protocol_probes array, and each function returns a
score and protocol name with which to find the corresponding driver.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block.c | 46 ++---
 block/probe/host_cdrom.c| 23 ++---
 block/probe/host_device.c   | 34 --
 block/raw-posix.c   |  3 ---
 block/raw-win32.c   |  1 -
 include/block/block_int.h   |  2 --
 include/block/probe.h   |  4 ++--
 scripts/modules/module_block.py | 12 ++-
 8 files changed, 76 insertions(+), 49 deletions(-)

diff --git a/block.c b/block.c
index 7e441fe..bc1046b 100644
--- a/block.c
+++ b/block.c
@@ -59,6 +59,7 @@
 
 typedef const char *BdrvProbeFunc(const uint8_t *buf, int buf_size,
   const char *filename, int *score);
+typedef const char *BdrvProbeDevFunc(const char *filename, int *score);
 
 static BdrvProbeFunc *format_probes[] = {
 bochs_probe,
@@ -76,6 +77,13 @@ static BdrvProbeFunc *format_probes[] = {
 vpc_probe
 };
 
+static BdrvProbeDevFunc *protocol_probes[] = {
+hdev_probe_device,
+#if defined(__FreeBSD__) || defined(__FreeBSD_kernel__) || defined(__linux__)
+cdrom_probe_device
+#endif
+};
+
 static QTAILQ_HEAD(, BlockDriverState) graph_bdrv_states =
 QTAILQ_HEAD_INITIALIZER(graph_bdrv_states);
 
@@ -95,6 +103,8 @@ static BlockDriverState *bdrv_open_inherit(const char 
*filename,
 /* If non-zero, use only whitelisted block drivers */
 static int use_bdrv_whitelist;
 
+static BlockDriver *bdrv_do_find_protocol(const char *protocol);
+
 #ifdef _WIN32
 static int is_windows_drive_prefix(const char *filename)
 {
@@ -487,25 +497,37 @@ int get_tmp_filename(char *filename, int size)
 static BlockDriver *find_hdev_driver(const char *filename)
 {
 int score_max = 0, score;
+const char *protocol_max = NULL;
+const char *protocol;
+BlockDriver *drv;
 size_t i;
-BlockDriver *drv = NULL, *d;
+
+for (i = 0; i < ARRAY_SIZE(protocol_probes); i++) {
+protocol = protocol_probes[i](filename, );
+if (score > score_max) {
+protocol_max = protocol;
+score_max = score;
+}
+}
+
+if (!protocol_max) {
+return NULL;
+}
+
+drv = bdrv_do_find_protocol(protocol_max);
+if (drv) {
+return drv;
+}
 
 for (i = 0; i < ARRAY_SIZE(block_driver_modules); ++i) {
-if (block_driver_modules[i].has_probe_device) {
+if (block_driver_modules[i].protocol_name &&
+!strcmp(block_driver_modules[i].protocol_name, protocol_max)) {
 block_module_load_one(block_driver_modules[i].library_name);
+break;
 }
 }
 
-QLIST_FOREACH(d, _drivers, list) {
-if (d->bdrv_probe_device) {
-score = d->bdrv_probe_device(filename);
-if (score > score_max) {
-score_max = score;
-drv = d;
-}
-}
-}
-
+drv = bdrv_do_find_protocol(protocol);
 return drv;
 }
 
diff --git a/block/probe/host_cdrom.c b/block/probe/host_cdrom.c
index 1886cad..3f7d863 100644
--- a/block/probe/host_cdrom.c
+++ b/block/probe/host_cdrom.c
@@ -1,22 +1,28 @@
 #include "qemu/osdep.h"
 #include "block/probe.h"
 
+static const char *protocol = "host_cdrom";
+
 #if defined(__FreeBSD__) || defined(__FreeBSD_kernel__)
-int cdrom_probe_device(const char *filename)
+const char *cdrom_probe_device(const char *filename, int *score)
 {
+assert(score);
 if (strstart(filename, "/dev/cd", NULL) ||
-strstart(filename, "/dev/acd", NULL))
-return 100;
+strstart(filename, "/dev/acd", NULL)) {
+*score = 100;
+return protocol;
+}
 return 0;
 }
 #elif defined(__linux__)
 #include 
 #include 
-int cdrom_probe_device(const char *filename)
+const char *cdrom_probe_device(const char *filename, int *score)
 {
 int fd, ret;
-int prio = 0;
 struct stat st;
+assert(score);
+*score = 0;
 
 fd = qemu_open(filename, O_RDONLY | O_NONBLOCK);
 if (fd < 0) {
@@ -29,12 +35,13 @@ int cdrom_probe_device(const char *filename)
 
 /* Attempt to detect via a CDROM specific ioctl */
 ret = ioctl(fd, CDROM_DRIVE_STATUS, CDSL_CURRENT);
-if (ret >= 0)
-prio = 100;
+if (ret >= 0) {
+*score = 100;
+}
 
 outc:
 qemu_close(fd);
 out:
-return prio;
+return protocol;
 }
 #endif
diff --git a/block/probe/host_device.c b/block/probe/host_device.c
index ebd969b..b4e4d20 100644
--- a/block/probe/host_device.c
+++ b/block/probe/host_device.c
@@ -2,29 +2,41 @@
 #include "block/probe.h"
 #include &qu

[Qemu-block] [PATCH v3 29/32] blockdev: Separate vpc probe from its driver

[Colin Lord]

Completes the separation of the vpc probe from the vpc driver. The
vpc probe now returns the format in addition to the score, allowing
correlation of the score and driver without the probe function being
part of the driver itself.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block.c   |  1 +
 block/probe/vpc.c | 15 +++
 block/vpc.c   |  1 -
 include/block/probe.h |  3 ++-
 4 files changed, 14 insertions(+), 6 deletions(-)

diff --git a/block.c b/block.c
index 9b839df..8226124 100644
--- a/block.c
+++ b/block.c
@@ -73,6 +73,7 @@ static BdrvProbeFunc *format_probes[] = {
 vdi_probe,
 vhdx_probe,
 vmdk_probe,
+vpc_probe
 };
 
 static QTAILQ_HEAD(, BlockDriverState) graph_bdrv_states =
diff --git a/block/probe/vpc.c b/block/probe/vpc.c
index afe8271..0fe8a65 100644
--- a/block/probe/vpc.c
+++ b/block/probe/vpc.c
@@ -1,9 +1,16 @@
 #include "qemu/osdep.h"
 #include "block/probe.h"
 
-int vpc_probe(const uint8_t *buf, int buf_size, const char *filename)
+const char *vpc_probe(const uint8_t *buf, int buf_size, const char *filename,
+  int *score)
 {
-if (buf_size >= 8 && !strncmp((char *)buf, "conectix", 8))
-   return 100;
-return 0;
+const char *format = "vpc";
+assert(score);
+if (buf_size >= 8 && !strncmp((char *)buf, "conectix", 8)) {
+*score = 100;
+return format;
+}
+
+*score = 0;
+return format;
 }
diff --git a/block/vpc.c b/block/vpc.c
index cb65022..cd87d8f 100644
--- a/block/vpc.c
+++ b/block/vpc.c
@@ -1050,7 +1050,6 @@ static BlockDriver bdrv_vpc = {
 .format_name= "vpc",
 .instance_size  = sizeof(BDRVVPCState),
 
-.bdrv_probe = vpc_probe,
 .bdrv_open  = vpc_open,
 .bdrv_close = vpc_close,
 .bdrv_reopen_prepare= vpc_reopen_prepare,
diff --git a/include/block/probe.h b/include/block/probe.h
index 13df1d3..7facb75 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -1,7 +1,6 @@
 #ifndef PROBE_H
 #define PROBE_H
 
-int vpc_probe(const uint8_t *buf, int buf_size, const char *filename);
 const char *bochs_probe(const uint8_t *buf, int buf_size, const char *filename,
 int *score);
 const char *cloop_probe(const uint8_t *buf, int buf_size, const char *filename,
@@ -26,5 +25,7 @@ const char *vhdx_probe(const uint8_t *buf, int buf_size, 
const char *filename,
int *score);
 const char *vmdk_probe(const uint8_t *buf, int buf_size, const char *filename,
int *score);
+const char *vpc_probe(const uint8_t *buf, int buf_size, const char *filename,
+  int *score);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v3 22/32] blockdev: Separate qcow probe from its driver

[Colin Lord]

Completes the separation of the qcow probe from the qcow driver. The
qcow probe now returns the format in addition to the score, allowing
correlation of the score and driver without the probe function being
part of the driver itself.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block.c   |  1 +
 block/probe/qcow.c| 16 +++-
 block/qcow.c  |  1 -
 include/block/probe.h |  3 ++-
 4 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/block.c b/block.c
index 0f4c9f6..236e562 100644
--- a/block.c
+++ b/block.c
@@ -66,6 +66,7 @@ static BdrvProbeFunc *format_probes[] = {
 block_crypto_probe_luks,
 dmg_probe,
 parallels_probe,
+qcow_probe,
 };
 
 static QTAILQ_HEAD(, BlockDriverState) graph_bdrv_states =
diff --git a/block/probe/qcow.c b/block/probe/qcow.c
index 6024d11..d249e02 100644
--- a/block/probe/qcow.c
+++ b/block/probe/qcow.c
@@ -3,14 +3,20 @@
 #include "block/probe.h"
 #include "block/driver/qcow.h"
 
-int qcow_probe(const uint8_t *buf, int buf_size, const char *filename)
+const char *qcow_probe(const uint8_t *buf, int buf_size, const char *filename,
+   int *score)
 {
+const char *format = "qcow";
 const QCowHeader *cow_header = (const void *)buf;
+assert(score);
 
 if (buf_size >= sizeof(QCowHeader) &&
 be32_to_cpu(cow_header->magic) == QCOW_MAGIC &&
-be32_to_cpu(cow_header->version) == QCOW_VERSION)
-return 100;
-else
-return 0;
+be32_to_cpu(cow_header->version) == QCOW_VERSION) {
+*score = 100;
+return format;
+}
+
+*score = 0;
+return format;
 }
diff --git a/block/qcow.c b/block/qcow.c
index 828749c..9475ed0 100644
--- a/block/qcow.c
+++ b/block/qcow.c
@@ -1002,7 +1002,6 @@ static QemuOptsList qcow_create_opts = {
 static BlockDriver bdrv_qcow = {
 .format_name   = "qcow",
 .instance_size = sizeof(BDRVQcowState),
-.bdrv_probe= qcow_probe,
 .bdrv_open = qcow_open,
 .bdrv_close= qcow_close,
 .bdrv_reopen_prepare= qcow_reopen_prepare,
diff --git a/include/block/probe.h b/include/block/probe.h
index 7383a64..3aeab2d 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -1,7 +1,6 @@
 #ifndef PROBE_H
 #define PROBE_H
 
-int qcow_probe(const uint8_t *buf, int buf_size, const char *filename);
 int qcow2_probe(const uint8_t *buf, int buf_size, const char *filename);
 int bdrv_qed_probe(const uint8_t *buf, int buf_size, const char *filename);
 int raw_probe(const uint8_t *buf, int buf_size, const char *filename);
@@ -19,5 +18,7 @@ const char *dmg_probe(const uint8_t *buf, int buf_size, const 
char *filename,
   int *score);
 const char *parallels_probe(const uint8_t *buf, int buf_size,
 const char *filename, int *score);
+const char *qcow_probe(const uint8_t *buf, int buf_size, const char *filename,
+   int *score);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v3 21/32] blockdev: Separate parallels probe from its driver

[Colin Lord]

Completes the separation of the parallels probe from the parallels
driver. The parallels probe now returns the format in addition to the
score, allowing correlation of the score and driver without the probe
function being part of the driver itself.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block.c |  1 +
 block/parallels.c   |  1 -
 block/probe/parallels.c | 18 +++---
 include/block/probe.h   |  3 ++-
 4 files changed, 14 insertions(+), 9 deletions(-)

diff --git a/block.c b/block.c
index 7dbe201..0f4c9f6 100644
--- a/block.c
+++ b/block.c
@@ -65,6 +65,7 @@ static BdrvProbeFunc *format_probes[] = {
 cloop_probe,
 block_crypto_probe_luks,
 dmg_probe,
+parallels_probe,
 };
 
 static QTAILQ_HEAD(, BlockDriverState) graph_bdrv_states =
diff --git a/block/parallels.c b/block/parallels.c
index 547373e..6994c41 100644
--- a/block/parallels.c
+++ b/block/parallels.c
@@ -709,7 +709,6 @@ static QemuOptsList parallels_create_opts = {
 static BlockDriver bdrv_parallels = {
 .format_name   = "parallels",
 .instance_size = sizeof(BDRVParallelsState),
-.bdrv_probe= parallels_probe,
 .bdrv_open = parallels_open,
 .bdrv_close= parallels_close,
 .bdrv_co_get_block_status = parallels_co_get_block_status,
diff --git a/block/probe/parallels.c b/block/probe/parallels.c
index 66cddea..3ebeddd 100644
--- a/block/probe/parallels.c
+++ b/block/probe/parallels.c
@@ -3,20 +3,24 @@
 #include "block/probe.h"
 #include "block/driver/parallels.h"
 
-int parallels_probe(const uint8_t *buf, int buf_size,
-   const char *filename)
+const char *parallels_probe(const uint8_t *buf, int buf_size,
+const char *filename, int *score)
 {
+const char *format = "parallels";
 const ParallelsHeader *ph = (const void *)buf;
+assert(score);
+*score = 0;
 
 if (buf_size < sizeof(ParallelsHeader)) {
-return 0;
+return format;
 }
 
 if ((!memcmp(ph->magic, HEADER_MAGIC, 16) ||
-   !memcmp(ph->magic, HEADER_MAGIC2, 16)) &&
-   (le32_to_cpu(ph->version) == HEADER_VERSION)) {
-return 100;
+!memcmp(ph->magic, HEADER_MAGIC2, 16)) &&
+(le32_to_cpu(ph->version) == HEADER_VERSION)) {
+*score = 100;
+return format;
 }
 
-return 0;
+return format;
 }
diff --git a/include/block/probe.h b/include/block/probe.h
index ef6629f..7383a64 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -1,7 +1,6 @@
 #ifndef PROBE_H
 #define PROBE_H
 
-int parallels_probe(const uint8_t *buf, int buf_size, const char *filename);
 int qcow_probe(const uint8_t *buf, int buf_size, const char *filename);
 int qcow2_probe(const uint8_t *buf, int buf_size, const char *filename);
 int bdrv_qed_probe(const uint8_t *buf, int buf_size, const char *filename);
@@ -18,5 +17,7 @@ const char *block_crypto_probe_luks(const uint8_t *buf, int 
buf_size,
 const char *filename, int *score);
 const char *dmg_probe(const uint8_t *buf, int buf_size, const char *filename,
   int *score);
+const char *parallels_probe(const uint8_t *buf, int buf_size,
+const char *filename, int *score);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v3 24/32] blockdev: Separate qed probe from its driver

[Colin Lord]

Completes the separation of the qed probe from the qed driver. The
qed probe now returns the format in addition to the score, allowing
correlation of the score and driver without the probe function being
part of the driver itself.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block.c   |  1 +
 block/probe/qed.c | 16 +++-
 block/qed.c   |  1 -
 include/block/probe.h |  3 ++-
 4 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/block.c b/block.c
index b4e347a..fee9f8c 100644
--- a/block.c
+++ b/block.c
@@ -68,6 +68,7 @@ static BdrvProbeFunc *format_probes[] = {
 parallels_probe,
 qcow_probe,
 qcow2_probe,
+bdrv_qed_probe,
 };
 
 static QTAILQ_HEAD(, BlockDriverState) graph_bdrv_states =
diff --git a/block/probe/qed.c b/block/probe/qed.c
index c902489..3fdb5d7 100644
--- a/block/probe/qed.c
+++ b/block/probe/qed.c
@@ -3,16 +3,22 @@
 #include "block/probe.h"
 #include "block/qed.h"
 
-int bdrv_qed_probe(const uint8_t *buf, int buf_size,
-  const char *filename)
+const char *bdrv_qed_probe(const uint8_t *buf, int buf_size,
+   const char *filename, int *score)
 {
+const char *format = "qed";
 const QEDHeader *header = (const QEDHeader *)buf;
+assert(score);
+*score = 0;
 
 if (buf_size < sizeof(*header)) {
-return 0;
+return format;
 }
+
 if (le32_to_cpu(header->magic) != QED_MAGIC) {
-return 0;
+return format;
 }
-return 100;
+
+*score = 100;
+return format;
 }
diff --git a/block/qed.c b/block/qed.c
index 21d8982..cf78ff0 100644
--- a/block/qed.c
+++ b/block/qed.c
@@ -1638,7 +1638,6 @@ static BlockDriver bdrv_qed = {
 .create_opts  = _create_opts,
 .supports_backing = true,
 
-.bdrv_probe   = bdrv_qed_probe,
 .bdrv_open= bdrv_qed_open,
 .bdrv_close   = bdrv_qed_close,
 .bdrv_reopen_prepare  = bdrv_qed_reopen_prepare,
diff --git a/include/block/probe.h b/include/block/probe.h
index 222185b..63edd74 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -1,7 +1,6 @@
 #ifndef PROBE_H
 #define PROBE_H
 
-int bdrv_qed_probe(const uint8_t *buf, int buf_size, const char *filename);
 int raw_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vdi_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vhdx_probe(const uint8_t *buf, int buf_size, const char *filename);
@@ -21,5 +20,7 @@ const char *qcow_probe(const uint8_t *buf, int buf_size, 
const char *filename,
int *score);
 const char *qcow2_probe(const uint8_t *buf, int buf_size, const char *filename,
 int *score);
+const char *bdrv_qed_probe(const uint8_t *buf, int buf_size,
+   const char *filename, int *score);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v3 27/32] blockdev: Separate vhdx probe from its driver

[Colin Lord]

Completes the separation of the vhdx probe from the vhdx driver. The
vhdx probe now returns the format in addition to the score, allowing
correlation of the score and driver without the probe function being
part of the driver itself.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block.c   |  1 +
 block/probe/vhdx.c| 12 +---
 block/vhdx.c  |  1 -
 include/block/probe.h |  3 ++-
 4 files changed, 12 insertions(+), 5 deletions(-)

diff --git a/block.c b/block.c
index fd97795..cc48279 100644
--- a/block.c
+++ b/block.c
@@ -71,6 +71,7 @@ static BdrvProbeFunc *format_probes[] = {
 bdrv_qed_probe,
 raw_probe,
 vdi_probe,
+vhdx_probe,
 };
 
 static QTAILQ_HEAD(, BlockDriverState) graph_bdrv_states =
diff --git a/block/probe/vhdx.c b/block/probe/vhdx.c
index 6c38aac..cbb74c1 100644
--- a/block/probe/vhdx.c
+++ b/block/probe/vhdx.c
@@ -12,10 +12,16 @@
  *
  *  Therefore, we probe by looking for the vhdxfile signature "vhdxfile"
  */
-int vhdx_probe(const uint8_t *buf, int buf_size, const char *filename)
+const char *vhdx_probe(const uint8_t *buf, int buf_size, const char *filename,
+   int *score)
 {
+const char *format = "vhdx";
+assert(score);
+
 if (buf_size >= 8 && !memcmp(buf, "vhdxfile", 8)) {
-return 100;
+*score = 100;
+return format;
 }
-return 0;
+*score = 0;
+return format;
 }
diff --git a/block/vhdx.c b/block/vhdx.c
index ba8adfe..2353569 100644
--- a/block/vhdx.c
+++ b/block/vhdx.c
@@ -1942,7 +1942,6 @@ static QemuOptsList vhdx_create_opts = {
 static BlockDriver bdrv_vhdx = {
 .format_name= "vhdx",
 .instance_size  = sizeof(BDRVVHDXState),
-.bdrv_probe = vhdx_probe,
 .bdrv_open  = vhdx_open,
 .bdrv_close = vhdx_close,
 .bdrv_reopen_prepare= vhdx_reopen_prepare,
diff --git a/include/block/probe.h b/include/block/probe.h
index b19b586..3e2e328 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -1,7 +1,6 @@
 #ifndef PROBE_H
 #define PROBE_H
 
-int vhdx_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vmdk_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vpc_probe(const uint8_t *buf, int buf_size, const char *filename);
 const char *bochs_probe(const uint8_t *buf, int buf_size, const char *filename,
@@ -24,5 +23,7 @@ const char *raw_probe(const uint8_t *buf, int buf_size, const 
char *filename,
   int *score);
 const char *vdi_probe(const uint8_t *buf, int buf_size, const char *filename,
   int *score);
+const char *vhdx_probe(const uint8_t *buf, int buf_size, const char *filename,
+   int *score);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v3 30/32] blockdev: Remove the .bdrv_probe field from BlockDrivers

[Colin Lord]

This commit finalizes the separation of the block driver and probe
function by removing the .bdrv_probe field from all BlockDrivers.
Probing is now accomplished solely by iterating over the array of probe
function pointers in the format_probes array.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block.c | 20 +---
 block/raw-posix.c   |  1 -
 include/block/block_int.h   |  1 -
 scripts/modules/module_block.py | 10 ++
 4 files changed, 3 insertions(+), 29 deletions(-)

diff --git a/block.c b/block.c
index 8226124..7e441fe 100644
--- a/block.c
+++ b/block.c
@@ -599,34 +599,16 @@ BlockDriver *bdrv_probe_all(const uint8_t *buf, int 
buf_size,
 const char *format_max = NULL;
 const char *format;
 size_t i;
-BlockDriver *drv = NULL, *d;
-
-for (i = 0; i < ARRAY_SIZE(block_driver_modules); ++i) {
-if (block_driver_modules[i].has_probe) {
-block_module_load_one(block_driver_modules[i].library_name);
-}
-}
-
-QLIST_FOREACH(d, _drivers, list) {
-if (d->bdrv_probe) {
-score = d->bdrv_probe(buf, buf_size, filename);
-if (score > score_max) {
-score_max = score;
-drv = d;
-}
-}
-}
 
 for (i = 0; i < ARRAY_SIZE(format_probes); i++) {
 format = format_probes[i](buf, buf_size, filename, );
 if (score > score_max) {
 score_max = score;
 format_max = format;
-drv = bdrv_find_format(format_max);
 }
 }
 
-return drv;
+return bdrv_find_format(format_max);
 }
 
 static int find_image_format(BlockDriverState *bs, const char *filename,
diff --git a/block/raw-posix.c b/block/raw-posix.c
index bef7a67..a6ad689 100644
--- a/block/raw-posix.c
+++ b/block/raw-posix.c
@@ -1936,7 +1936,6 @@ BlockDriver bdrv_file = {
 .protocol_name = "file",
 .instance_size = sizeof(BDRVRawState),
 .bdrv_needs_filename = true,
-.bdrv_probe = NULL, /* no probe for protocols */
 .bdrv_parse_filename = raw_parse_filename,
 .bdrv_file_open = raw_open,
 .bdrv_reopen_prepare = raw_reopen_prepare,
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 2057156..2bca115 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -99,7 +99,6 @@ struct BlockDriver {
 bool (*bdrv_recurse_is_first_non_filter)(BlockDriverState *bs,
  BlockDriverState *candidate);
 
-int (*bdrv_probe)(const uint8_t *buf, int buf_size, const char *filename);
 int (*bdrv_probe_device)(const char *filename);
 
 /* Any driver implementing this callback is expected to be able to handle
diff --git a/scripts/modules/module_block.py b/scripts/modules/module_block.py
index 4075574..18200e2 100644
--- a/scripts/modules/module_block.py
+++ b/scripts/modules/module_block.py
@@ -24,15 +24,13 @@ def get_string_struct(line):
 return data[2].replace('"', '')[:-1]
 
 def add_module(fheader, library, format_name, protocol_name,
-probe, probe_device):
+   probe_device):
 lines = []
 lines.append('.library_name = "' + library + '",')
 if format_name != "":
 lines.append('.format_name = "' + format_name + '",')
 if protocol_name != "":
 lines.append('.protocol_name = "' + protocol_name + '",')
-if probe:
-lines.append('.has_probe = true,')
 if probe_device:
 lines.append('.has_probe_device = true,')
 
@@ -52,20 +50,17 @@ def process_file(fheader, filename):
 format_name = get_string_struct(line)
 elif line.find(".protocol_name") != -1:
 protocol_name = get_string_struct(line)
-elif line.find(".bdrv_probe") != -1:
-probe = True
 elif line.find(".bdrv_probe_device") != -1:
 probe_device = True
 elif line == "};":
 add_module(fheader, library, format_name, protocol_name,
-probe, probe_device)
+   probe_device)
 found_start = False
 elif line.find("static BlockDriver") != -1:
 found_something = True
 found_start = True
 format_name = ""
 protocol_name = ""
-probe = False
 probe_device = False
 
 if not found_something:
@@ -93,7 +88,6 @@ static const struct {
 const char *format_name;
 const char *protocol_name;
 const char *library_name;
-bool has_probe;
 bool has_probe_device;
 } block_driver_modules[] = {''')
 
-- 
2.5.5

[Qemu-block] [PATCH v3 26/32] blockdev: Separate vdi probe from its driver

[Colin Lord]

Completes the separation of the vdi probe from the vdi driver. The
vdi probe now returns the format in addition to the score, allowing
correlation of the score and driver without the probe function being
part of the driver itself.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block.c   |  1 +
 block/probe/vdi.c | 13 -
 block/vdi.c   |  1 -
 include/block/probe.h |  3 ++-
 4 files changed, 11 insertions(+), 7 deletions(-)

diff --git a/block.c b/block.c
index a3d983f..fd97795 100644
--- a/block.c
+++ b/block.c
@@ -70,6 +70,7 @@ static BdrvProbeFunc *format_probes[] = {
 qcow2_probe,
 bdrv_qed_probe,
 raw_probe,
+vdi_probe,
 };
 
 static QTAILQ_HEAD(, BlockDriverState) graph_bdrv_states =
diff --git a/block/probe/vdi.c b/block/probe/vdi.c
index 9adf5e5..1dcf7ab 100644
--- a/block/probe/vdi.c
+++ b/block/probe/vdi.c
@@ -3,24 +3,27 @@
 #include "block/probe.h"
 #include "block/driver/vdi.h"
 
-int vdi_probe(const uint8_t *buf, int buf_size, const char *filename)
+const char *vdi_probe(const uint8_t *buf, int buf_size, const char *filename,
+  int *score)
 {
+const char *format = "vdi";
 const VdiHeader *header = (const VdiHeader *)buf;
-int ret = 0;
+assert(score);
+*score = 0;
 
 logout("\n");
 
 if (buf_size < sizeof(*header)) {
 /* Header too small, no VDI. */
 } else if (le32_to_cpu(header->signature) == VDI_SIGNATURE) {
-ret = 100;
+*score = 100;
 }
 
-if (ret == 0) {
+if (*score == 0) {
 logout("no vdi image\n");
 } else {
 logout("%s", header->text);
 }
 
-return ret;
+return format;
 }
diff --git a/block/vdi.c b/block/vdi.c
index e99ae92..445e2b8 100644
--- a/block/vdi.c
+++ b/block/vdi.c
@@ -849,7 +849,6 @@ static QemuOptsList vdi_create_opts = {
 static BlockDriver bdrv_vdi = {
 .format_name = "vdi",
 .instance_size = sizeof(BDRVVdiState),
-.bdrv_probe = vdi_probe,
 .bdrv_open = vdi_open,
 .bdrv_close = vdi_close,
 .bdrv_reopen_prepare = vdi_reopen_prepare,
diff --git a/include/block/probe.h b/include/block/probe.h
index b49663d..b19b586 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -1,7 +1,6 @@
 #ifndef PROBE_H
 #define PROBE_H
 
-int vdi_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vhdx_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vmdk_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vpc_probe(const uint8_t *buf, int buf_size, const char *filename);
@@ -23,5 +22,7 @@ const char *bdrv_qed_probe(const uint8_t *buf, int buf_size,
const char *filename, int *score);
 const char *raw_probe(const uint8_t *buf, int buf_size, const char *filename,
   int *score);
+const char *vdi_probe(const uint8_t *buf, int buf_size, const char *filename,
+  int *score);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v3 25/32] blockdev: Separate raw probe from its driver

[Colin Lord]

Completes the separation of the raw probe from the raw driver. The
raw probe now returns the format in addition to the score, allowing
correlation of the score and driver without the probe function being
part of the driver itself.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block.c   | 1 +
 block/probe/raw.c | 8 ++--
 block/raw_bsd.c   | 1 -
 include/block/probe.h | 3 ++-
 4 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/block.c b/block.c
index fee9f8c..a3d983f 100644
--- a/block.c
+++ b/block.c
@@ -69,6 +69,7 @@ static BdrvProbeFunc *format_probes[] = {
 qcow_probe,
 qcow2_probe,
 bdrv_qed_probe,
+raw_probe,
 };
 
 static QTAILQ_HEAD(, BlockDriverState) graph_bdrv_states =
diff --git a/block/probe/raw.c b/block/probe/raw.c
index 22c6bcb..9923bb6 100644
--- a/block/probe/raw.c
+++ b/block/probe/raw.c
@@ -1,10 +1,14 @@
 #include "qemu/osdep.h"
 #include "block/probe.h"
 
-int raw_probe(const uint8_t *buf, int buf_size, const char *filename)
+const char *raw_probe(const uint8_t *buf, int buf_size, const char *filename,
+  int *score)
 {
+const char *format = "raw";
+assert(score);
 /* smallest possible positive score so that raw is used if and only if no
  * other block driver works
  */
-return 1;
+*score = 1;
+return format;
 }
diff --git a/block/raw_bsd.c b/block/raw_bsd.c
index 8f49637..4b24a36 100644
--- a/block/raw_bsd.c
+++ b/block/raw_bsd.c
@@ -231,7 +231,6 @@ static int raw_probe_geometry(BlockDriverState *bs, 
HDGeometry *geo)
 
 BlockDriver bdrv_raw = {
 .format_name  = "raw",
-.bdrv_probe   = _probe,
 .bdrv_reopen_prepare  = _reopen_prepare,
 .bdrv_open= _open,
 .bdrv_close   = _close,
diff --git a/include/block/probe.h b/include/block/probe.h
index 63edd74..b49663d 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -1,7 +1,6 @@
 #ifndef PROBE_H
 #define PROBE_H
 
-int raw_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vdi_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vhdx_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vmdk_probe(const uint8_t *buf, int buf_size, const char *filename);
@@ -22,5 +21,7 @@ const char *qcow2_probe(const uint8_t *buf, int buf_size, 
const char *filename,
 int *score);
 const char *bdrv_qed_probe(const uint8_t *buf, int buf_size,
const char *filename, int *score);
+const char *raw_probe(const uint8_t *buf, int buf_size, const char *filename,
+  int *score);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v3 16/32] blockdev: Move vpc probe to its own file

[Colin Lord]

Isolates vpc probe as part of the modularization process.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block/Makefile.objs   | 2 +-
 block/probe/vpc.c | 9 +
 block/vpc.c   | 8 +---
 include/block/probe.h | 1 +
 4 files changed, 12 insertions(+), 8 deletions(-)
 create mode 100644 block/probe/vpc.c

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 4cecf68..2d2691e 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -26,7 +26,7 @@ block-obj-y += write-threshold.o
 block-obj-y += crypto.o
 block-obj-y += probe/bochs.o probe/cloop.o probe/luks.o probe/dmg.o
 block-obj-y += probe/parallels.o probe/qcow.o probe/qcow2.o probe/qed.o
-block-obj-y += probe/raw.o probe/vdi.o probe/vhdx.o probe/vmdk.o
+block-obj-y += probe/raw.o probe/vdi.o probe/vhdx.o probe/vmdk.o probe/vpc.o
 
 common-obj-y += stream.o
 common-obj-y += commit.o
diff --git a/block/probe/vpc.c b/block/probe/vpc.c
new file mode 100644
index 000..afe8271
--- /dev/null
+++ b/block/probe/vpc.c
@@ -0,0 +1,9 @@
+#include "qemu/osdep.h"
+#include "block/probe.h"
+
+int vpc_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+if (buf_size >= 8 && !strncmp((char *)buf, "conectix", 8))
+   return 100;
+return 0;
+}
diff --git a/block/vpc.c b/block/vpc.c
index 076a7ce..cb65022 100644
--- a/block/vpc.c
+++ b/block/vpc.c
@@ -26,6 +26,7 @@
 #include "qapi/error.h"
 #include "qemu-common.h"
 #include "block/block_int.h"
+#include "block/probe.h"
 #include "sysemu/block-backend.h"
 #include "qemu/module.h"
 #include "migration/migration.h"
@@ -179,13 +180,6 @@ static uint32_t vpc_checksum(uint8_t* buf, size_t size)
 }
 
 
-static int vpc_probe(const uint8_t *buf, int buf_size, const char *filename)
-{
-if (buf_size >= 8 && !strncmp((char *)buf, "conectix", 8))
-   return 100;
-return 0;
-}
-
 static void vpc_parse_options(BlockDriverState *bs, QemuOpts *opts,
   Error **errp)
 {
diff --git a/include/block/probe.h b/include/block/probe.h
index 392515d..6cf878b 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -14,5 +14,6 @@ int raw_probe(const uint8_t *buf, int buf_size, const char 
*filename);
 int vdi_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vhdx_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vmdk_probe(const uint8_t *buf, int buf_size, const char *filename);
+int vpc_probe(const uint8_t *buf, int buf_size, const char *filename);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v3 18/32] blockdev: Separate cloop probe from its driver

[Colin Lord]

Completes the separation of the cloop probe from the cloop driver. The
cloop probe now returns the format in addition to the score, allowing
correlation of the score and driver without the probe function being
part of the driver itself.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block.c   |  1 +
 block/cloop.c |  1 -
 block/probe/cloop.c   | 11 ---
 include/block/probe.h |  3 ++-
 4 files changed, 11 insertions(+), 5 deletions(-)

diff --git a/block.c b/block.c
index eab8a6e..baef612 100644
--- a/block.c
+++ b/block.c
@@ -62,6 +62,7 @@ typedef const char *BdrvProbeFunc(const uint8_t *buf, int 
buf_size,
 
 static BdrvProbeFunc *format_probes[] = {
 bochs_probe,
+cloop_probe,
 };
 
 static QTAILQ_HEAD(, BlockDriverState) graph_bdrv_states =
diff --git a/block/cloop.c b/block/cloop.c
index bf9fb75..b5db80b 100644
--- a/block/cloop.c
+++ b/block/cloop.c
@@ -264,7 +264,6 @@ static void cloop_close(BlockDriverState *bs)
 static BlockDriver bdrv_cloop = {
 .format_name= "cloop",
 .instance_size  = sizeof(BDRVCloopState),
-.bdrv_probe = cloop_probe,
 .bdrv_open  = cloop_open,
 .bdrv_co_preadv = cloop_co_preadv,
 .bdrv_close = cloop_close,
diff --git a/block/probe/cloop.c b/block/probe/cloop.c
index 955c29c..b9c2605 100644
--- a/block/probe/cloop.c
+++ b/block/probe/cloop.c
@@ -1,17 +1,22 @@
 #include "qemu/osdep.h"
 #include "block/probe.h"
 
-int cloop_probe(const uint8_t *buf, int buf_size, const char *filename)
+const char *cloop_probe(const uint8_t *buf, int buf_size, const char *filename,
+int *score)
 {
+const char *format = "cloop";
 const char *magic_version_2_0 = "#!/bin/sh\n"
 "#V2.0 Format\n"
 "modprobe cloop file=$0 && mount -r -t iso9660 /dev/cloop $1\n";
 int length = strlen(magic_version_2_0);
+assert(score);
 if (length > buf_size) {
 length = buf_size;
 }
 if (!memcmp(magic_version_2_0, buf, length)) {
-return 2;
+*score = 2;
+return format;
 }
-return 0;
+*score = 0;
+return format;
 }
diff --git a/include/block/probe.h b/include/block/probe.h
index 13c08bd..804f77c 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -1,7 +1,6 @@
 #ifndef PROBE_H
 #define PROBE_H
 
-int cloop_probe(const uint8_t *buf, int buf_size, const char *filename);
 int block_crypto_probe_luks(const uint8_t *buf, int buf_size,
 const char *filename);
 int dmg_probe(const uint8_t *buf, int buf_size, const char *filename);
@@ -16,5 +15,7 @@ int vmdk_probe(const uint8_t *buf, int buf_size, const char 
*filename);
 int vpc_probe(const uint8_t *buf, int buf_size, const char *filename);
 const char *bochs_probe(const uint8_t *buf, int buf_size, const char *filename,
 int *score);
+const char *cloop_probe(const uint8_t *buf, int buf_size, const char *filename,
+int *score);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v3 17/32] blockdev: Separate bochs probe from its driver

[Colin Lord]

Modifies the bochs probe to return the format name as well as the
score as the final step of separating the probe function from the
driver. This keeps the probe completely independent of the driver,
making future modularization easier to accomplish. Returning the format
name as well as the score allows the score to be correlated to the
driver without the probe function needing to be part of the driver.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block.c   | 19 +++
 block/bochs.c |  1 -
 block/probe/bochs.c   | 25 -
 include/block/probe.h |  3 ++-
 4 files changed, 37 insertions(+), 11 deletions(-)

diff --git a/block.c b/block.c
index 88a05b2..eab8a6e 100644
--- a/block.c
+++ b/block.c
@@ -25,6 +25,7 @@
 #include "trace.h"
 #include "block/block_int.h"
 #include "block/blockjob.h"
+#include "block/probe.h"
 #include "qemu/error-report.h"
 #include "module_block.h"
 #include "qemu/module.h"
@@ -56,6 +57,13 @@
 
 #define NOT_DONE 0x7fff /* used while emulated sync operation in progress 
*/
 
+typedef const char *BdrvProbeFunc(const uint8_t *buf, int buf_size,
+  const char *filename, int *score);
+
+static BdrvProbeFunc *format_probes[] = {
+bochs_probe,
+};
+
 static QTAILQ_HEAD(, BlockDriverState) graph_bdrv_states =
 QTAILQ_HEAD_INITIALIZER(graph_bdrv_states);
 
@@ -576,6 +584,8 @@ BlockDriver *bdrv_probe_all(const uint8_t *buf, int 
buf_size,
 const char *filename)
 {
 int score_max = 0, score;
+const char *format_max = NULL;
+const char *format;
 size_t i;
 BlockDriver *drv = NULL, *d;
 
@@ -595,6 +605,15 @@ BlockDriver *bdrv_probe_all(const uint8_t *buf, int 
buf_size,
 }
 }
 
+for (i = 0; i < ARRAY_SIZE(format_probes); i++) {
+format = format_probes[i](buf, buf_size, filename, );
+if (score > score_max) {
+score_max = score;
+format_max = format;
+drv = bdrv_find_format(format_max);
+}
+}
+
 return drv;
 }
 
diff --git a/block/bochs.c b/block/bochs.c
index 11da0fd..5c94bc6 100644
--- a/block/bochs.c
+++ b/block/bochs.c
@@ -230,7 +230,6 @@ static void bochs_close(BlockDriverState *bs)
 static BlockDriver bdrv_bochs = {
 .format_name   = "bochs",
 .instance_size = sizeof(BDRVBochsState),
-.bdrv_probe= bochs_probe,
 .bdrv_open = bochs_open,
 .bdrv_co_preadv = bochs_co_preadv,
 .bdrv_close= bochs_close,
diff --git a/block/probe/bochs.c b/block/probe/bochs.c
index 8adc09f..8206930 100644
--- a/block/probe/bochs.c
+++ b/block/probe/bochs.c
@@ -3,19 +3,26 @@
 #include "block/probe.h"
 #include "block/driver/bochs.h"
 
-int bochs_probe(const uint8_t *buf, int buf_size, const char *filename)
+const char *bochs_probe(const uint8_t *buf, int buf_size, const char *filename,
+int *score)
 {
+const char *format = "bochs";
 const struct bochs_header *bochs = (const void *)buf;
+assert(score);
+*score = 0;
 
-if (buf_size < HEADER_SIZE)
-   return 0;
+if (buf_size < HEADER_SIZE) {
+return format;
+}
 
 if (!strcmp(bochs->magic, HEADER_MAGIC) &&
-   !strcmp(bochs->type, REDOLOG_TYPE) &&
-   !strcmp(bochs->subtype, GROWING_TYPE) &&
-   ((le32_to_cpu(bochs->version) == HEADER_VERSION) ||
-   (le32_to_cpu(bochs->version) == HEADER_V1)))
-   return 100;
+!strcmp(bochs->type, REDOLOG_TYPE) &&
+!strcmp(bochs->subtype, GROWING_TYPE) &&
+((le32_to_cpu(bochs->version) == HEADER_VERSION) ||
+(le32_to_cpu(bochs->version) == HEADER_V1))) {
+*score = 100;
+return format;
+}
 
-return 0;
+return format;
 }
diff --git a/include/block/probe.h b/include/block/probe.h
index 6cf878b..13c08bd 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -1,7 +1,6 @@
 #ifndef PROBE_H
 #define PROBE_H
 
-int bochs_probe(const uint8_t *buf, int buf_size, const char *filename);
 int cloop_probe(const uint8_t *buf, int buf_size, const char *filename);
 int block_crypto_probe_luks(const uint8_t *buf, int buf_size,
 const char *filename);
@@ -15,5 +14,7 @@ int vdi_probe(const uint8_t *buf, int buf_size, const char 
*filename);
 int vhdx_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vmdk_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vpc_probe(const uint8_t *buf, int buf_size, const char *filename);
+const char *bochs_probe(const uint8_t *buf, int buf_size, const char *filename,
+int *score);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v3 04/32] blockdev: Move bochs probe into separate file

[Colin Lord]

This puts the bochs probe function into its own separate file as part of
the process of modularizing block drivers. Having the probe functions
separate from the rest of the driver allows us to probe without having
to potentially unnecessarily load the driver.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block/Makefile.objs  |  1 +
 block/bochs.c| 55 ++--
 block/probe/bochs.c  | 21 +
 include/block/driver/bochs.h | 40 
 include/block/probe.h|  6 +
 5 files changed, 70 insertions(+), 53 deletions(-)
 create mode 100644 block/probe/bochs.c
 create mode 100644 include/block/driver/bochs.h
 create mode 100644 include/block/probe.h

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 44a5416..bc0c2aa 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -24,6 +24,7 @@ block-obj-y += accounting.o dirty-bitmap.o
 block-obj-y += write-threshold.o
 
 block-obj-y += crypto.o
+block-obj-y += probe/bochs.o
 
 common-obj-y += stream.o
 common-obj-y += commit.o
diff --git a/block/bochs.c b/block/bochs.c
index 6c8d0f3..11da0fd 100644
--- a/block/bochs.c
+++ b/block/bochs.c
@@ -28,45 +28,11 @@
 #include "block/block_int.h"
 #include "qemu/module.h"
 #include "qemu/bswap.h"
+#include "block/driver/bochs.h"
+#include "block/probe.h"
 
 /**/
 
-#define HEADER_MAGIC "Bochs Virtual HD Image"
-#define HEADER_VERSION 0x0002
-#define HEADER_V1 0x0001
-#define HEADER_SIZE 512
-
-#define REDOLOG_TYPE "Redolog"
-#define GROWING_TYPE "Growing"
-
-// not allocated: 0x
-
-// always little-endian
-struct bochs_header {
-char magic[32]; /* "Bochs Virtual HD Image" */
-char type[16];  /* "Redolog" */
-char subtype[16];   /* "Undoable" / "Volatile" / "Growing" */
-uint32_t version;
-uint32_t header;/* size of header */
-
-uint32_t catalog;   /* num of entries */
-uint32_t bitmap;/* bitmap size */
-uint32_t extent;/* extent size */
-
-union {
-struct {
-uint32_t reserved;  /* for ??? */
-uint64_t disk;  /* disk size */
-char padding[HEADER_SIZE - 64 - 20 - 12];
-} QEMU_PACKED redolog;
-struct {
-uint64_t disk;  /* disk size */
-char padding[HEADER_SIZE - 64 - 20 - 8];
-} QEMU_PACKED redolog_v1;
-char padding[HEADER_SIZE - 64 - 20];
-} extra;
-} QEMU_PACKED;
-
 typedef struct BDRVBochsState {
 CoMutex lock;
 uint32_t *catalog_bitmap;
@@ -79,23 +45,6 @@ typedef struct BDRVBochsState {
 uint32_t extent_size;
 } BDRVBochsState;
 
-static int bochs_probe(const uint8_t *buf, int buf_size, const char *filename)
-{
-const struct bochs_header *bochs = (const void *)buf;
-
-if (buf_size < HEADER_SIZE)
-   return 0;
-
-if (!strcmp(bochs->magic, HEADER_MAGIC) &&
-   !strcmp(bochs->type, REDOLOG_TYPE) &&
-   !strcmp(bochs->subtype, GROWING_TYPE) &&
-   ((le32_to_cpu(bochs->version) == HEADER_VERSION) ||
-   (le32_to_cpu(bochs->version) == HEADER_V1)))
-   return 100;
-
-return 0;
-}
-
 static int bochs_open(BlockDriverState *bs, QDict *options, int flags,
   Error **errp)
 {
diff --git a/block/probe/bochs.c b/block/probe/bochs.c
new file mode 100644
index 000..8adc09f
--- /dev/null
+++ b/block/probe/bochs.c
@@ -0,0 +1,21 @@
+#include "qemu/osdep.h"
+#include "block/block_int.h"
+#include "block/probe.h"
+#include "block/driver/bochs.h"
+
+int bochs_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+const struct bochs_header *bochs = (const void *)buf;
+
+if (buf_size < HEADER_SIZE)
+   return 0;
+
+if (!strcmp(bochs->magic, HEADER_MAGIC) &&
+   !strcmp(bochs->type, REDOLOG_TYPE) &&
+   !strcmp(bochs->subtype, GROWING_TYPE) &&
+   ((le32_to_cpu(bochs->version) == HEADER_VERSION) ||
+   (le32_to_cpu(bochs->version) == HEADER_V1)))
+   return 100;
+
+return 0;
+}
diff --git a/include/block/driver/bochs.h b/include/block/driver/bochs.h
new file mode 100644
index 000..cd87256
--- /dev/null
+++ b/include/block/driver/bochs.h
@@ -0,0 +1,40 @@
+#ifndef BOCHS_H
+#define BOCHS_H
+
+#define HEADER_MAGIC "Bochs Virtual HD Image"
+#define HEADER_VERSION 0x0002
+#define HEADER_V1 0x0001
+#define HEADER_SIZE 512
+
+#define REDOLOG_TYPE "Redolog"
+#define GROWING_TYPE "Growing"
+
+// not allocated: 0x
+
+// always little-endian
+struct bochs_header {
+char magic[32]; /* "Bochs Virtual HD Image" */
+char type[16];

[Qemu-block] [PATCH v3 12/32] blockdev: Move raw probe to its own file

[Colin Lord]

Isolate raw probe as part of the modularization process.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block/Makefile.objs   |  1 +
 block/probe/raw.c | 10 ++
 block/raw_bsd.c   |  9 +
 include/block/probe.h |  1 +
 4 files changed, 13 insertions(+), 8 deletions(-)
 create mode 100644 block/probe/raw.c

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 6ed2689..693cc97 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -26,6 +26,7 @@ block-obj-y += write-threshold.o
 block-obj-y += crypto.o
 block-obj-y += probe/bochs.o probe/cloop.o probe/luks.o probe/dmg.o
 block-obj-y += probe/parallels.o probe/qcow.o probe/qcow2.o probe/qed.o
+block-obj-y += probe/raw.o
 
 common-obj-y += stream.o
 common-obj-y += commit.o
diff --git a/block/probe/raw.c b/block/probe/raw.c
new file mode 100644
index 000..22c6bcb
--- /dev/null
+++ b/block/probe/raw.c
@@ -0,0 +1,10 @@
+#include "qemu/osdep.h"
+#include "block/probe.h"
+
+int raw_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+/* smallest possible positive score so that raw is used if and only if no
+ * other block driver works
+ */
+return 1;
+}
diff --git a/block/raw_bsd.c b/block/raw_bsd.c
index 7f63791..8f49637 100644
--- a/block/raw_bsd.c
+++ b/block/raw_bsd.c
@@ -28,6 +28,7 @@
 
 #include "qemu/osdep.h"
 #include "block/block_int.h"
+#include "block/probe.h"
 #include "qapi/error.h"
 #include "qemu/option.h"
 
@@ -218,14 +219,6 @@ static void raw_close(BlockDriverState *bs)
 {
 }
 
-static int raw_probe(const uint8_t *buf, int buf_size, const char *filename)
-{
-/* smallest possible positive score so that raw is used if and only if no
- * other block driver works
- */
-return 1;
-}
-
 static int raw_probe_blocksizes(BlockDriverState *bs, BlockSizes *bsz)
 {
 return bdrv_probe_blocksizes(bs->file->bs, bsz);
diff --git a/include/block/probe.h b/include/block/probe.h
index e3e9934..053f961 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -10,5 +10,6 @@ int parallels_probe(const uint8_t *buf, int buf_size, const 
char *filename);
 int qcow_probe(const uint8_t *buf, int buf_size, const char *filename);
 int qcow2_probe(const uint8_t *buf, int buf_size, const char *filename);
 int bdrv_qed_probe(const uint8_t *buf, int buf_size, const char *filename);
+int raw_probe(const uint8_t *buf, int buf_size, const char *filename);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v3 19/32] blockdev: Separate luks probe from its driver

[Colin Lord]

Completes the separation of the luks probe from the crypto driver. The
luks probe now returns the format in addition to the score, allowing
correlation of the score and driver without the probe function being
part of the driver itself.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block.c   |  1 +
 block/crypto.c|  1 -
 block/probe/luks.c| 13 -
 include/block/probe.h |  4 ++--
 4 files changed, 11 insertions(+), 8 deletions(-)

diff --git a/block.c b/block.c
index baef612..d936da1 100644
--- a/block.c
+++ b/block.c
@@ -63,6 +63,7 @@ typedef const char *BdrvProbeFunc(const uint8_t *buf, int 
buf_size,
 static BdrvProbeFunc *format_probes[] = {
 bochs_probe,
 cloop_probe,
+block_crypto_probe_luks,
 };
 
 static QTAILQ_HEAD(, BlockDriverState) graph_bdrv_states =
diff --git a/block/crypto.c b/block/crypto.c
index 493dd69..6f37aec 100644
--- a/block/crypto.c
+++ b/block/crypto.c
@@ -549,7 +549,6 @@ static int block_crypto_create_luks(const char *filename,
 BlockDriver bdrv_crypto_luks = {
 .format_name= "luks",
 .instance_size  = sizeof(BlockCrypto),
-.bdrv_probe = block_crypto_probe_luks,
 .bdrv_open  = block_crypto_open_luks,
 .bdrv_close = block_crypto_close,
 .bdrv_create= block_crypto_create_luks,
diff --git a/block/probe/luks.c b/block/probe/luks.c
index 5c6427a..4c58586 100644
--- a/block/probe/luks.c
+++ b/block/probe/luks.c
@@ -15,9 +15,12 @@ static int block_crypto_probe_generic(QCryptoBlockFormat 
format,
 }
 }
 
-int block_crypto_probe_luks(const uint8_t *buf,
-   int buf_size,
-   const char *filename) {
-return block_crypto_probe_generic(Q_CRYPTO_BLOCK_FORMAT_LUKS,
-  buf, buf_size, filename);
+const char *block_crypto_probe_luks(const uint8_t *buf, int buf_size,
+const char *filename, int *score)
+{
+const char *format = "luks";
+assert(score);
+*score = block_crypto_probe_generic(Q_CRYPTO_BLOCK_FORMAT_LUKS,
+buf, buf_size, filename);
+return format;
 }
diff --git a/include/block/probe.h b/include/block/probe.h
index 804f77c..e3bf04e 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -1,8 +1,6 @@
 #ifndef PROBE_H
 #define PROBE_H
 
-int block_crypto_probe_luks(const uint8_t *buf, int buf_size,
-const char *filename);
 int dmg_probe(const uint8_t *buf, int buf_size, const char *filename);
 int parallels_probe(const uint8_t *buf, int buf_size, const char *filename);
 int qcow_probe(const uint8_t *buf, int buf_size, const char *filename);
@@ -17,5 +15,7 @@ const char *bochs_probe(const uint8_t *buf, int buf_size, 
const char *filename,
 int *score);
 const char *cloop_probe(const uint8_t *buf, int buf_size, const char *filename,
 int *score);
+const char *block_crypto_probe_luks(const uint8_t *buf, int buf_size,
+const char *filename, int *score);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v3 14/32] blockdev: Move vhdx probe to its own file

[Colin Lord]

Isolates vhdx probe as part of the modularization process.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block/Makefile.objs   |  2 +-
 block/probe/vhdx.c| 21 +
 block/vhdx.c  | 20 +---
 include/block/probe.h |  1 +
 4 files changed, 24 insertions(+), 20 deletions(-)
 create mode 100644 block/probe/vhdx.c

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 4a5bd88..89e0da4 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -26,7 +26,7 @@ block-obj-y += write-threshold.o
 block-obj-y += crypto.o
 block-obj-y += probe/bochs.o probe/cloop.o probe/luks.o probe/dmg.o
 block-obj-y += probe/parallels.o probe/qcow.o probe/qcow2.o probe/qed.o
-block-obj-y += probe/raw.o probe/vdi.o
+block-obj-y += probe/raw.o probe/vdi.o probe/vhdx.o
 
 common-obj-y += stream.o
 common-obj-y += commit.o
diff --git a/block/probe/vhdx.c b/block/probe/vhdx.c
new file mode 100644
index 000..6c38aac
--- /dev/null
+++ b/block/probe/vhdx.c
@@ -0,0 +1,21 @@
+#include "qemu/osdep.h"
+#include "block/probe.h"
+
+/*
+ * Per the MS VHDX Specification, for every VHDX file:
+ *  - The header section is fixed size - 1 MB
+ *  - The header section is always the first "object"
+ *  - The first 64KB of the header is the File Identifier
+ *  - The first uint64 (8 bytes) is the VHDX Signature ("vhdxfile")
+ *  - The following 512 bytes constitute a UTF-16 string identifiying the
+ *software that created the file, and is optional and diagnostic only.
+ *
+ *  Therefore, we probe by looking for the vhdxfile signature "vhdxfile"
+ */
+int vhdx_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+if (buf_size >= 8 && !memcmp(buf, "vhdxfile", 8)) {
+return 100;
+}
+return 0;
+}
diff --git a/block/vhdx.c b/block/vhdx.c
index f5605a2..ba8adfe 100644
--- a/block/vhdx.c
+++ b/block/vhdx.c
@@ -19,6 +19,7 @@
 #include "qapi/error.h"
 #include "qemu-common.h"
 #include "block/block_int.h"
+#include "block/probe.h"
 #include "sysemu/block-backend.h"
 #include "qemu/module.h"
 #include "qemu/crc32c.h"
@@ -273,25 +274,6 @@ static void vhdx_set_shift_bits(BDRVVHDXState *s)
 }
 
 /*
- * Per the MS VHDX Specification, for every VHDX file:
- *  - The header section is fixed size - 1 MB
- *  - The header section is always the first "object"
- *  - The first 64KB of the header is the File Identifier
- *  - The first uint64 (8 bytes) is the VHDX Signature ("vhdxfile")
- *  - The following 512 bytes constitute a UTF-16 string identifiying the
- *software that created the file, and is optional and diagnostic only.
- *
- *  Therefore, we probe by looking for the vhdxfile signature "vhdxfile"
- */
-static int vhdx_probe(const uint8_t *buf, int buf_size, const char *filename)
-{
-if (buf_size >= 8 && !memcmp(buf, "vhdxfile", 8)) {
-return 100;
-}
-return 0;
-}
-
-/*
  * Writes the header to the specified offset.
  *
  * This will optionally read in buffer data from disk (otherwise zero-fill),
diff --git a/include/block/probe.h b/include/block/probe.h
index f85c178..e901d8f 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -12,5 +12,6 @@ int qcow2_probe(const uint8_t *buf, int buf_size, const char 
*filename);
 int bdrv_qed_probe(const uint8_t *buf, int buf_size, const char *filename);
 int raw_probe(const uint8_t *buf, int buf_size, const char *filename);
 int vdi_probe(const uint8_t *buf, int buf_size, const char *filename);
+int vhdx_probe(const uint8_t *buf, int buf_size, const char *filename);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v3 09/32] blockdev: Move qcow probe to its own file

[Colin Lord]

Isolates qcow probe as part of the modularization process.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block/Makefile.objs |  2 +-
 block/probe/qcow.c  | 16 
 block/qcow.c| 31 ++-
 include/block/driver/qcow.h | 21 +
 include/block/probe.h   |  1 +
 5 files changed, 41 insertions(+), 30 deletions(-)
 create mode 100644 block/probe/qcow.c
 create mode 100644 include/block/driver/qcow.h

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 12607db..9458da8 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -25,7 +25,7 @@ block-obj-y += write-threshold.o
 
 block-obj-y += crypto.o
 block-obj-y += probe/bochs.o probe/cloop.o probe/luks.o probe/dmg.o
-block-obj-y += probe/parallels.o
+block-obj-y += probe/parallels.o probe/qcow.o
 
 common-obj-y += stream.o
 common-obj-y += commit.o
diff --git a/block/probe/qcow.c b/block/probe/qcow.c
new file mode 100644
index 000..6024d11
--- /dev/null
+++ b/block/probe/qcow.c
@@ -0,0 +1,16 @@
+#include "qemu/osdep.h"
+#include "block/block_int.h"
+#include "block/probe.h"
+#include "block/driver/qcow.h"
+
+int qcow_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+const QCowHeader *cow_header = (const void *)buf;
+
+if (buf_size >= sizeof(QCowHeader) &&
+be32_to_cpu(cow_header->magic) == QCOW_MAGIC &&
+be32_to_cpu(cow_header->version) == QCOW_VERSION)
+return 100;
+else
+return 0;
+}
diff --git a/block/qcow.c b/block/qcow.c
index 312af52..828749c 100644
--- a/block/qcow.c
+++ b/block/qcow.c
@@ -26,6 +26,8 @@
 #include "qemu-common.h"
 #include "qemu/error-report.h"
 #include "block/block_int.h"
+#include "block/probe.h"
+#include "block/driver/qcow.h"
 #include "sysemu/block-backend.h"
 #include "qemu/module.h"
 #include "qemu/bswap.h"
@@ -37,28 +39,11 @@
 /**/
 /* QEMU COW block driver with compression and encryption support */
 
-#define QCOW_MAGIC (('Q' << 24) | ('F' << 16) | ('I' << 8) | 0xfb)
-#define QCOW_VERSION 1
-
 #define QCOW_CRYPT_NONE 0
 #define QCOW_CRYPT_AES  1
 
 #define QCOW_OFLAG_COMPRESSED (1LL << 63)
 
-typedef struct QCowHeader {
-uint32_t magic;
-uint32_t version;
-uint64_t backing_file_offset;
-uint32_t backing_file_size;
-uint32_t mtime;
-uint64_t size; /* in bytes */
-uint8_t cluster_bits;
-uint8_t l2_bits;
-uint16_t padding;
-uint32_t crypt_method;
-uint64_t l1_table_offset;
-} QEMU_PACKED QCowHeader;
-
 #define L2_CACHE_SIZE 16
 
 typedef struct BDRVQcowState {
@@ -85,18 +70,6 @@ typedef struct BDRVQcowState {
 
 static int decompress_cluster(BlockDriverState *bs, uint64_t cluster_offset);
 
-static int qcow_probe(const uint8_t *buf, int buf_size, const char *filename)
-{
-const QCowHeader *cow_header = (const void *)buf;
-
-if (buf_size >= sizeof(QCowHeader) &&
-be32_to_cpu(cow_header->magic) == QCOW_MAGIC &&
-be32_to_cpu(cow_header->version) == QCOW_VERSION)
-return 100;
-else
-return 0;
-}
-
 static int qcow_open(BlockDriverState *bs, QDict *options, int flags,
  Error **errp)
 {
diff --git a/include/block/driver/qcow.h b/include/block/driver/qcow.h
new file mode 100644
index 000..c96ea24
--- /dev/null
+++ b/include/block/driver/qcow.h
@@ -0,0 +1,21 @@
+#ifndef QCOW_H
+#define QCOW_H
+
+#define QCOW_MAGIC (('Q' << 24) | ('F' << 16) | ('I' << 8) | 0xfb)
+#define QCOW_VERSION 1
+
+typedef struct QCowHeader {
+uint32_t magic;
+uint32_t version;
+uint64_t backing_file_offset;
+uint32_t backing_file_size;
+uint32_t mtime;
+uint64_t size; /* in bytes */
+uint8_t cluster_bits;
+uint8_t l2_bits;
+uint16_t padding;
+uint32_t crypt_method;
+uint64_t l1_table_offset;
+} QEMU_PACKED QCowHeader;
+
+#endif
diff --git a/include/block/probe.h b/include/block/probe.h
index f8b0984..5230da4 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -7,5 +7,6 @@ int block_crypto_probe_luks(const uint8_t *buf, int buf_size,
 const char *filename);
 int dmg_probe(const uint8_t *buf, int buf_size, const char *filename);
 int parallels_probe(const uint8_t *buf, int buf_size, const char *filename);
+int qcow_probe(const uint8_t *buf, int buf_size, const char *filename);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v3 00/32] Dynamic module loading for block drivers

[Colin Lord]

This is the next version of this patch series. The first three patches
in the series are mostly the same as they were last time, but with the
issues mentioned in the reviews fixed. Most notably this means much less
copy-paste happening in block.c.

The new changes, and the reason this series is so large, is a result of
isolating the block driver probe functions from the rest of the drivers.
This was mentioned in the reviews from the last version as something
that should be added. This allows the probes to be accessed without
needing to load all the modules. There are quite a few probes, which is
why there are so many patches in this series. I tried to keep the
patches where I was relocating the code separate from the patches where
the probe function signatures were changed.

The probe functions now return both a score and a format/protocol name,
which allows them to be completely separate from the drivers they
correspond to. I didn't put all the probe functions in the same file as
Fam suggested because it turns out there are some naming conflicts
between some macros commonly used in the probe functions. For instance,
QCowHeader is used in both qcow and qcow2 probes, but the meaning is
different between them. Having separate files makes resolving these
conflicts simple.

Also, just as a point of interest, I've done some basic profiling of my
own to see what performance benefits there are to be made. I recorded
the time to main of each option that you can enable/disable in the
configuration by enabling each option by itself while explicitly
disabling all other options. With each configuration I ran qemu 5 times
while recording the time to main. All the results are in milliseconds.

minimal config  2.772
vhdx2.776
attr2.786
coroutine-pool  2.836
bzip2.856
linux-aio   2.867
vnc-jpeg2.872
snappy  2.875
fdt 2.888
brlapi  2.916
vhost-net   2.922
bluez   2.937
cap-ng  2.940
libiscsi2.945
nettle  2.945
seccomp 2.973
numa2.983
lzo 2.994
usb-redir   3.005
cocoa   3.007
libnfs  3.026
vnc-sasl3.071
vnc-png 3.081
virtfs  3.081
uuid3.086
tpm 3.136
curses  3.145
gcrypt  3.190
smartcar3.242
rdma3.570
jemalloc3.584
glusterfs   3.597
sdl 3.677
libusb  3.758
libssh2 3.906
tcmalloc3.959
xen 4.151
xen-pci-passthrough 4.167
curl5.245
spice   5.289
gnutls  5.400
gtk 8.782
vte 9.764
rbd 44.315

Colin Lord (30):
  blockdev: prepare iSCSI block driver for dynamic loading
  blockdev: Move bochs probe into separate file
  blockdev: Move cloop probe to its own file
  blockdev: Move luks probe to its own file
  blockdev: Move dmg probe to its own file
  blockdev: Move parallels probe to its own file
  blockdev: Move qcow probe to its own file
  blockdev: Move qcow2 probe to its own file
  blockdev: Move qed probe to its own file
  blockdev: Move raw probe to its own file
  blockdev: Move vdi probe to its own file
  blockdev: Move vhdx probe to its own file
  blockdev: Move vmdk probe to its own file
  blockdev: Move vpc probe to its own file
  blockdev: Separate bochs probe from its driver
  blockdev: Separate cloop probe from its driver
  blockdev: Separate luks probe from its driver
  blockdev: Separate dmg probe from its driver
  blockdev: Separate parallels probe from its driver
  blockdev: Separate qcow probe from its driver
  blockdev: Separate qcow2 probe from its driver
  blockdev: Separate qed probe from its driver
  blockdev: Separate raw probe from its driver
  blockdev: Separate vdi probe from its driver
  blockdev: Separate vhdx probe from its driver
  blockdev: Separate vmdk probe from its driver
  blockdev: Separate vpc probe from its driver
  blockdev: Remove the .bdrv_probe field from BlockDrivers
  blockdev: Separate out bdrv_probe_device functions
  blockdev: Remove bdrv_probe_device field from BlockDriver

Marc Mari (2):
  blockdev: Add dynamic generation of module_block.h
  blockdev: Add dynamic module loading for block drivers

 Makefile |   7 ++
 block.c  | 181 +++
 block/Makefile.objs  |   4 +
 block/bochs.c|  56 +---
 block/cloop.c|  17 +---
 block/crypto.c   |  22 +
 block/dmg.c  |  17 +---
 block/iscsi.c|  36 
 block/parallels.c|  44 +-
 block/probe/bochs.c  |  28 ++
 block/probe/cloop.c  |  22 +
 block/probe

[Qemu-block] [PATCH v3 03/32] blockdev: Add dynamic module loading for block drivers

[Colin Lord]

From: Marc Mari <mar...@redhat.com>

Extend the current module interface to allow for block drivers to be loaded
dynamically on request.

The only block drivers that can be converted into modules are the drivers
that don't perform any init operation except for registering themselves.

All the necessary module information is located in a new structure found in
module_block.h

Signed-off-by: Marc Marí <mar...@redhat.com>
Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block.c   | 110 ++
 include/qemu/module.h |   3 ++
 util/module.c |  38 +
 3 files changed, 108 insertions(+), 43 deletions(-)

diff --git a/block.c b/block.c
index f4648e9..88a05b2 100644
--- a/block.c
+++ b/block.c
@@ -26,6 +26,7 @@
 #include "block/block_int.h"
 #include "block/blockjob.h"
 #include "qemu/error-report.h"
+#include "module_block.h"
 #include "qemu/module.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/qmp/qbool.h"
@@ -239,17 +240,40 @@ BlockDriverState *bdrv_new(void)
 return bs;
 }
 
-BlockDriver *bdrv_find_format(const char *format_name)
+static BlockDriver *bdrv_do_find_format(const char *format_name)
 {
 BlockDriver *drv1;
+
 QLIST_FOREACH(drv1, _drivers, list) {
 if (!strcmp(drv1->format_name, format_name)) {
 return drv1;
 }
 }
+
 return NULL;
 }
 
+BlockDriver *bdrv_find_format(const char *format_name)
+{
+BlockDriver *drv1;
+size_t i;
+
+drv1 = bdrv_do_find_format(format_name);
+if (drv1) {
+return drv1;
+}
+
+/* The driver isn't registered, maybe we need to load a module */
+for (i = 0; i < ARRAY_SIZE(block_driver_modules); ++i) {
+if (!strcmp(block_driver_modules[i].format_name, format_name)) {
+block_module_load_one(block_driver_modules[i].library_name);
+break;
+}
+}
+
+return bdrv_do_find_format(format_name);
+}
+
 static int bdrv_is_whitelisted(BlockDriver *drv, bool read_only)
 {
 static const char *whitelist_rw[] = {
@@ -443,8 +467,15 @@ int get_tmp_filename(char *filename, int size)
 static BlockDriver *find_hdev_driver(const char *filename)
 {
 int score_max = 0, score;
+size_t i;
 BlockDriver *drv = NULL, *d;
 
+for (i = 0; i < ARRAY_SIZE(block_driver_modules); ++i) {
+if (block_driver_modules[i].has_probe_device) {
+block_module_load_one(block_driver_modules[i].library_name);
+}
+}
+
 QLIST_FOREACH(d, _drivers, list) {
 if (d->bdrv_probe_device) {
 score = d->bdrv_probe_device(filename);
@@ -458,6 +489,19 @@ static BlockDriver *find_hdev_driver(const char *filename)
 return drv;
 }
 
+static BlockDriver *bdrv_do_find_protocol(const char *protocol)
+{
+BlockDriver *drv1;
+
+QLIST_FOREACH(drv1, _drivers, list) {
+if (drv1->protocol_name && !strcmp(drv1->protocol_name, protocol)) {
+return drv1;
+}
+}
+
+return NULL;
+}
+
 BlockDriver *bdrv_find_protocol(const char *filename,
 bool allow_protocol_prefix,
 Error **errp)
@@ -466,6 +510,7 @@ BlockDriver *bdrv_find_protocol(const char *filename,
 char protocol[128];
 int len;
 const char *p;
+size_t i;
 
 /* TODO Drivers without bdrv_file_open must be specified explicitly */
 
@@ -492,15 +537,25 @@ BlockDriver *bdrv_find_protocol(const char *filename,
 len = sizeof(protocol) - 1;
 memcpy(protocol, filename, len);
 protocol[len] = '\0';
-QLIST_FOREACH(drv1, _drivers, list) {
-if (drv1->protocol_name &&
-!strcmp(drv1->protocol_name, protocol)) {
-return drv1;
+
+drv1 = bdrv_do_find_protocol(protocol);
+if (drv1) {
+return drv1;
+}
+
+for (i = 0; i < ARRAY_SIZE(block_driver_modules); ++i) {
+if (block_driver_modules[i].protocol_name &&
+!strcmp(block_driver_modules[i].protocol_name, protocol)) {
+block_module_load_one(block_driver_modules[i].library_name);
+break;
 }
 }
 
-error_setg(errp, "Unknown protocol '%s'", protocol);
-return NULL;
+drv1 = bdrv_do_find_protocol(protocol);
+if (!drv1) {
+error_setg(errp, "Unknown protocol '%s'", protocol);
+}
+return drv1;
 }
 
 /*
@@ -521,8 +576,15 @@ BlockDriver *bdrv_probe_all(const uint8_t *buf, int 
buf_size,
 const char *filename)
 {
 int score_max = 0, score;
+size_t i;
 BlockDriver *drv = NULL, *d;
 
+for (i = 0; i < ARRAY_SIZE(block_driver_modules); ++i) {
+if (block_driver_modules[i].has_probe) {
+block_module_load_one(block_driver_modules[i].library_name);
+}
+}
+
 QLIST_FOREACH(d, _drivers, li

[Qemu-block] [PATCH v3 08/32] blockdev: Move parallels probe to its own file

[Colin Lord]

Isolate parallels probe as part of the modularization process.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block/Makefile.objs  |  1 +
 block/parallels.c| 43 ++--
 block/probe/parallels.c  | 22 
 include/block/driver/parallels.h | 26 
 include/block/probe.h|  1 +
 5 files changed, 52 insertions(+), 41 deletions(-)
 create mode 100644 block/probe/parallels.c
 create mode 100644 include/block/driver/parallels.h

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 1d744eb..12607db 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -25,6 +25,7 @@ block-obj-y += write-threshold.o
 
 block-obj-y += crypto.o
 block-obj-y += probe/bochs.o probe/cloop.o probe/luks.o probe/dmg.o
+block-obj-y += probe/parallels.o
 
 common-obj-y += stream.o
 common-obj-y += commit.o
diff --git a/block/parallels.c b/block/parallels.c
index d6a1a61..547373e 100644
--- a/block/parallels.c
+++ b/block/parallels.c
@@ -31,6 +31,8 @@
 #include "qapi/error.h"
 #include "qemu-common.h"
 #include "block/block_int.h"
+#include "block/probe.h"
+#include "block/driver/parallels.h"
 #include "sysemu/block-backend.h"
 #include "qemu/module.h"
 #include "qemu/bswap.h"
@@ -39,29 +41,6 @@
 
 /**/
 
-#define HEADER_MAGIC "WithoutFreeSpace"
-#define HEADER_MAGIC2 "WithouFreSpacExt"
-#define HEADER_VERSION 2
-#define HEADER_INUSE_MAGIC  (0x746F6E59)
-
-#define DEFAULT_CLUSTER_SIZE 1048576/* 1 MiB */
-
-
-// always little-endian
-typedef struct ParallelsHeader {
-char magic[16]; // "WithoutFreeSpace"
-uint32_t version;
-uint32_t heads;
-uint32_t cylinders;
-uint32_t tracks;
-uint32_t bat_entries;
-uint64_t nb_sectors;
-uint32_t inuse;
-uint32_t data_off;
-char padding[12];
-} QEMU_PACKED ParallelsHeader;
-
-
 typedef enum ParallelsPreallocMode {
 PRL_PREALLOC_MODE_FALLOCATE = 0,
 PRL_PREALLOC_MODE_TRUNCATE = 1,
@@ -536,24 +515,6 @@ exit:
 }
 
 
-static int parallels_probe(const uint8_t *buf, int buf_size,
-   const char *filename)
-{
-const ParallelsHeader *ph = (const void *)buf;
-
-if (buf_size < sizeof(ParallelsHeader)) {
-return 0;
-}
-
-if ((!memcmp(ph->magic, HEADER_MAGIC, 16) ||
-   !memcmp(ph->magic, HEADER_MAGIC2, 16)) &&
-   (le32_to_cpu(ph->version) == HEADER_VERSION)) {
-return 100;
-}
-
-return 0;
-}
-
 static int parallels_update_header(BlockDriverState *bs)
 {
 BDRVParallelsState *s = bs->opaque;
diff --git a/block/probe/parallels.c b/block/probe/parallels.c
new file mode 100644
index 000..66cddea
--- /dev/null
+++ b/block/probe/parallels.c
@@ -0,0 +1,22 @@
+#include "qemu/osdep.h"
+#include "block/block_int.h"
+#include "block/probe.h"
+#include "block/driver/parallels.h"
+
+int parallels_probe(const uint8_t *buf, int buf_size,
+   const char *filename)
+{
+const ParallelsHeader *ph = (const void *)buf;
+
+if (buf_size < sizeof(ParallelsHeader)) {
+return 0;
+}
+
+if ((!memcmp(ph->magic, HEADER_MAGIC, 16) ||
+   !memcmp(ph->magic, HEADER_MAGIC2, 16)) &&
+   (le32_to_cpu(ph->version) == HEADER_VERSION)) {
+return 100;
+}
+
+return 0;
+}
diff --git a/include/block/driver/parallels.h b/include/block/driver/parallels.h
new file mode 100644
index 000..512ef5f
--- /dev/null
+++ b/include/block/driver/parallels.h
@@ -0,0 +1,26 @@
+#ifndef PARALLELS_H
+#define PARALLELS_H
+
+#define HEADER_MAGIC "WithoutFreeSpace"
+#define HEADER_MAGIC2 "WithouFreSpacExt"
+#define HEADER_VERSION 2
+#define HEADER_INUSE_MAGIC  (0x746F6E59)
+
+#define DEFAULT_CLUSTER_SIZE 1048576/* 1 MiB */
+
+
+// always little-endian
+typedef struct ParallelsHeader {
+char magic[16]; // "WithoutFreeSpace"
+uint32_t version;
+uint32_t heads;
+uint32_t cylinders;
+uint32_t tracks;
+uint32_t bat_entries;
+uint64_t nb_sectors;
+uint32_t inuse;
+uint32_t data_off;
+char padding[12];
+} QEMU_PACKED ParallelsHeader;
+
+#endif
diff --git a/include/block/probe.h b/include/block/probe.h
index 267431d..f8b0984 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -6,5 +6,6 @@ int cloop_probe(const uint8_t *buf, int buf_size, const char 
*filename);
 int block_crypto_probe_luks(const uint8_t *buf, int buf_size,
 const char *filename);
 int dmg_probe(const uint8_t *buf, int buf_size, const char *filename);
+int parallels_probe(const uint8_t *buf, int buf_size, const char *filename);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v3 02/32] blockdev: Add dynamic generation of module_block.h

[Colin Lord]

From: Marc Mari <mar...@redhat.com>

To simplify the addition of new block modules, add a script that generates
module_block.h automatically from the modules' source code.

This script assumes that the QEMU coding style rules are followed.

Signed-off-by: Marc Marí <mar...@redhat.com>
Signed-off-by: Colin Lord <cl...@redhat.com>
---
 Makefile|   7 +++
 scripts/modules/module_block.py | 122 
 2 files changed, 129 insertions(+)
 create mode 100644 scripts/modules/module_block.py

diff --git a/Makefile b/Makefile
index c054bc6..29035ac 100644
--- a/Makefile
+++ b/Makefile
@@ -76,6 +76,8 @@ GENERATED_HEADERS += trace/generated-ust-provider.h
 GENERATED_SOURCES += trace/generated-ust.c
 endif
 
+GENERATED_HEADERS += module_block.h
+
 # Don't try to regenerate Makefile or configure
 # We don't generate any of them
 Makefile: ;
@@ -352,6 +354,11 @@ ivshmem-client$(EXESUF): $(ivshmem-client-obj-y) 
libqemuutil.a libqemustub.a
 ivshmem-server$(EXESUF): $(ivshmem-server-obj-y) libqemuutil.a libqemustub.a
$(call LINK, $^)
 
+module_block.h: $(SRC_PATH)/scripts/modules/module_block.py config-host.mak
+   $(call quiet-command,$(PYTHON) $< $@ \
+   $(addprefix $(SRC_PATH)/,$(patsubst %.mo,%.c,$(block-obj-m))), \
+   "  GEN   $@")
+
 clean:
 # avoid old build problems by removing potentially incorrect old files
rm -f config.mak op-i386.h opc-i386.h gen-op-i386.h op-arm.h opc-arm.h 
gen-op-arm.h
diff --git a/scripts/modules/module_block.py b/scripts/modules/module_block.py
new file mode 100644
index 000..4075574
--- /dev/null
+++ b/scripts/modules/module_block.py
@@ -0,0 +1,122 @@
+#!/usr/bin/python
+#
+# Module information generator
+#
+# Copyright Red Hat, Inc. 2015 - 2016
+#
+# Authors:
+#  Marc Mari <mar...@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2.
+# See the COPYING file in the top-level directory.
+
+from __future__ import print_function
+import sys
+import os
+
+def get_string_struct(line):
+data = line.split()
+
+# data[0] -> struct element name
+# data[1] -> =
+# data[2] -> value
+
+return data[2].replace('"', '')[:-1]
+
+def add_module(fheader, library, format_name, protocol_name,
+probe, probe_device):
+lines = []
+lines.append('.library_name = "' + library + '",')
+if format_name != "":
+lines.append('.format_name = "' + format_name + '",')
+if protocol_name != "":
+lines.append('.protocol_name = "' + protocol_name + '",')
+if probe:
+lines.append('.has_probe = true,')
+if probe_device:
+lines.append('.has_probe_device = true,')
+
+text = '\n\t'.join(lines)
+fheader.write('\n\t{\n\t' + text + '\n\t},')
+
+def process_file(fheader, filename):
+# This parser assumes the coding style rules are being followed
+with open(filename, "r") as cfile:
+found_something = False
+found_start = False
+library, _ = os.path.splitext(os.path.basename(filename))
+for line in cfile:
+if found_start:
+line = line.replace('\n', '')
+if line.find(".format_name") != -1:
+format_name = get_string_struct(line)
+elif line.find(".protocol_name") != -1:
+protocol_name = get_string_struct(line)
+elif line.find(".bdrv_probe") != -1:
+probe = True
+elif line.find(".bdrv_probe_device") != -1:
+probe_device = True
+elif line == "};":
+add_module(fheader, library, format_name, protocol_name,
+probe, probe_device)
+found_start = False
+elif line.find("static BlockDriver") != -1:
+found_something = True
+found_start = True
+format_name = ""
+protocol_name = ""
+probe = False
+probe_device = False
+
+if not found_something:
+print("No BlockDriver struct found in " + filename + ". \
+Is this really a module?", file=sys.stderr)
+sys.exit(1)
+
+def print_top(fheader):
+fheader.write('''/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
+/*
+ * QEMU Block Module Infrastructure
+ *
+ * Authors:
+ *  Marc Mari   <mar...@redhat.com>
+ */
+
+''')
+
+fheader.write('''#ifndef QEMU_MODULE_BLOCK_H
+#define QEMU_MODULE_BLOCK_H
+
+#include "qemu-common.h"
+
+static const struct {
+const char *format_name;
+const char *protocol_name;
+const char *library_name;
+bool has_probe;
+bool has_probe_device;
+} block_driver_modules[] = {

[Qemu-block] [PATCH v3 10/32] blockdev: Move qcow2 probe to its own file

[Colin Lord]

Isolates qcow2 probe as part of the modularization process.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block/Makefile.objs   |  2 +-
 block/probe/qcow2.c   | 16 
 block/qcow2.c | 13 +
 include/block/probe.h |  1 +
 4 files changed, 19 insertions(+), 13 deletions(-)
 create mode 100644 block/probe/qcow2.c

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 9458da8..23240d9 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -25,7 +25,7 @@ block-obj-y += write-threshold.o
 
 block-obj-y += crypto.o
 block-obj-y += probe/bochs.o probe/cloop.o probe/luks.o probe/dmg.o
-block-obj-y += probe/parallels.o probe/qcow.o
+block-obj-y += probe/parallels.o probe/qcow.o probe/qcow2.o
 
 common-obj-y += stream.o
 common-obj-y += commit.o
diff --git a/block/probe/qcow2.c b/block/probe/qcow2.c
new file mode 100644
index 000..56f4e82
--- /dev/null
+++ b/block/probe/qcow2.c
@@ -0,0 +1,16 @@
+#include "qemu/osdep.h"
+#include "block/block_int.h"
+#include "block/probe.h"
+#include "block/qcow2.h"
+
+int qcow2_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+const QCowHeader *cow_header = (const void *)buf;
+
+if (buf_size >= sizeof(QCowHeader) &&
+be32_to_cpu(cow_header->magic) == QCOW_MAGIC &&
+be32_to_cpu(cow_header->version) >= 2)
+return 100;
+else
+return 0;
+}
diff --git a/block/qcow2.c b/block/qcow2.c
index 23f666d..55639eb 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -27,6 +27,7 @@
 #include "qemu/module.h"
 #include 
 #include "block/qcow2.h"
+#include "block/probe.h"
 #include "qemu/error-report.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/qmp/qbool.h"
@@ -64,18 +65,6 @@ typedef struct {
 #define  QCOW2_EXT_MAGIC_BACKING_FORMAT 0xE2792ACA
 #define  QCOW2_EXT_MAGIC_FEATURE_TABLE 0x6803f857
 
-static int qcow2_probe(const uint8_t *buf, int buf_size, const char *filename)
-{
-const QCowHeader *cow_header = (const void *)buf;
-
-if (buf_size >= sizeof(QCowHeader) &&
-be32_to_cpu(cow_header->magic) == QCOW_MAGIC &&
-be32_to_cpu(cow_header->version) >= 2)
-return 100;
-else
-return 0;
-}
-
 
 /* 
  * read qcow2 extension and fill bs
diff --git a/include/block/probe.h b/include/block/probe.h
index 5230da4..f9dd36e 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -8,5 +8,6 @@ int block_crypto_probe_luks(const uint8_t *buf, int buf_size,
 int dmg_probe(const uint8_t *buf, int buf_size, const char *filename);
 int parallels_probe(const uint8_t *buf, int buf_size, const char *filename);
 int qcow_probe(const uint8_t *buf, int buf_size, const char *filename);
+int qcow2_probe(const uint8_t *buf, int buf_size, const char *filename);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v3 13/32] blockdev: Move vdi probe to its own file

[Colin Lord]

Isolates vdi probe as part of the modularization process.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block/Makefile.objs|  2 +-
 block/probe/vdi.c  | 26 +
 block/vdi.c| 69 ++
 include/block/driver/vdi.h | 49 
 include/block/probe.h  |  1 +
 5 files changed, 79 insertions(+), 68 deletions(-)
 create mode 100644 block/probe/vdi.c
 create mode 100644 include/block/driver/vdi.h

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 693cc97..4a5bd88 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -26,7 +26,7 @@ block-obj-y += write-threshold.o
 block-obj-y += crypto.o
 block-obj-y += probe/bochs.o probe/cloop.o probe/luks.o probe/dmg.o
 block-obj-y += probe/parallels.o probe/qcow.o probe/qcow2.o probe/qed.o
-block-obj-y += probe/raw.o
+block-obj-y += probe/raw.o probe/vdi.o
 
 common-obj-y += stream.o
 common-obj-y += commit.o
diff --git a/block/probe/vdi.c b/block/probe/vdi.c
new file mode 100644
index 000..9adf5e5
--- /dev/null
+++ b/block/probe/vdi.c
@@ -0,0 +1,26 @@
+#include "qemu/osdep.h"
+#include "block/block_int.h"
+#include "block/probe.h"
+#include "block/driver/vdi.h"
+
+int vdi_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+const VdiHeader *header = (const VdiHeader *)buf;
+int ret = 0;
+
+logout("\n");
+
+if (buf_size < sizeof(*header)) {
+/* Header too small, no VDI. */
+} else if (le32_to_cpu(header->signature) == VDI_SIGNATURE) {
+ret = 100;
+}
+
+if (ret == 0) {
+logout("no vdi image\n");
+} else {
+logout("%s", header->text);
+}
+
+return ret;
+}
diff --git a/block/vdi.c b/block/vdi.c
index 7d9ab9c..e99ae92 100644
--- a/block/vdi.c
+++ b/block/vdi.c
@@ -52,6 +52,8 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
 #include "block/block_int.h"
+#include "block/probe.h"
+#include "block/driver/vdi.h"
 #include "sysemu/block-backend.h"
 #include "qemu/module.h"
 #include "qemu/bswap.h"
@@ -59,14 +61,6 @@
 #include "qemu/coroutine.h"
 #include "qemu/cutils.h"
 
-#if defined(CONFIG_UUID)
-#include 
-#else
-/* TODO: move uuid emulation to some central place in QEMU. */
-#include "sysemu/sysemu.h" /* UUID_FMT */
-typedef unsigned char uuid_t[16];
-#endif
-
 /* Code configuration options. */
 
 /* Enable debug messages. */
@@ -92,16 +86,6 @@ typedef unsigned char uuid_t[16];
 #define SECTOR_SIZE 512
 #define DEFAULT_CLUSTER_SIZE (1 * MiB)
 
-#if defined(CONFIG_VDI_DEBUG)
-#define logout(fmt, ...) \
-fprintf(stderr, "vdi\t%-24s" fmt, __func__, ##__VA_ARGS__)
-#else
-#define logout(fmt, ...) ((void)0)
-#endif
-
-/* Image signature. */
-#define VDI_SIGNATURE 0xbeda107f
-
 /* Image version. */
 #define VDI_VERSION_1_1 0x00010001
 
@@ -163,33 +147,6 @@ static inline void uuid_unparse(const uuid_t uu, char *out)
 #endif
 
 typedef struct {
-char text[0x40];
-uint32_t signature;
-uint32_t version;
-uint32_t header_size;
-uint32_t image_type;
-uint32_t image_flags;
-char description[256];
-uint32_t offset_bmap;
-uint32_t offset_data;
-uint32_t cylinders; /* disk geometry, unused here */
-uint32_t heads; /* disk geometry, unused here */
-uint32_t sectors;   /* disk geometry, unused here */
-uint32_t sector_size;
-uint32_t unused1;
-uint64_t disk_size;
-uint32_t block_size;
-uint32_t block_extra;   /* unused here */
-uint32_t blocks_in_image;
-uint32_t blocks_allocated;
-uuid_t uuid_image;
-uuid_t uuid_last_snap;
-uuid_t uuid_link;
-uuid_t uuid_parent;
-uint64_t unused2[7];
-} QEMU_PACKED VdiHeader;
-
-typedef struct {
 /* The block map entries are little endian (even in memory). */
 uint32_t *bmap;
 /* Size of block (bytes). */
@@ -371,28 +328,6 @@ static int vdi_make_empty(BlockDriverState *bs)
 return 0;
 }
 
-static int vdi_probe(const uint8_t *buf, int buf_size, const char *filename)
-{
-const VdiHeader *header = (const VdiHeader *)buf;
-int ret = 0;
-
-logout("\n");
-
-if (buf_size < sizeof(*header)) {
-/* Header too small, no VDI. */
-} else if (le32_to_cpu(header->signature) == VDI_SIGNATURE) {
-ret = 100;
-}
-
-if (ret == 0) {
-logout("no vdi image\n");
-} else {
-logout("%s", header->text);
-}
-
-return ret;
-}
-
 static int vdi_open(BlockDriverState *bs, QDict *options, int flags,
 Error **errp)
 {
diff --git a/include/block/driver/vdi.h b/include/block/driver/vdi.h
new file mode 100644
index 000..4020c94
--- /dev/null
+++ b/include/block/driver/vdi.h
@@

[Qemu-block] [PATCH v3 07/32] blockdev: Move dmg probe to its own file

[Colin Lord]

Isolate dmg probe as part of the modularization process.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block/Makefile.objs   |  2 +-
 block/dmg.c   | 16 +---
 block/probe/dmg.c | 17 +
 include/block/probe.h |  1 +
 4 files changed, 20 insertions(+), 16 deletions(-)
 create mode 100644 block/probe/dmg.c

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 28a7ec3..1d744eb 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -24,7 +24,7 @@ block-obj-y += accounting.o dirty-bitmap.o
 block-obj-y += write-threshold.o
 
 block-obj-y += crypto.o
-block-obj-y += probe/bochs.o probe/cloop.o probe/luks.o
+block-obj-y += probe/bochs.o probe/cloop.o probe/luks.o probe/dmg.o
 
 common-obj-y += stream.o
 common-obj-y += commit.o
diff --git a/block/dmg.c b/block/dmg.c
index 06eb513..2c70687 100644
--- a/block/dmg.c
+++ b/block/dmg.c
@@ -25,6 +25,7 @@
 #include "qapi/error.h"
 #include "qemu-common.h"
 #include "block/block_int.h"
+#include "block/probe.h"
 #include "qemu/bswap.h"
 #include "qemu/error-report.h"
 #include "qemu/module.h"
@@ -66,21 +67,6 @@ typedef struct BDRVDMGState {
 #endif
 } BDRVDMGState;
 
-static int dmg_probe(const uint8_t *buf, int buf_size, const char *filename)
-{
-int len;
-
-if (!filename) {
-return 0;
-}
-
-len = strlen(filename);
-if (len > 4 && !strcmp(filename + len - 4, ".dmg")) {
-return 2;
-}
-return 0;
-}
-
 static int read_uint64(BlockDriverState *bs, int64_t offset, uint64_t *result)
 {
 uint64_t buffer;
diff --git a/block/probe/dmg.c b/block/probe/dmg.c
new file mode 100644
index 000..a40281b
--- /dev/null
+++ b/block/probe/dmg.c
@@ -0,0 +1,17 @@
+#include "qemu/osdep.h"
+#include "block/probe.h"
+
+int dmg_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+int len;
+
+if (!filename) {
+return 0;
+}
+
+len = strlen(filename);
+if (len > 4 && !strcmp(filename + len - 4, ".dmg")) {
+return 2;
+}
+return 0;
+}
diff --git a/include/block/probe.h b/include/block/probe.h
index 35a8d00..267431d 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -5,5 +5,6 @@ int bochs_probe(const uint8_t *buf, int buf_size, const char 
*filename);
 int cloop_probe(const uint8_t *buf, int buf_size, const char *filename);
 int block_crypto_probe_luks(const uint8_t *buf, int buf_size,
 const char *filename);
+int dmg_probe(const uint8_t *buf, int buf_size, const char *filename);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v3 05/32] blockdev: Move cloop probe to its own file

[Colin Lord]

Isolates cloop probing function as part of the modularization process.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block/Makefile.objs   |  2 +-
 block/cloop.c | 16 +---
 block/probe/cloop.c   | 17 +
 include/block/probe.h |  1 +
 4 files changed, 20 insertions(+), 16 deletions(-)
 create mode 100644 block/probe/cloop.c

diff --git a/block/Makefile.objs b/block/Makefile.objs
index bc0c2aa..3b98001 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -24,7 +24,7 @@ block-obj-y += accounting.o dirty-bitmap.o
 block-obj-y += write-threshold.o
 
 block-obj-y += crypto.o
-block-obj-y += probe/bochs.o
+block-obj-y += probe/bochs.o probe/cloop.o
 
 common-obj-y += stream.o
 common-obj-y += commit.o
diff --git a/block/cloop.c b/block/cloop.c
index ea5a92b..bf9fb75 100644
--- a/block/cloop.c
+++ b/block/cloop.c
@@ -25,6 +25,7 @@
 #include "qapi/error.h"
 #include "qemu-common.h"
 #include "block/block_int.h"
+#include "block/probe.h"
 #include "qemu/module.h"
 #include "qemu/bswap.h"
 #include 
@@ -44,21 +45,6 @@ typedef struct BDRVCloopState {
 z_stream zstream;
 } BDRVCloopState;
 
-static int cloop_probe(const uint8_t *buf, int buf_size, const char *filename)
-{
-const char *magic_version_2_0 = "#!/bin/sh\n"
-"#V2.0 Format\n"
-"modprobe cloop file=$0 && mount -r -t iso9660 /dev/cloop $1\n";
-int length = strlen(magic_version_2_0);
-if (length > buf_size) {
-length = buf_size;
-}
-if (!memcmp(magic_version_2_0, buf, length)) {
-return 2;
-}
-return 0;
-}
-
 static int cloop_open(BlockDriverState *bs, QDict *options, int flags,
   Error **errp)
 {
diff --git a/block/probe/cloop.c b/block/probe/cloop.c
new file mode 100644
index 000..955c29c
--- /dev/null
+++ b/block/probe/cloop.c
@@ -0,0 +1,17 @@
+#include "qemu/osdep.h"
+#include "block/probe.h"
+
+int cloop_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+const char *magic_version_2_0 = "#!/bin/sh\n"
+"#V2.0 Format\n"
+"modprobe cloop file=$0 && mount -r -t iso9660 /dev/cloop $1\n";
+int length = strlen(magic_version_2_0);
+if (length > buf_size) {
+length = buf_size;
+}
+if (!memcmp(magic_version_2_0, buf, length)) {
+return 2;
+}
+return 0;
+}
diff --git a/include/block/probe.h b/include/block/probe.h
index 6450ca1..ed1a60b 100644
--- a/include/block/probe.h
+++ b/include/block/probe.h
@@ -2,5 +2,6 @@
 #define PROBE_H
 
 int bochs_probe(const uint8_t *buf, int buf_size, const char *filename);
+int cloop_probe(const uint8_t *buf, int buf_size, const char *filename);
 
 #endif
-- 
2.5.5

[Qemu-block] [PATCH v3 01/32] blockdev: prepare iSCSI block driver for dynamic loading

[Colin Lord]

This commit moves the initialization of the QemuOptsList qemu_iscsi_opts
struct out of block/iscsi.c in order to allow the iscsi module to be
dynamically loaded.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block/iscsi.c | 36 
 vl.c  | 38 ++
 2 files changed, 38 insertions(+), 36 deletions(-)

diff --git a/block/iscsi.c b/block/iscsi.c
index 9bb5ff6..8246075 100644
--- a/block/iscsi.c
+++ b/block/iscsi.c
@@ -1880,45 +1880,9 @@ static BlockDriver bdrv_iscsi = {
 .bdrv_attach_aio_context = iscsi_attach_aio_context,
 };
 
-static QemuOptsList qemu_iscsi_opts = {
-.name = "iscsi",
-.head = QTAILQ_HEAD_INITIALIZER(qemu_iscsi_opts.head),
-.desc = {
-{
-.name = "user",
-.type = QEMU_OPT_STRING,
-.help = "username for CHAP authentication to target",
-},{
-.name = "password",
-.type = QEMU_OPT_STRING,
-.help = "password for CHAP authentication to target",
-},{
-.name = "password-secret",
-.type = QEMU_OPT_STRING,
-.help = "ID of the secret providing password for CHAP "
-"authentication to target",
-},{
-.name = "header-digest",
-.type = QEMU_OPT_STRING,
-.help = "HeaderDigest setting. "
-"{CRC32C|CRC32C-NONE|NONE-CRC32C|NONE}",
-},{
-.name = "initiator-name",
-.type = QEMU_OPT_STRING,
-.help = "Initiator iqn name to use when connecting",
-},{
-.name = "timeout",
-.type = QEMU_OPT_NUMBER,
-.help = "Request timeout in seconds (default 0 = no timeout)",
-},
-{ /* end of list */ }
-},
-};
-
 static void iscsi_block_init(void)
 {
 bdrv_register(_iscsi);
-qemu_add_opts(_iscsi_opts);
 }
 
 block_init(iscsi_block_init);
diff --git a/vl.c b/vl.c
index 9bb7f4c..cb640c2 100644
--- a/vl.c
+++ b/vl.c
@@ -506,6 +506,41 @@ static QemuOptsList qemu_fw_cfg_opts = {
 },
 };
 
+static QemuOptsList qemu_iscsi_opts = {
+.name = "iscsi",
+.head = QTAILQ_HEAD_INITIALIZER(qemu_iscsi_opts.head),
+.desc = {
+{
+.name = "user",
+.type = QEMU_OPT_STRING,
+.help = "username for CHAP authentication to target",
+},{
+.name = "password",
+.type = QEMU_OPT_STRING,
+.help = "password for CHAP authentication to target",
+},{
+.name = "password-secret",
+.type = QEMU_OPT_STRING,
+.help = "ID of the secret providing password for CHAP "
+"authentication to target",
+},{
+.name = "header-digest",
+.type = QEMU_OPT_STRING,
+.help = "HeaderDigest setting. "
+"{CRC32C|CRC32C-NONE|NONE-CRC32C|NONE}",
+},{
+.name = "initiator-name",
+.type = QEMU_OPT_STRING,
+.help = "Initiator iqn name to use when connecting",
+},{
+.name = "timeout",
+.type = QEMU_OPT_NUMBER,
+.help = "Request timeout in seconds (default 0 = no timeout)",
+},
+{ /* end of list */ }
+},
+};
+
 /**
  * Get machine options
  *
@@ -2982,6 +3017,9 @@ int main(int argc, char **argv, char **envp)
 qemu_add_opts(_icount_opts);
 qemu_add_opts(_semihosting_config_opts);
 qemu_add_opts(_fw_cfg_opts);
+#ifdef CONFIG_LIBISCSI
+qemu_add_opts(_iscsi_opts);
+#endif
 module_call_init(MODULE_INIT_OPTS);
 
 runstate_init();
-- 
2.5.5

Re: [Qemu-block] [Qemu-devel] [PATCH 1/3] blockdev: prepare iSCSI block driver for dynamic loading

[Colin Lord]

On 06/22/2016 09:22 PM, Fam Zheng wrote:
> On Wed, 06/22 17:35, Colin Lord wrote:
>> This commit moves the initialization of the QemuOptsList qemu_iscsi_opts
>> struct out of block/iscsi.c in order to allow it to be dynamically
>> loaded. Drivers that perform init operations other than registering
>> themselves can't be modularized, so this moves the initialization of
>> this struct into the main binary.
>>
>> Signed-off-by: Colin Lord <cl...@redhat.com>
>> ---
>>  block/iscsi.c | 36 
>>  vl.c  | 36 
>>  2 files changed, 36 insertions(+), 36 deletions(-)
>>
>> diff --git a/block/iscsi.c b/block/iscsi.c
>> index 7e78ade..6193499 100644
>> --- a/block/iscsi.c
>> +++ b/block/iscsi.c
>> @@ -1879,45 +1879,9 @@ static BlockDriver bdrv_iscsi = {
>>  .bdrv_attach_aio_context = iscsi_attach_aio_context,
>>  };
>>  
>> -static QemuOptsList qemu_iscsi_opts = {
>> -.name = "iscsi",
>> -.head = QTAILQ_HEAD_INITIALIZER(qemu_iscsi_opts.head),
>> -.desc = {
>> -{
>> -.name = "user",
>> -.type = QEMU_OPT_STRING,
>> -.help = "username for CHAP authentication to target",
>> -},{
>> -.name = "password",
>> -.type = QEMU_OPT_STRING,
>> -.help = "password for CHAP authentication to target",
>> -},{
>> -.name = "password-secret",
>> -.type = QEMU_OPT_STRING,
>> -.help = "ID of the secret providing password for CHAP "
>> -"authentication to target",
>> -},{
>> -.name = "header-digest",
>> -.type = QEMU_OPT_STRING,
>> -.help = "HeaderDigest setting. "
>> -"{CRC32C|CRC32C-NONE|NONE-CRC32C|NONE}",
>> -},{
>> -.name = "initiator-name",
>> -.type = QEMU_OPT_STRING,
>> -.help = "Initiator iqn name to use when connecting",
>> -},{
>> -.name = "timeout",
>> -.type = QEMU_OPT_NUMBER,
>> -.help = "Request timeout in seconds (default 0 = no timeout)",
>> -},
>> -{ /* end of list */ }
>> -},
>> -};
>> -
>>  static void iscsi_block_init(void)
>>  {
>>  bdrv_register(_iscsi);
>> -qemu_add_opts(_iscsi_opts);
>>  }
>>  
>>  block_init(iscsi_block_init);
>> diff --git a/vl.c b/vl.c
>> index 45eff56..4f04daa 100644
>> --- a/vl.c
>> +++ b/vl.c
>> @@ -526,6 +526,41 @@ static QemuOptsList qemu_fw_cfg_opts = {
>>  },
>>  };
>>  
>> +static QemuOptsList qemu_iscsi_opts = {
>> +.name = "iscsi",
>> +.head = QTAILQ_HEAD_INITIALIZER(qemu_iscsi_opts.head),
>> +.desc = {
>> +{
>> +.name = "user",
>> +.type = QEMU_OPT_STRING,
>> +.help = "username for CHAP authentication to target",
>> +},{
>> +.name = "password",
>> +.type = QEMU_OPT_STRING,
>> +.help = "password for CHAP authentication to target",
>> +},{
>> +.name = "password-secret",
>> +.type = QEMU_OPT_STRING,
>> +.help = "ID of the secret providing password for CHAP "
>> +"authentication to target",
>> +},{
>> +.name = "header-digest",
>> +.type = QEMU_OPT_STRING,
>> +.help = "HeaderDigest setting. "
>> +"{CRC32C|CRC32C-NONE|NONE-CRC32C|NONE}",
>> +},{
>> +.name = "initiator-name",
>> +.type = QEMU_OPT_STRING,
>> +.help = "Initiator iqn name to use when connecting",
>> +},{
>> +.name = "timeout",
>> +.type = QEMU_OPT_NUMBER,
>> +.help = "Request timeout in seconds (default 0 = no timeout)",
>> +},
>> +{ /* end of list */ }
>> +},
>> +};
>> +
>>  /**
>>   * Get machine options
>>   *
>> @@ -3006,6 +3041,7 @@ int main(int argc, char **argv, char **envp)
>>  qemu_add_opts(_icount_opts);
>>  qemu_add_opts(_semihosting_config_opts);
>>  qemu_add_opts(_fw_cfg_opts);
>> +qemu_add_opts(_iscsi_opts);
> 
> Should the new code still be conditional on CONFIG_LIBISCSI?  Because
> previously it was.

Yeah, I think that should still be the case. Thanks for catching that.
> 
> Fam
> 
>>  module_call_init(MODULE_INIT_OPTS);
>>  
>>  runstate_init();
>> -- 
>> 2.5.5
>>
>>

[Qemu-block] [PATCH 0/3] Dynamic module loading for block drivers

[Colin Lord]

This is v2 of the series I sent out last week. These are the changes I
made based on the feedback I got:
- Fixed typo and Marc's email address in the python script
- Moved registration of iscsi_opts into vl.c

What I didn't do:
- Remove copy-pasted loops

There was a bit of discussion about how to remove the need for the copy-
paste loops that are in block.c. I attempted to solve it by using
g_module_sym to load the BlockDriver struct directly at the time the
module gets loaded and returning it so that the loops were not
necessary. I accomplished this by adding a field to the struct which,
for a given format/protocol configuration, had the name of the
corresponding BlockDriver struct. Having the name allowed me to load the
symbol right out of the loaded module. However, it turns out that, at
least as far as I can tell, g_module_sym can't load the BlockDriver
structs in this way because they are declared static.

I tested the attempt to remove the copy-pasted loops by using the
qemu-iotests on it with ssh (which is modularized). The errors I got
were along the lines of:

can't open device ssh://[address removed]: Unknown protocol 'ssh'
Failed to find driver in module

To test my theory (I haven't had much luck finding reliable
documentation about this) that it was because they were static, I
changed the definition of the bdrv_ssh BlockDriver to not be static.
Unfortunately I still got errors, but I believe the drivers got loaded.
The errors were not the same, rather these ones were complaining about
the host key not matching the one in known_hosts. I've had this issue
while trying to set up ssh with qemu in the past, so I'm not quite as
worried about it (although I'd love to hear a fix), and more importantly
there aren't any messages about the driver not being found.

That hopefully explains some of the issues, and why I'm submitting this
with the duplicated loops still intact. If there are other ideas for
dealing with this my ears are open, but this is what I have for now.

Colin Lord (1):
  blockdev: prepare iSCSI block driver for dynamic loading

Marc Mari (2):
  blockdev: Add dynamic generation of module_block.h
  blockdev: Add dynamic module loading for block drivers

 .gitignore  |   1 +
 Makefile|  11 +++-
 block.c |  86 +++---
 block/iscsi.c   |  36 ---
 include/qemu/module.h   |   3 +
 scripts/modules/module_block.py | 134 
 util/module.c   |  37 +++
 vl.c|  36 +++
 8 files changed, 269 insertions(+), 75 deletions(-)
 create mode 100644 scripts/modules/module_block.py

-- 
2.5.5

[Qemu-block] [PATCH 1/3] blockdev: prepare iSCSI block driver for dynamic loading

[Colin Lord]

This commit moves the initialization of the QemuOptsList qemu_iscsi_opts
struct out of block/iscsi.c in order to allow it to be dynamically
loaded. Drivers that perform init operations other than registering
themselves can't be modularized, so this moves the initialization of
this struct into the main binary.

Signed-off-by: Colin Lord <cl...@redhat.com>
---
 block/iscsi.c | 36 
 vl.c  | 36 
 2 files changed, 36 insertions(+), 36 deletions(-)

diff --git a/block/iscsi.c b/block/iscsi.c
index 7e78ade..6193499 100644
--- a/block/iscsi.c
+++ b/block/iscsi.c
@@ -1879,45 +1879,9 @@ static BlockDriver bdrv_iscsi = {
 .bdrv_attach_aio_context = iscsi_attach_aio_context,
 };
 
-static QemuOptsList qemu_iscsi_opts = {
-.name = "iscsi",
-.head = QTAILQ_HEAD_INITIALIZER(qemu_iscsi_opts.head),
-.desc = {
-{
-.name = "user",
-.type = QEMU_OPT_STRING,
-.help = "username for CHAP authentication to target",
-},{
-.name = "password",
-.type = QEMU_OPT_STRING,
-.help = "password for CHAP authentication to target",
-},{
-.name = "password-secret",
-.type = QEMU_OPT_STRING,
-.help = "ID of the secret providing password for CHAP "
-"authentication to target",
-},{
-.name = "header-digest",
-.type = QEMU_OPT_STRING,
-.help = "HeaderDigest setting. "
-"{CRC32C|CRC32C-NONE|NONE-CRC32C|NONE}",
-},{
-.name = "initiator-name",
-.type = QEMU_OPT_STRING,
-.help = "Initiator iqn name to use when connecting",
-},{
-.name = "timeout",
-.type = QEMU_OPT_NUMBER,
-.help = "Request timeout in seconds (default 0 = no timeout)",
-},
-{ /* end of list */ }
-},
-};
-
 static void iscsi_block_init(void)
 {
 bdrv_register(_iscsi);
-qemu_add_opts(_iscsi_opts);
 }
 
 block_init(iscsi_block_init);
diff --git a/vl.c b/vl.c
index 45eff56..4f04daa 100644
--- a/vl.c
+++ b/vl.c
@@ -526,6 +526,41 @@ static QemuOptsList qemu_fw_cfg_opts = {
 },
 };
 
+static QemuOptsList qemu_iscsi_opts = {
+.name = "iscsi",
+.head = QTAILQ_HEAD_INITIALIZER(qemu_iscsi_opts.head),
+.desc = {
+{
+.name = "user",
+.type = QEMU_OPT_STRING,
+.help = "username for CHAP authentication to target",
+},{
+.name = "password",
+.type = QEMU_OPT_STRING,
+.help = "password for CHAP authentication to target",
+},{
+.name = "password-secret",
+.type = QEMU_OPT_STRING,
+.help = "ID of the secret providing password for CHAP "
+"authentication to target",
+},{
+.name = "header-digest",
+.type = QEMU_OPT_STRING,
+.help = "HeaderDigest setting. "
+"{CRC32C|CRC32C-NONE|NONE-CRC32C|NONE}",
+},{
+.name = "initiator-name",
+.type = QEMU_OPT_STRING,
+.help = "Initiator iqn name to use when connecting",
+},{
+.name = "timeout",
+.type = QEMU_OPT_NUMBER,
+.help = "Request timeout in seconds (default 0 = no timeout)",
+},
+{ /* end of list */ }
+},
+};
+
 /**
  * Get machine options
  *
@@ -3006,6 +3041,7 @@ int main(int argc, char **argv, char **envp)
 qemu_add_opts(_icount_opts);
 qemu_add_opts(_semihosting_config_opts);
 qemu_add_opts(_fw_cfg_opts);
+qemu_add_opts(_iscsi_opts);
 module_call_init(MODULE_INIT_OPTS);
 
 runstate_init();
-- 
2.5.5

[Qemu-block] [PATCH 2/3] blockdev: Add dynamic generation of module_block.h

[Colin Lord]

From: Marc Mari <mar...@redhat.com>

To simplify the addition of new block modules, add a script that generates
include/qemu/module_block.h automatically from the modules' source code.

This script assumes that the QEMU coding style rules are followed.

Signed-off-by: Marc Mari <mar...@redhat.com>
Signed-off-by: Colin Lord <cl...@redhat.com>
---
 .gitignore  |   1 +
 Makefile|   8 +++
 scripts/modules/module_block.py | 134 
 3 files changed, 143 insertions(+)
 create mode 100644 scripts/modules/module_block.py

diff --git a/.gitignore b/.gitignore
index 38ee1c5..06aa064 100644
--- a/.gitignore
+++ b/.gitignore
@@ -110,3 +110,4 @@ tags
 TAGS
 docker-src.*
 *~
+/include/qemu/module_block.h
diff --git a/Makefile b/Makefile
index ed4032a..8f8b6a2 100644
--- a/Makefile
+++ b/Makefile
@@ -76,6 +76,8 @@ GENERATED_HEADERS += trace/generated-ust-provider.h
 GENERATED_SOURCES += trace/generated-ust.c
 endif
 
+GENERATED_HEADERS += include/qemu/module_block.h
+
 # Don't try to regenerate Makefile or configure
 # We don't generate any of them
 Makefile: ;
@@ -352,6 +354,12 @@ ivshmem-client$(EXESUF): $(ivshmem-client-obj-y) 
libqemuutil.a libqemustub.a
 ivshmem-server$(EXESUF): $(ivshmem-server-obj-y) libqemuutil.a libqemustub.a
$(call LINK, $^)
 
+include/qemu/module_block.h: $(SRC_PATH)/scripts/modules/module_block.py 
config-host.mak
+   $(call quiet-command,$(PYTHON) \
+$(SRC_PATH)/scripts/modules/module_block.py \
+   $(SRC_PATH)/"./include/qemu/" $(addprefix $(SRC_PATH)/,$(patsubst 
%.mo,%.c,$(block-obj-m))), \
+   "  GEN   $@")
+
 clean:
 # avoid old build problems by removing potentially incorrect old files
rm -f config.mak op-i386.h opc-i386.h gen-op-i386.h op-arm.h opc-arm.h 
gen-op-arm.h
diff --git a/scripts/modules/module_block.py b/scripts/modules/module_block.py
new file mode 100644
index 000..2b5d24c
--- /dev/null
+++ b/scripts/modules/module_block.py
@@ -0,0 +1,134 @@
+#!/usr/bin/python
+#
+# Module information generator
+#
+# Copyright Red Hat, Inc. 2015
+#
+# Authors:
+#  Marc Mari <mar...@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2.
+# See the COPYING file in the top-level directory.
+
+from __future__ import print_function
+import sys
+import os
+
+def get_string_struct(line):
+data = line.split()
+
+# data[0] -> struct element name
+# data[1] -> =
+# data[2] -> value
+
+return data[2].replace('"', '')[:-1]
+
+def add_module(fheader, library, format_name, protocol_name,
+probe, probe_device):
+lines = []
+lines.append('.library_name = "' + library + '",')
+if format_name != "":
+lines.append('.format_name = "' + format_name + '",')
+if protocol_name != "":
+lines.append('.protocol_name = "' + protocol_name + '",')
+if probe:
+lines.append('.has_probe = true,')
+if probe_device:
+lines.append('.has_probe_device = true,')
+
+text = '\n\t'.join(lines)
+fheader.write('\n\t{\n\t' + text + '\n\t},')
+
+def process_file(fheader, filename):
+# This parser assumes the coding style rules are being followed
+with open(filename, "r") as cfile:
+found_something = False
+found_start = False
+library, _ = os.path.splitext(os.path.basename(filename))
+for line in cfile:
+if found_start:
+line = line.replace('\n', '')
+if line.find(".format_name") != -1:
+format_name = get_string_struct(line)
+elif line.find(".protocol_name") != -1:
+protocol_name = get_string_struct(line)
+elif line.find(".bdrv_probe") != -1:
+probe = True
+elif line.find(".bdrv_probe_device") != -1:
+probe_device = True
+elif line == "};":
+add_module(fheader, library, format_name, protocol_name,
+probe, probe_device)
+found_start = False
+elif line.find("static BlockDriver") != -1:
+found_something = True
+found_start = True
+format_name = ""
+protocol_name = ""
+probe = False
+probe_device = False
+
+if not found_something:
+print("No BlockDriver struct found in " + filename + ". \
+Is this really a module?", file=sys.stderr)
+sys.exit(1)
+
+def print_top(fheader):
+fheader.write('''/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
+/*
+ * QEMU Block Module Infrastructure
+ *
+ * Copyright Red Hat, Inc. 2015
+ *
+ * Authors

Re: [Qemu-block] [Qemu-devel] [PATCH 0/2] Dynamic module loading for block drivers

[Colin Lord]

On 06/21/2016 05:32 AM, Stefan Hajnoczi wrote:
> On Mon, Jun 20, 2016 at 11:32:38AM -0400, Colin Lord wrote:
>> On 06/17/2016 05:54 AM, Stefan Hajnoczi wrote:
>>> On Wed, Jun 15, 2016 at 02:40:53PM -0400, Colin Lord wrote:
>>>> 1) Denis Lunev suggested having block_module_load_one return the
>>>> loaded driver to reduce duplicated for loops in many of the functions
>>>> in block.c. I'd be happy to do this but wasn't completely sure how
>>>> error handling would happen in that case since currently the return
>>>> value is an integer error code. Would I switch to using the
>>>> error handling mechanisms provided in util/error.c?
>>>
>>> Yes, change "int foo(...)" to "MyObject *foo(..., Error **errp)".  The
>>> Error object allows functions to provide detailed, human-readable error
>>> messages so it can be a win.
>>>
>>> If this change would cause a lot of changes you can stop the refactoring
>>> from snowballing using error_setg_errno() to bridge new Error functions
>>> with old int -errno functions:
>>>
>>>   MyObject *foo(..., Error **errp)
>>>   {
>>>   /* I don't want propagate Error to all called functions yet, it
>>>* would snowball.  So just wrap up the errno:
>>>*/
>>>   ret = legacy_function(...);
>>>   if (ret < 0) {
>>>   error_setg_errno(errp, -ret, "legacy_function failed");
>>>   return NULL;
>>>   }
>>>   }
>>>
>>
>> So I started implementing this part (having block_module_load_one return
>> the module/driver) last Friday and in the process I realized that it is
>> not as simple as it seemed to me at first. The duplicated for loops this
>> was supposed to fix aren't the nicest thing, but I don't think that
>> returning the block/module directly is any better.
>>
>> The issue is that a module may contain multiple drivers, so
>> block_module_load_one would have to know exactly which driver to return,
>> which seems rather out of scope for that function. The function
>> registers multiple drivers when the module is loaded, so choosing just
>> one of them to return seems a little odd.
>>
>> An alternative way to do this is to return the entire module rather than
>> just the driver, and let the caller figure out which driver it needs
>> from the module. However, that would require a loop of some sort anyway
>> to examine all the drivers in the module, so we're kind of back where we
>> started. But it is actually a little worse than where we started I think
>> because as far as I can tell, to actually access the drivers through the
>> module, you need to know the name of the symbol you want (which I
>> believe is the name of the BlockDriver structs). I don't see a good way
>> to know the exact name of the struct that would be robust, so at this
>> point it seems like it may be better to just leave the duplicated for
>> loops in place.
> 
> I think the issue comes from the fact that you are considering something
> like load_block_module(const char *filename) as the API instead of
> request_block_driver(const char *driver_name).  In the latter case it's
> possible to return a BlockDriver pointer.  In the former it's not.
> 
> The request_block_driver() approach requires a mapping from block driver
> names to modules.  This can be achieved using a directory layout with
> symlinks (hmm...Windows portability?):
> 
>   /usr/lib/qemu/block/
> +--- sheepdog.so
> +--- by-protocol/
>   +--- sheepdog+unix -> ../sheepdog.so
> 
> request_block_driver() would look at
> /usr/lib/qemu/block/by-protocol/ to find the module file.
> 
> Stefan
> 
The question is, even if I was using request_block_driver(const char
*driver_name), I'm still not completely clear in your suggestion how I'm
supposed to get the driver name in the first place. I don't see where
I'd be getting that information from. I'd be happy to hear more about
it, but it also made me think of another possible solution: right now
the block_driver_modules array is being auto-generated by the python
script. Why not just add a "name" field to the struct so that each array
entry would actually know the name of the corresponding BlockDriver struct?

Then request_block_driver would take in the array entry (which is a
struct which currently has no name, which I will probably be fixing as
Paolo asked) and it could load the file using the library name (as
module_load_one does now). It could easily return the BlockDriver then
using the name field. It seems to me that this would work, and would be
a fairly minor change from how things are now (in particular I think
that symlinks wouldn't be necessary with this).

Re: [Qemu-block] [Qemu-devel] [PATCH 0/2] Dynamic module loading for block drivers

[Colin Lord]

On 06/17/2016 05:54 AM, Stefan Hajnoczi wrote:
> On Wed, Jun 15, 2016 at 02:40:53PM -0400, Colin Lord wrote:
>> 1) Denis Lunev suggested having block_module_load_one return the
>> loaded driver to reduce duplicated for loops in many of the functions
>> in block.c. I'd be happy to do this but wasn't completely sure how
>> error handling would happen in that case since currently the return
>> value is an integer error code. Would I switch to using the
>> error handling mechanisms provided in util/error.c?
> 
> Yes, change "int foo(...)" to "MyObject *foo(..., Error **errp)".  The
> Error object allows functions to provide detailed, human-readable error
> messages so it can be a win.
> 
> If this change would cause a lot of changes you can stop the refactoring
> from snowballing using error_setg_errno() to bridge new Error functions
> with old int -errno functions:
> 
>   MyObject *foo(..., Error **errp)
>   {
>   /* I don't want propagate Error to all called functions yet, it
>* would snowball.  So just wrap up the errno:
>*/
>   ret = legacy_function(...);
>   if (ret < 0) {
>   error_setg_errno(errp, -ret, "legacy_function failed");
> return NULL;
>   }
>   }
> 

So I started implementing this part (having block_module_load_one return
the module/driver) last Friday and in the process I realized that it is
not as simple as it seemed to me at first. The duplicated for loops this
was supposed to fix aren't the nicest thing, but I don't think that
returning the block/module directly is any better.

The issue is that a module may contain multiple drivers, so
block_module_load_one would have to know exactly which driver to return,
which seems rather out of scope for that function. The function
registers multiple drivers when the module is loaded, so choosing just
one of them to return seems a little odd.

An alternative way to do this is to return the entire module rather than
just the driver, and let the caller figure out which driver it needs
from the module. However, that would require a loop of some sort anyway
to examine all the drivers in the module, so we're kind of back where we
started. But it is actually a little worse than where we started I think
because as far as I can tell, to actually access the drivers through the
module, you need to know the name of the symbol you want (which I
believe is the name of the BlockDriver structs). I don't see a good way
to know the exact name of the struct that would be robust, so at this
point it seems like it may be better to just leave the duplicated for
loops in place.

Re: [Qemu-block] [Qemu-devel] [PATCH 2/2] blockdev: Add dynamic module loading for block drivers

[Colin Lord]

On 06/16/2016 10:05 AM, Paolo Bonzini wrote:
> 
> 
> On 16/06/2016 16:00, Colin Lord wrote:
>>>>> The only block drivers that can be converted into modules are the drivers
>>>>> that don't perform any init operation except for registering themselves. 
>>>>> This
>>>>> is why libiscsi has been disabled as a module.
>>>
>>> I don't think it has in this patch :) but you can also move the
>>> iscsi_opts registration to vl.c.
>>
>> Yeah I think Stefan mentioned this point in one of the earlier threads
>> but he said to do it in a separate commit, which I took to mean I
>> shouldn't include it here. Should I add it as a third part to this patch
>> series or leave it for a completely separate commit?
> 
> The patches in the series are left separate when including them in QEMU.
>  Therefore, a separate patch *is* (or will become :)) a separate commit.

Yep, mostly just wasn't sure whether it was considered to be related
enough to include with the other two.
> 
> Therefore, putting the change before this patch, or alternatively as the
> first patch in the series, will be fine.
> 
> Thanks,
> 
> Paolo
> 

Sounds good, I'll work on putting that in the next version then.

Colin

Re: [Qemu-block] [Qemu-devel] [PATCH 2/2] blockdev: Add dynamic module loading for block drivers

[Colin Lord]

On 06/15/2016 06:50 PM, Paolo Bonzini wrote:
> 
> 
> On 15/06/2016 20:40, Colin Lord wrote:
>>
>> The only block drivers that can be converted into modules are the drivers
>> that don't perform any init operation except for registering themselves. This
>> is why libiscsi has been disabled as a module.
> 
> I don't think it has in this patch :) but you can also move the
> iscsi_opts registration to vl.c.
> 
> Paolo
> 

Yeah I think Stefan mentioned this point in one of the earlier threads
but he said to do it in a separate commit, which I took to mean I
shouldn't include it here. Should I add it as a third part to this patch
series or leave it for a completely separate commit?

Colin