On Thu, Feb 02, 2023 at 10:04:20AM -0600, Eric Blake wrote: > On Wed, Feb 01, 2023 at 12:40:45PM +0000, Richard W.M. Jones wrote: > > Hi Eric, > > > > A couple of semi-related multi-conn questions. > > > > (1) nbdkit-curl-plugin does not currently advertise multi-conn. However > > forcing multi-conn makes it faster, eg: > > > > $ time nbdkit -r curl > > https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img > > timeout=2000 --run ' nbdcopy --no-extents -p $uri > > jammy-server-cloudimg-amd64.img ' > > █ 100% [****************************************] > > > > real 1m15.438s > > user 0m2.513s > > sys 0m4.063s > > I'm jealous of your connection. Even with bare wget (without nbdkit > in the middle), for some reason my computer is failing to connect when > trying IPv6. I then tried 'wget -4 ...', and didn't get the full file > until 5m37s. And when I tried to reproduce your nbdkit line, I got: > > nbdkit: curl[1]: error: pread: curl_easy_perform: Failed sending data to the > peer: Connection died, tried 5 times before giving up > nbdcopy: read at offset 245104640 failed: Input/output error > > at 6 minutes in. So slow speeds and some connection flakiness is not > helping me.
This was the original complaint in: https://github.com/kubevirt/containerized-data-importer/issues/2561 > > $ time nbdkit -r curl > > https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img > > --filter=multi-conn multi-conn-mode=unsafe timeout=2000 --run ' nbdcopy > > --no-extents -p $uri jammy-server-cloudimg-amd64.img ' > > Yay - I'm glad my multi-conn filter makes it easier to test things > like this! > > Should we tweak the docs in nbdkit-multi-conn-filter(1) to mention > that despite multi-conn-mode=unsafe being unsafe for a plugin that > does not have consistency, it is useful for timing tests on a plugin > where we suspect consistency is available in order to test timing to > see if it actually makes a difference? Yes, sounds good. > > █ 100% [****************************************] > > > > real 0m59.635s > > user 0m3.010s > > sys 0m4.538s > > > > (Note that this is line speed - wget also takes 59 seconds) > > > > I believe that multi-conn should be safe to enable in the curl plugin, > > certainly if we're in read-only mode (which is the most common case > > for this plugin). It's probably not safe when operating in read-write > > mode, because HTTP doesn't have any concept of flushing, but almost no > > web server supports writing. > > Agreed on both fronts; I see you have proposed this patch (and I acked > it) in the meantime. > > > > > (2) qemu's NBD client still doesn't seem to support multi-conn. This > > makes qemu-img convert very slow when reading from nbdkit, as in the > > example below. (Note the '.img' file on that website is actually a > > qcow2 file.) > > Correct - qemu does NOT do multi-queue (or multipath) client > connections. It is blindly using exactly one NBD client connection > per block device, and ignoring the server's advertisement of > multi-conn support. > > Teaching qemu to support a multipath paradigm is not a trivial task. > I'm wondering if the effort might be easier by teaching libblkio how > to do arbitrary multipath for any underlying block device, coupled > with current efforts to teach qemu how to target any arbitrary > libblkio target, so that qemu proper doesn't have to worry about the > intricacies of multipath; from there, we'd need a way for libblkio to > target NBD devices (probably by libnbd). In other words, instead of > trying to directly code client MULTI_CONN efforts into qemu proper, it > may be better to segregate things into more modular building blocks. > > > > > $ time nbdkit -r curl > > https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img > > --filter=multi-conn multi-conn-mode=unsafe timeout=2000 --run ' qemu-img > > convert -W -m 16 -p -f qcow2 $uri -O raw > > jammy-server-cloudimg-amd64.img.raw ' > > (100.00/100%) > > > > real 3m29.412s > > user 0m17.856s > > sys 0m8.218s > > > > Since we know from (1) above that copying could go at line speed when > > multi-conn is enabled, and also that qemu-img convert from the same > > local file takes only about 2 seconds, it seems like qemu-img convert > > above could go a lot faster if we allowed multi-conn. > > > > Is there any work on adding multi-conn support to qemu's NBD client? > > Not that I'm aware of at the moment, but we have proof that it may > prove fruitful to have someone spend time on. Kubernetes team are complaining that this particular case (stream qcow2 file from a website and convert it to raw without a temporary file) is slow. This is not a case that I've encountered before because it's not relevant to virt-v2v, but it does appear to be important. - * - * - I didn't want to complicate the original message with irrelevant stuff, but there's something else I want to mention now. If we just use qemu's curl driver (thus eliminating NBD & nbdkit from the mix), it gets even slower: $ time ~/d/qemu/build/qemu-img convert -p -W -f qcow2 'json:{ "file.readahead": 67108864, "file.driver": "http", "file.url": "http://oirase.annexia.org/tmp/jammy-server-cloudimg-amd64.qcow2";, "file.timeout":2000 }' -O raw jammy-server-cloudimg-amd64.img.raw (100.00/100%) real 3m53.923s user 0m13.751s sys 0m15.346s Now this is not something I'm personally concerned about (since I've long been arguing we should deprecate the qemu curl driver and use nbdkit), but it's also very surprising. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit _______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
