On 01/11/22 08:00, Laszlo Ersek wrote: > On 01/10/22 16:52, Richard W.M. Jones wrote: >> >> For the raw format local disk to local disk conversion, it's possible >> to regain most of the performance by adding >> --request-size=$(( 16 * 1024 * 1024 )) to the nbdcopy command. The >> patch below is not suitable for going upstream but it can be used for >> testing: >> >> diff --git a/v2v/v2v.ml b/v2v/v2v.ml >> index 47e6e937..ece3b7d9 100644 >> --- a/v2v/v2v.ml >> +++ b/v2v/v2v.ml >> @@ -613,6 +613,7 @@ and nbdcopy output_alloc input_uri output_uri = >> let cmd = ref [] in >> List.push_back_list cmd [ "nbdcopy"; input_uri; output_uri ]; >> List.push_back cmd "--flush"; >> + List.push_back cmd "--request-size=16777216"; >> (*List.push_back cmd "--verbose";*) >> if not (quiet ()) then List.push_back cmd "--progress"; >> if output_alloc = Types.Preallocated then List.push_back cmd >> "--allocated"; >> >> The problem is of course this is a pessimisation for other >> conversions. It's known to make at least qcow2 to qcow2, and all VDDK >> conversions worse. So we'd have to make it conditional on doing a raw >> format local conversion, which is a pretty ugly hack. Even worse, the >> exact size (16M) varies for me when I test this on different machines >> and HDDs vs SSDs. On my very fast AMD machine with an SSD, the >> nbdcopy default request size (256K) is fastest and larger sizes are a >> very slightly slower. >> >> I can imagine an "adaptive nbdcopy" which adjusts these parameters >> while copying in order to find the best performance. A little bit >> hard to implement ... >> >> I'm also still wondering exactly why a larger request size is better >> in this case. You can easily reproduce the effect using the attached >> test script and adjusting --request-size. You'll need to build the >> standard test guest, see part 1. > > (The following thought occurred to me last evening.) > > In modular v2v, we use multi-threaded nbdkit instances, and > multi-threaded nbdcopy instances. (IIUC.) I think: that should result in > quite a bit of thrashing, on both source and destination disks, no? That > should be especially visible on HDDs, but perhaps also on SSDs > (dependent on request size as you mention above). > > The worst is likely when both nbdcopy processes operate on the same > physical HDD (i.e., spinning rust). > > qemu-img is single-threaded, so even if reads from and writes to the > same physical hard disk, it kind of generates two "parallel" request > streams, which both the disk and the kernel's IO scheduler could cope > with more easily. According to the nbdcopy manual, the default thread > count is "number of processor cores available", the "sliding window of > requests" with a high thread count is likely undistinguishable from real > random access. > > Also I (vaguely?) gather that nbdcopy bypasses the page cache (or does > it only sync automatically at the end? I don't remember). If the page > cache is avoided, then the page cache has no chance to mitigate the > thrashing, especially on HDDs -- but even on SSDs, if the drive's > internal cache is not large enough (considering the individual request > size and the number of random requests flying in parallel), the > degradation should be visible. > > Can you tweak (i.e., lower) the thread count of both nbdcopy processes; > let's say to "1", for starters?
I meant to add: the "--request-size=16777216" option, *where it helps*, effectively serializes the request stream. A single request is not broken up into smaller *parallel* requests, thus, if you have huge (like 16MiB) requests, then that effectively de-randomizes the accesses in the sliding window. Thanks Laszlo _______________________________________________ Libguestfs mailing list [email protected] https://listman.redhat.com/mailman/listinfo/libguestfs
