It's finally time to implement one of the TODO items: we want to
support a thread pool of parallel readers from the client, in
order to allow multiple in-flight operations with potential
out-of-order completion. We also need at least one plugin that
supports parallel processing for testing the opt
Once we have multiple threads during parallel processing, we need
to be sure that any I/O error flagged by one thread prevents the
next thread from attempting I/O. Although we already have a
separate lock for reads and writes, it's easier if status is
shared by both actions, which needs yet anothe
I want to make my nbd forwarding plugin fully parallel - but to do
that, I first need to make nbdkit itself fully parallel ;)
With this series, I was finally able to demonstrate out-of-order
responses when using qemu-io (which is great at sending back-to-back
requests prior to waiting for response
Finish plumbing up everything we will need to process multiple
client requests in parallel after handshake is complete. Since
status is now global, and properly protected by a mutex, all
of the threads will eventually quit as soon as any of them
notices EOF or nbdkit detects a signal.
For ease of
In preparation for parallel processing, we need to be sure that
two threads belonging to the same connection cannot interleave
their I/O except at message boundaries. Add a mutex around
all reads and writes that must occur as a group (for now, there
is no contention for either mutex).
Signed-off-
If nbdkit receives SIGINT while we are reading from the client,
it's better to avoid calling into the plugin for any work that
will further stall our response to the signal, and instead just
immediately give the client an ESHUTDOWN error.
Signed-off-by: Eric Blake
---
src/connections.c | 11
_handle_request() was never returning anything but 0; and had to
communicate the error value through a parameter by reference.
As there is no return value that a plugin can use to declare the
connection useless, we can consolidate this into just returning
the error value. Meanwhile, we don't need
It is never a wise idea to pass stack-allocated storage to
another thread during pthread_create() unless you can guarantee
that the new thread will complete prior to the calling thread
returning and ending the lifetime of that storage. We were
violating this, with the result in a SEGV in the detac
These patches should be ready to go in now; I will also post my
work-in-progress for enabling full parallel handling that depends
on these, but with that series, I was still getting crashes or
hangs with test-socket-activation (I think I've nailed all the
crashes I've seen, but the hang is rather i
Directly returning storage in the module is risky, because it
forces callers to strdup() if they aren't sure of the lifetime.
Copying into the heap up front means that plugin_name() can
now be safely used even if unload happens in the meantime.
As long as we have only a single plugin in static sto
We can't guarantee what storage duration the caller's request
for a thread name has; and in fact, if the caller uses
plugin_name() for their thread name, then the moment .unload
is called, our threadlocal storage is pointing to la-la-land
and we get a nice SEGV while trying to print any debug messa
Since we construct our error/debug messages via multiple calls
to stdio primitives, we are at risk of multiple threads interleaving
their output if they try to output at once. Add a mutex to group
related outputs into an atomic chunk.
Signed-off-by: Eric Blake
---
src/errors.c | 30
On 11/16/2017 05:53 AM, Richard W.M. Jones wrote:
>
> ACK series, with one minor change:
>
> Please add a reference to the new manual page in the ‘SEE ALSO’
> section at the bottom of the following two files:
>
> docs/nbdkit.pod
This one is obvious; I also added:
diff --git i/docs/nbdkit.pod
On 11/14/2017 11:30 AM, Richard W.M. Jones wrote:
> This fixes the race conditions for me, using the test described here:
>
> https://www.redhat.com/archives/libguestfs/2017-September/msg00226.html
Running test-socket-activation in a loop, I've hit other races (some
provoked a bit more easily wit
On Thu, Nov 16, 2017 at 3:10 PM, Richard W.M. Jones wrote:
> On Thu, Nov 16, 2017 at 02:47:46PM +, Stefan Hajnoczi wrote:
>> The threads you observed are the thread pool that performs
>> preadv(2)/pwritev(2) syscalls. The Linux AIO API could be used instead
>> and does not use threads for rea
On Thu, Nov 16, 2017 at 3:12 PM, Richard W.M. Jones wrote:
> On Thu, Nov 16, 2017 at 03:51:29PM +0100, Peter Lieven wrote:
>> Afaik all writes to the same QCOW2 serialize because of the s->lock that
>> is held during the write. So its not suprising that there is no benefit from
>> mutliple threads
On Thu, Nov 16, 2017 at 05:30:48PM +, Stefan Hajnoczi wrote:
> On Thu, Nov 16, 2017 at 3:10 PM, Richard W.M. Jones wrote:
> > Both local filesystems, but on different SATA devices.
>
> Okay. I'm curious what the strace -f output looks like (only the
> preadv(2)/pwritev(2) syscalls are intere
On 11/16/2017 05:49 AM, Richard W.M. Jones wrote:
> On Wed, Nov 15, 2017 at 10:50:06AM -0600, Eric Blake wrote:
>> Tell automake that we want ./nbdkit to be kept up-to-date in the
>> developer's tree. Otherwise, after touching nbdkit.in, you would
>> have to remember to manually run 'make nbdkit'.
On Thu, Nov 16, 2017 at 02:47:46PM +, Stefan Hajnoczi wrote:
> The threads you observed are the thread pool that performs
> preadv(2)/pwritev(2) syscalls. The Linux AIO API could be used instead
> and does not use threads for read and write operations.
I guess if I used AIO then I wouldn't ge
On Thu, Nov 16, 2017 at 03:51:29PM +0100, Peter Lieven wrote:
> Afaik all writes to the same QCOW2 serialize because of the s->lock that
> is held during the write. So its not suprising that there is no benefit from
> mutliple threads as long as reading from the RAW file involves no delay.
> Which
On Wed, Nov 15, 2017 at 11:52:46AM +, Richard W.M. Jones wrote:
> [CC to qemu-devel since I'm obviously doing something wrong here,
> I'm just not sure what.]
>
> I was getting ready to add multiple threads to ‘qemu-img convert’ (the
> longest part of v2v conversions) when I noticed that it ha
Am 16.11.2017 um 15:47 schrieb Stefan Hajnoczi:
On Wed, Nov 15, 2017 at 11:52:46AM +, Richard W.M. Jones wrote:
[CC to qemu-devel since I'm obviously doing something wrong here,
I'm just not sure what.]
I was getting ready to add multiple threads to ‘qemu-img convert’ (the
longest part of v
On 11/16/2017 05:53 AM, Richard W.M. Jones wrote:
>
> ACK series, with one minor change:
>
> Please add a reference to the new manual page in the ‘SEE ALSO’
> section at the bottom of the following two files:
>
> docs/nbdkit.pod
> docs/nbdkit-plugin.pod
>
> It should be pretty clear what to
On 11/14/2017 01:23 PM, Eric Blake wrote:
> In preparation for allowing interleaved response, refactor the
> nbd forwarder so that writes are still done from the thread
> handling the original request from the client, but all reads are
> done by a dedicated reader thread. Control between the two
>
On 2017-11-16 11:08, Gandalf Corvotempesta wrote:
> 2017-11-15 23:55 GMT+01:00 Max Reitz :
>> https://xanclic.moe/convert-xva.rb -- does this work?
>> (It seems to works on the two example images I found...)
>>
>> An example is in the code, you use it like this:
>>
>> $ ./convert-xva.rb ~/Downloads
On 11/14/2017 01:23 PM, Eric Blake wrote:
> This is a minimal implementation of an NBD forwarder; it lets us
> convert between old and newstyle connections (great if a client
> expects one style but the real server only provides the other),
> or add TLS safety on top of a server without having to r
I think it's even possible to modify that shell script to pipe into
nbdkit and from there to ‘qemu-img convert’. I'm too lazy to actually
do that right now, but the basic idea is here:
https://rwmj.wordpress.com/2014/10/14/streaming-nbd-server/
You'll probably have to add ‘-m 1’ to the qemu-im
On Thu, 16 Nov 2017 13:56:16 +0100
Tomáš Golembiovský wrote:
> On Wed, 15 Nov 2017 21:41:20 +0100
> Gandalf Corvotempesta wrote:
>
> > 2017-11-15 21:29 GMT+01:00 Richard W.M. Jones :
> > > Gandalf, is there an XVA file publically available (pref. not
> > > too big) that we can look at?
> >
On Wed, 15 Nov 2017 21:41:20 +0100
Gandalf Corvotempesta wrote:
> 2017-11-15 21:29 GMT+01:00 Richard W.M. Jones :
> > Gandalf, is there an XVA file publically available (pref. not
> > too big) that we can look at?
>
> I can try to provide one, but it's simple:
>
> # tar tvf 20160630_124823_aa72
On Friday, 6 October 2017 15:52:16 CET Richard W.M. Jones wrote:
> If running the external command fails in "argv mode" (ie. when
> not using the shell), then exit with either 126 or 127 as defined
> by POSIX.
>
> This is mostly the same as what bash does, see
> execute_cmd.c:shell_execve in the b
On Sunday, 5 November 2017 20:42:41 CET Richard W.M. Jones wrote:
> This safe wrapper around Unix.openfile ensures that exceptions
> escaping cannot leave unclosed files.
>
> There are only a few places in the code where this wrapper can be used
> currently. There are other occurences of Unix.ope
On Sunday, 5 November 2017 20:42:40 CET Richard W.M. Jones wrote:
> These safe wrappers around Pervasives.open_in and Pervasives.open_out
> ensure that exceptions escaping cannot leave unclosed files.
> ---
Mostly LGTM, just one note/improvement.
> diff --git a/v2v/input_libvirt_vddk.ml b/v2v/inp
ACK series, with one minor change:
Please add a reference to the new manual page in the ‘SEE ALSO’
section at the bottom of the following two files:
docs/nbdkit.pod
docs/nbdkit-plugin.pod
It should be pretty clear what to change if you look at those files.
Thanks,
Rich.
--
Richard Jones
On Wed, Nov 15, 2017 at 10:50:06AM -0600, Eric Blake wrote:
> Tell automake that we want ./nbdkit to be kept up-to-date in the
> developer's tree. Otherwise, after touching nbdkit.in, you would
> have to remember to manually run 'make nbdkit'.
>
> Signed-off-by: Eric Blake
> ---
> Makefile.am |
On Wed, Nov 15, 2017 at 04:52:37PM -0600, Eric Blake wrote:
> No need to duplicate maintenance of export flag computation between
> old and new style handshakes.
>
> Signed-off-by: Eric Blake
> ---
> src/connections.c | 120
> +-
> 1 file chan
On Wed, Nov 15, 2017 at 04:16:49PM -0600, Eric Blake wrote:
> We had several inconsistencies from the NBD spec when diagnosing
> bad client messages:
> - FLUSH is not generally forbidden on a read-only export (so failing
> with EPERM is wrong) [meanwhile, if we don't advertise flush because
> plugi
On Thu, Nov 16, 2017 at 11:07:10AM +0100, Gandalf Corvotempesta wrote:
> 2017-11-16 11:01 GMT+01:00 Richard W.M. Jones :
> > As mentioned before you can use this to do a qemu-img convert using
> > captive nbdkit:
> >
> > $ nbdkit -U - \
> > perl script=./xva-reader.pl file=./debian8cloud.xv
2017-11-15 23:55 GMT+01:00 Max Reitz :
> https://xanclic.moe/convert-xva.rb -- does this work?
> (It seems to works on the two example images I found...)
>
> An example is in the code, you use it like this:
>
> $ ./convert-xva.rb ~/Downloads/stats-appliance-2.36.020502.xva Ref:73
It doesn't work
2017-11-16 11:01 GMT+01:00 Richard W.M. Jones :
> As mentioned before you can use this to do a qemu-img convert using
> captive nbdkit:
>
> $ nbdkit -U - \
> perl script=./xva-reader.pl file=./debian8cloud.xva size=4294967296 \
> --run 'qemu-img convert -f raw $nbd -O qcow2 /var/tmp/o
Here's my solution, as a nbdkit plugin written in Perl.
As with Max's solution I don't bother to parse the virtual size out of
the XML file, so you need to specify that on the command line
otherwise the disk will be truncated to the largest extent stored in
the file. Also the ‘.xva’ file must not
40 matches
Mail list logo
