Re: [PATCH v8 1/5] QIOChannel: Add flags on io_writev and introduce io_flush callback
Thanks for reviewing, Juan! On Fri, Feb 18, 2022 at 1:36 PM Juan Quintela wrote: > > Leonardo Bras wrote: > > Add flags to io_writev and introduce io_flush as optional callback to > > QIOChannelClass, allowing the implementation of zero copy writes by > > subclasses. > > > > How to use them: > > - Write data using > > qio_channel_writev*(...,QIO_CHANNEL_WRITE_FLAG_ZERO_COPY), > > - Wait write completion with qio_channel_flush(). > > > > Notes: > > As some zero copy write implementations work asynchronously, it's > > recommended to keep the write buffer untouched until the return of > > qio_channel_flush(), to avoid the risk of sending an updated buffer > > instead of the buffer state during write. > > > > As io_flush callback is optional, if a subclass does not implement it, then: > > - io_flush will return 0 without changing anything. > > > > Also, some functions like qio_channel_writev_full_all() were adapted to > > receive a flag parameter. That allows shared code between zero copy and > > non-zero copy writev, and also an easier implementation on new flags. > > > > Signed-off-by: Leonardo Bras > > Reviewed-by: Juan Quintela > > As everybody pointed out about the missing assertion... >
Re: [PATCH v8 1/5] QIOChannel: Add flags on io_writev and introduce io_flush callback
Leonardo Bras wrote: > Add flags to io_writev and introduce io_flush as optional callback to > QIOChannelClass, allowing the implementation of zero copy writes by > subclasses. > > How to use them: > - Write data using qio_channel_writev*(...,QIO_CHANNEL_WRITE_FLAG_ZERO_COPY), > - Wait write completion with qio_channel_flush(). > > Notes: > As some zero copy write implementations work asynchronously, it's > recommended to keep the write buffer untouched until the return of > qio_channel_flush(), to avoid the risk of sending an updated buffer > instead of the buffer state during write. > > As io_flush callback is optional, if a subclass does not implement it, then: > - io_flush will return 0 without changing anything. > > Also, some functions like qio_channel_writev_full_all() were adapted to > receive a flag parameter. That allows shared code between zero copy and > non-zero copy writev, and also an easier implementation on new flags. > > Signed-off-by: Leonardo Bras Reviewed-by: Juan Quintela As everybody pointed out about the missing assertion...
Re: [PATCH v8 1/5] QIOChannel: Add flags on io_writev and introduce io_flush callback
Hello Peter, On Mon, Feb 7, 2022 at 9:50 AM Peter Xu wrote: > > On Tue, Feb 01, 2022 at 03:28:59AM -0300, Leonardo Bras wrote: > > Add flags to io_writev and introduce io_flush as optional callback to > > QIOChannelClass, allowing the implementation of zero copy writes by > > subclasses. > > > > How to use them: > > - Write data using > > qio_channel_writev*(...,QIO_CHANNEL_WRITE_FLAG_ZERO_COPY), > > - Wait write completion with qio_channel_flush(). > > > > Notes: > > As some zero copy write implementations work asynchronously, it's > > recommended to keep the write buffer untouched until the return of > > qio_channel_flush(), to avoid the risk of sending an updated buffer > > instead of the buffer state during write. > > > > As io_flush callback is optional, if a subclass does not implement it, then: > > - io_flush will return 0 without changing anything. > > > > Also, some functions like qio_channel_writev_full_all() were adapted to > > receive a flag parameter. That allows shared code between zero copy and > > non-zero copy writev, and also an easier implementation on new flags. > > > > Signed-off-by: Leonardo Bras > > With Dan's comment addressed on removing the redundant assertion: > > Reviewed-by: Peter Xu > Thank you for reviewing! I think I am now missing reviewing only on patch 5/5 before sending the next version. Could you and/or Daniel help me with that? Just to check if I am missing anything? Best regards, Leo
Re: [PATCH v8 1/5] QIOChannel: Add flags on io_writev and introduce io_flush callback
On Tue, Feb 01, 2022 at 03:28:59AM -0300, Leonardo Bras wrote: > Add flags to io_writev and introduce io_flush as optional callback to > QIOChannelClass, allowing the implementation of zero copy writes by > subclasses. > > How to use them: > - Write data using qio_channel_writev*(...,QIO_CHANNEL_WRITE_FLAG_ZERO_COPY), > - Wait write completion with qio_channel_flush(). > > Notes: > As some zero copy write implementations work asynchronously, it's > recommended to keep the write buffer untouched until the return of > qio_channel_flush(), to avoid the risk of sending an updated buffer > instead of the buffer state during write. > > As io_flush callback is optional, if a subclass does not implement it, then: > - io_flush will return 0 without changing anything. > > Also, some functions like qio_channel_writev_full_all() were adapted to > receive a flag parameter. That allows shared code between zero copy and > non-zero copy writev, and also an easier implementation on new flags. > > Signed-off-by: Leonardo Bras With Dan's comment addressed on removing the redundant assertion: Reviewed-by: Peter Xu -- Peter Xu
Re: [PATCH v8 1/5] QIOChannel: Add flags on io_writev and introduce io_flush callback
Hello Daniel, thanks for reviewing!
On Tue, Feb 1, 2022 at 6:35 AM Daniel P. Berrangé wrote:
>
> On Tue, Feb 01, 2022 at 03:28:59AM -0300, Leonardo Bras wrote:
> > Add flags to io_writev and introduce io_flush as optional callback to
> > QIOChannelClass, allowing the implementation of zero copy writes by
> > subclasses.
> >
> > How to use them:
> > - Write data using
> > qio_channel_writev*(...,QIO_CHANNEL_WRITE_FLAG_ZERO_COPY),
> > - Wait write completion with qio_channel_flush().
> >
> > Notes:
> > As some zero copy write implementations work asynchronously, it's
> > recommended to keep the write buffer untouched until the return of
> > qio_channel_flush(), to avoid the risk of sending an updated buffer
> > instead of the buffer state during write.
> >
> > As io_flush callback is optional, if a subclass does not implement it, then:
> > - io_flush will return 0 without changing anything.
> >
> > Also, some functions like qio_channel_writev_full_all() were adapted to
> > receive a flag parameter. That allows shared code between zero copy and
> > non-zero copy writev, and also an easier implementation on new flags.
> >
> > Signed-off-by: Leonardo Bras
> > ---
> > include/io/channel.h| 38 -
> > chardev/char-io.c | 2 +-
> > hw/remote/mpqemu-link.c | 2 +-
> > io/channel-buffer.c | 1 +
> > io/channel-command.c| 1 +
> > io/channel-file.c | 1 +
> > io/channel-socket.c | 2 ++
> > io/channel-tls.c| 1 +
> > io/channel-websock.c| 1 +
> > io/channel.c| 53 +++--
> > migration/rdma.c| 1 +
> > scsi/pr-manager-helper.c| 2 +-
> > tests/unit/test-io-channel-socket.c | 1 +
> > 13 files changed, 92 insertions(+), 14 deletions(-)
> >
> > diff --git a/io/channel.c b/io/channel.c
> > index e8b019dc36..b8b99fdc4c 100644
> > --- a/io/channel.c
> > +++ b/io/channel.c
> > @@ -72,18 +72,32 @@ ssize_t qio_channel_writev_full(QIOChannel *ioc,
> > size_t niov,
> > int *fds,
> > size_t nfds,
> > +int flags,
> > Error **errp)
> > {
> > QIOChannelClass *klass = QIO_CHANNEL_GET_CLASS(ioc);
> >
> > -if ((fds || nfds) &&
> > -!qio_channel_has_feature(ioc, QIO_CHANNEL_FEATURE_FD_PASS)) {
> > +if (fds || nfds) {
> > +if (!qio_channel_has_feature(ioc, QIO_CHANNEL_FEATURE_FD_PASS)) {
> > +error_setg_errno(errp, EINVAL,
> > + "Channel does not support file descriptor
> > passing");
> > +return -1;
> > +}
> > +if (flags & QIO_CHANNEL_WRITE_FLAG_ZERO_COPY) {
> > +error_setg_errno(errp, EINVAL,
> > + "Zero Copy does not support file descriptor
> > passing");
> > +return -1;
> > +}
>
> Here you gracefully reject FD passing when zero copy is requested
> which is good.
>
> > +}
> > +
>
> > @@ -235,10 +249,16 @@ int qio_channel_writev_full_all(QIOChannel *ioc,
> >iov, niov,
> >0, iov_size(iov, niov));
> >
> > +if (flags & QIO_CHANNEL_WRITE_FLAG_ZERO_COPY) {
> > +assert(fds == NULL && nfds == 0);
> > +}
>
> But here you abort QEMU if FD passing is requested when zero copy
> is set.
>
> AFAICT, if you just delete this assert, the code to gracefully
> report errors will do the right thing.
Yeah, thatś right. This test is unnecessary since qio_channel_writev_full()
will be called and will return error if fds + zerocopy happens.
Good catch!
>
> Without the assert:
>
> Reviewed-by: Daniel P. Berrangé
>
Thanks!
I will wait for more feedback on other patches before sending the v9,
but it should not take too long this time.
Best regards,
Leo
Re: [PATCH v8 1/5] QIOChannel: Add flags on io_writev and introduce io_flush callback
On Tue, Feb 01, 2022 at 03:28:59AM -0300, Leonardo Bras wrote:
> Add flags to io_writev and introduce io_flush as optional callback to
> QIOChannelClass, allowing the implementation of zero copy writes by
> subclasses.
>
> How to use them:
> - Write data using qio_channel_writev*(...,QIO_CHANNEL_WRITE_FLAG_ZERO_COPY),
> - Wait write completion with qio_channel_flush().
>
> Notes:
> As some zero copy write implementations work asynchronously, it's
> recommended to keep the write buffer untouched until the return of
> qio_channel_flush(), to avoid the risk of sending an updated buffer
> instead of the buffer state during write.
>
> As io_flush callback is optional, if a subclass does not implement it, then:
> - io_flush will return 0 without changing anything.
>
> Also, some functions like qio_channel_writev_full_all() were adapted to
> receive a flag parameter. That allows shared code between zero copy and
> non-zero copy writev, and also an easier implementation on new flags.
>
> Signed-off-by: Leonardo Bras
> ---
> include/io/channel.h| 38 -
> chardev/char-io.c | 2 +-
> hw/remote/mpqemu-link.c | 2 +-
> io/channel-buffer.c | 1 +
> io/channel-command.c| 1 +
> io/channel-file.c | 1 +
> io/channel-socket.c | 2 ++
> io/channel-tls.c| 1 +
> io/channel-websock.c| 1 +
> io/channel.c| 53 +++--
> migration/rdma.c| 1 +
> scsi/pr-manager-helper.c| 2 +-
> tests/unit/test-io-channel-socket.c | 1 +
> 13 files changed, 92 insertions(+), 14 deletions(-)
>
> diff --git a/io/channel.c b/io/channel.c
> index e8b019dc36..b8b99fdc4c 100644
> --- a/io/channel.c
> +++ b/io/channel.c
> @@ -72,18 +72,32 @@ ssize_t qio_channel_writev_full(QIOChannel *ioc,
> size_t niov,
> int *fds,
> size_t nfds,
> +int flags,
> Error **errp)
> {
> QIOChannelClass *klass = QIO_CHANNEL_GET_CLASS(ioc);
>
> -if ((fds || nfds) &&
> -!qio_channel_has_feature(ioc, QIO_CHANNEL_FEATURE_FD_PASS)) {
> +if (fds || nfds) {
> +if (!qio_channel_has_feature(ioc, QIO_CHANNEL_FEATURE_FD_PASS)) {
> +error_setg_errno(errp, EINVAL,
> + "Channel does not support file descriptor
> passing");
> +return -1;
> +}
> +if (flags & QIO_CHANNEL_WRITE_FLAG_ZERO_COPY) {
> +error_setg_errno(errp, EINVAL,
> + "Zero Copy does not support file descriptor
> passing");
> +return -1;
> +}
Here you gracefully reject FD passing when zero copy is requested
which is good.
> +}
> +
> @@ -235,10 +249,16 @@ int qio_channel_writev_full_all(QIOChannel *ioc,
>iov, niov,
>0, iov_size(iov, niov));
>
> +if (flags & QIO_CHANNEL_WRITE_FLAG_ZERO_COPY) {
> +assert(fds == NULL && nfds == 0);
> +}
But here you abort QEMU if FD passing is requested when zero copy
is set.
AFAICT, if you just delete this assert, the code to gracefully
report errors will do the right thing.
Without the assert:
Reviewed-by: Daniel P. Berrangé
Regards,
Daniel
--
|: https://berrange.com -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
[PATCH v8 1/5] QIOChannel: Add flags on io_writev and introduce io_flush callback
Add flags to io_writev and introduce io_flush as optional callback to
QIOChannelClass, allowing the implementation of zero copy writes by
subclasses.
How to use them:
- Write data using qio_channel_writev*(...,QIO_CHANNEL_WRITE_FLAG_ZERO_COPY),
- Wait write completion with qio_channel_flush().
Notes:
As some zero copy write implementations work asynchronously, it's
recommended to keep the write buffer untouched until the return of
qio_channel_flush(), to avoid the risk of sending an updated buffer
instead of the buffer state during write.
As io_flush callback is optional, if a subclass does not implement it, then:
- io_flush will return 0 without changing anything.
Also, some functions like qio_channel_writev_full_all() were adapted to
receive a flag parameter. That allows shared code between zero copy and
non-zero copy writev, and also an easier implementation on new flags.
Signed-off-by: Leonardo Bras
---
include/io/channel.h| 38 -
chardev/char-io.c | 2 +-
hw/remote/mpqemu-link.c | 2 +-
io/channel-buffer.c | 1 +
io/channel-command.c| 1 +
io/channel-file.c | 1 +
io/channel-socket.c | 2 ++
io/channel-tls.c| 1 +
io/channel-websock.c| 1 +
io/channel.c| 53 +++--
migration/rdma.c| 1 +
scsi/pr-manager-helper.c| 2 +-
tests/unit/test-io-channel-socket.c | 1 +
13 files changed, 92 insertions(+), 14 deletions(-)
diff --git a/include/io/channel.h b/include/io/channel.h
index 88988979f8..c680ee7480 100644
--- a/include/io/channel.h
+++ b/include/io/channel.h
@@ -32,12 +32,15 @@ OBJECT_DECLARE_TYPE(QIOChannel, QIOChannelClass,
#define QIO_CHANNEL_ERR_BLOCK -2
+#define QIO_CHANNEL_WRITE_FLAG_ZERO_COPY 0x1
+
typedef enum QIOChannelFeature QIOChannelFeature;
enum QIOChannelFeature {
QIO_CHANNEL_FEATURE_FD_PASS,
QIO_CHANNEL_FEATURE_SHUTDOWN,
QIO_CHANNEL_FEATURE_LISTEN,
+QIO_CHANNEL_FEATURE_WRITE_ZERO_COPY,
};
@@ -104,6 +107,7 @@ struct QIOChannelClass {
size_t niov,
int *fds,
size_t nfds,
+ int flags,
Error **errp);
ssize_t (*io_readv)(QIOChannel *ioc,
const struct iovec *iov,
@@ -136,6 +140,8 @@ struct QIOChannelClass {
IOHandler *io_read,
IOHandler *io_write,
void *opaque);
+int (*io_flush)(QIOChannel *ioc,
+Error **errp);
};
/* General I/O handling functions */
@@ -228,6 +234,7 @@ ssize_t qio_channel_readv_full(QIOChannel *ioc,
* @niov: the length of the @iov array
* @fds: an array of file handles to send
* @nfds: number of file handles in @fds
+ * @flags: write flags (QIO_CHANNEL_WRITE_FLAG_*)
* @errp: pointer to a NULL-initialized error object
*
* Write data to the IO channel, reading it from the
@@ -260,6 +267,7 @@ ssize_t qio_channel_writev_full(QIOChannel *ioc,
size_t niov,
int *fds,
size_t nfds,
+int flags,
Error **errp);
/**
@@ -837,6 +845,7 @@ int qio_channel_readv_full_all(QIOChannel *ioc,
* @niov: the length of the @iov array
* @fds: an array of file handles to send
* @nfds: number of file handles in @fds
+ * @flags: write flags (QIO_CHANNEL_WRITE_FLAG_*)
* @errp: pointer to a NULL-initialized error object
*
*
@@ -846,6 +855,14 @@ int qio_channel_readv_full_all(QIOChannel *ioc,
* to be written, yielding from the current coroutine
* if required.
*
+ * If QIO_CHANNEL_WRITE_FLAG_ZERO_COPY is passed in flags,
+ * instead of waiting for all requested data to be written,
+ * this function will wait until it's all queued for writing.
+ * In this case, if the buffer gets changed between queueing and
+ * sending, the updated buffer will be sent. If this is not a
+ * desired behavior, it's suggested to call qio_channel_flush()
+ * before reusing the buffer.
+ *
* Returns: 0 if all bytes were written, or -1 on error
*/
@@ -853,6 +870,25 @@ int qio_channel_writev_full_all(QIOChannel *ioc,
const struct iovec *iov,
size_t niov,
int *fds, size_t nfds,
-Error **errp);
+int flags, Error **errp);
+
+/**
+ * qio_channel_flush:
+ * @ioc: the channel object
+ * @errp: pointer to a NULL-initialized error object
+ *
+ * Will block until every packet queued with
+ * qio_channel_writev_full() + QIO_CHANNEL_WRITE_FLAG_ZERO_COPY
+ * is sent, or return in case of any error.
+ *
+ * If
