Re: [PATCH v3 0/3] QIOChannel async_write & async_flush + MSG_ZEROCOPY + multifd
On Wed, Sep 29, 2021 at 4:23 PM Peter Xu wrote: > > On Wed, Sep 29, 2021 at 03:34:01PM -0300, Leonardo Bras Soares Passos wrote: > > You mean like I did in pre-V1, enabling ZEROCOPY for multifd it in QMP? > > Or is this something else? > > I mean any way to still be able to turn zerocopy off? E.g. when the user has > no > privilege on mlock. Thanks, > > -- > Peter Xu > Yeah, that makes sense in the new approach of failing when zerocopy is not possible. I will make sure to do the in v3. Best regards, Leonardo
Re: [PATCH v3 0/3] QIOChannel async_write & async_flush + MSG_ZEROCOPY + multifd
On Wed, Sep 29, 2021 at 03:34:01PM -0300, Leonardo Bras Soares Passos wrote: > You mean like I did in pre-V1, enabling ZEROCOPY for multifd it in QMP? > Or is this something else? I mean any way to still be able to turn zerocopy off? E.g. when the user has no privilege on mlock. Thanks, -- Peter Xu
Re: [PATCH v3 0/3] QIOChannel async_write & async_flush + MSG_ZEROCOPY + multifd
Hello Peter, On Tue, Sep 28, 2021 at 7:51 PM Peter Xu wrote: > > On Wed, Sep 22, 2021 at 07:24:20PM -0300, Leonardo Bras wrote: > > This patch series intends to enable MSG_ZEROCOPY in QIOChannel, and make > > use of it for multifd migration performance improvement. > > > > Patch #1 creates new callbacks for QIOChannel, allowing the implementation > > of asynchronous writing. > > > > Patch #2 implements async_write and async_flush on QIOChannelSocket, > > > > Patch #3 Makes use of async_write + async_flush to enable MSG_ZEROCOPY > > for migration using multifd nocomp. > > > > Results: > > So far, the resource usage of __sys_sendmsg() reduced 15 times, and the > > overall migration took 13-18% less time, based in synthetic workload. > > > > The objective is to reduce migration time in hosts with heavy cpu usage. > > My previous memory is that we'll add a capability bit for migration, so it'll > not be enabled until user specified it. Plan to do it in the next version? You mean like I did in pre-V1, enabling ZEROCOPY for multifd it in QMP? Or is this something else? > > It'll still be okay if you want to separate the work of (1) qio channel > zero-copy support on sockets, and (2) apply zero copy to migration, then that > can be done in part 2. Your call. :) Ok :) > > Thanks, > > -- > Peter Xu > Thank you Peter, Best regards, Leonardo
Re: [PATCH v3 0/3] QIOChannel async_write & async_flush + MSG_ZEROCOPY + multifd
On Wed, Sep 22, 2021 at 07:24:20PM -0300, Leonardo Bras wrote: > This patch series intends to enable MSG_ZEROCOPY in QIOChannel, and make > use of it for multifd migration performance improvement. > > Patch #1 creates new callbacks for QIOChannel, allowing the implementation > of asynchronous writing. > > Patch #2 implements async_write and async_flush on QIOChannelSocket, > > Patch #3 Makes use of async_write + async_flush to enable MSG_ZEROCOPY > for migration using multifd nocomp. > > Results: > So far, the resource usage of __sys_sendmsg() reduced 15 times, and the > overall migration took 13-18% less time, based in synthetic workload. > > The objective is to reduce migration time in hosts with heavy cpu usage. My previous memory is that we'll add a capability bit for migration, so it'll not be enabled until user specified it. Plan to do it in the next version? It'll still be okay if you want to separate the work of (1) qio channel zero-copy support on sockets, and (2) apply zero copy to migration, then that can be done in part 2. Your call. :) Thanks, -- Peter Xu
[PATCH v3 0/3] QIOChannel async_write & async_flush + MSG_ZEROCOPY + multifd
This patch series intends to enable MSG_ZEROCOPY in QIOChannel, and make use of it for multifd migration performance improvement. Patch #1 creates new callbacks for QIOChannel, allowing the implementation of asynchronous writing. Patch #2 implements async_write and async_flush on QIOChannelSocket, Patch #3 Makes use of async_write + async_flush to enable MSG_ZEROCOPY for migration using multifd nocomp. Results: So far, the resource usage of __sys_sendmsg() reduced 15 times, and the overall migration took 13-18% less time, based in synthetic workload. The objective is to reduce migration time in hosts with heavy cpu usage. --- Changes since v2: - Patch #1: One more fallback - Patch #2: Fall back to sync if fails to lock buffer memory in MSG_ZEROCOPY send. Changes since v1: - Reimplemented the patchset using async_write + async_flush approach. - Implemented a flush to be able to tell whenever all data was written. Leonardo Bras (3): QIOChannel: Add io_async_writev & io_async_flush callbacks QIOChannelSocket: Implement io_async_write & io_async_flush multifd: Send using asynchronous write on nocomp to send RAM pages. include/io/channel-socket.h | 2 + include/io/channel.h| 94 --- io/channel-socket.c | 176 ++-- io/channel.c| 66 +++--- migration/multifd.c | 3 +- 5 files changed, 300 insertions(+), 41 deletions(-) -- 2.33.0