Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-04-21 Thread Peter Xu
On Fri, Apr 21, 2023 at 08:48:02AM +0100, Daniel P. Berrangé wrote: > On Thu, Apr 20, 2023 at 03:19:39PM -0400, Peter Xu wrote: > > On Thu, Apr 20, 2023 at 10:02:43AM +0100, Daniel P. Berrangé wrote: > > > On Wed, Apr 19, 2023 at 03:07:19PM -0400, Peter Xu wrote: > > > > On Wed, Apr 19, 2023 at

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-04-21 Thread Daniel P . Berrangé
On Thu, Apr 20, 2023 at 03:19:39PM -0400, Peter Xu wrote: > On Thu, Apr 20, 2023 at 10:02:43AM +0100, Daniel P. Berrangé wrote: > > On Wed, Apr 19, 2023 at 03:07:19PM -0400, Peter Xu wrote: > > > On Wed, Apr 19, 2023 at 06:12:05PM +0100, Daniel P. Berrangé wrote: > > > > On Tue, Apr 18, 2023 at

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-04-20 Thread Peter Xu
On Thu, Apr 20, 2023 at 10:02:43AM +0100, Daniel P. Berrangé wrote: > On Wed, Apr 19, 2023 at 03:07:19PM -0400, Peter Xu wrote: > > On Wed, Apr 19, 2023 at 06:12:05PM +0100, Daniel P. Berrangé wrote: > > > On Tue, Apr 18, 2023 at 03:26:45PM -0400, Peter Xu wrote: > > > > On Tue, Apr 18, 2023 at

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-04-20 Thread Daniel P . Berrangé
On Wed, Apr 19, 2023 at 03:07:19PM -0400, Peter Xu wrote: > On Wed, Apr 19, 2023 at 06:12:05PM +0100, Daniel P. Berrangé wrote: > > On Tue, Apr 18, 2023 at 03:26:45PM -0400, Peter Xu wrote: > > > On Tue, Apr 18, 2023 at 05:58:44PM +0100, Daniel P. Berrangé wrote: > > > > Libvirt has multiple APIs

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-04-19 Thread Peter Xu
On Wed, Apr 19, 2023 at 06:12:05PM +0100, Daniel P. Berrangé wrote: > On Tue, Apr 18, 2023 at 03:26:45PM -0400, Peter Xu wrote: > > On Tue, Apr 18, 2023 at 05:58:44PM +0100, Daniel P. Berrangé wrote: > > > Libvirt has multiple APIs where it currently uses its migrate-to-file > > > approach > > >

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-04-19 Thread Daniel P . Berrangé
On Tue, Apr 18, 2023 at 03:26:45PM -0400, Peter Xu wrote: > On Tue, Apr 18, 2023 at 05:58:44PM +0100, Daniel P. Berrangé wrote: > > Libvirt has multiple APIs where it currently uses its migrate-to-file > > approach > > > > * virDomainManagedSave() > > > > This saves VM state to an libvirt

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-04-18 Thread Peter Xu
On Tue, Apr 18, 2023 at 05:58:44PM +0100, Daniel P. Berrangé wrote: > On Fri, Mar 31, 2023 at 12:27:48PM -0400, Peter Xu wrote: > > On Fri, Mar 31, 2023 at 05:10:16PM +0100, Daniel P. Berrangé wrote: > > > On Fri, Mar 31, 2023 at 11:55:03AM -0400, Peter Xu wrote: > > > > On Fri, Mar 31, 2023 at

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-04-18 Thread Daniel P . Berrangé
On Fri, Mar 31, 2023 at 12:27:48PM -0400, Peter Xu wrote: > On Fri, Mar 31, 2023 at 05:10:16PM +0100, Daniel P. Berrangé wrote: > > On Fri, Mar 31, 2023 at 11:55:03AM -0400, Peter Xu wrote: > > > On Fri, Mar 31, 2023 at 12:30:45PM -0300, Fabiano Rosas wrote: > > > > Peter Xu writes: > > > > > >

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-04-11 Thread Peter Xu
On Fri, Apr 07, 2023 at 12:36:24PM +0200, Claudio Fontana wrote: > On 4/6/23 18:46, Fabiano Rosas wrote: > > Peter Xu writes: > > > >> On Tue, Apr 04, 2023 at 05:10:52PM +0200, Claudio Fontana wrote: > >>> On 4/4/23 16:53, Peter Xu wrote: > On Tue, Apr 04, 2023 at 10:00:16AM +0200, Claudio

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-04-07 Thread Claudio Fontana
On 4/6/23 18:46, Fabiano Rosas wrote: > Peter Xu writes: > >> On Tue, Apr 04, 2023 at 05:10:52PM +0200, Claudio Fontana wrote: >>> On 4/4/23 16:53, Peter Xu wrote: On Tue, Apr 04, 2023 at 10:00:16AM +0200, Claudio Fontana wrote: > Hi Peter, Hi, Claudio, > > On

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-04-06 Thread Fabiano Rosas
Peter Xu writes: > On Tue, Apr 04, 2023 at 05:10:52PM +0200, Claudio Fontana wrote: >> On 4/4/23 16:53, Peter Xu wrote: >> > On Tue, Apr 04, 2023 at 10:00:16AM +0200, Claudio Fontana wrote: >> >> Hi Peter, >> > >> > Hi, Claudio, >> > >> >> >> >> On 4/3/23 21:26, Peter Xu wrote: >> >>> Hi,

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-04-04 Thread Peter Xu
On Tue, Apr 04, 2023 at 05:10:52PM +0200, Claudio Fontana wrote: > On 4/4/23 16:53, Peter Xu wrote: > > On Tue, Apr 04, 2023 at 10:00:16AM +0200, Claudio Fontana wrote: > >> Hi Peter, > > > > Hi, Claudio, > > > >> > >> On 4/3/23 21:26, Peter Xu wrote: > >>> Hi, Claudio, > >>> > >>> Thanks for

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-04-04 Thread Claudio Fontana
On 4/4/23 16:53, Peter Xu wrote: > On Tue, Apr 04, 2023 at 10:00:16AM +0200, Claudio Fontana wrote: >> Hi Peter, > > Hi, Claudio, > >> >> On 4/3/23 21:26, Peter Xu wrote: >>> Hi, Claudio, >>> >>> Thanks for the context. >>> >>> On Mon, Apr 03, 2023 at 09:47:26AM +0200, Claudio Fontana wrote:

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-04-04 Thread Peter Xu
On Tue, Apr 04, 2023 at 10:00:16AM +0200, Claudio Fontana wrote: > Hi Peter, Hi, Claudio, > > On 4/3/23 21:26, Peter Xu wrote: > > Hi, Claudio, > > > > Thanks for the context. > > > > On Mon, Apr 03, 2023 at 09:47:26AM +0200, Claudio Fontana wrote: > >> Hi, not sure if what is asked here is

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-04-04 Thread Claudio Fontana
Hi Peter, On 4/3/23 21:26, Peter Xu wrote: > Hi, Claudio, > > Thanks for the context. > > On Mon, Apr 03, 2023 at 09:47:26AM +0200, Claudio Fontana wrote: >> Hi, not sure if what is asked here is context in terms of the previous >> upstream discussions or our specific requirement we are trying

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-04-03 Thread Peter Xu
Hi, Claudio, Thanks for the context. On Mon, Apr 03, 2023 at 09:47:26AM +0200, Claudio Fontana wrote: > Hi, not sure if what is asked here is context in terms of the previous > upstream discussions or our specific requirement we are trying to bring > upstream. > > In terms of the specific

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-04-03 Thread Fabiano Rosas
David Hildenbrand writes: > On 03.04.23 16:41, Fabiano Rosas wrote: >> David Hildenbrand writes: >> >>> On 30.03.23 20:03, Fabiano Rosas wrote: Hi folks, I'm continuing the work done last year to add a new format of migration stream that can be used to migrate large guests

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-04-03 Thread David Hildenbrand
On 03.04.23 16:41, Fabiano Rosas wrote: David Hildenbrand writes: On 30.03.23 20:03, Fabiano Rosas wrote: Hi folks, I'm continuing the work done last year to add a new format of migration stream that can be used to migrate large guests to a single file in a performant way. This is an early

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-04-03 Thread Fabiano Rosas
David Hildenbrand writes: > On 30.03.23 20:03, Fabiano Rosas wrote: >> Hi folks, >> >> I'm continuing the work done last year to add a new format of >> migration stream that can be used to migrate large guests to a single >> file in a performant way. >> >> This is an early RFC with the

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-04-03 Thread Claudio Fontana
On 3/31/23 23:52, Peter Xu wrote: > On Fri, Mar 31, 2023 at 03:18:37PM -0300, Fabiano Rosas wrote: >> Peter Xu writes: >> >>> On Fri, Mar 31, 2023 at 05:10:16PM +0100, Daniel P. Berrangé wrote: On Fri, Mar 31, 2023 at 11:55:03AM -0400, Peter Xu wrote: > On Fri, Mar 31, 2023 at 12:30:45PM

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-04-03 Thread David Hildenbrand
On 30.03.23 20:03, Fabiano Rosas wrote: Hi folks, I'm continuing the work done last year to add a new format of migration stream that can be used to migrate large guests to a single file in a performant way. This is an early RFC with the previous code + my additions to support multifd and

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-03-31 Thread Peter Xu
On Fri, Mar 31, 2023 at 03:18:37PM -0300, Fabiano Rosas wrote: > Peter Xu writes: > > > On Fri, Mar 31, 2023 at 05:10:16PM +0100, Daniel P. Berrangé wrote: > >> On Fri, Mar 31, 2023 at 11:55:03AM -0400, Peter Xu wrote: > >> > On Fri, Mar 31, 2023 at 12:30:45PM -0300, Fabiano Rosas wrote: > >> >

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-03-31 Thread Fabiano Rosas
Peter Xu writes: > On Fri, Mar 31, 2023 at 05:10:16PM +0100, Daniel P. Berrangé wrote: >> On Fri, Mar 31, 2023 at 11:55:03AM -0400, Peter Xu wrote: >> > On Fri, Mar 31, 2023 at 12:30:45PM -0300, Fabiano Rosas wrote: >> > > Peter Xu writes: >> > > >> > > > On Fri, Mar 31, 2023 at 11:37:50AM

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-03-31 Thread Peter Xu
On Fri, Mar 31, 2023 at 05:10:16PM +0100, Daniel P. Berrangé wrote: > On Fri, Mar 31, 2023 at 11:55:03AM -0400, Peter Xu wrote: > > On Fri, Mar 31, 2023 at 12:30:45PM -0300, Fabiano Rosas wrote: > > > Peter Xu writes: > > > > > > > On Fri, Mar 31, 2023 at 11:37:50AM -0300, Fabiano Rosas wrote: >

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-03-31 Thread Daniel P . Berrangé
On Fri, Mar 31, 2023 at 11:55:03AM -0400, Peter Xu wrote: > On Fri, Mar 31, 2023 at 12:30:45PM -0300, Fabiano Rosas wrote: > > Peter Xu writes: > > > > > On Fri, Mar 31, 2023 at 11:37:50AM -0300, Fabiano Rosas wrote: > > >> >> Outgoing migration to file. NVMe disk. XFS filesystem. > > >> >> > >

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-03-31 Thread Peter Xu
On Fri, Mar 31, 2023 at 12:30:45PM -0300, Fabiano Rosas wrote: > Peter Xu writes: > > > On Fri, Mar 31, 2023 at 11:37:50AM -0300, Fabiano Rosas wrote: > >> >> Outgoing migration to file. NVMe disk. XFS filesystem. > >> >> > >> >> - Single migration runs of stopped 32G guest with ~90% RAM usage.

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-03-31 Thread Daniel P . Berrangé
On Fri, Mar 31, 2023 at 10:52:09AM -0400, Peter Xu wrote: > On Fri, Mar 31, 2023 at 11:37:50AM -0300, Fabiano Rosas wrote: > > >> Outgoing migration to file. NVMe disk. XFS filesystem. > > >> > > >> - Single migration runs of stopped 32G guest with ~90% RAM usage. Guest > > >> running

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-03-31 Thread Fabiano Rosas
Peter Xu writes: > On Fri, Mar 31, 2023 at 11:37:50AM -0300, Fabiano Rosas wrote: >> >> Outgoing migration to file. NVMe disk. XFS filesystem. >> >> >> >> - Single migration runs of stopped 32G guest with ~90% RAM usage. Guest >> >> running `stress-ng --vm 4 --vm-bytes 90% --vm-method all

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-03-31 Thread Peter Xu
On Fri, Mar 31, 2023 at 11:37:50AM -0300, Fabiano Rosas wrote: > >> Outgoing migration to file. NVMe disk. XFS filesystem. > >> > >> - Single migration runs of stopped 32G guest with ~90% RAM usage. Guest > >> running `stress-ng --vm 4 --vm-bytes 90% --vm-method all --verify -t > >> 10m -v`:

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-03-31 Thread Fabiano Rosas
Peter Xu writes: > On Thu, Mar 30, 2023 at 03:03:10PM -0300, Fabiano Rosas wrote: >> Hi folks, > > Hi, > >> >> I'm continuing the work done last year to add a new format of >> migration stream that can be used to migrate large guests to a single >> file in a performant way. >> >> This is an

Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-03-30 Thread Peter Xu
On Thu, Mar 30, 2023 at 03:03:10PM -0300, Fabiano Rosas wrote: > Hi folks, Hi, > > I'm continuing the work done last year to add a new format of > migration stream that can be used to migrate large guests to a single > file in a performant way. > > This is an early RFC with the previous code +

[RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram

2023-03-30 Thread Fabiano Rosas
Hi folks, I'm continuing the work done last year to add a new format of migration stream that can be used to migrate large guests to a single file in a performant way. This is an early RFC with the previous code + my additions to support multifd and direct IO. Let me know what you think! Here