Re: [pve-devel] Speed up PVE Backup

.com> Envoyé: Mardi 26 Juillet 2016 13:19:28 Objet: Re: [pve-devel] Speed up PVE Backup Hi, El 26/07/16 a las 10:32, Alexandre DERUMIER escribió: >>> There is no reason to flush a restored disk until just the end, really. >>> Issuing flushes every x MB could hurt other

Re: [pve-devel] Speed up PVE Backup

Hi, El 26/07/16 a las 10:32, Alexandre DERUMIER escribió: There is no reason to flush a restored disk until just the end, really. Issuing flushes every x MB could hurt other storages without need. I'm curious to see host memory usage of a big local file storage restore (100GB), with writeback

Re: [pve-devel] Speed up PVE Backup

nal - De: "Eneko Lacunza" <elacu...@binovo.es> À: "pve-devel" <pve-devel@pve.proxmox.com> Envoyé: Mardi 26 Juillet 2016 10:13:59 Objet: Re: [pve-devel] Speed up PVE Backup Hi, El 26/07/16 a las 10:04, Alexandre DERUMIER escribió: >>> I think qmrestore

Re: [pve-devel] Speed up PVE Backup

Hi, El 26/07/16 a las 10:04, Alexandre DERUMIER escribió: I think qmrestore isn't issuing any flush request (until maybe the end), Need to be checked! (but if I think we open restore block storage with writeback, so I hope we send flush) so for ceph storage backend we should set

Re: [pve-devel] Speed up PVE Backup

o Lacunza" <elacu...@binovo.es> À: "dietmar" <diet...@proxmox.com>, "pve-devel" <pve-devel@pve.proxmox.com> Envoyé: Jeudi 21 Juillet 2016 13:19:10 Objet: Re: [pve-devel] Speed up PVE Backup Hi, El 21/07/16 a las 09:34, Dietmar Maurer escribió: > >>

Re: [pve-devel] Speed up PVE Backup

mox.com>, "Eneko Lacunza" <elacu...@binovo.es> Envoyé: Mercredi 20 Juillet 2016 17:46:12 Objet: Re: [pve-devel] Speed up PVE Backup > This is called from restore_extents, where a comment precisely says "try > to write whole clusters to speedup restore", so this m

Re: [pve-devel] Speed up PVE Backup

Hi, El 21/07/16 a las 09:34, Dietmar Maurer escribió: But you can try to assemble larger blocks, and write them once you get an out of order block... Yes, this is the plan. I always thought the ceph libraries does (or should do) that anyways? (write combining) Reading the docs:

Re: [pve-devel] Speed up PVE Backup

> But I suppose they're mostly ordered? Yes - depends how much writes happens during backup... > > But you can try to assemble larger blocks, and write them once you get > > an out of order block... > Yes, this is the plan. > > I always thought the ceph libraries does (or should do) that

Re: [pve-devel] Speed up PVE Backup

El 20/07/16 a las 17:46, Dietmar Maurer escribió: This is called from restore_extents, where a comment precisely says "try to write whole clusters to speedup restore", so this means we're writing 64KB-8Byte chunks, which is giving a hard time to Ceph-RBD because this means lots of ~64KB IOPS.

Re: [pve-devel] Speed up PVE Backup

On 20/07/2016 4:24 PM, Eneko Lacunza wrote: Yesterday our 9-osd 3-node cluster restored a backup at 6MB/s... it was very boring, painfull and expensive to wait for it One of the reasons we migrated away from ceph - snapshot and backup restores were unusably slow. -- Lindsay Mathieson

Re: [pve-devel] Speed up PVE Backup

> This is called from restore_extents, where a comment precisely says "try > to write whole clusters to speedup restore", so this means we're writing > 64KB-8Byte chunks, which is giving a hard time to Ceph-RBD because this > means lots of ~64KB IOPS. > > So, I suggest the following solution

Re: [pve-devel] Speed up PVE Backup

Hi again, I've been looking around the backup/restore code a bit. I'm focused on restore acceleration on Ceph RBD right know. Sorry if I have something mistaken, I have never developed for Proxmox/Qemu. I see in line 563 of file

Re: [pve-devel] Speed up PVE Backup

Hi all, El 16/02/16 a las 15:52, Stefan Priebe - Profihost AG escribió: Am 16.02.2016 um 15:50 schrieb Dmitry Petuhov: 16.02.2016 13:20, Dietmar Maurer wrote: Storage Backend is ceph using 2x 10Gbit/s and i'm able to read from it with 500-1500MB/s. See below for an example. The backup

Re: [pve-devel] Speed up PVE Backup

backup system which uses the > drive-backup qmp routine to dump qcow2. We don't need the config files. > And this works with ceph and 4MB cluster size. > > Greets, > Stefan > >> - Mail original - >> De: "Stefan Priebe" <s.pri...@profihost.ag&g

Re: [pve-devel] Speed up PVE Backup

Envoyé: Mercredi 2 Mars 2016 08:28:52 Objet: Re: [pve-devel] Speed up PVE Backup Am 01.03.2016 um 14:05 schrieb Alexandre DERUMIER: >>> Yes it does. We already tested it - it just does not work with VMA. We >>> need to dump qcow2 files cause vma is crashing when the clu

Re: [pve-devel] Speed up PVE Backup

: "pve-devel" <pve-devel@pve.proxmox.com>, "dietmar" <diet...@proxmox.com> > Envoyé: Mardi 1 Mars 2016 11:55:21 > Objet: Re: [pve-devel] Speed up PVE Backup > > ) > > Am 01.03.2016 um 11:03 schrieb Alexandre DERUMIER: >> Hi, qemu devs have s

Re: [pve-devel] Speed up PVE Backup

ze(s) ? - Mail original - De: "Stefan Priebe" <s.pri...@profihost.ag> À: "pve-devel" <pve-devel@pve.proxmox.com>, "dietmar" <diet...@proxmox.com> Envoyé: Mardi 1 Mars 2016 11:55:21 Objet: Re: [pve-devel] Speed up PVE Backup ) Am 01.03.20

Re: [pve-devel] Speed up PVE Backup

;diet...@proxmox.com> > À: "aderumier" <aderum...@odiso.com>, "pve-devel" <pve-devel@pve.proxmox.com> > Envoyé: Vendredi 19 Février 2016 09:17:14 > Objet: Re: [pve-devel] Speed up PVE Backup > >> I wonder how perform the native qemu backup blockj

Re: [pve-devel] Speed up PVE Backup

: "dietmar" <diet...@proxmox.com> À: "aderumier" <aderum...@odiso.com>, "pve-devel" <pve-devel@pve.proxmox.com> Envoyé: Vendredi 19 Février 2016 09:17:14 Objet: Re: [pve-devel] Speed up PVE Backup > I wonder how perform the native qemu backup

Re: [pve-devel] Speed up PVE Backup

> I wonder how perform the native qemu backup blockjob vs proxmox vma backup > format ? We use the qemu backup blockjob, just slightly modified... ___ pve-devel mailing list pve-devel@pve.proxmox.com

Re: [pve-devel] Speed up PVE Backup

> Any reason to use 64k blocks and not something bigger? Backups should be > sequential on all storage backends so something bigger shouldn't hurt. Maybe you can raise that question on the qemu devel list? ___ pve-devel mailing list

Re: [pve-devel] Speed up PVE Backup

> Any reason to use 64k blocks and not something bigger? Again, take a look at the qemu backup code. I guess it is possible, but not trivial. ___ pve-devel mailing list pve-devel@pve.proxmox.com

Re: [pve-devel] Speed up PVE Backup

Hello Dietmar, Am 16.02.2016 um 14:55 schrieb Stefan Priebe - Profihost AG: Am 16.02.2016 um 12:58 schrieb Stefan Priebe - Profihost AG: Am 16.02.2016 um 11:55 schrieb Dietmar Maurer: Is it enough to just change these: The whole backup algorithm is based on 64KB blocksize, so it is not

Re: [pve-devel] Speed up PVE Backup

Am 16.02.2016 um 15:50 schrieb Dmitry Petuhov: > 16.02.2016 13:20, Dietmar Maurer wrote: >>> Storage Backend is ceph using 2x 10Gbit/s and i'm able to read from it >>> with 500-1500MB/s. See below for an example. >> The backup process reads 64KB blocks, and it seems this slows down ceph. >> This

Re: [pve-devel] Speed up PVE Backup

16.02.2016 13:20, Dietmar Maurer wrote: Storage Backend is ceph using 2x 10Gbit/s and i'm able to read from it with 500-1500MB/s. See below for an example. The backup process reads 64KB blocks, and it seems this slows down ceph. This is a known behavior, but I found no solution to speed it up.

Re: [pve-devel] Speed up PVE Backup

Am 16.02.2016 um 12:58 schrieb Stefan Priebe - Profihost AG: > Am 16.02.2016 um 11:55 schrieb Dietmar Maurer: >>> Is it enough to just change these: >> >> The whole backup algorithm is based on 64KB blocksize, so it >> is not trivial (or impossible?) to change that. >> >> Besides, I do not

Re: [pve-devel] Speed up PVE Backup

Am 16.02.2016 um 11:55 schrieb Dietmar Maurer: >> Is it enough to just change these: > > The whole backup algorithm is based on 64KB blocksize, so it > is not trivial (or impossible?) to change that. > > Besides, I do not understand why reading 64KB is slow - ceph libraries > should have/use a

Re: [pve-devel] Speed up PVE Backup

On Tue, 16 Feb 2016 11:55:07 +0100 (CET) Dietmar Maurer wrote: > > Besides, I do not understand why reading 64KB is slow - ceph libraries > should have/use a reasonable readahead cache to make it fast? > Due to the nature of the operation that reading is considered random

Re: [pve-devel] Speed up PVE Backup

> Is it enough to just change these: The whole backup algorithm is based on 64KB blocksize, so it is not trivial (or impossible?) to change that. Besides, I do not understand why reading 64KB is slow - ceph libraries should have/use a reasonable readahead cache to make it fast?

Re: [pve-devel] Speed up PVE Backup

Am 16.02.2016 um 11:20 schrieb Dietmar Maurer: >> Storage Backend is ceph using 2x 10Gbit/s and i'm able to read from it >> with 500-1500MB/s. See below for an example. > > The backup process reads 64KB blocks, and it seems this slows down ceph. > This is a known behavior, but I found no

Re: [pve-devel] Speed up PVE Backup

Am 16.02.2016 um 11:20 schrieb Dietmar Maurer: >> Storage Backend is ceph using 2x 10Gbit/s and i'm able to read from it >> with 500-1500MB/s. See below for an example. > > The backup process reads 64KB blocks, and it seems this slows down ceph. > This is a known behavior, but I found no

Re: [pve-devel] Speed up PVE Backup

> Storage Backend is ceph using 2x 10Gbit/s and i'm able to read from it > with 500-1500MB/s. See below for an example. The backup process reads 64KB blocks, and it seems this slows down ceph. This is a known behavior, but I found no solution to speed it up.

Re: [pve-devel] Speed up PVE Backup

Am 16.02.2016 um 11:02 schrieb Martin Waschbüsch: > >> Am 16.02.2016 um 10:32 schrieb Stefan Priebe - Profihost AG >> : >> >> Am 16.02.2016 um 09:57 schrieb Martin Waschbüsch: >>> Hi Stefan, >> This is PVE 3.4 running Qemu 2.4 >>> >>> To me this looks like the

Re: [pve-devel] Speed up PVE Backup

Stefan, > The output after 15 minutes is: > INFO: starting new backup job: vzdump 132 --remove 0 --mode snapshot > --storage vmbackup --node 1234 > INFO: Starting Backup of VM 132 (qemu) > INFO: status = running > INFO: update VM 132: -lock backup > INFO: backup mode: snapshot > INFO: ionice

Re: [pve-devel] Speed up PVE Backup

> Am 16.02.2016 um 10:32 schrieb Stefan Priebe - Profihost AG > : > > Am 16.02.2016 um 09:57 schrieb Martin Waschbüsch: >> Hi Stefan, > >>> This is PVE 3.4 running Qemu 2.4 >> >> To me this looks like the compression is the limiting factor? What speed do >> you get for

Re: [pve-devel] Speed up PVE Backup

Am 16.02.2016 um 09:57 schrieb Martin Waschbüsch: > Hi Stefan, >> This is PVE 3.4 running Qemu 2.4 > > To me this looks like the compression is the limiting factor? What speed do > you get for this NFS mount when just copying an existing file? Which compression? There is only FS compression on

Re: [pve-devel] Speed up PVE Backup

Am 16.02.2016 um 09:54 schrieb Andreas Steinel: > Hi Stefan, > > That's really slow. Yes > I use a similar setup, but with ZFS and I backup 6 nodes in parallel to > the storage and saturate the 1 GBit network connection. Currently vzdump / qemu is only uses around 100kb/s of the 10Gbit/s

Re: [pve-devel] Speed up PVE Backup

Hi Stefan, > Am 16.02.2016 um 09:22 schrieb Stefan Priebe - Profihost AG > : > > Hi, > > is there any way to speed up PVE Backups? > > I'm trying to evaluate the optimal method doing backups but they took > very long. > > I'm trying to use vzdump on top of nfs on top

Re: [pve-devel] Speed up PVE Backup

Hi Stefan, That's really slow. I use a similar setup, but with ZFS and I backup 6 nodes in parallel to the storage and saturate the 1 GBit network connection. I use LZOP on the Proxmox-side as best tradeoff between size and online-compression speed. On Tue, Feb 16, 2016 at 9:22 AM, Stefan

[pve-devel] Speed up PVE Backup

Hi, is there any way to speed up PVE Backups? I'm trying to evaluate the optimal method doing backups but they took very long. I'm trying to use vzdump on top of nfs on top of btrfs using zlib compression. The target FS it totally idle but the backup is running at a very low speed. The output