Hi, perhaps database-queries to calculate the files to backup take that time?
Udo Am 21.03.19 um 11:33 schrieb 'Stefan Krüger' via bareos-users: > Hello, > > our backup-client is a zfs-storage, which has 48 harddisk which is configured > as RAID10 with SSDs for read and write cache, before I backup my data I > create a snapshot. > > So far so good, my problem is that the incremental backup took around > 32minutes every time, no matter how many files or how many gigabytes, for me > the bottleneck is not the network, nor the backup storage, nor the client. > When the backup run the bareos-fd client has a cpu utilisation of around 15%, > iowait and also the load is around 0-1(%). > > SO can somebody tell me why is it so slow and how can i tune this? A > fullbackup (around 6,5TB) takes 26hours, I expect a read/backup speed of > >150MB/s. The Rate differs between 1709.4 KB/s and 143351.1 KB/s from my last > 50 inc Backups > > Here two examples: > > 23-Jan 02:00 bareos-dir JobId 1063: Start Backup JobId 1063, > Job=clouddata.2019-01-23_02.00.00_30 > 23-Jan 02:00 bareos-dir JobId 1063: There are no more Jobs associated with > Volume "Incremental-0188". Marking it purged. > 23-Jan 02:00 bareos-dir JobId 1063: All records pruned from Volume > "Incremental-0188"; marking it "Purged" > 23-Jan 02:00 bareos-dir JobId 1063: Recycled volume "Incremental-0188" > 23-Jan 02:00 bareos-dir JobId 1063: Using Device "FileStorage" to write. > 23-Jan 01:54 zfs-serv1-fd JobId 1063: DIR and FD clocks differ by -324 > seconds, FD automatically compensating. > 23-Jan 01:54 zfs-serv1-fd JobId 1063: shell command: run ClientBeforeJob > "/sbin/zfs snapshot -r nc_storage@bareos" > 23-Jan 02:00 bareos-sd JobId 1063: Recycled volume "Incremental-0188" on > device "FileStorage" (/backupdata), all previous data lost. > 23-Jan 02:00 bareos-dir JobId 1063: Max Volume jobs=1 exceeded. Marking > Volume "Incremental-0188" as Used. > 23-Jan 02:25 zfs-serv1-fd JobId 1063: shell command: run ClientAfterJob > "/sbin/zfs destroy -r nc_storage@bareos" > 23-Jan 02:31 bareos-sd JobId 1063: Elapsed time=00:31:12, Transfer rate=5.970 > M Bytes/second > 23-Jan 02:31 bareos-dir JobId 1063: sql_create.c:872 Insert of attributes > batch table done > 23-Jan 02:31 bareos-dir JobId 1063: Bareos bareos-dir 17.2.4 (21Sep17): > Build OS: x86_64-pc-linux-gnu debian Debian GNU/Linux 9.3 > (stretch) > JobId: 1063 > Job: clouddata.2019-01-23_02.00.00_30 > Backup Level: Incremental, since=2019-01-22 02:00:00 > Client: "zfs-serv1.my.domain.de" 17.2.4 (21Sep17) > x86_64-pc-linux-gnu,debian,Debian GNU/Linux 9.3 (stretch),Debian_9.0,x86_64 > FileSet: "backup-zfs-serv" 2018-03-06 12:19:01 > Pool: "Incremental" (From Job IncPool override) > Catalog: "MyCatalog" (From Client resource) > Storage: "File" (From Job resource) > Scheduled time: 23-Jan-2019 02:00:00 > Start time: 23-Jan-2019 02:00:00 > End time: 23-Jan-2019 02:31:13 > Elapsed time: 31 mins 13 secs > Priority: 10 > FD Files Written: 2,257 > SD Files Written: 2,257 > FD Bytes Written: 11,175,399,254 (11.17 GB) > SD Bytes Written: 11,175,894,854 (11.17 GB) > Rate: 5966.6 KB/s > Software Compression: 72.8 % (lz4) > VSS: no > Encryption: no > Accurate: no > Volume name(s): Incremental-0188 > Volume Session Id: 267 > Volume Session Time: 1539762885 > Last Volume Bytes: 11,189,726,587 (11.18 GB) > Non-fatal FD errors: 0 > SD Errors: 0 > FD termination status: OK > SD termination status: OK > Termination: Backup OK > > > > > 07-Mar 02:00 bareos-dir JobId 1180: Start Backup JobId 1180, > Job=clouddata.2019-03-07_02.00.00_06 > 07-Mar 02:00 bareos-dir JobId 1180: There are no more Jobs associated with > Volume "Incremental-0045". Marking it purged. > 07-Mar 02:00 bareos-dir JobId 1180: All records pruned from Volume > "Incremental-0045"; marking it "Purged" > 07-Mar 02:00 bareos-dir JobId 1180: Recycled volume "Incremental-0045" > 07-Mar 02:00 bareos-dir JobId 1180: Using Device "FileStorage" to write. > 07-Mar 01:59 zfs-serv1-fd JobId 1180: DIR and FD clocks differ by -30 > seconds, FD automatically compensating. > 07-Mar 01:59 zfs-serv1-fd JobId 1180: shell command: run ClientBeforeJob > "/sbin/zfs snapshot -r nc_storage@bareos" > 07-Mar 02:00 bareos-sd JobId 1180: Recycled volume "Incremental-0045" on > device "FileStorage" (/backupdata), all previous data lost. > 07-Mar 02:00 bareos-dir JobId 1180: Max Volume jobs=1 exceeded. Marking > Volume "Incremental-0045" as Used. > 07-Mar 02:31 zfs-serv1-fd JobId 1180: shell command: run ClientAfterJob > "/sbin/zfs destroy -r nc_storage@bareos" > 07-Mar 02:31 bareos-sd JobId 1180: Elapsed time=00:31:58, Transfer rate=9.753 > M Bytes/second > 07-Mar 02:32 bareos-dir JobId 1180: sql_create.c:872 Insert of attributes > batch table done > 07-Mar 02:32 bareos-dir JobId 1180: Bareos bareos-dir 17.2.4 (21Sep17): > Build OS: x86_64-pc-linux-gnu debian Debian GNU/Linux 9.3 > (stretch) > JobId: 1180 > Job: clouddata.2019-03-07_02.00.00_06 > Backup Level: Incremental, since=2019-03-06 02:00:01 > Client: "zfs-serv1.my.domain.de" 17.2.4 (21Sep17) > x86_64-pc-linux-gnu,debian,Debian GNU/Linux 9.3 (stretch),Debian_9.0,x86_64 > FileSet: "backup-zfs-serv" 2018-03-06 12:19:01 > Pool: "Incremental" (From Job IncPool override) > Catalog: "MyCatalog" (From Client resource) > Storage: "File" (From Job resource) > Scheduled time: 07-Mar-2019 02:00:00 > Start time: 07-Mar-2019 02:00:00 > End time: 07-Mar-2019 02:32:00 > Elapsed time: 32 mins > Priority: 10 > FD Files Written: 27,685 > SD Files Written: 27,685 > FD Bytes Written: 18,699,788,372 (18.69 GB) > SD Bytes Written: 18,707,482,165 (18.70 GB) > Rate: 9739.5 KB/s > Software Compression: 51.9 % (lz4) > VSS: no > Encryption: no > Accurate: no > Volume name(s): Incremental-0045 > Volume Session Id: 384 > Volume Session Time: 1539762885 > Last Volume Bytes: 18,725,855,238 (18.72 GB) > Non-fatal FD errors: 0 > SD Errors: 0 > FD termination status: OK > SD termination status: OK > Termination: Backup OK > -- You received this message because you are subscribed to the Google Groups "bareos-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to bareos-users+unsubscr...@googlegroups.com. To post to this group, send email to bareos-users@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
