On some systems, especially ones with many millions of files, the
stat(2) work of Bareos looking at files to decide if it has been
modified by far takes the most work. And that work is always the same
for each non-full backup.
If you have ZFS, maybe you also have access to DTrace? You could try
this one-liner:
# Count system calls by syscall, for PID 123 only:
dtrace -n 'syscall:::entry /pid == 123/ { @[probefunc] = count(); }'
Changing the '123' PID to the PID of bareos-fd. It will list the
syscalls by number of times each is called. If my guess is right, then
stat (or one of its variants) will be the number one.
If nothing else, DTrace is way cool.
On 3/21/19 5:33 AM, 'Stefan Krüger' via bareos-users wrote:
> Hello,
>
> our backup-client is a zfs-storage, which has 48 harddisk which is configured
> as RAID10 with SSDs for read and write cache, before I backup my data I
> create a snapshot.
>
> So far so good, my problem is that the incremental backup took around
> 32minutes every time, no matter how many files or how many gigabytes, for me
> the bottleneck is not the network, nor the backup storage, nor the client.
> When the backup run the bareos-fd client has a cpu utilisation of around 15%,
> iowait and also the load is around 0-1(%).
>
> SO can somebody tell me why is it so slow and how can i tune this? A
> fullbackup (around 6,5TB) takes 26hours, I expect a read/backup speed of
> >150MB/s. The Rate differs between 1709.4 KB/s and 143351.1 KB/s from my last
> 50 inc Backups
>
> Here two examples:
>
> 23-Jan 02:00 bareos-dir JobId 1063: Start Backup JobId 1063,
> Job=clouddata.2019-01-23_02.00.00_30
> 23-Jan 02:00 bareos-dir JobId 1063: There are no more Jobs associated with
> Volume "Incremental-0188". Marking it purged.
> 23-Jan 02:00 bareos-dir JobId 1063: All records pruned from Volume
> "Incremental-0188"; marking it "Purged"
> 23-Jan 02:00 bareos-dir JobId 1063: Recycled volume "Incremental-0188"
> 23-Jan 02:00 bareos-dir JobId 1063: Using Device "FileStorage" to write.
> 23-Jan 01:54 zfs-serv1-fd JobId 1063: DIR and FD clocks differ by -324
> seconds, FD automatically compensating.
> 23-Jan 01:54 zfs-serv1-fd JobId 1063: shell command: run ClientBeforeJob
> "/sbin/zfs snapshot -r nc_storage@bareos"
> 23-Jan 02:00 bareos-sd JobId 1063: Recycled volume "Incremental-0188" on
> device "FileStorage" (/backupdata), all previous data lost.
> 23-Jan 02:00 bareos-dir JobId 1063: Max Volume jobs=1 exceeded. Marking
> Volume "Incremental-0188" as Used.
> 23-Jan 02:25 zfs-serv1-fd JobId 1063: shell command: run ClientAfterJob
> "/sbin/zfs destroy -r nc_storage@bareos"
> 23-Jan 02:31 bareos-sd JobId 1063: Elapsed time=00:31:12, Transfer rate=5.970
> M Bytes/second
> 23-Jan 02:31 bareos-dir JobId 1063: sql_create.c:872 Insert of attributes
> batch table done
> 23-Jan 02:31 bareos-dir JobId 1063: Bareos bareos-dir 17.2.4 (21Sep17):
> Build OS: x86_64-pc-linux-gnu debian Debian GNU/Linux 9.3
> (stretch)
> JobId: 1063
> Job: clouddata.2019-01-23_02.00.00_30
> Backup Level: Incremental, since=2019-01-22 02:00:00
> Client: "zfs-serv1.my.domain.de" 17.2.4 (21Sep17)
> x86_64-pc-linux-gnu,debian,Debian GNU/Linux 9.3 (stretch),Debian_9.0,x86_64
> FileSet: "backup-zfs-serv" 2018-03-06 12:19:01
> Pool: "Incremental" (From Job IncPool override)
> Catalog: "MyCatalog" (From Client resource)
> Storage: "File" (From Job resource)
> Scheduled time: 23-Jan-2019 02:00:00
> Start time: 23-Jan-2019 02:00:00
> End time: 23-Jan-2019 02:31:13
> Elapsed time: 31 mins 13 secs
> Priority: 10
> FD Files Written: 2,257
> SD Files Written: 2,257
> FD Bytes Written: 11,175,399,254 (11.17 GB)
> SD Bytes Written: 11,175,894,854 (11.17 GB)
> Rate: 5966.6 KB/s
> Software Compression: 72.8 % (lz4)
> VSS: no
> Encryption: no
> Accurate: no
> Volume name(s): Incremental-0188
> Volume Session Id: 267
> Volume Session Time: 1539762885
> Last Volume Bytes: 11,189,726,587 (11.18 GB)
> Non-fatal FD errors: 0
> SD Errors: 0
> FD termination status: OK
> SD termination status: OK
> Termination: Backup OK
>
>
>
>
> 07-Mar 02:00 bareos-dir JobId 1180: Start Backup JobId 1180,
> Job=clouddata.2019-03-07_02.00.00_06
> 07-Mar 02:00 bareos-dir JobId 1180: There are no more Jobs associated with
> Volume "Incremental-0045". Marking it purged.
> 07-Mar 02:00 bareos-dir JobId 1180: All records pruned from Volume
> "Incremental-0045"; marking it "Purged"
> 07-Mar 02:00 bareos-dir JobId 1180: Recycled volume "Incremental-0045"
> 07-Mar 02:00 bareos-dir JobId 1180: Using Device "FileStorage" to write.
> 07-Mar 01:59 zfs-serv1-fd JobId 1180: DIR and FD clocks differ by -30
> seconds, FD automatically compensating.
> 07-Mar 01:59 zfs-serv1-fd JobId 1180: shell command: run ClientBeforeJob
> "/sbin/zfs snapshot -r nc_storage@bareos"
> 07-Mar 02:00 bareos-sd JobId 1180: Recycled volume "Incremental-0045" on
> device "FileStorage" (/backupdata), all previous data lost.
> 07-Mar 02:00 bareos-dir JobId 1180: Max Volume jobs=1 exceeded. Marking
> Volume "Incremental-0045" as Used.
> 07-Mar 02:31 zfs-serv1-fd JobId 1180: shell command: run ClientAfterJob
> "/sbin/zfs destroy -r nc_storage@bareos"
> 07-Mar 02:31 bareos-sd JobId 1180: Elapsed time=00:31:58, Transfer rate=9.753
> M Bytes/second
> 07-Mar 02:32 bareos-dir JobId 1180: sql_create.c:872 Insert of attributes
> batch table done
> 07-Mar 02:32 bareos-dir JobId 1180: Bareos bareos-dir 17.2.4 (21Sep17):
> Build OS: x86_64-pc-linux-gnu debian Debian GNU/Linux 9.3
> (stretch)
> JobId: 1180
> Job: clouddata.2019-03-07_02.00.00_06
> Backup Level: Incremental, since=2019-03-06 02:00:01
> Client: "zfs-serv1.my.domain.de" 17.2.4 (21Sep17)
> x86_64-pc-linux-gnu,debian,Debian GNU/Linux 9.3 (stretch),Debian_9.0,x86_64
> FileSet: "backup-zfs-serv" 2018-03-06 12:19:01
> Pool: "Incremental" (From Job IncPool override)
> Catalog: "MyCatalog" (From Client resource)
> Storage: "File" (From Job resource)
> Scheduled time: 07-Mar-2019 02:00:00
> Start time: 07-Mar-2019 02:00:00
> End time: 07-Mar-2019 02:32:00
> Elapsed time: 32 mins
> Priority: 10
> FD Files Written: 27,685
> SD Files Written: 27,685
> FD Bytes Written: 18,699,788,372 (18.69 GB)
> SD Bytes Written: 18,707,482,165 (18.70 GB)
> Rate: 9739.5 KB/s
> Software Compression: 51.9 % (lz4)
> VSS: no
> Encryption: no
> Accurate: no
> Volume name(s): Incremental-0045
> Volume Session Id: 384
> Volume Session Time: 1539762885
> Last Volume Bytes: 18,725,855,238 (18.72 GB)
> Non-fatal FD errors: 0
> SD Errors: 0
> FD termination status: OK
> SD termination status: OK
> Termination: Backup OK
>
--
You received this message because you are subscribed to the Google Groups
"bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to bareos-users+unsubscr...@googlegroups.com.
To post to this group, send email to bareos-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
