Re: too many lstat() syscalls, therefore too many IOPS
Ups, somehow I sent my e-mail twice... But at least now I know about
ionice; never needed it, but it's even installed by default (it's part
of util-linux, together with mount, fdisk, etc...).
I'm now browsing through the other utilities of this package, full of
small jewels. You never stop learning...
Thanks, Eric
On 12/05/2021 23:59, griffin tucker wrote:
there is a tool called ionice that comes with the util-linux package,
which should help other read/write processes if you are multitasking.
ionice shouldn't slow down rdiff-backup if there are no other io
processes going on.
On Thu, 13 May 2021 at 03:08, Eric L. Zolf wrote:
Hi,
first, I don't see anything surprising in what you describe, so all
normal AFAICJ.
Second, rdiff-backup needs to check each source file/directory and each
target, compare them and then copy (or not), so if you have some 2300
files to backup, that would sound about right. If the target or the
source file doesn't exist, it would give an error.
If the files are small or don't have changes, the lstat happen a lot and
nothing much else; this is typical random access. It gives a much
different access pattern than the copying of bigger files, where more
sequential is typically done to read/write the file's data.
There is no real way to improve the situation, rdiff-backup goes as fast
as it can and I personally don't know an I/O-equivalent of "nice" (and
if you limit the I/O, the backup will be even slower).
You could try the --no-fsync option to improve speed:
--fsync, --no-fsync [opt] do (or not) often sync the file system
(_not_ doing it is faster but can be dangerous)
And, yes, the `rdiff-backup-data/increments` directory is used by
rdiff-backup to keep track of file and directory changes.
Hope this helps,
Eric
On 12/05/2021 07:10, Andrei Enshin via Any discussion of rdiff-backup wrote:
Hi rdiff-backup folks,
Since recent, during backing up I can see spike in IOPS up to 500 which exhaust
limit of a VM. Therefore backup process takes very long. I've straced a bit and
what I can see is: many failed lstat() syscalls:
% time seconds usecs/call callserrors syscall
-- --- --- - -
42.710.040247 9 4608 1420 lstat
35.410.033370 12 2860 getdents
9.410.008865 6 1431 open
4.630.004363 3 1430 close
4.030.003797 3 1431 fstat
3.750.003536 2 1417 getuid
0.040.39 39 1 unlink
0.010.13 1 9 read
-- --- --- - -
100.000.094230 13187 1420 total
Seems rdiff-backup checks existence of some file/dir:
10:13:16 lstat("/some/path/rdiff-backup-data/increments/foo/bar", 0x7ffd832fa810) =
-1 ENOENT (No such file or directory) <0.20>
After backup is done, there is still no such file.
Seems the part in path - /rdiff-backup-data/increments/ - is some "config" for
rdiff-backup and probably it tryies to find something but can't?
What might be wrong in my setup? What would you recommend to check to solve the
issue if it is issue at all?
---
Best Regards,
Andrei Enshin
Re: too many lstat() syscalls, therefore too many IOPS
"Eric L. Zolf" wrote: > I personally don't know an I/O-equivalent of "nice". Try "ionice". ;-) -- Yves Bellefeuille
Re: too many lstat() syscalls, therefore too many IOPS
there is a tool called ionice that comes with the util-linux package,
which should help other read/write processes if you are multitasking.
ionice shouldn't slow down rdiff-backup if there are no other io
processes going on.
On Thu, 13 May 2021 at 03:08, Eric L. Zolf wrote:
>
> Hi,
>
> first, I don't see anything surprising in what you describe, so all
> normal AFAICJ.
>
> Second, rdiff-backup needs to check each source file/directory and each
> target, compare them and then copy (or not), so if you have some 2300
> files to backup, that would sound about right. If the target or the
> source file doesn't exist, it would give an error.
>
> If the files are small or don't have changes, the lstat happen a lot and
> nothing much else; this is typical random access. It gives a much
> different access pattern than the copying of bigger files, where more
> sequential is typically done to read/write the file's data.
>
> There is no real way to improve the situation, rdiff-backup goes as fast
> as it can and I personally don't know an I/O-equivalent of "nice" (and
> if you limit the I/O, the backup will be even slower).
>
> You could try the --no-fsync option to improve speed:
>
>--fsync, --no-fsync [opt] do (or not) often sync the file system
> (_not_ doing it is faster but can be dangerous)
>
> And, yes, the `rdiff-backup-data/increments` directory is used by
> rdiff-backup to keep track of file and directory changes.
>
> Hope this helps,
> Eric
>
> On 12/05/2021 07:10, Andrei Enshin via Any discussion of rdiff-backup wrote:
> >
> > Hi rdiff-backup folks,
> >
> > Since recent, during backing up I can see spike in IOPS up to 500 which
> > exhaust limit of a VM. Therefore backup process takes very long. I've
> > straced a bit and what I can see is: many failed lstat() syscalls:
> > % time seconds usecs/call callserrors syscall
> > -- --- --- - -
> > 42.710.040247 9 4608 1420 lstat
> > 35.410.033370 12 2860 getdents
> >9.410.008865 6 1431 open
> >4.630.004363 3 1430 close
> >4.030.003797 3 1431 fstat
> >3.750.003536 2 1417 getuid
> >0.040.39 39 1 unlink
> >0.010.13 1 9 read
> > -- --- --- - -
> > 100.000.094230 13187 1420 total
> > Seems rdiff-backup checks existence of some file/dir:
> > 10:13:16 lstat("/some/path/rdiff-backup-data/increments/foo/bar",
> > 0x7ffd832fa810) = -1 ENOENT (No such file or directory) <0.20>
> > After backup is done, there is still no such file.
> > Seems the part in path - /rdiff-backup-data/increments/ - is some "config"
> > for rdiff-backup and probably it tryies to find something but can't?
> >
> > What might be wrong in my setup? What would you recommend to check to solve
> > the issue if it is issue at all?
> >
> > ---
> > Best Regards,
> > Andrei Enshin
> >
>
Re: too many lstat() syscalls, therefore too many IOPS
Hi,
first, I don't see anything surprising in what you describe, so all
normal AFAICJ.
Second, rdiff-backup needs to check each source file/directory and each
target, compare them and then copy (or not), so if you have some 2300
files to backup, that would sound about right. If the target or the
source file doesn't exist, it would give an error.
If the files are small or don't have changes, the lstat happen a lot and
nothing much else; this is typical random access. It gives a much
different access pattern than the copying of bigger files, where more
sequential is typically done to read/write the file's data.
There is no real way to improve the situation, rdiff-backup goes as fast
as it can and I personally don't know an I/O-equivalent of "nice" (and
if you limit the I/O, the backup will be even slower).
You could try the --no-fsync option to improve speed:
--fsync, --no-fsync [opt] do (or not) often sync the file system
(_not_ doing it is faster but can be dangerous)
And, yes, the `rdiff-backup-data/increments` directory is used by
rdiff-backup to keep track of file and directory changes.
Hope this helps,
Eric
On 12/05/2021 07:10, Andrei Enshin via Any discussion of rdiff-backup wrote:
Hi rdiff-backup folks,
Since recent, during backing up I can see spike in IOPS up to 500 which exhaust limit of a VM. Therefore backup process takes very long. I've straced a bit and what I can see is: many failed lstat() syscalls:
% time seconds usecs/call callserrors syscall
-- --- --- - -
42.710.040247 9 4608 1420 lstat
35.410.033370 12 2860 getdents
9.410.008865 6 1431 open
4.630.004363 3 1430 close
4.030.003797 3 1431 fstat
3.750.003536 2 1417 getuid
0.040.39 39 1 unlink
0.010.13 1 9 read
-- --- --- - -
100.000.094230 13187 1420 total
Seems rdiff-backup checks existence of some file/dir:
10:13:16 lstat("/some/path/rdiff-backup-data/increments/foo/bar", 0x7ffd832fa810) =
-1 ENOENT (No such file or directory) <0.20>
After backup is done, there is still no such file.
Seems the part in path - /rdiff-backup-data/increments/ - is some "config" for
rdiff-backup and probably it tryies to find something but can't?
What might be wrong in my setup? What would you recommend to check to solve the
issue if it is issue at all?
---
Best Regards,
Andrei Enshin
Re: too many lstat() syscalls, therefore too many IOPS
On Wed, 2021-05-12 at 13:01 +0200, Eric L. Zolf wrote: ... > > There is no real way to improve the situation, rdiff-backup goes as > fast > as it can and I personally don't know an I/O-equivalent of "nice" > (and > if you limit the I/O, the backup will be even slower). The I/O equivalent of "nice" is "ionice" and you probably want to set it to the "idle" class, i.e. "ionice -c idle rdiff-backup ..." This will run when other activities are not doing I/O, and yes it will run slower, on a system that is doing other I/O tasks. Regards Frank
Re: too many lstat() syscalls, therefore too many IOPS
Hi,
first, I don't see anything surprising in what you describe, so all
normal AFAICJ.
Second, rdiff-backup needs to check each source file/directory and each
target, compare them and then copy (or not), so if you have some 2300
files to backup, that would sound about right. If the target or the
source file doesn't exist, it would give an error.
If the files are small or don't have changes, the lstat happen a lot and
nothing much else; this is typical random access. It gives a much
different access pattern than the copying of bigger files, where more
sequential is typically done to read/write the file's data.
There is no real way to improve the situation, rdiff-backup goes as fast
as it can and I personally don't know an I/O-equivalent of "nice" (and
if you limit the I/O, the backup will be even slower).
You could try the --no-fsync option to improve speed:
--fsync, --no-fsync [opt] do (or not) often sync the file system
(_not_ doing it is faster but can be dangerous)
And, yes, the `rdiff-backup-data/increments` directory is used by
rdiff-backup to keep track of file and directory changes.
Hope this helps,
Eric
On 12/05/2021 07:10, Andrei Enshin via Any discussion of rdiff-backup wrote:
Hi rdiff-backup folks,
Since recent, during backing up I can see spike in IOPS up to 500 which exhaust limit of a VM. Therefore backup process takes very long. I've straced a bit and what I can see is: many failed lstat() syscalls:
% time seconds usecs/call callserrors syscall
-- --- --- - -
42.710.040247 9 4608 1420 lstat
35.410.033370 12 2860 getdents
9.410.008865 6 1431 open
4.630.004363 3 1430 close
4.030.003797 3 1431 fstat
3.750.003536 2 1417 getuid
0.040.39 39 1 unlink
0.010.13 1 9 read
-- --- --- - -
100.000.094230 13187 1420 total
Seems rdiff-backup checks existence of some file/dir:
10:13:16 lstat("/some/path/rdiff-backup-data/increments/foo/bar", 0x7ffd832fa810) =
-1 ENOENT (No such file or directory) <0.20>
After backup is done, there is still no such file.
Seems the part in path - /rdiff-backup-data/increments/ - is some "config" for
rdiff-backup and probably it tryies to find something but can't?
What might be wrong in my setup? What would you recommend to check to solve the
issue if it is issue at all?
---
Best Regards,
Andrei Enshin
