From: Radosław Korzeniewski <rados...@korzeniewski.net>
> niedz., 10 lut 2019 o 20:54 <jes...@krogh.cc> napisał(a):
>> 1) When reading a catalog, loop over all files and issue a
>> posix_fasvise WILLNEED on the first 1MB of the file.
>> I have prototyped this outside bacula and it seem to work very
>> nicely and should be a small non-intrusive patch. It will allow the IO
>> stack to issue concurrently around the smaller files caching them in
>> memory. I have inspected the sourcecode and cannot find traces that this
>> should be in place allready.
> The code in question should be available in src/findlib/bfile.c:
> 1055 #if defined(HAVE_POSIX_FADVISE) && defined(POSIX_FADV_WILLNEED)
> 1056 /* If not RDWR or WRONLY must be Read Only */
> 1057 if (bfd->fid != -1 && !(flags & (O_RDWR|O_WRONLY))) {
> 1058 int stat = posix_fadvise(bfd->fid, 0, 0, POSIX_FADV_WILLNEED);
> 1059 Dmsg3(400, "Did posix_fadvise WILLNEED on %s fid=%d stat=%d\n",
> fname, bfd->fid, stat);
> 1060 }
> 1061 #endif
> It calls posix_fasvise with POSIX_FADV_WILLNEED flag but touch beginning of
> the file and not a first 1MB as you requested.
This is part of bopen, which is called when the file is opened for reading, as
far as I understand. The idea suggested by Jesper is to call fadvise on a larger
set of files well in advance, so the system can spend time bringing those files
into the page cache before the file daemon tries to read them. One method is
to call fadvise on all files in a directory before the fd loops over the dir to
read
them. It's simpler than parallelizing the file daemon itself, and yields a very
nice
speedup in our tests.
So instead of one loop that does
for file in dir:
fadvise_willneed
open
read
close
fadvise_dontneed
You have two loops:
for file in dir:
fadvise_willneed
for file in dir:
open
read
close
fadvise_dontneed
The first loop completes very quickly and will let the system utilize the
parallelization of the underlying storage to load in data very quickly.
--
Anders Ossowicki
_______________________________________________
Bacula-devel mailing list
Bacula-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-devel
