> The code in question should be available in src/findlib/bfile.c:
>
> 1055 #if defined(HAVE_POSIX_FADVISE) && defined(POSIX_FADV_WILLNEED)
> 1056 /* If not RDWR or WRONLY must be Read Only */
> 1057 if (bfd->fid != -1 && !(flags & (O_RDWR|O_WRONLY))) {
> 1058 int stat = posix_fadvise(bfd->fid, 0, 0, POSIX_FADV_WILLNEED);
> 1059 Dmsg3(400, "Did posix_fadvise WILLNEED on %s fid=%d stat=%d\n",
> fname, bfd->fid, stat);
> 1060 }
> 1061 #endif
>
> It calls posix_fasvise with POSIX_FADV_WILLNEED flag but touch beginning
> of
> the file and not a first 1MB as you requested.
This is sort of wrong -- if above hits a 2GB file.. I would prefer
readahead to work and not have the OS to read in everything. 0,0 is the
entire file.
>> 2) Thread out the filedaemon
>> Implement a X MB buffer in the filedaemon. could be 16 slots of
>> max 5MB, for files smaller than 5MB this serves as staging area
>> for the thread, haning it over to the master process.
>> Yes, this can be tuned in a lot of ways, but most of us with large
>> filesystems would easily sacrifice 5-10GB memory on the server, just for
>> speeding up this stuff.
>>
>
> I do not fully understand your idea. Why do you need a buffer and what is
> a
> "master process"? All I understand is that you want to add multiple
> threads
> to a single job, right?
Well.. code is hard to explain. What I'l like to do would to fork X
processes, allocate X slots in shared-memory (each slot < 10MB) then do
readdir and for f1,...fX I'd hand them off to a thread and read the data
into the shared-memory - signal back to the master thread that runs the
job to read data of the shared memory buffer and signal read of a new
file t the forked process... and so on. Thus serialized the datastream
in memory of the bacula-fd client before encryption, etc, etc.. and
only do it for "small files" as larger work fine allready and is better
streamed directly to the storage daemon.
> I think you should contact Bacula Systems to arrange this kind of
> sponsored development.
Would be great - and we'd like to sponsor development regardless of
who does it. As long as I can apply the patch and get it in production.
> I had a lot of discussions with Kern and Eric about "single job
> multithreaded processing" in the last 5 or 6 years. We designed in our
> minds almost unlimited number of ideas for this project.
Yes, there is many ways to nirvana here..
--
Jesper
_______________________________________________
Bacula-devel mailing list
Bacula-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-devel
