Hello,
Bacula was designed to handle a maximum of 10M files (Bacula 5.0.x).
Since then file systems have grown a lot and so has Bacula. We have
redesigned Bacula a number of times to be able to cope with 50M or more
files. From what I see, your problems are in the following areas:
1. You are running what I would call an antiquated Bacula -- moving to
5.2.x will probably not help much. To make a significant improvement
you should move up to version 7.4.7.
2. MySQL is a great database, but it is primarily designed to be fast
for web servers. It is also easy to use in Bacula. However, if you want
real performance you will need to use PostgreSQL. I switched about 5
years ago and have not regretted it at all.
3. For both MySQL and PostgreSQL, you need to do a certain minimum
tuning to make them preform well with Bacula.
4. To make building the restore tree fast, you need lots of memory. For
the number of files you have, you probably need 32GB. You need to
allocate up to 1/3 to PostgreSQL, which will use it only if it needs it.
5. Trying to make custom indexes usually results in disaster (very bad
performance) in some other area that is discovered later. The same goes
for programs such as PostgreSQL, if you give it too much memory it will
actually run slower. Stick to the developer's recommendations, which
unfortunately not configured by default.
6. You need to be 100% sure that batch insert is turned on (normally it
is by default).
7. Martin gave a good tip about "optimizespeed".
As far as I can tell, this problem has nothing to do with spooling and such.
Best regards,
Kern
On 04/12/2017 09:03 AM, Tom Yates wrote:
> I've got a fairly big filesystem (3TB, 15M files) of which I want to
> (test) restore a part. I know that if the backend DB is slow the
> "Building file list" stage can take some time, but I have it striped over
> a 5-SAS-disc RAID-0, and this step takes only about eight minutes.
>
> The problems start once I navigate to the directory I want restored
> (which admittedly contains the bulk of the files and about half the total
> space), and do an "add home".
>
> The current job has been stuck on this step for over 15 hours, now. When
> I strace bacula-dir I see a lot of:
>
> [pid 26711] write(6, "P\0\0\0\3SELECT FilenameId FROM File"..., 84) = 84
> [pid 26711] read(6, "\1\0\0\1\1@\0\0\2\3def\6bacula\10Filename\10Fi"...,
> 16384) = 102
> [pid 26711] poll([{fd=6, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
> [pid 26711] write(6, "m\0\0\0\3SELECT FileId, LStat, MD5 F"..., 113) = 113
> [pid 26711] read(6, "\1\0\0\1\0030\0\0\2\3def\6bacula\4File\4File\6F"...,
> 16384) = 249
> [pid 26711] poll([{fd=6, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
> [pid 26711] write(6, "P\0\0\0\3SELECT FilenameId FROM File"..., 84) = 84
> [pid 26711] read(6, "\1\0\0\1\1@\0\0\2\3def\6bacula\10Filename\10Fi"...,
> 16384) = 102
> [pid 26711] poll([{fd=6, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
> [pid 26711] write(6, "m\0\0\0\3SELECT FileId, LStat, MD5 F"..., 113) = 113
> [pid 26711] read(6, "\1\0\0\1\0030\0\0\2\3def\6bacula\4File\4File\6F"...,
> 16384) = 250
> [pid 26711] poll([{fd=6, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
>
> So I presume it's stepping through the built directory tree querying the
> database about each of these files. Problem is that any restore that
> takes ~24 hours just to kick off is not making my clients happy.
>
> The CentOS 6 server has 16GB of memory and does not seem short of it
> (negligible swap usage). We're currently using the CentOS 6 bacula
> packages, which are v5.0.0. I tried building 5.2.13 from source,
> upgrading, and running that, but it wasn't noticeably better, so I
> downgraded again. I'm happy to go to a still-later version if there is
> reason to think that this step is better optimised in that version. If
> building custom indexes would help, I'm open to that, too. If I'm doing
> something fundamentally stupid, it would be really useful to know!
>
> Apart from "don't restore your home area", does anyone have any advice?
> Thanks.
>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
