Hi, Thanks to all who responded and offered advice. I didn't want to get into technical details of the server and software in this thread and I do appreciate your time. I was just trying to get a "reality-check" that I wasn't trying to do something silly. The message I've gotten is that Bacula can work with the requirements I have, on the hardware that I have, and that the performance issues I'm seeing are because there is something currently wrong with my implementation.
Thanks, Rich. On Wed, 19 Feb 2014, r...@mbl.edu wrote: > Hi, > > I'm using accurate mode on my system and the mysql queries crawl along > causing the whole backup to run very slowly. > > I have 25 million files on a filesystem and I'm expected to be able to > restore any file that existed on the filesystem on any day for a period of > 1.5 years. > > Based on the performance of previous backup systems run on this server, I > split the filesystem up into four groups and created bacula jobs for each > group. Each group has its own dedicated logical library but all four jobs > use the same MySQL database. I've been using Bacula in accurate mode for > this system since Dec 20. While it was delightfully speedy at first over > time it has become quite slow. The MySQL queries used during the run seem > to be the bottleneck and I am worried the problem is insurmountable. > > I have been going through rounds of mysql tuning to try to increase > performance but it's a powerful computer with 24 cores (2 Xeon E5645s @ > 2.4GHz), mysql is using over 32G RAM, the database is on a SSD and MySQL > tmp is on a pair of striped SSDs. The query that hangs everything up > during the process takes more than 24 hours to get MySQL to 'explain' and > jobs take about 1.7 days to run at this point. (A job is running now and > the same query has been executing for over 62000 seconds.) I am using > MySQL 5.5 though, so I may be able to benefit from a more recent version. > > I've been running the MySQL sql-bench benchmarks on a number of different > servers I have available and this server seems to be quite fast matching > performance we get from our dedicated database servers with high speed > enterprise disk arrays and 128G RAM. (Except when dropping tables, the > dedicated systems are much faster than this server, but I don't believe > that is related to the slow query problem I am seeing.) > > I would chalk it up to a simple overestimation on my part of the ability > of accurate mode on a system this size but in my experience I have run > across quite a few statements about big bacula implementations that have > billions of rows in the File table. Mine currently only has 110 million or > so. I can't help but wonder if these references to large tables are on > systems that run in non-accurate mode and consequently don't perform the > complex queries that I'm seeing. > > I expected a accurate mode to be difficult to implement without incurring > a performance hit but I wonder if I'm not exceeding realistic expectations > of the file selection algorithm with this size filesystem. > > Thanks, > Rich. > > > -- Rich Fox Systems Administrator JBPC - Marine Biological Laboratory http://www.mbl.edu/jbpc 508-289-7669 - mbl-at-richfox.org ------------------------------------------------------------------------------ Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121054471&iu=/4140/ostg.clktrk _______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
