This sounds like a bug in bacula actually. It shouldn't follow recursion into the same structure, simply store the link and move on.
-Blake On Tue, Oct 21, 2014 at 4:17 AM, Ulrich Leodolter <ulrich.leodol...@obvsg.at> wrote: > Hi all, > > i found the root cause of the problem, it was simply a mysql performance > problem because auf special filesystem hierarchy on the users desktop. > > there was one directory which was recursively repeated inside itself. > > C:/Users/name/Desktop/Exercise Files/CSS Core Concepts > C:/Users/name/Desktop/Exercise Files/CSS Core Concepts/Exercise Files/CSS > Core Concepts/ > ... > > there was only one file inside "CSS Core Concepts" and six empty sub > directories > Chapter_01 to Chapter_06. this hierarchy was repeated up to path length of > 4834. > very strange, maybe a zip file containing symlinks pointing to . was > unzipped on desktop. > > the join on path in the batch insert seems to perform very badly comparing > about 27000 > long path names like that. > > INSERT INTO File (FileIndex, JobId, PathId, FilenameId, LStat, MD5, DeltaSeq) > SELECT batch.FileIndex, batch.JobId, Path.PathId, Filename.FilenameId, > batch.LStat, batch.MD5, batch.DeltaSeq FROM batch > JOIN Path ON (batch.Path = Path.Path) JOIN Filename ON (batch.Name = > Filename.Name) > > > now we removed the almost empty tree "C:/Users/name/Desktop/Exercise Files" > and i am sure tomorrow the backup will finish in time without problems. > > Best regards > Ulrich > > > > > On Mon, 2014-10-20 at 15:21 +0200, alejandro alfonso fernandez wrote: >> Hi! >> >> I agree with Martin. It's no a Bacula error, it's a MySQL problem >> >> I'm pretty sure that your /tmp partition becomes full, specially if you >> share both Bacula Spool Directory (are you using "SpoolData = yes"?) and >> MySQL temporary filesystem (both of them in /tmp by default) >> >> Try changing the "tmpdir" param of your MySQL server (my.cnf) to a bigger >> partition (don't forget restart the service to commit the change) >> >> Example: >> # point the following paths to different dedicated disks >> # tmpdir = /tmp/ >> tmpdir = /var/tmp/mysql >> >> Doing a "mysqlrepair" to test database integrity will be a good idea >> >> Best regards! >> >> On Mon, Oct 20, 2014 at 12:57 PM, Martin Simmons <mar...@lispworks.com> >> wrote: >> >> > >>>>> On Sun, 19 Oct 2014 19:02:57 +0200, Ulrich Leodolter said: >> > > >> > > Hello Dan, >> > > >> > > On Sat, 2014-10-18 at 13:32 -0400, Dan Langille wrote: >> > > > On Oct 18, 2014, >> > > > at 4:03 AM, Ulrich Leodolter <ulrich.leodol...@obvsg.at> wrote: >> > > > >> > > > > Hello, >> > > > > >> > > > > we have Win7 backup which does not come to an end within >> > MaxRunTime=12h. >> > > > > server runs 7.0.5 (28 July 2014), the client has installed the >> > > > > bacula-enterprise-win64-7.0.5.exe. but the problem started about 2 >> > > > > months ago, at that time windows client 5.2.10 was installed on the >> > > > > machine. >> > > > > >> > > > > the backup itself is about 100GB compressed and seems to finish >> > > > > on the client after about 6 hours, below are the last messages of >> > > > > the job before it gets stuck. >> > > > > >> > > > > 2014-10-18 03:18:09 troll-sd JobId 635821: Committing spooled data to >> > > > > Volume "Backup-0779". Despooling 1,692,736,419 bytes ... >> > > > > 2014-10-18 03:18:18 troll-sd JobId 635821: Despooling elapsed time = >> > 00:00:09, Transfer rate = 188.0 M Bytes/second >> > > > > 2014-10-18 03:18:19 troll-sd JobId 635821: Elapsed time=06:11:45, >> > Transfer rate=4.691 M Bytes/second >> > > > > 2014-10-18 03:18:22 troll-sd JobId 635821: Sending spooled attrs to >> > the Director. Despooling 603,449,667 bytes . >> > > > > >> > > > > mysql status at the same time: >> > > > > >> > > > > # echo "show full processlist" | mysql >> > > > > Id User Host db Command Time State Info >> > > > > 6854 bacula localhost bacula Sleep 522 >> > NULL >> > > > > 6873 bacula localhost bacula Query 21143 Sending >> > data INSERT INTO File (FileIndex, JobId, PathId, FilenameId, LStat, MD5, >> > DeltaSeq) SELECT batch.FileIndex, batch.JobId, Path.PathId, >> > Filename.FilenameId,batch.LStat, batch.MD5, batch.DeltaSeq FROM batch JOIN >> > Path ON (batch.Path = Path.Path) JOIN Filename ON (batch.Name = >> > Filename.Name) >> > > > > 6899 root localhost NULL Query 0 NULL >> > show full processlist >> > > > > >> > > > > >> > > > > we have a bunch of other clients (about 30), a mixture of linux, >> > win7 and mac powerpc. >> > > > > all other backups run without problems for years now. there are >> > even larger backups, >> > > > > in size and in number of files. >> > > > > >> > > > > >> > > > > does anyone have an idea why this single batch insert does not >> > complete? >> > > > > >> > > > > do i need to analyze the attrs spool file itself ? >> > > > > >> > > > > yesterday i optimized the bacula database, but that doesn't help. >> > > > > there must be something special in the attrs spool file which the >> > > > > mysql server can't handle. the server runs on standard CentOS 6.5 >> > x86_64. >> > > > >> > > > This is something which should be asked in the user mailing list, not >> > the devel mailing list. I am replying to that list instead. >> > > > >> > > >> > > Ok >> > > >> > > > Is this a large number of files? >> > > > >> > > >> > > about 750000, not very large. >> > > >> > > > I had something which took a while. I sped it up by giving PostgreSQL >> > more memory. Perhaps MySQL can do the same. >> > > > >> > > > Here’s what I did: >> > https://plus.google.com/+DanLangille/posts/AKXoRido3U1 >> > > > >> > > >> > > the mysql database is already optimized and has enough memory. >> > > backups up 4000000 files and 700GB run without problems. >> > > >> > > below are last job messages of the last failed jobs, it was canceled >> > > after 12 hours max run time. >> > > >> > > 2014-10-19 02:49:15 troll-sd JobId 635915: Elapsed time=05:43:32, >> > Transfer rate=5.095 M Bytes/second >> > > 2014-10-19 02:49:16 troll-sd JobId 635915: Sending spooled attrs to the >> > Director. Despooling 603,418,522 bytes ... >> > > 2014-10-19 09:05:43 troll-dir JobId 635915: Fatal error: Max run time >> > exceeded. Job canceled. >> > > 2014-10-19 09:41:17 troll-dir JobId 635915: Error: Bacula troll-dir >> > 7.0.5 (28Jul14): >> > > >> > > the database shows already about 600k files for this jobs. >> > > >> > > mysql> select count(*) from File where JobId = '635915' and LStat is not >> > > null; >> > > +----------+ >> > > | count(*) | >> > > +----------+ >> > > | 616848 | >> > > +----------+ >> > > 1 row in set (0.00 sec) >> > > >> > > >> > > is it possible that some special file attr brings the mysql datebase >> > > into troubles ? i really cant imagine. the mysql database is almost >> > > idle after first 600k have been inserted. there is no io traffic and >> > > cpu usage is also low. >> > > >> > > it seems i have to dump spooled attrs files tomorrow and compare to >> > > database to see at which attrs have been inserted and at which one it >> > > stops. >> > >> > I think that will be difficult, because the insert is a join of various >> > other >> > temporary tables so the order may be random. >> > >> > >> > > has anyone a better idea how to debug this problem? >> > > i am little bit lost :) because in the last 6 years since >> > > i am using bacula i never run into a problem like this. >> > >> > Maybe you can attach strace or gdb to the mysql process running this insert >> > statement to see what it is doing? It doesn't look like a Bacula problem, >> > but >> > it might be a bug in mysql. >> > >> > __Martin >> > >> > >> > ------------------------------------------------------------------------------ >> > Comprehensive Server Monitoring with Site24x7. >> > Monitor 10 servers for $9/Month. >> > Get alerted through email, SMS, voice calls or mobile push notifications. >> > Take corrective actions from your mobile device. >> > http://p.sf.net/sfu/Zoho >> > _______________________________________________ >> > Bacula-users mailing list >> > Bacula-users@lists.sourceforge.net >> > https://lists.sourceforge.net/lists/listinfo/bacula-users >> > > > > > ------------------------------------------------------------------------------ > Comprehensive Server Monitoring with Site24x7. > Monitor 10 servers for $9/Month. > Get alerted through email, SMS, voice calls or mobile push notifications. > Take corrective actions from your mobile device. > http://p.sf.net/sfu/Zoho > _______________________________________________ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users ------------------------------------------------------------------------------ Comprehensive Server Monitoring with Site24x7. Monitor 10 servers for $9/Month. Get alerted through email, SMS, voice calls or mobile push notifications. Take corrective actions from your mobile device. http://p.sf.net/sfu/Zoho _______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
