Hello Timothé, I did some tests and analysis for this hanging restore command error. It looks that the Bacula Bvfs restore part responsible for hardlinks causes it. This change came to Bacula in version 11.0.
Looking at this hardlinks part, as a workaround I created an index in the catalog that solved this hanging: CREATE INDEX file_jobid_fileindex_idx ON File (JobId, FileIndex); Creating this index can take some time specially if you have many files in the File table, but later the restore starts without waiting. Creating index is not a solution, better if the hardlinks part could be optimized if possible. For now I don't have any other option for that. For these hardlinks, in Bacula it is a common part for MySQL/MariaDB and PostgreSQL, but with MySQL this problem is much better visible. Best regards, Marcin Haba (gani) On Sat, 6 Apr 2024 at 18:55, Marcin Haba <ganius...@gmail.com> wrote: > Hello Timothé, > > Thanks for providing all details about this restore problem. > > I did tests on a set of 150K files with old Bacula 9.0.6 and the latest > 15.0.2. It looks that something has changed in Bacula MySQL/MariaDB support > in a version between because on the version 9 it works well, while on 15 it > hangs on the restore commands. > > Your restore using bconsole can work much better because it does not use > Bacula Bvfs interface. > > I will try to look at it deeper in the next few days. I will let know if I > find something. > > Best regards, > Marcin Haba (gani) > > On Thu, 4 Apr 2024 at 16:14, timothe pennec <timit...@hotmail.fr> wrote: > >> Hi, >> >> I'm using mariadb for database and it's him seems to be slow : >> top result : >> >> top - 16:08:51 up 2 days, 20:10, 6 users, load average: 0,97, 0,60, 0,29 >> Tasks: 307 total, 1 running, 306 sleeping, 0 stopped, 0 zombie >> %Cpu(s): 12,5 us, 0,0 sy, 0,0 ni, 87,3 id, 0,0 wa, 0,0 hi, 0,2 si, >> 0,0 st >> MiB Mem : 7936,5 total, 177,7 free, 592,8 used, 7166,1 >> buff/cache >> MiB Swap: 4096,0 total, 4095,0 free, 1,0 used. 7026,5 avail Mem >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> 29267 mysql 20 0 1346196 142604 23396 S 100,0 1,8 2:53.58 >> mariadbd >> >> I've 8 vCPU but only one is at 100 % (can see it with htop) all other at >> 0. >> >> I've enabled debug level 500 and i can see many log right after submit >> button "Run" and stuck at : >> >> bacula-dir: ua_restore.c:2015-0 JobId=4792 JobIds= FileIndex=10 >> [All 53000 files ...] >> bacula-dir: ua_restore.c:2015-0 JobId=4792 JobIds= FileIndex=53895 >> bacula-dir: ua_restore.c:2015-0 JobId=4792 JobIds= FileIndex=53896 >> bacula-dir: mysql.c:584-0 db_sql_query finished >> bacula-dir: ua_restore.c:1476-0 query=SELECT Path.Path, File.Filename >> FROM File JOIN Path USING (PathId) JOIN b210091 AS T ON (File.JobId = >> T.JobId AND File.FileIndex = T.FileIndex) WHERE File.Filename LIKE >> ':component_info_%' AND File.JobId IN (4792) >> bacula-dir: mysql.c:527-0 db_sql_query starts with SELECT Path.Path, >> File.Filename FROM File JOIN Path USING (PathId) JOIN b210091 AS T ON >> (File.JobId = T.JobId AND File.FileIndex = T.FileIndex) WHERE File.Filename >> LIKE ':component_info_%' AND File.JobId IN (4792) >> >> >> Any idea ? >> >> Thank you >> Timothé >> >> ------------------------------ >> *De :* Marcin Haba <ganius...@gmail.com> >> *Envoyé :* mercredi 3 avril 2024 21:53 >> *À :* timothe pennec <timit...@hotmail.fr> >> *Cc :* bacula-users@lists.sourceforge.net < >> bacula-users@lists.sourceforge.net> >> *Objet :* Re: [Bacula-users] [Bacularis/Baculum] Restore with web >> doesn't create job before 1 hour >> >> Hello Timothé, >> >> Thanks for your detailed description of the problem. >> >> For start I would propose to check what is going inside Bacula and the >> Catalog when this restore is trying to start. >> >> Yes, observing 'ps aux' is a good idea, that you did. Also useful can be >> 'top' command with watching what process takes the most resources. >> >> From the Bacula side you can set debug for Director by this bconsole >> command: >> >> setdebug level=500 tags=bvfs,sql trace=1 dir >> >> in the Bacula Director working directory you will see the >> youdirector-dir.trace file where this debug lines will be written. >> >> If your Catalog database is PostgreSQL you ran run in the time of the >> restore hanging this SQL query to see currently pending SQL queries: >> >> SELECT * FROM pg_stat_activity; >> >> Please let us know about the results. >> >> Good luck. >> >> Best regards, >> Marcin Haba (gani) >> >> On Wed, 3 Apr 2024 at 20:40, timothe pennec <timit...@hotmail.fr> wrote: >> >> Hi everyone, >> >> I've one issue with Bacularis (and same with Baculum) WEB when i follow >> the restore wizzard : >> >> - I select my client and let checked "Enable restore from copy job >> feature" >> - I select the Last full Backup of my client >> - I select one folder with add button (total of files about 55 000 >> and total syse about 4Go) >> - I select the same client and "/tm/restore" folder for testing >> - I let default setting for Restore job (RestoreFiles and no replace >> file and no relocate files) >> - I submit with "Run restore" >> >> >> When i submit nothing happen (i got only "loading" favicon and not >> redirect or validation message for create job). >> At the same time if i open new tab to default web page it seems the >> server not responding for a while and after a time the page loaded for the >> default tab and error "504 Gateway timeout" for restore tab >> If i ssh and launch bconsole and enter "status director" i can see that >> no job was created (terminated or running or whatever state). >> >> Here a "ps aux | grep bconsole" at this time : >> www-data 21059 0.0 0.0 2892 964 ? S 19:45 0:00 sh -c >> sudo /usr/bin/bconsole -c "/etc/bacula/bconsole.conf" -D bacula-dir 2>&1 >> <<END_OF_DATA gui on restore client="www.example.org" file="?b210091" >> replace="never" restorejob="RestoreFiles" where="/tmp/restore" yes quit >> END_OF_DATA >> root 21060 0.0 0.0 8488 5228 ? S 19:45 0:00 sudo >> /usr/bin/bconsole -c /etc/bacula/bconsole.conf -D bacula-dir >> root 21061 0.0 0.1 165928 10516 ? Sl 19:45 0:00 >> /usr/bin/bconsole -c /etc/bacula/bconsole.conf -D bacula-dir >> >> I'm running with bacula 15.0.2 with apt for server and clients all in >> Ubuntu 22.04 >> The result of "tail -f /opt/bacula/log/bacula.log" is empty for all >> process. >> >> For about 50min / 70 min nothing happen then i got job created ! >> >> For exemple the log for the test run at 17:03 (job created at 18:00 and >> all file copied in < 1 min) >> >> bacula-dir JobId 4831: End auto prune. >> bacula-dir JobId 4831: No Files found to prune. >> bacula-dir JobId 4831: Begin pruning Files. >> bacula-dir JobId 4831: No Jobs found to prune. >> bacula-dir JobId 4831: Begin pruning Jobs older than 6 months . >> bacula-dir JobId 4831: Bacula bacula-dir 15.0.2 (21Mar24): >> Build OS: x86_64-pc-linux-gnu-bacula ubuntu 22.04 >> JobId: 4831 >> Job: RestoreFiles.2024-04-02_18.00.36_48 >> Restore Client: "www.example.org" 15.0.2 (21Mar24) >> x86_64-pc-linux-gnu-bacula,ubuntu,22.04 >> Where: /tmp/restore >> Replace: Never >> Start time: 02-avril-2024 18:00:38 >> End time: 02-avril-2024 18:01:19 >> Elapsed time: 41 secs >> Files Expected: 53,888 >> Files Restored: 53,888 >> Bytes Restored: 4,066,211,495 (4.066 GB) >> Rate: 99175.9 KB/s >> FD Errors: 0 >> FD termination status: OK >> SD termination status: OK >> Termination: Restore OK >> bacula-sd JobId 4831: Elapsed time=00:00:41, Transfer rate=99.38 M >> Bytes/second >> bacula-sd JobId 4831: Forward spacing Volume "Monthlyvol-0017" to >> addr=22149617617 >> bacula-sd JobId 4831: Ready to read from volume "Monthlyvol-0017" on File >> device "FileStorage" (/backup). >> wwwexampleorg-fd JobId 4831: Connected to Storage at >> bacula.esx2.example.org:9103 with TLS >> bacula-dir JobId 4831: Connected to Client "www.example.org" at >> 2001:41d0:303:f006::3:9102 with TLS >> bacula-dir JobId 4831: Using Device "FileStorage" to read. >> bacula-dir JobId 4831: Connected to Storage >> "bacula.esx2.example.org_backup" at bacula.esx2.example.org:9103 with TLS >> bacula-dir JobId 4831: Restoring files from JobId(s) 4792 >> bacula-dir JobId 4831: Start Restore Job >> RestoreFiles.2024-04-02_18.00.36_48 >> >> If i try to run restore by the bconsole command and follow the prompt job >> is created instantaly. >> >> I didn't have test by the past the restore feature but i think something >> is wrong. >> >> Anyone can help me for this issue ? >> >> Best regards >> Timothé >> _______________________________________________ >> Bacula-users mailing list >> Bacula-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/bacula-users >> >> >> >> -- >> >> "Greater love hath no man than this, that a man lay down his life for his >> friends." Jesus Christ >> >> "Większej miłości nikt nie ma nad tę, jak gdy kto życie swoje kładzie za >> przyjaciół swoich." Jezus Chrystus >> >> > > -- > > "Greater love hath no man than this, that a man lay down his life for his > friends." Jesus Christ > > "Większej miłości nikt nie ma nad tę, jak gdy kto życie swoje kładzie za > przyjaciół swoich." Jezus Chrystus > > -- "Greater love hath no man than this, that a man lay down his life for his friends." Jesus Christ "Większej miłości nikt nie ma nad tę, jak gdy kto życie swoje kładzie za przyjaciół swoich." Jezus Chrystus
_______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
