Re: [Bacula-users] Bacula and memory usage
On 2020-01-28 11:06, Thomas Lohman wrote: >> Hello >> >> I am used to this principle with Linux but I don't understand why it just >> takes it when Bacula is working and it slows down the server so much that I >> can no longer access it in ssh. > > How is your storage allocated on the server? i.e. how are things > partitioned with regard to your backup disks and your database? If your > DB is located on the same physical disks as your OS and/or your actual > backup data then you could see such "freeze ups" while Bacula is running > due to I/O limitations. I find it helps to separate the OS, DB data and > any Bacula storage volumes so they are all on separate disk devices if > possible - separate controllers even better. In particular it is a VERY BAD IDEA to have your catalog DB and your Bacula backup storage on the same physical devices. Both of those are high-I/O operations. You can EASILY saturate your I/O with a configuration like this, especially if you're writing to RAID5 where every write requires a parity calculation. -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
Hello I am used to this principle with Linux but I don't understand why it just takes it when Bacula is working and it slows down the server so much that I can no longer access it in ssh. How is your storage allocated on the server? i.e. how are things partitioned with regard to your backup disks and your database? If your DB is located on the same physical disks as your OS and/or your actual backup data then you could see such "freeze ups" while Bacula is running due to I/O limitations. I find it helps to separate the OS, DB data and any Bacula storage volumes so they are all on separate disk devices if possible - separate controllers even better. --tom ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
On 1/28/2020 9:45 AM, Jean Mark Orfali wrote:
Hello
I am used to this principle with Linux but I don't understand why it just takes
it when Bacula is working and it slows down the server so much that I can no
longer access it in ssh.
Typically,
- failing drive,
- saturated network,
- broken name resolution.
If it only happens when bacula is running, I'd first run `iostat -dmx 5
5` and see if the drive where "File1" is located is clocking high wait
time and near-100% utilization.
If yes, throw it away and replace with a TLER ("NAS") HDD.
Dima
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
On 2020-01-28 10:45, Jean Mark Orfali wrote: > Hello > > I am used to this principle with Linux but I don't understand why it just > takes it when Bacula is working and it slows down the server so much that I > can no longer access it in ssh. 1 - Presumably only Bacula is doing enough file access on this server that it needs to cache that many files. 2 - A slowdown such as you describe has nothing to do with file caching in memory. Memory is FAST. That's the POINT of caching recently opened files in memory. Have you looked at your storage I/O on this server? What is its physical and logical storage configuration? Do you know how to use iostat? Have you ruled out the case that your server is simply I/O bound - which is to say, you are saturating the I/O capacity of your physical storage? -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
On 2020-01-28 10:08, Jean Mark Orfali wrote: > Hello Phil and Thomas, > > Here is the result of the bconsol. > > > Select daemon type for status (1-5): 5 > bacula-dir Version: 7.0.5 (28 July 2014) x86_64-redhat-linux-gnu redhat > Enterprise release > Daemon started 27-Jan-20 14:53. Jobs: run=0, running=5 mode=0,0 > Heap: heap=270,336 smbytes=652,990 max_bytes=737,558 bufs=557 max_bufs=563 > > Scheduled Jobs: > Level Type Pri Scheduled Job Name Volume > === > IncrementalBackup10 28-Jan-20 23:00BackOVH*unknown* > IncrementalBackup10 28-Jan-20 23:00BackphppROD*unknown* > IncrementalBackup10 28-Jan-20 23:00BackServeurFichier *unknown* > IncrementalBackup10 28-Jan-20 23:00BackBlanchard *unknown* > > > Running Jobs: > Console connected at 27-Jan-20 15:36 > Console connected at 28-Jan-20 10:01 > JobId Type Level Files Bytes Name Status > == > 75 Back Full 0 0 BackphppROD is running > 76 Back Incr 0 0 BackOVH is waiting on > Storage "File1" > 77 Back Full 0 0 BackphppROD is waiting on max > Client jobs > 78 Back Full 0 0 BackServeurFichier is waiting on > Storage "File1" > 79 Back Full 0 0 BackBlanchard is waiting on > Storage "File1" > OK, this looks pretty clear that something is not working as intended in your Storage configuration. You have a full backup running on BackphppROD, and your Storage is configured such that nothing else can write while that job is running. What's more, you have a second job waiting on BackphppROD that can't run. Now we have some idea where to look. For one thing, you should probably be using the following directives in your Director configuration: Allow Duplicate Jobs = no Cancel Queued Duplicates = yes What this does is tell the Director not to queue or run any new job on a client while another copy of that job, at the same or different level, is already running. This will prevent the situation above where you have a Full backup job BackphppROD running, and another Full backup copy (probably promoted from Incremental) queued to run as soon as it finishes. Now on to your Storage configuration. Let's see... What it looks like to me is that your fundamental problem here is that you have each client assigned its own POOL, with its own set of Volumes, but you only have one Storage device, which can only have one Volume open at a time. This means that EVERY job will block on ANY other running job because your Storage can only service one client at a time since your Clients each have their own unique Pools. (I see you've set up the virtual autochanger, but I don't use that and have no experience with it, so I can't speak about it.) Do you have an OPERATIONAL NEED to keep the backup data from different Clients separated on different Volumes? -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
Hello I am used to this principle with Linux but I don't understand why it just takes it when Bacula is working and it slows down the server so much that I can no longer access it in ssh. Jean Mark Orfali • Sys admin T. 1-877-258-4542 P. 227 bravad.ca/Fabrique Avis de confidentialité Le contenu de ce message ainsi que du ou des fichiers qui y sont joints est strictement confidentiel et destiné exclusivement à son ou sa destinataire. Si vous n’êtes pas cette personne, nous attirons votre attention sur le fait qu’il est strictement interdit de copier, de faire suivre ou d’utiliser les informations contenues dans ce courriel. Si vous l’avez reçu par erreur, nous vous remercions de nous le faire savoir. -Message d'origine- De : Phil Stracchino Envoyé : Tuesday, 28 January, 2020 10:36 AM À : bacula-users@lists.sourceforge.net Objet : Re: [Bacula-users] Bacula and memory usage On 2020-01-28 09:49, Thomas Lohman wrote: > > >> %Cpu(s): 0.1 us, 0.2 sy, 0.0 ni, 52.9 id, 46.5 wa, 0.0 hi, 0.2 si, 0.0 >> st >> KiB Mem : 29987532 total, 220092 free, 697356 used, 29070084 buff/cache >> KiB Swap: 15138812 total, 15138812 free,0 used. 28880936 avail Mem > > It looks like your memory is being used by the Linux file cache. This > is typical and if the system needs the memory for something else, it > will use it. Yup, agreed. This is something that often looks alarming to people coming from the Windows world, but in thew Unix world it is *perfectly normal*. Memory that you're not using is memory that isn't doing anything for you, and if you have free memory Unix will put it to use for something useful — caching files — until it needs it for something else. -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
On 2020-01-28 09:49, Thomas Lohman wrote: > > >> %Cpu(s): 0.1 us, 0.2 sy, 0.0 ni, 52.9 id, 46.5 wa, 0.0 hi, 0.2 si, 0.0 >> st >> KiB Mem : 29987532 total, 220092 free, 697356 used, 29070084 buff/cache >> KiB Swap: 15138812 total, 15138812 free,0 used. 28880936 avail Mem > > It looks like your memory is being used by the Linux file cache. This is > typical and if the system needs the memory for something else, it will > use it. Yup, agreed. This is something that often looks alarming to people coming from the Windows world, but in thew Unix world it is *perfectly normal*. Memory that you're not using is memory that isn't doing anything for you, and if you have free memory Unix will put it to use for something useful — caching files — until it needs it for something else. -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
cted at: 28-Jan-20 16:01 No Jobs running. Terminated Jobs: JobId LevelFiles Bytes Status FinishedName == 42 Full 2 0 OK 22-Jan-20 20:42 BackServeurFichier 432 0 OK 22-Jan-20 20:43 RestoreOVH 57 Full167,746475.5 G Error27-Jan-20 19:43 BackServeurFichier Connecting to Client 162.241.5.144-fd at 162.241.5.144:9102 bacula-fd3 Version: 5.2.13 (19 February 2013) x86_64-redhat-linux-gnu redhat (Core) Daemon started 22-Jan-20 15:03. Jobs: run=3 running=0. Heap: heap=32,768 smbytes=23,481 max_bytes=319,714 bufs=56 max_bufs=109 Sizeof: boffset_t=8 size_t=8 debug=0 trace=0 Running Jobs: Director connected at: 28-Jan-20 10:01 No Jobs running. Terminated Jobs: JobId LevelFiles Bytes Status FinishedName == 44 Full 2 0 OK 22-Jan-20 15:08 BackBlanchard 452 0 OK 22-Jan-20 15:09 RestoreOVH 46 Full 1,749167.8 G Error23-Jan-20 09:32 BackBlanchard * Jean Mark Orfali • Sys admin T. 1-877-258-4542 P. 227 bravad.ca/Fabrique Avis de confidentialité Le contenu de ce message ainsi que du ou des fichiers qui y sont joints est strictement confidentiel et destiné exclusivement à son ou sa destinataire. Si vous n’êtes pas cette personne, nous attirons votre attention sur le fait qu’il est strictement interdit de copier, de faire suivre ou d’utiliser les informations contenues dans ce courriel. Si vous l’avez reçu par erreur, nous vous remercions de nous le faire savoir. -Message d'origine- De : Thomas Lohman Envoyé : Tuesday, 28 January, 2020 9:50 AM À : bacula-users@lists.sourceforge.net Objet : Re: [Bacula-users] Bacula and memory usage > %Cpu(s): 0.1 us, 0.2 sy, 0.0 ni, 52.9 id, 46.5 wa, 0.0 hi, 0.2 si, 0.0 > st > KiB Mem : 29987532 total, 220092 free, 697356 used, 29070084 buff/cache > KiB Swap: 15138812 total, 15138812 free,0 used. 28880936 avail Mem It looks like your memory is being used by the Linux file cache. This is typical and if the system needs the memory for something else, it will use it. As mentioned in my previous e-mail, can you run status within the director (bconsole) and see what the clients are doing when the backups are running? Is bacula actually backing anything up? The first thing to determine is if there is a problem/malfunction or if possibly your backups are simply taking too long to run (due to data total, # of files, etc.). --tom ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
%Cpu(s): 0.1 us, 0.2 sy, 0.0 ni, 52.9 id, 46.5 wa, 0.0 hi, 0.2 si, 0.0 st KiB Mem : 29987532 total, 220092 free, 697356 used, 29070084 buff/cache KiB Swap: 15138812 total, 15138812 free,0 used. 28880936 avail Mem It looks like your memory is being used by the Linux file cache. This is typical and if the system needs the memory for something else, it will use it. As mentioned in my previous e-mail, can you run status within the director (bconsole) and see what the clients are doing when the backups are running? Is bacula actually backing anything up? The first thing to determine is if there is a problem/malfunction or if possibly your backups are simply taking too long to run (due to data total, # of files, etc.). --tom ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
On 2020-01-28 08:30, Jean Mark Orfali wrote: > Hello Phil, > > I restarted the backups last night and automatically the memory increased to > 96% of use. 28.33GIB. Here are the results for mysqltuner and top. > Performance Metrics > --- > [--] Up for: 17h 30m 18s (109K q [1.735 qps], 53 conn, TX: 19M, RX: 639M) > [--] Reads / Writes: 5% / 95% > [--] Binary logging is disabled > [--] Physical Memory : 28.6G > [--] Max MySQL memory: 987.0M OK, so MariaDB is still using inder 1GB of RAM. So that's not the problem. > > top - 08:26:22 up 17:32, 2 users, load average: 5.43, 5.28, 5.53 > Tasks: 219 total, 1 running, 209 sleeping, 0 stopped, 9 zombie > %Cpu(s): 0.1 us, 0.2 sy, 0.0 ni, 52.9 id, 46.5 wa, 0.0 hi, 0.2 si, 0.0 > st > KiB Mem : 29987532 total, 220092 free, 697356 used, 29070084 buff/cache > KiB Swap: 15138812 total, 15138812 free,0 used. 28880936 avail Mem > > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND > 2093 bacula20 0 763968 6688 3100 S 15.6 0.0 23:33.51 bacula-sd > 6 root 20 0 0 0 0 S 2.3 0.0 3:10.41 > ksoftirqd/0 >93 root 20 0 0 0 0 S 0.3 0.0 0:16.80 kswapd0 > 2088 bacula20 0 1070748 9480 3820 S 0.3 0.0 2:39.34 bacula-dir > And according to top, bacula is using almost no memory either. In fact according to top, nothing is using any significiant amount of CPU *OR* memory. Upon what evidence are you basing your belief that you have a memory usage problem? Because top says you don't. -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
0 0 0 0 S 0.0 0.0 0:00.57 watchdog/0 12 root rt 0 0 0 0 S 0.0 0.0 0:00.31 watchdog/1 13 root rt 0 0 0 0 S 0.0 0.0 0:00.04 migration/1 14 root 20 0 0 0 0 S 0.0 0.0 0:00.02 ksoftirqd/1 16 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H 17 root rt 0 0 0 0 S 0.0 0.0 0:00.31 watchdog/2 18 root rt 0 0 0 0 S 0.0 0.0 0:00.01 migration/2 19 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/2 21 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/2:0H 22 root rt 0 0 0 0 S 0.0 0.0 0:00.31 watchdog/3 23 root rt 0 0 0 0 S 0.0 0.0 0:00.04 migration/3 24 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/3 26 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:0H 27 root rt 0 0 0 0 S 0.0 0.0 0:00.31 watchdog/4 28 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/4 29 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/4 31 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/4:0H 32 root rt 0 0 0 0 S 0.0 0.0 0:00.31 watchdog/5 33 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/5 34 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/5 36 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/5:0H 37 root rt 0 0 0 0 S 0.0 0.0 0:00.31 watchdog/6 38 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/6 39 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/6 41 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/6:0H Jean Mark Orfali • Sys admin T. 1-877-258-4542 P. 227 bravad.ca/Fabrique Avis de confidentialité Le contenu de ce message ainsi que du ou des fichiers qui y sont joints est strictement confidentiel et destiné exclusivement à son ou sa destinataire. Si vous n’êtes pas cette personne, nous attirons votre attention sur le fait qu’il est strictement interdit de copier, de faire suivre ou d’utiliser les informations contenues dans ce courriel. Si vous l’avez reçu par erreur, nous vous remercions de nous le faire savoir. -Message d'origine- De : Phil Stracchino Envoyé : Monday, 27 January, 2020 8:09 PM À : Jean Mark Orfali ; bacula-users@lists.sourceforge.net Objet : Re: [Bacula-users] Bacula and memory usage On 2020-01-27 19:39, Jean Mark Orfali wrote: > Sorry, I meant to say: I had to reboot the server. > Her is the result of top: So having rebooted the server, there is CURRENTLY nothing to see. You can't diagnose a problem when it isn't there. If your backup jobs stall again, please try to report what is happening on the system when the jobs are actually stalled. We need to be able to see what is going wrong to be able to help you diagnose the problem. -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
On 2020-01-27 19:39, Jean Mark Orfali wrote: > Sorry, I meant to say: I had to reboot the server. > Her is the result of top: So having rebooted the server, there is CURRENTLY nothing to see. You can't diagnose a problem when it isn't there. If your backup jobs stall again, please try to report what is happening on the system when the jobs are actually stalled. We need to be able to see what is going wrong to be able to help you diagnose the problem. -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
Sorry, I meant to say: I had to reboot the server. Her is the result of top: top - 19:38:12 up 4:44, 2 users, load average: 3.67, 5.03, 4.62 Tasks: 202 total, 2 running, 200 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.1 us, 0.2 sy, 0.0 ni, 92.0 id, 7.6 wa, 0.0 hi, 0.1 si, 0.0 st KiB Mem : 29987532 total, 224088 free, 653832 used, 29109612 buff/cache KiB Swap: 15138812 total, 15138812 free,0 used. 28932832 avail Mem PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND 2093 bacula20 0 395308 4300 3100 S 5.3 0.0 5:09.74 bacula-sd 5634 root 20 0 162036 2316 1524 R 0.7 0.0 0:00.03 top 2088 bacula20 0 554584 7232 3828 S 0.3 0.0 0:24.96 bacula-dir 5591 root 20 0 0 0 0 S 0.3 0.0 0:00.31 kworker/0:2 1 root 20 0 191156 4096 2512 S 0.0 0.0 0:02.76 systemd 2 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kthreadd 4 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H 6 root 20 0 0 0 0 R 0.0 0.0 0:53.52 ksoftirqd/0 7 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh 9 root 20 0 0 0 0 S 0.0 0.0 0:03.34 rcu_sched 10 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 lru-add-drain 11 root rt 0 0 0 0 S 0.0 0.0 0:00.13 watchdog/0 12 root rt 0 0 0 0 S 0.0 0.0 0:00.08 watchdog/1 13 root rt 0 0 0 0 S 0.0 0.0 0:00.02 migration/1 14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 16 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H 17 root rt 0 0 0 0 S 0.0 0.0 0:00.08 watchdog/2 18 root rt 0 0 0 0 S 0.0 0.0 0:00.01 migration/2 19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2 21 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/2:0H 22 root rt 0 0 0 0 S 0.0 0.0 0:00.08 watchdog/3 23 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/3 24 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3 26 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:0H 27 root rt 0 0 0 0 S 0.0 0.0 0:00.08 watchdog/4 28 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/4 29 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/4 31 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/4:0H 32 root rt 0 0 0 0 S 0.0 0.0 0:00.08 watchdog/5 33 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/5 34 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/5 35 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/5:0 36 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/5:0H 37 root rt 0 0 0 0 S 0.0 0.0 0:00.08 watchdog/6 38 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/6 39 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/6 41 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/6:0H 42 root rt 0 0 0 0 S 0.0 0.0 0:00.08 watchdog/7 Jean Mark Orfali • Sys admin T. 1-877-258-4542 P. 227 bravad.ca/Fabrique Avis de confidentialité Le contenu de ce message ainsi que du ou des fichiers qui y sont joints est strictement confidentiel et destiné exclusivement à son ou sa destinataire. Si vous n’êtes pas cette personne, nous attirons votre attention sur le fait qu’il est strictement interdit de copier, de faire suivre ou d’utiliser les informations contenues dans ce courriel. Si vous l’avez reçu par erreur, nous vous remercions de nous le faire savoir. -Message d'origine- De : Phil Stracchino Envoyé : Monday, 27 January, 2020 7:02 PM À : Jean Mark Orfali ; bacula-users@lists.sourceforge.net Objet : Re: [Bacula-users] Bacula and memory usage On 2020-01-27 18:50, Jean Mark Orfali wrote: > Hello Phil, > > It indicates less than 24 hours of function because I had to leave it to > access SSH. The RAM was too high and I had no access to anything ... I do not > have Plesk or Cpanel on the server. Only what is needed to run Bacula. I left > a single backup and the memory went up to 96% ... "Had to leave it to access SSH"? What exactly do you mean by that? What processes does top tell you are the highest memory users? -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485
Re: [Bacula-users] Bacula and memory usage
On 2020-01-27 18:50, Jean Mark Orfali wrote: > Hello Phil, > > It indicates less than 24 hours of function because I had to leave it to > access SSH. The RAM was too high and I had no access to anything ... I do not > have Plesk or Cpanel on the server. Only what is needed to run Bacula. I left > a single backup and the memory went up to 96% ... "Had to leave it to access SSH"? What exactly do you mean by that? What processes does top tell you are the highest memory users? -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
Hello Phil, It indicates less than 24 hours of function because I had to leave it to access SSH. The RAM was too high and I had no access to anything ... I do not have Plesk or Cpanel on the server. Only what is needed to run Bacula. I left a single backup and the memory went up to 96% ... Jean Mark Orfali • Sys admin T. 1-877-258-4542 P. 227 bravad.ca/Fabrique Avis de confidentialité Le contenu de ce message ainsi que du ou des fichiers qui y sont joints est strictement confidentiel et destiné exclusivement à son ou sa destinataire. Si vous n’êtes pas cette personne, nous attirons votre attention sur le fait qu’il est strictement interdit de copier, de faire suivre ou d’utiliser les informations contenues dans ce courriel. Si vous l’avez reçu par erreur, nous vous remercions de nous le faire savoir. -Message d'origine- De : Phil Stracchino Envoyé : Monday, 27 January, 2020 6:27 PM À : Jean Mark Orfali ; bacula-users@lists.sourceforge.net Objet : Re: [Bacula-users] Bacula and memory usage On 2020-01-27 15:34, Jean Mark Orfali wrote: > Hello Phil, > > Her is the result : OK, there's good news and bad news here. The good news is, your Bacula DB is, so far, so tiny that you're getting by just fine with out-of-the-box MariaDB default settings. In fact, it's smaller than it *SHOULD* be if you're backing up 6TB of data. The bad news is, you said your backups have been running for five days, but mysqltuner says mariadb has been up less than 24 hours. There's two obvious possibilities here. 1: mariadb keeps crashing. If this is the case, you need to find out why. 2: something keeps restarting mariadb. You're not running Plesk, are you...? Check your MariaDB error log and syslog. If you're saying that system memory usage is constantly high, what MAY be happening here is that your system is running out of memory and the kernel is killing mysqld because it sees it as the largest single consumer of memory. (Though with only a 128.0MB InnoDB buffer pool I would find that surprising.) What ELSE runs on your Bacula server? Are you running a webserver on it as well? Do you have resource limits set on the webserver to stop it from spawning httpd processes until it consumes all free memory? (A common error.) Whatever the reason, if the running mysqld process goes away while the Director is trying to read from or write to it, Bacula is going to have a problem. If MariaDB *really is* being constantly restarted before any jobs can finish, then that's a large part of why your jobs can't finish, and you need to solve that problem before you can get anywhere further. -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
On 2020-01-27 15:34, Jean Mark Orfali wrote: > Hello Phil, > > Her is the result : OK, there's good news and bad news here. The good news is, your Bacula DB is, so far, so tiny that you're getting by just fine with out-of-the-box MariaDB default settings. In fact, it's smaller than it *SHOULD* be if you're backing up 6TB of data. The bad news is, you said your backups have been running for five days, but mysqltuner says mariadb has been up less than 24 hours. There's two obvious possibilities here. 1: mariadb keeps crashing. If this is the case, you need to find out why. 2: something keeps restarting mariadb. You're not running Plesk, are you...? Check your MariaDB error log and syslog. If you're saying that system memory usage is constantly high, what MAY be happening here is that your system is running out of memory and the kernel is killing mysqld because it sees it as the largest single consumer of memory. (Though with only a 128.0MB InnoDB buffer pool I would find that surprising.) What ELSE runs on your Bacula server? Are you running a webserver on it as well? Do you have resource limits set on the webserver to stop it from spawning httpd processes until it consumes all free memory? (A common error.) Whatever the reason, if the running mysqld process goes away while the Director is trying to read from or write to it, Bacula is going to have a problem. If MariaDB *really is* being constantly restarted before any jobs can finish, then that's a large part of why your jobs can't finish, and you need to solve that problem before you can get anywhere further. -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
mediate / 71 locks) Performance schema [--] Performance schema is disabled. [--] Memory used by P_S: 0B [--] Sys schema isn't installed. ThreadPool Metrics [--] ThreadPool stat is enabled. [--] Thread Pool Size: 12 thread(s). [--] Using default value is good enough for your version (5.5.64-MariaDB) MyISAM Metrics [!!] Key buffer used: 18.2% (24M used / 134M cache) [OK] Key buffer size / total MyISAM indexes: 128.0M/100.0K InnoDB Metrics [--] InnoDB is enabled. [--] InnoDB Thread Concurrency: 0 [!!] InnoDB File per table is not activated [OK] InnoDB buffer pool / data size: 128.0M/1.6M [!!] Ratio InnoDB log file size / InnoDB Buffer pool size (7.8125 %): 5.0M * 2/128.0M should be equal to 25% [OK] InnoDB buffer pool instances: 1 [--] InnoDB Buffer Pool Chunk Size not used or defined in your version [OK] InnoDB Read buffer efficiency: 92.15% (4863 hits/ 5277 total) [!!] InnoDB Write Log efficiency: 100% (5 hits/ 5 total) [OK] InnoDB log waits: 0.00% (0 waits / 10 writes) AriaDB Metrics [--] AriaDB is enabled. [OK] Aria pagecache size / total Aria indexes: 128.0M/1B TokuDB Metrics [--] TokuDB is disabled. XtraDB Metrics [--] XtraDB is disabled. Galera Metrics [--] Galera is disabled. Replication Metrics --- [--] Galera Synchronous replication: NO [--] No replication slave(s) for this server. [--] Binlog format: STATEMENT [--] XA support enabled: ON [--] Semi synchronous replication Master: Not Activated [--] Semi synchronous replication Slave: Not Activated [--] This is a standalone server Recommendations --- General recommendations: Control warning line(s) into /var/log/mariadb/mariadb.log file MySQL was started within the last 24 hours - recommendations may be inaccurate Enable the slow query log to troubleshoot bad queries Reduce or eliminate unclosed connections and network issues Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1 Set thread_cache_size to 4 as a starting value For MySQL 5.6.2 and lower, Max combined innodb_log_file_size should have a ceiling of (4096MB / log files in group) - 1MB. Before changing innodb_log_file_size and/or innodb_log_files_in_group read this: https://bit.ly/2TcGgtU Variables to adjust: SET innodb_stats_on_metadata = OFF query_cache_size (=0) query_cache_type (=0) query_cache_limit (> 1M, or use smaller result sets) thread_cache_size (start at 4) innodb_file_per_table=ON innodb_log_file_size should be (=16M) if possible, so InnoDB total log files size equals to 25% of buffer pool size. [root@localhost ~]# Jean Mark Orfali • Sys admin T. 1-877-258-4542 P. 227 bravad.ca/Fabrique Avis de confidentialité Le contenu de ce message ainsi que du ou des fichiers qui y sont joints est strictement confidentiel et destiné exclusivement à son ou sa destinataire. Si vous n’êtes pas cette personne, nous attirons votre attention sur le fait qu’il est strictement interdit de copier, de faire suivre ou d’utiliser les informations contenues dans ce courriel. Si vous l’avez reçu par erreur, nous vous remercions de nous le faire savoir. -Message d'origine- De : Phil Stracchino Envoyé : Monday, 27 January, 2020 2:31 PM À : Jean Mark Orfali ; bacula-users@lists.sourceforge.net Objet : Re: [Bacula-users] Bacula and memory usage On 2020-01-27 14:11, Jean Mark Orfali wrote: > Hello, > > Thank you for your reply. Here is the missing information. My Bacula server > and the four clients are on Linux Centos 7 servers. I use Webmin version > 1.941 to access bacula. The bacula version is 7.0.5. The SQL server is a > MariaDB version 5.5.64. The server has 30TB of hard drive and 30GB of memory. > Backups are saved in a directory directly on the backup server. No backup is > kept on clients side. At the moment there is 6 TB of data to backup. On each > of the 4 clients I have an incremental backup task schedule every day at 11 > p.m. Right now I have 4 backups running for 5 days and 14 waiting. Jean, How is that MariaDB instance configured? As a quick-and-dirty first approximation, what does my
Re: [Bacula-users] Bacula and memory usage
"
# Media Type = DDS-4
# Archive Device = /dev/nsa1
# AutomaticMount = yes; # when device opened, read it
# AlwaysOpen = yes
# Offline On Unmount = no
# Hardware End of Medium = no
# BSF at EOM = yes
# Backward Space Record = no
# Fast Forward Space File = no
# TWO EOF = yes
# #
# # New alert command in Bacula 9.0.0
# # Note: you must have the sg3_utils (rpms) or the
# #sg3-utils (deb) installed on your system.
# #and you must set the correct control device that
# #corresponds to the Archive Device
# Control Device = /dev/sg?? # must be SCSI ctl for /dev/nsa1
# Alert Command = "/usr/libexec/bacula/tapealert %l"
#
# If you have smartctl, enable this, it has more info than tapeinfo
# Alert Command = "sh -c 'smartctl -H -l error %c'"
#}
#
# Send all messages to the Director,
# mount messages also are sent to the email address
#
Messages {
Name = Standard
director = bacula-dir = all
}
===
Bconsole.conf
#
# Bacula User Agent (or Console) Configuration File
#
# Copyright (C) 2000-2015 Kern Sibbald
# License: BSD 2-Clause; see file LICENSE-FOSS
#
Director {
Name = bacula-dir
#Name = BackOVH-fd
DIRport = 9101
address = 51.79.119.27
Password = ""
}
On the client side
Bacula-fd.conf
#
# Default Bacula File Daemon Configuration file
#
# For Bacula release 5.2.13 (19 February 2013) -- redhat (Core)
#
# There is not much to change here except perhaps the
# File daemon Name to
#
#
# List Directors who are permitted to contact this File daemon
#
Director {
#Name = bacula-dir
Name = bacula-dir
Password = ""
}
#
# Restricted Director, used by tray-monitor to get the
# status of the file daemon
#
Director {
Name = bacula-mon
Password = ""
Monitor = yes
}
#
# "Global" File daemon configuration specifications
#
FileDaemon { # this is me
#Name = 54.39.187.124-fd
Name = bacula-fd2
FDAddress = 54.39.187.124
FDport = 9102 # where we listen for the director
WorkingDirectory = /var/spool/bacula
Pid Directory = /var/run
Maximum Concurrent Jobs = 100
}
# Send all messages except skipped files back to Director
Messages {
Name = Standard
director = BackupServer-dir = all, !skipped, !restored
}
bconsole.conf
#
# Bacula User Agent (or Console) Configuration File
#
Director {
Name = bacula-dir
DIRport = 9101
address = 54.39.187.124
Password = "xxxx"
Jean Mark Orfali . Sys admin
T. 1-877-258-4542 P. 227
bravad.ca/Fabrique
Avis de confidentialité
Le contenu de ce message ainsi que du ou des fichiers qui y sont joints est
strictement confidentiel et destiné exclusivement à son ou sa destinataire. Si
vous n'êtes pas cette personne, nous attirons votre attention sur le fait qu'il
est strictement interdit de copier, de faire suivre ou d'utiliser les
informations contenues dans ce courriel. Si vous l'avez reçu par erreur, nous
vous remercions de nous le faire savoir.
-Message d'origine-
De : Phil Stracchino
Envoyé : Monday, 27 January, 2020 11:25 AM
À : bacula-users@lists.sourceforge.net
Objet : Re: [Bacula-users] Bacula and memory usage
On 2020-01-27 10:38, Jean Mark Orfali wrote:
Hello,
I have a Bacula server and 4 clients. The server has 8 cpu and 30 GIB
of ram. I don't know why, it's been 5 days since the first backup
started and they are still not finished. The ram on the server is
still used at almost 100%, 29.30GIB. Do you know why Bacula uses so much memory?
thank you so much
This is not nearly enough information from which to diagnose the problem.
What back-end catalog database are you running? How is it configured?
Does it have enough memory? How much data are you backing up, and where are
you backing it up to? What are the concurrency settings on ALL of your Bacula
daemons? What is the Director currently *doing*? What jobs are running, what
jobs are pending, what jobs are scheduled? This would probably be enough to
start troubleshooting.
Tell us how your system is set up, at high level, and give us an overview of
your configuration (with passwords obscured, of course).
One unfortunate beginner mistake that I have seen people run into is to run
disk-to-disk backups stored on one of the filesystems that is backed up,
leading to a job that will effectively run forever as it recursively backs up
its own backups.
Another common mistake is to put the catalog on a fresh-out-of-the-box,
unconfigured, untuned MySQL instance. The only thing that ca
Re: [Bacula-users] Bacula and memory usage
On 2020-01-27 14:11, Jean Mark Orfali wrote: > Hello, > > Thank you for your reply. Here is the missing information. My Bacula server > and the four clients are on Linux Centos 7 servers. I use Webmin version > 1.941 to access bacula. The bacula version is 7.0.5. The SQL server is a > MariaDB version 5.5.64. The server has 30TB of hard drive and 30GB of memory. > Backups are saved in a directory directly on the backup server. No backup is > kept on clients side. At the moment there is 6 TB of data to backup. On each > of the 4 clients I have an incremental backup task schedule every day at 11 > p.m. Right now I have 4 backups running for 5 days and 14 waiting. Jean, How is that MariaDB instance configured? As a quick-and-dirty first approximation, what does mysqltuner tell you? -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
smartctl, enable this, it has more info than tapeinfo
# Alert Command = "sh -c 'smartctl -H -l error %c'"
#}
#
# Send all messages to the Director,
# mount messages also are sent to the email address
#
Messages {
Name = Standard
director = bacula-dir = all
}
===
Bconsole.conf
#
# Bacula User Agent (or Console) Configuration File
#
# Copyright (C) 2000-2015 Kern Sibbald
# License: BSD 2-Clause; see file LICENSE-FOSS
#
Director {
Name = bacula-dir
#Name = BackOVH-fd
DIRport = 9101
address = 51.79.119.27
Password = ""
}
On the client side
Bacula-fd.conf
#
# Default Bacula File Daemon Configuration file
#
# For Bacula release 5.2.13 (19 February 2013) -- redhat (Core)
#
# There is not much to change here except perhaps the
# File daemon Name to
#
#
# List Directors who are permitted to contact this File daemon
#
Director {
#Name = bacula-dir
Name = bacula-dir
Password = ""
}
#
# Restricted Director, used by tray-monitor to get the
# status of the file daemon
#
Director {
Name = bacula-mon
Password = ""
Monitor = yes
}
#
# "Global" File daemon configuration specifications
#
FileDaemon { # this is me
#Name = 54.39.187.124-fd
Name = bacula-fd2
FDAddress = 54.39.187.124
FDport = 9102 # where we listen for the director
WorkingDirectory = /var/spool/bacula
Pid Directory = /var/run
Maximum Concurrent Jobs = 100
}
# Send all messages except skipped files back to Director
Messages {
Name = Standard
director = BackupServer-dir = all, !skipped, !restored
}
bconsole.conf
#
# Bacula User Agent (or Console) Configuration File
#
Director {
Name = bacula-dir
DIRport = 9101
address = 54.39.187.124
Password = ""
Jean Mark Orfali . Sys admin
T. 1-877-258-4542 P. 227
bravad.ca/Fabrique
Avis de confidentialité
Le contenu de ce message ainsi que du ou des fichiers qui y sont joints est
strictement confidentiel et destiné exclusivement à son ou sa destinataire. Si
vous n'êtes pas cette personne, nous attirons votre attention sur le fait qu'il
est strictement interdit de copier, de faire suivre ou d'utiliser les
informations contenues dans ce courriel. Si vous l'avez reçu par erreur, nous
vous remercions de nous le faire savoir.
-Message d'origine-
De : Phil Stracchino
Envoyé : Monday, 27 January, 2020 11:25 AM
À : bacula-users@lists.sourceforge.net
Objet : Re: [Bacula-users] Bacula and memory usage
On 2020-01-27 10:38, Jean Mark Orfali wrote:
> Hello,
>
>
>
> I have a Bacula server and 4 clients. The server has 8 cpu and 30 GIB
> of ram. I don't know why, it's been 5 days since the first backup
> started and they are still not finished. The ram on the server is
> still used at almost 100%, 29.30GIB. Do you know why Bacula uses so much
> memory?
>
> thank you so much
This is not nearly enough information from which to diagnose the problem.
What back-end catalog database are you running? How is it configured?
Does it have enough memory? How much data are you backing up, and where are
you backing it up to? What are the concurrency settings on ALL of your Bacula
daemons? What is the Director currently *doing*? What jobs are running, what
jobs are pending, what jobs are scheduled? This would probably be enough to
start troubleshooting.
Tell us how your system is set up, at high level, and give us an overview of
your configuration (with passwords obscured, of course).
One unfortunate beginner mistake that I have seen people run into is to run
disk-to-disk backups stored on one of the filesystems that is backed up,
leading to a job that will effectively run forever as it recursively backs up
its own backups.
Another common mistake is to put the catalog on a fresh-out-of-the-box,
unconfigured, untuned MySQL instance. The only thing that can be said of the
typical fresh-from-the-box default MySQL configuration file is that it's
usually no longer actively harmful. Whichever database you are using for your
catalog, make sure it is properly configured and tuned before you start.
(And don't use SQLite as a production catalog DB. Seriously.)
--
Phil Stracchino
Babylon Communications
ph...@caerllewys.net
p...@co.ordinate.org
Landline: +1.603.293.8485
Mobile: +1.603.998.6958
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and memory usage
On 2020-01-27 10:38, Jean Mark Orfali wrote: > Hello, > > > > I have a Bacula server and 4 clients. The server has 8 cpu and 30 GIB of > ram. I don't know why, it's been 5 days since the first backup started > and they are still not finished. The ram on the server is still used at > almost 100%, 29.30GIB. Do you know why Bacula uses so much memory? > > thank you so much This is not nearly enough information from which to diagnose the problem. What back-end catalog database are you running? How is it configured? Does it have enough memory? How much data are you backing up, and where are you backing it up to? What are the concurrency settings on ALL of your Bacula daemons? What is the Director currently *doing*? What jobs are running, what jobs are pending, what jobs are scheduled? This would probably be enough to start troubleshooting. Tell us how your system is set up, at high level, and give us an overview of your configuration (with passwords obscured, of course). One unfortunate beginner mistake that I have seen people run into is to run disk-to-disk backups stored on one of the filesystems that is backed up, leading to a job that will effectively run forever as it recursively backs up its own backups. Another common mistake is to put the catalog on a fresh-out-of-the-box, unconfigured, untuned MySQL instance. The only thing that can be said of the typical fresh-from-the-box default MySQL configuration file is that it's usually no longer actively harmful. Whichever database you are using for your catalog, make sure it is properly configured and tuned before you start. (And don't use SQLite as a production catalog DB. Seriously.) -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Bacula and memory usage
Hello, I have a Bacula server and 4 clients. The server has 8 cpu and 30 GIB of ram. I don't know why, it's been 5 days since the first backup started and they are still not finished. The ram on the server is still used at almost 100%, 29.30GIB. Do you know why Bacula uses so much memory? thank you so much [cid:image001.png@01D2B43F.D4E82A50] Jean Mark Orfali * Sys admin T. 1-877-258-4542 P. 227 bravad.ca/Fabrique Avis de confidentialité Le contenu de ce message ainsi que du ou des fichiers qui y sont joints est strictement confidentiel et destiné exclusivement à son ou sa destinataire. Si vous n'êtes pas cette personne, nous attirons votre attention sur le fait qu'il est strictement interdit de copier, de faire suivre ou d'utiliser les informations contenues dans ce courriel. Si vous l'avez reçu par erreur, nous vous remercions de nous le faire savoir. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
