Hi, sorry for top posting, but I think it might be better to leave the org. mail as it was.
The memory bacula-sd uses seems to be growing constantly over time (500 MB atm). http://img163.imageshack.us/img163/7928/baculasdmem.png This is no problem right now, but I just started an restore of 13 files and server started swapping again. bacula-sd's memory usage began to boost over 2,5 GB RAM. I can reproduce this ever time I try to restore these files. http://img696.imageshack.us/img696/4163/baculadirmem.png Mem: 4063148k total, 3633076k used, 430072k free, 4776k buffers Swap: 1951856k total, 1532320k used, 419536k free, 701724k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15974 bacula 20 0 3498m 2.5g 1756 S 0 63.6 8:34.59 bacula-dir 13098 postgres 20 0 534m 7680 7372 S 0 0.2 1:34.63 postgres 8583 root 20 0 65928 3116 2488 S 0 0.1 0:00.10 sshd 4636 snmp 20 0 51336 2580 1216 S 0 0.1 51:18.09 snmpd 8585 root 20 0 18852 1968 1392 S 0 0.0 0:00.00 bash 13087 postgres 20 0 533m 1936 1776 S 0 0.0 4:38.30 postgres 12796 bacula 20 0 503m 1480 964 S 0 0.0 797:06.17 bacula-sd The directory tree for the backup was build from a full, a differential and one incemental backup. The full backup is rather large with 7 million files. +--------+-------+-----------+-------------------+---------------------+-------------------+ | jobid | level | jobfiles | jobbytes | starttime | volumename | +--------+-------+-----------+-------------------+---------------------+-------------------+ | 20,161 | F | 7,263,056 | 3,467,485,693,617 | 2010-03-07 00:41:44 | A00206L4 | | 20,161 | F | 7,263,056 | 3,467,485,693,617 | 2010-03-07 00:41:44 | A00165L4 | | 20,161 | F | 7,263,056 | 3,467,485,693,617 | 2010-03-07 00:41:44 | A00204L4 | | 20,161 | F | 7,263,056 | 3,467,485,693,617 | 2010-03-07 00:41:44 | A00205L4 | | 20,770 | D | 565,881 | 259,586,228,222 | 2010-03-28 00:45:11 | 06D127L3 | | 20,810 | I | 12,932 | 16,134,152,294 | 2010-03-30 00:28:26 | vu0em003-inc-0472 | | 20,810 | I | 12,932 | 16,134,152,294 | 2010-03-30 00:28:26 | vu0em003-inc-0572 | +--------+-------+-----------+-------------------+---------------------+-------------------+ It took only 2-3 minutes to build the tree, so the speed is no problem. My question: is this memory usage expected for restore jobs of this size? I've never noticed it before 3.x. Accurate Backups are not active for this client/backup. Ralf PS: sorry if this might be better placed on the -users list. > Hi, > > since a few weeks I see the memory usage of my two bacula-sd's growing > and the server starts swapping. > > I had two cases where the server ran out of memory and a running job > was not able to change the tape because mtx didn't get any memory. At > that time 8 GB of swap was used. > > This is a graph of the swap space: > > http://img693.imageshack.us/img693/8061/baculasdswap.png > > And the top output from yesterday, before I restarted the SD. > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 14240 bacula 20 0 990m 437m 1264 S 0 11.0 1699:02 > /usr/sbin/bacula-sd > > And this is how the memory usage for bacula-sd looks now: > > http://img89.imageshack.us/img89/875/baculasdvumem004.png > > VUMEM004-sd Version: 3.0.3 (18 October 2009) x86_64-pc-linux-gnu debian 5.0.3 > Daemon started 17-Mär010 14:45, 52 Jobs run since started. > Heap: heap=19,267,584 smbytes=18,630,339 max_bytes=18,984,066 bufs=1,495 > max_bufs=1,715 > Sizes: boffset_t=8 size_t=8 int32_t=4 int64_t=8 > > I guess it will take 1-2 weeks until the memory usage starts growing > excessivly again. > > There are not many backups running concurrently and only 10-20 jobs at > all. But there are a couple of very large backups: > > Terminated Jobs: > JobId Level Files Bytes Status Finished Name > ====================================================================== > 20387 Full 4,841 2.254 T Error 15-Mär010 21:35 VU0EF005-Volume2 > 20406 Full 7,047 3.192 T OK 16-Mär010 15:37 VU0EF005-Volume2 > 20405 Full 8,422 4.117 T OK 16-Mär010 19:55 VU0EF005-Volume1 > > or > > +--------+----------+---------------------+------+-------+-----------+-------------------+-----------+ > | jobid | name | starttime | type | level | jobfiles | > jobbytes | jobstatus | > +--------+----------+---------------------+------+-------+-----------+-------------------+-----------+ > | 20,161 | VU0EM003 | 2010-03-07 00:41:44 | B | F | 7,263,056 | > 3,467,485,693,617 | T | > > > > > I haven't seen something like this in the last 2-3 years and I'm not > sure which change is the cause for it. > > What has changed: > > * Updated from Debian Etch to Lenny (AMD64) > * Update to psql 8.4 > * Updated from bacula 2.4.4 to 3.0.2 and later 3.0.3. Because there > was no 3.0.3 Lenny package, I've built my own deb package. > > > I know the Kaboom chapter in the manual: > http://www.bacula.org/en/dev-manual/What_Do_When_Bacula.html > > It is more about bacula crashing which is not the case right now. Is > there a way to see which parts of the SD are using memory during > "normal operation"? > > > Ralf > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Bacula-devel mailing list > Bacula-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-devel ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev _______________________________________________ Bacula-devel mailing list Bacula-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-devel
