On Tuesday 30 March 2010 11:12:25 Ralf Gross wrote: > Hi, > > sorry for top posting, but I think it might be better to leave the org. > mail as it was. > > The memory bacula-sd uses seems to be growing constantly over time (500 MB > atm). > > http://img163.imageshack.us/img163/7928/baculasdmem.png > > This is no problem right now, but I just started an restore of 13 files and > server started swapping again. bacula-sd's memory usage began to boost over > 2,5 GB RAM. I can reproduce this ever time I try to restore these files.
Normally, there is no reason why a SD would ever use at a maximum more than a few hundred megabytes of memory. In fact, the data below indicates that the SD is using 19MB, which is quite normal and reasonable. So there is a big difference between what your system is reporting and what the SD thinks it is using. Can you tell me how big the .bsr file was that the Director sent to the SD? > > http://img696.imageshack.us/img696/4163/baculadirmem.png > > > Mem: 4063148k total, 3633076k used, 430072k free, 4776k buffers > Swap: 1951856k total, 1532320k used, 419536k free, 701724k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 15974 bacula 20 0 3498m 2.5g 1756 S 0 63.6 8:34.59 bacula-dir > 13098 postgres 20 0 534m 7680 7372 S 0 0.2 1:34.63 postgres > 8583 root 20 0 65928 3116 2488 S 0 0.1 0:00.10 sshd > 4636 snmp 20 0 51336 2580 1216 S 0 0.1 51:18.09 snmpd > 8585 root 20 0 18852 1968 1392 S 0 0.0 0:00.00 bash > 13087 postgres 20 0 533m 1936 1776 S 0 0.0 4:38.30 postgres > 12796 bacula 20 0 503m 1480 964 S 0 0.0 797:06.17 bacula-sd > > > The directory tree for the backup was build from a full, a differential > and one incemental backup. The full backup is rather large with 7 million > files. > > > +--------+-------+-----------+-------------------+---------------------+--- >----------------+ > > | jobid | level | jobfiles | jobbytes | starttime | > | volumename | > > +--------+-------+-----------+-------------------+---------------------+--- >----------------+ > > | 20,161 | F | 7,263,056 | 3,467,485,693,617 | 2010-03-07 00:41:44 | > | A00206L4 | 20,161 | F | 7,263,056 | 3,467,485,693,617 | > | 2010-03-07 00:41:44 | A00165L4 | 20,161 | F | 7,263,056 | > | 3,467,485,693,617 | 2010-03-07 00:41:44 | A00204L4 | 20,161 | F > | | 7,263,056 | 3,467,485,693,617 | 2010-03-07 00:41:44 | A00205L4 > | | 20,770 | D | 565,881 | 259,586,228,222 | 2010-03-28 00:45:11 > | | 06D127L3 | 20,810 | I | 12,932 | 16,134,152,294 | > | 2010-03-30 00:28:26 | vu0em003-inc-0472 | 20,810 | I | 12,932 | > | 16,134,152,294 | 2010-03-30 00:28:26 | vu0em003-inc-0572 | > > +--------+-------+-----------+-------------------+---------------------+--- >----------------+ > > > It took only 2-3 minutes to build the tree, so the speed is no problem. > > > My question: is this memory usage expected for restore jobs of this size? > I've never noticed it before 3.x. Accurate Backups are not active for this > client/backup. The memory usage for restoring from a 3TB backup could be rather large in the Director if you have not configured your Storage daemon properly. Can you tell me how many JobMedia records there are for JobId 20161? What is the output device? Tape, disk, ??? This will give an idea if your SD is properly configured. Kern > > Ralf > > > PS: sorry if this might be better placed on the -users list. > > > Hi, > > > > since a few weeks I see the memory usage of my two bacula-sd's growing > > and the server starts swapping. > > > > I had two cases where the server ran out of memory and a running job > > was not able to change the tape because mtx didn't get any memory. At > > that time 8 GB of swap was used. > > > > This is a graph of the swap space: > > > > http://img693.imageshack.us/img693/8061/baculasdswap.png > > > > And the top output from yesterday, before I restarted the SD. > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 14240 bacula 20 0 990m 437m 1264 S 0 11.0 1699:02 > > /usr/sbin/bacula-sd > > > > And this is how the memory usage for bacula-sd looks now: > > > > http://img89.imageshack.us/img89/875/baculasdvumem004.png > > > > VUMEM004-sd Version: 3.0.3 (18 October 2009) x86_64-pc-linux-gnu debian > > 5.0.3 Daemon started 17-Mär010 14:45, 52 Jobs run since started. > > Heap: heap=19,267,584 smbytes=18,630,339 max_bytes=18,984,066 bufs=1,495 > > max_bufs=1,715 Sizes: boffset_t=8 size_t=8 int32_t=4 int64_t=8 > > > > I guess it will take 1-2 weeks until the memory usage starts growing > > excessivly again. > > > > There are not many backups running concurrently and only 10-20 jobs at > > all. But there are a couple of very large backups: > > > > Terminated Jobs: > > JobId Level Files Bytes Status Finished Name > > ====================================================================== > > 20387 Full 4,841 2.254 T Error 15-Mär010 21:35 > > VU0EF005-Volume2 20406 Full 7,047 3.192 T OK 16-Mär010 > > 15:37 VU0EF005-Volume2 20405 Full 8,422 4.117 T OK > > 16-Mär010 19:55 VU0EF005-Volume1 > > > > or > > > > +--------+----------+---------------------+------+-------+-----------+--- > >----------------+-----------+ > > > > | jobid | name | starttime | type | level | jobfiles | > > | jobbytes | jobstatus | > > > > +--------+----------+---------------------+------+-------+-----------+--- > >----------------+-----------+ > > > > | 20,161 | VU0EM003 | 2010-03-07 00:41:44 | B | F | 7,263,056 | > > | 3,467,485,693,617 | T | > > > > I haven't seen something like this in the last 2-3 years and I'm not > > sure which change is the cause for it. > > > > What has changed: > > > > * Updated from Debian Etch to Lenny (AMD64) > > * Update to psql 8.4 > > * Updated from bacula 2.4.4 to 3.0.2 and later 3.0.3. Because there > > was no 3.0.3 Lenny package, I've built my own deb package. > > > > > > I know the Kaboom chapter in the manual: > > http://www.bacula.org/en/dev-manual/What_Do_When_Bacula.html > > > > It is more about bacula crashing which is not the case right now. Is > > there a way to see which parts of the SD are using memory during > > "normal operation"? > > > > > > Ralf > > > > ------------------------------------------------------------------------- > >----- Download Intel® Parallel Studio Eval > > Try the new software tools for yourself. Speed compiling, find bugs > > proactively, and fine-tune applications for parallel performance. > > See why Intel Parallel Studio got high marks during beta. > > http://p.sf.net/sfu/intel-sw-dev > > _______________________________________________ > > Bacula-devel mailing list > > Bacula-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/bacula-devel > > --------------------------------------------------------------------------- >--- Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Bacula-devel mailing list > Bacula-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-devel ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev _______________________________________________ Bacula-devel mailing list Bacula-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-devel
