First off thank you everyone in the group for your useful and insightful suggestions. I was just about to throw in the towel when the solution presented itself.
For all you 3ware users out there this might be of help to you. Steve you gave me the nudge I needed the problem was with the high Iowait time. I went looking through many other forums and about IO wait time when I came across the following line "[sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA" The person who posted it was also complaining of slow performance but his he thought was related to the DPO/FUA error. I remember seeing this same line and wanted to check his error against mine. This is where I found the smoking gun. "[sdc] Write cache: enabled, read cache: disabled, doesn't support DPO or FUA" After finding this line it was a quick search of the 3ware forum to find out how to enable the read caching. 1. Upgrade 3DM2 web tool and tw_cli This was necessary to see if my current firmware supported the read cache setting. (of course it did not) 2. Upgrade the firmware on the raid card I would call 3ware just in case as you have to make sure that you current firmware can be upgraded in one single upgrade or if you need to patch to previous releases first. Also you need to make sure that your current driver will support the new firmware and finally backup backup backup. They 3ware team could not stress this one enough. 3. Enable the read cache via either 3dm2 or tw_cli. After upgrading the firmware and enabling read caching I followed some of these links to further increase performance. http://www.3ware.com/kb/article.aspx?id=11050 http://www.3ware.com/KB/article.aspx?id=15244 http://www.3ware.com/KB/article.aspx?id=10889 I am now getting between 30 and 50 megs/s. The slower speed is due the sheer number of files being backed up (2.1M files in 1.7TB) Thanks again. -H -----Original Message----- From: Steve Ellis [mailto:el...@brouhaha.com] Sent: Thursday, December 10, 2009 10:21 AM To: bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] Network transfer Speed On 12/10/2009 9:33 AM, Hayden Katzenellenbogen wrote: > Steve, > > Here is a quick snap of my top during a full backup. > > top - 09:32:35 up 1 day, 18:38, 1 user, load average: 11.41, 11.60, > 10.75 > Tasks: 161 total, 1 running, 160 sleeping, 0 stopped, 0 zombie > Cpu0 : 0.3%us, 0.3%sy, 0.0%ni, 0.0%id, 99.4%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu1 : 13.0%us, 1.4%sy, 0.0%ni, 18.1%id, 66.9%wa, 0.0%hi, 0.6%si, > 0.0%st > Cpu2 : 1.9%us, 0.3%sy, 0.0%ni, 0.0%id, 97.8%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu3 : 0.6%us, 1.3%sy, 0.0%ni, 22.7%id, 74.8%wa, 0.0%hi, 0.6%si, > 0.0%st > Cpu4 : 0.6%us, 0.3%sy, 0.0%ni, 0.0%id, 99.1%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu5 : 0.6%us, 0.0%sy, 0.0%ni, 20.8%id, 78.3%wa, 0.0%hi, 0.3%si, > 0.0%st > Cpu6 : 0.4%us, 0.7%sy, 0.0%ni, 52.5%id, 46.5%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu7 : 0.6%us, 0.3%sy, 0.0%ni, 7.6%id, 91.5%wa, 0.0%hi, 0.0%si, > 0.0%st > Mem: 4057152k total, 4035996k used, 21156k free, 14372k buffers > Swap: 8000360k total, 1536k used, 7998824k free, 3852936k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 6118 root 20 0 60388 2212 1296 S 24 0.1 102:35.41 bacula-fd > > 6240 root 20 0 130m 2092 1348 S 3 0.1 227:19.75 bacula-sd > > 23214 hayden 20 0 18992 1308 932 R 1 0.0 0:00.24 top > > > There are many occasions where the bacula-sd and bacula-fd have between > 10-50% and all the wait percentages are through the roof as you see > above. Also as you can see the load average is also through the roof. > > -H Hayden- I'm not sure what else you might be running on this box, but I've noticed on mine that my memory usage recently went up significantly, and I started seeing high load average peaks when only bacula (or the db behind it) should have been generating high load. It didn't seem that the bacula daemons themselves were using significantly more memory than previously, and I hadn't changed my bacula version anyway. Further, I had not changed any configuration from what had been running smoothly before. In my case, after detuning my MySQL config to use less memory, my load averages improved, and so did bacula performance. I have not dug into this very far yet, but my guess is that either the kernel behavior has changed recently with some update I applied (perhaps significantly increased kernel buffering??), or some library used by lots of programs now uses more memory than previously. Certainly, my box, with only 2 processors shouldn't need as much physical RAM as yours, but both your 8-core box and my 2-core box have 4Gigs of RAM. I have no silver bullets here, but if see load average spike dramatically during a backup, and see very high wait percentages on your CPUs during backups, then you might look to see what programs are using the most VM, and seeing if they can either be temporarily turned off or detuned to use less memory. An even better answer might be to try adding RAM to your machine (assuming its running a 64-bit OS), but the first approach could be done without taking the box down and without any expense at least just to test the theory. Hope that helps, -se ------------------------------------------------------------------------ ------ Return on Information: Google Enterprise Search pays you back Get the facts. http://p.sf.net/sfu/google-dev2dev _______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users ------------------------------------------------------------------------------ Return on Information: Google Enterprise Search pays you back Get the facts. http://p.sf.net/sfu/google-dev2dev _______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users