Re: [Bacula-users] LTO-3 getting the lead out.
On Mon, 04 Jan 2010 13:38:47 -0500 Brian Debelius bdebel...@intelesyscorp.com wrote: I am using batch inserts into MySQL. The database is on a different RAID1 volume. On 1/4/2010 1:21 PM, Richard Scobie wrote: Brian Debelius wrote: Shamless bump. Does anyone have any insight into this? Thanks brian- The difference between the dd tests and bacula backups is that catalog writes are occuring in the latter. Perhaps you have a database bottleneck? Is the database stored on the RAID also? The next thing to check is the block size that Bacula is using. Try increasing your Bacula block size and Bacula network block size. Search for Ralf Gross's notes on LTO-4 performance in the archives. http://www.google.com/search?q=ralf+gross+lto-4 Mainly something like this: Maximum Block Size = 262144 Maximum Network Buffer Size = 262144 Maximum File Size = 5G Keep in mind that tapes written with different block sizes will be incompatible, I think. So you won't be able to read your old backups unless you remove that new block size from the sd config, I think. Regards, -- Alex Chekholko ch...@pcbi.upenn.edu -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO-3 getting the lead out.
Incompatible? That's not good. I guess I will have to purge the tape and run the copy jobs again. I just set the SD Maximum Block Size to 1 megabyte, and this seems to improved things a lot. During a copy job, when I got a storage status from bconsole it said the rate was 54,008,673 Bytes/sec. To me that says it is maxing out the tape drive. The curious thing now is, when I received the job history, it said the rate was something like 35,000,000 Bytes/sec. Does the time that it takes for the spooled attributes to despool, effect the reported throughput? If thats the case, I don't like that. It makes it appear as the throughout is lower then it actually is. I will have to try the Network Buffer Size change. Is there anyway test for an ideal size? Thanks Brian- On 1/5/2010 2:21 PM, Alex Chekholko wrote: On Mon, 04 Jan 2010 13:38:47 -0500 Brian Debeliusbdebel...@intelesyscorp.com wrote: I am using batch inserts into MySQL. The database is on a different RAID1 volume. On 1/4/2010 1:21 PM, Richard Scobie wrote: Brian Debelius wrote: Shamless bump. Does anyone have any insight into this? Thanks brian- The difference between the dd tests and bacula backups is that catalog writes are occuring in the latter. Perhaps you have a database bottleneck? Is the database stored on the RAID also? The next thing to check is the block size that Bacula is using. Try increasing your Bacula block size and Bacula network block size. Search for Ralf Gross's notes on LTO-4 performance in the archives. http://www.google.com/search?q=ralf+gross+lto-4 Mainly something like this: Maximum Block Size = 262144 Maximum Network Buffer Size = 262144 Maximum File Size = 5G Keep in mind that tapes written with different block sizes will be incompatible, I think. So you won't be able to read your old backups unless you remove that new block size from the sd config, I think. Regards, -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO-3 getting the lead out.
...and if this post is still valid, this may be the maximum speed (55M/s) given that the LTO-3 and disk are on the same SD (even though tape compression is on.) http://www.mail-archive.com/bacula-de...@lists.sourceforge.net/msg01246.html [Kern]...so I forget the formulas for calculating this, but basically, you must wait to get data from the disk, then wait to write it to the tape, and that slows down both. To run at full speed, you need two threads, one that reads from disk and buffers in memory, then a second thread that writes to tape... On 1/5/2010 2:21 PM, Alex Chekholko wrote: On Mon, 04 Jan 2010 13:38:47 -0500 Brian Debeliusbdebel...@intelesyscorp.com wrote: I am using batch inserts into MySQL. The database is on a different RAID1 volume. On 1/4/2010 1:21 PM, Richard Scobie wrote: Brian Debelius wrote: Shamless bump. Does anyone have any insight into this? Thanks brian- The difference between the dd tests and bacula backups is that catalog writes are occuring in the latter. Perhaps you have a database bottleneck? Is the database stored on the RAID also? The next thing to check is the block size that Bacula is using. Try increasing your Bacula block size and Bacula network block size. Search for Ralf Gross's notes on LTO-4 performance in the archives. http://www.google.com/search?q=ralf+gross+lto-4 Mainly something like this: Maximum Block Size = 262144 Maximum Network Buffer Size = 262144 Maximum File Size = 5G Keep in mind that tapes written with different block sizes will be incompatible, I think. So you won't be able to read your old backups unless you remove that new block size from the sd config, I think. Regards, -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO-3 getting the lead out.
Shamless bump. Does anyone have any insight into this? Thanks brian- On 12/28/2009 2:12 PM, Brian Debelius wrote: Hi, I am trying to improve my LTO-3 throughput. I recently installed a SAS Tandberg LTO-3 drive. I am using Bacula 3.0.3 in Ubuntu. I backup to disk files, and then copy to tape. The disk files are 4GB in size. They are stored on a 4 disk RAID 5 array. The raid and the tape drive are in the same system. dd'ing a file from the raid to /dev/null yields: #dd if=vol-0572 of=/dev/null 8388576+1 records in 8388576+1 records out 4294951097 bytes (4.3 GB) copied, 45.0316 s, 95.4 MB/s disabling compression on the tape drive, and dd'ing the file from raid to /dev/nst1 yields: #dd if=vol-0572 of=/dev/nst1 bs=64K 65535+1 records in 65535+1 records out 4294951097 bytes (4.3 GB) copied, 77.9708 s, 55.1 MB/s enabling compression on the tape drive, and dd'ing the file from raid to /dev/nst1 yields: #dd if=vol-0572 of=/dev/nst1 bs=64K 65535+1 records in 65535+1 records out 4294951097 bytes (4.3 GB) copied, 43.1465 s, 99.5 MB/s These are the copy job throughput rates. The parent jobs of these three run concurrently, so all three jobs are in the same disk files. Rate: 5GB @ 22282.1 KB/s Rate: 8GB @ 22787.8 KB/s Rate: 5GB @ 19648.1 KB/s With the next two jobs, they do not run concurrent, so each disk file only contains the data of one job. Rate: 290GB @ 25024.7 KB/s Rate: 58GB @ 27998.8 KB/s This seems slower then it should be. brian- -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO-3 getting the lead out.
Brian Debelius wrote: Shamless bump. Does anyone have any insight into this? Thanks brian- The difference between the dd tests and bacula backups is that catalog writes are occuring in the latter. Perhaps you have a database bottleneck? Is the database stored on the RAID also? Regards, Richard -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO-3 getting the lead out.
I am using batch inserts into MySQL. The database is on a different RAID1 volume. On 1/4/2010 1:21 PM, Richard Scobie wrote: Brian Debelius wrote: Shamless bump. Does anyone have any insight into this? Thanks brian- The difference between the dd tests and bacula backups is that catalog writes are occuring in the latter. Perhaps you have a database bottleneck? Is the database stored on the RAID also? Regards, Richard -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
