Re: [Bacula-users] Tape access speed with LTO3
On 11/9/06, John Drescher [EMAIL PROTECTED] wrote: Try if=/dev/zero and if=/dev/random AB Note that depending on the platform where you try, /dev/urandom as a source might be faster than /dev/random. (/dev/random blocks until more entropy can be obtained says e.g. the Solaris man page) -- Maxx - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tape access speed with LTO3
On Thu, 9 Nov 2006, John Drescher wrote: Have you tried a test of the raw tape speed? When I first got mt lto-2 library I did a few tests like the following time dd if=/dev/sda of=/dev/nst0 bs=1G count=10 and I got around 35MB/s. Try if=/dev/zero and if=/dev/random I did not do /dev/zero because of hardware compression. I guess I could turn that off but isn't /dev/random 1 byte at a time?? It's fast enough on most systems. Alternatively, use btape and the fill command. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tape access speed with LTO3
Alan Brown wrote (2006/11/10): I did not do /dev/zero because of hardware compression. I guess I could turn that off but isn't /dev/random 1 byte at a time?? It's fast enough on most systems. Really? My experience is that the speed of /dev/random and /dev/urandom in FreeBSD is far from to be sufficient for LTO3. # dd if=/dev/random of=/dev/null bs=64k count=1 1+0 records in 1+0 records out 65536 bytes transferred in 20.414294 secs (32102996 bytes/sec) # dd if=/dev/urandom of=/dev/null bs=64k count=1 1+0 records in 1+0 records out 65536 bytes transferred in 20.418363 secs (32096599 bytes/sec) Processor: CPU: Genuine Intel(R) CPU 3.20GHz (2793.02-MHz 686-class CPU). -- Rudolf Cejka cejkar at fit.vutbr.cz http://www.fit.vutbr.cz/~cejkar Brno University of Technology, Faculty of Information Technology Bozetechova 2, 612 66 Brno, Czech Republic - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tape access speed with LTO3
On Fri, 10 Nov 2006, Rudolf Cejka wrote: Alan Brown wrote (2006/11/10): I did not do /dev/zero because of hardware compression. I guess I could turn that off but isn't /dev/random 1 byte at a time?? It's fast enough on most systems. Really? My experience is that the speed of /dev/random and /dev/urandom in FreeBSD is far from to be sufficient for LTO3. If this is the case then one needs to use some pseudorandom/non-compressable data stream which _is_ fast enough. AB - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tape access speed with LTO3
The top speed I've achieved with an IBM 3581 L38 was around 30 MB/s on a test (btest). With bacula, local disk (sata): JobId: 1384 Job:srv-backup.2006-10-27_22.05.07 Backup Level: Full Client: srv-backup x86_64-unknown-linux-gnu,suse,10.0 FileSet:srv-backup-fs 2006-05-11 11:18:34 Pool: Default Storage:lto Scheduled time: 27-Oct-2006 22:05:06 Start time: 28-Oct-2006 03:49:40 End time: 28-Oct-2006 04:33:46 Elapsed time: 44 mins 6 secs Priority: 500 FD Files Written: 216,994 SD Files Written: 216,994 FD Bytes Written: 48,200,304,853 (48.20 GB) SD Bytes Written: 48,231,485,273 (48.23 GB) Rate: 18216.3 KB/s Software Compression: None Volume name(s): bac-0001 Volume Session Id: 31 Volume Session Time:1161792332 Last Volume Bytes: 222,847,029,453 (222.8 GB) Non-fatal FD errors:0 SD Errors: 0 FD termination status: OK SD termination status: OK Termination:Backup OK and over a Gb net: 29-Oct 04:27 srv-backup-dir: Bacula 1.38.11 (28Jun06): 29-Oct-2006 04:27:53 JobId: 1387 Job:bd-unrc-full.2006-10-29_04.01.00 Backup Level: Full Client: bd-unrc x86_64-unknown-linux-gnu,suse,10.0 FileSet:bd-unrc_full-fs 2006-07-14 12:55:52 Pool: Default Storage:lto Scheduled time: 29-Oct-2006 04:01:00 Start time: 29-Oct-2006 04:01:24 End time: 29-Oct-2006 04:27:53 Elapsed time: 26 mins 29 secs Priority: 10 FD Files Written: 131,196 SD Files Written: 131,196 FD Bytes Written: 31,528,273,511 (31.52 GB) SD Bytes Written: 31,545,629,755 (31.54 GB) Rate: 19841.6 KB/s Software Compression: None Volume name(s): bac-0001 Volume Session Id: 32 Volume Session Time:1161792332 Last Volume Bytes: 254,420,061,578 (254.4 GB) Non-fatal FD errors:0 SD Errors: 0 FD termination status: OK SD termination status: OK Termination:Backup OK I believe the bottleneck are the disks/network. El Miércoles, 8 de Noviembre de 2006 10:00, Adam Huffman escribió: What sort of backup speed should I expect when using Bacula with an LTO3 library? I'm seeing nothing like the advertised 60-80MB/s. Adam -- Cris. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tape access speed with LTO3
On 11/9/06, Carlos Cristóbal Sabroe Yde [EMAIL PROTECTED] wrote: The top speed I've achieved with an IBM 3581 L38 was around 30 MB/s on a test(btest).Have you tried a test of the raw tape speed? When I first got mt lto-2 library I did a few tests like the following time dd if=/dev/sda of=/dev/nst0 bs=1G count=10and I got around 35MB/s.With my drive (LTO-2) which I believe is supposed backup at (20 to 40MB/s) I generally get around 21MB/s with bacula on most backups from a raid6 array on a remote system over a gigabit network with the database on a third machine and with MD5 signitures and medical image data that generally compresses around 1.3/1 to 1.5 /1John - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tape access speed with LTO3
On Thu, 9 Nov 2006, John Drescher wrote: Have you tried a test of the raw tape speed? When I first got mt lto-2 library I did a few tests like the following time dd if=/dev/sda of=/dev/nst0 bs=1G count=10 and I got around 35MB/s. Try if=/dev/zero and if=/dev/random AB - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tape access speed with LTO3
From: Adam Huffman [EMAIL PROTECTED] What sort of backup speed should I expect when using Bacula with an LTO3 library? I'm seeing nothing like the advertised 60-80MB/s. The tape drive speed is not the bottleneck, then. The number of files saved (or, seen from another point, their average size) has a large influence. For instance I get : FD Files Written: 11,608 SD Files Written: 11,608 FD Bytes Written: 1,593,770,921 (1.593 GB) SD Bytes Written: 1,595,662,849 (1.595 GB) Rate: 34647.2 KB/s FD Files Written: 179,040 SD Files Written: 179,040 FD Bytes Written: 13,268,207,376 (13.26 GB) SD Bytes Written: 13,292,405,638 (13.29 GB) Rate: 22915.7 KB/s FD Files Written: 154,664 SD Files Written: 154,664 FD Bytes Written: 3,127,551,736 (3.127 GB) SD Bytes Written: 3,148,432,104 (3.148 GB) Rate: 9992.2 KB/s All are full backups on a LTO3 through a Gbit/s ethernet with no particular tuning. I don't know if this is mainly due to the filesystem on the client or the index database on the server. The second result above is the server itself. Same hardware as the first one, relatively similar average file size, but noticeably slower when both non-tape-related IOs combine (at least, that's my guess). - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tape access speed with LTO3
On 11/9/06, Alan Brown [EMAIL PROTECTED] wrote: On Thu, 9 Nov 2006, John Drescher wrote: Have you tried a test of the raw tape speed? When I first got mt lto-2 library I did a few tests like the following time dd if=/dev/sda of=/dev/nst0 bs=1G count=10 and I got around 35MB/s.Try if=/dev/zero and if=/dev/randomABI did not do /dev/zero because of hardware compression. I guess I could turn that off but isn't /dev/random 1 byte at a time?? John - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tape access speed with LTO3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Adam Huffman wrote: Is the data compressable? A mixture - some already compressed, some images, some XML data etc. The library and the disks are on the same fibre channel fabric, so I wouldn't expect that to be a bottleneck. I'm using SQLite for the catalog- would that really make much of a difference? In this case, who knows, but the default answer is yes. SQLite3 particularly, but I believe 2 isn't known for its speed either. - -- _ _ _ _ ___ _ _ _ |Y#| | | |\/| | \ |\ | | |Ryan Novosielski - Systems Programmer III |$| |__| | | |__/ | \| _| |[EMAIL PROTECTED] - 973/972.0922 (2-0922) \__/ Univ. of Med. and Dent.|IST/AST - NJMS Medical Science Bldg - C630 -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFU5twmb+gadEcsb4RArfIAKCGW0qa8z+GgvUoyzdCkYi3Sn6aUgCfeLKN qScAG7m7D3nZVZ/HJDwMPIs= =GHNQ -END PGP SIGNATURE- - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tape access speed with LTO3
I have a 7-disk raid fibre attached and a lto3 scsi attached to my sd, and I get the same throughput for both. I run 10 jobs (two from each client) to each store concurrently. I don't think the medium or the connection are the holdup. My network throughput at that time varies around 750 mbps. I'm running postgres, director, sd, and catalog on the internal scsi hardware raid 5 volume. FWIW. On Thu, 2006-11-09 at 16:19 -0500, Ryan Novosielski wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Adam Huffman wrote: Is the data compressable? A mixture - some already compressed, some images, some XML data etc. The library and the disks are on the same fibre channel fabric, so I wouldn't expect that to be a bottleneck. I'm using SQLite for the catalog- would that really make much of a difference? In this case, who knows, but the default answer is yes. SQLite3 particularly, but I believe 2 isn't known for its speed either. - -- _ _ _ _ ___ _ _ _ |Y#| | | |\/| | \ |\ | | |Ryan Novosielski - Systems Programmer III |$| |__| | | |__/ | \| _| |[EMAIL PROTECTED] - 973/972.0922 (2-0922) \__/ Univ. of Med. and Dent.|IST/AST - NJMS Medical Science Bldg - C630 -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFU5twmb+gadEcsb4RArfIAKCGW0qa8z+GgvUoyzdCkYi3Sn6aUgCfeLKN qScAG7m7D3nZVZ/HJDwMPIs= =GHNQ -END PGP SIGNATURE- - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tape access speed with LTO3
On 11/9/06, Don MacArthur [EMAIL PROTECTED] wrote: I have a 7-disk raid fibre attached and a lto3 scsi attached to my sd,and I get the same throughput for both.I run 10 jobs (two from eachclient) to each store concurrently. Both what? What backup speeds are you getting?Thanks,John - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tape access speed with LTO3
On 11/9/06, Don MacArthur [EMAIL PROTECTED] wrote: The jobs are written to both the raid and the tape at almost identicalspeeds.I don't know what my total throughput is,but the transfer rate for each job is about 1.5mbps to 3mbps.For a few minutes after reading this I was very confused about that but after reading your first post again where you said that you run 10 concurrent jobs so these numbers make sense... John - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tape access speed with LTO3
As a reminder, the director/sd/database system is supporting 20 concurrent jobs, 10 to tape and 10 to raid. It's a 2-cpu hp dl380 G4, 3.4GHz cpus and 3.5Gb ram. On Thu, 2006-11-09 at 17:29 -0500, John Drescher wrote: On 11/9/06, Don MacArthur [EMAIL PROTECTED] wrote: The jobs are written to both the raid and the tape at almost identical speeds. I don't know what my total throughput is,but the transfer rate for each job is about 1.5mbps to 3mbps. For a few minutes after reading this I was very confused about that but after reading your first post again where you said that you run 10 concurrent jobs so these numbers make sense... John - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Tape access speed with LTO3
What sort of backup speed should I expect when using Bacula with an LTO3 library? I'm seeing nothing like the advertised 60-80MB/s. Adam - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tape access speed with LTO3
On Wed, 8 Nov 2006, Adam Huffman wrote: What sort of backup speed should I expect when using Bacula with an LTO3 library? I'm seeing nothing like the advertised 60-80MB/s. What speeds _are_ you seeing? Is the data compressable? - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tape access speed with LTO3
On 08/11/06, Alan Brown [EMAIL PROTECTED] wrote: On Wed, 8 Nov 2006, Adam Huffman wrote: What sort of backup speed should I expect when using Bacula with an LTO3 library? I'm seeing nothing like the advertised 60-80MB/s. What speeds _are_ you seeing? Around the 30MB/s mark, according to Bacula logs. Is the data compressable? A mixture - some already compressed, some images, some XML data etc. The library and the disks are on the same fibre channel fabric, so I wouldn't expect that to be a bottleneck. I'm using SQLite for the catalog- would that really make much of a difference? Adam - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tape access speed with LTO3
Adam Huffman wrote: What sort of backup speed should I expect when using Bacula with an LTO3 library? I'm seeing nothing like the advertised 60-80MB/s. Adam - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users Have you tried the tool mentioned here: https://selfservice.talisma.com/display/2n/index.asp?c=18cpc=TJVN0Y442nJq41250384h8TsoO3RuDHpbReVIcid=11cat=catURL=r=0.2119867 What is the result(rate) when testing with btape? Did you implement spooling? http://bacula.org/dev-manual/Data_Spooling.html -- Hristo Benev IT Manager WAVEROAD Partners in Telecommunications 514-935-2020 x225 T 514-935-1001 F www.waveroad.ca [EMAIL PROTECTED] - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
