Re: [Bacula-users] LTO3 tape capacity (variable?)
Thank you everyone for your help! Oracle replaced the drive and while it's not running with as high a throughput as I would like, it's at least up at the 60MB/s (random data) that my other drives are at, rather than it's previous 30MB/s. I'm still going to experiment with some of the ideas that were tossed out and see if I can't get even better throughput of for bacula. thanks again, Stephen On 10/2/12 2:47 AM, Alan Brown wrote: On 02/10/12 01:35, Stephen Thompson wrote: Correction, the non-problem drive has a higher ECC fast error count, but the problem drive has a significantly higher Corrective algorithm invocations count. What that means is that it rewrote the data, which accounts for the lower throughput. LTO drives read as they write and if there are errors, they write again. If a cleaning tape doesn't work then you need to get the drive looked at/replaced under warranty. On 10/1/12 5:33 PM, Stephen Thompson wrote: On 10/1/12 4:06 PM, Alan Brown wrote: On 01/10/12 23:38, Stephen Thompson wrote: More importantly, I realized that my testing 6 months ago was not on all 4 of my drives, but only 2 of them. Today, I discovered one of my drives (untested in the past) is getting 1/2 the throughput for random data writes as the others!! smartctl -a /dev/sg(drive) will tell you a lot Put a cleaning tape in it Cleaning tape did not improve results. I see some errors in the counter log on the problem drive, but I see even more errors on another drive which isn't having a throughput problem (specifically SL500 Drive 1 is the lower throughput, but C4 Drive 1 actually has a higher error count). SL500 Drive 0 (~60MB/s random data throughput) = Error counter log: Errors Corrected by Total Correction GigabytesTotal ECC rereads/errors algorithm processeduncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 00 0 0 0 0.000 0 write: 00 0 0 0 0.000 0 SL500 Drive 1 (~30MB/s random data throughput) = Error counter log: Errors Corrected by Total Correction GigabytesTotal ECC rereads/errors algorithm processeduncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 00 0 0 0 0.000 0 write: 104540 0 0 821389 0.000 0 C4 Drive 0 (~60MB/s random data throughput) == Error counter log: Errors Corrected by Total Correction GigabytesTotal ECC rereads/errors algorithm processeduncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 20 0 0 2 0.000 0 write: 00 0 0 0 0.000 0 C4 Drive 1 (~60MB/s random data throughput) == Error counter log: Errors Corrected by Total Correction GigabytesTotal ECC rereads/errors algorithm processeduncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 20 0 0 2 0.000 0 write: 189610 0 0 48261 0.000 0 Stephen -- Stephen Thompson Berkeley Seismological Laboratory step...@seismo.berkeley.edu215 McCone Hall # 4760 404.538.7077 (phone) University of California, Berkeley 510.643.5811 (fax) Berkeley, CA 94720-4760 -- Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
On 02/10/12 01:35, Stephen Thompson wrote: Correction, the non-problem drive has a higher ECC fast error count, but the problem drive has a significantly higher Corrective algorithm invocations count. What that means is that it rewrote the data, which accounts for the lower throughput. LTO drives read as they write and if there are errors, they write again. If a cleaning tape doesn't work then you need to get the drive looked at/replaced under warranty. On 10/1/12 5:33 PM, Stephen Thompson wrote: On 10/1/12 4:06 PM, Alan Brown wrote: On 01/10/12 23:38, Stephen Thompson wrote: More importantly, I realized that my testing 6 months ago was not on all 4 of my drives, but only 2 of them. Today, I discovered one of my drives (untested in the past) is getting 1/2 the throughput for random data writes as the others!! smartctl -a /dev/sg(drive) will tell you a lot Put a cleaning tape in it Cleaning tape did not improve results. I see some errors in the counter log on the problem drive, but I see even more errors on another drive which isn't having a throughput problem (specifically SL500 Drive 1 is the lower throughput, but C4 Drive 1 actually has a higher error count). SL500 Drive 0 (~60MB/s random data throughput) = Error counter log: Errors Corrected by Total Correction GigabytesTotal ECC rereads/errors algorithm processeduncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 00 0 0 0 0.000 0 write: 00 0 0 0 0.000 0 SL500 Drive 1 (~30MB/s random data throughput) = Error counter log: Errors Corrected by Total Correction GigabytesTotal ECC rereads/errors algorithm processeduncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 00 0 0 0 0.000 0 write: 104540 0 0 821389 0.000 0 C4 Drive 0 (~60MB/s random data throughput) == Error counter log: Errors Corrected by Total Correction GigabytesTotal ECC rereads/errors algorithm processeduncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 20 0 0 2 0.000 0 write: 00 0 0 0 0.000 0 C4 Drive 1 (~60MB/s random data throughput) == Error counter log: Errors Corrected by Total Correction GigabytesTotal ECC rereads/errors algorithm processeduncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 20 0 0 2 0.000 0 write: 189610 0 0 48261 0.000 0 Stephen -- Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
Alan Brown wrote (2012/10/01): 2Mb is the maximum block size I've found to work on LTO* - it's a bacula limit, not an LTO one. Or it could be a limit of an operating system or of used hw interface. Increasing from 65535 will improve throughput significantly. There have been discussions about this in the last few years. Sometimes yes, sometimes no. It depends on many things, like hardware of your server, used interface type and speed and used operating system. I'm not sure just about one thing right now - on which level rewrites are done, either on a block level, or on something smaller. -- Rudolf Cejka cejkar at fit.vutbr.cz http://www.fit.vutbr.cz/~cejkar Brno University of Technology, Faculty of Information Technology Bozetechova 2, 612 66 Brno, Czech Republic -- Got visibility? Most devs has no idea what their production app looks like. Find out how fast your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219671;13503038;y? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
Hi, I ran some btape tests today to verify that I'd be improving throughput by changing blocksize from 256KB to 2MB and found that this does indeed appear to be true in terms of increasing compression efficiency, but it doesn't seem to affect incompressible data much, if at all. Still, it seems worth changing and I thank you for pointing me in that direction. More importantly, I realized that my testing 6 months ago was not on all 4 of my drives, but only 2 of them. Today, I discovered one of my drives (untested in the past) is getting 1/2 the throughput for random data writes as the others!! btape *speed file_size=4 nb_file=4 skip_raw SL500 Drive 0 SL500 Drive 1 C4 Drive 0 C4 Drive 1 256KB block size: Zeros = 92.86 MB/s 92.36 MB/s 91.38 MB/s 92.86 MB/s Random= 63.16 MB/s 27.53 MB/s 63.39 MB/s 63.60 MB/s 2MB block size: Zeros = 123.5 MB/s 122.7 MB/s 122.7 MB/s 122.7 MB/s Random= 62.24 MB/s 28.44 MB/s 63.62 MB/s 63.62 MB/s ^ thanks, Stephen On 09/28/2012 05:08 AM, Alan Brown wrote: On 28/09/12 02:38, Stephen Thompson wrote: Aren't these considered reasonable settings for LTO3? Maximum block size = 262144 # 256kb Maximum File Size = 2gb Not really. Change maximum file size to 10Gb and maximum block size to 2M You _must_ set all open tapes to used and restart the storage daemon when changing the block size. Bacula can't cope with varying maximum sizes on a tape Even with those changes, if you have a lot of small, incompressible files you'll see high tape overheads. thanks for the help! Stephen -- Stephen Thompson Berkeley Seismological Laboratory step...@seismo.berkeley.edu215 McCone Hall # 4760 404.538.7077 (phone) University of California, Berkeley 510.643.5811 (fax) Berkeley, CA 94720-4760 -- Got visibility? Most devs has no idea what their production app looks like. Find out how fast your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219671;13503038;y? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
Hi, I ran some btape tests today to verify that I'd be improving throughput by changing blocksize from 256KB to 2MB and found that this does indeed appear to be true in terms of increasing compression efficiency, but it doesn't seem to affect incompressible data much, if at all. Still, it seems worth changing and I thank you for pointing me in that direction. More importantly, I realized that my testing 6 months ago was not on all 4 of my drives, but only 2 of them. Today, I discovered one of my drives (untested in the past) is getting 1/2 the throughput for random data writes as the others!! Is it definitely LTO3 and definitely using LTO3 media? LTO2 was about half the speed, including using LTO2 media in an LTO3 drive. James -- Got visibility? Most devs has no idea what their production app looks like. Find out how fast your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219671;13503038;y? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
On 10/01/2012 03:52 PM, James Harper wrote: Hi, I ran some btape tests today to verify that I'd be improving throughput by changing blocksize from 256KB to 2MB and found that this does indeed appear to be true in terms of increasing compression efficiency, but it doesn't seem to affect incompressible data much, if at all. Still, it seems worth changing and I thank you for pointing me in that direction. More importantly, I realized that my testing 6 months ago was not on all 4 of my drives, but only 2 of them. Today, I discovered one of my drives (untested in the past) is getting 1/2 the throughput for random data writes as the others!! Is it definitely LTO3 and definitely using LTO3 media? LTO2 was about half the speed, including using LTO2 media in an LTO3 drive. James Yes, all 4 drives are HP Ultrium 3 drives. And the same LTO3 bacula volume was used in all 4 testing runs today. All drives are connected via 2Gb fiber. All tests were done independent of each other with no other activity on the backup server during the time of the testing. Stephen -- Stephen Thompson Berkeley Seismological Laboratory step...@seismo.berkeley.edu215 McCone Hall # 4760 404.538.7077 (phone) University of California, Berkeley 510.643.5811 (fax) Berkeley, CA 94720-4760 -- Got visibility? Most devs has no idea what their production app looks like. Find out how fast your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219671;13503038;y? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
On 01/10/12 23:38, Stephen Thompson wrote: More importantly, I realized that my testing 6 months ago was not on all 4 of my drives, but only 2 of them. Today, I discovered one of my drives (untested in the past) is getting 1/2 the throughput for random data writes as the others!! smartctl -a /dev/sg(drive) will tell you a lot Put a cleaning tape in it -- Got visibility? Most devs has no idea what their production app looks like. Find out how fast your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219671;13503038;y? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
On 10/1/12 4:06 PM, Alan Brown wrote: On 01/10/12 23:38, Stephen Thompson wrote: More importantly, I realized that my testing 6 months ago was not on all 4 of my drives, but only 2 of them. Today, I discovered one of my drives (untested in the past) is getting 1/2 the throughput for random data writes as the others!! smartctl -a /dev/sg(drive) will tell you a lot Put a cleaning tape in it Cleaning tape did not improve results. I see some errors in the counter log on the problem drive, but I see even more errors on another drive which isn't having a throughput problem (specifically SL500 Drive 1 is the lower throughput, but C4 Drive 1 actually has a higher error count). SL500 Drive 0 (~60MB/s random data throughput) = Error counter log: Errors Corrected by Total Correction GigabytesTotal ECC rereads/errors algorithm processeduncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 00 0 0 0 0.000 0 write: 00 0 0 0 0.000 0 SL500 Drive 1 (~30MB/s random data throughput) = Error counter log: Errors Corrected by Total Correction GigabytesTotal ECC rereads/errors algorithm processeduncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 00 0 0 0 0.000 0 write: 104540 0 0 821389 0.000 0 C4 Drive 0 (~60MB/s random data throughput) == Error counter log: Errors Corrected by Total Correction GigabytesTotal ECC rereads/errors algorithm processeduncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 20 0 0 2 0.000 0 write: 00 0 0 0 0.000 0 C4 Drive 1 (~60MB/s random data throughput) == Error counter log: Errors Corrected by Total Correction GigabytesTotal ECC rereads/errors algorithm processeduncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 20 0 0 2 0.000 0 write: 189610 0 0 48261 0.000 0 Stephen -- Stephen Thompson Berkeley Seismological Laboratory step...@seismo.berkeley.edu215 McCone Hall # 4760 404.538.7077 (phone) University of California, Berkeley 510.643.5811 (fax) Berkeley, CA 94720-4760 -- Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
Correction, the non-problem drive has a higher ECC fast error count, but the problem drive has a significantly higher Corrective algorithm invocations count. On 10/1/12 5:33 PM, Stephen Thompson wrote: On 10/1/12 4:06 PM, Alan Brown wrote: On 01/10/12 23:38, Stephen Thompson wrote: More importantly, I realized that my testing 6 months ago was not on all 4 of my drives, but only 2 of them. Today, I discovered one of my drives (untested in the past) is getting 1/2 the throughput for random data writes as the others!! smartctl -a /dev/sg(drive) will tell you a lot Put a cleaning tape in it Cleaning tape did not improve results. I see some errors in the counter log on the problem drive, but I see even more errors on another drive which isn't having a throughput problem (specifically SL500 Drive 1 is the lower throughput, but C4 Drive 1 actually has a higher error count). SL500 Drive 0 (~60MB/s random data throughput) = Error counter log: Errors Corrected by Total Correction GigabytesTotal ECC rereads/errors algorithm processeduncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 00 0 0 0 0.000 0 write: 00 0 0 0 0.000 0 SL500 Drive 1 (~30MB/s random data throughput) = Error counter log: Errors Corrected by Total Correction GigabytesTotal ECC rereads/errors algorithm processeduncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 00 0 0 0 0.000 0 write: 104540 0 0 821389 0.000 0 C4 Drive 0 (~60MB/s random data throughput) == Error counter log: Errors Corrected by Total Correction GigabytesTotal ECC rereads/errors algorithm processeduncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 20 0 0 2 0.000 0 write: 00 0 0 0 0.000 0 C4 Drive 1 (~60MB/s random data throughput) == Error counter log: Errors Corrected by Total Correction GigabytesTotal ECC rereads/errors algorithm processeduncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 20 0 0 2 0.000 0 write: 189610 0 0 48261 0.000 0 Stephen -- Stephen Thompson Berkeley Seismological Laboratory step...@seismo.berkeley.edu215 McCone Hall # 4760 404.538.7077 (phone) University of California, Berkeley 510.643.5811 (fax) Berkeley, CA 94720-4760 -- Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
Alan Brown wrote (2012/09/28): Aren't these considered reasonable settings for LTO3? Maximum block size = 262144 # 256kb Maximum File Size = 2gb Not really. Hi, I have Maximum Block Size = 65536 Maximum File Size = 4gb and without any problem. All tapes have over 400,000,000,000 bytes, except 2 of 116, where I think that one was forced to change, so just 1 of 116 has under 400 GB (around 380 GB). Almost all tapes are 7 years old now. My typical end of tape mesage is like End of medium on Volume Bytes=618,539,200,512 Blocks=9,438,187 Change maximum file size to 10Gb and maximum block size to 2M However, then please test readibility of these data on all systems you could possibly use. Increasing blocks size is potentially dangerous, so before increasing it, be sure, that you can read all these data back. -- Rudolf Cejka cejkar at fit.vutbr.cz http://www.fit.vutbr.cz/~cejkar Brno University of Technology, Faculty of Information Technology Bozetechova 2, 612 66 Brno, Czech Republic -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://ad.doubleclick.net/clk;258768047;13503038;j? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
On 30/09/12 22:42, Cejka Rudolf wrote: Hi, I have Maximum Block Size = 65536 Maximum File Size = 4gb and without any problem. All tapes have over 400,000,000,000 bytes, except 2 of 116, where I think that one was forced to change, so just 1 of 116 has under 400 GB (around 380 GB). Almost all tapes are 7 years old now. My typical end of tape mesage is like End of medium on Volume Bytes=618,539,200,512 Blocks=9,438,187 Change maximum file size to 10Gb and maximum block size to 2M However, then please test readibility of these data on all systems you could possibly use. Increasing blocks size is potentially dangerous, so before increasing it, be sure, that you can read all these data back. 2Mb is the maximum block size I've found to work on LTO* - it's a bacula limit, not an LTO one. Increasing from 65535 will improve throughput significantly. There have been discussions about this in the last few years. -- Got visibility? Most devs has no idea what their production app looks like. Find out how fast your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219671;13503038;y? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
On 28/09/12 02:38, Stephen Thompson wrote: Aren't these considered reasonable settings for LTO3? Maximum block size = 262144 # 256kb Maximum File Size = 2gb Not really. Change maximum file size to 10Gb and maximum block size to 2M You _must_ set all open tapes to used and restart the storage daemon when changing the block size. Bacula can't cope with varying maximum sizes on a tape Even with those changes, if you have a lot of small, incompressible files you'll see high tape overheads. thanks for the help! Stephen -- Got visibility? Most devs has no idea what their production app looks like. Find out how fast your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219671;13503038;y? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
On 26/09/12 22:37, Stephen Thompson wrote: I think I pointed this out before, but I also have used and new tapes with 400-800Gb on them. It seems really hit or miss, though the tapes with 400Gb or less are probably a 1/3 of my tapes. The other 2/3 have above 400Gb. If you have small blocking factors and a lot of small files it's possible that bacula overheads are high (it reports actual data on tape, not data+overheads) In my experience, tapes which achieve high compression factors are usually full of incremental backups containing highly repetitive data such as logfiles. Tape capacities are quoted in Gb = 1*10^9, while bacula uses GiB for its reporting, but the kind of underrun you're seeing can't be entirely explained by that small difference. I've occasionally seen LTO5 tapes marked as full when they're well short of expected capacity but this generally happens on a drive which is about to request a cleaning tape. Marking them as append has a 50:50 chance of allowing them to continue and generally they're fine when recycled. Have you tried btape to test the tape in question, or used dd to see how many bytes are actually on the tape? Smartctl will help too. -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://ad.doubleclick.net/clk;258768047;13503038;j? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
On 09/25/2012 10:43 AM, Alan Brown wrote: On 25/09/12 17:43, Stephen Thompson wrote: Our Sun/Oracle service engineer claims that our drives do not require cleaning tapes. Does that sound legit? In general: true (as in, Don't do it as a scheduled item), but all LTO drives require cleaning tapes from time to time and sometimes benefit from loading one even if the clean light isn't on. It primarily depends on the cleanliness of the room where the drive is. Our throughput is pretty reasonable for our hardware -- we do use disk staging and get something like 60Mb/s to tape. 60Mb/s is _slow_ for LTO3. You need to take a serious look at what you're using as stage disk and consider using a raid0 array of SSDs in order to keep up. Lastly, the tapes that get 200 vs 800 are from the same batch of tapes, same number of uses, and used by the same pair of SL500 drives. That's primarily why I wondered if it could be data dependent (or a bacula bug). What happens if you mark the volumes as append and put them back in the library? I haven't had a lot of time to look into this today, but I do this quick test and it immediately marks the volume Full again. 27-Sep 14:20 sd-SL500 JobId 260069: Volume FB0763 previously written, moving to end of data. 27-Sep 14:21 sd-SL500 JobId 260069: Ready to append to end of Volume FB0763 at file=110. 27-Sep 14:21 sd-SL500 JobId 260069: Spooling data ... 27-Sep 14:21 sd-SL500 JobId 260069: Job write elapsed time = 00:00:01, Transfer rate = 759.3 K Bytes/second 27-Sep 14:21 sd-SL500 JobId 260069: Committing spooled data to Volume FB0763. Despooling 762,358 bytes ... 27-Sep 14:21 sd-SL500 JobId 260069: End of Volume FB0763 at 110:1 on device SL500-Drive-0 (/dev/SL500-Drive-0). Write of 262144 bytes got -1. 27-Sep 14:21 sd-SL500 JobId 260069: Re-read of last block succeeded. 27-Sep 14:21 sd-SL500 JobId 260069: End of medium on Volume FB0763 Bytes=219,730,936,832 Blocks=838,207 at 27-Sep-2012 14:21. 27-Sep 14:21 sd-SL500 JobId 260069: 3307 Issuing autochanger unload slot 36, drive 0 command. I've seen transient scsi errors result in tapes being marked as full. What does smartctl show for the drive and tape in question? (run this against the /dev/sg of the tape drive) -- Stephen Thompson Berkeley Seismological Laboratory step...@seismo.berkeley.edu215 McCone Hall # 4760 404.538.7077 (phone) University of California, Berkeley 510.643.5811 (fax) Berkeley, CA 94720-4760 -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://ad.doubleclick.net/clk;258768047;13503038;j? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
On 27/09/12 22:25, Stephen Thompson wrote: What happens if you mark the volumes as append and put them back in the library? I haven't had a lot of time to look into this today, but I do this quick test and it immediately marks the volume Full again. Then it really is full and the rest is down to overheads. Consider using larger block sizes. -- Got visibility? Most devs has no idea what their production app looks like. Find out how fast your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219671;13503038;y? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
On 9/27/12 6:17 PM, Alan Brown wrote: On 27/09/12 22:25, Stephen Thompson wrote: What happens if you mark the volumes as append and put them back in the library? I haven't had a lot of time to look into this today, but I do this quick test and it immediately marks the volume Full again. Then it really is full and the rest is down to overheads. Consider using larger block sizes. Aren't these considered reasonable settings for LTO3? Maximum block size = 262144 # 256kb Maximum File Size = 2gb thanks for the help! Stephen -- Stephen Thompson Berkeley Seismological Laboratory step...@seismo.berkeley.edu215 McCone Hall # 4760 404.538.7077 (phone) University of California, Berkeley 510.643.5811 (fax) Berkeley, CA 94720-4760 -- Got visibility? Most devs has no idea what their production app looks like. Find out how fast your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219671;13503038;y? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
On 09/25/2012 02:29 PM, Cejka Rudolf wrote: Stephen Thompson wrote (2012/09/25): The tape in question have only been used once or twice. Do you mean just one or two drive loads and unloads? Yes, I mean the tapes have only been in a drive once or twice, possibly for a dozen sequential jobs while in the drive, but only in and out of the drive once or twice. I have seen this 200-300Gb capacity on new tapes as well as used. I see it in both my SL500 library as well as my C4 library, which is a combined 4 LTO3 drives (2 in each library). The library is a StorageTek whose SLConsole reports no media (or drive) errors, though I will look into those linux-based tools. There are several types of errors, recoverable and non-recoverable, and I'm afraid that you see just non-recoverable, but it is too late to see them. Our Sun/Oracle service engineer claims that our drives do not require cleaning tapes. Does that sound legit? If you are interested, you can study http://www.tarconis.com/documentos/LTO_Cleaning_wp.pdf ;o) So in HP case, it is possible to agree. However, you still have to have atleast one cleaning cartridge prepared ;o) Our throughput is pretty reasonable for our hardware -- we do use disk staging and get something like 60Mb/s to tape. HP LTO-3 drive can slow down physical speed to 27 MB/s, IBM LTO-3 to 40 MB/s. Native speed is 80 MB/s, bot all these speeds are after compression. If you have 60 MB/s before compression and there are some places with somewhat better compression than 2:1, then you are not able to feed HP LTO-3. For IBM drive, it is suffucient to have places with just 2:1 to need repositions. Lastly, the tapes that get 200 vs 800 are from the same batch of tapes, same number of uses, and used by the same pair of SL500 drives. That's primarily why I wondered if it could be data dependent (or a bacula bug). And what about the reason to switch to the next tape? Do you have something like this in your reports? 22-Sep 02:22 backup-sd JobId 74990: End of Volume 1 at 95:46412 on device drive0 (/dev/nsa0). Write of 65536 bytes got 0. 22-Sep 02:22 backup-sd JobId 74990: Re-read of last block succeeded. 22-Sep 02:22 backup-sd JobId 74990: End of medium on Volume 1 Bytes=381,238,317,056 Blocks=5,817,238 at 22-Sep-2012 02:22. Here's an example of a tape that had one job and only wrote ~278Gb to the tape: 10-Sep 10:08 sd-SL500 JobId 256773: Recycled volume FB0095 on device SL500-Drive-1 (/dev/SL500-Drive-1), all previous data lost. 10-Sep 10:08 sd-SL500 JobId 256773: New volume FB0095 mounted on device SL500-Drive-1 (/dev/SL500-Drive-1) at 10-Sep-2012 10:08. 10-Sep 13:02 sd-SL500 JobId 256773: End of Volume FB0095 at 149:5906 on device SL500-Drive-1 (/dev/SL500-Drive-1). Write of 262144 bytes got -1. 10-Sep 13:02 sd-SL500 JobId 256773: Re-read of last block succeeded. 10-Sep 13:02 sd-SL500 JobId 256773: End of medium on Volume FB0095 Bytes=299,532,813,312 Blocks=1,142,627 at 10-Sep-2012 13:02. Do not you use something from the following things in bacula configuration? UseVolumeOnce Maximum Volume Jobs Maximum Volume Bytes Volume Use Duration ? No, none of those are configured. Stephen -- Stephen Thompson Berkeley Seismological Laboratory step...@seismo.berkeley.edu215 McCone Hall # 4760 404.538.7077 (phone) University of California, Berkeley 510.643.5811 (fax) Berkeley, CA 94720-4760 -- How fast is your code? 3 out of 4 devs don\\\'t know how their code performs in production. Find out how slow your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219672;13503038;z? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
On 09/26/2012 02:35 PM, Stephen Thompson wrote: On 09/25/2012 02:29 PM, Cejka Rudolf wrote: Stephen Thompson wrote (2012/09/25): The tape in question have only been used once or twice. Do you mean just one or two drive loads and unloads? Yes, I mean the tapes have only been in a drive once or twice, possibly for a dozen sequential jobs while in the drive, but only in and out of the drive once or twice. I have seen this 200-300Gb capacity on new tapes as well as used. I think I pointed this out before, but I also have used and new tapes with 400-800Gb on them. It seems really hit or miss, though the tapes with 400Gb or less are probably a 1/3 of my tapes. The other 2/3 have above 400Gb. I see it in both my SL500 library as well as my C4 library, which is a combined 4 LTO3 drives (2 in each library). The library is a StorageTek whose SLConsole reports no media (or drive) errors, though I will look into those linux-based tools. There are several types of errors, recoverable and non-recoverable, and I'm afraid that you see just non-recoverable, but it is too late to see them. Our Sun/Oracle service engineer claims that our drives do not require cleaning tapes. Does that sound legit? If you are interested, you can study http://www.tarconis.com/documentos/LTO_Cleaning_wp.pdf ;o) So in HP case, it is possible to agree. However, you still have to have atleast one cleaning cartridge prepared ;o) Our throughput is pretty reasonable for our hardware -- we do use disk staging and get something like 60Mb/s to tape. HP LTO-3 drive can slow down physical speed to 27 MB/s, IBM LTO-3 to 40 MB/s. Native speed is 80 MB/s, bot all these speeds are after compression. If you have 60 MB/s before compression and there are some places with somewhat better compression than 2:1, then you are not able to feed HP LTO-3. For IBM drive, it is suffucient to have places with just 2:1 to need repositions. Lastly, the tapes that get 200 vs 800 are from the same batch of tapes, same number of uses, and used by the same pair of SL500 drives. That's primarily why I wondered if it could be data dependent (or a bacula bug). And what about the reason to switch to the next tape? Do you have something like this in your reports? 22-Sep 02:22 backup-sd JobId 74990: End of Volume 1 at 95:46412 on device drive0 (/dev/nsa0). Write of 65536 bytes got 0. 22-Sep 02:22 backup-sd JobId 74990: Re-read of last block succeeded. 22-Sep 02:22 backup-sd JobId 74990: End of medium on Volume 1 Bytes=381,238,317,056 Blocks=5,817,238 at 22-Sep-2012 02:22. Here's an example of a tape that had one job and only wrote ~278Gb to the tape: 10-Sep 10:08 sd-SL500 JobId 256773: Recycled volume FB0095 on device SL500-Drive-1 (/dev/SL500-Drive-1), all previous data lost. 10-Sep 10:08 sd-SL500 JobId 256773: New volume FB0095 mounted on device SL500-Drive-1 (/dev/SL500-Drive-1) at 10-Sep-2012 10:08. 10-Sep 13:02 sd-SL500 JobId 256773: End of Volume FB0095 at 149:5906 on device SL500-Drive-1 (/dev/SL500-Drive-1). Write of 262144 bytes got -1. 10-Sep 13:02 sd-SL500 JobId 256773: Re-read of last block succeeded. 10-Sep 13:02 sd-SL500 JobId 256773: End of medium on Volume FB0095 Bytes=299,532,813,312 Blocks=1,142,627 at 10-Sep-2012 13:02. Do not you use something from the following things in bacula configuration? UseVolumeOnce Maximum Volume Jobs Maximum Volume Bytes Volume Use Duration ? No, none of those are configured. Stephen -- Stephen Thompson Berkeley Seismological Laboratory step...@seismo.berkeley.edu215 McCone Hall # 4760 404.538.7077 (phone) University of California, Berkeley 510.643.5811 (fax) Berkeley, CA 94720-4760 -- How fast is your code? 3 out of 4 devs don\\\'t know how their code performs in production. Find out how slow your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219672;13503038;z? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
Zitat von Stephen Thompson step...@seismo.berkeley.edu: Thanks for the info, John. Is there anyone else in the bacula community with LTO3's seeing this behaviour? I don't believe (but am not 100% sure) that I'm having any hardware-related issues. Not sure what to make of this. About 25% of tapes in a monthly run (70 tapes) are under the 400Gb native, but then the other 75% are above it, some even hitting the 800Gb top. As said by others. The native capacity is 400Gb and Bacula simply writes to the tape until the write does not succed which Bacula take as tape full. If you have some (full) tapes with less than 400Gb you either have some older tapes (LTO-2) or problem with your hardware or tapes. The tapes with more than 400Gb simply contains data which can be better compressed by the built-in compression of the drive. Regards Andreas -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
Op 25/09/2012 2:03, James Harper schreef: Hello all, This is not likely a bacula questions, but in the chance that it is, or the experience on this list, I figured I would ask. We've been using LTO3 tapes with bacula for a few years now. Recently I've noticed how variable our tape capacity it, ranging from 200-800 Gb. Is that strictly governed by the compressibility of the actual data being backed up? Or is there some chance that bacula isn't squeezing as much onto my tapes as I would expect? 200Gb is not very much! I don't think this explains your issue, but LTO drives will write the data to the tape, and then immediately read it again (the read head is placed such that this is possible). If the read is bad the drive will rewrite the data. This ensures that you get a good write, but obviously decreases the effective capacity of your tape. Your tapes would have to be pretty worn out to drop the capacity to 25% though. The tape and/or drive should record the margin and other figures, but I don't know of any Linux tools to read that information. James The tools depend on the brand. For HP drives ltt is nice and works flawlessly on linux command-line. It reports drive errors, tape errors for the past 4 tapes, tape and drive margin, and a lot of other info. http://h18006.www1.hp.com/products/storageworks/ltt/ I know IBM has their own tool (ITDT), but as far as I know there is no command-line version of their tool. Other brands, no clue. Regards, Jeremy DISCLAIMER http://www.schaubroeck.be/maildisclaimer.htm -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
On 25/09/12 09:08, Jeremy Maes wrote: The tape and/or drive should record the margin and other figures, but I don't know of any Linux tools to read that information. The tools depend on the brand. For HP drives ltt is nice and works flawlessly on linux command-line. It reports drive errors, tape errors for the past 4 tapes, tape and drive margin, and a lot of other info. http://h18006.www1.hp.com/products/storageworks/ltt/ I know IBM has their own tool (ITDT), but as far as I know there is no command-line version of their tool. Other brands, no clue. Smartctl gives some usable information on each tape - certainly enough to assess if it's bad or not. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
We've been using LTO3 tapes with bacula for a few years now. Recently I've noticed how variable our tape capacity it, ranging from 200-800 Gb. Is that strictly governed by the compressibility of the actual data being backed up? Hello, the lower bound 200 GB on 400 GB LTO-3 tapes is not possible due to the drive compression, because it always compares, if compressed data are shorter that original. In other case, it writes data uncompressed. So, in all cases, you should see atleast 400 000 000 000 bytes written on all tapes. Or is there some chance that bacula isn't squeezing as much onto my tapes as I would expect? 200Gb is not very much! In bacula, look mainly for the reasons, why there is just 200 GB written. If the tape is full, think about these: - Weared tapes. Typical tape service life is written as 200 full cycles. However, read http://www.xma4govt.co.uk/Libraries/Manufacturer/ultriumwhitepaper_EEE.sflb where they experienced problems with some tapes just only after 30 cycles! How many cycles could you have with your tapes? - Do you use disk staging, so that tape writes are done at full speed? Do you have a good disk staging? Considering using SSDs for staging is very wise. If data rate is lower that 1/3 to 1/2 of native tape speed (based on drive vendor, HP or IBM), then drive has to perform tape repositions, which means another important excessive drive and tape wearing. My experiences are, that even HW RAID-0 with four 10k disks could not be sufficient and when there are data writes and reads in parallel, it could not put 80 MB/s to the drive, typically just 50-70 MB/s, which is still acceptable for LTO-3, but not good. Currently, I have 4 x 450 GB SSDs HW RAID-0 with over 1500 GB/s without problem running writes and reads in parallel and just after that I hope that it is really sufficient for = LTO-3 staging and putting drives and tapes wearing to minimum. - Dirty heads. You can enforce cleaning cycle, but then return to the two points above and other suggestiong, like using some monitoring like ltt on Linux (or I have some home made reporting tool using camcontrol on FreeBSD), where it would be possible to ensure, that your problem are weared tapes, or something else. Best regards. -- Rudolf Cejka cejkar at fit.vutbr.cz http://www.fit.vutbr.cz/~cejkar Brno University of Technology, Faculty of Information Technology Bozetechova 2, 612 66 Brno, Czech Republic -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
Thanks everyone for the suggestions, they at least give me somewhere to look, as I was running low on ideas. More info... The tape in question have only been used once or twice. The library is a StorageTek whose SLConsole reports no media (or drive) errors, though I will look into those linux-based tools. Our Sun/Oracle service engineer claims that our drives do not require cleaning tapes. Does that sound legit? Our throughput is pretty reasonable for our hardware -- we do use disk staging and get something like 60Mb/s to tape. Lastly, the tapes that get 200 vs 800 are from the same batch of tapes, same number of uses, and used by the same pair of SL500 drives. That's primarily why I wondered if it could be data dependent (or a bacula bug). thanks! Stephen On 09/25/12 02:29, Cejka Rudolf wrote: We've been using LTO3 tapes with bacula for a few years now. Recently I've noticed how variable our tape capacity it, ranging from 200-800 Gb. Is that strictly governed by the compressibility of the actual data being backed up? Hello, the lower bound 200 GB on 400 GB LTO-3 tapes is not possible due to the drive compression, because it always compares, if compressed data are shorter that original. In other case, it writes data uncompressed. So, in all cases, you should see atleast 400 000 000 000 bytes written on all tapes. Or is there some chance that bacula isn't squeezing as much onto my tapes as I would expect? 200Gb is not very much! In bacula, look mainly for the reasons, why there is just 200 GB written. If the tape is full, think about these: - Weared tapes. Typical tape service life is written as 200 full cycles. However, read http://www.xma4govt.co.uk/Libraries/Manufacturer/ultriumwhitepaper_EEE.sflb where they experienced problems with some tapes just only after 30 cycles! How many cycles could you have with your tapes? - Do you use disk staging, so that tape writes are done at full speed? Do you have a good disk staging? Considering using SSDs for staging is very wise. If data rate is lower that 1/3 to 1/2 of native tape speed (based on drive vendor, HP or IBM), then drive has to perform tape repositions, which means another important excessive drive and tape wearing. My experiences are, that even HW RAID-0 with four 10k disks could not be sufficient and when there are data writes and reads in parallel, it could not put 80 MB/s to the drive, typically just 50-70 MB/s, which is still acceptable for LTO-3, but not good. Currently, I have 4 x 450 GB SSDs HW RAID-0 with over 1500 GB/s without problem running writes and reads in parallel and just after that I hope that it is really sufficient for = LTO-3 staging and putting drives and tapes wearing to minimum. - Dirty heads. You can enforce cleaning cycle, but then return to the two points above and other suggestiong, like using some monitoring like ltt on Linux (or I have some home made reporting tool using camcontrol on FreeBSD), where it would be possible to ensure, that your problem are weared tapes, or something else. Best regards. -- Stephen Thompson Berkeley Seismological Laboratory step...@seismo.berkeley.edu215 McCone Hall # 4760 404.538.7077 (phone) University of California, Berkeley 510.643.5811 (fax) Berkeley, CA 94720-4760 -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
On 25/09/12 17:43, Stephen Thompson wrote: Our Sun/Oracle service engineer claims that our drives do not require cleaning tapes. Does that sound legit? In general: true (as in, Don't do it as a scheduled item), but all LTO drives require cleaning tapes from time to time and sometimes benefit from loading one even if the clean light isn't on. It primarily depends on the cleanliness of the room where the drive is. Our throughput is pretty reasonable for our hardware -- we do use disk staging and get something like 60Mb/s to tape. 60Mb/s is _slow_ for LTO3. You need to take a serious look at what you're using as stage disk and consider using a raid0 array of SSDs in order to keep up. Lastly, the tapes that get 200 vs 800 are from the same batch of tapes, same number of uses, and used by the same pair of SL500 drives. That's primarily why I wondered if it could be data dependent (or a bacula bug). What happens if you mark the volumes as append and put them back in the library? I've seen transient scsi errors result in tapes being marked as full. What does smartctl show for the drive and tape in question? (run this against the /dev/sg of the tape drive) -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
On 9/25/12 12:43 PM, Stephen Thompson step...@seismo.berkeley.edu wrote: Thanks everyone for the suggestions, they at least give me somewhere to look, as I was running low on ideas. More info... The tape in question have only been used once or twice. The library is a StorageTek whose SLConsole reports no media (or drive) errors, though I will look into those linux-based tools. Our Sun/Oracle service engineer claims that our drives do not require cleaning tapes. Does that sound legit? Our throughput is pretty reasonable for our hardware -- we do use disk staging and get something like 60Mb/s to tape. Lastly, the tapes that get 200 vs 800 are from the same batch of tapes, same number of uses, and used by the same pair of SL500 drives. That's primarily why I wondered if it could be data dependent (or a bacula bug). thanks! Stephen I've found LTO drives of any variety rarely need cleaning. I've found that one cleaning tape will usually be sufficient for the life of a library. Your field support may very well be right. Patti Clark Linux Administrator Research and Development Systems Support Oak Ridge National Laboratory -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
On 09/25/2012 10:43 AM, Alan Brown wrote: On 25/09/12 17:43, Stephen Thompson wrote: Our Sun/Oracle service engineer claims that our drives do not require cleaning tapes. Does that sound legit? In general: true (as in, Don't do it as a scheduled item), but all LTO drives require cleaning tapes from time to time and sometimes benefit from loading one even if the clean light isn't on. It primarily depends on the cleanliness of the room where the drive is. Our throughput is pretty reasonable for our hardware -- we do use disk staging and get something like 60Mb/s to tape. 60Mb/s is _slow_ for LTO3. You need to take a serious look at what you're using as stage disk and consider using a raid0 array of SSDs in order to keep up. Why do you say that's slow when the max speed appears to be 80? http://en.wikipedia.org/wiki/Linear_Tape-Open Lastly, the tapes that get 200 vs 800 are from the same batch of tapes, same number of uses, and used by the same pair of SL500 drives. That's primarily why I wondered if it could be data dependent (or a bacula bug). What happens if you mark the volumes as append and put them back in the library? I've seen transient scsi errors result in tapes being marked as full. What does smartctl show for the drive and tape in question? (run this against the /dev/sg of the tape drive) -- Stephen Thompson Berkeley Seismological Laboratory step...@seismo.berkeley.edu215 McCone Hall # 4760 404.538.7077 (phone) University of California, Berkeley 510.643.5811 (fax) Berkeley, CA 94720-4760 -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
On 09/25/2012 11:17 AM, Konstantin Khomoutov wrote: On Tue, 25 Sep 2012 11:00:07 -0700 Stephen Thompson step...@seismo.berkeley.edu wrote: 60Mb/s is _slow_ for LTO3. You need to take a serious look at what you're using as stage disk and consider using a raid0 array of SSDs in order to keep up. Why do you say that's slow when the max speed appears to be 80? http://en.wikipedia.org/wiki/Linear_Tape-Open It's quite logical, that to not starve the consumer, the producer should be at least as fast or faster, so you have to provide at least 80 Mb/s sustained read rate from your spooling media to be sure the tape drive is kept busy. No, I mean, there's slow and there's __SLOW__. He seemed to be indicating that it was unacceptably slow. I understand it's not optimal. Stephen -- Stephen Thompson Berkeley Seismological Laboratory step...@seismo.berkeley.edu215 McCone Hall # 4760 404.538.7077 (phone) University of California, Berkeley 510.643.5811 (fax) Berkeley, CA 94720-4760 -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
Zitat von Stephen Thompson step...@seismo.berkeley.edu: On 09/25/2012 10:43 AM, Alan Brown wrote: On 25/09/12 17:43, Stephen Thompson wrote: Our Sun/Oracle service engineer claims that our drives do not require cleaning tapes. Does that sound legit? In general: true (as in, Don't do it as a scheduled item), but all LTO drives require cleaning tapes from time to time and sometimes benefit from loading one even if the clean light isn't on. It primarily depends on the cleanliness of the room where the drive is. Our throughput is pretty reasonable for our hardware -- we do use disk staging and get something like 60Mb/s to tape. 60Mb/s is _slow_ for LTO3. You need to take a serious look at what you're using as stage disk and consider using a raid0 array of SSDs in order to keep up. Why do you say that's slow when the max speed appears to be 80? http://en.wikipedia.org/wiki/Linear_Tape-Open The maximum *native* speed with uncompressed data is 80Mb/s. As the drives do compression to get more data to the tape the speed to deliver data must be higher. If you use the optimistic 2:1 compression from marketing your tape drive is actually crawling with 30Mb/s which is not even 50% of the native speed. If you suspekt bacula bug you can still use dd to fill the tape and see what you get in speed/capacity. Regards Andreas -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
Stephen Thompson wrote (2012/09/25): The tape in question have only been used once or twice. Do you mean just one or two drive loads and unloads? The library is a StorageTek whose SLConsole reports no media (or drive) errors, though I will look into those linux-based tools. There are several types of errors, recoverable and non-recoverable, and I'm afraid that you see just non-recoverable, but it is too late to see them. Our Sun/Oracle service engineer claims that our drives do not require cleaning tapes. Does that sound legit? If you are interested, you can study http://www.tarconis.com/documentos/LTO_Cleaning_wp.pdf ;o) So in HP case, it is possible to agree. However, you still have to have atleast one cleaning cartridge prepared ;o) Our throughput is pretty reasonable for our hardware -- we do use disk staging and get something like 60Mb/s to tape. HP LTO-3 drive can slow down physical speed to 27 MB/s, IBM LTO-3 to 40 MB/s. Native speed is 80 MB/s, bot all these speeds are after compression. If you have 60 MB/s before compression and there are some places with somewhat better compression than 2:1, then you are not able to feed HP LTO-3. For IBM drive, it is suffucient to have places with just 2:1 to need repositions. Lastly, the tapes that get 200 vs 800 are from the same batch of tapes, same number of uses, and used by the same pair of SL500 drives. That's primarily why I wondered if it could be data dependent (or a bacula bug). And what about the reason to switch to the next tape? Do you have something like this in your reports? 22-Sep 02:22 backup-sd JobId 74990: End of Volume 1 at 95:46412 on device drive0 (/dev/nsa0). Write of 65536 bytes got 0. 22-Sep 02:22 backup-sd JobId 74990: Re-read of last block succeeded. 22-Sep 02:22 backup-sd JobId 74990: End of medium on Volume 1 Bytes=381,238,317,056 Blocks=5,817,238 at 22-Sep-2012 02:22. Do not you use something from the following things in bacula configuration? UseVolumeOnce Maximum Volume Jobs Maximum Volume Bytes Volume Use Duration ? -- Rudolf Cejka cejkar at fit.vutbr.cz http://www.fit.vutbr.cz/~cejkar Brno University of Technology, Faculty of Information Technology Bozetechova 2, 612 66 Brno, Czech Republic -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
I've found LTO drives of any variety rarely need cleaning. I've found that one cleaning tape will usually be sufficient for the life of a library. Your field support may very well be right. The drive has an internal cleaning mechanism that is activated on load http://en.wikipedia.org/wiki/Linear_Tape-Open#Cleaning which is normally enough to keep the heads clean. In my experience, when the drive starts telling you it needs cleaning, it's probably failing. James -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] LTO3 tape capacity (variable?)
Hello all, This is not likely a bacula questions, but in the chance that it is, or the experience on this list, I figured I would ask. We've been using LTO3 tapes with bacula for a few years now. Recently I've noticed how variable our tape capacity it, ranging from 200-800 Gb. Is that strictly governed by the compressibility of the actual data being backed up? Or is there some chance that bacula isn't squeezing as much onto my tapes as I would expect? 200Gb is not very much! thanks, Stephen -- Stephen Thompson Berkeley Seismological Laboratory step...@seismo.berkeley.edu215 McCone Hall # 4760 404.538.7077 (phone) University of California, Berkeley 510.643.5811 (fax) Berkeley, CA 94720-4760 -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
This is not likely a bacula questions, but in the chance that it is, or the experience on this list, I figured I would ask. We've been using LTO3 tapes with bacula for a few years now. Recently I've noticed how variable our tape capacity it, ranging from 200-800 Gb. Is that strictly governed by the compressibility of the actual data being backed up? Or is there some chance that bacula isn't squeezing as much onto my tapes as I would expect? 200Gb is not very much! These tapes are 400GB native. If you get substantially less than that you have a configuration problem (you set limits on the volume size or duration) or a hardware problem. Compression should be handled entirely and automatically by the tape drive. Bacula does not enable or disable hardware compression it just passes the data to the drive and writes as much as it can up until it hits its first hardware error. At that point bacula calls the tape full and verifies that it can read the last block. I believe if it can't read the last block this block will be the first block written on the next volume. John -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
Thanks for the info, John. Is there anyone else in the bacula community with LTO3's seeing this behaviour? I don't believe (but am not 100% sure) that I'm having any hardware-related issues. Not sure what to make of this. About 25% of tapes in a monthly run (70 tapes) are under the 400Gb native, but then the other 75% are above it, some even hitting the 800Gb top. Stephen On 09/24/2012 12:02 PM, John Drescher wrote: This is not likely a bacula questions, but in the chance that it is, or the experience on this list, I figured I would ask. We've been using LTO3 tapes with bacula for a few years now. Recently I've noticed how variable our tape capacity it, ranging from 200-800 Gb. Is that strictly governed by the compressibility of the actual data being backed up? Or is there some chance that bacula isn't squeezing as much onto my tapes as I would expect? 200Gb is not very much! These tapes are 400GB native. If you get substantially less than that you have a configuration problem (you set limits on the volume size or duration) or a hardware problem. Compression should be handled entirely and automatically by the tape drive. Bacula does not enable or disable hardware compression it just passes the data to the drive and writes as much as it can up until it hits its first hardware error. At that point bacula calls the tape full and verifies that it can read the last block. I believe if it can't read the last block this block will be the first block written on the next volume. John -- Stephen Thompson Berkeley Seismological Laboratory step...@seismo.berkeley.edu215 McCone Hall # 4760 404.538.7077 (phone) University of California, Berkeley 510.643.5811 (fax) Berkeley, CA 94720-4760 -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO3 tape capacity (variable?)
Hello all, This is not likely a bacula questions, but in the chance that it is, or the experience on this list, I figured I would ask. We've been using LTO3 tapes with bacula for a few years now. Recently I've noticed how variable our tape capacity it, ranging from 200-800 Gb. Is that strictly governed by the compressibility of the actual data being backed up? Or is there some chance that bacula isn't squeezing as much onto my tapes as I would expect? 200Gb is not very much! I don't think this explains your issue, but LTO drives will write the data to the tape, and then immediately read it again (the read head is placed such that this is possible). If the read is bad the drive will rewrite the data. This ensures that you get a good write, but obviously decreases the effective capacity of your tape. Your tapes would have to be pretty worn out to drop the capacity to 25% though. The tape and/or drive should record the margin and other figures, but I don't know of any Linux tools to read that information. James -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
