Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
Nils Juergens writes: > for me (with LTO-3) increasing Maximum block size has had quite the > impact on performance. > > Sadly, with the new setting I had trouble reading my tapes so switched > back to the old configuration. in Linux, you can configure your tape drive to accept variable block size with "mt setblk 0". without a variable block size, I have found that you need to set minimum and maximum block size to the same value, or else Bacula may write an, e.g., 66 KiB block if you have configured minimum block size to be 64 KiB. at least my LTO-5 drives will not accept a 66 KiB write after "mt setblk 65536". oh, and I'm using max block size of 512 KiB, since that is what HP's own tape testing tool (hp_ltt) is using. -- Kjetil T. Homme Redpill Linpro AS - Changing the game -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
On Aug 21, 2012, at 12:53 PM, lst_ho...@kwsoft.de wrote: > BTW: Do you need some Hardware? We have a Overland LXB Powerloader > (DLT-8000) near the dustbin ;-) I am going to guess it is in Germany and I know I'm in USA. :) -- Dan Langille - http://langille.org -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
Zitat von Phil Stracchino : > On 08/21/12 13:10, Alan Brown wrote: >> On 21/08/12 17:50, Dan Langille wrote: >>> All my backups are in my basement. Including the tapes. I really >>> should move the latest full backups somewhere else. >> >> Preferably "sooner" rather than "later". >> >> One of our staff was burgled recently. The thieves took his personal >> laptop as well as the USB hard drives he'd been using for backups >> (on a shelf above the computer). He isn't a happy bunny. > > I don't currently have any offsite storage for my backups. My > contingency plan in case of a house fire includes grabbing the entire > stack of LTO4 tapes off the shelf over my desk if time permits. Most of > my contingency plans for burglary involve the burglar saying something > like "OH SHIT." I doubt burglar will grab for you tapes, i would be more worry about who is grabbing the tapes if you are asleep or on vacation. But maby you have a policy which does not permit fire during vacation or outside business hours ;-) Andreas -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
Zitat von Alan Brown : > On 21/08/12 15:40, Sven Tegethoff wrote: > >> At my previous company, we used to do a primary backup on a local backup >> server, and then copied it over a 100mbit WAN line to a secondary >> offsite server at a colocation center over the course of the day. > > If I was to do this, the 100Mb/s WAN would be saturated 24*7 - as it is > we average 200Mb/s and spike to 800Mb/s without putting backup traffic > over our WAN circuit. > > The scenario you describe is a small enough system that tape isn't > necessary. > >> It's especially attractive if you're in an >> environment that requires a lot of restores, because disk backups are >> instantly available when you need them. > > Backups are for disaster recovery. > > If you have "an environment that requires a lot of restores" (ie, ID10T > users who delete stuff they need) then you should look at > multigenerational filesystems which allow users to haul their own rear > ends out of the fire. > > OTOH, there's nothing like telling user that you can't restore their > critical document for a few days to make them pay attention to good > housekeeping practices (we have a policy of making users wait at least > 48 hours. This is fully backed by management because it encourages users > to take greater care of their data) :-) :-) :-) BOFH, it's you??? I will take it on the next meeting agenda. Andreas -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
Zitat von Alan Brown : > On 21/08/12 17:46, lst_ho...@kwsoft.de wrote: > >>> That means there's some room for improvement in despooling speed, >>> but the big bottleneck at the moment is disk-fd and fd->sd, not >>> sd->tape - the best achieved there is a sustained 52MB/s and that >>> virtually maxes out a 1Gb/s NIC. Even if the network block sizing is >>> optimized, I need to look at 10Gb NICs and for simultaneous >>> spool/despooling. >> >> We plan to takle this one with parallel running jobs doing spooling to >> SSD and despooling to tape. As far as i understand spooling/despooling >> can happen in parallel if using different jobs. > > I already do this (7 drives, 6 pools), but it still means individual > TB-class jobs take too long to run. Glad to here that we are not the first to try :-) We also will have this problem at some point because 2/3 of our data is on a big filer (~2TB). But maybe it will help to create more than one job for this machine. >>> WRT "offsite backups" - I'm more inclined to use a good firesafe in >>> another building than pass media to a 3rd party company. Far too >>> many people backup to tape but then don't take care of the media. >> >> Our's are in a Bank safe at the other end of the town and in the >> future they are also encrypted thanks to Bacula :-) > > In the present they can be encrypted too, for LTO4 and higher. > Hardware encryption is a lot faster than software. But the client side encryption has the advantage to secure the data already when traveling the network/leaving the machine. It could also prevent some malicous "restore on other machine" scenario. But admittetly this decreases overall speed, nothing comes for free. Regards and thanks for the insights Andreas -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
On 08/21/12 13:10, Alan Brown wrote: > On 21/08/12 17:50, Dan Langille wrote: >> All my backups are in my basement. Including the tapes. I really >> should move the latest full backups somewhere else. > > Preferably "sooner" rather than "later". > > One of our staff was burgled recently. The thieves took his personal > laptop as well as the USB hard drives he'd been using for backups > (on a shelf above the computer). He isn't a happy bunny. I don't currently have any offsite storage for my backups. My contingency plan in case of a house fire includes grabbing the entire stack of LTO4 tapes off the shelf over my desk if time permits. Most of my contingency plans for burglary involve the burglar saying something like "OH SHIT." -- Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355 ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org Renaissance Man, Unix ronin, Perl hacker, SQL wrangler, Free Stater It's not the years, it's the mileage. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
On 21/08/12 15:40, Sven Tegethoff wrote: > At my previous company, we used to do a primary backup on a local backup > server, and then copied it over a 100mbit WAN line to a secondary > offsite server at a colocation center over the course of the day. If I was to do this, the 100Mb/s WAN would be saturated 24*7 - as it is we average 200Mb/s and spike to 800Mb/s without putting backup traffic over our WAN circuit. The scenario you describe is a small enough system that tape isn't necessary. > It's especially attractive if you're in an > environment that requires a lot of restores, because disk backups are > instantly available when you need them. Backups are for disaster recovery. If you have "an environment that requires a lot of restores" (ie, ID10T users who delete stuff they need) then you should look at multigenerational filesystems which allow users to haul their own rear ends out of the fire. OTOH, there's nothing like telling user that you can't restore their critical document for a few days to make them pay attention to good housekeeping practices (we have a policy of making users wait at least 48 hours. This is fully backed by management because it encourages users to take greater care of their data) -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
On 21/08/12 17:50, Dan Langille wrote: >> Assuming your DLTs are DLT8000, 12 hours is about spot-on for the raw >> capacity and speed of the tape (40GB raw + 7GB/hour) > > In my case, DLT7000. DLT7k is only slightly slower >> WRT "offsite backups" - I'm more inclined to use a good firesafe in another >> building than pass media to a 3rd party company. Far too many people backup >> to tape but then don't take care of the media. > > All my backups are in my basement. Including the tapes. I really should move > the latest full backups somewhere else. Preferably "sooner" rather than "later". One of our staff was burgled recently. The thieves took his personal laptop as well as the USB hard drives he'd been using for backups (on a shelf above the computer). He isn't a happy bunny. AB -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
On 21/08/12 17:46, lst_ho...@kwsoft.de wrote: >> That means there's some room for improvement in despooling speed, >> but the big bottleneck at the moment is disk-fd and fd->sd, not >> sd->tape - the best achieved there is a sustained 52MB/s and that >> virtually maxes out a 1Gb/s NIC. Even if the network block sizing is >> optimized, I need to look at 10Gb NICs and for simultaneous >> spool/despooling. > > We plan to takle this one with parallel running jobs doing spooling to > SSD and despooling to tape. As far as i understand spooling/despooling > can happen in parallel if using different jobs. I already do this (7 drives, 6 pools), but it still means individual TB-class jobs take too long to run. >> WRT "offsite backups" - I'm more inclined to use a good firesafe in >> another building than pass media to a 3rd party company. Far too >> many people backup to tape but then don't take care of the media. > > Our's are in a Bank safe at the other end of the town and in the > future they are also encrypted thanks to Bacula :-) In the present they can be encrypted too, for LTO4 and higher. Hardware encryption is a lot faster than software. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
Zitat von Dan Langille : > On 2012-08-21 05:42, Alan Brown wrote: >> On 21/08/12 08:37, lst_ho...@kwsoft.de wrote: >>> Some short tests with the "Maximum Block Size" set, show transfer >>> speed to the LTO-4 of 87MBytes/sec with 256K and 98MBytes/sec with 1M >>> with Bacula encrypted data, so with this we are reaching the >>> theoretical uncompressed speed of the LTO-4 device by ~20%. It would >>> be really helpful if there is some developer to say if there is any >>> downside of large block sizes? >> >> I'm not a dev, however I use 2Mb block size (this is the largest bacula >> currently supports) > > I suspect I'm an edge case with tape. I'm using old technology and old > hardware. That's what I can afford on a home network. > > I wasn't sure what block size I'm using on my DLT-7000 tape library, but > I see it's variable > > $ sudo mt -f /dev/nsa0 status > Mode Density Blocksize bpi Compression > Current: 0x1b:DLTapeIV(35GB)variable 85937IDRC > -available modes- > 0:0x1b:DLTapeIV(35GB)variable 85937IDRC > 1:0x1b:DLTapeIV(35GB)variable 85937IDRC > 2:0x1b:DLTapeIV(35GB)variable 85937IDRC > 3:0x1b:DLTapeIV(35GB)variable 85937IDRC > - > Current Driver State: at rest. > - > File Number: 0 Record Number: 0Residual Count 0 > > All my tape jobs involve data on local HDD. That is, I backup to > disk first, then > copy to tape. The HDD and the tape are both connected to the same bacula-sd. > >> The only downside is that you _must_ mark all active tapes as "used" >> before changing block size. > > Setting them as USED is fine by me. I write my tapes until they are FULL. > Checking the recycling algorithm documentation[1], this will not affect > pruning. > >> Bacula will be able to read the old tapes, but it cannot handle maximum >> block size changing midway through a tape. > > I found one reference[2] after a bunch of searching, but I should look more: > > ### > The DLT drive default data block transfer size is 4KB (4096 bytes). > To achieve better performance, adjust block size to 32K bytes or > higher when using a fixed block device. > ### > > My full backups, roughly 90GB, take about 12 hours to copy to tape. Thus, > my only incentive to change blocksize is to avoid shoe-shine, which I do not > think is happening at present. That's around 2MByte/sec? Even DLT-7000 should do better, according to wikipedia it should be able to do around 5MByte/sec. BTW: Do you need some Hardware? We have a Overland LXB Powerloader (DLT-8000) near the dustbin ;-) Regards Andreas -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
On Aug 21, 2012, at 12:38 PM, Alan Brown wrote: > On 21/08/12 16:04, Dan Langille wrote: > >> ### >> The DLT drive default data block transfer size is 4KB (4096 bytes). >> To achieve better performance, adjust block size to 32K bytes or >> higher when using a fixed block device. >> ### > > What you want to know is how big it can go - and the only way to know that is > to experiment. > > Assuming your DLTs are DLT8000, 12 hours is about spot-on for the raw > capacity and speed of the tape (40GB raw + 7GB/hour) In my case, DLT7000. > I'm seeing unspool speeds of about 150MB/sec to LTO5 using 2MB block sizes > (that's max, smaller block sizes get used too), and tests show the spool > (raid0 4 * Intel 64GB SLCs) is capable of something like 800MB/s r/w with > several streams running (1400MB/s single sustained read to /dev/null) > > That means there's some room for improvement in despooling speed, but the big > bottleneck at the moment is disk-fd and fd->sd, not sd->tape - the best > achieved there is a sustained 52MB/s and that virtually maxes out a 1Gb/s > NIC. Even if the network block sizing is optimized, I need to look at 10Gb > NICs and for simultaneous spool/despooling. > > > WRT "offsite backups" - I'm more inclined to use a good firesafe in another > building than pass media to a 3rd party company. Far too many people backup > to tape but then don't take care of the media. All my backups are in my basement. Including the tapes. I really should move the latest full backups somewhere else. -- Dan Langille http://langille.org/ -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
Zitat von Alan Brown : > On 21/08/12 16:04, Dan Langille wrote: > >> ### >> The DLT drive default data block transfer size is 4KB (4096 bytes). >> To achieve better performance, adjust block size to 32K bytes or >> higher when using a fixed block device. >> ### > > What you want to know is how big it can go - and the only way to > know that is to experiment. > > Assuming your DLTs are DLT8000, 12 hours is about spot-on for the > raw capacity and speed of the tape (40GB raw + 7GB/hour) > > I'm seeing unspool speeds of about 150MB/sec to LTO5 using 2MB block > sizes (that's max, smaller block sizes get used too), and tests show > the spool (raid0 4 * Intel 64GB SLCs) is capable of something like > 800MB/s r/w with several streams running (1400MB/s single sustained > read to /dev/null) > > That means there's some room for improvement in despooling speed, > but the big bottleneck at the moment is disk-fd and fd->sd, not > sd->tape - the best achieved there is a sustained 52MB/s and that > virtually maxes out a 1Gb/s NIC. Even if the network block sizing is > optimized, I need to look at 10Gb NICs and for simultaneous > spool/despooling. We plan to takle this one with parallel running jobs doing spooling to SSD and despooling to tape. As far as i understand spooling/despooling can happen in parallel if using different jobs. > WRT "offsite backups" - I'm more inclined to use a good firesafe in > another building than pass media to a 3rd party company. Far too > many people backup to tape but then don't take care of the media. Our's are in a Bank safe at the other end of the town and in the future they are also encrypted thanks to Bacula :-) Regards Andreas -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
On 21/08/12 16:04, Dan Langille wrote: > ### > The DLT drive default data block transfer size is 4KB (4096 bytes). > To achieve better performance, adjust block size to 32K bytes or > higher when using a fixed block device. > ### What you want to know is how big it can go - and the only way to know that is to experiment. Assuming your DLTs are DLT8000, 12 hours is about spot-on for the raw capacity and speed of the tape (40GB raw + 7GB/hour) I'm seeing unspool speeds of about 150MB/sec to LTO5 using 2MB block sizes (that's max, smaller block sizes get used too), and tests show the spool (raid0 4 * Intel 64GB SLCs) is capable of something like 800MB/s r/w with several streams running (1400MB/s single sustained read to /dev/null) That means there's some room for improvement in despooling speed, but the big bottleneck at the moment is disk-fd and fd->sd, not sd->tape - the best achieved there is a sustained 52MB/s and that virtually maxes out a 1Gb/s NIC. Even if the network block sizing is optimized, I need to look at 10Gb NICs and for simultaneous spool/despooling. WRT "offsite backups" - I'm more inclined to use a good firesafe in another building than pass media to a 3rd party company. Far too many people backup to tape but then don't take care of the media. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
On 2012-08-21 05:42, Alan Brown wrote: > On 21/08/12 08:37, lst_ho...@kwsoft.de wrote: >> Some short tests with the "Maximum Block Size" set, show transfer >> speed to the LTO-4 of 87MBytes/sec with 256K and 98MBytes/sec with >> 1M >> with Bacula encrypted data, so with this we are reaching the >> theoretical uncompressed speed of the LTO-4 device by ~20%. It would >> be really helpful if there is some developer to say if there is any >> downside of large block sizes? > > I'm not a dev, however I use 2Mb block size (this is the largest > bacula > currently supports) I suspect I'm an edge case with tape. I'm using old technology and old hardware. That's what I can afford on a home network. I wasn't sure what block size I'm using on my DLT-7000 tape library, but I see it's variable $ sudo mt -f /dev/nsa0 status Mode Density Blocksize bpi Compression Current: 0x1b:DLTapeIV(35GB)variable 85937IDRC -available modes- 0:0x1b:DLTapeIV(35GB)variable 85937IDRC 1:0x1b:DLTapeIV(35GB)variable 85937IDRC 2:0x1b:DLTapeIV(35GB)variable 85937IDRC 3:0x1b:DLTapeIV(35GB)variable 85937IDRC - Current Driver State: at rest. - File Number: 0 Record Number: 0Residual Count 0 All my tape jobs involve data on local HDD. That is, I backup to disk first, then copy to tape. The HDD and the tape are both connected to the same bacula-sd. > The only downside is that you _must_ mark all active tapes as "used" > before changing block size. Setting them as USED is fine by me. I write my tapes until they are FULL. Checking the recycling algorithm documentation[1], this will not affect pruning. > Bacula will be able to read the old tapes, but it cannot handle > maximum > block size changing midway through a tape. I found one reference[2] after a bunch of searching, but I should look more: ### The DLT drive default data block transfer size is 4KB (4096 bytes). To achieve better performance, adjust block size to 32K bytes or higher when using a fixed block device. ### My full backups, roughly 90GB, take about 12 hours to copy to tape. Thus, my only incentive to change blocksize is to avoid shoe-shine, which I do not think is happening at present. [1] - http://www.bacula.org/5.2.x-manuals/en/main/main/Automatic_Volume_Recycling.html [2] - http://techpubs.sgi.com/library/dynaweb_docs/hdwr/SGI_Admin/books/DLT_OG/sgi_html/ch05.html -- Dan Langille - http://langille.org/ -- Dan Langille - http://langille.org/ -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
Zitat von Sven Tegethoff : > On 21.08.2012 16:07, lst_ho...@kwsoft.de wrote: >>> Not very many. The vast majority of bacula installations use disk >>> volumes, not tape - which is why debugging tape problems can take so >>> long. >> I always wonder how they handle offsite backup for disaster recovery... >> Maybe the plan is simply to go bankrupt in case of fire/water and so on ;-) > > At my previous company, we used to do a primary backup on a local backup > server, and then copied it over a 100mbit WAN line to a secondary > offsite server at a colocation center over the course of the day. It > still had risks that true offline backups don't have, but the > "fire/water" case was very well covered by that, and it required a lot > less manual "babysitting" than your typical tape backup. Tape backups > are very high-maintenance in comparison, and disks have become so fast > and cheap, it's often easier to build a couple of raids, and keep > multiple copies around that'll only need intervention when disks fail or > you run out of space. It's especially attractive if you're in an > environment that requires a lot of restores, because disk backups are > instantly available when you need them. > > Really, the times when tapes where THE one and only choice for backups > are long over. True, but you first need at least a 100mbit line to something some distance away from the building. We also use B2D for daily backups but have no choice for a cheap, high bandwidth offsite location where we can push some 3-4TB per week. Regards Andreas -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
On 21.08.2012 16:07, lst_ho...@kwsoft.de wrote: >> Not very many. The vast majority of bacula installations use disk >> volumes, not tape - which is why debugging tape problems can take so >> long. > I always wonder how they handle offsite backup for disaster recovery... > Maybe the plan is simply to go bankrupt in case of fire/water and so on ;-) At my previous company, we used to do a primary backup on a local backup server, and then copied it over a 100mbit WAN line to a secondary offsite server at a colocation center over the course of the day. It still had risks that true offline backups don't have, but the "fire/water" case was very well covered by that, and it required a lot less manual "babysitting" than your typical tape backup. Tape backups are very high-maintenance in comparison, and disks have become so fast and cheap, it's often easier to build a couple of raids, and keep multiple copies around that'll only need intervention when disks fail or you run out of space. It's especially attractive if you're in an environment that requires a lot of restores, because disk backups are instantly available when you need them. Really, the times when tapes where THE one and only choice for backups are long over. yours, -- Best Regards, Sven Tegethoff -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
Zitat von Alan Brown : > On 21/08/12 13:00, lst_ho...@kwsoft.de wrote: > >> Of course but the manual as far as i understand explicitely discourage >> altering the value for Block Size with modern hardware: "On most >> modern tape drives, you will not need to specify this directive.." > > That's on the basis that 64k is universal. Some older drives can't > even handle blocks this big. > >> With this in mind i really wonder how many Bacula installations setup >> the last 5 years are running with shoe-shining tape drives despite the >> fact that Bacula can do better. > > Not very many. The vast majority of bacula installations use disk > volumes, not tape - which is why debugging tape problems can take so > long. I always wonder how they handle offsite backup for disaster recovery... Maybe the plan is simply to go bankrupt in case of fire/water and so on ;-) Regards Andreas -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
On 21/08/12 13:00, lst_ho...@kwsoft.de wrote: > Of course but the manual as far as i understand explicitely discourage > altering the value for Block Size with modern hardware: "On most > modern tape drives, you will not need to specify this directive.." That's on the basis that 64k is universal. Some older drives can't even handle blocks this big. > With this in mind i really wonder how many Bacula installations setup > the last 5 years are running with shoe-shining tape drives despite the > fact that Bacula can do better. Not very many. The vast majority of bacula installations use disk volumes, not tape - which is why debugging tape problems can take so long. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
Zitat von Alan Brown : > On 21/08/12 12:27, lst_ho...@kwsoft.de wrote: > >> So i still wonder why the manual recommend not to change this value? > > Because some (older) drive types don't like it and will break. Of course but the manual as far as i understand explicitely discourage altering the value for Block Size with modern hardware: "On most modern tape drives, you will not need to specify this directive.." So it should be more of don't alter this value for old tape drives and don't alter the value after first tape usage or you will not be able to restore. >> The problem of not mixing the blocksize is obvious so if there is not >> other downside one should choose a larger value with any recent tape >> drive, no? > > It's only obvious to those experienced with tape. Inexperienced > admins are likely to end up with non-restorable tapes due to mixed > blocksizes on one volume and not realise what they've done until > it's far too late. Given the complexity of backup/restore and the knowledge required a big *warning* in the manual should be sufficient i guess... > LTO drives can handle blocksize up to 16Mb. I suspect that moving to > 4 or 8Mb would result in better throughput but that isn't an option > at the moment. > > Similarly, the 64kb network block size isn't really large enough for > Gb/10Gb-scale networks but in my experience setting larger values > causes problems inside bacula. > > Alan With this in mind i really wonder how many Bacula installations setup the last 5 years are running with shoe-shining tape drives despite the fact that Bacula can do better. Thanks for your help on this one Moving to next part of the puzzle ;-) Andreas -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
On 21/08/12 12:27, lst_ho...@kwsoft.de wrote: > So i still wonder why the manual recommend not to change this value? Because some (older) drive types don't like it and will break. > The problem of not mixing the blocksize is obvious so if there is not > other downside one should choose a larger value with any recent tape > drive, no? It's only obvious to those experienced with tape. Inexperienced admins are likely to end up with non-restorable tapes due to mixed blocksizes on one volume and not realise what they've done until it's far too late. LTO drives can handle blocksize up to 16Mb. I suspect that moving to 4 or 8Mb would result in better throughput but that isn't an option at the moment. Similarly, the 64kb network block size isn't really large enough for Gb/10Gb-scale networks but in my experience setting larger values causes problems inside bacula. Alan -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
Zitat von Alan Brown : > On 21/08/12 08:37, lst_ho...@kwsoft.de wrote: >> Some short tests with the "Maximum Block Size" set, show transfer >> speed to the LTO-4 of 87MBytes/sec with 256K and 98MBytes/sec with >> 1M with Bacula encrypted data, so with this we are reaching the >> theoretical uncompressed speed of the LTO-4 device by ~20%. It >> would be really helpful if there is some developer to say if there >> is any downside of large block sizes? > > I'm not a dev, however I use 2Mb block size (this is the largest > bacula currently supports) > > The only downside is that you _must_ mark all active tapes as "used" > before changing block size. > > Bacula will be able to read the old tapes, but it cannot handle > maximum block size changing midway through a tape. So i still wonder why the manual recommend not to change this value? The problem of not mixing the blocksize is obvious so if there is not other downside one should choose a larger value with any recent tape drive, no? Regards Andreas -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
On 21/08/12 08:37, lst_ho...@kwsoft.de wrote: > Some short tests with the "Maximum Block Size" set, show transfer > speed to the LTO-4 of 87MBytes/sec with 256K and 98MBytes/sec with 1M > with Bacula encrypted data, so with this we are reaching the > theoretical uncompressed speed of the LTO-4 device by ~20%. It would > be really helpful if there is some developer to say if there is any > downside of large block sizes? I'm not a dev, however I use 2Mb block size (this is the largest bacula currently supports) The only downside is that you _must_ mark all active tapes as "used" before changing block size. Bacula will be able to read the old tapes, but it cannot handle maximum block size changing midway through a tape. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
Zitat von lst_ho...@kwsoft.de: > Zitat von Nils Juergens : > >> Hello Andreas, >> >> Am 20.08.2012 17:29, schrieb lst_ho...@kwsoft.de:> As the tape is >> altering speed all the time with Bacula i would like to >>> know if there is a possibility to get closer to the dd values and what >>> could be the bottleneck in such a setup? >> >> for me (with LTO-3) increasing Maximum block size has had quite the >> impact on performance. >> >> Sadly, with the new setting I had trouble reading my tapes so switched >> back to the old configuration. >> >> I suspect you may get around the problem if you re-label all your tapes >> after switching the block size, however I have not testet this. Maybe >> someone can shed some light on the whole block size business? :-) > > Would really nice indeed. As far as i understand the block size used > as default (and max.) if nothing is specified is 64,512 Bytes. This > looks a little bit low for high-speed tape and i'm not sure what the > downside of higher values could be or why the manual state that there > is no need to set it explicit (higher). > As i'm in the testing phase i will have a look if higher values lead > to any trouble. Anyone aware why the default of 126*512 Bytes was > choosen? > Some short tests with the "Maximum Block Size" set, show transfer speed to the LTO-4 of 87MBytes/sec with 256K and 98MBytes/sec with 1M with Bacula encrypted data, so with this we are reaching the theoretical uncompressed speed of the LTO-4 device by ~20%. It would be really helpful if there is some developer to say if there is any downside of large block sizes? Regards Andreas -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
Zitat von Alan Brown : > On 20/08/12 16:29, lst_ho...@kwsoft.de wrote: >> Hello List >> >> this has been discussed already some times on the list but as i have >> ssen no real conclusion i will ask for opinion again. The setup is >> Bacula 5.2.10 on Ubuntu 12.04 LTS (x64) with a LTO-4 drive connected >> through Adaptec 29320LPE, a 320MByte/sec PCI-Express controller. To >> prevent the LTO-4 tape from shining/whining around we use the Data >> spool feature on a SSD drive (Intel Postville Refresh 120GB) with >> "Maximum Spool Size = 20GB" and "Maximum File Size = 4GB". This works >> as expected beside the fact that the transfer spped max. out at >> ~84MBytes/sec with encrypted data (hard to further compress) and >> 86MBytes/sec without encryption when using Bacula. To test if the >> hardware is to blame i have done some simple dd to tape like this >> >> "dd if=/some/file of=/dev/nst0 bs=xxK" > > Test the speed of writing to/from disk as well as your tape speeds. > > SSDs have high IOPS/seek rates but they're not always as fast for > sequential operations as you might expect. That's why i have done the "dd" tests from the same "disk" which yield the 140MBytes/sec. Other tests with hdparm/bonnie and simple cp yield to something in the line of 260MBytes/sec which looks like the Sata-2 transfer limit. So saturating the tape speed should be possible, no? Regards Andreas -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
Zitat von Nils Juergens : > Hello Andreas, > > Am 20.08.2012 17:29, schrieb lst_ho...@kwsoft.de:> As the tape is > altering speed all the time with Bacula i would like to >> know if there is a possibility to get closer to the dd values and what >> could be the bottleneck in such a setup? > > for me (with LTO-3) increasing Maximum block size has had quite the > impact on performance. > > Sadly, with the new setting I had trouble reading my tapes so switched > back to the old configuration. > > I suspect you may get around the problem if you re-label all your tapes > after switching the block size, however I have not testet this. Maybe > someone can shed some light on the whole block size business? :-) Would really nice indeed. As far as i understand the block size used as default (and max.) if nothing is specified is 64,512 Bytes. This looks a little bit low for high-speed tape and i'm not sure what the downside of higher values could be or why the manual state that there is no need to set it explicit (higher). As i'm in the testing phase i will have a look if higher values lead to any trouble. Anyone aware why the default of 126*512 Bytes was choosen? Regards Andreas -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
On 20/08/12 16:29, lst_ho...@kwsoft.de wrote: > Hello List > > this has been discussed already some times on the list but as i have > ssen no real conclusion i will ask for opinion again. The setup is > Bacula 5.2.10 on Ubuntu 12.04 LTS (x64) with a LTO-4 drive connected > through Adaptec 29320LPE, a 320MByte/sec PCI-Express controller. To > prevent the LTO-4 tape from shining/whining around we use the Data > spool feature on a SSD drive (Intel Postville Refresh 120GB) with > "Maximum Spool Size = 20GB" and "Maximum File Size = 4GB". This works > as expected beside the fact that the transfer spped max. out at > ~84MBytes/sec with encrypted data (hard to further compress) and > 86MBytes/sec without encryption when using Bacula. To test if the > hardware is to blame i have done some simple dd to tape like this > > "dd if=/some/file of=/dev/nst0 bs=xxK" Test the speed of writing to/from disk as well as your tape speeds. SSDs have high IOPS/seek rates but they're not always as fast for sequential operations as you might expect. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Linux and the LTO-4 tape speed
Hello Andreas, Am 20.08.2012 17:29, schrieb lst_ho...@kwsoft.de:> As the tape is altering speed all the time with Bacula i would like to > know if there is a possibility to get closer to the dd values and what > could be the bottleneck in such a setup? for me (with LTO-3) increasing Maximum block size has had quite the impact on performance. Sadly, with the new setting I had trouble reading my tapes so switched back to the old configuration. I suspect you may get around the problem if you re-label all your tapes after switching the block size, however I have not testet this. Maybe someone can shed some light on the whole block size business? :-) kind regards, Nils -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Bacula Linux and the LTO-4 tape speed
Hello List this has been discussed already some times on the list but as i have ssen no real conclusion i will ask for opinion again. The setup is Bacula 5.2.10 on Ubuntu 12.04 LTS (x64) with a LTO-4 drive connected through Adaptec 29320LPE, a 320MByte/sec PCI-Express controller. To prevent the LTO-4 tape from shining/whining around we use the Data spool feature on a SSD drive (Intel Postville Refresh 120GB) with "Maximum Spool Size = 20GB" and "Maximum File Size = 4GB". This works as expected beside the fact that the transfer spped max. out at ~84MBytes/sec with encrypted data (hard to further compress) and 86MBytes/sec without encryption when using Bacula. To test if the hardware is to blame i have done some simple dd to tape like this "dd if=/some/file of=/dev/nst0 bs=xxK" which lead to ~110MBytes/sec with bs=64K and ~140MBytes/sec with bs=1M As the tape is altering speed all the time with Bacula i would like to know if there is a possibility to get closer to the dd values and what could be the bottleneck in such a setup? It would also be helpful if someone has compareable results for LTO-4 with Bacula to see if we are on the right track. Many Thanks Andreas -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
