Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Simon Templar]

In my case using spooling didn’t prevent shoe-shining; it just introduced long 
pauses while data was spooled. I think all this means is that I can read from 
my data sources faster than my tape can write.

So far the only change I made to help with shoe-shining was to set Max File 
Size to a large number (mine is now set to 20g, after first trying 3gb then 
5gb). This one change alone is probably responsible for most of the performance 
increase that I’ve been able to achieve thus far. I’d like to test and tune 
more, but I’m still wrestling with things like tape mount timeouts (no 3rd 
shift operators), job run time timeouts, etc…

-Simon


> On Mar 10, 2016, at 11:09 PM, Kern Sibbald  wrote:
> 
> I have not tried this, but one thing that may help a lot is to turn on 
> data spooling for the tape device. This will probably not speed up the 
> process but should prevent that tape shoe-shine (start and stopping).

--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785111=/4140___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Simon Templar]

Dan,

I’m new to bacula, and arguably not very smart, but I’ve been struggling with 
tape drive performance pretty much since the moment I got the configurations to 
a functional state so I’ll share my learnings thus far.

Can you hear your tape drive? If so, do you hear lots of stops and starts when 
bacula is running the job? When mine was writing I’d hear lots of stops and 
starts (shoe-shining, I believe it’s called) followed by long pauses of silence.

I have a single LTO6 drive, and so far I’ve made 2 changes to my setup that 
brought my average write speed from less than 60 MB/s to between 130-160 MB/s.

First, the long pauses with no activity were found to be related to spooling. I 
started by spooling to an idle set of 6 sata drives in a raidz, then later 
switched to an ssd, and with either the behavior was that the tape drive would 
be completely idle while the data was spooled, then the data was written to the 
drive, then when it all was written the tape drive was completely idle again 
while data was spooled. Needless to say this added a long time to the backups 
for our platform. Our data ultimately resides on another platform (a freenas) 
and the goal is to replicate it to our backup server, then write to tape for 
archival purposes. It turns out that I can write the 15tb directly from the nfs 
mount to tape faster than I can spool it from local disk to ssd then write to 
tape. I’m sure something’s wrong with my setup, but if you’re spooling it’s 
worth taking a look to see if you have long tape drive idle times while the 
data is spooled. I would have guessed that the spool directory should be kept 
as full as possible WHILE the spooled data is written to tape, but that’s not 
the behavior I’m seeing.

The second thing I did was add:
Maximum File Size = 20GB

to my bacula-sd.conf. I probably need to tune this further, and it might make 
restores slower (at least for individual files?) but it drastically cut down on 
my drive starts and stops. Together, the maximum file size and elimination of 
spooling more than doubled my average write speed in large backups.

-Simon

P.S. Is your tar write speed as fast as you’d expect for your LTO drive?


> On Mar 9, 2016, at 6:14 PM, Dan Langille  wrote:
> 
> I have a copy to tape job which copies from disk to tape using Bacula 7.4.0 
> and PostgreSQL 9.4 on FreeBSD 10.2
> 
> Everything is within one SD
> 
> Full details at https://gist.github.com/dlangille/2341a6da8f9ee836270c 
> 
> 
> The job summary:
> 
>  Start time: 09-Mar-2016 19:41:54
>  End time:   09-Mar-2016 20:51:02
>  Elapsed time:   1 hour 9 mins 8 secs
>  Priority:   410
>  SD Files Written:   1
>  SD Bytes Written:   52,897,660,928 (52.89 GB)
>  Rate:   12752.6 KB/s
> 
> If I tar the volumes directly to tape, it takes only 16 minutes.
> 
> 
> $ time sudo tar -cf /dev/nsa1 IncrAuto-4525 IncrAuto-4320 IncrAuto-4324 
> IncrAuto-4321 \
> > IncrAuto-4055 IncrAuto-4322 IncrAuto-4319 IncrAuto-3969 IncrAuto-3972 \
> > IncrAuto-3973 IncrAuto-3971 IncrAuto-4058
> 
> real15m47.508s
> user0m5.844s
> sys 1m51.834s
> 
> Spooling attributes is trivial:
> 
> 09-Mar 20:51 crey-sd JobId 232945: Sending spooled attrs to the Director. 
> Despooling 321 bytes ...
> 09-Mar 20:51 bacula-dir JobId 232944: Bacula bacula-dir 7.4.0 (16Jan16):
> 
> I am not sure where to look to figure this out.
> 
> Full job output:
> 
> 
> 09-Mar 19:41 bacula-dir JobId 232944: Warning: FileSet MD5 digest not found.
> 09-Mar 19:41 bacula-dir JobId 232944: The following 1 JobId was chosen to be 
> copied: 232778
> 09-Mar 19:41 bacula-dir JobId 232944: Copying using JobId=232778 
> Job=BackupCatalog.2016-03-08_03.05.16_52
> 09-Mar 19:41 bacula-dir JobId 232944: Bootstrap records written to 
> /usr/local/bacula/working/bacula-dir.restore.107.bsr
> 09-Mar 19:41 bacula-dir JobId 232944: Start Copying JobId 232944, 
> Job=CopyToTape-Full-Just-One-tape-01b.2016-03-09_19.41.51_42
> 09-Mar 19:41 bacula-dir JobId 232944: Using Device "vDrive-0" to read.
> 09-Mar 19:41 bacula-dir JobId 232945: Using Device "LTO_0" to write.
> 09-Mar 19:41 crey-sd JobId 232944: Ready to read from volume "IncrAuto-4525" 
> on file device "vDrive-0" (/usr/local/bacula/volumes).
> 09-Mar 19:41 crey-sd JobId 232944: Forward spacing Volume "IncrAuto-4525" to 
> file:block 0:1207281748.
> 09-Mar 19:47 crey-sd JobId 232944: End of Volume at file 1 on device 
> "vDrive-0" (/usr/local/bacula/volumes), Volume "IncrAuto-4525"
> 09-Mar 19:47 crey-sd JobId 232944: Ready to read from volume "IncrAuto-4320" 
> on file device "vDrive-0" (/usr/local/bacula/volumes).
> 09-Mar 19:47 crey-sd JobId 232944: Forward spacing Volume "IncrAuto-4320" to 
> file:block 0:3844390440.
> 09-Mar 19:49 crey-sd JobId 232944: End of Volume at file 1 on device 
> "vDrive-0" (/usr/local/bacula/volumes), Volume "IncrAuto-4320"
> 09-Mar 19:49 

Re: [Bacula-users] Trouble setting up Bacula on FreeBSD

[Simon Templar]

ape btape fill test. 
Which would always fail. I was going nuts trying to get it to work and wasting 
many hours per iteration until I read the following sentence on the 
aforementioned page:

Please use only the simple single tape option because the multiple tape option 
still doesn't work totally correctly

After learning from the output asked for by Cejka Rudolf and others I landed on 
a configuration that looked okay, passed a single tape btape fill test, and I 
then ran a backup. And it mostly worked.

I say mostly because I suffered a lot of shoe-shining, so I’ll probably start a 
new thread with some spooling questions, but right now I’m reading directly 
from an nfs mount without spooling and writing to the tape at an average of 
130MB/s without undue tape stops and starts, which is okay for now.

Thanks again,
Simon

P.S. My current bacula-sd.conf file:

#

Storage {
  Name = UralTape
  WorkingDirectory = "/usr/local/bacula/working"
  Pid Directory= "/var/run"
}
 
Director {
  Name = Director-Ural
  Password = “#"
}
 
Device {
  Name   = QuantumLTO
  Media Type = LTO6
  Device Type = Tape
  ArchiveDevice = /dev/nsa0
  Description = "LTO-6 for FreeBSD"
  LabelMedia = yes
  AutomaticMount = yes;   # when device opened, read it
  AlwaysOpen = yes
  RemovableMedia = yes
  Offline On Unmount = no
  Hardware End of Medium = no   # Noted as FreeBSD specific in Problems 
section of manual
  BSF at EOM = yes  # Support List Recommendation
  Backward Space Record = yes   # Support List Recommendation
  Fast Forward Space File = yes # Noted as FreeBSD specific in Problems 
section of manual
  TWO EOF = yes # Support List Recommendation
#Minimum Block Size = 64512
#Maximum Block Size = 64512
Maximum File Size = 20GB
# for data spooling, which even using SSD causes slowdown
# tape stops completely while filling SSD, then writes fast to tape, then stops 
while data spools
# average is about half of using direct read from nfs mount
#Maximum Spool Size = 75gb
#Maximum Job Spool Size = 70gb
#Spool Directory = /usr/local/bacula/spool
#  If you have smartctl, enable this, it has more info than tapeinfo
#  Alert Command = "sh -c 'smartctl -H -l error %c'"


}
 
Messages {
  Name = Standard
  director = Director-Ural = all
}






> On Mar 1, 2016, at 6:34 PM, Cejka Rudolf <cej...@fit.vutbr.cz> wrote:
> 
> Simon Templar wrote (2016/03/01):
>> Perhaps I should have included this information in the first email,
>> but I???ve tried several combinations of parameters in the
>> bacula-sd.conf, and it failed each time. Here are the parts I???ve changed:
> 
> And perhaps include latest output of btape -c... /dev/nsa0 "test" command.
> 
> What is your FreeBSD version?
> 
> What is the output of
> 
> mt -f /dev/nsa0 status -v
> 
> mt -f /dev/nsa0 geteotmodel
> 
> mt -f /dev/nsa0 param -l
> 
> ?
> 
> -- 
> Rudolf Cejka  http://www.fit.vutbr.cz/~cejkar
> Brno University of Technology, Faculty of Information Technology
> Bozetechova 2, 612 66  Brno, Czech Republic

--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Trouble setting up Bacula on FreeBSD

[Simon Templar]

Hello, and thank you all for your suggestions.

Just to make sure my hardware is still working I installed gtar and dumped 
about 10tb to tape, and it completed without errors.

Perhaps I should have included this information in the first email, but I’ve 
tried several combinations of parameters in the bacula-sd.conf, and it failed 
each time. Here are the parts I’ve changed:

First try:
Offline On Unmount = no
Hardware End of Medium = no
BSF at EOM = yes
Backward Space Record = no
Fast Forward Space File = no
TWO EOF = yes

Second Try:
 Device {
  Name   = QuantumLTO
  Media Type = LTO6
  Device Type = Tape
  ArchiveDevice = /dev/nsa0
  LabelMedia = yes
#  Random Access  = yes
  Requires Mount = no
  AutomaticMount = yes
  RemovableMedia = yes
  Always Open = yes
  Offline On Unmount = no
  Hardware End of Medium = yes
  BSF at EOM = yes
  Backward Space Record = yes
  Fast Forward Space File = yes
  TWO EOF = yes

Third Try:
Device {
  Name   = QuantumLTO
  Media Type = LTO6
  Device Type = Tape
  ArchiveDevice = /dev/nsa0
  Description = "LTO-6 for FreeBSD"
  LabelMedia = yes
  AutomaticMount = yes;   # when device opened, read it
  AlwaysOpen = yes
  RemovableMedia = yes
  Offline On Unmount = no
  Hardware End of Medium = no
  BSF at EOM = yes
  Backward Space Record = no
  Fast Forward Space File = no
  TWO EOF = yes

Fourth Try:
Device {
  Name   = QuantumLTO
  Media Type = LTO6
  Device Type = Tape
  ArchiveDevice = /dev/nsa0
  Description = "LTO-6 for FreeBSD"
  LabelMedia = yes
  AutomaticMount = yes;   # when device opened, read it
  AlwaysOpen = yes
  RemovableMedia = yes
  Offline On Unmount = no
  Hardware End of Medium = no   # Noted as FreeBSD specific
  BSF at EOM = no   # Noted as FreeBSD specific
  Backward Space Record = no# Noted as FreeBSD specific
  Fast Forward Space File = yes # Noted as FreeBSD specific
  TWO EOF = no  # Noted as FreeBSD specific

And today, as I type this I’m trying a btape fill test with:
Device {
  Name   = QuantumLTO
  Media Type = LTO6
  Device Type = Tape
  ArchiveDevice = /dev/nsa0
  Description = "LTO-6 for FreeBSD"
  LabelMedia = yes
  AutomaticMount = yes;   # when device opened, read it
  AlwaysOpen = yes
  RemovableMedia = yes
  Offline On Unmount = no
  Hardware End of Medium = no   # Noted as FreeBSD specific
  BSF at EOM = yes  # Support List Recommendation
  Backward Space Record = yes   # Support List Recommendation
  Fast Forward Space File = yes # Noted as FreeBSD specific
  TWO EOF = yes # Support List Recommendation
Minimum Block Size = 64512
Maximum Block Size = 64512

Which is similar to my third try, with the exceptions of Backward Space Record, 
Fast Forward Space File, and also the addition of the Min and Max Block Size 
directives.

So at this point my question is this: is there a smart way to identify the 
correct entries for my hardware? I’ve re-read the docs from quantum but clearly 
I’m still flailing. The trial-and-error approach wouldn’t be so bad if it 
didn’t take 11 hours to write 2.5tb, then a short while to begin the second 
tape, then begin the unfill portion of the test, then eventually fail. It’s 
basically a full calendar day with each config file guess iteration. 

As an aside, gtar wrote to tape at almost exactly twice the speed that I’m 
seeing in the btape test. My disk arrays are showing 300+mb/s for random reads, 
so they’re not the bottleneck. Almost immediately after I get this multi-tape 
dump and restore working in bacula I’ll be needing to address the horrible 
speeds I’m seeing. I do notice a lot of  “shoe shining” while running btape 
fill, and nothing is being written to the spool directory (which is ssd) so 
maybe the real backups will be faster. We’ll see.

Thanks again,
Simon


> On Mar 1, 2016, at 8:42 AM, Martin Simmons <mar...@lispworks.com> wrote:
> 
>>>>>> On Tue, 1 Mar 2016 14:07:55 +0100, Cejka Rudolf said:
>> 
>> Simon Templar wrote (2016/02/29):
>>> I've set up bacula 7.2 on my FreeBSD server using Postgres as the 
>>> backend database.
>> 
>> Which FreeBSD version? According to the mt status it seems to be
>> sufficiently fresh.
>> 
>>> Device {
>>> ...
>>>   Hardware End of Medium = no   # Noted as FreeBSD specific
>>>   BSF at EOM = no   # Noted as FreeBSD specific
>>>   Backward Space Record = no# Noted as FreeBSD specific
>>>   Fast Forward Space File = yes # Noted as FreeBSD specific
>>>   TWO EOF = no  # Noted as FreeBSD specific
>> 
>> Where did you get this?
>> 
>> Try this:
>>

[Bacula-users] Trouble setting up Bacula on FreeBSD

[Simon Templar]

Or perhaps a more accurate subject would be "Trouble successfully using 
Bacula on FreeBSD"...

I've set up bacula 7.2 on my FreeBSD server using Postgres as the 
backend database.

My tape drive info:
mt -f /dev/nsa0 status
Drive: sa0:  Serial Number: ##
-
Mode  Density  Blocksize  bpi  Compression
Current:  0x5a:LTO-6   variable   384607   enabled (0x1)
-
Current Driver State: at rest.
-
Partition:   0  Calc File Number:   0 Calc Record Number: 0
Residual:0  Reported File Number:   0 Reported Record Number: 0
Flags: BOP

I've iterated through the config files until they seemed to be correct 
(though I admit I'm over my head with the bacula-sd.conf specifics, and 
I've tried and failed with examples that I've found for LTO drives and 
also with FreeBSD-specific options), then dumped and recreated the 
tables and deleted the old .state files so I had a "clean slate". I ran 
btape test which was successful, but when I run btape fill it 
successfully writes to both tapes, then fails on the read/unfill portion 
with this output:

Reading the first 1 records from 0:0.
1 records read now at 1:5084
Reposition from 1:5084 to 2501:6988
Reading block 6988.

The blocks differ at byte 0

 The last block written and the block
that was read back differ. The test FAILED 
This must be corrected before you use Bacula
to write multi-tape Volumes.
btape: btape.c:2585-0 Autochanger returned: 0
Mount second tape. Press enter when ready:
btape: btape.c:2588-0
29-Feb 09:46 btape JobId 0: Ready to read from volume "TestVolume2" on 
tape device "QuantumLTO" (/dev/nsa0).
Reposition from 0:0 to 0:1
Reading block 1.

The first block on the second tape matches.

Reposition from 0:2 to 0:1001
Reading block 1001.

The blocks differ at byte 0

 The last block written and the block
that was read back differ. The test FAILED 
This must be corrected before you use Bacula
to write multi-tape Volumes.
Bacula interrupted by signal 11: Segmentation violation
Kaboom! btape, btape got signal 11 - Segmentation violation at 
29-Feb-2016 09:46:33. Attempting traceback.
Kaboom! exepath=/root
Calling: /root/btraceback /root/btape 1377 /tmp
It looks like the traceback worked...
Dumping: /tmp/btape.1377.lockdump
btape: lockmgr.c:1158-0 lockmgr disabled
Segmentation fault (core dumped)

My current bacula-sd.conf file:

Storage {
   Name = UralTape
   WorkingDirectory = "/usr/local/bacula/working"
   Pid Directory= "/var/run"
}

Director {
   Name = Director-Ural
   Password = "#"
}

Device {
   Name   = QuantumLTO
   Media Type = LTO6
   Device Type = Tape
   ArchiveDevice = /dev/nsa0
   Description = "LTO-6 for FreeBSD"
   LabelMedia = yes
   AutomaticMount = yes;   # when device opened, read it
   AlwaysOpen = yes
   RemovableMedia = yes
   Offline On Unmount = no
   Hardware End of Medium = no   # Noted as FreeBSD specific
   BSF at EOM = no   # Noted as FreeBSD specific
   Backward Space Record = no# Noted as FreeBSD specific
   Fast Forward Space File = yes # Noted as FreeBSD specific
   TWO EOF = no  # Noted as FreeBSD specific
# for data spooling
Maximum Spool Size = 75gb
Maximum Job Spool Size = 75gb
Spool Directory = /usr/local/bacula/spool
#  If you have smartctl, enable this, it has more info than tapeinfo
#  Alert Command = "sh -c 'smartctl -H -l error %c'"


}

Messages {
   Name = Standard
   director = Director-Ural = all
}

I saved each version of my bacula-sd.conf file and can share them, but I 
think the primary (if not only?) differences are the lines with the 
comments about being FreeBSD-specific.

In the past I've used this same platform while running centos to create 
multi-tape tars of xfs filesystems, and also used ltfs without issue. It 
was recently that we moved to FreeBSD and this tape backup solution is 
the last component I'm wrestling with.

Any suggestions?

Thanks,
Simon






--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users