Re: [Bacula-users] Trouble setting up Bacula on FreeBSD

2016-03-04 Thread Simon Templar 
It’s been an interesting few days with my bacula system.

Thanks for all the help and the suggestions; I think I’m over the major hurdle 
of having a non-functioning system. I’ll get to the current config, but first 
let me answer the questions that were asked:

My freebsd version is 10.2-RELEASE-p12

mt -f /dev/nsa0 geteotmodel
/dev/nsa0: the model is 2 filemarks at EOT

mt -f /dev/nsa0 status -v
Drive: sa0:  Serial Number: HU1305UGMF
-
Mode  Density  Blocksize  bpi  Compression
Current:  0x5a:LTO-6   variable   384607   enabled (0x1)
-
Current Driver State: at rest.
-
Partition:   0  Calc File Number: 1937 Calc Record Number: 0
Residual:0  Reported File Number: 1937 Reported Record Number: 38720002
Flags: EOP
-
Tape I/O parameters:
 Maximum I/O size allowed by driver and controller (maxio): 131072 bytes
 Maximum I/O size reported by controller (cpi_maxio): 262144 bytes
 Maximum block size supported by tape drive and media (max_blk): 16777215 bytes
 Minimum block size supported by tape drive and media (min_blk): 1 bytes
 Block granularity supported by tape drive and media (blk_gran): 0 bytes
 Maximum possible I/O size (max_effective_iosize): 131072 bytes

mt -f /dev/nsa0 param -l
sili: 0 (Suppress an error on underlength variable reads)
eot_warn: 0 (Return an error to warn that end of tape is approaching)
protection.protection_supported: 1 (Set to 1 if protection information is 
supported)
protection.prot_method: 0 (Current Protection Method)
protection.pi_length: 0 (Length of Protection Information)
protection.lbp_w: 0 (Check Protection on Writes)
protection.lbp_r: 0 (Check and Include Protection on Reads)
protection.rbdp: 0 (Transfer Protection Information for RECOVER BUFFERED DATA 
command)

Output of btape fill command:

01-Mar 19:39 btape JobId 0: End of Volume "TestVolume1" at 1937:1 on device 
"QuantumLTO" (/dev/nsa0). Write of 64512 bytes got 0.
01-Mar 19:39 btape JobId 0: Re-read of last block succeeded.
btape: btape.c:2713-0 Last block at: 1936:1 this_dev_block_num=1
btape: btape.c:2748-0 End of tape 1937:0. Volume Bytes=2,497,779,809,280. Write 
rate = 79.01 MB/s
01-Mar 19:39 btape JobId 0: End of medium on Volume "TestVolume1" 
Bytes=2,497,779,809,280 Blocks=38,718,065 at 01-Mar-2016 19:39.
btape: btape.c:3074-0 Autochanger returned: 0
Mount blank Volume on device "QuantumLTO" (/dev/nsa0) and press return when 
ready:
btape: btape.c:3079-0
Wrote Volume label for volume "TestVolume2".
01-Mar 20:14 btape JobId 0: Wrote label to prelabeled Volume "TestVolume2" on 
tape device "QuantumLTO" (/dev/nsa0)
01-Mar 20:14 btape JobId 0: New volume "TestVolume2" mounted on device 
"QuantumLTO" (/dev/nsa0) at 01-Mar-2016 20:14.
btape: btape.c:2316-0 Wrote 1000 blocks on second tape. Done.
Done writing 0 records ...
Wrote End of Session label.
btape: btape.c:2385-0 Wrote state file last_block_num1=1 
last_block_num2=1001
btape: btape.c:2403-0

20:14:44 Done filling tapes at 0:1003. Now beginning re-read of first tape ...
btape: btape.c:2481-0 Enter do_unfill
btape: btape.c:2519-0 Autochanger returned: 0
Mount first tape. Press enter when ready:
btape: btape.c:2522-0
01-Mar 20:16 btape JobId 0: Ready to read from volume "TestVolume1" on tape 
device "QuantumLTO" (/dev/nsa0).
Rewinding.
Reading the first 1 records from 0:0.
1 records read now at 1:5084
Reposition from 1:5084 to 1936:1
Reading block 1.
Error reading block: ERR=block.c:484 Read zero bytes Vol=TestVolume1 at -1 on 
device "QuantumLTO" (/dev/nsa0).

Bacula interrupted by signal 11: Segmentation violation
Kaboom! btape, btape got signal 11 - Segmentation violation at 01-Mar-2016 
20:18:23. Attempting traceback.
Kaboom! exepath=/colossus
Calling: /colossus/btraceback /colossus/btape 15075 /tmp
It looks like the traceback worked...
Dumping: /tmp/btape.15075.lockdump
btape: lockmgr.c:1158-0 lockmgr disabled
Segmentation fault (core dumped)

dmesg output:

(sa0:mps0:0:0:0): 10240-byte tape record bigger than supplied buffer
(sa0:mps0:0:0:0): 64512-byte tape record bigger than supplied buffer
pid 17958 (btape), uid 0: exited on signal 11 (core dumped)
pid 15075 (btape), uid 0: exited on signal 11 (core dumped)


Also asked along the way was where I got the suggestions for my configuration 
attempts, and one of the pages I used for reference was:
http://www.bacula.org/5.2.x-manuals/en/problems/problems/Testing_Your_Tape_Drive.html
 


I think I started with a configuration (an incorrect one) and after a btape 
test succeeded I tried a backup, which failed. After that failure I’d change 
the bacula-sd.conf, re-run btape test, then run a multi-tape btape fill test. 
Which would always fail. I was going nuts trying to get it to work and wasting 
many hours per iteration

Re: [Bacula-users] Trouble setting up Bacula on FreeBSD

2016-03-01 Thread Cejka Rudolf 
Simon Templar wrote (2016/03/01):
> Perhaps I should have included this information in the first email,
> but I???ve tried several combinations of parameters in the
> bacula-sd.conf, and it failed each time. Here are the parts I???ve changed:

And perhaps include latest output of btape -c... /dev/nsa0 "test" command.

What is your FreeBSD version?

What is the output of

mt -f /dev/nsa0 status -v

mt -f /dev/nsa0 geteotmodel

mt -f /dev/nsa0 param -l

?

-- 
Rudolf Cejka  http://www.fit.vutbr.cz/~cejkar
Brno University of Technology, Faculty of Information Technology
Bozetechova 2, 612 66  Brno, Czech Republic

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Trouble setting up Bacula on FreeBSD

2016-03-01 Thread Simon Templar 
Hello, and thank you all for your suggestions.

Just to make sure my hardware is still working I installed gtar and dumped 
about 10tb to tape, and it completed without errors.

Perhaps I should have included this information in the first email, but I’ve 
tried several combinations of parameters in the bacula-sd.conf, and it failed 
each time. Here are the parts I’ve changed:

First try:
Offline On Unmount = no
Hardware End of Medium = no
BSF at EOM = yes
Backward Space Record = no
Fast Forward Space File = no
TWO EOF = yes

Second Try:
 Device {
  Name   = QuantumLTO
  Media Type = LTO6
  Device Type = Tape
  ArchiveDevice = /dev/nsa0
  LabelMedia = yes
#  Random Access  = yes
  Requires Mount = no
  AutomaticMount = yes
  RemovableMedia = yes
  Always Open = yes
  Offline On Unmount = no
  Hardware End of Medium = yes
  BSF at EOM = yes
  Backward Space Record = yes
  Fast Forward Space File = yes
  TWO EOF = yes

Third Try:
Device {
  Name   = QuantumLTO
  Media Type = LTO6
  Device Type = Tape
  ArchiveDevice = /dev/nsa0
  Description = "LTO-6 for FreeBSD"
  LabelMedia = yes
  AutomaticMount = yes;   # when device opened, read it
  AlwaysOpen = yes
  RemovableMedia = yes
  Offline On Unmount = no
  Hardware End of Medium = no
  BSF at EOM = yes
  Backward Space Record = no
  Fast Forward Space File = no
  TWO EOF = yes

Fourth Try:
Device {
  Name   = QuantumLTO
  Media Type = LTO6
  Device Type = Tape
  ArchiveDevice = /dev/nsa0
  Description = "LTO-6 for FreeBSD"
  LabelMedia = yes
  AutomaticMount = yes;   # when device opened, read it
  AlwaysOpen = yes
  RemovableMedia = yes
  Offline On Unmount = no
  Hardware End of Medium = no   # Noted as FreeBSD specific
  BSF at EOM = no   # Noted as FreeBSD specific
  Backward Space Record = no# Noted as FreeBSD specific
  Fast Forward Space File = yes # Noted as FreeBSD specific
  TWO EOF = no  # Noted as FreeBSD specific

And today, as I type this I’m trying a btape fill test with:
Device {
  Name   = QuantumLTO
  Media Type = LTO6
  Device Type = Tape
  ArchiveDevice = /dev/nsa0
  Description = "LTO-6 for FreeBSD"
  LabelMedia = yes
  AutomaticMount = yes;   # when device opened, read it
  AlwaysOpen = yes
  RemovableMedia = yes
  Offline On Unmount = no
  Hardware End of Medium = no   # Noted as FreeBSD specific
  BSF at EOM = yes  # Support List Recommendation
  Backward Space Record = yes   # Support List Recommendation
  Fast Forward Space File = yes # Noted as FreeBSD specific
  TWO EOF = yes # Support List Recommendation
Minimum Block Size = 64512
Maximum Block Size = 64512

Which is similar to my third try, with the exceptions of Backward Space Record, 
Fast Forward Space File, and also the addition of the Min and Max Block Size 
directives.

So at this point my question is this: is there a smart way to identify the 
correct entries for my hardware? I’ve re-read the docs from quantum but clearly 
I’m still flailing. The trial-and-error approach wouldn’t be so bad if it 
didn’t take 11 hours to write 2.5tb, then a short while to begin the second 
tape, then begin the unfill portion of the test, then eventually fail. It’s 
basically a full calendar day with each config file guess iteration. 

As an aside, gtar wrote to tape at almost exactly twice the speed that I’m 
seeing in the btape test. My disk arrays are showing 300+mb/s for random reads, 
so they’re not the bottleneck. Almost immediately after I get this multi-tape 
dump and restore working in bacula I’ll be needing to address the horrible 
speeds I’m seeing. I do notice a lot of  “shoe shining” while running btape 
fill, and nothing is being written to the spool directory (which is ssd) so 
maybe the real backups will be faster. We’ll see.

Thanks again,
Simon


> On Mar 1, 2016, at 8:42 AM, Martin Simmons  wrote:
> 
>> On Tue, 1 Mar 2016 14:07:55 +0100, Cejka Rudolf said:
>> 
>> Simon Templar wrote (2016/02/29):
>>> I've set up bacula 7.2 on my FreeBSD server using Postgres as the 
>>> backend database.
>> 
>> Which FreeBSD version? According to the mt status it seems to be
>> sufficiently fresh.
>> 
>>> Device {
>>> ...
>>>   Hardware End of Medium = no   # Noted as FreeBSD specific
>>>   BSF at EOM = no   # Noted as FreeBSD specific
>>>   Backward Space Record = no# Noted as FreeBSD specific
>>>   Fast Forward Space File = yes # Noted as FreeBSD specific
>>>   TWO EOF = no  # Noted as FreeBSD specific
>> 
>> Where did you get this?
>> 
>> Try this:
>> 
>>  BSF at EOM = yes
>>  TWO EOF = yes
>>  Hardware End of Medium = no   # maybe not needed now, but there is no
>># performance impact, so let's start with it
>>  Backward

Re: [Bacula-users] Trouble setting up Bacula on FreeBSD

2016-03-01 Thread Martin Simmons 
> On Tue, 1 Mar 2016 14:07:55 +0100, Cejka Rudolf said:
> 
> Simon Templar wrote (2016/02/29):
> > I've set up bacula 7.2 on my FreeBSD server using Postgres as the 
> > backend database.
> 
> Which FreeBSD version? According to the mt status it seems to be
> sufficiently fresh.
> 
> > Device {
> > ...
> >Hardware End of Medium = no   # Noted as FreeBSD specific
> >BSF at EOM = no   # Noted as FreeBSD specific
> >Backward Space Record = no# Noted as FreeBSD specific
> >Fast Forward Space File = yes # Noted as FreeBSD specific
> >TWO EOF = no  # Noted as FreeBSD specific
> 
> Where did you get this?
> 
> Try this:
> 
>   BSF at EOM = yes
>   TWO EOF = yes
>   Hardware End of Medium = no   # maybe not needed now, but there is no
> # performance impact, so let's start with it
>   Backward Space Record = yes   # not needed, changed to yes is the default
>   Fast Forward Space File = yes # not needed, yes is the default

There is no "correct" value for all hardware.  I needed TWO EOF = yes etc for
DDS drive on FreeBSD 4, but for an HP LTO-1 drive on FreeBSD 8 I needed this:

  Hardware End of Medium = no
  BSF at EOM = no
  Backward Space Record = yes
  Backward Space File = yes
  Fast Forward Space File = yes
  TWO EOF = no

__Martin

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Trouble setting up Bacula on FreeBSD

2016-03-01 Thread Cejka Rudolf 
Simon Templar wrote (2016/02/29):
> I've set up bacula 7.2 on my FreeBSD server using Postgres as the 
> backend database.

Which FreeBSD version? According to the mt status it seems to be
sufficiently fresh.

> Device {
> ...
>Hardware End of Medium = no   # Noted as FreeBSD specific
>BSF at EOM = no   # Noted as FreeBSD specific
>Backward Space Record = no# Noted as FreeBSD specific
>Fast Forward Space File = yes # Noted as FreeBSD specific
>TWO EOF = no  # Noted as FreeBSD specific

Where did you get this?

Try this:

  BSF at EOM = yes
  TWO EOF = yes
  Hardware End of Medium = no   # maybe not needed now, but there is no
# performance impact, so let's start with it
  Backward Space Record = yes   # not needed, changed to yes is the default
  Fast Forward Space File = yes # not needed, yes is the default

-- 
Rudolf Cejka  http://www.fit.vutbr.cz/~cejkar
Brno University of Technology, Faculty of Information Technology
Bozetechova 2, 612 66  Brno, Czech Republic

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Trouble setting up Bacula on FreeBSD

2016-03-01 Thread Ana Emília M . Arruda 
Hello Simon,

I´m not a FreeBSD expert, but from here:
http://lists.freebsd.org/pipermail/freebsd-scsi/2003-June/000190.html, it
seems you should configure BSF at EOM = yes (if I am not misunderstanding).

Best regards,
Ana

On Mon, Feb 29, 2016 at 5:19 PM, Simon Templar 
wrote:

> Or perhaps a more accurate subject would be "Trouble successfully using
> Bacula on FreeBSD"...
>
> I've set up bacula 7.2 on my FreeBSD server using Postgres as the
> backend database.
>
> My tape drive info:
> mt -f /dev/nsa0 status
> Drive: sa0:  Serial Number: ##
> -
> Mode  Density  Blocksize  bpi  Compression
> Current:  0x5a:LTO-6   variable   384607   enabled (0x1)
> -
> Current Driver State: at rest.
> -
> Partition:   0  Calc File Number:   0 Calc Record Number: 0
> Residual:0  Reported File Number:   0 Reported Record Number: 0
> Flags: BOP
>
> I've iterated through the config files until they seemed to be correct
> (though I admit I'm over my head with the bacula-sd.conf specifics, and
> I've tried and failed with examples that I've found for LTO drives and
> also with FreeBSD-specific options), then dumped and recreated the
> tables and deleted the old .state files so I had a "clean slate". I ran
> btape test which was successful, but when I run btape fill it
> successfully writes to both tapes, then fails on the read/unfill portion
> with this output:
>
> Reading the first 1 records from 0:0.
> 1 records read now at 1:5084
> Reposition from 1:5084 to 2501:6988
> Reading block 6988.
>
> The blocks differ at byte 0
>
>  The last block written and the block
> that was read back differ. The test FAILED 
> This must be corrected before you use Bacula
> to write multi-tape Volumes.
> btape: btape.c:2585-0 Autochanger returned: 0
> Mount second tape. Press enter when ready:
> btape: btape.c:2588-0
> 29-Feb 09:46 btape JobId 0: Ready to read from volume "TestVolume2" on
> tape device "QuantumLTO" (/dev/nsa0).
> Reposition from 0:0 to 0:1
> Reading block 1.
>
> The first block on the second tape matches.
>
> Reposition from 0:2 to 0:1001
> Reading block 1001.
>
> The blocks differ at byte 0
>
>  The last block written and the block
> that was read back differ. The test FAILED 
> This must be corrected before you use Bacula
> to write multi-tape Volumes.
> Bacula interrupted by signal 11: Segmentation violation
> Kaboom! btape, btape got signal 11 - Segmentation violation at
> 29-Feb-2016 09:46:33. Attempting traceback.
> Kaboom! exepath=/root
> Calling: /root/btraceback /root/btape 1377 /tmp
> It looks like the traceback worked...
> Dumping: /tmp/btape.1377.lockdump
> btape: lockmgr.c:1158-0 lockmgr disabled
> Segmentation fault (core dumped)
>
> My current bacula-sd.conf file:
>
> Storage {
>Name = UralTape
>WorkingDirectory = "/usr/local/bacula/working"
>Pid Directory= "/var/run"
> }
>
> Director {
>Name = Director-Ural
>Password = "#"
> }
>
> Device {
>Name   = QuantumLTO
>Media Type = LTO6
>Device Type = Tape
>ArchiveDevice = /dev/nsa0
>Description = "LTO-6 for FreeBSD"
>LabelMedia = yes
>AutomaticMount = yes;   # when device opened, read it
>AlwaysOpen = yes
>RemovableMedia = yes
>Offline On Unmount = no
>Hardware End of Medium = no   # Noted as FreeBSD specific
>BSF at EOM = no   # Noted as FreeBSD specific
>Backward Space Record = no# Noted as FreeBSD specific
>Fast Forward Space File = yes # Noted as FreeBSD specific
>TWO EOF = no  # Noted as FreeBSD specific
> # for data spooling
> Maximum Spool Size = 75gb
> Maximum Job Spool Size = 75gb
> Spool Directory = /usr/local/bacula/spool
> #  If you have smartctl, enable this, it has more info than tapeinfo
> #  Alert Command = "sh -c 'smartctl -H -l error %c'"
>
>
> }
>
> Messages {
>Name = Standard
>director = Director-Ural = all
> }
>
> I saved each version of my bacula-sd.conf file and can share them, but I
> think the primary (if not only?) differences are the lines with the
> comments about being FreeBSD-specific.
>
> In the past I've used this same platform while running centos to create
> multi-tape tars of xfs filesystems, and also used ltfs without issue. It
> was recently that we moved to FreeBSD and this tape backup solution is
> the last component I'm wrestling with.
>
> Any suggestions?
>
> Thanks,
> Simon
>
>
>
>
>
>
>
> --
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot