Re: [Bacula-users] Simple disk storage devices migrations/backups

[Martin Simmons]

> On Wed, 1 Apr 2020 16:02:59 +0200, Pierre Bernhardt said:
> 
> I think to correct the problems on existing backups a bscan -m -s should also
> fix this issues permanently so a patch of bscan is needed, too.

AFAICS (I've not tried it), bscan will not fail if it gets these errors.  It
will print the error messages but will still update the database.

__Martin


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Simple disk storage devices migrations/backups

[Pierre Bernhardt]

Am 01.04.20 um 15:19 schrieb Martin Simmons:
> Yes, I think 1 is the best solution, but it will not fix existing backups.
> 
> Migration could also be changed to allow certain types of error, like restore
> does with the "Restore OK -- with errors" status.
I think to correct the problems on existing backups a bscan -m -s should also
fix this issues permanently so a patch of bscan is needed, too.

Cheers,
Pierre



___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Simple disk storage devices migrations/backups

[Martin Simmons]

> On Wed, 1 Apr 2020 11:36:22 +0200, Pierre Bernhardt said:
> 
> Am 01.04.20 um 00:25 schrieb Pierre Bernhardt:
> > Am 01.04.20 um 00:11 schrieb Pierre Bernhardt:
> > So now I restarted again a migration to check the result.
> 
> Here the content from the migration job:
> The migration back to the tape has now been finshed without problems:

Great!

> By the way, do you have an idea how it should be fixed? I think, this 
> workaround is not
> the correct solution, isn't it?

Yes, the workaround is a bad solution because it might hide real errors.

> In my opinion at the time the job has to been written and splitted to the 
> disks there two
> solutions possible:
> At the end the unexpected file write has been identified and before the next 
> disk has to been
> ask for:
> 1. The short block should be truncated and not written to the database
> or
> 2. The short block should be ignored to write to the database.
> or
> 3. The short block should be marked as invalid in the database and should be 
> ignored.
> 
> After this step the block should be rewritten on the next volume.
> 
> 1. looks like me for the cleanest solution because in read this block is not 
> there
> 
> 2. looks like me a little bit problematic because if eg. bscan is used it 
> must also
> fix/ignore this block.
> 
> 3. this is a good solution if there exists already mechanism in all tools and 
> bacule
> to ignore such parts in backup volume data.

Yes, I think 1 is the best solution, but it will not fix existing backups.

Migration could also be changed to allow certain types of error, like restore
does with the "Restore OK -- with errors" status.

__Martin


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Simple disk storage devices migrations/backups

[Pierre Bernhardt]

Am 01.04.20 um 00:25 schrieb Pierre Bernhardt:
> Am 01.04.20 um 00:11 schrieb Pierre Bernhardt:
> So now I restarted again a migration to check the result.

Here the content from the migration job:
The migration back to the tape has now been finshed without problems:
01-Apr 00:19 backup-dir JobId 47775: The following 1 JobId was chosen to be 
migrated: 47704
01-Apr 00:19 backup-dir JobId 47775: Migration using JobId=47704 
Job=nihilnihil_home.2020-03-21_20.23.31_49
01-Apr 00:19 backup-dir JobId 47775: Start Migration JobId 47775, 
Job=MigrateFile2Drive.2020-04-01_00.19.07_04
01-Apr 00:19 backup-dir JobId 47775: Using Device "DiskStorage2" to read.
01-Apr 00:19 backup-sd JobId 47775: Ready to read from volume "DISK016" on File 
device "DiskStorage2" (/media/baculadisk2).
01-Apr 00:19 backup-sd JobId 47775: Forward spacing Volume "DISK016" to addr=217
01-Apr 05:27 backup-sd JobId 47775: block.c:682 [SE0208] Volume data has error 
at 0:0! Short block of 57010 bytes on device "DiskStorage2" 
(/media/baculadisk2) discarded.
01-Apr 05:27 backup-sd JobId 47775: read_records.c:160 block.c:682 [SE0208] 
Volume data has error at 0:0! Short block of 57010 bytes on device 
"DiskStorage2" (/media/baculadisk2) discarded.
01-Apr 05:27 backup-sd JobId 47775: End of Volume "DISK016" at 
addr=972406571008 on device "DiskStorage2" (/media/baculadisk2).
01-Apr 05:28 backup-sd JobId 47775: Ready to read from volume "DISK017" on File 
device "DiskStorage2" (/media/baculadisk2).
01-Apr 05:28 backup-sd JobId 47775: Forward spacing Volume "DISK017" to addr=213
01-Apr 06:27 backup-sd JobId 47775: End of Volume "DISK017" at 
addr=110838477984 on device "DiskStorage2" (/media/baculadisk2).
01-Apr 06:27 backup-sd JobId 47775: Elapsed time=06:07:56, Transfer rate=49.02 
M Bytes/second
01-Apr 08:30 backup-dir JobId 47775: Bacula backup-dir 9.4.2 (04Feb19):
  Build OS:   x86_64-pc-linux-gnu debian buster/sid
  Prev Backup JobId:  47704
  Prev Backup Job:nihilnihil_home.2020-03-21_20.23.31_49
  New Backup JobId:   47776
  Current JobId:  47775
  Current Job:MigrateFile2Drive.2020-04-01_00.19.07_04
  Backup Level:   Full
  Client: backup-fd
  FileSet:"Full Set" 2017-10-09 08:53:50
  Read Pool:  "Migrate" (From Job resource)
  Read Storage:   "Disk2" (From Pool resource)
  Write Pool: "Monthly" (From Job Pool's NextPool resource)
  Write Storage:  "FibreCAT TX48 S2" (From Job Pool's NextPool resource)
  Catalog:"MyCatalog" (From Client resource)
  Start time: 01-Apr-2020 00:19:11
  End time:   01-Apr-2020 08:25:40
  Elapsed time:   8 hours 6 mins 29 secs
  Priority:   21
  SD Files Written:   1,030,385
  SD Bytes Written:   1,082,331,572,757 (1.082 TB)
  Rate:   37080.1 KB/s
  Volume name(s): LTO40025|LTO40026
  Volume Session Id:  1
  Volume Session Time:1585692943
  Last Volume Bytes:  270,297,861,120 (270.2 GB)
  SD Errors:  0
  SD termination status:  OK
  Termination:Migration OK

Here from migration-backup job:
01-Apr 00:19 backup-dir JobId 47776: Recycled current volume "LTO40025"
01-Apr 00:19 backup-dir JobId 47776: Using Device "HP
Ultrium
4-2" to write.
01-Apr 00:19 backup-sd JobId 47776: Recycled volume "LTO40025" on Tape device 
"HP Ultrium 4-2" (/dev/tape/by-id/scsi-HU19145705-nst), all previous data lost.
01-Apr 04:30 backup-sd JobId 47776: [SI0202] End of Volume "LTO40025" at 
812:15489 on device "HP Ultrium 4-2" (/dev/tape/by-id/scsi-HU19145705-nst). 
Write of 64512 bytes got -1.
01-Apr 04:30 backup-sd JobId 47776: Re-read of last block succeeded.
01-Apr 04:30 backup-sd JobId 47776: End of medium on Volume "LTO40025" 
Bytes=812,947,258,368 Blocks=12,601,488 at 01-Apr-2020 04:30.
01-Apr 04:30 backup-sd JobId 47776: 3307 Issuing autochanger "unload Volume 
LTO40025, Slot 17, Drive 1" command.
01-Apr 04:32 backup-dir JobId 47776: Recycled volume "LTO40026"
01-Apr 04:32 backup-sd JobId 47776: 3304 Issuing autochanger "load Volume 
LTO40026, Slot 9, Drive 1" command.
01-Apr 04:34 backup-sd JobId 47776: 3305 Autochanger "load Volume LTO40026, 
Slot 9, Drive 1", status is OK.
01-Apr 04:34 backup-sd JobId 47776: Recycled volume "LTO40026" on Tape device 
"HP Ultrium 4-2" (/dev/tape/by-id/scsi-HU19145705-nst), all previous data lost.
01-Apr 04:34 backup-sd JobId 47776: New volume "LTO40026" mounted on device "HP 
Ultrium 4-2" (/dev/tape/by-id/scsi-HU19145705-nst) at 01-Apr-2020 04:34.
01-Apr 06:30 backup-sd JobId 47776: Elapsed time=06:07:02, Transfer rate=49.14 
M Bytes/second
01-Apr 06:30 backup-sd JobId 47776: Sending spooled attrs to the Director. 
Despooling 300,838,146 bytes ...

I've compared the file which was stored in the short block: The restored file
from the migration to tape has the same sha1sum as the one which I restored
from the two disks and the original file.

So 

Re: [Bacula-users] Simple disk storage devices migrations/backups

[Bahman Sharzad Asr]

i running bacula 9.0.6 on ubuntu 18.04. Backup of clients is ok. but get 
error for catalog backup: Cannot find any appendable volumes.


And in my libbacsd-9.0.6.so:

strings libbacsd-9.0.6.so | grep short
Volume data error at %s! Very short block of %d bytes on device %s 
discarded.


i created a volume with label comman from bconsol., but did not help.

On 4/1/20 12:25 AM, Pierre Bernhardt wrote:

Am 01.04.20 um 00:11 schrieb Pierre Bernhardt:

In which installed files is block.c and read_records.c used?
I compiled all but repaced only the bacula-sd files. Maybe the modification
is located in another file and not in the binary?

Ok, I found the file by using grep:

root@backup:/var/lib/bacula# strings /usr/lib/bacula/libbacsd-9.4.2.so |grep 
Short
[SE0208] Volume data has error at %u:%u! Short block of %d bytes on device %s 
discarded.

The file is installed by debian-common deb so now I installed this new created
installation deb. As you can see I've added also a …has… in the message so we 
should
now see the modified in the output of the job.

So now I restarted again a migration to check the result.

Cheers,
Pierre



___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

--



Bahman Sharzad

Systemadministrator


Process Factory ApS

Rosenørns Allé 9, 5. sal

DK-1970 Frederiksberg C

CVR: 30905970


M: +45 60 16 65 93

E: bahman.sharzad 
@process-factory.dk 



W: www.process-factory.dk 


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Simple disk storage devices migrations/backups

[Pierre Bernhardt]

Am 01.04.20 um 00:11 schrieb Pierre Bernhardt:
> In which installed files is block.c and read_records.c used?
> I compiled all but repaced only the bacula-sd files. Maybe the modification
> is located in another file and not in the binary?
Ok, I found the file by using grep:

root@backup:/var/lib/bacula# strings /usr/lib/bacula/libbacsd-9.4.2.so |grep 
Short
[SE0208] Volume data has error at %u:%u! Short block of %d bytes on device %s 
discarded.

The file is installed by debian-common deb so now I installed this new created
installation deb. As you can see I've added also a …has… in the message so we 
should
now see the modified in the output of the job.

So now I restarted again a migration to check the result.

Cheers,
Pierre



___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Simple disk storage devices migrations/backups

[Pierre Bernhardt]

Am 31.03.20 um 22:07 schrieb Pierre Bernhardt:
>> Your change to use M_INFO looks correct, but the block.c:682 message in the
>> log still says "Error:" so you are still running the original code.  Did you
>> run "make" and "make install" after changing the code?  Did they complete
>> without errors (you might need to run "make install" as root)?  Did you
>> restart the bacula-sd after that?
> I compiled all on my workstation by the debian way, means I created a modified
> deb, transfered it to the backup server and installed it by dpkg -i.
> I checked also the md5sum of the bin before and after the installation and
> the original installation differs from the new compiled version.

In which installed files is block.c and read_records.c used?
I compiled all but repaced only the bacula-sd files. Maybe the modification
is located in another file and not in the binary?

Cheers,
Pierre



___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Simple disk storage devices migrations/backups

[Pierre Bernhardt]

Am 31.03.20 um 17:05 schrieb Martin Simmons:
>> On Tue, 31 Mar 2020 12:40:04 +0200, Pierre Bernhardt said:
>>
>> Am 30.03.20 um 16:12 schrieb Martin Simmons:
>> Hello,
>>

>>
>> I think the return line should be corrected by changing the false? I'm 
>> virgin coding
>> c++ ;-)
>>
>>if (block->block_len > block->read_len) {
>>   dev->dev_errno = EIO;
>>   Mmsg4(dev->errmsg, _("[SE0208] Volume has data error at %u:%u! Short 
>> block of %d bytes on device %s discarded.\n"),
>>  dev->file, dev->block_num, block->read_len, dev->print_name());
>>   Jmsg(jcr, M_INFO, 0, "%s", dev->errmsg);
>>   dev->set_short_block();
>>   block->read_len = block->binbuf = 0;
>>   return true; /* return error */
>>}
> 
> No, returning true will not work correctly -- the calling function must get
> false for a short block.
Ok. For the moment I test it without any return value, but found allready 
another
problem with the new binary, but this is another point.

> Your change to use M_INFO looks correct, but the block.c:682 message in the
> log still says "Error:" so you are still running the original code.  Did you
> run "make" and "make install" after changing the code?  Did they complete
> without errors (you might need to run "make install" as root)?  Did you
> restart the bacula-sd after that?
I compiled all on my workstation by the debian way, means I created a modified
deb, transfered it to the backup server and installed it by dpkg -i.
I checked also the md5sum of the bin before and after the installation and
the original installation differs from the new compiled version.

> I also notice that there is another use of M_ERROR at line 160 of
> read_records.c that causes a second error message:
> 
>Jmsg1(jcr, M_ERROR, 0, "%s", dev->errmsg);
> 
> This also needs to be changed to M_INFO.
Will do it also and retest it. Also I will modify the message a little bit
so I can check the message is really from the modified binary.

Cheers,
Pierre



___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Simple disk storage devices migrations/backups

[Martin Simmons]

> On Tue, 31 Mar 2020 12:40:04 +0200, Pierre Bernhardt said:
> 
> Am 30.03.20 um 16:12 schrieb Martin Simmons:
> Hello,
> 
> > You could try temporarily hacking bacula-sd to report the short block as an
> > info message.  In src/stored/block.c, change M_ERROR to M_INFO in these 
> > lines:
> > 
> >   Mmsg4(dev->errmsg, _("[SE0208] Volume data error at %u:%u! Short 
> > block of %d bytes on device %s discarded.\n"),
> >  dev->file, dev->block_num, block->read_len, dev->print_name());
> >   Jmsg(jcr, M_ERROR, 0, "%s", dev->errmsg);
> It looks like it is not enought ot change M_ERROR to M_INFO because the job 
> will still
> fail and so the metadata still wont migrate to the new job:
> 
> 31-Mar 01:48 backup-dir JobId 47771: The following 1 JobId was chosen to be 
> migrated: 47704
> 31-Mar 01:48 backup-dir JobId 47771: Migration using JobId=47704 
> Job=nihilnihil_home.2020-03-21_20.23.31_49
> 31-Mar 01:48 backup-dir JobId 47771: Start Migration JobId 47771, 
> Job=MigrateFile2Drive.2020-03-31_01.48.24_05
> 31-Mar 01:49 backup-dir JobId 47771: Using Device "DiskStorage2" to read.
> 31-Mar 01:53 backup-sd JobId 47771: Ready to read from volume "DISK016" on 
> File device "DiskStorage2" (/media/baculadisk2).
> 31-Mar 01:53 backup-sd JobId 47771: Forward spacing Volume "DISK016" to 
> addr=217
> 31-Mar 07:00 backup-sd JobId 47771: Error: block.c:682 [SE0208] Volume data 
> error at 0:0! Short block of 57010 bytes on device "DiskStorage2" 
> (/media/baculadisk2) discarded.
> 31-Mar 07:00 backup-sd JobId 47771: Error: read_records.c:160 block.c:682 
> [SE0208] Volume data error at 0:0! Short block of 57010 bytes on device 
> "DiskStorage2" (/media/baculadisk2) discarded.
> 31-Mar 07:00 backup-sd JobId 47771: End of Volume "DISK016" at 
> addr=972406571008 on device "DiskStorage2" (/media/baculadisk2).
> 31-Mar 07:01 backup-sd JobId 47771: Ready to read from volume "DISK017" on 
> File device "DiskStorage2" (/media/baculadisk2).
> 31-Mar 07:01 backup-sd JobId 47771: Forward spacing Volume "DISK017" to 
> addr=213
> 31-Mar 08:00 backup-sd JobId 47771: End of Volume "DISK017" at 
> addr=110838477984 on device "DiskStorage2" (/media/baculadisk2).
> 31-Mar 08:00 backup-sd JobId 47771: Elapsed time=06:07:58, Transfer 
> rate=49.02 M Bytes/second
> 31-Mar 10:01 backup-dir JobId 47771: Warning: Found errors during the 
> migration process. The original job 47704 will be kept in the catalog and the 
> Migration job will be marked in Error
> 31-Mar 10:01 backup-dir JobId 47771: Error: bsock.c:388 Wrote 4 bytes to 
> Storage daemon:backup.localnet.cosmicstars.de:9103, but only 0 accepted.
> 31-Mar 10:01 backup-dir JobId 47771: Error: Bacula backup-dir 9.4.2 (04Feb19):
>   Build OS:   x86_64-pc-linux-gnu debian buster/sid
>   Prev Backup JobId:  47704
>   Prev Backup Job:nihilnihil_home.2020-03-21_20.23.31_49
>   New Backup JobId:   47772
>   Current JobId:  47771
>   Current Job:MigrateFile2Drive.2020-03-31_01.48.24_05
>   Backup Level:   Full
>   Client: backup-fd
>   FileSet:"Full Set" 2017-10-09 08:53:50
>   Read Pool:  "Migrate" (From Job resource)
>   Read Storage:   "Disk2" (From Pool resource)
>   Write Pool: "Monthly" (From Job Pool's NextPool resource)
>   Write Storage:  "FibreCAT TX48 S2" (From Job Pool's NextPool 
> resource)
>   Catalog:"MyCatalog" (From Client resource)
>   Start time: 31-Mar-2020 01:49:20
>   End time:   31-Mar-2020 10:01:43
>   Elapsed time:   8 hours 12 mins 23 secs
>   Priority:   21
>   SD Files Written:   1,030,385
>   SD Bytes Written:   1,082,331,572,757 (1.082 TB)
>   Rate:   36635.8 KB/s
>   Volume name(s): LTO40025|LTO40026
>   Volume Session Id:  1
>   Volume Session Time:1585612030
>   Last Volume Bytes:  270,297,861,120 (270.2 GB)
>   SD Errors:  2
>   SD termination status:  OK
>   Termination:*** Migration Error ***
> 
> 
> 
> I think the return line should be corrected by changing the false? I'm virgin 
> coding
> c++ ;-)
> 
>if (block->block_len > block->read_len) {
>   dev->dev_errno = EIO;
>   Mmsg4(dev->errmsg, _("[SE0208] Volume has data error at %u:%u! Short 
> block of %d bytes on device %s discarded.\n"),
>  dev->file, dev->block_num, block->read_len, dev->print_name());
>   Jmsg(jcr, M_INFO, 0, "%s", dev->errmsg);
>   dev->set_short_block();
>   block->read_len = block->binbuf = 0;
>   return true; /* return error */
>}

No, returning true will not work correctly -- the calling function must get
false for a short block.

Your change to use M_INFO looks correct, but the block.c:682 message in the
log still says "Error:" so you are still running the original code.  Did you
run "make" and "make install" after changing the code?  Did they complete

Re: [Bacula-users] Simple disk storage devices migrations/backups

[Pierre Bernhardt]

Am 30.03.20 um 16:12 schrieb Martin Simmons:
Hello,

> You could try temporarily hacking bacula-sd to report the short block as an
> info message.  In src/stored/block.c, change M_ERROR to M_INFO in these lines:
> 
>   Mmsg4(dev->errmsg, _("[SE0208] Volume data error at %u:%u! Short block 
> of %d bytes on device %s discarded.\n"),
>  dev->file, dev->block_num, block->read_len, dev->print_name());
>   Jmsg(jcr, M_ERROR, 0, "%s", dev->errmsg);
It looks like it is not enought ot change M_ERROR to M_INFO because the job 
will still
fail and so the metadata still wont migrate to the new job:

31-Mar 01:48 backup-dir JobId 47771: The following 1 JobId was chosen to be 
migrated: 47704
31-Mar 01:48 backup-dir JobId 47771: Migration using JobId=47704 
Job=nihilnihil_home.2020-03-21_20.23.31_49
31-Mar 01:48 backup-dir JobId 47771: Start Migration JobId 47771, 
Job=MigrateFile2Drive.2020-03-31_01.48.24_05
31-Mar 01:49 backup-dir JobId 47771: Using Device "DiskStorage2" to read.
31-Mar 01:53 backup-sd JobId 47771: Ready to read from volume "DISK016" on File 
device "DiskStorage2" (/media/baculadisk2).
31-Mar 01:53 backup-sd JobId 47771: Forward spacing Volume "DISK016" to addr=217
31-Mar 07:00 backup-sd JobId 47771: Error: block.c:682 [SE0208] Volume data 
error at 0:0! Short block of 57010 bytes on device "DiskStorage2" 
(/media/baculadisk2) discarded.
31-Mar 07:00 backup-sd JobId 47771: Error: read_records.c:160 block.c:682 
[SE0208] Volume data error at 0:0! Short block of 57010 bytes on device 
"DiskStorage2" (/media/baculadisk2) discarded.
31-Mar 07:00 backup-sd JobId 47771: End of Volume "DISK016" at 
addr=972406571008 on device "DiskStorage2" (/media/baculadisk2).
31-Mar 07:01 backup-sd JobId 47771: Ready to read from volume "DISK017" on File 
device "DiskStorage2" (/media/baculadisk2).
31-Mar 07:01 backup-sd JobId 47771: Forward spacing Volume "DISK017" to addr=213
31-Mar 08:00 backup-sd JobId 47771: End of Volume "DISK017" at 
addr=110838477984 on device "DiskStorage2" (/media/baculadisk2).
31-Mar 08:00 backup-sd JobId 47771: Elapsed time=06:07:58, Transfer rate=49.02 
M Bytes/second
31-Mar 10:01 backup-dir JobId 47771: Warning: Found errors during the migration 
process. The original job 47704 will be kept in the catalog and the Migration 
job will be marked in Error
31-Mar 10:01 backup-dir JobId 47771: Error: bsock.c:388 Wrote 4 bytes to 
Storage daemon:backup.localnet.cosmicstars.de:9103, but only 0 accepted.
31-Mar 10:01 backup-dir JobId 47771: Error: Bacula backup-dir 9.4.2 (04Feb19):
  Build OS:   x86_64-pc-linux-gnu debian buster/sid
  Prev Backup JobId:  47704
  Prev Backup Job:nihilnihil_home.2020-03-21_20.23.31_49
  New Backup JobId:   47772
  Current JobId:  47771
  Current Job:MigrateFile2Drive.2020-03-31_01.48.24_05
  Backup Level:   Full
  Client: backup-fd
  FileSet:"Full Set" 2017-10-09 08:53:50
  Read Pool:  "Migrate" (From Job resource)
  Read Storage:   "Disk2" (From Pool resource)
  Write Pool: "Monthly" (From Job Pool's NextPool resource)
  Write Storage:  "FibreCAT TX48 S2" (From Job Pool's NextPool resource)
  Catalog:"MyCatalog" (From Client resource)
  Start time: 31-Mar-2020 01:49:20
  End time:   31-Mar-2020 10:01:43
  Elapsed time:   8 hours 12 mins 23 secs
  Priority:   21
  SD Files Written:   1,030,385
  SD Bytes Written:   1,082,331,572,757 (1.082 TB)
  Rate:   36635.8 KB/s
  Volume name(s): LTO40025|LTO40026
  Volume Session Id:  1
  Volume Session Time:1585612030
  Last Volume Bytes:  270,297,861,120 (270.2 GB)
  SD Errors:  2
  SD termination status:  OK
  Termination:*** Migration Error ***



I think the return line should be corrected by changing the false? I'm virgin 
coding
c++ ;-)

   if (block->block_len > block->read_len) {
  dev->dev_errno = EIO;
  Mmsg4(dev->errmsg, _("[SE0208] Volume has data error at %u:%u! Short 
block of %d bytes on device %s discarded.\n"),
 dev->file, dev->block_num, block->read_len, dev->print_name());
  Jmsg(jcr, M_INFO, 0, "%s", dev->errmsg);
  dev->set_short_block();
  block->read_len = block->binbuf = 0;
  return true; /* return error */
   }



> I suggest you also report this as a bug to https://bugs.bacula.org/.
Yes, I will do it later after I have some more information.
By debian buster I have another issue to compile the source. But this is 
another point.

Cheers,
Pierre



___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Simple disk storage devices migrations/backups

[Martin Simmons]

> On Fri, 27 Mar 2020 21:48:29 +0100, Pierre Bernhardt said:
> 
> Am 27.03.20 um 21:26 schrieb Pierre Bernhardt:
> > Am 27.03.20 um 18:26 schrieb Martin Simmons:
> 
> >>> Any idea how I can extract the single data für fileindex 932145 from the 
> >>> disks for comparing?
> >>> If the short block is only repeated as whole block on the next disk the 
> >>> problem could be fixed
> >>> by modify the database so the short block will not be read or by 
> >>> truncating at short block on
> >>> the DISK016 (?)
> >> You can find the filename for fileindex 932145 by running bscan -r -vv and
> >> then do a restore for that filename.
> 
> > By the way the restore (with bconsole?) will be work without getting a 
> > problem? I will
> > try to do it.
> 
> 27-Mar 21:27 backup-sd JobId 47756: Ready to read from volume "DISK016" on 
> File device "DiskStorage2" (/media/baculadisk2).
> 27-Mar 21:27 backup-sd JobId 47756: Forward spacing Volume "DISK016" to 
> addr=971937834340
> 27-Mar 21:28 backup-sd JobId 47756: Error: block.c:682 [SE0208] Volume data 
> error at 0:0! Short block of 57010 bytes on device "DiskStorage2" 
> (/media/baculadisk2) discarded.
> 27-Mar 21:28 backup-sd JobId 47756: Error: read_records.c:160 block.c:682 
> [SE0208] Volume data error at 0:0! Short block of 57010 bytes on device 
> "DiskStorage2" (/media/baculadisk2) discarded.
> 27-Mar 21:28 backup-sd JobId 47756: End of Volume "DISK016" at 
> addr=972406571008 on device "DiskStorage2" (/media/baculadisk2).
> 27-Mar 21:28 backup-sd JobId 47756: Ready to read from volume "DISK017" on 
> File device "DiskStorage2" (/media/baculadisk2).
> 27-Mar 21:28 backup-sd JobId 47756: Forward spacing Volume "DISK017" to 
> addr=213
> 27-Mar 21:28 backup-sd JobId 47756: End of Volume "DISK017" at addr=645332 on 
> device "DiskStorage2" (/media/baculadisk2).
> 27-Mar 21:28 backup-sd JobId 47756: Elapsed time=00:00:15, Transfer 
> rate=134.3 K Bytes/second
> 27-Mar 21:28 backup-dir JobId 47756: Bacula backup-dir 9.4.2 (04Feb19):
>   Build OS:   x86_64-pc-linux-gnu debian buster/sid
>   JobId:  47756
>   Job:RestoreFiles.2020-03-27_21.27.41_46
>   Restore Client: nihilnihil-fd
>   Where:  /tmp/restore
>   Replace:Always
>   Start time: 27-Mar-2020 21:27:44
>   End time:   27-Mar-2020 21:28:07
>   Elapsed time:   23 secs
>   Files Expected: 1
>   Files Restored: 1
>   Bytes Restored: 2,122,031 (2.122 MB)
>   Rate:   92.3 KB/s
>   FD Errors:  0
>   FD termination status:  OK
>   SD termination status:  OK
>   Termination:Restore OK -- with errors   27-Mar 21:28 backup-dir 
> JobId 47756: Begin pruning Jobs older than 99 years .
> 27-Mar 21:28 backup-dir JobId 47756: No Jobs found to prune.
> 27-Mar 21:28 backup-dir JobId 47756: Begin pruning Files.
> 27-Mar 21:28 backup-dir JobId 47756: No Files found to prune.
> 27-Mar 21:28 backup-dir JobId 47756: End auto prune.
> 
> Looks like restoring has been worked although only with the shot block notice 
> error message.
> 
> Now use a newer backup from tape to restore the same file and compare them 
> with sha256sum.
> Content and meta data are looks like correct after restore.

Good, so the short block was written to the next volume as expected.

> So my problem is only, because of failed migration because of short block 
> notice, the
> migrated database entries will not transfered after the migration back from 
> disk to tape
> has been proceed.
> 
> Is there a way so I can finish the migration maybe by a workaround?

You could try temporarily hacking bacula-sd to report the short block as an
info message.  In src/stored/block.c, change M_ERROR to M_INFO in these lines:

  Mmsg4(dev->errmsg, _("[SE0208] Volume data error at %u:%u! Short block of 
%d bytes on device %s discarded.\n"),
 dev->file, dev->block_num, block->read_len, dev->print_name());
  Jmsg(jcr, M_ERROR, 0, "%s", dev->errmsg);

I suggest you also report this as a bug to https://bugs.bacula.org/.

__Martin


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Simple disk storage devices migrations/backups

[Pierre Bernhardt]

Am 27.03.20 um 21:26 schrieb Pierre Bernhardt:
> Am 27.03.20 um 18:26 schrieb Martin Simmons:

>>> Any idea how I can extract the single data für fileindex 932145 from the 
>>> disks for comparing?
>>> If the short block is only repeated as whole block on the next disk the 
>>> problem could be fixed
>>> by modify the database so the short block will not be read or by truncating 
>>> at short block on
>>> the DISK016 (?)
>> You can find the filename for fileindex 932145 by running bscan -r -vv and
>> then do a restore for that filename.

> By the way the restore (with bconsole?) will be work without getting a 
> problem? I will
> try to do it.

27-Mar 21:27 backup-sd JobId 47756: Ready to read from volume "DISK016" on File 
device "DiskStorage2" (/media/baculadisk2).
27-Mar 21:27 backup-sd JobId 47756: Forward spacing Volume "DISK016" to 
addr=971937834340
27-Mar 21:28 backup-sd JobId 47756: Error: block.c:682 [SE0208] Volume data 
error at 0:0! Short block of 57010 bytes on device "DiskStorage2" 
(/media/baculadisk2) discarded.
27-Mar 21:28 backup-sd JobId 47756: Error: read_records.c:160 block.c:682 
[SE0208] Volume data error at 0:0! Short block of 57010 bytes on device 
"DiskStorage2" (/media/baculadisk2) discarded.
27-Mar 21:28 backup-sd JobId 47756: End of Volume "DISK016" at 
addr=972406571008 on device "DiskStorage2" (/media/baculadisk2).
27-Mar 21:28 backup-sd JobId 47756: Ready to read from volume "DISK017" on File 
device "DiskStorage2" (/media/baculadisk2).
27-Mar 21:28 backup-sd JobId 47756: Forward spacing Volume "DISK017" to addr=213
27-Mar 21:28 backup-sd JobId 47756: End of Volume "DISK017" at addr=645332 on 
device "DiskStorage2" (/media/baculadisk2).
27-Mar 21:28 backup-sd JobId 47756: Elapsed time=00:00:15, Transfer rate=134.3 
K Bytes/second
27-Mar 21:28 backup-dir JobId 47756: Bacula backup-dir 9.4.2 (04Feb19):
  Build OS:   x86_64-pc-linux-gnu debian buster/sid
  JobId:  47756
  Job:RestoreFiles.2020-03-27_21.27.41_46
  Restore Client: nihilnihil-fd
  Where:  /tmp/restore
  Replace:Always
  Start time: 27-Mar-2020 21:27:44
  End time:   27-Mar-2020 21:28:07
  Elapsed time:   23 secs
  Files Expected: 1
  Files Restored: 1
  Bytes Restored: 2,122,031 (2.122 MB)
  Rate:   92.3 KB/s
  FD Errors:  0
  FD termination status:  OK
  SD termination status:  OK
  Termination:Restore OK -- with errors   27-Mar 21:28 backup-dir 
JobId 47756: Begin pruning Jobs older than 99 years .
27-Mar 21:28 backup-dir JobId 47756: No Jobs found to prune.
27-Mar 21:28 backup-dir JobId 47756: Begin pruning Files.
27-Mar 21:28 backup-dir JobId 47756: No Files found to prune.
27-Mar 21:28 backup-dir JobId 47756: End auto prune.

Looks like restoring has been worked although only with the shot block notice 
error message.

Now use a newer backup from tape to restore the same file and compare them with 
sha256sum.
Content and meta data are looks like correct after restore.

So my problem is only, because of failed migration because of short block 
notice, the
migrated database entries will not transfered after the migration back from 
disk to tape
has been proceed.

Is there a way so I can finish the migration maybe by a workaround?

Cheers,
Pierre



___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Simple disk storage devices migrations/backups

[Pierre Bernhardt]

Am 27.03.20 um 18:26 schrieb Martin Simmons:
Hi,

>> Any idea how I can extract the single data für fileindex 932145 from the 
>> disks for comparing?
>> If the short block is only repeated as whole block on the next disk the 
>> problem could be fixed
>> by modify the database so the short block will not be read or by truncating 
>> at short block on
>> the DISK016 (?)
> You can find the filename for fileindex 932145 by running bscan -r -vv and
> then do a restore for that filename.
I found it on database with

select client.name, job.jobid, job.level, path.path, filename.name, file.lstat 
from file, filename, path, job, client where 
file.filenameid=filename.filenameid and file.pathid=path.pathid and 
file.jobid=job.jobid and job.clientid=client.clientid and job.jobid=47704 and 
file.fileindex=932145 order by job.endtime desc;

(Sorry for the long line ;-)

By the way the restore (with bconsole?) will be work without getting a problem? 
I will
try to do it.

Cheers,
Pierre



___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Simple disk storage devices migrations/backups

[Martin Simmons]

> On Thu, 26 Mar 2020 23:31:43 +0100, Pierre Bernhardt said:
> 
> Am 26.03.20 um 16:48 schrieb Martin Simmons:
> > Looks like a bug to me, but a possible workaround is to limit the size of 
> > your
> Me2. If the short block is correctly identified at writing it should be 
> repeated
> or should be finished on the new disk. In both cases the data on the disks 
> should
> be ok. Only the reading has maybe than a problem.
> But if the tail data of the short block is lost, the data on the disks should 
> be
> incomplete

I think the short block should be repeated on the new disk.

> 21-Mar 20:23 backup-dir JobId 47704: Using Volume "DISK016" from 'Scratch' 
> pool.
> 21-Mar 20:23 backup-dir JobId 47704: Using Device "DiskStorage2" to write.
> 21-Mar 20:23 backup-sd JobId 47704: Wrote label to prelabeled Volume 
> "DISK016" on File device "DiskStorage2" (/media/baculadisk2)
> 22-Mar 07:47 backup-sd JobId 47704: [SI0201] Out of freespace caused End of 
> Volume "DISK016" at 972406513998 on device "DiskStorage2" 
> (/media/baculadisk2). Write of 64512 bytes got 57010.
> 22-Mar 07:47 backup-sd JobId 47704: End of medium on Volume "DISK016" 
> Bytes=972,406,513,998 Blocks=15,073,266 at 22-Mar-2020 07:47.
> 22-Mar 07:47 backup-sd JobId 47704:

OK, this look as expected.

> By the way I created a bscan. Here the lines about the disk change:
> 
> 
> bscan: bscan.c:442-0 Record: SessId=42 SessTim=1584646035 FileIndex=932145 
> Stream=23 len=62656
> bscan: bscan.c:442-0 Record: SessId=42 SessTim=1584646035 FileIndex=932145 
> Stream=23 len=62336
> bscan: bscan.c:442-0 Record: SessId=42 SessTim=1584646035 FileIndex=932145 
> Stream=23 len=61632
> bscan: bscan.c:442-0 Record: SessId=42 SessTim=1584646035 FileIndex=932145 
> Stream=23 len=62528
> bscan: bscan.c:442-0 Record: SessId=42 SessTim=1584646035 FileIndex=932145 
> Stream=23 len=62672
> 25-Mar 23:27 bscan JobId 0: Error: block.c:682 [SE0208] Volume data error at 
> 0:0! Short block of 57010 bytes on device "FileTmpVol" (/media/baculadisk2) 
> discarded.
> ...
> 
> Any idea how I can extract the single data für fileindex 932145 from the 
> disks for comparing?
> If the short block is only repeated as whole block on the next disk the 
> problem could be fixed
> by modify the database so the short block will not be read or by truncating 
> at short block on
> the DISK016 (?)

You can find the filename for fileindex 932145 by running bscan -r -vv and
then do a restore for that filename.

__Martin


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Simple disk storage devices migrations/backups

[Pierre Bernhardt]

Am 26.03.20 um 16:48 schrieb Martin Simmons:
> Looks like a bug to me, but a possible workaround is to limit the size of your
Me2. If the short block is correctly identified at writing it should be repeated
or should be finished on the new disk. In both cases the data on the disks 
should
be ok. Only the reading has maybe than a problem.
But if the tail data of the short block is lost, the data on the disks should be
incomplete

> disk volumes (see Maximum Volume Bytes) to avoid filling the disks during the
> backup.  This will avoid the "short block" when you migrate.
Yes, but I have different size of disks (320 GB to 2 TB) and cannot change each
time the max size.

> BTW, can you post the log from jobid 47704 as well?

Migration job:

21-Mar 20:23 backup-dir JobId 47703: The following 1 JobId was chosen to be 
migrated: 46802
21-Mar 20:23 backup-dir JobId 47703: Migration using JobId=46802 
Job=nihilnihil_home.2020-01-05_23.50.01_29
21-Mar 20:23 backup-dir JobId 47703: Start Migration JobId 47703, 
Job=Migrate2FileTmpVol.2020-03-21_20.23.31_48
21-Mar 20:23 backup-dir JobId 47703: Using Device "HP
Ultrium
4-2" to read.
21-Mar 20:23 backup-sd JobId 47703: 3307 Issuing autochanger "unload Volume 
LTO40027, Slot 5, Drive 1" command.
21-Mar 20:27 backup-sd JobId 47703: 3304 Issuing autochanger "load Volume 
LTO40026, Slot 9, Drive 1" command.
21-Mar 20:28 backup-sd JobId 47703: 3305 Autochanger "load Volume LTO40026, 
Slot 9, Drive 1", status is OK.
21-Mar 20:28 backup-sd JobId 47703: Ready to read from volume "LTO40026" on 
Tape device "HP Ultrium 4-2" (/dev/tape/by-id/scsi-HU19145705-nst).
21-Mar 20:28 backup-sd JobId 47703: Forward spacing Volume "LTO40026" to 
addr=19:1457
22-Mar 00:36 backup-sd JobId 47703: End of Volume "LTO40026" at addr=628:5763 
on device "HP Ultrium 4-2" (/dev/tape/by-id/scsi-HU19145705-nst).
22-Mar 00:37 backup-sd JobId 47703: 3307 Issuing autochanger "unload Volume 
LTO40026, Slot 9, Drive 1" command.
22-Mar 00:38 backup-sd JobId 47703: 3304 Issuing autochanger "load Volume 
LTO40025, Slot 17, Drive 1" command.
22-Mar 00:40 backup-sd JobId 47703: 3305 Autochanger "load Volume LTO40025, 
Slot 17, Drive 1", status is OK.
22-Mar 00:40 backup-sd JobId 47703: Ready to read from volume "LTO40025" on 
Tape device "HP Ultrium 4-2" (/dev/tape/by-id/scsi-HU19145705-nst).
22-Mar 00:40 backup-sd JobId 47703: Forward spacing Volume "LTO40025" to 
addr=1:11779
22-Mar 05:33 backup-sd JobId 47703: End of Volume "LTO40025" at addr=0:0 on 
device "HP Ultrium 4-2" (/dev/tape/by-id/scsi-HU19145705-nst).
22-Mar 05:33 backup-sd JobId 47703: 3307 Issuing autochanger "unload Volume 
LTO40025, Slot 17, Drive 1" command.
22-Mar 05:34 backup-sd JobId 47703: 3304 Issuing autochanger "load Volume 
LTO40027, Slot 5, Drive 1" command.
22-Mar 05:36 backup-sd JobId 47703: 3305 Autochanger "load Volume LTO40027, 
Slot 5, Drive 1", status is OK.
22-Mar 05:36 backup-sd JobId 47703: Ready to read from volume "LTO40027" on 
Tape device "HP Ultrium 4-2" (/dev/tape/by-id/scsi-HU19145705-nst).
22-Mar 05:36 backup-sd JobId 47703: Forward spacing Volume "LTO40027" to 
addr=0:1
22-Mar 10:32 backup-sd JobId 47703: End of Volume "LTO40027" at addr=298:0 on 
device "HP Ultrium 4-2" (/dev/tape/by-id/scsi-HU19145705-nst).
22-Mar 10:32 backup-sd JobId 47703: Elapsed time=14:03:10, Transfer rate=21.39 
M Bytes/second
22-Mar 12:21 backup-dir JobId 47703: Bacula backup-dir 9.4.2 (04Feb19):
  Build OS:   x86_64-pc-linux-gnu debian buster/sid
  Prev Backup JobId:  46802
  Prev Backup Job:nihilnihil_home.2020-01-05_23.50.01_29
  New Backup JobId:   47704
  Current JobId:  47703
  Current Job:Migrate2FileTmpVol.2020-03-21_20.23.31_48
  Backup Level:   Full
  Client: backup-fd
  FileSet:"Full Set" 2017-10-09 08:53:50
  Read Pool:  "Monthly" (From Job resource)
  Read Storage:   "FibreCAT TX48 S2" (From Pool resource)
  Write Pool: "Migrate" (From Job Pool's NextPool resource)
  Write Storage:  "Disk2" (From Job Pool's NextPool resource)
  Catalog:"MyCatalog" (From Client resource)
  Start time: 21-Mar-2020 20:23:34
  End time:   22-Mar-2020 12:16:45
  Elapsed time:   15 hours 53 mins 11 secs
  Priority:   21
  SD Files Written:   1,030,385
  SD Bytes Written:   1,082,331,572,757 (1.082 TB)
  Rate:   18924.9 KB/s
  Volume name(s): DISK016|DISK017
  Volume Session Id:  41
  Volume Session Time:1584646035
  Last Volume Bytes:  110,838,413,473 (110.8 GB)
  SD Errors:  0
  SD termination status:  OK
  Termination:Migration OK

Related Backup-Migration Job:


21-Mar 20:23 backup-dir JobId 47704: Using Volume "DISK016" from 'Scratch' pool.
21-Mar 20:23 backup-dir JobId 47704: Using Device "DiskStorage2" to write.
21-Mar 20:23 backup-sd JobId 47704: Wrote label to prelabeled Volume "DISK016" 

Re: [Bacula-users] Simple disk storage devices migrations/backups

[Pierre Bernhardt]

Am 26.03.20 um 14:06 schrieb Josh Fisher:
> On 3/25/2020 3:23 PM, Pierre Bernhardt wrote:
> And what is the Autochanger / Device configurations for the disk storage in 
> bacula-sd.conf?

Device {
  Name = DiskStorage1
  Media Type = Disk
  Device Type = File
  Archive Device = /media/baculadisk1
  LabelMedia = yes;   # lets Bacula label unlabeled media
  AutoChanger = No;
  Random Access = Yes;
  AutomaticMount = yes;   # when device opened, read it
  RemovableMedia = yes;
  AlwaysOpen = Yes;
  Requires Mount = yes;
  Spool Directory = "/tmp"
  Maximum Spool Size = 17179869184
  Maximum Job Spool Size = 17179869184
  Mount Point = /media/baculadisk1
  Mount Command = "/usr/bin/pmount %m"
  Unmount Command = "/usr/bin/pumount %m"
  #Free Space Command = "echo `/bin/df %m | /usr/bin/tr -s \  | /usr/bin/cut -d 
\  -f 4 | /usr/bin/tail -1`*1024 | /usr/bin/bc"
}

Cheers,
Pierre



___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Simple disk storage devices migrations/backups

[Martin Simmons]

Looks like a bug to me, but a possible workaround is to limit the size of your
disk volumes (see Maximum Volume Bytes) to avoid filling the disks during the
backup.  This will avoid the "short block" when you migrate.

BTW, can you post the log from jobid 47704 as well?

__Martin


> On Wed, 25 Mar 2020 20:23:20 +0100, Pierre Bernhardt said:
> 
> Am 24.03.20 um 22:39 schrieb Pierre Bernhardt:
> >
> Today I tried to migrate again the job which uses two disk files.
> But now I tried to put both files in one directory (I used my second
> bay to mount the DISK017 file) and used a symbolic link:
> 
> -rw-rw-r-- 1 bacula tape 972406571008 Mar 24 17:10 /media/baculadisk1/DISK017
> -rw-rw-r-- 1 bacula tape 972406571008 Mar 22 07:47 /media/baculadisk2/DISK016
> lrwxrwxrwx 1 root   root   26 Mar 25 07:32 /media/baculadisk2/DISK017 
> -> /media/baculadisk1/D
> 
> By the way the migration situation doesn't have been changed:
> 
> Here the messages from the backup and from the migration job:
> 
> 
> Backup (Migration) Job:
> 25-Mar 07:48 backup-dir JobId 47755: Recycled volume "LTO40026"
> 25-Mar 07:48 backup-dir JobId 47755: Using Device "HP
Ultrium
4-2" to write.
> 25-Mar 07:48 backup-sd JobId 47755: 3307 Issuing autochanger "unload Volume 
> LTO40030, Slot 8, Drive 1" command.
> 25-Mar 07:49 backup-sd JobId 47755: 3304 Issuing autochanger "load Volume 
> LTO40026, Slot 9, Drive 1" command.
> 25-Mar 07:51 backup-sd JobId 47755: 3305 Autochanger "load Volume LTO40026, 
> Slot 9, Drive 1", status is OK.
> 25-Mar 07:51 backup-sd JobId 47755: Recycled volume "LTO40026" on Tape device 
> "HP Ultrium 4-2" (/dev/tape/by-id/scsi-HU19145705-nst), all previous data 
> lost.
> 25-Mar 12:02 backup-sd JobId 47755: [SI0202] End of Volume "LTO40026" at 
> 813:1 on device "HP Ultrium 4-2" (/dev/tape/by-id/scsi-HU19145705-nst). Write 
> of 64512 bytes got -1.
> 25-Mar 12:02 backup-sd JobId 47755: Re-read of last block succeeded.
> 25-Mar 12:02 backup-sd JobId 47755: End of medium on Volume "LTO40026" 
> Bytes=812,948,032,512 Blocks=12,601,500 at 25-Mar-2020 12:02.
> 25-Mar 12:02 backup-sd JobId 47755: 3307 Issuing autochanger "unload Volume 
> LTO40026, Slot 9, Drive 1" command.
> 25-Mar 12:04 backup-dir JobId 47755: Recycled volume "LTO40025"
> 25-Mar 12:04 backup-sd JobId 47755: 3301 Issuing autochanger "loaded? drive 
> 1" command.
> 25-Mar 12:04 backup-sd JobId 47755: 3302 Autochanger "loaded? drive 1", 
> result: nothing loaded.
> 25-Mar 12:04 backup-sd JobId 47755: 3304 Issuing autochanger "load Volume 
> LTO40025, Slot 17, Drive 1" command.
> 25-Mar 12:05 backup-sd JobId 47755: 3305 Autochanger "load Volume LTO40025, 
> Slot 17, Drive 1", status is OK.
> 25-Mar 12:05 backup-sd JobId 47755: Recycled volume "LTO40025" on Tape device 
> "HP Ultrium 4-2" (/dev/tape/by-id/scsi-HU19145705-nst), all previous data 
> lost.
> 25-Mar 12:05 backup-sd JobId 47755: New volume "LTO40025" mounted on device 
> "HP Ultrium 4-2" (/dev/tape/by-id/scsi-HU19145705-nst) at 25-Mar-2020 12:05.
> 25-Mar 14:01 backup-sd JobId 47755: Elapsed time=06:06:37, Transfer 
> rate=49.20 M Bytes/second
> 25-Mar 14:01 backup-sd JobId 47755: Sending spooled attrs to the Director. 
> Despooling 300,838,146 bytes ...
> 
> 
> Migration job:
> 
> 
> 25-Mar 07:48 backup-dir JobId 47754: The following 1 JobId was chosen to be 
> migrated: 47704
> 25-Mar 07:48 backup-dir JobId 47754: Migration using JobId=47704 
> Job=nihilnihil_home.2020-03-21_20.23.31_49
> 25-Mar 07:48 backup-dir JobId 47754: Start Migration JobId 47754, 
> Job=MigrateFile2Drive.2020-03-25_07.48.01_43
> 25-Mar 07:48 backup-dir JobId 47754: Using Device "DiskStorage2" to read.
> 25-Mar 07:51 backup-sd JobId 47754: Ready to read from volume "DISK016" on 
> File device "DiskStorage2" (/media/baculadisk2).
> 25-Mar 07:51 backup-sd JobId 47754: Forward spacing Volume "DISK016" to 
> addr=217
> 25-Mar 12:59 backup-sd JobId 47754: Error: block.c:682 [SE0208] Volume data 
> error at 0:0! Short block of 57010 bytes on device "DiskStorage2" 
> (/media/baculadisk2) discarded.
> 25-Mar 12:59 backup-sd JobId 47754: Error: read_records.c:160 block.c:682 
> [SE0208] Volume data error at 0:0! Short block of 57010 bytes on device 
> "DiskStorage2" (/media/baculadisk2) discarded.
> 25-Mar 12:59 backup-sd JobId 47754: End of Volume "DISK016" at 
> addr=972406571008 on device "DiskStorage2" (/media/baculadisk2).
> 25-Mar 13:00 backup-sd JobId 47754: Ready to read from volume "DISK017" on 
> File device "DiskStorage2" (/media/baculadisk2).
> 25-Mar 13:00 backup-sd JobId 47754: Forward spacing Volume "DISK017" to 
> addr=213
> 25-Mar 13:59 backup-sd JobId 47754: End of Volume "DISK017" at 
> addr=110838477984 on device "DiskStorage2" (/media/baculadisk2).
> 25-Mar 13:59 backup-sd JobId 47754: Elapsed time=06:08:08, Transfer 
> rate=49.00 M Bytes/second
> 25-Mar 15:51 backup-dir JobId 47754: Warning: Found errors during the 
> migration process. The original job 47704 will be kept in the catalog and the 
> 

Re: [Bacula-users] Simple disk storage devices migrations/backups

[Josh Fisher]

On 3/25/2020 3:23 PM, Pierre Bernhardt wrote:

Am 24.03.20 um 22:39 schrieb Pierre Bernhardt:
Today I tried to migrate again the job which uses two disk files.
But now I tried to put both files in one directory (I used my second
bay to mount the DISK017 file) and used a symbolic link:

-rw-rw-r-- 1 bacula tape 972406571008 Mar 24 17:10 /media/baculadisk1/DISK017
-rw-rw-r-- 1 bacula tape 972406571008 Mar 22 07:47 /media/baculadisk2/DISK016
lrwxrwxrwx 1 root   root   26 Mar 25 07:32 /media/baculadisk2/DISK017 
-> /media/baculadisk1/D

By the way the migration situation doesn't have been changed:



And what is the Autochanger / Device configurations for the disk storage 
in bacula-sd.conf?





___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Simple disk storage devices migrations/backups

[Pierre Bernhardt]

Am 24.03.20 um 22:39 schrieb Pierre Bernhardt:
>
Today I tried to migrate again the job which uses two disk files.
But now I tried to put both files in one directory (I used my second
bay to mount the DISK017 file) and used a symbolic link:

-rw-rw-r-- 1 bacula tape 972406571008 Mar 24 17:10 /media/baculadisk1/DISK017
-rw-rw-r-- 1 bacula tape 972406571008 Mar 22 07:47 /media/baculadisk2/DISK016
lrwxrwxrwx 1 root   root   26 Mar 25 07:32 /media/baculadisk2/DISK017 
-> /media/baculadisk1/D

By the way the migration situation doesn't have been changed:

Here the messages from the backup and from the migration job:


Backup (Migration) Job:
25-Mar 07:48 backup-dir JobId 47755: Recycled volume "LTO40026"
25-Mar 07:48 backup-dir JobId 47755: Using Device "HP
Ultrium
4-2" to write.
25-Mar 07:48 backup-sd JobId 47755: 3307 Issuing autochanger "unload Volume 
LTO40030, Slot 8, Drive 1" command.
25-Mar 07:49 backup-sd JobId 47755: 3304 Issuing autochanger "load Volume 
LTO40026, Slot 9, Drive 1" command.
25-Mar 07:51 backup-sd JobId 47755: 3305 Autochanger "load Volume LTO40026, 
Slot 9, Drive 1", status is OK.
25-Mar 07:51 backup-sd JobId 47755: Recycled volume "LTO40026" on Tape device 
"HP Ultrium 4-2" (/dev/tape/by-id/scsi-HU19145705-nst), all previous data lost.
25-Mar 12:02 backup-sd JobId 47755: [SI0202] End of Volume "LTO40026" at 813:1 
on device "HP Ultrium 4-2" (/dev/tape/by-id/scsi-HU19145705-nst). Write of 
64512 bytes got -1.
25-Mar 12:02 backup-sd JobId 47755: Re-read of last block succeeded.
25-Mar 12:02 backup-sd JobId 47755: End of medium on Volume "LTO40026" 
Bytes=812,948,032,512 Blocks=12,601,500 at 25-Mar-2020 12:02.
25-Mar 12:02 backup-sd JobId 47755: 3307 Issuing autochanger "unload Volume 
LTO40026, Slot 9, Drive 1" command.
25-Mar 12:04 backup-dir JobId 47755: Recycled volume "LTO40025"
25-Mar 12:04 backup-sd JobId 47755: 3301 Issuing autochanger "loaded? drive 1" 
command.
25-Mar 12:04 backup-sd JobId 47755: 3302 Autochanger "loaded? drive 1", result: 
nothing loaded.
25-Mar 12:04 backup-sd JobId 47755: 3304 Issuing autochanger "load Volume 
LTO40025, Slot 17, Drive 1" command.
25-Mar 12:05 backup-sd JobId 47755: 3305 Autochanger "load Volume LTO40025, 
Slot 17, Drive 1", status is OK.
25-Mar 12:05 backup-sd JobId 47755: Recycled volume "LTO40025" on Tape device 
"HP Ultrium 4-2" (/dev/tape/by-id/scsi-HU19145705-nst), all previous data lost.
25-Mar 12:05 backup-sd JobId 47755: New volume "LTO40025" mounted on device "HP 
Ultrium 4-2" (/dev/tape/by-id/scsi-HU19145705-nst) at 25-Mar-2020 12:05.
25-Mar 14:01 backup-sd JobId 47755: Elapsed time=06:06:37, Transfer rate=49.20 
M Bytes/second
25-Mar 14:01 backup-sd JobId 47755: Sending spooled attrs to the Director. 
Despooling 300,838,146 bytes ...


Migration job:


25-Mar 07:48 backup-dir JobId 47754: The following 1 JobId was chosen to be 
migrated: 47704
25-Mar 07:48 backup-dir JobId 47754: Migration using JobId=47704 
Job=nihilnihil_home.2020-03-21_20.23.31_49
25-Mar 07:48 backup-dir JobId 47754: Start Migration JobId 47754, 
Job=MigrateFile2Drive.2020-03-25_07.48.01_43
25-Mar 07:48 backup-dir JobId 47754: Using Device "DiskStorage2" to read.
25-Mar 07:51 backup-sd JobId 47754: Ready to read from volume "DISK016" on File 
device "DiskStorage2" (/media/baculadisk2).
25-Mar 07:51 backup-sd JobId 47754: Forward spacing Volume "DISK016" to addr=217
25-Mar 12:59 backup-sd JobId 47754: Error: block.c:682 [SE0208] Volume data 
error at 0:0! Short block of 57010 bytes on device "DiskStorage2" 
(/media/baculadisk2) discarded.
25-Mar 12:59 backup-sd JobId 47754: Error: read_records.c:160 block.c:682 
[SE0208] Volume data error at 0:0! Short block of 57010 bytes on device 
"DiskStorage2" (/media/baculadisk2) discarded.
25-Mar 12:59 backup-sd JobId 47754: End of Volume "DISK016" at 
addr=972406571008 on device "DiskStorage2" (/media/baculadisk2).
25-Mar 13:00 backup-sd JobId 47754: Ready to read from volume "DISK017" on File 
device "DiskStorage2" (/media/baculadisk2).
25-Mar 13:00 backup-sd JobId 47754: Forward spacing Volume "DISK017" to addr=213
25-Mar 13:59 backup-sd JobId 47754: End of Volume "DISK017" at 
addr=110838477984 on device "DiskStorage2" (/media/baculadisk2).
25-Mar 13:59 backup-sd JobId 47754: Elapsed time=06:08:08, Transfer rate=49.00 
M Bytes/second
25-Mar 15:51 backup-dir JobId 47754: Warning: Found errors during the migration 
process. The original job 47704 will be kept in the catalog and the Migration 
job will be marked in Error
25-Mar 15:51 backup-dir JobId 47754: Error: bsock.c:388 Wrote 4 bytes to 
Storage daemon:backup.localnet.cosmicstars.de:9103, but only 0 accepted.
25-Mar 15:51 backup-dir JobId 47754: Error: Bacula backup-dir 9.4.2 (04Feb19):
  Build OS:   x86_64-pc-linux-gnu debian buster/sid
  Prev Backup JobId:  47704
  Prev Backup Job:nihilnihil_home.2020-03-21_20.23.31_49
  New Backup JobId:   47755
  Current JobId:  47754
  Current Job:

[Bacula-users] Simple disk storage devices migrations/backups

[Pierre Bernhardt]

Hello,

I try to migrate a bigger job from three tapes to two disks.
I use a USB3-SATA Disk swapping station which have at all the same SCSI-ID
so I can use same /dev-node.
I use a mountpoint /median/baculadisk2
Each Disk has only write rights by root user so the bacula user can only
write on files which allready exists.
The Labeled file instead has write/read rights for bacula:tape so
bacula can write, clear and read the file, but cannot create new
files in this mount point.
To label a new disk I create a fs, mount it in /baculadisk2
and create a empty file like DISK018 and set the permissions
-rw-rw for bacula:tape.
A label command for the diskstorage2 device with name DISK018
will be then finished successfully and the file can be filled up
as long enough space is available on the disk.

If a job will needs more space than is available, the job will be
fill up the disk and then request for a new volume. So I unmount
the volume in bacula by unmounting the configured storage device.
Than I put out the USB-3 plug, switch of the bay, replace the
disk by another one (e.g. a labeled disk), switch on the bay and
plug in the USB connector.
Then the disk inode is created so a bacula mount command will
mount the filesystem and the labeled file is available in
/media/baculadisk2. The backup/migrate job will continue in
this volume (file) as long the backup/migration is finished.

Problem:
I want to migrate the job now back from the two disks to tapes,
so I configured a job which should do it by selecting jobid.

For first all looks fine: The jobs starts and bacula asks for
the first of the two disks. I mount the requested disks and
the migration starts.
After a couple of ours the job arrives the end of the file
(volume) and asks for the next disk.
I umount the 1st and mount the 2nd disk and the job continues.

By the way the migration job ends with an error which looks like
identified by a problem which was produced at the end of
the first disk, because the last block was truncated?
If so I wonder me that the migration to the disk has been
worked without problems.

Here the jobs output which I got via mail. First the migration
from tape to disk then back from disk to tape.
Maybe you can explain what the problem is.
Maybe it is only a single random problem hopefully it is not
a problem because of truncated last block of 1st disk.
In my opinion like on tapes it should reread and than rewritten
to the 2nd disk.




21-Mar 20:23 backup-dir JobId 47703: The following 1 JobId was chosen to be 
migrated: 46802
21-Mar 20:23 backup-dir JobId 47703: Migration using JobId=46802 
Job=nihilnihil_home.2020-01-05_23.50.01_29
21-Mar 20:23 backup-dir JobId 47703: Start Migration JobId 47703, 
Job=Migrate2FileTmpVol.2020-03-21_20.23.31_48
21-Mar 20:23 backup-dir JobId 47703: Using Device "HP
Ultrium
4-2" to read.
21-Mar 20:23 backup-sd JobId 47703: 3307 Issuing autochanger "unload Volume 
LTO40027, Slot 5, Drive 1" command.
21-Mar 20:27 backup-sd JobId 47703: 3304 Issuing autochanger "load Volume 
LTO40026, Slot 9, Drive 1" command.
21-Mar 20:28 backup-sd JobId 47703: 3305 Autochanger "load Volume LTO40026, 
Slot 9, Drive 1", status is OK.
21-Mar 20:28 backup-sd JobId 47703: Ready to read from volume "LTO40026" on 
Tape device "HP Ultrium 4-2" (/dev/tape/by-id/scsi-HU19145705-nst).
21-Mar 20:28 backup-sd JobId 47703: Forward spacing Volume "LTO40026" to 
addr=19:1457
22-Mar 00:36 backup-sd JobId 47703: End of Volume "LTO40026" at addr=628:5763 
on device "HP Ultrium 4-2" (/dev/tape/by-id/scsi-HU19145705-nst).
22-Mar 00:37 backup-sd JobId 47703: 3307 Issuing autochanger "unload Volume 
LTO40026, Slot 9, Drive 1" command.
22-Mar 00:38 backup-sd JobId 47703: 3304 Issuing autochanger "load Volume 
LTO40025, Slot 17, Drive 1" command.
22-Mar 00:40 backup-sd JobId 47703: 3305 Autochanger "load Volume LTO40025, 
Slot 17, Drive 1", status is OK.
22-Mar 00:40 backup-sd JobId 47703: Ready to read from volume "LTO40025" on 
Tape device "HP Ultrium 4-2" (/dev/tape/by-id/scsi-HU19145705-nst).
22-Mar 00:40 backup-sd JobId 47703: Forward spacing Volume "LTO40025" to 
addr=1:11779
22-Mar 05:33 backup-sd JobId 47703: End of Volume "LTO40025" at addr=0:0 on 
device "HP Ultrium 4-2" (/dev/tape/by-id/scsi-HU19145705-nst).
22-Mar 05:33 backup-sd JobId 47703: 3307 Issuing autochanger "unload Volume 
LTO40025, Slot 17, Drive 1" command.
22-Mar 05:34 backup-sd JobId 47703: 3304 Issuing autochanger "load Volume 
LTO40027, Slot 5, Drive 1" command.
22-Mar 05:36 backup-sd JobId 47703: 3305 Autochanger "load Volume LTO40027, 
Slot 5, Drive 1", status is OK.
22-Mar 05:36 backup-sd JobId 47703: Ready to read from volume "LTO40027" on 
Tape device "HP Ultrium 4-2" (/dev/tape/by-id/scsi-HU19145705-nst).
22-Mar 05:36 backup-sd JobId 47703: Forward spacing Volume "LTO40027" to 
addr=0:1
22-Mar 10:32 backup-sd JobId 47703: End of Volume "LTO40027" at addr=298:0 on 
device "HP Ultrium 4-2" (/dev/tape/by-id/scsi-HU19145705-nst).
22-Mar 10:32 backup-sd JobId 47703: