Re: [Bacula-users] Restore job forward space volume - usb file volume
Yes you right I reload the volume with bscan. And the bsr is created at the restore stage. I've now understood the high value of the bsr file. That's why some of my restore are quicker than other. And big volumes don't help it. Thanks a lot. -Message d'origine- De : Martin Simmons Envoyé : lundi 4 septembre 2023 19:16 À : Lionel PLASSE Cc : rados...@korzeniewski.net; bacula-users@lists.sourceforge.net Objet : Re: [Bacula-users] Restore job forward space volume - usb file volume I think the problem is the VolAddr=0-1239074631492 line in the bsr file. Normally it would be a more precise range, but that range will be slow. Was bscan used to reload data into the catalog for this volume some in the past? That can cause this problem with VolAddr. __Martin ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Restore job forward space volume - usb file volume
1642266413 FI=1 > > BUSTER_BKP-sd: match_bsr.c:498-9868 Enter match_all > > BUSTER_BKP-sd: match_bsr.c:618-9868 OK match_volume=T1 > > BUSTER_BKP-sd: match_bsr.c:508-9868 OK bsr match bsr_vol=T1 read_vol=T1 > > BUSTER_BKP-sd: match_bsr.c:706-9868 match_voladdr: saddr=0 > eaddr=1239074631492 recaddr=64728 sfile=0 efile=288 recfile=0 > > BUSTER_BKP-sd: match_bsr.c:709-9868 OK match voladdr=64728 > > BUSTER_BKP-sd: match_bsr.c:521-9868 Fail on sesstime. bsr=1649883149 > rec=1642266413 > > BUSTER_BKP-sd: match_bsr.c:608-9868 Leave match all 0 > > > > > > > > The SD trace for a newer backup job restored instantly : > > > > BUSTER_BKP-sd: label.c:191-9867 VolHdr.VerNum=11 OK. > > BUSTER_BKP-sd: label.c:211-9867 Compare Vol names: VolName=HEBDO1 hdr=HEBDO1 > > > > Volume Label: > > Adata : 0 > > Id: Bacula 1.0 immortal > > VerNo : 11 > > VolName : HEBDO1 > > PrevVolName : > > VolFile : 0 > > LabelType : VOL_LABEL > > LabelSize : 184 > > PoolName : DIFF > > MediaType : USB > > PoolType : Backup > > HostName : BUSTER-BKP > > Date label written: 14-janv.-2022 19:58 > > BUSTER_BKP-sd: label.c:261-9867 Leave read_volume_label() VOL_OK > > BUSTER_BKP-sd: file_dev.c:71-9867 Enter: virtual bool DEVICE::rewind(DCR*) > > BUSTER_BKP-sd: label.c:274-9867 Call reserve_volume=HEBDO1 > > BUSTER_BKP-sd: vol_mgr.c:381-9867 enter reserve_volume=HEBDO1 drive="BACKUP" > (/BKP/bacubak) > > BUSTER_BKP-sd: vol_mgr.c:286-9867 new Vol=HEBDO1 slot=0 at 7fb444071538 > dev="BACKUP" (/BKP/bacubak) > > BUSTER_BKP-sd: vol_mgr.c:548-9867 set in_use. vol=HEBDO1 dev="BACKUP" > (/BKP/bacubak) > > BUSTER_BKP-sd: label.c:289-9867 Leave: virtual int > DEVICE::read_dev_volume_label(DCR*) > > BUSTER_BKP-sd: acquire.c:246-9867 Got correct volume. VOL_OK: HEBDO1 > > BUSTER_BKP-sd: acquire.c:354-9867 dcr=7fb444070378 dev=7fb44c001318 > > BUSTER_BKP-sd: acquire.c:355-9867 MediaType dcr=USB dev=USB > > BUSTER_BKP-sd: acquire.c:357-9867 Leave: bool acquire_device_for_read(DCR*) > > BUSTER_BKP-sd: read_records.c:436-9867 Enter: BSR* > position_to_first_file(JCR*, DCR*, BSR*) > > BUSTER_BKP-sd: match_bsr.c:231-9867 use_pos=1 repos=1 > > BUSTER_BKP-sd: match_bsr.c:618-9867 OK match_volume=HEBDO1 > > BUSTER_BKP-sd: read_records.c:451-9867 pos_to_first_file from addr=0 to > 3634588568 > > BUSTER_BKP-sd: file_dev.c:107-9867 = lseek to 3634588568 > > BUSTER_BKP-sd: read_records.c:455-9867 Leave: BSR* > position_to_first_file(JCR*, DCR*, BSR*) > > BUSTER_BKP-sd: block.c:520-9867 Pos for read=3634588568 3634588568 > > BUSTER_BKP-sd: block.c:545-9867 Read() adata=0 vol=HEBDO1 nbytes=256000 > pos=3634588568 > > BUSTER_BKP-sd: block_util.c:456-9867 set block=7fb44406fc40 adata=0 > binbuf=255976 > > > > > > De : Radosław Korzeniewski > Envoyé : jeudi 31 août 2023 18:30 > À : Lionel PLASSE > Cc : Martin Simmons ; bacula-users@lists.sourceforge.net > Objet : Re: [Bacula-users] Restore job forward space volume - usb file volume > > > > Hi, > > > > pon., 28 sie 2023 o 10:13 Lionel PLASSE <mailto:pla...@cofiem.fr> > napisał(a): > > You're right, I say it was slow from my own apreciation. > The Volume file is 1 TB. And the restoration job was 65 GB. > > And unfortunately, I don't have the exact timing for each steps , but I was > stuck in the "forwarding spacing " for half an hour before I stopped watching. > > > > Could you share a restore BSR file for this restore? At this file you can > check what parts of the volume will be "fast-forwarded" which for a file on a > block device (usb disk) means Bacula just executes lseek(2) which takes > almost "no time"; and what Bacula will read block by block. > > > > best regards > > -- > > Radosław Korzeniewski > rados...@korzeniewski.net <mailto:rados...@korzeniewski.net> > > ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Restore job forward space volume - usb file volume
Hi, pon., 28 sie 2023 o 10:13 Lionel PLASSE napisał(a): > You're right, I say it was slow from my own apreciation. > The Volume file is 1 TB. And the restoration job was 65 GB. > > And unfortunately, I don't have the exact timing for each steps , but I > was stuck in the "forwarding spacing " for half an hour before I stopped > watching. > Could you share a restore BSR file for this restore? At this file you can check what parts of the volume will be "fast-forwarded" which for a file on a block device (usb disk) means Bacula just executes lseek(2) which takes almost "no time"; and what Bacula will read block by block. best regards -- Radosław Korzeniewski rados...@korzeniewski.net ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Restore job forward space volume - usb file volume
It is unlikely that smaller volumes will be faster. I don't see any evidence that it depends on the position of the data. How long did the backup of these file take to run? External USB drives can be slow, especially if connected by USB 2.0. __Martin >>>>> On Mon, 28 Aug 2023 07:53:06 +, Lionel PLASSE said: > > You're right, I say it was slow from my own apreciation. > The Volume file is 1 TB. And the restoration job was 65 GB. > > And unfortunately, I don't have the exact timing for each steps , but I was > stuck in the "forwarding spacing " for half an hour before I stopped watching. > > Elapsed time: 1 hour 16 mins 30 secs > Files Expected: 26 > Files Restored: 26 > Bytes Restored: 63,581,189,026 (63.58 GB) > Rate: 13852.1 KB/s > FD Errors: 0 > FD termination status: OK > SD termination status: OK > Termination:Restore OK > BUSTER_BKP-sd JobId 9784: Elapsed time=01:11:20, Transfer rate=14.85 M > Bytes/second > BUSTER_BKP-sd JobId 9784: Forward spacing Volume "T1" to addr=249122650420 > > Ok, But in my memories this step o was much faster to perform ? > > So, as I understand it, it depends on the position of the data in the restore > job in the volume. > Can it be faster if I use smaller but more numerous volumes (if a smaller > volume contains all the data of the restore job at once of course) > > The overall Rate is good for me. > > Thank for informations, > > > -Message d'origine- > De : Martin Simmons > Envoyé : vendredi 25 août 2023 18:05 > À : Lionel PLASSE > Cc : bacula-users@lists.sourceforge.net > Objet : Re: [Bacula-users] Restore job forward space volume - usb file volume > >>>>> On Thu, 24 Aug 2023 09:32:38 +, Lionel PLASSE said: > > > > Hello, > > > > For usb harddrive and file media volume, when I do a restore job I get a > > long waiting step : "Forward spacing Volume "T1" to addr=249122650420" > > I remember I managed to configure the storage resource to quickly restore > > sdd drives. > > > > Should I use fastforward, blockpositionning and HardwareEndOfFile for USB > > disks and file volume? > > How to avoid this long forwarding when not a tape device? > > How do you know it is slow in the forwarding? It might have done that > quickly and is slow in the next operation. > > How much data are you restoring? > > You can try using 'status storage' and 'status client' a few times in > bconsole to monitor the progress of the restore. > > __Martin > ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Restore job forward space volume - usb file volume
You're right, I say it was slow from my own apreciation. The Volume file is 1 TB. And the restoration job was 65 GB. And unfortunately, I don't have the exact timing for each steps , but I was stuck in the "forwarding spacing " for half an hour before I stopped watching. Elapsed time: 1 hour 16 mins 30 secs Files Expected: 26 Files Restored: 26 Bytes Restored: 63,581,189,026 (63.58 GB) Rate: 13852.1 KB/s FD Errors: 0 FD termination status: OK SD termination status: OK Termination:Restore OK BUSTER_BKP-sd JobId 9784: Elapsed time=01:11:20, Transfer rate=14.85 M Bytes/second BUSTER_BKP-sd JobId 9784: Forward spacing Volume "T1" to addr=249122650420 Ok, But in my memories this step o was much faster to perform ? So, as I understand it, it depends on the position of the data in the restore job in the volume. Can it be faster if I use smaller but more numerous volumes (if a smaller volume contains all the data of the restore job at once of course) The overall Rate is good for me. Thank for informations, -Message d'origine- De : Martin Simmons Envoyé : vendredi 25 août 2023 18:05 À : Lionel PLASSE Cc : bacula-users@lists.sourceforge.net Objet : Re: [Bacula-users] Restore job forward space volume - usb file volume >>>>> On Thu, 24 Aug 2023 09:32:38 +, Lionel PLASSE said: > > Hello, > > For usb harddrive and file media volume, when I do a restore job I get a > long waiting step : "Forward spacing Volume "T1" to addr=249122650420" > I remember I managed to configure the storage resource to quickly restore sdd > drives. > > Should I use fastforward, blockpositionning and HardwareEndOfFile for USB > disks and file volume? > How to avoid this long forwarding when not a tape device? How do you know it is slow in the forwarding? It might have done that quickly and is slow in the next operation. How much data are you restoring? You can try using 'status storage' and 'status client' a few times in bconsole to monitor the progress of the restore. __Martin ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Restore job forward space volume - usb file volume
> On Fri, 25 Aug 2023 13:22:47 -0400, Josh Fisher said: > > On 8/25/23 12:06, Martin Simmons wrote: > >> On Thu, 24 Aug 2023 15:51:18 -0400, Josh Fisher via Bacula-users said: > >> > >>Probably you have compression and/or encryption turned on. In > >> that case, Bacula cannot simply fseek to the offset. It has to > >> decompress and/or decrypt all data in order to find it, making restores > >> far slower than backups. > > The compression and/or encryption is done within each block, so tnat doesn't > > affect seek time. > > > Interesting. So after decompression and decryption, does the > uncompressed/decrypted data contain a partial block(s), or are the > compressed/encrypted blocks originally written with variable block sizes > so that the original data is handled as fixed size blocks? Except for the last block in a backup, I think file volumes use a fixed block size (the default is 64512 bytes), but it doesn't really matter. The compression/encryption occurs within the records, which are then packed into blocks. If you run 'bls -v -v -k ..path.to.a.file.volume..' then it will show the blocks and records like this: Blk=0 blen=246 bVer=2 SessId=16 SessTim=1691737690 bls: block_util.c:105-0 Dump block 800d68200: adata=0 size=246 BlkNum=0 Hdrcksum=8adb2881 cksum=8adb2881 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=VOL_LABEL Strm=0 len=210 reclen=0 Blk=1 blen=64512 bVer=2 SessId=16 SessTim=1691737690 bls: block_util.c:105-0 Dump block 800d68200: adata=0 size=64512 BlkNum=1 Hdrcksum=d01c4958 cksum=d01c4958 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=SOS_LABEL Strm=98043 len=177 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=1 Strm=UATTR len=102 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=1 Strm=GZIP len=226 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=1 Strm=MD5 len=16 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=2 Strm=UATTR len=104 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=2 Strm=GZIP len=1633 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=2 Strm=MD5 len=16 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=3 Strm=UATTR len=113 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=3 Strm=GZIP len=5748 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=3 Strm=GZIP len=5978 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=3 Strm=GZIP len=4677 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=3 Strm=GZIP len=4378 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=3 Strm=GZIP len=5937 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=3 Strm=GZIP len=4950 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=3 Strm=GZIP len=6003 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=3 Strm=GZIP len=5222 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=3 Strm=GZIP len=4809 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=3 Strm=GZIP len=5911 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=3 Strm=GZIP len=5435 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=3 Strm=GZIP len=5339 reclen=0 Blk=2 blen=64512 bVer=2 SessId=16 SessTim=1691737690 bls: block_util.c:105-0 Dump block 800d68200: adata=0 size=64512 BlkNum=2 Hdrcksum=d28c863e cksum=d28c863e bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=3 Strm=contGZIP len=2526 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=3 Strm=GZIP len=5652 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=3 Strm=GZIP len=6024 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=3 Strm=GZIP len=6674 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=3 Strm=GZIP len=4829 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=3 Strm=MD5 len=16 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=4 Strm=UATTR len=121 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=4 Strm=GZIP len=19296 reclen=0 bls: block_util.c:130-0Rec: VId=16 VT=1691737690 FI=4 Strm=GZIP len=23650 reclen=0 __Martin ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Restore job forward space volume - usb file volume
On 8/25/23 12:06, Martin Simmons wrote: On Thu, 24 Aug 2023 15:51:18 -0400, Josh Fisher via Bacula-users said: Probably you have compression and/or encryption turned on. In that case, Bacula cannot simply fseek to the offset. It has to decompress and/or decrypt all data in order to find it, making restores far slower than backups. The compression and/or encryption is done within each block, so tnat doesn't affect seek time. Interesting. So after decompression and decryption, does the uncompressed/decrypted data contain a partial block(s), or are the compressed/encrypted blocks originally written with variable block sizes so that the original data is handled as fixed size blocks? __Martin ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Restore job forward space volume - usb file volume
> On Thu, 24 Aug 2023 15:51:18 -0400, Josh Fisher via Bacula-users said: > > On 8/24/23 05:32, Lionel PLASSE wrote: > > Hello, > > > > For usb harddrive and file media volume, when I do a restore job I get a > > long waiting step : "Forward spacing Volume "T1" to addr=249122650420" > > I remember I managed to configure the storage resource to quickly restore > > sdd drives. > > > > Should I use fastforward, blockpositionning and HardwareEndOfFile for USB > > disks and file volume? > > How to avoid this long forwarding when not a tape device? > > > I don't think any of those affect file type storage (random access > devices). Agreed. > Probably you have compression and/or encryption turned on. In > that case, Bacula cannot simply fseek to the offset. It has to > decompress and/or decrypt all data in order to find it, making restores > far slower than backups. The compression and/or encryption is done within each block, so tnat doesn't affect seek time. __Martin ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Restore job forward space volume - usb file volume
> On Thu, 24 Aug 2023 09:32:38 +, Lionel PLASSE said: > > Hello, > > For usb harddrive and file media volume, when I do a restore job I get a > long waiting step : "Forward spacing Volume "T1" to addr=249122650420" > I remember I managed to configure the storage resource to quickly restore sdd > drives. > > Should I use fastforward, blockpositionning and HardwareEndOfFile for USB > disks and file volume? > How to avoid this long forwarding when not a tape device? How do you know it is slow in the forwarding? It might have done that quickly and is slow in the next operation. How much data are you restoring? You can try using 'status storage' and 'status client' a few times in bconsole to monitor the progress of the restore. __Martin ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Restore job forward space volume - usb file volume
On 8/24/23 05:32, Lionel PLASSE wrote: Hello, For usb harddrive and file media volume, when I do a restore job I get a long waiting step : "Forward spacing Volume "T1" to addr=249122650420" I remember I managed to configure the storage resource to quickly restore sdd drives. Should I use fastforward, blockpositionning and HardwareEndOfFile for USB disks and file volume? How to avoid this long forwarding when not a tape device? I don't think any of those affect file type storage (random access devices). Probably you have compression and/or encryption turned on. In that case, Bacula cannot simply fseek to the offset. It has to decompress and/or decrypt all data in order to find it, making restores far slower than backups. . ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Restore job forward space volume - usb file volume
Hello, For usb harddrive and file media volume, when I do a restore job I get a long waiting step : "Forward spacing Volume "T1" to addr=249122650420" I remember I managed to configure the storage resource to quickly restore sdd drives. Should I use fastforward, blockpositionning and HardwareEndOfFile for USB disks and file volume? How to avoid this long forwarding when not a tape device? PLASSE Lionel | Administrateur systèmes et réseaux 221 Allée de Fétan 01600 TREVOUX | https://www.google.fr/maps/place/221+Allée+de+Fétan,+01600+Trévoux/@45.9445028,4.7581844,17z/data=!4m13!1m7!3m6!1s0x47f4906e98463259:0x32e33a7755284538!2zMjEyIEFsbMOpZSBkZSBGw6l0YW4sIDAxNjAwIFRyw6l2b3V4!3b1!8m2!3d45.9444281!4d4.7578088!3m4!1s0x47f4906e98463259:0x32e33a7755284538!8m2!3d45.9444281!4d4.7578088?hl=fr Tel : 04.37.49.91.39 pla...@cofiem.fr https://www.cofiem.fr/ | https://www.cofiem-robotics.fr/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
