Re: [Bacula-users] Virtual tapes or virtual disks
https://www.bacula.org/whitepapers/CommunityDiskBackup.pdf Other options is to use software for vtl, as you have been told: mhvtl or Quadstor. Each one with its compression algorithm. On Fri, Jan 21, 2022 at 9:23 AM Peter Milesson via Bacula-users < bacula-users@lists.sourceforge.net> wrote: > Hi folks, > > I am building a new backup server that is going to replace the old one. > The old backup server uses Bacula ver. 9.2.2 with a virtual tape library > (mhvtl). I have found mhvtl a bit tricky, mostly when updating the OS > (CentOS 7.8), as it is necessary to recompile the mhvtl kernel driver. > Mhvtl also seems a bit outdated, with intermittent development and > updates, also problematic with newer and coming Linux kernel versions. I > also want to jump off the RedHat train, as I see it deviate more and > more from mainstream Linux. Therefore, I would prefer to choose another > solution. > > I have studied the Bacula documentation and it seems to be possible to > use disk based backup with auto changer role. I plan to use volumes with > a size of around 200Gbytes, making the setup fairly flexible, not making > a too big hole when volumes need to be purged. Total disk space is > around 30TB. > > If somebody has got experience with disk based, multi volume Bacula > backup, I would be grateful about some information (tips, what to > expect, pitfalls, etc.). > > Best regards, > > Peter > > > ___ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users > -- # # Sistema Operativo: Debian # #Caracas, Venezuela # # ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
On 1/27/22 14:45, Pedro Oliveira wrote: Why not use mhtvl or quadstor https://quadstor.com/virtual-tape-library.html http://www.mhvtl.com they are nice solutions and integrate well with Bacula It depends on what is needed. These VTLs are more complex and use a kernel module that I don't think is in the mainline kernel. They are not as convenient as vchanger for backing up to removable hot-swap media. For backing up to a single filesystem, Bacula has a native disk autochanger. On the other hand, they can be used as a iSCSI-attached SAN device for large networks with multiple subnets and multiple Bacula Storage Daemons. It just depends on the environment. Josh Fisher escreveu em qui., 27/01/2022 às 18:50 : On 1/26/22 12:42, dmitri maziuk wrote: > On 2022-01-26 11:06 AM, Peter Milesson via Bacula-users wrote: >> > ... >> Your way of explaining the reasoning of why to use smaller file >> volumes, is very appreciated. > ... > >> The only thing I haven't found out is how to preallocate the number >> of volumes needed. Maybe there is no need, if the volumes are created >> automagically. Most of the RAID array will be used by Bacula, just >> leaving a couple of percent as free space. > > If you use actual disks as "magazines" with vchanger, you need to > pre-label the volumes. If you use just one big filesystem, you can let > bacula do it for you (last I looked that functionality didn't work w/ > autochangers). Automatic labeling doesn't work, but vchanger supports barcodes like a tape autochanger. The virtual barcodes are just the filenames of the volume files on the disk "magazine". So the 'label barcodes' command works with vchanger the same as a tape autochanger. The vchanger 'createvols' command both creates the volume files and then invokes bconsole and issues the label barcodes command. A single command creates and labels the volume files, so it is not so bad. > > If you use disk "magazines" you also need to consider the whole-disk > failure. If you use one big filesystem, use RAID (of course) to guard > against those. But then you should look at the number of file volumes: > some filesystems handle large numbers of directory entries better than > others and you may want to balance the volume file size vs the number > of directory entries. That is true if physical disks are used as magazines. Vchanger mostly targets removable drives, such as USB, RDX, or hot-swap SAS JBOD. In that case, exposure to whole-disk failure is mitigated by having multiple backups and using Copy jobs. But it is also possible to use vchanger with iSCSI volumes as magazines. I use iSCSI for some magazines and portable USB drives for other magazines that are used for offline and off-site storage. The iSCSI volumes are on RAID10/LVM, but they could easily be ZFS volumes too. > > For single filesystem, I suggest using ZFS instead of a traditional > RAID if you can: you can later grow it on-line by replacing disks w/ > bigger ones when (not if) you need to. > > Dima > > > ___ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- -- Cumprimentos, Pedro Oliveira Rua Antonio Botto | Nº 23 | 1º A | 2950-565 Quinta do Anjo tel +351 218 440 100 | mobile +351 916 111 464 *Aviso de Confidencialidade:* Esta mensagem é exclusivamente destinada ao seu destinatário, podendo conter informação CONFIDENCIAL, cuja divulgação está expressamente vedada nos termos da lei. Caso tenha recepcionado indevidamente esta mensagem, solicitamos-lhe que nos comunique esse mesmo facto por esta via ou para o telefone +351 916111464 devendo apagar o seu conteúdo de imediato. This message is intended exclusively for its addressee. It may contain CONFIDENTIAL information protected by law. If this message has been received by error, please notify us via e-mail or by telephone +351916111464 and delete it immediately. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
Why not use mhtvl or quadstor https://quadstor.com/virtual-tape-library.html http://www.mhvtl.com they are nice solutions and integrate well with Bacula Josh Fisher escreveu em qui., 27/01/2022 às 18:50 : > > On 1/26/22 12:42, dmitri maziuk wrote: > > On 2022-01-26 11:06 AM, Peter Milesson via Bacula-users wrote: > >> > > ... > >> Your way of explaining the reasoning of why to use smaller file > >> volumes, is very appreciated. > > ... > > > >> The only thing I haven't found out is how to preallocate the number > >> of volumes needed. Maybe there is no need, if the volumes are created > >> automagically. Most of the RAID array will be used by Bacula, just > >> leaving a couple of percent as free space. > > > > If you use actual disks as "magazines" with vchanger, you need to > > pre-label the volumes. If you use just one big filesystem, you can let > > bacula do it for you (last I looked that functionality didn't work w/ > > autochangers). > > > Automatic labeling doesn't work, but vchanger supports barcodes like a > tape autochanger. The virtual barcodes are just the filenames of the > volume files on the disk "magazine". So the 'label barcodes' command > works with vchanger the same as a tape autochanger. The vchanger > 'createvols' command both creates the volume files and then invokes > bconsole and issues the label barcodes command. A single command creates > and labels the volume files, so it is not so bad. > > > > > > If you use disk "magazines" you also need to consider the whole-disk > > failure. If you use one big filesystem, use RAID (of course) to guard > > against those. But then you should look at the number of file volumes: > > some filesystems handle large numbers of directory entries better than > > others and you may want to balance the volume file size vs the number > > of directory entries. > > > That is true if physical disks are used as magazines. Vchanger mostly > targets removable drives, such as USB, RDX, or hot-swap SAS JBOD. In > that case, exposure to whole-disk failure is mitigated by having > multiple backups and using Copy jobs. > > But it is also possible to use vchanger with iSCSI volumes as magazines. > I use iSCSI for some magazines and portable USB drives for other > magazines that are used for offline and off-site storage. The iSCSI > volumes are on RAID10/LVM, but they could easily be ZFS volumes too. > > > > > > For single filesystem, I suggest using ZFS instead of a traditional > > RAID if you can: you can later grow it on-line by replacing disks w/ > > bigger ones when (not if) you need to. > > > > Dima > > > > > > ___ > > Bacula-users mailing list > > Bacula-users@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/bacula-users > > > ___ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users > -- -- Cumprimentos, Pedro Oliveira Rua Antonio Botto | Nº 23 | 1º A | 2950-565 Quinta do Anjo tel +351 218 440 100 | mobile +351 916 111 464 *Aviso de Confidencialidade:* Esta mensagem é exclusivamente destinada ao seu destinatário, podendo conter informação CONFIDENCIAL, cuja divulgação está expressamente vedada nos termos da lei. Caso tenha recepcionado indevidamente esta mensagem, solicitamos-lhe que nos comunique esse mesmo facto por esta via ou para o telefone +351 916111464 devendo apagar o seu conteúdo de imediato. This message is intended exclusively for its addressee. It may contain CONFIDENTIAL information protected by law. If this message has been received by error, please notify us via e-mail or by telephone +351916111464 and delete it immediately. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
On 1/26/22 12:42, dmitri maziuk wrote: On 2022-01-26 11:06 AM, Peter Milesson via Bacula-users wrote: ... Your way of explaining the reasoning of why to use smaller file volumes, is very appreciated. ... The only thing I haven't found out is how to preallocate the number of volumes needed. Maybe there is no need, if the volumes are created automagically. Most of the RAID array will be used by Bacula, just leaving a couple of percent as free space. If you use actual disks as "magazines" with vchanger, you need to pre-label the volumes. If you use just one big filesystem, you can let bacula do it for you (last I looked that functionality didn't work w/ autochangers). Automatic labeling doesn't work, but vchanger supports barcodes like a tape autochanger. The virtual barcodes are just the filenames of the volume files on the disk "magazine". So the 'label barcodes' command works with vchanger the same as a tape autochanger. The vchanger 'createvols' command both creates the volume files and then invokes bconsole and issues the label barcodes command. A single command creates and labels the volume files, so it is not so bad. If you use disk "magazines" you also need to consider the whole-disk failure. If you use one big filesystem, use RAID (of course) to guard against those. But then you should look at the number of file volumes: some filesystems handle large numbers of directory entries better than others and you may want to balance the volume file size vs the number of directory entries. That is true if physical disks are used as magazines. Vchanger mostly targets removable drives, such as USB, RDX, or hot-swap SAS JBOD. In that case, exposure to whole-disk failure is mitigated by having multiple backups and using Copy jobs. But it is also possible to use vchanger with iSCSI volumes as magazines. I use iSCSI for some magazines and portable USB drives for other magazines that are used for offline and off-site storage. The iSCSI volumes are on RAID10/LVM, but they could easily be ZFS volumes too. For single filesystem, I suggest using ZFS instead of a traditional RAID if you can: you can later grow it on-line by replacing disks w/ bigger ones when (not if) you need to. Dima ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
On 2022-01-26 20:13, dmitri maziuk wrote: On 2022-01-26 12:57 PM, Josip Deanovic wrote: The number of files per directory is far bigger and is unlikely to get reached, especially not for this use case. The limit is one thing, the scaling is another. I agree: 40TB of 10GB files is not enough to see the slow-down on any modern system, you'd need an order of magnitude more files to get there. Still it's something to be aware of when deciding on volume size. 40 TB is 40960 GB which would give 4096 files, 10 GB in size. Order of magnitude would be 40960 files which is still nothing. Right now on my laptop I have 291794 files and 34481 directories and that's only under /usr. I had systems with hundreds of millions of files on UFS2 (FreeBSD) and systems with billions of files on ext3 (Linux) and that was like 15 years ago. As far as I can remember there were no issues with read/write performance related to the number of files. The issue was backup which would take a lot of time to traverse the whole file system. This is a problem common to all hierarchical databases without some kind of indexing employed to deal with the issue. As long the full path of a file is known, I don't think the read/write performance of a file would change noticeably with the increase of number of files on the file system. Modern file systems are using directory indexing so even searching through a file system doesn't take too long but it's common sense that the time needed to perform a lookup would increase (not necessary linearly) with the number of files on the file system. In any case, Bacula knows the path names of the file volumes and doesn't need to search the file system. I can't imagine the setup where the number of files on the local file system containing Bacula file volumes would pose a problem. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
On 2022-01-26 12:57 PM, Josip Deanovic wrote: The number of files per directory is far bigger and is unlikely to get reached, especially not for this use case. The limit is one thing, the scaling is another. I agree: 40TB of 10GB files is not enough to see the slow-down on any modern system, you'd need an order of magnitude more files to get there. Still it's something to be aware of when deciding on volume size. Dima ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
I'm having a RAID5 array of about 40TB in size. A separate RAID controller card handles the disks. I'm planning to use the normal ext4 file system. It's standard and well known, most probably not the fastest though. That will not have any great impact, as there is a 4TB NVMe SSD drive, which takes the odd of the slow physical disk performance. Hi, I'd recommend if you're going to use RAID that you at least use a RAID-6 configuration. You don't want to risk losing all your backups if you have a drive fail and then during the rebuilding of the RAID-5, you happen to have another drive failure/error. cheers, --tom ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
On 2022-01-26 18:42, dmitri maziuk wrote: If you use actual disks as "magazines" with vchanger, you need to pre-label the volumes. If you use just one big filesystem, you can let bacula do it for you (last I looked that functionality didn't work w/ autochangers). If you use disk "magazines" you also need to consider the whole-disk failure. If you use one big filesystem, use RAID (of course) to guard against those. But then you should look at the number of file volumes: some filesystems handle large numbers of directory entries better than others and you may want to balance the volume file size vs the number of directory entries. Regarding the number of directory entries... It is common to see the file system limit of 32000 directories per directory. The number of files per directory is far bigger and is unlikely to get reached, especially not for this use case. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
On 2022-01-26 11:59 AM, Peter Milesson via Bacula-users wrote: ... I'm having a RAID5 array of about 40TB in size. A separate RAID controller card handles the disks. I'm planning to use the normal ext4 file system. It's standard and well known, most probably not the fastest though. That will not have any great impact, as there is a 4TB NVMe SSD drive, which takes the odd of the slow physical disk performance. Yeah, we gave up on hardware RAID controllers long ago, but YMMV. As for SSDs, if you spool the jobs you can run them in parallel to spool->volume stream. You'd have to look at the numbers for your setup but generally despooling off the SSD over the bus runs just fine while clients are spooling to it over the network. Dima ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
On 2022-01-26 18:06, Peter Milesson via Bacula-users wrote: I'm used to fixed volume sizes from the tape drives, I feel comfortable with it, and I do not need to relearn a lot to configure the Bacula system. The only thing I haven't found out is how to preallocate the number of volumes needed. Maybe there is no need, if the volumes are created automagically. Most of the RAID array will be used by Bacula, just leaving a couple of percent as free space. When using mhvtl, I started a script with the tape size and number of tapes I wanted, and the corresponding tape directories and volumes were created on the fly. Thanks Josip! You are welcome. I would like to point out that different requirements people may have will dictate different approaches. Regarding preallocation of the voluems, if there is a way to do it I am not aware of it. However, if you define maximum volume size and the maximum number of volumes in the pool, you should be able to calculate the space needed. Just leave some free space like 2x size of a volume and you should be good. Later, when you use all the volumes you will see if there is enough space to create yet another volume. You can chose to label volumes by yourself or leave that to Bacula. It's up to you. If you intend to recycle your volumes automatically, make sure that your retention periods are short enough to expire before all the volumes are used. Otherwise Bacula will not be able to perform backup. The alternative would be to force the recycle of the oldest volume but this doesn't happen by default, this option must be explicitly turned on. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
On 26.01.2022 18:42, dmitri maziuk wrote: On 2022-01-26 11:06 AM, Peter Milesson via Bacula-users wrote: ... Your way of explaining the reasoning of why to use smaller file volumes, is very appreciated. ... The only thing I haven't found out is how to preallocate the number of volumes needed. Maybe there is no need, if the volumes are created automagically. Most of the RAID array will be used by Bacula, just leaving a couple of percent as free space. If you use actual disks as "magazines" with vchanger, you need to pre-label the volumes. If you use just one big filesystem, you can let bacula do it for you (last I looked that functionality didn't work w/ autochangers). If you use disk "magazines" you also need to consider the whole-disk failure. If you use one big filesystem, use RAID (of course) to guard against those. But then you should look at the number of file volumes: some filesystems handle large numbers of directory entries better than others and you may want to balance the volume file size vs the number of directory entries. For single filesystem, I suggest using ZFS instead of a traditional RAID if you can: you can later grow it on-line by replacing disks w/ bigger ones when (not if) you need to. Dima Thanks for your input Dima. I'm having a RAID5 array of about 40TB in size. A separate RAID controller card handles the disks. I'm planning to use the normal ext4 file system. It's standard and well known, most probably not the fastest though. That will not have any great impact, as there is a 4TB NVMe SSD drive, which takes the odd of the slow physical disk performance. Best regards, Peter ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
On 2022-01-26 11:06 AM, Peter Milesson via Bacula-users wrote: ... Your way of explaining the reasoning of why to use smaller file volumes, is very appreciated. ... The only thing I haven't found out is how to preallocate the number of volumes needed. Maybe there is no need, if the volumes are created automagically. Most of the RAID array will be used by Bacula, just leaving a couple of percent as free space. If you use actual disks as "magazines" with vchanger, you need to pre-label the volumes. If you use just one big filesystem, you can let bacula do it for you (last I looked that functionality didn't work w/ autochangers). If you use disk "magazines" you also need to consider the whole-disk failure. If you use one big filesystem, use RAID (of course) to guard against those. But then you should look at the number of file volumes: some filesystems handle large numbers of directory entries better than others and you may want to balance the volume file size vs the number of directory entries. For single filesystem, I suggest using ZFS instead of a traditional RAID if you can: you can later grow it on-line by replacing disks w/ bigger ones when (not if) you need to. Dima ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
On 26.01.2022 0:02, Josip Deanovic wrote: On 23.01.2022 11:37, Radosław Korzeniewski wrote: Hello, pt., 21 sty 2022 o 14:22 Peter Milesson via Bacula-users napisał(a): If somebody has got experience with disk based, multi volume Bacula backup, I would be grateful about some information (tips, what to expect, pitfalls, etc.). The best IMVHO (but not the only mine) is to configure one job = one volume. You will get no real benefit to limit the size of a single volume. In the single volume = single job configuration you can set up job retention very easily as purging a volume will purge a single job only. It is not required to "wait" a particular volume to fill up to start retention. Purging a volume affects a single job only. And finally you end up with a way less number of volumes then when limiting its size to i.e. 10G. There are many different approaches which can fit different requirements. I don't see the benefit of having a single job per volume as Bacula is tracking media, files, jobs and everything else. That's why Bacula has a catalog which allows the backup system to determine the location and state of volumes, jobs, files, etc. To logically separate backup data I use pools and leave the rest to Bacula. When Bacula needs a particular file volume, if it's available Bacula will simply use it and if it's not or if we are using tape volume which is currently not in the tape drive/library, Bacula will ask for the volume by name. The number of smaller file volumes (e.g. 10GB) is not an issue as Bacula is handling them correctly and automatically (provided that Bacula is correctly configured, of course). I'll go through few examples where smaller file volumes (e.g. 10GB) could prove useful: 1. If the catalog database get corrupted or completely lost, due to the the small size, it's easier and faster to handle and determine volumes which contain database backup. That makes the process of importing the data into a new catalog database using a tool such as bscan easier. 2. Similar to 1), it is easier to manage small file volumes and extract particular jobs from a volume using bextract tool. 3. If the space is an issue (as it usually is), bigger volumes tend to eat more space which cannot be reused (volume cannot be recycled) as long as the volume contains a single job we want to preserve. 4. Although I don't like that approach, sometimes people chose to sync or copy whole file volumes to a secondary location using the usual tools such as rsync, cp and similar. In such case it is better to keep file volumes small. 5. When recycling a file volume, it will take longer time to wipe bigger file volume. If a volume is smaller it will take less time to recycle ensuring more time windows where other tasks could benefit from I/O performance. In case of large file volumes all other tasks would have to fight for the opportunity to access the file system and that gets more obvious when a slow network file system is being used. 6. In case of any kind of corruption of a file volume due to the file system corruption or damage in transport, it is likely that less data will be lost in case of a smaller file volume. And again, it's easier to handle smaller file volume when trying to recover pieces of data. Regards! Great Post! Your way of explaining the reasoning of why to use smaller file volumes, is very appreciated. The truth is, most files are fairly small. Particularly files created by office users. They range from a few kbytes up to some tens of megabytes. Videos can be huge, but I guess most companies handle instruction videos and similar, and not full blown movies. This type of content very seldom exceed 1GB. So a 10Gbyte volume limit seems to be a good balance. I'm used to fixed volume sizes from the tape drives, I feel comfortable with it, and I do not need to relearn a lot to configure the Bacula system. The only thing I haven't found out is how to preallocate the number of volumes needed. Maybe there is no need, if the volumes are created automagically. Most of the RAID array will be used by Bacula, just leaving a couple of percent as free space. When using mhvtl, I started a script with the tape size and number of tapes I wanted, and the corresponding tape directories and volumes were created on the fly. Thanks Josip! Best regards, Peter ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
On 23.01.2022 11:37, Radosław Korzeniewski wrote: Hello, pt., 21 sty 2022 o 14:22 Peter Milesson via Bacula-users napisał(a): If somebody has got experience with disk based, multi volume Bacula backup, I would be grateful about some information (tips, what to expect, pitfalls, etc.). The best IMVHO (but not the only mine) is to configure one job = one volume. You will get no real benefit to limit the size of a single volume. In the single volume = single job configuration you can set up job retention very easily as purging a volume will purge a single job only. It is not required to "wait" a particular volume to fill up to start retention. Purging a volume affects a single job only. And finally you end up with a way less number of volumes then when limiting its size to i.e. 10G. There are many different approaches which can fit different requirements. I don't see the benefit of having a single job per volume as Bacula is tracking media, files, jobs and everything else. That's why Bacula has a catalog which allows the backup system to determine the location and state of volumes, jobs, files, etc. To logically separate backup data I use pools and leave the rest to Bacula. When Bacula needs a particular file volume, if it's available Bacula will simply use it and if it's not or if we are using tape volume which is currently not in the tape drive/library, Bacula will ask for the volume by name. The number of smaller file volumes (e.g. 10GB) is not an issue as Bacula is handling them correctly and automatically (provided that Bacula is correctly configured, of course). I'll go through few examples where smaller file volumes (e.g. 10GB) could prove useful: 1. If the catalog database get corrupted or completely lost, due to the the small size, it's easier and faster to handle and determine volumes which contain database backup. That makes the process of importing the data into a new catalog database using a tool such as bscan easier. 2. Similar to 1), it is easier to manage small file volumes and extract particular jobs from a volume using bextract tool. 3. If the space is an issue (as it usually is), bigger volumes tend to eat more space which cannot be reused (volume cannot be recycled) as long as the volume contains a single job we want to preserve. 4. Although I don't like that approach, sometimes people chose to sync or copy whole file volumes to a secondary location using the usual tools such as rsync, cp and similar. In such case it is better to keep file volumes small. 5. When recycling a file volume, it will take longer time to wipe bigger file volume. If a volume is smaller it will take less time to recycle ensuring more time windows where other tasks could benefit from I/O performance. In case of large file volumes all other tasks would have to fight for the opportunity to access the file system and that gets more obvious when a slow network file system is being used. 6. In case of any kind of corruption of a file volume due to the file system corruption or damage in transport, it is likely that less data will be lost in case of a smaller file volume. And again, it's easier to handle smaller file volume when trying to recover pieces of data. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
On 25.01.2022 9:31, Lionel PLASSE wrote: I’ve made a rotation system like this With one Bacula storage “backup” corresponding one Bacula device “Backup”<=> root directory /pool-bacula/ automounted by udev. And 3 pool full, diff, incr attached to this device I have 8 usb disk xfs formatted with a xfs label “Bacula” (better than uuid to configure udev rules) so udev can mount it automatically when plugged to the Debian system. I don't use Bacula for usb disk mounting task On each disk I label one volume: 3 for each 3 month full rotation volumes , Four for each 4 weeks diff rotation one for the daily incremental auto purged each week By configuring the good retention period between week days and month and by adding the correct number on jobs volume on each one, you can easily configure a schedule with 3 steps , one for full attached to full pool on each first Wednesday a month for example, one for diff attached to diff pool each week from 2nd Wednesday to 5th Wednesday and the third for incremental each day from Monday to Thursday. By this schedule you can keep a great number of backups so each day you can restore the previous until the Monday with incremental backups, each week kept by the differental backups and the 3 full montly too. De : Radosław Korzeniewski Envoyé : dimanche 23 janvier 2022 11:37 À : Peter Milesson Cc : bacula-users@lists.sourceforge.net Objet : Re: [Bacula-users] Virtual tapes or virtual disks Hello, pt., 21 sty 2022 o 14:22 Peter Milesson via Bacula-users <mailto:bacula-users@lists.sourceforge.net> napisał(a): If somebody has got experience with disk based, multi volume Bacula backup, I would be grateful about some information (tips, what to expect, pitfalls, etc.). The best IMVHO (but not the only mine) is to configure one job = one volume. You will get no real benefit to limit the size of a single volume. In the single volume = single job configuration you can set up job retention very easily as purging a volume will purge a single job only. It is not required to "wait" a particular volume to fill up to start retention. Purging a volume affects a single job only. And finally you end up with a way less number of volumes then when limiting its size to i.e. 10G. best regards -- Radosław Korzeniewski mailto:rados...@korzeniewski.net Hi Lionel, thanks for the information. I've tried your way of doing backups after throwing out the physical tape drive. It just wasn't workable. The first thing is having office personnel properly managing the different USB drives (same problem as with tapes). Second, backup to USB drives takes forever, and there is not time enough for a full backup to complete comes Monday morning, not to speak of the few times there were backup glitches and the necessity to repeat the full backup. So the only solution for me is backup to a huge fixed storage. Now I'm planning how to implement volume management on this fixed storage. Best regards, Peter ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
On 23.01.2022 11:37, Radosław Korzeniewski wrote: Hello, pt., 21 sty 2022 o 14:22 Peter Milesson via Bacula-users napisał(a): If somebody has got experience with disk based, multi volume Bacula backup, I would be grateful about some information (tips, what to expect, pitfalls, etc.). The best IMVHO (but not the only mine) is to configure one job = one volume. You will get no real benefit to limit the size of a single volume. In the single volume = single job configuration you can set up job retention very easily as purging a volume will purge a single job only. It is not required to "wait" a particular volume to fill up to start retention. Purging a volume affects a single job only. And finally you end up with a way less number of volumes then when limiting its size to i.e. 10G. best regards -- Radosław Korzeniewski rados...@korzeniewski.net Hi Radosław, thanks for your input. I'm still in the planning phase for the new backup server, so all information is valuable. Excuse me some elementary questions. How do you define volumes in the pool for your configuration? I've been using Bacula a little more than 11 years, but up till now I've used virtual tapes with fixed sizes. Once setup, it's just ticking, and the original configuration stays. I use a one year retention scheme, and that will not change. I have got no reason to purge jobs within that period and after a year, I would like to do as little work as possible, having old volumes purged automatically, when there is no more room on the RAID array. Using tapes, this was automatic. When all volumes were full, the volume with the oldest write time is purged, and reused. Best regards, Peter ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
Yes, one volume, one disk, plus all the Bacula conf files and the bootstrap bsr file PLASSE Lionel | Administrateur systŠmes et r‚seaux 221 All‚e de F‚tan 01600 TREVOUX | Plan d'accŠs Tel : 04.37.49.91.39 pla...@cofiem.fr www.cofiem.fr | www.cofiem-robotics.fr -Message d'origine- De : Josh Fisher Envoyé : mardi 25 janvier 2022 15:10 À : Lionel PLASSE ; bacula-users@lists.sourceforge.net Objet : Re: [Bacula-users] Virtual tapes or virtual disks On 1/25/22 03:31, Lionel PLASSE wrote: > I’ve made a rotation system like this > > With one Bacula storage “backup” corresponding one Bacula device > “Backup”<=> root directory /pool-bacula/ automounted by udev. > And 3 pool full, diff, incr attached to this device > > I have 8 usb disk xfs formatted with a xfs label “Bacula” (better than > uuid to configure udev rules) so udev can mount it automatically when > plugged to the Debian system. I don't use Bacula for usb disk mounting > task > > On each disk I label one volume: > 3 for each 3 month full rotation volumes , Four for each 4 weeks > diff rotation one for the daily incremental auto purged each week To be clear, you are using only a single volume per USB disk? > > By configuring the good retention period between week days and month and > by adding the correct number on jobs volume on each one, you can easily > configure a schedule with 3 steps , > one for full attached to full pool on each first Wednesday a month for > example, > one for diff attached to diff pool each week from 2nd Wednesday to 5th > Wednesday > and the third for incremental each day from Monday to Thursday. > > By this schedule you can keep a great number of backups so each day you can > restore the previous until the Monday with incremental backups, each week > kept by the differental backups and the 3 full montly too. > > De : Radosław Korzeniewski > Envoyé : dimanche 23 janvier 2022 11:37 > À : Peter Milesson > Cc : bacula-users@lists.sourceforge.net > Objet : Re: [Bacula-users] Virtual tapes or virtual disks > > Hello, > > pt., 21 sty 2022 o 14:22 Peter Milesson via Bacula-users > <mailto:bacula-users@lists.sourceforge.net> napisał(a): > If somebody has got experience with disk based, multi volume Bacula > backup, I would be grateful about some information (tips, what to > expect, pitfalls, etc.). > > The best IMVHO (but not the only mine) is to configure one job = one volume. > You will get no real benefit to limit the size of a single volume. > In the single volume = single job configuration you can set up job retention > very easily as purging a volume will purge a single job only. > It is not required to "wait" a particular volume to fill up to start > retention. Purging a volume affects a single job only. And finally you end up > with a way less number of volumes then when limiting its size to i.e. 10G. > > best regards > -- > Radosław Korzeniewski > mailto:rados...@korzeniewski.net > > ___ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
On 1/25/22 03:31, Lionel PLASSE wrote: I’ve made a rotation system like this With one Bacula storage “backup” corresponding one Bacula device “Backup”<=> root directory /pool-bacula/ automounted by udev. And 3 pool full, diff, incr attached to this device I have 8 usb disk xfs formatted with a xfs label “Bacula” (better than uuid to configure udev rules) so udev can mount it automatically when plugged to the Debian system. I don't use Bacula for usb disk mounting task On each disk I label one volume: 3 for each 3 month full rotation volumes , Four for each 4 weeks diff rotation one for the daily incremental auto purged each week To be clear, you are using only a single volume per USB disk? By configuring the good retention period between week days and month and by adding the correct number on jobs volume on each one, you can easily configure a schedule with 3 steps , one for full attached to full pool on each first Wednesday a month for example, one for diff attached to diff pool each week from 2nd Wednesday to 5th Wednesday and the third for incremental each day from Monday to Thursday. By this schedule you can keep a great number of backups so each day you can restore the previous until the Monday with incremental backups, each week kept by the differental backups and the 3 full montly too. De : Radosław Korzeniewski Envoyé : dimanche 23 janvier 2022 11:37 À : Peter Milesson Cc : bacula-users@lists.sourceforge.net Objet : Re: [Bacula-users] Virtual tapes or virtual disks Hello, pt., 21 sty 2022 o 14:22 Peter Milesson via Bacula-users <mailto:bacula-users@lists.sourceforge.net> napisał(a): If somebody has got experience with disk based, multi volume Bacula backup, I would be grateful about some information (tips, what to expect, pitfalls, etc.). The best IMVHO (but not the only mine) is to configure one job = one volume. You will get no real benefit to limit the size of a single volume. In the single volume = single job configuration you can set up job retention very easily as purging a volume will purge a single job only. It is not required to "wait" a particular volume to fill up to start retention. Purging a volume affects a single job only. And finally you end up with a way less number of volumes then when limiting its size to i.e. 10G. best regards -- Radosław Korzeniewski mailto:rados...@korzeniewski.net ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
I’ve made a rotation system like this With one Bacula storage “backup” corresponding one Bacula device “Backup”<=> root directory /pool-bacula/ automounted by udev. And 3 pool full, diff, incr attached to this device I have 8 usb disk xfs formatted with a xfs label “Bacula” (better than uuid to configure udev rules) so udev can mount it automatically when plugged to the Debian system. I don't use Bacula for usb disk mounting task On each disk I label one volume: 3 for each 3 month full rotation volumes , Four for each 4 weeks diff rotation one for the daily incremental auto purged each week By configuring the good retention period between week days and month and by adding the correct number on jobs volume on each one, you can easily configure a schedule with 3 steps , one for full attached to full pool on each first Wednesday a month for example, one for diff attached to diff pool each week from 2nd Wednesday to 5th Wednesday and the third for incremental each day from Monday to Thursday. By this schedule you can keep a great number of backups so each day you can restore the previous until the Monday with incremental backups, each week kept by the differental backups and the 3 full montly too. De : Radosław Korzeniewski Envoyé : dimanche 23 janvier 2022 11:37 À : Peter Milesson Cc : bacula-users@lists.sourceforge.net Objet : Re: [Bacula-users] Virtual tapes or virtual disks Hello, pt., 21 sty 2022 o 14:22 Peter Milesson via Bacula-users <mailto:bacula-users@lists.sourceforge.net> napisał(a): If somebody has got experience with disk based, multi volume Bacula backup, I would be grateful about some information (tips, what to expect, pitfalls, etc.). The best IMVHO (but not the only mine) is to configure one job = one volume. You will get no real benefit to limit the size of a single volume. In the single volume = single job configuration you can set up job retention very easily as purging a volume will purge a single job only. It is not required to "wait" a particular volume to fill up to start retention. Purging a volume affects a single job only. And finally you end up with a way less number of volumes then when limiting its size to i.e. 10G. best regards -- Radosław Korzeniewski mailto:rados...@korzeniewski.net ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
Hello, pt., 21 sty 2022 o 14:22 Peter Milesson via Bacula-users < bacula-users@lists.sourceforge.net> napisał(a): > If somebody has got experience with disk based, multi volume Bacula > backup, I would be grateful about some information (tips, what to > expect, pitfalls, etc.). > The best IMVHO (but not the only mine) is to configure one job = one volume. You will get no real benefit to limit the size of a single volume. In the single volume = single job configuration you can set up job retention very easily as purging a volume will purge a single job only. It is not required to "wait" a particular volume to fill up to start retention. Purging a volume affects a single job only. And finally you end up with a way less number of volumes then when limiting its size to i.e. 10G. best regards -- Radosław Korzeniewski rados...@korzeniewski.net ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
On 21.01.2022 15:46, Josip Deanovic wrote: On 2022-01-21 15:29, Peter Milesson via Bacula-users wrote: thanks for your input. Today, I am using 160GB virtual tape volumes with mhvtl and that seems to be a good trade off. 10GB volumes in the new setting would be a nightmare to manage. That would be around 3,000 volumes! The volumes will be on a physical RAID set, but I will anyway use an SSD drive for spooling. One of the bottlenecks today is the slow throughput of the physical disks. It is not worth investing in the old rig, as it is anyway ripe for decommission. I have many thousands of file volumes and I don't see any problem managing them. They are being managed by Bacula. Regards! Hi Josip, Thanks for the information. I will have a look how to implement it. Best regards, Peter ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
On 2022-01-21 15:29, Peter Milesson via Bacula-users wrote: thanks for your input. Today, I am using 160GB virtual tape volumes with mhvtl and that seems to be a good trade off. 10GB volumes in the new setting would be a nightmare to manage. That would be around 3,000 volumes! The volumes will be on a physical RAID set, but I will anyway use an SSD drive for spooling. One of the bottlenecks today is the slow throughput of the physical disks. It is not worth investing in the old rig, as it is anyway ripe for decommission. I have many thousands of file volumes and I don't see any problem managing them. They are being managed by Bacula. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
On 21.01.2022 14:56, Josip Deanovic wrote: On 2022-01-21 14:03, Peter Milesson via Bacula-users wrote: Hi folks, I am building a new backup server that is going to replace the old one. The old backup server uses Bacula ver. 9.2.2 with a virtual tape library (mhvtl). I have found mhvtl a bit tricky, mostly when updating the OS (CentOS 7.8), as it is necessary to recompile the mhvtl kernel driver. Mhvtl also seems a bit outdated, with intermittent development and updates, also problematic with newer and coming Linux kernel versions. I also want to jump off the RedHat train, as I see it deviate more and more from mainstream Linux. Therefore, I would prefer to choose another solution. I have studied the Bacula documentation and it seems to be possible to use disk based backup with auto changer role. I plan to use volumes with a size of around 200Gbytes, making the setup fairly flexible, not making a too big hole when volumes need to be purged. Total disk space is around 30TB. If somebody has got experience with disk based, multi volume Bacula backup, I would be grateful about some information (tips, what to expect, pitfalls, etc.). Hi Peter, I am using file volumes for many years and I can't recall any special tips or pitfalls worth mentioning. Make sure the directory ownership and permissions are correct, that is, that storage daemon is able to access it (R/W). There are some differences regarding options such as Random Access, Removable Media and Device Type. As disk is not a sequential access type of media, you should set the Random Access option to yes. Unlike tape, a disk is not removable media so you may want to set RemovableMedia option to no. Regarding the volume size, I always chose size of 10 GB. You will need to test whether there is a gain of employing a SpoolDirectory for your disk based backup. If your file system containing file volumes is being mounted over a network (e.g. using iSCSI or NFS), it might be a good idea to use a local spool directory. Regards! Hi Josip, thanks for your input. Today, I am using 160GB virtual tape volumes with mhvtl and that seems to be a good trade off. 10GB volumes in the new setting would be a nightmare to manage. That would be around 3,000 volumes! The volumes will be on a physical RAID set, but I will anyway use an SSD drive for spooling. One of the bottlenecks today is the slow throughput of the physical disks. It is not worth investing in the old rig, as it is anyway ripe for decommission. Best regards, Peter ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
On 2022-01-21 14:03, Peter Milesson via Bacula-users wrote: Hi folks, I am building a new backup server that is going to replace the old one. The old backup server uses Bacula ver. 9.2.2 with a virtual tape library (mhvtl). I have found mhvtl a bit tricky, mostly when updating the OS (CentOS 7.8), as it is necessary to recompile the mhvtl kernel driver. Mhvtl also seems a bit outdated, with intermittent development and updates, also problematic with newer and coming Linux kernel versions. I also want to jump off the RedHat train, as I see it deviate more and more from mainstream Linux. Therefore, I would prefer to choose another solution. I have studied the Bacula documentation and it seems to be possible to use disk based backup with auto changer role. I plan to use volumes with a size of around 200Gbytes, making the setup fairly flexible, not making a too big hole when volumes need to be purged. Total disk space is around 30TB. If somebody has got experience with disk based, multi volume Bacula backup, I would be grateful about some information (tips, what to expect, pitfalls, etc.). Hi Peter, I am using file volumes for many years and I can't recall any special tips or pitfalls worth mentioning. Make sure the directory ownership and permissions are correct, that is, that storage daemon is able to access it (R/W). There are some differences regarding options such as Random Access, Removable Media and Device Type. As disk is not a sequential access type of media, you should set the Random Access option to yes. Unlike tape, a disk is not removable media so you may want to set RemovableMedia option to no. Regarding the volume size, I always chose size of 10 GB. You will need to test whether there is a gain of employing a SpoolDirectory for your disk based backup. If your file system containing file volumes is being mounted over a network (e.g. using iSCSI or NFS), it might be a good idea to use a local spool directory. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Virtual tapes or virtual disks
Hi folks, I am building a new backup server that is going to replace the old one. The old backup server uses Bacula ver. 9.2.2 with a virtual tape library (mhvtl). I have found mhvtl a bit tricky, mostly when updating the OS (CentOS 7.8), as it is necessary to recompile the mhvtl kernel driver. Mhvtl also seems a bit outdated, with intermittent development and updates, also problematic with newer and coming Linux kernel versions. I also want to jump off the RedHat train, as I see it deviate more and more from mainstream Linux. Therefore, I would prefer to choose another solution. I have studied the Bacula documentation and it seems to be possible to use disk based backup with auto changer role. I plan to use volumes with a size of around 200Gbytes, making the setup fairly flexible, not making a too big hole when volumes need to be purged. Total disk space is around 30TB. If somebody has got experience with disk based, multi volume Bacula backup, I would be grateful about some information (tips, what to expect, pitfalls, etc.). Best regards, Peter ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
