Re: [Bacula-users] Doubts about Bacula
On 4/16/19 1:22 PM, Josip Deanovic wrote: > The usual way to do it is a shared storage. > Servers would need to be able to connect to the storage and see the > exported LUN over fibre channel, iSCSI or similar protocol. And what I was saying is that at this point DRBD would be very low on my list of shared storage options for this application. -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu signature.asc Description: OpenPGP digital signature ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Doubts about Bacula
On Tuesday 2019-04-16 12:40:02 Dimitri Maziuk via Bacula-users wrote: > On 4/16/19 10:04 AM, Josip Deanovic wrote: > > NFS and DRBD are not really comparable that way. > > The easy way you migrate a running VM from one host to another is to > have the image on an NFS mounted on both hosts. Yes, that's one way to do it. > The hard way involves copying the actual image file, however, if you use > ZFS you may get away with only copying a small incremental snapshot. Copying the image around is the most inconvenient way to do it. > Last but not least, you can set it up on DRBD and make sure you get your > HA stack set up to migrate everything over correctly. With all the > network-controlled power outlets and the rest of the bloat you need to > make corosync/pacemaker run. Yes, that's somewhat complicated way to do it. The usual way to do it is a shared storage. Servers would need to be able to connect to the storage and see the exported LUN over fibre channel, iSCSI or similar protocol. If no simpler solutions are available, a locking mechanism (usually a component of a cluster infrastructure) would need to be employed to ensure that only one process is using the LUN at any time to avoid corruption. > Personally I believe in simple stupid; all my VMs live behind door > number 1. I totally agree that solutions should be as simple as possible. But not at the cost of stability and data integrity. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Doubts about Bacula
On Tuesday 2019-04-16 12:28:45 Dimitri Maziuk via Bacula-users wrote: > On 4/16/19 10:27 AM, Josh Fisher wrote: > > Running jobs will fail, but the automated "Reschedule On Error" > > feature > > allows restarting them after the fail-over. Also, fail-over doesn't > > affect scheduled jobs that haven't started yet at the time of > > fail-over. Putting the OS on DRBD and running in a VM (or container) > > allows continuation of backup services without operator intervention. > > What is wrong with that? > > Nothing, I just doubt migrating the VM itself is all that seamless and > stable without operator intervention. Admittedly, I have not played with > that in years... but last I looked if you actually *failed* without an > orderly VM shutdown, booting it up on the other node was a crap shoot. It's working just fine even with opensource technology for years. I used live migration of KVM virtual machines in 2010 and it worked without problems. While working in the virtual machine using ssh, a user wouldn't even notice that the virtual machine was migrated. The only problem for was a particular cluster component that was unstable. RHEV (RedHat Enterprise Virtualization) or oVirt solved that by introducing a solution that didn't relay on complex and unstable cluster components. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Doubts about Bacula
Not sure how this happened: > The easy way you migrate a running from one host to another VM is have -- it was meant to be "a running VM from one host to another". 8-\ boggle -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu signature.asc Description: OpenPGP digital signature ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Doubts about Bacula
On 4/16/19 10:04 AM, Josip Deanovic wrote: > NFS and DRBD are not really comparable that way. The easy way you migrate a running from one host to another VM is have the image on an NFS mounted on both hosts. The hard way involves copying the actual image file, however, if you use ZFS you may get away with only copying a small incremental snapshot. Last but not least, you can set it up on DRBD and make sure you get your HA stack set up to migrate everything over correctly. With all the network-controlled power outlets and the rest of the bloat you need to make corosync/pacemaker run. Personally I believe in simple stupid; all my VMs live behind door number 1. -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu signature.asc Description: OpenPGP digital signature ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Doubts about Bacula
On 4/16/19 10:27 AM, Josh Fisher wrote: > > > Running jobs will fail, but the automated "Reschedule On Error" feature > allows restarting them after the fail-over. Also, fail-over doesn't > affect scheduled jobs that haven't started yet at the time of fail-over. > Putting the OS on DRBD and running in a VM (or container) allows > continuation of backup services without operator intervention. What is > wrong with that? Nothing, I just doubt migrating the VM itself is all that seamless and stable without operator intervention. Admittedly, I have not played with that in years... but last I looked if you actually *failed* without an orderly VM shutdown, booting it up on the other node was a crap shoot. -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu signature.asc Description: OpenPGP digital signature ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Doubts about Bacula
On Tuesday 2019-04-16 11:27:54 Josh Fisher wrote: > On 4/16/2019 10:45 AM, Dmitri Maziuk via Bacula-users wrote: > > I would put everything out of drbd volume because quite frankly I > > don't > > see the point. I don't think you can fail over in a middle of a > > backup, > > and without that, why not just put OS on NFS? -- or ZFS and send > > incremental snapshot as part of your manual failover. Using drbd for > > backup storage is just a waste of disk. > > Running jobs will fail, but the automated "Reschedule On Error" feature > allows restarting them after the fail-over. Also, fail-over doesn't > affect scheduled jobs that haven't started yet at the time of fail-over. > Putting the OS on DRBD and running in a VM (or container) allows > continuation of backup services without operator intervention. What is > wrong with that? Nothing is wrong with that if one needs it and can afford it. > I agree that putting volume files on DRBD is wasteful, since running > jobs will always fail at fail-over, but putting just the OS on DRBD > doesn't use much disk space, and it certainly isn't a waste when it > reduces or eliminates operator intervention. When used properly DRBD can ensure redundancy. Often people want to have secondary backup and archives. One can copy backup jobs or volumes to a remote site and archive them at some point (my choice) but one can also chose to sync the blocks of the underlying block device to a remote location which could serve the purpose if the goal is to protect yourself from hardware error on the primary site. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Doubts about Bacula
Josip DeanovicOn Tuesday 2019-04-16 17:27:48 wrote: > On Tuesday 2019-04-16 12:13:55 Heitor Faria wrote: > > Speaking of Bacula HA, I've been deploying a scenario with relative > > success. Primary Director & SD have copy jobs routines to a Secondary > > Remote SD that also has an independent working Director. Both director > > can access the Secondary SD. An Admin Job with a Shell Script daily > > bscans all volumes in to the Secondary Director and its catalog. All > > bscanned volumes comes with the Archived status, so they are basically > > Read-Only. Advantage: you can restore jobs from both environments, any > > time. => > > http://bacula.us/bacula-server-and-backups-replication-for-high-availa > > b > > ility/ Perhaps, a "bscan all" bconsole command would be a nice feature > > to sync all disk based volumes to catalog and improve the proccess a > > little bit more. > > Have you tried to actually restore something from the secondary SD? > > I am asking because I have reported some bugs related to bsr files > but I don't remember the detail any more. > > But since you are doing bscan on secondary SD it might not be affected > by the bug I have mentioned. Forgot to say that the bug I have mentioned was reported for the version 7.4 if I am not mistaken. It could be fixed in newer versions. Regards -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Doubts about Bacula
On Tuesday 2019-04-16 12:13:55 Heitor Faria wrote: > Speaking of Bacula HA, I've been deploying a scenario with relative > success. Primary Director & SD have copy jobs routines to a Secondary > Remote SD that also has an independent working Director. Both director > can access the Secondary SD. An Admin Job with a Shell Script daily > bscans all volumes in to the Secondary Director and its catalog. All > bscanned volumes comes with the Archived status, so they are basically > Read-Only. Advantage: you can restore jobs from both environments, any > time. => > http://bacula.us/bacula-server-and-backups-replication-for-high-availab > ility/ Perhaps, a "bscan all" bconsole command would be a nice feature > to sync all disk based volumes to catalog and improve the proccess a > little bit more. Have you tried to actually restore something from the secondary SD? I am asking because I have reported some bugs related to bsr files but I don't remember the detail any more. But since you are doing bscan on secondary SD it might not be affected by the bug I have mentioned. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Doubts about Bacula
On 4/16/2019 10:45 AM, Dmitri Maziuk via Bacula-users wrote: On Mon, 15 Apr 2019 23:24:10 -0300 Marcio Demetrio Bacci wrote: 5. Currently the OS and Backup disks are on the same DRBD volume, so would it be better to put the OS disk out of the DRBD volume? (the VM has frequently crashing what makes me think that excessive writing on the disk may be impacting the OS) I would put everything out of drbd volume because quite frankly I don't see the point. I don't think you can fail over in a middle of a backup, and without that, why not just put OS on NFS? -- or ZFS and send incremental snapshot as part of your manual failover. Using drbd for backup storage is just a waste of disk. Running jobs will fail, but the automated "Reschedule On Error" feature allows restarting them after the fail-over. Also, fail-over doesn't affect scheduled jobs that haven't started yet at the time of fail-over. Putting the OS on DRBD and running in a VM (or container) allows continuation of backup services without operator intervention. What is wrong with that? I agree that putting volume files on DRBD is wasteful, since running jobs will always fail at fail-over, but putting just the OS on DRBD doesn't use much disk space, and it certainly isn't a waste when it reduces or eliminates operator intervention. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Doubts about Bacula
Hello All, >> 5. Currently the OS and Backup disks are on the same DRBD volume, so >> would it be better to put the OS disk out of the DRBD volume? (the VM >> has frequently crashing what makes me think that excessive writing on >> the disk may be impacting the OS) > > I would put everything out of drbd volume because quite frankly I don't > see the point. I don't think you can fail over in a middle of a backup, > and without that, why not just put OS on NFS? -- or ZFS and send > incremental snapshot as part of your manual failover. Using drbd for > backup storage is just a waste of disk. Speaking of Bacula HA, I've been deploying a scenario with relative success. Primary Director & SD have copy jobs routines to a Secondary Remote SD that also has an independent working Director. Both director can access the Secondary SD. An Admin Job with a Shell Script daily bscans all volumes in to the Secondary Director and its catalog. All bscanned volumes comes with the Archived status, so they are basically Read-Only. Advantage: you can restore jobs from both environments, any time. => http://bacula.us/bacula-server-and-backups-replication-for-high-availability/ Perhaps, a "bscan all" bconsole command would be a nice feature to sync all disk based volumes to catalog and improve the proccess a little bit more. Regards, -- MSc Heitor Faria CEO Bacula LATAM mobile1: + 1 909 655-8971 mobile2: + 55 61 98268-4220 [ https://www.linkedin.com/in/msc-heitor-faria-5ba51b3 ] [ http://www.bacula.com.br/ ] América Latina [ http://bacula.lat/ | bacula.lat ] | [ http://www.bacula.com.br/ | bacula.com.br ] ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Doubts about Bacula
On Tuesday 2019-04-16 09:45:29 Dmitri Maziuk via Bacula-users wrote: > On Mon, 15 Apr 2019 23:24:10 -0300 > > Marcio Demetrio Bacci wrote: > > 5. Currently the OS and Backup disks are on the same DRBD volume, so > > would it be better to put the OS disk out of the DRBD volume? (the VM > > has frequently crashing what makes me think that excessive writing on > > the disk may be impacting the OS) > > I would put everything out of drbd volume because quite frankly I don't > see the point. I don't think you can fail over in a middle of a backup, > and without that, why not just put OS on NFS? -- or ZFS and send > incremental snapshot as part of your manual failover. Using drbd for > backup storage is just a waste of disk. NFS and DRBD are not really comparable that way. DRBD is block level replication and you can achieve redundancy at the block level. NFS is a network file system which could be setup on top of the DRBD provided block device and doesn't provide redundancy. > I'd also try to run bacula in a container instead of VM at this point. > but that's just me. It all depends on his environment. -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Doubts about Bacula
On Mon, 15 Apr 2019 23:24:10 -0300 Marcio Demetrio Bacci wrote: > 5. Currently the OS and Backup disks are on the same DRBD volume, so > would it be better to put the OS disk out of the DRBD volume? (the VM > has frequently crashing what makes me think that excessive writing on > the disk may be impacting the OS) I would put everything out of drbd volume because quite frankly I don't see the point. I don't think you can fail over in a middle of a backup, and without that, why not just put OS on NFS? -- or ZFS and send incremental snapshot as part of your manual failover. Using drbd for backup storage is just a waste of disk. I'd also try to run bacula in a container instead of VM at this point. but that's just me. -- Dmitri Maziuk ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Doubts about Bacula
On 4/15/2019 10:50 PM, Gary R. Schmidt wrote: On 16/04/2019 12:24, Marcio Demetrio Bacci wrote: ... 2. Is there any restriction on mounting the Bacula VM on a DRBD volume in the hypervisor. In this VM I will implement ZFS deduplication for my backups? I /think/ you are asking if Bacula can handle fail-over during backup? Short answer is "no". (I am willing to accept corrections on this, but AFAICS Bacula doesn't checkpoint its output.) I would say "mostly", rather than "no". On fail-over, running jobs will eventually fail, since the FD clients will not reconnect and their open TCP sockets will timeout. AFAIK, there is no transparent TCP connection migration. The virtual IP is migrated, but not the currently opened TCP connections. However, this is not necessarily a bad thing. Bacula can be configured to automatically re-run failed jobs, and there was some reason for the fail-over in the first place. Do we really trust the first part of the backup that was made while running on the failing cluster node? I have experimented with running Dir and SD in a KVM VM on a two-node Corosync/Pacemaker cluster using active/passive DRBD storage for the VM's OS. Backup storage was via iSCSI provided by an existing NAS. I found no major issues. On fail-over, Pacemaker attempts to initiate a shutdown on the failing node. If the shutdown succeeds, then the bacula-dir and bacula-sd services are stopped in the normal way and it affects running jobs exactly the same as if those services were stopped manually. If the shutdown fails, then the node is STONITH'd and it is the same effect as pulling the power on the node. In that case, the clients' actively running jobs are orphaned. Eventually, their TCP sockets timeout. In either case, the running jobs are not successful, but as I stated earlier, I don't see that as a major reason not to run Bacula in a VM. Currently, I run Dir and SD in a VM on a four-node cluster, but without fail-over for the Bacula VM. The VM and its DRBD storage have to be brought up manually, but it allows a single catalog, a single IP, and a single set of config, log, and spool files that can be brought up in seconds on any one of the nodes. The reason for this is because I do not have a FC-attached tape library or other HA capable backup device. To bring up on another node I have to move my backup devices (USB-attached RDS and portable hard drives plus SATA drives in removable drive bays using vchanger). Based on how it worked with iSCS NAS storage, I believe it would also work well with HA capable tape hardware on FC (or iSCSI?). It should also work (ie. jobs running at fail-over will fail, but can be automatically re-run) when storing backups on DRBD attached locally to the VM. Of course, it doubles the storage space needed for volume files and so will be considerable. Since the running jobs will fail at fail-over, I see no reason for volume files to be HA. They just need to be accessible to the VM, for example on iSCSI-attached NAS. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Doubts about Bacula
Marcio Demetrio Bacci kirjoitti 16.4.2019 5:24: I have some doubts about Bacula, however I did not find answers on the Internet or in the books for the following doubts: 1. Is there any problem in using Bacula virtualized? (I didn't find anything saying otherwise) I run my Bacula server as an LXC container in an CentOS 7 server. No problems so far. A dedicated box might be better for some, but this works for me. 2. Is there any restriction on mounting the Bacula VM on a DRBD volume in the hypervisor. In this VM I will implement ZFS deduplication for my backups? No idea and because of that I'd just try and see. But I see not an obvious reason for that to not work. 3. Is there any limitation on the size of Backups with Bacula Community version? (I did not find information about any limitations) No. 4. Having a few jobs with 300 GB or more, though I set up my volumes on Bacula with 100 GB, so would you recommend increasing the size of the volumes to 200 GB or more? I use 4.6 GB for my Volumes, and my storage size is a 8 TB Archive Disk (SMR). That has a historical reason as once upon a time I stored them to DVD-RW. There is this to ponder: when you recycle a Volume, all data in the Volume is lost. If you have 200 GB Volumes, you will Lose 200 GB in every Volume recycle... To me that matters. The smaller the Volume the better. 5. Currently the OS and Backup disks are on the same DRBD volume, so would it be better to put the OS disk out of the DRBD volume? (the VM has frequently crashing what makes me think that excessive writing on the disk may be impacting the OS) I have no idea about this. -- Jari Fredriksson Bitwell Oy +358 400 779 440 ja...@bitwell.fi Dev: https://www.bitwell.fi Ops: https://www.bitwell.biz -- ja...@iki.fi ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Doubts about Bacula
On Monday 2019-04-15 23:24:10 Marcio Demetrio Bacci wrote: > I have some doubts about Bacula, however I did not find answers on the > Internet or in the books for the following doubts: > > 1. Is there any problem in using Bacula virtualized? (I didn't find > anything saying otherwise) No. I have several setups on vmware and kvm. Even few on raspberry-pi and they are all running for at least 5 years, some of them 10+. > 2. Is there any restriction on mounting the Bacula VM on a DRBD volume > in the hypervisor. In this VM I will implement ZFS deduplication for my > backups? No. As long as Bacula can use the file system as usual, Bacula will not be affected by the fact you are using DRBD as it just provides another block device you are using for your file system of choice (in your case ZFS). But you might want to read some articles on the DRBD + ZFS setup if you haven't done that already. E.g. http://cedric.dufour.name/blah/IT/ZfsOnTopOfDrbd.html > 3. Is there any limitation on the size of Backups with Bacula Community > version? (I did not find information about any limitations) No. Bacula Systems is kind enough to not impose such limits on Bacula Community version and it was never a case. > 4. Having a few jobs with 300 GB or more, though I set up my volumes on > Bacula with 100 GB, so would you recommend increasing the size of the > volumes to 200 GB or more? I prefer to keep them at 10G so that I can maintain them easier in case I need to deal with the volumes manually. > 5. Currently the OS and Backup disks are on the same DRBD volume, so > would it be better to put the OS disk out of the DRBD volume? (the VM > has frequently crashing what makes me think that excessive writing on > the disk may be impacting the OS) Generally, it is good idea to keep OS and the data separated. That way you can optimize its devices differently performance and security-wise. But if your DRBD is the only thing that gives you redundancy then it's better to keep them both on the DRBD. However, the crashing VM is really bad problem and is not something that should be ignored or considered as nuisance. You should attend to solving this problem before proceeding with improvements of other systems that rely on that VM and its stability. Good luck. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
