Re: [Bacula-users] Recovery from only one storage
On 2023-04-11 07:30, Tomasz Świderski wrote: Can someone help me on this matter? W dniu 5.04.2023 o 13:07, Tomasz Świderski pisze: I have 2 jobs in Bacula connected to one client. Job 1 makes a copy to the hard drive (storage1) and Job 2 makes a copy to the magnetic tape (storage2). How during recovery, indicate that it should only recover e.g. from storage2? My scenario assumes, for example, that storage 1 is not available and I have to recover only from magnetic tapes (storage2). When in Baculum I choose, for example, Job2, and click Recovery, it shows me from my two storage on the list of required media. Hello Tomasz, You can use command "restore copies" in the bconsole tool. Then you can chose the time of the targeted restore and targeted client. From the output you will be able to see jobs with their respective MediaType which can be used to determine the pool of a particular job. Once you determine the job IDs you want to restore, you can use the command like this: "restore jobid=130286,130836,131808,131826,132591" in the bconsole tool. The rest of the restore goes as usual. It is a bit complicated procedure but it works. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] password for user bacula
On 2023-04-11 04:51, Yateen Shaligram Bhagat (Nokia) wrote: Thanks Bill, much appreciated. I assume adding password and shell to bacula user does not cause any harm to the bacula functionality Hello, It will not harm bacula functionality but I doubt you need to do that. If you need some commands to be executed by user bacula from shell scripts, I would recommend utilizing sudo command. In addition to what Bill said, if you just need to become bacula user temporary, you could use the command: su -s /bin/bash - bacula It is useful when you want to check if the user can actually access some file or directory or when debugging bacula services. Regards -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula 13 & PostgreSQL 15 ?
On 2023-02-03 08:36, Frédéric F. wrote: Hello Ana, Ok it's what I'm going to do. I can't say anything for Bacula 13 but I am running Bacula 9.6 on Postgres 15 for several months without problems. Before Postgres 15 I used Postgres 14, 13 and 11, also without problem. Regards -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Q. Copy jobs - how to select all uncopied backups jobs that ran in the last 24 hours?
On 2023-01-20 00:42, Justin Case wrote: Thank you so much Josip, that sounds perfect. I will run tests with your query. Better check the whole thread because it seems that I have modified the query few months later: https://sourceforge.net/p/bacula/mailman/bacula-users/thread/9049365.ZOPTLNicxr%40libria/#msg34336222 -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Q. Copy jobs - how to select all uncopied backups jobs that ran in the last 24 hours?
On 2023-01-19 22:56, Justin Case wrote: Hi all, I am trying to keep a copy of all backup jobs on a tier-2 storage. My first approach was to copy all yet uncopied jobs - but in the beginning that would be a prohibitive number of copy jobs (hundreds!) I think it would work for me if I would let Bacula copy all yet uncopied jobs that ran within the last 24 hours. It seems that the the only selection type supporting such a thing would be SQLQuery. Or is there another selection that would make it easier to do this? Searching the manuals I couldn’t find the schema that I would be able to use for the SQL query. Whee is this documented? Are there any examples for this selection typ for copy or migration jobs? Hello, Please check this thread from archives: https://sourceforge.net/p/bacula/mailman/message/34336222/ In that archive thread there is an SQL query that selects only the latest full job along with its differential and all the incremental jobs that haven't been copied yet. If you copy only the jobs executed in the last 24h you will not be able to restore them (using normal restore procedures) without the related full backup jobs. Regards -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Copy backups to more than one storage
On 2023-01-17 17:45, Bill Arlofski via Bacula-users wrote: On 1/17/23 06:05, Ivan Villalba via Bacula-users wrote: How can I run two differnet copy job that copies the same jobid with the PoolUncopiedJobs ? You can't. The PoolUncopiedJobs does exactly what its name suggests: It copies jobs in a pool that have not been copied to some other pool. If you want to copy the same backup jobs to more than one other pool, you will need to use `Selection type = SQLQuery` and then use an appropriate SQL query for the `SelectionPattern` to generate a list of JobIds to run the second set of copies. I am using PoolUncopiedJobs with a RunAfterJob which executes a script when the job successfully finishes. The script inserts the job ID of a copied job into a table created specifically for that purpose. The original job ID can be passed to a shell script using "%I". This requires additional admin job that starts jobs that will perform a copy using the list of job IDs from the table. Those copy jobs also use RunAfterJob calling a shell script that removes a job ID from the table in case the copy job finished successfully. That way it is possible to copy jobs as many time as needed. It is possible to use different pools which is useful in case one need to maintain several pools which contain all the jobs stored on different media which also rotate several times per week. I like the level of flexibility Bacula offers. It is worth mentioning that copy job builds a list of jobs that need to be copied at the time of job execution. Regards -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Backup without files records
On 2023-01-07 13:03, Yateen Shaligram Bhagat (Nokia) wrote: Hi All, I am curious to know if a backup job can be run without file records being cataloged in the backend database. I know about AutoPrune mechanism for files, but that happens after the backup job is over for file records older than the retention time. But we don't want the file records to get cataloged at all. Hello, I doubt this is possible. The point of the catalog database is that Bacula can track the status of a job and its files, related volumes and so on. Without the data in the database Bacula would not be able to compare the status of the files and wouldn't be able to perform incremental backup. For example, once all the data related to a specific job or volume is pruned, in order to be able to restore the files using Bacula, the data on the volume would first need to be inserted back into the database using bscan tool. -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Change volume retention periods?
On 2022-11-30 20:00, Agiofws Holylight wrote: Thank you for the clarification You are welcome. Regards -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Change volume retention periods?
On 2022-11-30 18:34, Agiofws Holylight wrote: But the ""The Recycle Oldest Volume = yes record" will respect retention dates right? I'm just asking through this thread because I have about 16 volumes in my increment pool and none have expired so the job is just waiting for a new volume with the right retention period or something like that. Thank you C.m Yes. Recycle Oldest Volume will honor retention period while Purge Oldest Volume will not. Since you have only 16 volumes, there is a danger that a lot of useful data could be lost along with the oldest volume in case you configure Bacula to recycle the oldest volume without honoring the retention period. For example, some server could suddenly bring a lot of new data into backup and that could lead to an undesired recycling of multiple volumes, resulting in loss of data previously stored into the pool of volumes. Malicious actors could also recognize such vulnerability and use it to destroy the backup as a preparation for a larger cyberattack (e.g. ransomware). This is one of the reasons why companies and people who can afford it, make several backup copies using several different media, including WORM tapes. Regards -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Change volume retention periods?
On 2022-11-30 13:34, Agiofws Holylight wrote: So the Directive "Purge Oldest Volume = yes" will ignore retention settings on the volume? Yes. Here is the relevant part from Bacula documentation: "The Purge Oldest Volume = yes record in the Pool resource permits a forced recycling of the oldest Volume when a new one is needed. N.B. This record ignores retention periods! We highly recommend not to use this record, but instead use Recycle Oldest Volume." "The Recycle Oldest Volume = yes record in the Pool resource tells Bacula to Prune the oldest volume in the Pool, and if all files were pruned to recycle this volume and use it." More on the subject of Bacula volume management: https://www.bacula.org/11.0.x-manuals/en/main/Basic_Volume_Management.html Regards -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Change volume retention periods?
On 2022-11-30 11:02, Agiofws Holylight wrote: Hello is the way forcing Bacula to use the oldest volume in a pool ignoring retention periods? I have a kind of situation where due to the size all the data being backed up bacula used more than one volume per job so now my oldest volumes have not exceeded the retention. So bacula will not recycle the volumes for the pool Hello You can do that but there is a potential risk of dropping useful backup data in case your regular backup takes substantial number of volumes in a pool. This is what I am using (it is useful for file volumes): Maximum Volume Bytes = 10G Maximum Volumes = # to be calculated from disk space # and Maximum Volume Bytes (make sure # to leave extra space for at least # 2 x Maximum Volume Bytes) AutoPrune = yes Volume Retention = Recycle = yes Purge Oldest Volume = yes Regards -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Storage Management
On 2022-11-25 16:00, Nick Bright wrote: [...] inelegant. The system should automatically be pruning or at least allowing me to overwrite the old data - it ought to be cyclical - use the available disk space (17T) to back up systems, letting old backups fall off automatically so that new backups can take their place. This is exactly what I am doing using these directives in the pool resource configuration: Maximum Volume Bytes = 10G Maximum Volumes = # to be calculated from disk space # and Maximum Volume Bytes (make sure # to leave extra space for at least # 2 x Maximum Volume Bytes) AutoPrune = yes Volume Retention = Recycle = yes Purge Oldest Volume = yes The configuration above would recycle the oldest volume when needed, regardless of the retention period. This is potentially dangerous. You should have enough volumes in your pool to minimize the chance of losing useful (relatively recent) backup data. Regards -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Storage Management
On 2022-11-23 20:02, Nick Bright wrote: Thank you for the replies. The physical disk has about 2% free space (446GB) so it shouldn't be locked preventing any maintenance from occurring due to insufficient storage. Hi Nick If your file system is configured to reserve some percentage of blocks for root user, you will not be able to use all the space in case your bacula-sd is running as a non-root user. For example, with ext3/ext4 file system you might check that using the command: dumpe2fs -h /your/file/system | grep "Reserved block count" Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Notifications doesn't work
On 2022-11-21 13:16, Gina Costa wrote:
Hi,
I’m using bacula 9 over CentOS. I need receive notifications about job
failures but this notifications doesn’t work.
The configuration file bacula-dir.conf has the followup Messages
directive:
***
Messages {
Name = Standard
mailcommand = "/usr/sbin/bsmtp -h 192.168.0.66 -f "(Bacula) <%r>" -s
"Bacula: %t %e of %c %l" %r"
operatorcommand = "/usr/sbin/bsmtp -h 192.168.0.66 -f "(Bacula) <%r>"
-s "Bacula: Intervention needed for %j" %r"
mail = gina.co...@uc.pt = all, !skipped
operator = gina.co...@uc.pt = mount
console = all, !skipped, !saved
append = "/var/log/bacula/bacula.log" = all, !skipped
catalog = all
}
***
To test whether the mail sending is working as it should, I do the
following command, and this works fine, I receive the test message :
echo "This is test message." | /opt/bacula/bin/bsmtp -h 192.168.0.66
-f bac...@uc.pt -s "Test" gina.co...@uc.pt -d 50
Can anyone help me?
Maybe you need to escape special characters like quotes.
For example:
mailcommand = "/usr/sbin/bsmtp -h 192.168.0.66 -f \"\(Bacula\) <%r>\" -s
\"Bacula: %t %e of %c %l\" %r"
operatorcommand = "/usr/sbin/bsmtp -h 192.168.0.66 -f \"\(Bacula\)
<%r>\" -s \"Bacula: Intervention needed for %j\" %r"
Regards!
--
Josip Deanovic
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tricky restore issue (bacula 11)
On 2022-10-31 12:46, Uwe Schuerkamp wrote: Hi folks, due to user error part of a file system on one of our backup clients was compressed by a find command gone haywire. Among them are /usr/bin/bash & others (kernel, initrd, grub, root, authorized_keys etc), leading to a situation where we cannot log into the system remotely nor via the vmware console function anymore. I've tried to restore /usr/bin/bash by its uncompressed version as the bacula-fd on the client is still running (today's incr. backup worked fine, too), but once the restore starts I'm getting the following error: 31-Oct 12:23 deniol2378-sd JobId 23316: Forward spacing Volume "deX-0183" to addr=269 31-Oct 12:23 deniol2378-sd JobId 23316: Elapsed time=00:00:01, Transfer rate=484.9 K Bytes/second 31-Oct 12:22 bacula-fd JobId 23316: Error: create_file.c:227 Could not create /usr/bin/bash: ERR=Permission denied 31-Oct 12:23 deniol2378-dir JobId 23316: Bacula deYY-dir 13.0.0 (04Jul22): the bacula-fd runs as root on the client so it should have no issue creating the file in the local filesystem... any idea what could be causing this? It's also weird that I'm seeing a "restore OK" message at the end of the job output: Replace:Always Start time: 31-Oct-2022 12:23:02 End time: 31-Oct-2022 12:23:02 Elapsed time: 1 sec Files Expected: 1 Files Restored: 1 Bytes Restored: 0 (0 B) Rate: 0.0 KB/s FD Errors: 0 FD termination status: OK SD termination status: OK Termination:Restore OK I'm clutching at straws here naturally, however before we rebuild the VM in question from a full backup I'd like to try as many options as possible. What makes this case even worse is that LVM is being used for / & other fileystems, so we cannot simply mount those drives on another VM (3 physical volumes) to repair the damage, right? Thanks for reading this far & for your help, Hello, It might be that your file systems are now mounted read-only and bacula-fd cannot write to file systems. Are you able to connect to the system and run any command? If not, you will have to boot the system using boot iso image, prepare network, prepare LVM, mount file systems, copy the necessary bacula files, chroot into the file system, start bacula-fd and restore the data. If yes, you could try to remount file systems rw. E.g. mount -o remount,rw / If os is heavily broken, you could help yourself using the statically linked busybox tool. -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Workaround to get Bacula FD running on Ubuntu 22.04 Jammy using Debian 11 Bullseye packages
On 2022-09-28 17:15, Matthew Weatherford wrote: Hi Folks, We've got Bacula Client working on Ubunut 22.04 systems (Jammy) and I thought I'd share my script with you all. [...] Hi Matthew, I don't understand what was the problem with the Bacula FD on Ubuntu. Is that something Ubuntu specific? For Debian 10.12 and Bacula 9.6.7 all one would need to do is to use Debian buster-backports repository and install needed bacula-* packages as usual. Regards -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Silly question from newbie
On 2022-08-26 01:11, Bill Arlofski via Bacula-users wrote: On 8/25/22 15:54, Josip Deanovic wrote: You also mentioned that your backup is going to disks and to tapes. Unless something changed in the newer versions of Bacula, the space on the tape volumes as well as the data on the disk/file volumes will not be reclaimed until the whole volume is recycled. Hello Josip, Nothing has changed. The above is still a true statement. :) Thanks for confirming it. It would complicate things greatly otherwise. Regards -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Silly question from newbie
On 2022-08-25 23:30, Pollard, Jim wrote: Maybe another option would be do a migrate job to external media for enough of the volumes that contain the incrementals. That way If things go completely south I could still restore. I’ll have to take a look at the schedules. I’m feeling like the first thing bacula is going to do is determine that incrementals are missing and initiate fulls. Is that sounding right? When you say "migrate" do you refer to Bacula migrate job or do you refer to moving file volumes to a different file system? You can move some file volumes or the whole directory containing Bacula file volumes to a separate file system and either use symlinks, mount the new file system to the same mountpoint or reconfigure bacula-sd. If you remove an incremental job, Bacula will simply continue backing up data since the last known successful full, differential or incremental backup. It will decide that full backup is needed only in case no successful full backup is known to exist. Bacula will determine this by looking into the database. Regards -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Silly question from newbie
On 2022-08-25 23:16, Pollard, Jim wrote: [...] From: Udo Kaune Date: Thursday, August 25, 2022 at 4:05 PM To: Pollard, Jim Subject: Re: [Bacula-users] Silly question from newbie Am 25.08.22 um 22:52 schrieb Pollard, Jim: I see. So if my incrementals were daily then I would be able to restore to a specific day. Otherwise my only options would be the last full, or the last full + the differential? Correct. If you aim for (just) getting back the latest version of everything you might remove the "in-betweens". How far apart are your Full backups? Please always cc to the list so that the community may benefit from your findings. One should be careful here. Differential backups will hold changes since the last successful full backup. Incremental will hold changes since the last successful full, differential or incremental backup. Please check the Job Resource -> Level -> Incremental here: https://www.bacula.org/11.0.x-manuals/en/main/Configuring_Director.html If you lose "in-betweens", e.g., you remove differential backups, you will lose the data covered between the time the differential backup and the incremental backup that preceded it. The incremental backup following the differential backup will cover only the changes since the last successful backup, in this case differential. So, in short, if you don't want to lose data, I would suggest to avoid deleting "in-betweens". Also, unless Bacula is configured specifically the way that backup jobs end up on their own file volumes (e.g. single backup per volume), deleting a backup job will not free the space in the volume. Only the metadata related to the deleted job will be deleted from the database. You also mentioned that your backup is going to disks and to tapes. Unless something changed in the newer versions of Bacula, the space on the tape volumes as well as the data on the disk/file volumes will not be reclaimed until the whole volume is recycled. Regards -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] restore options (yesterday, last week, last month)
On 2022-07-19 19:00, Clouse, Raymond [JT4 LLC] wrote: Can Bacula restore files based on date? That is, can I specify I want file.txt from yesterday, or from last week, or from last month? Is there a way to set up backups to preserve files in this manner? Hi Raymond Yes it can. Of course, you will have to perform backup on a daily basis. Here is the relevant documentation: https://www.bacula.org/11.0.x-manuals/en/main/Restore_Command.html Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Beta Release 11.3.4
On 2022-06-03 20:34, Bill Arlofski via Bacula-users wrote:
Hello Josip,
This new StorageGroup feature allows you to specify more than one
Storage in a Job or Pool.
For example:
8<
Job {
Name = Test
Storage = Storage_1, Storage_2, Storage_3, ...
StorageGroupPolicy = (ListedOrder|LeastUsed)
...
}
8<
Then, when Bacula starts a job, if it cannot reach the first one for
some reason, or the first one rejects the job (ie: some
mis-configuration issue, or devices are disabled on it), then the next
one is tried, and so on until it finds one that works.
The `StorageGroupPolicy` setting allows you to tell Bacula what
criteria to use when selecting
the storage. The default is
`ListedOrder` and will be used if not specified.
The `LeastUsed` policy checks all of the specified Storages and
chooses the one with the least number of jobs currently
running on it.
I understand there are more `StorageGroupPolicy` options being planned.
Hope this helps,
Bill
This is very nice.
Bill, thank you for explaining it.
Regards!
--
Josip Deanovic
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Beta Release 11.3.4
On 2022-06-03 15:00, Eric Bollengier via Bacula-users wrote: Hello, We are pleased to announce the new BETA release of the next major Bacula version 11.3.4 to both the Bacula website (www.bacula.org) and to SourceForge. Thanks for the valuable feedback. [...] New Features: - Job Storage group support Hello Eric, A lot of new features, useful fixes and changes. That's very nice. Would it be possible to migrate from 9.6 to 13.x sometimes in the future? Could you please say few words about "Job Storage group support" feature. What exactly is that? I tried to search for it on the net, including the documentation but couldn't find anything that would explain what Job Storage group support might be. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Copy Job of a bunch of backups?
On 2022-05-27 11:27, Pierre Bernhardt wrote: Hello, I create a full backup each 1st Sunday, diff backups of rest of Sundays, incremental backups each day. I would create a copy after the first day of a month backup has been made. The data should be copied from the last backup however it has been made. So for the moment I create a restore before a given date/time and create a full backup of these restore to a special tape. Is there a "virtual copy" possible which which I can configure to create one copy job based on backup before a specified time which made the same without having restore them before? Hi Pierre The solution would be to use SQLQuery selection type in your Copy job. These are the options you are looking for: Selection Type = SQLQuery Selection Pattern = "" You just need to devise a query that will return the list of Job IDs you want to copy in the first column of the query output. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Question about differential backup
On 2022-05-24 20:17, egoitz--- via Bacula-users wrote: Good afternoon, One more question, that I have seen it difficult to be absolutely sure just with the documentation available on the net and my Bacula books bought. If I have : Full Differential 001 Differential 002 Differential 003 If I want to restore my machine to Diferential 003 situation... do I need the Full job (obviously) + Differential 001 + Differential 002 + Differential 003 or it's OK if I use Full job (obviously) + Differential 003 ?. Thanks a lot mates! Hi Differential backup covers differences since the latest full job. So, in your example you would need the appropriate Full job and the "Differential 003" job. The idea behind differential jobs is to speed up the restore by avoiding the need of a number of incremental jobs which could reside on different volumes. If you are using only volumes of a file type (stored on a disk storage) and your volumes are always available, then you probably wouldn't benefit from differential backups as disk storage is quite fast and Bacula can open volumes as needed. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula client with several policies retention
On 2022-05-11 17:43, Guillermo Martin via Bacula-users wrote: Hi everyone! Does anyone know if is it possible to configure a bacula client with different policies retention? For example.. 5 years for logs files and 3 months for the rest files. So this client will have 2 file retention, 2 job retention and 2 volume retention. Is it possible to configure it in bacula? Hello Since both, file retention and job retention are set per-client I believe it is not possible. Note that if your file retention is shorter than the job retention and your files expire but their job does not, you will still be able to perform a restore of all the files but will not be able to browse and select specific files for the restoration. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] About bacula-dir.conmsg
On Monday 2022-05-02 15:36:53 Bill Arlofski via Bacula-users wrote: > On 5/2/22 07:54, Guilherme Santos wrote: > > Hey guys, what's up? > > > > Could someone tell me more about bacula-dir.conmsg? Mine is about > > 21G... Why this archive has this size? Thanks!! > > That file is the on-disk bconsole buffer of console messages. > > > You can do one of either: > > 1. Type 'messages' in a bconsole session and watch 21GB of log files > scroll by (not recommended lol) > > 2. Stop the Director, and delete the file, then restart the Director. There is yet another option. 3. Trunkate the file: > /path/to/the/file Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacularis - Bacula web interface
On 2022-05-09 04:36, Marcin Haba wrote: On Sun, 8 May 2022 at 20:51, Josip Deanovic wrote: I interpret that note as an implicit statement that no files unrelated to the selected job will be shown. This is also the most logical decision (at least to me). Also, if full backup is selected, only the files related to that specific backup job should be displayed. Thanks for your idea to the file version box. I think it might be possible to limit listed file versions to versions belonging to selected backup(s). I noted it in my local feature request list. Just noticed that when "Selected backup" method is used, one can search for the file on the given client and select the right job considering the time which is also shown in the rightmost column. This is where user can look for different versions of a file if specific file is what he is looking for. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacularis - Bacula web interface
On 2022-05-08 18:29, Marcin Haba wrote: Hello Josip, On Sun, 8 May 2022 at 15:27, Josip Deanovic wrote: Hello! I have compared Baculum and Bacularis web interfaces using their demo sites. Many thanks for your response and feedback. As always any fresh look at the project is precious and it helps to see it using other pair of eyes. Glad to be of help. I am not sure if Baculum improved its look since the last time I looked into it within a year or is it just me. It is hard to spot the differences and from what I have seen on the demo site, they poses the same features. Bacularis just moved few buttons under existing menus and that's pretty much all. I am curious, what is the actual functional difference and benefit of using one over another? Comparing visually two demos it can look as you described. Bacularis is a fork of Baculum so it contains Baculum functions. To better know the differences, if I could advise something, I would propose to read the Bacularis website. The news section is a good place for start the same as the documentation. I also encourage to visit the GitHub project. Thank you, I will follow your advice if I get more time. Unfortunately I am short with time, that's the reason why I compared project's demo sites instead of performing real life tests. While comparing Baculum and Bacularis demo sites I noticed few strange behaviors which could be caused by a bug. For example, with Baculum, when using restore and selecting either the exact job or chose the latest backup, in both cases at the end one would be presented with multiple files with the same name and location but from different backup job. One could then select multiple of those files with the same name for restoration using "Add" link or using drag-and-drop feature. It is versions box where you can choose version of selected file. It lists all versions of file for selected client, path and file. That doesn't seem intuitive. Why would one expect to see all the files for the client right after the step where one get to chose the exact backup job and on the page it says that it is about: - Backup client: server.baculus.pl-fd - Job type: Backup - Job name: BackupClient1 - Status: Ok - Restore time point: 2022-05-08 20:00:18 When I test the restore procedure, the first step asks me to chose the backup client. On the second step there is a list of available jobs to chose from and only a single job can be selected (selection by radio buttons). There is also a note written in bold which says: "Note: if you select incremental or differential backup, on the next step will be also loaded all directories and files from older backups required to do the job restore. In other words, the selected backup determines time point from which will be loaded the selected backup and other older backups (incremental, differential) backups up till closest full backup." And that note is perfectly fine as one cannot restore files using just incremental or differential backup job. A full backup job is required. I interpret that note as an implicit statement that no files unrelated to the selected job will be shown. This is also the most logical decision (at least to me). Also, if full backup is selected, only the files related to that specific backup job should be displayed. That makes no sense. When you select from file versions box files to restore, you can see yellow tooltips for each file. In this tooltip is information about jobid for file together with few other file version properties. Yes, I noticed that. That's how I confirmed that these are files belonging to multiple unrelated jobs. In fact, on the test server there are only full backup jobs available and since I selected a single full backup jobs I didn't expect to see multiple versions of a file existing in other backup jobs performed for the same backup client. Which job would be used to restore the files added for the restoration? There will be restored file version from the latest job. Ok, that's good but a warning message about two files of a different version selected for the restore might be even better. Maybe a user could be prevented to add another file with the same location/filename to the list until the file already in the list is removed. Also, if I may suggest a feature request, it would be nice to have tooltip similar to the one you have described, implemented for the files already in the list of files selected for the restoration. That would help observing and checking the version of the file previously added to the list. With Bacularis the case is almost identical with the difference that at the last step one cannot add multiple files but can still use drag-and-drop to add them to the list for restoration. It is something to fix. Thanks for catching it. You are welcome. Apart from that, both tools have issues showing schedule configuration for a job. This ha
Re: [Bacula-users] Bacularis - Bacula web interface
On 2022-05-08 08:44, Marcin Haba wrote: Hello Everybody, I am writing to let you know about a new web interface project related to Bacula. It is called Bacularis. It has been created to provide the Bacula web interface in an easy way, with simple installation and configuration. Bacularis is a friendly fork of Baculum. One of the Bacularis goals is to simplify Baculum features that they could be used not only by users with stronger Bacula skills but also by beginners or intermediate users. From the internal architecture side there has been done significant changes. In Bacularis everything is available in one application (API and Web) and during installation users can choose what components they would like to configure. The interface can be installed by binary packages (deb, rpm) or by Composer. I would like to invite you to the Bacularis project website. You can find there much more information about this interface: https://bacularis.app Best regards, Marcin Haba (gani) Hello! I have compared Baculum and Bacularis web interfaces using their demo sites. It is hard to spot the differences and from what I have seen on the demo site, they poses the same features. Bacularis just moved few buttons under existing menus and that's pretty much all. I am curious, what is the actual functional difference and benefit of using one over another? While comparing Baculum and Bacularis demo sites I noticed few strange behaviors which could be caused by a bug. For example, with Baculum, when using restore and selecting either the exact job or chose the latest backup, in both cases at the end one would be presented with multiple files with the same name and location but from different backup job. One could then select multiple of those files with the same name for restoration using "Add" link or using drag-and-drop feature. That makes no sense. Which job would be used to restore the files added for the restoration? With Bacularis the case is almost identical with the difference that at the last step one cannot add multiple files but can still use drag-and-drop to add them to the list for restoration. Apart from that, both tools have issues showing schedule configuration for a job. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] poor utilization of tape space
On 2022-03-08 22:16, Alan Polinsky wrote: Josip: Your help was stellar. All is working properly. Damned how I missed that one. Glad you made it work. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] poor utilization of tape space
On 2022-03-08 19:43, Alan Polinsky wrote: Josip: That is obviously the problem, but I'm not sure why it has occurred, or how to correct it. These are standard LTO2 tapes, that were previously used in a different setting. I used an mt command, mt -f /dev/nst0 weof to write an end of file mark at the begining so I could use bacula to re-label them. All of the tapes are marked with a maxvolbytes of 53,687,091,200. volbytes is set to 64512 in each of them. I guess something was changed significantly between the 5.X version of bacula and 9.6, but it also might be the upgrade of Slackware. I also don't recall the shoeshine parameter being enabled in the prior release. In my current setup, which I will probably have to re-create, my spool files grow quite large. Thank you for pointing my search to a satisfactory solution. Ok, now that we know the cause we can devise a solution. Check your Bacula configuration files for option Maximum Volume Bytes and remove if from pool resources used for tapes. Don't forget to reload or restart bacula-dir daemon. After that, use the update command from bconsole tool. In the prompt chose the option "Pool from resource". That option will instruct Bacula to apply current pool configuration to already created pools. After that, use the update command again and chose "Volume parameters". The next step will allow you to chose the option "Maximum Volume Bytes". After that you will be allowed to chose the pool and the volume name or ID. If the number of volumes you need to update is too big for manual fixing, check the bconsole documentation and help for its update command. It is possible to select all volumes in a specified pool. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] poor utilization of tape space
On 2022-03-08 15:23, Alan Polinsky wrote: I have recently moved from an old version of Bacula FROM THE 5.X series to 9.6, along with my move from Slackware 14.2 to Slackware 15, along with a change to Mariadb 10.5. (I would first like to thank Phil Stracchino for his help in getting things configured properly.) I am backing up to LTO2 tapes which should have a minimum of about 200 gigs of storage. I seem to be getting requests to mount a new tape after about 80 gigs. Initially I thought that perhaps a tape had some sort of defect, but that was disproved when I put in a previously unused tape. Perhaps I have not configured some parameter properly. Can someone suggest what parameter I might be able to change or add to assure more complete tape utilization? (I should mention that the tape drive wrote complete tapes under the old release.) Hi Alan I would suggest to use the bconsole command llist to check the current media parameters. Specifically, check for variables like maxvolbytes, maxvolfiles and maxvoljobs. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Fwd: Job copy question
On Saturday 2022-02-26 07:48:52 Marcelo Slon wrote: > Thanks a lot for your answers. > I will analyze and do my tests. > I'm using Bacula Community version 9.6.7 with PostgreSQL, and in the > database I found a 'status' table with the following data: > |jobstatus|jobstatuslong |severity| > |-|---|| > |C|Created, not yet running |15 | > |R|Running|15 | > |B|Blocked|15 | > |T|Completed successfully |10 | > |E|Terminated with errors |25 | > |e|Non-fatal error|20 | > |f|Fatal error|100 | > |D|Verify found differences |15 | > |A|Canceled by user |90 | > |F|Waiting for Client |15 | > |S|Waiting for Storage daemon |15 | > |m|Waiting for new media || > |M|Waiting for media mount|15 | > |s|Waiting for storage resource |15 | > |j|Waiting for job resource |15 | > |c|Waiting for client resource|15 | > |d|Waiting on maximum jobs|15 | > |t|Waiting on start time |15 | > |p|Waiting on higher priority jobs|15 | > |a|SD despooling attributes |15 | > |i|Doing batch insert file records|15 | > |I|Incomplete Job |25 | > > As you can see it doesn't have the 'W' status that you use in the SQL > query. Does this jobstatus exist in my bacula version or is this table > incomplete? I am not sure what is that table used for but I found it can be found in Bacula documentation: # For 9.6.x https://www.bacula.org/9.6.x-manuals/en/developers/Catalog_Services.html#SECTION001014000 # For 11.0.x https://www.bacula.org/11.0.x-manuals/en/main/Job_status.html#blb:director:job:status JobStatus "W" means "Terminated normally with warnings". For example, it could happen when the job has normal status but some script produced an output (but still return 0). -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Job copy question
On 2022-02-26 00:09, Bill Arlofski via Bacula-users wrote:
Thanks Josip!
I will go look for that post. Sounds like a useful SQL query for
implementing Copy jobs when a history of other jobs already
exists in a pool you'd like to copy from.
I found it in my configurations. I think this is it.
Sorry for bad SQL indentation.
SELECT DISTINCT JobId,StartTime,Type,Level,Name,PriorJobId
FROM Job
WHERE Name = 'YOUR_JOB_NAME'
AND JobBytes > 0
AND ((Level='F' AND JobStatus IN ('T', 'W') AND StartTime >=
(SELECT DISTINCT StartTime
FROM Job
WHERE Name = 'YOUR_JOB_NAME'
AND Level='F'
AND JobStatus IN ('T', 'W')
ORDER BY StartTime DESC LIMIT 1))
OR (Level='D'
AND JobStatus IN ('T', 'W')
AND StartTime >=
(SELECT DISTINCT StartTime
FROM Job
WHERE Name = 'YOUR_JOB_NAME'
AND Level='F'
AND JobStatus IN ('T', 'W')
ORDER BY StartTime DESC LIMIT 1))
OR (Level='I'
AND JobStatus IN ('T', 'W')
AND StartTime >=
(SELECT DISTINCT StartTime
FROM Job
WHERE Name = 'YOUR_JOB_NAME'
AND Level='D'
AND JobStatus IN ('T', 'W')
ORDER BY StartTime DESC LIMIT 1)
OR StartTime >=
(SELECT DISTINCT StartTime
FROM Job
WHERE Name = 'YOUR_JOB_NAME'
AND Level='F'
AND JobStatus IN ('T', 'W')
ORDER BY StartTime DESC LIMIT 1)))
AND NOT JobId = ANY
(select PriorJobId
FROM Job
WHERE Name = 'YOUR_JOB_NAME'
AND PriorJobId <> 0)
AND Type<>'C'
ORDER BY JobId;
Note that in the above query there are six places where
YOUR_JOB_NAME should be replaced with the actual job name.
This used to work on Bacula 7.4. Didn't test it on newer
Bacula versions.
Regards!
--
Josip Deanovic
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Job copy question
On 2022-02-25 23:01, Bill Arlofski via Bacula-users wrote: [...] When customers start doing copy jobs and want to use the convenient "PoolUncopiedJobs" feature, I usually recommend to start by using a series of SQL queries instead. The idea is to configure the Copy job to select jobs between two old dates, then once that batch is done being copied, adjust the SQL query to the next range of dates and continue this process until the current date is reached. Then, the Copy job can be changed to use the PoolUncopiedJobs feature and only the jobs that have not been copied yet will be copied moving forward. The other suggestion I make is, create a new pool and start sending all of your backups there. Then, you can immediately start using the PoolUncopiedJobs feature since only these new jobs will need to be copied. This second idea is OK, only if it is not necessary to copy all the old jobs and you'd like to implement Copy jobs, starting now... And, for testing, just to make sure your Copy jobs are configured correctly, you can make the Copy job select just one specific jobid with the SQL query being simply: SELECT '1234' (where 1234 is the small backup job you want to test your Copy job configuration with. Several years ago I posted to this list an SQL query that copies only the jobs that weren't already copied, while starting from the last successful Full job. The post could still be found in the mailinglist archives. I used it when I had to start using Copy jobs but didn't want to copy the whole pool as it was quite large. -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Job copy question
On 2022-02-25 16:05, Marcelo Slon wrote:
Hi,
I have configured a copy job with the Selection Type = PoolUncopiedJobs
directive.
Is there a way to list the jobs that have not yet been copied before
running this job in the console or by SQL query? Where is this
information
stored in the database?
Hi Marcelo,
There is no place in the database that holds that specific information.
Instead, there is a complex SQL query that returns the list of job ids
upon query execution.
For Bacula version 9.6.7 source it can be found in the file
src/dird/mac.c
or src/dird/mac_sql.c, I don't remember which one.
The query looks like this (I added few columns in the output
but Bacula requires only the first one to be the jobid column):
SELECT DISTINCT
Job.JobId,Job.StartTime,Job.Name,Job.job,Job.JobBytes,Job.JobStatus,Job.Level
FROM Job,Pool
WHERE Pool.Name = '%1'
AND Pool.PoolId = Job.PoolId
AND Job.Type = 'B'
AND Job.JobStatus IN ('T','W')
AND Job.jobBytes > 0
AND Job.JobId NOT IN
(SELECT PriorJobId
FROM Job
WHERE Type IN ('B','C')
AND Job.JobStatus IN ('T','W')
AND PriorJobId != 0)
ORDER by Job.StartTime;
Note that %1 in the query above must be replaced with the name
of your pool.
To make this knowledge more useful, you could add that
query to your /etc/bacula/query.sql file which would allow
you to use bconsole query command.
You still need to modify it a bit to make bconsole query
command ask you for the name of the pool.
The complete query definition modified for use in query.sql
file would look like this (the number 21 means that this is
the 21st query definition in my /etc/bacula/query.sql file):
# 21
: List the successfully completed backup jobs which have not been copied
before
*Enter Pool name:
SELECT DISTINCT
Job.JobId,Job.StartTime,Job.Name,Job.job,Job.JobBytes,Job.JobStatus,Job.Level
FROM Job,Pool
WHERE Pool.Name = '%1'
AND Pool.PoolId = Job.PoolId
AND Job.Type = 'B'
AND Job.JobStatus IN ('T','W')
AND Job.jobBytes > 0
AND Job.JobId NOT IN
(SELECT PriorJobId
FROM Job
WHERE Type IN ('B','C')
AND Job.JobStatus IN ('T','W')
AND PriorJobId != 0)
ORDER by Job.StartTime;
Regards!
--
Josip Deanovic
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Deleting Read Only Volumes
On 2022-02-21 11:46, Chris Wilkinson wrote:
Following a failure of a storage device, I have a number of records
marked as ‘read-only’ in the database.
The backup jobs has been moved to a new storage so these records are
now redundant and I’d like to clean them out.
I looked through bconsole commands but didn’t find a command to do
this except by finding volumes with read only status and deleting them
one by one. Is there a semi-automated way to find and delete read-only
records?
Hi Chris
I think that bconsole delete command would not accept more than
one volume name or volume id.
You could get the list of your read-only volumes from the database
and then use shell one-liner which would send the delete command
through bconsole in a loop.
Instead of retrieving the list of volumes from the database you
could also get them from bconsole + some grepping or awking.
Something like this would give you the list of the names of your
volumes in the Read-Only state from all available pools (you might
want to limit the list to a specific pool using pool=your-pool-name):
echo "list volumes" | bconsole | grep Read-Only | awk '{print $4}'
Then you can take that list of volume names and in a similar manner
call bconsole tool with the delete command in a loop.
Regards!
--
Josip Deanovic
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Reusing schedule resource
On 2022-02-16 11:55, Chris Wilkinson wrote: Is it possible to define two jobs to use the same schedule resource, i.e. have two jobs to run using the same schedule? Hi Chris, Yes, it's possible and it's a common thing. It doesn't seem like that is possible since the schedule resource requires to specify the pools that are used which would usually be different. I wonder if there is a logical reason for requiring schedules to define the pools used when this info is already in the job resource? Definition of a Schedule resource doesn't require pool to be specified. Multiple jobs can use the same pool. This is also a common thing. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Too Big? 11.0.5/FreeBSD Ports
On 2022-02-10 05:21, Larry Rosenman wrote: [snip] Looks like it was my firewall not allowing an ESTABLISHED session over 2 hours. I've increased it to 1 Day, and all seems(!) better. You might want to turn on Heart Beat Interval, both in the File daemon and the Storage daemon. You can set it to a minute or maybe 5 minutes or so. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
On 2022-01-26 20:13, dmitri maziuk wrote: On 2022-01-26 12:57 PM, Josip Deanovic wrote: The number of files per directory is far bigger and is unlikely to get reached, especially not for this use case. The limit is one thing, the scaling is another. I agree: 40TB of 10GB files is not enough to see the slow-down on any modern system, you'd need an order of magnitude more files to get there. Still it's something to be aware of when deciding on volume size. 40 TB is 40960 GB which would give 4096 files, 10 GB in size. Order of magnitude would be 40960 files which is still nothing. Right now on my laptop I have 291794 files and 34481 directories and that's only under /usr. I had systems with hundreds of millions of files on UFS2 (FreeBSD) and systems with billions of files on ext3 (Linux) and that was like 15 years ago. As far as I can remember there were no issues with read/write performance related to the number of files. The issue was backup which would take a lot of time to traverse the whole file system. This is a problem common to all hierarchical databases without some kind of indexing employed to deal with the issue. As long the full path of a file is known, I don't think the read/write performance of a file would change noticeably with the increase of number of files on the file system. Modern file systems are using directory indexing so even searching through a file system doesn't take too long but it's common sense that the time needed to perform a lookup would increase (not necessary linearly) with the number of files on the file system. In any case, Bacula knows the path names of the file volumes and doesn't need to search the file system. I can't imagine the setup where the number of files on the local file system containing Bacula file volumes would pose a problem. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
On 2022-01-26 18:42, dmitri maziuk wrote: If you use actual disks as "magazines" with vchanger, you need to pre-label the volumes. If you use just one big filesystem, you can let bacula do it for you (last I looked that functionality didn't work w/ autochangers). If you use disk "magazines" you also need to consider the whole-disk failure. If you use one big filesystem, use RAID (of course) to guard against those. But then you should look at the number of file volumes: some filesystems handle large numbers of directory entries better than others and you may want to balance the volume file size vs the number of directory entries. Regarding the number of directory entries... It is common to see the file system limit of 32000 directories per directory. The number of files per directory is far bigger and is unlikely to get reached, especially not for this use case. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
On 2022-01-26 18:06, Peter Milesson via Bacula-users wrote: I'm used to fixed volume sizes from the tape drives, I feel comfortable with it, and I do not need to relearn a lot to configure the Bacula system. The only thing I haven't found out is how to preallocate the number of volumes needed. Maybe there is no need, if the volumes are created automagically. Most of the RAID array will be used by Bacula, just leaving a couple of percent as free space. When using mhvtl, I started a script with the tape size and number of tapes I wanted, and the corresponding tape directories and volumes were created on the fly. Thanks Josip! You are welcome. I would like to point out that different requirements people may have will dictate different approaches. Regarding preallocation of the voluems, if there is a way to do it I am not aware of it. However, if you define maximum volume size and the maximum number of volumes in the pool, you should be able to calculate the space needed. Just leave some free space like 2x size of a volume and you should be good. Later, when you use all the volumes you will see if there is enough space to create yet another volume. You can chose to label volumes by yourself or leave that to Bacula. It's up to you. If you intend to recycle your volumes automatically, make sure that your retention periods are short enough to expire before all the volumes are used. Otherwise Bacula will not be able to perform backup. The alternative would be to force the recycle of the oldest volume but this doesn't happen by default, this option must be explicitly turned on. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
On 23.01.2022 11:37, Radosław Korzeniewski wrote: Hello, pt., 21 sty 2022 o 14:22 Peter Milesson via Bacula-users napisał(a): If somebody has got experience with disk based, multi volume Bacula backup, I would be grateful about some information (tips, what to expect, pitfalls, etc.). The best IMVHO (but not the only mine) is to configure one job = one volume. You will get no real benefit to limit the size of a single volume. In the single volume = single job configuration you can set up job retention very easily as purging a volume will purge a single job only. It is not required to "wait" a particular volume to fill up to start retention. Purging a volume affects a single job only. And finally you end up with a way less number of volumes then when limiting its size to i.e. 10G. There are many different approaches which can fit different requirements. I don't see the benefit of having a single job per volume as Bacula is tracking media, files, jobs and everything else. That's why Bacula has a catalog which allows the backup system to determine the location and state of volumes, jobs, files, etc. To logically separate backup data I use pools and leave the rest to Bacula. When Bacula needs a particular file volume, if it's available Bacula will simply use it and if it's not or if we are using tape volume which is currently not in the tape drive/library, Bacula will ask for the volume by name. The number of smaller file volumes (e.g. 10GB) is not an issue as Bacula is handling them correctly and automatically (provided that Bacula is correctly configured, of course). I'll go through few examples where smaller file volumes (e.g. 10GB) could prove useful: 1. If the catalog database get corrupted or completely lost, due to the the small size, it's easier and faster to handle and determine volumes which contain database backup. That makes the process of importing the data into a new catalog database using a tool such as bscan easier. 2. Similar to 1), it is easier to manage small file volumes and extract particular jobs from a volume using bextract tool. 3. If the space is an issue (as it usually is), bigger volumes tend to eat more space which cannot be reused (volume cannot be recycled) as long as the volume contains a single job we want to preserve. 4. Although I don't like that approach, sometimes people chose to sync or copy whole file volumes to a secondary location using the usual tools such as rsync, cp and similar. In such case it is better to keep file volumes small. 5. When recycling a file volume, it will take longer time to wipe bigger file volume. If a volume is smaller it will take less time to recycle ensuring more time windows where other tasks could benefit from I/O performance. In case of large file volumes all other tasks would have to fight for the opportunity to access the file system and that gets more obvious when a slow network file system is being used. 6. In case of any kind of corruption of a file volume due to the file system corruption or damage in transport, it is likely that less data will be lost in case of a smaller file volume. And again, it's easier to handle smaller file volume when trying to recover pieces of data. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
On 2022-01-21 15:29, Peter Milesson via Bacula-users wrote: thanks for your input. Today, I am using 160GB virtual tape volumes with mhvtl and that seems to be a good trade off. 10GB volumes in the new setting would be a nightmare to manage. That would be around 3,000 volumes! The volumes will be on a physical RAID set, but I will anyway use an SSD drive for spooling. One of the bottlenecks today is the slow throughput of the physical disks. It is not worth investing in the old rig, as it is anyway ripe for decommission. I have many thousands of file volumes and I don't see any problem managing them. They are being managed by Bacula. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual tapes or virtual disks
On 2022-01-21 14:03, Peter Milesson via Bacula-users wrote: Hi folks, I am building a new backup server that is going to replace the old one. The old backup server uses Bacula ver. 9.2.2 with a virtual tape library (mhvtl). I have found mhvtl a bit tricky, mostly when updating the OS (CentOS 7.8), as it is necessary to recompile the mhvtl kernel driver. Mhvtl also seems a bit outdated, with intermittent development and updates, also problematic with newer and coming Linux kernel versions. I also want to jump off the RedHat train, as I see it deviate more and more from mainstream Linux. Therefore, I would prefer to choose another solution. I have studied the Bacula documentation and it seems to be possible to use disk based backup with auto changer role. I plan to use volumes with a size of around 200Gbytes, making the setup fairly flexible, not making a too big hole when volumes need to be purged. Total disk space is around 30TB. If somebody has got experience with disk based, multi volume Bacula backup, I would be grateful about some information (tips, what to expect, pitfalls, etc.). Hi Peter, I am using file volumes for many years and I can't recall any special tips or pitfalls worth mentioning. Make sure the directory ownership and permissions are correct, that is, that storage daemon is able to access it (R/W). There are some differences regarding options such as Random Access, Removable Media and Device Type. As disk is not a sequential access type of media, you should set the Random Access option to yes. Unlike tape, a disk is not removable media so you may want to set RemovableMedia option to no. Regarding the volume size, I always chose size of 10 GB. You will need to test whether there is a gain of employing a SpoolDirectory for your disk based backup. If your file system containing file volumes is being mounted over a network (e.g. using iSCSI or NFS), it might be a good idea to use a local spool directory. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula ACL
On 2022-01-12 06:28, Shaligram Bhagat, Yateen (Nokia - IN/Bangalore) wrote: Hi All, We have backup being taken for filesets of different users (users co-located on a single Linux host as well as individual user with his/her private Linux host) At the moment any user can restore other user's data through bconsole/BAT How can we put some kind of ACL's to ensure that a given user has restore access only to his/her fileset? We have Bacula 9.4.4 on Centos. Hi Shaligram, You could add separate Console resources to your bacula-dir.conf and configure them with different passwords (and SSL/TLS if you are using transport encryption). You could then define which Console is allowed to access which file daemon, file set etc. Here is the relevant documentation page for your version (9.4.x): https://www.bacula.org/9.4.x-manuals/en/main/Configuring_Director.html#SECTION002019 Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula for QNAP?
On 2022-01-02 12:46, Radosław Korzeniewski wrote: Hello, sob., 1 sty 2022 o 23:52 Phil Stracchino napisał(a): (...) /usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: ../lib/libbac.a(plugins.o): in function `load_plugins(void*, void*, char const*, char const*, bool (*)(Plugin*))': /netstore/src/bacula-11.0.5/src/lib/plugins.c:150: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking As I understand this issue is that glibc does not support dlopen with static linking as the code which implements dlopen is dynamically linked only. I could be very wrong about my understanding. /usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: cannot find -lssl /usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: cannot find -lcrypto /usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: cannot find -lcap /usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: cannot find -llzo2 collect2: error: ld returned 1 exit status make[1]: *** [Makefile:240: static-bacula-fd] Error 1 Do we actually still support static builds? I think it is not very well maintained as very few users require and use it. The solution to it would be to disable a plugin support. I'm not sure if such an option is currently available. The problem is with the openssl support. File daemon would probably statically compile without openssl support. Last time I did that was with the 7.x version of bacula. I would certainly like to be able to statically compile file daemon. It would make preparations for bare metal restore using Bacula much simpler. Right now, if encryption is used for transport between bacula components, one would need to copy all the libraries file daemon depends on and all their dependencies including openssl libraries which are loaded using dlopen(3) and of course, all their dependencies and so on... Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] How are "new" files determined?
On 2021-12-30 11:37, Gary R. Schmidt wrote: On 30/12/2021 20:03, Sebastian Suchanek wrote: Hi everyone! How exactly are "new" files, which have to be backed up, determined by Bacula? The reason for asking: Recently, I migrated my main data to a new set of hard drives. I did so by using "rsync -aH" which *should* leave all file timestamps etc. untouched. After the migration was finished, also all paths etc. were exactly the same as before, just with a new set of HDDs "underneath". However, in the next scheduled Bacula run (which was scheduled as Incremental), Bacula stared to backup *everything* of the abovementioned data. (I cancelled the job and will resume doing backups on this particular fileset with next regular Full backup which is scheduled for the upcoming Sunday anyway.) So - what went wrong here and how could this have been avoided? The inodes *all* changed, it is effectively a new file system, it will need to be fully backed up. The documentation says that Bacula decides what files to backup for Incremental and Differential backup by comparing the change (st_ctime) and modification (st_mtime) times of the file to the time the last backup completed. I performed a simple test exactly a month ago. What changes during the restore is the change time (st_ctime). That's why Bacula decides to backup restored files with the next Incremental backup (after the restore). The case with rsync copy is similar to the the restoration. st_ctime is used to keep the time of the last change on the inode (e.g. ownership, permissions, hard link count and maybe something else). Rsync option -a (archive) includes options -r, -l, -p, -t, -g, -o, -D. None of these options will preserve change time thus bacula will see all the files as candidates for backup. As Gary said, this is a new file system with newly created inodes so all the files now probably have the change time that corresponds to the time when the rsync did the copy. This can be checked with the stat(1) tool. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Backup Windows 10 and selective restore after changing to Linux Mint
On 2021-12-30 12:59, Earl Harper wrote: Backup Windows 10 and selective restore after changing to Linux Mint Hello, I want to backup my Windows 10 computer before I erase it and set up Linux Mint on it. Then I would like to be able to selectively restore files from the windows 10 backup to the new linux mint system. I know that I could just copy some file over but, knowing my luck, I would miss something important. Will any of these programs allow me to do this? I know that I would need a copy for windows to do the backup and a copy for linux to restore any files. My fear is can the windows and linux programs open the data for the other? Hi Earl If migration from Windows to Linux is the only reason for the backup you plan to perform then I would say that Bacula is huge overkill in this case. Especially if you don't have previous experience with Bacula. Linux has support for NTFS and FAT access (read/write) but to access files on Linux file systems from Windows you would need to install tools and drivers which would bring support for a specific Linux file systems on Windows OS. Also, when copying files between NTFS or FAT file systems to Linux file systems, you will not get the exact mapping of the permissions and you might lose some other file system specific metadata. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula-fd version
On 2021-12-29 03:18, Richard Couture wrote: The client bacula-fd is version 7.4.3 (18 June 2016) and the director and storage daemons are version 9.6.7 (10 December 2020) Is that too great a version disparity? It's not. I have director and storage daemon of version 9.6.7 and few file daemons of version 5.0.0 which I plan to upgrade soon. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] make_catalog_backup stopped working
On 2021-12-28 17:33, Graham Dicker wrote: Looking good. A full backup completed and make_catalog_backup performed flawlessly, which was the original problem. All done manually. Tomorrow morning it'll be back on automatic and I have every expectation that when I finish breakfast there will be emails waiting for me reporting successful backups. I can't tell you how grateful I am for all the help from Graham Sparks, Phil Stracchino and Josip Deanovic. Thank you very much guys, ever in your debt. Glad to hear it worked for you. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] make_catalog_backup stopped working
On 2021-12-28 14:18, Graham Dicker wrote: On Tuesday, 28 December 2021 12:03:05 GMT Josip Deanovic wrote: > Is dropping user bacula the action that emptied all the Bacula tables > of data? > I know very little about the behaviour of mysql. Anyway, the tables > were still > there but empty so I created a new volume and Bacula is now happily > backing up > to it. It will take some hours yet to finish that. Ok, so everything is in order now? Dropping user using "drop user" command in mysql client wouldn't cause deletion or truncation of data from the tables. I just did "drop user bacula" and that only removed row from mysql.user table that looked like: Host | User --+--- % | bacula When I issued the command "drop user 'bacula'@'localhost', the other row containing bacula user password was deleted as well: Host | User --+--- localhost | bacula Tables and the data inside the tables were left intact as it should be. I was mistaken. The data from my backups going back for months is still all there and I can see it using mysql. Using bconsole Bacula can see the tables (or did it create new ones?) but not the data in them. That's why it told me it needed a new volume. If you performed import of the data at some point, this is what created the tables (and probably populated them with the data unless the backup didn't contain the data). It all depends on exact actions you performed. Remember that you have been using dedicated akonadi mysql instance which kept the data in the /home/graham/.local/share/akonadi/db_data/ directory. Now your mysql instance keeps the data in the /var/lib/mysql which is the default for most Linux distributions and MySQL installations. -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] make_catalog_backup stopped working
On 2021-12-28 12:41, Graham Dicker wrote: On Tuesday, 28 December 2021 11:21:19 GMT Josip Deanovic wrote: Could send the output of this command: mysql -u root -e "show databases; show variables where variable_name='datadir' or variable_name='socket';" This will output all your databases and in addition, it will output the datadir and the socket your mysql is using. This will help us to determine that you are indeed not using akonadi instance. If everything is ok you would just need to enter the correct database credentials into the bacula-dir.conf and there should be no problems. Instead of dbaddress you would just need to specify dbsocket. If you don't specify dbsocket, bacula will use the default mysql socket which might nor might not be the correct path for your mysql instance (depending on the mysql setup). .++ .| Database | .++ .| bacula | .| information_schema | .| mysql | .| performance_schema | .| test | .++ .+---+---+ .| Variable_name | Value | .+---+---+ .| datadir | /var/lib/mysql/ | .| socket| /run/mysql/mysql.sock | .+---+---+ Ok, everything looks good here. This is the correct mysql instance and not the one dedicated to akonadi. Is dropping user bacula the action that emptied all the Bacula tables of data? I know very little about the behaviour of mysql. Anyway, the tables were still there but empty so I created a new volume and Bacula is now happily backing up to it. It will take some hours yet to finish that. Ok, so everything is in order now? Dropping user using "drop user" command in mysql client wouldn't cause deletion or truncation of data from the tables. I just did "drop user bacula" and that only removed row from mysql.user table that looked like: Host | User --+--- % | bacula When I issued the command "drop user 'bacula'@'localhost', the other row containing bacula user password was deleted as well: Host | User --+--- localhost | bacula Tables and the data inside the tables were left intact as it should be. -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] make_catalog_backup stopped working
On 2021-12-28 11:20, Graham Dicker wrote: On Tuesday, 28 December 2021 09:31:46 GMT Josip Deanovic wrote: Could you please send the output of this command: ps auxww | grep -i mysql Yes of course - see below. But the mariadb service actually starts on boot. akonadi doesn't start at all until I run KMail, at which point it presumably connects to the already running mariadb/mysqldb. I have to do that because I need to have Bacula running long before I get out of bed and use KMail ;-). Of course. :-) mysql 1263 0.0 1.6 1595128 128768 ? Ssl 08:41 0:00 /usr/sbin/ mysqld --defaults-file=/etc/my.cnf --user=mysql graham2511 0.0 1.5 2484768 123896 ? Sl 08:43 0:01 /usr/sbin/ mysqld --defaults-file=/home/graham/.local/share/akonadi/mysql.conf -- datadir=/home/graham/.local/share/akonadi/db_data/ --socket=/run/user/1000/ akonadi/mysql.socket --pid-file=/run/user/1000/akonadi/mysql.pid That shows that there are two mysql instances as it should be. One is started by akonadi and the other by the system (user mysql). That is ok and this is how it should be. As I thought, akonadi is using its own mysql config file and its own datadir. This is ok. Could send the output of this command: mysql -u root -e "show databases; show variables where variable_name='datadir' or variable_name='socket';" This will output all your databases and in addition, it will output the datadir and the socket your mysql is using. This will help us to determine that you are indeed not using akonadi instance. If everything is ok you would just need to enter the correct database credentials into the bacula-dir.conf and there should be no problems. Instead of dbaddress you would just need to specify dbsocket. If you don't specify dbsocket, bacula will use the default mysql socket which might nor might not be the correct path for your mysql instance (depending on the mysql setup). -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] make_catalog_backup stopped working
On 2021-12-28 10:23, Graham Dicker wrote: On Monday, 27 December 2021 18:56:32 GMT Phil Stracchino wrote: On 12/27/21 13:38, Graham Dicker wrote: > On Monday, 27 December 2021 18:19:23 GMT Phil Stracchino wrote: > > The blazes? If mysql.user is a VIEW, something is VERY BADLY > > WRONG. > > Are you still using Akonadi's MySQL instance? > > Not as far as I can tell. I changed /etc/my.cnf to say: > > port = 3306 And you are certain that the mysqld listening on that port is NOT Akonadi's? If Akonadi's mysqld is listening on port 3306, and you install a second mysqld, it will be unable to bind port 3306. You cannot have two different processes listening on the same port. Try this command as root: lsof -i | grep 3306 and see which mysql process is listening on that port. Then look at the process tree (ps axf) and see what that process belongs to. The command lsof -i | grep 3306 as root returns nothing lsof -i -U | grep mysql returns this COMMANDPID USER FD TYPE DEVICE SIZE/OFF NODE NAME mysqld1263 mysql 58u IPv4 35060 0t0 TCP localhost:mysql (LISTEN) mysqld1263 mysql 59u unix 0x8c5c0484e600 0t0 35061 /run/ mysql/mysql.sock type=STREAM mysqld1263 mysql 102u unix 0x8c5b56e50cc0 0t0 65617 /run/ mysql/mysql.sock type=STREAM mysqld2511 graham 17u unix 0x8c5bce1d72c0 0t0 39643 /run/ user/1000/akonadi/mysql.socket type=STREAM mysqld2511 graham 26u unix 0x8c5bdbc69540 0t0 39665 /run/ user/1000/akonadi/mysql.socket type=STREAM mysqld2511 graham 123u unix 0x8c5bdbc6aa80 0t0 40731 /run/ user/1000/akonadi/mysql.socket type=STREAM mysqld2511 graham 124u unix 0x8c5bce1d2640 0t0 39676 /run/ user/1000/akonadi/mysql.socket type=STREAM mysqld2511 graham 125u unix 0x8c5bb77e4c80 0t0 43085 /run/ user/1000/akonadi/mysql.socket type=STREAM mysqld2511 graham 126u unix 0x8c5b60ce0440 0t0 66284 /run/ user/1000/akonadi/mysql.socket type=STREAM mysqld2511 graham 138u unix 0x8c5c05025d80 0t0 51606 /run/ user/1000/akonadi/mysql.socket type=STREAM mysql 3666 root3u unix 0x8c5b56e550c0 0t0 60155 type=STREAM poking about in the process tree returns lots of cryptic commands. The number 3306 doesn't feature anywhere except /etc/my.cnf and bacula-dir.conf even though I'm running mysql, KMail and Bacula at the time. It would seem that you are not running "normal" mysql service. It seems that you are running only the one started by akonadi. Could you please send the output of this command: ps auxww | grep -i mysql -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] make_catalog_backup stopped working
On 2021-12-27 21:07, Dmitri Maziuk wrote: On 12/27/2021 2:02 PM, Josip Deanovic wrote: On 2021-12-27 19:19, Phil Stracchino wrote: ... The blazes? If mysql.user is a VIEW, something is VERY BADLY WRONG. Are you still using Akonadi's MySQL instance? Yes, that doesn't look normal. I hope it's not something Suse specific. Google sez it's one of the many improvements in mariadb 10.4. Does bacula support mariadb at all? Hmm... That's something I wasn't aware of. It seems that I get to learn something today as well. I don't think that such change would affect bacula support for mariadb but there might be some other changes as well. -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] make_catalog_backup stopped working
On 2021-12-27 21:01, Dmitri Maziuk wrote: Thank you, I learned something today (not that I wanted to: I try to stay away from that POS as much as I can, but that doesn't work 100%). Now I know why you had to grant to both 'root'@'localhost' and 'root'@'127.0.0.1' -- in addition to 'root'@'%' of course. Special care should be used when allowing access from other hosts. For example that 'root'@'%' would allow root access from any IP. Most people continue to keep mysql root user without password which might prove as a bad practice, especially in constellation with incorrectly set access permissions (grants). Also, all traffic is plain-text and very few people decide to configure TLS encryption although MySQL supports is for decades. -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] make_catalog_backup stopped working
On 2021-12-27 19:19, Phil Stracchino wrote: On 12/27/21 12:10, Josip Deanovic wrote: On 2021-12-27 14:24, Graham Dicker wrote: I can't get those commands to work MariaDB [mysql]> CREATE USER 'bacula'@'localhost' IDENTIFIED BY 'xxx'; ERROR 1396 (HY000): Operation CREATE USER failed for 'bacula'@'localhost' MariaDB [mysql]> UPDATE mysql.user SET Host='localhost' WHERE User='bacula'; ERROR 1356 (HY000): View 'mysql.user' references invalid table(s) or column(s) The blazes? If mysql.user is a VIEW, something is VERY BADLY WRONG. Are you still using Akonadi's MySQL instance? Yes, that doesn't look normal. I hope it's not something Suse specific. -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] make_catalog_backup stopped working
On 2021-12-27 19:56, Phil Stracchino wrote: On 12/27/21 13:38, Graham Dicker wrote: On Monday, 27 December 2021 18:19:23 GMT Phil Stracchino wrote: > The blazes? If mysql.user is a VIEW, something is VERY BADLY WRONG. > Are you still using Akonadi's MySQL instance? Not as far as I can tell. I changed /etc/my.cnf to say: port = 3306 And you are certain that the mysqld listening on that port is NOT Akonadi's? If Akonadi's mysqld is listening on port 3306, and you install a second mysqld, it will be unable to bind port 3306. You cannot have two different processes listening on the same port. Try this command as root: lsof -i | grep 3306 and see which mysql process is listening on that port. Then look at the process tree (ps axf) and see what that process belongs to. Remember that if you install a second mysqld, it will need to have its own separate mysql configuration file(s), and listen on its own separate socket and its own separate port, and you will have to configure bacula to use that socket and/or port. Personally I'm not a fan of PostgreSQL. Speaking as a computer scientist there are things about its internals that make me want to tear my hair out. But in this case you may actually have an easier time dumping your catalog schema from MySQL and importing it into PostgreSQL than in keeping two distinct MySQL instances properly separate when (and I do not intend any offense by this) it is clear that you don't really understand how to manage MySQL. The other option might be to set up a Bacula catalog DB on a different host that is not running Akonadi, migrate your catalog DB to that instance, and then configure the Director to talk to that host. I don't know how things look on Suse these days but on Centos 7.x There is only one MySQL installation on the system (meaning that akonadi doesn't bring its own mysql binaries and that is a good thing). Akonadi is using mormal mysql binaries to start its dedicated mysql instance using its own config file, datadir and unix socket. So it is unlikely that mysql used by akonadi is listening on the port 3306/TCP. -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] make_catalog_backup stopped working
On 2021-12-27 19:38, Graham Dicker wrote: On Monday, 27 December 2021 18:19:23 GMT Phil Stracchino wrote: On 12/27/21 12:10, Josip Deanovic wrote: > On 2021-12-27 14:24, Graham Dicker wrote: >> I can't get those commands to work >> >> MariaDB [mysql]> CREATE USER 'bacula'@'localhost' IDENTIFIED BY 'xxx'; >> ERROR 1396 (HY000): Operation CREATE USER failed for >> 'bacula'@'localhost' >> MariaDB [mysql]> UPDATE mysql.user SET Host='localhost' WHERE >> User='bacula'; >> ERROR 1356 (HY000): View 'mysql.user' references invalid table(s) or >> column(s) The blazes? If mysql.user is a VIEW, something is VERY BADLY WRONG. Are you still using Akonadi's MySQL instance? Not as far as I can tell. I changed /etc/my.cnf to say: port = 3306 I realized (eventually) that I couldn't create another user bacula without dropping the existing one first. So I dropped user bacula but now I can't create it again. create user 'bacula2'@'localhost'; works but create user 'bacula'@'localhost'; results in ERROR 1396 (HY000): Operation CREATE USER failed for 'bacula'@'localhost' Did you use drop user or did you manually remove user from the user table in the mysql database? If you manually deleted user from the mysql database, you need to issue the command: flush privileges; -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] make_catalog_backup stopped working
On 2021-12-27 19:28, Dmitri Maziuk wrote:
On 12/27/2021 7:24 AM, Graham Dicker wrote:
...
If I change the Catalog definition from
dbname = "bacula"; dbuser = "bacula"; dbpassword = ""
to
dbname = "bacula"; dbuser = "root"; dbpassword = ""
then Bacula does work but it doesn't "see" the volumes already in the
database
and wants
to create new ones when I try to run a backup.
'bacula'@'%' is supposed to match bacula@ANY_HOST ('%' being the SQL
wildcard) but in my limited mysql experience it never worked 100% of
the time.
'localhost' and '%' are special keywords.
'localhost' applies only to unix socket.
'%' has a meaning of any AF_INTET socket (any IP address or host)
So, the '%' does not include 'localhost'.
--
Josip Deanovic
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] make_catalog_backup stopped working
Privileges for user bacula granted on database bacula. ## This is an example of data import from the bacula_db.sql file [root@test ~]# mysql -h localhost -u bacula -p bacula < bacula_db.sql Enter password: [root@test ~]# echo $? 0 [root@test ~]# /usr/libexec/bacula/make_catalog_backup.pl -m MyCatalog bacula.BaseFiles OK bacula.CDImagesOK bacula.Client OK bacula.CountersOK bacula.Device OK bacula.FileOK bacula.FileSet OK bacula.FilenameOK bacula.Job OK bacula.JobHistoOK bacula.JobMediaOK bacula.LocationOK bacula.LocationLog OK bacula.Log OK bacula.Media OK bacula.MediaType OK bacula.PathOK bacula.PathHierarchy OK bacula.PathVisibility OK bacula.PoolOK bacula.RestoreObject OK bacula.SnapshotOK bacula.Status OK bacula.Storage OK bacula.UnsavedFilesOK bacula.Version OK Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] make_catalog_backup stopped working
On 2021-12-27 12:55, Graham Dicker wrote: I tried (while running mysql as root) MariaDB [(none)]> UPDATE mysql.db SET Host='localhost' WHERE User='bacula'; that worked OK but this didn't MariaDB [(none)]> UPDATE mysql.user SET Host='localhost' WHERE User='bacula'; ERROR 1356 (HY000): View 'mysql.user' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them I'm sure Bacula never used to involve this much mucking about! The configure and build process used to take care of all this didn't it? But this is not about bacula, this is about basic mysql administration. Bacula is completely innocent here. :-) So, what you need to do is to start mysql service as usual and connect to that mysql instance as user root. If you cannot connet to that mysql instance like this: mysql -u root ... then this is another problem (assuming you didn't setup root password already). After you connect to the correct mysql instance as root, you will be able to "enter" the database you need which is in this case "mysql" database so you would need to issue this command: "use mysql;" After that you would need to perform update you already tried: "update user set Host='localhost' where User='bacula';" And after that you would need to issue the command "flush privileges;" which is in this case necessary because you are modifying data on the mysql system tables. Instead of all of the above, you could just add bacula user in mysql client like this: CREATE USER 'bacula'@'localhost' IDENTIFIED BY 'your password'; Then you could create bacula database using the command in mysql client: "create database if not exists bacula;" After that you would need to use the GRANT command mentioned in previous posts. The purpose of GRANT command is to give privileges over the "bacula" mysql database to the "bacula" mysql user. When this is done, you should be able to connect as user "bacula" to the database "bacula" using mysql client like this: mysql -h localhost -u bacula -p bacula After that, you should be able to import your bacula data into the bacula database using mysql client like this: mysql -h localhost -u bacula -p bacula < bacula_db_backup.sql The only thing left to would be to update your bacula-dir.conf accordingly and that's it. Note that for mysql client there is a difference between localhost and 127.0.0.1. Localhost means that the client will be using unix socket whereas the 127.0.0.1 will instruct the mysql client to open TCP connection to 127.0.0.1:3306 (by default). Apologies for typos if there are any, I am writing in a hurry. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] make_catalog_backup stopped working
On 2021-12-27 10:34, Graham Dicker wrote: [...] Thank you for the information. mysql> select user, host from mysql.user; returned the following (without the leading dots) : . +-+---+ . | User| Host | . +-+---+ . | bacula | % | . | | localhost | . | mariadb.sys | localhost | . | mysql | localhost | . | root| localhost | . | | vivaldi | . +-+---+ I don't use passwords in my Bacula config files so I tried: MariaDB [(none)]> GRANT ALL PRIVILEGES ON bacula.* TO 'bacula'@'localhost'; but it returned: ERROR 1133 (28000): Can't find any matching row in the user table That's because you don't have the row that looks like: . +-+---+ . | User| Host | . +-+---+ . | bacula | localhost | You have only: - +-+---+ . | User| Host | . +-+---+ . | bacula | % | Which is nit the same thus the error "Can't find any matching row in the user table". Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] make_catalog_backup stopped working
On 2021-12-26 18:19, Phil Stracchino wrote: If so: mysql> GRANT ALL PRIVILEGES ON bacula.* TO 'bacula'@'localhost' (or whatever DB user you want bacula to connect as) IDENTIFIED BY 'bacula password goes here'; mysql> FLUSH PRIVILEGES; When using MySQL roles and grant statement to change privileges, the command "flush privileges" is not necessary but it doesn't hurt. When modifying mysql/user and similar tables then it is necessary to issue "flush privileges" in order to re-read and apply changes made to the tables. If there is not an existing user for bacula, run the same command, but make up a new username. (For sanity's sake I suggest using 'bacula', then you'll never be in any doubt what it is.) Then go back into your Bacula director config, set the mysql credentials to what you just set above, restart the Director, and try again. KMail/akonadi is a separate issue. I'm not a fan of services that set up their own embedded mysql instances. Multiple mysql (or most other database) instances running on the same host will ALWAYS, *ALWAYS*, perform worse, for all services, than a single properly tuned instance utilizing the same resources. Unfortunately, many years ago KDE PIM application suite moved to akonadi which uses and manages its own mysql instance. That made it awfully slow and unstable, hard to debug and understand and even harder to trace the problem and fix but the wheel of "progress" doesn't care much about functionality. Also, you mentioned "embedded instance" but I am not sure if this can be called embedded instance because the there is an actual mysql daemon running. It just uses dedicated config file and datadir. MySQL used to support embedded instance which meant that you could access your data without daemon running. This was a long time ago and I don't know if such support still exists in MySQL. I remember that the data file needed to be preallocated to about 32MB which was too much for my purposes. That's when I learned about SQLite and moved in different direction. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] make_catalog_backup stopped working
On 2021-12-26 13:47, Graham Dicker wrote: Thank you Josip this is very helpful. I have restored everything I had tinkered with back to what it was, that is: Removed specification of port and socket from /etc/my.cnf Removed specification of port and socket from bacula-dir.conf both of this were previously specifying the akonadi socket and port 3306 now './bacula' start from a console as root puts this message in the log: 26-Dec 09:40 bacula-dir JobId 0: Fatal error: Could not open Catalog "MyCatalog", database "bacula". If I run mysql from a console as root then 'use bacula' succeeds and I can access the tables but if i try the same thing as a normal user I get: MariaDB [(none)]> use bacula; KMail/akonadi behaviour is still not normal. I have to click the Kmail icon, then the desktop, then KMail icon again to get it to start, and at least the first time of collecting my POP3 email takes a lot longer than usual. Graham I see that my attempts to paste messages copied from a console window into an email are not working as I expect. Trying again...The message above should have read: 26-Dec 09:39 bacula-dir JobId 0: Fatal error: Could not open Catalog "MyCatalog", database "bacula". 26-Dec 09:39 bacula-dir JobId 0: Fatal error: mysql.c:293 Unable to connect to MySQL server. Database=bacula User=bacula MySQL connect failed either server not running or your authorization is incorrect. 26-Dec 09:39 bacula-dir ERROR TERMINATION Please correct configuration file: /opt/bacula/etc/bacula-dir.conf Ok. So, leave akonadi to use its own mysql instance and don't use it for anything else. This is the default setup on the Centos/RHEL and I assume that Suse is doing the same in that regards. So, you need to configure your separate (I would say normal) mysql service which you can use for all your purposes (including bacula). That would mean that you would need to configure your /etc/my.cnf to point to the correct mysql data directory (e.g. datadir=/var/lib/mysql), you would need to define either some socket (e.g. socket=/var/lib/mysql/mysql.sock) or IP and port. Then you would need to enable and start that service using the the usual procedure that applies for you OS (probably using systemctl tool as this is the most common thing these days, on Linux distributions). Then you would need to create the database you plan to use for bacula and its database user. After that you will need to populate it with the bacula tables. Be careful to use the correct credentials when connecting to the database (make sure you are using the correct mysql service and not the one dedicated to akonadi). After that, you will need to reconfigure your bacula-dir.conf the way it uses correct connection info and user credentials. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] make_catalog_backup stopped working
On Friday 2021-12-24 13:32:42 Graham Dicker wrote: > Thanks for the tip Phil, which I will bear in mind if I ever get Bacula > working again as it should (and has done for the past 10 years or more). > Up until about a year ago my workstation would boot itself up when > woken by the motherboard timer and Bacula would start running as a > service and do the necessary backups. The first thing that stopped > working was the service. There seemed to be some problem with timing > between Bacula and mysql. I got around it by dropping the service and > using crontab to start mysql and Bacula in that order. Then a few > months ago that started failing due to some disagreement between > akonadi, Bacula and mysql as to where the mysql socket was and which > port was to be used. I solved that by total surrender to akonadi and > told Bacula to use the socket created by akonadi. Now I don't know > what's gone wrong. KMail behaviour has also changed in the last few day > also; I have to click on it twice to make it start up. I think OpenSuse > 15.3 must have had some dodgy updates recently. That would explain why your database disappeared. It seems that you have been using mysql instance dedicated to akonadi service. I don't know how things are setup on Suse but on Centos/RHEL, akonadi service would by default bring up its own mysql instance, with its own mysql config file and data directory. It would also use its own socket. I wouldn't use that mysql instance for anything else but akonadi. I strongly suggest that you configure and bring up normal mysql service which can safely be used for bacula and other services as you se fit. If configured correctly you will not lose your databases any more. The timing between mysql and Bacula you have mentioned is likely caused by the akonadi which starts its mysql instance and that happens only during the graphical login or even after you start the "kontact" or similar application such as "kmail". Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] make_catalog_backup stopped working
On Thursday 2021-12-23 14:42:12 Graham Dicker wrote: > Hi all > > I'm running Bacula on a desk machine with OpenSuse 15.3 and Bacula > 11.0.4. > > The make_catalog_backup command has stopped working following a complete > loss and restore of my Bacula database. > I don't know how the database vanished but I restored it from the last > successful backup thus: > > ./bextract -b ../working/BackupCatalog.bsr /media/seagate4tb /tmp > mysql > create database bacula; > use bacula; > source bacula.sql; > grant_mysql_privileges > > But now although my main backup works fine the backup of the catalog > gets this error: > > 23-Dec 13:39 vivaldi-dir JobId 65: BeforeJob: mysqldump: Got error: > 1044: "Access denied for user ''@'localhost' to database 'bacula'" when > selecting the database > > Any clues? Thanks in advance. Hello Graham! The problem here is the access to the MySQL database. Maybe the script that does the catalog backup doesn't use the correct database user. Maybe you are using non-default location of the bacula director configuration file? The error says: "Access denied for user ''@'localhost'". As you can see the username part is empty. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] make_catalog_backup stopped working
On Thursday 2021-12-23 13:04:13 Phil Stracchino wrote: > On 12/23/21 11:19, Graham Dicker wrote: > > Thank you for your reply Graham. > > > > I don't think Bacula itself has any problems, it is only the script > > mysqldump that has a problem. The script is just supposed to create a > > dump of the database and (if it is successful) then Bacula goes on to > > first back it up and then delete it. > > I'm gonna be very blunt here. mysqldump is a piece of obsolete garbage > and nobody should be using it any more. It CANNOT, period, CORRECTLY > back up both My ISAM and InnoDB tables in the same dump. CANNOT. > (Nobody should be using the MyISAM storage engine any more either, but > that's another issue.) > > Take a look at mydumper instead. It is better than mysqldump in almost > every imaginable way. It is parallelized, storage engine aware, and > immensely easier to do partial restores with (i.e a single schema or a > handful of tables). It is what mysqldump SHOULD have evolved into > fifteen years ago. I would suggest XtraBackup in case of MySQL Percona and Mariabackup in case of MariaDB 10.1 and newer. The tool is supported by the Percona and MariaDB software distribution. It produces no downtime and supports incremental backups. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Copy job restores, again
On Saturday 2021-12-25 00:12:09 Phil Stracchino wrote: > For the first time I'm trying to run a restore directly from a copy job, > after a complete failure of my NAS. I've got a second storage daemon > temporarily installed on my workstation, I have the external disk > chassis that holds my rotating archive copy sets attached to a > temporary server running a clean Solaris 11.4 install, and I have the > most recent full backup set copy set mounted. And I'm trying to do a > test restore of 30 or so small files. > > Status dir says: > > JobId Type Level Files Bytes Name Status > == > 34562 Rest Rest 0 0 Restore is waiting > for a mount request > > > Status storage says: > > > Reading: Full Restore job Restore JobId=34562 > Volume="ARCHIVE-20211206-14:00" > pool="Scratch" device="ArchiveCopy" (/arcpool) newbsr=0 > Files=0 Bytes=0 AveBytes/sec=0 LastBytes/sec=0 > FDReadSeqNo=7 in_msg=7 out_msg=7 fd=6 > Director connected at: 24-Dec-21 23:59 > > > Jobs waiting to reserve a drive: > > > Terminated Jobs: > JobId LevelFiles Bytes Status FinishedName > === > 34558 Rest 0 0 Cancel 24-Dec-21 23:17 Restore > 34559 Rest 0 0 Cancel 24-Dec-21 23:45 Restore > > > Device status: > > Device File: "ArchiveCopy" (/arcpool) is not open. > Device is BLOCKED waiting for mount of volume > "ARCHIVE-20211206-14:00", Pool:Scratch > Media type: File > Available Space=711.4 GB > > > And if I try to mount: > > *mount > Automatically selected Catalog: Catalog > Using Catalog "Catalog" > Automatically selected Storage: babylon5-archive > 3906 File device ""ArchiveCopy" (/arcpool)" is always mounted. > > > > There is one complication. The machine running the storage daemon does > not have enough local storage to copy the ARCHIVE-20211206-14:00 volume > over. And I've had no luck so far trying to get a working development > environment onto the server that has the external chassis attached. I > can't attach it to the machine running the storage daemon because > OpenZFS and Solaris ZFS zpools are incompatible. Thank you OH SO VERY > MUCH ORACLE. > > So, the zfs filesystem containing the archive file is remotely mounted > via NFS. Hello Phil > Is that what's causing the problem here? I don't think that NFS would be a problem here assuming it's correctly setup. I would doublecheck ownership and permissions of files and directories. The way how NFS maps UIDs and GIDs could also affect user access (check option no_root_squash). > Does anyone know a solution? I will assume that you already know about the "copies" option and that you need to find and use the ID of the copy job instead of the ID of a copied job. If that's not the case I would suggest to look into the thread "[Bacula-users] Restoring from a copy job" started at 2021-11-29 21:01 where I successfully managed to perform a restore from a copied job. > I don't suppose anyone has a set of Bacula 11.0.5 packages for Solaris > 11.4 amd64 compiled...? I haven't been using Bacula on Solaris for quite some time so I can't help you here. I hope you'll find the solution. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] New Bacula Server with multiple disks
On 2021-12-16 13:01, Josh Fisher wrote: IMHO, the use of RAID is critical, and not optional, when LVM / single filesystem is to be used. A single disk failure would affect the entire filesystem. A 4-disk group is 4 times as likely to fail. If the disks are in hot swap bays, then I would consider using vchanger (https://sourceforge.net/projects/vchanger/), which allows using all disks as one virtual tape library and also allows swapping disks in and out for archival or off-site storage use. Vchanger is intended for removable RDX/USB disks, but works the same for hot swap SATA. I agree. BTW, LVM supports redundancy as well. Check lvmraid(7). Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] New Bacula Server with multiple disks
On 2021-12-16 10:44, mac-eduki2.co.uk wrote: Good day I hope this message finds you all well. I have been running a small Bacula server with a single disk for a number of years now and it has become a vital part of our small business. I setup our original server with some help from the community and am very grateful to anyone who freely shares their knowledge. I am using Debian 9 and Bacula 7.4.4 I have 4 storage drives mounted as backup1-4 We have 20 plus clients and each one has their own pool and label. currently if I go to my single storage device all my client backup files are listed there I first question is: Can Bacula use my 4 disks in the same way filling up backup1 and than using backup2 etc? Would it be better to share out my clients between the 4 devices (disks) - and if so, should they each have a different file type for example file1 file2 etc? My approach is to figure out the storage first. If anyone has experience and can offer help on how a small Bacula server with multiple devices (disks) should be configured I would be most grateful. Hello Brad Bacula cannot fill your disk drives as you described but you can achieve the similar effect using other means. If I were you I would use LVM to include all the drives into a single LVM volume group. That would allow you to create a LVM logical volume with a single large file system you could use with Bacula as you see fit. It would also be a good idea to consider using RAID or similar kind of redundancy to prevent loss of data in case of a drive failure. Other options would include either use of symlinks or separate storage resource configurations. If you chose to use different storage devices for this purpose, you should make sure that they are using different MediaType. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Restoring from a copy job
On Wednesday 2021-12-01 09:11:43 Eric Bollengier via Bacula-users wrote: > Hello, > > On 11/30/21 22:31, Phil Stracchino wrote: > > On 11/30/21 14:54, Josip Deanovic wrote: > >> On Tuesday 2021-11-30 12:07:37 Phil Stracchino wrote: > >>> The question then ceases to be "I have twenty possible volumes from > >>> which your restore could be performed, which five of the twenty > >>> volumes > >>> do you want to use?", and instead becomes, "I have four possible > >>> SOURCES to perform this restore from, please pick *one restore > >>> source*.">> > >> The restore command of the bconsole tool already has the "pool" > >> command > >> line argument which could be used for volume selection. > >> > >> In my case with Bacula 9.6.7 it didn't work with copy jobs. It would > >> be > >> nice to make it work with newer Bacula versions (e.g. 11.x). > >> > >> Eric mentioned that MediaType might potentially be used in the > >> selection algorithm. I am not sure if this is necessary if we get a > >> working "pool=" argument. > > > > Indeed, and Pool is probably a better discriminator in this case than > > MediaType. In my case, the two relevant pools for Full backups are > > Full-Disk and Full-Archive, and both are Media Type = File. In any > > random hypothetical environment with copy jobs, I would expect it to > > be far more likely that Copy jobs will be in a different Pool than > > that they will have a different media type. > I don't know, personally, I'm using different pools to have different > retention, so, we have a simple case where all jobs are in one pool > (from Full to the last Incr), but it's a corner case as well. > > We can add a pool= or a mediatype= kind of keyword in the restore > command line, however, it's far from being very friendly as what we can > find in Baculum for example. I will think about it and your ideas are > welcome. But the command line parameter "pool=" already exists in bconsole tool. It just doesn't work in my 9.6.7 setup. Not sure if it would work with 11.x version of Bacula. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Restoring from a copy job
On Tuesday 2021-11-30 12:07:37 Phil Stracchino wrote: > > However, in a case where there are two or more complete sets of jobs on > different pools or devices, both containing all of the jobs necessary to > perform the desired restore, it SEEMS that it should be conceptually > simple to list the jobs and sources and allow the user/administrator to > select a restore SOURCE, then automatically select the volumes and jobs > required to perform the restore from that source. This seems much > simpler than trying to choose from all possible sets of jobs/volumes. > > The question then ceases to be "I have twenty possible volumes from > which your restore could be performed, which five of the twenty volumes > do you want to use?", and instead becomes, "I have four possible SOURCES > to perform this restore from, please pick *one restore source*." The restore command of the bconsole tool already has the "pool" command line argument which could be used for volume selection. In my case with Bacula 9.6.7 it didn't work with copy jobs. It would be nice to make it work with newer Bacula versions (e.g. 11.x). Eric mentioned that MediaType might potentially be used in the selection algorithm. I am not sure if this is necessary if we get a working "pool=" argument. -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Restore Session
Josip DeanovicOn Tuesday 2021-11-30 10:27:08 wrote: > On Tuesday 2021-11-30 15:29:42 Gary R. Schmidt wrote: > > Also, be aware that you have created a completely new file system, all > > the inodes will be different, and the access/modify/change/birth times > > on all the directories will be set to when the new directory was > > created - i.e. when it started to be populated by the restore. > > > > And everything will have different dev and rdev values. > > The documentation says that Bacula decides what files to backup for > Incremental and Differential backup by comparing the change (st_ctime) > and modification (st_mtime) times of the file to the time the last > backup completed. > > Could it be that the "Accurate" option changes that behavior when used > in the Job resource? > > If that is the case then it could be controlled by the "Accurate" option > used in the FileSet resource. Just did a simple test. What changes during the restore is the change time (st_ctime). That's why Bacula decides to backup restored files with the next Incremental backup. There is an option "mtimeonly=yes" which could be used in the FileSet. This would solve the problem Neil currently has to deal with. I would suggest to remove that option once it is not needed any more as it could have undesirable effects on the long run. Adding and removing of this option to the FileSet resource will not trigger promotion of Incremental job to Full unless Include or Exclude lists are changed as well. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Restoring from a copy job
On Tuesday 2021-11-30 04:54:35 Bill Arlofski via Bacula-users wrote: > Hello Josip, > > I did a little research and found the old Mantis ticket I was referring > to... > > However, the result from the conversation with the developers was as > follows: > > > The "copies" option is doing only what is described in the manual: > 8< > If the keyword copies is present on the command line, Bacula > Enterprise will DISPLAY the list of all copies for selected jobs. > 8< > > So, if you actually you want to restore from the copies, you need to > cancel the restore at the point where the copy jobids are displayed, > then run the restore job again but instead of specifying the client and > fileset you can specify the copy jobids as follows: > > * restore jobid=1,2,3,4,5 (where 1,2,3,4,5 are the jobids of the copy > jobs) > > It is not the way I would expect or hope the 'copies' option would > function, but at least it is "functioning as documented"™ :) > > Hope this helps. :) Thank you Bill, it helped and restore from copy jobs from tape pool works without issues now when I am properly using it. I expected the "copies" argument to work exactly as the documentation stated: "display the list of all copies for selected jobs" but I missed and completely ignored the "copyjobid" column given in the list. I was steered in the wrong direction by the part of the documentation covering restore command line arguments which says: "- pool=pool-name - specify a Pool name to be used for selection of Volumes when specifying option 5 and 6 (restore current system, and restore current system before given date). This permits you to have several Pools, possibly one offsite, and to select the Pool to be used for restoring." That part with specifying pool (in my case tape1) doesn't work although I am using separate storage, device and media type for that specific pool. Maybe there is a bug regarding that because when I specify a pool, after I select option 5 and then client and its particular job, it immediately returns this output: "No Full backup before 2021-11-30 09:42:58 found." Thanks again, you helped a lot and I can confirm that restoring from copy jobs is working with Bacula 9.6.7 when jobid= is used with the values from the "copyjobid" column listed by "copies" argument. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Restore Session
On Tuesday 2021-11-30 15:29:42 Gary R. Schmidt wrote: > Also, be aware that you have created a completely new file system, all > the inodes will be different, and the access/modify/change/birth times > on all the directories will be set to when the new directory was > created - i.e. when it started to be populated by the restore. > > And everything will have different dev and rdev values. The documentation says that Bacula decides what files to backup for Incremental and Differential backup by comparing the change (st_ctime) and modification (st_mtime) times of the file to the time the last backup completed. Could it be that the "Accurate" option changes that behavior when used in the Job resource? If that is the case then it could be controlled by the "Accurate" option used in the FileSet resource. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Restoring from a copy job
On Tuesday 2021-11-30 08:51:40 Radosław Korzeniewski wrote: > Hello, > > wt., 30 lis 2021 o 06:01 Bill Arlofski via Bacula-users < > > bacula-users@lists.sourceforge.net> napisał(a): > > It is not the way I would expect or hope the 'copies' option would > > function, but at least it is "functioning as documented"™ :) > > The way my customers are using it is when "originals" are not available > for restore, then restore from copies works perfectly as expected. So, > when doing copy for offline, off-site storage, disaster recovery, long > term, low-cost storage, etc. etc. Yes. In my case I needed it to test the offsite storage, media and backup itself. While I was googling for the solution, I have found that people often misinterpret the documentation which says that Bacula will automatically promote copy jobs if the records about original jobs are purged from the catalog. It seems that people read the last part as: "if the original jobs are not available" which is not correct. Their records must be deleted from the database. > It is not perfect and does not cover all use cases you can imagine for > sure. But do the real work when required. I am fine with it as long there is a way to perform restore from copy jobs. I don't mind if it takes an extra step as long as it works. :-) Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Restoring from a copy job
On Tuesday 2021-11-30 09:19:51 Marcin Haba wrote: > Hello Josip, > > Baculum supports restore from copy jobs. It is a function in the > restore wizard that default is enabled. So you can just select copy > jobs and restore from them all or selected files. This function is > available from Baculum version 9.6.0. > > If you prefer doing it using Bconsole, you can use Bvfs restore > commands for that. But please be careful with .bvfs_get_jobids command > in version lower than 11.0.3, because there was a bug in this command > with copy jobs: > > https://bugs.bacula.org/view.php?id=2500 > > Besides that it works well. Thank you for confirming it works on 9.6.x. I am currently not using Baculum but Bill helped me to find the solution. In short, the solution is: 1. use "restore copies" to get the list of IDs of copy jobs 2. use "jobid=copyjob1,copyjob2" to select the copy jobs listed in the copyjobid column Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Restoring from a copy job
-sd-tape1
Maximum Concurrent Jobs = 10
TLS Enable = yes
TLS Require = yes
TLS CA Certificate File = /etc/bacula/certs/localdomainBackupCAChain.crt
TLS Certificate =
/etc/bacula/certs/dir-storage-client_p-bkp-1.bkp.localdomain.crt
TLS Key = /etc/bacula/certs/dir-storage-client_p-bkp-1.bkp.localdomain.key
}
Pool {
Name = "tape1"
Pool Type = Backup
Storage = "p-bkp-1-sd-tape1"
AutoPrune = yes
Volume Retention = 12 months
Recycle = no
Label Format = "tapevol-"
}
Kind regards!
--
Josip Deanovic
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Which OS is best for bacula ?
On Sunday 2021-07-11 18:31:24 Kern Sibbald wrote: > Hello Sven, > > Yes, I am aware of the FHS policy, which is perfectly fine for 99.9% of > all installed programs, but not ideal for Bacula. Hello, There is nothing special about Bacula in that regard. Many (if not most) programs require modifications to conform to a specific standard unless the program is following particular rules from the very beginning. Many programs are modified by package maintainers who sometimes have to come up with patches for programs to work as expected because authors sometimes chose to hardcode paths and other options which should be configurable. We are dealing here with two opposite opinions and two different points of view because two types of people are involved here: developers and sysadmins. For a developer the easiest solution is to keep everything under a single directory and to ship a bunch of libraries with the program distribution. This keeps program more predictive and the deployment less complicated. Developers usually don't have to bother with the software deployment and the service maintenance. Every bit of work they manage to spare during the application development by keeping their environment simple will usually get payed by a sysadmin during the deployment and service maintenance. On Unix (and Linux) /opt directory used to be a directory which usually contained optional software distributions provided by a third party companies or vendors. It is rarely used by OpenSource tools or applications. Unless there is no other way (e.g. there is a commercial software distribution with a specific requirements), /opt shouldn't be used by a software distribution. Using /opt is usually a sign of a bad support (e.g. packaging and testing) for a specific system. Note that I am talking in general, not about Bacula specifically. Software that tend to use /opt for its installation usually brings several commonly available libraries locked to a specific version. Providing commonly available libraries already available on the system with the particular software distribution and linking programs with such libraries makes it good thing for the application stability and simplicity of its deployment and ultimately vendor support. However, from a non-developer view there are no visible benefits. >From sysadmin view, it looks like yet another nonstandard software installation with an unusual update procedure which usually turns out to be more of a migration and less of a simple minor version update. Another problem is that it is most likely that libraries shipped with a third-party software will receive their security and bugfix updates far too slow or more commonly, will not receive any updates at all which adds to an overall potential system vulnerability footprint. Also, shipping such common libraries will duplicate the code used on the system. Putting software under /opt could also complicate a diskless setup a bit. Now imagine the horror on the system if most of the services on the system would follow such practice. Such system would be very hard to maintain and the code sharing in memory would be pretty low. Directory structure on Linux systems is the only thing that was relatively well standardized throughout the Linux distributions since the earliest days of Linux and it is interesting to see that what is considered to be a properly packaged software for one person could at the same time be described as "spreading it all over the filesystem" for somebody else. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] How to configure Bacula with S3
On Thursday 2021-06-17 08:01:59 Wanderlei Huttel wrote: > Hi Josip, > > I forgot to mention it before. > I'm running Debian 10 Buster Interesting. I just tested it on new Debian 10 installation. I tested it by putting a path in /etc/ld.so.conf before and after the include line. I have also tested it with the separate config file in the /etc/ld.so.conf.d directory. In all cases it worked as expected and the cache was properly updated. I checked the content of the cache file with "ldconfig -p" command. -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] How to configure Bacula with S3
On Thursday 2021-06-17 07:25:35 Wanderlei Huttel wrote: > Hello Josip > > I don't know why, but I've run ldconfig several time and didn't worked, > only when I remove the /etc/ld.so.cache that s3 worked. Hi Wanderlei I am curious what operating system are you using? Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] How to configure Bacula with S3
On Wednesday 2021-06-16 16:44:34 Wanderlei Huttel wrote: > Hello Erick > > I guess I've discovered the problem. > I had to remove the libs cache "rm /etc/ld.so.cache" and just > run "ldconfig" to create a new cache and then running command "s3" > worked fine. > > I guess my bacula configuration (director and storage are OK), I've made > a backup and it worked. Hello! You don't have to remove /etc/ld.so.cache file. Just run ldconfig. By default only libraries found in /lib and /usr/lib will be cached. Recent Linux distributions removed lib directory and replaced it with a symlink pointing to usr/lib in the root directory. Suse 9 and probably 10 and later, used to run ldconfig with a specific config file with every package install or update thus ignoring /etc/ld.so.conf file which is just one of many bad decisions they did. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] I need to reuse volume1 inmediately
On Friday 2021-06-04 21:40:41 Denis Morejón wrote: > Hello: > I have built a bacula 11 successfuly. I am testing saving a few files to > a file type device, using a little Pool of up to 3 volumes of 100Mb > each one. I need to see how bacula creates the three volumes > automatically as I repeat a job some times in order to fill up the > three volumes. And It happens indeed. The three volumes get full. But > then, a new job execution remains running for a long time and need to > be stopped. It seems that the first volume can not be rewriten. Even > when directive volume retention = 0 is set on the pool. File and Job > retention = 0 is set on the client resource. What else should I do ? I > mean, to rewrite the first volume automatically. Hello! I think others already mentioned that Bacula will honor retention in order to prevent you from unintentionally loosing your backup data. However, what you are trying to achieve can be done like this: Volume Retention = 12 months AutoPrune = yes Maximum Volumes = 3 Recycle = yes Purge Oldest Volume = yes # will reuse the oldest volume regardless # the Retention periods Note that for some changes in the pool to take effect (e.g. pool retention or volume sizes) you have to use the "update pool" command. It is not enough to reload Bacula configuration. Notice that I have used option "Maximum Volumes = 3" which notifies Bacula that there are total of three volumes and when there are no available volumes left, Bacula will reuse the oldest one. I believe that in your case Bacula waited for new volume to appear like it would with the real tape volume because existing volumes were protected by retention period and were not yet pruned. If you chose this path of recycling the oldest volume, note that there is a potential for an attacker to exploit that approach by creating a huge data that needs to be backed up, resulting in recycling of all the volumes thus rendering backup useless. These days number and proficiency of ransomware attacks increases every year and this shouldn't be taken lightly. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula-users Digest, Vol 181, Issue 21
On Sunday 2021-05-23 11:53:25 Joseph Zatarski wrote: > In response to Bill, Phil, and Josip, I have checked the logs to be > sure and the job is not being upgraded to full. I can't say for > certain that the files haven't been touched by anything, but I'll be > checking some attributes and doing some comparisons between what bacula > has in the DB for those files, and what they are on disk. I believe > last time I did such a thing, I came to the conclusion that the only > difference I saw was in the device number field of the stat(2) > attribute blob. I'll follow up again once I've looked at it. So, if Bacula didn't upgrade incremental job to a full backup job and you only got suspiciously large backup, I would suspect that problem lays somewhere else. I had to check your original post to refresh my memory. You asked about the change of the device number of the storage device... I am pretty sure it doesn't affect Bacula and the way it determines which files differ from those already backed up. Bacula deals with files and directories that reside on the file system you have decided to backup and it doesn't check the device numbers of the underlaying device. But the fact that you have noticed unreasonably large amount of files retrieved by the incremental job might mean that something else is changing "attributes" of the files. Here are few things you might wan to consider: - Are you using normal file system (e.g. ext4, xfs) or some experimental file system? - Is the OS you are trying to backup hardware machine or virtual machine? - Is your system time correct and your ntpd/chronyd properly configured. Bacula is trying to take into account the time difference between server and client but still, it's worth checking the system time. You might also want to manually synchronize your hardware clock with your system clock using the command "hwclock -w" in case it is not regularly performed. - What kind of data are getting copied by the job? Are we talking about the files that belong to OS or some custom user data? - If the data that get copied belong to the OS than it would be a good thing to check if there were some unattended upgrades before the backup. - Also, prelinking (in case it's turned on and forced to run) could modify large number of system files. - Are you using selinux or some other subsystem or tools that could trigger a change of large number of files? - Have you noticed any warnings or errors in your logs and dmesg output lately? - Have you noticed any corrupted files or databases on that file system you are trying to backup? > It would also be useful to know what the default options are for the > "Accurate = " flags in the fileset. Unfortunately, I don't see any > listing of default options in the documentation, beyond when they > implemented the feature and it pops up in the change summary sections. > At least back then, it was mtime, ctime, and file size. I'll have to > work on the assumption that it has not changed since then. In the Bacula documentation of version 5..0.x https://www.bacula.org/5.0.x-manuals/en/main/main.pdf in section 4.5 it says: -BEGIN In previous versions, the accurate code used the file creation and modification times to determine if a file was modified or not. Now you can specify which attributes to use (time, size, checksum, permission, owner, group, . . . ), similar to the Verify options. -END- -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] noticing an occasional incremental backup act like full
On Sunday 2021-05-23 14:31:51 Bill Arlofski via Bacula-users wrote: > On Sunday, May 23, 2021 8:25 AM, Bill Arlofski wrote: > > Check this out: > > 8< > > 23-May 08:12 speedy-dir JobId 6: The FileSet "Full Set" was modified > > on 2021-05-23 08:12:32, this is after the last successful backup on > > 2021-05-22 17:35:16. > > > > 23-May 08:12 speedy-dir JobId 6: No prior or suitable Full backup > > found in catalog for the current FileSet. Doing FULL backup. > > 8< > > I just checked and I see that this was implemented in June 2019. :) I must have got you wrong. I thought you are referring to "No prior or suitable Full backup". -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] noticing an occasional incremental backup act like full
On Sunday 2021-05-23 14:25:37 Bill Arlofski via Bacula-users wrote: > Right, good point! And when a job (typically set to Level = Incremental) > is first run, this will always happen too. :) > > > Oh! And I just remembered that I had made an internal Bacula Enterprise > feature request some time ago. It was accepted and implemented, so > Option 1 is even better than I explained because now it tells you very > clearly why there are "no prior suitable Full backups". > > Check this out: > 8< > 23-May 08:12 speedy-dir JobId 6: The FileSet "Full Set" was modified on > 2021-05-23 08:12:32, this is after the last successful backup on > 2021-05-22 17:35:16. > > 23-May 08:12 speedy-dir JobId 6: No prior or suitable Full backup found > in catalog for the current FileSet. Doing FULL backup. 8< > > > I am looking forward to Joe's feedback in this thread. :) This message is very helpful. I don't even remember when it first appeared. -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] noticing an occasional incremental backup act like full
On Saturday 2021-05-22 21:14:20 Bill Arlofski via Bacula-users wrote: > Hello Joe, > > I can think of three things that can cause this to happen, and at least > two of them will show some evidence in the job log > > > Option 1: > - > You have edited the Fileset, and the "IgnoreFilesetChanges = yes" option > was not set at the top of the Fileset. In this case Bacula will log as > one of the first lines in the job's log something like this: 8< > 05-May 11:05 bacula-dir JobId 38357: No prior or suitable Full backup > found in catalog. Doing FULL backup. 8< > > Additionally, in the Summary, where it shows the "Backup Level:" it will > also tell you the job was upgraded from an Inc or Diff 8< > Backup Level: Full (upgraded from Differential) > 8< > > One other thing in the summary to help diagnose if an edit of the > Fileset was the cause is that the "Fileset:" line shows the last time > the Fileset that was used in the job was edited: > 8< > FileSet:"HomeFileSet" 2015-07-03 12:35:32 > 8< > > > Option 2: > - > Something or someone has 'touched' all of the files in the tree. I have > seen this a few times where someone implements a script - perhaps to do > some virus scan or something - and the script or program modifies the > attributes of every single file, causing Bacula to (correctly) noticed > a change and backup each file. > > To mitigate this type of case, you can set specific attributes to check > in the Fileset with the "Accurate = " setting - where is a > list of characters to enable the attributes you want to check. Using > this feature you can tell Bacula to ignore attributes that may get > modified in cases like I described. > > > Option 3: > - > You have set the "MaxFullInterval" in a Job or JobDefs resource. > Personally, I like and use this feature. It is a simple way to help > "spread out" my Full backups from all kicking off at the same time each > month. And, since I don't really care what day or time my Fulls run as > long as they run once a month - this is a prefect feature for me. > > I set it for about 33 days, and then set all of my Jobs to have "Level = > Incremental", and my Schedules are just simple "run every day at x > time" with no mention of running a Full on the First Sunday of a month > and Incrementals every other day. > > Then, when 33 days have passed, instead of doing the specified > Incremental or Differential level configured in the Job/JobDefs, Bacula > will log this in the job log: > 8< > 12-May 23:00 bacula-dir JobId 38614: Max Full Interval exceeded. Doing > FULL backup. 8< > > and then this in the Summary: > 8< > Backup Level: Full (upgraded from Differential) > 8< > > Also, if I kick of a Full for whatever reason, this starts the > MaxFullInterval counter for this job again at this time, adding a > little more entropy into the mix of when Fulls are kicked off. > > Hope this helps, There is another option. Option 4: ----- If no valid full backup for the job could be found in the catalog, the backup job will be upgraded from incremental or differential to full backup. The message would be the same as the one described in option 1. This could happen if the job is prematurely pruned (before a new full backup has been performed). Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Backup catalog error
On Wednesday 2021-05-05 20:53:42 Peter Milesson wrote: > Hi folks, > > After upgrading to Bacula 9.2.2, the following backup error occurs when > the daily catalog backup is run: [...] > > The normal backups seem to terminate without any errors. It's quite > clear that the bacula.sql file cannot be created or written, which > triggers the error message. The spool directory /spool/bacula is > otherwise used by Bacula during normal backups without any error > messages. The directory /spool/bacula is owned by bacula:bacula and with > permissions 0750. > > I assume that the MySQL database bacula is backed up in this job. If it > is so, I do not worry very much, as there is a cron job run separately, > that backs up the bacula database to an external server. > > I'm grateful for any input concerning this problem. Hello! Are you sure it is not defined in your FileSet named "Catalog"? Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] After upgrading Library firmware drives not working
On Tuesday 2021-03-30 11:40:06 Christian Lehmann wrote: > Dear Vigrant, > > but as you can see from your output, the changer is now sg8, not sg2. > > Best, > > Christian Recent Linux based systems might also support device names based on the device ID. For example: /dev/tape/by-id/scsi- Using device path like that would preserve its name and path through system reboots. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula 11.0.1 running on Ubunto 20.10 - bacula-dir service starts and exits immediately without error message
On Thursday 2021-03-18 02:57:51 Ismael Matos wrote: > Hello Everyone, > > I'm new to Bacula, trying to backup a couple of Ubuntu and Windows 10 > machines. I'm also no expert in Linux/Ubuntu. > > After installation of the latest version available, and the initial > configuration, I'm stuck with this situation: > - all 3 bacula-* services start automatically at boot without problems > or error messages, > - but the bacula-dir.service exits immediately. Again no error or > diagnostic messages, > - bacula-dir runs 100% OK when started manually from the console. > > Would anyone be able to help and point me in the right direction to get > bacula-dir service working automatically? > > Thanks and regards, > > Ismael In such cases I always do the next few steps before anything else: 1. Check if Bacula services are running or not ps aux | grep -i bacula 2. Perform configuration syntax check using Bacula daemon programs # for bacula file daemon bacula-fd -t # for bacula storage daemon bacula-sd -t # for bacula director daemon bacula-dir -t 3. If that doesn't reveal the issue, try to run the daemon manually in foreground mode without systemd or init scripts. Make sure you are doing this as the same user a particular Bacula daemon is supposed to run (this is important). For example, you could temporary become "bacula" user using this command: sudo su - bacula -s /bin/bash If you are already root, leave out the sudo command. Check the output of the "bacula-fd --help" command to see the list the daemon options. Since you have mentioned that you are experienced in Linux administration this might be a bit difficult for you. But then again, this also an opportunity to gain new knowledge. :-) -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Error on despooling attributes in catreq.c: wanted xxx bytes, maximum permitted 10000000 bytes
On Monday 2021-02-22 16:06:42 David Brodbeck wrote: > On Wed, Feb 17, 2021 at 6:12 AM Josip Deanovic > > wrote: > > It's interesting that the job had almost 2GB of attributes to > > despool. That's quite a large amount of attributes. > > I have a few jobs that despool that much, and one that despools over 18 > GB. The machine in question has 51 million files. Just to be sure... Are you talking about the attribute spool file that gets created in running directory and not about the spool file that gets created in the spool directory? Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Error on despooling attributes in catreq.c: wanted xxx bytes, maximum permitted 10000000 bytes
On Tuesday 2021-02-16 12:22:51 Uwe Schuerkamp wrote: > Hi folks, > > thanks for all your suggestions. I compiled 9.6.7 from source and > "patched" catreq.c to increase the maximum attribute spool size to 3x > the original value. When mentioning the original value, are you referring to 1000, the hardcoded value) or to the 1969368434 bytes which was the reported value when the job failed? It's interesting that the job had almost 2GB of attributes to despool. That's quite a large amount of attributes. I am not sure if 5M files and directories could account for the attribute spool file of 1.8GB in size. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Error on despooling attributes in catreq.c: wanted xxx bytes, maximum permitted 10000000 bytes
On Monday 2021-02-15 18:47:35 Martin Simmons wrote: > >>>>> On Mon, 15 Feb 2021 09:56:33 +0100, Uwe Schuerkamp said: > > Hi folks, > > > > during a largish backup job (3,8TB windows fileserver, about 5m files > > & directories) I'm seeing this error when the attributes are being > > despooled once the job completes: > > > > Fatal error: catreq.c:762 fread attr spool error. Wanted 1969368434 > > bytes, maximum permitted 1000 bytes > > How large is the attribute spool file? > > This could be a bug. > > Is it repeatable? If so, does it repeat with this same number > (1969368434)? > > __Martin I don't know why some of my e-mails don't reach the e-mail list. I wrote today that the size of 1000 is hardcoded i nthe source of catreq.c. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Can Bacula Director 9.4.2 backup - Client FD 5.2.6
On 2020-12-29 11:30, mau...@gmx.ch wrote: Hello It's possible to backup a client that has the FD 5.2.6, with Director 9.4.2? Thanks for possible answer I believe it is. I am using Director and Storage daemon of version 9.6.6 with clients of multiple versions: - 9.6.6 - 5.2.13 - 5.0.0 Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Write Bootstrap Directive Syntax
On 2020-11-04 16:22, Chris Wilkinson wrote: The main manual describes the write bootstrap file directive in several places as either “write bootstrap =“ or “writebootstrap =“ (in 2 places). Is one/either/both of these acceptable syntax? I tested both of these with bacula-dir -t and neither flagged a syntax error. Best -Chris- Both are acceptable. You could even use "Write Bootstrap". Check the "Upper and Lower Case and Spaces" section on the following link: https://www.bacula.org/7.4.x-manuals/en/main/Customizing_Configuration_F.html#SECTION001622000 Regards -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Hung jobs in Bacula 9.6.5: Apparently fixed in 9.6.6
On 2020-10-12 20:39, Phil Stracchino wrote: [...] I *THINK* I can safely state that whatever the root cause of the problem in 9.6.5, it is fixed in 9.6.6. Since it is already reported that a TLS issue in 9.6.5 is fixed in 9.6.6, I'm going to *speculate* (with the caveat that I have no definite proof) that both issues were actually networking problems, that jobs were hanging because communication with the clients silently failed, and that the fix for the TLS problem ALSO fixed the hung-jobs problem by fixing the client communication failures. It is worth mentioning that that Bacula client 9.6.5 worked without TLS related issues on Centos 8. I have observed the TLS issue only on Centos 6 and Centos 7 with that Bacula version. On Centos 6 (Bacula client 5.0.0) and Centos 7 (Bacula client 5.2.13) there were no TLS issues either. With Bacula 9.6.6 (compiled with the same options, in the same environment) TLS issues were solved for all Centos versions but I didn't thoroughly test it yet. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula 9.6.5 TLS issue - solved in 9.6.6
On 2020-10-08 15:54, Phil Stracchino wrote: On 10/8/20 9:11 AM, Josip Deanovic wrote: Do you have to turn off attribute spooling with 9.6.3 and 9.6.6? Disabling attribute spooling will inflict noticeable performance degradation. Unfortunately, yes, because the attribute spooling code — at least for the MySQL driver — is broken. It caches all of the attribute data in a temporary table until the job is done, then dumps it all into the DB at once, ignoring the configured write batch size. If the job copies more than 128K files, this exceeds Galera 3's hard writeset limit. I see. Thank you for explaining it. I used to use MySQL for a long time and had no problems because I didn't use Galera MySQL cluster with Bacula. If it honored the batch size setting, it would be perfectly fine. That said, I probably would not have done the spooling that way in the first place. I would have cached the attribute data in memory until I had $BATCHSIZE records, then written them directly to the DB in a batch. I honestly think this would perform better than saving them all until the end of the job and then ogging the DB with potentially millions of records at once. That is ALWAYS a bad idea. I'd write and offer a patch — in fact I'd overhaul the entire MySQL driver — but I don't know nearly enough C++. I don't know how these things are implemented in Bacula. It's possible that Bacula team did it because they thought that it would help setting up HA for the Bacula director daemon. In case one is using database cluster in round-robin setup, one of the master nodes could start lagging which could have unpredictable effects on most applications (unless synchronous communication is in use). With some applications, depending on how they utilize database, it could lead to some kind of interlocks which would need to be solved by the cluster software or otherwise it could lead to long or indefinite timeouts. Round-robin with database nodes (master-master) is usually fine for applications that produce small queries and don't have to create awfully complex relations. Otherwise, database cluster software would need to take care about locking which brings in the question of synchronous communication and the overall performance gain from such setup. I am aware that some decisions in Bacula regarding dealing with database connections are not the best. For example, if you use a Copy which is configured to select like 300 jobs to copy, Bacula would open 600 connections to the database. Those connections would stay opened until jobs are finished. For each Copy job that completes, two connections would get released. And if your database has connection limit set below the number of connections Bacula temporary needs, Bacula-dir would segfault. I have experienced it with Postgres and I have found old posts in the mailing list archives claiming that the same problem exists with MySQL as well. There are few ways to workaround the problem with too many connections but Bacula director shouldn't segfault. -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula 9.6.5 TLS issue - solved in 9.6.6
On 2020-10-08 14:56, Phil Stracchino wrote: Well, I understand that perspective, but this problem appeared instantly as soon as I upgraded to 9.6.5, with a DB configuration I've been using for years without a hiccup. I could turn the problem on and off like a lightswitch by updating *ONLY* the Director to 9.6.5 or rolling it back to 9.6.3, even if connecting directly to the local DB node without using HAproxy. 9.6.5 Director, even without HAproxy: About one in three jobs hang. 9.6.3 Director, even WITH Haproxy: No hangs. So far, 9.6.6. is behaving like 9.6.3. Interesting. In that case you are probably right. Are you using MariaDB cluster because you need high availability or because you want to achieve higher database throughput? High availability, fundamentally. I'd honestly prefer to be using Percona XtraDB Cluster, but there is no working, maintained ebuild for Gentoo Linux — and enterprise customers ARE going to try to use it against HA clusters, so we'd better be sure it works. Which it does, as long as attribute spooling is disabled (and you're not using Director 9.6.5). Do you have to turn off attribute spooling with 9.6.3 and 9.6.6? Disabling attribute spooling will inflict noticeable performance degradation. -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
