Hi all, Thank you for the assistance with debugging this issue! I created an issue here to track this further so other people don't end up with the same problem: https://github.com/bareos/bareos/issues/2600
On Thursday, 19 March 2026 at 09:25:49 UTC+1 Patrick Hasler wrote: > OK, we still have the same issue here. The backup jobs are still running > but the volumes are already in a much larger size than they should be. > > I verified the sql query as well. Here the list is also empty. But > interestingly this behavior is only present for one of the affected > clients. For another one where the incrementals are also way too large, I'm > able to get a list of files. > For this client, I copied the list to the client and calculated the total > file size, but it's not even close to the size of the related Bareos > volume. So Bareos is definitely doing something wrong here. Note that we > only use volumes once for a single job so the size can not be related to > other jobs. > > One guess I had was that Bareos is somehow not keeping track of what's in > the former incrementals and is including the same files in each incremental > again and again. But the incrementals are still smaller than the full > system, so Bareos is not backing up the full file system each time. > The biggest affected system is around 8 TB large, but the backups are > always around 1.15 TB, with the amount of files always being nearly > identical with every job (805066, 811411, 810618). > > How likely is it that the database encoding would cause these problems? > Approximately a month ago we shrank the Catalog database (according to > https://docs.bareos.org/TasksAndConcepts/CatalogMaintenance.html#compacting-your-postgresql-database) > > because the PostgreSQL Autovacuum was not properly configured and the DB > just kept growing more and more. > This lead to the warning regarding the database encoding. But we haven't > seen any issues since then. The weird incremental growth started last week. > > I'm currently waiting for one long-running consolidate job to finish but > once that is done, I would dump the database and recreate it according to > https://docs.bareos.org/IntroductionAndTutorial/InstallingBareos.html#id7 > before adding the dump again. > > It looks like the consolidate job will keep running for at least another > hour before I can "fix" the DB. If we could have a look at the system > together in the meantime, that would be much appreciated. > > > > > On Wednesday, 18 March 2026 at 17:42:29 UTC+1 Bruno Friedmann > (bruno-at-bareos) wrote: > >> yes estimate level=incremental will only show what will be saved. >> >> for a given jobid you can also ask directly the database >> in bconsole >> >> .sql query="select p.path || f.name from file f left join path p on >> p.pathid=f.pathid where jobid=12345;" >> >> That should return something normally. >> >> On Wednesday, 18 March 2026 at 16:47:52 UTC+1 Patrick Hasler wrote: >> >>> >>> > I'm not yet sure how reliable is a volume name like >>> >> Created new Volume "Customer-A/AI_Inc_8263" in catalog. >>> > Is that one file in /mnt/bareos/backup or dir/file ? >>> >>> It's an actual directory structure. In the end this will result in a >>> sub-directory in /mnt/bareos/backup with the customer name ("Customer-A) >>> and then the Bareos volume files in it. In this example AI_Inc_8263 is a >>> file in the directory /mnt/bareos/backup/Customer-A. >>> This has worked reliably over years and was the suggested setup by >>> Andreas Rogge. For our use-case this is very helpful because Bareos does >>> not have a feature for "grouping" hosts together and we need to be able to >>> tell how much storage a customer is using. >>> >>> I would be very surprised if the issue is on the storage's side. The >>> backups are properly written and the volumes have the exact file size that >>> the WebUI and job log are reporting as well. It's just not clear what the >>> actual content is and why they are so huge out of nowhere. >>> >>> For now I've started a new consolidate job which resulted in a >>> successful virtual full backup for some of the affected hosts. So my hope >>> would be that the next incremental which will trigger tonight will be of >>> regular size. >>> >>> If that would not be the case, would it be an option to have a look at >>> the setup together tomorrow? If the problem persist we would need to find a >>> solution as soon as possible, because we are now in a situation where we >>> cannot trust that Bareos is actually doing proper backups of our machines. >>> It refuses to give us the content of what's being backed up and it's >>> filling up our storage endpoints with huge data with unknown content. >>> >>> In any case, I'll let you guys know tomorrow what the status is. >>> >>> > if the incremental has everyday such number of changes you might want >>> to check them with estimate listing >>> >>> The estimate command (estimate level=incremental >>> job=Job_Customer-A_Daily_AlwaysIncremental_client-01 accurate=yes listing) >>> gives me a proper file list. If `level=incremental` is specified, should it >>> only print files that have changed and would be included in the next >>> incremental job, or will it still give a full list of all files included in >>> the related FileSet to that job? Because the output looks like it gives us >>> the full file tree of that system. >>> >>> Here are the command outputs: >>> *estimate level=incremental >>> job=Job_Customer-A_Daily_AlwaysIncremental_client-01 accurate=yes listing >>> ... (huge list of files) >>> 2000 OK estimate files=1,523,195 bytes=2,095,857,085,860 >>> >>> >>> >>> >>> On Wednesday, 18 March 2026 at 15:52:53 UTC+1 Bruno Friedmann >>> (bruno-at-bareos) wrote: >>> >>> I hope you noticed that your database is using the wrong encoding and >>> should be fixed as soon as possible >>> > Warning: Encoding error for database "bareos". Wanted SQL_ASCII, got >>> UTF8 >>> Try to follow our documentation when you initialize the catalog. >>> >>> >>> Due to the use of folder/file in volume names maybe you are required to >>> quotes that when asking list files from volume and the webui can't ? >>> >>> I'm not yet sure how reliable is a volume name like >>> > Created new Volume "Customer-A/AI_Inc_8263" in catalog. >>> Is that one file in /mnt/bareos/backup or dir/file ? >>> >>> > Consolidation jobs keep their original start/end time. >>> You mean consolidated jobs ;-) the consolidation job has its own >>> start/end time ... >>> >>> if the incremental has everyday such number of changes you might want to >>> check them with estimate listing >>> >>> >>> On Wednesday, 18 March 2026 at 11:42:34 UTC+1 Patrick Hasler wrote: >>> >>> I've attached the job log for one of these jobs. Nothing unusual from >>> what I can tell. >>> >>> >Are you sure you didn't get caught by the fact that after >>> consolidation, history is rewritten, and older jobid get new files in due >>> to the consolidation? >>> >>> These are not consolidation jobs, but the regular running backup jobs. >>> Consolidation jobs would start with "Start Virtual Backup" in the >>> joblog. This is also visible by the timestamps in the screenshots. >>> Consolidation jobs keep their original start/end time. >>> >>> On Wednesday, 18 March 2026 at 11:32:24 UTC+1 Bruno Friedmann >>> (bruno-at-bareos) wrote: >>> >>> You may want to check what the joblog contain. >>> >>> All my AI job are showing the files backup'ed when there's some, and >>> also all my volumes show me jobid stored on. >>> >>> Are you sure you didn't get caught by the fact that after consolidation, >>> history is rewritten, and older jobid get new files in due to the >>> consolidation ? >>> >>> On Wednesday, 18 March 2026 at 11:27:35 UTC+1 Patrick Hasler wrote: >>> >>> Hi Bruno, >>> >>> Thank you for the quick reply! The list is actually empty: >>> *list files jobid=93451 >>> * >>> >>> I tried it for mutiple jobids. For older job with the regular file size >>> the command is working as expected and I get a list of files. >>> >>> >>> >>> >>> On Wednesday, 18 March 2026 at 11:17:18 UTC+1 Bruno Friedmann >>> (bruno-at-bareos) wrote: >>> >>> You can see what files have been backup'ed by using >>> bconsole >>> list files jobid=12345 >>> >>> On Wednesday, 18 March 2026 at 10:16:57 UTC+1 Patrick Hasler wrote: >>> >>> We noticed a quiet urgent issue on some of our larger systems where we >>> do backup for with Bareos. Suddenly every new incremental backup is huge >>> and also ruffly has the same size every time (See screenshots below). >>> Usually I would suspect something changing on the system (For example a >>> DB backup dump cronjob) but as this affects multiple completely unrelated >>> systems, this can be ruled out. There is no indication of any kind of large >>> file changes on all the affected hosts, so the question is: What is Bareos >>> actually backing up here? >>> >>> If I go to the restore section to check what files were actually backed >>> up by the job, the "File selection" section just stays empty, so we don't >>> have any idea what's in the backups. Due to there huge size we also cannot >>> just restore them to the client to check their content and as we use >>> encryption we cannot simply extract the data from the volumes on the >>> Storage Daemon's side. >>> >>> There was no recent Bareos software upgrade, so I don't think that this >>> is a new software bug. Also the affected clients use different versions of >>> the File Daemon (24.0.7, 25.0.2), while the Director and Storage Daemon >>> both use 25.0.2. >>> >>> What could be an explanation for this kind of behavior? I mentioned >>> Always Incremental because I'm pretty sure that the problem is somehow >>> related to it as the new huge incrementals started showing up after the >>> most recent VirtualFull. But this is not the first VirtualFull, so I'm not >>> sure why this behavior suddenly started.[image: >>> bareos_huge_incrementals-1.png] >>> >>> [image: bareos_huge_incrementals-2.png] >>> >>> -- You received this message because you are subscribed to the Google Groups "bareos-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/bareos-users/bb2bcba7-a1e6-4520-88e1-dbb6904be720n%40googlegroups.com.
