Re: [Bacula-users] Duplicate Backups with OneFS=no and the directory winding up in the list multiple times?
Hello, One other point about this. Semi-automatic detection of backing up files multiple times could be done with the estimate bconsole command turning on listing, then processing the output through sort and looking for duplicates. Perhaps an interesting "feature" might be to add another option to estimate that would do exactly what I mentioned above then display all the duplicates. Once a proper FileSet is made, there should really be no need to go to the extra expense of looking for duplicates. Best regards, Kern On 6/5/19 12:46 AM, Larry Rosenman wrote: On 06/04/2019 5:25 pm, Chandler wrote: Larry Rosenman wrote on 6/4/19 14:56: Is there a way to have bacula see if it's backed up a path already in this run and not do that? What you could try is to do a restore job, then pick "list jobs where a given file is saved" and you could get that listing. Then in the director backup settings, you can update the Fileset resource so it doesn't have duplicates and add "Ignore Fileset Changes = yes" to prevent another Full backup from running, but ideally you would run a new Full backup after updating the Fileset resource. HTH, I've already made that change, and the new Full's are running/have run. I was just wondering if it would make sense for Bacula to become smarter about this, and be helpful and not duplicate the files. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Duplicate Backups with OneFS=no and the directory winding up in the list multiple times?
Hello, Yes, Bacula could become much smarter about this, but it requires two things: 1. Bacula will require *significantly* more resources (CPU and memory) to do such a backup. 2. You would need to find a programmer interested in writing the necessary code -- it is not a monster project, but it is not trivial either. Much of the code for such a project already exists in the Accurate code of the FD. Best regards, Kern On 6/5/19 12:46 AM, Larry Rosenman wrote: On 06/04/2019 5:25 pm, Chandler wrote: Larry Rosenman wrote on 6/4/19 14:56: Is there a way to have bacula see if it's backed up a path already in this run and not do that? What you could try is to do a restore job, then pick "list jobs where a given file is saved" and you could get that listing. Then in the director backup settings, you can update the Fileset resource so it doesn't have duplicates and add "Ignore Fileset Changes = yes" to prevent another Full backup from running, but ideally you would run a new Full backup after updating the Fileset resource. HTH, I've already made that change, and the new Full's are running/have run. I was just wondering if it would make sense for Bacula to become smarter about this, and be helpful and not duplicate the files. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Duplicate Backups with OneFS=no and the directory winding up in the list multiple times?
> On Tue, 04 Jun 2019 18:08:16 -0500, Larry Rosenman said: > > On 06/04/2019 6:05 pm, Chandler wrote: > > Larry Rosenman wrote on 6/4/19 15:46: > >> I was just wondering if it would make sense for Bacula to become > >> smarter about this, > >> and be helpful and not duplicate the files. > > > > Well bacula will do what you tell it, so it's up to you to figure out > > if there are duplicated entries in your Fileset. > > > > > > > > I was kind of hoping it would sort the list into the include/exclude > lists, and especially > if OneFS = yes is set, only hit a path ONCE. > > But I guess not. > > Oh well, was worth asking. OneFS = yes will prevent bacula from descending into mounted filesystems, so will prevent duplicates of those. There is no detection of duplicates within a single filesystem though. You can check for duplicate files in the catalog using this query (works on PostgreSQL at least): select dup.jobid, path.path, filename.name, countof from (select jobid, pathid, filenameid, count(*) as countof from file group by jobid, pathid, filenameid having count(*)>1) as dup inner join path on dup.pathid = path.pathid inner join filename on dup.filenameid = filename.filenameid where not filename.name = '' limit 100; Remove the limit 100 if you want to see them all. __Martin ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Duplicate Backups with OneFS=no and the directory winding up in the list multiple times?
On 06/04/2019 6:05 pm, Chandler wrote: Larry Rosenman wrote on 6/4/19 15:46: I was just wondering if it would make sense for Bacula to become smarter about this, and be helpful and not duplicate the files. Well bacula will do what you tell it, so it's up to you to figure out if there are duplicated entries in your Fileset. I was kind of hoping it would sort the list into the include/exclude lists, and especially if OneFS = yes is set, only hit a path ONCE. But I guess not. Oh well, was worth asking. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: l...@lerctr.org US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Duplicate Backups with OneFS=no and the directory winding up in the list multiple times?
Larry Rosenman wrote on 6/4/19 15:46: I was just wondering if it would make sense for Bacula to become smarter about this, and be helpful and not duplicate the files. Well bacula will do what you tell it, so it's up to you to figure out if there are duplicated entries in your Fileset. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Duplicate Backups with OneFS=no and the directory winding up in the list multiple times?
On 06/04/2019 5:25 pm, Chandler wrote: Larry Rosenman wrote on 6/4/19 14:56: Is there a way to have bacula see if it's backed up a path already in this run and not do that? What you could try is to do a restore job, then pick "list jobs where a given file is saved" and you could get that listing. Then in the director backup settings, you can update the Fileset resource so it doesn't have duplicates and add "Ignore Fileset Changes = yes" to prevent another Full backup from running, but ideally you would run a new Full backup after updating the Fileset resource. HTH, I've already made that change, and the new Full's are running/have run. I was just wondering if it would make sense for Bacula to become smarter about this, and be helpful and not duplicate the files. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: l...@lerctr.org US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Duplicate Backups with OneFS=no and the directory winding up in the list multiple times?
Larry Rosenman wrote on 6/4/19 14:56: Is there a way to have bacula see if it's backed up a path already in this run and not do that? What you could try is to do a restore job, then pick "list jobs where a given file is saved" and you could get that listing. Then in the director backup settings, you can update the Fileset resource so it doesn't have duplicates and add "Ignore Fileset Changes = yes" to prevent another Full backup from running, but ideally you would run a new Full backup after updating the Fileset resource. HTH, Chandler / Systems Administrator Arizona Genomics Institute University Of Arizona www.genome.arizona.edu ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Duplicate Backups with OneFS=no and the directory winding up in the list multiple times?
Greetings, I just discovered that I was backing up certain directories on my system(s) multiple times due to having OneFS=no and the paths winding up in the file list multiple times. Is there a way to have bacula see if it's backed up a path already in this run and not do that? I realize this is a degenerate case, but I figure I'd ask. This is with 9.4.3 on FreeBSD from Ports. Thanks, Larry Rosenman -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: l...@lerctr.org US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
