Re: [Bacula-users] bacula 3.0.3 maintain ctime on restored files?
In the case here it's rather a large issue (granted due to other problems like hardware or software issues that require restores of data and is NOT something that I want to continue, however we are living in an imperfect world) but to give you an idea the dataset size is about 30-40TiB with restores anywhere from 2TiB to a full restore to back that back up again not only takes a LOT of tapes which is costly it also takes a LOT of time (day/weeks) where nothing else can be run. What I have been doing which is just painful is do a full restore, then have to do a full backup right after then continue so a restore is actually the time of about 2x of a full. (for a full set this is about 9-10 days on LTO4). This has happened about 3 times so far in the past 2 months. Have you considered breaking up such a big dataset into pieces that only take up perhaps 5 tapes or less each? I know it can be a pain to manage, but it seems like that would go a long way toward. The other benefit is that if a tape breaks during the restore (or is just discovered to be unreadable) you can still get 100% of the other datasets restored. IMHO, any time you realize you can't back up your fileset in one day or even one weekend that's a good opportunity to take a step back and re-think things. Perhaps you might even need to take the step of scheduling the fulls so they don't occur on the same schedule. I realize you that your periodic audits of your filesets to make sure you're not missing something become a lot more complicated, but a painful audit once per quarter seems like a lot better than what you've described above. -- Throughout its 18-year history, RSA Conference consistently attracts the world's best and brightest in the field, creating opportunities for Conference attendees to learn about information security's most important issues through interactions with peers, luminaries and emerging and established companies. http://p.sf.net/sfu/rsaconf-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula 3.0.3 maintain ctime on restored files?
Yes, unfortunately the directory names and the amount of data in them varies dramatically (image file processing for each project) so there is no real way to break it apart without a large chance of missing something.Ideally I would like to have the SD multi-plex the files to different tape drives (i.e. get linear improvement with the number of drives) but haven't looked into it as it's only a problem with full backups/restores which /should/ be infrequent except now when we're having other problems. On 01/15/2010 09:02, Bob Hetzel wrote: In the case here it's rather a large issue (granted due to other problems like hardware or software issues that require restores of data and is NOT something that I want to continue, however we are living in an imperfect world) but to give you an idea the dataset size is about 30-40TiB with restores anywhere from 2TiB to a full restore to back that back up again not only takes a LOT of tapes which is costly it also takes a LOT of time (day/weeks) where nothing else can be run. What I have been doing which is just painful is do a full restore, then have to do a full backup right after then continue so a restore is actually the time of about 2x of a full. (for a full set this is about 9-10 days on LTO4). This has happened about 3 times so far in the past 2 months. Have you considered breaking up such a big dataset into pieces that only take up perhaps 5 tapes or less each? I know it can be a pain to manage, but it seems like that would go a long way toward. The other benefit is that if a tape breaks during the restore (or is just discovered to be unreadable) you can still get 100% of the other datasets restored. IMHO, any time you realize you can't back up your fileset in one day or even one weekend that's a good opportunity to take a step back and re-think things. Perhaps you might even need to take the step of scheduling the fulls so they don't occur on the same schedule. I realize you that your periodic audits of your filesets to make sure you're not missing something become a lot more complicated, but a painful audit once per quarter seems like a lot better than what you've described above. -- Throughout its 18-year history, RSA Conference consistently attracts the world's best and brightest in the field, creating opportunities for Conference attendees to learn about information security's most important issues through interactions with peers, luminaries and emerging and established companies. http://p.sf.net/sfu/rsaconf-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Throughout its 18-year history, RSA Conference consistently attracts the world's best and brightest in the field, creating opportunities for Conference attendees to learn about information security's most important issues through interactions with peers, luminaries and emerging and established companies. http://p.sf.net/sfu/rsaconf-dev2dev___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula 3.0.3 maintain ctime on restored files?
On Wed, 13 Jan 2010 20:18:19 -0600, Steve Costaras said: Ok, found out why when I do a restore of files bacula keeps thinking that they are 'new' and will back them up again. Seems that bacula changes ctime to the time of the restore of the file not the original ctime. atime mtime are properly set on the files at restore but not ctime. I didn't see anything in the on-line docs, is there a flag in the fileset directive to keep ALL time values (atime, mtime, ctime) to be exactly as they were on the original file when doing a restore? No, there is no way to do that. It is impossible to modify the ctime on unix without hacking the disk directly. __Martin -- Throughout its 18-year history, RSA Conference consistently attracts the world's best and brightest in the field, creating opportunities for Conference attendees to learn about information security's most important issues through interactions with peers, luminaries and emerging and established companies. http://p.sf.net/sfu/rsaconf-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula 3.0.3 maintain ctime on restored files?
Ok, then let me ask this in a different way. How do you prevent Bacula from backing up files that were just restored? I see the mtimeonly flag in the fileset options but there are many caveats about using it as you will miss other files that may have been copied over that have retained mtimes from before the last backup. Since bacula does an MD5/SHA1 hash of all files I assumed (wrongly it seems) that it would be smart enough to not back up files that it already had backed up and are on tape. On 01/14/2010 15:11, Martin Simmons wrote: On Wed, 13 Jan 2010 20:18:19 -0600, Steve Costaras said: Ok, found out why when I do a restore of files bacula keeps thinking that they are 'new' and will back them up again. Seems that bacula changes ctime to the time of the restore of the file not the original ctime. atime mtime are properly set on the files at restore but not ctime. I didn't see anything in the on-line docs, is there a flag in the fileset directive to keep ALL time values (atime, mtime, ctime) to be exactly as they were on the original file when doing a restore? No, there is no way to do that. It is impossible to modify the ctime on unix without hacking the disk directly. __Martin -- Throughout its 18-year history, RSA Conference consistently attracts the world's best and brightest in the field, creating opportunities for Conference attendees to learn about information security's most important issues through interactions with peers, luminaries and emerging and established companies. http://p.sf.net/sfu/rsaconf-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Throughout its 18-year history, RSA Conference consistently attracts the world's best and brightest in the field, creating opportunities for Conference attendees to learn about information security's most important issues through interactions with peers, luminaries and emerging and established companies. http://p.sf.net/sfu/rsaconf-dev2dev___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula 3.0.3 maintain ctime on restored files?
Steve Costaras wrote: On 01/14/2010 15:11, Martin Simmons wrote: On Wed, 13 Jan 2010 20:18:19 -0600, Steve Costaras said: Ok, found out why when I do a restore of files bacula keeps thinking that they are 'new' and will back them up again. Seems that bacula changes ctime to the time of the restore of the file not the original ctime. atime mtime are properly set on the files at restore but not ctime. I didn't see anything in the on-line docs, is there a flag in the fileset directive to keep ALL time values (atime, mtime, ctime) to be exactly as they were on the original file when doing a restore? No, there is no way to do that. It is impossible to modify the ctime on unix without hacking the disk directly. Please reply at the bottom of your message. It makes it much easier to follow the issue. Ok, then let me ask this in a different way. How do you prevent Bacula from backing up files that were just restored? It appears you can't. At least, not easily. I see the mtimeonly flag in the fileset options but there are many caveats about using it as you will miss other files that may have been copied over that have retained mtimes from before the last backup. Since bacula does an MD5/SHA1 hash of all files I assumed (wrongly it seems) that it would be smart enough to not back up files that it already had backed up and are on tape. Smart enough? Sheesh. ;) That hash is to ensure the file is restored properly. And for verfication. To do what you want is not easy. How big of a problem is this for you? Have you looked into the Virtual Backups, although I'm not sure this would help you. -- Throughout its 18-year history, RSA Conference consistently attracts the world's best and brightest in the field, creating opportunities for Conference attendees to learn about information security's most important issues through interactions with peers, luminaries and emerging and established companies. http://p.sf.net/sfu/rsaconf-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula 3.0.3 maintain ctime on restored files?
On 01/14/2010 15:59, Dan Langille wrote: Steve Costaras wrote: I see the mtimeonly flag in the fileset options but there are many caveats about using it as you will miss other files that may have been copied over that have retained mtimes from before the last backup. Since bacula does an MD5/SHA1 hash of all files I assumed (wrongly it seems) that it would be smart enough to not back up files that it already had backed up and are on tape. Smart enough? Sheesh. ;) That hash is to ensure the file is restored properly. And for verfication. To do what you want is not easy. How big of a problem is this for you? Have you looked into the Virtual Backups, although I'm not sure this would help you. :) Well I figured it would be relatively easy as an option (since the hash is in the database and when a file is read from disk for backup (since it's planning on backing it up anyway it would need to read the file, if the file name hash match those that are in the database the file could be skipped. (for 'real' completeness and to keep in line w/ the accurate option perhaps update the database with permissions et al on the inode but since the content matches that would save a lot of tapes). In the case here it's rather a large issue (granted due to other problems like hardware or software issues that require restores of data and is NOT something that I want to continue, however we are living in an imperfect world) but to give you an idea the dataset size is about 30-40TiB with restores anywhere from 2TiB to a full restore to back that back up again not only takes a LOT of tapes which is costly it also takes a LOT of time (day/weeks) where nothing else can be run. What I have been doing which is just painful is do a full restore, then have to do a full backup right after then continue so a restore is actually the time of about 2x of a full. (for a full set this is about 9-10 days on LTO4). This has happened about 3 times so far in the past 2 months. -- Throughout its 18-year history, RSA Conference consistently attracts the world's best and brightest in the field, creating opportunities for Conference attendees to learn about information security's most important issues through interactions with peers, luminaries and emerging and established companies. http://p.sf.net/sfu/rsaconf-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula 3.0.3 maintain ctime on restored files?
Steve Costaras wrote: On 01/14/2010 15:59, Dan Langille wrote: Steve Costaras wrote: I see the mtimeonly flag in the fileset options but there are many caveats about using it as you will miss other files that may have been copied over that have retained mtimes from before the last backup. Since bacula does an MD5/SHA1 hash of all files I assumed (wrongly it seems) that it would be smart enough to not back up files that it already had backed up and are on tape. Smart enough? Sheesh. ;) That hash is to ensure the file is restored properly. And for verfication. To do what you want is not easy. How big of a problem is this for you? Have you looked into the Virtual Backups, although I'm not sure this would help you. :) Well I figured it would be relatively easy as an option (since the hash is in the database and when a file is read from disk for backup (since it's planning on backing it up anyway it would need to read the file, if the file name hash match those that are in the database the file could be skipped. (for 'real' completeness and to keep in line w/ the accurate option perhaps update the database with permissions et al on the inode but since the content matches that would save a lot of tapes). In the database... not on the client. That's the issue. Let us not discuss this here. It is not a trivial problem to do correctly. In the case here it's rather a large issue (granted due to other problems like hardware or software issues that require restores of data and is NOT something that I want to continue, however we are living in an imperfect world) but to give you an idea the dataset size is about 30-40TiB with restores anywhere from 2TiB to a full restore to back that back up again not only takes a LOT of tapes which is costly it also takes a LOT of time (day/weeks) where nothing else can be run. What I have been doing which is just painful is do a full restore, then have to do a full backup right after then continue so a restore is actually the time of about 2x of a full. (for a full set this is about 9-10 days on LTO4). This has happened about 3 times so far in the past 2 months. I would start a new thread. How to avoid backing up restored files. This one has gone its course. -- Throughout its 18-year history, RSA Conference consistently attracts the world's best and brightest in the field, creating opportunities for Conference attendees to learn about information security's most important issues through interactions with peers, luminaries and emerging and established companies. http://p.sf.net/sfu/rsaconf-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula 3.0.3 maintain ctime on restored files?
Steve Costaras wrote: Ok, found out why when I do a restore of files bacula keeps thinking that they are 'new' and will back them up again. Seems that bacula changes ctime to the time of the restore of the file not the original ctime. atime mtime are properly set on the files at restore but not ctime. It seems? Have you verified by looking at ctime? What OS? I didn't see anything in the on-line docs, is there a flag in the fileset directive to keep ALL time values (atime, mtime, ctime) to be exactly as they were on the original file when doing a restore? (I did see the keepatime flag but from reading that only affects atime?) Note, if you use this feature, when Bacula resets the access time, the change time (st_ctime) will automatically be modified by the system, so on the next incremental job, the file will be backed up even if it has not changed. As a consequence, you will probably also want to use mtimeonly = yes as well as keepatime (thanks to Rudolf Cejka for this tip). -- Throughout its 18-year history, RSA Conference consistently attracts the world's best and brightest in the field, creating opportunities for Conference attendees to learn about information security's most important issues through interactions with peers, luminaries and emerging and established companies. http://p.sf.net/sfu/rsaconf-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula 3.0.3 maintain ctime on restored files?
On 01/13/2010 20:36, Dan Langille wrote:
Steve Costaras wrote:
Ok, found out why when I do a restore of files bacula keeps thinking
that they are 'new' and will back them up again. Seems that bacula
changes ctime to the time of the restore of the file not the original
ctime. atime mtime are properly set on the files at restore but
not ctime.
It seems? Have you verified by looking at ctime?
What OS?
Yes, sorry for the ambiguity. ctime is set to the time of the restore
of the particular file. atime/mtime show that the times are set to the
original file's times.
OS is linux
Linux loki 2.6.24-26-server #1 SMP Tue Dec 1 18:26:43 UTC 2009 x86_64
GNU/Linux
file system is XFS; using SQLlite3 as the database but that shouldn't
matter here.
I didn't see anything in the on-line docs, is there a flag in the
fileset directive to keep ALL time values (atime, mtime, ctime) to be
exactly as they were on the original file when doing a restore?
(I did see the keepatime flag but from reading that only affects atime?)
Note, if you use this feature, when Bacula resets the access time,
the change time (st_ctime) will automatically be modified by the
system, so on the next incremental job, the file will be backed up
even if it has not changed. As a consequence, you will probably also
want to use mtimeonly = yes as well as keepatime (thanks to Rudolf
Cejka for this tip).
I saw that comment which is why I mentioned it as what I'm seeing is the
opposite. I do not have keepatime set at all and it IS being set to
current time at the time of restore.
-
FileSet {
Name = loki-FileSet
Ignore FileSet Changes = yes
Include {
Options {
accurate=mcs1
checkfilechanges=yes
hardlinks=yes
noatime=yes
onefs=yes
recurse=yes
signature=SHA1
sparse=yes
verify=pns1
}
File = /
File = /boot
File = /home
File = /var/ftp
}
Exclude {
File = /cdrom
File = /dev
File = /lost+found
File = /media
File = /opt/bacula/var/bacula/spool
File = /opt/bacula/var/bacula/working
File = /proc
File = /sys
File = /tmp
File = /var/ftp/tmp
File = /var/lock
File = /var/run
File = /var/tmp
File = /.fsck
File = /.journal
}
}
--
Throughout its 18-year history, RSA Conference consistently attracts the
world's best and brightest in the field, creating opportunities for Conference
attendees to learn about information security's most important issues through
interactions with peers, luminaries and emerging and established companies.
http://p.sf.net/sfu/rsaconf-dev2dev___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
