Re: rather simple question around dumpcycle
Am 19.11.19 um 06:12 schrieb Jon LaBadie: > On Sun, Nov 17, 2019 at 12:16:18PM +0100, Stefan G. Weichinger wrote: >> Am 15.11.19 um 20:32 schrieb Jon LaBadie: >> >>> I was not thinking of any algorithm. Just a weekend crontab entries like: >>> >>> amadmin ... force ... ; amdump ... >> >> Yes, I understood that. But that "manual force" always disturbs the >> schedule built by the amanda algorithm, right? >> > > Would "amdump -o no-record ..." or somesuch correct that? > >> At least that was my observation, the force didn't work out reliably, I >> assume because of too much data in relation to the media size and number >> of tapes. > > True, whether same or separate config ;) everything changed yesterday: server died, mobo flaky or so Now up on a fallback hardware ... and noticed that I didn't have a DLE containing the amanda DBs and indexes (only /etc with the configs) Not fatal, but ugly. I could restore stuff from tapes already ... make sure you have that encryption passphrase somewhere else ... That situation is stress. Glad the data wasn't touched ... it's on the SAN storage and OK.
Re: rather simple question around dumpcycle
On Sun, Nov 17, 2019 at 12:16:18PM +0100, Stefan G. Weichinger wrote: > Am 15.11.19 um 20:32 schrieb Jon LaBadie: > > > I was not thinking of any algorithm. Just a weekend crontab entries like: > > > > amadmin ... force ... ; amdump ... > > Yes, I understood that. But that "manual force" always disturbs the > schedule built by the amanda algorithm, right? > Would "amdump -o no-record ..." or somesuch correct that? > At least that was my observation, the force didn't work out reliably, I > assume because of too much data in relation to the media size and number > of tapes. True, whether same or separate config ;) jl -- Jon H. LaBadie j...@jgcomp.com 11226 South Shore Rd. (703) 787-0688 (H) Reston, VA 20190 (703) 935-6720 (C)
Re: rather simple question around dumpcycle
Am 15.11.19 um 20:32 schrieb Jon LaBadie: > I was not thinking of any algorithm. Just a weekend crontab entries like: > > amadmin ... force ... ; amdump ... Yes, I understood that. But that "manual force" always disturbs the schedule built by the amanda algorithm, right? At least that was my observation, the force didn't work out reliably, I assume because of too much data in relation to the media size and number of tapes.
Re: rather simple question around dumpcycle
On Tue, Nov 12, 2019 at 11:15:24AM +0100, Stefan G. Weichinger wrote: > Am 12.11.19 um 00:12 schrieb Jon LaBadie: > > > What about a single config incremental only. > > > > Via crontab, force level 0's when you want, > > maybe every other weekend FRI & SAT. > > > > The SAT dump serves as level 0's for the incrementals. > > > > The FRI dump can be used for your archiving as needed. > > Take the tape out of use but retain logs etc. Add a > > new tape to keep the same # of tapes in use. > > Well, yes, I tried that already. > > But it was always hard to get the algorithm do lev0 on weekends. > I was not thinking of any algorithm. Just a weekend crontab entries like: amadmin ... force ... ; amdump ... jl -- Jon H. LaBadie j...@jgcomp.com 11226 South Shore Rd. (703) 787-0688 (H) Reston, VA 20190 (703) 935-6720 (C)
Re: rather simple question around dumpcycle
> On Nov 12, 2019, at 10:53 AM, Stefan G. Weichinger wrote: > > Am 12.11.19 um 15:27 schrieb Stefan G. Weichinger: > >> I think that should be pointed out somewhere in the docs ... maybe I >> should write something (or you ;-)) > > The more I think of this I suggest that the default value of the > > property "GNUTAR-LISTDIR" > > should be "/var/lib/amanda/gnutar-lists/$config/" > > anyway. Right? > > > Probably, yes. I _think_ I still have some configs that use old fashioned DUMP rather than TAR. I assumed they still “recorded” to the one /etc/dumpdates (??) file, and would definitely mess up the other configuration, as described above. It does sound like one could separate the TAR record location. Deb Baddorf Fermilab
Re: rather simple question around dumpcycle
Am 12.11.19 um 15:27 schrieb Stefan G. Weichinger: > I think that should be pointed out somewhere in the docs ... maybe I > should write something (or you ;-)) The more I think of this I suggest that the default value of the property "GNUTAR-LISTDIR" should be "/var/lib/amanda/gnutar-lists/$config/" anyway. Right?
Re: rather simple question around dumpcycle
Am 12.11.19 um 14:55 schrieb Nathan Stratton Treadway:
> The danger in your particular case is that the "weekly" run will update
> the level 0 snapshot file for each DLE, and then the any "daily" runs
> that do level 1 dumps the following week will only include files changed
> after that "weekly" level 0 -- so if you were to try to recover using
> only the "daily" dumps, there could be some files that are missing
*DANGER* !
> You can check for this sort of clash by looking at the dates of the
> files in the gnutar-listdir/gnutar-list-dir directory, which defaults to
> /var/lib/amanda/gnutar-lists/ in the Debian/Ubuntu Amanda package. If
> you check there early in the week and you see a mix of dates from the
> previous week ("daily" config runs) and over the weekend (the "weekly"
> runs), then you are affected.
>
> I haven't ever tried "record no", but skimming the source code it
> appears that this works by preventing amanda from renaming the working
> incremental snapshot file, "_.new" back to "_".
> So I guess as long as the only files dated over the weekend are ".new"
> files, you are okay; those files are just "orphans" will be ignored by
> later runs.
>
> (If you don't want to use "record no" for one of the configs and are
> using amgtar, it's easy to avoid the conflicting-snapshot-files problem
> by setting separate directories for each config in the amanda.conf
> amgtar application defintion, e.g. add the config name at the end of the
> path, so for config "TestBackup" I have:
>
> define application-tool app_amgtar {
> comment "amgtar"
> plugin "amgtar"
> property "GNUTAR-LISTDIR" "/var/lib/amanda/gnutar-lists/TestBackup/"
> property "CHECK-DEVICE" "NO"
> }
>
> You do have to manually create that directory for each config as part of
> getting a new client machine set up in the disklist, though. [amcheck
> will warn you, if I remember correctly.]
yes, it does. Phew, I definitely wasn't aware of the fact that the 2
configs conflict here!
I decided to set GNUTAR-LISTDIR now for "weekly" and will do as well for
"daily" asap.
Maybe that will lead to more balance here ...
I prefer this over setting "record no".
thanks!
I think that should be pointed out somewhere in the docs ... maybe I
should write something (or you ;-))
Re: rather simple question around dumpcycle
On Tue, Nov 12, 2019 at 11:20:49 +0100, Stefan G. Weichinger wrote:
> Am 11.11.19 um 22:10 schrieb Debra S Baddorf:
> > My dumptype definition includes
>
> > strategy noinc #don???t even calculate partial sizes
> > skip-incr yes # belt and suspenders
> > record no # daily incrementals are based on the daily level-0s.
> > # If I let this record, I think that messes the daily
> > incrementals.
>
> That "record no" ... hm, really?
> I mean, afaik there are NO incrementals done here anyway, right?
When you are running multiple configs together over time which have the
same DLEs, you do need to be very careful of the GNU tar incremental
snapshot files (assuming that you are using either GNUTAR or amgtar,
obviously).
The danger in your particular case is that the "weekly" run will update
the level 0 snapshot file for each DLE, and then the any "daily" runs
that do level 1 dumps the following week will only include files changed
after that "weekly" level 0 -- so if you were to try to recover using
only the "daily" dumps, there could be some files that are missing
You can check for this sort of clash by looking at the dates of the
files in the gnutar-listdir/gnutar-list-dir directory, which defaults to
/var/lib/amanda/gnutar-lists/ in the Debian/Ubuntu Amanda package. If
you check there early in the week and you see a mix of dates from the
previous week ("daily" config runs) and over the weekend (the "weekly"
runs), then you are affected.
I haven't ever tried "record no", but skimming the source code it
appears that this works by preventing amanda from renaming the working
incremental snapshot file, "_.new" back to "_".
So I guess as long as the only files dated over the weekend are ".new"
files, you are okay; those files are just "orphans" will be ignored by
later runs.
(If you don't want to use "record no" for one of the configs and are
using amgtar, it's easy to avoid the conflicting-snapshot-files problem
by setting separate directories for each config in the amanda.conf
amgtar application defintion, e.g. add the config name at the end of the
path, so for config "TestBackup" I have:
define application-tool app_amgtar {
comment "amgtar"
plugin "amgtar"
property "GNUTAR-LISTDIR" "/var/lib/amanda/gnutar-lists/TestBackup/"
property "CHECK-DEVICE" "NO"
}
You do have to manually create that directory for each config as part of
getting a new client machine set up in the disklist, though. [amcheck
will warn you, if I remember correctly.]
For GNUTAR, you have to configure this in per-config amanda-client.conf
files on each client, so it's more of a pain.)
Nathan
Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic region
Ray Ontko & Co. - Software consulting services - http://www.ontko.com/
GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239
Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
Re: rather simple question around dumpcycle
Am 11.11.19 um 22:10 schrieb Debra S Baddorf: > My alternate config is run once a month, but I think my params would work > for your > “every weekend” too. > dumpcycle 0 # I plan to run it whenever I want to > tapecycle 9000 # I will always give you a fresh tape. You would say 31. > runtimes 3# the max you’ll allow to catch all the level-0 backups OK, runtapes, as mentioned ;-) > My dumptype definition includes > strategy noinc #don’t even calculate partial sizes > skip-incr yes # belt and suspenders > record no # daily incrementals are based on the daily level-0s. > # If I let this record, I think that messes the daily > incrementals. That "record no" ... hm, really? I mean, afaik there are NO incrementals done here anyway, right? I set most of your suggestions now, some of them were in use already. thanks, I wait for friday then.
Re: rather simple question around dumpcycle
Am 12.11.19 um 00:12 schrieb Jon LaBadie: > What about a single config incremental only. > > Via crontab, force level 0's when you want, > maybe every other weekend FRI & SAT. > > The SAT dump serves as level 0's for the incrementals. > > The FRI dump can be used for your archiving as needed. > Take the tape out of use but retain logs etc. Add a > new tape to keep the same # of tapes in use. Well, yes, I tried that already. But it was always hard to get the algorithm do lev0 on weekends. Some DLEs failed and it was a hassle to get a defined set of tapes with lev0 only. So I tried to solve that with a separate config that does only lev0 from the start.
Re: rather simple question around dumpcycle
> On Nov 11, 2019, at 4:24 PM, Stefan G. Weichinger wrote: > > Am 11.11.19 um 22:10 schrieb Debra S Baddorf: > >> runtimes 3# the max you’ll allow to catch all the level-0 backups > > that parameter is not recognized here ;-) > > Aside from that: thanks a lot for your feedback, Debra I thing that was an “autocorrect” aka “monkey correct” runtapes (Yup, I WATCHED it change it to the wrong thing, this time. But I insisted!) Deb
Re: rather simple question around dumpcycle
On Mon, Nov 11, 2019 at 05:34:14PM +0100, Stefan G. Weichinger wrote: > > At a customer we try to run 2 configs in parallel. > > I discussed this setup here in the past, and bring it back up again ;-) > > - > > We run a daily config with 58 tapes. > This is run on weekdays 1-4 only. > > Until now we had: > > dumpcycle 2 weeks > runspercycle 8 > > Would it be better to have > > dumpcycle 8 days ? > > - > > The 2nd config runs on FR and SAT: "weekly". With full backups only. > > I want to force lev0 here and archive a set of tapes every 3 months -> > quartal and year archive sets. > > I have 31 active tapes there right now, a full set of data takes around > 4-5 tapes. > > What parameters do you users use for such a setup? > > dumpcycle? runspercycle? > > - > > I'd appreciate optimization tips here. It seems my dumpcycles are too > short so I get issues with too much data dumped per week etc >>> End of included message <<< What about a single config incremental only. Via crontab, force level 0's when you want, maybe every other weekend FRI & SAT. The SAT dump serves as level 0's for the incrementals. The FRI dump can be used for your archiving as needed. Take the tape out of use but retain logs etc. Add a new tape to keep the same # of tapes in use. Jon -- Jon H. LaBadie j...@jgcomp.com 11226 South Shore Rd. (703) 787-0688 (H) Reston, VA 20190 (703) 935-6720 (C)
Re: rather simple question around dumpcycle
Am 11.11.19 um 22:10 schrieb Debra S Baddorf: > runtimes 3# the max you’ll allow to catch all the level-0 backups that parameter is not recognized here ;-) Aside from that: thanks a lot for your feedback, Debra
Re: rather simple question around dumpcycle
> On Nov 11, 2019, at 10:34 AM, Stefan G. Weichinger wrote: > > > At a customer we try to run 2 configs in parallel. > > I discussed this setup here in the past, and bring it back up again ;-) > > - > > We run a daily config with 58 tapes. > This is run on weekdays 1-4 only. > > Until now we had: > > dumpcycle 2 weeks > runspercycle 8 > > Would it be better to have > > dumpcycle 8 days ? > > - > > The 2nd config runs on FR and SAT: "weekly". With full backups only. > > I want to force lev0 here and archive a set of tapes every 3 months -> > quartal and year archive sets. > > I have 31 active tapes there right now, a full set of data takes around > 4-5 tapes. > > What parameters do you users use for such a setup? > > dumpcycle? runspercycle? > > - > > I'd appreciate optimization tips here. It seems my dumpcycles are too > short so I get issues with too much data dumped per week etc I run a similar config: daily for everyday (we include Saturday and Sunday too, cuz some people work then & make important changes - no big difference for you). I think you are good with config1 being “2 weeks, runspercycle 8”. That means it will do a FULL on each disk at least once every 2 weeks.Doing it once every 8 days is NOT really what you want. My alternate config is run once a month, but I think my params would work for your “every weekend” too. dumpcycle 0 # I plan to run it whenever I want to tapecycle 9000 # I will always give you a fresh tape. You would say 31. runtimes 3# the max you’ll allow to catch all the level-0 backups My dumptype definition includes strategy noinc #don’t even calculate partial sizes skip-incr yes # belt and suspenders record no # daily incrementals are based on the daily level-0s. # If I let this record, I think that messes the daily incrementals. And in true belt-plus-suspender-plus physically holding up the pants: I run a cronjob before each archive run that does amadmin force * (test this a little, manually — I found I needed to do force *.. ) Hope this is a little help. Deb Baddorf Fermilab
rather simple question around dumpcycle
At a customer we try to run 2 configs in parallel. I discussed this setup here in the past, and bring it back up again ;-) - We run a daily config with 58 tapes. This is run on weekdays 1-4 only. Until now we had: dumpcycle 2 weeks runspercycle 8 Would it be better to have dumpcycle 8 days ? - The 2nd config runs on FR and SAT: "weekly". With full backups only. I want to force lev0 here and archive a set of tapes every 3 months -> quartal and year archive sets. I have 31 active tapes there right now, a full set of data takes around 4-5 tapes. What parameters do you users use for such a setup? dumpcycle? runspercycle? - I'd appreciate optimization tips here. It seems my dumpcycles are too short so I get issues with too much data dumped per week etc
