Re: [BackupPC-users] Different Blackout Periods
Hi, I used one file per message. The amount of messages are about 100 per hour. In my setup now, I can see that the number of files are increasing every incremental backup (the incr backup self takes 0.5mins). IncrLevels is set to 1. If read the doc about this value but I think I didn't understand this right. In my case, were I takes (7*24) around 168 incr backups between two full backups, which value were a good choice for IncrLevels? Is it 1,2,3,4,5...,167? I get a little bit confused because my english is not very well. Thank you for helping! Greetings Alex -Ursprüngliche Nachricht- Von: Les Mikesell [mailto:lesmikes...@gmail.com] Gesendet: Freitag, 3. April 2015 18:16 An: General list for user discussion, questions and support Betreff: Re: [BackupPC-users] Different Blackout Periods On Thu, Apr 2, 2015 at 4:25 AM, Alexander Rehbein alexander.rehb...@fmex.de wrote: I've looked in the archive. Sometimes the incr backup is only 20mb, sometimes 200mb. Perhaps it is an better idea to only backup /var/vmail directory hourly instead of the full root path. First, make sure you understand how backuppc works - and the type of your mail files. If you use the old mbox mail format where messages are appended to a single growing mailbox file, and use rsync, only the difference will be transfered, but there is some CPU work involved to compute the changes, and on the server side a complete new file copy will be constructed and stored separately. Xfer methods other than rsync will send the whole changed file.If your mail storage uses one message per file like maildir format, rsync will only send the new files (whether doing a full or incremental) and the ones where the content matches the previous copy will all be pooled without taking additional space. Non-rsync xfers will send only new files on incrementals and everything on fulls, but existing content is still found and pooled in storage. Note that 'new' means newer than the previous full unless you have configured incremental levels. So, there won't be much difference in the storage used regardless of your mix of incremental/full runs. The main difference will be in the time they take to complete and possibly the performance impact on the target system. An incremental rsync run quickly skips fields where the length/timestamp match the previous copy where the fulls do a full read of the target content to do a block-checksum comparison, Also, the fulls will rebuild the archive directory tree which may be time-consuming with many small files, but that becomes the new base for incrementals, making the next ones more efficient. -- Les Mikesell lesmikes...@gmail.com -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Different Blackout Periods
Hello Falko, thank you. The bbc Option is a very good idea for archive the mails. I will have a look. Until now I setup two hosts in backuppc. One which only backup mails hourly and one which backup the whole server daily without maildir. Greetings Alex -Ursprüngliche Nachricht- Von: Falko Trojahn [mailto:n...@trojahn.de] Gesendet: Freitag, 3. April 2015 15:56 An: General list for user discussion, questions and support Betreff: Re: [BackupPC-users] Different Blackout Periods Hello Alexander, there's a slightly off-topic suggestion for you. Alexander Rehbein wrote on 02.04.2015 11:25: You told something about an delta. Are diff backups better for me? I want to do the following: I have an Mailserver which should do a full backup every week. It could happen that I delete an email faulty, so I decided to backup the mails every hour. This I want do with incr backups. If a mail get lost I can restore this Mail from the backup archive. I've looked in the archive. Sometimes the incr backup is only 20mb, sometimes 200mb. Perhaps it is an better idea to only backup /var/vmail directory hourly instead of the full root path. According to your needs, may be there is another possibility apart from stressing your mail server with hourly backups. And, your statement If a mail get lost I can restore this Mail from the backup archive. is only partly true. If both the mail arrives and is deleted between two incremental backups, it will get lost, too. So what can you do is: use the always_bcc feature of your MTA - e.g. if you use dovecot with postfix, have a look at http://www.postfix.org/ADDRESS_REWRITING_README.html#auto_bcc and put it in master.cf in the cleanup process. I assume your backuppc server is physically on another host. There are several possibilities: - send all incoming and outgoing mail of the server to one or several bcc accounts, using fetchmail on the backup host to fetch the mail and deliver it * to local account * to a file, e.g. using procmail - send all incoming and outgoing mail per domain or per user directly to an archive host, which must not be the same as the backuppc host, but can = backup the archive host using backuppc as usual Find attached an example .procmailrc which we use together with one incoming and one outgoing bcc account per server to split important mail from not so relevant ones. If there are several mail processes on the server (content/antivirus filters etc.) make sure you generate the bcc mails only once. The generated files can be gzipped at end of month, and even archived (to fullfill the archiving duty in germany: http://de.wikipedia.org/wiki/E-Mail-Archivierung ) HTH Falko # === ~/.procmailrc === PATH=$HOME/bin:/usr/bin:/bin:/usr/local/bin MAILDIR=$HOME/Mail # Youd better make sure it exists YEAR=`date +%Y` MONTH=`date +%m` DEFAULT=$MAILDIR/IN-$YEAR-$MONTH # for outgoing control user: # DEFAULT=$MAILDIR/OUT-$YEAR-$MONTH LOGFILE=$MAILDIR/_maillog-$YEAR-$MONTH :0: * ^X-Spam-Status: Yes,.*$ SPAM-$YEAR-$MONTH :0: * ^X-Quarantine-ID: .*$ BADHEADER-$YEAR-$MONTH :0: * ^From: logcheck@ SYSTEM-$YEAR-$MONTH :0: * ^From: root@ SYSTEM-$YEAR-$MONTH :0: * ^List-Id: .*$ MAILINGLIST-$YEAR-$MONTH -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Different Blackout Periods
One more idea. Instead of saving so much incr backups, perhaps it would be better to keep for example only 3 and delete the other, because, as you told me, the incr backups hold every changed file since the full backup. -Ursprüngliche Nachricht- Von: Les Mikesell [mailto:lesmikes...@gmail.com] Gesendet: Freitag, 3. April 2015 18:16 An: General list for user discussion, questions and support Betreff: Re: [BackupPC-users] Different Blackout Periods On Thu, Apr 2, 2015 at 4:25 AM, Alexander Rehbein alexander.rehb...@fmex.de wrote: I've looked in the archive. Sometimes the incr backup is only 20mb, sometimes 200mb. Perhaps it is an better idea to only backup /var/vmail directory hourly instead of the full root path. First, make sure you understand how backuppc works - and the type of your mail files. If you use the old mbox mail format where messages are appended to a single growing mailbox file, and use rsync, only the difference will be transfered, but there is some CPU work involved to compute the changes, and on the server side a complete new file copy will be constructed and stored separately. Xfer methods other than rsync will send the whole changed file.If your mail storage uses one message per file like maildir format, rsync will only send the new files (whether doing a full or incremental) and the ones where the content matches the previous copy will all be pooled without taking additional space. Non-rsync xfers will send only new files on incrementals and everything on fulls, but existing content is still found and pooled in storage. Note that 'new' means newer than the previous full unless you have configured incremental levels. So, there won't be much difference in the storage used regardless of your mix of incremental/full runs. The main difference will be in the time they take to complete and possibly the performance impact on the target system. An incremental rsync run quickly skips fields where the length/timestamp match the previous copy where the fulls do a full read of the target content to do a block-checksum comparison, Also, the fulls will rebuild the archive directory tree which may be time-consuming with many small files, but that becomes the new base for incrementals, making the next ones more efficient. -- Les Mikesell lesmikes...@gmail.com -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Different Blackout Periods
Thank you, this is a really large answer. However my English is not very
well I hope I understand your answer right.
At the moment the solution is only doing incr backups and full via cron?
You told something about an delta. Are diff backups better for me? I want to
do the following:
I have an Mailserver which should do a full backup every week. It could
happen that I delete an email faulty, so I decided to backup the mails every
hour. This I want do with incr backups. If a mail get lost I can restore
this Mail from the backup archive.
I've looked in the archive. Sometimes the incr backup is only 20mb,
sometimes 200mb. Perhaps it is an better idea to only backup /var/vmail
directory hourly instead of the full root path.
It also would be nice if some option like 'backupType' = 'full' would be
implemented. I could image something like a name you can setup for an
backup, so you can setup blackout periods different. For example:
BackupSet1 (full every week) with own blackout periods
BackupSet2 (incr full root path daily) with own blackout periods
BackupSet3 (incr /var/vmail path hourly) with own blackout periods
If no specials blackout periods defined, the default system blackout periods
would be used.
Greetings,
Alex
-Ursprüngliche Nachricht-
Von: Holger Parplies [mailto:wb...@parplies.de]
Gesendet: Mittwoch, 1. April 2015 23:38
An: Les Mikesell
Cc: General list for user discussion, questions and support
Betreff: Re: [BackupPC-users] Different Blackout Periods
Hi,
Les Mikesell wrote on 2015-04-01 13:25:56 -0500 [Re: [BackupPC-users]
Different Blackout Periods]:
On Wed, Apr 1, 2015 at 1:03 PM, alexander.rehb...@fmex.de wrote:
I want to setup the following backup plan. Every week one full
backup and every hour an incremental backup.
This works but i have a problem with the balackout periods. Full
backups should only done at night. is it possible to configure this?
There is no way to specify different blackouts for fulls vs.
incrementals.
true.
However fulls won't happen until the FullPeriod time has expired since
the last one, so if you force one at an appropriate time, subsequent
runs will happen at about the same time a week later.
That is not strictly true. If I remember correctly, a new full backup will
not happen until the FullPeriod has expired since *completion* of the last
one which might be significantly later, depending on the time the backup
takes and backup concurrency issues. Usually, you can compensate for that
with the schedule, blackouts, and an appropriately lower FullPeriod, to keep
your backups from wandering through the day.
Here, however, blackouts can't be used. What you suggest is very much
equivalent to implement the effect of blackouts by just forgetting about
them. If blackouts are a requirement, i.e. full backups *must* only be done
at night, rather than it would be rather convenient if they mostly run at
night until something unexpected happens, when we will be happy to manually
intervene to correct the timing, then this is not a solution.
I've got a small patch which I would have first expected would do what you
want, however the task is not as trivial as it seems. BackupPC schedules an
incremental backup or a full one as appropriate. When a full one is due, it
is rescheduled until it completes successfully, perhaps being skipped
several times due to blackouts. In the mean time, there is no point in
attempting incremental backups, because the same blackouts would apply. What
you really want to happen when a full run is skipped due to a blackout
period applying only to full backups is an incremental backup to be run
instead. However, a manually scheduled full backup should probably not be
automatically turned into an incremental. Overlapping blackout periods for
full and incremental backups should work as expected, i.e. an incremental
replacement backup should still be restricted by blackouts for incrementals,
regardless of the order in which blackouts are specified. So there are some
things to keep in mind.
An additional problem is that, programmatically, BackupPC currently handles
blackouts *before* making the decision whether a full or incremental backup
is needed. There may be a good reason for that (such as avoiding pings
during blackouts - pingCmd might attempt to wake up the remote host) or not,
I'd have to take a closer look. In any case, it would be significantly more
intrusive than changing two lines of code as my first attempt would have.
Configuration wise, I would want to add an optional hash member backupType
with supported values 'full', 'incr' or 'all', defaulting to 'all', which
would make the blackout period apply to backups of the selected type(s),
e.g.:
$Conf{BlackoutPeriods} = [
{
'weekDays' = [ 1 .. 7 ],
'hourBegin' = 4,
'hourEnd' = 2,
'backupType' = 'full',
},
[BackupPC-users] Different Blackout Periods
Hello,
I want to setup the following backup plan. Every week one full backup
and every hour an incremental backup.
This works but i have a problem with the balackout periods. Full backups
should only done at night. is it possible to configure this?
My actually config looks like:
$Conf{FullKeepCnt} = [
'4',
'0',
'4',
'0',
'0',
'2'
];
$Conf{FullPeriod} = '6.97';
$Conf{IncrKeepCnt} = '150';
$Conf{IncrPeriod} = '0.04';
$Conf{BlackoutPeriods} = [
{
'hourEnd' = '2',
'weekDays' = [
'1',
'2',
'3',
'4',
'5',
'6',
'7'
],
'hourBegin' = '4'
}
];
Greetings
Alexander Rehbein
--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
