Re: [Zope] backup full instances

2007-10-28 Thread Jonas Meurer
On 27/10/2007 Andreas Jung wrote:
 This is not the point. rsync will run into if the file to be synced changes
 in-between the sync operation. This will happen with the Data.fs - it won't
 happen with a static copy. An inconsistent copy of the Data.fs is not the 
 problem since invalid transaction entries will be discarded by the ZODB.
 So rsyncing a copy of the Data.fs it the way to go.

Ah, now i got it. Thanks for your patience ;-)

it's interesting that the current backups (rsync the Data.fs directly)
never caused any problems. i guess that's due to the fact that the
Data.fs infact never changed during the backups yet.

i'll change that to backup a local copy of the Data.fs instead in
future. a simple solution would be to run 'cp Data.fs Data.fs.safe'
for every instance just before backuppc starts the rsync process.

thanks for your help,
 jonas


signature.asc
Description: Digital signature
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] backup full instances

2007-10-28 Thread Jonas Meurer
On 27/10/2007 Tino Wildenhain wrote:
 Nobody is wrong. Your #debian guys just had not all informations.
 Zope is always appending to Data.fs, copy (cp) works linear, so
 it would always maintain a consistent state of this file.
 (It either copies before or after the last append operation)

 rsync on the other hand is very efficient by calculating only
 differences of the file contents to be copied. This may or may
 be not following the order of blocks in the file. In the latter
 case it could try to sync wrong information. (You would need
 a special rsync which would only transfer new blocks at the end
 in their given order - you could script something like this
 using dd, gzip/zcat, ssh.

that finally made the difference between rsync and local copy
clear to me ;-)

i'll change the backups to rsync a local copy in the future.

thanks,
 jonas
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] backup full instances

2007-10-27 Thread Jonas Meurer
On 26/10/2007 Andreas Jung wrote:
 Is it safe in general to copy Data.fs while the instance is running? Or
 might I end up with a broken Data.fs, which zope is not able to read at
 all?

 If you create a local copy, it is safe. If you rsync a running Data.fs, it 
 is not safe. In this case you create a local copy and rsync the copy.

what's the difference between local copy and rsync except for the time
it takes?
but if rsync is unsafe only because it takes long time, and changes on the
instance during the copy process could cause a corrupted Data.fs to be
backuped, then theoretically this could happen for a local copy as
well, right? only that the probability would be far smaller.

 For incremental backups: check the repozo.py script.

That one look interesting. thanks for the hint!

greetings,
 jonas


signature.asc
Description: Digital signature
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] backup full instances

2007-10-27 Thread Andreas Jung



--On 27. Oktober 2007 16:29:18 +0200 Jonas Meurer [EMAIL PROTECTED] 
wrote:



On 26/10/2007 Andreas Jung wrote:

Is it safe in general to copy Data.fs while the instance is running? Or
might I end up with a broken Data.fs, which zope is not able to read at
all?


If you create a local copy, it is safe. If you rsync a running Data.fs,
it  is not safe. In this case you create a local copy and rsync the copy.


what's the difference between local copy and rsync except for the time
it takes?
but if rsync is unsafe only because it takes long time, and changes on the
instance during the copy process could cause a corrupted Data.fs to be
backuped, then theoretically this could happen for a local copy as
well, right?


Think twice about your last sentence. What should cause a local *copy*
to be changed??

-aj

pgpL7jXIiuTOY.pgp
Description: PGP signature
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] backup full instances

2007-10-27 Thread Jonas Meurer
On 27/10/2007 Andreas Jung wrote:
 If you create a local copy, it is safe. If you rsync a running Data.fs,
 it  is not safe. In this case you create a local copy and rsync the copy.

 but if rsync is unsafe only because it takes long time, and changes on the
 instance during the copy process could cause a corrupted Data.fs to be
 backuped, then theoretically this could happen for a local copy as
 well, right?

 Think twice about your last sentence. What should cause a local *copy*
 to be changed??

zope might write to the Data.fs while it is copied, thus an inconsistent
copy would be backuped, even inside one filesystem.

i've asked in #debian on freenode as i wasn't sure, here's the log:

 mejo if i copy a large file inside a mounted filesystem (ext3), is it
possible that the file is changed during the copy process?
 Wyzard mejo: Yes, it's possible that something else can write to the
file while you're copying it
 mejo because i asked on the zope-users mailinglist if i could backup
the global Data.fs (database) while the daemon is running, and
someone answered:
 mejo If you create a local copy, it is safe. If you rsync a running
Data.fs it  is not safe. In this case you create a local copy
and rsync the copy.
 Wyzard mejo: Making a local copy is faster, so it'd be safer, but
still not completely safe
 mejo exactly, that's what i thought as well.
 mejo but when i wrote that, he replied:
 mejo Think twice about your last sentence. What should cause a local
*copy* to be changed??
 mejo so he's wrong?
 Wyzard I'd say he's wrong
 Wyzard A local copy isn't instantaneous, and Zope changes the file
while it's being read
 wols_ he is wrong yes
 mejo thanks. is it ok for you if i quote you in my reply mail?
 Wyzard mejo: sure
 wols_ mejo: while copying zope could change the database and create
an inconsistent state

greetings,
 jonas


signature.asc
Description: Digital signature
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] backup full instances

2007-10-27 Thread knitti
On 10/27/07, Jonas Meurer [EMAIL PROTECTED] wrote:
 zope might write to the Data.fs while it is copied, thus an inconsistent
 copy would be backuped, even inside one filesystem.

if you insist on copying a file, _and_ happen to be on linux with some
lvm volumes, you can snapshot that volume with Data.fs on it (throw
perhaps a sync before, don't know if it makes a difference). So you get
the copy at one point in time. you can also rsync from that.

--knitti
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] backup full instances

2007-10-27 Thread Andreas Jung



--On 27. Oktober 2007 18:18:23 +0200 Jonas Meurer [EMAIL PROTECTED] 
wrote:



On 27/10/2007 Andreas Jung wrote:

If you create a local copy, it is safe. If you rsync a running Data.fs,
it  is not safe. In this case you create a local copy and rsync the
copy.


but if rsync is unsafe only because it takes long time, and changes on
the instance during the copy process could cause a corrupted Data.fs to
be backuped, then theoretically this could happen for a local copy as
well, right?


Think twice about your last sentence. What should cause a local *copy*
to be changed??


zope might write to the Data.fs while it is copied, thus an inconsistent
copy would be backuped, even inside one filesystem.


This is not the point. rsync will run into if the file to be synced 
changes

in-between the sync operation. This will happen with the Data.fs - it won't
happen with a static copy. An inconsistent copy of the Data.fs is not the 
problem since invalid transaction entries will be discarded by the ZODB.

So rsyncing a copy of the Data.fs it the way to go.


-aj

pgpfgonwoDECt.pgp
Description: PGP signature
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] backup full instances

2007-10-27 Thread Tino Wildenhain

Hi,

Jonas Meurer schrieb:

On 27/10/2007 Andreas Jung wrote:

If you create a local copy, it is safe. If you rsync a running Data.fs,
it  is not safe. In this case you create a local copy and rsync the copy.

but if rsync is unsafe only because it takes long time, and changes on the
instance during the copy process could cause a corrupted Data.fs to be
backuped, then theoretically this could happen for a local copy as
well, right?

Think twice about your last sentence. What should cause a local *copy*
to be changed??


zope might write to the Data.fs while it is copied, thus an inconsistent
copy would be backuped, even inside one filesystem.

i've asked in #debian on freenode as i wasn't sure, here's the log:

 mejo if i copy a large file inside a mounted filesystem (ext3), is it
possible that the file is changed during the copy process?
 Wyzard mejo: Yes, it's possible that something else can write to the
file while you're copying it
 mejo because i asked on the zope-users mailinglist if i could backup
the global Data.fs (database) while the daemon is running, and
someone answered:
 mejo If you create a local copy, it is safe. If you rsync a running
Data.fs it  is not safe. In this case you create a local copy
and rsync the copy.
 Wyzard mejo: Making a local copy is faster, so it'd be safer, but
still not completely safe
 mejo exactly, that's what i thought as well.
 mejo but when i wrote that, he replied:
 mejo Think twice about your last sentence. What should cause a local
*copy* to be changed??
 mejo so he's wrong?
 Wyzard I'd say he's wrong
 Wyzard A local copy isn't instantaneous, and Zope changes the file
while it's being read
 wols_ he is wrong yes
 mejo thanks. is it ok for you if i quote you in my reply mail?
 Wyzard mejo: sure
 wols_ mejo: while copying zope could change the database and create
an inconsistent state


Nobody is wrong. Your #debian guys just had not all informations.
Zope is always appending to Data.fs, copy (cp) works linear, so
it would always maintain a consistent state of this file.
(It either copies before or after the last append operation)

rsync on the other hand is very efficient by calculating only
differences of the file contents to be copied. This may or may
be not following the order of blocks in the file. In the latter
case it could try to sync wrong information. (You would need
a special rsync which would only transfer new blocks at the end
in their given order - you could script something like this
using dd, gzip/zcat, ssh.

Regards
Tino





___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] backup full instances

2007-10-27 Thread Dieter Maurer
Jonas Meurer wrote at 2007-10-26 16:57 +0200:
 ...
The main problem with this backup approach is, that the zope instance is
not stopped for the backup at all, thus the Data.fs might not be up to
date.

Why do you think that this is a problem?

Of course, your backup process may read a partial transaction record --
it will be dropped automatically when the storage file is opened.
Of course, your backup process may not have backed up the state
at the end of the backup -- but it will have backed up the state at
the start of the backup.

Thus, all you can have as a problem: the effective backup may
not exceed the state at it was when the backup started.

I think, this should be good enough.



-- 
Dieter
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] backup full instances

2007-10-26 Thread Jonathan


- Original Message - 
From: Jonas Meurer [EMAIL PROTECTED]

To: zope-users zope@zope.org
Sent: Friday, October 26, 2007 10:57 AM
Subject: [Zope] backup full instances



Hello,

We have several different zope instances running on our servers, and
currently the only backup we do is a daily copy of the Data.fs.

The main problem with this backup approach is, that the zope instance is
not stopped for the backup at all, thus the Data.fs might not be up to
date.

Is it safe in general to copy Data.fs while the instance is running? Or
might I end up with a broken Data.fs, which zope is not able to read at
all?

And which other ways to you know to backup an entire instance? Maybe
sync it with ZEO to another server in realtime? Or use some export
method?


Have a look at:

http://wiki.zope.org/ZODB/FileStorageBackup


Jonathan
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] backup full instances

2007-10-26 Thread Andreas Jung



--On 26. Oktober 2007 16:57:02 +0200 Jonas Meurer [EMAIL PROTECTED] 
wrote:




Is it safe in general to copy Data.fs while the instance is running? Or
might I end up with a broken Data.fs, which zope is not able to read at
all?


If you create a local copy, it is safe. If you rsync a running Data.fs, it 
is not safe. In this case you create a local copy and rsync the copy.




And which other ways to you know to backup an entire instance? Maybe
sync it with ZEO to another server in realtime? Or use some export
method?



http://www.zope.com/products/zope_replication_services.html

Check also for zeoraid from Gocept.

For incremental backups: check the repozo.py script.

-aj

pgpU92i8GAnpL.pgp
Description: PGP signature
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] backup full instances

2007-10-26 Thread Sascha Welter
(Fri, Oct 26, 2007 at 04:57:02PM +0200) Jonas Meurer wrote/schrieb/egrapse:
 And which other ways to you know to backup an entire instance? Maybe
 sync it with ZEO to another server in realtime? Or use some export
 method?

repozo.py

Also don't forget to backup all the products and zope.conf once in a
while.

Regards,

Sascha

___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )