Re: [Zope] Re: Questions about demostorage

2006-12-18 Thread Chris Withers

Duncan Booth wrote:

Urm, that would involve taking a complete copy of the db at open
time... 


Not knowing the details of the implementation I don't know that it would. 


Knowing the details as I do, I'm telling you it would...

Got that, but the problem is that the test database isn't exactly identical 
to the live one (e.g. the ldap config points to the test ldap server) but 


Then fix the flakey test environment rather than applying even more 
fragile machinery in an attempt to ignore the real problem ;-)


I'm sure we could script the tweaks needed. More significantly is the pain 
involved in copying 10Gb from the live system onto dev on more than an 
occasional basis.


It's still worth it...

The problem I'm trying to address is that sometimes we make mistakes when 
updating the live server because of minor differences between the test and 
live environments.


Ah, a flakey upgrade process too ;-) Take a look at Stepper. One of it's 
main uses is to write upgrade scripts than can be consistently run when 
doing upgrades and test upgrades on a development environment.


took some time and caused site errors until it was complete. I have a 
script which I believe should do the migration quickly but I want to be 
absolutely positive it is going to work without any hitches.


In your position, I would have little to no confidence that using 
DemoStorage in the way you suggest would meet your requirements...


cheers,

Chris

--
Simplistix - Content Management, Zope  Python Consulting
   - http://www.simplistix.co.uk

___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )


[Zope] Re: Questions about demostorage

2006-12-15 Thread Duncan Booth
Chris Withers [EMAIL PROTECTED] wrote:
 (So in other words, the 
 connection from the demostorage front-end to the zeo-client isn't
 frozen at the point when the storage is opened which might have been
 a plausible alternative).
 
 Urm, that would involve taking a complete copy of the db at open
 time... 

Not knowing the details of the implementation I don't know that it would. 
I'd have thought it would involve copying the index rather than the data, 
and since I don't know the implementation of the index it might only need 
to do a copy-on-write as and when parts of the index change.

 But really, what you're doing is not what DemoStorage was 
 designed for and will likely end in tears or a lot of hair pulling...

I know, that's why I'm trying to establish the boundaries of what I can 
expect from it.

 What I'm hoping to achieve is a way of testing upgrades on our live
 system in an environment as close as possible to the real live
 environment, but without actually affecting the live system in any
 way. Demostorage seems to offer that as an option.
 
 It really doesn't ;-) Do this kinda testing the same way that everyone
 does: build yourself a uat environment that mirrors your production 
 environment, maybe with a few less zeo clients, run it off a copy of 
 your production zodb and then test there...

Got that, but the problem is that the test database isn't exactly identical 
to the live one (e.g. the ldap config points to the test ldap server) but 
I'm sure we could script the tweaks needed. More significantly is the pain 
involved in copying 10Gb from the live system onto dev on more than an 
occasional basis.

The problem I'm trying to address is that sometimes we make mistakes when 
updating the live server because of minor differences between the test and 
live environments. So on the live servers we have a separate zope instance 
for final testing. It has its own copy of the Products folder (checked out 
from the relevant svn tag) and when we are happy that the checkout works as 
expected I just rsync that Products folder onto the live Products.

I'm not suggesting running the system on demostorage for long periods of 
time. Simply long enough to check that whatever weird buildout changes we 
make are doing are going to work. That means that the only guarantee I 
really need is that the live system won't be affected by any changes to the 
test system; if the test system becomes internally inconsistent and falls 
over I'll just restart it and try again.

For example, we are currently running with an ancient svn checkout of 
CacheFu. I know, from playing on the dev system, that migrating from that 
to [a slightly patched copy of] the current version is possible, but it 
took some time and caused site errors until it was complete. I have a 
script which I believe should do the migration quickly but I want to be 
absolutely positive it is going to work without any hitches. If I can do a 
dry run in the live environment then I should be able to get that 
confidence.

___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


[Zope] Re: Questions about demostorage

2006-12-15 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Duncan Booth wrote:
 Can someone explain to me what is and is not safe to do with demostorage?
 
 Say I have a zope setup involving a zeo server and clients, and I create 
 another zope instance with demostorage wrapped around both zeoclient 
 mounts (the catalogs are mounted separately from the rest of the content).
 The main zope setup remains active while I'm using the demostorage setup, 
 so content will be added/edited or deleted in the zeo zodb.
 
 A bit of experimenting seems to indicate if I edit an object in the main 
 system, the demostorage version picks up the changes until such time as I 
 edit the same object in the demostorage instance. (So in other words, the 
 connection from the demostorage front-end to the zeo-client isn't frozen at 
 the point when the storage is opened which might have been a plausible 
 alternative).

THat is correct.  DemoStorage itself won't write to the backing layer,
but other clients (your main appservers, for instance) might.  If you
want to freeze the backing layer, then you need to use the 'stop'
parameter when creating the FileStorage.  Unfortunately, it doesn't seem
to be exposed at the ZConfig level.  More unfortunately for your case,
it is not available at all for ZEO.ClientStorage.

 What I'm a little worried about is the catalog. I'd guess that there is no 
 guarantee that the demostorage instance gets a catalog which is consistent 
 with the rest of the data (so I could easily get site errors in the 
 demostorage instance), but there should be no danger that any changes I 
 make can affect the live site and restarting the demo instance would get it 
 back into a consistent state, at least for a while. Is that a reasonable 
 interpretation?

If both storages are wrapped as DemoStorages, yes.

 What I'm hoping to achieve is a way of testing upgrades on our live system 
 in an environment as close as possible to the real live environment, but 
 without actually affecting the live system in any way. Demostorage seems to 
 offer that as an option.

I would just bite the bullet and use repozo to replicate your database
to the dev system.



Tres.
- --
===
Tres Seaver  +1 540-429-0999  [EMAIL PROTECTED]
Palladion Software   Excellence by Designhttp://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFFgroV+gerLs4ltQ4RAj+TAJ0SrqNuehRvQoNH7tEPDdqY54LDowCffVgx
ormTTUilfWeGDY42tlktFKo=
=F87X
-END PGP SIGNATURE-

___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )