On Tue, Jan 22, 2008 at 12:54:52PM -0500, Jeremy Roberts wrote: > It's true, I am looking for an easy way to move both content space objects > and software space objects between ZODB storages. > > I'd considered simply copying Data.fs files but I learned that there is a > small but possible risk of making a copy while a transaction is incomplete, > risking data integrity.
My understanding is that this is perfectly safe: ZODB will ignore the incomplete transaction record at the end of the Data.fs, so you'll get a snapshot of earlier state. If you try to make any changes, ZODB will overwrite that fragment with a new transaction record and truncate the Data.fs. (If my understanding is incorrect, I'd like someone to thwack me with a cluebat.) > I then stumbled upon the repozo.py script which can safely handle making > copies of Data.fs files while the database is in use. Unfortunately it > doesn't seemed to be included in my linux Zope 3.3.1 install. It lives in ZODB/scripts in the source tree. Or, alternatively, here: svn://svn.zope.org/repos/main/ZODB/trunk/src/ZODB/scripts/repozo.py > I am only just starting to play with eggs, but I did find that the > repozo.py script is packaged with a recent ZODB3-3.8.0b3 egg install. > > Even so, this approach (replicating entire Data.fs files) affords no > control over the objects that are transfered between storages - it's all or > nothing. This is not a deal breaker for many types of development > situations, but I anticipate finding myself in other types of situations > where it would be useful to be able to be able to transfer a few objects > here and a few objects there - even if it required some premeditation on my > part in terms of the z3 components available to me in the instances to make > it possible. I miss Zope2-style .zexp files every now and then. > My needs stem from my desire to provide the client with a dev instance, a > production instance, and a sane controlled process for moving not only file > system code (easy: darcs), but also content objects in ZODB or ZEO storage > between the instances. fssync would've solved that problem (and more -- it would let you keep your data objects in a revision control system). You could probably cobble some solution from zc.copy. > PS Is there a way to mount multiple Data.fs files in a Zope3 instance via > configuration? I've only seen it done with code in the debugger prompt. > > I have found description of how to acheive this with configuration in Z2 > (mount-point directive in zope.conf's zodb element), but not Z3... > > If I had that capability, the only problem left for me would be the > reference vs deep copy problem which occurs when using copy operations > through the ZMI between storages. zope.location.pickling.locationCopy and zc.copy solve that problem. (A hopefully unrelated accident: I used Zope 2.9 ZMI to *cut* and paste an object across storages -- from a TemporaryStorage to a FileStorage. The next day I discovered that my object kept some reference to TemporaryStorage objects that got garbage-collected in the meantime. Oops.) > I had a friend recently point me towards > the python __deepcopy__ api which I haven't looked at yet, and zc.copy. Do look at zc.copy (I just did). It works by (smartly) making a pickle that includes some objects and references other objects, and then unpickling it in the next breath. What you need to do to get Zope2-style object import/export is take that logic, verify that all objects are included in the pickle and there are no external references (other than a single __parent__ reference from the original object you're exporting, which you should convert to None before pickling) and then write that pickle to a file (or return it as the response body to a GET request). On the other system just unpickle and stick it in a container somewhere. But beware of the big can of worms you're opening: - pickles can execute arbitrary code, so be sure you only unpickle strings received from trusted sources - ZODB schema is implicit and not explicit, so make sure you're not letting your users break your system by importing objects with incompatible attributes (from earlier/later DB generations). - if you use catalogs or indexes, make sure you'll update them properly after importing a pickled bundle of objects - in general, be very wary of direct references between objects that may cross the export boundaries - beware of objects providing ILocation that aren't properly attached (i.e. have None as the __parent__) -- these will never be copied/exported In general, I'd say it's impossible to provide reliable object copying (or import/export) without knowing (or limiting) what your application stores in the DB. Marius Gedminas -- No sane person should use frame buffers if they have the choice. -- Linus Torvalds on lkml
Description: Digital signature
_______________________________________________ Zope3-users mailing list Zope3email@example.com http://mail.zope.org/mailman/listinfo/zope3-users