Re: [Zope3-Users] Copy / Move ZODB Content From Dev Zope3Instance to Production Zope3 Instance?
We do something similar but with our Zope2 instances. With Zope3 we don't have a ZODB and do all distribution using Debian packages. So how does this answer the question? He's talking about ZODB instances themselves without having to use Postgres or any other data solution other than the Data.fs itself. My recommendation is to either export all of the data that you want as object and do a mass import. Or to simply copy around the Data.fs which would probably not be the best solution but unless someone responds with a better way :-); the only solution. Thanks, Christopher Warner ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Copy / Move ZODB Content From Dev Zope3Instance to Production Zope3 Instance?
On Tue, 2008-01-22 at 18:16 +0100, Brian Sutherland wrote: On Tue, Jan 22, 2008 at 12:08:10PM -0500, Christopher Warner wrote: We do something similar but with our Zope2 instances. With Zope3 we don't have a ZODB and do all distribution using Debian packages. So how does this answer the question? He's talking about ZODB instances themselves without having to use Postgres or any other data solution other than the Data.fs itself. My recommendation is to either export all of the data that you want as object and do a mass import. Or to simply copy around the Data.fs which would probably not be the best solution but unless someone responds with a better way :-); the only solution. Perhaps you could have 2 Data.fs in your instance mounted at different points. One for code and the other for data. Then you just copy the code Data.fs around and leave the data Data.fs in place. Eventually you'd have to create data to test the code. I guess the real issue is once you're complete with all of the code if you should begin a move into production to input the data there. The only problem with this is that doing so really defeats the purpose of production. I guess it would make sense to have a stage to production sort of setup where before you hit production you have it in staging for data entry. Then from there push it into production. Thanks, Christopher Warner ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Copy / Move ZODB Content From Dev Zope3Instance to Production Zope3 Instance?
On Tue, Jan 22, 2008 at 12:08:10PM -0500, Christopher Warner wrote: We do something similar but with our Zope2 instances. With Zope3 we don't have a ZODB and do all distribution using Debian packages. So how does this answer the question? He's talking about ZODB instances themselves without having to use Postgres or any other data solution other than the Data.fs itself. My recommendation is to either export all of the data that you want as object and do a mass import. Or to simply copy around the Data.fs which would probably not be the best solution but unless someone responds with a better way :-); the only solution. Perhaps you could have 2 Data.fs in your instance mounted at different points. One for code and the other for data. Then you just copy the code Data.fs around and leave the data Data.fs in place. Thanks, Christopher Warner -- Brian Sutherland ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Copy / Move ZODB Content From Dev Zope3Instance to Production Zope3 Instance?
Christopher Warner wrote: On Tue, 2008-01-22 at 18:16 +0100, Brian Sutherland wrote: On Tue, Jan 22, 2008 at 12:08:10PM -0500, Christopher Warner wrote: We do something similar but with our Zope2 instances. With Zope3 we don't have a ZODB and do all distribution using Debian packages. So how does this answer the question? He's talking about ZODB instances themselves without having to use Postgres or any other data solution other than the Data.fs itself. My recommendation is to either export all of the data that you want as object and do a mass import. Or to simply copy around the Data.fs which would probably not be the best solution but unless someone responds with a better way :-); the only solution. Perhaps you could have 2 Data.fs in your instance mounted at different points. One for code and the other for data. Then you just copy the code Data.fs around and leave the data Data.fs in place. Eventually you'd have to create data to test the code. I guess the real issue is once you're complete with all of the code if you should begin a move into production to input the data there. The only problem with this is that doing so really defeats the purpose of production. I guess it would make sense to have a stage to production sort of setup where before you hit production you have it in staging for data entry. Then from there push it into production. Thanks, Christopher Warner Thanks for the input Chris, and Brian. It's true, I am looking for an easy way to move both content space objects and software space objects between ZODB storages. I'd considered simply copying Data.fs files but I learned that there is a small but possible risk of making a copy while a transaction is incomplete, risking data integrity. I then stumbled upon the repozo.py script which can safely handle making copies of Data.fs files while the database is in use. Unfortunately it doesn't seemed to be included in my linux Zope 3.3.1 install. I am only just starting to play with eggs, but I did find that the repozo.py script is packaged with a recent ZODB3-3.8.0b3 egg install. Even so, this approach (replicating entire Data.fs files) affords no control over the objects that are transfered between storages - it's all or nothing. This is not a deal breaker for many types of development situations, but I anticipate finding myself in other types of situations where it would be useful to be able to be able to transfer a few objects here and a few objects there - even if it required some premeditation on my part in terms of the z3 components available to me in the instances to make it possible. My needs stem from my desire to provide the client with a dev instance, a production instance, and a sane controlled process for moving not only file system code (easy: darcs), but also content objects in ZODB or ZEO storage between the instances. An illustration may, um... illustrate: App is developed on the dev instance. App works great on dev. woo! App is launched to production using Data.fs replication. New feature is requested. Development of new feature proceeds on dev instance. (Meanwhile - site users are making new content objects on production.) Client tests new feature by adding the site content objects to the dev instance ZODB that the new feature was designed to allow to be added. New feature works great on dev, new content is on dev, now it's time to launch... What now? I have new content objects on production and dev. This is all part of my desire to think responsibly about my development processes that involve my dev instance, a dev instance for the clients team of technologists, and a production instance. -Jer PS Is there a way to mount multiple Data.fs files in a Zope3 instance via configuration? I've only seen it done with code in the debugger prompt. I have found description of how to acheive this with configuration in Z2 (mount-point directive in zope.conf's zodb element), but not Z3... If I had that capability, the only problem left for me would be the reference vs deep copy problem which occurs when using copy operations through the ZMI between storages. I had a friend recently point me towards the python __deepcopy__ api which I haven't looked at yet, and zc.copy. As I know more... ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Copy / Move ZODB Content From Dev Zope3Instance to Production Zope3 Instance?
On Tue, Jan 22, 2008 at 12:54:52PM -0500, Jeremy Roberts wrote: It's true, I am looking for an easy way to move both content space objects and software space objects between ZODB storages. I'd considered simply copying Data.fs files but I learned that there is a small but possible risk of making a copy while a transaction is incomplete, risking data integrity. My understanding is that this is perfectly safe: ZODB will ignore the incomplete transaction record at the end of the Data.fs, so you'll get a snapshot of earlier state. If you try to make any changes, ZODB will overwrite that fragment with a new transaction record and truncate the Data.fs. (If my understanding is incorrect, I'd like someone to thwack me with a cluebat.) I then stumbled upon the repozo.py script which can safely handle making copies of Data.fs files while the database is in use. Unfortunately it doesn't seemed to be included in my linux Zope 3.3.1 install. It lives in ZODB/scripts in the source tree. Or, alternatively, here: svn://svn.zope.org/repos/main/ZODB/trunk/src/ZODB/scripts/repozo.py I am only just starting to play with eggs, but I did find that the repozo.py script is packaged with a recent ZODB3-3.8.0b3 egg install. Even so, this approach (replicating entire Data.fs files) affords no control over the objects that are transfered between storages - it's all or nothing. This is not a deal breaker for many types of development situations, but I anticipate finding myself in other types of situations where it would be useful to be able to be able to transfer a few objects here and a few objects there - even if it required some premeditation on my part in terms of the z3 components available to me in the instances to make it possible. I miss Zope2-style .zexp files every now and then. My needs stem from my desire to provide the client with a dev instance, a production instance, and a sane controlled process for moving not only file system code (easy: darcs), but also content objects in ZODB or ZEO storage between the instances. fssync would've solved that problem (and more -- it would let you keep your data objects in a revision control system). You could probably cobble some solution from zc.copy. PS Is there a way to mount multiple Data.fs files in a Zope3 instance via configuration? I've only seen it done with code in the debugger prompt. I have found description of how to acheive this with configuration in Z2 (mount-point directive in zope.conf's zodb element), but not Z3... If I had that capability, the only problem left for me would be the reference vs deep copy problem which occurs when using copy operations through the ZMI between storages. zope.location.pickling.locationCopy and zc.copy solve that problem. (A hopefully unrelated accident: I used Zope 2.9 ZMI to *cut* and paste an object across storages -- from a TemporaryStorage to a FileStorage. The next day I discovered that my object kept some reference to TemporaryStorage objects that got garbage-collected in the meantime. Oops.) I had a friend recently point me towards the python __deepcopy__ api which I haven't looked at yet, and zc.copy. Do look at zc.copy (I just did). It works by (smartly) making a pickle that includes some objects and references other objects, and then unpickling it in the next breath. What you need to do to get Zope2-style object import/export is take that logic, verify that all objects are included in the pickle and there are no external references (other than a single __parent__ reference from the original object you're exporting, which you should convert to None before pickling) and then write that pickle to a file (or return it as the response body to a GET request). On the other system just unpickle and stick it in a container somewhere. But beware of the big can of worms you're opening: - pickles can execute arbitrary code, so be sure you only unpickle strings received from trusted sources - ZODB schema is implicit and not explicit, so make sure you're not letting your users break your system by importing objects with incompatible attributes (from earlier/later DB generations). - if you use catalogs or indexes, make sure you'll update them properly after importing a pickled bundle of objects - in general, be very wary of direct references between objects that may cross the export boundaries - beware of objects providing ILocation that aren't properly attached (i.e. have None as the __parent__) -- these will never be copied/exported In general, I'd say it's impossible to provide reliable object copying (or import/export) without knowing (or limiting) what your application stores in the DB. Marius Gedminas -- No sane person should use frame buffers if they have the choice. -- Linus Torvalds on lkml signature.asc Description: Digital signature ___ Zope3-users mailing list
Re: [Zope3-Users] Transforming formlib field data on save
On Tue, Jan 22, 2008 at 12:28:22PM +1300, andrew wrote:
I'm struggling with how to transform some input data into an object
property using an auto-generated formlib edit form, schema, etc. I see
how the simple flow works, i.e.:
1. Create a schema
2. Build the form from the schema
3. Use applyChanges to update context object properties with input data
But what if I want to do something other than just set an object
property ? In this case, I'm working on a content page editor where the
underlying data is stored as the content of a page template, so I want
to be able to map a form content field to / from the page template
content instead of a property on the page template.
You can always write a custom action that stores the data wherever you
want.
class MyPage(form.EditForm):
form_fields = form.Fields(...)
actions = form.Actions() # remove the default Apply action
# and replace it with a custom one:
@form.action(_(Apply), condition=form.haveInputWidgets)
def apply_action(self, action, data):
self.context.pt_edit(data['content'])
self.status = _('Saved changes')
You can also combine manual actions with formlib's applyChanges, if you
want to -- just make sure you remove those fields you don't want formlib
to process from the data dict before you call applyChanges.
@form.action(_(Apply), condition=form.haveInputWidgets)
def apply_action(self, action, data):
self.context.pt_edit(data.pop('content'))
form.applyChanges(self.context, self.form_fields, data)
self.status = _('Saved changes')
I'm leaving some details (such as sending IObjectModified events, or not
doing anything if you haven't actually changed any of the values) as an
exercise for the reader. Read the zope.formlib.form source code for
examples of that.
Anyone got a simple example of how to go about this ? I'm guessing it's
something to do with providing an adapter on the content field so I can
override the get/set methods of the field, but I'm stuck at that point.
If you want to do it via adapters, here you go:
class ITemplateSource(Interface):
source = SourceText(title=uTemplate source)
class TemplateSource(object):
adapts(IMyTemplateInterface)
provides(ITemplateSource)
def __init__(self, context):
self.context = context
def _getSource(self):
return self.context.read()
def _setSource(self, new_source)
self.context.pt_edit(new_source, 'text/html')
source = property(_getSource, _setSource)
# Make sure to provideAdapter(TemplateSource) or the equivalent in
# ZCML for the following to work:
class MyPage(form.EditForm):
form_fields = form.Fields(..., ITemplateSource)
Since ITemplateSource.source is defined in the ITemplateSource
interface, formlib will try to adapt your context to ITemplateSource.
Disclaimer: all code in this email is untested.
HTH,
Marius Gedminas
--
An algorithm must be seen to be believed.
-- D.E. Knuth
signature.asc
Description: Digital signature
___
Zope3-users mailing list
Zope3-users@zope.org
http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] InvalidObjectReference error
On Mon, Jan 21, 2008 at 11:32:13AM +0100, Frank Burkhardt wrote: my zope 3 server contains a modified principal folder which is updated on a regular basis via XMLRPC. This usually works fine but after some days, the update process makes the zope server raise this error: InvalidObjectReference: Attempt to store a reference to an object from a separate connection to the same database or multidatabase Restarting the server fixes the problem for some days. I'm quite sure, this is some bug in my update code but I don't really know, what I've to look for. Does anyone have an idea? I suspect you're abusing global variables. Zope has one ZODB connection per thread. Somehow an object loaded/stored in one thread is being referenced from a different thread. The most common way to pass data between threads is to use globals (or accidental shared state, e.g. class attributes, or mutable default method arguments). Marius Gedminas -- Westheimer's Discovery: A couple of months in the laboratory can frequently save a couple of hours in the library. signature.asc Description: Digital signature ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Status of zsync
On Sat, Jan 19, 2008 at 04:46:47PM +0100, Achim Domma wrote: I'm interested in zsync, but cannot find much information via google. I'm using Zope 3.4b2 and there are docs for zsync but the tool seems not to be included!? It was never fully implemented. I guess people forgot to remove the docs when they removed the nonfunctional prototype from the source tree. I'm developing a small zope app and I want to sync my ZPT pages with the filesystem. Or is there another way to develop ZPT pages via filesystem using a text editor like vim? If you're developing a Zope 3 app, then all of your application, including page templates, should be developed on the filesystem. Use ViewPageTemplateFile for templates instead of ZPTPage. Marius Gedminas -- Some people around here wouldn't recognize subtlety if it hit them on the head. signature.asc Description: Digital signature ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] ImportError running bin/zopectl run myScript.py
On Thu, Jan 17, 2008 at 05:46:56PM +0100, Achim Domma wrote: I try to run a script on my zope instance via bin/zopectl run myScript.py. I get the following error: Traceback (most recent call last): File /Users/domma/develop/zope/demo/bin/scriptzope, line 22, in ? import paths ImportError: No module named paths I cannot find a paths.py on my system. Any hint what I'm doing wrong? I haven't seen any reply to this, so I'll bite, even though I've never used bin/zopectl run in my life. It appears that you've found a bug. There's no paths.py on my system either. The bin/debugzope script doesn't have these two lines, so I suggest you remove them: import paths paths.addPackagePaths() I've filed a bug: https://bugs.launchpad.net/zope3/+bug/185219 Marius Gedminas -- Get a life? Well, once I nearly found one, but the link was broken. signature.asc Description: Digital signature ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
[Zope3-Users] Buildout and the Data.fs
When developing and/or deploying using Buildout, how do you deal with
the Data.fs? I'm particularly concerned with running on different
environments (desktop development, server development, staging,
deployment) as I don't want the Data.fs to be under source control nor
inside the application package at all, really. In some of my playing
around, I've often blown away the local Data.fs, either by doing a
'git clean' or just by tinkering with my buildout.cfg.
I know that buildout supports extending other buildout.cfg files. Is
the best option to have a different buildout.cfg for every
environment?
As a side question then, is buildout:directory relative to a config
file, or relative to where buildout is being run? I'd like to have a
base buildout.cfg in the base, but put the others in a sub-dir.
ie:
myapp/buildout.cfg
--
...
; generates zope.conf
[zope.conf]
recipe = z3c.recipe.filetemplate
files = zope.conf
here = ${buildout:directory}
site_definition = site.zcml
zodb = ${database:zconfig}
devmode = on
myapp/etc/jeffdesktop.cfg:
---
[buildout]
extends = ../buildout.cfg
[database]
recipe = zc.recipe.filestorage
path = ${buildout:directory}/var/Data.fs
myapp/etc/deployment.cfg
---
[buildout]
extends = ../buildout.cfg
[database]
; can't remember the ZEO configs, but
; let's pretend this one sets up a ZEO client.
-
I should be able to run::
myapp buildout -c etc/jeffdesktop.cfg
And it would run relative to 'myapp', I would imagine. Anyways - is
this the best option, bad idea, better ideas? I've been basing my
latest buildout experiments on the style generated by zopeproject
(using 'paster serve', none of the zope3instance/whatsit recipes, as I
still don't really understand those and the zopeproject one seems
simpler). I've modified it by playing with some of the 'filetemplate'
recipes to try to keep any generated ZCML, zope.conf, whatever, in a
format that's easier for me to see and understand; I think I'm finally
coming up with a working buildout based setup!! Well, I've got one
working, but now I'm trying to understand it better and polish it up.
Thanks,
Jeff Shell
___
Zope3-users mailing list
Zope3-users@zope.org
http://mail.zope.org/mailman/listinfo/zope3-users
[Zope3-Users] Zope on WSGI for Deployment - best to still use proxy behind apache? Good/bad servers?
While continuing to experiment with buildout options and restructuring one of our applications to be self-contained, I started experimenting with a setup based on what zopeproject generates, which includes being able to run via 'paster serve'. I was pleasantly surprised to see Zope run on different servers, including CherryPy. Our deployments for the past many years have been behind Apache via mod_proxy. I imagine this combination would continue to work just fine with Zope running on a different server. Seeing this post about Trying on different WSGI Servers: http://blog.repoze.org/fitting_room-20071029.html got me to thinking about running on something different if we could get better performance, although my quick tests didn't produce much difference between `egg:Paste#http`, CherryPy, or ZServer. About the only thing that I saw of any interest was warnings about more threads running than there were available ZODB connections (seen on CherryPy, and I think on Paste#http too), but I didn't spend any time configuring the server settings aside from the serving-port. Is there any preferred server to use or not use? why? And ... mod_wsgi. I don't know much about it, but is it better to run through that than to go through mod_proxy? Or nginx? We had one new customer totally surprise us with their traffic/load (mostly solved, for now, due to some aggressive caching) and I'm interested in trying to find new setups. Thanks, Jeff Shell ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
