Re: [Zope] Versioned connectors from ZODB
Tim, . . . Versions solves this for me. Maybe like death would solve my problem with overdue taxes wink. I did get the versioned connections to work (so far), BUT, I will definately take your word on it and seek another solution :) snip Like, e.g., in the ZODB 3.2 line, otherdb = ZODB.DB(storage, cache_size=100, pool_size=2) Then connections obtained via otherdb.open() will hang if two threads already have connections from `otherdb` (that's the effect of `pool_size`), and will have ZODB memory caches that strive to keep no more than 100 objects in memory across transaction boundaries (the effect of `cache_size`). snip to double check: otherdb = ZODB.DB(existingdb._storage, cache_size=100, pool_size=2) is ok? It seems that you can create more than one DB instance that shares one storage object. I hit upon the idea of creating another DB instance and sharing the storage object myself yesterday, but wasn't sure what the repurcussions will be. Your post answers most of my questions. I have one left, though: if I do decide to share the storage object (and not go ZEO for whatever reason), will the caches between the two DB objects not get out of sync? In other words, will one DB object know to invalidate objects in it's caches should that object be changed through another DB instance? I know ZEO does this for you, but I'd like to know what the case would be for two DBs in one process. My other option is to create the connections by hand (that way I can control the cache size easily) and keep my own little pool of connections with a modified close method that does not put my connections back into the normal pool. But I'm afraid I may end up with a new can of worms that way. This answer assumes you're using ZODB directly. I don't know details of how to spell it from within a Zope application (if that's what you need -- unsure). I use the ZODB directly, but from within Zope. The connections are used in long-running processes that are not nescesarily browser-triggered. Some of them are scheduled events that are started up in their own thread. From there the need to get new connections to the ZODB. I have quite a bit of experience working safely with multiple threads and the ZODB, so I'm sure I have that part right. My problem had more to do with cache contamination and reserving special connections for specific processes. Thanks for the reply Etienne ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Versioned connectors from ZODB
ZODB versions are deprecated, unsupported, buggy and hard to use. Don't use them. Florent Etienne Labuschagne [EMAIL PROTECTED] wrote: If I get a versioned connection from the ZODB: conn = Zope.DB.open(version=myVersion) root = conn.root() app = root['Application'] # do some stuff get_transaction().commit() conn.close() Are the changes now in a version? How do I get those changes rolled into the trunk version of the ZODB? I guess all objects changed in the version will now be locked to that version until I apply the version changes to the trunk? -- Florent Guillaume, Nuxeo (Paris, France) CTO, Director of RD +33 1 40 33 71 59 http://nuxeo.com [EMAIL PROTECTED] ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Versioned connectors from ZODB
Please stay on the list. On 11 Jul 2005, at 16:19, Mark Barratt wrote: Florent Guillaume wrote: ZODB versions are deprecated, unsupported, buggy and hard to use. Don't use them. Understood. Alternative mechanisms which achieve the same object? Well that depends on your objective, and you haven't told us what you want to do from a functional point of view. Florent -- Florent Guillaume, Nuxeo (Paris, France) CTO, Director of RD +33 1 40 33 71 59 http://nuxeo.com [EMAIL PROTECTED] ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Versioned connectors from ZODB
On 7/11/05, Florent Guillaume [EMAIL PROTECTED] wrote: ZODB versions are deprecated, unsupported, buggy and hard to use. Don't use them. Florent And as I understand, so are temporary connections too. That leaves me with getting a normal ZODB connection from the pool which I don't want to do. I really need a temporary connection that I can discard. This connection can have a much smaller cache than the normal connections as it makes very little difference in the speed of data loading. Second prize is a connection that will only be used by a specific process and never used for other processes. Versions solves this for me. I can check out a connection and keep it aside only for data loading. But this means that I waste precious memory on a connection that does not really need to cache the amount of objects that the other connections should. In my case, this translates to using 1GB of RAM on one connection that gets used once a day. Please believe me that I really need a special connection. For those who really want to know why, below is an attempt at an explanation why: In the application that I have written, I want to be able to get connections that are not part of the normal connection pool. Once my process is finished, I can store these connections for later use, or discard them. Currently my application uses the normal connections in the pool. The problem is that this process contaminates the cache of the connections with objects that are not used in normal client application use (I use a thick client). This means that the client applications are extremely slow the next day and that it takes a long time before the cache contains the often used objects again. From there the reason why I DON'T want to use the connections for my once a day data loading process. My ZODB contains about 700`000 objects. A connection caches about 60`000 objects to give satisfactory client speed. To start up the client before the cache is initialized, takes about 5 minutes. Once the cache is populated, it takes a client seconds to start up. Data loading invalidates all of this, but is worse than a clean cache in that it takes long for the new objects in the cache to be flushed and replaced by the often used objects again. Data loading does not need such a big cache since it mostly loads data into the ZODB. Unfortunately, the loaded objects also end up in the cache. Why do I need so many objects in the cache? Some searches cannot be done with a mere ZCatalog search and have to run through a subset of all the objects. These tend to fit nicely in the cache. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Versioned connectors from ZODB
[Etienne Labuschagne] ... I really need a temporary connection that I can discard. This connection can have a much smaller cache than the normal connections as it makes very little difference in the speed of data loading. Second prize is a connection that will only be used by a specific process and never used for other processes. Versions solves this for me. Maybe like death would solve my problem with overdue taxes wink. Connection pools are associated with DB instances, so if you want connections with different characteristics, create another DB instance. Like, e.g., in the ZODB 3.2 line, otherdb = ZODB.DB(storage, cache_size=100, pool_size=2) Then connections obtained via otherdb.open() will hang if two threads already have connections from `otherdb` (that's the effect of `pool_size`), and will have ZODB memory caches that strive to keep no more than 100 objects in memory across transaction boundaries (the effect of `cache_size`). This is easiest if you're using ZEO (ClientStorage), because doing otherdb.close() also calls close() on the DB's storage. If you, e.g., share a FileStorage directly across multiple DBs, closing any one of the DBs will close the FileStorage across all the DBs using that FileStorage. ZEO makes it easy to open multiple ClientStorage's on top of of a single FileStorage, which can be closed independently. If you never close otherdb, this isn't an issue. This answer assumes you're using ZODB directly. I don't know details of how to spell it from within a Zope application (if that's what you need -- unsure). ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
