[ZODB-Dev] Re: Invalidating caches for module refresh doesn't seem to work
Jim Fulton wrote: On Jan 29, 2007, at 5:46 PM, Philipp von Weitershausen wrote: After refreshing a product, Zope 2 uses the following stanza in App.RefreshFuncs.autoRefresh() to let the ZODB know that it should invalidate its pickle caches: ... refresh products from ZODB import Connection Connection.resetCaches() transaction.commit() jar._resetCache() transaction.begin() That is really weird code. It looks very brittle. That way, persistent objects will use the newly reimported classes instead of continuing to use the old, no longer current classes. I suppose that was the intent. For grok I tried to implement the same mechanism and copied that code almost verbatimly, only to find out it doesn't seem to work. Persistent objects will still be instances of the classes that should have been thrown away during the re-import of modules. Testing product refresh with both Zope 2.9 and 2.10 produces errors for me while Zope 2.8 works, which leads me to the assumption that a cache invalidation bug was introduced after ZODB 3.4. Is anybody else seeing this? The way that connections were managed changes quite a bit in 3.4. I'm a bit surprised nobody else has complained about this so far... Maybe people finally stopped using product-refresh because it doesn't work reliably. What do I have to do to make it work reliably? I'd like to avoid having to restart all of Zope. (Note that this is grok on Zope3, not necessarily Zope 2). Help with tracking this down would be greatly appreciated. Shane originally wrote this. Maybe he could help out. I'm not interested myself. In fact, I'l be happy to see this code get ripped out. If it stays in it needs a doctest to prevent future regression and to explain how to use it, including what it's semantics and limitations are. So, are there any alternatives that let me refresh modules w/o server restarts AND make the ZODB aware of this? Would closing and reopening the database altogether be an option? I would assume it would leave the connections that other threads have to the database in a funny state, but I don't understand a whole lot about ZODB internals so I'm just guessing. -- http://worldcookery.com -- Professional Zope documentation and training Next Zope 3 training at Camp5: http://trizpug.org/boot-camp/camp5 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Invalidating caches for module refresh doesn't seem to work
After refreshing a product, Zope 2 uses the following stanza in App.RefreshFuncs.autoRefresh() to let the ZODB know that it should invalidate its pickle caches: ... refresh products from ZODB import Connection Connection.resetCaches() transaction.commit() jar._resetCache() transaction.begin() That way, persistent objects will use the newly reimported classes instead of continuing to use the old, no longer current classes. For grok I tried to implement the same mechanism and copied that code almost verbatimly, only to find out it doesn't seem to work. Persistent objects will still be instances of the classes that should have been thrown away during the re-import of modules. Testing product refresh with both Zope 2.9 and 2.10 produces errors for me while Zope 2.8 works, which leads me to the assumption that a cache invalidation bug was introduced after ZODB 3.4. Is anybody else seeing this? I'm a bit surprised nobody else has complained about this so far... Help with tracking this down would be greatly appreciated. -- http://worldcookery.com -- Professional Zope documentation and training Next Zope 3 training at Camp5: http://trizpug.org/boot-camp/camp5 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: 64-bit BTrees
Fred Drake wrote: I have a need for 64-bit BTrees (at least for IOBTree and OIBTree), and I'm not the first. I've created a feature development branch for this, and checked in my initial implementation. I've modified the existing code to use PY_LONG_LONG instead of int for the key and/or value type; there's no longer a 32-bit version in the modified code. Any Python int or long that can fit in 64 bits is accepted; ValueError is raised for values that require 65 bits (or more). Keys and values that can be reported as Python ints are, and longs are only returned when the value cannot be converted to a Python int. This can have a substantial effect on memory consumption, since keys and/or values now take twice the space. There may be performance issues as well, but those have not been tested. There are new unit tests, but more are likely needed. If you're interested in getting the code from Subversion, it's available at: svn://svn.zope.org/repos/main/ZODB/branches/fdrake-64bits/ Ideally, this or some variation on this could be folded back into the main development for ZODB. If this is objectionable, making 64-bit btrees available would require introducing new versions of the btrees (possibly named LLBTree, LOBTree, and OLBTree). -1 to the L*BTree flavours. The 'long' type is disappearing from Python anyways, why should the implementation detail worry us regarding BTrees? +1 to making the I*BTree flavours 64bit-aware. If Jim is right, existing pickles will be read back just fine and as long as the integers stay under 32 bit, they won't even be bigger. The only real implication therefore is the pushing around twice as many bits in memory. Dieter estimates 20% to 35% slowdown for the C algorithms (whatever that means), Tim seems to think it won't have such a big effect. I guess we'll only know after some benchmarks. Philipp ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
