Re: [ZODB-Dev] Twisted, ZODB and zc.twist
On Jul 29, 2011, at 4:38 AM, Erik Allik wrote: Thanks for another clear answer! I'll simply ignore zc.twist (for now at least) and set up a classic thread pool. I'll still probably use zc.twist's source as an example of how to write transaction retrial on ConflicErrors and ClientDisconnecteds as I haven't found anything similar elsewhere. Cool. FWIW, http://pypi.python.org/pypi/zc.queue does something similar without threads. I prefer it (despite the fact that I said that this testing approach has known weaknesses) if your use cases allow it. Gary Erik 2011/7/29 Gary Poster gary.pos...@gmail.com: On Jul 28, 2011, at 6:38 PM, Erik Allik wrote: Gary, since you're the author of zc.twist (as I understand), I take this as an exhaustive answer, and thank you :) However, I'm still curious to know zc.twist would provide any benefit when used together with a thread pool. The thread examples in its README (which I find hard to understand) indicate that there must be some meaning to using zc.twist with threads. The threads in the README are there to let me explore a running reactor within the doctest. They are not anything to emulate. The README overstates zc.twist's usefulness. As you point out, it blocks. If you assume that the ZODB is functioning normally, it won't block much but with standard definitions of blocking, it fails. For zc.async, that's fine--but a while True loop and a time.sleep() to give some breathing room probably would have been fine, with hindsight, and much simpler. The use of Twisted is a very questionable convenience, and the way zc.async uses Twisted is not to be copied 99% or more of the time. Gary Thanks again and best regards, Erik 2011/7/29 Gary Poster gary.pos...@gmail.com: Hello Erik. As the author, I'll suggest that you move on to a ZODB connection pool used by the usual Twisted-provided thread pool. zc.twist is maybe interesting and maybe clever, but of very limited usefulness. I suspect it is only used in zc.async, and that's probably as it should be. It's a case of premature generalization. If you are going to use Twisted in anything like a normal way, and the ZODB in anything like a normal way, use a connection pool and threads. Best, Gary On Jul 28, 2011, at 4:12 PM, Erik Allik wrote: Hello everyone, I hope this is the right mailing list to post this question to. I'm trying to build an Twisted application that uses ZODB and I've also found zc.twist. I've read its README as well as the source code to try to understand how exactly and why I should use zc.twist, and also asked the same questions on irc.freenode.net #zope but without a conclusive answer. What I'm not able to understand is whether using zc.twist will allow me to do all ZODB access in the main Twisted thread, or if still need to set up a thread pool and deferToThread any code that accesses the DB. The README of zc.twist has: Everything can be done within the main thread, so it can be full-bore Twisted usage, without threads. However, looking at the source code of zc.twist, I cannot find any reference to async communication (callbacks/polling) with ZODB, so it's difficult to conclude that all DB access is non-blocking and can be done in a single thread. Also, there are some examples with threads in the README. I would highly appreciate if somebody with a clearer understanding of this could explain to me whether I in fact need to set up a thread pool, and in any case, what exactly is zc.twist for regardless of whether it needs to be used with a threadpool or not. Thanks in advance! Regards, Erik Allik P.S. I'd be very thankful for any references to open source projects/code using ZODB with Twisted, or even zc.twist (besides zc.async). ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Twisted, ZODB and zc.twist
Hello Erik. As the author, I'll suggest that you move on to a ZODB connection pool used by the usual Twisted-provided thread pool. zc.twist is maybe interesting and maybe clever, but of very limited usefulness. I suspect it is only used in zc.async, and that's probably as it should be. It's a case of premature generalization. If you are going to use Twisted in anything like a normal way, and the ZODB in anything like a normal way, use a connection pool and threads. Best, Gary On Jul 28, 2011, at 4:12 PM, Erik Allik wrote: Hello everyone, I hope this is the right mailing list to post this question to. I'm trying to build an Twisted application that uses ZODB and I've also found zc.twist. I've read its README as well as the source code to try to understand how exactly and why I should use zc.twist, and also asked the same questions on irc.freenode.net #zope but without a conclusive answer. What I'm not able to understand is whether using zc.twist will allow me to do all ZODB access in the main Twisted thread, or if still need to set up a thread pool and deferToThread any code that accesses the DB. The README of zc.twist has: Everything can be done within the main thread, so it can be full-bore Twisted usage, without threads. However, looking at the source code of zc.twist, I cannot find any reference to async communication (callbacks/polling) with ZODB, so it's difficult to conclude that all DB access is non-blocking and can be done in a single thread. Also, there are some examples with threads in the README. I would highly appreciate if somebody with a clearer understanding of this could explain to me whether I in fact need to set up a thread pool, and in any case, what exactly is zc.twist for regardless of whether it needs to be used with a threadpool or not. Thanks in advance! Regards, Erik Allik P.S. I'd be very thankful for any references to open source projects/code using ZODB with Twisted, or even zc.twist (besides zc.async). ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Twisted, ZODB and zc.twist
On Jul 28, 2011, at 6:38 PM, Erik Allik wrote: Gary, since you're the author of zc.twist (as I understand), I take this as an exhaustive answer, and thank you :) However, I'm still curious to know zc.twist would provide any benefit when used together with a thread pool. The thread examples in its README (which I find hard to understand) indicate that there must be some meaning to using zc.twist with threads. The threads in the README are there to let me explore a running reactor within the doctest. They are not anything to emulate. The README overstates zc.twist's usefulness. As you point out, it blocks. If you assume that the ZODB is functioning normally, it won't block much but with standard definitions of blocking, it fails. For zc.async, that's fine--but a while True loop and a time.sleep() to give some breathing room probably would have been fine, with hindsight, and much simpler. The use of Twisted is a very questionable convenience, and the way zc.async uses Twisted is not to be copied 99% or more of the time. Gary Thanks again and best regards, Erik 2011/7/29 Gary Poster gary.pos...@gmail.com: Hello Erik. As the author, I'll suggest that you move on to a ZODB connection pool used by the usual Twisted-provided thread pool. zc.twist is maybe interesting and maybe clever, but of very limited usefulness. I suspect it is only used in zc.async, and that's probably as it should be. It's a case of premature generalization. If you are going to use Twisted in anything like a normal way, and the ZODB in anything like a normal way, use a connection pool and threads. Best, Gary On Jul 28, 2011, at 4:12 PM, Erik Allik wrote: Hello everyone, I hope this is the right mailing list to post this question to. I'm trying to build an Twisted application that uses ZODB and I've also found zc.twist. I've read its README as well as the source code to try to understand how exactly and why I should use zc.twist, and also asked the same questions on irc.freenode.net #zope but without a conclusive answer. What I'm not able to understand is whether using zc.twist will allow me to do all ZODB access in the main Twisted thread, or if still need to set up a thread pool and deferToThread any code that accesses the DB. The README of zc.twist has: Everything can be done within the main thread, so it can be full-bore Twisted usage, without threads. However, looking at the source code of zc.twist, I cannot find any reference to async communication (callbacks/polling) with ZODB, so it's difficult to conclude that all DB access is non-blocking and can be done in a single thread. Also, there are some examples with threads in the README. I would highly appreciate if somebody with a clearer understanding of this could explain to me whether I in fact need to set up a thread pool, and in any case, what exactly is zc.twist for regardless of whether it needs to be used with a threadpool or not. Thanks in advance! Regards, Erik Allik P.S. I'd be very thankful for any references to open source projects/code using ZODB with Twisted, or even zc.twist (besides zc.async). ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] big list (was Re: Making ZODB / ZEO faster)
On Dec 7, 2009, at 11:08 AM, Erik Dahl wrote: ... Our slow loading object was a persistent with a regular list inside of the main pickle. Objects that the list pointed to were persistent which I believe means that hey will load separately. In general we have tried to make our persistent objects reasonably large to lower the amount of load round trips. I haven't actually checked its size yet but it will be interesting to see. Even if the list holds just persistent objects, if there are a lot of them, just loading the references can still be a big problem. That would point to refactoring that part of the app, and/or investigating if the characteristics of things like zc.queue or zc.blist would help. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Ruby/Smalltalk OODB
Uh-oh, I'm implicated! (see below) On Jun 3, 2008, at 12:53 PM, Sean Allen wrote: ... If you do that in gemstone, there is only one copy of Order B, no matter what variable in what dictionary you come at it from. And its drop dead simple. I looked at implementing that with zodb and moved along. I'm confused. This has been the way the ZODB worked for a long time, unless I'm really missing something in your description. i tried to do this: create customer that has order so that i can have different extents type situations... store customer in one dictionary. store order in another. if i pulled the order back out from the order dictionary and modified it then pulled the customer out, the customers order was no longer in sync with what came out of the order dictionary. the reference was lost on serialization. original in memory objects were fine, those that came back out from zodb werent. i'm going to quote the initial email i sent with the idea in general and the followup i got and i then tried it out to make sure i hadnt asked the question wrong, and yeah... what i wanted to do, wasnt easily done. the quotes: The biggest concern I have is how do to the layout/storage so that this slightly contrived example works: Product has a brand. There are many brands. How do I store so that I can find all products simply and all brands simply and also so that changes in a brand instance are reflected when the product instance is deserialized. By 'simply' I mean that it doesnt really work on our end to have to walk all Products looking for unique brands. Should just be able to go directly in and get said brands ( using keys() or similar call ). If I create 'brand' and 'product' as btrees, then if i do something like some_product.brand.name = 'something entirely different' and that brand already exists in 'brand', would it be updated? are references maintained in that fashion? do we have to handle manually on update and creation? Note that we would just be using ZODB not Zope in this scenario. Back references are not maintained automatically. I'd identify two classic solutions to this sort of thing. One is to make a custom mapping (using a BTree as the inner data structure) that maintains back-references when objects are placed in them or removed. zope.app(.container? .folder? I'd have to look) has code that does this, along with firing events. For simple stories like the one you describe here, that's what I'd probably recommend. It works to the strengths of the ZODB, which particularly shines in terms of readability when you just need to walk a tree of attributes to get what you want. The other is to keep an external index, a la zc.extrinsicreference or zc.relation. zc.extrinsicreference does not have too many dependencies beyond ZODB, and as long as zope.app.keyreference doesn't drag much along with it, might be usable as a library. That said, it's also very simple, and could be used as a model for you, even if you don't use it directly. It would also be a reasonable choice for a simple situation like the one you describe. It relies on events to update its data structures. zc.relation an almost-released-revision of zc.relationship that drastically reduces dependencies--actually, it has no additional dependencies to ZODB, as you can see at http://svn.zope.org/zc.relation/trunk/setup.py?view=markup . It's also a bit overwhelming and low-level: see the README:http://svn.zope.org/zc.relation/trunk/src/zc/relation/README.txt?view=auto . It doesn't hook anything up for you: you set the relationship catalog up and you arrange for it to be updated, via events or direct messages. That said, if you need its power, it is well- tested and would be a good choice for some jobs from at least some perspectives (caveat read-or: I'm the author). Now in the context of this discussion, I see that I misread you. I apologize. This works out of the box: You have a Product class and a Brand class. Both inherit from persistent.Persistent. You have a persistent-aware mapping such as a BTree. In the root of your ZODB, put two BTrees, one for your products and one for your brands. Create some Brand instances and put them in the brand mapping. Create some Product instances and put them in the product mapping. Assign some of the brands you have made as attributes of the products. Now, product.brand.foo = 'bar' (in any thread or any ZEO client) will be changing the same effective object (let's call it a record) as the one in the brand mapping you have in the root of your db. I believe that this is what you are talking about. Again, I apologize for not reading your original question closely enough. What you *can't* do out of the box is ask hey, what products have an attribute that points to this brand?. That's a back-reference, and that needs solutions like the ones to which I was
Re: [ZODB-Dev] Re: so your coming from the rdbms world tutorial?
On Mar 20, 2008, at 11:04 PM, Sean Allen wrote: I've been playing with this and I'm still having a disconnect 3 week later. 1. I love the zc.relation stuff it answers one area of stuff I hadn't even gotten to yet. cool :-) 2. I'm just totally missing something less cool. :-/ If I have Customers and Orders and I want to be able update all of them independently of their relationships ( so that if an order is updated, when i get the customer at some later time, it has the updated order amongst that relation ) but I can't figure it out. I tried looking at the zope.app.folder stuff but I keep getting lost in the zope aspects of it and am having a hard time seeing the forest for the trees. Is there some more general ready on the patterns used that you know of? I think if I understood the idea more in an abstract sense, I could get a lot more out of the folder implementation. Here's a simple, dumb example that parallels the folder stuff. See the Dict class in http://svn.zope.org/zc.async/branches/dev/src/zc/async/utils.py?rev=84657view=auto . You put something in the dict, and __setitem__ slams a name and a parent. The other mutating methods should do the right thing as well in terms of updating the back reference. So, completely randomly and arbitrarily, but to try and make a parallel, what if customers were a dict of orders, and when you made an order it was associated with only one customer, and you put the order in the customer. This may be bizarre--what's the key? can more than one customer be associated with an order? But that would mean that customer.values() would get all of the customer's orders, and order.parent would be the associated customer. This example subclasses zc.dict, a super simple package that only depends on ZODB, btw. It does not have full dict behavior, as Jim likes to point out, because items are stored by BTree sorting, not hashes, but it looks like a dict otherwise. Maybe it can help you out. If you want to try out this example, copy the code out and get the zc.dict egg and give it a try. Anyway, this pattern of directly manipulating back-references is good for intrinsic relationships like customers and orders. Hope that helps a bit, Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Anyone mind looking over a bit of code?
Hi Kenneth. First, make sure you do a transaction.begin() (or transaction.abort()) before you view your database for a given operation. This will sync invalidations for a given transaction, even if it will only be a read transaction, and reset the MVCC view. The partial-view behavior you describe sounds like it might have been possible pre-MVCC, but seems pretty unlikely now, unless you are committing after each individual insertion rather than after the aggregated insertion. ...Actually, looking through your code in newData, that appears to be what you are doing: if the object is iterable, you recursively call newData on the sub-elements, with a commit for each insertion. In that case, it would be reasonable for a concurrent reader to get part of the insertions while the writer continues to write. Because of MVCC, the reader will not see the subsequent insertions until it makes another transaction commit/abort/begin. There are some various other oddities to my eye--why do a ``while`` on the BTree insertion when you don't mutate the key?--but they don't look like the cause of the behavior you describe. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Zeo Server as a single point of failure
On Mar 10, 2008, at 6:34 PM, Kenneth Miller wrote: Hello all, I'm currently developing my application ontop of zodb/zeo, and the idea of having the Zeo server the only single point of failure still bothers me. My question is, has anyone thought of any solutions for this. My idea was to have a few or more background processes that kept multiple servers synced and concurrent. Any thoughts? - Zope Corp sells ZRS (http://www.zope.com/products/zope_replication_services.html ) - Shane Hathaway's RelStorage (http://wiki.zope.org/ZODB/RelStorage) promises to accomplish some of the same sort of thing with (for instance) Postgres tools though the page I listed says that this is untested. - Christian Theune is working on ZEO RAID: http://pypi.python.org/pypi/gocept.zeoraid/dev Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: so your coming from the rdbms world tutorial?
On Feb 27, 2008, at 10:03 AM, Sean Allen wrote: On Feb 27, 2008, at 5:33 AM, Laurence Rowe wrote: Sean Allen wrote: been looking for anything along those lines. in particular, strategies and gotchas for how to store objects. everything i've found is basically just a single type of object being stored. i'm really interested in tutorials and information on the best ways to setup large complicated hierarchies and all the various gotchas etc. anything like that exist? A good jumping off point is the wiki: http://wiki.zope.org/ZODB/Documentation The beauty of using the ZODB is that it really makes no difference whether you are storing homogeneous or heterogeneous hierarchies of objects. If you are likely to have a large number of objects in any one folder then make sure it is BTree based. This is a scalable data structure, so the whole list does not have to be loaded into memory to access a single child object. Been over that. Still have serious questions. And yes, we have a large number of objects, something in the area of 16 million at present if you were to map each relational table row to an object. The biggest concern I have is how do to the layout/storage so that this slightly contrived example works: Product has a brand. There are many brands. How do I store so that I can find all products simply and all brands simply and also so that changes in a brand instance are reflected when the product instance is deserialized. By 'simply' I mean that it doesnt really work on our end to have to walk all Products looking for unique brands. Should just be able to go directly in and get said brands ( using keys() or similar call ). If I create 'brand' and 'product' as btrees, then if i do something like some_product.brand.name = 'something entirely different' and that brand already exists in 'brand', would it be updated? are references maintained in that fashion? do we have to handle manually on update and creation? Note that we would just be using ZODB not Zope in this scenario. Back references are not maintained automatically. I'd identify two classic solutions to this sort of thing. One is to make a custom mapping (using a BTree as the inner data structure) that maintains back-references when objects are placed in them or removed. zope.app(.container? .folder? I'd have to look) has code that does this, along with firing events. For simple stories like the one you describe here, that's what I'd probably recommend. It works to the strengths of the ZODB, which particularly shines in terms of readability when you just need to walk a tree of attributes to get what you want. The other is to keep an external index, a la zc.extrinsicreference or zc.relation. zc.extrinsicreference does not have too many dependencies beyond ZODB, and as long as zope.app.keyreference doesn't drag much along with it, might be usable as a library. That said, it's also very simple, and could be used as a model for you, even if you don't use it directly. It would also be a reasonable choice for a simple situation like the one you describe. It relies on events to update its data structures. zc.relation an almost-released-revision of zc.relationship that drastically reduces dependencies--actually, it has no additional dependencies to ZODB, as you can see at http://svn.zope.org/zc.relation/trunk/setup.py?view=markup . It's also a bit overwhelming and low-level: see the README: http://svn.zope.org/zc.relation/trunk/src/zc/relation/README.txt?view=auto . It doesn't hook anything up for you: you set the relationship catalog up and you arrange for it to be updated, via events or direct messages. That said, if you need its power, it is well-tested and would be a good choice for some jobs from at least some perspectives (caveat read-or: I'm the author). HTH Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
On Nov 1, 2007, at 4:25 PM, Matt Hamilton wrote: David Binger dbinger at mems-exchange.org writes: On Nov 1, 2007, at 7:05 AM, Matt Hamilton wrote: Ie we perhaps look at a catalog data structure in which writes are initially done to some kind of queue then moved to the BTrees at a later point. A suggestion: use a pair of BTrees, one with a high branching factor (bucket size) and one with a very low branching factor. Force all writes into the tree with little buckets. Make every search look in both trees. Consolidate occasionally. An interesting idea. Surely we need the opposite though, and that is an additional BTree with a very large bucket size, as we want to minimize the chance of a bucket split when inserting? Then we occasionally consolidate and move the items in the original BTree with the regular bucket size/ branch factor. maybe. haven't thought it through, but worth thinking about. idle thought I should probably not share: you could use a Bucket directly for that--it will never split at all, and has the conflict resolution behavior. (strangely, I'm not idle at all, but rather overwhelmingly busy ;-) ) Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Memory Errors reading large ZODB
On Aug 2, 2007, at 9:33 AM, Jim Fulton wrote: On Aug 2, 2007, at 7:51 AM, Chris Withers wrote: Gary Poster wrote: you can call cache minimize after a threshold.. maybe every 100 iterations. sounds good, assuming you know you are not writing. I've used this trick loads, especially for huge datastructure migrations where writing is happening. I wonder why I haven't bumped into problems? I'm not sure what Gary was referring to. I don't think there are any problems with writing. OTOH, modified objects won't be removed from memory. Perhaps that was what he was referring to. Yup. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problems with ZODB3-3.9.0_dev_r77011
On Jul 19, 2007, at 7:11 AM, Tobias Rodäbel wrote: Hi, zope.app.keyreference-3.5.0_dev_r77018-py2.4.egg requires ZODB3=3.9.0-dev-r77011 But there might be a caching problem within ZODB3-3.9.0 dev r77011, my debug session tells: /development/Zope3/MyProject/eggs/ZODB3-3.9.0_dev_r77011-py2.4- macosx-10.4-ppc.egg/ZODB/Connection.py(644)_store_objects() - raise (Pdb) obj zope.app.file.image.Image object at 0x3faadb0 (Pdb) oid '\x00\x00\x00\x00\x00\x00\x00\xc6' (Pdb) self._cache[oid] *** KeyError: '\x00\x00\x00\x00\x00\x00\x00\xc6' (Pdb) self._cache[oid] = obj *** TypeError: Cache values must be persistent objects. But zope.app.file.image.Image should be persistent. (Pdb) dir (obj) ['__annotations__', '__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__', '__getstate__', '__hash__', '__implemented__', '__init__', '__module__', '__new__', '__providedBy__', '__provides__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__slotnames__', '__str__', '__weakref__', '_data', '_getData', '_height', '_p_activate', '_p_changed', '_p_deactivate', '_p_delattr', '_p_getattr', '_p_invalidate', '_p_jar', '_p_mtime', '_p_oid', '_p_serial', '_p_setattr', '_p_state', '_setData', '_size', '_width', 'contentType', 'data', 'getImageSize', 'getSize'] Looking forward to some hints or help, Hi Tobias. The ZODB 3.9 dev version is only different from 3.8 in some conflict resolution code, for which I am responsible. Some thoughts. - I haven't seen any errors like this yet. That's just a data point, and certainly does not necessarily invalidate your report. - Is this consistently reproduceable, or intermittent? Unless you are intentionally creating a conflict in a test, any errors in the changes in 3.9 would be more likely to be intermittent. - Even better, can you construct a small, distributable test case? That would certainly invite more help. - Have you tried to reproduce with the most recent zope.app.keyreference in the 3.4 line and the most recent ZODB 3.8 line? If so, that might get Jim's attention, and would rule out the relatively small changes in the 3.9 dev egg. Unless you like riding the bleeding edge, I might suggest using those earlier versions for now anyway. Gary___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Failing tests of ZODB on Linux 64-bit
On May 9, 2007, at 1:35 PM, Christian Theune wrote: Hey, my new buildbot just discovered this: Failure in test test32 (BTrees.tests.testBTrees.FamilyTest) Traceback (most recent call last): File /usr/lib64/python2.4/unittest.py, line 260, in run testMethod() File /data/buildbot/slave/ZODB/build/src/BTrees/tests/ testBTrees.py, line 1712, in test32 self.assert_(BTrees.family32.maxint + 1 not in s) File /usr/lib64/python2.4/unittest.py, line 309, in failUnless if not expr: raise self.failureException, msg AssertionError That's mine. I'll try to get my 64 bit machine working again-- upgrade issues have killed it ATM. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Failing tests of ZODB on Linux 64-bit
On May 9, 2007, at 1:40 PM, Gary Poster wrote: On May 9, 2007, at 1:35 PM, Christian Theune wrote: Hey, my new buildbot just discovered this: Failure in test test32 (BTrees.tests.testBTrees.FamilyTest) Traceback (most recent call last): File /usr/lib64/python2.4/unittest.py, line 260, in run testMethod() File /data/buildbot/slave/ZODB/build/src/BTrees/tests/ testBTrees.py, line 1712, in test32 self.assert_(BTrees.family32.maxint + 1 not in s) File /usr/lib64/python2.4/unittest.py, line 309, in failUnless if not expr: raise self.failureException, msg AssertionError That's mine. I'll try to get my 64 bit machine working again-- upgrade issues have killed it ATM. 64 bit machine was alive enough to repro and fix, at least. I checked in the fix (just a test change) to SVN (r75659). I have not released a ZODB egg because I don't think you need one for buildbot and Jim says he will be releasing one with some more important changes soon. Thanks Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] DisconnectedError
Hi. ZODB does not currently work with Py2.5. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: another reason to stop supporting versions
I assume you are more interested in seeing if you can get someone to raise a compelling -1 than in hearing a chorus of +1s. But FWIW, +1 in getting rid of versions. It sounds like it would simplify the code significantly, which would be its own feature. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Threads, ZODB, and how to install ZODB without distutils
On Feb 11, 2007, at 7:29 PM, Manuel Vazquez Acosta wrote: My scenario is akin a consumer-producer with shared buffer. Consumers pull items from the buffer whilst producers put items in the buffer. The buffer is an OOBTree along with an IOBTree which gives serial numbers to the keys of the OOBTree. I'm not sure if this is a match, but if you follow Chris' suggestions (which I recommend generally) then you might want to look at zc.queue: http://cheeseshop.python.org/pypi/zc.queue/ . Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Threads, ZODB, and how to install ZODB without distutils
On Feb 12, 2007, at 10:01 AM, Manuel Vazquez Acosta wrote: Hi Gary, Although the shared buffer indeed has a queue-like interface there's a catch for my needs. I need that pull method to retrieve the item in the head and to move the head forward, and not to remove the item from the buffer. Just like zc.queue I need a single copy of a item in the queue, but I also need this uniqueness over the time, so that any item which has being consumed already, does not get back to buffer. Ah, OK. FWIW, you could get the same functionality with a wrapper around a zc.queue pretty easily, of course, with at least a couple of variants off the top of my head (both involving an additional data structure that you maintain, and supplying your own `put` and `pull` that wrap the zc.queue methods, either with composition or inheritance). If, instead, you move to a multi-threaded model following the same approach you outlined before (sequential int keys, IIUC) you'll need to be careful of write conflict errors. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] We need to pay down debt
On Oct 5, 2006, at 9:37 AM, Jim Fulton wrote: Russ Ferriday wrote: Do you have some favorite documents that you would point me to as most accurate or up to date? No, because I haven't used ZODB documentation much myself. :) Decent clues will be enough to get started. - Doc in test or source files - web pages, - forums chats, - email threads The ZODB wiki which is here, in case you didn't know http://www.zope.org/Wikis/ZODB/FrontPage (or Google for ZODB wiki - I'm feeling lucky ;-) has some pointers In particular, I usually point people here (which is a link from the wiki): http://www.zope.org/Wikis/ZODB/FrontPage/guide/index.html Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Threads and Connections
On Jul 26, 2006, at 12:52 PM, Chris McDonough wrote: FWIW I believe by default at least, open ZODB connections are tied to the thread which did the opening (they are returned to a pool when closed and reused, possibly in another thread). And indeed each connection does have a cache; caching is one of the primary responsibilities of a connection object. See the documentation in ZODB/interfaces.py for IConnection. I *think* what is happening here is that you are committing the transaction devoted to the current connection/thread, and trying to close a connection that has pending changes from another thread. So when you do transaction.abort(), it's aborting the transaction involving the current thread, not the one associated with the other connection. That said, I'm not entirely sure what to tell you to do here; there used to be an API named setLocalTransaction or somesuch that allowed you to control the one-thread-per-connection policy wrt to a transaction minimally, IIRC. This API has disappeared, probably replaced with something more flexible, but I'm not sure what that is. I suspect it may have something to do with the transaction_manager parameter to DB.open() however. Open the database with an explicit transaction_manager if you want to manage these yourself, instead of using the thread default. If you have a connection, get the transaction_manager off of the connection. Can't find it in the interface, unfortunately, but I believe it to be reliable. All that said, Chris S, it sounds like you might be attacking a solved problem--and *might* be adding some serious unnecessary complexity. Maybe you ought to take a step back and see if anyone has a general approach to what you want? For instance, you said The reason why I'm doing this is because I'm trying to update the classes of persistent objects loaded into memory. Want to elaborate on this a bit? It sounds like stuff other folks have done, but you might want to give more detail. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: getting the object ID (_p_oid attribute)
On Jun 26, 2006, at 3:16 PM, Florent Guillaume wrote: On 26 Jun 2006, at 21:11, Robert Gravina wrote: On 2006/06/27, at 3:49, Benji York wrote: Robert Gravina wrote: I just tried loading a persisted object interactively and noticed that although the _p_oid doesn't print out as anything (and hence I always thought it was empty in my debugging prints), it isn't actually None! Can anyone explain this? (here p is my persisted object) p._p_oid '\x00\x00\x00\x00\x00\x00\x00\x08' print p._p_oid p._p_oid is None False What would you expect to see if you printed out seven null characters and a backspace? Hahaha - that's a good point! I was expecting IDs to look, well, something like asdf23asdf. Well, anyway thankyou! I seemed to have solved this problem. I was able to write a __eq__ function like this: def __eq__(self, other): if isinstance(other, name of my class): if hasattr(other,_p_oid) and other._p_oid != None and (other._p_oid == self._p_oid): return True else: return False and now can compare objects for equality after the (Twisted) client edits them and sends them back. You can shorten that to def __eq__(self, other): return aq_base(self) is aq_base(other) Heh, good point. And you can ditch the aq_base if you don't use acquisition-based classes. (or don't have proxies generally--Zope 3 has a number of them, for instance.) (this wouldn't work if you had the same persistent object from different connections...but don't go there!) Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Zope-Annce] Technical Preview of Blob support in ZODB
On Apr 29, 2006, at 6:08 AM, Christian Theune wrote: Hi everybody, the ZODB team is proud to announce a technical preview of a Blob implementation for ZODB. Hey. This seems like there's no way that it will make it into the next ZODB release, right? I assume no one is using it in production, on the basis of the branch TODO. Does anyone have a hard need to get this production-ready (address the TODOs) by a certain date? Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] edge case question about _p_deactiveate and modified objects
On Mar 24, 2006, at 9:11 AM, Chris Withers wrote: Hi All, What happens if you call _p_deactivate() on a modified object but before the modifications have been committed? Do the modifications get lost or will they still be saved when the next commit happens after the _p_deactivate()? Under normal operation, the _p_deactivate call is ignored on a modified object, so the changes are not lost (but _p_deactivate accomplished nothing). However, there is currently a bug that if you call _p_deactivate on a new persistent object that has a database connection then the whole thing will just be lost. I have a TODO to make a small test case of this and put it in the collector. AIUI from Jim, fixing this reliably will require some API changes. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: Possible contributions to ZODB
On Feb 3, 2006, at 9:12 AM, Florent Guillaume wrote: Gary Poster wrote: On Jan 26, 2006, at 10:16 PM, Gary Poster wrote: I have some code that might be worth putting into ZODB. There are three bits. - Two flavors of persistent queue. [...] I'm a bit surprised there was so little discussion: ZC has needed a persistent queue several times in the past. Whether the queue would be based on what I proposed or on something else someone else has, it seems like it would be a good thing to have. However, with only Julien replying I don't think that's enough to push it further. I'll keep it in zasync until/unless some better option comes around. Like Julien I'd like to encourage you to put it in the core. I'd say, in the persistent module. OK: I'll push this a bit more then. Thanks to you and Julien for your encouragement, and for Julien's offer of help. If you want unit testing without ZODB dependency, I think you should just synthesize states by hand, and test the conflict resolution method. True, that would work; and since I agree that the queue belongs in persistent, that's probably what I should do. Because in fact in you current unit tests, you're testing much more, including the ZODB mechanisms that detect concurrently modified objects in different connections and regenerate old state from the database, etc. It's not strictly needed (and is a lot of setup). True. I still like what I did, because I think it gives a fuller story for someone reading the doctest (and I don't mind the fact that the test relies on ZODB, despite classic unit test advice), but it can't go in persistent, so I think the point is moot. I'll push this farther along. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Possible contributions to ZODB
I have some code that might be worth putting into ZODB. There are three bits. - Two flavors of persistent queue. Both have a conflict resolution policy to be about as lenient as I could make it. They don't expect to ever hold multiple instances of equivalent values at the same time. One flavor puts all of the contents in a single data structure, while the other divides it up into smaller persistent objects, a la BTree/Buckets. These might go in the persistent package, but tests require ZODB, so I'm not sure. See http://svn.zope.org/Sandbox/poster/zope.zasync/queue.py?view=auto and http://svn.zope.org/Sandbox/poster/zope.zasync/queue.txt?view=auto - Simple, somewhat inefficient approach to pickling methods. I'm not sure where this might go; maybe the persistent package again? See http://svn.zope.org/Sandbox/poster/zope.zasync/methodpickle.py?view=auto and http://svn.zope.org/Sandbox/poster/zope.zasync/methodpickle.txt? view=auto - Hacked version of MappingStorage (barely) appropriate for testing conflict resolution code. It might go in ZODB/tests/utils. I'd need something like it to include the persistent queue tests. See http://svn.zope.org/Sandbox/poster/zope.zasync/tests.py?view=auto Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: [Zope] DateTime mess
On Dec 1, 2005, at 11:38 AM, Tim Peters wrote: ... I know Gary Poster gave a lot of thought to making pickles for the timezone info in Zope3 efficient too. For some definition of a lot of thought. :-) The pickle for pytz.utc is now relatively small (though still adds a non-trivial percentage addition--30%ish?--to a naive datetime IIRC). That's as far as that bit goes. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: [Zope] DateTime mess
On Dec 1, 2005, at 12:04 PM, Tim Peters wrote: Note that we have yet to use a new strategy for shrinking pickle sizes: a few years ago Python's pickle code grew support for extension codes, a registry of class/type names that _can_ be referenced by short (as short as 2 bytes) new pickle codes, instead of embedding the module and class name into every pickle, over and over again. I don't recall the exact numbers numbers, but some years ago Jeremy analyzed a customer Data.fs, and discovered that at least half of it consisted of repetitions of the string BTrees.OOBTree.OOBTree ;-) That's the kind of thing the extension code pickle mechanism was intended to address; it's a simple and cheap compression gimmick, but so far unused. Yes, I remembered this, and just refreshed my memory. This is the last mention I see in the archives as to ZODB use of protocol 2 (i.e., it doesn't, and prior to Py 2.3.4 it couldn't). http://mail.zope.org/pipermail/zodb-dev/2004-December/008259.html Is that still accurate--that is, does ZODB still not use protocol 2? Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Ordering before commit hooks
On Aug 5, 2005, at 6:20 AM, Julien Anguenot wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, I would like to order the before commit hooks on a transaction by being able to specify an order for a subscriber at registration time. For the moment, they are called in the order they've been registred. I like this idea, and was just idly wishing for it this morning. The order argument could be an integer with a default value to 0 added to the registration method. I usually like the word 'priority' for this sort of value, rather than 'order'. To me 'order' implies unique value--i.e., with 'order' I am mildly surprised to learn that two entries may have the same order. I am not surprised that two entries may share a priority. If negative values are acceptable then a default value of 0 is OK by me. We could just insert the subscriber wihin the queue according to this order parameter if found *or* sorting the queue according to this parameter before flushing the queue and calling the hooks. I think it would be ideal if before commit hooks with the same priority continue to be in the order in which they were added. There are a number of ways to accomplish this and I expect you can choose a very reasonable one :-) Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Ordering before commit hooks
On Aug 5, 2005, at 2:33 PM, Tim Peters wrote: [Julien Anguenot] ... The order argument could be an integer with a default value to 0 added to the registration method. [Gary Poster] I usually like the word 'priority' for this sort of value, rather than 'order'. To me 'order' implies unique value--i.e., with 'order' I am mildly surprised to learn that two entries may have the same order. I am not surprised that two entries may share a priority. It's wholly unclear to me, given a callback with associated value 0 and another with associated value 999, which one is supposed to be invoked first. If the parameter is called priority, then I'd expect 999 to win; if it's called order, then 0. To each his own, I suppose--'first priority' and 'last priority' is my model, in which, at the least, a priority of 1 would be first priority and a priority of 999 is a reasonable last priority (acknowledging a positive infinity as something that would take the honor of being reliably last priority, then). A priority of 0, and a negative priority, start to muddle the model a bit, certainly. But anyway, if you don't see 'priority' in that way, then yes, it is perhaps unclear in a different dimension than 'order' is unclear. :-) Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Reloading product in 2.8
On Jul 12, 2005, at 1:46 PM, Jens Vagelpohl wrote: On 12 Jul 2005, at 18:17, Lennart Regebro wrote: previous Zope version, which I used, was 2.6.1, but it was very old version and I decided to jump to 2.8.0. I took problems with autorefresh. What can I do to solve my problems? If it creates problems then simply don't use it. That's not a very good answer. I disagree. I agree with Jens that it is unlikely to ever work correctly 100% of the time. I agree with Lennart that leaving it not working is not a good answer. I think it should be removed as a feature. Making every developer using ZODB eventually learn not to trust the feature seems unnecessarily painful. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
