>> Indeed I want to sync the same BTree between threads. I am adding >> persistence and acknowledgement to the Python queue. >I don't understand what you mean by: >- "sync": ZODB is probably not a good backend to provide synchronisation as a > feature, ie it's not a "lock server" and does what it can to avoid locking > (optimistic transaction, working best on use cases where conflicts are > unlikely by design). With sync I referred to the sync method mentioned earlier, seeing the same object in all the threads. The locking is handled by the Queue class mutex in the get/put methods. This is in the context of a series of articles about queues (urls at the end)
>- "same BTree" vs. "Python queue": I don't get the link here. A ZODB BTree > involved in an alternate implementation of Python's "Queue" module ? > Sorry, more precisely a BTree as a way to implement a deque instead of the collections.deque inside the Python queue (for 2.x Python Queue class). I am modifying the _get/_put internal methods and adding acknowledgement. >> Is that expensive in terms of performance? > Hard to tell without knowing the implementation and (maybe) use cases. It is a persistent queue with acknowledgment. The issue I see is the sync of the BTree between threads. Consumers/Producers get/put elements, one at a time, so before that operation can take place it will need to sync to have the most updated version of the BTree in all threads. > Some points out the top of my head which should be worth considering: > - In one of my projects, Pure Python Queue.Queue showed (single producer, > single consumer) bad lock contention on the producer side (!). I'm talking > about 30% of execution time being spent waiting on lock on producer side, > and 40% on consumer side, IIRC, with quite fast code executed on each > side. This is because of Queue API providing current size in number of > items, which requires serialises all accesses. Is this in the context of Python 3.y? because it is a multiprocess Queue or on Python 2.x? > I the case of the queue element counter protected by that producer-hurting > lock, using BTree.Length makes sense (warning: I didn't think it through, > and it might very well depend on your implementation). In the context of my article I am eliminating the size. > - Note that ZODB's typical backend (ZODB.FileStorage) keeps a history of all > object states it ever saw. Depending on your usage, you might not need that > history to be preserved, and using an "history free" backend would serve you > better. OTOH, doing that prevents conflict resolution from working (removing > all interest from using BTree.Length). Packing can help, but on a large > enough database (compared to available RAM) it might take a significant > amount of time. There is a way to have a "history free" storage? obviously in the context of ZODB. The context of these questions is a series of articles about queues. Previously I've been using DyBASE but since it doesn't provide a transaction per thread I moved to ZODB. I needed transactions per thread for using acknowledgement but not for persistent queues with just put/get methods. The article is available at: http://blog.databigbang.com/persisting-native-python-queues/ Thanks, Sebastian _______________________________________________ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
