>> Indeed I want to sync the same BTree between threads. I am adding
>> persistence and acknowledgement to the Python queue.
>I don't understand what you mean by:
>- "sync": ZODB is probably not a good backend to provide synchronisation as
> feature, ie it's not a "lock server" and does what it can to avoid
> (optimistic transaction, working best on use cases where conflicts are
> unlikely by design).
With sync I referred to the sync method mentioned earlier, seeing the same
object in all the threads. The locking is handled by the Queue class mutex
in the get/put methods. This is in the context of a series of articles about
queues (urls at the end)
>- "same BTree" vs. "Python queue": I don't get the link here. A ZODB BTree
> involved in an alternate implementation of Python's "Queue" module ?
Sorry, more precisely a BTree as a way to implement a deque instead of the
collections.deque inside the Python queue (for 2.x Python Queue class). I am
modifying the _get/_put internal methods and adding acknowledgement.
>> Is that expensive in terms of performance?
> Hard to tell without knowing the implementation and (maybe) use cases.
It is a persistent queue with acknowledgment. The issue I see is the sync of
the BTree between threads. Consumers/Producers get/put elements, one at a
time, so before that operation can take place it will need to sync to have
the most updated version of the BTree in all threads.
> Some points out the top of my head which should be worth considering:
> - In one of my projects, Pure Python Queue.Queue showed (single producer,
> single consumer) bad lock contention on the producer side (!). I'm
> about 30% of execution time being spent waiting on lock on producer side,
> and 40% on consumer side, IIRC, with quite fast code executed on each
> side. This is because of Queue API providing current size in number of
> items, which requires serialises all accesses.
Is this in the context of Python 3.y? because it is a multiprocess Queue or
on Python 2.x?
> I the case of the queue element counter protected by that
> lock, using BTree.Length makes sense (warning: I didn't think it through,
> and it might very well depend on your implementation).
In the context of my article I am eliminating the size.
> - Note that ZODB's typical backend (ZODB.FileStorage) keeps a history of
> object states it ever saw. Depending on your usage, you might not need
> history to be preserved, and using an "history free" backend would serve
> better. OTOH, doing that prevents conflict resolution from working
> all interest from using BTree.Length). Packing can help, but on a large
> enough database (compared to available RAM) it might take a significant
> amount of time.
There is a way to have a "history free" storage? obviously in the context of
The context of these questions is a series of articles about queues.
Previously I've been using DyBASE but since it doesn't provide a transaction
per thread I moved to ZODB. I needed transactions per thread for using
acknowledgement but not for persistent queues with just put/get methods. The
article is available at:
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org