>> Indeed I want to sync the same BTree between threads. I am adding 
>> persistence and acknowledgement to the Python queue.
>I don't understand what you mean by:
>- "sync": ZODB is probably not a good backend to provide synchronisation as
>  feature, ie it's not a "lock server" and does what it can to avoid
>  (optimistic transaction, working best on use cases where conflicts are
>  unlikely by design).
With sync I referred to the sync method mentioned earlier, seeing the same
object in all the threads. The locking is handled by the Queue class mutex
in the get/put methods. This is in the context of a series of articles about
queues (urls at the end)

>- "same BTree" vs. "Python queue": I don't get the link here. A ZODB BTree
>  involved in an alternate implementation of Python's "Queue" module ?
Sorry, more precisely a BTree as a way to implement a deque instead of the
collections.deque inside the Python queue (for 2.x Python Queue class). I am
modifying the _get/_put internal methods and adding acknowledgement.

>> Is that expensive in terms of performance?
> Hard to tell without knowing the implementation and (maybe) use cases.
It is a persistent queue with acknowledgment. The issue I see is the sync of
the BTree between threads. Consumers/Producers get/put elements, one at a
time, so before that operation can take place it will need to sync to have
the most updated version of the BTree in all threads.

> Some points out the top of my head which should be worth considering:
> - In one of my projects, Pure Python Queue.Queue showed (single producer,
>  single consumer) bad lock contention on the producer side (!). I'm
>  about 30% of execution time being spent waiting on lock on producer side,
>  and 40% on consumer side, IIRC, with quite fast code executed on each
>  side. This is because of Queue API providing current size in number of
>  items, which requires serialises all accesses.
Is this in the context of Python 3.y? because it is a multiprocess Queue or
on Python 2.x?

>  I the case of the queue element counter protected by that
>  lock, using BTree.Length makes sense (warning: I didn't think it through,
>  and it might very well depend on your implementation).
In the context of my article I am eliminating the size.

> - Note that ZODB's typical backend (ZODB.FileStorage) keeps a history of
>  object states it ever saw. Depending on your usage, you might not need
>  history to be preserved, and using an "history free" backend would serve
>  better. OTOH, doing that prevents conflict resolution from working
>  all interest from using BTree.Length). Packing can help, but on a large
>  enough database (compared to available RAM) it might take a significant
>  amount of time.
There is a way to have a "history free" storage? obviously in the context of

The context of these questions is a series of articles about queues.
Previously I've been using DyBASE but since it doesn't provide a transaction
per thread I moved to ZODB. I needed transactions per thread for using
acknowledgement but not for persistent queues with just put/get methods. The
article is available at:


For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org

Reply via email to