Re: [ZODB-Dev] Threads, ZODB, and how to install ZODB without distutils

2007-02-13 Thread Gary Poster


On Feb 11, 2007, at 7:29 PM, Manuel Vazquez Acosta wrote:

My scenario is akin a consumer-producer with shared buffer.  
Consumers pull items from the buffer whilst producers put items in  
the buffer. The buffer is an OOBTree along with an IOBTree which  
gives serial numbers to the keys of the OOBTree.


I'm not sure if this is a match, but if you follow Chris' suggestions  
(which I recommend generally) then you might want to look at  
zc.queue: http://cheeseshop.python.org/pypi/zc.queue/ .


Gary

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Threads, ZODB, and how to install ZODB without distutils

2007-02-13 Thread Manuel Vazquez Acosta

Hi Gary,

Although the shared buffer indeed has a queue-like interface there's a 
catch for my needs. I need that pull method to retrieve the item in the 
head and to move the head forward, and not to remove the item from the 
buffer. Just like zc.queue I need a single copy of a item in the queue, 
but I also need this uniqueness over the time, so that any item which 
has being consumed already, does not get back to buffer.


Thanks for your advice, I haven't seen zc.queue, so you comments are 
really an improvement to my knowledge. Besides I'm going to take a peek 
at zc.queue and compare it with my own implementation.


Thanks and best regards,
Manuel.


Gary Poster wrote:


On Feb 11, 2007, at 7:29 PM, Manuel Vazquez Acosta wrote:

My scenario is akin a consumer-producer with shared buffer. Consumers 
pull items from the buffer whilst producers put items in the buffer. 
The buffer is an OOBTree along with an IOBTree which gives serial 
numbers to the keys of the OOBTree.


I'm not sure if this is a match, but if you follow Chris' suggestions 
(which I recommend generally) then you might want to look at zc.queue: 
http://cheeseshop.python.org/pypi/zc.queue/ .


Gary



___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Threads, ZODB, and how to install ZODB without distutils

2007-02-13 Thread Gary Poster


On Feb 12, 2007, at 10:01 AM, Manuel Vazquez Acosta wrote:


Hi Gary,

Although the shared buffer indeed has a queue-like interface  
there's a catch for my needs. I need that pull method to retrieve  
the item in the head and to move the head forward, and not to  
remove the item from the buffer. Just like zc.queue I need a single  
copy of a item in the queue, but I also need this uniqueness over  
the time, so that any item which has being consumed already, does  
not get back to buffer.


Ah, OK.

FWIW, you could get the same functionality with a wrapper around a  
zc.queue pretty easily, of course, with at least a couple of variants  
off the top of my head (both involving an additional data structure  
that you maintain, and supplying your own `put` and `pull` that wrap  
the zc.queue methods, either with composition or inheritance).


If, instead, you move to a multi-threaded model following the same  
approach you outlined before (sequential int keys, IIUC) you'll need  
to be careful of write conflict errors.


Gary
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Threads, ZODB, and how to install ZODB without distutils

2007-02-13 Thread Marius Gedminas
On Sun, Feb 11, 2007 at 07:29:58PM -0500, Manuel Vazquez Acosta wrote:
 The other question is about compiling ZODB without using the 
 out-of-the-box distutils installation procedure. I'm also playing with 
 Zope and Plone, so I have several instances on the same machine. I think 
 installing with distutils may cause conflicts with the Zope instances. 
 Am I right? If so, then how should I install ZODB side-by-side the 
 consumer-producer application?

This sounds like a job for zc.buildout.

Marius Gedminas
-- 
Special bonus feature: absolutely nowhere in RetchMail's code is there an
arbitrary 3-second sleep(). Wow! What other mail retriever can say that? (Hint:
not fetchmail.)
-- http://open.nit.ca/wiki/index.php?page=RetchMail


signature.asc
Description: Digital signature
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Threads, ZODB, and how to install ZODB without distutils

2007-02-13 Thread Dieter Maurer
Manuel Vazquez Acosta wrote at 2007-2-11 19:29 -0500:
 ...
I have read that each thread should have its own connection to the DB. 
However, in my case, each thread should be aware of what is actually in 
the DB at all times. So I wonder if I can shared the connection between 
those threads as long as I take the means to protect it (i.e RLock).

You might be able to do that -- but you would loose almost
all benefits of the ZODB: an earlier discussion with Jim and Jeremy
indicated that not only writes would need such locking but reads as
well

I think, you should forget about using the same ZODB connection
in different threads...



-- 
Dieter
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Threads, ZODB, and how to install ZODB without distutils

2007-02-11 Thread Chris McDonough

On Feb 11, 2007, at 7:29 PM, Manuel Vazquez Acosta wrote:


Hi all,

I'm writing a small application for which I want to use ZODB's  
persistence machinery. I have a couple of question, though.


I have read that each thread should have its own connection to the  
DB. However, in my case, each thread should be aware of what is  
actually in the DB at all times. So I wonder if I can shared the  
connection between those threads as long as I take the means to  
protect it (i.e RLock).


I am not certain, actually.  The mechanism for doing this is in  
there.  The interfaces.py file states:


Synchronization
---

A Connection instance is not thread-safe.  It is designed to
support a thread model where each thread has its own transaction.
If an application has more than one thread that uses the
connection or the transaction the connection is registered with,
the application should provide locking.

And it goes on to talk about the sync=False parameter vaguely.

But the story doesnt really end there... there is also the capability  
to register a different transaction manager at DB open time.. the  
default one assumes one transaction per thread.


I'd need to dig through all this code to reunderstand it, but it  
looks possible.




My scenario is akin a consumer-producer with shared buffer.  
Consumers pull items from the buffer whilst producers put items in  
the buffer. The buffer is an OOBTree along with an IOBTree which  
gives serial numbers to the keys of the OOBTree.


The typical way to do this using one-connection-per-thread and a  
threaded txn manager would be to call transaction.commit() when  
mutating the shared data structures.  To detect changes to the  
objects provided by a connection (due to other threads committing  
during the connection's transaction), you can connection.sync().   
Zope calls transaction.begin() at the beginning of a request and  
either transaction.abort() (on failure) or transaction.commit() (on  
success) at its termination.  It never needs to do an explict  
connection.sync() because it never wants to do a dirty read (each  
request has its own connection and sees its connection data as  
canonical for the duration of its existence).  Zope handles conflicts  
by retrying the request.  But Zope has it easy because the web has  
natural transaction boundaries that don't always exist in GUI apps or  
other apps with a different request/response cycle.




The other question is about compiling ZODB without using the out-of- 
the-box distutils installation procedure. I'm also playing with  
Zope and Plone, so I have several instances on the same machine. I  
think installing with distutils may cause conflicts with the Zope  
instances. Am I right? If so, then how should I install ZODB side- 
by-side the consumer-producer application?


The easiest thing to do here is to install a separate Python instance  
for each project.  Alternately if you're on UNIX, you can use  
virtual python (http://peak.telecommunity.com/dist/virtual- 
python.py) which makes a tree of symlinks that acts like a new Python  
instance but has its own site packages directory in which distutils  
stuff is installed.


If you're on Windows, I believe Zope/Plone ship with their own copy  
of Python on that platform, so installing to the system python  
shouldnt create a conflict.


- C

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Threads, ZODB, and how to install ZODB without distutils

2007-02-11 Thread Manuel Vazquez Acosta

Chris,

Thanks for your response.  I will try the locking first and the see if I 
can implement a transaction manager of my own.


I will read about virtualizing Python, I'm a debian user :))

Thanks and best regards,
Manuel.


Chris McDonough wrote:

On Feb 11, 2007, at 7:29 PM, Manuel Vazquez Acosta wrote:


Hi all,

I'm writing a small application for which I want to use ZODB's 
persistence machinery. I have a couple of question, though.


I have read that each thread should have its own connection to the 
DB. However, in my case, each thread should be aware of what is 
actually in the DB at all times. So I wonder if I can shared the 
connection between those threads as long as I take the means to 
protect it (i.e RLock).


I am not certain, actually.  The mechanism for doing this is in 
there.  The interfaces.py file states:


Synchronization
---

A Connection instance is not thread-safe.  It is designed to
support a thread model where each thread has its own transaction.
If an application has more than one thread that uses the
connection or the transaction the connection is registered with,
the application should provide locking.

And it goes on to talk about the sync=False parameter vaguely.

But the story doesnt really end there... there is also the capability 
to register a different transaction manager at DB open time.. the 
default one assumes one transaction per thread.


I'd need to dig through all this code to reunderstand it, but it looks 
possible.




My scenario is akin a consumer-producer with shared buffer. Consumers 
pull items from the buffer whilst producers put items in the buffer. 
The buffer is an OOBTree along with an IOBTree which gives serial 
numbers to the keys of the OOBTree.


The typical way to do this using one-connection-per-thread and a 
threaded txn manager would be to call transaction.commit() when 
mutating the shared data structures.  To detect changes to the objects 
provided by a connection (due to other threads committing during the 
connection's transaction), you can connection.sync().  Zope calls 
transaction.begin() at the beginning of a request and either 
transaction.abort() (on failure) or transaction.commit() (on success) 
at its termination.  It never needs to do an explict connection.sync() 
because it never wants to do a dirty read (each request has its own 
connection and sees its connection data as canonical for the duration 
of its existence).  Zope handles conflicts by retrying the request.  
But Zope has it easy because the web has natural transaction 
boundaries that don't always exist in GUI apps or other apps with a 
different request/response cycle.




The other question is about compiling ZODB without using the 
out-of-the-box distutils installation procedure. I'm also playing 
with Zope and Plone, so I have several instances on the same machine. 
I think installing with distutils may cause conflicts with the Zope 
instances. Am I right? If so, then how should I install ZODB 
side-by-side the consumer-producer application?


The easiest thing to do here is to install a separate Python instance 
for each project.  Alternately if you're on UNIX, you can use virtual 
python (http://peak.telecommunity.com/dist/virtual-python.py) which 
makes a tree of symlinks that acts like a new Python instance but has 
its own site packages directory in which distutils stuff is installed.


If you're on Windows, I believe Zope/Plone ship with their own copy of 
Python on that platform, so installing to the system python shouldnt 
create a conflict.


- C




___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev