Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache

2006-10-09 Thread Dieter Maurer
Jim Fulton wrote at 2006-10-9 13:37 -0400:
>Dieter Maurer wrote:
>> Jim Fulton wrote at 2006-10-6 17:04 -0400:
>...
>>> Or consider object activation and deactivation. If a ghost is
>>> shared among multiple threads, then __setstate__ could
>>> be called from separate threads.
>> 
>> But, why should this be a problem? They would install the same
>> state.
>
>Because __setstate__ is written assuming that no other thread is
>calling it.

And where is the this assumption used?

  The object is already created. "__setstate__" must store
  the state given in its argument in the instance dict or
  slots.
  This looks harmless, even in a multi threaded environment.

> Threaded programming is tricky.  Code designed for a
>threaded environment has to take the possibility of of multiple
>simultaneous calls into account.  Traditionally, persistent object
>implementors haven't had to do that.

Yes.

If we really implement this idea, then we have to look
into the implementation carefully.

>>> And of course, there's the possibility that buggy software
>>> might try to mutate the objects.  How would you prevent it?
>> 
>> The special read-only connection with shared cache would raise 
>> an exception instead of joining the transaction.
>
>That's too late.

I have not looked at the "Persistent.__setattr__", but we
could easily arrange the code that this is *BEFORE* the actual
assignment. It is difficult to believe that this could be too late.

>> Of course, other threads might already have been affected
>> (e.g. in the example above of modifying non-persistent data).
>> But, bugs can have nasty effects
>
>ZODB applications count on isolation.  IMO, it isn't acceptable
>to give up on the isolation guarantee.

If part of the application is buggy, then some guarantees are no
longer available.

> 
>What if the exception is swallowed?

Then the attribute might retain its old (unchanged) value (see above).

However, swallowing essential exceptions is always dangerous.
Currently (without the feature),
you can get inconsistent persistent state -- which is
comparably nasty to multi thread problems.

> What would you do? Reload the state?

As mentioned above: perform the connection callback before the actual
assignment. Then, there will be no assignment when the connection
is unwilling to let it through.

>> This fails for:
>> 
>>   * volatile attributes -- I have no easy solution
>
>Conceivably the persistence machinery could disallow
>assignments to _v_ attrs.

But, it should do this only for some special connections...
>...
>My intuition is still that sharing objects between threads will
>introduce a host of subtle bugs.

Yes, you may be right...



-- 
Dieter
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables

2006-10-09 Thread Jim Fulton

Dieter Maurer wrote:

Jim Fulton wrote at 2006-10-9 13:22 -0400:

...
Hm, I don't why the use cases imply setting it only at the class
level, but OK.  I'll just take that as a given. So, since _p_sticky
is set at the class level, why store it on the instances?




...


use case 3, set on instance level.


I missed that. Sorry.


To make cache garbage collection as fast as possible,
I have proposed to actually store the sticky information in the
C object structure.


I don't like the idea of adding that overhead to every
persistent object just for this use case.  OTOH, we only
need a bit and I think we had a few spare bits.  So maybe
this wouldn't require any new space.


Jim

--
Jim Fulton   mailto:[EMAIL PROTECTED]   Python Powered!
CTO  (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com   http://www.zope.org
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables

2006-10-09 Thread Dieter Maurer
Jim Fulton wrote at 2006-10-9 13:22 -0400:
> ...
>Hm, I don't why the use cases imply setting it only at the class
>level, but OK.  I'll just take that as a given. So, since _p_sticky
>is set at the class level, why store it on the instances?

Use case 2, set on class level;
use case 3, set on instance level.


To make cache garbage collection as fast as possible,
I have proposed to actually store the sticky information in the
C object structure. In the garbage collection loop,
this gives an "if (...obj->p_sticky...)" instead
of

   isSticky = PyObject_GetAttr(obj, _p_sticky_name)
   if (...isSticky...) {}

   Py_XDECREF(isSticky)

This part of the proposal (storing "_p_sticky" in the the
C header of persistent objects) is not essentail.


-- 
Dieter
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache

2006-10-09 Thread Tim Peters

...

|Jim Fulton]

Or consider object activation and deactivation. If a ghost is
shared among multiple threads, then __setstate__ could
be called from separate threads.


[Dieter Maurer]

But, why should this be a problem? They would install the same
state.


__setstate__ implementations are normally never written to be
thread-safe, and can leave the state temporarily insane /during/
construction; IOW, they expect to run "atomically", as does virtually
all code that isn't deliberately & carefully written to be
thread-safe.  This goes deep; here's an example:

   http://mail.zope.org/pipermail/zodb-dev/2005-April/008747.html

   I believe I identified one cause for catastrophic thread-race
   problems here (sharing a Connection across multiple threads, or
   for any other reason sharing an in-memory persistent object across
   non-serialized threads), even when objects are never modified.  It
   doesn't look shallow to me.
   ...
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache

2006-10-09 Thread Jim Fulton

Dieter Maurer wrote:

Jim Fulton wrote at 2006-10-6 17:04 -0400:

...

Or consider object activation and deactivation. If a ghost is
shared among multiple threads, then __setstate__ could
be called from separate threads.


But, why should this be a problem? They would install the same
state.


Because __setstate__ is written assuming that no other thread is
calling it.  Threaded programming is tricky.  Code designed for a
threaded environment has to take the possibility of of multiple
simultaneous calls into account.  Traditionally, persistent object
implementors haven't had to do that.


And of course, there's the possibility that buggy software
might try to mutate the objects.  How would you prevent it?


The special read-only connection with shared cache would raise 
an exception instead of joining the transaction.


That's too late.


Of course, other threads might already have been affected
(e.g. in the example above of modifying non-persistent data).
But, bugs can have nasty effects


ZODB applications count on isolation.  IMO, it isn't acceptable
to give up on the isolation guarantee.


You couldn't wait until commit if there was the possibility that
other threads might see the errant changes.


In most cases (when an attribute of a persistent object is changed),
we could be able to detect the forbidden modification
before it happens:

  The modification goes through "Persistent.__setattr__"
  which informs the associated connection which joins the transaction.
  Thus, if the connection does not want that its objects are changed,
  it raises an exception instead of joining the transaction.


Good point. That helps.

What if the exception is swallowed?  What would you do? Reload the state?
Maybe this could be made to work. I don't know.


This fails for:

  * volatile attributes -- I have no easy solution


Conceivably the persistence machinery could disallow
assignments to _v_ attrs.

...

My intuition is still that sharing objects between threads will
introduce a host of subtle bugs.

Jim

--
Jim Fulton   mailto:[EMAIL PROTECTED]   Python Powered!
CTO  (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com   http://www.zope.org
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] We need to pay down debt

2006-10-09 Thread Dieter Maurer
Jim Fulton wrote at 2006-10-5 08:50 -0400:
>Russ Ferriday wrote:
>> I'll provide some time for this, Jim.
>> I'm no expert on the ZODB, which might be spun as an advantage, and I'm 
>> prepared to play a supporting role cleaning up doctests, or helping with 
>> doc organization. This means I don't mind gathering spippets from those 
>> who have them and pulling them together.
>
>Awesome.  Perhaps you can start thinking about how to organize the doctests.
>
>I think that there are 2 (or 3) main audiences to consider:
>
>- Users of ZODB
>
>   These folks need guidance on how to use ZODB to develop applications,
>   including pitfalls.

I do not think that doctests are the optimal means for those target group.
A guide, such as Andrew's guide, seems more appropriate in my view.



-- 
Dieter
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] We need to pay down debt

2006-10-09 Thread Dieter Maurer
Russ Ferriday wrote at 2006-10-5 13:45 +0100:
>I'll provide some time for this, Jim.
>I'm no expert on the ZODB, which might be spun as an advantage, and  
>I'm prepared to play a supporting role cleaning up doctests, or  
>helping with doc organization. This means I don't mind gathering  
>spippets from those who have them and pulling them together.

If you have questions I will try to provide answers :-)



-- 
Dieter
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] We need to pay down debt

2006-10-09 Thread Dieter Maurer
Jim Fulton wrote at 2006-10-5 08:32 -0400:
> ...
>I think this is an area where a lot of volunteers could make
>contributions.  Perhaps we could even schedule some ZODB "Doc days",
>similar bug days, but with a different emphasis.
>
>I won't insist that new work should wait for this effort

Good!

> although
>I'd like to. :)

Sad.

> Certainly, I've refrained from pursuing some
>ideas of mine in large part because I think we need some foundation work
>first.
>
>Thoughts?

I am ready to write doctests for new features I develop
(although I like unit tests much more).

I am not inclined to work on documentation primarily.



-- 
Dieter
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables

2006-10-09 Thread Jim Fulton

Dieter Maurer wrote:

Jim Fulton wrote at 2006-10-6 16:55 -0400:

...

As explained in the proposal:

  We have 3 use cases for volatile attributes:

I didn't ask why you use volatile attributes.

I asked why _p_sticky needs to be stored on each instance,
since it is set at the class level.


It is set at class level only for one of the 3 use cases -- as explained
in the original proposal *AND* in the repetition of the 3 use cases


Hm, I don't why the use cases imply setting it only at the class
level, but OK.  I'll just take that as a given. So, since _p_sticky
is set at the class level, why store it on the instances?

Jim

--
Jim Fulton   mailto:[EMAIL PROTECTED]   Python Powered!
CTO  (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com   http://www.zope.org
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache

2006-10-09 Thread Dieter Maurer
Jim Fulton wrote at 2006-10-6 17:04 -0400:
> ...
>> Thread-safety is not an issue for read-only data.
>
>Yes it is.  An object that is read-only from a persistence
>point of view isn't read-only at the Python level.

Okay: thread-safety is not an issue for (truely) read-only data.
It is potentially an issue for only partially read-only data
such as read-only only from a persistence point of view.

I agree that we have spoken about a read-only storage such
that they may be thread-safety issues in this context.

> Objects
>often have other data.  Take volatile data as an example.

In fact, they are the only example (if we disregard bugs
modifying non-persistent attributes without informing
the containing persistent object).
But a single example may be enough.

>Or consider object activation and deactivation. If a ghost is
>shared among multiple threads, then __setstate__ could
>be called from separate threads.

But, why should this be a problem? They would install the same
state.

>And of course, there's the possibility that buggy software
>might try to mutate the objects.  How would you prevent it?

The special read-only connection with shared cache would raise 
an exception instead of joining the transaction.

Of course, other threads might already have been affected
(e.g. in the example above of modifying non-persistent data).
But, bugs can have nasty effects

>You couldn't wait until commit if there was the possibility that
>other threads might see the errant changes.

In most cases (when an attribute of a persistent object is changed),
we could be able to detect the forbidden modification
before it happens:

  The modification goes through "Persistent.__setattr__"
  which informs the associated connection which joins the transaction.
  Thus, if the connection does not want that its objects are changed,
  it raises an exception instead of joining the transaction.

This fails for:

  * volatile attributes -- I have no easy solution

  * modification of mutable non-persistent data

This would work, too, provided that

 obj._p_changed = True

where called *before* the modification.

Calling it before the modification is in principle
necessary (the get the modification thrown away when
the transaction is aborted) but unfortunately, almost
all legacy code does it after the modification.

But, this simple is a bug -- and bugs can have
non-local effects.



-- 
Dieter
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables

2006-10-09 Thread Dieter Maurer
Jim Fulton wrote at 2006-10-6 16:55 -0400:
> ...
>> As explained in the proposal:
>> 
>>   We have 3 use cases for volatile attributes:
>
>I didn't ask why you use volatile attributes.
>
>I asked why _p_sticky needs to be stored on each instance,
>since it is set at the class level.

It is set at class level only for one of the 3 use cases -- as explained
in the original proposal *AND* in the repetition of the 3 use cases

>...
>
>>> - This isn't unique to _v_ attributes.  An object could
>>>   have other non-persistent data that is precious.
>> 
>> No: only "_v_" attributes are lost at deactivation
>
>No, normally all state is lost at deactivation, including _v_
>attributes.  Of course, most other state can be reloaded
>from the database.

Thus, it is in some way persistent, isn't it?



-- 
Dieter
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


[ZODB-Dev] Re: Deadlock in ZODB/DB.py?

2006-10-09 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Christian Zagrodnick wrote:
> Hi there,
> 
> the DB.open method makes me wonder how it works
> (http://svn.zope.org/ZODB/trunk/src/ZODB/DB.py?rev=69551&view=auto)
> 
> It locks with self._a() and calls self._connectionMap which locks too. I
> cannot imagine a way this could possibly work.
> 
> I guess the lock should be released before the _connectionMap call?
> 
> Interestingly both all the tests pass and ZODB works...

Note the following in the __init__()::

# Allocate lock.
x = threading.RLock()
self._a = x.acquire
self._r = x.release

The lock is a "reentrant" lock[1], which means that subsequent locks
from within the same thread do not deadlock;  instead, they increment
the lock count.

[1] http://docs.python.org/lib/module-threading.html#l2h-3418


Tres.
- --
===
Tres Seaver  +1 202-558-7113  [EMAIL PROTECTED]
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFFKm1r+gerLs4ltQ4RAkmjAKCQFc0CwZeWRmffajADcHqBpsY9qQCguZ12
1dFN7jpri+CtgfiGR86zH7w=
=uhXI
-END PGP SIGNATURE-

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Deadlock in ZODB/DB.py?

2006-10-09 Thread Tim Peters

[Christian Zagrodnick]

the DB.open method makes me wonder how it works (http://svn.zope.org/
ZODB/trunk/src/ZODB/DB.py?rev=69551&view=auto)

It locks with self._a() and calls self._connectionMap which locks
too. I cannot imagine a way this could possibly work.


Look at __init__():

   x = threading.RLock()
   self._a = x.acquire
   self._r = x.release

That is, it's a reentrant lock.  See the Python docs for RLock().


I guess the lock should be released before the _connectionMap call?


No.


Interestingly both all the tests pass and ZODB works...


That was a good clue ;-)
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables

2006-10-09 Thread Jim Fulton

Chris Withers wrote:

Jim Fulton wrote:

- I wonder if an argument could be made than we shouldn't
  implicitly deactivate an object that has been accessed in a
  a transaction while the transaction is still running.


Would this prevent ZODB from ever promising not to use more than a 
certain amount of memory?


Yes, and non-necessarily.  :)

The only way that ZODB could keep such a promise would be to
disallow loading new objects, generating errors under some circumstances.

Certainly, as Dieter pointed out, this would make it harder to
remove objects from memory.

The biggest zodb-related performance problems I've seen are when a 
scripter writes code that drags way more objects into memory than any 
sane script should. The creates a HUGE python process which never 
releases the memory back to the os (I believe that may be fixed in 
Python 2.5?) which causes all kinds of performance problems...


It should be possible to prevent scripters shooting themselves in the foot!


Sorry, that's impossible.

Jim

--
Jim Fulton   mailto:[EMAIL PROTECTED]   Python Powered!
CTO  (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com   http://www.zope.org
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables

2006-10-09 Thread Chris Withers

Jim Fulton wrote:

- I wonder if an argument could be made than we shouldn't
  implicitly deactivate an object that has been accessed in a
  a transaction while the transaction is still running.


Would this prevent ZODB from ever promising not to use more than a 
certain amount of memory?


The biggest zodb-related performance problems I've seen are when a 
scripter writes code that drags way more objects into memory than any 
sane script should. The creates a HUGE python process which never 
releases the memory back to the os (I believe that may be fixed in 
Python 2.5?) which causes all kinds of performance problems...


It should be possible to prevent scripters shooting themselves in the foot!

cheers,

Chris

--
Simplistix - Content Management, Zope & Python Consulting
   - http://www.simplistix.co.uk

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


[ZODB-Dev] Deadlock in ZODB/DB.py?

2006-10-09 Thread Christian Zagrodnick

Hi there,

the DB.open method makes me wonder how it works (http://svn.zope.org/ 
ZODB/trunk/src/ZODB/DB.py?rev=69551&view=auto)


It locks with self._a() and calls self._connectionMap which locks  
too. I cannot imagine a way this could possibly work.


I guess the lock should be released before the _connectionMap call?

Interestingly both all the tests pass and ZODB works...


--
Christian Zagrodnick

gocept gmbh & co. kg  ·  forsterstrasse 29 · 06112 halle/saale
www.gocept.com · fon. +49 345 12298894 · fax. +49 345 12298891






smime.p7s
Description: S/MIME cryptographic signature
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev