Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables

2006-10-11 Thread Dieter Maurer
Jim Fulton wrote at 2006-10-9 15:04 -0400:
 ...
 To make cache garbage collection as fast as possible,
 I have proposed to actually store the sticky information in the
 C object structure.

I don't like the idea of adding that overhead to every
persistent object just for this use case.  OTOH, we only
need a bit and I think we had a few spare bits.  So maybe
this wouldn't require any new space.

That's how I implemented it in our local Zope copy.

#define cPersistent_HEAD \
PyObject_HEAD \
PyObject *jar; \
PyObject *oid; \
PerCache *cache; \
CPersistentRing ring; \
char serial[8]; \
signed char state; \
/* DM 2005-08-22: sticky support */ \
/* unsigned char reserved[3]; */ \
unsigned char sticky; /* we need only a bit; but byte access is cheaper */\
unsigned char reserved[2];

As you can see, I used one of the three reserved chars -- and did not
increase the size of persistent objects.


-- 
Dieter
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables

2006-10-10 Thread Chris Withers

Jim Fulton wrote:

Chris Withers wrote:

Jim Fulton wrote:

- I wonder if an argument could be made than we shouldn't
  implicitly deactivate an object that has been accessed in a
  a transaction while the transaction is still running.


Would this prevent ZODB from ever promising not to use more than a 
certain amount of memory?


Yes, and non-necessarily.  :)


Don't follow what that means...


The only way that ZODB could keep such a promise would be to
disallow loading new objects, generating errors under some circumstances.


...or to bump objects out of the cache mid-way through a transaction, 
generating errors if there are no non-modified objects that can be 
unloaded, hence my interest in the _v_ and _p_sticky discussion...


The biggest zodb-related performance problems I've seen are when a 
scripter writes code that drags way more objects into memory than any 
sane script should. The creates a HUGE python process which never 
releases the memory back to the os (I believe that may be fixed in 
Python 2.5?) which causes all kinds of performance problems...


It should be possible to prevent scripters shooting themselves in the 
foot!


Sorry, that's impossible.


Well, okay, but the behaviour in this area is pretty crummy right now. 
We have an object cache that doesn't know anything about object size, 
and which is also happy to balloon to using all the memory on a machine, 
sometimes to the point where all you can do is get the machine hosters 
to hit the big red button :-(


I'm sure things _can_ be better than this, I just wish I could help...

Chris

--
Simplistix - Content Management, Zope  Python Consulting
   - http://www.simplistix.co.uk
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables

2006-10-09 Thread Chris Withers

Jim Fulton wrote:

- I wonder if an argument could be made than we shouldn't
  implicitly deactivate an object that has been accessed in a
  a transaction while the transaction is still running.


Would this prevent ZODB from ever promising not to use more than a 
certain amount of memory?


The biggest zodb-related performance problems I've seen are when a 
scripter writes code that drags way more objects into memory than any 
sane script should. The creates a HUGE python process which never 
releases the memory back to the os (I believe that may be fixed in 
Python 2.5?) which causes all kinds of performance problems...


It should be possible to prevent scripters shooting themselves in the foot!

cheers,

Chris

--
Simplistix - Content Management, Zope  Python Consulting
   - http://www.simplistix.co.uk

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables

2006-10-09 Thread Jim Fulton

Chris Withers wrote:

Jim Fulton wrote:

- I wonder if an argument could be made than we shouldn't
  implicitly deactivate an object that has been accessed in a
  a transaction while the transaction is still running.


Would this prevent ZODB from ever promising not to use more than a 
certain amount of memory?


Yes, and non-necessarily.  :)

The only way that ZODB could keep such a promise would be to
disallow loading new objects, generating errors under some circumstances.

Certainly, as Dieter pointed out, this would make it harder to
remove objects from memory.

The biggest zodb-related performance problems I've seen are when a 
scripter writes code that drags way more objects into memory than any 
sane script should. The creates a HUGE python process which never 
releases the memory back to the os (I believe that may be fixed in 
Python 2.5?) which causes all kinds of performance problems...


It should be possible to prevent scripters shooting themselves in the foot!


Sorry, that's impossible.

Jim

--
Jim Fulton   mailto:[EMAIL PROTECTED]   Python Powered!
CTO  (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com   http://www.zope.org
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables

2006-10-09 Thread Jim Fulton

Dieter Maurer wrote:

Jim Fulton wrote at 2006-10-6 16:55 -0400:

...

As explained in the proposal:

  We have 3 use cases for volatile attributes:

I didn't ask why you use volatile attributes.

I asked why _p_sticky needs to be stored on each instance,
since it is set at the class level.


It is set at class level only for one of the 3 use cases -- as explained
in the original proposal *AND* in the repetition of the 3 use cases


Hm, I don't why the use cases imply setting it only at the class
level, but OK.  I'll just take that as a given. So, since _p_sticky
is set at the class level, why store it on the instances?

Jim

--
Jim Fulton   mailto:[EMAIL PROTECTED]   Python Powered!
CTO  (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com   http://www.zope.org
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables

2006-10-09 Thread Dieter Maurer
Jim Fulton wrote at 2006-10-9 13:22 -0400:
 ...
Hm, I don't why the use cases imply setting it only at the class
level, but OK.  I'll just take that as a given. So, since _p_sticky
is set at the class level, why store it on the instances?

Use case 2, set on class level;
use case 3, set on instance level.


To make cache garbage collection as fast as possible,
I have proposed to actually store the sticky information in the
C object structure. In the garbage collection loop,
this gives an if (...obj-p_sticky...) instead
of

   isSticky = PyObject_GetAttr(obj, _p_sticky_name)
   if (...isSticky...) {}

   Py_XDECREF(isSticky)

This part of the proposal (storing _p_sticky in the the
C header of persistent objects) is not essentail.


-- 
Dieter
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables

2006-10-06 Thread Jim Fulton

Dieter Maurer wrote:

I have put an enhancement proposal at

  http://www.zope.org/Wikis/ZODB/VolatileAttributeLifetimeGarantee

It allows applications to declare that they need for 
some (so called sticky) objects a garanteed lifetime for their

volatile attributes that extends until at least the next
transaction boundary.


This extension makes safe the use of volatile attributes for
the following cases:

  * the storage of database connections in Shared.DC.ZRDB.Connection

  * the former use of _v_skindata in Products.CMFCore.Skinnable
(now replaced by a much more complex, less efficient and not
completely correct solution)

  * the use of _v_is_cp  in Archetypes.BaseObject
(and other use of volatile attributes for control purposes).

This statement implies that the current use is unsafe.


Feedback is welcome -- either in the Wiki or here.


I'm a little uneasy about baking this policy so deeply into
the infrastructure.  I wonder if the use case can be handled
another way.

A persistent object can override _p_deactivate.  For example:

def _p_deactivate(self):
pass

prevents an object from being turned into a ghost unless it is
invalidated.

You could implement your sticky attribute at the application level:


def _p_deactivate(self):
if getattr(self, '_p_sticky', False):
   return
Persistent._p_deactivate(self)

You could provide any policy you want, without making the policy part
of ZODB.

Jim

--
Jim Fulton   mailto:[EMAIL PROTECTED]   Python Powered!
CTO  (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com   http://www.zope.org
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables

2006-10-06 Thread Dieter Maurer
Jim Fulton wrote at 2006-10-6 12:10 -0400:
 ...
I'm a little uneasy about baking this policy so deeply into
the infrastructure.  I wonder if the use case can be handled
another way.

A persistent object can override _p_deactivate.  For example:

 def _p_deactivate(self):
 pass

prevents an object from being turned into a ghost unless it is
invalidated.

I think that
this will not work -- unless more drastical changes are done to
persistent:

   When we invalidate an object, it must get deactivated.

   Now, currently invalidating a cache entry calls
   the object's _p_invalidate which calls
   Per_set_changed(obj, NULL) which uses _p_deactivate
   for the deactivation.

   With a definition like the above, the object will never
   again be deactivated -- not even when it is invalidated.
  

You could implement your sticky attribute at the application level:


 def _p_deactivate(self):
 if getattr(self, '_p_sticky', False):
return
 Persistent._p_deactivate(self)

You could provide any policy you want, without making the policy part
of ZODB.

But, such an object would never again be deactivated (unless
_p_sticky is reset). In my proposal, such objects may be
invalidated at transaction boundaries.

This may not be a big problem, as there are probably not too many
objects that have _p_sticky defined permanently
(Shared.DC.ZRDB.Connection and
Products.CMFCore.Skinnable.ObjectManager would be examples).



-- 
Dieter
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables

2006-10-06 Thread Jim Fulton

Dieter Maurer wrote:

Jim Fulton wrote at 2006-10-6 12:10 -0400:

...
I'm a little uneasy about baking this policy so deeply into
the infrastructure.  I wonder if the use case can be handled
another way.

A persistent object can override _p_deactivate.  For example:

def _p_deactivate(self):
pass

prevents an object from being turned into a ghost unless it is
invalidated.


I think that
this will not work -- unless more drastical changes are done to
persistent:

   When we invalidate an object, it must get deactivated.

   Now, currently invalidating a cache entry calls
   the object's _p_invalidate which calls
   Per_set_changed(obj, NULL) which uses _p_deactivate
   for the deactivation.

   With a definition like the above, the object will never
   again be deactivated -- not even when it is invalidated.


That is clearly a bug that is easily fixed.




You could implement your sticky attribute at the application level:


def _p_deactivate(self):
if getattr(self, '_p_sticky', False):
   return
Persistent._p_deactivate(self)

You could provide any policy you want, without making the policy part
of ZODB.


But, such an object would never again be deactivated (unless
_p_sticky is reset). In my proposal, such objects may be
invalidated at transaction boundaries.

This may not be a big problem, as there are probably not too many
objects that have _p_sticky defined permanently
(Shared.DC.ZRDB.Connection and
Products.CMFCore.Skinnable.ObjectManager would be examples).


Good point.  I left off that you could register a callback on transaction
commit to clear the sticky flag.

Jim

--
Jim Fulton   mailto:[EMAIL PROTECTED]   Python Powered!
CTO  (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com   http://www.zope.org
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables

2006-10-06 Thread Jim Fulton

Dieter Maurer wrote:

Jim Fulton wrote at 2006-10-6 15:08 -0400:

...

You could implement your sticky attribute at the application level:


def _p_deactivate(self):
if getattr(self, '_p_sticky', False):
   return
Persistent._p_deactivate(self)

You could provide any policy you want, without making the policy part
of ZODB.

But, such an object would never again be deactivated (unless
_p_sticky is reset). In my proposal, such objects may be
invalidated at transaction boundaries.

This may not be a big problem, as there are probably not too many
objects that have _p_sticky defined permanently
(Shared.DC.ZRDB.Connection and
Products.CMFCore.Skinnable.ObjectManager would be examples).

Good point.  I left off that you could register a callback on transaction
commit to clear the sticky flag.


Not a nice perspective -- for too reasons:

  *  When would I register the callback?


When you set the volatile data that needed to be sticky.


 Sure, the callback should not need to look at all objects in
 the cache to find the sticky ones.

 This means, the callback would need to be registered for
 each object potentially relevant at transaction boundary time.


Yes, any object that needed to be sticky would need to have a callback
registered.


 In principle, all objects in the cache can be relevant...


All objects are sticky?


  *  The examples above have their _p_sticky defined at
 class level -- as they make vital use of volatile attributes
 and need a garanteed lifetime for them until transaction boundary.


Hm, so stickyness is a property of the class?  Why store _p_sticky on
the instance then?


 This means we need an additional callback activated after
 the cache garbage collection to get rid of the instance
 level _p_sticky again.


I don't really follow this. I'm not sure I need to.

I have a feeling that there's some deeper issue lurking here.
I'll have to think about this some more.

Some observations:

- This isn't unique to _v_ attributes.  An object could
  have other non-persistent data that is precious.

- I wonder if an argument could be made than we shouldn't
  implicitly deactivate an object that has been accessed in a
  a transaction while the transaction is still running.

Jim

--
Jim Fulton   mailto:[EMAIL PROTECTED]   Python Powered!
CTO  (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com   http://www.zope.org
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables

2006-10-06 Thread Dieter Maurer
Jim Fulton wrote at 2006-10-6 16:02 -0400:
Dieter Maurer wrote:
 Jim Fulton wrote at 2006-10-6 15:08 -0400:
 ...
 You could implement your sticky attribute at the application level:


 def _p_deactivate(self):
 if getattr(self, '_p_sticky', False):
return
 Persistent._p_deactivate(self)

 You could provide any policy you want, without making the policy part
 of ZODB.
 But, such an object would never again be deactivated (unless
 _p_sticky is reset). In my proposal, such objects may be
 invalidated at transaction boundaries.

 This may not be a big problem, as there are probably not too many
 objects that have _p_sticky defined permanently
 (Shared.DC.ZRDB.Connection and
 Products.CMFCore.Skinnable.ObjectManager would be examples).
 Good point.  I left off that you could register a callback on transaction
 commit to clear the sticky flag.
 
 Not a nice perspective -- for too reasons:
 
   *  When would I register the callback?

When you set the volatile data that needed to be sticky.

You miss my Class level sticky declaration use case...


Registering when I set the _v_ variable would probably not be enough
at least not when the registration is valid only for one transaction --
what I expect (but am not sure of):

   The _p_sticky would need to be deactivated at any transaction
   boundary as long as the object is in the cache -- independent
   of whether or not I have set any _v_ variable in the transaction.

 ...
  Sure, the callback should not need to look at all objects in
  the cache to find the sticky ones.
 
  This means, the callback would need to be registered for
  each object potentially relevant at transaction boundary time.

Yes, any object that needed to be sticky would need to have a callback
registered.

And this is difficult when the registration is only active for a single
transaction: the object may remain in the cache without me touching
it in the following transactions. Nevertheless, it should be possible
to garbage collect it at transaction boundaries.

If, on the other hand, the registration is active until explicitely canceled,
I explicitely need to cancel it when the object is flushed from
the cache. I would have to add respective code in _p_deactivate
(which I have already overridden) and in _p_invalidate (which
I would need to override as well).

  In principle, all objects in the cache can be relevant...

All objects are sticky?

The Class level sticky declaration use case means that sticky
objects can come into existence without any explicit action.

Thus, while not all objects are sticky, any object in the cache could
be sticky.

   *  The examples above have their _p_sticky defined at
  class level -- as they make vital use of volatile attributes
  and need a garanteed lifetime for them until transaction boundary.

Hm, so stickyness is a property of the class?  Why store _p_sticky on
the instance then?

As explained in the proposal:

  We have 3 use cases for volatile attributes:

1.  cache -- does not need a garanteed lifetime

2.  uses like Shared.DC.ZRDB.Connection and
Products.CMFCore.Skinnable.SkinnableObjectManager:
Require necessarily garanteed lifetime for volatile
attributes.

Most secure, if _p_sticky is defined on class level.

3.  Additional control flow information as in the
_v_isc[o]p[y] use by Archetypes.

The _p_sticky is set temporarily on the instance
to temporarily protect the _v_ attribute.

Setting it on class level is not adequate (as a potentially
huge number of objects would become sticky).

  This means we need an additional callback activated after
  the cache garbage collection to get rid of the instance
  level _p_sticky again.

I don't really follow this. I'm not sure I need to.

I have a feeling that there's some deeper issue lurking here.
I'll have to think about this some more.

Okay, will wait until then...

Some observations:

- This isn't unique to _v_ attributes.  An object could
   have other non-persistent data that is precious.

No: only _v_ attributes are lost at deactivation

(provided the object was informed about changes of its non-persistent data).

- I wonder if an argument could be made than we shouldn't
   implicitly deactivate an object that has been accessed in a
   a transaction while the transaction is still running.

If we made this argument, then my sticky proposal would immediately be
void.

However, we explicitely call cache garbage collection at
savepoint boundaries in order to allow large transactions accessing
huge numbers of objects (such as catalog operations) to
keep their memory requirements small.

Moreover, Chris Withers is keen to see an aggressive cache replacement
policy where objects are deactivated immediately as soon the
configured cache limits are reached -- again in order to keep
memory consumption low even for large transactions.

And, finally, Toby 

Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables

2006-10-06 Thread Jim Fulton

Dieter Maurer wrote:

Jim Fulton wrote at 2006-10-6 16:02 -0400:

Dieter Maurer wrote:

Jim Fulton wrote at 2006-10-6 15:08 -0400:

...

You could implement your sticky attribute at the application level:


def _p_deactivate(self):
if getattr(self, '_p_sticky', False):
   return
Persistent._p_deactivate(self)

You could provide any policy you want, without making the policy part
of ZODB.

But, such an object would never again be deactivated (unless
_p_sticky is reset). In my proposal, such objects may be
invalidated at transaction boundaries.

This may not be a big problem, as there are probably not too many
objects that have _p_sticky defined permanently
(Shared.DC.ZRDB.Connection and
Products.CMFCore.Skinnable.ObjectManager would be examples).

Good point.  I left off that you could register a callback on transaction
commit to clear the sticky flag.

Not a nice perspective -- for too reasons:

  *  When would I register the callback?

When you set the volatile data that needed to be sticky.


You miss my Class level sticky declaration use case...


Registering when I set the _v_ variable would probably not be enough
at least not when the registration is valid only for one transaction --
what I expect (but am not sure of):

   The _p_sticky would need to be deactivated at any transaction
   boundary as long as the object is in the cache -- independent
   of whether or not I have set any _v_ variable in the transaction.


...

 Sure, the callback should not need to look at all objects in
 the cache to find the sticky ones.

 This means, the callback would need to be registered for
 each object potentially relevant at transaction boundary time.

Yes, any object that needed to be sticky would need to have a callback
registered.


And this is difficult when the registration is only active for a single
transaction: the object may remain in the cache without me touching
it in the following transactions. Nevertheless, it should be possible
to garbage collect it at transaction boundaries.

If, on the other hand, the registration is active until explicitely canceled,
I explicitely need to cancel it when the object is flushed from
the cache. I would have to add respective code in _p_deactivate
(which I have already overridden) and in _p_invalidate (which
I would need to override as well).


 In principle, all objects in the cache can be relevant...

All objects are sticky?


The Class level sticky declaration use case means that sticky
objects can come into existence without any explicit action.

Thus, while not all objects are sticky, any object in the cache could
be sticky.


  *  The examples above have their _p_sticky defined at
 class level -- as they make vital use of volatile attributes
 and need a garanteed lifetime for them until transaction boundary.

Hm, so stickyness is a property of the class?  Why store _p_sticky on
the instance then?


As explained in the proposal:

  We have 3 use cases for volatile attributes:


I didn't ask why you use volatile attributes.

I asked why _p_sticky needs to be stored on each instance,
since it is set at the class level.

...


- This isn't unique to _v_ attributes.  An object could
  have other non-persistent data that is precious.


No: only _v_ attributes are lost at deactivation


No, normally all state is lost at deactivation, including _v_
attributes.  Of course, most other state can be reloaded
from the database.

Jim

--
Jim Fulton   mailto:[EMAIL PROTECTED]   Python Powered!
CTO  (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com   http://www.zope.org
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables

2006-09-29 Thread Sidnei da Silva
| Feedback is welcome -- either in the Wiki or here.

I've seen an informal version of this by Dieter on a mailing list
somewhere (might even have been this list) ages ago and even thought
that it was already implemented. I really like the proposal, +1 on it.

-- 
Sidnei da Silva
Enfold Systemshttp://enfoldsystems.com
Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables

2006-09-29 Thread Dieter Maurer
Sidnei da Silva wrote at 2006-9-29 22:13 -0300:
| Feedback is welcome -- either in the Wiki or here.

I've seen an informal version of this by Dieter on a mailing list
somewhere (might even have been this list) ages ago and even thought
that it was already implemented.

It is implemented in our (local) copy of Zope since about the beginning
of 2005. I have put patches into the collector -- but they never
made it into the core (and patches tend to break, when they are not
incorporated soon).

I am now a Zope committer and can directly work on the core --
thus patches are no longer needed.



-- 
Dieter
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev