Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache
Jim Fulton wrote at 2006-10-9 13:37 -0400: >Dieter Maurer wrote: >> Jim Fulton wrote at 2006-10-6 17:04 -0400: >... >>> Or consider object activation and deactivation. If a ghost is >>> shared among multiple threads, then __setstate__ could >>> be called from separate threads. >> >> But, why should this be a problem? They would install the same >> state. > >Because __setstate__ is written assuming that no other thread is >calling it. And where is the this assumption used? The object is already created. "__setstate__" must store the state given in its argument in the instance dict or slots. This looks harmless, even in a multi threaded environment. > Threaded programming is tricky. Code designed for a >threaded environment has to take the possibility of of multiple >simultaneous calls into account. Traditionally, persistent object >implementors haven't had to do that. Yes. If we really implement this idea, then we have to look into the implementation carefully. >>> And of course, there's the possibility that buggy software >>> might try to mutate the objects. How would you prevent it? >> >> The special read-only connection with shared cache would raise >> an exception instead of joining the transaction. > >That's too late. I have not looked at the "Persistent.__setattr__", but we could easily arrange the code that this is *BEFORE* the actual assignment. It is difficult to believe that this could be too late. >> Of course, other threads might already have been affected >> (e.g. in the example above of modifying non-persistent data). >> But, bugs can have nasty effects > >ZODB applications count on isolation. IMO, it isn't acceptable >to give up on the isolation guarantee. If part of the application is buggy, then some guarantees are no longer available. > >What if the exception is swallowed? Then the attribute might retain its old (unchanged) value (see above). However, swallowing essential exceptions is always dangerous. Currently (without the feature), you can get inconsistent persistent state -- which is comparably nasty to multi thread problems. > What would you do? Reload the state? As mentioned above: perform the connection callback before the actual assignment. Then, there will be no assignment when the connection is unwilling to let it through. >> This fails for: >> >> * volatile attributes -- I have no easy solution > >Conceivably the persistence machinery could disallow >assignments to _v_ attrs. But, it should do this only for some special connections... >... >My intuition is still that sharing objects between threads will >introduce a host of subtle bugs. Yes, you may be right... -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables
Dieter Maurer wrote: Jim Fulton wrote at 2006-10-9 13:22 -0400: ... Hm, I don't why the use cases imply setting it only at the class level, but OK. I'll just take that as a given. So, since _p_sticky is set at the class level, why store it on the instances? ... use case 3, set on instance level. I missed that. Sorry. To make cache garbage collection as fast as possible, I have proposed to actually store the sticky information in the C object structure. I don't like the idea of adding that overhead to every persistent object just for this use case. OTOH, we only need a bit and I think we had a few spare bits. So maybe this wouldn't require any new space. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables
Jim Fulton wrote at 2006-10-9 13:22 -0400:
> ...
>Hm, I don't why the use cases imply setting it only at the class
>level, but OK. I'll just take that as a given. So, since _p_sticky
>is set at the class level, why store it on the instances?
Use case 2, set on class level;
use case 3, set on instance level.
To make cache garbage collection as fast as possible,
I have proposed to actually store the sticky information in the
C object structure. In the garbage collection loop,
this gives an "if (...obj->p_sticky...)" instead
of
isSticky = PyObject_GetAttr(obj, _p_sticky_name)
if (...isSticky...) {}
Py_XDECREF(isSticky)
This part of the proposal (storing "_p_sticky" in the the
C header of persistent objects) is not essentail.
--
Dieter
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache
... |Jim Fulton] Or consider object activation and deactivation. If a ghost is shared among multiple threads, then __setstate__ could be called from separate threads. [Dieter Maurer] But, why should this be a problem? They would install the same state. __setstate__ implementations are normally never written to be thread-safe, and can leave the state temporarily insane /during/ construction; IOW, they expect to run "atomically", as does virtually all code that isn't deliberately & carefully written to be thread-safe. This goes deep; here's an example: http://mail.zope.org/pipermail/zodb-dev/2005-April/008747.html I believe I identified one cause for catastrophic thread-race problems here (sharing a Connection across multiple threads, or for any other reason sharing an in-memory persistent object across non-serialized threads), even when objects are never modified. It doesn't look shallow to me. ... ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache
Dieter Maurer wrote: Jim Fulton wrote at 2006-10-6 17:04 -0400: ... Or consider object activation and deactivation. If a ghost is shared among multiple threads, then __setstate__ could be called from separate threads. But, why should this be a problem? They would install the same state. Because __setstate__ is written assuming that no other thread is calling it. Threaded programming is tricky. Code designed for a threaded environment has to take the possibility of of multiple simultaneous calls into account. Traditionally, persistent object implementors haven't had to do that. And of course, there's the possibility that buggy software might try to mutate the objects. How would you prevent it? The special read-only connection with shared cache would raise an exception instead of joining the transaction. That's too late. Of course, other threads might already have been affected (e.g. in the example above of modifying non-persistent data). But, bugs can have nasty effects ZODB applications count on isolation. IMO, it isn't acceptable to give up on the isolation guarantee. You couldn't wait until commit if there was the possibility that other threads might see the errant changes. In most cases (when an attribute of a persistent object is changed), we could be able to detect the forbidden modification before it happens: The modification goes through "Persistent.__setattr__" which informs the associated connection which joins the transaction. Thus, if the connection does not want that its objects are changed, it raises an exception instead of joining the transaction. Good point. That helps. What if the exception is swallowed? What would you do? Reload the state? Maybe this could be made to work. I don't know. This fails for: * volatile attributes -- I have no easy solution Conceivably the persistence machinery could disallow assignments to _v_ attrs. ... My intuition is still that sharing objects between threads will introduce a host of subtle bugs. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] We need to pay down debt
Jim Fulton wrote at 2006-10-5 08:50 -0400: >Russ Ferriday wrote: >> I'll provide some time for this, Jim. >> I'm no expert on the ZODB, which might be spun as an advantage, and I'm >> prepared to play a supporting role cleaning up doctests, or helping with >> doc organization. This means I don't mind gathering spippets from those >> who have them and pulling them together. > >Awesome. Perhaps you can start thinking about how to organize the doctests. > >I think that there are 2 (or 3) main audiences to consider: > >- Users of ZODB > > These folks need guidance on how to use ZODB to develop applications, > including pitfalls. I do not think that doctests are the optimal means for those target group. A guide, such as Andrew's guide, seems more appropriate in my view. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] We need to pay down debt
Russ Ferriday wrote at 2006-10-5 13:45 +0100: >I'll provide some time for this, Jim. >I'm no expert on the ZODB, which might be spun as an advantage, and >I'm prepared to play a supporting role cleaning up doctests, or >helping with doc organization. This means I don't mind gathering >spippets from those who have them and pulling them together. If you have questions I will try to provide answers :-) -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] We need to pay down debt
Jim Fulton wrote at 2006-10-5 08:32 -0400: > ... >I think this is an area where a lot of volunteers could make >contributions. Perhaps we could even schedule some ZODB "Doc days", >similar bug days, but with a different emphasis. > >I won't insist that new work should wait for this effort Good! > although >I'd like to. :) Sad. > Certainly, I've refrained from pursuing some >ideas of mine in large part because I think we need some foundation work >first. > >Thoughts? I am ready to write doctests for new features I develop (although I like unit tests much more). I am not inclined to work on documentation primarily. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables
Dieter Maurer wrote: Jim Fulton wrote at 2006-10-6 16:55 -0400: ... As explained in the proposal: We have 3 use cases for volatile attributes: I didn't ask why you use volatile attributes. I asked why _p_sticky needs to be stored on each instance, since it is set at the class level. It is set at class level only for one of the 3 use cases -- as explained in the original proposal *AND* in the repetition of the 3 use cases Hm, I don't why the use cases imply setting it only at the class level, but OK. I'll just take that as a given. So, since _p_sticky is set at the class level, why store it on the instances? Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache
Jim Fulton wrote at 2006-10-6 17:04 -0400: > ... >> Thread-safety is not an issue for read-only data. > >Yes it is. An object that is read-only from a persistence >point of view isn't read-only at the Python level. Okay: thread-safety is not an issue for (truely) read-only data. It is potentially an issue for only partially read-only data such as read-only only from a persistence point of view. I agree that we have spoken about a read-only storage such that they may be thread-safety issues in this context. > Objects >often have other data. Take volatile data as an example. In fact, they are the only example (if we disregard bugs modifying non-persistent attributes without informing the containing persistent object). But a single example may be enough. >Or consider object activation and deactivation. If a ghost is >shared among multiple threads, then __setstate__ could >be called from separate threads. But, why should this be a problem? They would install the same state. >And of course, there's the possibility that buggy software >might try to mutate the objects. How would you prevent it? The special read-only connection with shared cache would raise an exception instead of joining the transaction. Of course, other threads might already have been affected (e.g. in the example above of modifying non-persistent data). But, bugs can have nasty effects >You couldn't wait until commit if there was the possibility that >other threads might see the errant changes. In most cases (when an attribute of a persistent object is changed), we could be able to detect the forbidden modification before it happens: The modification goes through "Persistent.__setattr__" which informs the associated connection which joins the transaction. Thus, if the connection does not want that its objects are changed, it raises an exception instead of joining the transaction. This fails for: * volatile attributes -- I have no easy solution * modification of mutable non-persistent data This would work, too, provided that obj._p_changed = True where called *before* the modification. Calling it before the modification is in principle necessary (the get the modification thrown away when the transaction is aborted) but unfortunately, almost all legacy code does it after the modification. But, this simple is a bug -- and bugs can have non-local effects. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables
Jim Fulton wrote at 2006-10-6 16:55 -0400: > ... >> As explained in the proposal: >> >> We have 3 use cases for volatile attributes: > >I didn't ask why you use volatile attributes. > >I asked why _p_sticky needs to be stored on each instance, >since it is set at the class level. It is set at class level only for one of the 3 use cases -- as explained in the original proposal *AND* in the repetition of the 3 use cases >... > >>> - This isn't unique to _v_ attributes. An object could >>> have other non-persistent data that is precious. >> >> No: only "_v_" attributes are lost at deactivation > >No, normally all state is lost at deactivation, including _v_ >attributes. Of course, most other state can be reloaded >from the database. Thus, it is in some way persistent, isn't it? -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: Deadlock in ZODB/DB.py?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Christian Zagrodnick wrote: > Hi there, > > the DB.open method makes me wonder how it works > (http://svn.zope.org/ZODB/trunk/src/ZODB/DB.py?rev=69551&view=auto) > > It locks with self._a() and calls self._connectionMap which locks too. I > cannot imagine a way this could possibly work. > > I guess the lock should be released before the _connectionMap call? > > Interestingly both all the tests pass and ZODB works... Note the following in the __init__():: # Allocate lock. x = threading.RLock() self._a = x.acquire self._r = x.release The lock is a "reentrant" lock[1], which means that subsequent locks from within the same thread do not deadlock; instead, they increment the lock count. [1] http://docs.python.org/lib/module-threading.html#l2h-3418 Tres. - -- === Tres Seaver +1 202-558-7113 [EMAIL PROTECTED] Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFKm1r+gerLs4ltQ4RAkmjAKCQFc0CwZeWRmffajADcHqBpsY9qQCguZ12 1dFN7jpri+CtgfiGR86zH7w= =uhXI -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Deadlock in ZODB/DB.py?
[Christian Zagrodnick] the DB.open method makes me wonder how it works (http://svn.zope.org/ ZODB/trunk/src/ZODB/DB.py?rev=69551&view=auto) It locks with self._a() and calls self._connectionMap which locks too. I cannot imagine a way this could possibly work. Look at __init__(): x = threading.RLock() self._a = x.acquire self._r = x.release That is, it's a reentrant lock. See the Python docs for RLock(). I guess the lock should be released before the _connectionMap call? No. Interestingly both all the tests pass and ZODB works... That was a good clue ;-) ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables
Chris Withers wrote: Jim Fulton wrote: - I wonder if an argument could be made than we shouldn't implicitly deactivate an object that has been accessed in a a transaction while the transaction is still running. Would this prevent ZODB from ever promising not to use more than a certain amount of memory? Yes, and non-necessarily. :) The only way that ZODB could keep such a promise would be to disallow loading new objects, generating errors under some circumstances. Certainly, as Dieter pointed out, this would make it harder to remove objects from memory. The biggest zodb-related performance problems I've seen are when a scripter writes code that drags way more objects into memory than any sane script should. The creates a HUGE python process which never releases the memory back to the os (I believe that may be fixed in Python 2.5?) which causes all kinds of performance problems... It should be possible to prevent scripters shooting themselves in the foot! Sorry, that's impossible. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables
Jim Fulton wrote: - I wonder if an argument could be made than we shouldn't implicitly deactivate an object that has been accessed in a a transaction while the transaction is still running. Would this prevent ZODB from ever promising not to use more than a certain amount of memory? The biggest zodb-related performance problems I've seen are when a scripter writes code that drags way more objects into memory than any sane script should. The creates a HUGE python process which never releases the memory back to the os (I believe that may be fixed in Python 2.5?) which causes all kinds of performance problems... It should be possible to prevent scripters shooting themselves in the foot! cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Deadlock in ZODB/DB.py?
Hi there, the DB.open method makes me wonder how it works (http://svn.zope.org/ ZODB/trunk/src/ZODB/DB.py?rev=69551&view=auto) It locks with self._a() and calls self._connectionMap which locks too. I cannot imagine a way this could possibly work. I guess the lock should be released before the _connectionMap call? Interestingly both all the tests pass and ZODB works... -- Christian Zagrodnick gocept gmbh & co. kg · forsterstrasse 29 · 06112 halle/saale www.gocept.com · fon. +49 345 12298894 · fax. +49 345 12298891 smime.p7s Description: S/MIME cryptographic signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
