Re: [ZODB-Dev] Extenuated memory
Antonio Beamud Montero wrote at 2006-3-23 09:26 +0100: > ... >But my server doesn't free any memory, now it uses 55Mb of resident >size... How I can debug this "references"? It has no chance -- and this is neither the fault of Python nor of the ZODB but of "C"'s standard memory management: "C" does not use memory compaction (if it would, development would be much more difficult, as any memory use would need communication with the compacter (in the form of: I am currently using this memory, do not move it aways while I do). Thus, uses memory snippets cannot be moved in order to have all free memory at the end of the heap. "*nix" has 2 ways to allocate memory: * the "brk" system call that can add and remove memory from the end of the data/heap segment * and "mmap" which can add large memory blocks to the process address space. "mmap" is only reasonable for large blocks, smaller blocks are usually allocated via the "brk" system call -- and this memory can only be given back to the OS if a contiguous block is free at the end of the data/segment heap. The "C" level memory management decides when to give memory back this way (and how large the free block must be before it does). As occupied memory cannot be moved, its chances to give back large chunks are limited... This problem plages *ALL* C programs that intermittendly require large amounts of memory (but know in very large blocks which could use "mmap"). -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Weird ZODB error on Zope startup :-(
Chris Withers wrote at 2006-3-24 16:44 +: >Anyone ever see this error before and have a clue what it means? > File "lib/python/Zope2/Startup/__init__.py", line 47, in startup starter.setConfiguration(cfg) File "lib/python/Zope2/App/startup.py", line 100, in startup File "lib/python/Products/ZODBMountPoint/Mount.py", line 226, in close File "lib/python/ZODB/Connection.py", line 240, in close ZODB.POSException.ConnectionStateError: Cannot close a connection joined to a transaction A persistency bug -- probably in some component: The error occurs because something closes the connection without previously aborting/committing the transaction. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Weird ZODB error on Zope startup :-(
[Chris Withers] > Anyone ever see this error before and have a clue what it means? > > >>> File "lib/python/Zope2/Startup/__init__.py", line 47, in startup > >>> starter.setConfiguration(cfg) > >>> File "lib/python/Zope2/App/startup.py", line 100, in startup > >>> File "lib/python/Products/ZODBMountPoint/Mount.py", line 226, in close > >>> File "lib/python/ZODB/Connection.py", line 240, in close > >>> ZODB.POSException.ConnectionStateError: Cannot close a connection joined > >>> to a transaction It means that some object in the Connection was in a modified state when an attempt to close the Connection was made. The current transaction must be committed or aborted first -- ZODB can't guess whether the pending changes should be committed or thrown away, so it won't let the app close the Connection until the app decides that for itself. Which version of Zope? Note that ZODBMountPoint isn't part of ZODB -- it's part of Zope, and gets rewritten a lot ;-) That's why the Zope version is important. If you want to make progress, look in SVN history for who rewrote it most recently for the version of Zope you're using. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Weird ZODB error on Zope startup :-(
Anyone ever see this error before and have a clue what it means? File "lib/python/Zope2/Startup/__init__.py", line 47, in startup starter.setConfiguration(cfg) File "lib/python/Zope2/App/startup.py", line 100, in startup File "lib/python/Products/ZODBMountPoint/Mount.py", line 226, in close File "lib/python/ZODB/Connection.py", line 240, in close ZODB.POSException.ConnectionStateError: Cannot close a connection joined to a transaction cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] edge case question about _p_deactiveate and modified objects
On Mar 24, 2006, at 9:11 AM, Chris Withers wrote: Hi All, What happens if you call _p_deactivate() on a modified object but before the modifications have been committed? Do the modifications get lost or will they still be saved when the next commit happens after the _p_deactivate()? Under normal operation, the _p_deactivate call is ignored on a modified object, so the changes are not lost (but _p_deactivate accomplished nothing). However, there is currently a bug that if you call _p_deactivate on a new persistent object that has a database connection then the whole thing will just be lost. I have a TODO to make a small test case of this and put it in the collector. AIUI from Jim, fixing this reliably will require some API changes. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Conflict resolving persistent queue?
On Mar 23, 2006, at 3:21 PM, Chris Withers wrote: Hi All, I was wondering whether anyone had implemented a FIFO persistent queue class which has the following conflict resolution strategy: two concurrent adds: adds both new items to the end of the queue in a time-based order one add and one remove happening concurrently: add the new item and remove the removed item two concurrent removes: re-raise the conflict error. http://svn.zope.org/Sandbox/poster/zope.zasync/queue.py? rev=41465&view=auto Slight differences: - if you use the bucket-based one (better for potentially large queues) then it's not reliably time based order; if you use the simpler one then you get what you want. - it only pukes if you remove the same one (since, though it defaults to FIFO, it allows you to remove ones later down the queue if desired). I can't imagine it'd be hard to knock one up in python if not, in which case: - is there any similar python code I can look at for examples? - how would I write unit tests for the above 3 cases? This was the approach I preferred: http://svn.zope.org/Sandbox/poster/zope.zasync/queue.txt? rev=41464&view=auto I want to put this in the ZODB release (though time is running out). When I proposed it, Florent suggested that it go in persistent and that the tests no longer rely on ZODB, but test the conflict resolution code with artificial entries. This makes sense in that context, and it's what I plan/hope to do. That said, the approach I took in the doctest is also useful, since it is more real-world. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] edge case question about _p_deactiveate and modified objects
Hi All, What happens if you call _p_deactivate() on a modified object but before the modifications have been committed? Do the modifications get lost or will they still be saved when the next commit happens after the _p_deactivate()? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Extenuated memory
El vie, 24-03-2006 a las 10:25 -0300, Sidnei da Silva escribió: > On Fri, Mar 24, 2006 at 02:20:06PM +0100, Antonio Beamud Montero wrote: > | > | I think I will remove ZODB from my server and use a SQL Object mapper > | > | like SQLObject to see if I can get an stable server, or the dbapi > | > | directly.. my experience with ZODB and threads has been a nightmare :( > | > | > Don't let yourself be turned down so quickly. > All languages and all their tools have their 'problems'. If you don't > understand how they work or what they do you will never be able to > write a system that performs well and has good memory management. well, I have read a lot of python internals, but now I haven't any way to solve the memory management without a patch over the interpreter, i.e. something like: zodb.forceGC() to force the garbage collector to free the unused memory... My only hope is to apply the 2.5a1 patch and wait to works... > No language or tool will 'magically' give you that, no matter what > others say *wink*. Yes, but I need something more predictable, not by use a DB (as ZODB) in the correct way, my memory starts to grow without control... finally I think I'll return to C :-) Greetings ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Before-abort hook?
Not an option in this case I am afraid. I am more after a general solution for transaction participants that need to manage external (to ZODB) resources. The fact that persistent attributes are destroyed before a registered TM gets a chance to do its work is unfortunate. Stefan On 24. Mär 2006, at 11:24, Tino Wildenhain wrote: Do you remember, when you use the tempfile module, you get a filehandle with a "nameless" file which automatically vanishes if you close the handle. Maybe this solves the problem for you? -- Anything that happens, happens. --Douglas Adams ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Extenuated memory
On Fri, Mar 24, 2006 at 02:20:06PM +0100, Antonio Beamud Montero wrote: | > | I think I will remove ZODB from my server and use a SQL Object mapper | > | like SQLObject to see if I can get an stable server, or the dbapi | > | directly.. my experience with ZODB and threads has been a nightmare :( | > | | > | A lot of thanks for your help, but this server is in a production/stable | > | system, and use unstable systems give me fear... (what's the next | > | problem?) | > | > Note that SQLObject has it's internal 'cache' of objects, and if you | > don't handle that carefully you might end up with memory bloat just as well! | > | | Ok, first I'll make stress tests to check that, and with the raw | dbapi... if this problems continues I'll change to other | language... :-( (I've been using python for last 6 years, but with this | problems I cannot continue using it...) Don't let yourself be turned down so quickly. All languages and all their tools have their 'problems'. If you don't understand how they work or what they do you will never be able to write a system that performs well and has good memory management. No language or tool will 'magically' give you that, no matter what others say *wink*. -- Sidnei da Silva Enfold Systems, Inc. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Extenuated memory
> | I think I will remove ZODB from my server and use a SQL Object mapper > | like SQLObject to see if I can get an stable server, or the dbapi > | directly.. my experience with ZODB and threads has been a nightmare :( > | > | A lot of thanks for your help, but this server is in a production/stable > | system, and use unstable systems give me fear... (what's the next > | problem?) > > Note that SQLObject has it's internal 'cache' of objects, and if you > don't handle that carefully you might end up with memory bloat just as well! > Ok, first I'll make stress tests to check that, and with the raw dbapi... if this problems continues I'll change to other language... :-( (I've been using python for last 6 years, but with this problems I cannot continue using it...) Greetings. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Extenuated memory
On Fri, Mar 24, 2006 at 11:12:19AM +0100, Antonio Beamud Montero wrote: | El jue, 23-03-2006 a las 15:02 -0500, Tim Peters escribió: | > [Antonio Beamud Montero] | > > But my server doesn't free any memory, now it uses 55Mb of resident | > > size... How I can debug this "references"? | > | > Many layers are involved in memory management, from ZODB through | > Python down to your platform C's malloc/free implementation and ending | > in your OS kernel Unless you become knowledgable about all of them | > (and vital details vary wildly across specific platform C and OS | > versions), chances are decent you'll never find a correct answer. | | I think I will remove ZODB from my server and use a SQL Object mapper | like SQLObject to see if I can get an stable server, or the dbapi | directly.. my experience with ZODB and threads has been a nightmare :( | | A lot of thanks for your help, but this server is in a production/stable | system, and use unstable systems give me fear... (what's the next | problem?) Note that SQLObject has it's internal 'cache' of objects, and if you don't handle that carefully you might end up with memory bloat just as well! -- Sidnei da Silva Enfold Systems, Inc. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Before-abort hook?
Stefan H. Holek schrieb: Hi all, The ExtFile product I maintain keeps its payload on the filesystem. All operations are performed using temporary files which are then saved or deleted on transaction commit or abort respectively. Do you remember, when you use the tempfile module, you get a filehandle with a "nameless" file which automatically vanishes if you close the handle. Maybe this solves the problem for you? Regards Tino ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Extenuated memory
El jue, 23-03-2006 a las 15:02 -0500, Tim Peters escribió: > [Antonio Beamud Montero] > > But my server doesn't free any memory, now it uses 55Mb of resident > > size... How I can debug this "references"? > > Many layers are involved in memory management, from ZODB through > Python down to your platform C's malloc/free implementation and ending > in your OS kernel Unless you become knowledgable about all of them > (and vital details vary wildly across specific platform C and OS > versions), chances are decent you'll never find a correct answer. I think I will remove ZODB from my server and use a SQL Object mapper like SQLObject to see if I can get an stable server, or the dbapi directly.. my experience with ZODB and threads has been a nightmare :( A lot of thanks for your help, but this server is in a production/stable system, and use unstable systems give me fear... (what's the next problem?) Greetings. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Before-abort hook?
Hi all, The ExtFile product I maintain keeps its payload on the filesystem. All operations are performed using temporary files which are then saved or deleted on transaction commit or abort respectively. To this end an ExtFile provides _finish and _abort methods which are hooked up to a TM. Since Zope 2.8 (MVCC) I have a problem with aborts. On abort the machinery apparently nukes an ExtFile's persistent attributes, and I lose all information I need to clean up the external resources (filename, path, etc). What I need is a way to guarantee my cleanup code is called *before* the persistent part of my object vanishes. I currently manipulate the transaction's _resources list directly [1], but that's obviously not desirable. I have seen synchronizers, but I am unable to tell whether a beforeCompletion will end up to be a commit or abort. We now have before-commit hooks, and it seems what I need is a before- abort hook as well. Thoughts appreciated, Stefan [1] http://dev.plone.org/collective/browser/ExtFile/trunk/TM.py -- Anything that happens, happens. --Douglas Adams ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
