Re: [Zope-dev] zope2 webdav memory usage...
On Jan 10, 2008, at 8:58 PM, Chris McDonough wrote: Hi folks, Your note today pointing at code I wrote made me pay closer attention. :) While messing with Zope2's webdav implementation, I ran across this bit of memory-management code: dflag = hasattr(ob, '_p_changed') and (ob._p_changed == None) ... stuff ... if dflag: ob._p_deactivate() I actually think this should be: dflag = not getattr(ob, '_p_changed', None) ... stuff ... if dflag: ob._p_deactivate() .. because when _p_changed on a persistent object (as I read it in the persistence interface file) is None, the object is a ghost. The object will never be a ghost here because non-_p_ attributes are looked up on it before we check for _p_changed, and even if it was a ghost, deactivating a ghost (to turn it into a ghost) is just not useful. I think the only time we don't want to deactivate it is if it has been changed (when _p_changed is True). Or at least that seems to be the intent. Does this sound right? No. The important thing to note above is that the check is made *before* .. stuff ... The idea is that we want to free any objects we loaded while doing whatever we are doing. We check if an object is a ghost before doing something and then, if it was a ghost before, we try to make it a ghost when we're done. We try to leave the object as we found it. With your change, we'd be ghostifying objects that were presumably useful, because they were in the cache, and we'd be failing to ghostify objects that we resurrected just for the operation. I have a feeling the answer will be crickets, but I figure what the heck. :) Jim -- Jim Fulton Zope Corporation ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Zope Tests: 4 OK, 1 Failed
Summary of messages to the zope-tests list. Period Thu Jan 10 12:00:00 2008 UTC to Fri Jan 11 12:00:00 2008 UTC. There were 5 messages: 5 from Zope Unit Tests. Test failures - Subject: FAILED (failures=1) : Zope-trunk Python-2.4.4 : Linux From: Zope Unit Tests Date: Thu Jan 10 21:00:18 EST 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-January/008935.html Tests passed OK --- Subject: OK : Zope-2.7 Python-2.3.6 : Linux From: Zope Unit Tests Date: Thu Jan 10 20:54:17 EST 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-January/008931.html Subject: OK : Zope-2.8 Python-2.3.6 : Linux From: Zope Unit Tests Date: Thu Jan 10 20:55:47 EST 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-January/008932.html Subject: OK : Zope-2.9 Python-2.4.4 : Linux From: Zope Unit Tests Date: Thu Jan 10 20:57:17 EST 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-January/008933.html Subject: OK : Zope-2.10 Python-2.4.4 : Linux From: Zope Unit Tests Date: Thu Jan 10 20:58:47 EST 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-January/008934.html ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] 2 failures on the KGS (z3c.rml zope.app.publication)
Hi, I was trying to read and understand zope.release and zope.kgs (to be able to add zope.app.locales 3.4.1 into the upcoming zope 3.4). So I've tried to run all the tests of the KGS : - $ ./bin/test -vpc1 Running tests at level 1 Test-module import failures: Module: ZEO.tests.testZEO IOError: [Errno 2] No such file or directory: '/home/dadafkas/projets/zope3/zope3.4/test/eggs/ZODB3-3.8.0b4-py2.4-linux-i686.egg/ZEO/tests/registerDB.test' Module: z3c.rml.tests.test_rml Traceback (most recent call last): File /home/dadafkas/projets/zope3/zope3.4/test/eggs/z3c.rml-0.7.1-py2.4.egg/z3c/rml/tests/test_rml.py, line 19, in ? import PIL ImportError: No module named PIL Running unit tests: Running: 3966/10840 (36.6%) ... ml-0.7.1-py2.4.egg/z3c/rml/tests/../pagetemplate.txt Failure in test /home/dadafkas/projets/zope3/zope3.4/test/eggs/z3c.rml-0.7.1-py2.4.egg/z3c/rml/tests/../pagetemplate.txt Failed doctest test for pagetemplate.txt File /home/dadafkas/projets/zope3/zope3.4/test/eggs/z3c.rml-0.7.1-py2.4.egg/z3c/rml/tests/../pagetemplate.txt, line 0 -- File /home/dadafkas/projets/zope3/zope3.4/test/eggs/z3c.rml-0.7.1-py2.4.egg/z3c/rml/tests/../pagetemplate.txt, line 41, in pagetemplate.txt Failed example: from z3c.rml import pagetemplate Exception raised: Traceback (most recent call last): File /home/dadafkas/projets/zope3/zope3.4/test/eggs/tmpHxshRy/zope.testing-3.5.1-py2.4.egg/zope/testing/doctest.py, line 1356, in __run File doctest pagetemplate.txt[3], line 1, in ? from z3c.rml import pagetemplate File /home/dadafkas/projets/zope3/zope3.4/test/eggs/z3c.rml-0.7.1-py2.4.egg/z3c/rml/pagetemplate.py, line 20, in ? from z3c.rml import rml2pdf File /home/dadafkas/projets/zope3/zope3.4/test/eggs/z3c.rml-0.7.1-py2.4.egg/z3c/rml/rml2pdf.py, line 24, in ? from z3c.rml import document, interfaces File /home/dadafkas/projets/zope3/zope3.4/test/eggs/z3c.rml-0.7.1-py2.4.egg/z3c/rml/document.py, line 22, in ? import reportlab.pdfgen.canvas ImportError: No module named reportlab.pdfgen.canvas 4909/10840 (45.3%) ... etup-3.4.1-py2.4.egg/zope/app/appsetup/bootstrap.txtzope.app.appsetup: Security policy is not configured. Please make sure that securitypolicy.zcml is included in site.zcml immediately before principals.zcml zope.app.appsetup: Security policy is not configured. Please make sure that securitypolicy.zcml is included in site.zcml immediately before principals.zcml zope.app.appsetup: Security policy is not configured. Please make sure that securitypolicy.zcml is included in site.zcml immediately before principals.zcml 6434/10840 (59.4%) evolve (zope.app.generations.generations)zope.app.generations: Failed to evolve database to generation 4 for app1 Traceback (most recent call last): File /home/dadafkas/projets/zope3/zope3.4/test/eggs/tmpmkOgSQ/zope.app.generations-3.4.1-py2.4.egg/zope/app/generations/generations.py, line 415, in evolve File doctest zope.app.generations.generations.evolve[1], line 12, in evolve raise ValueError(generation) ValueError: 4 zope.app.generations: Failed to evolve database to generation 4 for app1 Traceback (most recent call last): File /home/dadafkas/projets/zope3/zope3.4/test/eggs/tmpmkOgSQ/zope.app.generations-3.4.1-py2.4.egg/zope/app/generations/generations.py, line 415, in evolve File doctest zope.app.generations.generations.evolve[1], line 12, in evolve raise ValueError(generation) ValueError: 4 zope.app.generations: Failed to evolve database to generation 4 for app1 Traceback (most recent call last): File /home/dadafkas/projets/zope3/zope3.4/test/eggs/tmpmkOgSQ/zope.app.generations-3.4.1-py2.4.egg/zope/app/generations/generations.py, line 415, in evolve File doctest zope.app.generations.generations.evolve[1], line 12, in evolve raise ValueError(generation) ValueError: 4 zope.app.generations: Failed to evolve database to generation 4 for app1 Traceback (most recent call last): File /home/dadafkas/projets/zope3/zope3.4/test/eggs/tmpmkOgSQ/zope.app.generations-3.4.1-py2.4.egg/zope/app/generations/generations.py, line 415, in evolve File doctest zope.app.generations.generations.evolve[1], line 12, in evolve raise ValueError(generation) ValueError: 4 zope.app.generations: Failed to evolve database to generation 4 for app1 Traceback (most recent call last): File /home/dadafkas/projets/zope3/zope3.4/test/eggs/tmpmkOgSQ/zope.app.generations-3.4.1-py2.4.egg/zope/app/generations/generations.py, line 415, in evolve File doctest zope.app.generations.generations.evolve[1], line 12, in evolve raise ValueError(generation) ValueError: 4 zope.app.generations: Failed to evolve database to generation 4 for app1 Traceback (most recent call last): File
Re: [Zope-dev] zope2 webdav memory usage...
On Jan 11, 2008, at 9:24 AM, Jim Fulton wrote: While messing with Zope2's webdav implementation, I ran across this bit of memory-management code: dflag = hasattr(ob, '_p_changed') and (ob._p_changed == None) ... stuff ... if dflag: ob._p_deactivate() I actually think this should be: dflag = not getattr(ob, '_p_changed', None) ... stuff ... if dflag: ob._p_deactivate() .. because when _p_changed on a persistent object (as I read it in the persistence interface file) is None, the object is a ghost. The object will never be a ghost here because non-_p_ attributes are looked up on it before we check for _p_changed, and even if it was a ghost, deactivating a ghost (to turn it into a ghost) is just not useful. I think the only time we don't want to deactivate it is if it has been changed (when _p_changed is True). Or at least that seems to be the intent. Does this sound right? No. The important thing to note above is that the check is made *before* .. stuff ... The idea is that we want to free any objects we loaded while doing whatever we are doing. We check if an object is a ghost before doing something and then, if it was a ghost before, we try to make it a ghost when we're done. We try to leave the object as we found it. With your change, we'd be ghostifying objects that were presumably useful, because they were in the cache, and we'd be failing to ghostify objects that we resurrected just for the operation. I see. The earliest place I've seen this code is almost ten years old, it's amazing you remember. The intent was misunderstood over the years, because code like this got into various places: for ob in obj.listDAVObjects(): if hasattr(ob,meta_type): if ob.meta_type==Broken Because Product is Gone: continue dflag=hasattr(ob, '_p_changed') and (ob._p_changed == None) the (ob._p_changed == None) will never be true here as we will have woken it up by looking for 'meta_type'. This is what got me confused initially. With your change, we'd be ghostifying objects that were presumably useful, because they were in the cache, and we'd be failing to ghostify objects that we resurrected just for the operation. The former is true, the latter no. Objects that are resurrected just for the operation would be ghostified too. Only objects that were _p_changed = True would not be. But it doesn't matter, I just needed to understand the intent, and now I do. I don't intend to check my competing example code in, I'll just fix the code around the places where the pattern was misunderstood. Related but fuzzier... is it your expectation that the amount of memory used for a database walk routine that tries to do memory management via some combination of connection.cacheMinimize-or- cacheGC() every-n-iterations (no calls to individual objects' _p_deactivate) should be close to one which uses _p_deactivate aggressively against the objects being walked? In my experience, no combination of the aggregate cache management calls seems to work nearly as well as aggressively _p_deactivate-ing walked objects directly while walking a large object tree (at least under ZODB 3.6). It seems like doing cacheMinimize, etc just doesn't really have much effect on real memory usage during the walk when it's the only thing used. It's a difficult thing to test, as you need a truly huge database to finaly see the failure mode (which is that you run out of RAM ;-), but that's my experience anyway. - C ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] 3.4 controlled-packages
I'm trying to figure out where (in the svn) the following file is: http://download.zope.org/zope3.4/controlled-packages.cfg Isn't it supposed to be the same as the one found in zope.release (trunk or 3.4.0b2 tag) ? Where does the difference come from? Christophe ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope2 webdav memory usage...
On Jan 11, 2008, at 1:08 PM, Chris McDonough wrote: ... Related but fuzzier... is it your expectation that the amount of memory used for a database walk routine that tries to do memory management via some combination of connection.cacheMinimize-or- cacheGC() every-n-iterations (no calls to individual objects' _p_deactivate) should be close to one which uses _p_deactivate aggressively against the objects being walked? In my experience, no combination of the aggregate cache management calls seems to work nearly as well as aggressively _p_deactivate-ing walked objects directly while walking a large object tree (at least under ZODB 3.6). It seems like doing cacheMinimize, etc just doesn't really have much effect on real memory usage during the walk when it's the only thing used. It's a difficult thing to test, as you need a truly huge database to finaly see the failure mode (which is that you run out of RAM ;-), but that's my experience anyway. Python isn't good at returning memory to the OS, so you really can't free it. Calling _p_deactivate along the way prevents it from growing much in the first place. BTW, in situations in which you don't know which objects to deactivate, an alternative is to call cacheGC on the connection frequently. This is fairly inexpensive and incremental. Jim -- Jim Fulton Zope Corporation ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
