[Zope-dev] RE: [Geeks] Re: Interface Meta Data proposal
[Christopher Petrilli] I notice you mention post/pre conditions (something that UML obviously talks about). I wonder if we want to do a bit of research on Eiffle and it's contractual description. The only thing I wonder is if some of this is actually useful programatically, if that makes sense? It's great info, but is it useful at runtime? Eiffel takes this all very seriously, and supplies several variants of assertions that are individually togglable. Preconditions verify a method is called according to its docs; postconditions verify a method does what it's advertised to do; anyone who takes debugging seriously is writing verification code of that kind anyway, and Eiffel automates it to an extraordinary degree. Note that's only the tip of "the contract" part: inherent in any contracting scheme is the ability to sub-contract. A subcontractor cannot require more, nor deliver less, than the original contract specifies. So, in Eiffel, for a subclass S that overrides a base class B's method M, the preconditions for S.M are magically OR'ed with the preconditions for B.M (S.M can't require more than B.M, but it may require less), and the postconditions for S.M are magically AND'ed with the postconditions for B.M (S.M can't deliver less than B.M, but it may deliver more). The language guarantees to keep all this stuff straight for you, and the doc-generation system for Eiffel knows all about it too. This isn't a collection of random debugging features in Eiffel: it's all in support of a particular formal theory of program design. if-you-can't-check-a-thing-at-runtime-you-can't-know-whether- it's-been-satisfied-ly y'rs - tim ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] RE: [ZODB-Dev] [BUG] maybe yet unknown BTREES bug
[Dieter Maurer] We have run in an apparent BTrees bug (with the BTrees version included in Zope 2.5.1). ... We will install the BTrees package from ZODB-3.2a1 and see whether the problem disappears. I will report back... There are two new kinds of sanity checks you can make there: 1. somebtree._check() This is a new method on BTree objects, which does extensive checks of internal inter-pointer consistency. It raises AssertionError if internal pointers have gotten into an inconsistent state. This used to be possible in the presence of (unlikely mixtures of) inserts and deletes. 2. from BTrees.check import check check(somebtree) This is a brand new checker that does a different kind of consistency check, verifying subtle invariants involving the key *values* (not the internal C pointers) in a BTree (or TreeSet). Again, it raises AssertionError if anything is wrong there. If anything is wrong, from BTrees.check import display display(somebtree) displays the internal bucket structure of a BTree, to stdout. It can clarify the nature of the damage. There are no known ways of creating a damaged BTree anymore, short of (so far) hypothetical scenarios involving multiple processes somehow managing to get hold of stale data without realizing it. 3.2a1 fixes a newly-discovered but extremely unlikely bug in BTree conflict resolution. I don't know that it's ever been seen in real life (it was provoked by a multithreaded stress test I wrote) -- if it ever did happen, the symptom would have been a segfault, not silent BTree corruption. We try to insert a value into an BTrees.OOBTree.OOBTree instance with tree[k]= v and get an exceptions.KeyError on 13384379A0pfvqTmtEQ. That's indeed peculiar. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] RE: [ZODB-Dev] False alarm (was: [BUG] maybe yet unknown BTREES bug)
[[EMAIL PROTECTED]] It has been a false alarm. Sorry! The same problem occured with the new BTrees module. Therefore, I checked the error again and found, that the problem was not caused by tree[k] = v but by a nearby del tree[k] (which, of course, is entitled to raise KeyError). When we had analysed the error for the first time, we happened to look at some slightly outdated source. I appreciate the follwup, Dieter! I've gone half blind searching the source for any way a BTree __setitem__ could have raised KeyError. I didn't find one, even if the BTree were corrupted. But that doesn't mean it couldn't happen wink, so knowing it wasn't actually happening is a relief. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] RE: [Zope3-dev] How (in)secure is Zope?
[Christian Tismer] ... I don't mean to offend anybody by this, it is just a very simple question which I cannot answer alone. There may be a simple question hiding in this, but it's hard to find wink. You try: how secure is sendmail? how secure is ssh? how secure is Python? Answer those simple(?!) questions in the way you're looking for, and maybe someone can do the same wrt Zope. As is, you *appear* to be asking for a one-word summary of an encyclopedia. Big wink. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] RE: [Zope3-dev] How (in)secure is Zope?
[Christian Tismer] ... p.s.: sendmail? ssh? Python? Security exploits are discussed in the bugtraq list. I can find them all in the list archive. What about Zope? It is not in bugtraq. The obvious conclusion is that no security hole has ever been discovered in Zope. Whether that's a *correct* conclusion I take no position on, just that it's the obvious conclusion wink. I think the answers given on the list were quite useful, thanks to you all! You're welcome. Don't forget that Zope's source is open, too: lots of eyeballs have scanned it, so the potentially dangerous modes of operation seem well-known in the community. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] RE: [Zope-Coders] Merging the new-install-branch
[Fred L. Drake, Jr.] I'm planning to merge the new-install-branch efforts recently discussed on this list into the Zope 2 trunk tomorrow. ... http://mail.zope.org/pipermail/zope-dev/2003-March/018999.html Perhaps the most immediate change is that the Windows tests may become completely wedged, since we've not had time to do any testing on Windows yet. If anyone has time to do so (and a suitable platform), please don't hesitate to send bug reports before the merge! I'm running the Windows tests on this branch (Python 2.2.2 + new-install-branch + Win2000), and it's going well. We found one Windows-specific problem, and Fred checked in a fix for that. Two ZEO tests are failing, but that appears to be a cross-platform problem introduced yesterday and not unique to the branch. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: segfaults in cPersistence under 2.6
[Anthony Baxter] *sigh* So I tried a fresh checkout on a different box, no crash. Completely nuked old checkout (which was a new checkout last night), checked out again, rebuilt, no crash. Beats me. I'm going to blame the electrician who took out power to our floor this morning, because it's that or cosmic rays. I hate these. Causes for stuff like this in the past included: + make clean not cleaning everything it should have cleaned. + Compiling not recompiling everything it should have recompiled. + Wild load or store in Zope or Python's C code. These *can* be extremely touchy, appearing to go away for weeks at a time, and never occurring on some peoples' machines. I've only seen one instance of that last one triggered by Zope, which turned out to be a bug in Python (fixed long ago) specific to the intersection of Python's new-style classes and Zope's ExtensionClass. Jeremy and I saw cases of ring corruption while putting together the experimental ZODB 3.3, but they were 100% reproducible on all platforms, and turned out to have clear causes (some of the C objects we backported from ZODB4 were getting ghostified by the ZODB4 code without removing themselves from the ZODB3 ring). So ring corruption rang a bell for me, but not one that makes any sense in your context. Keep your eyes open, Anthony! If it happens again, I want to know. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Very severe memory leak
[Paul Winkler] Maybe. I'd really like to run with python 2.2. What is involved in the rigorous analysis? Is there a plan I could follow? [Shane Hathaway] We need to follow the same process we used to move to Python 2.1. http://dev.zope.org/DevHome/Wikis/DevSite/Projects/SupportPython21/FrontPage This time it should be simpler, though. We first need to produce a migration assessment, similar to this one: http://dev.zope.org/Wikis/DevSite/Projects/SupportPython21/MigrationAssessme nt Basically, for each significant change made to Python, we need to evaluate its impact on Zope. Simply putting up a wiki page listing all changes between Python 2.1 and 2.2 would be a good start. If you want to, you could even include changes up to 2.3, so we don't have to do this work again to support Python 2.3. I'd actually recommend skipping 2.2 and going straight to 2.3 -- 2.3 is basically 2.2 + 16 months of bugfixes and speedups, a very solid release. It will create more problems for Zope because 2.3 starts complaining about more deprecated practices that 2.2.3 still lets slide, but they should be minor problems at worst. There are deeper benefits too. For example, the experimental ZODB3 3.3 alpha release requires Python 2.3; because of a new-style class bug that still exists in Python 2.2.3, it's not at all trivial to get ZODB 3.3 to work with Python 2.2.3. I'd dearly love to skip that pointlessly time-consuming exercise. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Why not ZODB 3.1.3 with Zope 2.6.2?
[Paul Winkler] I downloaded the hidden 2.6.2b6 release (from http://zope.org/Products/Zope/2.6.2b6/Zope/view) and had a look in lib/python/ZODB/__init__.py, which gives its version as ZODB 3.1.2. Is there any reason it's not ZODB 3.1.3? Because 3.1.3 hasn't yet been tested with that particular version of Zope. Maybe just an oversight? No, but it's becoming one wink. I could easily check any ZODB-3.1.3 changes to the Zope-2.6 branch, unless somebody knows a reason not to do so. The changes are delicate, ZODB3 3.1.4 is probably coming soon, and all these changes need also to be ported to the ZODB3 HEAD, ZODB3-3_2-branch, ZODB4, the experimental ZODB3 3.3 branch, and God only knows how many Zope branches at this point. If you want to take all those on, cool, else it's probably better to wait for Jeremy and me to do them in a controlled way. Also, why doesn't ZODB/Tools come with zope? There's some really handy stuff in there, some of which gets mentioned on the zope list from time to time; it would be nice if newbies could get hold of it easily. Leaving that to someone else. On a related note, NEWS.txt in ZODB 3.1.3 mentions that there is a new Tools/README.txt ... but it's not actually in the tarball :-) Oops! A last-second addition that didn't get added to MANIFEST. That's repaired on ZODB3-3_1-branch now. Thanks for pointing it out. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Cache growing during single REQUEST
[Toby Dickenson] You get a ReadConflictError when loading an object if it has been modified since the start of the transaction. This exception therefore becomes increasingly likely as time progresses since the start of the transaction. [Chris Withers] What's the thing Zope Corp are touting as the long term solution to this problem? Multiversion concurrency control (MVCC for short) is the next step. If no other crises intervene, Jeremy and I will start implementing that on the ZODB3 3.2 branch (most likely that branch -- can't swear to it) soon (ditto). You can google on the phrase to get a ton of more-or-less abstract overviews of the concept. The short course in ZODB is that, when MVCC is in effect, a read will return the state of the object as of the time the current transaction began, even if the object has subsequently been modified by some other transaction. The overriding concern in all schemes is that you don't see inconsistent data. The current ReadConflictError prevents you from seeing inconsistent data by preventing you from loading objects that have changed since your current transaction began. MVCC prevents it by (possibly) delivering non-current object states. I don't think either can be viewed as a pure win. The ways in which ReadConflictError loses are obvious to people here because they've experienced them. The ways in which MVCC loses will become obvious later 0.9 wink. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Cache growing during single REQUEST
[Tim] ... The ways in which MVCC loses will become obvious later 0.9 wink. [Chris Withers] Any ideas what they'll be yet? Probably none for many apps. You'll be working with possibly non-current data, so think of ways your apps could possibly be damaged by that. For example, you're Bill Gates, using ZODB to track all your assets. A summary report takes hours to generate, and by the time you get it, perhaps a few of your billion-dollar overseas accounts were wiped out in the wee hours by an adverse court judgment, but the total you get added in the account values as of the time the report-generating transaction began. Oops. To the extent that MVCC hides that you're working with non-current data, to that extent also does an app relying on current data become vulnerable. When Bill is contemplating fleeing the country during turbulent times, he presumably needs to know how much cash he has right now, not what he had last night. Most apps aren't like that, but a one-size-fits-all policy for long-running transactions (like Bill's) doesn't exist. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Cache growing during single REQUEST
[Tim] ... The short course in ZODB is that, when MVCC is in effect, a read will return the state of the object as of the time the current transaction began, even if the object has subsequently been modified by some other transaction. [Dieter Maurer] Whoever wants to use it right now: the no more ReadConflictErrors patch on http://www.dieter.handshake.de/pyprojects/zope does precisely this (for storages that support history information). How has that been working out for people (not limited to Dieter)? That's indeed what we're after, although Jeremy has in mind deeper/broader changes aimed at being more efficient than digging thru history. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] current best of breed ZEO for Zope 2.6.2 in production?
* ZEO 2.0? * any one of the ZEO releases at http://www.zope.org/Products/ZEO ? All of the releases on this page, including 2.0, are obsolete. It says so at the top of the page. I thought that was easy to miss, so I changed the heading from Zope Enterprise Objects to Zope Enterprise Objects (OBSOLETE) It could probably use more text explaining what to do instead. I'd do that if I thought I knew wink. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Re: current best of breed ZEO for Zope 2.6.2 inproduction?
[Jeremy] ... On the other hand, Jim asked me today how someone would navigate from the zope.org home page to the current ZODB release. I don't have any idea. Anyone else know? I just use Google. There are several ways, probably the easiest starting from the Developers/Zope Projects link in the left-side navigation bar, then following the ZODB link. Another easy way is to type ZODB in the Search box, then follow the top result. A more circuitous route starts with Download/Zope Products. But I use Google instead of site navigation aids on all public sites now, so those are the *last* things I'd try wink. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Re: [Zope] Using 2.3.2 for Zope 2.7
[Chris McDonough] Some people at ZC have made pretty compelling arguments to make Python 2.3.2 the recommended version of Python to use with Zope 2.7 final. I'm wondering if other people have a strong feeling about this either way. [george donnelly] yes please. :) python 2.3 seems a lot faster to me and even quite stable so far. or did you mean 2.2.3? Chris did mean 2.3.2, which is still unfamiliar to most because it was just released today: http://www.python.org/2.3.2/ This came very soon after the 2.3.1 release, and just fixes a few gross but obscure platform-specific packaging mistakes in 2.3.1. In reality and with benefit of hindsight, what was released as 2.3.1 should have been released as 2.3.1c1 (release candidate 1), and then 2.3.2 should have been released as 2.3.1 final. Regardless, 2.3.2 is the best Python in existence (by any criterion other than identical to 1.5.2 wink). ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Re: [Zope] Using 2.3.2 for Zope 2.7
[Chris McDonough] Currently, Zope still claims it works with 2.2.X (via the configure script's acceptable versions feature). Actually, 2.2, 2.2.1, and 2.2.2 aren't acceptable for Zope even now, because of Zope-critical Python bugs first fixed in 2.2.3. There's no version of 2.2 with a fix for the rare RH9 Unicode-segfault bug Jeremy stumbled into a few weeks ago (while running Zope tests), and there may never be. Should we continue to make that claim true by not depending on any Python 2.3-specific features in the Zope core? I don't think there are a lot of super-compelling core and/or library differences between Pythons 2.3.2 and 2.2.3 that would make this a hardship on core Zope developers. I don't know whether it's a hardship for anyone to install 2.3.2 but not to install 2.2.3. Seems unlikely wink. In the spambayes project, we've found that people just can't stop themselves from using the new-in-2.3 enumerate() builtin, and the new-in-2.3 Sets module. They're generally useful. Since 2.2.3 is slower and buggier than 2.3.2, and may be the last of the 2.2 line, I'm not sure we'd be doing anyone a real favor by facilitating hanging on to 2.2.3. [Fred L. Drake, Jr.] Requiring 2.3 (any flavor) would allow us to drop the copy of the logging package from Zope 2.7 and newer. There are greater benefits for Zope 3, where we have several modules and packages laying around that would no longer be needed (logging, csv, gettext). Plus I copy 2.3's strptime.py module into Zope3 now (and have to edit it each time I synch up to get rid of its enumerate() call), and Zope3 has a different all-Python implementation of the new-in-2.3 all-C datetime module. The C version of datetime is more desirable due to its comparative memory frugality. One more: We're trying to move toward replacing ExtensionClass with new-style classes. This is straightforward under 2.3, but there's still a relevant glitch in 2.2.3 that appears to make it much harder (that's why the code on zodb3-devel-branch works fine under 2.3 but segfaults left and right under 2.2.3; zodb3-devel-branch is quiet now, but will probably become important again). I also don't know that we should consider 2.3.1 acceptable for any version of Zope. 2.3.1 is missing os.fsync() on POSIX systems (a gross mistake that snuck into 2.3.1), and MvL's arguments notwithstanding, ZODB wants to use os.fsync() on POSIX systems. 2.3.1 was a mistake, but a mistake that got fixed quickly. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] testSetupServers() in Zope-2_7-branch?
== ERROR: testSetupServers (Zope.Startup.tests.testStarter.ZopeStarterTestCase) -- Traceback (most recent call last): File C:\Code\Zope-2_7-branch\lib/python\Zope\Startup\tests\testStarter.py, line 194, in testSetupServers self.assertRaises(ZConfig.ConfigurationError, starter.setupServers) File C:\PYTHON23\lib\unittest.py, line 289, in failUnlessRaises callableObj(*args, **kwargs) File C:\Code\Zope-2_7-branch\lib/python\Zope\Startup\__init__.py, line 187,in setupServers servers.append(server.create()) File C:\Code\Zope-2_7-branch\lib/python\ZServer\datatypes.py, line 93, in cr eate logger_object=access_logger) File C:\Code\Zope-2_7-branch\lib/python\ZServer\FTPServer.py, line 600, in __init__ ftp_server.__init__(self, None, *args, **kw) File C:\Code\Zope-2_7-branch\lib/python\ZServer\medusa\ftp_server.py, line 727, in __init__ self.bind ((self.ip, self.port)) File C:\PYTHON23\lib\asyncore.py, line 300, in bind return self.socket.bind(addr) File string, line 1, in bind TypeError: an integer is required That was on Windows Python 2.3.2, but I don't know whether it matters. The second half of testSetupServers() appears to be setting up an error case, but it's not dying in the way the test expects. Instead self.ip == '18092 # conflict' and self.port == None and socket.bind has no idea what to make of that. It raises TypeError. The test is expecting ZConfig.ConfigurationError. I figure someone can figure this out faster than I can -- don't be shy wink. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Re: [Zope] Using 2.3.2 for Zope 2.7
[Fred L. Drake, Jr.] In Python 2.3.x, when a class is defined the __name__ of the encompassing module is looked up. I suspect Python Scripts can be fixed by defining __name__ in the globals dictionary in which the code is executed. Thanks, Fred! I added this info to the collector entry: http://collector.zope.org/Zope/1074 ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] debugging memory leaks
[Seb Bacon] ... I know from the refcounts in Zope that items of class Foo are definitely leaking, yet when I do a sys.getobjects(0, Foo) I get nothing back. If Foo is an old-style class, then every instance of Foo has type InstanceType (and so does every instance of every other old-style class): class Foo: pass type(Foo()) type 'instance' import types types.InstanceType type 'instance' types.InstanceType is type(Foo()) True getobjects() filters on type, so nothing will ever match Foo as a type. If you can change Foo to a new-style class (most easily by inheriting from object, in a recent-enough Python), life gets easier: class Foo(object): pass type(Foo()) class '__main__.Foo' Then getobjects() can filter on Foo as a type. Classes and types before Python 2.2 are distinct concepts (barring Zope ExtensionClass complications). Navigating all 10-ish references one by one seems a bit daunting. Na, with list comprehension syntax (for brevity -- you can do the same with a for-loop, of course), foos = [x for x in sys.getobjects(0) if isinstance(x, types.InstanceType) and x.__class__ is Foo] will extract just the Foo instances (if Foo is an old-style class). So I'm not sure where to go from here. Debugging memory leaks can be hard, in any language. Another place to look for ideas is in the top-level test.py from a current Zope HEAD checkout (or 2.7 branch, or Zope3). If you're able to run your code in a loop, the TrackRefs class in test.py automates some measure of identifying what (if anything) is leaking. We've changed many internal ZODB and ZEO classes to new-style classes primarily just so this test.py's -r option is more useful in identifying the source of leaks. Some yielded easily to analysis, others slobbered on for part-time weeks. There are two common culprits: 1. Some class keeps a list, or dict, of all instances ever created. These are obvious once found, but can be surprisingly hard to locate. Of course the instances never go away then until the class goes away. Sometimes it's due to leftover debugging code someone forgot to delete again. 2. Reference cycles. Big topic wink. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] debugging memory leaks
[Seb Bacon] ... Seeing as the suspect leaker contains code like: other = Foo() other.reciprocal = self self.reciprocal = other I fear the worst ;-) ...but my (naive?) reading of the documentation was that reference cycles are cleaned out by the garbage collector, *unless* they define a __del__ (which is not the case here). How am I wrong? You're reading the docs correctly. It's not necessarily cycles directly involving Foo objects that causes Foo objects to leak, it can be instead that some other (non-Foo) objects in cycles can't be collected, from which the Foo objects are in turn reachable. When an object O can't be collected, then neither can any object reachable from O. gc.get_referrers() can be used to find objects that refer to a given Foo instance. It's also possible that a something S refers to a Foo instance where S doesn't participate in cyclic gc. Then any cycle containing S is immortal, regardless of whether __del__ methods are defined in the cycle, and also then gc.get_referrers() can't reveal S's existence. Sometimes such an S is in the Python core, or in Zope's C code, although the more recent the release the less likely that is (more more kinds of objects have been added to cyclic gc over time). Are you sure that *only* Foo objects are leaking? It's pretty rare, when there's a leak, to see only one kind of object leaking. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] ThreadLock module?
[Chris Withers] Anyone know what the difference is between Zope's ThreadLock module and the standard python Lock objects from the threading module? A ThreadLock.ThreadLock is reentrant but a threading.Lock isn't: once a thread has acquired a ThreadLock, that thread can acquire that ThreadLock again, any number of times, without blocking. This is useful, e.g., for a lock that wants to give a thread exclusive access to an object's methods, where some of the methods may invoke other methods of the object. That isn't true of a threading.Lock: each attempt to acquire an already-acquired Lock blocks until the Lock is released, regardless of which thread may be trying to acquire. The other difference is that an acquired ThreadLock can be released only by the thread that acquired it, while any thread can release an acquired threading.Lock. A threading.RLock (note the R) has the same semantics as a ThreadLock.ThreadLock, though. Zope's ThreadLock.c predates Python's threading.py, and Python didn't offer any form of reentrant lock before threading.py (although you could build your own on top of Python's thread module's lock, which is in fact how threading.RLock and ThreadLock.ThreadLock are implemented). ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] KeywordIndex performance / multiunion
[Seb Bacon] But main the reason I'm posting is to wonder if there any reason not to use the multiunion operator instead of the union operator in UnIndex.py... it should be faster, right? It seems a touch faster in some informal tests. [Casey Duncan] Yes, it probably should be used. I think it is much faster when unioning very small and very large sets as well. Tim would know better, tho. time.clock() knows best wink. multiunion() has a systemic algorithmic advantage when applied to at least 3 input sets: its runtime is proportional to the sum of the sizes of the input sets. A plain union of two sets also has runtime proportional to the sum of the sizes of its input sets, but if all you've got is plain two-argument union, it's not possible to unite N sets in time proportional to the sum of their sizes. You have to pick some pair to unite first, and at worst that result has a size equal to the sum of the sizes of its inputs, and tnat larger intermediate result has to feed in again to a later union (etc). OTOH, two-argument union crawls over its input sets once each. multiunion() crawls over each of its inputs essentially six or seven times each (once to extract the values, once for each of 4 passes of a bytewise radix sort, once over the sorted result to weed out duplicates, and once again over the duplicate-free sorted result to stuff the integers back into a set object). That implies the relative timing results can be data-dependent -- and they are. The more input sets you have, though, the harder it is for a chain of two-input unions to overcome multiunion's ever-growing algorithmic advantage. Of course, if the data isn't already in memory, none of that matters unless you've got many sets to unite (data fetch time will dominate). ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] RE: [Zope-Coders] apply()'s PendingDeprecationWarning
[Fred L. Drake, Jr.] ... Guido has conceded that these warnings can be removed from Python, so I've removed them for the upcoming Python 2.3.3 release as well as for the future Python 2.4. Just thought everyone would be thrilled to know. ;-) I am! Then again, I didn't want apply() to get deprecated to begin with. taking-thrills-where-i-can-find-'em-ly y'rs - tim ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] To the brave folks trying Zope HEAD...
[Jeremy] I think the attribute error on data is just a symptom of the BTree failure. The object has no data attribute, because ZODB failed to unpickle the object that would be assigned to data. I don't know what happened with backwards compatibility for the old BTree classes. When Jim and I discussed it a while back, I thought we concluded that we would provide a script that would convert all the old BTree instances to new BTrees instance. But I don't know if that decision stuck or if the script was ever written. Jim asked about it on zope-dev, and got no response; see: Can we forsake database backward compatability on the head for a while? http://mail.zope.org/pipermail/zope-dev/2003-November/021055.html ... This would mean that old databases would not be useable with the CVS head in the near term. Would this cause anyone any problems? I'm happy no one responded to this. Silence is assent, :) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] To the brave folks trying Zope HEAD...
[Jeremy Hylton] I don't know if there is any documentation for BTree, I don't know of any dcos for (the old) BTree either. so the code will need to be your guide. I checked the code and each of the BTree types does support items(). So walk over the database, find each top-level BTree object, extract its items, put them in a new BTrees-style BTree, then save that in place of the old object. Then it may not be necessary to extract the items -- a constructor for a BTrees-style BTree object: /** ** Accepts a sequence of 2-tuples, or any object with an items() ** method that returns an iterable object producing 2-tuples. */ IOW, the constructor should invoke .items() for you -- just pass it the old-style BTree. Of course this has to be done under a version of ZODB that still *has* old-style BTree support! I don't know whether the old-style BTree comes in different type-dependent flavors, either. If not, an old BTree should probably become a new OOBTree, and an old IntSet a new IISet. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Zope 2.7 broken on Windows
[Andrew Veitch] http://collector.zope.org/Zope/1128 Not being able to use the calendar module on Zope 2.7 breaks our MailManager and more importantly the CMF! ... Looking at this a bit more I'm guessing that Python's new datetime module was removed to avoid conflict with Zope's DateTime on Window's case insensitive file system. I believe it's just a symptom of a more pervasive problem: if you look at Control Panel in your 2.7b3 Windows installation, you'll see that it identifies the Zope version as 2.7b2. Something went wrong with the Windows installer build process, packaging an odd mix of new and old code. I'm sure this will be fixed in the next Zope 2.7 Windows beta. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] cPickleCache endless loop...
[Mario Lorenz] we have spent most of the day tracking down obscure hangs of Zope (2.6.4rc1) under python2.1.3 on a RHEL3 machine. Based on what you say next, it sure sounds like this isn't unique to 2.6.4rc1. Did the same code work under some previous release? The infinite loop appears to be an inherent property of this iteration of the cPickleCache design, and that's not new in 2.6.4rc1. The problem seems to be a logic flaw somewhere related to the cPickleCache, when using a destructor in a Zope object that accesses itself. In our case(shortened to the offending line): def __del__(self): print About to destroy: , self.id What seems to happen is that the self.id access causes the object to be cached again, causing scan_gc_items() to run in circles. Based on eyeballing the C code, the ring is a list of objects in cache, ordered from least recently used to most recently used. scan_gc_items traverses this list once, from LRU to MRU, ghosting ghostifiable objects until the target number of non-ghost objects remaining is reached, or the entire list has been traversed. It looks like ghostifying your self triggers self.__del__(). Then the __del__ method unghostifies self, which has the side effect of moving self to the MRU end of the ring, which in turn means the list traversal will visit self *again*. When it does, same thing happens all over again, ad infinitum. Any ideas on how to best fix this? As the docs Chris pointed you at say, persistent objects shouldn't have __del__ methods. If the by-eyeball analysis above is correct, if a persistent object does have a __del__ method referencing an attribute of self, an infinite loop in scan_gc_items() is inevitable. So I only see 3 workarounds short of rewriting the C code: 1. Lose the __del__ method (recommended). 2. If you need a __del__ method (it's hard to imagine why, since it will get called whenever the object is ghostified, and has nothing to do with the object's actual lifetime), don't reference any persistent objects (and esp. not self) within it. 3. Recompile with MUCH_RING_CHECKING defined. Then scan_gc_items will give up after max(cache_size * 10, 1) iterations instead of running forever. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] ZODB load state error
[Tibor Tolgyesi] I just found this in my Zope log file. After this error the Zope system stopped. No wonder, but certainly not because of physical memory error. What could be the problem? My config is: AiX 5.2 Python 2.3.2 Zope 2.7.0-b3 Apache 1.3.26 The error message: 2004-01-25T02:55:21 ERROR(200) ZODB Couldn't load state for 3ef2 Traceback (most recent call last): File /opt/freeware/zope/lib/python/ZODB/Connection.py, line 564, in setstate self._set_ghost_state(obj, p) File /opt/freeware/zope/lib/python/ZODB/Connection.py, line 603, in _set_ghost_state state = unpickler.load() MemoryError If anyone has seen the same error, and especially if you have a reproducible test case, please add your info to the collector issue Tibor opened about it: http://zope.org/Collectors/Zope/1204 It's most likely that something got corrupted, but there's not enough info about this kind of error yet to guess exactly where or how or why. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] cPickleCache endless loop...
[Tim Peters] It looks like ghostifying your self triggers self.__del__(). Then the __del__ method unghostifies self, which has the side effect of moving self to the MRU end of the ring, which in turn means the list traversal will visit self *again*. When it does, same thing happens all over again, ad infinitum. [Toby Dickenson] Not necessaralily ad infinitum. It will only run forever if the number of __del__-resurrecting objects in the cache is larger than the cache target size. It's actually that the number of __del__-resurrecting objects *plus* the number of non-ghostifiable objects in cache is larger than the cache target size, right? Does that fit with your scenario? Similarly wink, except that if there's a large number of non-ghostifiable objects (more than the cache target size), then only one __del__-resurrected object is enough to provoke an infinite loop. 2. If you need a __del__ method (it's hard to imagine why, since it will get called whenever the object is ghostified, and has nothing to do with the object's actual lifetime), don't reference any persistent objects (and esp. not self) within it. or 2b as jeremy suggested, put your __del__ on a non-persistent sub object. Yup. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Zope 2.7.0 rc2 + python 2.3.3 problem
[Florent Guillaume] I'm having a weird failure here using Zope 2.7.0 rc2 and python 2.3.3 (Debian). When I execute a python script with the code: foo = range(1.0) I get this: Error Type: TypeError Error Value: unsubscriptable object Traceback (innermost last): * Module ZPublisher.Publish, line 104, in publish * Module warnings, line 57, in warn * Module warnings, line 63, in warn_explicit Instead of the expected: DeprecationWarning: integer argument expected, got float I don't know if it's python's fault or Zope's... But I guess python assumes too much about something as it fails in its standard libraries. If i use the code for i in range(1.0): pass The traceback is a bit longer: Error Type: TypeError Error Value: unsubscriptable object Traceback (innermost last): * Module ZPublisher.Publish, line 100, in publish * Module ZPublisher.mapply, line 88, in mapply * Module ZPublisher.Publish, line 40, in call_object * Module Shared.DC.Scripts.Bindings, line 306, in __call__ * Module Shared.DC.Scripts.Bindings, line 343, in _bindAndExec * Module Products.PythonScripts.PythonScript, line 318, in _exec * Module None, line 1, in testbug PythonScript at /testbug Line 1 * Module AccessControl.ZopeGuards, line 210, in next * Module warnings, line 57, in warn * Module warnings, line 63, in warn_explicit TypeError: unsubscriptable object Looks messy. It's dying here (in Python's warnings.py): def warn_explicit(message, category, filename, lineno, module=None, registry=None): if module is None: module = filename if module[-3:].lower() == .py: HERE * filename and module are both None at this point. That can't happen when running real Python. The globals set up for running the script appear not to contain a '__file__' key, and have a '__name__' key explicitly set to None. If it set either of these to something useful, or didn't have a '__name__' key explicitly set to None, warning.warn() would have been able to make up *something* for warn_explicit's filename and/or module arguments. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Minor windows intaller bug and a question
[Chris McDonough, on the C:\Zope-Instance default]
You can change the instance dir manually, no? You mean it's a bug
that it doesn't detect your system drive and instead always uses C
as the default, right? If so, yes, I agree, but likely it won't get
fixed too soon as it's so minor.
The right drive letter is the value of InnoSetup's {sd} (system drive)
constant, but I don't know how to feed that into the Pascal code that sets
this up.
[Lennart Regebro]
I find the default value strange. The old installer put it into the
program files directory, which seem to me to be the reasonable place.
Opinions on this?
PITA. If you want to adhere to Windows standards, programs belong under
Program Files, but a Zope instance is more data so really belongs in some
confusing place like $WINDOWS_DIRECTORY\Application Data\Zope
Corporation\Zope-Instance\ (under an English version of Windows), or even
under a user-specific variant of that.
That makes it maximally difficult for you to find your data, and the
embedded spaces in the path make it maximally difficult for you to use
command-line tools too (e.g., maybe you want to do an fsdump on your
Data.fs). I'd much rather have this stuff where it's easy to find, and easy
to use from a DOS box. The installer *lets* you put it anywhere; the
default should favor ease of understanding and use.
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Minor windows intaller bug and a question
[Tim Peters]
The right drive letter is the value of InnoSetup's {sd} (system
drive) constant, but I don't know how to feed that into the Pascal
code that sets this up.
[Chris McDonough]
I'm sure it's possible, I can probably figure it out. I'll put it on
my todo list.
Staring at the docs, I think this Pascal expression:
ExpandConstant('xyz{sd}xyz')
would yield the string
xyzC:xyz
on my box today. I don't have access to any box where the system drive
isn't C:, though -- PITA to test.
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] 2.7.0 corrupt Data.fs
[Thomas Anderson]
I'm seeing changes to the ZODB getting lost on reboot. I think it's
because there are no fsync() calls being issued by Zope or even by
zopectl on stopping Zope but I can't be sure. I'm using
/instance/bin/zopectl {start|stop} for controlling Zope.
I added a sync command to my /etc/init.d/umountfs script (Debian
Woody) and this seems to have fixed the problem. I know postgresql
syncs to disk for every transaction by default, does Zope?
ZODB does, as you can see in your installation's FileStorage.py
FileStorage._finish method.
Which version of Python are you using? If you're using one of the early 2.3
releases, that would explain it; from the NEWS file for Python 2.3.2:
- A bug in the autoconf machinery meant that os.fsync was never
available.
2.3.3 should be used now.
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Core Dump (Zope 2.7, Python 2.3.3, FreeBSD 4.5)
[Jeffrey P Shell] We had similar problems with Python 2.1.3 (it's a pretty infamous Python+FreeBSD problem, AFAICT), and I thought that it was patched in the Python core by now as I thought I heard something along those lines a while back, but I may have been hearing about the patch being applied to the BSD 'ports' collection. I don't think anything relevant changed in the Python core, apart from special-casing regexp tests on FreeBSD to have ever-lower expectations. Googling suggests that the BSD 'ports' collection (I have no understanding of what that means, though) added a Makefile option WANT_HUGE_STACK_SIZE which, if specified, causes FreeBSD to give threads the same stack size most other OSes give by default. It's an interesting meaning for huge, anyway wink. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Re: Zope 2.7 and objects turning into None
#0 0x6257255 in select_select (self=0x0, args=0x3d641f7c) at /usr/local/src/python_release23-maint/Modules/selectmodule.c:171 171 } (gdb) bt #0 0x6257255 in select_select (self=0x0, args=0x3d641f7c) at /usr/local/src/python_release23-maint/Modules/selectmodule.c:171 [Dieter Maurer] It is almost surely not this thread that got the SIGSEGV (unless it is a C runtime stack overflow, quite common on BSD systems). I've got serious wonders about that: 1. The full stack trace had more than 350 frames, which is excessive even for Zope 0.9 wink. 2. Line 171 of selectmodule.c in Python branch release23-maint is the closing curly brace of internal function set2list(). When a debugger dies pointing to a function's closing curly, it can mean that the generated return-from-function C code is blowing up due to stack corruption. But not enough info to say for sure ... ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Re: Zope 2.7 and objects turning into None
[Dieter Maurer] This means that the current transaction references a connection that is meanwhile closed. Some earlier request forgot to commit or abort the transaction; therefore, the transaction contains still references to the former connection. I had to fix product installation code because Zope forgot an abort when the installation of a product did not succeed. Apparently, this is not your problem. When a connection is closed while modifications are pending for that connection, what *should* happen? Jeremy I were just talking about that, and agree the current benign neglect sucks -- a subsequent attempt to commit the transaction can't succeed, and will fail in one of these obscure ways. It certainly wouldn't be right to commit the current transaction by magic when a connection is closed. It doesn't seem right to abort it by magic either, since it could, e.g., be that the connection being closed has no changes pending, but that other connections do have changes pending in the transaction, and silently aborting them could be a real surprise. I think I'd like to raise an exception when a connection is closed but changes from that connection are pending in the current transaction. Would anyone object to that? I doubt it was ever *intended* that transactions could be left in unusable states (or, if it was intended, the design has proven flawed in this respect). ZODB 4 kept track of enough stuff so that it was easy to do this there; ZODB 3 would require more work. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] RE: python.exe vs pythonw.exe difference?
[moving this from comp.lang.python to zope-dev; it really belongs on a
Zope list, although schemes to change what pythonw does probably belong
on python-dev]
[Emile van Sebille [EMAIL PROTECTED]]
I've possibly narrowed a problem I'm having running zope as a service
on winxp pro sp 1 in that when started from a command line as:
c:\zope\v27\lib\python\python.exe \
c:\zope\v27\lib\python\zope\startup\run.py -C \
c:\zope\v27\instance\etc\zope.conf
[tim inserted backslashes above, to make the line structure clear]
it starts up just fine (although now running from the console).
But when I start it with:
c:\zope\v27\lib\python\pythonw.exe \
c:\zope\v27\lib\python\zope\startup\run.py -C \
c:\zope\v27\instance\etc\zope.conf
it dies after about 30 seconds.
It wouldn't surprise me that I'm doing something it doesn't like (I'm
spawning additional processes from within a product but it worked
fine with 2.5),
It's unclear what it means, in it wworked fine with 2.5. For example,
do you mean that the second command line, using pythonw.exe explicitly from
a DOS box worked fine, or do you mean that running Zope as a service on XP
Pro SP1 worked fine, or ...?
but I'm somewhat at a loss as to debugging it in that when run as a
console app it works fine, but when run windowless it doesn't.
Did you look in your Zope log file(s) for tracebacks?
Do I have to write out check points to a file? or is there some way
to use Mark Hammonds process debugging tools? Or is this a bug,
known or otherwise?
[Thomas Heller]
It has been reported that writing to the original sys.stdout (and
maybe also sys.stderr) sooner or later raises an IOError when running
pythonw.exe, unless these are redirected. Could this be the problem?
It could, although I have no idea what WinXP does (and don't have access to
XP). Here's a Python program to try:
import sys
if 1: # edit to 1 for stdout, 0 for stderr
console = sys.stdout
else:
console = sys.stderr
import traceback
tb = file('tb.txt', 'w')
try:
i = 0
while True:
i += 1
console.write('.')
except:
print tb, Died when trying to write byte, i
traceback.print_exc(file=tb)
tb.close()
Under Win98SE, and regardless of whether it writes to stdout or stderr, it
dies when run under pythonw, and tb.txt contains this after:
Died when trying to write byte 4097
Traceback (most recent call last):
File wr.py, line 14, in ?
console.write('.')
IOError: [Errno 9] Bad file descriptor
The point of pythonw.exe is that no console is created or inherited, and the
default stdin, stdout and stderr provided by MS C in that case are unusable
(although the output flavors can appear to be usable until some secret MS
limit is exceeded -- at least under Win98SE).
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] RE: python.exe vs pythonw.exe difference?
[Thomas Heller]
[I'm currently reading python-list via the gmane nntp interface, I
don't know whether there really is a gmane.comp.web.zope.devel
newsgroup]
I don't know either, so I'll copy you directly.
[Tim]
...
Here's a Python program to try:
import sys
if 1: # edit to 1 for stdout, 0 for stderr
console = sys.stdout
else:
console = sys.stderr
import traceback
tb = file('tb.txt', 'w')
try:
i = 0
while True:
i += 1
console.write('.')
except:
print tb, Died when trying to write byte, i
traceback.print_exc(file=tb)
tb.close()
Under Win98SE, and regardless of whether it writes to stdout or
stderr, it dies when run under pythonw, and tb.txt contains this
after:
Died when trying to write byte 4097
Traceback (most recent call last):
File wr.py, line 14, in ?
console.write('.')
IOError: [Errno 9] Bad file descriptor
[Thomas]
I get exactly the same, on Win XP Pro, both for sys.stdout and
sys.stderr.
That's good to know! Thanks.
...
Since it seems XP shows the same behaviour than win98SE, has the
behaviour of Python changed? Were IOErrors ignored on sys.stdout or
sys.stderr in earlier versions?
No, the same program fails the same way here under pythonw 2.2.3 and 2.1.3
(with s/file/open/ and s/True/1/). The OP wasn't clear about what it
meant, though (in it used to work). I guess it's most likely he meant
running Zope 2.5 as a service used to work, not that running Zope 2.5 by
hand from a DOS box with pythonw used to work, but don't know. It's
certainly possible that something relevant changed in Zope, and/or in how
Zope tries to live with Windows services.
IIRC, /F first proposed that pythonw.exe should create a console to
have a place to show tracebacks. Sounds like a good idea to me.
Some way of having pythonw not drop output into the bit bucket has sounded
like a good idea to everyone for about a decade now wink.
ideas-ain't-code-ly y'rs - tim
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] RE: python.exe vs pythonw.exe difference?
[Emile van Sebille] ... At this point, I won't be back to that site before next week, although I may take some time to test this weekend. It sounds like I should rework any areas that spew output to the console. Is there something better than checking os.path.basename(sys.executable)? You could use the logging module all the time instead of trying to make the destination conditional on the executable name. Bad errors should probably be reported to the Windows Event Log service too; doing a serious job of making a thing a Windows Service is painful. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIGSession problems - blocker - our site dies - need help of experienceZope developer, please)
[Chris McDonough] ... I'd really rather just figure out why the code is failing in the first place. I'd just rather not mask the problem until I understand the cause. That may never happen, of course, but a man can dream. I definitely want to know it if there's still a way remaining to provoke conflict resolution into creating insane BTrees, although I only care if it's a most-recent version of ZODB (3.1.5, 3.2.1, or HEAD) (earlier versions have known, relevant bugs that have been fixed). BTrees appear sensitive to tiny timing holes just because they're complicated data structures and are involved in conflict resolution a lot. But apart from the bugs in the BTree implmentation fixed a lng time ago, no other corruption bug we've squashed since then actually had anything to do with BTrees -- they were general timing holes that could corrupt anything at all involved in conflict resolution (generally hard-to-provoke failure of invalidation to keep caches consistent). ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Zopeservice and sitecustomize
[Sake]
I have Zope and Activestate Python installed together in the same
win-xp machine. Everything works fine until I've learned that I can
put sys.setdefaultencoding('cp874') into sitecustomize.py to
accomodate my native language coding. Since I do that, my Zope 2.7.0
service can no longer start. I can start it manually from
runzope.bat, but it never start through the windows system service.
A full day investigation reveal that the trouble cause by the missing
of the 'SOFTWARE_HOME\lib\python' in the system environment's
PYTHONPATH. The runzope.bat set that up before then execution of
Zope.Startup.run.py, hence it run fine. But zopeservice.py rely on
the SOFTWARE_HOME\bin\Lib\site-packages\sitecustomize.py to set up
the correct PYTHONPATH. Here is the code inside Zope's
sitecustomize.py
Add Zope packages in Windows binary distro to sys.path
automagically import sys
import os
try:
sp = __file__
except:
sp = None
if sp:
dn = os.path.dirname
swhome = os.path.join(dn(dn(dn(dn(sp, 'lib', 'python')
if os.path.exists(swhome):
sys.path.insert(0, swhome)
Unluckily, this sitecustomize.py is now masked with my
sitecustomize.py inside Activestate's site-package directory, which
actually get loaded by Zope via the Python registry load path
(HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.3\PythonPath) instead
of the expected one.
It's actually a different bug: Python normally never looks at the value of
the PythonPath registry key, and that's not why your sitecustomize.py is
found first. That's actually a side effect of ActiveState installing
win32all: if you look *under*
HKLM\Software\Python\PythonCore\2.3\PythonPath, you'll find subkeys
Pythonwin, win32, and win32com. It's actually the win32com subkey that puts
your ActiveState Python's lib\site-packages into sys.path. It's then a bug
in Zope that it lets that dirty trick hide its own lib\site-packages: Zope
ships with its own Python, and *intends* to insulate you completely (in both
directions) from whatever other Python(s) you may happen to have installed
on the machine. So when that bug gets fixed, your sitecustomize.py will
never get executed.
BTW, sys.setdefaultencoding() is almost never a good solution; working with
Unicode instead usually is.
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Zopeservice and sitecustomize
[Chris McDonough] ... From this (and without a Windows machine in front of me), I can't really make any sense out of why your Activestate Python's sitecustomize.py is being found instead of Zope's Python sitecustomize.py if you're running Zope using the Zope Python install. I suspect it may be because of placement of your sitecustomize.py file and the rule named We look in the registry for application paths, but that's a guess. Yes, and the application path is specifically win32com, which ActiveState installed. That has the side effect of putting the ActiveState Python's Lib/site-packages into sys.path, and it so happens that it ends up before Zope's Lib/site-packages. That's why Sake's sitecustomize.py is found first. It also so happens that all of ActiveState's win32all appears before any of Zope's attempt to supply its own win32all, so there are multiple bugs here. AFIAK, these subtle sys.path glitches are in all of Zope Corp's Windows offerings, except for the very recent ZRS-for-Windows 1.4 release. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Re: ...but I want to access 'a particular tuple' inthat context!
[Tres Seaver]
The 'iteritems' method of a dictionary returns an object of type
'dictionary-iterator'; AccessControl.ZopeGuards makes no container
assertions about that type, although it *does* permit calling the
'iteritems' method which returns an instance of it.
I find it interesting that that module wraps 'iterkeys' and
'itervalues' in its 'get_iter' checker, but allows unrestricted
access to 'iteritems'.
[Jamie Heilman]
Yeah I saw that, which is why I asked about it, I couldn't decide if
it was left sort of half-baked on purpose or not. I assume not, but I
wanted to make sure.
It was deliberate. iterkeys() and itervalues() reveal objects held by the
dict directly, and so require security checks before handing those objects
out. iteritems() returns 2-tuples, though, and there's no security
implication there until you do something to extract the contained keys and
values from the 2-tuples. The security checks in the iteritems() case occur
in all the ways of picking apart the 2-tuples.
The following patch will make your use case work (it would need to be
prettied up for Python 2.3):
[a patch that inserted
ContainerAssertions[type({}.iteritems())] = 1
]
Unfortunately,
type({}.iteritems()) is type({}.iterkeys()) is type({}.itervalues())
They all return a dictionary-iterator object. The .next() method of a
dictionary-iterator object is safe to hand out for the iteritems flavor
(again because it returns 2-tuples), but the .next() methods of the others
aren't safe (again because they reveal contained objects directly).
OK, but really I'm more interested in having this supported in Zope
proper,
Sure -- I'm sure Tres was just trying to get you unstuck in a hurry.
I can always use .items() instead of .iteritems() and soak the
associated costs if I have to. Surely making iteritems use a guarded
interator is the Right thing, yes?
The thing going wrong here is subtler than that. For example, change
iteritems() to itervalues() in your original example, and I bet it still
won't work. OTOH, if you use list(yourdict.iterkeys()) instead, I bet it
will work.
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Re: circular referenced persistent objects
[Sandor] This is still a question. Is there any way, to determine how many times a zodb persistent object is referenced? ZODB itself doesn't keep track of that, although it's possible to write a storage that does. FileStorage does not. BerkeleyStorage did (past tense because Zope Corp has given up on BerkeleyStorage for now). sys.getrefcount seems to tell me only those references which are currently loaded in the memory. Yes, sys.getrefcount() is a Python builtin, and only knows about in-memory Python objects. Python (and its builtins) doesn't know anything about persistence. Also another very interesting question is that who referring to a specific object. Like at the start, it would be possible to write a storage that saves away that kind of inverted index, but I don't know of any that do. Given a particular flavor of storage, and detailed knowledge of its implementation, you can write scripts to crawl over the storage and compute anything you like. For FileStorage, the standard tools fsrefs.py and fsdump.py illustrate two ways of picking apart an .fs file. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] zLOG is dead
[Chris McDonough] There probably are no log-trawling tools. The output generated by zLOG is basically unparseable. Alas, that hasn't stopped people from writing trawlers to analyze ZEO server and client logs. That one's going to be my headache to fix(*), and has some urgency since ZODB/ZEO's use of zLOG is the only reason Zope3 still needs the zLOG package. people-who-like-regexps-don't-understand-perverse-ly y'rs - tim (*) Unless someone decides to do it before I get to it. False hope is a way of life wink. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Re: The bleak Future of Zope?
[Max M] Or perhaps an automated nightly Windows build. [Stephan Richter] We have talked about it many times before, but simply lack the bandwidth. Maybe you could provide this for Cygwin? [Max M] Argh ... that wasn't fair. Ok I will try and find some time to look into it. A problem is that every platform has its own unique bag of miserable quirks. Case in point: before we released ZODB 3.3a3 last Friday (which is also the ZODB in the current Zope2 and Zope3 CVS HEADs), I tried to run the ZODB/ZEO test suite under Cygwin on WinXP Pro. Disaster is a fair assessment -- every time the test framework tried to spawn a ZEO process, it died instantly, with a Cygwin-specific message I didn't understand. So you need to be a real platform fan to get a minority platform to work; while I like Cygwin well enough, I rarely use it, and don't have time or interest to pursue it as a hobby. Maybe this is (still) relevant to building Zope under Cygwin, maybe not: http://www.zope.org/Members/dgeorgieff/howto_zope_cvs_on_cygwin/index_html What exactly is needed? I routinely compile Zope2 and Zope3 HEAD on Windows, using MSVC 6. I can't make time to set up a fancy snapshot procedure, but if all people want is (e.g.) a zip file containing the .pyd files, uploading those once a week wouldn't be a significant time sink. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Re: The bleak Future of Zope?
[Max M] Well, yeah. I installed cygwin and all the devolpment tools. About 800 Megs. I could have sorted it, but I wouldn't risk missing libraries, tools etc. and harddisk is cheap. Same here (although my old laptop doesn't have enough disk space remaining to download the whole thing). Python compiled fine, both with and without ./configure --with-threads Z3 also compiled without a hickup. Python 2.3.3 comes with current Cygwin, so there shouldn't be a need to build Python (or maybe there is? I don't know; the one that comes with Cygwin has threads enabled already: $ python Python 2.3.3 (#1, Dec 30 2003, 08:29:25) [GCC 3.3.1 (cygming special)] on cygwin Type help, copyright, credits or license for more information. import thread def f(): ...print 'hi!' ... thread.start_new_thread(f, ()) 10852896 hi! ). I didn't have any problems compiling anything, I hit instant disasters whenever code tried to spawn a new process (mystery errors under WinXP Pro, segfault and system freeze under Win98SE). But when I tried to go to http://localhost:8080; or http://localhost:8080/manage; I just got a A system error occurred. message, and a the following log entry: 2004-04-22T08:47:13 ERROR root PageTemplateFile: Error in template: Compilation failed exceptions.SyntaxError: invalid syntax (string, line 1) Which is sort of non-helpfull. Sorry, no clues here. Perhaps someone else knows how to get Cygwin to work. ... What exactly is needed? I routinely compile Zope2 and Zope3 HEAD on Windows, using MSVC 6. I can't make time to set up a fancy snapshot procedure, but if all people want is (e.g.) a zip file containing the .pyd files, uploading those once a week wouldn't be a significant time sink. AS far as I can see that should be enough. If the compiled files, in their directory structure, could just be dropped on top of the python structure from cvs/subversion I expect that would be enough? No way to tell without trying. I don't know whether you're building Zope2 or Zope3, but since this is the zope-dev list I assume the former. Try http://zope.org/Members/tim_one/Zope2-20040422.zip/file_view and let us know what happens! As the comment there says, it's just .pyd files from Zope2 HEAD, compiled with MSVC 6. This is from an inplace (setup.py build_ext -i) build on Windows, from a current Zope HEAD checkout. As far as I can see from a quick manual scan of the directory structure that's how the code is structured now? The compiled files are not under version control, and so would not be overwritten by updating from cvs/subversion. That's correct. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Re: The bleak Future of Zope?
[Tim Peters] ... No way to tell without trying. I don't know whether you're building Zope2 or Zope3, but since this is the zope-dev list I assume the former. Try http://zope.org/Members/tim_one/Zope2-20040422.zip/file_view and let us know what happens! As the comment there says, it's just .pyd files from Zope2 HEAD, compiled with MSVC 6. This is from an inplace (setup.py build_ext -i) build on Windows, from a current Zope HEAD checkout. FYI, there's a similar zip file now containing the same kind of thing for a current Zope3 checkout (s/Zope2/Zope3/ in the URL). If this is good enough for people trying to work from CVS on Windows, let me know and I'll update them from time to time, and maybe move them to a saner location. If it's not good enough, sorry, but I don't anticipate having enough time to do more than this (which is just a trivial zip+upload step beyond the builds I have to do anyway). ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Nightly Zope 3 Binary Compiles for WIndows
[Chris Withers] Saw my name mentioned earlier but not sure whether Tim has solved the problem... Can't say -- I put up .pyds for current Zope 2 and Zope 3 HEAD, but haven't heard whether anyone tried them. I'd be happy to set up a nightly (or weekly, let me know which would be better) scheduled task [...] that checked out he latest HEAD of Zope 3, compiled it and PUT it up to my Zope.org member area I expect that would be helpful, and also helpful for the Zope 2 HEAD, but it's not clear what you would upload. For example, just the .pyds, or the entire codebase, or...? (I suspect the slowest part of this process will actually be uploading it to Zope.org If it's just the .pyds, the upload is small and goes fast, and only *needs* to be done when Zope's C code changes (infrequent). If it's the entire codebase, then, ya, it will go slower, and needs doing more often. ... Would it be helpful to get the nightly Windows tests running again? Yes! For both HEADs. I stopped as a corollory of setting up my own company, but also because of lack of perceived support from people who could fix Windows problems and the lack of caring of developers who only developed for Linux and broke stuff on Windows... ahs any of that changed? I can't know whether your perceptions have changed, but guess that they haven't wink. Of course most developers are still on Linux, and break the tests on Windows routinely; that's not going to change (the things that go wrong on Windows don't make sense to Linux programmers -- e.g., the idea that you can't rename or delete an open file just isn't in their view of the world). So progress on Windows occurs in spurts, usually only when one of the few active contributors on Windows can make spare time for it. There are currently 7 failures on native Windows in Zope 2 HEAD. I opened a Collector report about them because I don't know how to fix them. The Zope3 tests are in better Windows shape today, and it's likely that all Zope3 tests (unit and functional, including the --all tests) will pass on native Windows today. Exceptions I know about: + One test will never pass on Win98SE (it opens more sockets simultaneously than Win98SE can handle). + One of the --all ZEO tests often fails on my hyper-threaded Pentium box, but never fails anywhere else, and never fails if I disable hyper-threading in the BIOS. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope3-dev] Re: [Zope-dev] Re: The bleak Future of Zope?
[Max M, tests a zip file of .pyd files in Zope3, on Windows] I finally got around to testing this, and it works *exactly* as I hoped. I downloaded Z3 from CVS, and Tim's zip file. I unpacked the zip file into the Z3 directory, and it started the first time. Yippee! That's what I expected, but it's always a pleasant surprise when software works 0.8 wink. I don't know about others, but I don't need anything more than this. If either Chris or Tim would put that structure up at a saner location, I will write a How-to about it. Basically I just need a web adress that stays current to use in the How-to, so that I could point to somewhere like: http://zope.org/Members/tim_one/z3builds Where they could get the latest zip file. http://zope.org/Members/tim_one/Zope3-pyd.zip is in place now. I'll put up a similarly named one for Zope2 this afternoon (EDT). They'll be replaced from time to time (when Zope 2/3's C code changes), but will retain theses names. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Z2 HEAD in rough shape for b/c
[Chris McDonough] ... - The ZODBMountPoint product relies on a method of Connection objects named _getMountedConnection, which apparently no longer exists. Connection objects never had such a method, but they still do wink. This breaks any mounted databases (which breaks dbtab, which breaks sessions, which breaks lots of other things). Hopefully this is simple to fix, I haven't looked yet. I expect Shane will look at this in a week or two (when he settles down from moving) -- but can't know that. There was some horrid patching going on in ZODBMountPoint.py, which reached into various ZODB internals and replaced them. Jeremy refactored that, to make MountConnection a subclass of Connection, and MountConnection._getMountedConnection() is still there. There's still horrid patching going on, in that ZODBMountPoint.py reaches into ZODB.DB.DB and forces it to create MountConnection connections instead of Connection connections. Perhaps this isn't being done soon enough in whatever exactly it is you were trying. It would probably make most sense to move some of this code into ZODB (instead of replacing ZODB guts by magic). I think Shane wants to do that. I wish I could say it's peculiar that the Zope HEAD tests all pass despite the problems you're finding. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] RE: [Zope3-dev] RE: [ZODB-Dev] Subversion repository layout
[Tim Peters] ... IOW, the existing subversion docs cover the standard layout quite well. If we do something unique, I'm afraid it becomes another piece of folklore that will be impossible to guess and difficult to find out about. [Martijn Faassen] I don't know much about subversion, but this strikes me as a good argument for sticking with the standard layout. The I don't know part is exactly why I would prefer a standard layout. I didn't know anything about subversion, but the online book looks very good: http://svnbook.red-bean.com/ After reading (parts of) that, I thought I knew what the standard layout looked like, and why. Appendix A is especially for CVS users, and has a giant warning box saying: Warning Since Subversion treats branches and tags as ordinary directories, always remember to check out the trunk http://svn.example.com/repos/calc/trunk/) of your project, and not the project itself (http://svn.example.com/repos/calc/). If you make the mistake of checking out the project itself, you'll wind up with a working copy that contains a copy of your project for every branch and tag you have [That is, providing you don't run out of disk space before your checkout finishes.] The standard layout is assumed, like it is there, all over the book. Now if we change it to something we like better, the people doing the change will understand it completely, but few others will, and those relying on the svnbook docs to get up to speed will discover (probably the hard way) that all the examples in the book are wrong for Zope's unique layout. So I think it would do more harm than good, unless it does a lot more good than just saving me from typing an extra trunk/ now again. In the standard layout, each project has a trunk, branches, and tags subdirectory, containing what you already think they contain -- it's quite logical and elegant. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] RE: [ZODB-Dev] Subversion repository layout
[Kapil Thangavelu] ... I like the layouts Jim's presented (specifically #2 of 3), i think when considering the subversion docs, the important distinctions are made between the directories used for branches and tags, as long as that information is clearly communicated the semantics are exactly the same. the subversion docs themselves outline multiple repository structures (for example the single project layout), Sorry, I haven't seen that. The Choosing a Repository Layout section does talk about many ways to organize repositories, like multiple repositories vs one, and putting all projects in a repository at top level vs grouping them into related subtrees. But in all these cases, the only structure for a project they discuss or illustrate has project/trunk, project/branches, and project/tags structure. although they do recommend a standard structure, the docs go through great lengths to convey a semantic understanding of subversion as a versioned filesystem, not a magic functional notion as is common with cvs. i honestly dont think anyone coming from/to a subversion system will have problems as long as the location of the trunk, tags, and branches directories for a project are clearly identifiable. I haven't used svn, and I'm more concerned about people like me wink: the docs assume a specific project (not repository) layout throughout. I'm not interested in svn for its own sake, it's just something I'll need to do my job. The closer the docs match the system I have to work with, the easier it will be to get started. I can't say Jim's suggestions are bad, or good -- I can't judge them since I haven't used svn (you?). Going against the recommendation of the people who designed and coded the system seems a dubious step on the face of it, though. quoting the svn docs. Lay out your repository in whatever way you see fit. Subversion does not expect or enforce a layout schema-in its eyes, a directory is a directory is a directory. Ultimately, you should choose the repository arrangement that meets the needs of the people who work on the projects that live there. That's at the end of the Choosing a Repository Layout section. As above, that section discusses and shows nothing but the standard per-project layout; the layout choices they do discuss in that section are the ones mentioned above (how to organize projects in relation to each other, and in relation to repositories). ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] performance tuning of ZODB
[Leonardo Rochael Almeida] Please add this issue to the Collector, including the test (prefereably without the twisted bits) [Syver Enstad] Eh, what is the Collector? The zope.org Collectors are here: http://www.zope.org/Collectors/ and you want the Zope Collector: http://zope.org/Collectors/Zope/ It's where bug (or issue) reports are stored and managed. Maling lists don't work for this purpose. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] RE: [Zope3-dev] RE: [ZODB-Dev] Subversion repository layout
[Kapil Thangavelu] [snip debating over what the book says] sigh.. debating over what the book says isn't very productive. That's for sure wink. my conclusions at the end of my previous email, namely that what this layout will accomplish for the zopeorg repository in terms of avoiding renames of checkouts will likely be fairly limited in pratice, still win me out against deviating from the standard layouts. I think your points about stitching stuff together are crucial, since a lot of that indeed occurs in the Zope world. You have svn experience too, while I don't, so I'm happy to yield to that. The other crucial thing is that we document clearly what we're doing. I said before that newcomers to svn won't understand the alternative structures, and I still believe that. By sheer coincidence, someone on comp.lang.python today posted this: I'm experimenting with svn. What is the best way to set up the original project, anticipating importing to a truck-and-branch world? When I start I have: myproject/ doc/ mypackage/ stable.py changing.py test/ go_test To do branches, I think I'm supposed to get to end up with: myproject/ trunk/ doc/ mypackage/ stable.py test/ go_test branches/ branch1/ mypackage/ changing.py test/ go_test ... If that demonstrates a lack of understanding of how svn truly works, what that really demonstrates is that *of course* newcomers to svn don't understand how it truly works, and I think it's hard to come away from a first pass over the svn book without believing this specific directory structure is more magical than it really is. monkey-see-monkey-do-is-a-tough-default-to-overcome-ly y'rs - tim ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Change in repository approach to software sharing
[Jim Fulton] The Zope project includes a number of interrelated subprojects, such as: - Zope 2 - Zope 3 - ZODB - ZConfig Software from the ZODB and ZConfig projects are shared by Zope 2 and Zope 3. Note that ZODB also depends on ZConfig. We want this sharing to be very convenient for people working on Zope 2 and Zope 3. We don't want users of the Zope 2 and Zope 3 repository to have to do separate checkouts for ZODB and ZConfig. CVS supports such software sharing through it's module system. The module system has some flaws, so we use symbolic links instead. In a response to: http://dev.zope.org/Zope3/MovingSCMToSubversion John Belmonte has suggested that Zope 2 and Zope 3 should depend on specific versions of shared packages, rather than on the head. I'm inclined to agree. Didn't we endure a lot of pain over the last year to get away from that model? At one point there were 7 lines of ZODB I was wrestling with every day. For example, there was the ZODB 3.1 branch, which in theory corresponded to the Zope 2.6 branch, but in practice never matched it. Life got much better (for both Zope 2.6 maintenance and ZODB 3.1 maintenance) when we abandoned the ZODB3-3_1-branch branch of ZODB3 and did all ZODB 3.1 maintenance directly on the Zope-2_6-branch branch of Zope. Likewise for ZODB3-3_2-branch versus Zope-2_7-branch: the former was abandoned, and life got better for everyone because of that. Currently the HEAD of the ZODB module is the same as the HEAD view of ZODB when viewed from the Zope and Zope3 modules, which isn't a pure win, but isn't a pure loss either. People working on ZODB and ZConfig have to test their changes against both Zope 2 and Zope 3 to avoid breakage. I don't see that this part can change, no matter how names are shuffled. This is very burdensome That part either -- all the tests in all contexts still need to be run. and causes much pain when they get it wrong. Ditto, although you're (re)introducing mechanism that alleviates this at the cost of increasing the number of ZODB snapshots in active use. Fortunately, subversion provides a mechanism for sharing specific revisions. We'll be able to have the convenience of getting ZODB and ZConfig (and other shared software) when we do a checkout *and* we'll be able to control what parts we get. To see how this will work, we'll look at ZConfig as an example (because it has a single package) of reusable software that we will include in Zope 3. In the repository, we'll have a top-level Zope3 project directory, with the standard svn subdirectories trunk, branches, and tags. We'll also have a top-level ZConfig project directory. The trunk of the ZConfig Python package will be in ZConfig/trunk/src/ZConfig. If we create a tag T1 of ZConfig, then the Python package for that tag will be in ZConfig/tags/T1/src/ZConfig. Now, when we set up the Zope 3 repository, we will create the ZConfig package in Zope 3 by copying a *tag* from the ZConfig project: svn copy svn+ssh://svn.zope.org/repos/ZConfig/tags/T1/src/ZConfig \ svn+ssh://svn.zope.org/repos/Zope3/trunk/src/ZConfig -m 'Bring ZConfig T1 into main branch' Then, whenever someone checks out the Z3 trunk, they'll get the ZConfig from T1. I don't yet know enough about svn to guess how the next step works: since ZODB all by itself depends on ZConfig, presumably similar stitching of ZConfig will take place in the top-level ZODB project. Now when we go on to stitch ZODB into Z3, how will ZODB's attempt to stitch in its own version of ZConfig play with Z3's attempt to stitch in T1 above? I'm not picturing how this works. Maybe it's simple! If we need to, we can even make Zope3-local changes to ZConfig. That would bloat ZConfig maintenance headaches exponentially. I'm confident of that, because of ZODB experience with (at least) 7 concurrently active ZODB snapshots -- it's simply intractable to keep straight what should and shouldn't get cross-ported across this set; mistakes and oversights are as much the norm as the exception then; the non-ZODB people doing stitching of ZODB into their projects won't be confident about exactly which tag they should stitch in; expediency will cause them to stitch in whatever seems to work at the time; and then the latter becomes an active snapshot that needs to be maintained too. Don't make Zope3-local changes to ZConfig or ZODB, though, and at least that nest of rats can be sidestepped. Later, we may decide to upgrade the Zope 3 head to use ZConfig tag 3. Ya, and the Zope head may decide to use ZConfig tag 44, while the Zope 2.8 maintenance branch decides to use ZConfig tag 37 despite that ZConfig 3.3.1 (which is supposed to go out with the next Zope 2.8 release) is actually released at tag 39. The dark side of flexibility is that it creates that many more ways to get out of synch. At that point, we can recopy from the tag, or we can merge changes
RE: [Zope-dev] Re: Policy for Collector-Issues 545 and 1217?
[Jamie Heilman] here's the patch I'd have attached to http://zope.org/Collectors/Zope/1217 if the collector could collect FYI, I attached the patch to the collector report (nothing magical -- it just worked for me). ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] is threading in zope useful ?
[sathya] I read somewhere that each zope thread acquires the global python interpreter lock before processing a request and until it releases it the other zope threads are essentially locked out. The Python GIL (global interpreter lock) affects all code written in Python: only one thread at a time can interpret Python bytecodes in a given process. That's true in or out of Zope. Does that mean zope threads are eventually serialized and there are no benefits to be gained from concurrent execution ? No. The GIL effectively serializes threads doing pure computation in Python, but many pieces of the Python implementation release the GIL around calls to known-threadsafe C libraries. The most important examples of that are I/O, of many kinds. For example, if a Python thread opens a socket, the GIL is released around the C-level code that does the low-level socket open, which allows other Python threads to proceed in parallel (of course the thread that does the C-level socket open waits for the latter to finish; it reacquires the GIL after the socket operation is complete and it's ready to go back to interpreting Python bytecodes). Because Zope does a lot of network I/O, this form of parallelism can be very useful for it. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Re: [Zope3-dev] ATTENTION! cvs to subversion transitiontomorrow
[Jim Fulton] ... You will be able to do read-only anonymous checkouts like so: svn co svn://svn.zope.org/repos/main/project/trunk For example: svn co svn://svn.zope.org/repos/main/ZConfig/trunk FYI, I tried that on Windows (XP), and it worked fine. One glitch, which may be all over the place: some of the text files got checked out with Windows line ends, but most did not. For example, 14 of the 19 *.txt files in ZConfig ended up with Windows line ends, but none of the 37 *.py files did. Ack, no, none of the checked-out .txt files did either. The .txt files that had Windows line ends were all created by svn for its own purposes (README.txt files in .svn directories). I'm not sure what to do about this. Best I can tell from the docs so far, svn wants a svn:eol-style property added to every line-oriented file, with value native in order to get platform-sane line-end conversions. The doc's explanation of the effect of that matches my understanding of what CVS does for all non-binary files, which is usually exactly right. I noticed that Fredrik Lundh complained about something similar here: http://effbot.org/zone/subversion.htm ... Properties are nice, but having to use three different commands to check in a text file from Windows is pretty annoying. Looks like svn *expected* us to do this by setting enable-auto-props during the intial imports, with a bunch of [auto-props] settings in a config file; like [auto-props] *.c = svn:eol-style=native *.cpp = svn:eol-style=native *.h = svn:eol-style=native *.py = svn:eol-style=native *.dsp = svn:eol-style=CRLF *.dsw = svn:eol-style=CRLF *.sh = svn:eol-style=native;svn:executable *.txt = svn:eol-style=native *.png = svn:mime-type=image/png *.jpg = svn:mime-type=image/jpeg I think we'll have to develop a standard set of config file settings like that for committers to add to their personal svn configs -- or can that be done on the server side? To do a writable checkout (if you are a contributor who has submitted a version 1.1 contributor's agreement), you will use svn+ssh: svn co svn+ssh://svn.zope.org/repos/main/project/trunk Is that supposed work already? All I've been able to get out of it is, e.g., C:\Codesvn co svn+ssh://svn.zope.org/repos/main/ZConfig/trunk szc2 svn: Connection closed unexpectedly where the error msg appears very quickly (usually well under a second). ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-Coders] RE: [Zope-dev] Re: [Zope3-dev] ATTENTION! cvs tosubversion transitiontomorrow
[Jim] svn co svn+ssh://svn.zope.org/repos/main/project/trunk [Tim] Is that supposed [to] work already? All I've been able to get out of it is, e.g., C:\Codesvn co svn+ssh://svn.zope.org/repos/main/ZConfig/trunk szc2 svn: Connection closed unexpectedly where the error msg appears very quickly (usually well under a second). Jeez Louise -- I peed away the whole night on this. Read about 70 messages on various lists with various people on Windows having the symptom. Most threads eventually fizzled out without a cure being found. The others had a variety of causes, none of which applied to me. Finally hit paydirt at the very end of: http://www.contactor.se/~dast/svnusers/archive-2004-02/1053.shtml which is a guide to setting up an svn *server* on Windows. The ultimate problem was that my user name on zope.org is tim_one, but that's not what svn on Windows thinks my user name is. In fact, I still have no idea what svn makes up for that, but it ain't tim_one. Things that don't work: + The svn --username switch: svn co --username tim_one svn+ssh://svn.zope.org/repos/main/ZConfig/trunk svn: Connection closed unexpectedly The link above warned that this wouldn't work, which opened the floodgates. + Embedding my user name in the URL: svn co svn+ssh://[EMAIL PROTECTED]/repos/main/ZConfig/trunk svn: Connection closed unexpectedly + Setting the USERNAME envar (which WinXP sets automatically, and here I'm overriding it): set USERNAME=tim_one svn co svn+ssh://svn.zope.org/repos/main/ZConfig/trunk svn: Connection closed unexpectedly + Setting the Unixish LOGNAME envar, in poke-and-hope desperation: set LOGNAME=tim_one svn co svn+ssh://svn.zope.org/repos/main/ZConfig/trunk svn: Connection closed unexpectedly + Adding host svn.zope.org user tim_one to an .ssh\config file. I think that *should* have worked, but it's probably a limitation of the specific for-Windows ssh.exe I'm using here (but which has worked fine with CVS for years, and without a config file). Finally (give up wink?), what does work: + Setting the SVN_SSH envar appropriately: set SVN_SSH=ssh -l tim_one svn co svn+ssh://svn.zope.org/repos/main/ZConfig/trunk A trunk\info.py A trunk\datatypes.py ... Aarghghgh. Here's another that works, and this is the best: + Adding Host svn.zope.org User tim_one to an .ssh\config file. I tried that after reading a man page for SSH_CONFIG. It claimed that the keywords were case-insensitive, so I figured whoever wrote the Windows implementation must have screwed that up -- and, apparently, they did! all-obvious-to-the-most-casual-observer-ly y'rs - tim ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] RE: ATTENTION! cvs tosubversion transitiontomorrow
One more on access from Windows. TortoiseSVN will probably be more popular on Windows than the cmdline version of svn -- it's a slick integration of intelligent svn context-menus and icon overlays into the Windows Explorer GUI, including a nice GUI diff/merge component. Turns out it has its own kinds of svn+ssh access problems, and some of the articles on the web about worming around them are plain crazy. I'll skip all that and just explain a clean way that works (which I stumbled into on my own): TortoiseSVN comes with its own implementation of ssh, which appears to be a renamed version of plink.exe from the popular PuTTY suite of Windows connection tools. It's not documented anywhere, but it turns out TortoiseSVN accepts a PuTTY session name for the hostname portion of an svn+ssh URL. That's the key. So download the PuTTY tools too (I expect most developers on Windows already have them). This is what I did: + Ran putty.exe, and created a new session in the GUI: - Under Session, used Host Name svn.zope.org, and selected the SSH protocol. - Under Connection, put tim_one in the auto-login username box. - Under Connection - SSH - Auth, entered the path to my private key file. - Back under Session, saved the session under name svnzope (this is arbitrary). - Closed putty. That's a one-time setup dance. PuTTY saves this config info in the Windows registry, where other programs can get at it via the session name you chose, and turns out that's enough for TortoiseSVN. Now when using any Tortoise action where a svn+ssh svn.zope.org URL is needed, just use svnzope (or whatever name you picked for your session) instead of svn.zope.org. For example, svn+ssh://svnzope/repos/main/ZConfig/trunk instead of svn+ssh://svn.zope.org/repos/main/ZConfig/trunk If you have an SSH passphrase, you can also run PuTTY's pageant.exe to supply it for you for as long as you leave pageant running (pageant is like the Unixish ssh-agent). Bottom line: while none of it is documented or obvious, there are clean and effective ways to use ssh+svn on Windows, both via cmdline and via the TortoiseSVN GUI, and even if your Windows username differs from your zope.org username. All the *obvious* ways of using SSH with client-side svn on Windows didn't work for me -- although they all do work for me (and have worked, for years) using SSH with client-side cvs on Windows. Disclaimer: All my experiments were on WinXP Pro. It's possible that other flavors of Windows will act differently, although I don't expect that in the cases talked about here. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Re: [Collector] Strange reject policy
[Chris McDonough] I think there needs to be another category named wontfix that doesn't imply that it will ever be fixed like deferred seems to. This category should also be selected in the default search settings. +1. The Python bug tracker has a WontFix, and it's proved valuable in practice to distinguish that from fixed and rejected. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Segfault and Deadlock
[EMAIL PROTECTED] Hi Zope (and Python) experts! There seems to be a problem when an external python module segfaults during a zope request. The remaining worker threads are deadlocked. Maybe, maybe not. Python (and so also Zope) use platform-native thread facilities, and what happens when SIGSEGV gets signaled is mostly up to them. That's why you see different behavior, e.g., between Linux with NPTL and Linux without NPTL: the OS and C runtime determine exceptional thread semantics, and Python isn't the operating system. Suppose a thread dies while holding the GIL (Python's global interpreter lock). Will the GIL be released so that another thread (including the main thread) can continue? There's no general answer to that. I expect that under *most* platform threading implementations, all threads will be dead in the water then, because threads are intentionally (by the OS and C runtime) lightweight objects under most implementations, and don't save away enough info to make it *possible* for the platform thread runtime to recover gracefully in case of thread disaster. The natural (least effort) behavior is for the system to kill off the thread simply ignoring whatever resources it may be holding. In that case, all Python threads remaining will hang forever waiting to acquire the GIL. I expect the best that can be done, short of heroic effort (like writing your own platform thread implementation), is to document what the various thread implementations actually do. ... The reason is the way python handles threads on some systems (RedHat-7.3, kernel 2.4.20, without NPTL). If you search the Python implementation, you'll find that there's nothing different in what Python does depending on whether NPTL is present. On any system, all Python asks of the platform thread gimmicks is (a) a way to start a thread, and (b) a way to implement Python lock semantics. On any POSIX system, #b is done with POSIX semaphores #if defined(_POSIX_SEMAPHORES) !defined(HAVE_BROKEN_POSIX_SEMAPHORES) else #b is done with a combination of POSIX mutexes and POSIX condition variables. It could be that whether POSIX semaphores are available on Linux depends on whether NPTL is in place -- I don't know. But if so, that may be the relevant difference. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] How To run current Zope 2 3 from CVS on Windows
[Max M] As I said, I would write a How-To in getting Zope 3 up and running on Windows, given the binaries that Tim has made. http://www.mxm.dk/papers/run-z3-cvs-wthout-compiler/ Feel free to comment. Nicely done! Thank you for doing this. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Subversion
[Chris Withers] ... There's a svn property you can set on a higher level folder in the repository that can control a mapping for file extensions to this property, IIRC. I am hazy on it but I know it's possible. If so, it's not documented. Perhaps you're thinking of the svn:ignore property? glob-based eol-style property addition can be specified in your svn config file's auto-props section, like [auto-props] *.c = svn:eol-style=native *.png = svn:mime-type=image/png but there's no provision for sharing such personal settings with other people (the config file belongs to the user, not to the repository). ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Segfault and Deadlock
[Dieter Maurer] The reason why I believe Python is to blame: Then this should really move to a Python bug tracker. With Python 2.1.3, a SIGSEGV in one thread killed them all; with Python 2.3.3, a SIGSEGV in one thread kills one of them (the main thread, not the thread that got the SIGSEGV) but brings the others in a funny state. This is on the same OS (Linux 2.4 kernel without NPTL). Apparently, Python's handling of SIGSEGV signals changed between 2.1.3 and 2.3.3. SIGSEGV is mentioned only in Python's signalmodule.c. You can use ViewCVS to show a diff between the 2.1.3 state of that (tag r213) and current HEAD. I don't see any possibly relevant differences: http://cvs.sf.net/viewcvs.py/python/python/dist/src/Modules/signalmodule.c In an earlier post, someone reported that Python explicitely blocks most signals in non-main threads. I'm not clear on exactly what blocked means. The comments at the top of signalmodule.c say: ... When threads are supported, we want the following semantics: - only the main thread can set a signal handler - any thread can get a signal handler - signals are only delivered to the main thread ... That's the intent. I verified that in the SIGSEGV case above, all remaining threads had SIGSEGV blocked. I may try to change Python to not block SIGSEGV and see whether we get again the old Python 2.1.3 behaviour. The relevant change is probably in Python/thread_pthread.h. Guido added a call to pthread_sigmask (or sigprocmask, depending on how broken the platform pthread support is ...), to PyThread__init_thread(), in revision 2.33. The checkin comment begins: Add SF patch #468347 -- mask signals for non-main pthreads, by Jason Lowe: This patch updates Python/thread_pthread.h to mask all signals for any thread created. This will keep all signals masked for any thread that isn't the initial thread. For Solaris and Linux, the two platforms I was able to test it on, it solves bug #465673 (pthreads need signal protection) and probably will solve bug #219772 (Interactive Interpreter+ Thread - core dump at exit). That was added before 2.1.3, but looks like it didn't get backported to the 2.1.3 maintenance branch before 2.1.3 was released. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Segfault and Deadlock
[EMAIL PROTECTED] [... snip good explanations ...] In order to get LinuxThreads to support the Python's threading semantics, what probably needs to be done is to have PyThread_init_thread set all handlers to call kill(main_thread, sig) to signal the main thread. If someone cares enough to work up a patch, Python's patch tracker is open all night: http://sf.net/tracker/?atid=305470group_id=5470 ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Segfault and Deadlock
[EMAIL PROTECTED], on special-casing LinuxThreads] I might be willing to try my hand at this, but I could use a tiny bit of guidance. (If you don't mind.) I don't mind wink, but I haven't run on Linux since 1994, and have lost track of how Unixish special-casing is done in Python since then. Best advice is to start with a bug report on Python's bug tracker, and perhaps a msg to mailto:[EMAIL PROTECTED] I think Martin v. Löwis is currently most knowledgeable about messy config issues in Python. It seems that the patch should only be activated for LinuxThreads, and should be tested for in configure. Sounds plausible, but I wouldn't know. Is it reasonable to test for a LinuxThreads specific function (like pthread_kill_other_threads_np). Should I create a functional test that test tries to cause the LinuxThread specific behavior (cause a deadlock) and the notice the problem and fix it.Should I use the glibc feature getconf GNU_LIBPTHREAD_VERSION? I don't know what's available in LinuxThreads *to* test. Most packages have some God-awful preprocessor #define to key off of. Also don't know whether the specific breakage at issue here is unique to LinuxThreads. The first is easiest to test for, but seems a little error prone. (what if someone else adds the non-standard function in order to ease porting from Linux? What if someone comes up with a LinuxThreads update that solves this problem?) Its testing a feature that is related to the feature I want info for, but not the troublesome behavior itself. I expect that's why most people settle for testing a package-specific #define. It's also why there's always at least some resistance to patches that do key off goofy symbols: the #ifdef'ed code will probably remain there forever, regardless of whether the problem still exists. So: The second solution seems to be one step away from the halting problem (although it might be able to be done with block signal_a, send signal_a, send signal_b, if signal_b is caught but not signal_a, then signals are not rerouted across threads.) An autoconf-able test that checks for the actual bad behavior would be best. ... Is it reasonable to put a LinuxThreads specific replacement SET_THREAD_SIGMASK in thread_pthread.h? Yes. There are already a slew of system specific defines, and the differences don't seem extreme enough to make a separate thread_linuxthreads.h Fully agreed. LinuxThreads is primarily pthreads with a bug. That makes it qualitatively the same as all other pthreads implementations wink. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Segfault and Deadlock
As Andrew Langmead has already discovered, the LinuxThreads issue with SIGSEGV was reported on the Python bug tracker almost a year ago (well, reported, but not diagnosed): SIGSEGV causes hung threads (Linux) http://www.python.org/sf/756924 Looks like: can't CNTRL-C when running os.system in a thread http://www.python.org/sf/756940 is related. python-dev'ers, do we have a release manager for 2.3.4 (I didn't see a resolution to the brouhaha at the end of March)? If so, is 2.3.4 still planned for this month? tick-tock-tick-tock-ing-ly y'rs - tim ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Subversion
[Chris Withers] ... Is it worth asking on the SVN lists how hard this would be to implement? I mean, we have the svn:ignore property, and we have the svn:eol-style property, what we want is a combination of those two, how hard can it be? 0.5 wink I don't think we want a combination of the two: like .cvsignore, svn:ignore only applies to the contents of a single directory (the one on which svn:ignore is set). We'd really like something global (e.g., .py files are text files throughout all of Zope's code). The svn config file is global (or global enough wink). Jim and I bugged the Subversion folks last week. Ben Collins-Sussman replied (in part): http://www.contactor.se/~dast/svnusers/archive-2004-04/1373.shtml At this point, the user community has made it absolutely clear that we really need a repository-side configuration which somehow 'broadcasts' runtime configuration options to clients, such as auto-props. Yes, it's burdensome to force every client to hand- maintain auto-props. For now, you'll need to choose between that burden, or the burden of occasionally fixing a mistake when someone forgets to activate native eol style. There's another difference between cvs and svn here: if you forget to mark a file as binary in cvs, it can be hard to recover from. If a Windows user checked it in, the file in the repository may be corrupt. If a Linux user checked it in, and a Windows user grabbed it before -kb was added, the Windows copy may be corrupt, and cvs won't repair that by magic when -kb is added (a cvs update doesn't consider the Windows working copy to be out of date). svn's story is much better (perfect, in fact) when forgetting to add eol-style: regardless of which kind of platform did the commit, the property can be added after the fact by anyone, and svn will automatically repair working copies on all platforms. Because (most) svn properties are versioned, adding eol-style is enough to convince svn that pre-eol-style copies are out of date. Nobody even needs to bother running dos2unix or unix2dos; just adding the property is enough (and when the person who adds the property commits the change, svn fiddles the line ends on their working copy (if needed) by magic too). ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Re: Nightly Zope 3 Binary Compiles for WIndows
[Jim] ... If they have the .pyd files, they can run Zope and fix problems they find in the .py files. They have the .pyd files now. I'm keeping them up to date, and Max M wrote a clear howto (which references the .pyd Zip files on zope.org): http://www.mxm.dk/papers/run-z3-cvs-wthout-compiler/ easier-to-do-it-than-argue-about-it0.7-wink-ly y'rs - tim ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Re: [Collector] Strange reject policy
[Ken Manheimer] All the actions are verbs, won't fix is not a verb. Can you suggest a verb the more clearly indicates the result is won't fix? Sorry, I got lost on the first sentence: what difference does it make to anything whether they're verbs, adjectives, a mix, ...? They're all just cryptically abbreviated answers to the question what do you want to do with this report?. We want to resolve it. Sorry, we want to reject it. Sorry, but while it is arguably a bug, we won't fix it. If someone claims they can't understand what won't fix means, I'll be sympathetic. Until then, won't fix sounds perfect to me. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] RE: [Python 2.3.4c1] nasty LinuxThread problem not solved
[Dieter Maurer] I just checked that Python 2.3.4c1 does not yet fix our LinuxThread-Crash problem -- the problem that lets a multi-threaded application enter a curious state when one on the threads crashes. I'm sorry to report that 2.3.4 final won't fix it either. The active Python bug report is here: [ 756924 ] SIGSEGV causes hung threads (Linux) http://www.python.org/sf/756924 And this despite the patch for this problem in Python's collector... There's a patch that squashes the specific symptom you have in mind, but at the cost of other breakage -- the original patch was added for a reason too. Read the bug report. It's complicated, and too complicated to slam in any change here for 2.3.4 (according to Python's release manager, and according to Guido). Unfortunately, I am no python-dev subscriber. Can you check please whether this patch has a chance to become part of the official Python 2.3.4. Doesn't look like it. If not, we would be forced to maintain our own Python version as Python's current crashing behaviour with LinuxThreads is unacceptable. Please add your concerns to the bug report. There are still open issues in coming up with a correct patch, but if those are resolved and enough people care, it should be enough to justify a 2.3.5 release. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] http://collector.zope.org/Zope/1347 - Zeo help
[Laurence] Sorry if this is a faq or otherwise covered elsewhere, but I've encountered quite a big problem with Zeo in my environment. I'm running FreeBSD 5.2.1 (Current as of last night) with Zope 2.7.0, Postgresql 7.4.2, Python 2.3.3 and psycopg-1.1.13. We have been developing with Zope using a single Zope server without Zeo and the time have come to migrate onto Zeo. Everything seemed to come accross smoothly (We brought up the new server in a single Data.fs config) and everything seemed fine so we moved onto setting up Zeo with the Data.fs. It seems that if we nest ZPT's more then three deep (ZPTA includes ZPTB which includes ZPTC) we get a seg fault and the Zope process dies (The Zeo server is fine). If we load ZPTB or comment out the inclusion of ZPTC everything works fine. It's when we encounter 3 or more layers that the server seg faults. 3 deep blows up, 2 deep doesn't sure makes it sound like the notorious FreeBSD allocates tiny stacks for threads business. Google on freebsd zope stack for many hits on this. Haven't used FreeBSD myself, and don't know what the current favored way to increase the thread stack size on FreeBSD might be. BTW, using ZEO increases the stack space you need, to deal with the extra (compared to not using ZEO) layers of across-the-network communication needed. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] RE: [Python 2.3.4c1] nasty LinuxThread problem not solved
[Tim Petere] There's a patch that squashes the specific symptom you have in mind, but at the cost of other breakage -- the original patch was added for a reason too. [Dieter Maurer] I verified that http://sourceforge.net/tracker/index.php?func=detailaid=949332group_id=54 70atid=305470 indeed fixes the problem. It fixes a problem, yes, and let's be clear that it does do by avoiding provoking a bug in LinuxThreads. If LinuxThreads had a POSIX-compliant implementation of signals, this discussion wouldn't be happening. It might introduce other subtle problems but at least none that are revealed by Python's regression test suite... The comments on the bug report are extensive: [ 756924 ] SIGSEGV causes hung threads (Linux) http://www.python.org/sf/756924 Guido applied Jason Lowe's original signal-blocking patch because he was persuaded it fixed significant thread problems at the time. Everyone (including Jason) now agrees that patch was too extreme, but the platform problems it intended to address still exist. It's certainly true that Python's test suite doesn't cover all endcase threading+signal interaction behaviors across dozens of incompatible thread implementations, and many such problems are exposed by GNU readline, which is plain difficult to test except interactively. So it goes. Moreover, I doubt that such problems will be significant in practise: As above, the original signal-blocking patch was added for reasons in practice that appeared sufficient at the time. If you want to argue that, the right (helpful) place to do so is in a comment attached to the bug report. ... As an automatic restart after a crash is vital for our productive Zope2 installations, we will probably bite the bullet and maintain our own Python version. The 2.3.4 release manager rejected any change in this area for 2.3.4, and Guido agreed with that decision. 2.3.4 is just days away now, and there are several issues on several quite different platforms that need to be addressed simultaneously. Talking to them (via the Python bug report) may change their minds, but (a) I doubt it, and (b) nobody on zope-dev can change this. Adding comments to the bug report will still help to get it resolved for 2.3.5. An alternative to maintaining your own Python, and/or your own Linux, is to move to the current Linux thread implementation (NPTL), which doesn't have the LinuxThread signal bug that's the deeper cause of Zope's problems (on Linux boxes using LinuxThreads). ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] RE: [Python 2.3.4c1] nasty LinuxThread problem not solved
[Dieter Maurer] You have snipped my explanation why I am convinced that the patch can only improve things! Yes, because zope-dev isn't a useful place to discuss this complicated Python issue. If you missed it the first two times wink, let me suggest again that you add your comments to the bug report: that's the only place where the people fixing this problem, and the people making the release decisions, will see what you have to say (I have no say in what happens here, and I'm not working on resolving this issue either -- all I did is agitate to do something for 2.3.4, but I lost that battle). I have not argued that there was no case to block *some* signals, As the discussion in the bug report makes clearer, it's unclear whether Python should be blocking any signals anywhere. just not the ones that the operating system uses to signal major problems -- SIGSEGV, SIGBUS, ... and friends. The patch states that the pthreads standard says that such signals should not be blocked. This is a Python issue independent of the bug in LinuxThreads. Python has its own threading model, which has to work across several thread implementations besides just pthreads. It doesn't *intend* to mimic the native platform thread gimmicks, it has to build on them. If LinuxThreads handled signals correctly (according to the pthreads standard), there wouldn't be a problem here. Our system administrators have been sceptical to switch to NPTL support. They say, there are still some problems about it. I don't know, but most things I've read about the *current* NPTL are very positive (orders of magnitude faster in some stress tests than LinuxThreads, and much better conformance to the standard). Earlier versions of NPTL got worse press. I will reraise the question and see what my colleagues feel as the less problematic way: use NPTL or our own Python version. It would sure help if people running Zope on an NPTL system spoke up here! ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] RE: [Python 2.3.4c1] nasty LinuxThread problem notsolved
[Chris McDonough] I run Zope on an NPTL system (Fedora Core 1), but it's just a development box; as a result it doesn't get much load and doesn't stay up indefinitely. It's better than knowing nothing wink. If that's the box you do overnight test runs on too, that's also interesting, since you frequently see failures in certain ZEO tests that are rarely (if ever) seen elsewhere. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] cvs down ??
[robert rottermann] I try to access the Zope cvs anonymously according to the instructions. However it fails. Has anything changed, or is this temporarily. :pserver: access is disabled for now, because of recently announced security holes. It will be enabled again when (and if?) CVS is repaired and installed. [EMAIL PROTECTED]:/home/zope/Products cvs -d :pserver:[EMAIL PROTECTED]:/cvs-repository co Zope cvs [checkout aborted]: connect to cvs.zope.org:2401 failed: Connection Refused Maybe better news: if you want Zope HEAD, that's been moved from CVS to SVN. Install an svn client, and do svn co svn://svn.zope.org/repos/main/Zope/trunk Zope ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] cvs down ??
[EMAIL PROTECTED] thanks for your answer, what I would like to download is CMFSetup I think that's still in CVS only, and so long as :pserver: access remains disabled you simply can't get it that way. You could go to http://cvs.zope.org/CMF/CMFSetup/ and use the Download tarball link near the bottom, though. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Help: __getstate__ overriding
[Syver Enstad]
I have a Persistent derived class where I want to upgrade from using a
PersistentList to a BTrees.IOBTree.IOBTree.
_articleList is the old Persistent list
_oidsToArticles is the new IOBTree.
def __setstate__(self, state):
articleList = state.get('_articleList')
if articleList:
oidsToArticles = self.makeBTree()
for each in articleList:
oidsToArticles[each.oid()] = each
del state['_articleList']
state['_oidsToArticles'] = oidsToArticles
Persistent.__setstate__(self, state)
This seems to work fine, and I am adding more objects to
_oidsToArticles. The problem is that when I commit the transaction,
nothing is saved.
Quick guess (untested, untried, just from eyeballing this snippet quickly):
nothing is marking self itself as being changed. The only things getting
changed are the throw-away in-memory 'state' dict, the throw-away in-memory
self.__dict__, and a throw-away in-memory BTree. The primary purpose of
p.__setstate__ is to activate a ghost, after which point p is in the
up-to-date state (i.e., not changed -- calling Persistent.__setstate__(self,
...) isn't going to mark self as changed, and nothing in the code above
modifies self itself). If self doesn't look changed, its state in the DB
won't change during commit(), and the DB will still reference your old
PersistentList.
Perhaps shuffling the code around would work, a la:
Persistent.__setstate__(self, state)
articleList = state.get('_articleList')
if articleList is not None:
# make the BTree in local oidsToArticles as above, then
del self._articlelist
self._oidsToArticles = oidsToArticles
The thinking there is that then you've truly modified self, after self has
been activated.
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Help: __getstate__ overriding
[Syver Enstad, wants to switch an attribute of a Persitent subclass
From PersistentList to an IOBTree]
[Tim, guessing]
Quick guess (untested, untried, ...
...
Perhaps shuffling the code around would work, a la:
Persistent.__setstate__(self, state)
articleList = state.get('_articleList')
if articleList is not None:
# make the BTree in local oidsToArticles as above, then
del self._articlelist
self._oidsToArticles = oidsToArticles
The thinking there is that then you've truly modified self, after self has
been activated.
Nope, sorry, not a chance. You have truly modified self then, but
__setstate__ is called by magic as part of activation (unghostifying), and
the activation machinery resets self._p_changed after it calls __setstate__.
So same result as your code: the in-memory version of self has the BTree,
but commit() doesn't change anything about self in the database (again
because self._p_changed is false -- and will be no matter what you try to do
in __setstate__()).
This appears to be one of those severely underdocumented minefields. The
best older thread I found is here:
http://mail.zope.org/pipermail/zodb-dev/2002-March/002442.html
but it doesn't actually spell out something that works in the way you
want.
One possibility is to use any of these __setstate__ implementations
temporarily, in a one-shot database conversion script: load every object of
your subclass's type, and for each one obj, after loading it do
obj._p_changed = True
get_transaction().commit()
Note again that setting _p_changed *inside* __setstate__ won't do any good.
Note too that Jim Fulton has a recent proposal for Zope3 in this area:
http://tinyurl.com/ypkhk
[Zope URLs are generally so long my mailer refuses to keep them on one line]
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Help: __getstate__ overriding
[Dieter Maurer] I think, this is a ZODB buglet: It should set _p_changed = 0 before it calls __setstate__ and not afterwards... I don't know; Jim (or Jeremy) may know the reasoning here, but I don't. Activation currently sets the state to changed *before* calling __setstate__ too. A comment says this is to prevent recursively calling _PyPersist_Load, and that makes sense to me, else unbounded recursion could occur. Given this abuse of _p_changed, and that the purpose of activating a ghost is to transition it to the up-to-date state, making mutations to the persistent state inside __setstate__ seems hard to accommodate. On ZODB head, Jeremy also documented that __setstate__ can't affect the persistent state (in persistent/interfaces.py's IPersistent). Whether this is deliberate design, or a consequence of the current implementation, I can't say. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Help: __getstate__ overriding
[Tim Peters] ... On ZODB head, Jeremy also documented that __setstate__ can't affect the persistent state (in persistent/interfaces.py's IPersistent). Whether this is deliberate design, or a consequence of the current implementation, I can't say. [Dieter Maurer] A potential problem: it *can* change the persistent state but it does not do it reliably. It does precisely when the object is later modified again: in this case modifications made in __setstate__ become persistent, otherwise, they do not. I agree, but the docs are cryptic (because so brief) enough that I can read them as being correct or incorrect: def __setstate__(state): Set the object state data Note that this does not affect the object's persistent state. There are enough ambiguities there to fill a whole thread wink. It is true that a call to __setstate__() alone doesn't affect the object's state in the DB. It needs more words. OTOH, I'd rather dream up a way to make object changes made in __setstate__() persistent, if desired. For example, __setstate__'s return value is ignored now. Perhaps we could say that if it returns True, then the object should be considered to be in the changed state rather than in the uptodate state. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Help: __getstate__ overriding
[Syver Enstad] I was aware that __setstate__ doesn't allow me to commit my changes after only loading the object into memory (__setstate__ is called). I may have explained myself unclearly (not a native english speaker/writer), I don't think that matters much: English instead of code is always ambiguous, even for native writers. the problem is that it won't save my changes when I add more items to the _oidsToArticles BTree long after __setstate__ time. Right. I tried to explain that in my first reply. It's expected. You have to get _p_changed set on your *object*. It doesn't matter how much you mutate new objects attached *to* the object you loaded, you have to get _p_changed set to 1 on the original object. Example code: # articleDb is an ArticleDb and has the __setstate__ method articleDb = connection.root()['articledb'] At this point articleDb._p_changed is not 1, and nothing you do later changes this fact. for each in articleDb.articles() # loop through all article print each It's obvious that this doesn't set articleDb._p_changed, right? computer = Computer(1030) articleDb.addArticle(computer) That adds something to the BTree that is bound to an attribute of articleDb, but still doesn't set _p_changed on articleDb (mutating the BTree does not mutate the object containing the BTree). So far as the *database* is concerned, there's still no path from the root object to the BTree your __setstate__() created, so the BTree never gets committed. If you do articleDb._p_changed = 1 at this point too, then your changes will get stored. connection.commit() # nothing is saved Right. As far as the persistence machinery is concerned, articleDb itself never changed, so there was no reason to store it. Since it didn't get stored, the BTree you attached to it in __setstate__ doesn't get stored either. Everything works smoothly until connection.commit(). The __setstate__ method correctly converts to using the BTree and the addArticle method adds a new computer to ArticleDb. The problem is that, while adding the BTree to articleDb did change (did set _p_changed to 1) articleDb, because that happened as a side effect of loading the object, the unghostification machinery clears _p_changed after __setstate__ returns. Then nothing you did after that set articleDb._p_changed to a true value. If I instead create a new ArticleDb instance with the BTree instead of using __setstate__ the new computer is saved as it should. Yes. If you had called __setstate__ *directly*, it would also work (contradicting the current minimal docs, but so it goes). The problem is that unghostifying by magic clears _p_changed. In none of the cases is articleDb._p_changed == True, When you create an object directly, it's not associated with a Connection (a jar), and some of the persistence machinery is sidestepped then. New objects only get into the database if you attach them to an object that's already in the database, and the latter object is marked changed. Think about that wink, and the purpose of the root() object will become clearer. but when the ArticleDb is created cleanly instead of upgraded, commit works anyway. Sorry, I'm not clear on what that means. In any case, it doesn't really matter how an object gets created, what matters is whether the persistence machinery sees a path to it from a changed object already in the database. Since this works if the object that holds _oidsToArticles is created from scratch, it seems that ZODB isn't properly aware of the _oidsToArticles attribute since it was created in __setstate__ instead of in __init__. Both ways set _p_changed *at the time* oidsToArticles is bound to the new BTree. The difference is that uhghostification deliberately clears _p_changed after calling __setstate__. I would really like to know exactly what happens here as the reason I am utilizing an object database is to be able to refactor the database with a minimum of fuss, and I would expect __setstate__ to allow me to make additional attributes in my instances. Calling __setstate__ as a side effect of a persistent load does not, by itself, support making changes that persist. It apparently wasn't designed to, either (see the other msgs in this thread, particularly from ChrisM). ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] More Transience weirdness in 2.7.1b1
[Steve Jibson] It seems that there may still be some strangeness with Transience. We're running Zope-2.7.1b1 (Python 2.3.3, Fedora Core 1) ... So, I hook-up to the ZMI and click on *temp_folder*, then click on *session_data (Session Data Container)* and I get the following error: Traceback (innermost last): Module ZPublisher.Publish, line 101, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module App.special_dtml, line 62, in __call__ Module DocumentTemplate.DT_String, line 474, in __call__ Module Products.Transience.Transience, line 709, in nudge Module Products.Transience.Transience, line 555, in _gc RuntimeError: the bucket being iterated changed size FWIW, the RuntimeError is raised in the bowels of the BTree implementation, and can occur if someone performs a size-changing mutation on a BTree while iterating over the BTree. A discussion, and concrete example, can be found in the Iteration and Mutation section (5.3.2) here: http://zope.org/Wikis/ZODB/FrontPage/guide/node6.html ChrisM will be familiar with the cause wink. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Remove BDBStorage from 2.7 branch as well?
[Stefan H. Holek] As BDBStorage has been removed from HEAD quite some time ago, I was wondering whether it would be possible/advisable to remove it from 2.7 branch as well. Would it? I think it would, because there's no resource to maintain it. BDBStorage was released as an experiment: Zope Corp doesn't have the internal resource to make it production-quality, and we knew that. The experiment consisted of releasing the code anyway, to see whether the community wanted it enough to complete it. That didn't happen, so it's a mass of dead code now. OTOH, removing BDBStorage isn't entirely straightforward, as tendrils reach into everything from setup to dark corners of the test suite, and BDBStorage's primary author has a different job now (i.e., the person best able to remove it correctly and completely isn't available to do so). That said, the BDB-based storages remain a great *idea*, and the code is in good shape. There's still faint hope in my heart that someone will adopt the code and make it a production-quality option. Realistically, that's not going to happen, in which case the BDB code is an attractive nuisance everywhere it remains. In all, it's easiest now to leave it in 2.7; it will be gone in 2.8 regardless. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Re: status of Zope versus zope?
[Martijn Faassen] Yes, I'm doing something much like that and it works fine on Linux. I'm just wondering how this de-confuses Python's import system on a case-insensitive file system. Python imports are case-sensitive on all platforms (this wasn't always true, but is true in recent Pythons). What's impossible in a case-insensitive filesystem is having two things (files or directories) in the same directory whose names differ only in case. I'm not sure that answers your question, but since it's the only answer there is, you'll have to adjust your question to match wink. To get concrete, if sys.path is ['/A', '/B'], files /A/Hello.py and /B/HellO.py exist, A and B are otherwise empty, and you do import HellO then /B/HellO.py will get loaded, Windows or not, and import hello will fail, also Windows or not. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Prohibiting Spams on the List
[Mohsen Moeeni] Fristly excuse me for cross-posting. Actually, I did not like to be the person who comes this up however the amount of spams which is propagated thru the lists is annoying. I wonder why the guys at Zope Inc. does not tweak the list settings so posting is only allowed by the members. Perhaps just the weight of history, and/or trying to keep the barriers to participation as low as possible. Speaking as zodb-dev's list admin, I'd be happy to restrict posting to members, *if* people don't object to the idea. I spend an increasingly resented part of each day doing list-admin triage (for zodb-dev and about 6 other mailing lists). A predictable irritation is that Mailman's answer to member? is based on the posting's sender's email address, but many people post from more than one account. If you want the default action for non-member posts to be hold for review, I won't do it -- I can't afford the additional time sink. reject is the no-manual-intervention default action, but it would place a new burden on members posting from non-member accounts (their posts would be rejected, at least until they asked the list admin to add their other posting accounts to Mailman's list of non-members to accept -- and they would need to do that on a per-list basis). ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope] RE: [Zope-dev] Prohibiting Spams on the List
[Philip Kilner] ... Does Mailman have a nomail facility that subscribers could set /themselves/ on secondary accounts? There are different versions of Mailman (of course), but at least recent ones offer this. If you go to your personal list subscription page on the web, a checkbox to suspend delivery is the first of the listed checkbox options. That seems a reasonable workaround to me. As to why this comes up, I expect my own case is typical: I have 6 POP3 accounts with various providers. I can retrieve email from all of them when logged in with any of the providers, but SMTP access (for sending email) is often limited to the specific account I'm logged into. For example, I cannot use my Zope Corp SMTP account, or my Comcast SMTP account, when logged in to my MSN account from a dialup line in a hotel room. It doesn't matter whose fault that is (although I suspect it's MSN's ...), it's just the fact of the matter. I don't have dialup facilities thru Zope Corp or Comcast, so when I'm on the road and restricted to dialup access, I send mail via MSN or Hotmail, or not at all. I'm told that posting from my Hotmail account doesn't project a professional image wink. The same kind of thing happens on the one list I admin where non-member posts are held for review (this is the PSF Board mailing list, where non-member posts can be legitimate, e.g. a company asking about becoming a PSF sponsor). There are fewer than a dozen members on that list, and the legit traffic is usually light, so accommodating two or ten email addresses per member was a finite task there. In gentler times, your concern would be wonderful - in the current climate, you'll drown if you can't automate it. Save yourself, while you can! ;-) If there's spam on Zope's mailing lists, I never see it(*), so my only stake in this is in a list-admin capacity. (*) I never see it because I started this project -- and it works wink: http://www.spambayes.org/ ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] zope-dev list policies
Over on the zope and zope-dev lists, there's currently agitation to make them members-only mailing lists. The point is that spam could not get thru then (unless posted by a member). What would zodb-dev members like? Posting by a list member would not be affected, unless you attempted to send a message from an email account other than one you subscribed with. In the latter case, your message would be bounced back to you. You could worm around that by subscribing to zodb-dev with that address too, then going to your list membership page on the web and checking the box to suspend email delivery on that account. Then you could post using that account too, but wouldn't also get zodb-dev email on that account. I'm the list admin for zodb-dev, and don't have a preference. If you do, and it's strong enough to scream it, shout. The loudest screamer will win wink. By default, I won't change the current policy (anyone can post here, member or not). ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope.Com Geeks] Re: [Zope-dev] zope-dev list policies
[Ken Manheimer] I noticed this when it went initially went by, but didn't have time to follow up. The upshot is that there is absolutely no way *under the current arrangement* that this is going to happen. I can see a way to swing it, requiring earnest volunteer effort. Here are the details. I think you have something different in mind than was being discussed. Members only comes in several flavors. You seem to have the ... and non-member posts are held for moderator review flavor in mind. That wasn't suggested. Two other flavors were: - ... and non-member posts are rejected. No messages are held for moderator review then. A would-be poster with a legitimate email address gets an auto-generated rejection reply msg. Since most rejection msgs would go to bogus addresses on spam and virus email, m.z.o gets another bounce back for most attempts to send a rejection reply. - ... and non-member posts are discarded. No messages are held for moderator review then. Non-member posts go to the bit bucket, without comment or recourse. Being the administrator of many of the zope lists (probably over ten and below twenty), i am already dismayed by the challenge of the typically thirty to one hundred held spam messages, bounces, and other effluvia i have to handle *per day*. I do not know how many of the legitimate list messages would additionally be held and require more attention (with the current mailman implementation, it takes a lot more fuss to approve a held message than to discard it), but the load is already untenable, so one more is too many. There is an option, however. It's possible to add moderators to lists, separate from list administration privileges. I would be willing to set the lists to hold non-member postings, *if* there were volunteer moderators that would actually take care of some significant portion of the load - ie, i would not have to approve one non-member (alternate address) posting. (I would not mind occasionally approving a non-member/alt-addr posting if the volunteers reduced the spam/bounce handling efforts in the process.) In my (limited but real wink) experience, this doesn't work. Without a single clear owner, postings held for review eventually grow to unmanageable bulk. Nobody enjoys the moderation task, it does consume time, and when there are multiple moderators they all eventually reach a point of believing that someone else can handle it for a while. After a few days go by like that, a co-moderator who is able to make some time logs in and finds such a backlog that they decide they have more urgent work to attend to. Then it snowballs out of control. We had a clear example of this about a month ago, when the backlog of python-help messages waiting for review reached thousands. At that point the only realistic option was to discard all of them, effectively making python-help the ... and non-member posts are discarded list flavor. The only ... and non-member posts are held for review list I moderate that works is the PSF Board mailing list. That works because I'm the only moderator, legit traffic on it is very light, and I know enough Visual Basic to automate the reject/approve process without leaving Outlook wink. ... That's the situation. Are there people that would be willing to volunteer for moderation duties? (Say which lists when you reply - and make sure to cc me directly, since i can't read most of the lists i moderate.) The rub is that mailing lists are open 24 hours per day, 365.2425 days per year, and virus/spam traffic keeps increasing. Good intentions get crushed by that reality. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope.Com Geeks] Re: [Zope-dev] zope-dev list policies
[Ken Manheimer] In either mode, essentially, list members would be able to get postings to the list only from their registered account. Or accounts. When I've faced a list like this as a user, I've subscribed multiple times, once from each account I'm likely to post from, but set the no delivery option on all but the primary (IMO) subscription. This is something users can do on their own. I don't have a confident guess about whether that would be prohibitive to any or many. I suppose we could try it and see whether how it sits with people. So far, the people who post from only one account have insisted it won't make real trouble for anyone wink. I don't know how it works for people posting from gmane. There's also the incidental considerations - both modes have drawbacks. Yes, but all modes have drawbacks, including the status quo. ... So i could see giving a try to non-member-posts-rejected, That would actually lighten *our* (list admin) loads. Nobody is sympathetic to that except us, of course. Even with the current open list policy, I still get a ridiculous number of messages held for review. if the membership thinks the added inconvenience is worth the reduced spam. There won't be consensus on this. Votes on zodb-dev have been about evenly split, and I don't expect that will change. I have the impression, though, that the spam on most of the high-traffic zope.org maillists is relatively low-proportion. Am i mistaken? No idea -- my personal SB filter spares me from seeing almost all junk email from all sources, and I get a few hundred of them per day. By far the biggest source is bounces to [EMAIL PROTECTED], due to viruses and spam forged to appear as if sent from [EMAIL PROTECTED] Compared to that load, every other source is in the noise for me. ... Reject (actually, discard) is pretty easy - you just have to reply to a particular attachment in the held-message notice. Approval is the same process (at least under a current-enough Mailman), except you need to put an Approved: list password line in the headers, or as the body of the reply. My VBA code knows the list passwords for the various mailing lists, and discard/approve is just a matter of clicking a button for me (one button for discard, another for approve). It still takes real time to open and review the messages Mailman is asking about, though. (I **wish** the confirmation message for the discard would indicate that a discard happened - instead, it says Confirmation succeeded, which is nearly worse than no feedback at all, ... Even worse, it says exactly the same thing for an approved message. But wouldn't Barry be a better recipient for this rant wink? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] RE: [ZODB-Dev] [Warning] Zope/ZEO clients: subprocesses can lead tonon-deterministic message loss
[Dieter Maurer] ATTENTION: Crosspost -- Reply-To set to '[EMAIL PROTECTED]' Which I've honored. Today, I hit a nasty error. The error affects applications under Unix (and maybe Windows) which * use an asyncore mainloop thread (and maybe other asyncore applications) Zope and many ZEO clients belong to this class Note a possible complication: ZEO monkey-patches asyncore, replacing its loop() function with one of its own. This is done in ZODB's ThreadedAsync/LoopCallback.py. and * create subprocesses (via fork and system, popen or friends if they use fork internally (they do under Unix but I think not under Windows)). It may be an issue under Cygwin, but not under native Windows, which supports no way to clone a process; file descriptors may get inherited by child processes on Windows, but no code runs by magic. The error can cause non-deterministic loss of messages (HTTP requests, ZEO server responses, ...) destined for the parent process. It also can cause the same output to be send several times over sockets. The error is explained as follows: asyncore maintains a map from file descriptors to handlers. The asyncore main loop waits for any file descriptor to become active and then calls the corresponding handler. There's a key related point, though: asyncore.loop() terminates if it sees that the map has become empty. This appears to have consequences for the correctness of workarounds. For example, this is Python's current asyncore loop (the monkey-patched one ZEO installs is similar in this respect): def loop(timeout=30.0, use_poll=False, map=None): if map is None: map = socket_map if use_poll and hasattr(select, 'poll'): poll_fun = poll2 else: poll_fun = poll while map: poll_fun(timeout, map) If map becomes empty, loop() exits. When a process forks the complete state, including file descriptors, threads and memory state is copied and the new process executes in this copied state. We now have 2 asyncore threads waiting for the same events. Sam Rushing created asyncore as an alternative to threaded approaches; mixing asyncore with threads is a nightmare; throwing forks into the pot too is a good working definition of hell wink. File descriptors are shared between parent and child. When the child reads from a file descriptor from its parent, it steals the corresponding message: the message will not reach the parent. While file descriptors are shared, memory state is separate. Therefore, pending writes can be performed by both parent and child -- leading to duplicate writes to the same file descriptor. A workaround it to deactivate asyncore before forking (or system, popen, ...) and reactivate it afterwards: as exemplified in the following code: from asyncore import socket_map saved_socket_map = socket_map.copy() socket_map.clear() # deactivate asyncore As noted above, this may (or may not) cause asyncore.loop() to plain stop, in parent and/or in child process. If there aren't multiple threads, it's safe, but presumably you have multiple threads in mind, in which case behavior seems unpredictable (will the parent process's thread running asyncore.loop() notice that the map has become empty before the code below populates the map again? asyncore.loop() will or won't stop in the parent depending on that timing accident). pid = None try: pid = fork() if (pid == 0): # child # ... finally: if pid != 0: socket_map.update(saved_socket_map) # reactivate asyncore Another approach I've seen is to skip mucking with socket_map directly, and call asyncore.close_all() first thing in the child process. Of course that's vulnerable to vagaries of thread scheduling too, if asyncore is running in a thread other than the one doing the fork() call. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] RE: [ZODB-Dev] [Warning] Zope/ZEO clients: subprocesses can lead tonon-deterministic message loss
[sathya] so can we safely assume that zeo does not mix the asyncore implementation with forks or threads and hence does not suffer from the child concurrently operating on sockets along with parent syndrome that dieter is experiencing ? appreciate any clarifications. It's normal for a ZEO application to run asyncore in its own thread. I don't really understand what Dieter is seeing, though: [Dieter] When a process forks the complete state, including file descriptors, threads and memory state is copied and the new process executes in this copied state. We now have 2 asyncore threads waiting for the same events. A problem is that it's *not* the case that a POSIX fork() clones all threads. Only the thread calling fork() exists in the child process. There's a brief but clear discussion of that here: http://www.opengroup.org/onlinepubs/009695399/functions/fork.html POSIX doesn't even have a way to *ask* that all threads be duplicated, for reasons explained there. Last I heard, Dieter was running LinuxThreads, which fail to meet the POSIX thread spec in several respects. But, AFAICT, fork() under LinuxThreads is the same as POSIX in this particular respect (since threads are distinct processes under LinuxThreads, it would be bizarre if a fork() cloned multiple processes!). I believe native Solaris threads act as Dieter describes, though (fork() clones all native Solaris threads). Dieter, can you clarify which OS(es) and thread package(s) you're using here? Do the things you're doing that call fork() (directly or indirectly) actually run from the thread running asyncore.loop()? That's the only way a POSIX fork() should end up with a clone of the thread running the asyncore loop. But then the subsequent exec (if you're doing system() or popen()) should wipe out the cloned asyncore code before the child process returns to asyncore. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: RE: [Zope-dev] RE: [ZODB-Dev] [Warning] Zope/ZEO clients: subprocesses can lead tonon-deterministic message loss
[Dieter Maurer] The problem occured in a ZEO client which called asyncore.poll in the forked subprocess. This poll deterministically stole ZEO server invalidation messages from the parent. I'm sorry, but this is still too vague to guess what happened. - Which operating system was in use? - Which thread package? - In the ZEO client that called fork(), did it call fork() directly, or indirectly as the result of a system() or popen() call? Or what? I'd like to understand a specific failure before rushing to generalization. - In the ZEO client that called fork() (whether directly or indirectly), was fork called *from* the thread running ZEO's asyncore loop, or from a different thread? I read the Linux fork manual page and found: fork creates a child process that differs from the parent process only in its PID and PPID, and in the fact that resource utilizations are set to 0. File locks and pending signals are not inherited. ... The fork call conforms to SVr4, SVID, POSIX, X/OPEN, BSD 4.3 If it conforms to POSIX (as it says it does), then fork() also has to satisfy the huge list of requirements I referenced before: http://www.opengroup.org/onlinepubs/009695399/functions/fork.html That page is the current POSIX spec for fork(). I concluded that if the only difference is in the PID/PPID and resource utilizations, there is no difference in the threads between parent and child. Except that if you're running non-POSIX LinuxThreads, a thread *is* a process (there's a one-to-one relationship under LinuxThreads, not the many-to-one relationship in POSIX), in which case no difference in threads is trivially true. This would mean that the wide spread asyncore.mainloop threads could suffer the same message loss and message duplication. That's why all sane wink threading implementations do what POSIX does on a fork(). fork() and threading don't really mix well under POSIX either, but the fork+exec model for starting a new process is an historical burden that bristles with subtle problems in a multithreaded world; POSIX introduced posix_spawn() and posix_spawnp() for sane(r) process creation, ironically moving closer to what most non-Unix systems have always done to create a new process. I did not observe a message loss/duplication in any application with an asyncore.mainloop thread. I don't understand. You said that you *have* seen message loss/duplication in a ZEO client, and I assume the ZEO client was running an asyncore thread. If so, then you have seen loss/duplication in an application with an asyncore thread. Or are you saying that you haven't seen loss/duplication under the specific Linux flavor whose man page you quoted, but have seen it under some other (so far unidentified) system? Maybe, the Linux fork manual page is only not precise with respect to threads and the problem does not occur in applications with a standard asyncore.mainloop thread. That fork manpage is clearly missing a mountain of crucial details (or it's not telling the truth about being POSIX-compliant). fork() is historically poorly documented, though. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] [Warning] Zope/ZEO clients: subprocesses can lead tonon-deterministic message loss:MAYBE NOT
[sathya] thanks for the clarification below and also the pointers to the posix behaviour of fork. The Warning about Zope/ZEO clients in the subject line certainly caused some alarm bells to go off. I am assuming now that dieters description below of using forks does not gel with the ZOPE/ZEO process model i.e there are no forks being called within the code to cause the asyncore thread to be cloned I'm still not clear on either exactly what Dieter did, or on what kind of system. In any case, he wasn't talking about forks in Zope/ZEO directly, but about forks in *application* code. The presence (or absence) of those depends on what applications do, not on what Zope/ZEO do. ( even if one were using a non posix compliant thread lib like native solaris it would still be a non issue ). I don't think we know enough yet to say. I've never used native Solaris threads, and they're so different from POSIX threads in this respect that I'm not going to guess about how ugly life can be with them. I'm sure there's no problem here with native Windows threads. I don't *see* a way for there to be a problem under POSIX thread semantics, but then I'm still guessing too much about what Dieter actually did. If I were you, I'd ignore this thread until it reaches a conclusion -- if there were widespread problems here, we would have heard about them before in the multiple years Zope and ZEO have been in production (and even under Solaris, I believe most people chose to use Sun's POSIX threads implementation, not the native Solaris threads). ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Warning] Zope/ZEO clients: subprocesses can lead tonon-deterministic message loss: would like to duplicate this
[sathya] ... The zeoclient causes threads to be created but there are no forks or system calls as far as I can tell (or strace for that matter) Can you please point out where in the zeo code does forking occur ? I will try and duplicate this condition. ZEO and ZEO never fork -- they wouldn't be portable if they did (only Unixish systems have fork()). The only forking you'll find here is in startup scripts specific to Unixish systems. Dieter is talking about forks (whether direct or indirect) in *application* code, not in the Zope/ZEO cores. If application code running in a Zope/ZEO process forks, and Zope/ZEO are running asyncore in a thread, and the fork() implementation does clone all threads (which does not happen under Windows or POSIX), then of course the fork() the application code does is going to create a clone of the asyncore thread too. That said, it would indeed be helpful if Dieter revealed enough details about what his app did, and on which kind of system, so it would be possible for others to try to duplicate his results. I have no doubt that he is seeing problems, BTW -- Dieter has an excellent track record. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [Warning] Zope/ZEO clients: subprocesses can lead tonon-deterministic message loss: ONLY IF YOUR APP CODE FORKS
[sathya] tim thanks for confirming it. Wont loose sleep over it now. I did not mean to sound like questioning anybodys track record. No, it didn't sound like you were. I mentioned that Dieter has an excellent track record because *I'm* giving him a hard time here wink. I'm sure he's seeing problems, but I'm frustrated by the lack of concrete information about how they were provoked and on which kind of system. Since we have ZEO clusters in production it raised alarm bells thats all. Its good to know the problem MAY occur only if u fork in your own app code A potential problem is that Unixish systems typically fork under the covers for things that look utterly harmless to an application writer -- like the os.system() and os.popen() Dieter mentioned. fork()ing can create horrible problems for threaded applications (which is essentially *why* POSIX only clones the thread calling fork() -- that creates problems too, but the POSIX spec I referenced before discusses all this, so I won't repeat it here). and the core zeo/zope code by itself is not contributing to it. I don't think they are, but there's still not enough info here to say for sure. I guess as a general rule of thumb its not a good idea to use forks in zope application or product code anyway (unless you know what you are doing.) A more general rule of thumb is to avoid forking a threaded application of any kind. It's easy to follow that rule on OSes that don't have fork cough, but unfortunately tricky to avoid on those that do have fork. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] [Warning] Zope/ZEO clients: subprocesses can lead tonon-deterministic message loss
[Dieter Maurer] The problem occured in a ZEO client which called asyncore.poll in the forked subprocess. This poll deterministically stole ZEO server invalidation messages from the parent. [Tim Peters] I'm sorry, but this is still too vague to guess what happened. [Dieter Maurer] Even when I sometimes make errors, my responses usually contain all relevant information. I agree, but for whatever reason I'm having a very hard time following this message thread. - Which operating system was in use? The ZEO client application mentioned above is almost independent of the operating system -- beside the fact, that is uses fork (and therefore requires the OS to support it). The OS is important because the semantics of fork() depend on the OS. Therefore, I did not mention that the application was running on Linux 2. OK, so, e.g., the Solaris fork() semantics play no role in the actual damage you saw. - Which thread package? The application mentioned above does not use any thread. Therefore, it is independent of the thread package. Would it use threads it were LinuxThreads (but it does not). You said the app was a ZEO client, and, if that's so, it uses multiple threads whether or not your part of the app creates threads of its own. For example, a ZEO client creates a new thread just to connect to a ZEO server. If this is a ZEO client that never connects to a ZEO server, then perhaps threads are wholly irrelevant. There is no mystery at all that the application lost ZEO server invalidation messages. It directly follows from the fork semantics with respect to file descriptors. I can believe that's the truth, but I confess I still don't see how. The problem I saw for wider Zope/ZEO client usage came alone from reading the Linux fork manual page which indicates (or at least can be interpreted) that child and parent have the same threads. There was no concrete observation that messages are lost/duplicated in this szenario. Good! Thanks. Meanwhile, I checked that fork under Linux with LinuxThreads behaves with respect to threads as dictated by the POSIX standard: the forked process has a single thread and does not inherit other threads from its parent. I will soon check how our Solaris version of Python behaves. If this, too, has only one thread, I will apologize for the premature warning... Solaris offers (or imposes 0.9 wink) choices that don't exist on other platforms. One Solaris choice is whether you link Python with native Solaris threads, or with the Sun POSIX pthreads library. Another choice is whether you call Solaris fork() or Solaris fork1() (note that Python exposes fork1() on platforms that have it -- fork1() clones only the calling threading). The dangerous combination is Solaris threads + Solaris fork(). The other 3 combinations are harmless in this respect. Note that even using Solaris threads, it doesn't follow that places where Linux calls fork() under the covers are also places Solaris calls fork() under the covers. For example, Solaris system() calls Solaris vfork() under the covers, which differs from Solaris fork() in several key respects (and also differs from Solaris fork1()). The most relevant way vfork() differs from fork() under Solaris is that vfork() only clones the calling thread. - In the ZEO client that called fork(), did it call fork() directly, or indirectly as the result of a system() or popen() call? Or what? The ZEO client as the basic structure: while 1: work_to_do = get_work(...) for work in work_to_do: pid = fork() if pid == 0: do_work(work) # will not return sleep(...) do_work opens a new ZEO connection. get_work and do_work use asyncore.poll to synchronize with incoming messages from ZEO -- no asyncore.mainloop around. The poll in do_work has stolen ZEO invalidation messages destined for the parent such that get_work has read old state and returned work items already completed. That is the problem I saw. Well, don't do that then wink. All this is easy to understand, (almost) platform independent and independant of the thread library. I still wouldn't say it's easy to understand. While the thread that calls fork isn't running an asyncore loop, it must still be the case that asyncore in the parent has a non-empty map -- yes? If it had an empty map, the child processes would start with a clean slate (map), and so wouldn't pick up socket traffic meant for the parent. If that's so, it looks like just clearing asyncore's map in the child (before do_work()) would solve the (main) problem. *Iff* a thread library lets a forked child inherit all threads then the problem I announced in this Warning thread can occur, as it then behaves similarly to my application above (with an automatic rather than a explicit poll). I still don't want to rush to generalizations; as above, even on Solaris with native Solaris
Re: [Zope-dev] Re: hasattr geddon
[Shane Hathaway]
Hmm. I just heard about this hasattr geddon. hasattr is *good*. Why are
we fixing hasattr and bare excepts
Well, bare excepts are generally bad news in any application -- they
almost always turn out to catch more than the author intended
(including things like SystemExit and KeyboardInterrupt, which are
rarely intended to be swallowed).
when the real problem is ZODB? ZODB
should *not* be sensitive to the way the application handles ConflictErrors.
When a ConflictError (or ReadConflictError) occurs, the transaction should
fall into an uncommitable state. From this state, you can only abort the
transaction; any attempts to write an object or commit cause another
ConflictError.
AFAIK, that is the way the current releases of ZODB work. I'm not
sure what write an object means there, though. I think I know what
commit means. The logic in Connection.commit() raises
ReadConflictError upon an attempt to commit an object that previously
suffered a conflict during the transaction. How does that differ from
what you want?
If you know of a case where this doesn't work, please open a Collector
report. I can *believe* there's a problem -- I just don't know of one
(intended preconditions and postconditions aren't documented, and the
implementation has grown so many clever(?) special cases I'm rarely
100% sure of what it's trying to accomplish).
Then, only code that can guarantee that the attempted
transaction is complete should actually abort the transaction, and
fortunately ZPublisher fits that role. Today, the abort is implicit, and
that's the mistake that has caused us to litter the code with knowledge of
ConflictErrors. With the uncommitable state, it would not matter if the
application swallows ConflictErrors.
That said, I was surprised to discover that Python 2.3 implements hasattr this
way (from bltinmodule.c):
v = PyObject_GetAttr(v, name);
if (v == NULL) {
PyErr_Clear();
Py_INCREF(Py_False);
return Py_False;
}
Py_DECREF(v);
Py_INCREF(Py_True);
return Py_True;
hasattr has always been like that (that exact code, or a workalike
spelling), and that's always how hasattr has been documented to work
(the oldest docs I have handy are for Python 1.4).
It should not swallow all errors,
Guido doesn't agree; hoping that Python will change here is probably
futile (Guido has said several times that hasattr() should never raise
an exception, and that matches how it's always been documented and
implemented).
especially now that descriptors make computed attributes quite common.
getattr() only recently started catching only AttributeErrors, but apparently
hasattr is lagging behind.
The implementation of 2-argument getattr() didn't match the docs until
Python 2.2, when someone finally noticed that the implementation
didn't match the docs. Then the getattr() implementation was
repaired, to match the docs. The docs have never been clear (and
still aren't, IMO) about what 3-argument getattr() does with
exceptions. I think the current implementation is the most reasonable
one, with 3-argument getattr() propagating any exceptions the
2-argument form would have propagated, except for AttributeError --
but the docs don't really say that's what 3-arg getattr() does.
I suggest the consistency between getattr and hasattr should be fixed in Python,
not Zope.
They weren't intended to treat exceptions the same way, so an argument
based on inconsistency in that area won't make progress. As I said
recently in zodb-dev,
Zope and ZODB do incredibly complex stuff as side effects in what Guido
surely thought of as tiny hooks. He had in mind that hasattr()
might look for
a key in a dict or list, and suppress a harmless KeyError or IndexError,
not start World War III and then send an endless sequence of Terminators
back in time to change the outcome wink.
BTW, there's a ton of existing code that relies on hasattr()
suppressing things like KeyError and IndexError, so I doubt Python
would change here even if Guido agreed hasattr() was wrong to suppress
all exceptions. But that debate belongs on python-dev.
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
