Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Mon, 2004-05-17 at 13:00, Dieter Maurer wrote: > Chris McDonough wrote at 2004-5-15 13:04 -0400: > > ... > >Dieter, do you think you can read this patch and give a thumbs up or > >down on it? > > The patch looks good. Great, thanks for looking at it. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Chris McDonough wrote at 2004-5-15 13:04 -0400: > ... >Dieter, do you think you can read this patch and give a thumbs up or >down on it? The patch looks good. >On a different subject, the publisher probably shouldn't pass around >traceback objects (e.g. when it calls into err_hook) as Tres believes >that may be a memory leak waiting to happen. The traceback is vital for error analysis. It may not be necessary that ZPublisher touches the traceback but we will definitely need access to it during error handling. -- Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Fri, 2004-05-14 at 22:49, Michael Dunstan wrote: > hmm... I suspect there are few holes in this... > > - Need to make sure there is no get_transaction().begin() in > zpublisher_exception_hook() of lib/python/Zope/App/startup.py > >That begin() would mean that the error handling is not in fact part > of the original transaction? Right. > - An exception within err_hook() (aside from a Retry) will mean that > abort() is never called. Ya, thanks. Dieter, do you think you can read this patch and give a thumbs up or down on it? (If SteveA and/or Toby are reading, please do same?) On a different subject, the publisher probably shouldn't pass around traceback objects (e.g. when it calls into err_hook) as Tres believes that may be a memory leak waiting to happen. I will first apply the error-in-main-transaction patch to the 2.7 branch and worry about that later however. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On 14/05/2004, at 10:06 AM, Chris McDonough wrote: I am tempted to check the following into the 2.7 branch and HEAD: - "error occurs in same transaction as main request" patch to Publish.py. See http://www.plope.com/Members/chrism/ publishpy_errorinmaintrainsaction.patch/file_view for the patch. hmm... I suspect there are few holes in this... - Need to make sure there is no get_transaction().begin() in zpublisher_exception_hook() of lib/python/Zope/App/startup.py That begin() would mean that the error handling is not in fact part of the original transaction? - An exception within err_hook() (aside from a Retry) will mean that abort() is never called. Here is another attempt at publishpy_errorinmaintrainsaction.patch: Index: lib/python/ZPublisher/Publish.py === RCS file: /cvs-repository/Zope/lib/python/ZPublisher/Publish.py,v retrieving revision 1.164.2.2 diff -u -r1.164.2.2 Publish.py --- lib/python/ZPublisher/Publish.py17 Nov 2003 22:34:19 - 1.164.2.2 +++ lib/python/ZPublisher/Publish.py15 May 2004 02:34:12 - @@ -105,7 +105,6 @@ return response except: -if transactions_manager: transactions_manager.abort() # DM: provide nicer error message for FTP sm = None @@ -119,29 +118,36 @@ if err_hook is not None: -if parents: parents=parents[0] try: -return err_hook(parents, request, -sys.exc_info()[0], -sys.exc_info()[1], -sys.exc_info()[2], -) -except Retry: -# We need to try again -if not request.supports_retry(): +if parents: parents=parents[0] +try: return err_hook(parents, request, sys.exc_info()[0], sys.exc_info()[1], sys.exc_info()[2], ) -newrequest=request.retry() -request.close() # Free resources held by the request. -try: -return publish(newrequest, module_name, after_list, debug) -finally: -newrequest.close() +except Retry: +if not request.supports_retry(): +return err_hook(parents, request, +sys.exc_info()[0], +sys.exc_info()[1], +sys.exc_info()[2], +) +finally: +if transactions_manager: transactions_manager.abort() + + +newrequest=request.retry() +request.close() # Free resources held by the request. +try: +return publish(newrequest, module_name, after_list, debug) +finally: +newrequest.close() + +else: +if transactions_manager: transactions_manager.abort() +raise -else: raise def publish_module_standard(module_name, Index: lib/python/Zope/App/startup.py === RCS file: /cvs-repository/Zope/lib/python/Zope/App/startup.py,v retrieving revision 1.8.2.3 diff -u -r1.8.2.3 startup.py --- lib/python/Zope/App/startup.py 20 Dec 2003 04:48:36 - 1.8.2.3 +++ lib/python/Zope/App/startup.py 15 May 2004 02:34:13 - @@ -176,8 +176,6 @@ published=app.__bobo_traverse__(REQUEST).__of__( RequestContainer(REQUEST)) -get_transaction().begin() # Just to be sure. - published=getattr(published, 'im_self', published) while 1: f=getattr(published, 'raise_standardErrorMessage', None) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Largely due to Michael I believe I have isolated and fixed every reported sessioning error except this (still-difficult-to-reproduce-but-definitely-still-existing) KeyError bug in temporary storage. I can let the test rig run for several hours; it happens maybe once every hour or two, so I've not gotten the provocation of it down to a science yet. It doesn't occur when the transient object container is placed in a FileStorage. I am tempted to check the following into the 2.7 branch and HEAD: - "error occurs in same transaction as main request" patch to Publish.py. See http://www.plope.com/Members/chrism/publishpy_errorinmaintrainsaction.patch/file_view for the patch. - new Transience implementation (changed only slightly from chrism-sessiongeddon-branch) ... and worry about the TemporaryStorage bug as a separate issue. Thoughts? On Thu, 2004-05-13 at 04:05, Chris McDonough wrote: > On Wed, 2004-05-12 at 22:54, Michael Dunstan wrote: > > On 11/05/2004, at 4:54 PM, Chris McDonough wrote: > > > > > On Mon, 2004-04-26 at 18:34, Michael Dunstan wrote: > > >> I removed all the above hacks leaving just the > > >> one that comments out the construction of the to_notify list. Reran > > >> the > > >> test rig. And have yet to see any KeyErrors. > > > > > > Have you seen any since the time you reported this? > > > > Since the last report I have not run been running the test rig. I > > Needed my laptop back so I could get on with a few other things. :-( > > > > So I'm going to see if I can show others how to recreate the test rig > > that I'm using here. A trivial site and the script for testing that > > site are attached. > > Thanks very much for this, I'm running it at the moment. So far, with > the transaction isolation patch (the one which does not run the error in > its own transaction) and the "new" Transience.py, and an > unmodified-from-stock TemporaryStorage I haven't yet been able to > provoke any unseemly errors out of transience... ... as I typed > that, I just saw one go by! ;-) Excellent. I will diagnose. > > > - C > > > > ___ > Zope-Dev maillist - [EMAIL PROTECTED] > http://mail.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://mail.zope.org/mailman/listinfo/zope-announce > http://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Wed, 2004-05-12 at 22:54, Michael Dunstan wrote: > On 11/05/2004, at 4:54 PM, Chris McDonough wrote: > > > On Mon, 2004-04-26 at 18:34, Michael Dunstan wrote: > >> I removed all the above hacks leaving just the > >> one that comments out the construction of the to_notify list. Reran > >> the > >> test rig. And have yet to see any KeyErrors. > > > > Have you seen any since the time you reported this? > > Since the last report I have not run been running the test rig. I > Needed my laptop back so I could get on with a few other things. :-( > > So I'm going to see if I can show others how to recreate the test rig > that I'm using here. A trivial site and the script for testing that > site are attached. Thanks very much for this, I'm running it at the moment. So far, with the transaction isolation patch (the one which does not run the error in its own transaction) and the "new" Transience.py, and an unmodified-from-stock TemporaryStorage I haven't yet been able to provoke any unseemly errors out of transience... ... as I typed that, I just saw one go by! ;-) Excellent. I will diagnose. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On 11/05/2004, at 4:54 PM, Chris McDonough wrote: On Mon, 2004-04-26 at 18:34, Michael Dunstan wrote: I removed all the above hacks leaving just the one that comments out the construction of the to_notify list. Reran the test rig. And have yet to see any KeyErrors. Have you seen any since the time you reported this? (I chatted about this with Chris on #zope.) Since the last report I have not run been running the test rig. I Needed my laptop back so I could get on with a few other things. :-( So I'm going to see if I can show others how to recreate the test rig that I'm using here. A trivial site and the script for testing that site are attached. (Reminder - I'm using Zope 2.7.0 here - HEAD just is not ready for such games. :-) SessionRigExtensions.py and sessionrig.zexp make up the site. sessionrig.zexp is simply a bunch of external methods (with a folder to provide some separation of a standard_error_message that makes use of sessions). SessionRigExtensions.py has all the code for the external methods. TestSessionRig.py is the script which attacks zope. It should just work as is. (Unless the site you just imported is not found at http://localhost:8080/sessionrig. If that's the case then simply edit the 'path' in the code.) Don't expect the output of this script to say anything conclusive. Instead you should be tailing + greping through the event log for things of interest. Now the knobs that you should tweak to so that you see things of interest within minutes/hours rather than days/weeks... (The actual values may depend on the hardware that your running this on.) * etc/zope.conf: session-timeout-minutes 1 maximum-number-of-session-objects 1000 * lib/python/Products/Tranience: PERIOD = 1 #20 # attempt housekeeping every PERIOD seconds SPARE_BUCKETS = 2 #15 # number of buckets to keep spare * lib/python/tempstorage/TempoaryStorage.py: # keep old object revisions for CONFLICT_CACHE_MAXAGE seconds CONFLICT_CACHE_MAXAGE = 1 #60 # garbage collect conflict cache every CONFLICT_CACHE_GCEVERY seconds CONFLICT_CACHE_GCEVERY = 1 #60 * lib/python/Products/SiteErrorLog/SiteErrorLog.py: _rate_restrict_burst = 10 #5 (I wanted to see repeated errors in the event log. There is probably a more obvious way to do that. But that's what I'm using at the moment. Took me a while to actually find out this was why I was not seeing all the exceptions in the event log. Perhaps things like these should be exposed via zconfig? Makes a useful medium to document such knobs to promote awareness. As well as allowing people that are pushing zope into new corners to have a simpler way to manage these kinds of settings.) * TestSessionRig.py - there are few things that you can tweak within here too. (Well the whole script is up for grabs - hack away - I did.) As the script is it should expose problems with lack of transaction isolation for standard_error_message. I do tend to run several instances of this script at once. And in my case (as the url suggests) I run this on the same host as zope is running on. However to expose the KeyError thrown from TempoaryStorage I found that I did not really need much 'sleepiness' and replaced all the start()'s at the end of the script with start(200,1,100,10). (I suspect that you don't even need to support cookies in the client for this error. Sufficient to hit any URL that simply creates a session object on the server. And hit that URL very hard.) Another effective knob that you may try playing with is the nap duration used to slow down a session write: slowWrite = path + '/write?nap:float=0.5' hmm... that's probably enough to help someone recreate a KeyError or two. Michael. sessionrig.zexp Description: Binary data SessionRigExtensions.py Description: Binary data TestSessionRig.py Description: Binary data ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Tue, 2004-05-11 at 13:04, Dieter Maurer wrote: > Hi Chris, > > Chris McDonough wrote at 2004-5-11 00:54 -0400: > > ... > >So what do folks think of this error-case transaction isolation patch? > >Michael's original patch is preserved here (although it didn't survive > >cut and paste from my mail client in a pristine way, you'll get the > >idea): > > > >http://www.plope.com/Members/chrism/transaction_isolation_error.patch/file_view > > > >Personally it looks fine to me. It wraps each error invocation in its > >own transaction. > > Please revisit the discussion (mostly between Toby and me) on > "zope-dev" about how to fix this problem. > > Toby convinced me that error handling should take place > in the same transaction as the original request (and not in its > own). > > His main argument: the traceback can contain references to persistent > objects that should not be there. If they are written to ZODB > in any way, all kinds of dubious inconsistencies can occur. > > Meanwhile, I saw several questions in the mailing lists of > people that wanted to access the SESSION object during > error handling (probably) because it contained useful information for > error handling. > This is impossible when the transaction is aborted before > error handling. Right. I will try to work up another patch then. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Hi Chris, Chris McDonough wrote at 2004-5-11 00:54 -0400: > ... >So what do folks think of this error-case transaction isolation patch? >Michael's original patch is preserved here (although it didn't survive >cut and paste from my mail client in a pristine way, you'll get the >idea): > >http://www.plope.com/Members/chrism/transaction_isolation_error.patch/file_view > >Personally it looks fine to me. It wraps each error invocation in its >own transaction. Please revisit the discussion (mostly between Toby and me) on "zope-dev" about how to fix this problem. Toby convinced me that error handling should take place in the same transaction as the original request (and not in its own). His main argument: the traceback can contain references to persistent objects that should not be there. If they are written to ZODB in any way, all kinds of dubious inconsistencies can occur. Meanwhile, I saw several questions in the mailing lists of people that wanted to access the SESSION object during error handling (probably) because it contained useful information for error handling. This is impossible when the transaction is aborted before error handling. -- Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Whew. I finally got a chance to review at least a bit of Michael's excellent work... On Mon, 2004-04-26 at 18:34, Michael Dunstan wrote: > Okay - I've gone ahead and implemented a flavour of transaction > isolation for errors. (See below for patch.) That seems to have done > the trick just nicely. I have not seen any more KeyError's thrown from > Transience.py. Yay! So what do folks think of this error-case transaction isolation patch? Michael's original patch is preserved here (although it didn't survive cut and paste from my mail client in a pristine way, you'll get the idea): http://www.plope.com/Members/chrism/transaction_isolation_error.patch/file_view Personally it looks fine to me. It wraps each error invocation in its own transaction. We need to do something like this to fix problems exposed by session writes (or any other database writes) that occur during an error. I know there were some opinions about this before as to whether the error should occur as part of the preceding transaction or whether the error should be wrapped in its own transaction, but I've forgotten the pros and cons (I've even forgotten which side I was on at that point ;-). Does anyone have a strong opinion? I think I remember that there were issues here with ZPatterns, mail-on-error, and some other things. > However there is another KeyError reported in 848 too. This one is > thrown from TempoaryStoarge. After some further tweaking of my test rig > I was able to produce such KeyErrors quite frequently. Using Zope 2.7.0 > and recent the rewrite of Transience.py: > > http://cvs.zope.org/*checkout*/Products/Transience/Transience.py? > rev=1.32.12.2.2.2&only_with_tag=chrism-sessiongeddon > > errors look like: > TemporaryStorage.py", line 94, in load > s=self._index[oid] > KeyError: '\x00\x00\x00\x00\x00\x00\x08\x82' > > So far it looks like commenting out the construction of the to_notify > list in _gc makes this go away. So _gc looks like: > > def _gc(self, now=None): > if self._timeout_slices == 0: > return # dont do gc if there is no timeout > > if now is None: > now = getCurrentTimeslice() # for unit tests > max_ts = now - (PERIOD * (self._timeout_slices + 1)) > keys = self._data.keys(None, max_ts) > > #to_notify = [] > > for key in list(self._data.keys(None, max_ts)): > assert(key <= max_ts) > STRICT and _assert(self._data.has_key(key)) > #for v in self._data[key].values(): > #to_notify.append(v) > del self._data[key] > > #for v in to_notify: > #self.notifyDel(v) > > Note that the test rig does not set onAdd or onDelete. I was unable to verify/review this, but I believe you! However, I think the above might be fixing the symptom rather than the underlying problem. I was able to provoke these errors out of temporary storage in a completely different way. There is a "session stress test" in the Sessions product (stresstests/stressTestMultipleThreads) that kicks off multiple threads emulating simultaneous reads and writes to a temporary storage that exposes some key errors out of Transience as well. I took a look at this and I have a sneaky suspicion that the "conflict cache" in TemporaryStorage may be causing problems. TemporaryStorage attempts to emulate a revisioning/undoing storage (which it isn't), where older revisions of objects are kept around in order to play with the conflict resolution strategy in ConflictResolvingStorage. I think TemporaryStorage fails to clear this cache at some appropriate times (particularly when an object is gc'ed in the storage by the _takeOutGarbage method... TemporaryStorage does its own form of in-band "packing" via reference counts). In the amount of time I could devote to this, I was unable to prove or disprove this objectively, but looking at the code made me believe it was possible. > Along the way there where a few things that I was not too sure about so > I tried fiddling with to see what would happen: > > (1) Use of hasattr in Transience.py. hasattr should be avoided? See > http://mail.zope.org/pipermail/zodb-dev/2003-October/005968.html. I did > change usage of hasattr to match the recipe given by Tim. And added > some logging to explicitly look for ConflictError's at those points. > Found a few of these in the logs. But that did not solve the KeyErrors. Good point, the hasattrs should all be changed to getattrs. > (2) _p_resolveConflict of Increaser of Transience does not always > return the max of the three given states. I replaced the implementation > with return max(old, state1, state2). But that did not solve the > KeyErrors. Whoops. ;-) Thanks! > > (3) Use of try/except in TempoaryStorage. For example: > > try: del referenceCount[oid] > except: pass > > I removed all such try/except wrappings of del. But t
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On 11/03/2004, at 7:25 PM, Chris McDonough wrote: On Wed, 2004-03-10 at 23:14, michael wrote: Thanks for your excellent isolation here. I don't get a KeyError thrown until *after* at least one rendering of standard_error_message that includes an access to a session variable. And I'm only getting KeyError's for sessions that have had a chance to expire. And that KeyError will be thrown whenever any access is made for the session. (Previously I thought I was *only* seeing KeyError's being thrown from standard_error_message.) I think after provide transaction isolation for the duration of an error, we will not see this symptom occur again. Okay - I've gone ahead and implemented a flavour of transaction isolation for errors. (See below for patch.) That seems to have done the trick just nicely. I have not seen any more KeyError's thrown from Transience.py. Yay! http://collector.zope.org/Zope/848 has a couple of examples of such KeyErrors thrown from Transience. These KeyError were a very painful user experience as either the server needed to be restarted or the user needed to delete their cookie for that server from their browser cookie jar. However there is another KeyError reported in 848 too. This one is thrown from TempoaryStoarge. After some further tweaking of my test rig I was able to produce such KeyErrors quite frequently. Using Zope 2.7.0 and recent the rewrite of Transience.py: http://cvs.zope.org/*checkout*/Products/Transience/Transience.py? rev=1.32.12.2.2.2&only_with_tag=chrism-sessiongeddon errors look like: 2004-04-27T08:24:16 BLATHER(-100) OFS Exception while rendering an error message Traceback (most recent call last): File "/Users/michael/builds/zope/Zope/lib/python/OFS/SimpleItem.py", line 204, in raise_standardErrorMessage v = s(**kwargs) File "/Users/michael/builds/zope/Zope/lib/python/Shared/DC/Scripts/ Bindings.py", line 306, in __call__ return self._bindAndExec(args, kw, None) File "/Users/michael/builds/zope/Zope/lib/python/Shared/DC/Scripts/ Bindings.py", line 343, in _bindAndExec return self._exec(bound_data, args, kw) File "/Users/michael/builds/zope/Zope/lib/python/Products/PythonScripts/ PythonScript.py", line 318, in _exec result = f(*args, **kw) File "Script (Python)", line 12, in standard_error_message File "/Users/michael/builds/zope/Zope/lib/python/ZPublisher/HTTPRequest.py", line 1218, in __getattr__ v = self.get(key, default, returnTaints=returnTaints) File "/Users/michael/builds/zope/Zope/lib/python/ZPublisher/HTTPRequest.py", line 1178, in get if callable(v): v = v() File "/Users/michael/builds/zope/Zope/lib/python/Products/Sessions/ SessionDataManager.py", line 93, in getSessionData return self._getSessionDataObject(key) File "/Users/michael/builds/zope/Zope/lib/python/Products/Sessions/ SessionDataManager.py", line 180, in _getSessionDataObject ob = container.new_or_existing(key) File "/Users/michael/builds/zope/Zope/lib/python/Products/Transience/ Transience.py", line 504, in new_or_existing item = self.get(key, _marker) File "/Users/michael/builds/zope/Zope/lib/python/Products/Transience/ Transience.py", line 332, in get item = self._move_item(k, current_ts, _marker) File "/Users/michael/builds/zope/Zope/lib/python/Products/Transience/ Transience.py", line 208, in _move_item self._gc(current_ts) File "/Users/michael/builds/zope/Zope/lib/python/Products/Transience/ Transience.py", line 429, in _gc for key in list(self._data.keys(None, max_ts)): File "/Users/michael/builds/zope/Zope/lib/python/ZODB/Connection.py", line 561, in setstate p, serial = self._storage.load(oid, self._version) File "/Users/michael/builds/zope/Zope/lib/python/tempstorage/ TemporaryStorage.py", line 94, in load s=self._index[oid] KeyError: '\x00\x00\x00\x00\x00\x00\x08\x82' So far it looks like commenting out the construction of the to_notify list in _gc makes this go away. So _gc looks like: def _gc(self, now=None): if self._timeout_slices == 0: return # dont do gc if there is no timeout if now is None: now = getCurrentTimeslice() # for unit tests max_ts = now - (PERIOD * (self._timeout_slices + 1)) keys = self._data.keys(None, max_ts) #to_notify = [] for key in list(self._data.keys(None, max_ts)): assert(key <= max_ts) STRICT and _assert(self._data.has_key(key)) #for v in self._data[key].values(): #to_notify.append(v) del self._data[key] #for v in to_notify: #self.notifyDel(v) Note that the test rig does not set onAdd or onDelete. Along the way there where a few things that I was not too sure about so I tried fiddling with to see what would happen: (1) Use of hasattr in Transience.py. hasattr should be avoided? See http://mail.zope.org/pipermail/zodb-dev/2003-October/00
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Chris McDonough wrote at 2004-3-11 23:56 -0500: >On Thu, 2004-03-11 at 07:29, Toby Dickenson wrote: > >> I have applications that send emails in error handlers. MailHost isnt >> transactional today, but Ive always wanted it to be. > >I suppose that's true. If someone else doesn't do it first, during the >bug day we'll try to support this use case, then, unless it turns out to >be very problematic. I would abort the transaction after error handling. If someone needs a commit, he can call "get_transaction().commit()" himself. -- Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Thu, 2004-03-11 at 07:29, Toby Dickenson wrote: > I have applications that send emails in error handlers. MailHost isnt > transactional today, but Ive always wanted it to be. I suppose that's true. If someone else doesn't do it first, during the bug day we'll try to support this use case, then, unless it turns out to be very problematic. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Thu, 2004-03-11 at 08:32, [EMAIL PROTECTED] wrote: > Hi Chris, > > On Tue, 9 Mar 2004, Chris McDonough wrote: > > > > But it seems I forget to install new Transience module to the new Zope > > > instance. Should I? > > Yes, please! > > Sorry, it does not seems to work on my first attempt: Right. Could you delete your Session.fs file and retry? - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Hi Chris, On Tue, 9 Mar 2004, Chris McDonough wrote: > > But it seems I forget to install new Transience module to the new Zope > > instance. Should I? > Yes, please! Sorry, it does not seems to work on my first attempt: Traceback (innermost last): * Module ZPublisher.Publish, line 163, in publish_module_standard * Module Products.iHotfix, line 80, in new_publish * Module ZPublisher.Publish, line 127, in publish * Module Zope.App.startup, line 203, in zpublisher_exception_hook * Module ZPublisher.Publish, line 100, in publish * Module ZPublisher.mapply, line 88, in mapply * Module ZPublisher.Publish, line 40, in call_object * Module OFS.DTMLDocument, line 128, in __call__ URL: http://www.chalkface.com/custom/index_html/manage_main Physical Path:/www.chalkface.com/ZWarehouse_0.8/custom/index_html * Module DocumentTemplate.DT_String, line 474, in __call__ * Module OFS.DTMLDocument, line 121, in __call__ URL: http://www.chalkface.com/custom/index.html/manage_main Physical Path:/www.chalkface.com/ZWarehouse_0.8/custom/index.html * Module DocumentTemplate.DT_String, line 474, in __call__ * Module DocumentTemplate.DT_Let, line 76, in render * Module OFS.DTMLDocument, line 121, in __call__ URL: http://www.chalkface.com/catalog/html/zwarehouse_html_header/manage_main Physical Path:/www.chalkface.com/ZWarehouse_0.8/catalog/html/zwarehouse_html_header * Module DocumentTemplate.DT_String, line 474, in __call__ * Module DocumentTemplate.DT_Util, line 201, in eval __traceback_info__: cart_functions * Module , line 1, in * Module Shared.DC.Scripts.Bindings, line 306, in __call__ * Module Shared.DC.Scripts.Bindings, line 343, in _bindAndExec * Module Products.PythonScripts.PythonScript, line 318, in _exec * Module None, line 16, in setSessionByRequest.py Line 16 * Module ZPublisher.HTTPRequest, line 1218, in __getattr__ * Module ZPublisher.HTTPRequest, line 1178, in get * Module Products.Sessions.SessionDataManager, line 93, in getSessionData * Module Products.Sessions.SessionDataManager, line 180, in _getSessionDataObject * Module Products.Transience.Transience, line 504, in new_or_existing * Module Products.Transience.Transience, line 332, in get * Module Products.Transience.Transience, line 204, in _move_item * Module Products.Transience.Transience, line 359, in _roll AttributeError: _max_timeslice -- Alex V. Koval http://www.halogen-dg.com/ http://www.zwarehouse.org/ ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Thursday 11 March 2004 12:19, Chris McDonough wrote: > > What if you want to make a change in the error handler? > > If anyone had been depending on this behavior in their current error > handlers, they would have seen weirdnesses in their applications like > those being reported now. I have applications that send emails in error handlers. MailHost isnt transactional today, but Ive always wanted it to be. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Thu, 2004-03-11 at 06:52, Chris Withers wrote: > Well, what if you want to make a note in some object (say the error_log), that > something bad happened? The error_log doesn't use persistent objects to keep its messages. > What if you want to make a change in the error handler? If anyone had been depending on this behavior in their current error handlers, they would have seen weirdnesses in their applications like those being reported now. And no one has really complained, AFAICT. That leads me to believe it's a case of YAGNI (or even YNNI -- you never needed it). > My view is that the error handler should oeprate in it's own transaction, and be > aborted if any exceptions are raised in it... I guess this is a reasonable strategy too, but I just wonder if it's worth the bother. I guess you might want to write to a transactional relational database in your error page, but as far as I can tell no one does this, because if they did they would probably be complaining about lost records every so often. Unless they just never noticed. > ...that said, I guess if you _really_ wanted to make changes in your error > handler, would anything bad happen if your proposed changes are made, but the > user's error handling code does a manaual get_transaction().commit() itself? Not that I can tell. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Chris McDonough wrote: I think the transaction in which the error code operates in should be aborted. Well, what if you want to make a note in some object (say the error_log), that something bad happened? What if you want to make a change in the error handler? My view is that the error handler should oeprate in it's own transaction, and be aborted if any exceptions are raised in it... There is no other sane thing to do. I think this might be as easy as adding a few strategic get_transaction().abort() calls to various cases in Zope/App/startup.py's zpublisher_exception_hook. I don't have the time to untangle that mess at the moment, but I will enter a collector issue in. ...that said, I guess if you _really_ wanted to make changes in your error handler, would anything bad happen if your proposed changes are made, but the user's error handling code does a manaual get_transaction().commit() itself? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On 11/03/2004, at 7:25 PM, Chris McDonough wrote: On Wed, 2004-03-10 at 23:14, michael wrote: Thanks for your excellent isolation here. I don't get a KeyError thrown until *after* at least one rendering of standard_error_message that includes an access to a session variable. And I'm only getting KeyError's for sessions that have had a chance to expire. And that KeyError will be thrown whenever any access is made for the session. (Previously I thought I was *only* seeing KeyError's being thrown from standard_error_message.) I think after provide transaction isolation for the duration of an error, we will not see this symptom occur again. That's the first step, at least. We are going to have a "bugathon" at the Zope 2 sprint at PyCon 9 days from now, and I will put this high on the list. Would you be willing to test a Zope 2.7 version that contains a fix? Sure. Michael ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Wed, 2004-03-10 at 23:14, michael wrote: Thanks for your excellent isolation here. > I don't get a KeyError thrown until *after* at least one rendering of > standard_error_message that includes an access to a session variable. > And I'm only getting KeyError's for sessions that have had a chance to > expire. And that KeyError will be thrown whenever any access is made > for the session. (Previously I thought I was *only* seeing KeyError's > being thrown from standard_error_message.) I think after provide transaction isolation for the duration of an error, we will not see this symptom occur again. That's the first step, at least. We are going to have a "bugathon" at the Zope 2 sprint at PyCon 9 days from now, and I will put this high on the list. Would you be willing to test a Zope 2.7 version that contains a fix? Thanks! - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On 11/03/2004, at 1:07 PM, Chris McDonough wrote: On Wed, 2004-03-10 at 18:30, michael wrote: On closer inspection I have another small detail - I'm only seeing these KeyError's for sessions that have had a chance to expire and when trying to access the session from standard_error_message. Can you provide a complete traceback for the error message? I'm guessing that it happens during session garbage collection but I can't be sure without the traceback. hmmm... continuing to refine what I think I'm seeing :-O I don't get a KeyError thrown until *after* at least one rendering of standard_error_message that includes an access to a session variable. And I'm only getting KeyError's for sessions that have had a chance to expire. And that KeyError will be thrown whenever any access is made for the session. (Previously I thought I was *only* seeing KeyError's being thrown from standard_error_message.) I can't yet say from what I'm seeing if the session accessed in standard_error_message which somehow corrupts the consistency of transience is the same session for which the KeyError is thrown further down the track. Here is an example traceback when the KeyError is thrown from a Script (Python) that reads the session variable. Followed by a similar traceback thrown when Zope tries to read that same sesssion variable in my standard_error_message: 2004-03-11T16:13:03 ERROR(200) SiteError http://localhost:8080/test/read Traceback (most recent call last): File "/Users/michael/Desktop/Zope-2.7.0/lib/python/ZPublisher/Publish.py", line 101, in publish request, bind=1) File "/Users/michael/Desktop/Zope-2.7.0/lib/python/ZPublisher/mapply.py", line 88, in mapply if debug is not None: return debug(object,args,context) File "/Users/michael/Desktop/Zope-2.7.0/lib/python/ZPublisher/Publish.py", line 41, in call_object result=apply(object,args) # Type s to step into published object. File "/Users/michael/Desktop/Zope-2.7.0/lib/python/Shared/DC/Scripts/ Bindings.py", line 306, in __call__ return self._bindAndExec(args, kw, None) File "/Users/michael/Desktop/Zope-2.7.0/lib/python/Shared/DC/Scripts/ Bindings.py", line 343, in _bindAndExec return self._exec(bound_data, args, kw) File "/Users/michael/Desktop/Zope-2.7.0/lib/python/Products/PythonScripts/ PythonScript.py", line 318, in _exec result = f(*args, **kw) File "Script (Python)", line 2, in read File "/Users/michael/Desktop/Zope-2.7.0/lib/python/ZPublisher/ HTTPRequest.py", line 1218, in __getattr__ v = self.get(key, default, returnTaints=returnTaints) File "/Users/michael/Desktop/Zope-2.7.0/lib/python/ZPublisher/ HTTPRequest.py", line 1178, in get if callable(v): v = v() File "/Users/michael/Desktop/Zope-2.7.0/lib/python/Products/Sessions/ SessionDataManager.py", line 93, in getSessionData return self._getSessionDataObject(key) File "/Users/michael/Desktop/Zope-2.7.0/lib/python/Products/Sessions/ SessionDataManager.py", line 180, in _getSessionDataObject ob = container.new_or_existing(key) File "/Users/michael/Desktop/Zope-2.7.0/lib/python/Products/Transience/ Transience.py", line 176, in new_or_existing item = self.get(key, notfound) File "/Users/michael/Desktop/Zope-2.7.0/lib/python/Products/Transience/ Transience.py", line 809, in get v = self._data[b].get(k, notfound) KeyError: 1078974700 -- 2004-03-11T16:13:04 BLATHER(-100) OFS Exception while rendering an error message Traceback (most recent call last): File "/Users/michael/Desktop/Zope-2.7.0/lib/python/OFS/SimpleItem.py", line 204, in raise_standardErrorMessage v = s(**kwargs) File "/Users/michael/Desktop/Zope-2.7.0/lib/python/Shared/DC/Scripts/ Bindings.py", line 306, in __call__ return self._bindAndExec(args, kw, None) File "/Users/michael/Desktop/Zope-2.7.0/lib/python/Shared/DC/Scripts/ Bindings.py", line 343, in _bindAndExec return self._exec(bound_data, args, kw) File "/Users/michael/Desktop/Zope-2.7.0/lib/python/Products/PythonScripts/ PythonScript.py", line 318, in _exec result = f(*args, **kw) File "Script (Python)", line 12, in standard_error_message File "/Users/michael/Desktop/Zope-2.7.0/lib/python/ZPublisher/ HTTPRequest.py", line 1218, in __getattr__ v = self.get(key, default, returnTaints=returnTaints) File "/Users/michael/Desktop/Zope-2.7.0/lib/python/ZPublisher/ HTTPRequest.py", line 1178, in get if callable(v): v = v() File "/Users/michael/Desktop/Zope-2.7.0/lib/python/Products/Sessions/ SessionDataManager.py", line 93, in getSessionData return self._getSessionDataObject(key) File "/Users/michael/Desktop/Zope-2.7.0/lib/python/Products/Sessions/ SessionDataManager.py", line 180, in _getSessionDataObject ob = container.new_or_existing(key) File "/Users/michael/Desktop/Zope-2.7.0/lib/python/Products/Transience/ Transience.py", line 176, in new_or_existing item = self.get(key
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Wed, 2004-03-10 at 18:30, michael wrote: > On closer inspection I have another small detail - I'm only seeing > these KeyError's for sessions that have had a chance to expire and when > trying to access the session from standard_error_message. Can you provide a complete traceback for the error message? I'm guessing that it happens during session garbage collection but I can't be sure without the traceback. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On 11/03/2004, at 11:48 AM, Chris McDonough wrote: On Wed, 2004-03-10 at 15:20, Dieter Maurer wrote: michael wrote at 2004-3-10 15:22 +1300: ... I have been trying on and off to recreate this error via brute force loading of the simplest possible site that uses sessions. I failed to see this particular KeyError *until* I tried reading a session variable from standard_error_message. Now I can recreate these quite frequently. This is a well known problem in Zope, lengthy discussed but apparently still not yet fixed... The problem is that error handling starts after the main transaction has been aborted. Aborting or committing a transaction implicitely starts a new transaction. If error handling causes objects to be registered in the transaction (because it writes some objects and this can happen for session variables even when they are only read), this transaction is tainted. As the transaction is not aborted after error handling, the transaction and the modified cache are spilled (independently!) into a later request. As the connection/cache may later end up in a different thread, objects can be invalidated asynchronously. This is disastrous. Zope *MUST* abort (or commit) the transaction after error handling!!! I think the transaction in which the error code operates in should be aborted. There is no other sane thing to do. I think this might be as easy as adding a few strategic get_transaction().abort() calls to various cases in Zope/App/startup.py's zpublisher_exception_hook. I don't have the time to untangle that mess at the moment, but I will enter a collector issue in. On closer inspection I have another small detail - I'm only seeing these KeyError's for sessions that have had a chance to expire and when trying to access the session from standard_error_message. Also I'm seeing behaviour that suggests my first guess is not right. In other words - I'm seeing some healthy behaviour when a conflict error is thrown while attempting to render the standard_error_message. Michael ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Nevermind. http://zope.org/Collectors/Zope/789 and http://zope.org/Collectors/Zope/786 The bug neglector is really living up to its name lately (not pointing fingers, mea culpa). - C On Wed, 2004-03-10 at 15:20, Dieter Maurer wrote: > michael wrote at 2004-3-10 15:22 +1300: > > ... > >I have been trying on and off to recreate this error via brute force > >loading of the simplest possible site that uses sessions. I failed to > >see this particular KeyError *until* I tried reading a session variable > >from standard_error_message. Now I can recreate these quite frequently. > > This is a well known problem in Zope, lengthy discussed > but apparently still not yet fixed... > > The problem is that error handling starts after the main > transaction has been aborted. Aborting or committing > a transaction implicitely starts a new transaction. > > If error handling causes objects to be registered > in the transaction (because it writes some objects > and this can happen for session variables even when they > are only read), this transaction is tainted. > As the transaction is not aborted after error handling, > the transaction and the modified cache are spilled > (independently!) into a later request. > > As the connection/cache may later end up in a different thread, > objects can be invalidated asynchronously. This is disastrous. > > Zope *MUST* abort (or commit) the transaction after error handling!!! ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Wed, 2004-03-10 at 15:20, Dieter Maurer wrote: > michael wrote at 2004-3-10 15:22 +1300: > > ... > >I have been trying on and off to recreate this error via brute force > >loading of the simplest possible site that uses sessions. I failed to > >see this particular KeyError *until* I tried reading a session variable > >from standard_error_message. Now I can recreate these quite frequently. > > This is a well known problem in Zope, lengthy discussed > but apparently still not yet fixed... > > The problem is that error handling starts after the main > transaction has been aborted. Aborting or committing > a transaction implicitely starts a new transaction. > > If error handling causes objects to be registered > in the transaction (because it writes some objects > and this can happen for session variables even when they > are only read), this transaction is tainted. > As the transaction is not aborted after error handling, > the transaction and the modified cache are spilled > (independently!) into a later request. > > As the connection/cache may later end up in a different thread, > objects can be invalidated asynchronously. This is disastrous. > > Zope *MUST* abort (or commit) the transaction after error handling!!! I think the transaction in which the error code operates in should be aborted. There is no other sane thing to do. I think this might be as easy as adding a few strategic get_transaction().abort() calls to various cases in Zope/App/startup.py's zpublisher_exception_hook. I don't have the time to untangle that mess at the moment, but I will enter a collector issue in. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
michael wrote at 2004-3-10 15:22 +1300: > ... >I have been trying on and off to recreate this error via brute force >loading of the simplest possible site that uses sessions. I failed to >see this particular KeyError *until* I tried reading a session variable >from standard_error_message. Now I can recreate these quite frequently. This is a well known problem in Zope, lengthy discussed but apparently still not yet fixed... The problem is that error handling starts after the main transaction has been aborted. Aborting or committing a transaction implicitely starts a new transaction. If error handling causes objects to be registered in the transaction (because it writes some objects and this can happen for session variables even when they are only read), this transaction is tainted. As the transaction is not aborted after error handling, the transaction and the modified cache are spilled (independently!) into a later request. As the connection/cache may later end up in a different thread, objects can be invalidated asynchronously. This is disastrous. Zope *MUST* abort (or commit) the transaction after error handling!!! -- Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On 10/03/2004, at 6:29 AM, Chris McDonough wrote: On Tue, 2004-03-09 at 06:36, Alex V. Koval wrote: I have enabled 400 error_log ex exceptions to keep, and during 3 days I got 2 errors on the site: TimeUsername (User Id) Exception 16:35:01Anonymous User (None) KeyError: 1078763620 21:59:05Anonymous User (None) KeyError: 1078696720 I have been trying on and off to recreate this error via brute force loading of the simplest possible site that uses sessions. I failed to see this particular KeyError *until* I tried reading a session variable from standard_error_message. Now I can recreate these quite frequently. My *guess* is that this error happens when a ZODB conflict error is thrown from Transience while rendering standard_error_message. I need to do some more testing and probably worth detailing just how I'm doing the tests. Currently using Zope 2.7.0, Python 2.3.3. Yet to try with the Transience branch. Or use FileStorage. Just happy to have actually provoked this error in some kind of test arena :-) Michael ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Tue, 2004-03-09 at 06:36, Alex V. Koval wrote: > I have enabled 400 error_log ex exceptions to keep, and during 3 days I > got 2 errors > on the site: > > Time Username (User Id) Exception > 16:35:01 Anonymous User (None) KeyError: 1078763620 > 21:59:05 Anonymous User (None) KeyError: 1078696720 This is fewer of these than you were receiving before? > > But it seems I forget to install new Transience module to the new Zope > instance. Should I? Yes, please! ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Hi Chris, On Fri, 05 Mar 2004 13:08:01 -0500, Chris McDonough <[EMAIL PROTECTED]> wrote: I am monitoring site now, and will tell you about the results. OK, many thanks! Running For: 3 days 4 hours 28 min 29 sec. I have enabled 400 error_log ex exceptions to keep, and during 3 days I got 2 errors on the site: TimeUsername (User Id) Exception 16:35:01Anonymous User (None) KeyError: 1078763620 21:59:05Anonymous User (None) KeyError: 1078696720 But it seems I forget to install new Transience module to the new Zope instance. Should I? -- Alex V. Koval http://www.halogen-dg.com http://www.zwarehouse.orgTraceback (innermost last): Module ZPublisher.Publish, line 100, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 40, in call_object Module OFS.DTMLDocument, line 128, in __call__ - - URL: http://www.chalkface.com/custom/index_html/manage_main - Physical Path: /www.chalkface.com/ZWarehouse_0.8/custom/index_html Module DocumentTemplate.DT_String, line 474, in __call__ Module OFS.DTMLDocument, line 121, in __call__ - - URL: http://www.chalkface.com/custom/index.html/manage_main - Physical Path: /www.chalkface.com/ZWarehouse_0.8/custom/index.html Module DocumentTemplate.DT_String, line 474, in __call__ Module DocumentTemplate.DT_Let, line 76, in render Module OFS.DTMLDocument, line 121, in __call__ - - URL: http://www.chalkface.com/catalog/html/zwarehouse_html_header/manage_main - Physical Path: /www.chalkface.com/ZWarehouse_0.8/catalog/html/zwarehouse_html_header Module DocumentTemplate.DT_String, line 474, in __call__ Module DocumentTemplate.DT_Util, line 201, in eval - __traceback_info__: cart_functions Module , line 1, in Module Shared.DC.Scripts.Bindings, line 306, in __call__ Module Shared.DC.Scripts.Bindings, line 343, in _bindAndExec Module Products.PythonScripts.PythonScript, line 318, in _exec Module None, line 16, in setSessionByRequest.py - - Line 16 Module ZPublisher.HTTPRequest, line 1218, in __getattr__ Module ZPublisher.HTTPRequest, line 1178, in get Module Products.Sessions.SessionDataManager, line 93, in getSessionData Module Products.Sessions.SessionDataManager, line 180, in _getSessionDataObject Module Products.Transience.Transience, line 176, in new_or_existing Module Products.Transience.Transience, line 809, in get KeyError: 1078696720 Traceback (innermost last): Module ZPublisher.Publish, line 100, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 40, in call_object Module OFS.DTMLDocument, line 128, in __call__ - - URL: http://www.chalkface.com/custom/index.html/manage_main - Physical Path: /www.chalkface.com/ZWarehouse_0.8/custom/index.html Module DocumentTemplate.DT_String, line 474, in __call__ Module DocumentTemplate.DT_Let, line 76, in render Module OFS.DTMLDocument, line 121, in __call__ - - URL: http://www.chalkface.com/catalog/html/zwarehouse_html_header/manage_main - Physical Path: /www.chalkface.com/ZWarehouse_0.8/catalog/html/zwarehouse_html_header Module DocumentTemplate.DT_String, line 474, in __call__ Module DocumentTemplate.DT_Util, line 201, in eval - __traceback_info__: cart_functions Module , line 1, in Module Shared.DC.Scripts.Bindings, line 306, in __call__ Module Shared.DC.Scripts.Bindings, line 343, in _bindAndExec Module Products.PythonScripts.PythonScript, line 318, in _exec Module None, line 16, in setSessionByRequest.py - - Line 16 Module ZPublisher.HTTPRequest, line 1218, in __getattr__ Module ZPublisher.HTTPRequest, line 1178, in get Module Products.Sessions.SessionDataManager, line 93, in getSessionData Module Products.Sessions.SessionDataManager, line 180, in _getSessionDataObject Module Products.Transience.Transience, line 176, in new_or_existing Module Products.Transience.Transience, line 809, in get KeyError: 1078763620 ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Fri, 2004-03-05 at 09:15, [EMAIL PROTECTED] wrote: > Installed. I had to install new zope instance first, then import > old content via zexp & recreate manually all admins. Because when you > change the running zope config in Zope 2.7.0 it has no impact on > temp_folder - Sessions.fs is not created when the temporary storage > was already in place. Wow. Sorry about that. I would have thought changing out the storage would just work, but I didn't try it myself. > Lets see, next test will be on Monday, many users come only on working > days. > > I also got many complatins from customer, so I wrote a test suite > which is logging to a site each 2 minutes, and if there are problem, > it sends emails to all of admins, so they come and restart zope. Ouch. > > With prev. Transience, when this keyError appear, it was seen > only by 1,2,3 users. With new Transience, when error starts > to appear, nobody is able to see any page which uses Session - > all sessions became broken. Ouch again. > > I am monitoring site now, and will tell you about the results. OK, many thanks! - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Hi Chris, On Wed, 3 Mar 2004, Chris McDonough wrote: > With this text: > > > # Temporary storage database (for sessions) > > path $INSTANCE/var/Sessions.fs > > mount-point /temp_folder > container-class Products.TemporaryFolder.TemporaryContainer > Installed. I had to install new zope instance first, then import old content via zexp & recreate manually all admins. Because when you change the running zope config in Zope 2.7.0 it has no impact on temp_folder - Sessions.fs is not created when the temporary storage was already in place. Lets see, next test will be on Monday, many users come only on working days. I also got many complatins from customer, so I wrote a test suite which is logging to a site each 2 minutes, and if there are problem, it sends emails to all of admins, so they come and restart zope. With prev. Transience, when this keyError appear, it was seen only by 1,2,3 users. With new Transience, when error starts to appear, nobody is able to see any page which uses Session - all sessions became broken. I am monitoring site now, and will tell you about the results. -- Alex V. Koval http://www.halogen-dg.com/ http://www.zwarehouse.org/ ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Hi Chris, On Wed, 3 Mar 2004, Chris McDonough wrote: > (boldly crossposting this to zodb-dev, please respond on one list or the > other but not both) > > That error *appears* to be caused by reaching a state that is impossible > to reach. The code in question is: > > for key in list(self._data.keys(None, max_ts)): > assert(key <= max_ts) > STRICT and _assert(self._data.has_key(key)) > for v in self._data[key].values(): > to_notify.append(v) > del self._data[key] > I was not working yesterday, now I found a big thread about the problem here :) Thats good that people are interested into resolving of this bug. I will read all the mails now, and will try to help to resolve it, since we have system where the high load causes such problems. By the way, just a few minutes ago I have found another session error, with a little different traceback then reported ago, so I am posting it here, just in case it helps you understand the prob. I am still thinking may be something wrong with my code? -- traceback -- ═ Site Error An error was encountered while publishing this resource. KeyError Sorry, a site error occurred. Traceback (innermost last): ∙Module ZPublisher.Publish, line 163, in publish_module_standard ∙Module Products.iHotfix, line 80, in new_publish ∙Module ZPublisher.Publish, line 127, in publish ∙Module Zope.App.startup, line 203, in zpublisher_exception_hook ∙Module ZPublisher.Publish, line 100, in publish ∙Module ZPublisher.mapply, line 88, in mapply ∙Module ZPublisher.Publish, line 40, in call_object ∙Module OFS.DTMLDocument, line 128, in __call__ URL: http://www.chalkface.com/custom/index_html/manage_main Physical Path:/www.chalkface.com/ZWarehouse_0.8/custom/index_html ∙Module DocumentTemplate.DT_String, line 474, in __call__ ∙Module OFS.DTMLDocument, line 121, in __call__ URL: http://www.chalkface.com/custom/index.html/manage_main Physical Path:/www.chalkface.com/ZWarehouse_0.8/custom/index.html ∙Module DocumentTemplate.DT_String, line 474, in __call__ ∙Module DocumentTemplate.DT_Let, line 76, in render ∙Module OFS.DTMLDocument, line 121, in __call__ URL: http://www.chalkface.com/catalog/html/zwarehouse_html_header/ manage_main Physical Path:/www.chalkface.com/ZWarehouse_0.8/catalog/html/ zwarehouse_html_header ∙Module DocumentTemplate.DT_String, line 474, in __call__ ∙Module DocumentTemplate.DT_Util, line 201, in eval __traceback_info__: cart_functions ∙Module , line 1, in ∙Module Shared.DC.Scripts.Bindings, line 306, in __call__ ∙Module Shared.DC.Scripts.Bindings, line 343, in _bindAndExec ∙Module Products.PythonScripts.PythonScript, line 318, in _exec ∙Module None, line 16, in setSessionByRequest.py Line 16 ∙Module ZPublisher.HTTPRequest, line 1218, in __getattr__ ∙Module ZPublisher.HTTPRequest, line 1178, in get ∙Module Products.Sessions.SessionDataManager, line 93, in getSessionData ∙Module Products.Sessions.SessionDataManager, line 180, in _getSessionDataObject ∙Module Products.Transience.Transience, line 494, in new_or_existing ∙Module Products.Transience.Transience, line 304, in __setitem__ KeyError: 1078473960 (Also, an error occurred while attempting to render the standard error message.) Troubleshooting Suggestions ∙ The URL may be incorrect. ∙ The parameters passed to this resource may be incorrect. ∙ A resource that this resource relies on may be encountering an error. For more detailed information about the error, please refer to error log. If the error persists please contact the site maintainer. Thank you for your patience. -- Alex V. Koval http://www.halogen-dg.com/ http://www.zwarehouse.org/ ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Wed, 2004-03-03 at 22:53, John Belmonte wrote: > If I'm following this thread correctly, isn't the code failing because > the BTree is corrupted (that is, BTrees.check.check chokes)? If that's > the case then you're certainly right to avoid masking the problem. We don't know that it's corrupted for sure yet because the problem has not yet made itself repeatable in isolation, and has only appeared in one production setup (Alex's). Hopefully Alex will instrument his code to do the check when it chokes and we'll know more then. I've also asked him to change out the storage that's being used to hold this data to a FileStorage. I have some fear about the current storage code that he is using to store this data (TemporaryStorage), as it is not typically tested via unit tests for simultaneous access and does some hairy inplace garbage collection that other storages don't do. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Chris McDonough wrote: On Wed, 2004-03-03 at 22:20, Casey Duncan wrote: for key in list(self._data.keys(None, max_ts)): assert(key <= max_ts) STRICT and _assert(self._data.has_key(key)) for v in self._data[key].values(): to_notify.append(v) del self._data[key] Maybe you could use items() and two loops instead; to_rm = [] for key, val in self._data.items(None, max_ts): for v in val.values(): to_notify.append(v) to_rm.append(key) for key in to_rm: try: del self._data[key] except Keyerror: pass # Somebody else deleted it first I don't think that could raise a KeyError... Well, the real bit of magic there is the "try.. except KeyError: pass" stanza. Believe me, I'm tempted to stick that in, but this is the kind of voodoo that got me in to a lot of trouble in the older version of this code (there was reams upon reams of voodoo in the old code), so I'd really rather just figure out why the code is failing in the first place. I'd just rather not mask the problem until I understand the cause. That may never happen, of course, but a man can dream. If I'm following this thread correctly, isn't the code failing because the BTree is corrupted (that is, BTrees.check.check chokes)? If that's the case then you're certainly right to avoid masking the problem. -John -- http:// if ile.org/ ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Wed, 2004-03-03 at 22:20, Casey Duncan wrote: > > > for key in list(self._data.keys(None, max_ts)): > > > assert(key <= max_ts) > > > STRICT and _assert(self._data.has_key(key)) > > > for v in self._data[key].values(): > > > to_notify.append(v) > > > del self._data[key] > > Maybe you could use items() and two loops instead; > > to_rm = [] > for key, val in self._data.items(None, max_ts): > for v in val.values(): > to_notify.append(v) > to_rm.append(key) > for key in to_rm: > try: > del self._data[key] > except Keyerror: >pass # Somebody else deleted it first > > I don't think that could raise a KeyError... Well, the real bit of magic there is the "try.. except KeyError: pass" stanza. Believe me, I'm tempted to stick that in, but this is the kind of voodoo that got me in to a lot of trouble in the older version of this code (there was reams upon reams of voodoo in the old code), so I'd really rather just figure out why the code is failing in the first place. I'd just rather not mask the problem until I understand the cause. That may never happen, of course, but a man can dream. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
Chris McDonough wrote at 2004-3-3 04:55 -0500: >(boldly crossposting this to zodb-dev, please respond on one list or the >other but not both) > >That error *appears* to be caused by reaching a state that is impossible >to reach. The code in question is: > >for key in list(self._data.keys(None, max_ts)): >assert(key <= max_ts) >STRICT and _assert(self._data.has_key(key)) >for v in self._data[key].values(): >to_notify.append(v) >del self._data[key] > >The line that says "for v in self._data[key].values()" is the line that >throws the KeyError. But it should be impossible for the code to throw >a KeyError for the expression "self._data[key]" because the "keys" >method of the _data IOBTree just told us that the key named by "key" was >one of its keys via the range search; it should be an invariant. If we had a low conflict connection, I would understand how this could happen: All BTree leaves are chained together. The "keys" method uses this chain to enumerate the keys. "[]" on the other hand, uses the tree structure to locate its key. Assume that a parallel thread removes a key and commits the transaction while we are in the for loop above. We may read the old state (with the later deleted key) for "keys". During the "for", we receive invalidations for the nodes affected by the deletion. When we access "[key]" we may try to load a tree node which is not yet in the ZODB cache and meanwhile invalidated. When we suppress the resulting "ReadConflictError", we may not find "key" (as it is by now deleted). In my "Transience" implementation, I ignore this exceptional case. I do use a low conflict connection and have to be prepared for this situation. Furthermore, the situation is not problematic: I want to determine sessions that should be deleted. Someone else already did it -- this is fine. No need to do it twice... -- Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Wed, 2004-03-03 at 04:55, Chris McDonough wrote: > (boldly crossposting this to zodb-dev, please respond on one list or the > other but not both) > > That error *appears* to be caused by reaching a state that is impossible > to reach. The code in question is: > > for key in list(self._data.keys(None, max_ts)): > assert(key <= max_ts) > STRICT and _assert(self._data.has_key(key)) > for v in self._data[key].values(): > to_notify.append(v) > del self._data[key] I don't have much context for this question. It's definitely the case that a corrupt BTree there are keys you can reach using keys(), which follows the bucket next pointers, that can't reach using a lookup, which follows child pointers down through the interior nodes. If you could call the check functions on the BTrees in question. That's object._check() to check C internals and BTrees.check.check() to check value based consistency. So how is the BTree is question used? If the test is failing here, it seems most likely that the BTree was corrupted by a write somewhere else. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
(boldly crossposting this to zodb-dev, please respond on one list or the other but not both) That error *appears* to be caused by reaching a state that is impossible to reach. The code in question is: for key in list(self._data.keys(None, max_ts)): assert(key <= max_ts) STRICT and _assert(self._data.has_key(key)) for v in self._data[key].values(): to_notify.append(v) del self._data[key] The line that says "for v in self._data[key].values()" is the line that throws the KeyError. But it should be impossible for the code to throw a KeyError for the expression "self._data[key]" because the "keys" method of the _data IOBTree just told us that the key named by "key" was one of its keys via the range search; it should be an invariant. Note that in the line above that starts "STRICT and _assert...", I do the paranoid check there as there *have* been cases where BTrees range searches lied in the past. STRICT is not true in your case (it's turned off), so that check never gets run on your system, but if it had, it might have raised an assertion error. I haven't been able to provoke that kind of thing in my own stress tests, unfortunately. I have been proven to be at fault about this sort of thing before, but I've been a good boy and I believe I've applied all of the lessons I learned in the past to the newest code, so I unfortunately again have to reach the conclusion that there is something afoul in the BTrees code, provoked only under high stress scenarios. It's also appears to be very difficult to reproduce. In the end, this means to you that... well.. you've got two choices. a) continue using ZODB-based sessions and helping us (me) to track it down, living with the consequences of the errors in the meantime or b) use a different session implementation. I would prefer "a" but I do need to warn you that this might *never* get solved because the failure mode appears to be so intermittent that it's extremely expensive (in the dollars-and-cents sense) to pin down and ultimately fix, and that may prevent me (and ZC) from doing so. But with a lot of help from other interested people (like yourself) we may be able to coax the failure out of obscurity and squish it without breaking the bank. ;-) Assuming you're interested, what can you do? Well, you could find out a little about the BTrees module in Zope CVS, particularly the "check" module which has code to check a BTree for corruption, and instrument the Transience code to run the check code in the places it seems to be coming up with errors before bombing out. If it's not corrupt, well.. I'm not sure what that means, but it would appear to be a problem with the BTrees range search functions. If it is corrupt, it might exonerate the range search functions. Rinse, lather, repeat with other checks in the code, such as reporting the internal state of the BTree when the error occurs (which I've forgotten how to do, but a maillist search should help), providing information about when conflict errors were raised right before the error, and so on. It's very difficult to provide a concrete "type this, type that" set of steps for this sort of thing due to the latency involved in remote debugging an extremely hard to reproduce failure, so if you want to help best, since you're the person who has access to the machine where the failure appears to be reproducible (and hopefully the motive to want to fix it), you should familiarize yourself with the Transience code and the BTrees APIs and use a bit of inductive logic to help me isolate the problem. If you'd rather not, I can understand that too. ;-) HTH, - C On Wed, 2004-03-03 at 03:18, [EMAIL PROTECTED] wrote: > Chris, > > No, just a few minutes ago I got this again: > > Time 2004/03/03 07:45:04.662 GMT > User Name (User Id) Anonymous User (None) > Request URL http://www.chalkface.com/catalog/html/custom/index_html > Exception TypeKeyError > Exception Value 1078236460 > > Traceback (innermost last): > > * Module ZPublisher.Publish, line 100, in publish > * Module ZPublisher.mapply, line 88, in mapply > * Module ZPublisher.Publish, line 40, in call_object > * Module OFS.DTMLDocument, line 128, in __call__ > > URL: http://www.chalkface.com/custom/index_html/manage_main > Physical Path:/www.chalkface.com/ZWarehouse_0.8/custom/index_html > * Module DocumentTemplate.DT_String, line 474, in __call__ > * Module OFS.DTMLDocument, line 121, in __call__ > > URL: http://www.chalkface.com/custom/index.html/manage_main > Physical Path:/www.chalkface.com/ZWarehouse_0.8/custom/index.html > * Module DocumentTemplate.DT_String, line 474, in __call__ > * Module DocumentTemplate.DT_Let, line 76, in render > * Module OFS.DTMLDocument, line 121, in __call__ > > URL: > http://www.chalkface.com/catalog/html/zwarehouse_html_header/manage_m
