Re: [Zope-dev] Sessions and long requests = conflicts? Mcdutils?
On 4/12/07, Maciej Wisniowski [EMAIL PROTECTED] wrote:
Hi
Currently I have some problems with our application (Zope2.8.4)
and with Conflict Errors in sessions.
In general if we have few concurrent requests that are running
sometimes for 3-4 minutes (and they're touching session inside)
I get a lot of conflict errors with Inceraser, OOBTree, Length2 etc.
These are errors like:
serial starded with xxx serial currently commited
Of course I know that it is best to move such long processes to
something external with Async, lovely.Remotetask or built
in Oracle jobs, but so far I have to deal with this state of
our application (as I'm not the author ot this but rather
something like an zope admin/supporter for this app).
I think that ConflictErrors are caused because of sessions
implementation, especially: timeslices, current bucket etc.
as written in Transience/HowTransienceWorks.stx.
Changing session_resolution_seconds to big value
helps here. By default we had session_resolution_seconds
set to 300.
You could keep experimenting with values to reduce the chances of
conflicts. Perhaps sessions that last for days. With resolution of
hours. Disabling inband housekeeping.
Note that a session-timeout-minutes of 0 enables a slightly different
approach which has a little less active structure.
I wonder how this happens. If I have two requests that at the beginning
modify it's sessions like: session.set('aa', 1), and then call long
running ZSQLMethods then (if meanwhile timeslice has changed because
of too short session_resolution_seconds):
- first request that finishes finds that there is new 'current' bucket
and moves it's session object and second request's session object to
new 'current' bucket and commits this
- second request finishes and finds that it's session object is not
the same as it was at the beginning (because it was moved to 'current'
bucket)??
Anybody can say if I'm right here?
I don't think session mechanics operates like that at the end of a
transaction. More generally what is happening is that the second
transaction is trying to commit data that was changed by an earlier
transaction after second transaction read that data. In this case the
data is various bits of the internals that make up sessions and
transience storage.
Very careful use of explicit transaction commits may be all that you
need in your application. For example, make all your edits of the
session data early in the request and then commit the transaction.
Then proceed with the longer operation. Might be that destroys the
consistency of your application data though.
-- Michael
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] OT: zc.buildout / ploneout
On 4/6/07, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Hi, I have inherited an existing buildout architecture developed in-house which is becoming difficult to support and maintain. There are about 25 setup tasks in total. I am considering moving it to using zc.buildout. Two questions: - is there a central repository of recipes anywhere? for example, one of our current build tasks is a squid installation, which, i think would be of common interest if converted to a recipe. so far I've only come across zope3-based recipes, the zope2 installer, and ploneout. Have a look in the cheeseshop under Framework :: Buildout http://cheeseshop.python.org/pypi?:action=browsec=512 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope] Florent Guillaume's database Conflict Error on session objects patch not working ?
On 2/3/07, yacine chaouche [EMAIL PROTECTED] wrote: I am experiencing issues regarding a database ConflictError on session objects. Can you detail all the various getSesion and setSession methods of: http://mail.zope.org/pipermail/zope/2007-January/170167.html I use zope 2.9.0 python 2.4.2. It's worth trying the latest of each of those. Cheers Michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: Re: [Fwd: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)]
On 10/29/06, Yoshinori Okuji [EMAIL PROTECTED] wrote: IIRC, Jim noted that it would be better to backport zope3's, while our patches are for zope2's. See http://mail.zope.org/pipermail/zope-dev/2006-May/027503.html for a short thread on successfully using xmlpickle. With that recipe and patch I was able to import and export a Plone site at the time. Michael ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZODB.POSException.ConflictError on attribute assignment
On 9/1/06, [EMAIL PROTECTED]
[EMAIL PROTECTED] wrote:
I'm getting a ZODB.POSException.ConflictError in the following code in a
Custom Zope Product, and I'm hoping someone can enlighten me on how to
fix it.
# nd is a folder with a sub-folder Categories
# NDCategory is a custom Zope product I wrote that does not have
anything called bogus defined.
categories = nd.Categories.objectValues('NDCategory')
for cat in categories:
cat.bogus = []
Feels like you might be miss interpreting what a ConflictError is.
With some hand waving... ConflictErrors are _not_ a mechanism for
prohibiting attribute assignment because that attribute already
exists. Instead ConflictErrors are a mechanism to prohibit inadvertent
concurrent updates of the same attribute. (Or in older ZODB's, even
concurrent reading of an attribute that is being written too.)
Check the traceback for your ConflictError to get some details about
just what is in conflict. Also might be worth a look in the access
logs to see what else might be happening on the server at about the
same time.
Cheers
Michael
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Hotfix-20060821 does not apply to zope 2.7.0 to 2.7.2?
From what I can tell the feature that is Hotfixed does not appear until Zope 2.7.3. If the Hotfix is installed in Zope 2.7.0 through to 2.7.2 then it shows up as a broken product. No real harm done. The product import breaks with: Traceback (most recent call last): File /usr/local/www/Zope/lib/python/OFS/Application.py, line 660, in import_product product=__import__(pname, global_dict, global_dict, silly) File /usr/local/www/Zope/Atrac/Products/Hotfix_20060821/__init__.py, line 8, in ? from docutils.parsers.rst.directives import tables ImportError: cannot import name tables And we see that ``tables`` not show up till Zope 2.7.3 via: http://cvs.zope.org/Zope/lib/python/docutils/parsers/rst/directives/Attic/tables.py?hideattic=0only_with_tag=Zope-2_7-branch And a casual glance around docutils before ``tables`` was introducded suggests that feature was not part of another module. Cheers Michael ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope] Inavlid Refernce Counts in Temporary Storage
On 7/14/06, Philip Kilner [EMAIL PROTECTED] wrote: For the last three days, we have been plagued by slowdowns and temporary lock-ups, which have on a couple of score occasions culminated in a Conflict Error, indicting transactional problems in the temporary storage. On four or five occasions, Zope has apparently locked up, and a restart has been required. The conflict error's may well just be a by-product of the slow down. Roughly speaking, the longer a request takes the higher the chances are of conflict error. There are various tools for trying to see what zope is doing. Google for: debug spinning zope. cheers michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Trying to trap ConflictError
On 7/7/06, Jonathan [EMAIL PROTECTED] wrote: I haven't been able to determine how the oid's are generated, other than little-endian 64-bit unsigned integers that will be assigned more or less sequentially, but I can't figure out how the same oid is being used twice (on the other hand I could be totally off-base as I am really way over my head here!). An oid is an identifier for a persistent state of an object. As the state of an object evolves from transaction to transaction it gets a new oid for each state. Two concurrent transactions that involve the same object will start with the same state, i.e. the same oid, for that object. Some detail about how BTrees behave can be found in http://www.zope.org/Wikis/ZODB/BTreeConflictResolution Also might be illustrative to run fsdump to see what is happening in your transactions. See http://www.zope.org/Wikis/ZODB/FileStorageBackup. You'll need FileStorage for that. And that only shows you details of successful transactions. cheers michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Trying to trap ConflictError
On 7/7/06, Michael Dunstan [EMAIL PROTECTED] wrote: An oid is an identifier for a persistent state of an object. As the state of an object evolves from transaction to transaction it gets a new oid for each state. Two concurrent transactions that involve the same object will start with the same state, i.e. the same oid, for that object. Sorry. Ignore that. :-( Note to self, don't even read email before morning caffeine intake. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Trying to trap ConflictError
On 7/6/06, Jonathan [EMAIL PROTECTED] wrote: I am still investing, and have found errors like: 2006-07-04T14:48:12 ERROR ZODB.Connection Couldn't load state for 0x1c Traceback (most recent call last): File /usr/local/Zope-2.9.2/lib/python/ZODB/Connection.py, line 732, in setstate self._setstate(obj) File /usr/local/Zope-2.9.2/lib/python/ZODB/Connection.py, line 768, in _setstate p, serial = self._storage.load(obj._p_oid, self._version) File /usr/local/Zope-2.9.2/lib/python/tempstorage/TemporaryStorage.py, line 104, in load s=self._index[oid] KeyError: '\x00\x00\x00\x00\x00\x00\x00\x1c' in the event.log. From my googling research, it appears that these type of errors may be due to insufficient memory (but I am not sure and am still investigating to see if this is the cause of the conflict errors). TempoaryStorage.py has a few tuning constants. The above error reads as though RECENTLY_GC_OIDS_LEN is too small for your application. It may well be that the default tuning of TempoaryStorage does not match your application very well. It would be useful to try FileStorage to help decide on where to focus attention. If FileStorage behaves a lot better then that also suggests that TempoaryStorage is miss-tuned for your application. Cheers Michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope-dev] zope.xmlpickle
On 5/26/06, Jim Fulton [EMAIL PROTECTED] wrote: It would be nice to integrate this back into the export/import machinery. Yeah, I'd like to think I'll be able to find time to put this into Zope 2.11. Any suggestions on getting a successful dumps of a Missing.Value? You need someone with pickle zen to look at this. I don't have time right now myself. If you don't find any other takers, then put an issue in the Zope 3 collector. How about review of a trivial patch that is attached? Michael missing-hash.patch Description: Binary data ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Zope3-dev] December release post-mortem
On 1/20/06, Tim Peters [EMAIL PROTECTED] wrote: In all, zope.testing is a poster child for the value of package development outside of a Zope tree. I've been very happy using zope.testing with several non zope projects. Including how easy it is to follow and distribute that package as needed for those projects. Same goes for zdaemon. And of course ZConfig. Michael ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope] More on understanding conflicts
On 12/22/05, Dennis Allison [EMAIL PROTECTED] wrote: The issue here has had to do with session variables and their iteraction with the persistence mechanism and conflicts and multiple threads for the same session. Chris McDonough has pointed out that session variables can cause writes and write conflicts even if the only Zope level access is a read. I thought MVCC works out of the box for Zope 2.8.4 which uses ZODB 2.3.4. Am I wrong? The real surprise was that you can get a write conflict from a pair of session variable reads! The missing detail here is that reading a session object causes a write to the database to update the last access time for that session object. So what looks like a plain old read to the application code above, in-fact includes a write just below the surface. (This is a design choice so that stale session objects can be removed.) There are implementation details of TransientObject that attempt to mitigate some of the pain: - The access time has a WRITEGRANULARITY. That can help a bit by avoiding writing altogether some of the time. But still there is need to write (and so potential contention) each time a boundary of that granularity is crossed. - And _p_resolveConflict of TransientObject will most often pick a state to avoid a conflict error. When it does there is the performance cost of writing to database but there will be no ConflictError. However, I've suggested in the past that you should comment out _p_resolveConflict while debugging consistency problems. What is the current state of your _p_resolveConflict now? _p_resolveConflict includes a DIWM'ly component. Perhaps you still need to avoid that component of _p_resolveConflict? In that case a _p_resolveConflict could be engineered that can resolve the case of just updates to the access time? But still bails out when presented with two states that have explicit updates to maintain consistency. Though that's all just conjecture till we see your tracebacks for your conflicts. michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: Zope Persistence (was: XML-RPC within ZOPE)
On 12/19/05, Jan-Ole Esleben [EMAIL PROTECTED] wrote: 1. In the example, just setting _p_changed=1 does _not_ lead to a conflict error. With the ineffectual code above it (that never gets executed) it _does_. So there _is_ some implicit magical stuff going on and ZOPE tries to take care that only subobjects change (but incompletely)! I strongly doubt it. Zope does not inspect code. There must be a problem in your testing. Note that if self.a is a standard list, the self.a.append(1) doesn't have any impact on the persistence mechanism or transactions either. Please, try it out. Delete the if clause and the append in it. Little bit tricky to try out as testers need to guess what all the missing code is. Also, for this kind of code demonstration, rather than directions for commenting/uncommenting and relying on the tester to restart the server between trails, provide different methods or even different classes. Meanwhile I'm camping over at I-strongly-doubt-it. 3. It is especially confusing that ZOPE behaves differently when using XML-RPC calls. From what you say, it should be the same within the ZOPE system as when using XML-RPC. It gets more complicated with XML-RPC though! The successive XML-RPC call you describe provoke new transactions, surely you're aware of that? Whereas just calling a function of course doesn't. I'm aware of that. But ZOPE offers XML-RPC and as there is nothing in the documentation about such complex interactions. Also, I was directly responding to what the previous poster had written. Zope provides for XML-RPC as a server. And it does so with internal transaction support. But there is no extra transactional framework for XML-RPC clients (which your example has). Or for that matter acting as an XML-RPC server nested within some external transaction (which your example has). Either design to avoid this or get your hands dirty with transactions. Some starting points for examples would include any of the relational database adapters. Also in the same space is MaildropHost. michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: Zope Persistence (was: XML-RPC within ZOPE)
On 12/19/05, Jan-Ole Esleben [EMAIL PROTECTED] wrote: Little bit tricky to try out as testers need to guess what all the missing code is. Any standard persistent ZOPE product wrapped around this will do. These are the only methods in a ZOPE product that inherits from Item, Persistent, RoleManager and Implicit. Also, for this kind of code demonstration, rather than directions for commenting/uncommenting and relying on the tester to restart the server between trails, provide different methods or even different classes. If I did that, people would say somethings wrong with your testing and they would have to do a line-by-line comparison and still have room to doubt my statement. /me shurgs. I prefer code that runs. :-) But there is no extra transactional framework for XML-RPC clients (which your example has). Or for that matter acting as an XML-RPC server nested within some external transaction (which your example has). I was initially talking about a more complex situation that has been gradually simplified. ZOPE should be able to act as an XML-RPC client to the outside world. As to me not understanding what you all are saying: I believe I understand, I know that theoretically - from a merely technical viewpoint, it works, I just look at it from a different angle. Of course you have to know a framework and its limitations, but this error I stumbled upon strikes me as really complex (even though you keep repeating it's very simple if you just acknowledge xyz), for instance because the docs aren't perfect and say that You must explicitly signal any changes made to mutable attributes (such as instances, lists, and dictionaries) or use persistent versions of mutable objects, like ZODB.PersistentMapping (see below for more information on PersistentMapping.) - the _first_ option being mutable attributes! Also, there is no below. I believe you when you say that it is really very simple and I shouldn't have to worry if I follow the right principles, that you then go on to state (principles that aren't really that explicit anywhere - such as don't use XML-RPC to talk to ZOPE from ZOPE and don't use mutable subobjects at all). However, this is a further bit of distance between a ZOPE user and the Python he knows, and it also isn't as true for complex situations with complex requirements that I could probably just jot down in dumber frameworks such as RoR or Django or TurboGears. Because I know without having to check myself where I am designing Data, I explicitly _make_ changes (to the DB) and am thus very aware of where complex interactions might occur etc. This is probably one of those cases where no one is really right because a mixture of viewpoints is involved, I feel a bit as if we were talking about wether Lisp is better than Smalltalk. However, even if it is only the docs that are lacking I think it would be sensible to acknowledge that as a problem. It obviously was a problem for me, as of course was the complexity of my setup and several other factors. I am not an inexperienced programmer, so I would think that just because other people rarely stumble upon it it is not an insignificant problem. Imagine designing a large system based on some only marginally false assumptions about ZOPE transactions, persistence and XML-RPC interaction, and then running into this in a key component. Joining many transactional systems into one mega transactional beast is requires attention to details. How about getting away from this notion a mega transaction with nested transactions? For example, make use of transaction.commit() before making XML-RPC calls. Though you well need to decide what happens in the event of a ConflictError being thrown from somewhere else and causing the initial request to be retried. michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope-dev] Re: sessions in the presence of conflicts
On 12/16/05, Dennis Allison [EMAIL PROTECTED] wrote: MAYBE CONFLICTS AND THEIR RESOLUTION ARE NOT THE ROOT CAUSE OF THE SESSION VARIABLE PROBLEM. The observed problem is that session variables suddenly disappear. Perhaps your app is tripping over some bug in conflict handling. But I'd say it is worth entertain other ideas too. For now, just comment out all of TransientObject._p_resolveConflict and if you still get errors then you know you have to look elsewhere. (Sure, some of that elsewhere may well include the conflict resolution code above _p_resolveConflict.) Your application and sessions should cope just fine in the face of any ConflictError. ConflictError's are an essential part of the machinery that keeps data state consistent. As Chris mentions, look at how your using sessions and some of the assumptions you might be making. Might be useful to try with sessions that don't timeout, set session-timeout-minutes to 0. And try maximum-number-of-session-objects of 0. Also trying the turning those knobs the other way, session-timeout-minutes of 1 and maximum-number-of-session-objects of 2. For now, stay focused on making sure you maintain a consistent state. Only once you have a consistent state then is it worth trying to improve the rate of ConflictErrors. (Which in your case of sessions lasting for many hours I think the numbers you quote elsewhere are too small. And, yeah, an alternative implementation for _p_resolveConflict might help there. Personally I prefer the simple approach of just commenting that out completely and living with a slightly higher conflict count.) Might be worth trying without ZEO in the mix. Definitely worth the effort, if you have not already, to recreate the whole system on a separate host that you can feel comfortable making changes to. Then you can happily tune the various knobs downwards which may help with trying to observe the problem. For example session-timeout-minutes of 1. cheers michael ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope] resolving conflict errors
On 12/9/05, Dennis Allison [EMAIL PROTECTED] wrote: The problem I am trying to resolve appears to be load related. The observed symptom is that (some) session variables spontaneously disappear. There appears to be some connection to conflicts, but the exact mechanism and the relationship is not yet clear. A small possibility is that you are being bitten by the DWIM'ly nature of TransientObjects conflict resolution where the last modified state is chosen over the others. If you think this is biting you then try commenting out _p_resolveConflict of TransientObject. That's bound to increase the rate of conflict errors but should provide you with a consistent session state. Perhaps useful as a debugging step. michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] resolving conflict errors
On 12/11/05, Dennis Allison [EMAIL PROTECTED] wrote: Good idea, but it is hard to do in a production environment with a never lose data model. Have a go at recreating the problems you are seeing on a development host. SessionRig can be used to mount a brute force attack of the session machinery. You'll need to tune that somewhat to your particular application. http://cvs.zope.org/Packages/SessionRig michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope-dev] Please vote about conflict errors logging
How about adding a URL to the logged message? Something like See http://zope.org/ConflictError; detailing the difference between a retried and unresovled ConflictErrors and how to recognise them in the event.log. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope] conflicts, sessions, and programming best practices guidelines
On 11/18/05, Chris McDonough [EMAIL PROTECTED] wrote: On Nov 17, 2005, at 4:46 AM, Dennis Allison wrote: Thank Chris, good pointsB. The session variable timeout is currently several hours as requird by our application. The timeout resolution remains at its default value. Making the resolution higher has been observed to reduce conflicts (see dunny's chart). If you sessions live for several hours then perhaps you want a increase session-resolution-seconds to something like 1800 (30 minutes) or more. Also worth seeing if increasing WRITEGRANULARITY of Products.Transience.TransientObject helps you. I have not dorked with disabling the inband housekeeping. I don't see how it would impact on our problem. Am I missing something? Turning off inband housekeeping has the potential to reduce the numnber of conflicts because less work is done during normal sessioning operations. Instead of doing the work to figure out whether it needs to clean up after itself during normal operations it relies on an external process. And it is not that difficult to do. Michael. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Help!!! Zope won't start -- StorageSystemError
On 8/28/05, Dan Gaibel [EMAIL PROTECTED] wrote: Hi folks, This is my first post to this list and I'm desperate for help. I'm running Zope 2.6.1 with Python 2.1.3. My Zope instance died today and won't come back. When I try to start it, I get: 2005-08-27T20:58:15 PANIC(300) z2 Startup exception Traceback (innermost last): File /home/lrc/z2.py, line 497, in ? File string, line 1, in ? File /home/lrc/lib/python/Zope/__init__.py, line 37, in ? File /home/lrc/lib/python/ZODB/FileStorage.py, line 213, in __init__ File /home/lrc/lib/python/ZODB/lock_file.py, line 33, in lock_file StorageSystemError: Could not lock the database file. There must be another process that has opened the file. lock_file.py has a simple try/except block around that line. As a debugging step remove the try/except block and see if you get more verbose exception details. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope-dev] Re: ZServer response RFC compliance improvement
On 8/22/05, Tres Seaver [EMAIL PROTECTED] wrote: Are we sure that we won't be breaking the rather large possible set of installed servers running behind Apache 1.3.x with the bug for which adding the content length was a workaround? I understand that this bug was resolved in Apache 1.3.27 [1]. Which is a few years old now. And outdated by several security releases since then. Also OFS.Image been patched as of Zope 2.7.1 [2] in such a way that it would have already tripped a combination of old-ish Apache and new-ish Zope. Though ZServer was still throwing in a Content-Length: 0. (Which I read as sufficient to provoke the bug in Apache 1.3.27.) [1] http://www.apacheweek.com/issues/02-10-04 [2] http://cvs.zope.org/Zope/lib/python/OFS/Attic/Image.py?hideattic=0only_with_tag=Zope-2_7-branch#rev1.145.2.7 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope] Zope 2.8.1 on Mac OS X tiger Server
On 2005-08-18, at 06:35 GMT+12:00, Bernd Dorn wrote: On 17.08.2005, at 20:22, Garito wrote: Hi again! Sorry but Zope is executable, perhaps another thing? Any idea? hm, have you set the effective user in zope.conf? you need this if you run zopectl as root as root try to execute /System/Library/StartupItems/Zope/Zope from the commandline and have a look what if something is looged into your event.log without any messages i have no clue Also worth having a look through the system log using say Console.app. Cheers, Michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope 2.8 and conflict errors
On 2005-08-04, at 02:32 GMT+12:00, Maciej Wisniowski wrote: All errors are with OOBTree and as I can see with tempstorage.TemporaryStorage. They look like: li Module ZPublisher.Publish, line 187, in publish_module_standard/li li Module ZPublisher.Publish, line 161, in publish/li li Module ZPublisher.Publish, line 161, in publish/li li Module ZPublisher.Publish, line 161, in publish/li li Module ZPublisher.Publish, line 151, in publish/li li Module Zope2.App.startup, line 158, in zpublisher_exception_hook/li li Module ZPublisher.Publish, line 119, in publish/li li Module Zope2.App.startup, line 215, in commit/li li Module transaction._manager, line 84, in commit/li li Module transaction._transaction, line 381, in commit/li li Module transaction._transaction, line 379, in commit/li li Module transaction._transaction, line 424, in _commitResources/li li Module ZODB.Connection, line 462, in commit/li li Module ZODB.Connection, line 503, in _commit/li li Module ZODB.Connection, line 526, in _store_objects/li li Module tempstorage.TemporaryStorage, line 200, in store/li /ulConflictError: database conflict error (oid 0x0944, class BTrees._OOBTree.OOBTree, serial this txn started with 0x035f134dc11a2977 2005-08-03 12:29:45.258327, serial currently committed 0x035f1357051c7199 2005-08-03 12:39:01.197916) On 2005-08-04, at 09:02 GMT+12:00, Maciej Wisniowski wrote: so I hope that maybe next Zope 2.8.x release will be using MVCC for OOBTree (if this is the point?). Does anybody know something about that? That looks like a write conflict. MVCC helps to lower/eliminate the rate of read conflicts. One option to reduce the rate of this write conflict is to tune the session machinery to suit. For example use session-resolution-seconds of say 300 seconds. michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] accessing session data error
On 22/06/2005, at 4:50 AM, Leticia Larrosa wrote:
Hi:
Thanks to Michael Dunstan. I test with the recomendation of
Michael but I still get the error. In pages where I get the error,
if I wait some seconds and refresh the pages, I obtain the correct
information (no get the error).
I thinks that the problem maybe are similar to the guess of
Michael, but I test breaking up ``init2`` into two scripts and
the error don't disappear. :(
Any help will be useful.
Thanks in advance,
Leticia Larrosa
Hi Leticia,
Just double check that you are redirecting from the init2-first-half
to init2-second-half. Using something like the following in init2-
first-half::
response.redirect('init2-second-half')
rather than directly calling init2-second-half.
Cheers
Michael
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] accessing session data error
Hi Leticia,
The mixture of ``response.write()`` including JavaScript to drive the
browser to a new location along with writing objects in ZODB all
within the same transaction can break some of the promises that you'd
normally expect Zope and a browser to keep.
My guess is that browser is being told to visit ``end`` before the
transaction started by ``init2`` has finished. Perhaps a conflict
error is causing that transaction to take longer than you expect.
Normally this would not trip you up as your browser does not see
anything until after the transaction has finished. (In this normal
mode you might see a conflict error rendered in the browser if the
publication machinery had to give up.)
For your example try breaking up ``init2`` into two scripts. The
first that writes the session data and then redirects the browser to
the second. The second then does not touch session data and manages
all the ``response.write()``s. And remove the unused ``sessionData =
context.REQUEST.SESSION`` from ``external``.
Cheers
Michael
On 21/06/2005, at 6:20 AM, Leticia Larrosa wrote:
Hi all:
I think that I found a ZOPE bug. Is really very important for me
know what can I do for avoid it. The first 9 steps are to recreate
the situation and the other steps are to provoke the error.
Step 1: Crate a folder called reproducingError in Zope Interface
Manage
Step 2: create a page template called index_html in the
reproducingError folder with the following content:
!-- index_html page template --
form name=form1 method=post action=reproducingError/init
input type=text name=user
input type=submit name=Submit value=Submit
/form
Step 3: create a python script called init in the
reproducingError folder with the following content:
# init script
sessionData = container.REQUEST.SESSION
sessionData.set(userId, container.REQUEST.user)
return context.init2_html()
Step 4: create a page template called init2_html in the
reproducingError folder with the following content:
!-- init2_html page template --
a href=init2passing face 2/a
Step 5: create a python script called init2 in the
reproducingError folder with the following content:
# init2 script
request = context.REQUEST
sessionData = request.SESSION
sessionData.set('idService', 'testService')
context.first()
return context.external(context)
Step 6: create a page template called first_html in the
reproducingError folder with the following content:
First Response
Step 7: create a python script called first in the
reproducingError folder with the following content:
# first script
response = container.REQUEST.RESPONSE
str = container.first_html()
response.write(str)
response.flush()
Step 8: create an external method called external in the
reproducingError folder with the following content, in where the
function of the external method is communicationResponse and the
module is any name that you give to the file:
# external method
import time
def communicationResponse(context):
response = context.REQUEST.RESPONSE
sessionData = context.REQUEST.SESSION
strDetailEnd = scriptwindow.location='reproducingError/end'/
script
for item in range(1, 40):
time.sleep(3)
response.write(Response iteration: + str(item))
response.flush()
response.write(strDetailEnd)
response.flush()
Step 9: create a python script called end in the
reproducingError folder with the following content:
# end script
idService = context.REQUEST.SESSION['idService']
return Fin - + idService
Step 10: open two explorer in wich call the object
reproducingError (Example: localhost:8080/reproducingError)
Step 11: enter an example text in each one.
Step 12: clink in the link of one of them, and 5 second later clink
on the link of the other explorer. The waiting for around 5 second
is necesary because with other time interval not allways get the
error.
Stpe 13: and wait the Response iteration... Please if the error
don't appear in any of the pages, begins at Step 9 again. and try
to count to 5 witch other velocity :)
The correct result is:
Fin - testService
The error is:
Site Error
An error was encountered while publishing this resource.
Error Type: KeyError
Error Value: 'idService'
The traceback:
Traceback (innermost last):
Module ZPublisher.Publish, line 101, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 39, in call_object
Module Shared.DC.Scripts.Bindings, line 306, in __call__
Module Shared.DC.Scripts.Bindings, line 343, in _bindAndExec
Module Products.PythonScripts.PythonScript, line 323, in _exec
Module None, line 2, in end
- PythonScript at /approach/reproducingError/end
- Line 2
Module AccessControl.ZopeGuards, line 67, in guarded_getitem
Module Products.Transience.TransientObject, line 176, in __getitem__
KeyError: 'idService'
I will appreciate if you try until the error appear, because I not
[Zope-dev] transience and gc.set_threshold(1)
I'm seeing test failures in transience when running the tests for Zope-2_7-branch (python 2.3.3) with garbage collection threshold0 set to 1. Seems to break in slightly different places from run to run. Here is one example: % python test.py -vv --dir lib/python/Products/Transience/tests -g 1 testPathological gc threshold: (1, 10, 10) Running unit tests at level 1 Running unit tests from /Zope/lib/python/Products/Transience/tests testPathologicalLeftBranching (Products.Transience.tests.testTransientObjectContainer.TestTransientObj ectContainer) ... ERROR testPathologicalRightBranching (Products.Transience.tests.testTransientObjectContainer.TestTransientObj ectContainer) ... ERROR == ERROR: testPathologicalLeftBranching (Products.Transience.tests.testTransientObjectContainer.TestTransientObj ectContainer) -- Traceback (most recent call last): File /Zope/lib/python/Products/Transience/tests/ testTransientObjectContainer.py, line 193, in testPathologicalLeftBranching map(self.t.__delitem__, revr) File /Zope/lib/python/Products/Transience/Transience.py, line 470, in __delitem__ del bucket[k] KeyError: 946 == ERROR: testPathologicalRightBranching (Products.Transience.tests.testTransientObjectContainer.TestTransientObj ectContainer) -- Traceback (most recent call last): File /Zope/lib/python/Products/Transience/tests/ testTransientObjectContainer.py, line 183, in testPathologicalRightBranching map(self.t.__delitem__, r) File /Zope/lib/python/Products/Transience/Transience.py, line 470, in __delitem__ del bucket[k] KeyError: 69 -- Ran 2 tests in 4.659s FAILED (errors=2) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] transience and gc.set_threshold(1)
On 6/10/2004, at 4:06 AM, Tim Peters wrote: [Michael Dunstan] I'm seeing test failures in transience when running the tests for Zope-2_7-branch (python 2.3.3) with garbage collection threshold0 set to 1. snip Whenever I see a test module import fauxtime, I run screaming in terror. This seems to be like running the multiprocess ZEO tests (from which I run screaming daily wink): depending on a gazillion timing accidents you can't control, sometimes they just fail. I ran the tests above in a loop a few hundred times on my box just now, with no failures, Python 2.3.4 and command line: test.py -vvLd -g1 --dir lib/python/Products/Transience/tests . testPathological There's a big difference between my runs and yours, though: running the two tests took about 4.6 seconds on your box, but takes about 1.8 seconds on mine. Now, for example, if I add oldtime.sleep(3) # approximately 4.6 - 1.8, on the high side right before map(self.t.__delitem__, r) in testPathologicalRightBranching, then testPathologicalRightBranching fails every time in a way similar to your reported failure -- and even without -g1 on the command line. If I change the sleep to 1.5, then (with -g1 again) sometimes it passes, sometimes not, depending on how busy the box is doing other stuff at the same time. So that's A Problem. Any test that adds a bunch of stuff to a TransientObjectContainer and expects to *find* it all there later is implicitly assuming that the test runs fast enough so that containees don't vanish before their containedness is checked. I suspect you're seeing problems with -g1 here just because -g1 grossly slows the tests. Doh! For some reason I had written that explanation off before posting. The pattern of missing keys did not seem consistent enough at the time. But, as you say, as soon as I comment out the use of faxtime and run those same two tests they both pass just fine. thanks michael ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] session problem again?
On 29/09/2004, at 11:07 AM, Matteo Memelli wrote: Hello I include the conversation that I had today with Chris: I'm using Zope 2.6b1+ with ZWarehouse e-commerce product (ZWarehouse-20040522 version 0.8) I tried upgrading to Zope 2.7 but ZWarehouse gave me many problems so I decided to try applying your patches for Sessioning problems that I have. Hmm.. I think that the error that zope gives to me is very known to you... I tried to copy the whole directories Transience, Session and ZPublisher from both Zope-2.7.3b1-0 and Zope 2.7.2 but without success I also copied the file startup.py from lib/python/Zope/App/ ... Am I missing anything? Yes, the ZODB directory also... although I reallly have no idea whether this will work under 2.6. Really, to be sure you've got all the right files, it'd be best if you could upgrade to 2.7.3... maybe we could help fix whatever in ZWarehouse doesn't work under there (move this conversation to the Zope list [EMAIL PROTECTED]) To be thorough, you will also need to delete and recreate any persistent session data container you've got laying around (unless it's in temp_folder, that's in RAM anyway and gets recreated on reboot). My customer is complaining so much and I really don't know what to do! Hoping in your answer...thank you in advance Matteo Memelli Site Error An error was encountered while publishing this resource. KeyError Sorry, a site error occurred. Traceback (innermost last): * Module ZPublisher.Publish, line 175, in publish_module_standard * Module Products.iHotfix, line 83, in new_publish * Module ZPublisher.Publish, line 132, in publish * Module Zope.App.startup, line 203, in zpublisher_exception_hook * Module ZPublisher.Publish, line 101, in publish * Module ZPublisher.mapply, line 88, in mapply * Module ZPublisher.Publish, line 39, in call_object * Module OFS.DTMLDocument, line 128, in __call__ DTMLDocument instance at 41725260 URL: http://www.arte-luce.com/catalog/html/catalog.html/manage_main Physical Path:/arteluce/catalog/html/catalog.html * Module DocumentTemplate.DT_String, line 474, in __call__ * Module DocumentTemplate.DT_Let, line 76, in render * Module OFS.DTMLDocument, line 121, in __call__ DTMLDocument instance at 4172b6b0 URL: http://www.arte-luce.com/catalog/html/zwarehouse_html_header/ manage_main Physical Path:/arteluce/catalog/html/zwarehouse_html_header * Module DocumentTemplate.DT_String, line 474, in __call__ * Module DocumentTemplate.DT_Util, line 201, in eval __traceback_info__: cart_functions * Module string, line 1, in expression * Module Shared.DC.Scripts.Bindings, line 306, in __call__ * Module Shared.DC.Scripts.Bindings, line 343, in _bindAndExec * Module Products.PythonScripts.PythonScript, line 318, in _exec * Module None, line 16, in setSessionByRequest.py PythonScript at /arteluce/catalog/cart_functions/setSessionByRequest.py Line 16 * Module ZPublisher.HTTPRequest, line 1218, in __getattr__ * Module ZPublisher.HTTPRequest, line 1178, in get * Module Products.Sessions.SessionDataManager, line 94, in getSessionData * Module Products.Sessions.SessionDataManager, line 181, in _getSessionDataObject * Module Products.Transience.Transience, line 847, in new_or_existing * Module Products.Transience.Transience, line 450, in __setitem__ KeyError: 1096365080 (Also, an error occurred while attempting to render the standard error message.) ___ Troubleshooting Suggestions * The URL may be incorrect. * The parameters passed to this resource may be incorrect. * A resource that this resource relies on may be encountering an error. For more detailed information about the error, please refer to error log. If the error persists please contact the site maintainer. Thank you for your patience. Hmmm.. that's an odd place to get that error. I'm surprised. Does it only happen after some period of inactivity on the site? - C FOLLOWING Chris's advice first I tried to copy also ZODB directory from Zope 2.7.3 but still the same error... so I Installed Zope-2.7.3b1-0 (IS THIS THE RIGHT VERSION THAT SHOULD SOLVE SESSION PROBLEMS?) and reinstalled with success the following packages: ZWarehouse 0.8 iHotfix-0.4.0 itools-0.4.6 Localizer-1.1.0a4 ZMySQLDA-2.0.8 MySQL-python-1.0.0 TranslationService-0.4 THEN imported my customer's web site but I still have the same problem... whats going on... what am i missing? For every change I made I
Re: [Zope-dev] Transaction problems and publisher
On 25/08/2004, at 12:55 AM, Florent Guillaume wrote: I've reopened bug 789 (http://collector.zope.org/Zope/789) because we observed problems in Zope 2.7.2 with transaction abort by the publisher. The problems can make objects disappear or provoke POSKeyErrors. Here is a patch for publisher that may help. I'm currently running some tests for sessions and transience. And this patch seems to iron out a lot problems that have been reported with transience of recent. (I failed to dance in just the right way to attach this patch to the collector.) michael publisher-abort.patch Description: Binary data ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: hasattr geddon
On 12/07/2004, at 12:29 PM, Chris McDonough wrote: FWIW, as far as I understand the prevent commit on conflict behavior *is* currently the behavior for caught ReadConflictErrors. The only time ZODB doesn't exhibit this behavior is *during the commit of a transaction*. If a commit is attempted at this point and a conflict error is caught unintentionally, the following happens (as explained by Tim at http://mail.zope.org/pipermail/zodb-dev/2004-May/007404.html : - the attempt to commit fails, raising a write conflict error - the write conflict error is unintentionally caught (whether by hasattr or a bare except or by whatever) - a new transaction begins implicitly (as if get_transaction().abort() or get_transaction().begin() were called) - ... whatever happens after this happens .. Since it's extremely unlkely for a write conflict error to be caught unintentionally in Zope (it would require an application to perform a get_transaction().commit() going outside the boundaries of the default transaction manager), it's an obscure failure mode. If the transaction was not put into a committable state after a caught exception error, there would be no problem. But to be fair, I don't think there is much of a problem right now, as very few Zope apps manage their own transactions (except for the Catalog... gulp ;-). But it would be nice if ZODB didn't do the implicit get_transaction().abort() after a write conflict error and left the transaction in an uncommittable state instead... or at least had an option to do this. So... it sounds like ZODB will take care of itself in the face of ReadConflictErrors swallowed by bare excepts (and hasattr's). (Unless your managing your own transactions.) One penalty of bare excepts swallowing a ConflictError is running more code than is needed. The transaction is going to be thrown away eventually. Is there problem with seeing bogus exceptions in code after a swallowed ConflictError? I'm thinking that when zope catches any exception in a transaction it should check the state of the transaction to decide if should treat this as a ConflictError or report the exception it caught. Consider a contrived example:: def setFoo(self, value): try: self.foo = value except: pass self.fooUpdated() Assume that the except swallows some ConflictError then an exception is raised from fooUpdated() but not a ConflictError, lets say a KeyError. Sure, ZODB is tough enough to look after itself in this case. What about the developer trying to interpret the KeyError that is written to event.log? (Hmm... I should really check that is what you could potentially see with a current Zope :-) Michael ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Conflict errors on BDBMinimal storage
On 18/05/2004, at 6:57 AM, Gerhard Schmidt wrote: On Mon, May 17, 2004 at 01:38:15PM -0400, Chris McDonough wrote: AFAIK Zope is the only appserver that even makes an attempt at allowing session data to be shared between servers like this. Most other appservers require you to set up session affinity in your load balancer and have a session database on each appserver. and nealy every system failed to do so. How do you implement this session affinity. Based on source IP. At the moment every large ISP uses multiple Proxyserver so every request comes from a different IP so the affinity doesn´t work for them. The Proxyserver provided by our Computing Center is also split on at least 8 Hosts, so everyone using this proxy would lose session infomation, getting very strange effekts. Perhaps you can get some relief by using pound as your load balancer. It is quite capable at sniffing out user sessions including watching the Cookie header. See http://www.apsis.ch/pound/ Michael ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On 18/05/2004, at 10:03 PM, Andrea Patuzzo wrote: Hi, dear developers: We are now randomly getting this kind of error: [...] File /usr/local/Zope270CVS/lib/python/Products/Transience/Transience.py, line 341, in __setitem__ current_bucket = self._data[current_ts] KeyError: 1084872640 This one was with the latest publish.py / startup.py patch and the new Transience.py implementation, on a yesterday's Zope-2_7-branch checkout. New Zope instance with FileStorage does not solve the problem. Same happens with the patches applied to the official Zope-2.7.0 release. Note: once the error happens, zope has to be restarted to function again. Before applying patches we were getting get errors (instead of __setitem__) and we just had to close browser to get back to work. Here we have to restart Zope (with TemporaryStorage) or delete Sessions.fs and restart (with FileStorage). My guess at the moment is that somewhere a ConflictError is being swallowed. There are a few things that can be responsible for this: - Use of hasattr on Persistent objects. - try/except blocks that don't raise ConflictErrors. From your trace that looks like a Plone site. A very quick glance through Plone and its friends indicates that there may be some chance of either of the above causing some problems. Although Plone 2.0.1 includes the following fix: - Fixed bare 'except:' statements in Plone so we don't swallow ZODB ConflictErrors. Mostly by adding 'except ConflictError: raise'. [stefan] Someone please shoot me down and claim that Plone and friends all do the right thing. ;-) Michael ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Conflict errors on BDBMinimal storage
On 19/05/2004, at 8:36 AM, Chris McDonough wrote: As a result, I am coming to believe that along with the errors as part of main transaction patch for 2.7.1, I should also make the retry-on-conflict-error policy pluggable for those who really desperately need to slow their Zope systems to a conflict-induced crawl. They can bump it up to 100 as far as I care and if it works for them, great. Or this could be used to avoid retries altogether. And at the very least it would advertise the fact that zope does retry requests. Michael. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On 18/05/2004, at 5:42 AM, Chris McDonough wrote: On Mon, 2004-05-17 at 13:06, Steve Jibson wrote: Also of note: 1 - After having the system run for 45 minutes, I had 8 similar errors. Some were on different web pages and some had different keys that were causing the error. 2 - I have also traced through Z2.log and followed the same path through the web site that produced one of the errors and I did not get an error. 3 - Since changing this server to use FileStorage (1hr 39min ago), there has not been a single error. Is there anything I can do to help here? A small reproducible test case would help if you still believe this error is not in your own application. Looks like session data can expire prematurely. See attached files for small changes to the test rig that reports cases where context.session_data_manager.hasSessionData() is False. Michael SessionRigExtensions.py Description: Binary data TestSessionRig.py Description: Binary data ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On 14/05/2004, at 10:06 AM, Chris McDonough wrote: I am tempted to check the following into the 2.7 branch and HEAD: - error occurs in same transaction as main request patch to Publish.py. See http://www.plope.com/Members/chrism/ publishpy_errorinmaintrainsaction.patch/file_view for the patch. hmm... I suspect there are few holes in this... - Need to make sure there is no get_transaction().begin() in zpublisher_exception_hook() of lib/python/Zope/App/startup.py That begin() would mean that the error handling is not in fact part of the original transaction? - An exception within err_hook() (aside from a Retry) will mean that abort() is never called. Here is another attempt at publishpy_errorinmaintrainsaction.patch: Index: lib/python/ZPublisher/Publish.py === RCS file: /cvs-repository/Zope/lib/python/ZPublisher/Publish.py,v retrieving revision 1.164.2.2 diff -u -r1.164.2.2 Publish.py --- lib/python/ZPublisher/Publish.py17 Nov 2003 22:34:19 - 1.164.2.2 +++ lib/python/ZPublisher/Publish.py15 May 2004 02:34:12 - @@ -105,7 +105,6 @@ return response except: -if transactions_manager: transactions_manager.abort() # DM: provide nicer error message for FTP sm = None @@ -119,29 +118,36 @@ if err_hook is not None: -if parents: parents=parents[0] try: -return err_hook(parents, request, -sys.exc_info()[0], -sys.exc_info()[1], -sys.exc_info()[2], -) -except Retry: -# We need to try again -if not request.supports_retry(): +if parents: parents=parents[0] +try: return err_hook(parents, request, sys.exc_info()[0], sys.exc_info()[1], sys.exc_info()[2], ) -newrequest=request.retry() -request.close() # Free resources held by the request. -try: -return publish(newrequest, module_name, after_list, debug) -finally: -newrequest.close() +except Retry: +if not request.supports_retry(): +return err_hook(parents, request, +sys.exc_info()[0], +sys.exc_info()[1], +sys.exc_info()[2], +) +finally: +if transactions_manager: transactions_manager.abort() + + +newrequest=request.retry() +request.close() # Free resources held by the request. +try: +return publish(newrequest, module_name, after_list, debug) +finally: +newrequest.close() + +else: +if transactions_manager: transactions_manager.abort() +raise -else: raise def publish_module_standard(module_name, Index: lib/python/Zope/App/startup.py === RCS file: /cvs-repository/Zope/lib/python/Zope/App/startup.py,v retrieving revision 1.8.2.3 diff -u -r1.8.2.3 startup.py --- lib/python/Zope/App/startup.py 20 Dec 2003 04:48:36 - 1.8.2.3 +++ lib/python/Zope/App/startup.py 15 May 2004 02:34:13 - @@ -176,8 +176,6 @@ published=app.__bobo_traverse__(REQUEST).__of__( RequestContainer(REQUEST)) -get_transaction().begin() # Just to be sure. - published=getattr(published, 'im_self', published) while 1: f=getattr(published, 'raise_standardErrorMessage', None) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On 11/05/2004, at 4:54 PM, Chris McDonough wrote: On Mon, 2004-04-26 at 18:34, Michael Dunstan wrote: I removed all the above hacks leaving just the one that comments out the construction of the to_notify list. Reran the test rig. And have yet to see any KeyErrors. Have you seen any since the time you reported this? (I chatted about this with Chris on #zope.) Since the last report I have not run been running the test rig. I Needed my laptop back so I could get on with a few other things. :-( So I'm going to see if I can show others how to recreate the test rig that I'm using here. A trivial site and the script for testing that site are attached. (Reminder - I'm using Zope 2.7.0 here - HEAD just is not ready for such games. :-) SessionRigExtensions.py and sessionrig.zexp make up the site. sessionrig.zexp is simply a bunch of external methods (with a folder to provide some separation of a standard_error_message that makes use of sessions). SessionRigExtensions.py has all the code for the external methods. TestSessionRig.py is the script which attacks zope. It should just work as is. (Unless the site you just imported is not found at http://localhost:8080/sessionrig. If that's the case then simply edit the 'path' in the code.) Don't expect the output of this script to say anything conclusive. Instead you should be tailing + greping through the event log for things of interest. Now the knobs that you should tweak to so that you see things of interest within minutes/hours rather than days/weeks... (The actual values may depend on the hardware that your running this on.) * etc/zope.conf: session-timeout-minutes 1 maximum-number-of-session-objects 1000 * lib/python/Products/Tranience: PERIOD = 1 #20 # attempt housekeeping every PERIOD seconds SPARE_BUCKETS = 2 #15 # number of buckets to keep spare * lib/python/tempstorage/TempoaryStorage.py: # keep old object revisions for CONFLICT_CACHE_MAXAGE seconds CONFLICT_CACHE_MAXAGE = 1 #60 # garbage collect conflict cache every CONFLICT_CACHE_GCEVERY seconds CONFLICT_CACHE_GCEVERY = 1 #60 * lib/python/Products/SiteErrorLog/SiteErrorLog.py: _rate_restrict_burst = 10 #5 (I wanted to see repeated errors in the event log. There is probably a more obvious way to do that. But that's what I'm using at the moment. Took me a while to actually find out this was why I was not seeing all the exceptions in the event log. Perhaps things like these should be exposed via zconfig? Makes a useful medium to document such knobs to promote awareness. As well as allowing people that are pushing zope into new corners to have a simpler way to manage these kinds of settings.) * TestSessionRig.py - there are few things that you can tweak within here too. (Well the whole script is up for grabs - hack away - I did.) As the script is it should expose problems with lack of transaction isolation for standard_error_message. I do tend to run several instances of this script at once. And in my case (as the url suggests) I run this on the same host as zope is running on. However to expose the KeyError thrown from TempoaryStorage I found that I did not really need much 'sleepiness' and replaced all the start()'s at the end of the script with start(200,1,100,10). (I suspect that you don't even need to support cookies in the client for this error. Sufficient to hit any URL that simply creates a session object on the server. And hit that URL very hard.) Another effective knob that you may try playing with is the nap duration used to slow down a session write: slowWrite = path + '/write?nap:float=0.5' hmm... that's probably enough to help someone recreate a KeyError or two. Michael. sessionrig.zexp Description: Binary data SessionRigExtensions.py Description: Binary data TestSessionRig.py Description: Binary data ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.X Session problems
On 7/05/2004, at 5:15 AM, Kris Erickson wrote: No, that's not the problem; in THEORY that's what is happening, but in reality there is no way that this is the case; We just unrolled a registration system with participation rates at or around 100 to 200 participants per month; At any given time, monitoring the session data container, there are *at most* 1 or 2 items in the transient object container--EXCEPT when it spikes... I have seen such spikes occur (in a corner case) where some breads of web robots were aggressively hitting a page that used sessions. These robots did not bother to return the cookie handed out by the server. Each page hit effectively constructs a new session. Have a look through your access logs to see if can see signs of something similar happening. Not all web robots are created equal. I ended up sniffing for the user agent and returning a page that does not use sessions for the offending robots. (From memory, robots.txt was not useful for this bread.) Alternatively you can set the maximum-number-of-session-objects to something a lot higher and see if you can just live through the bot invasion. Michael. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Zope 2.X Session problems
On 7/05/2004, at 4:39 PM, Tres Seaver wrote: Michael Dunstan wrote: On 7/05/2004, at 5:15 AM, Kris Erickson wrote: No, that's not the problem; in THEORY that's what is happening, but in reality there is no way that this is the case; We just unrolled a registration system with participation rates at or around 100 to 200 participants per month; At any given time, monitoring the session data container, there are *at most* 1 or 2 items in the transient object container--EXCEPT when it spikes... I have seen such spikes occur (in a corner case) where some breads of web robots were aggressively hitting a page that used sessions. These robots did not bother to return the cookie handed out by the server. Each page hit effectively constructs a new session. Have a look through your access logs to see if can see signs of something similar happening. Not all web robots are created equal. I ended up sniffing for the user agent and returning a page that does not use sessions for the offending robots. (From memory, robots.txt was not useful for this bread.) Alternatively you can set the maximum-number-of-session-objects to something a lot higher and see if you can just live through the bot invasion. Even better, avoid writing to the session on each request! Your application will be *much* happier if you write to the session only when the human makes a gesture; neither bots nor casually-browsing humans will consume sessions, but only session keys (which are cheap). Yup - certainly that is a whole lot better if you can arrange for that. Michael. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On 11/03/2004, at 7:25 PM, Chris McDonough wrote: On Wed, 2004-03-10 at 23:14, michael wrote: Thanks for your excellent isolation here. I don't get a KeyError thrown until *after* at least one rendering of standard_error_message that includes an access to a session variable. And I'm only getting KeyError's for sessions that have had a chance to expire. And that KeyError will be thrown whenever any access is made for the session. (Previously I thought I was *only* seeing KeyError's being thrown from standard_error_message.) I think after provide transaction isolation for the duration of an error, we will not see this symptom occur again. Okay - I've gone ahead and implemented a flavour of transaction isolation for errors. (See below for patch.) That seems to have done the trick just nicely. I have not seen any more KeyError's thrown from Transience.py. Yay! http://collector.zope.org/Zope/848 has a couple of examples of such KeyErrors thrown from Transience. These KeyError were a very painful user experience as either the server needed to be restarted or the user needed to delete their cookie for that server from their browser cookie jar. However there is another KeyError reported in 848 too. This one is thrown from TempoaryStoarge. After some further tweaking of my test rig I was able to produce such KeyErrors quite frequently. Using Zope 2.7.0 and recent the rewrite of Transience.py: http://cvs.zope.org/*checkout*/Products/Transience/Transience.py? rev=1.32.12.2.2.2only_with_tag=chrism-sessiongeddon errors look like: 2004-04-27T08:24:16 BLATHER(-100) OFS Exception while rendering an error message Traceback (most recent call last): File /Users/michael/builds/zope/Zope/lib/python/OFS/SimpleItem.py, line 204, in raise_standardErrorMessage v = s(**kwargs) File /Users/michael/builds/zope/Zope/lib/python/Shared/DC/Scripts/ Bindings.py, line 306, in __call__ return self._bindAndExec(args, kw, None) File /Users/michael/builds/zope/Zope/lib/python/Shared/DC/Scripts/ Bindings.py, line 343, in _bindAndExec return self._exec(bound_data, args, kw) File /Users/michael/builds/zope/Zope/lib/python/Products/PythonScripts/ PythonScript.py, line 318, in _exec result = f(*args, **kw) File Script (Python), line 12, in standard_error_message File /Users/michael/builds/zope/Zope/lib/python/ZPublisher/HTTPRequest.py, line 1218, in __getattr__ v = self.get(key, default, returnTaints=returnTaints) File /Users/michael/builds/zope/Zope/lib/python/ZPublisher/HTTPRequest.py, line 1178, in get if callable(v): v = v() File /Users/michael/builds/zope/Zope/lib/python/Products/Sessions/ SessionDataManager.py, line 93, in getSessionData return self._getSessionDataObject(key) File /Users/michael/builds/zope/Zope/lib/python/Products/Sessions/ SessionDataManager.py, line 180, in _getSessionDataObject ob = container.new_or_existing(key) File /Users/michael/builds/zope/Zope/lib/python/Products/Transience/ Transience.py, line 504, in new_or_existing item = self.get(key, _marker) File /Users/michael/builds/zope/Zope/lib/python/Products/Transience/ Transience.py, line 332, in get item = self._move_item(k, current_ts, _marker) File /Users/michael/builds/zope/Zope/lib/python/Products/Transience/ Transience.py, line 208, in _move_item self._gc(current_ts) File /Users/michael/builds/zope/Zope/lib/python/Products/Transience/ Transience.py, line 429, in _gc for key in list(self._data.keys(None, max_ts)): File /Users/michael/builds/zope/Zope/lib/python/ZODB/Connection.py, line 561, in setstate p, serial = self._storage.load(oid, self._version) File /Users/michael/builds/zope/Zope/lib/python/tempstorage/ TemporaryStorage.py, line 94, in load s=self._index[oid] KeyError: '\x00\x00\x00\x00\x00\x00\x08\x82' So far it looks like commenting out the construction of the to_notify list in _gc makes this go away. So _gc looks like: def _gc(self, now=None): if self._timeout_slices == 0: return # dont do gc if there is no timeout if now is None: now = getCurrentTimeslice() # for unit tests max_ts = now - (PERIOD * (self._timeout_slices + 1)) keys = self._data.keys(None, max_ts) #to_notify = [] for key in list(self._data.keys(None, max_ts)): assert(key = max_ts) STRICT and _assert(self._data.has_key(key)) #for v in self._data[key].values(): #to_notify.append(v) del self._data[key] #for v in to_notify: #self.notifyDel(v) Note that the test rig does not set onAdd or onDelete. Along the way there where a few things that I was not too sure about so I tried fiddling with to see what would happen: (1) Use of hasattr in Transience.py. hasattr should be avoided? See http://mail.zope.org/pipermail/zodb-dev/2003-October/005968.html. I did change usage
Re: [Zope-dev] Re: ZPT for CSS, anyone?
On 31/03/2004, at 7:31 AM, Tres Seaver wrote:
Overall, I agree with you that stylesheets are best kept static.
However, consider what happens to cacheability when the URLs of of
images referenced in a document are relative; the same problem can
afflict stylesheets, particularly in a setup where virtual hosting is
in play.
I strongly favor DTML over ZPT for those cases where you need to
generate dynamic plain text (mail, CSS, Javascript, etc.) Neither
will be transparent to tools like Dreamweaver, but then again I
can't imagine *any* markup that would be transparent; it would pretty
much have to be embedded in whatever comment syntax the underlying
language defines.
hmmm... would a similar idea to TAL ease the pain? A sprinkling of new
CSS properties along the lines of...
p {
padding: 1.5em;
foo-property: padding context/padding;
}
Could even apply similar ideas for i18n for properties such as
content.
Michael
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
