Re: [Zope] Florent Guillaume's database Conflict Error on session objects patch not working ?
On 2/3/07, yacine chaouche <[EMAIL PROTECTED]> wrote: I am experiencing issues regarding a database ConflictError on session objects. Can you detail all the various getSesion and setSession methods of: http://mail.zope.org/pipermail/zope/2007-January/170167.html I use zope 2.9.0 python 2.4.2. It's worth trying the latest of each of those. Cheers Michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Inavlid Refernce Counts in Temporary Storage
On 7/14/06, Philip Kilner <[EMAIL PROTECTED]> wrote: For the last three days, we have been plagued by slowdowns and temporary lock-ups, which have on a couple of score occasions culminated in a Conflict Error, indicting transactional problems in the temporary storage. On four or five occasions, Zope has apparently "locked up", and a restart has been required. The conflict error's may well just be a by-product of the slow down. Roughly speaking, the longer a request takes the higher the chances are of conflict error. There are various tools for trying to see what zope is doing. Google for: debug spinning zope. cheers michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Trying to trap ConflictError
On 7/7/06, Michael Dunstan <[EMAIL PROTECTED]> wrote: An oid is an identifier for a persistent state of an object. As the state of an object evolves from transaction to transaction it gets a new oid for each state. Two concurrent transactions that involve the same object will start with the same state, i.e. the same oid, for that object. Sorry. Ignore that. :-( Note to self, don't even read email before morning caffeine intake. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Trying to trap ConflictError
On 7/7/06, Jonathan <[EMAIL PROTECTED]> wrote: I haven't been able to determine how the oid's are generated, other than "little-endian 64-bit unsigned integers that will be assigned more or less sequentially", but I can't figure out how the same oid is being used twice (on the other hand I could be totally off-base as I am really way over my head here!). An oid is an identifier for a persistent state of an object. As the state of an object evolves from transaction to transaction it gets a new oid for each state. Two concurrent transactions that involve the same object will start with the same state, i.e. the same oid, for that object. Some detail about how BTrees behave can be found in http://www.zope.org/Wikis/ZODB/BTreeConflictResolution Also might be illustrative to run fsdump to see what is happening in your transactions. See http://www.zope.org/Wikis/ZODB/FileStorageBackup. You'll need FileStorage for that. And that only shows you details of successful transactions. cheers michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Trying to trap ConflictError
On 7/6/06, Jonathan <[EMAIL PROTECTED]> wrote: I am still investing, and have found errors like: 2006-07-04T14:48:12 ERROR ZODB.Connection Couldn't load state for 0x1c Traceback (most recent call last): File "/usr/local/Zope-2.9.2/lib/python/ZODB/Connection.py", line 732, in setstate self._setstate(obj) File "/usr/local/Zope-2.9.2/lib/python/ZODB/Connection.py", line 768, in _setstate p, serial = self._storage.load(obj._p_oid, self._version) File "/usr/local/Zope-2.9.2/lib/python/tempstorage/TemporaryStorage.py", line 104, in load s=self._index[oid] KeyError: '\x00\x00\x00\x00\x00\x00\x00\x1c' in the event.log. From my googling research, it appears that these type of errors may be due to insufficient memory (but I am not sure and am still investigating to see if this is the cause of the conflict errors). TempoaryStorage.py has a few tuning constants. The above error reads as though RECENTLY_GC_OIDS_LEN is too small for your application. It may well be that the default tuning of TempoaryStorage does not match your application very well. It would be useful to try FileStorage to help decide on where to focus attention. If FileStorage behaves a lot better then that also suggests that TempoaryStorage is miss-tuned for your application. Cheers Michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] More on understanding conflicts
On 12/22/05, Dennis Allison <[EMAIL PROTECTED]> wrote: > The issue here has had to do with session variables and their iteraction > with the persistence mechanism and conflicts and multiple threads for the > same session. Chris McDonough has pointed out that session variables can > cause writes and write conflicts even if the only Zope level access is a > read. I thought MVCC works out of the box for Zope 2.8.4 which uses ZODB > 2.3.4. Am I wrong? > > The real surprise was that you can get a write conflict from a pair of > session variable reads! The missing detail here is that reading a session object causes a write to the database to update the last access time for that session object. So what looks like a plain old read to the application code above, in-fact includes a write just below the surface. (This is a design choice so that stale session objects can be removed.) There are implementation details of TransientObject that attempt to mitigate some of the pain: - The access time has a WRITEGRANULARITY. That can help a bit by avoiding writing altogether some of the time. But still there is need to write (and so potential contention) each time a boundary of that granularity is crossed. - And _p_resolveConflict of TransientObject will most often pick a state to avoid a conflict error. When it does there is the performance cost of writing to database but there will be no ConflictError. However, I've suggested in the past that you should comment out _p_resolveConflict while debugging consistency problems. What is the current state of your _p_resolveConflict now? _p_resolveConflict includes a DIWM'ly component. Perhaps you still need to avoid that component of _p_resolveConflict? In that case a _p_resolveConflict could be engineered that can resolve the case of just updates to the access time? But still bails out when presented with two states that have explicit updates to maintain consistency. Though that's all just conjecture till we see your tracebacks for your conflicts. michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: Zope Persistence (was: XML-RPC within ZOPE)
On 12/19/05, Jan-Ole Esleben <[EMAIL PROTECTED]> wrote: > However, even if > it is only the docs that are lacking I think it would be sensible to > acknowledge that as a problem. Zope 3 is the literate child of Zope 2. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: Zope Persistence (was: XML-RPC within ZOPE)
On 12/19/05, Jan-Ole Esleben <[EMAIL PROTECTED]> wrote: > > Little bit tricky to try out as testers need to guess what all the > > missing code is. > > Any standard persistent ZOPE product wrapped around this will do. > These are the only methods in a ZOPE product that inherits from Item, > Persistent, RoleManager and Implicit. > > > Also, for this kind of code demonstration, rather > > than directions for commenting/uncommenting and relying on the tester > > to restart the server between trails, provide different methods or > > even different classes. > > If I did that, people would say "somethings wrong with your testing" > and they would have to do a line-by-line comparison and still have > room to doubt my statement. /me shurgs. I prefer code that runs. :-) > > But there is no extra transactional framework for XML-RPC clients > > (which your example has). Or for that matter acting as an XML-RPC > > server nested within some external transaction (which your example > > has). > > I was initially talking about a more complex situation that has been > gradually simplified. ZOPE should be able to act as an XML-RPC client > to the outside world. > > As to me not understanding what you all are saying: I believe I > understand, I know that theoretically - from a merely "technical" > viewpoint, it works, I just look at it from a different angle. Of > course you have to know a framework and its limitations, but this > error I stumbled upon strikes me as really complex (even though you > keep repeating it's very simple if you just acknowledge xyz), for > instance because the docs "aren't perfect" and say that > "You must explicitly signal any changes made to mutable attributes > (such as instances, lists, and dictionaries) or use persistent > versions of mutable objects, like ZODB.PersistentMapping (see below > for more information on PersistentMapping.)" - the _first_ option > being mutable attributes! Also, there is no "below". > I believe you when you say that it is really very simple and I > shouldn't have to worry if I follow the right principles, that you > then go on to state (principles that aren't really that explicit > anywhere - such as "don't use XML-RPC to talk to ZOPE from ZOPE" and > "don't use mutable subobjects at all"). However, this is a further bit > of distance between a ZOPE user and the Python he knows, and it also > isn't as true for complex situations with complex requirements that I > could probably just jot down in "dumber" frameworks such as RoR or > Django or TurboGears. Because I know without having to check myself > where I am designing Data, I explicitly _make_ changes (to the DB) and > am thus very aware of where complex interactions might occur etc. This > is probably one of those cases where no one is really "right" because > a mixture of viewpoints is involved, I feel a bit as if we were > talking about wether Lisp is better than Smalltalk. However, even if > it is only the docs that are lacking I think it would be sensible to > acknowledge that as a problem. It obviously was a problem for me, as > of course was the complexity of my setup and several other factors. I > am not an inexperienced programmer, so I would think that just because > other people rarely stumble upon it it is not an insignificant > problem. Imagine designing a large system based on some only > marginally false assumptions about ZOPE transactions, persistence and > XML-RPC interaction, and then running into this in a key component. Joining many transactional systems into one mega transactional beast is requires attention to details. How about getting away from this notion a mega transaction with nested transactions? For example, make use of transaction.commit() before making XML-RPC calls. Though you well need to decide what happens in the event of a ConflictError being thrown from somewhere else and causing the initial request to be retried. michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: Zope Persistence (was: XML-RPC within ZOPE)
On 12/19/05, Jan-Ole Esleben <[EMAIL PROTECTED]> wrote: > > > 1. In the example, just setting _p_changed=1 does _not_ lead to a > > > conflict error. With the ineffectual code above it (that never gets > > > executed) it _does_. So there _is_ some implicit magical stuff going > > > on and ZOPE tries to take care that only subobjects change (but > > > incompletely)! > > I strongly doubt it. Zope does not "inspect code". There must be a > > problem in your testing. Note that if self.a is a standard list, the > > self.a.append(1) doesn't have any impact on the persistence mechanism or > > transactions either. > > Please, try it out. Delete the if clause and the append in it. Little bit tricky to try out as testers need to guess what all the missing code is. Also, for this kind of code demonstration, rather than directions for commenting/uncommenting and relying on the tester to restart the server between trails, provide different methods or even different classes. Meanwhile I'm camping over at I-strongly-doubt-it. > > > 3. It is especially confusing that ZOPE behaves differently when using > > > XML-RPC calls. From what you say, it should be the same within the > > > ZOPE system as when using XML-RPC. It gets more complicated with > > > XML-RPC though! > > The successive XML-RPC call you describe provoke new transactions, > > surely you're aware of that? Whereas just calling a function of course > > doesn't. > > I'm aware of that. But ZOPE offers XML-RPC and as there is nothing in > the documentation about such complex interactions. Also, I was > directly responding to what the previous poster had written. Zope provides for XML-RPC as a server. And it does so with internal transaction support. But there is no extra transactional framework for XML-RPC clients (which your example has). Or for that matter acting as an XML-RPC server nested within some external transaction (which your example has). Either design to avoid this or get your hands dirty with transactions. Some starting points for examples would include any of the relational database adapters. Also in the same space is MaildropHost. michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] resolving conflict errors
On 12/11/05, Dennis Allison <[EMAIL PROTECTED]> wrote: > > Good idea, but it is hard to do in a production environment with a "never > lose data" model. Have a go at recreating the problems you are seeing on a development host. SessionRig can be used to mount a brute force attack of the session machinery. You'll need to tune that somewhat to your particular application. http://cvs.zope.org/Packages/SessionRig michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] resolving conflict errors
On 12/9/05, Dennis Allison <[EMAIL PROTECTED]> wrote: > The problem I am trying to resolve appears to be load related. The > observed symptom is that (some) session variables spontaneously disappear. > There appears to be some connection to conflicts, but the exact mechanism > and the relationship is not yet clear. A small possibility is that you are being bitten by the DWIM'ly nature of TransientObjects conflict resolution where the last modified state is chosen over the others. If you think this is biting you then try commenting out _p_resolveConflict of TransientObject. That's bound to increase the rate of conflict errors but should provide you with a consistent session state. Perhaps useful as a debugging step. michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] conflicts, sessions, and programming best practices guidelines
On 11/18/05, Chris McDonough <[EMAIL PROTECTED]> wrote: > > On Nov 17, 2005, at 4:46 AM, Dennis Allison wrote: > > > > > Thank Chris, good pointsB. > > > > The session variable timeout is currently several hours as requird > > by our > > application. The timeout resolution remains at its default value. > > Making the resolution higher has been observed to reduce conflicts > (see dunny's chart). If you sessions live for several hours then perhaps you want a increase session-resolution-seconds to something like 1800 (30 minutes) or more. Also worth seeing if increasing WRITEGRANULARITY of Products.Transience.TransientObject helps you. > > I have > > not dorked with disabling the inband housekeeping. I don't see how it > > would impact on our problem. Am I missing something? > > Turning off inband housekeeping has the potential to reduce the > numnber of conflicts because less work is done during normal > sessioning operations. Instead of doing the work to figure out > whether it needs to clean up after itself during "normal" operations > it relies on an external process. And it is not that difficult to do. Michael. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Help!!! Zope won't start -- StorageSystemError
On 8/28/05, Dan Gaibel <[EMAIL PROTECTED]> wrote: > Hi folks, > > This is my first post to this list and I'm desperate for help. I'm running > Zope 2.6.1 with Python 2.1.3. My Zope instance died today and won't come > back. When I try to start it, I get: > > 2005-08-27T20:58:15 PANIC(300) z2 Startup exception > Traceback (innermost last): >File /home/lrc/z2.py, line 497, in ? >File , line 1, in ? >File /home/lrc/lib/python/Zope/__init__.py, line 37, in ? >File /home/lrc/lib/python/ZODB/FileStorage.py, line 213, in __init__ >File /home/lrc/lib/python/ZODB/lock_file.py, line 33, in lock_file > StorageSystemError: Could not lock the database file. There must be > another process that has opened the file. lock_file.py has a simple try/except block around that line. As a debugging step remove the try/except block and see if you get more verbose exception details. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope 2.8.1 on Mac OS X tiger Server
On 2005-08-18, at 06:35 GMT+12:00, Bernd Dorn wrote: On 17.08.2005, at 20:22, Garito wrote: Hi again! Sorry but Zope is executable, perhaps another thing? Any idea? hm, have you set the effective user in zope.conf? you need this if you run zopectl as root as root try to execute /System/Library/StartupItems/Zope/Zope from the commandline and have a look what if something is looged into your event.log without any messages i have no clue Also worth having a look through the system log using say Console.app. Cheers, Michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope 2.8 and conflict errors
On 2005-08-04, at 02:32 GMT+12:00, Maciej Wisniowski wrote: All errors are with OOBTree and as I can see with tempstorage.TemporaryStorage. They look like: Module ZPublisher.Publish, line 187, in publish_module_standard Module ZPublisher.Publish, line 161, in publish Module ZPublisher.Publish, line 161, in publish Module ZPublisher.Publish, line 161, in publish Module ZPublisher.Publish, line 151, in publish Module Zope2.App.startup, line 158, in zpublisher_exception_hook Module ZPublisher.Publish, line 119, in publish Module Zope2.App.startup, line 215, in commit Module transaction._manager, line 84, in commit Module transaction._transaction, line 381, in commit Module transaction._transaction, line 379, in commit Module transaction._transaction, line 424, in _commitResources Module ZODB.Connection, line 462, in commit Module ZODB.Connection, line 503, in _commit Module ZODB.Connection, line 526, in _store_objects Module tempstorage.TemporaryStorage, line 200, in store ConflictError: database conflict error (oid 0x0944, class BTrees._OOBTree.OOBTree, serial this txn started with 0x035f134dc11a2977 2005-08-03 12:29:45.258327, serial currently committed 0x035f1357051c7199 2005-08-03 12:39:01.197916) On 2005-08-04, at 09:02 GMT+12:00, Maciej Wisniowski wrote: so I hope that maybe next Zope 2.8.x release will be using MVCC for OOBTree (if this is the point?). Does anybody know something about that? That looks like a write conflict. MVCC helps to lower/eliminate the rate of read conflicts. One option to reduce the rate of this write conflict is to tune the session machinery to suit. For example use session-resolution-seconds of say 300 seconds. michael ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] accessing session data error
On 22/06/2005, at 4:50 AM, Leticia Larrosa wrote:
Hi:
Thanks to Michael Dunstan. I test with the recomendation of
Michael but I still get the error. In pages where I get the error,
if I wait some seconds and refresh the pages, I obtain the correct
information (no get the error).
I thinks that the problem maybe are similar to the guess of
Michael, but I test "breaking up ``init2`` into two scripts" and
the error don't disappear. :(
Any help will be useful.
Thanks in advance,
Leticia Larrosa
Hi Leticia,
Just double check that you are redirecting from the init2-first-half
to init2-second-half. Using something like the following in init2-
first-half::
response.redirect('init2-second-half')
rather than directly calling init2-second-half.
Cheers
Michael
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] accessing session data error
Hi Leticia,
The mixture of ``response.write()`` including JavaScript to drive the
browser to a new location along with writing objects in ZODB all
within the same transaction can break some of the promises that you'd
normally expect Zope and a browser to keep.
My guess is that browser is being told to visit ``end`` before the
transaction started by ``init2`` has finished. Perhaps a conflict
error is causing that transaction to take longer than you expect.
Normally this would not trip you up as your browser does not see
anything until after the transaction has finished. (In this normal
mode you might see a conflict error rendered in the browser if the
publication machinery had to give up.)
For your example try breaking up ``init2`` into two scripts. The
first that writes the session data and then redirects the browser to
the second. The second then does not touch session data and manages
all the ``response.write()``s. And remove the unused ``sessionData =
context.REQUEST.SESSION`` from ``external``.
Cheers
Michael
On 21/06/2005, at 6:20 AM, Leticia Larrosa wrote:
Hi all:
I think that I found a ZOPE bug. Is really very important for me
know what can I do for avoid it. The first 9 steps are to recreate
the situation and the other steps are to provoke the error.
Step 1: Crate a folder called "reproducingError" in Zope Interface
Manage
Step 2: create a page template called "index_html" in the
"reproducingError" folder with the following content:
"
"
Step 3: create a python script called "init" in the
"reproducingError" folder with the following content:
"
# init script
sessionData = container.REQUEST.SESSION
sessionData.set("userId", container.REQUEST.user)
return context.init2_html()
"
Step 4: create a page template called "init2_html" in the
"reproducingError" folder with the following content:
"
passing face 2
"
Step 5: create a python script called "init2" in the
"reproducingError" folder with the following content:
"
# init2 script
request = context.REQUEST
sessionData = request.SESSION
sessionData.set('idService', 'testService')
context.first()
return context.external(context)
"
Step 6: create a page template called "first_html" in the
"reproducingError" folder with the following content:
"
First Response
"
Step 7: create a python script called "first" in the
"reproducingError" folder with the following content:
"
# first script
response = container.REQUEST.RESPONSE
str = container.first_html()
response.write(str)
response.flush()
"
Step 8: create an external method called "external" in the
"reproducingError" folder with the following content, in where the
function of the external method is "communicationResponse" and the
module is any name that you give to the file:
"
# external method
import time
def communicationResponse(context):
response = context.REQUEST.RESPONSE
sessionData = context.REQUEST.SESSION
strDetailEnd = "
