Re: [Zope-dev] Sessions and long requests = conflicts? Mcdutils?

2007-04-11 Thread Michael Dunstan

On 4/12/07, Maciej Wisniowski [EMAIL PROTECTED] wrote:

Hi

Currently I have some problems with our application (Zope2.8.4)
and with Conflict Errors in sessions.
In general if we have few concurrent requests that are running
sometimes for 3-4 minutes (and they're touching session inside)
I get a lot of conflict errors with Inceraser, OOBTree, Length2 etc.
These are errors like:

serial starded with xxx serial currently commited 

Of course I know that it is best to move such long processes to
something external with Async, lovely.Remotetask or built
in Oracle jobs, but so far I have to deal with this state of
our application (as I'm not the author ot this but rather
something like an zope admin/supporter for this app).

I think that ConflictErrors are caused because of sessions
implementation, especially: timeslices, current bucket etc.
as written in Transience/HowTransienceWorks.stx.
Changing session_resolution_seconds to big value
helps here. By default we had session_resolution_seconds
set to 300.


You could keep experimenting with values to reduce the chances of
conflicts. Perhaps sessions that last for days. With resolution of
hours. Disabling inband housekeeping.

Note that a session-timeout-minutes of 0 enables a slightly different
approach which has a little less active structure.



I wonder how this happens. If I have two requests that at the beginning
modify it's sessions like: session.set('aa', 1), and then call long
running ZSQLMethods then (if meanwhile timeslice has changed because
of too short session_resolution_seconds):

- first request that finishes finds that there is new 'current' bucket
  and moves it's session object and second request's session object to
  new 'current' bucket and commits this

- second request finishes and finds that it's session object is not
  the same as it was at the beginning (because it was moved to 'current'
  bucket)??


Anybody can say if I'm right here?


I don't think session mechanics operates like that at the end of a
transaction. More generally what is happening is that the second
transaction is trying to commit data that was changed by an earlier
transaction after second transaction read that data. In this case the
data is various bits of the internals that make up sessions and
transience storage.

Very careful use of explicit transaction commits may be all that you
need in your application. For example, make all your edits of the
session data early in the request and then commit the transaction.
Then proceed with the longer operation. Might be that destroys the
consistency of your application data though.

-- Michael
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] OT: zc.buildout / ploneout

2007-04-08 Thread Michael Dunstan

On 4/6/07, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote:

Hi,

I have inherited an existing buildout architecture developed in-house
which is becoming difficult to support and maintain.  There are about
25 setup tasks in total.  I am considering moving it to using
zc.buildout.  Two questions:

 - is there a central repository of recipes anywhere?  for example,
one of our current build tasks is a squid installation, which, i think
would be of common interest if converted to a recipe.  so far I've
only come across zope3-based recipes, the zope2 installer, and
ploneout.


Have a look in the cheeseshop under Framework :: Buildout

 http://cheeseshop.python.org/pypi?:action=browsec=512
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope] Florent Guillaume's database Conflict Error on session objects patch not working ?

2007-02-03 Thread Michael Dunstan

On 2/3/07, yacine chaouche [EMAIL PROTECTED] wrote:

I am experiencing issues regarding a database ConflictError on session objects.


Can you detail all the various getSesion and setSession methods of:

 http://mail.zope.org/pipermail/zope/2007-January/170167.html


I use zope 2.9.0 python 2.4.2.


It's worth trying the latest of each of those.

Cheers
Michael
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )


Re: Re: [Fwd: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)]

2006-11-05 Thread Michael Dunstan

On 10/29/06, Yoshinori Okuji [EMAIL PROTECTED] wrote:

IIRC, Jim noted that it would be better to
backport zope3's, while our patches are for zope2's.


See http://mail.zope.org/pipermail/zope-dev/2006-May/027503.html for a
short thread on successfully using xmlpickle. With that recipe and
patch I was able to import and export a Plone site at the time.

Michael
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] ZODB.POSException.ConflictError on attribute assignment

2006-09-01 Thread Michael Dunstan

On 9/1/06, [EMAIL PROTECTED]
[EMAIL PROTECTED] wrote:


I'm getting a ZODB.POSException.ConflictError in the following code in a
Custom Zope Product, and I'm hoping someone can enlighten me on how to
fix it.

# nd is a folder with a sub-folder Categories
# NDCategory is a custom Zope product I wrote that does not have
anything called bogus defined.
categories = nd.Categories.objectValues('NDCategory')
for cat in categories:
cat.bogus = []


Feels like you might be miss interpreting what a ConflictError is.

With some hand waving... ConflictErrors are _not_ a mechanism for
prohibiting attribute assignment because that attribute already
exists. Instead ConflictErrors are a mechanism to prohibit inadvertent
concurrent updates of the same attribute. (Or in older ZODB's, even
concurrent reading of an attribute that is being written too.)

Check the traceback for your ConflictError to get some details about
just what is in conflict. Also might be worth a look in the access
logs to see what else might be happening on the server at about the
same time.

Cheers
Michael
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Hotfix-20060821 does not apply to zope 2.7.0 to 2.7.2?

2006-08-23 Thread Michael Dunstan

From what I can tell the feature that is Hotfixed does not appear

until Zope 2.7.3. If the Hotfix is installed in Zope 2.7.0 through to
2.7.2 then it shows up as a broken product. No real harm done.

The product import breaks with:

Traceback (most recent call last):
 File /usr/local/www/Zope/lib/python/OFS/Application.py, line 660,
in import_product
   product=__import__(pname, global_dict, global_dict, silly)
 File /usr/local/www/Zope/Atrac/Products/Hotfix_20060821/__init__.py,
line 8, in ?
   from docutils.parsers.rst.directives import tables
ImportError: cannot import name tables

And we see that ``tables`` not show up till Zope 2.7.3 via:

http://cvs.zope.org/Zope/lib/python/docutils/parsers/rst/directives/Attic/tables.py?hideattic=0only_with_tag=Zope-2_7-branch

And a casual glance around docutils before ``tables`` was introducded
suggests that feature was not part of another module.

Cheers
Michael
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope] Inavlid Refernce Counts in Temporary Storage

2006-07-16 Thread Michael Dunstan

On 7/14/06, Philip Kilner [EMAIL PROTECTED] wrote:

For the last three days, we have been plagued by slowdowns and temporary
lock-ups, which have on a couple of score occasions culminated in a
Conflict Error, indicting transactional problems in the temporary
storage. On four or five occasions, Zope has apparently locked up, and
a restart has been required.


The conflict error's may well just be a by-product of the slow down.
Roughly speaking, the longer a request takes the higher the chances
are of conflict error.

There are various tools for trying to see what zope is doing. Google
for: debug spinning zope.

cheers
michael
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] Trying to trap ConflictError

2006-07-06 Thread Michael Dunstan

On 7/7/06, Jonathan [EMAIL PROTECTED] wrote:

I haven't been able to determine how the oid's are generated, other than
little-endian 64-bit unsigned integers that will be assigned more or less
sequentially, but I can't figure out how the same oid is being used twice
(on the other hand I could be totally off-base as I am really way over my
head here!).


An oid is an identifier for a persistent state of an object. As the
state of an object evolves from transaction to transaction it gets a
new oid for each state. Two concurrent transactions that involve the
same object will start with the same state, i.e. the same oid, for
that object.

Some detail about how BTrees behave can be found in
http://www.zope.org/Wikis/ZODB/BTreeConflictResolution

Also might be illustrative to run fsdump to see what is happening in
your transactions. See
http://www.zope.org/Wikis/ZODB/FileStorageBackup. You'll need
FileStorage for that. And that only shows you details of successful
transactions.

cheers
michael
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] Trying to trap ConflictError

2006-07-06 Thread Michael Dunstan

On 7/7/06, Michael Dunstan [EMAIL PROTECTED] wrote:

An oid is an identifier for a persistent state of an object. As the
state of an object evolves from transaction to transaction it gets a
new oid for each state. Two concurrent transactions that involve the
same object will start with the same state, i.e. the same oid, for
that object.


Sorry. Ignore that. :-(

Note to self, don't even read email before morning caffeine intake.
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] Trying to trap ConflictError

2006-07-05 Thread Michael Dunstan

On 7/6/06, Jonathan [EMAIL PROTECTED] wrote:


I am still investing, and have found errors like:

2006-07-04T14:48:12 ERROR ZODB.Connection Couldn't load state for 0x1c
Traceback (most recent call last):
  File /usr/local/Zope-2.9.2/lib/python/ZODB/Connection.py, line 732, in
setstate
self._setstate(obj)
  File /usr/local/Zope-2.9.2/lib/python/ZODB/Connection.py, line 768, in
_setstate
p, serial = self._storage.load(obj._p_oid, self._version)
  File /usr/local/Zope-2.9.2/lib/python/tempstorage/TemporaryStorage.py,
line 104, in load
s=self._index[oid]
KeyError: '\x00\x00\x00\x00\x00\x00\x00\x1c'

in the event.log.   From my googling research, it appears that these type of
errors may be due to insufficient memory (but I am not sure and am still
investigating to see if this is the cause of the conflict errors).


TempoaryStorage.py has a few tuning constants. The above error reads
as though RECENTLY_GC_OIDS_LEN is too small for your application. It
may well be that the default tuning of TempoaryStorage does not match
your application very well.

It would be useful to try FileStorage to help decide on where to focus
attention. If FileStorage behaves a lot better then that also suggests
that TempoaryStorage is miss-tuned for your application.

Cheers
Michael
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope-dev] zope.xmlpickle

2006-05-26 Thread Michael Dunstan

On 5/26/06, Jim Fulton [EMAIL PROTECTED] wrote:

It would be nice to integrate this back into the export/import
machinery.


Yeah, I'd like to think I'll be able to find time to put this into Zope 2.11.


 Any suggestions on getting a successful dumps of a Missing.Value?

You need someone with pickle zen to look at this.  I don't have time
right now myself.  If you don't find any other takers, then put an
issue in the Zope 3 collector.


How about review of a trivial patch that is attached?

Michael


missing-hash.patch
Description: Binary data
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: [Zope3-dev] December release post-mortem

2006-01-19 Thread Michael Dunstan
On 1/20/06, Tim Peters [EMAIL PROTECTED] wrote:
 In all, zope.testing is a poster child for the value of package
 development outside of a Zope tree.

I've been very happy using zope.testing with several non zope
projects. Including how easy it is to follow and distribute that
package as needed for those projects.

Same goes for zdaemon. And of course ZConfig.

Michael
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope] More on understanding conflicts

2005-12-21 Thread Michael Dunstan
On 12/22/05, Dennis Allison [EMAIL PROTECTED] wrote:
 The issue here has had to do with session variables and their iteraction
 with the persistence mechanism and conflicts and multiple threads for the
 same session.  Chris McDonough has pointed out that session variables can
 cause writes and write conflicts even if the only Zope level access is a
 read.  I thought MVCC works out of the box for Zope 2.8.4 which uses ZODB
 2.3.4. Am I wrong?

 The real surprise was that you can get a write conflict from a pair of
 session variable reads!

The missing detail here is that reading a session object causes a
write to the database to update the last access time for that session
object. So what looks like a plain old read to the application code
above, in-fact includes a write just below the surface. (This is a
design choice so that stale session objects can be removed.)

There are implementation details of TransientObject that attempt to
mitigate some of the pain:

- The access time has a WRITEGRANULARITY. That can help a bit by
avoiding writing altogether some of the time. But still there is need
to write (and so potential contention) each time a boundary of that
granularity is crossed.

- And _p_resolveConflict of TransientObject will most often pick a
state to avoid a conflict error. When it does there is the performance
cost of writing to database but there will be no ConflictError.
However, I've suggested in the past that you should comment out
_p_resolveConflict while debugging consistency problems. What is the
current state of your _p_resolveConflict now? _p_resolveConflict
includes a DIWM'ly component. Perhaps you still need to avoid that
component of _p_resolveConflict? In that case a _p_resolveConflict
could be engineered that can resolve the case of just updates to the
access time? But still bails out when presented with two states that
have explicit updates to maintain consistency.

Though that's all just conjecture till we see your tracebacks for your
conflicts.

michael
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] Re: Zope Persistence (was: XML-RPC within ZOPE)

2005-12-18 Thread Michael Dunstan
On 12/19/05, Jan-Ole Esleben [EMAIL PROTECTED] wrote:
   1. In the example, just setting _p_changed=1 does _not_ lead to a
   conflict error. With the ineffectual code above it (that never gets
   executed) it _does_. So there _is_ some implicit magical stuff going
   on and ZOPE tries to take care that only subobjects change (but
   incompletely)!
  I strongly doubt it. Zope does not inspect code. There must be a
  problem in your testing. Note that if self.a is a standard list, the
  self.a.append(1) doesn't have any impact on the persistence mechanism or
  transactions either.

 Please, try it out. Delete the if clause and the append in it.

Little bit tricky to try out as testers need to guess what all the
missing code is. Also, for this kind of code demonstration, rather
than directions for commenting/uncommenting and relying on the tester
to restart the server between trails, provide different methods or
even different classes.

Meanwhile I'm camping over at I-strongly-doubt-it.

   3. It is especially confusing that ZOPE behaves differently when using
   XML-RPC calls. From what you say, it should be the same within the
   ZOPE system as when using XML-RPC. It gets more complicated with
   XML-RPC though!
  The successive XML-RPC call you describe provoke new transactions,
  surely you're aware of that? Whereas just calling a function of course
  doesn't.

 I'm aware of that. But ZOPE offers XML-RPC and as there is nothing in
 the documentation about such complex interactions. Also, I was
 directly responding to what the previous poster had written.

Zope provides for XML-RPC as a server. And it does so with internal
transaction support.

But there is no extra transactional framework for XML-RPC clients
(which your example has). Or for that matter acting as an XML-RPC
server nested within some external transaction (which your example
has).

Either design to avoid this or get your hands dirty with transactions.
Some starting points for examples would include any of the relational
database adapters. Also in the same space is MaildropHost.


michael
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] Re: Zope Persistence (was: XML-RPC within ZOPE)

2005-12-18 Thread Michael Dunstan
On 12/19/05, Jan-Ole Esleben [EMAIL PROTECTED] wrote:
  Little bit tricky to try out as testers need to guess what all the
  missing code is.

 Any standard persistent ZOPE product wrapped around this will do.
 These are the only methods in a ZOPE product that inherits from Item,
 Persistent, RoleManager and Implicit.

  Also, for this kind of code demonstration, rather
  than directions for commenting/uncommenting and relying on the tester
  to restart the server between trails, provide different methods or
  even different classes.

 If I did that, people would say somethings wrong with your testing
 and they would have to do a line-by-line comparison and still have
 room to doubt my statement.

/me shurgs. I prefer code that runs. :-)

  But there is no extra transactional framework for XML-RPC clients
  (which your example has). Or for that matter acting as an XML-RPC
  server nested within some external transaction (which your example
  has).

 I was initially talking about a more complex situation that has been
 gradually simplified. ZOPE should be able to act as an XML-RPC client
 to the outside world.

 As to me not understanding what you all are saying: I believe I
 understand, I know that theoretically - from a merely technical
 viewpoint, it works, I just look at it from a different angle. Of
 course you have to know a framework and its limitations, but this
 error I stumbled upon strikes me as really complex (even though you
 keep repeating it's very simple if you just acknowledge xyz), for
 instance because the docs aren't perfect and say that
 You must explicitly signal any changes made to mutable attributes
 (such as instances, lists, and dictionaries) or use persistent
 versions of mutable objects, like ZODB.PersistentMapping (see below
 for more information on PersistentMapping.) - the _first_ option
 being mutable attributes! Also, there is no below.
 I believe you when you say that it is really very simple and I
 shouldn't have to worry if I follow the right principles, that you
 then go on to state (principles that aren't really that explicit
 anywhere - such as don't use XML-RPC to talk to ZOPE from ZOPE and
 don't use mutable subobjects at all). However, this is a further bit
 of distance between a ZOPE user and the Python he knows, and it also
 isn't as true for complex situations with complex requirements that I
 could probably just jot down in dumber frameworks such as RoR or
 Django or TurboGears. Because I know without having to check myself
 where I am designing Data, I explicitly _make_ changes (to the DB) and
 am thus very aware of where complex interactions might occur etc. This
 is probably one of those cases where no one is really right because
 a mixture of viewpoints is involved, I feel a bit as if we were
 talking about wether Lisp is better than Smalltalk. However, even if
 it is only the docs that are lacking I think it would be sensible to
 acknowledge that as a problem. It obviously was a problem for me, as
 of course was the complexity of my setup and several other factors. I
 am not an inexperienced programmer, so I would think that just because
 other people rarely stumble upon it it is not an insignificant
 problem. Imagine designing a large system based on some only
 marginally false assumptions about ZOPE transactions, persistence and
 XML-RPC interaction, and then running into this in a key component.

Joining many transactional systems into one mega transactional beast
is requires attention to details.

How about getting away from this notion a mega transaction with nested
transactions? For example, make use of transaction.commit() before
making XML-RPC calls. Though you well need to decide what happens in
the event of a ConflictError being thrown from somewhere else and
causing the initial request to be retried.


michael
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope-dev] Re: sessions in the presence of conflicts

2005-12-15 Thread Michael Dunstan
On 12/16/05, Dennis Allison [EMAIL PROTECTED] wrote:
 MAYBE CONFLICTS AND THEIR RESOLUTION ARE NOT THE ROOT CAUSE OF THE SESSION
 VARIABLE PROBLEM.  The observed problem is that session variables suddenly
 disappear.

Perhaps your app is tripping over some bug in conflict handling. But
I'd say it is worth entertain other ideas too. For now, just comment
out all of TransientObject._p_resolveConflict and if you still get
errors then you know you have to look elsewhere.

(Sure, some of that elsewhere may well include the conflict resolution
code above _p_resolveConflict.)

Your application and sessions should cope just fine in the face of any
ConflictError. ConflictError's are an essential part of the machinery
that keeps data state consistent.

As Chris mentions, look at how your using sessions and some of the
assumptions you might be making. Might be useful to try with sessions
that don't timeout, set session-timeout-minutes to 0. And try
maximum-number-of-session-objects of 0. Also trying the turning those
knobs the other way, session-timeout-minutes of 1 and
maximum-number-of-session-objects of 2.

For now, stay focused on making sure you maintain a consistent state.
Only once you have a consistent state then is it worth trying to
improve the rate of ConflictErrors. (Which in your case of sessions
lasting for many hours I think the numbers you quote elsewhere are too
small. And, yeah, an alternative implementation for _p_resolveConflict
might help there. Personally I prefer the simple approach of just
commenting that out completely and living with a slightly higher
conflict count.)

Might be worth trying without ZEO in the mix.

Definitely worth the effort, if you have not already, to recreate the
whole system on a separate host that you can feel comfortable making
changes to. Then you can happily tune the various knobs downwards
which may help with trying to observe the problem. For example
session-timeout-minutes of 1.

cheers
michael
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope] resolving conflict errors

2005-12-10 Thread Michael Dunstan
On 12/9/05, Dennis Allison [EMAIL PROTECTED] wrote:
 The problem I am trying to resolve appears to be load related.  The
 observed symptom is that (some) session variables spontaneously disappear.
 There appears to be some connection to conflicts, but the exact mechanism
 and the relationship is not yet clear.

A small possibility is that you are being bitten by the DWIM'ly nature
of TransientObjects conflict resolution where the last modified state
is chosen over the others. If you think this is biting you then try
commenting out _p_resolveConflict of TransientObject. That's bound to
increase the rate of conflict errors but should provide you with a
consistent session state. Perhaps useful as a debugging step.

michael
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] resolving conflict errors

2005-12-10 Thread Michael Dunstan
On 12/11/05, Dennis Allison [EMAIL PROTECTED] wrote:

 Good idea, but it is hard to do in a production environment with a never
 lose data model.

Have a go at recreating the problems you are seeing on a development
host. SessionRig can be used to mount a brute force attack of the
session machinery. You'll need to tune that somewhat to your
particular application.

  http://cvs.zope.org/Packages/SessionRig

michael
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope-dev] Please vote about conflict errors logging

2005-12-06 Thread Michael Dunstan
How about adding a URL to the logged message? Something like

 See http://zope.org/ConflictError;

detailing the difference between a retried and unresovled
ConflictErrors and how to recognise them in the event.log.
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope] conflicts, sessions, and programming best practices guidelines

2005-11-18 Thread Michael Dunstan
On 11/18/05, Chris McDonough [EMAIL PROTECTED] wrote:

 On Nov 17, 2005, at 4:46 AM, Dennis Allison wrote:

 
  Thank Chris, good pointsB.
 
  The session variable timeout is currently several hours as requird
  by our
  application. The timeout resolution remains at its default value.

 Making the resolution higher has been observed to reduce conflicts
 (see dunny's chart).

If you sessions live for several hours then perhaps you want a
increase session-resolution-seconds to something like 1800 (30
minutes) or more.

Also worth seeing if increasing WRITEGRANULARITY of
Products.Transience.TransientObject helps you.

  I have
  not dorked with disabling the inband housekeeping.  I don't see how it
  would impact on our problem.  Am I missing something?

 Turning off inband housekeeping has the potential to reduce the
 numnber of conflicts because less work is done during normal
 sessioning operations.  Instead of doing the work to figure out
 whether it needs to clean up after itself during normal operations
 it relies on an external process.

And it is not that difficult to do.

Michael.
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] Help!!! Zope won't start -- StorageSystemError

2005-08-27 Thread Michael Dunstan
On 8/28/05, Dan Gaibel [EMAIL PROTECTED] wrote:
 Hi folks,
 
 This is my first post to this list and I'm desperate for help. I'm running
 Zope 2.6.1 with Python 2.1.3. My Zope instance died today and won't come
 back. When I try to start it, I get:
 
 2005-08-27T20:58:15 PANIC(300) z2 Startup exception
 Traceback (innermost last):
File /home/lrc/z2.py, line 497, in ?
File string, line 1, in ?
File /home/lrc/lib/python/Zope/__init__.py, line 37, in ?
File /home/lrc/lib/python/ZODB/FileStorage.py, line 213, in __init__
File /home/lrc/lib/python/ZODB/lock_file.py, line 33, in lock_file
 StorageSystemError: Could not lock the database file.  There must be
 another process that has opened the file.

lock_file.py has a simple try/except block around that line. As a
debugging step remove the try/except block and see if you get more
verbose exception details.
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope-dev] Re: ZServer response RFC compliance improvement

2005-08-22 Thread Michael Dunstan
On 8/22/05, Tres Seaver [EMAIL PROTECTED] wrote:
 Are we sure that we won't be breaking the rather large possible set of
 installed servers running behind Apache 1.3.x with the bug for which
 adding the content length was a workaround?

I understand that this bug was resolved in Apache 1.3.27 [1]. Which is
a few years old now. And outdated by several security releases since
then.

Also OFS.Image been patched as of Zope 2.7.1 [2] in such a way that it
would have already tripped a combination of old-ish Apache and new-ish
Zope. Though ZServer was still throwing in a Content-Length: 0.
(Which I read as sufficient to provoke the bug in Apache  1.3.27.)

[1] http://www.apacheweek.com/issues/02-10-04

[2] 
http://cvs.zope.org/Zope/lib/python/OFS/Attic/Image.py?hideattic=0only_with_tag=Zope-2_7-branch#rev1.145.2.7
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope] Zope 2.8.1 on Mac OS X tiger Server

2005-08-17 Thread Michael Dunstan

On 2005-08-18, at 06:35 GMT+12:00, Bernd Dorn wrote:

On 17.08.2005, at 20:22, Garito wrote:

Hi again!
Sorry but Zope is executable, perhaps another thing?
Any idea?


hm, have you set the effective user in zope.conf? you need this if  
you run zopectl as root


as root try to execute

/System/Library/StartupItems/Zope/Zope

from the commandline

and have a look what if something is looged into your event.log

without any messages i have no clue



Also worth having a look through the system log using say Console.app.

Cheers,
Michael

___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] Zope 2.8 and conflict errors

2005-08-03 Thread Michael Dunstan

On 2005-08-04, at 02:32 GMT+12:00, Maciej Wisniowski wrote:
All errors are with OOBTree and as I can see with  
tempstorage.TemporaryStorage. They look like:


li  Module ZPublisher.Publish, line 187, in  
publish_module_standard/li

li  Module ZPublisher.Publish, line 161, in publish/li
li  Module ZPublisher.Publish, line 161, in publish/li
li  Module ZPublisher.Publish, line 161, in publish/li
li  Module ZPublisher.Publish, line 151, in publish/li
li  Module Zope2.App.startup, line 158, in  
zpublisher_exception_hook/li

li  Module ZPublisher.Publish, line 119, in publish/li
li  Module Zope2.App.startup, line 215, in commit/li
li  Module transaction._manager, line 84, in commit/li
li  Module transaction._transaction, line 381, in commit/li
li  Module transaction._transaction, line 379, in commit/li
li  Module transaction._transaction, line 424, in  
_commitResources/li

li  Module ZODB.Connection, line 462, in commit/li
li  Module ZODB.Connection, line 503, in _commit/li
li  Module ZODB.Connection, line 526, in _store_objects/li
li  Module tempstorage.TemporaryStorage, line 200, in store/li
/ulConflictError: database conflict error (oid 0x0944, class  
BTrees._OOBTree.OOBTree, serial this txn started with  
0x035f134dc11a2977 2005-08-03 12:29:45.258327, serial currently  
committed 0x035f1357051c7199 2005-08-03 12:39:01.197916)


On 2005-08-04, at 09:02 GMT+12:00, Maciej Wisniowski wrote:


so I hope that maybe next Zope 2.8.x release will be using MVCC for  
OOBTree (if this is the point?). Does anybody know something about  
that?


That looks like a write conflict. MVCC helps to lower/eliminate the  
rate of read conflicts.


One option to reduce the rate of this write conflict is to tune the  
session machinery to suit. For example use session-resolution-seconds  
of say 300 seconds.


michael
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] accessing session data error

2005-06-21 Thread Michael Dunstan

On 22/06/2005, at 4:50 AM, Leticia Larrosa wrote:


Hi:

 Thanks to Michael Dunstan. I test with the recomendation of  
Michael but I still get the error. In pages where I get the error,  
if I wait some seconds and refresh the pages, I obtain the correct  
information (no get the error).


I thinks that the problem maybe are similar to the guess of  
Michael, but I test breaking up ``init2`` into two scripts and  
the error don't disappear. :(


Any help will be useful.

Thanks in advance,

Leticia Larrosa


Hi Leticia,

Just double check that you are redirecting from the init2-first-half  
to init2-second-half. Using something like the following in init2- 
first-half::


  response.redirect('init2-second-half')

rather than directly calling init2-second-half.

Cheers
Michael
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] accessing session data error

2005-06-20 Thread Michael Dunstan

Hi Leticia,

The mixture of ``response.write()`` including JavaScript to drive the  
browser to a new location along with writing objects in ZODB all  
within the same transaction can break some of the promises that you'd  
normally expect Zope and a browser to keep.


My guess is that browser is being told to visit ``end`` before the  
transaction started by ``init2`` has finished. Perhaps a conflict  
error is causing that transaction to take longer than you expect.  
Normally this would not trip you up as your browser does not see  
anything until after the transaction has finished. (In this normal  
mode you might see a conflict error rendered in the browser if the  
publication machinery had to give up.)


For your example try breaking up ``init2`` into two scripts. The  
first that writes the session data and then redirects the browser to  
the second. The second then does not touch session data and manages  
all the ``response.write()``s. And remove the unused ``sessionData =  
context.REQUEST.SESSION`` from ``external``.


Cheers
Michael

On 21/06/2005, at 6:20 AM, Leticia Larrosa wrote:


Hi all:

I think that I found a ZOPE bug. Is really very important for me  
know what can I do for avoid it. The first 9 steps are to recreate  
the situation and the other steps are to provoke the error.


Step 1: Crate a folder called reproducingError in Zope Interface  
Manage


Step 2: create a page template called index_html in the  
reproducingError folder with the following content:


!-- index_html page template --
form name=form1 method=post action=reproducingError/init
  input type=text name=user
  input type=submit name=Submit value=Submit
/form


Step 3: create a python script called init in the  
reproducingError folder with the following content:


# init script
sessionData = container.REQUEST.SESSION
sessionData.set(userId, container.REQUEST.user)
return context.init2_html()


Step 4: create a page template called init2_html in the  
reproducingError folder with the following content:


!-- init2_html page template --
a href=init2passing face 2/a


Step 5: create a python script called init2 in the  
reproducingError folder with the following content:


# init2 script
request = context.REQUEST
sessionData = request.SESSION
sessionData.set('idService', 'testService')
context.first()
return context.external(context)


Step 6: create a page template called first_html in the  
reproducingError folder with the following content:


First Response


Step 7: create a python script called first in the  
reproducingError folder with the following content:


# first script
response = container.REQUEST.RESPONSE
str = container.first_html()
response.write(str)
response.flush()


Step 8: create an external method called external in the  
reproducingError folder with the following content, in where the  
function of the external method is communicationResponse and the  
module is any name that you give to the file:


# external method
import time
def communicationResponse(context):
response = context.REQUEST.RESPONSE
sessionData = context.REQUEST.SESSION
strDetailEnd = scriptwindow.location='reproducingError/end'/ 
script


for item in range(1, 40):
time.sleep(3)
response.write(Response iteration: + str(item))
response.flush()
response.write(strDetailEnd)
response.flush()


Step 9: create a python script called end in the  
reproducingError folder with the following content:


# end script
idService = context.REQUEST.SESSION['idService']
return Fin -  + idService


Step 10: open two explorer in wich call the object  
reproducingError (Example: localhost:8080/reproducingError)


Step 11: enter an example text in each one.

Step 12: clink in the link of one of them, and 5 second later clink  
on the link of the other explorer. The waiting for around 5 second  
is necesary because with other time interval not allways get the  
error.


Stpe 13: and wait the Response iteration...  Please if the error  
don't appear in any of the pages, begins at Step 9 again. and try  
to count to 5 witch other velocity :)


The correct result is:

Fin - testService


The error is:

Site Error
An error was encountered while publishing this resource.
Error Type: KeyError
Error Value: 'idService'

The traceback:

Traceback (innermost last):
  Module ZPublisher.Publish, line 101, in publish
  Module ZPublisher.mapply, line 88, in mapply
  Module ZPublisher.Publish, line 39, in call_object
  Module Shared.DC.Scripts.Bindings, line 306, in __call__
  Module Shared.DC.Scripts.Bindings, line 343, in _bindAndExec
  Module Products.PythonScripts.PythonScript, line 323, in _exec
  Module None, line 2, in end
   - PythonScript at /approach/reproducingError/end
   - Line 2
  Module AccessControl.ZopeGuards, line 67, in guarded_getitem
  Module Products.Transience.TransientObject, line 176, in __getitem__
KeyError: 'idService'


I will appreciate if you try until the error appear, because I not  

[Zope-dev] transience and gc.set_threshold(1)

2004-10-05 Thread Michael Dunstan
I'm seeing test failures in transience when running the tests for  
Zope-2_7-branch (python 2.3.3) with garbage collection threshold0 set  
to 1.

Seems to break in slightly different places from run to run. Here is  
one example:

% python test.py -vv --dir lib/python/Products/Transience/tests -g 1   
testPathological
gc threshold: (1, 10, 10)
Running unit tests at level 1
Running unit tests from /Zope/lib/python/Products/Transience/tests
testPathologicalLeftBranching  
(Products.Transience.tests.testTransientObjectContainer.TestTransientObj 
ectContainer) ... ERROR
testPathologicalRightBranching  
(Products.Transience.tests.testTransientObjectContainer.TestTransientObj 
ectContainer) ... ERROR

==
ERROR: testPathologicalLeftBranching  
(Products.Transience.tests.testTransientObjectContainer.TestTransientObj 
ectContainer)
--
Traceback (most recent call last):
  File  
/Zope/lib/python/Products/Transience/tests/ 
testTransientObjectContainer.py, line 193, in  
testPathologicalLeftBranching
map(self.t.__delitem__, revr)
  File /Zope/lib/python/Products/Transience/Transience.py, line 470,  
in __delitem__
del bucket[k]
KeyError: 946

==
ERROR: testPathologicalRightBranching  
(Products.Transience.tests.testTransientObjectContainer.TestTransientObj 
ectContainer)
--
Traceback (most recent call last):
  File  
/Zope/lib/python/Products/Transience/tests/ 
testTransientObjectContainer.py, line 183, in  
testPathologicalRightBranching
map(self.t.__delitem__, r)
  File /Zope/lib/python/Products/Transience/Transience.py, line 470,  
in __delitem__
del bucket[k]
KeyError: 69

--
Ran 2 tests in 4.659s
FAILED (errors=2)
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] transience and gc.set_threshold(1)

2004-10-05 Thread Michael Dunstan
On 6/10/2004, at 4:06 AM, Tim Peters wrote:
[Michael Dunstan]
I'm seeing test failures in transience when running the tests for
Zope-2_7-branch (python 2.3.3) with garbage collection threshold0 set
to 1.
snip
Whenever I see a test module import fauxtime, I run screaming in
terror.  This seems to be like running the multiprocess ZEO tests
(from which I run screaming daily wink):  depending on a gazillion
timing accidents you can't control, sometimes they just fail.
I ran the tests above in a loop a few hundred times on my box just
now, with no failures, Python 2.3.4 and command line:
test.py -vvLd -g1 --dir lib/python/Products/Transience/tests . 
testPathological

There's a big difference between my runs and yours, though:  running
the two tests took about 4.6 seconds on your box, but takes about 1.8
seconds on mine.  Now, for example, if I add
oldtime.sleep(3)  # approximately 4.6 - 1.8, on the high side
right before
map(self.t.__delitem__, r)
in testPathologicalRightBranching, then testPathologicalRightBranching
fails every time in a way similar to your reported failure -- and even
without -g1 on the command line.
If I change the sleep to 1.5, then (with -g1 again) sometimes it
passes, sometimes not, depending on how busy the box is doing other
stuff at the same time.
So that's A Problem.  Any test that adds a bunch of stuff to a
TransientObjectContainer and expects to *find* it all there later is
implicitly assuming that the test runs fast enough so that containees
don't vanish before their containedness is checked.  I suspect you're
seeing problems with -g1 here just because -g1 grossly slows the
tests.
Doh! For some reason I had written that explanation off before posting. 
The pattern of missing keys did not seem consistent enough at the time.

But, as you say, as soon as I comment out the use of faxtime and run 
those same two tests they both pass just fine.

thanks
michael
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] session problem again?

2004-09-28 Thread Michael Dunstan
On 29/09/2004, at 11:07 AM, Matteo Memelli wrote:
Hello I include the conversation that I had today with Chris:
I'm using Zope 2.6b1+ with ZWarehouse e-commerce product
(ZWarehouse-20040522 version 0.8)
I tried upgrading to Zope 2.7 but ZWarehouse gave me many problems so
I decided to try applying your patches for Sessioning problems that I
have.
Hmm..
I think that the error that zope gives to me is very known to you... I
tried to copy the whole directories Transience, Session and ZPublisher
from both
Zope-2.7.3b1-0 and Zope 2.7.2 but without success I also copied the
file startup.py from lib/python/Zope/App/ ...
Am I missing anything?
Yes, the ZODB directory also... although I reallly have no idea whether
this will work under 2.6.
Really, to be sure you've got all the right files, it'd be best if you
could upgrade to 2.7.3... maybe we could help fix whatever in  
ZWarehouse
doesn't work under there (move this conversation to the Zope list
[EMAIL PROTECTED])

To be thorough, you will also need to delete and recreate any  
persistent
session data container you've got laying around (unless it's in
temp_folder, that's in RAM anyway and gets recreated on reboot).

My customer is complaining so much and I really don't know what to do!
Hoping in your answer...thank you in advance
Matteo Memelli

Site Error
An error was encountered while
publishing this resource.
KeyError
Sorry, a site error occurred.
Traceback (innermost last):
  * Module ZPublisher.Publish,
line 175, in
publish_module_standard
  * Module Products.iHotfix,
line 83, in new_publish
  * Module ZPublisher.Publish,
line 132, in publish
  * Module Zope.App.startup,
line 203, in
zpublisher_exception_hook
  * Module ZPublisher.Publish,
line 101, in publish
  * Module ZPublisher.mapply,
line 88, in mapply
  * Module ZPublisher.Publish,
line 39, in call_object
  * Module OFS.DTMLDocument,
line 128, in __call__
DTMLDocument instance at
41725260
URL:
http://www.arte-luce.com/catalog/html/catalog.html/manage_main
Physical
Path:/arteluce/catalog/html/catalog.html
  * Module
DocumentTemplate.DT_String,
line 474, in __call__
  * Module
DocumentTemplate.DT_Let,
line 76, in render
  * Module OFS.DTMLDocument,
line 121, in __call__
DTMLDocument instance at
4172b6b0
URL:
http://www.arte-luce.com/catalog/html/zwarehouse_html_header/ 
manage_main
Physical
Path:/arteluce/catalog/html/zwarehouse_html_header
  * Module
DocumentTemplate.DT_String,
line 474, in __call__
  * Module
DocumentTemplate.DT_Util,
line 201, in eval
__traceback_info__:
cart_functions
  * Module string, line 1, in
expression
  * Module
Shared.DC.Scripts.Bindings,
line 306, in __call__
  * Module
Shared.DC.Scripts.Bindings,
line 343, in _bindAndExec
  * Module
Products.PythonScripts.PythonScript, line 318, in _exec
  * Module None, line 16, in
setSessionByRequest.py
PythonScript at
/arteluce/catalog/cart_functions/setSessionByRequest.py
Line 16
  * Module
ZPublisher.HTTPRequest,
line 1218, in __getattr__
  * Module
ZPublisher.HTTPRequest,
line 1178, in get
  * Module
Products.Sessions.SessionDataManager, line 94, in
getSessionData
  * Module
Products.Sessions.SessionDataManager, line 181, in
_getSessionDataObject
  * Module
Products.Transience.Transience, line 847, in new_or_existing
  * Module
Products.Transience.Transience, line 450, in __setitem__
KeyError: 1096365080 (Also, an
error occurred while attempting to
render the standard error message.)
___
Troubleshooting Suggestions
  * The URL may be incorrect.
  * The parameters passed to
this resource may be
incorrect.
  * A resource that this
resource relies on may be
encountering an error.
For more detailed information about
the error, please refer to error
log.
If the error persists please
contact the site maintainer. Thank
you for your patience.

Hmmm.. that's an odd place to get that error.  I'm surprised.  Does it
only happen after some period of inactivity on the site?
- C
FOLLOWING Chris's advice first I tried to copy also ZODB directory from
Zope 2.7.3 but still the same error...
so I Installed Zope-2.7.3b1-0 (IS THIS THE RIGHT VERSION THAT SHOULD
SOLVE SESSION PROBLEMS?) and reinstalled with success the following
packages:
ZWarehouse 0.8
iHotfix-0.4.0
itools-0.4.6
Localizer-1.1.0a4
ZMySQLDA-2.0.8
MySQL-python-1.0.0
TranslationService-0.4
THEN imported my customer's web site but I still have the same
problem... whats going on... what am i missing?
For every change I made I 

Re: [Zope-dev] Transaction problems and publisher

2004-08-24 Thread Michael Dunstan
On 25/08/2004, at 12:55 AM, Florent Guillaume wrote:
I've reopened bug 789 (http://collector.zope.org/Zope/789) because we 
observed problems in Zope 2.7.2 with transaction abort by the 
publisher.

The problems can make objects disappear or provoke POSKeyErrors.
Here is a patch for publisher that may help.
I'm currently running some tests for sessions and transience. And this 
patch seems to iron out a lot problems that have been reported with 
transience of recent.

(I failed to dance in just the right way to attach this patch to the 
collector.)

michael


publisher-abort.patch
Description: Binary data
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: hasattr geddon

2004-08-18 Thread Michael Dunstan
On 12/07/2004, at 12:29 PM, Chris McDonough wrote:
FWIW, as far as I understand the prevent commit on conflict behavior
*is* currently the behavior for caught ReadConflictErrors.  The only
time ZODB doesn't exhibit this behavior is *during the commit of a
transaction*.  If a commit is attempted at this point and a conflict
error is caught unintentionally, the following happens (as explained by
Tim at http://mail.zope.org/pipermail/zodb-dev/2004-May/007404.html :
- the attempt to commit fails, raising a write conflict error
- the write conflict error is unintentionally caught (whether by 
hasattr
or a bare except or by whatever)

- a new transaction begins implicitly (as if 
get_transaction().abort()
or get_transaction().begin() were called)

- ... whatever happens after this happens ..
Since it's extremely unlkely for a write conflict error to be caught
unintentionally in Zope (it would require an application to perform a
get_transaction().commit() going outside the boundaries of the 
default
transaction manager), it's an obscure failure mode.

If the transaction was not put into a committable state after a caught
exception error, there would be no problem.  But to be fair, I don't
think there is much of a problem right now, as very few Zope apps 
manage
their own transactions (except for the Catalog... gulp ;-).  But it
would be nice if ZODB didn't do the implicit 
get_transaction().abort()
after a write conflict error and left the transaction in an
uncommittable state instead... or at least had an option to do this.
So... it sounds like ZODB will take care of itself in the face of 
ReadConflictErrors swallowed by bare excepts (and hasattr's). (Unless 
your managing your own transactions.)

One penalty of bare excepts swallowing a ConflictError is running more 
code than is needed. The transaction is going to be thrown away 
eventually.

Is there problem with seeing bogus exceptions in code after a swallowed 
ConflictError?

I'm thinking that when zope catches any exception in a transaction it 
should check the state of the transaction to decide if should treat 
this as a ConflictError or report the exception it caught.

Consider a contrived example::
def setFoo(self, value):
try:
self.foo = value
except:
pass
self.fooUpdated()
Assume that the except swallows some ConflictError then an exception is 
raised from fooUpdated()
but not a ConflictError, lets say a KeyError. Sure, ZODB is tough 
enough to look after itself in this case. What about the developer 
trying to interpret the KeyError that is written to event.log?

(Hmm... I should really check that is what you could potentially see 
with a current Zope :-)

Michael
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Conflict errors on BDBMinimal storage

2004-05-18 Thread Michael Dunstan
On 18/05/2004, at 6:57 AM, Gerhard Schmidt wrote:
On Mon, May 17, 2004 at 01:38:15PM -0400, Chris McDonough wrote:
AFAIK Zope is the only appserver that even makes an attempt at 
allowing
session data to be shared between servers like this.  Most other
appservers require you to set up session affinity in your load
balancer and have a session database on each appserver.
and nealy every system failed to do so. How do you implement this
session affinity. Based on source IP. At the moment every large ISP
uses multiple Proxyserver so every request comes from a different
IP so the affinity doesn´t work for them. The Proxyserver provided by
our Computing Center is also split on at least 8 Hosts, so everyone
using this proxy would lose session infomation, getting very strange
effekts.
Perhaps you can get some relief by using pound as your load balancer. 
It is quite capable at sniffing out user sessions including watching 
the Cookie header. See http://www.apsis.ch/pound/

Michael
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)

2004-05-18 Thread Michael Dunstan
On 18/05/2004, at 10:03 PM, Andrea Patuzzo wrote:
Hi, dear developers:
We are now randomly getting this kind of error:
[...]
  File 
/usr/local/Zope270CVS/lib/python/Products/Transience/Transience.py,
line 341, in __setitem__
current_bucket = self._data[current_ts]
KeyError: 1084872640


This one was with the latest publish.py / startup.py patch and the new
Transience.py implementation, on a yesterday's Zope-2_7-branch 
checkout.
New Zope instance with FileStorage does not solve the problem.
Same happens with the patches applied to the official Zope-2.7.0 
release.

Note: once the error happens, zope has to be restarted to function 
again.
Before applying patches we were getting get errors (instead of
__setitem__) and we just had to close browser to get back to work.
Here we have to restart Zope (with TemporaryStorage) or delete 
Sessions.fs
and restart (with FileStorage).
My guess at the moment is that somewhere a ConflictError is being 
swallowed. There are a few things that can be responsible for this:

- Use of hasattr on Persistent objects.
- try/except blocks that don't raise ConflictErrors.
From your trace that looks like a Plone site. A very quick glance 
through Plone and its friends indicates that there may be some chance 
of either of the above causing some problems. Although Plone 2.0.1 
includes the following fix:

- Fixed bare 'except:' statements in Plone so we don't swallow
  ZODB ConflictErrors. Mostly by adding 'except ConflictError: 
raise'.
  [stefan]

Someone please shoot me down and claim that Plone and friends all do 
the right thing. ;-)

Michael
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: Conflict errors on BDBMinimal storage

2004-05-18 Thread Michael Dunstan
On 19/05/2004, at 8:36 AM, Chris McDonough wrote:
As a result, I am coming to believe that along with the errors as part
of main transaction patch for 2.7.1, I should also make the
retry-on-conflict-error policy pluggable for those who really
desperately need to slow their Zope systems to a conflict-induced
crawl.  They can bump it up to 100 as far as I care and if it works for
them, great.
Or this could be used to avoid retries altogether. And at the very 
least it would advertise the fact that zope does retry requests.

Michael.
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)

2004-05-17 Thread Michael Dunstan
On 18/05/2004, at 5:42 AM, Chris McDonough wrote:
On Mon, 2004-05-17 at 13:06, Steve Jibson wrote:
Also of note:
1 - After having the system run for 45 minutes, I had 8 similar 
errors.
  Some were on different web pages and some had different keys that
were causing the error.
2 - I have also traced through Z2.log and followed the same path 
through
the web site that produced one of the errors and I did not get an 
error.
3 - Since changing this server to use FileStorage (1hr 39min ago), 
there
has not been a single error.

Is there anything I can do to help here?
A small reproducible test case would help if you still believe this
error is not in your own application.
Looks like session data can expire prematurely. See attached files for 
small changes to the test rig that reports cases where 
context.session_data_manager.hasSessionData() is False.

Michael



SessionRigExtensions.py
Description: Binary data


TestSessionRig.py
Description: Binary data
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)

2004-05-14 Thread Michael Dunstan
On 14/05/2004, at 10:06 AM, Chris McDonough wrote:
I am tempted to check the following into the 2.7 branch and HEAD:
- error occurs in same transaction as main request patch
  to Publish.py.  See
http://www.plope.com/Members/chrism/ 
publishpy_errorinmaintrainsaction.patch/file_view
 for the patch.
hmm... I suspect there are few holes in this...
- Need to make sure there is no get_transaction().begin() in  
zpublisher_exception_hook() of lib/python/Zope/App/startup.py

  That begin() would mean that the error handling is not in fact part  
of the original transaction?

- An exception within err_hook() (aside from a Retry) will mean that  
abort() is never called.

Here is another attempt at publishpy_errorinmaintrainsaction.patch:
Index: lib/python/ZPublisher/Publish.py
===
RCS file: /cvs-repository/Zope/lib/python/ZPublisher/Publish.py,v
retrieving revision 1.164.2.2
diff -u -r1.164.2.2 Publish.py
--- lib/python/ZPublisher/Publish.py17 Nov 2003 22:34:19 -   
1.164.2.2
+++ lib/python/ZPublisher/Publish.py15 May 2004 02:34:12 -
@@ -105,7 +105,6 @@

 return response
 except:
-if transactions_manager: transactions_manager.abort()
 # DM: provide nicer error message for FTP
 sm = None
@@ -119,29 +118,36 @@
 if err_hook is not None:
-if parents: parents=parents[0]
 try:
-return err_hook(parents, request,
-sys.exc_info()[0],
-sys.exc_info()[1],
-sys.exc_info()[2],
-)
-except Retry:
-# We need to try again
-if not request.supports_retry():
+if parents: parents=parents[0]
+try:
 return err_hook(parents, request,
 sys.exc_info()[0],
 sys.exc_info()[1],
 sys.exc_info()[2],
 )
-newrequest=request.retry()
-request.close()  # Free resources held by the request.
-try:
-return publish(newrequest, module_name,  
after_list, debug)
-finally:
-newrequest.close()
+except Retry:
+if not request.supports_retry():
+return err_hook(parents, request,
+sys.exc_info()[0],
+sys.exc_info()[1],
+sys.exc_info()[2],
+)
+finally:
+if transactions_manager: transactions_manager.abort()
+
+
+newrequest=request.retry()
+request.close()  # Free resources held by the request.
+try:
+return publish(newrequest, module_name, after_list,  
debug)
+finally:
+newrequest.close()
+
+else:
+if transactions_manager: transactions_manager.abort()
+raise

-else: raise
 def publish_module_standard(module_name,
Index: lib/python/Zope/App/startup.py
===
RCS file: /cvs-repository/Zope/lib/python/Zope/App/startup.py,v
retrieving revision 1.8.2.3
diff -u -r1.8.2.3 startup.py
--- lib/python/Zope/App/startup.py  20 Dec 2003 04:48:36 -   
1.8.2.3
+++ lib/python/Zope/App/startup.py  15 May 2004 02:34:13 -
@@ -176,8 +176,6 @@
 published=app.__bobo_traverse__(REQUEST).__of__(
 RequestContainer(REQUEST))

-get_transaction().begin() # Just to be sure.
-
 published=getattr(published, 'im_self', published)
 while 1:
 f=getattr(published, 'raise_standardErrorMessage', None)
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)

2004-05-12 Thread Michael Dunstan
On 11/05/2004, at 4:54 PM, Chris McDonough wrote:

On Mon, 2004-04-26 at 18:34, Michael Dunstan wrote:
I removed all the above hacks leaving just the
one that comments out the construction of the to_notify list. Reran 
the
test rig. And have yet to see any KeyErrors.
Have you seen any since the time you reported this?
(I chatted about this with Chris on #zope.)

Since the last report I have not run been running the test rig. I 
Needed my laptop back so I could get on with a few other things. :-(

So I'm going to see if I can show others how to recreate the test rig 
that I'm using here. A trivial site and the script for testing that 
site are attached.

(Reminder - I'm using Zope 2.7.0 here - HEAD just is not ready for such 
games. :-)

SessionRigExtensions.py and sessionrig.zexp make up the site. 
sessionrig.zexp is simply a bunch of external methods (with a folder to 
provide some separation of a standard_error_message that makes use of 
sessions). SessionRigExtensions.py has all the code for the external 
methods.

TestSessionRig.py is the script which attacks zope. It should just work 
as is. (Unless the site you just imported is not found at 
http://localhost:8080/sessionrig. If that's the case then simply edit 
the 'path' in the code.) Don't expect the output of this script to say 
anything conclusive. Instead you should be tailing + greping through 
the event log for things of interest.

Now the knobs that you should tweak to so that you see things of 
interest within minutes/hours rather than days/weeks... (The actual 
values may depend on the hardware that your running this on.)

* etc/zope.conf:

session-timeout-minutes 1
maximum-number-of-session-objects 1000
* lib/python/Products/Tranience:

PERIOD = 1 #20 # attempt housekeeping every PERIOD seconds
SPARE_BUCKETS = 2 #15 # number of buckets to keep spare
* lib/python/tempstorage/TempoaryStorage.py:

# keep old object revisions for CONFLICT_CACHE_MAXAGE seconds
CONFLICT_CACHE_MAXAGE = 1 #60
# garbage collect conflict cache every CONFLICT_CACHE_GCEVERY seconds
CONFLICT_CACHE_GCEVERY = 1 #60
* lib/python/Products/SiteErrorLog/SiteErrorLog.py:

_rate_restrict_burst = 10 #5

(I wanted to see repeated errors in the event log. There is probably a 
more obvious way to do that. But that's what I'm using at the moment. 
Took me a while to actually find out this was why I was not seeing all 
the exceptions in the event log. Perhaps things like these should be 
exposed via zconfig? Makes a useful medium to document such knobs to 
promote awareness. As well as allowing people that are pushing zope 
into new corners to have a simpler way to manage these kinds of 
settings.)

* TestSessionRig.py - there are few things that you can tweak within 
here too. (Well the whole script is up for grabs - hack away - I did.)

As the script is it should expose problems with lack of transaction 
isolation for standard_error_message. I do tend to run several 
instances of this script at once. And in my case (as the url suggests) 
I run this on the same host as zope is running on.

However to expose the KeyError thrown from TempoaryStorage I found that 
I did not really need much 'sleepiness' and replaced all the start()'s 
at the end of the script with start(200,1,100,10). (I suspect that you 
don't even need to support cookies in the client for this error. 
Sufficient to hit any URL that simply creates a session object on the 
server. And hit that URL very hard.)

Another effective knob that you may try playing with is the nap 
duration used to slow down a session write:

slowWrite = path + '/write?nap:float=0.5'

hmm... that's probably enough to help someone recreate a KeyError or 
two.

Michael.



sessionrig.zexp
Description: Binary data


SessionRigExtensions.py
Description: Binary data


TestSessionRig.py
Description: Binary data



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Zope 2.X Session problems

2004-05-06 Thread Michael Dunstan
On 7/05/2004, at 5:15 AM, Kris Erickson wrote:

No, that's not the problem;
in THEORY that's what is happening, but in reality there is no way 
that this is the case;
We just unrolled a registration system with participation rates at or 
around 100 to 200 participants per month;
At any given time, monitoring the session data container, there are 
*at most* 1 or 2 items in the transient object container--EXCEPT when 
it spikes...
I have seen such spikes occur (in a corner case) where some breads of 
web robots were aggressively hitting a page that used sessions. These 
robots did not bother to return the cookie handed out by the server. 
Each page hit effectively constructs a new session.

Have a look through your access logs to see if can see signs of 
something similar happening.

Not all web robots are created equal. I ended up sniffing for the user 
agent and returning a page that does not use sessions for the offending 
robots. (From memory, robots.txt was not useful for this bread.) 
Alternatively you can set the maximum-number-of-session-objects to 
something a lot higher and see if you can just live through the bot 
invasion.

Michael.

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: Zope 2.X Session problems

2004-05-06 Thread Michael Dunstan
On 7/05/2004, at 4:39 PM, Tres Seaver wrote:

Michael Dunstan wrote:
On 7/05/2004, at 5:15 AM, Kris Erickson wrote:
No, that's not the problem;
in THEORY that's what is happening, but in reality there is no way 
that this is the case;
We just unrolled a registration system with participation rates at 
or around 100 to 200 participants per month;
At any given time, monitoring the session data container, there are 
*at most* 1 or 2 items in the transient object container--EXCEPT 
when it spikes...
I have seen such spikes occur (in a corner case) where some breads of 
web robots were aggressively hitting a page that used sessions. These 
robots did not bother to return the cookie handed out by the server. 
Each page hit effectively constructs a new session.
Have a look through your access logs to see if can see signs of 
something similar happening.
Not all web robots are created equal. I ended up sniffing for the 
user agent and returning a page that does not use sessions for the 
offending robots. (From memory, robots.txt was not useful for this 
bread.) Alternatively you can set the 
maximum-number-of-session-objects to something a lot higher and see 
if you can just live through the bot invasion.
Even better, avoid writing to the session on each request!  Your 
application will be *much* happier if you write to the session only 
when the human makes a gesture;  neither bots nor casually-browsing 
humans will consume sessions, but only session keys (which are cheap).
Yup - certainly that is a whole lot better if you can arrange for that.

Michael.

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)

2004-04-26 Thread Michael Dunstan
On 11/03/2004, at 7:25 PM, Chris McDonough wrote:

On Wed, 2004-03-10 at 23:14, michael wrote:

Thanks for your excellent isolation here.

I don't get a KeyError thrown until *after* at least one rendering of
standard_error_message that includes an access to a session variable.
And I'm only getting KeyError's for sessions that have had a chance to
expire. And that KeyError will be thrown whenever any access is made
for the session. (Previously I thought I was *only* seeing KeyError's
being thrown from standard_error_message.)
I think after provide transaction isolation for the duration of an
error, we will not see this symptom occur again.
Okay - I've gone ahead and implemented a flavour of transaction  
isolation for errors. (See below for patch.) That seems to have done  
the trick just nicely. I have not seen any more KeyError's thrown from  
Transience.py. Yay!

http://collector.zope.org/Zope/848 has a couple of examples of such  
KeyErrors thrown from Transience. These KeyError were a very painful  
user experience as either the server needed to be restarted or the user  
needed to delete their cookie for that server from their browser cookie  
jar.

However there is another KeyError reported in 848 too. This one is  
thrown from TempoaryStoarge. After some further tweaking of my test rig  
I was able to produce such KeyErrors quite frequently. Using Zope 2.7.0  
and recent the rewrite of Transience.py:

http://cvs.zope.org/*checkout*/Products/Transience/Transience.py? 
rev=1.32.12.2.2.2only_with_tag=chrism-sessiongeddon

errors look like:

2004-04-27T08:24:16 BLATHER(-100) OFS Exception while rendering an  
error message
Traceback (most recent call last):
  File /Users/michael/builds/zope/Zope/lib/python/OFS/SimpleItem.py,  
line 204, in raise_standardErrorMessage
v = s(**kwargs)
  File  
/Users/michael/builds/zope/Zope/lib/python/Shared/DC/Scripts/ 
Bindings.py, line 306, in __call__
return self._bindAndExec(args, kw, None)
  File  
/Users/michael/builds/zope/Zope/lib/python/Shared/DC/Scripts/ 
Bindings.py, line 343, in _bindAndExec
return self._exec(bound_data, args, kw)
  File  
/Users/michael/builds/zope/Zope/lib/python/Products/PythonScripts/ 
PythonScript.py, line 318, in _exec
result = f(*args, **kw)
  File Script (Python), line 12, in standard_error_message
  File  
/Users/michael/builds/zope/Zope/lib/python/ZPublisher/HTTPRequest.py,  
line 1218, in __getattr__
v = self.get(key, default, returnTaints=returnTaints)
  File  
/Users/michael/builds/zope/Zope/lib/python/ZPublisher/HTTPRequest.py,  
line 1178, in get
if callable(v): v = v()
  File  
/Users/michael/builds/zope/Zope/lib/python/Products/Sessions/ 
SessionDataManager.py, line 93, in getSessionData
return self._getSessionDataObject(key)
  File  
/Users/michael/builds/zope/Zope/lib/python/Products/Sessions/ 
SessionDataManager.py, line 180, in _getSessionDataObject
ob = container.new_or_existing(key)
  File  
/Users/michael/builds/zope/Zope/lib/python/Products/Transience/ 
Transience.py, line 504, in new_or_existing
item = self.get(key, _marker)
  File  
/Users/michael/builds/zope/Zope/lib/python/Products/Transience/ 
Transience.py, line 332, in get
item = self._move_item(k, current_ts, _marker)
  File  
/Users/michael/builds/zope/Zope/lib/python/Products/Transience/ 
Transience.py, line 208, in _move_item
self._gc(current_ts)
  File  
/Users/michael/builds/zope/Zope/lib/python/Products/Transience/ 
Transience.py, line 429, in _gc
for key in list(self._data.keys(None, max_ts)):
  File /Users/michael/builds/zope/Zope/lib/python/ZODB/Connection.py,  
line 561, in setstate
p, serial = self._storage.load(oid, self._version)
  File  
/Users/michael/builds/zope/Zope/lib/python/tempstorage/ 
TemporaryStorage.py, line 94, in load
s=self._index[oid]
KeyError: '\x00\x00\x00\x00\x00\x00\x08\x82'

So far it looks like commenting out the construction of the to_notify  
list in _gc makes this go away. So _gc looks like:

def _gc(self, now=None):
if self._timeout_slices == 0:
return # dont do gc if there is no timeout
if now is None:
now = getCurrentTimeslice() # for unit tests
max_ts = now  - (PERIOD * (self._timeout_slices + 1))
keys = self._data.keys(None, max_ts)
#to_notify = []

for key in list(self._data.keys(None, max_ts)):
assert(key = max_ts)
STRICT and _assert(self._data.has_key(key))
#for v in self._data[key].values():
#to_notify.append(v)
del self._data[key]
#for v in to_notify:
#self.notifyDel(v)
Note that the test rig does not set onAdd or onDelete.

Along the way there where a few things that I was not too sure about so  
I tried fiddling with to see what would happen:

(1) Use of hasattr in Transience.py. hasattr should be avoided? See  
http://mail.zope.org/pipermail/zodb-dev/2003-October/005968.html. I did  
change usage 

Re: [Zope-dev] Re: ZPT for CSS, anyone?

2004-03-30 Thread Michael Dunstan
On 31/03/2004, at 7:31 AM, Tres Seaver wrote:
Overall, I agree with you that stylesheets are best kept static. 
However, consider what happens to cacheability when the URLs of of 
images referenced in a document are relative;  the same problem can 
afflict stylesheets, particularly in a setup where virtual hosting is 
in play.

I strongly favor DTML over ZPT for those cases where you need to 
generate dynamic plain text (mail, CSS, Javascript, etc.)  Neither 
will be transparent to tools like Dreamweaver, but then again I 
can't imagine *any* markup that would be transparent;  it would pretty 
much have to be embedded in whatever comment syntax the underlying 
language defines.
hmmm... would a similar idea to TAL ease the pain? A sprinkling of new 
CSS properties along the lines of...

p {
padding: 1.5em;
foo-property: padding context/padding;
}
Could even apply similar ideas for i18n for properties such as 
content.

Michael



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )