Re: [Zope-dev] Re: Conflict errors on BDBMinimal storage

2004-05-18 Thread Dario Lopez-Kästen
Tres Seaver wrote:
I recently reworked Anthony Baxter's SQLSessions product for a 
consulting customer
<...snip...>
What I would like to do is revisit the effort, using the new 
PluginRegistry code released as part of the PluggableAuthService 
release;  I am, however, pretty well booked up for the next few 
weeks. I would be glad to advise anybody who felt called to take on 
such an effort, however.
<...snip...>
I actually need this whithin the next couple of weeks - only one caveat: 
I'd have to do a serious read-up on sessioning, which I don't understand 
at all and the PluggableAuthService product, which I never heard of 
until I read about it.

We have something similar to SQL-Sessions. However, our security 
machinery is built on top of Zopes own (in parallell not replacing it). 
It is loosely modelled around a mix of Zope's way of doing things with a 
hint of XUF (we couldn't use XUF at the time, because it wasn't stable 
enough back then).

One of the goals wiht the current rewrite of one of our apps is to go 
back doing things the Zope-way, by using Zope's own security mechanisms. 
We have working solutions for authentication, but the sessioning part 
remains - especically since we, like in Steve's setup expire sessions on 
the server rather than on the browser end.

So... any hints on where to begin reading for the completely-lost newbie?
/dario
--
-- ---
Dario Lopez-KÃsten, IT Systems & Services Chalmers University of Tech.
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: Conflict errors on BDBMinimal storage

2004-05-18 Thread Chris McDonough
On Tue, 2004-05-18 at 16:36, Chris McDonough wrote:
> In my testing, I'm able to see that naive sessioning applications (ones
> that hit the session on every request, whether they really need to or
> not) seem to fail down due to the "abort after 3 conflicts in a row"
> hardcoded policy under extremely high load (~ 40-60 req/sec or so). 
> This will happen regardless of what code I put in the sessioning
> machinery; attempting to prevent it there is a lost cause: it gets a bit
> like trying to prove the Heisenberg Uncertainty Principal or trying to
> predict the future.

Just as a followup: when I observed this, it was because I had all of my
extra debugging code turned on.  True to the Uncertainty Principle, the
execution of the debugging code caused a raft of extra conflicts. 
Turning off the debugging code leads to a much rosier scenario.  In
cases where it was raising conflicts to the end user, it now operates
without doing so.  I still need to do some really hardcore testing of
the thing to see exactly where it buckles, but I can no longer provoke
it trivially.

> As a result, I am coming to believe that along with the "errors as part
> of main" transaction patch for 2.7.1, I should also make the
> retry-on-conflict-error policy pluggable for those who really
> desperately need to slow their Zope systems to a conflict-induced
> crawl.  They can bump it up to 100 as far as I care and if it works for
> them, great.  The more selective people can wait around for a different
> (non-ZODB-based) sessioning implementation or change their code to not
> pound the snot out of the sessioning machinery unnecessarily.

Michael Dunstan also had another good idea.  At least one knob exists in
the source code to increase/decrease the "resolution" of a part of the
sessioning/transience code.  Basically this knob allows you to choose a
tradeoff between the frequency of a class of writes done by the session
code and the "accuracy" of the session "finalization".

Setting the knob higher will decrease the aggregate number of database
writes that would need to be done over any given period of time in order
to keep sessions "current", which might lead to fewer conflicts.  On the
flip side, it would mean that the session might not be "finalized" on
time (sessions are "finalized" when they are garbage collected; sessions
are only allowed to be garbage collected every "n" seconds where n is
the knob's setting).  This would typically be meaningful only to people
who use the "Script to call at session deletion time" feature of
sessions; that script might be called much later than you think it might
be called if you set this value high enough.  There's currently no
guarantee that it will be called exactly when a session expires even
with the default value, but setting the resolution higher will decrease
that probability even further.

Since that feature isn't used very much (at least I don't think it is),
I think this is an ok tradeoff for a lot of people.  I'd need to make
sure that it actually helps something before doing so, but assuming it
did, it would be put into the config file for 2.7.1 as something like
"session-resolution-seconds" or somesuch.

For those of you with conflict problems now, this knob is the PERIOD
variable in Transience.py... you might be able to get away with setting
it higher now to see if that helps anything, especially those people
trying to use sessions+ZEO.  It's preset to 20, which means 20 seconds. 
Try 120 instead, or something around there.  Just don't set it to a
number higher than the timeout value for the TOC (which is in minutes,
remember to multiply it by 60) or all sorts of insanity will happen. 
You'll need to restart Zope and recreate your transient object container
after doing this, as doing this will effectively invalidate all data
previously in the container and using old data left over from running it
under the default PERIOD value might make it go a little insane too.

> As far as I'm concerned, the new Transience stuff is looking really good
> for the common case.  I haven't been able to reproduce any of the
> corruption issues that happened with the older implementation.

This is still true (it's been almost an hour since I posted it, I think
that's a new record!)

- c




___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: Conflict errors on BDBMinimal storage

2004-05-18 Thread Michael Dunstan
On 19/05/2004, at 8:36 AM, Chris McDonough wrote:
As a result, I am coming to believe that along with the "errors as part
of main" transaction patch for 2.7.1, I should also make the
retry-on-conflict-error policy pluggable for those who really
desperately need to slow their Zope systems to a conflict-induced
crawl.  They can bump it up to 100 as far as I care and if it works for
them, great.
Or this could be used to avoid retries altogether. And at the very 
least it would advertise the fact that zope does retry requests.

Michael.
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)

2004-05-18 Thread Michael Dunstan
On 18/05/2004, at 10:03 PM, Andrea Patuzzo wrote:
Hi, dear developers:
We are now randomly getting this kind of error:
[...]
  File 
"/usr/local/Zope270CVS/lib/python/Products/Transience/Transience.py",
line 341, in __setitem__
current_bucket = self._data[current_ts]
KeyError: 1084872640


This one was with the latest publish.py / startup.py patch and the new
Transience.py implementation, on a yesterday's Zope-2_7-branch 
checkout.
New Zope instance with FileStorage does not solve the problem.
Same happens with the patches applied to the official Zope-2.7.0 
release.

Note: once the error happens, zope has to be restarted to function 
again.
Before applying patches we were getting "get" errors (instead of
"__setitem__") and we just had to close browser to get back to work.
Here we have to restart Zope (with TemporaryStorage) or delete 
Sessions.fs
and restart (with FileStorage).
My guess at the moment is that somewhere a ConflictError is being 
swallowed. There are a few things that can be responsible for this:

- Use of hasattr on Persistent objects.
- try/except blocks that don't raise ConflictErrors.
From your trace that looks like a Plone site. A very quick glance 
through Plone and its friends indicates that there may be some chance 
of either of the above causing some problems. Although Plone 2.0.1 
includes the following fix:

- Fixed bare 'except:' statements in Plone so we don't swallow
  ZODB ConflictErrors. Mostly by adding 'except ConflictError: 
raise'.
  [stefan]

Someone please shoot me down and claim that Plone and friends all do 
the right thing. ;-)

Michael
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Threads dying on zope 2.7.0 on one Zeo setup, ZServer exception

2004-05-18 Thread Dieter Maurer
Heimo Laukkanen wrote at 2004-5-18 14:10 +0300:
> ...
>At the moment with testing setup under load test ZServer dies - leaving 
>all but one thread at almost 0% cpu and that one thread working in high 
>loads of 90%. Testing similar zeo-setup with development instances do 
>not produce this.

Provided, you work under Linux, you can try to attach
to the process with GDB and analyse the thread behaviour.

I use the functions "ps" (Print String) and "pfr" (Print FRame)
during such analysis. They are defined by:

def ps
x/s ({PyStringObject}$arg0)->ob_sval
end

def pfr
ps f->f_code->co_filename
ps f->f_code->co_name
p f->f_lineno
end

You can apply "ps" to an string object.

You can call "pfr" in an "eval_frame" frame context.

-- 
Dieter

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Conflict errors on BDBMinimal storage

2004-05-18 Thread Dieter Maurer
Gerhard Schmidt wrote at 2004-5-17 21:01 +0200:
> ...
>> However, due to a ZEO weakness, its conflict resolution
>> currently does not work (ZEO no longer supports "INSTANCE_HOME").
>
>Is that a bug or intetional. 

I expect it to be a bug.

I plan to fix it soon -- but currently, I have lots of things to do.
May take some months...

-- 
Dieter

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: Conflict errors on BDBMinimal storage

2004-05-18 Thread Tres Seaver
robert rottermann wrote:
On Tue, 2004-05-18 at 14:15, Tres Seaver wrote:
 

I recently reworked Anthony Baxter's SQLSessions product for a 
consulting customer, essentially modernizing the code and making it 
compliant with the new Sessions interface (I dropped the bits which 
do the work currently done by BrowserIDManager).  It took me about 
six hours or so, including writing unit tests.  Unfortunately, I 
don't have access to that code, due to the nature of the consulting 
arrangement.

What I would like to do is revisit the effort, using the new 
PluginRegistry code released as part of the PluggableAuthService 
release;  I am, however, pretty well booked up for the next few 
weeks. I would be glad to advise anybody who felt called to take on 
such an effort, however.
  
I do not know whether I understand this email correctly. So mabe I am 
off topic.
We would like to use  thre PluggableAuthService to replace a userfolder 
that authenticates against a an email server (calling a web service to 
do so) and then caches authentification and other user info in a session.
So we would like to dig into what both of you seem not to have time to do.
How can we get at PluggableAuthService?
I did not find it anywhere.
The PluggableAuthService product is available in CVS:
   http://cvs.zope.org/Products/PluggableAuthService
It depends on the PluginRegistry product:
   http://cvs.zope.org/Products/PluginRegistry
Both are "releasable", but not yet released (I just reminded Zac to cut 
the release tarballs).  PAS has a SessionAuthHelper plugin which should 
serve as a model for the part of your problem which revolves around the 
session;  you would need to write an "authenticator" plugin which 
validated credentials against your web service.

And we would of course gladly accept Tres offer for advise.
I was referring to using the PluginRegistry, which is a lower-level 
utility, to (re)write the SQLSessions product, which needs to be 
"pluggable" in order to accomodate different backends.  The PIR is a 
Zope 2 analog to the Zope 3 "local utility" service:  it stores a 
perstent map of the plugins registered / activated for a set of 
application-defined plugin interfaces.

Tres.
--
===
Tres Seaver[EMAIL PROTECTED]
Zope Corporation  "Zope Dealers"   http://www.zope.com
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: Conflict errors on BDBMinimal storage

2004-05-18 Thread Chris McDonough
On Tue, 2004-05-18 at 14:58, Chris McDonough wrote:
> On Tue, 2004-05-18 at 14:15, Tres Seaver wrote:
> > I recently reworked Anthony Baxter's SQLSessions product for a 
> > consulting customer, essentially modernizing the code and making it 
> > compliant with the new Sessions interface (I dropped the bits which do 
> > the work currently done by BrowserIDManager).  It took me about six 
> > hours or so, including writing unit tests.  Unfortunately, I don't have 
> > access to that code, due to the nature of the consulting arrangement.
> > 
> > What I would like to do is revisit the effort, using the new 
> > PluginRegistry code released as part of the PluggableAuthService 
> > release;  I am, however, pretty well booked up for the next few weeks. 
> > I would be glad to advise anybody who felt called to take on such an 
> > effort, however.
> 
> I would also like to see this happen.  I don't have the time to help
> implement this right now, but I may be able to work it in to an ongoing
> engagement at some point in the next few months.

In my testing, I'm able to see that naive sessioning applications (ones
that hit the session on every request, whether they really need to or
not) seem to fail down due to the "abort after 3 conflicts in a row"
hardcoded policy under extremely high load (~ 40-60 req/sec or so). 
This will happen regardless of what code I put in the sessioning
machinery; attempting to prevent it there is a lost cause: it gets a bit
like trying to prove the Heisenberg Uncertainty Principal or trying to
predict the future.

As a result, I am coming to believe that along with the "errors as part
of main" transaction patch for 2.7.1, I should also make the
retry-on-conflict-error policy pluggable for those who really
desperately need to slow their Zope systems to a conflict-induced
crawl.  They can bump it up to 100 as far as I care and if it works for
them, great.  The more selective people can wait around for a different
(non-ZODB-based) sessioning implementation or change their code to not
pound the snot out of the sessioning machinery unnecessarily.

As far as I'm concerned, the new Transience stuff is looking really good
for the common case.  I haven't been able to reproduce any of the
corruption issues that happened with the older implementation.

- C



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: Conflict errors on BDBMinimal storage

2004-05-18 Thread robert rottermann
Chris McDonough wrote:
On Tue, 2004-05-18 at 14:15, Tres Seaver wrote:
 

I recently reworked Anthony Baxter's SQLSessions product for a 
consulting customer, essentially modernizing the code and making it 
compliant with the new Sessions interface (I dropped the bits which do 
the work currently done by BrowserIDManager).  It took me about six 
hours or so, including writing unit tests.  Unfortunately, I don't have 
access to that code, due to the nature of the consulting arrangement.

What I would like to do is revisit the effort, using the new 
PluginRegistry code released as part of the PluggableAuthService 
release;  I am, however, pretty well booked up for the next few weeks. 
I would be glad to advise anybody who felt called to take on such an 
effort, however.
   

I would also like to see this happen.  I don't have the time to help
implement this right now, but I may be able to work it in to an ongoing
engagement at some point in the next few months.
- C

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

 

I do not know whether I understand this email correctly. So mabe I am 
off topic.
We would like to use  thre PluggableAuthService to replace a userfolder 
that authenticates against a an email server (calling a web service to 
do so) and then caches authentification and other user info in a session.
So we would like to dig into what both of you seem not to have time to do.
How can we get at PluggableAuthService?
I did not find it anywhere.
And we would of course gladly accept Tres offer for advise.

Robert
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: Conflict errors on BDBMinimal storage

2004-05-18 Thread Chris McDonough
On Tue, 2004-05-18 at 14:15, Tres Seaver wrote:
> I recently reworked Anthony Baxter's SQLSessions product for a 
> consulting customer, essentially modernizing the code and making it 
> compliant with the new Sessions interface (I dropped the bits which do 
> the work currently done by BrowserIDManager).  It took me about six 
> hours or so, including writing unit tests.  Unfortunately, I don't have 
> access to that code, due to the nature of the consulting arrangement.
> 
> What I would like to do is revisit the effort, using the new 
> PluginRegistry code released as part of the PluggableAuthService 
> release;  I am, however, pretty well booked up for the next few weeks. 
> I would be glad to advise anybody who felt called to take on such an 
> effort, however.

I would also like to see this happen.  I don't have the time to help
implement this right now, but I may be able to work it in to an ongoing
engagement at some point in the next few months.

- C



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)

2004-05-18 Thread Chris McDonough
On Tue, 2004-05-18 at 13:11, Steve Jibson wrote:
> Well, after much log reading, I have found that the KeyError we got last 
> night was OUR fault.

Thanks for reporting this, it's reasurring.  I was fixing to go insane.
;-)

> I will fix the problem with our app, then I want to change back to 
> TemporaryStorage and watch the system some more.  I'll keep you posted.

Great, thanks.

> Here's the brief explanation of our problem (you can skip it if you like):
> 
> A user logged in and did some stuff then left his browser for almost an 
> hour.  When he returned and tried to do more stuff, he was no longer in 
> the ExUserFolder's credential cache and his session had expired.  He was 
> forced to log in again.  Upon supplying his ID and password, he was sent 
> to the "loginSuccess" page.  This is the one that calls our method to 
> set up his user session.  The Z2.log shows a 302 result code on this 
> page.

Ooh, that's bad.  I guess that means that your users might not be
logging in as whom they think they're logging in as...  CookieCrumbler
handles this by injecting Pragma: No-Cache and the HTTP/1.1 equivalent
into the header (currently on *every* request, not so great actually).

> His browser had the "loginSuccess" page in cache, so it did not 
> request it again and his session was never re-created.
> 
> Score one for Chris's suggestion on how we should be setting up the 
> user's session.  For now, however, I think I'll just add the 
> please-don't-cache-me header stuff to the RESPONSE.

Right... although caching+auth is kind of tricky.   I wouldn't be
surprised if you found other places where caching is defeating some
other intentions you had.  If the site is meant to be accessed only by
authenticated users, I'd be tempted to arrange for the no-cache headers
to be injected by the main template itself.

I don't understand your setup 100%, although it sounds interesting.  By
your description of it sounds like you're implementing a bit of your own
security machinery (a user can either be an "admin" or a normal user,
depending on some setting he chooses at login time, even though it's
physically only one account).  But maybe an alternate way to implement
the machinery to do this is to do the following:

- Associate "member data" with each user.  This is built in to
  CMF, or you can roll your own if you're not using CMF
  (maybe by manually using a
  Transient Object Container in a FileStorage with the timeout set
  to 0... it basically then just becomes a big dictionary).
  Store user "preferences" there.  Give each user a default
  set of preferences for first-time login.

- Allow the user to log in with just a username and password
  (don't require him to enter his profile data on each login).

- In the session "add notify" script, copy his prefs into his
  session.  Your existing code will then work normally.

- In the session "delete notify" script, copy the prefs data
  in the session back into the member data.

Just an idea... 

- C



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: Conflict errors on BDBMinimal storage

2004-05-18 Thread Tres Seaver
Chris McDonough wrote:
Hi Dr. Wagner,

thanks for the hint with minimal storage. however the same problem 
occurs with filestorage and is even worse.

we know very well, that we cannot avoid the appearance of conflict 
errors in particular with zeo, and we understand what causes them.

what wonders us is, that we find some contributions about this errors in 
the mailing lists, but no one seems to have a real problem with it. but 
we have! we are not worried about some errors appearing in a log, but we 
are using plone2 witch currently 30 concurrent authenticated users, so 
that this error occurs permanently, thus decreasing system performance 
due to several retries and moreover leading to very frequent nasty 
'POS.Exception Errors' in the browser, whenever four successive tries 
fail (almost every 100. request).

this is not acceptable, and i  wonder wheter there are any plone sites 
with more than a handful of authenticated users that experience the same 
problem.  we found that one (there are more) product responsible for it 
is the placeless translation service, which heavily writes to the session.

Are you using ZEO for your session storage or are you using a local
storage?  (You hint at it above but you don't actually say one way or
the other.)

so we desperately need a solution for this problem, maybe uncouple the 
session of the locking mechanism or whatever, and this should be a 
problem that should be addressed by the core team. because this is a 
strong reason (and so far the only one) to avoid using zope/plone in a 
large user environment.

I also need reliable sessioning for my own customers, so I am working
towards fixing the issues I know about.  But the issues I know about may
not be the same issues as those that you want fixed.  You can ensure
your issues are addressed by devoting resources towards the problem, be
that time+expertise or money or both.
I recently reworked Anthony Baxter's SQLSessions product for a 
consulting customer, essentially modernizing the code and making it 
compliant with the new Sessions interface (I dropped the bits which do 
the work currently done by BrowserIDManager).  It took me about six 
hours or so, including writing unit tests.  Unfortunately, I don't have 
access to that code, due to the nature of the consulting arrangement.

What I would like to do is revisit the effort, using the new 
PluginRegistry code released as part of the PluggableAuthService 
release;  I am, however, pretty well booked up for the next few weeks. 
I would be glad to advise anybody who felt called to take on such an 
effort, however.

Tres.
--
===
Tres Seaver[EMAIL PROTECTED]
Zope Corporation  "Zope Dealers"   http://www.zope.com
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)

2004-05-18 Thread Steve Jibson
Well, after much log reading, I have found that the KeyError we got last 
night was OUR fault.

I will fix the problem with our app, then I want to change back to 
TemporaryStorage and watch the system some more.  I'll keep you posted.

Here's the brief explanation of our problem (you can skip it if you like):
A user logged in and did some stuff then left his browser for almost an 
hour.  When he returned and tried to do more stuff, he was no longer in 
the ExUserFolder's credential cache and his session had expired.  He was 
forced to log in again.  Upon supplying his ID and password, he was sent 
to the "loginSuccess" page.  This is the one that calls our method to 
set up his user session.  The Z2.log shows a 302 result code on this 
page.  His browser had the "loginSuccess" page in cache, so it did not 
request it again and his session was never re-created.

Score one for Chris's suggestion on how we should be setting up the 
user's session.  For now, however, I think I'll just add the 
please-don't-cache-me header stuff to the RESPONSE.

Steve Jibson wrote:
Good morning.
I just got in and checked on my customer's system.  In the past 22 1/2 
hours they've had 15000 page hits and last night at about 9:30, ONE 
person got a KeyError.  Actually, this same person got twenty KeyErrors 
over a period of about 45 seconds.  I'm downloading their log files now 
and plan to spend some time this morning going through them.

Anyway, it appears that I was wrong when I said that the problem doesn't 
show up when I use FileStorage (although it does seem to happen less 
frequently -- but who can be sure of anything at this point?).

In answer to your questions earlier, Chris, we set up the user session 
at login time because we make the user answer some questions at login 
time that determine which portions of the interface to present to 
him/her.  For example, using the same login id and password, a user may 
choose to login as an administrator or as a normal user.  We store this 
choice and other info based on this choice in the session.  Also, we 
don't rely on the browser to time out the authentication cookie.  Once a 
user authenticates with ExUserFolder, ExUserFolder keeps their 
credentials in a cache until they have been inactive for 10 minutes (the 
timer resets with each cache hit).  If their credentials are not in the 
cache, rather than looking them up again, the user is logged out and 
must re-authenticate.  It seems like a reasonable way to handle logins 
and sessions.

In addition to going through log files, I will spend some more time 
today making sure we're not doing something stupid in our app.

Thanks again (to Chris, Michael, Alex and everyone else who has lost 
sleep over this session stuff).  I'll keep you posted on any new 
information I find.

Steve
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)

2004-05-18 Thread Max M
Andrea Patuzzo wrote:
We are now randomly getting this kind of error:

Note: once the error happens, zope has to be restarted to function again.
Before applying patches we were getting "get" errors (instead of
"__setitem__") and we just had to close browser to get back to work.

That's right. Under Plone 2.0 and Zope 2.7 I had the same kind of 
problem. But when I restarted the browser it would be solved.

That could point in the direction of a session id being somehow involved.
--
hilsen/regards Max M, Denmark
http://www.mxm.dk/
IT's Mad Science
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)

2004-05-18 Thread Steve Jibson
Good morning.
I just got in and checked on my customer's system.  In the past 22 1/2 
hours they've had 15000 page hits and last night at about 9:30, ONE 
person got a KeyError.  Actually, this same person got twenty KeyErrors 
over a period of about 45 seconds.  I'm downloading their log files now 
and plan to spend some time this morning going through them.

Anyway, it appears that I was wrong when I said that the problem doesn't 
show up when I use FileStorage (although it does seem to happen less 
frequently -- but who can be sure of anything at this point?).

In answer to your questions earlier, Chris, we set up the user session 
at login time because we make the user answer some questions at login 
time that determine which portions of the interface to present to 
him/her.  For example, using the same login id and password, a user may 
choose to login as an administrator or as a normal user.  We store this 
choice and other info based on this choice in the session.  Also, we 
don't rely on the browser to time out the authentication cookie.  Once a 
user authenticates with ExUserFolder, ExUserFolder keeps their 
credentials in a cache until they have been inactive for 10 minutes (the 
timer resets with each cache hit).  If their credentials are not in the 
cache, rather than looking them up again, the user is logged out and 
must re-authenticate.  It seems like a reasonable way to handle logins 
and sessions.

In addition to going through log files, I will spend some more time 
today making sure we're not doing something stupid in our app.

Thanks again (to Chris, Michael, Alex and everyone else who has lost 
sleep over this session stuff).  I'll keep you posted on any new 
information I find.

Steve
Chris McDonough wrote:
On Mon, 2004-05-17 at 23:08, Chris McDonough wrote:
There indeed is a minor off-by-one error: it manifests itself as
sessions timing out at most 20 seconds early.
But there is also a deeper issue which involves the fact that a session
data object is not properly removed from an older bucket when it "moves"
due to being accessed in a later timeslice; the symptom only appears
when a browser id is "reused" to start a session after it was used to
start an older one that had timed out normally.  I've got almost no clue
why this happens at this point, but I'm working on it.  Ugh.  This is
almost certainly what Steve is experiencing.

I take that back.  Actually, I think I was just reading the test results
and debug output wrong.  It appears to be operating normally except for
the off-by-one problem (which is minor).  I need to jack up the tests to
do some comparisons of data values; currently I'm just testing to ensure
that *something* is in the session.. I need to test if the "right" thing
is in the session over time.
- C

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)

2004-05-18 Thread Andrea Patuzzo
Hi, dear developers:

We are now randomly getting this kind of error:


2004-05-18T11:30:41 ERROR(200) SiteError
http://XXX/prefs_error_log_form
Traceback (most recent call last):
  File "/usr/local/Zope270CVS/lib/python/ZPublisher/Publish.py", line 100,
in publish
request, bind=1)
  File "/usr/local/Zope270CVS/lib/python/ZPublisher/mapply.py", line 88, in
mapply
if debug is not None: return debug(object,args,context)
  File "/usr/local/Zope270CVS/lib/python/ZPublisher/Publish.py", line 40, in
call_object
result=apply(object,args) # Type s to step into published object.
  File "/usr/local/Zope270CVS/lib/python/Shared/DC/Scripts/Bindings.py",
line 306, in __call__
return self._bindAndExec(args, kw, None)
  File "/usr/local/Zope270CVS/lib/python/Shared/DC/Scripts/Bindings.py",
line 343, in _bindAndExec
return self._exec(bound_data, args, kw)
  File "/zope_instances/patuzzo.ch/Products/CMFCore/FSPageTemplate.py", line
191, in _exec
result = self.pt_render(extra_context=bound_names)
  File "/zope_instances/patuzzo.ch/Products/CMFCore/FSPageTemplate.py", line
124, in pt_render
result = FSPageTemplate.inheritedAttribute('pt_render')(
  File
"/usr/local/Zope270CVS/lib/python/Products/PageTemplates/PageTemplate.py",
line 96, in pt_render
tal=not source, strictinsert=0)()
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 189,
in __call__
self.interpret(self.program)
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 233,
in interpret
handlers[opcode](self, args)
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 663,
in do_useMacro
self.interpret(macro)
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 233,
in interpret
handlers[opcode](self, args)
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 408,
in do_optTag_tal
self.do_optTag(stuff)
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 393,
in do_optTag
return self.no_tag(start, program)
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 388,
in no_tag
self.interpret(program)
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 233,
in interpret
handlers[opcode](self, args)
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 663,
in do_useMacro
self.interpret(macro)
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 233,
in interpret
handlers[opcode](self, args)
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 408,
in do_optTag_tal
self.do_optTag(stuff)
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 393,
in do_optTag
return self.no_tag(start, program)
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 388,
in no_tag
self.interpret(program)
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 233,
in interpret
handlers[opcode](self, args)
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 663,
in do_useMacro
self.interpret(macro)
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 233,
in interpret
handlers[opcode](self, args)
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 629,
in do_condition
self.interpret(block)
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 233,
in interpret
handlers[opcode](self, args)
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 406,
in do_optTag_tal
self.no_tag(stuff[-2], stuff[-1])
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 388,
in no_tag
self.interpret(program)
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 233,
in interpret
handlers[opcode](self, args)
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 605,
in do_loop_tal
self.interpret(block)
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 233,
in interpret
handlers[opcode](self, args)
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 290,
in do_startTag
ok, name, s = attrAction(self, item)
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 369,
in attrAction_tal
translated = self.translate(msgid or value, value, {})
  File "/usr/local/Zope270CVS/lib/python/TAL/TALInterpreter.py", line 615,
in translate
msgid, i18ndict, default=default)
  File "/usr/local/Zope270CVS/lib/python/Products/PageTemplates/TALES.py",
line 264, in translate
target_language=target_language)
  File
"/zope_instances/patuzzo.ch/Products/PlacelessTranslationService/PlacelessTr
anslationService.py", line 109, in translate
return service.translate(domain, msgid, mapping, context,
target_language, default)
  File
"/zope_instances/patuzzo.ch/Products/PlacelessTranslationService/PlacelessTr
anslationService.py", line 423, in translate
catalogs = self.getCatal

[Zope-dev] Threads dying on zope 2.7.0 on one Zeo setup, ZServer exception

2004-05-18 Thread Heimo Laukkanen
Hi ya,
I noticed some weird bahivour that was bringing a setup down on our
servers on testing installation. In our setup we create development 
sandboxes with makefile and in similar fashion also testing/production 
servers.

Our testing/production setup has a little bit different config files 
where from standard config all the commented out lines have been 
removed. Also compared to development instances the instance home is in 
other place. Other than that setups are same.

At the moment with testing setup under load test ZServer dies - leaving 
all but one thread at almost 0% cpu and that one thread working in high 
loads of 90%. Testing similar zeo-setup with development instances do 
not produce this.

We are a bit scared on this, since we do not have clue on what causes 
following behaviour - and eventhough we see that we can get it to work, 
we would like to have an idea on what might be behind this - so that it 
does not bite our butt again.

Below is log snip and traceback of the exceptions when other threads go 
down.

In log-file we get:
2004-05-17T16:47:17 ERROR(200) ZServer uncaptured python exception,
closing channel 
127.0.0.1:47855 at 0x42fdb36c channel#: 589 requests:>
(socket.error:(104, 'Connection reset by peer')
[/opt/portaali/ContentManagement-1.0/python-2.3.3/lib/python2.3/asynchat.py|initiate_send|218] 

[/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZServer/medusa/http_server.py|send|417] 

[/opt/portaali/ContentManagement-1.0/python-2.3.3/lib/python2.3/asyncore.py|send|337])
2004-05-17T16:48:49 ERROR(200) ZServer uncaptured python exception,
closing channel 
127.0.0.1:47924 at 0x430538ac channel#: 651 requests:>
(socket.error:(104, 'Connection reset by peer')
[/opt/portaali/ContentManagement-1.0/python-2.3.3/lib/python2.3/asynchat.py|initiate_send|218] 

[/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZServer/medusa/http_server.py|send|417] 

[/opt/portaali/ContentManagement-1.0/python-2.3.3/lib/python2.3/asyncore.py|send|337])
2004-05-17T16:50:05 ERROR(200) ZServer uncaptured python exception,
closing channel 
127.0.0.1:47961 at 0x42f3ba4c channel#: 683 requests:>
(socket.error:(104, 'Connection reset by peer')
[/opt/portaali/ContentManagement-1.0/python-2.3.3/lib/python2.3/asynchat.py|initiate_send|218] 

[/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZServer/medusa/http_server.py|send|417] 

[/opt/portaali/ContentManagement-1.0/python-2.3.3/lib/python2.3/asyncore.py|send|337])

In tracback we will get:
2004-05-18T11:44:14 INFO(0) Zope Ready to handle requests
Unhandled exception in thread started by 
ZServer.PubCore.ZServerPublisher.ZServerPublisher at 0x405435cc>
Traceback (most recent call last):
  File
"/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZServer/PubCore/ZServerPublisher.py", 

line 23, in __init__
response=response)
  File
"/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZPublisher/Publish.py", 

line 372, in publish_module
environ, debug, request, response)
  File
"/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZPublisher/Publish.py", 

line 173, in publish_module_standard
request.response.exception()
AttributeError: 'NoneType' object has no attribute 'exception'
Unhandled exception in thread started by 
ZServer.PubCore.ZServerPublisher.ZServerPublisher at 0x405435cc>
Traceback (most recent call last):
  File
"/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZServer/PubCore/ZServerPublisher.py", 

line 23, in __init__
response=response)
  File
"/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZPublisher/Publish.py", 

line 372, in publish_module
environ, debug, request, response)
  File
"/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZPublisher/Publish.py", 

line 173, in publish_module_standard
request.response.exception()
AttributeError: 'NoneType' object has no attribute 'exception'
Unhandled exception in thread started by 
ZServer.PubCore.ZServerPublisher.ZServerPublisher at 0x405435cc>
Traceback (most recent call last):
  File
"/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZServer/PubCore/ZServerPublisher.py", 

line 23, in __init__
response=response)
  File
"/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZPublisher/Publish.py", 

line 372, in publish_module
environ, debug, request, response)
  File
"/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZPublisher/Publish.py", 

line 173, in publish_module_standard
request.response.exception()
AttributeError: 'NoneType' object has no attribute 'exception'
Unhandled exception in thread started by 
ZServer.PubCore.ZServerPublisher.ZServerPublisher at 0x405435cc>
Traceback (most recent call last):
  File
"/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZServer/PubCore/ZServerPublisher.py", 

line 23, in __init__
response=response)
  File
"/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZPublisher/Publish.py", 

line 372, in publi

Re: [Zope-dev] Re: [ZODB-Dev] Re: BTrees strangeness (was Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)

2004-05-18 Thread Chris McDonough
On Mon, 2004-05-17 at 23:08, Chris McDonough wrote:
> There indeed is a minor off-by-one error: it manifests itself as
> sessions timing out at most 20 seconds early.
> 
> But there is also a deeper issue which involves the fact that a session
> data object is not properly removed from an older bucket when it "moves"
> due to being accessed in a later timeslice; the symptom only appears
> when a browser id is "reused" to start a session after it was used to
> start an older one that had timed out normally.  I've got almost no clue
> why this happens at this point, but I'm working on it.  Ugh.  This is
> almost certainly what Steve is experiencing.

I take that back.  Actually, I think I was just reading the test results
and debug output wrong.  It appears to be operating normally except for
the off-by-one problem (which is minor).  I need to jack up the tests to
do some comparisons of data values; currently I'm just testing to ensure
that *something* is in the session.. I need to test if the "right" thing
is in the session over time.

- C



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Conflict errors on BDBMinimal storage

2004-05-18 Thread Chris McDonough
Hi Dr. Wagner,

> thanks for the hint with minimal storage. however the same problem 
> occurs with filestorage and is even worse.
> 
> we know very well, that we cannot avoid the appearance of conflict 
> errors in particular with zeo, and we understand what causes them.
> 
> what wonders us is, that we find some contributions about this errors in 
> the mailing lists, but no one seems to have a real problem with it. but 
> we have! we are not worried about some errors appearing in a log, but we 
> are using plone2 witch currently 30 concurrent authenticated users, so 
> that this error occurs permanently, thus decreasing system performance 
> due to several retries and moreover leading to very frequent nasty 
> 'POS.Exception Errors' in the browser, whenever four successive tries 
> fail (almost every 100. request).
> 
> this is not acceptable, and i  wonder wheter there are any plone sites 
> with more than a handful of authenticated users that experience the same 
> problem.  we found that one (there are more) product responsible for it 
> is the placeless translation service, which heavily writes to the session.

Are you using ZEO for your session storage or are you using a local
storage?  (You hint at it above but you don't actually say one way or
the other.)

> so we desperately need a solution for this problem, maybe uncouple the 
> session of the locking mechanism or whatever, and this should be a 
> problem that should be addressed by the core team. because this is a 
> strong reason (and so far the only one) to avoid using zope/plone in a 
> large user environment.

I also need reliable sessioning for my own customers, so I am working
towards fixing the issues I know about.  But the issues I know about may
not be the same issues as those that you want fixed.  You can ensure
your issues are addressed by devoting resources towards the problem, be
that time+expertise or money or both.

- C



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )