Re: [web2py] Re: Long-running controllers

2011-01-20 Thread António Ramos
Please add to the book your suggestion. 2011/1/19 Massimo Di Pierro > Your code does not block the server. Only blocks the session > associated to your browser to guarantee integrity of the data in your > session. If you know that the block function takes time and it does > not need to save sess

[web2py] Re: Long-running controllers

2011-01-19 Thread cjrh
On Jan 19, 10:05 pm, Anthony wrote: > I'm not sure about that, but I found this great Monty Python reference while > looking into it: > >     def bring_out_your_dead(self): >         # Remove dead threads from the pool Awesome!

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread Jonathan Lundell
On Jan 19, 2011, at 7:15 AM, Anthony wrote: > On Wednesday, January 19, 2011 1:35:09 AM UTC-5, Jonathan Lundell wrote: > Actually, there's a bug in the documentation (or else in Session). You have > to say: > > session.forget(response) > > or else the session doesn't get unlocked > > There

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread Jonathan Lundell
On Jan 19, 2011, at 1:45 PM, ae wrote: > > I understand getting a different session is expected. I'm talking > about different behavior in how requests are dispatched to threads. > The above scenarios for example which leads me to believe you can't > reliably use long running functions. Again, i

[web2py] Re: Long-running controllers

2011-01-19 Thread ae
I understand getting a different session is expected. I'm talking about different behavior in how requests are dispatched to threads. The above scenarios for example which leads me to believe you can't reliably use long running functions.

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread Jonathan Lundell
On Jan 19, 2011, at 12:49 PM, ae wrote: > > I can say that using two different browsers (FF & Chrome) from the > same host does not exhibit the same behavior as two tabs/windows from > the same browser (same profile...). Yes, that's expected. Browsers don't share their cookie jars, so there's no

[web2py] Re: Long-running controllers

2011-01-19 Thread ae
I can say that using two different browsers (FF & Chrome) from the same host does not exhibit the same behavior as two tabs/windows from the same browser (same profile...). On Jan 19, 1:46 pm, ron_m wrote: > I remember session.forget() documented as a performance enhancement - no > plans to alter

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread ron_m
Okay, I get it, Storage id a dict but adds a __getattr__ which returns None on missing key. Nothing like learning in public -- blush.

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread Jonathan Lundell
On Jan 19, 2011, at 12:32 PM, ae wrote: > > I think the default minimum is 8 and it seems there are 3 other > 'overhead' threads. (if I just start web2py without any thread > options, ps shows 11 threads.) The default minimum is 10 (set in Rocket). I think you're seeing the main thread plus the

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread Jonathan Lundell
On Jan 19, 2011, at 12:32 PM, ron_m wrote: > I was talking about the _unlock call at the top of _try_store_on_disk > > def _try_store_on_disk(self, request, response): > if not hasattr(os,'mkdir'): return > if response._dbtable_and_field \ > or not response.sess

[web2py] Re: Long-running controllers

2011-01-19 Thread ae
I think the default minimum is 8 and it seems there are 3 other 'overhead' threads. (if I just start web2py without any thread options, ps shows 11 threads.) If I do --maxthreads=2, I get 11 threads total. If I do --minthreads=2 --maxthreads=2, I get 5 total (3 + 2?) So, since the default is 8 a

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread ron_m
I was talking about the _unlock call at the top of _try_store_on_disk def _try_store_on_disk(self, request, response): if not hasattr(os,'mkdir'): return if response._dbtable_and_field \ or not response.session_id \ or self._forget: s

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread Jonathan Lundell
On Jan 19, 2011, at 11:48 AM, ae wrote: > > --maxthreads does nothing if you don't specify --minthreads, so that > should be fixed one way or another. I don't know if that problem is > in web2py or Rocket. I can't find a problem. How did it present itself? One thing to keep in mind is that the

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread Anthony
On Wednesday, January 19, 2011 2:58:33 PM UTC-5, Jonathan Lundell wrote: > > Also, I notice the command line options section of the book ( > http://web2py.com/book/default/chapter/04?search=shell#Command-Line-Options) > still refers to the --numthreads option -- should that be replaced with the

[web2py] Re: Long-running controllers

2011-01-19 Thread Anthony
On Wednesday, January 19, 2011 2:51:25 PM UTC-5, ae wrote: > > Now that I think about it, it seems likely that somewhere something's > doing: > > if maxthreads < minthreads: > maxthreads = minthreads I'm not sure about that, but I found this great Monty Python reference while looking i

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread Jonathan Lundell
On Jan 19, 2011, at 11:43 AM, Anthony wrote: > On Wednesday, January 19, 2011 2:33:55 PM UTC-5, Jonathan Lundell wrote: > On Jan 19, 2011, at 11:23 AM, ron_m wrote: > > Maybe it is a Windows only issue, I was reviewing the flock(2) man page and > > it isn't a problem in UNIX systems. I haven't loo

[web2py] Re: Long-running controllers

2011-01-19 Thread ae
Now that I think about it, it seems likely that somewhere something's doing: if maxthreads < minthreads: maxthreads = minthreads On Jan 19, 2:48 pm, ae wrote: > --maxthreads does nothing if you don't specify --minthreads, so that > should be fixed one way or another.  I don't know if that p

[web2py] Re: Long-running controllers

2011-01-19 Thread ae
--maxthreads does nothing if you don't specify --minthreads, so that should be fixed one way or another. I don't know if that problem is in web2py or Rocket.

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread Anthony
On Wednesday, January 19, 2011 2:33:55 PM UTC-5, Jonathan Lundell wrote: > > On Jan 19, 2011, at 11:23 AM, ron_m wrote: > > Maybe it is a Windows only issue, I was reviewing the flock(2) man page > and it isn't a problem in UNIX systems. I haven't looked at the Windows docs > but remember file a

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread Jonathan Lundell
On Jan 19, 2011, at 11:23 AM, ron_m wrote: > Maybe it is a Windows only issue, I was reviewing the flock(2) man page and > it isn't a problem in UNIX systems. I haven't looked at the Windows docs but > remember file access is more restrictive on that system. For now it is only a > suspicion, I d

[web2py] Re: Long-running controllers

2011-01-19 Thread Massimo Di Pierro
Good catch. Thanks Jonathan. On Jan 19, 12:45 pm, Jonathan Lundell wrote: > On Jan 19, 2011, at 10:26 AM, ron_m wrote: > > > If the response is left off the session.forget() parameter list it defaults > > to None. The end result then is the session._forget state variable is set > > True but the

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread ron_m
Maybe it is a Windows only issue, I was reviewing the flock(2) man page and it isn't a problem in UNIX systems. I haven't looked at the Windows docs but remember file access is more restrictive on that system. For now it is only a suspicion, I don't use Windows so would have to do a lot to test

[web2py] Re: Long-running controllers

2011-01-19 Thread ae
On Jan 19, 10:14 am, ae wrote: > Well, maybe not.  I just downloaded 1.91.6 and I couldn't replicate > the behavior.  We use 1.67.1 in production and what I tested on might > have been even older than that. So I'm back to not sure again. When I tested earlier I used the -- maxthreads option, but

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread Jonathan Lundell
On Jan 19, 2011, at 11:06 AM, ron_m wrote: > There is another subtle problem. If the file close is not done after the > unlock which is how it works now then the file is still open. It is not > possible to get the exclusive lock in session.connect() until the file is > closed so the second reque

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread ron_m
There is another subtle problem. If the file close is not done after the unlock which is how it works now then the file is still open. It is not possible to get the exclusive lock in session.connect() until the file is closed so the second request remains blocked.

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread Jonathan Lundell
On Jan 19, 2011, at 10:52 AM, ron_m wrote: > I saw that too last night but it was late. It would be better to set > response.session_file to None which will force a file close since the file > object goes out of scope. Adding a del in _unlock will cause AttributeError > in the _try_store_on_disk

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread ron_m
It would be better to set response.session_file to None in _unlock because the second _unlock call will generate an AttributeError when testing for response.session_file at the top of _unlock.

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread ron_m
I saw that too last night but it was late. It would be better to set response.session_file to None which will force a file close since the file object goes out of scope. Adding a del in _unlock will cause AttributeError in the _try_store_on_disk when the request-response cycle is finishing up.

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread Jonathan Lundell
On Jan 19, 2011, at 10:32 AM, Anthony wrote: > So, I guess the question is, what should the book (and the 'call' function in > the scaffolding app) be recommending, session.forget() or > session.forget(response)? At the very least, the difference between the two > should be explained in the book

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread ron_m
I remember session.forget() documented as a performance enhancement - no plans to alter session on this call, do this to save the write. I think this would be the most common use. Adding the response parameter to unlock the session file is only required if there is a need for concurrency in the

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread Jonathan Lundell
On Jan 19, 2011, at 10:26 AM, ron_m wrote: > If the response is left off the session.forget() parameter list it defaults > to None. The end result then is the session._forget state variable is set > True but the session file is not unlocked in the _unlock function. This would > enhance performan

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread Anthony
So, I guess the question is, what should the book (and the 'call' function in the scaffolding app) be recommending, session.forget() or session.forget(response)? At the very least, the difference between the two should be explained in the book. On Wednesday, January 19, 2011 1:26:06 PM UTC-5,

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread ron_m
If the response is left off the session.forget() parameter list it defaults to None. The end result then is the session._forget state variable is set True but the session file is not unlocked in the _unlock function. This would enhance performance by bypassing the writing of the session file at

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread Jonathan Lundell
On Jan 19, 2011, at 6:50 AM, ae wrote: > > > > On Jan 19, 9:17 am, Anthony wrote: >> On Wednesday, January 19, 2011 8:18:25 AM UTC-5, ae wrote: >> >>> I have lots of users and when one thread takes a long time, some users >>> can keep accessing controller functions and some can't. Once the lo

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread Anthony
On Wednesday, January 19, 2011 1:35:09 AM UTC-5, Jonathan Lundell wrote: > > Actually, there's a bug in the documentation (or else in Session). You have > to say: > > session.forget(response) > > or else the session doesn't get unlocked > There are at least four occurrences of session.forg

[web2py] Re: Long-running controllers

2011-01-19 Thread ae
On Jan 18, 11:21 pm, Anthony wrote: > I tried this (the session.forget() version), and sayjunk is still being > blocked for 30 seconds while waiting for blockme to finish (only when in the > same browser -- I can load sayjunk in a different browser, which starts a > different session). Am I miss

[web2py] Re: Long-running controllers

2011-01-19 Thread Anthony
Hmm, this sounds contrary to what Massimo and Jonathan have been saying. >From their explanations, it sounds like user A should not be blocking user B (i.e., their requests should not be sharing a thread). On Wednesday, January 19, 2011 9:50:35 AM UTC-5, ae wrote: > > > On Jan 19, 9:17 am, Ant

[web2py] Re: Long-running controllers

2011-01-19 Thread ae
On Jan 19, 9:17 am, Anthony wrote: > On Wednesday, January 19, 2011 8:18:25 AM UTC-5, ae wrote: > > > I have lots of users and when one thread takes a long time, some users > > can keep accessing controller functions and some can't.  Once the long- > > running controller finishes, everyone can a

[web2py] Re: Long-running controllers

2011-01-19 Thread ae
> No, a cookie is associated with a session. Agreed. I'm pretty much using 'cookie', 'user', 'session', and 'browser' as interchangeable for my purposes. A cookie is associated to a session, but somehow a session is associated to a thread. Either that or I'm seeing some other effect for which I

[web2py] Re: Long-running controllers

2011-01-19 Thread ae
On Jan 18, 11:21 pm, Anthony wrote: > Am I missing something? Yes. You're probably running web2py with the default number of threads (30 I think?). Your 2nd browser window's cookie is unlikely to get associated to the same thread. I have lots of users and when one thread takes a long time, som

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread ron_m
Oh crud, I didn't see the self._forget state variable test at the front of _try-store_on_disk, sorry for the noise and response.session_file is a file object so having it go out of scope will close it. Ron

Re: [web2py] Re: Long-running controllers

2011-01-19 Thread ron_m
I think there is another bug as well. The _unlock function only unlocks the session file if it exists but leaves response.session_file defined. Later when the request cycle finishes the _try_store_on_disk function will still see response.session_file as active and write it but this is now dan

Re: [web2py] Re: Long-running controllers

2011-01-18 Thread Jonathan Lundell
On Jan 18, 2011, at 6:50 PM, ae wrote: > > On Jan 18, 9:17 pm, Jonathan Lundell wrote: >> On Jan 18, 2011, at 5:41 PM, ae wrote: >> >> >> >>> On Jan 18, 11:22 am, Jonathan Lundell wrote: >> When you say "anyone associated with the thread", do you mean other requests using some sha

Re: [web2py] Re: Long-running controllers

2011-01-18 Thread Jonathan Lundell
On Jan 18, 2011, at 7:58 PM, Massimo Di Pierro wrote: > > Your code does not block the server. Only blocks the session > associated to your browser to guarantee integrity of the data in your > session. If you know that the block function takes time and it does > not need to save session, you can d

[web2py] Re: Long-running controllers

2011-01-18 Thread Anthony
I tried this (the session.forget() version), and sayjunk is still being blocked for 30 seconds while waiting for blockme to finish (only when in the same browser -- I can load sayjunk in a different browser, which starts a different session). Am I missing something? On Tuesday, January 18, 2011

[web2py] Re: Long-running controllers

2011-01-18 Thread Massimo Di Pierro
Your code does not block the server. Only blocks the session associated to your browser to guarantee integrity of the data in your session. If you know that the block function takes time and it does not need to save session, you can do: def blockme(): session.forget() import time time.

[web2py] Re: Long-running controllers

2011-01-18 Thread ae
On Jan 18, 9:17 pm, Jonathan Lundell wrote: > On Jan 18, 2011, at 5:41 PM, ae wrote: > > > > > On Jan 18, 11:22 am, Jonathan Lundell wrote: > > >> When you say "anyone associated with the thread", do you mean other > >> requests using some shared, locked resource (like the session)? Or > >> som

Re: [web2py] Re: Long-running controllers

2011-01-18 Thread Jonathan Lundell
On Jan 18, 2011, at 5:41 PM, ae wrote: > > On Jan 18, 11:22 am, Jonathan Lundell wrote: > >> When you say "anyone associated with the thread", do you mean other requests >> using some shared, locked resource (like the session)? Or something else? > > Browser sessions seem to get associated to

[web2py] Re: Long-running controllers

2011-01-18 Thread ae
On Jan 18, 5:06 pm, cjrh wrote: > > I suggest you present some code to this group that shows how laborious > it is, and ask for simplifications.  I am not a javascript guru, but > some do lurk here and that would be the best way to make it less > laborious.  web2py itself does have some helpers

[web2py] Re: Long-running controllers

2011-01-18 Thread ae
On Jan 18, 11:22 am, Jonathan Lundell wrote: > > When you say "anyone associated with the thread", do you mean other requests > using some shared, locked resource (like the session)? Or something else? Browser sessions seem to get associated to a thread. As long as that thread is busy the use

[web2py] Re: Long-running controllers

2011-01-18 Thread cjrh
On Jan 18, 5:39 pm, ae wrote: > How do people deal with controller functions that take a long time? 1) If you must do something that takes time, start it as a separate process, and that process toggles a flag that says the job is complete. The user can be notified of the completion via email. T