[web2py] Re: Apache, Wsgi problem
That's fine. But it makes your opinion irrelevant. Very few program in isolation.
[web2py] Re: Apache, Wsgi problem
On Monday, January 10, 2011 6:36:18 AM UTC+2, Anthony wrote: On Sunday, January 9, 2011 1:24:02 AM UTC-5, Graham Dumpleton wrote: You guys really just got to learn to do your own thing and not treating it like a crusade where you have to convert the world. But that's exactly the problem -- web2py is trying to do its own thing, ...so far so good... and there is a vocal clique of Pythonistas who don't like that thing and want to stop it. ...and here we get it wrong. Why should we *care* about this vocal clique? Arguing on reddit and other forums about the minutae of tit-for-tat features is almost entirely pointless. It convinces nobody, *least* of all the newbies who are still trying to decide on which framework to use. And this is something else that bothers me. *Why* should we care how many users we have? It seems to me an entirely pointless objective. I care only about trying to make web2py *my* favourite platform for web apps. In an online discussion between proponents of different frameworks, or different *anything* for that matter, there is precious little to be gained from arguing the case. That is just how it is. If we say we just want to do our own thing, then let's do just that. Seriously, going after spurious comments in every reddit or reddit-alike is *harmful*. Getting the truth out there has a lot of side effects, and they're not all good. The best, and IMO only real way to promote web2py is to facilitate the building of awesome websites. That is the whole ball game, so let's focus on that. This is, I suppose, obligatory: http://xkcd.com/386/ Graham is in a unique position in that wsgi is used by all the frameworks and so he wears a badge that allows him to move freely between them without carrying any baggage, and he clearly gets to see how each of the communities function. It is important for us to listen to what he has to say, because we will not get such feedback from anywhere else I can think of.
[web2py] Re: Apache, Wsgi problem
On Friday, July 8, 2011 3:25:45 AM UTC-4, cjrh wrote: On Monday, January 10, 2011 6:36:18 AM UTC+2, Anthony wrote: On Sunday, January 9, 2011 1:24:02 AM UTC-5, Graham Dumpleton wrote: You guys really just got to learn to do your own thing and not treating it like a crusade where you have to convert the world. But that's exactly the problem -- web2py is trying to do its own thing, ...so far so good... and there is a vocal clique of Pythonistas who don't like that thing and want to stop it. ...and here we get it wrong. Why should we *care* about this vocal clique? Well, we probably shouldn't care too much, but that doesn't mean we shouldn't care at all. Note, the unfounded criticisms are effectively impugning both the technical and personal integrity of web2py's developers and users, in very public forums. This can have negative professional (and personal) consequences. If you have chosen to build an app with web2py and your client, manager, or investor reads bad things about web2py, you may lose their confidence and favor. So, occassional polite and professional corrections may be appropriate, especially on purely factual misunderstandings (rather than issues of opinion and taste). *Why* should we care how many users we have? Again, this is not the most important thing, but the number of users is not entirely without consequence either. Generally, more users will translate into more expert contributors and volunteers, which will contribute to the development of the framework and its ecosystem and to its long-term sustainability (i.e., more features, better testing and performance, better documentation, more plugins and applications, better support, more third-party hosting and development tool support, etc.). In many cases, when you select a framework, you want some confidence that it will still be actively maintained several years hence, and that confidence will be bolstered if there is a sizable and active community around it. The best, and IMO only real way to promote web2py is to facilitate the building of awesome websites. That is the whole ball game, so let's focus on that. Absolutely. :-)
[web2py] Re: Apache, Wsgi problem
On Friday, 8 July 2011 16:05:54 UTC+2, Anthony wrote: On Friday, July 8, 2011 3:25:45 AM UTC-4, cjrh wrote: On Monday, January 10, 2011 6:36:18 AM UTC+2, Anthony wrote: On Sunday, January 9, 2011 1:24:02 AM UTC-5, Graham Dumpleton wrote: You guys really just got to learn to do your own thing and not treating it like a crusade where you have to convert the world. But that's exactly the problem -- web2py is trying to do its own thing, ...so far so good... and there is a vocal clique of Pythonistas who don't like that thing and want to stop it. ...and here we get it wrong. Why should we *care* about this vocal clique? Note, the unfounded criticisms are effectively impugning both the technical and personal integrity of web2py's developers and users, in very public forums. This can have negative professional (and personal) consequences. If you have chosen to build an app with web2py and your client, manager, or investor reads bad things about web2py, you may lose their confidence and favor. Haters gonna hate :) *Why* should we care how many users we have? Again, this is not the most important thing, but the number of users is not entirely without consequence either. Generally, more users will translate into more expert contributors and volunteers, which will contribute to the development of the framework and its ecosystem and to its long-term sustainability (i.e., more features, better testing and performance, better documentation, more plugins and applications, better support, more third-party hosting and development tool support, etc.). This is almost definitely not true. Or rather, it certainly hasn't been my experience anyway. The number of people in a project or community appears to have no bearing whatsoever on the quality of the code or the abilities of contributors. In many cases, when you select a framework, you want some confidence that it will still be actively maintained several years hence, and that confidence will be bolstered if there is a sizable and active community around it. I still don't see why this is my problem, or indeed even yours, or anyone else's for that matter. I do not see why I need to care about whether some pointy-hair boss thinks web2py will be actively maintained in the future. I have selected web2py on technical merit. I do not care how active the community is, for I have the source code. In truth, I have selected web2py because my value system around coding has a large intersection with Massimo's, IMO. I value the conveniences that he built into web2py early on; conveniences that other frameworks left as an exercise to the reader. I think that matters. I think that remains a selling point for web2py. I think it is unfortunate that he uses the word Enterprise to indicate that those conveniences exist, and I think that that causes a lot more confusion than it should. But our PR face is way, way too defensive. I understand that *we *might not see it like that, but if someone like Graham happens point out that that is how it looks from the outside, then we should take note of that. IMO there is very little merit in just about anything that goes through reddit. So why bother wasting time on it? I am not sure I even want to attract too many coders that spend time reading reddit anyway. Apologies upfront if that offends anyone.
[web2py] Re: Apache, Wsgi problem
On Friday, July 8, 2011 3:47:09 PM UTC-4, cjrh wrote: *Why* should we care how many users we have? Again, this is not the most important thing, but the number of users is not entirely without consequence either. Generally, more users will translate into more expert contributors and volunteers, which will contribute to the development of the framework and its ecosystem and to its long-term sustainability (i.e., more features, better testing and performance, better documentation, more plugins and applications, better support, more third-party hosting and development tool support, etc.). This is almost definitely not true. Or rather, it certainly hasn't been my experience anyway. The number of people in a project or community appears to have no bearing whatsoever on the quality of the code or the abilities of contributors. It's not just about the core framework, but the larger ecosystem. Right now there are several hosting services in beta dedicated specifically to hosting Django apps. Pycharm includes special functionality to support Django development. There are lots of libraries and reusable apps specifically for Django. (Not to mention what you can find in the Rails, PHP, or ASP.NET worlds.) So, I think there can be some benefits to a larger community and user base, though such benefits may not matter to everyone. In many cases, when you select a framework, you want some confidence that it will still be actively maintained several years hence, and that confidence will be bolstered if there is a sizable and active community around it. I still don't see why this is my problem, or indeed even yours, or anyone else's for that matter. I do not see why I need to care about whether some pointy-hair boss thinks web2py will be actively maintained in the future. I have selected web2py on technical merit. I do not care how active the community is, for I have the source code. Yes, but not everyone wants to worry about having to take over development and maintenance of their app's framework should it be abandoned by its core developers. Why do we worry about backward compatibility? Backward compatibility only matters if the framework is actively being developed to begin with, so concern about backward compatibility follows from a concern about long-term sustainability. I agree that technical merit is critical, but it's not the only consideration, at least not for all users. In truth, I have selected web2py because my value system around coding has a large intersection with Massimo's, IMO. I value the conveniences that he built into web2py early on; conveniences that other frameworks left as an exercise to the reader. I think that matters. I think that remains a selling point for web2py. Hear, hear! But our PR face is way, way too defensive. I understand that *we *might not see it like that, but if someone like Graham happens point out that that is how it looks from the outside, then we should take note of that. Note, this thread is from back in January, and in the past few months, there has been relatively little negative reddit activity, so perhaps we have already moved on to better days. :-)
[web2py] Re: Apache, Wsgi problem
Sorry cjrh, but the number of users and the perception of the project DO matter. Some of us are building businesses around Web2py and need to feel comfortable that the project has longevity, that we can hire and retain engineers, that we can find resources if we need them, etc. Not everyone is doing just their own one-person projects. I don't really care if engineering candidates have Web2py experience or not but I do care if any perceptual issues handicap recruiting. You need to think slightly bigger than your own personal projects.
[web2py] Re: Apache, Wsgi problem
Have I missed some new controversial reddit thread? On Jul 8, 4:18 pm, pbreit pbreitenb...@gmail.com wrote: Sorry cjrh, but the number of users and the perception of the project DO matter. Some of us are building businesses around Web2py and need to feel comfortable that the project has longevity, that we can hire and retain engineers, that we can find resources if we need them, etc. Not everyone is doing just their own one-person projects. I don't really care if engineering candidates have Web2py experience or not but I do care if any perceptual issues handicap recruiting. You need to think slightly bigger than your own personal projects.
[web2py] Re: Apache, Wsgi problem
No, or rather, not that I am aware. We are discussing, I think, the principal of the matter.
[web2py] Re: Apache, Wsgi problem
On Friday, 8 July 2011 23:18:57 UTC+2, pbreit wrote: You need to think slightly bigger than your own personal projects. Why should I? The bigger community does not concern me. I contribute to web2py for entirely selfish reasons, because it want it to be the best tool for me to use for my own personal (actually work) projects. I do not care one iota about what other people may or may not think about web2py. It doesn't concern me even one bit. I have tried the various web frameworks, and I have made a decision based on the relative merits to focus on web2py. I don't see AT ALL why I need to think any bigger than my own current needs.
Re: [web2py] Re: Apache, Wsgi problem
For some reason I missed the rest of this discussion. Just a quick follow up. + I upgraded psycopg2 from 2.0.7 (default in Debian 5) to the latest version, 2.4. This manual installation was smoother than I thought. + Result: preliminary stress tests show that the problem *disappeared*. No more WSGI premature script error.
Re: [web2py] Re: Apache, Wsgi problem
On Mar 4, 2011, at 9:31 AM, VP wrote: For some reason I missed the rest of this discussion. Just a quick follow up. + I upgraded psycopg2 from 2.0.7 (default in Debian 5) to the latest version, 2.4. This manual installation was smoother than I thought. + Result: preliminary stress tests show that the problem *disappeared*. No more WSGI premature script error. Thanks for the report. I wonder if it might be worth a warning message if an app actually uses (not just imports) psycopg2 and the version is 2.0.7 or earlier. It seems to be fairly widespread.
[web2py] Re: Apache, Wsgi problem
To be precise, RSS started with about 32MB (with an apache restart) and starts crawling up to 80MB/process with aforementioned testing. And If I remember correctly, it stayed that way over night and did not get released. As I said, I will do further testing with this toy example and with my real app and report further findings. On Mar 4, 11:31 am, VP vtp2...@gmail.com wrote: For some reason I missed the rest of this discussion. Just a quick follow up. + I upgraded psycopg2 from 2.0.7 (default in Debian 5) to the latest version, 2.4. This manual installation was smoother than I thought. + Result: preliminary stress tests show that the problem *disappeared*. No more WSGI premature script error.
Re: [web2py] Re: Apache, Wsgi problem
On Mar 4, 2011, at 9:46 AM, VP wrote: To be precise, RSS started with about 32MB (with an apache restart) and starts crawling up to 80MB/process with aforementioned testing. And If I remember correctly, it stayed that way over night and did not get released. As I said, I will do further testing with this toy example and with my real app and report further findings. It'd be useful to know whether it's stable at 80MB or it keeps growing. On Mar 4, 11:31 am, VP vtp2...@gmail.com wrote: For some reason I missed the rest of this discussion. Just a quick follow up. + I upgraded psycopg2 from 2.0.7 (default in Debian 5) to the latest version, 2.4. This manual installation was smoother than I thought. + Result: preliminary stress tests show that the problem *disappeared*. No more WSGI premature script error.
[web2py] Re: Apache, Wsgi problem
oops, it looks like that last post was supposed to be for the other thread (potential memory leaks). On Mar 4, 12:28 pm, Jonathan Lundell jlund...@pobox.com wrote: On Mar 4, 2011, at 9:46 AM, VP wrote: To be precise, RSS started with about 32MB (with an apache restart) and starts crawling up to 80MB/process with aforementioned testing. And If I remember correctly, it stayed that way over night and did not get released. As I said, I will do further testing with this toy example and with my real app and report further findings. It'd be useful to know whether it's stable at 80MB or it keeps growing. On Mar 4, 11:31 am, VP vtp2...@gmail.com wrote: For some reason I missed the rest of this discussion. Just a quick follow up. + I upgraded psycopg2 from 2.0.7 (default in Debian 5) to the latest version, 2.4. This manual installation was smoother than I thought. + Result: preliminary stress tests show that the problem *disappeared*. No more WSGI premature script error.
Re: [web2py] Re: Apache, Wsgi problem
Question: Could provide you provide an app that generates the error? Answer: Check Appliance Customer Relationship Management. db = DAL(postgres://postgres: Error Snapshot : type 'exceptions.RuntimeError'(Failure to connect, tried 5 times: global name 'psycopg2' is not defined)
[web2py] Re: Apache, Wsgi problem
The problem here is that you did not install the psycopg2 driver for postgresql. On Feb 16, 2:28 am, Ahmed Bani taz...@gmail.com wrote: Question: Could provide you provide an app that generates the error? Answer: Check Appliance Customer Relationship Management. db = DAL(postgres://postgres: Error Snapshot : type 'exceptions.RuntimeError'(Failure to connect, tried 5 times: global name 'psycopg2' is not defined)
[web2py] Re: Apache, Wsgi problem
Problem solved:
# Download : hg clone https://web2py.googlecode.com/hg/ web2py# Follow the
instructions:
Title : Using psycopg2 with virtualenv on UbuntuLink:
http://www.saltycrane.com/blog/2009/07/using-psycopg2-virtualenv-ubuntu-jaunty/
Check your syntax :
# *db* = *DAL*('*postgres*://myself:mypass@127.0.0.1:5432/mydatabase')
Enjoy it!
Now, do we have to migrate web2py database and reimport it again? :
Short tutorial to migrate web2py database - model and
datahttp://alexandremas.blogspot.com/2009/11/short-tutorial-to-migrate-web2py.html
[web2py] Re: Apache, Wsgi problem
Looks like Jonathan may be the winner here. I think VP raised the
issue so he should decide whether the issue is solved.
Is anybody else having problems after upgrade to the latest psycopg2?
If this problem was dear to you any symbolic contribution towards the
prize will be appreciated.
Massimo
On Feb 10, 8:19 am, Massimo Di Pierro massimo.dipie...@gmail.com
wrote:
I offer $300 to whoever can identify within 3 weeks and without
ambiguity the cause of this problem. The bounty applies even if the
problem turns out to be not in web2py but in one of the Python
modules, in Apache or in the mod_wsgi implementation. It does not
apply if the problem is due to known bug that has already been fixed
by the responsible party (for example if you are using an old un-
patched python version like 2.5.0 or an old mod_wsgi version).
On Feb 10, 12:38 am, VP vtp2...@gmail.com wrote:
Alright people short answer: I think I figured this out (at least
with my configuration)
After testing various configurations, here's the result with: ab -kc
100 -t 20https://domain.com/imageblog/default/index/ (same
imageblog app, 100 connections, 20 seconds stress test).
Two things you will notice with this result.
1. ZERO failed request. No more wsgi premature script error
2. Complete requests is 1234, 61 requests/second on average.
Compare to prior configuration: 588 total requests, 29 requests/
sec on average. Not to mention 15 failed requests due to wsgi
premature script errors!!!
This is insane!!!
So how did I configure this? here it is:
WSGIDaemonProcess web2py user=username group=username \
display-name=%{GROUP} processes=5 threads=1
The important option being 5 processes, 1 thread.
With this configuration, my real app also did not get wsgi premature
script errors anymore. And guess what... the requests/sec
triples
I am still curious about this. While my real app can possibly be not
thread-safe, but the imageblog app should be thread safe (the index
was simply a listing of images, i.e. read only). Why would there be a
problem with more than 1 thread?
Document Path: /imageblog/default/index
Document Length: 13083 bytes
Concurrency Level: 100
Time taken for tests: 20.008 seconds
Complete requests: 1234
Failed requests: 0
Write errors: 0
Keep-Alive requests: 1234
Total transferred: 16827432 bytes
HTML transferred: 16171262 bytes
Requests per second: 61.68 [#/sec] (mean)
Time per request: 1621.377 [ms] (mean)
Time per request: 16.214 [ms] (mean, across all concurrent
requests)
Transfer rate: 821.33 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 9.4 0 73
Processing: 82 481 890.5 317 5475
Waiting: 76 443 878.7 274 5393
Total: 82 483 894.7 317 5503
Percentage of the requests served within a certain time (ms)
50% 317
66% 342
75% 360
80% 372
90% 416
95% 489
98% 5351
99% 5397
100% 5503 (longest request)
Re: [web2py] Re: Apache, Wsgi problem
On Feb 16, 2011, at 3:07 PM, Massimo Di Pierro wrote:
Looks like Jonathan may be the winner here. I think VP raised the
issue so he should decide whether the issue is solved.
Is anybody else having problems after upgrade to the latest psycopg2?
If this problem was dear to you any symbolic contribution towards the
prize will be appreciated.
It need only be symbolic; I think that this falls into the known bug
category. I found it by Googling.
To be more certain, it'd be good if someone could revert psycogpg2 (with no
other changes) and reproduce the problem.
Massimo
On Feb 10, 8:19 am, Massimo Di Pierro massimo.dipie...@gmail.com
wrote:
I offer $300 to whoever can identify within 3 weeks and without
ambiguity the cause of this problem. The bounty applies even if the
problem turns out to be not in web2py but in one of the Python
modules, in Apache or in the mod_wsgi implementation. It does not
apply if the problem is due to known bug that has already been fixed
by the responsible party (for example if you are using an old un-
patched python version like 2.5.0 or an old mod_wsgi version).
On Feb 10, 12:38 am, VP vtp2...@gmail.com wrote:
Alright people short answer: I think I figured this out (at least
with my configuration)
After testing various configurations, here's the result with: ab -kc
100 -t 20https://domain.com/imageblog/default/index/ (same
imageblog app, 100 connections, 20 seconds stress test).
Two things you will notice with this result.
1. ZERO failed request. No more wsgi premature script error
2. Complete requests is 1234, 61 requests/second on average.
Compare to prior configuration: 588 total requests, 29 requests/
sec on average. Not to mention 15 failed requests due to wsgi
premature script errors!!!
This is insane!!!
So how did I configure this? here it is:
WSGIDaemonProcess web2py user=username group=username \
display-name=%{GROUP} processes=5 threads=1
The important option being 5 processes, 1 thread.
With this configuration, my real app also did not get wsgi premature
script errors anymore. And guess what... the requests/sec
triples
I am still curious about this. While my real app can possibly be not
thread-safe, but the imageblog app should be thread safe (the index
was simply a listing of images, i.e. read only). Why would there be a
problem with more than 1 thread?
Document Path: /imageblog/default/index
Document Length:13083 bytes
Concurrency Level: 100
Time taken for tests: 20.008 seconds
Complete requests: 1234
Failed requests:0
Write errors: 0
Keep-Alive requests:1234
Total transferred: 16827432 bytes
HTML transferred: 16171262 bytes
Requests per second:61.68 [#/sec] (mean)
Time per request: 1621.377 [ms] (mean)
Time per request: 16.214 [ms] (mean, across all concurrent
requests)
Transfer rate: 821.33 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect:01 9.4 0 73
Processing:82 481 890.53175475
Waiting: 76 443 878.72745393
Total: 82 483 894.73175503
Percentage of the requests served within a certain time (ms)
50%317
66%342
75%360
80%372
90%416
95%489
98% 5351
99% 5397
100% 5503 (longest request)
Re: [web2py] Re: Apache, Wsgi problem
Greetings. I have tried to replicate the error and have had no exicto. Environment: virtual server 1: debian ssl web2py virtual server 2: debian ssl postgresql Application: simple queries and throwing 2 results. tool: Jmeter: Threads 1200 0.2 seconds upload Díaz Luis TSU Analisis de Sistemas Universidad de Carabobo Facultad de Odontologíahttp://www.odontologia.uc.edu.ve/index.php?option=com_contentview=articleid=102Itemid=85 2011/2/12 Luis Díaz diazluis2...@gmail.com Translation error .. sorry. I could not .. might wonder if any of you? Díaz Luis TSU Analisis de Sistemas Universidad de Carabobo Facultad de Odontologíahttp://www.odontologia.uc.edu.ve/index.php?option=com_contentview=articleid=102Itemid=85 2011/2/12 Jonathan Lundell jlund...@pobox.com On Feb 12, 2011, at 8:46 AM, ron_m wrote: FWIW I am on Ubuntu 10.04 with all patches applied and could not reproduce the above problem when I tried. The version of python-psycopg2 from the standard repository is 2.0.13. Not an iron-clad guarantee because something else could be at play but at least a data point. I also drove the application a lot harder from ab with 10x number of requests to try and reproduce but still nothing. The fact it doesn't fail when processes=5 and threads=1 agrees well with the GIL problem - without threads there is no need to hold the GIL. Yes, it's a useful data point. The best test will be to have an environment that actually fails with psycopg2 2.0.7, and have the failure disappear with an updated version (and no other changes), and to get the error back again with 2.0.7.
Re: [web2py] Re: Apache, Wsgi problem
On Feb 14, 2011, at 9:48 AM, Luis Díaz wrote: Greetings. I have tried to replicate the error and have had no exicto. Thanks. What version of psycopg2 are you using? Environment: virtual server 1: debian ssl web2py virtual server 2: debian ssl postgresql Application: simple queries and throwing 2 results. tool: Jmeter: Threads 1200 0.2 seconds upload Díaz Luis TSU Analisis de Sistemas Universidad de Carabobo Facultad de Odontología 2011/2/12 Luis Díaz diazluis2...@gmail.com Translation error .. sorry. I could not .. might wonder if any of you? Díaz Luis TSU Analisis de Sistemas Universidad de Carabobo Facultad de Odontología 2011/2/12 Jonathan Lundell jlund...@pobox.com On Feb 12, 2011, at 8:46 AM, ron_m wrote: FWIW I am on Ubuntu 10.04 with all patches applied and could not reproduce the above problem when I tried. The version of python-psycopg2 from the standard repository is 2.0.13. Not an iron-clad guarantee because something else could be at play but at least a data point. I also drove the application a lot harder from ab with 10x number of requests to try and reproduce but still nothing. The fact it doesn't fail when processes=5 and threads=1 agrees well with the GIL problem - without threads there is no need to hold the GIL. Yes, it's a useful data point. The best test will be to have an environment that actually fails with psycopg2 2.0.7, and have the failure disappear with an updated version (and no other changes), and to get the error back again with 2.0.7.
Re: [web2py] Re: Apache, Wsgi problem
I have installed: python-psycopg2/lenny update 2.0.7-4 Question: What is the funsion of psycopg2? Díaz Luis TSU Analisis de Sistemas Universidad de Carabobo Facultad de Odontologíahttp://www.odontologia.uc.edu.ve/index.php?option=com_contentview=articleid=102Itemid=85 2011/2/14 Jonathan Lundell jlund...@pobox.com On Feb 14, 2011, at 9:48 AM, Luis Díaz wrote: Greetings. I have tried to replicate the error and have had no exicto. Thanks. What version of psycopg2 are you using? Environment: virtual server 1: debian ssl web2py virtual server 2: debian ssl postgresql Application: simple queries and throwing 2 results. tool: Jmeter: Threads 1200 0.2 seconds upload Díaz Luis TSU Analisis de Sistemas Universidad de Carabobo Facultad de Odontologíahttp://www.odontologia.uc.edu.ve/index.php?option=com_contentview=articleid=102Itemid=85 2011/2/12 Luis Díaz diazluis2...@gmail.com Translation error .. sorry. I could not .. might wonder if any of you? Díaz Luis TSU Analisis de Sistemas Universidad de Carabobo Facultad de Odontologíahttp://www.odontologia.uc.edu.ve/index.php?option=com_contentview=articleid=102Itemid=85 2011/2/12 Jonathan Lundell jlund...@pobox.com On Feb 12, 2011, at 8:46 AM, ron_m wrote: FWIW I am on Ubuntu 10.04 with all patches applied and could not reproduce the above problem when I tried. The version of python-psycopg2 from the standard repository is 2.0.13. Not an iron-clad guarantee because something else could be at play but at least a data point. I also drove the application a lot harder from ab with 10x number of requests to try and reproduce but still nothing. The fact it doesn't fail when processes=5 and threads=1 agrees well with the GIL problem - without threads there is no need to hold the GIL. Yes, it's a useful data point. The best test will be to have an environment that actually fails with psycopg2 2.0.7, and have the failure disappear with an updated version (and no other changes), and to get the error back again with 2.0.7.
Re: [web2py] Re: Apache, Wsgi problem
ready to investigate: psycopg2 http://wiki.postgresql.org/wiki/Using_psycopg2_with_PostgreSQL The error could be caused by a bad query developed? Could provide an app that generates the error? Díaz Luis TSU Analisis de Sistemas Universidad de Carabobo Facultad de Odontologíahttp://www.odontologia.uc.edu.ve/index.php?option=com_contentview=articleid=102Itemid=85 2011/2/14 Luis Díaz diazluis2...@gmail.com I have installed: python-psycopg2/lenny update 2.0.7-4 Question: What is the funsion of psycopg2? Díaz Luis TSU Analisis de Sistemas Universidad de Carabobo Facultad de Odontologíahttp://www.odontologia.uc.edu.ve/index.php?option=com_contentview=articleid=102Itemid=85 2011/2/14 Jonathan Lundell jlund...@pobox.com On Feb 14, 2011, at 9:48 AM, Luis Díaz wrote: Greetings. I have tried to replicate the error and have had no exicto. Thanks. What version of psycopg2 are you using? Environment: virtual server 1: debian ssl web2py virtual server 2: debian ssl postgresql Application: simple queries and throwing 2 results. tool: Jmeter: Threads 1200 0.2 seconds upload Díaz Luis TSU Analisis de Sistemas Universidad de Carabobo Facultad de Odontologíahttp://www.odontologia.uc.edu.ve/index.php?option=com_contentview=articleid=102Itemid=85 2011/2/12 Luis Díaz diazluis2...@gmail.com Translation error .. sorry. I could not .. might wonder if any of you? Díaz Luis TSU Analisis de Sistemas Universidad de Carabobo Facultad de Odontologíahttp://www.odontologia.uc.edu.ve/index.php?option=com_contentview=articleid=102Itemid=85 2011/2/12 Jonathan Lundell jlund...@pobox.com On Feb 12, 2011, at 8:46 AM, ron_m wrote: FWIW I am on Ubuntu 10.04 with all patches applied and could not reproduce the above problem when I tried. The version of python-psycopg2 from the standard repository is 2.0.13. Not an iron-clad guarantee because something else could be at play but at least a data point. I also drove the application a lot harder from ab with 10x number of requests to try and reproduce but still nothing. The fact it doesn't fail when processes=5 and threads=1 agrees well with the GIL problem - without threads there is no need to hold the GIL. Yes, it's a useful data point. The best test will be to have an environment that actually fails with psycopg2 2.0.7, and have the failure disappear with an updated version (and no other changes), and to get the error back again with 2.0.7.
Re: [web2py] Re: Apache, Wsgi problem
On Feb 14, 2011, at 10:24 AM, Luis Díaz wrote: ready to investigate: psycopg2 http://wiki.postgresql.org/wiki/Using_psycopg2_with_PostgreSQL The error could be caused by a bad query developed? Could provide an app that generates the error? I'm not using it myself; I'm not sure what triggers the error. I assume that there's more information earlier in this thread. Díaz Luis TSU Analisis de Sistemas Universidad de Carabobo Facultad de Odontología 2011/2/14 Luis Díaz diazluis2...@gmail.com I have installed: python-psycopg2/lenny update 2.0.7-4 Question: What is the funsion of psycopg2? Díaz Luis TSU Analisis de Sistemas Universidad de Carabobo Facultad de Odontología 2011/2/14 Jonathan Lundell jlund...@pobox.com On Feb 14, 2011, at 9:48 AM, Luis Díaz wrote: Greetings. I have tried to replicate the error and have had no exicto. Thanks. What version of psycopg2 are you using? Environment: virtual server 1: debian ssl web2py virtual server 2: debian ssl postgresql Application: simple queries and throwing 2 results. tool: Jmeter: Threads 1200 0.2 seconds upload Díaz Luis TSU Analisis de Sistemas Universidad de Carabobo Facultad de Odontología 2011/2/12 Luis Díaz diazluis2...@gmail.com Translation error .. sorry. I could not .. might wonder if any of you? Díaz Luis TSU Analisis de Sistemas Universidad de Carabobo Facultad de Odontología 2011/2/12 Jonathan Lundell jlund...@pobox.com On Feb 12, 2011, at 8:46 AM, ron_m wrote: FWIW I am on Ubuntu 10.04 with all patches applied and could not reproduce the above problem when I tried. The version of python-psycopg2 from the standard repository is 2.0.13. Not an iron-clad guarantee because something else could be at play but at least a data point. I also drove the application a lot harder from ab with 10x number of requests to try and reproduce but still nothing. The fact it doesn't fail when processes=5 and threads=1 agrees well with the GIL problem - without threads there is no need to hold the GIL. Yes, it's a useful data point. The best test will be to have an environment that actually fails with psycopg2 2.0.7, and have the failure disappear with an updated version (and no other changes), and to get the error back again with 2.0.7.
[web2py] Re: Apache, Wsgi problem
In any case, please check whether upgrading postgresql for you solves the problem. It is possible that mysqldb has/had the same issue. I am very skeptical this is a web2py issue. So we should try isolate the problem. Massimo On Feb 12, 12:12 am, Anthony abasta...@gmail.com wrote: On Saturday, February 12, 2011 1:03:48 AM UTC-5, Jonathan Lundell wrote: On Feb 10, 2011, at 6:19 AM, Massimo Di Pierro wrote: I offer $300 to whoever can identify within 3 weeks and without ambiguity the cause of this problem. The bounty applies even if the problem turns out to be not in web2py but in one of the Python modules, in Apache or in the mod_wsgi implementation. It does not apply if the problem is due to known bug that has already been fixed by the responsible party (for example if you are using an old un- patched python version like 2.5.0 or an old mod_wsgi version). psycopg2 2.0.7 is the version in use in this case. 2.0.8 had this in the change log: * psycopg/pqpath.c (_pq_fetch_tuples): Don't call Python APIs without holding the GIL. Suggestive, no? There's some evidence on the web that this may have led to Segmentation fault - premature end of script headers errors. 2.3.2 is current Though someone reported the same problem with MySQL:https://groups.google.com/d/msg/web2py/AcCDl5eJnds/AXNyKO6kmBwJ
Re: [web2py] Re: Apache, Wsgi problem
On Feb 11, 2011, at 10:12 PM, Anthony wrote: On Saturday, February 12, 2011 1:03:48 AM UTC-5, Jonathan Lundell wrote: On Feb 10, 2011, at 6:19 AM, Massimo Di Pierro wrote: I offer $300 to whoever can identify within 3 weeks and without ambiguity the cause of this problem. The bounty applies even if the problem turns out to be not in web2py but in one of the Python modules, in Apache or in the mod_wsgi implementation. It does not apply if the problem is due to known bug that has already been fixed by the responsible party (for example if you are using an old un- patched python version like 2.5.0 or an old mod_wsgi version). psycopg2 2.0.7 is the version in use in this case. 2.0.8 had this in the change log: * psycopg/pqpath.c (_pq_fetch_tuples): Don't call Python APIs without holding the GIL. Suggestive, no? There's some evidence on the web that this may have led to Segmentation fault - premature end of script headers errors. 2.3.2 is current Though someone reported the same problem with MySQL: https://groups.google.com/d/msg/web2py/AcCDl5eJnds/AXNyKO6kmBwJ Part of the problem is that this is a very general error. Lots of things can cause a segmentation fault of the wsgi request thread, which in turn leads to that message. Psycopg2 2.0.7 may well have a problem that causes it, and there may well be other ways for it to happen as well. It's important to eliminate the known problems (like this one) so that if another fault comes along, we know it's something else. Unfortunately we can't carry these adapters along in web2py, since they're not pure/portable Python.
Re: [web2py] Re: Apache, Wsgi problem
Greetings. I could create a web2py.app.error.w2p and detail the specifications of the environment, I will try to put him eye to the issue. I am a rookie .. but I would try Díaz Luis TSU Analisis de Sistemas Universidad de Carabobo Facultad de Odontologíahttp://www.odontologia.uc.edu.ve/index.php?option=com_contentview=articleid=102Itemid=85 2011/2/12 Jonathan Lundell jlund...@pobox.com On Feb 11, 2011, at 10:12 PM, Anthony wrote: On Saturday, February 12, 2011 1:03:48 AM UTC-5, Jonathan Lundell wrote: On Feb 10, 2011, at 6:19 AM, Massimo Di Pierro wrote: I offer $300 to whoever can identify within 3 weeks and without ambiguity the cause of this problem. The bounty applies even if the problem turns out to be not in web2py but in one of the Python modules, in Apache or in the mod_wsgi implementation. It does not apply if the problem is due to known bug that has already been fixed by the responsible party (for example if you are using an old un- patched python version like 2.5.0 or an old mod_wsgi version). psycopg2 2.0.7 is the version in use in this case. 2.0.8 had this in the change log: * psycopg/pqpath.c (_pq_fetch_tuples): Don't call Python APIs without holding the GIL. Suggestive, no? There's some evidence on the web that this may have led to Segmentation fault - premature end of script headers errors. 2.3.2 is current Though someone reported the same problem with MySQL: https://groups.google.com/d/msg/web2py/AcCDl5eJnds/AXNyKO6kmBwJ Part of the problem is that this is a very general error. Lots of things can cause a segmentation fault of the wsgi request thread, which in turn leads to that message. Psycopg2 2.0.7 may well have a problem that causes it, and there may well be other ways for it to happen as well. It's important to eliminate the known problems (like this one) so that if another fault comes along, we know it's something else. Unfortunately we can't carry these adapters along in web2py, since they're not pure/portable Python.
Re: [web2py] Re: Apache, Wsgi problem
FWIW I am on Ubuntu 10.04 with all patches applied and could not reproduce the above problem when I tried. The version of python-psycopg2 from the standard repository is 2.0.13. Not an iron-clad guarantee because something else could be at play but at least a data point. I also drove the application a lot harder from ab with 10x number of requests to try and reproduce but still nothing. The fact it doesn't fail when processes=5 and threads=1 agrees well with the GIL problem - without threads there is no need to hold the GIL.
Re: [web2py] Re: Apache, Wsgi problem
On Feb 12, 2011, at 8:33 AM, Luis Díaz wrote: I could create a web2py.app.error.w2p and detail the specifications of the environment, I will try to put him eye to the issue. I am a rookie .. but I would try Thanks, Luis.
Re: [web2py] Re: Apache, Wsgi problem
On Feb 12, 2011, at 8:46 AM, ron_m wrote: FWIW I am on Ubuntu 10.04 with all patches applied and could not reproduce the above problem when I tried. The version of python-psycopg2 from the standard repository is 2.0.13. Not an iron-clad guarantee because something else could be at play but at least a data point. I also drove the application a lot harder from ab with 10x number of requests to try and reproduce but still nothing. The fact it doesn't fail when processes=5 and threads=1 agrees well with the GIL problem - without threads there is no need to hold the GIL. Yes, it's a useful data point. The best test will be to have an environment that actually fails with psycopg2 2.0.7, and have the failure disappear with an updated version (and no other changes), and to get the error back again with 2.0.7.
Re: [web2py] Re: Apache, Wsgi problem
Translation error .. sorry. I could not .. might wonder if any of you? Díaz Luis TSU Analisis de Sistemas Universidad de Carabobo Facultad de Odontologíahttp://www.odontologia.uc.edu.ve/index.php?option=com_contentview=articleid=102Itemid=85 2011/2/12 Jonathan Lundell jlund...@pobox.com On Feb 12, 2011, at 8:46 AM, ron_m wrote: FWIW I am on Ubuntu 10.04 with all patches applied and could not reproduce the above problem when I tried. The version of python-psycopg2 from the standard repository is 2.0.13. Not an iron-clad guarantee because something else could be at play but at least a data point. I also drove the application a lot harder from ab with 10x number of requests to try and reproduce but still nothing. The fact it doesn't fail when processes=5 and threads=1 agrees well with the GIL problem - without threads there is no need to hold the GIL. Yes, it's a useful data point. The best test will be to have an environment that actually fails with psycopg2 2.0.7, and have the failure disappear with an updated version (and no other changes), and to get the error back again with 2.0.7.
Re: [web2py] Re: Apache, Wsgi problem
On Feb 10, 2011, at 6:19 AM, Massimo Di Pierro wrote: I offer $300 to whoever can identify within 3 weeks and without ambiguity the cause of this problem. The bounty applies even if the problem turns out to be not in web2py but in one of the Python modules, in Apache or in the mod_wsgi implementation. It does not apply if the problem is due to known bug that has already been fixed by the responsible party (for example if you are using an old un- patched python version like 2.5.0 or an old mod_wsgi version). psycopg2 2.0.7 is the version in use in this case. 2.0.8 had this in the change log: * psycopg/pqpath.c (_pq_fetch_tuples): Don't call Python APIs without holding the GIL. Suggestive, no? There's some evidence on the web that this may have led to Segmentation fault - premature end of script headers errors. 2.3.2 is current.
Re: [web2py] Re: Apache, Wsgi problem
On Saturday, February 12, 2011 1:03:48 AM UTC-5, Jonathan Lundell wrote: On Feb 10, 2011, at 6:19 AM, Massimo Di Pierro wrote: I offer $300 to whoever can identify within 3 weeks and without ambiguity the cause of this problem. The bounty applies even if the problem turns out to be not in web2py but in one of the Python modules, in Apache or in the mod_wsgi implementation. It does not apply if the problem is due to known bug that has already been fixed by the responsible party (for example if you are using an old un- patched python version like 2.5.0 or an old mod_wsgi version). psycopg2 2.0.7 is the version in use in this case. 2.0.8 had this in the change log: * psycopg/pqpath.c (_pq_fetch_tuples): Don't call Python APIs without holding the GIL. Suggestive, no? There's some evidence on the web that this may have led to Segmentation fault - premature end of script headers errors. 2.3.2 is current Though someone reported the same problem with MySQL: https://groups.google.com/d/msg/web2py/AcCDl5eJnds/AXNyKO6kmBwJ
[web2py] Re: Apache, Wsgi problem
Great!!
I also had threads=25 and changed this to threads=1 processes=5,
so it makes sense that I was encountering the same problem. It sounds
like something in web2py might not be thread-safe. The next time I
run a production test I will report if this fixes the problem.
On Feb 10, 2:38 pm, VP vtp2...@gmail.com wrote:
Alright people short answer: I think I figured this out (at least
with my configuration)
After testing various configurations, here's the result with: ab -kc
100 -t 20https://domain.com/imageblog/default/index/ (same
imageblog app, 100 connections, 20 seconds stress test).
Two things you will notice with this result.
1. ZERO failed request. No morewsgipremature script error
2. Complete requests is 1234, 61 requests/second on average.
Compare to prior configuration: 588 total requests, 29 requests/
sec on average. Not to mention 15 failed requests due towsgi
premature script errors!!!
This is insane!!!
So how did I configure this? here it is:
WSGIDaemonProcess web2py user=username group=username \
display-name=%{GROUP} processes=5 threads=1
The important option being 5 processes, 1 thread.
With this configuration, my real app also did not getwsgipremature
script errors anymore. And guess what... the requests/sec
triples
I am still curious about this. While my real app can possibly be not
thread-safe, but the imageblog app should be thread safe (the index
was simply a listing of images, i.e. read only). Why would there be
aproblemwith more than 1 thread?
Document Path: /imageblog/default/index
Document Length: 13083 bytes
Concurrency Level: 100
Time taken for tests: 20.008 seconds
Complete requests: 1234
Failed requests: 0
Write errors: 0
Keep-Alive requests: 1234
Total transferred: 16827432 bytes
HTML transferred: 16171262 bytes
Requests per second: 61.68 [#/sec] (mean)
Time per request: 1621.377 [ms] (mean)
Time per request: 16.214 [ms] (mean, across all concurrent
requests)
Transfer rate: 821.33 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 9.4 0 73
Processing: 82 481 890.5 317 5475
Waiting: 76 443 878.7 274 5393
Total: 82 483 894.7 317 5503
Percentage of the requests served within a certain time (ms)
50% 317
66% 342
75% 360
80% 372
90% 416
95% 489
98% 5351
99% 5397
100% 5503 (longest request)
[web2py] Re: Apache, Wsgi problem
This is a very important test. I believe web2py to be thread safe and
in fact we thread.lock every access to shared objects even if
operations are atomic, yet I will review it in detail.
Massimo
On Feb 10, 12:38 am, VP vtp2...@gmail.com wrote:
Alright people short answer: I think I figured this out (at least
with my configuration)
After testing various configurations, here's the result with: ab -kc
100 -t 20https://domain.com/imageblog/default/index/ (same
imageblog app, 100 connections, 20 seconds stress test).
Two things you will notice with this result.
1. ZERO failed request. No more wsgi premature script error
2. Complete requests is 1234, 61 requests/second on average.
Compare to prior configuration: 588 total requests, 29 requests/
sec on average. Not to mention 15 failed requests due to wsgi
premature script errors!!!
This is insane!!!
So how did I configure this? here it is:
WSGIDaemonProcess web2py user=username group=username \
display-name=%{GROUP} processes=5 threads=1
The important option being 5 processes, 1 thread.
With this configuration, my real app also did not get wsgi premature
script errors anymore. And guess what... the requests/sec
triples
I am still curious about this. While my real app can possibly be not
thread-safe, but the imageblog app should be thread safe (the index
was simply a listing of images, i.e. read only). Why would there be a
problem with more than 1 thread?
Document Path: /imageblog/default/index
Document Length: 13083 bytes
Concurrency Level: 100
Time taken for tests: 20.008 seconds
Complete requests: 1234
Failed requests: 0
Write errors: 0
Keep-Alive requests: 1234
Total transferred: 16827432 bytes
HTML transferred: 16171262 bytes
Requests per second: 61.68 [#/sec] (mean)
Time per request: 1621.377 [ms] (mean)
Time per request: 16.214 [ms] (mean, across all concurrent
requests)
Transfer rate: 821.33 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 9.4 0 73
Processing: 82 481 890.5 317 5475
Waiting: 76 443 878.7 274 5393
Total: 82 483 894.7 317 5503
Percentage of the requests served within a certain time (ms)
50% 317
66% 342
75% 360
80% 372
90% 416
95% 489
98% 5351
99% 5397
100% 5503 (longest request)
[web2py] Re: Apache, Wsgi problem
I offer $300 to whoever can identify within 3 weeks and without
ambiguity the cause of this problem. The bounty applies even if the
problem turns out to be not in web2py but in one of the Python
modules, in Apache or in the mod_wsgi implementation. It does not
apply if the problem is due to known bug that has already been fixed
by the responsible party (for example if you are using an old un-
patched python version like 2.5.0 or an old mod_wsgi version).
On Feb 10, 12:38 am, VP vtp2...@gmail.com wrote:
Alright people short answer: I think I figured this out (at least
with my configuration)
After testing various configurations, here's the result with: ab -kc
100 -t 20https://domain.com/imageblog/default/index/ (same
imageblog app, 100 connections, 20 seconds stress test).
Two things you will notice with this result.
1. ZERO failed request. No more wsgi premature script error
2. Complete requests is 1234, 61 requests/second on average.
Compare to prior configuration: 588 total requests, 29 requests/
sec on average. Not to mention 15 failed requests due to wsgi
premature script errors!!!
This is insane!!!
So how did I configure this? here it is:
WSGIDaemonProcess web2py user=username group=username \
display-name=%{GROUP} processes=5 threads=1
The important option being 5 processes, 1 thread.
With this configuration, my real app also did not get wsgi premature
script errors anymore. And guess what... the requests/sec
triples
I am still curious about this. While my real app can possibly be not
thread-safe, but the imageblog app should be thread safe (the index
was simply a listing of images, i.e. read only). Why would there be a
problem with more than 1 thread?
Document Path: /imageblog/default/index
Document Length: 13083 bytes
Concurrency Level: 100
Time taken for tests: 20.008 seconds
Complete requests: 1234
Failed requests: 0
Write errors: 0
Keep-Alive requests: 1234
Total transferred: 16827432 bytes
HTML transferred: 16171262 bytes
Requests per second: 61.68 [#/sec] (mean)
Time per request: 1621.377 [ms] (mean)
Time per request: 16.214 [ms] (mean, across all concurrent
requests)
Transfer rate: 821.33 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 9.4 0 73
Processing: 82 481 890.5 317 5475
Waiting: 76 443 878.7 274 5393
Total: 82 483 894.7 317 5503
Percentage of the requests served within a certain time (ms)
50% 317
66% 342
75% 360
80% 372
90% 416
95% 489
98% 5351
99% 5397
100% 5503 (longest request)
Re: [web2py] Re: Apache, Wsgi problem
Good work.
You mentioned that you're using Debian Lenny. What version of Python? 2.5.2?
And what version of psycopg2?
On Feb 9, 2011, at 10:38 PM, VP wrote:
Alright people short answer: I think I figured this out (at least
with my configuration)
After testing various configurations, here's the result with: ab -kc
100 -t 20 https://domain.com/imageblog/default/index/ (same
imageblog app, 100 connections, 20 seconds stress test).
Two things you will notice with this result.
1. ZERO failed request. No more wsgi premature script error
2. Complete requests is 1234, 61 requests/second on average.
Compare to prior configuration: 588 total requests, 29 requests/
sec on average. Not to mention 15 failed requests due to wsgi
premature script errors!!!
This is insane!!!
So how did I configure this? here it is:
WSGIDaemonProcess web2py user=username group=username \
display-name=%{GROUP} processes=5 threads=1
The important option being 5 processes, 1 thread.
With this configuration, my real app also did not get wsgi premature
script errors anymore. And guess what... the requests/sec
triples
I am still curious about this. While my real app can possibly be not
thread-safe, but the imageblog app should be thread safe (the index
was simply a listing of images, i.e. read only). Why would there be a
problem with more than 1 thread?
Document Path: /imageblog/default/index
Document Length:13083 bytes
Concurrency Level: 100
Time taken for tests: 20.008 seconds
Complete requests: 1234
Failed requests:0
Write errors: 0
Keep-Alive requests:1234
Total transferred: 16827432 bytes
HTML transferred: 16171262 bytes
Requests per second:61.68 [#/sec] (mean)
Time per request: 1621.377 [ms] (mean)
Time per request: 16.214 [ms] (mean, across all concurrent
requests)
Transfer rate: 821.33 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect:01 9.4 0 73
Processing:82 481 890.53175475
Waiting: 76 443 878.72745393
Total: 82 483 894.73175503
Percentage of the requests served within a certain time (ms)
50%317
66%342
75%360
80%372
90%416
95%489
98% 5351
99% 5397
100% 5503 (longest request)
[web2py] Re: Apache, Wsgi problem
The modules I used are all standard via installation with apt-get.
Python is 2.5.2, and psycopg2 is 2.0.7
On Feb 10, 9:58 am, Jonathan Lundell jlund...@pobox.com wrote:
Good work.
You mentioned that you're using Debian Lenny. What version of Python? 2.5.2?
And what version of psycopg2?
On Feb 9, 2011, at 10:38 PM, VP wrote:
Alright people short answer: I think I figured this out (at least
with my configuration)
After testing various configurations, here's the result with: ab -kc
100 -t 20https://domain.com/imageblog/default/index/ (same
imageblog app, 100 connections, 20 seconds stress test).
Two things you will notice with this result.
1. ZERO failed request. No more wsgi premature script error
2. Complete requests is 1234, 61 requests/second on average.
Compare to prior configuration: 588 total requests, 29 requests/
sec on average. Not to mention 15 failed requests due to wsgi
premature script errors!!!
This is insane!!!
So how did I configure this? here it is:
WSGIDaemonProcess web2py user=username group=username \
display-name=%{GROUP} processes=5 threads=1
The important option being 5 processes, 1 thread.
With this configuration, my real app also did not get wsgi premature
script errors anymore. And guess what... the requests/sec
triples
I am still curious about this. While my real app can possibly be not
thread-safe, but the imageblog app should be thread safe (the index
was simply a listing of images, i.e. read only). Why would there be a
problem with more than 1 thread?
Document Path: /imageblog/default/index
Document Length: 13083 bytes
Concurrency Level: 100
Time taken for tests: 20.008 seconds
Complete requests: 1234
Failed requests: 0
Write errors: 0
Keep-Alive requests: 1234
Total transferred: 16827432 bytes
HTML transferred: 16171262 bytes
Requests per second: 61.68 [#/sec] (mean)
Time per request: 1621.377 [ms] (mean)
Time per request: 16.214 [ms] (mean, across all concurrent
requests)
Transfer rate: 821.33 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 9.4 0 73
Processing: 82 481 890.5 317 5475
Waiting: 76 443 878.7 274 5393
Total: 82 483 894.7 317 5503
Percentage of the requests served within a certain time (ms)
50% 317
66% 342
75% 360
80% 372
90% 416
95% 489
98% 5351
99% 5397
100% 5503 (longest request)
Re: [web2py] Re: Apache, Wsgi problem
On Feb 10, 2011, at 11:18 AM, VP vtp2...@gmail.com wrote:
The modules I used are all standard via installation with apt-get.
Python is 2.5.2, and psycopg2 is 2.0.7
Upgrade psycopg2 to at least 2.0.8 and see if that doesn't fix the fault.
On Feb 10, 9:58 am, Jonathan Lundell jlund...@pobox.com wrote:
Good work.
You mentioned that you're using Debian Lenny. What version of Python? 2.5.2?
And what version of psycopg2?
On Feb 9, 2011, at 10:38 PM, VP wrote:
Alright people short answer: I think I figured this out (at least
with my configuration)
After testing various configurations, here's the result with: ab -kc
100 -t 20https://domain.com/imageblog/default/index/ (same
imageblog app, 100 connections, 20 seconds stress test).
Two things you will notice with this result.
1. ZERO failed request. No more wsgi premature script error
2. Complete requests is 1234, 61 requests/second on average.
Compare to prior configuration: 588 total requests, 29 requests/
sec on average. Not to mention 15 failed requests due to wsgi
premature script errors!!!
This is insane!!!
So how did I configure this? here it is:
WSGIDaemonProcess web2py user=username group=username \
display-name=%{GROUP} processes=5 threads=1
The important option being 5 processes, 1 thread.
With this configuration, my real app also did not get wsgi premature
script errors anymore. And guess what... the requests/sec
triples
I am still curious about this. While my real app can possibly be not
thread-safe, but the imageblog app should be thread safe (the index
was simply a listing of images, i.e. read only). Why would there be a
problem with more than 1 thread?
Document Path: /imageblog/default/index
Document Length:13083 bytes
Concurrency Level: 100
Time taken for tests: 20.008 seconds
Complete requests: 1234
Failed requests:0
Write errors: 0
Keep-Alive requests:1234
Total transferred: 16827432 bytes
HTML transferred: 16171262 bytes
Requests per second:61.68 [#/sec] (mean)
Time per request: 1621.377 [ms] (mean)
Time per request: 16.214 [ms] (mean, across all concurrent
requests)
Transfer rate: 821.33 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect:01 9.4 0 73
Processing:82 481 890.53175475
Waiting: 76 443 878.72745393
Total: 82 483 894.73175503
Percentage of the requests served within a certain time (ms)
50%317
66%342
75%360
80%372
90%416
95%489
98% 5351
99% 5397
100% 5503 (longest request)
[web2py] Re: Apache, Wsgi problem
Alright people short answer: I think I figured this out (at least
with my configuration)
After testing various configurations, here's the result with: ab -kc
100 -t 20 https://domain.com/imageblog/default/index/ (same
imageblog app, 100 connections, 20 seconds stress test).
Two things you will notice with this result.
1. ZERO failed request. No more wsgi premature script error
2. Complete requests is 1234, 61 requests/second on average.
Compare to prior configuration: 588 total requests, 29 requests/
sec on average. Not to mention 15 failed requests due to wsgi
premature script errors!!!
This is insane!!!
So how did I configure this? here it is:
WSGIDaemonProcess web2py user=username group=username \
display-name=%{GROUP} processes=5 threads=1
The important option being 5 processes, 1 thread.
With this configuration, my real app also did not get wsgi premature
script errors anymore. And guess what... the requests/sec
triples
I am still curious about this. While my real app can possibly be not
thread-safe, but the imageblog app should be thread safe (the index
was simply a listing of images, i.e. read only). Why would there be a
problem with more than 1 thread?
Document Path: /imageblog/default/index
Document Length:13083 bytes
Concurrency Level: 100
Time taken for tests: 20.008 seconds
Complete requests: 1234
Failed requests:0
Write errors: 0
Keep-Alive requests:1234
Total transferred: 16827432 bytes
HTML transferred: 16171262 bytes
Requests per second:61.68 [#/sec] (mean)
Time per request: 1621.377 [ms] (mean)
Time per request: 16.214 [ms] (mean, across all concurrent
requests)
Transfer rate: 821.33 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect:01 9.4 0 73
Processing:82 481 890.53175475
Waiting: 76 443 878.72745393
Total: 82 483 894.73175503
Percentage of the requests served within a certain time (ms)
50%317
66%342
75%360
80%372
90%416
95%489
98% 5351
99% 5397
100% 5503 (longest request)
[web2py] Re: Apache, Wsgi problem
Massimo: Have you gotten a chance to look into this issue (related to scalability)? Thanks.
[web2py] Re: Apache, Wsgi problem
Not yet. I do not have the problem on my production server. I need to setup a new VM to test this. Hopefully tomorrow. Massmimo On Jan 22, 1:13 pm, VP vtp2...@gmail.com wrote: Massimo: Have you gotten a chance to look into this issue (related to scalability)? Thanks.
[web2py] Re: Apache, Wsgi problem
UPDATE: I am able to reproduce this Premature end of script headers:
wsgihandler.py error with a slightly modified app based on the image
blog in the web2py book. I modified it so that images are shown in
the index page, instead of having just the titles listed.
Testing using: ab -kc 100 -t 20 https://domain.com/imageblog/default/index/
If I am simply doing this, there's no error. BUT when I was running
this test, while simultaneously browsing the website, clicking on
images, etc., then the error appeared, consistently.
Monitoring available RAM while running this test showed system RAM
peeked at about 500MB, still a few hundreds MB available. This is on
the same server as the other app. VPS. 768MB RAM. 1GB RAM
Burstable. About 129 apache2 processes running and 11,000 files
openning.
sudo ps aux | grep apache2 | wc
1291675 11847
sudo lsof | wc
10982 106870 1313126
=
Here's a typical Apache Bench result:
Server Software:Apache/2.2.9
Server Port:443
SSL/TLS Protocol: TLSv1/SSLv3,DHE-RSA-AES256-SHA,1024,256
Document Path: /imageblog/default/index/
Document Length:13130 bytes
Concurrency Level: 100
Time taken for tests: 20.004 seconds
Complete requests: 588
Failed requests:15
(Connect: 0, Receive: 0, Length: 15, Exceptions: 0)
Write errors: 0
Non-2xx responses: 15
Keep-Alive requests:573
Total transferred: 7875930 bytes
HTML transferred: 7564607 bytes
Requests per second:29.39 [#/sec] (mean)
Time per request: 3401.991 [ms] (mean)
Time per request: 34.020 [ms] (mean, across all concurrent
requests)
Transfer rate: 384.50 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect:0 912 2146.5 07265
Processing:86 2234 1303.3 27814427
Waiting: 85 2136 1281.0 27594316
Total: 86 3146 2410.6 3017 10478
Percentage of the requests served within a certain time (ms)
50% 3017
66% 3222
75% 3939
80% 4029
90% 7089
95% 9409
98% 10098
99% 10223
100% 10478 (longest request)
=
Relevant excerpt of db.py:
db = DAL('postgres://username:password@localhost:5432/imageblog',
pool_size=20)
=
Model:
db.define_table('image',
Field('title'),
Field('file', 'upload'))
db.define_table('comment',
Field('image_id', db.image),
Field('author'),
Field('email'),
Field('body', 'text'))
db.image.title.requires = IS_NOT_IN_DB(db, db.image.title)
db.comment.image_id.requires = IS_IN_DB(db, db.image.id, '%(title)s')
db.comment.author.requires = IS_NOT_EMPTY()
db.comment.email.requires = IS_EMAIL()
db.comment.body.requires = IS_NOT_EMPTY()
=
Controller:
def index():
images = db().select(db.image.ALL, orderby=db.image.title)
return dict(images=images)
@auth.requires_login()
def add():
form = crud.create(db.image, next = URL('index'))
return dict(form=form)
def show():
image = db(db.image.id==request.args(0)).select().first()
form = SQLFORM(db.comment)
form.vars.image_id = image.id
if form.accepts(request.vars, session):
response.flash = 'your comment is posted'
comments = db(db.comment.image_id==image.id).select()
return dict(image=image, comments=comments, form=form)
def user():
exposes:
http:///[app]/default/user/login
http:///[app]/default/user/logout
http:///[app]/default/user/register
http:///[app]/default/user/profile
http:///[app]/default/user/retrieve_password
http:///[app]/default/user/change_password
use @auth.requires_login()
@auth.requires_membership('group name')
@auth.requires_permission('read','table name',record_id)
to decorate functions that need access control
return dict(form=auth())
def download():
allows downloading of uploaded files
http:///[app]/default/download/[filename]
return response.download(request,db)
def call():
exposes services. for example:
http:///[app]/default/call/jsonrpc
decorate with @services.jsonrpc the functions to expose
supports xml, json, xmlrpc, jsonrpc, amfrpc, rss, csv
session.forget()
return service()
=
View (index.html)
{{extend 'layout.html'}}
h1Current Images/h1
{{for image in images:}}
div class=entry style=padding:1em; border-bottom:1px dashed
#999;
div class=entrytitle style=font-size:150%;{{=A(image.title,
_href=URL(show, args=image.id))}}/div
div class=entryimageimg src={{=URL('download',
args=image.file)}}/div
/div
{{pass}}
=
Apache's site-enabled/000-default:
VirtualHost *:80
WSGIDaemonProcess web2py
[web2py] Re: Apache, Wsgi problem
What is curious is that RAM is still available, with this error. Monitoring CPU (using top) shows CPU % is about 70% for apache2 (70% is probably capped by my VPS host, given to each VPS slice). And this occurs for a very simple app. I hope Massimo or someone else can reproduce this error with this app. Note that while testing with ab, you might have to browse the site, clicking on images, etc. Otherwise, ab just it the same domain name repeatedly.
[web2py] Re: Apache, Wsgi problem
I forgot to mention that in this imageblog app, I only have 10 images, and no comment. I.e. a very small database. Here are the 10 images: http://imgur.com/u6gwulkJNwjDsjQxHIIN99AqEII5Pvh1gxagEcoJru2ucY3583C
[web2py] Re: Apache, Wsgi problem
Thanks this is very helpful and I will try reproduce it.
Are you running the trunk version or which other version?
Massimo
On Jan 19, 12:45 pm, VP vtp2...@gmail.com wrote:
UPDATE: I am able to reproduce this Premature end of script headers:
wsgihandler.py error with a slightly modified app based on the image
blog in the web2py book. I modified it so that images are shown in
the index page, instead of having just the titles listed.
Testing using: ab -kc 100 -t 20https://domain.com/imageblog/default/index/
If I am simply doing this, there's no error. BUT when I was running
this test, while simultaneously browsing the website, clicking on
images, etc., then the error appeared, consistently.
Monitoring available RAM while running this test showed system RAM
peeked at about 500MB, still a few hundreds MB available. This is on
the same server as the other app. VPS. 768MB RAM. 1GB RAM
Burstable. About 129 apache2 processes running and 11,000 files
openning.
sudo ps aux | grep apache2 | wc
129 1675 11847
sudo lsof | wc
10982 106870 1313126
=
Here's a typical Apache Bench result:
Server Software: Apache/2.2.9
Server Port: 443
SSL/TLS Protocol: TLSv1/SSLv3,DHE-RSA-AES256-SHA,1024,256
Document Path: /imageblog/default/index/
Document Length: 13130 bytes
Concurrency Level: 100
Time taken for tests: 20.004 seconds
Complete requests: 588
Failed requests: 15
(Connect: 0, Receive: 0, Length: 15, Exceptions: 0)
Write errors: 0
Non-2xx responses: 15
Keep-Alive requests: 573
Total transferred: 7875930 bytes
HTML transferred: 7564607 bytes
Requests per second: 29.39 [#/sec] (mean)
Time per request: 3401.991 [ms] (mean)
Time per request: 34.020 [ms] (mean, across all concurrent
requests)
Transfer rate: 384.50 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 912 2146.5 0 7265
Processing: 86 2234 1303.3 2781 4427
Waiting: 85 2136 1281.0 2759 4316
Total: 86 3146 2410.6 3017 10478
Percentage of the requests served within a certain time (ms)
50% 3017
66% 3222
75% 3939
80% 4029
90% 7089
95% 9409
98% 10098
99% 10223
100% 10478 (longest request)
=
Relevant excerpt of db.py:
db = DAL('postgres://username:password@localhost:5432/imageblog',
pool_size=20)
=
Model:
db.define_table('image',
Field('title'),
Field('file', 'upload'))
db.define_table('comment',
Field('image_id', db.image),
Field('author'),
Field('email'),
Field('body', 'text'))
db.image.title.requires = IS_NOT_IN_DB(db, db.image.title)
db.comment.image_id.requires = IS_IN_DB(db, db.image.id, '%(title)s')
db.comment.author.requires = IS_NOT_EMPTY()
db.comment.email.requires = IS_EMAIL()
db.comment.body.requires = IS_NOT_EMPTY()
=
Controller:
def index():
images = db().select(db.image.ALL, orderby=db.image.title)
return dict(images=images)
@auth.requires_login()
def add():
form = crud.create(db.image, next = URL('index'))
return dict(form=form)
def show():
image = db(db.image.id==request.args(0)).select().first()
form = SQLFORM(db.comment)
form.vars.image_id = image.id
if form.accepts(request.vars, session):
response.flash = 'your comment is posted'
comments = db(db.comment.image_id==image.id).select()
return dict(image=image, comments=comments, form=form)
def user():
exposes:
http:///[app]/default/user/login
http:///[app]/default/user/logout
http:///[app]/default/user/register
http:///[app]/default/user/profile
http:///[app]/default/user/retrieve_password
http:///[app]/default/user/change_password
use @auth.requires_login()
@auth.requires_membership('group name')
@auth.requires_permission('read','table name',record_id)
to decorate functions that need access control
return dict(form=auth())
def download():
allows downloading of uploaded files
http:///[app]/default/download/[filename]
return response.download(request,db)
def call():
exposes services. for example:
http:///[app]/default/call/jsonrpc
decorate with @services.jsonrpc the functions to expose
supports xml, json, xmlrpc, jsonrpc, amfrpc, rss, csv
session.forget()
return service()
=
View (index.html)
{{extend 'layout.html'}}
h1Current Images/h1
{{for image in images:}}
div class=entry style=padding:1em; border-bottom:1px dashed
#999;
div class=entrytitle
[web2py] Re: Apache, Wsgi problem
Yes, this echos my experiences exactly! Using apache ab benchmark alone would NOT trigger the error. I had plenty of RAM available. Seems to be a concurrency bug. On Jan 19, 10:53 am, VP vtp2...@gmail.com wrote: What is curious is that RAM is still available, with this error. Monitoring CPU (using top) shows CPU % is about 70% for apache2 (70% is probably capped by my VPS host, given to each VPS slice). And this occurs for a very simple app. I hope Massimo or someone else can reproduce this error with this app. Note that while testing with ab, you might have to browse the site, clicking on images, etc. Otherwise, ab just it the same domain name repeatedly.
[web2py] Re: Apache, Wsgi problem
I am running the official version 1.91.6.
On Jan 19, 1:24 pm, Massimo Di Pierro massimo.dipie...@gmail.com
wrote:
Thanks this is very helpful and I will try reproduce it.
Are you running the trunk version or which other version?
Massimo
On Jan 19, 12:45 pm, VP vtp2...@gmail.com wrote:
UPDATE: I am able to reproduce this Premature end of script headers:
wsgihandler.py error with a slightly modified app based on the image
blog in the web2py book. I modified it so that images are shown in
the index page, instead of having just the titles listed.
Testing using: ab -kc 100 -t 20https://domain.com/imageblog/default/index/
If I am simply doing this, there's no error. BUT when I was running
this test, while simultaneously browsing the website, clicking on
images, etc., then the error appeared, consistently.
Monitoring available RAM while running this test showed system RAM
peeked at about 500MB, still a few hundreds MB available. This is on
the same server as the other app. VPS. 768MB RAM. 1GB RAM
Burstable. About 129 apache2 processes running and 11,000 files
openning.
sudo ps aux | grep apache2 | wc
129 1675 11847
sudo lsof | wc
10982 106870 1313126
=
Here's a typical Apache Bench result:
Server Software: Apache/2.2.9
Server Port: 443
SSL/TLS Protocol: TLSv1/SSLv3,DHE-RSA-AES256-SHA,1024,256
Document Path: /imageblog/default/index/
Document Length: 13130 bytes
Concurrency Level: 100
Time taken for tests: 20.004 seconds
Complete requests: 588
Failed requests: 15
(Connect: 0, Receive: 0, Length: 15, Exceptions: 0)
Write errors: 0
Non-2xx responses: 15
Keep-Alive requests: 573
Total transferred: 7875930 bytes
HTML transferred: 7564607 bytes
Requests per second: 29.39 [#/sec] (mean)
Time per request: 3401.991 [ms] (mean)
Time per request: 34.020 [ms] (mean, across all concurrent
requests)
Transfer rate: 384.50 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 912 2146.5 0 7265
Processing: 86 2234 1303.3 2781 4427
Waiting: 85 2136 1281.0 2759 4316
Total: 86 3146 2410.6 3017 10478
Percentage of the requests served within a certain time (ms)
50% 3017
66% 3222
75% 3939
80% 4029
90% 7089
95% 9409
98% 10098
99% 10223
100% 10478 (longest request)
=
Relevant excerpt of db.py:
db = DAL('postgres://username:password@localhost:5432/imageblog',
pool_size=20)
=
Model:
db.define_table('image',
Field('title'),
Field('file', 'upload'))
db.define_table('comment',
Field('image_id', db.image),
Field('author'),
Field('email'),
Field('body', 'text'))
db.image.title.requires = IS_NOT_IN_DB(db, db.image.title)
db.comment.image_id.requires = IS_IN_DB(db, db.image.id, '%(title)s')
db.comment.author.requires = IS_NOT_EMPTY()
db.comment.email.requires = IS_EMAIL()
db.comment.body.requires = IS_NOT_EMPTY()
=
Controller:
def index():
images = db().select(db.image.ALL, orderby=db.image.title)
return dict(images=images)
@auth.requires_login()
def add():
form = crud.create(db.image, next = URL('index'))
return dict(form=form)
def show():
image = db(db.image.id==request.args(0)).select().first()
form = SQLFORM(db.comment)
form.vars.image_id = image.id
if form.accepts(request.vars, session):
response.flash = 'your comment is posted'
comments = db(db.comment.image_id==image.id).select()
return dict(image=image, comments=comments, form=form)
def user():
exposes:
http:///[app]/default/user/login
http:///[app]/default/user/logout
http:///[app]/default/user/register
http:///[app]/default/user/profile
http:///[app]/default/user/retrieve_password
http:///[app]/default/user/change_password
use @auth.requires_login()
@auth.requires_membership('group name')
@auth.requires_permission('read','table name',record_id)
to decorate functions that need access control
return dict(form=auth())
def download():
allows downloading of uploaded files
http:///[app]/default/download/[filename]
return response.download(request,db)
def call():
exposes services. for example:
http:///[app]/default/call/jsonrpc
decorate with @services.jsonrpc the functions to expose
supports xml, json, xmlrpc, jsonrpc, amfrpc, rss, csv
session.forget()
return
[web2py] Re: Apache, Wsgi problem
It seems the links to the images don't work anymore. You can just picked any 10 images, but again the ones I used are: http://i.imgur.com/u6gwu.jpg http://i.imgur.com/kJNwj.jpg http://i.imgur.com/DsjQx.jpg http://i.imgur.com/HIIN9.jpg http://i.imgur.com/9AqEI.jpg http://i.imgur.com/I5Pvh.jpg http://i.imgur.com/1gxag.jpg http://i.imgur.com/EcoJr.jpg http://i.imgur.com/u2ucY.jpg http://i.imgur.com/3583C.jpg
[web2py] Re: Apache, Wsgi problem
Are you using connection pooling? Can I see the line db = DAL(...)? On Jan 16, 2:24 pm, VP vtp2...@gmail.com wrote: Kenneth (see post above) had the same problem with MySQL. On Jan 16, 3:55 am, ron_m ron.mco...@gmail.com wrote: Is there any chance to try another server based database such as MySQL. Since SQLite did not exhibit a problem that is a valid data point but it is an embedded database so called directly by the web2py server code instead of using a driver. There is a possibility there is a problem with the driver on PostgreSQL if the connections are being reused in a pool. This is very difficult to find without some further logging. I gather the page request you see failing as a user simply never completes. I saw this once while testing the application but the loading is only myself running tests. I use PostgreSQL in this app but I also have a version that runs on MySQL which has only a connection string difference plus the content of the databases directory. Maybe I can try this tomorrow, it is almost 2 am here so I don't want to start something new right now. I just need to learn how to use ab which I take it isn't going to be much to figure out.
[web2py] Re: Apache, Wsgi problem
Here it is with private information replaced with generic information.
if request.env.web2py_runtime_gae:# if running on Google
App Engine
db = DAL('gae') # connect to Google
BigTable
session.connect(request, response, db = db) # and store sessions
and tickets there
else: # else use a normal
relational database
db = DAL('postgres://name:password@localhost:5432/db')
#db = DAL('sqlite://storage.sqlite')
On Jan 17, 1:19 pm, Massimo Di Pierro massimo.dipie...@gmail.com
wrote:
Are you using connection pooling? Can I see the line db = DAL(...)?
On Jan 16, 2:24 pm, VP vtp2...@gmail.com wrote:
Kenneth (see post above) had the same problem with MySQL.
On Jan 16, 3:55 am, ron_m ron.mco...@gmail.com wrote:
Is there any chance to try another server based database such as MySQL.
Since SQLite did not exhibit a problem that is a valid data point but it
is
an embedded database so called directly by the web2py server code instead
of
using a driver. There is a possibility there is a problem with the driver
on
PostgreSQL if the connections are being reused in a pool. This is very
difficult to find without some further logging. I gather the page request
you see failing as a user simply never completes. I saw this once while
testing the application but the loading is only myself running tests. I
use
PostgreSQL in this app but I also have a version that runs on MySQL which
has only a connection string difference plus the content of the databases
directory. Maybe I can try this tomorrow, it is almost 2 am here so I
don't
want to start something new right now. I just need to learn how to use ab
which I take it isn't going to be much to figure out.
[web2py] Re: Apache, Wsgi problem
You should really have
db = DAL('postgres://name:password@localhost:5432/db',pool_size=20)
The reason is that client-server databases may set a max to number of
open connections and it takes time to perform the 3-way handshake to
establish a new connection at every http request. Without pooling you
may hit the max sooner than you should.
This will also make your app faster.
Massimo
On Jan 17, 1:39 pm, VP vtp2...@gmail.com wrote:
Here it is with private information replaced with generic information.
if request.env.web2py_runtime_gae: # if running on Google
App Engine
db = DAL('gae') # connect to Google
BigTable
session.connect(request, response, db = db) # and store sessions
and tickets there
else: # else use a normal
relational database
db = DAL('postgres://name:password@localhost:5432/db')
# db = DAL('sqlite://storage.sqlite')
On Jan 17, 1:19 pm, Massimo Di Pierro massimo.dipie...@gmail.com
wrote:
Are you using connection pooling? Can I see the line db = DAL(...)?
On Jan 16, 2:24 pm, VP vtp2...@gmail.com wrote:
Kenneth (see post above) had the same problem with MySQL.
On Jan 16, 3:55 am, ron_m ron.mco...@gmail.com wrote:
Is there any chance to try another server based database such as MySQL.
Since SQLite did not exhibit a problem that is a valid data point but
it is
an embedded database so called directly by the web2py server code
instead of
using a driver. There is a possibility there is a problem with the
driver on
PostgreSQL if the connections are being reused in a pool. This is very
difficult to find without some further logging. I gather the page
request
you see failing as a user simply never completes. I saw this once while
testing the application but the loading is only myself running tests. I
use
PostgreSQL in this app but I also have a version that runs on MySQL
which
has only a connection string difference plus the content of the
databases
directory. Maybe I can try this tomorrow, it is almost 2 am here so I
don't
want to start something new right now. I just need to learn how to use
ab
which I take it isn't going to be much to figure out.
[web2py] Re: Apache, Wsgi problem
Thanks. I fixed this. The performance seems to be a little better
(requests/second), but the error is still there.
On Jan 17, 2:20 pm, Massimo Di Pierro massimo.dipie...@gmail.com
wrote:
You should really have
db = DAL('postgres://name:password@localhost:5432/db',pool_size=20)
The reason is that client-server databases may set a max to number of
open connections and it takes time to perform the 3-way handshake to
establish a new connection at every http request. Without pooling you
may hit the max sooner than you should.
This will also make your app faster.
Massimo
On Jan 17, 1:39 pm, VP vtp2...@gmail.com wrote:
Here it is with private information replaced with generic information.
if request.env.web2py_runtime_gae: # if running on Google
App Engine
db = DAL('gae') # connect to Google
BigTable
session.connect(request, response, db = db) # and store sessions
and tickets there
else: # else use a normal
relational database
db = DAL('postgres://name:password@localhost:5432/db')
# db = DAL('sqlite://storage.sqlite')
On Jan 17, 1:19 pm, Massimo Di Pierro massimo.dipie...@gmail.com
wrote:
Are you using connection pooling? Can I see the line db = DAL(...)?
On Jan 16, 2:24 pm, VP vtp2...@gmail.com wrote:
Kenneth (see post above) had the same problem with MySQL.
On Jan 16, 3:55 am, ron_m ron.mco...@gmail.com wrote:
Is there any chance to try another server based database such as
MySQL.
Since SQLite did not exhibit a problem that is a valid data point but
it is
an embedded database so called directly by the web2py server code
instead of
using a driver. There is a possibility there is a problem with the
driver on
PostgreSQL if the connections are being reused in a pool. This is very
difficult to find without some further logging. I gather the page
request
you see failing as a user simply never completes. I saw this once
while
testing the application but the loading is only myself running tests.
I use
PostgreSQL in this app but I also have a version that runs on MySQL
which
has only a connection string difference plus the content of the
databases
directory. Maybe I can try this tomorrow, it is almost 2 am here so I
don't
want to start something new right now. I just need to learn how to
use ab
which I take it isn't going to be much to figure out.
[web2py] Re: Apache, Wsgi problem
I have pool_size=100, and get the error.
On Jan 17, 12:20 pm, Massimo Di Pierro massimo.dipie...@gmail.com
wrote:
You should really have
db = DAL('postgres://name:password@localhost:5432/db',pool_size=20)
The reason is that client-server databases may set a max to number of
open connections and it takes time to perform the 3-way handshake to
establish a new connection at every http request. Without pooling you
may hit the max sooner than you should.
This will also make your app faster.
Massimo
On Jan 17, 1:39 pm, VP vtp2...@gmail.com wrote:
Here it is with private information replaced with generic information.
if request.env.web2py_runtime_gae: # if running on Google
App Engine
db = DAL('gae') # connect to Google
BigTable
session.connect(request, response, db = db) # and store sessions
and tickets there
else: # else use a normal
relational database
db = DAL('postgres://name:password@localhost:5432/db')
# db = DAL('sqlite://storage.sqlite')
On Jan 17, 1:19 pm, Massimo Di Pierro massimo.dipie...@gmail.com
wrote:
Are you using connection pooling? Can I see the line db = DAL(...)?
On Jan 16, 2:24 pm, VP vtp2...@gmail.com wrote:
Kenneth (see post above) had the same problem with MySQL.
On Jan 16, 3:55 am, ron_m ron.mco...@gmail.com wrote:
Is there any chance to try another server based database such as
MySQL.
Since SQLite did not exhibit a problem that is a valid data point but
it is
an embedded database so called directly by the web2py server code
instead of
using a driver. There is a possibility there is a problem with the
driver on
PostgreSQL if the connections are being reused in a pool. This is very
difficult to find without some further logging. I gather the page
request
you see failing as a user simply never completes. I saw this once
while
testing the application but the loading is only myself running tests.
I use
PostgreSQL in this app but I also have a version that runs on MySQL
which
has only a connection string difference plus the content of the
databases
directory. Maybe I can try this tomorrow, it is almost 2 am here so I
don't
want to start something new right now. I just need to learn how to
use ab
which I take it isn't going to be much to figure out.
[web2py] Re: Apache, Wsgi problem
Can you share any more info you have about the errors?
Does it happen with every action? Is there one in particular? What
does it do? Do you use session? Do you use cache? Can you show us any
code? Can you build a minimalist app to reproduce it? Are there a
minimum number of steps to reproduce it?
Massimo
On Jan 17, 5:51 pm, Michael Toomim too...@gmail.com wrote:
I have pool_size=100, and get the error.
On Jan 17, 12:20 pm, Massimo Di Pierro massimo.dipie...@gmail.com
wrote:
You should really have
db = DAL('postgres://name:password@localhost:5432/db',pool_size=20)
The reason is that client-server databases may set a max to number of
open connections and it takes time to perform the 3-way handshake to
establish a new connection at every http request. Without pooling you
may hit the max sooner than you should.
This will also make your app faster.
Massimo
On Jan 17, 1:39 pm, VP vtp2...@gmail.com wrote:
Here it is with private information replaced with generic information.
if request.env.web2py_runtime_gae: # if running on Google
App Engine
db = DAL('gae') # connect to Google
BigTable
session.connect(request, response, db = db) # and store sessions
and tickets there
else: # else use a normal
relational database
db = DAL('postgres://name:password@localhost:5432/db')
# db = DAL('sqlite://storage.sqlite')
On Jan 17, 1:19 pm, Massimo Di Pierro massimo.dipie...@gmail.com
wrote:
Are you using connection pooling? Can I see the line db = DAL(...)?
On Jan 16, 2:24 pm, VP vtp2...@gmail.com wrote:
Kenneth (see post above) had the same problem with MySQL.
On Jan 16, 3:55 am, ron_m ron.mco...@gmail.com wrote:
Is there any chance to try another server based database such as
MySQL.
Since SQLite did not exhibit a problem that is a valid data point
but it is
an embedded database so called directly by the web2py server code
instead of
using a driver. There is a possibility there is a problem with the
driver on
PostgreSQL if the connections are being reused in a pool. This is
very
difficult to find without some further logging. I gather the page
request
you see failing as a user simply never completes. I saw this once
while
testing the application but the loading is only myself running
tests. I use
PostgreSQL in this app but I also have a version that runs on MySQL
which
has only a connection string difference plus the content of the
databases
directory. Maybe I can try this tomorrow, it is almost 2 am here so
I don't
want to start something new right now. I just need to learn how to
use ab
which I take it isn't going to be much to figure out.
[web2py] Re: Apache, Wsgi problem
The problem for me is that this occurs on a webapp used by mechanical
turk, and it fails when I have hundreds of mechanical turkers using my
app... which only happens when I pay them hundreds of dollars. So
it's hard to reproduce right now without hundreds of dollars.
I am excited to try using VP's ab benchmark to try to reproduce the
error more cheaply. When I can do that, and get a minimal system
exhibiting the error, I will let you know.
I'm also currently using an older version of web2py. One pre-DAL.
On Jan 17, 12:20 pm, Massimo Di Pierro massimo.dipie...@gmail.com
wrote:
You should really have
db = DAL('postgres://name:password@localhost:5432/db',pool_size=20)
The reason is that client-server databases may set a max to number of
open connections and it takes time to perform the 3-way handshake to
establish a new connection at every http request. Without pooling you
may hit the max sooner than you should.
This will also make your app faster.
Massimo
On Jan 17, 1:39 pm, VP vtp2...@gmail.com wrote:
Here it is with private information replaced with generic information.
if request.env.web2py_runtime_gae: # if running on Google
App Engine
db = DAL('gae') # connect to Google
BigTable
session.connect(request, response, db = db) # and store sessions
and tickets there
else: # else use a normal
relational database
db = DAL('postgres://name:password@localhost:5432/db')
# db = DAL('sqlite://storage.sqlite')
On Jan 17, 1:19 pm, Massimo Di Pierro massimo.dipie...@gmail.com
wrote:
Are you using connection pooling? Can I see the line db = DAL(...)?
On Jan 16, 2:24 pm, VP vtp2...@gmail.com wrote:
Kenneth (see post above) had the same problem with MySQL.
On Jan 16, 3:55 am, ron_m ron.mco...@gmail.com wrote:
Is there any chance to try another server based database such as
MySQL.
Since SQLite did not exhibit a problem that is a valid data point but
it is
an embedded database so called directly by the web2py server code
instead of
using a driver. There is a possibility there is a problem with the
driver on
PostgreSQL if the connections are being reused in a pool. This is very
difficult to find without some further logging. I gather the page
request
you see failing as a user simply never completes. I saw this once
while
testing the application but the loading is only myself running tests.
I use
PostgreSQL in this app but I also have a version that runs on MySQL
which
has only a connection string difference plus the content of the
databases
directory. Maybe I can try this tomorrow, it is almost 2 am here so I
don't
want to start something new right now. I just need to learn how to
use ab
which I take it isn't going to be much to figure out.
[web2py] Re: Apache, Wsgi problem
How old web2py? We have had bugs in the past that may cause your
problem.
You should try upgrade.
Massimo
On Jan 17, 6:58 pm, Michael Toomim too...@gmail.com wrote:
The problem for me is that this occurs on a webapp used by mechanical
turk, and it fails when I have hundreds of mechanical turkers using my
app... which only happens when I pay them hundreds of dollars. So
it's hard to reproduce right now without hundreds of dollars.
I am excited to try using VP's ab benchmark to try to reproduce the
error more cheaply. When I can do that, and get a minimal system
exhibiting the error, I will let you know.
I'm also currently using an older version of web2py. One pre-DAL.
On Jan 17, 12:20 pm, Massimo Di Pierro massimo.dipie...@gmail.com
wrote:
You should really have
db = DAL('postgres://name:password@localhost:5432/db',pool_size=20)
The reason is that client-server databases may set a max to number of
open connections and it takes time to perform the 3-way handshake to
establish a new connection at every http request. Without pooling you
may hit the max sooner than you should.
This will also make your app faster.
Massimo
On Jan 17, 1:39 pm, VP vtp2...@gmail.com wrote:
Here it is with private information replaced with generic information.
if request.env.web2py_runtime_gae: # if running on Google
App Engine
db = DAL('gae') # connect to Google
BigTable
session.connect(request, response, db = db) # and store sessions
and tickets there
else: # else use a normal
relational database
db = DAL('postgres://name:password@localhost:5432/db')
# db = DAL('sqlite://storage.sqlite')
On Jan 17, 1:19 pm, Massimo Di Pierro massimo.dipie...@gmail.com
wrote:
Are you using connection pooling? Can I see the line db = DAL(...)?
On Jan 16, 2:24 pm, VP vtp2...@gmail.com wrote:
Kenneth (see post above) had the same problem with MySQL.
On Jan 16, 3:55 am, ron_m ron.mco...@gmail.com wrote:
Is there any chance to try another server based database such as
MySQL.
Since SQLite did not exhibit a problem that is a valid data point
but it is
an embedded database so called directly by the web2py server code
instead of
using a driver. There is a possibility there is a problem with the
driver on
PostgreSQL if the connections are being reused in a pool. This is
very
difficult to find without some further logging. I gather the page
request
you see failing as a user simply never completes. I saw this once
while
testing the application but the loading is only myself running
tests. I use
PostgreSQL in this app but I also have a version that runs on MySQL
which
has only a connection string difference plus the content of the
databases
directory. Maybe I can try this tomorrow, it is almost 2 am here so
I don't
want to start something new right now. I just need to learn how to
use ab
which I take it isn't going to be much to figure out.
[web2py] Re: Apache, Wsgi problem
1.74.5. I will upgrade when I can reproduce the problem locally.
On Jan 17, 5:13 pm, Massimo Di Pierro massimo.dipie...@gmail.com
wrote:
How old web2py? We have had bugs in the past that may cause your
problem.
You should try upgrade.
Massimo
On Jan 17, 6:58 pm, Michael Toomim too...@gmail.com wrote:
The problem for me is that this occurs on a webapp used by mechanical
turk, and it fails when I have hundreds of mechanical turkers using my
app... which only happens when I pay them hundreds of dollars. So
it's hard to reproduce right now without hundreds of dollars.
I am excited to try using VP's ab benchmark to try to reproduce the
error more cheaply. When I can do that, and get a minimal system
exhibiting the error, I will let you know.
I'm also currently using an older version of web2py. One pre-DAL.
On Jan 17, 12:20 pm, Massimo Di Pierro massimo.dipie...@gmail.com
wrote:
You should really have
db = DAL('postgres://name:password@localhost:5432/db',pool_size=20)
The reason is that client-server databases may set a max to number of
open connections and it takes time to perform the 3-way handshake to
establish a new connection at every http request. Without pooling you
may hit the max sooner than you should.
This will also make your app faster.
Massimo
On Jan 17, 1:39 pm, VP vtp2...@gmail.com wrote:
Here it is with private information replaced with generic information.
if request.env.web2py_runtime_gae: # if running on Google
App Engine
db = DAL('gae') # connect to Google
BigTable
session.connect(request, response, db = db) # and store sessions
and tickets there
else: # else use a normal
relational database
db = DAL('postgres://name:password@localhost:5432/db')
# db = DAL('sqlite://storage.sqlite')
On Jan 17, 1:19 pm, Massimo Di Pierro massimo.dipie...@gmail.com
wrote:
Are you using connection pooling? Can I see the line db = DAL(...)?
On Jan 16, 2:24 pm, VP vtp2...@gmail.com wrote:
Kenneth (see post above) had the same problem with MySQL.
On Jan 16, 3:55 am, ron_m ron.mco...@gmail.com wrote:
Is there any chance to try another server based database such as
MySQL.
Since SQLite did not exhibit a problem that is a valid data point
but it is
an embedded database so called directly by the web2py server code
instead of
using a driver. There is a possibility there is a problem with
the driver on
PostgreSQL if the connections are being reused in a pool. This is
very
difficult to find without some further logging. I gather the page
request
you see failing as a user simply never completes. I saw this once
while
testing the application but the loading is only myself running
tests. I use
PostgreSQL in this app but I also have a version that runs on
MySQL which
has only a connection string difference plus the content of the
databases
directory. Maybe I can try this tomorrow, it is almost 2 am here
so I don't
want to start something new right now. I just need to learn how
to use ab
which I take it isn't going to be much to figure out.
[web2py] Re: Apache, Wsgi problem
Is there any chance to try another server based database such as MySQL. Since SQLite did not exhibit a problem that is a valid data point but it is an embedded database so called directly by the web2py server code instead of using a driver. There is a possibility there is a problem with the driver on PostgreSQL if the connections are being reused in a pool. This is very difficult to find without some further logging. I gather the page request you see failing as a user simply never completes. I saw this once while testing the application but the loading is only myself running tests. I use PostgreSQL in this app but I also have a version that runs on MySQL which has only a connection string difference plus the content of the databases directory. Maybe I can try this tomorrow, it is almost 2 am here so I don't want to start something new right now. I just need to learn how to use ab which I take it isn't going to be much to figure out.
[web2py] Re: Apache, Wsgi problem
I can't seem to get it to generate an error running on PostgreSQL and the apache logs show nothing. I upped the ab time ( -t ) to 100 and then used the application. It wasn't very responsive but then the ab average was 8+sec on the home page which just displays a message. I am using the default apache-wsgi installer for ubuntu. I set the connection pool to 25 on the DAL constructor. Here is the ab output camcentral@dev64:/var/log/apache2$ ab -kc 200 -t 100 http://192.168.210.5/ccims This is ApacheBench, Version 2.3 $Revision: 655654 $ Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 192.168.210.5 (be patient) Finished 1786 requests Server Software:Apache/2.2.14 Server Hostname:192.168.210.5 Server Port:80 Document Path: /ccims Document Length:9838 bytes Concurrency Level: 200 Time taken for tests: 100.018 seconds Complete requests: 1786 Failed requests:0 Write errors: 0 Keep-Alive requests:1786 Total transferred: 18383447 bytes HTML transferred: 17570668 bytes Requests per second:17.86 [#/sec] (mean) Time per request: 11200.252 [ms] (mean) Time per request: 56.001 [ms] (mean, across all concurrent requests) Transfer rate: 179.49 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:01 3.2 0 19 Processing: 384 7986 2609.6 8224 21216 Waiting: 384 7986 2609.6 8224 21216 Total:388 7987 2611.3 8224 21235 Percentage of the requests served within a certain time (ms) 50% 8224 66% 8327 75% 8411 80% 8502 90% 10175 95% 11982 98% 14889 99% 16970 100% 21235 (longest request)
[web2py] Re: Apache, Wsgi problem
Kenneth (see post above) had the same problem with MySQL. On Jan 16, 3:55 am, ron_m ron.mco...@gmail.com wrote: Is there any chance to try another server based database such as MySQL. Since SQLite did not exhibit a problem that is a valid data point but it is an embedded database so called directly by the web2py server code instead of using a driver. There is a possibility there is a problem with the driver on PostgreSQL if the connections are being reused in a pool. This is very difficult to find without some further logging. I gather the page request you see failing as a user simply never completes. I saw this once while testing the application but the loading is only myself running tests. I use PostgreSQL in this app but I also have a version that runs on MySQL which has only a connection string difference plus the content of the databases directory. Maybe I can try this tomorrow, it is almost 2 am here so I don't want to start something new right now. I just need to learn how to use ab which I take it isn't going to be much to figure out.
[web2py] Re: Apache, Wsgi problem
To follow up on this issue, I think it might be related to db and/or DAL. + I don't think it relates to exhausting RAM or too many open files. Using lsof to monitor open files during stress test, with 200 concurrent channels (ab), I witnessed up to 11,000 open files (most are apache2). There are several dozens of failed requests, causing by premature end of script wsgi. What is interesting is that I was able to cause these errors even with only 20 concurrent channels (with only about 3000-4000 open files). This is under postgres. + I could not cause the error with sqlite or on a page (controller) that has only 1 db call. + How was I able to cause this error with only 20 concurrent channels? First, I observed that ab is quite simple in that it hits the same page again and again. So I wanted a more realistic test (with more complex behavior). Without any other tools, I decided to do 2 things simultaneously: (1) ab with 20 concurrent channels, and (2) manually (ajax) search for items using the search form on the website; search will perform several db queries which ab does not. Well, the result is that there were several failed requests (resulting from this error) even with only 20 concurrent channels (which is ridiculous). + Another anecdote. I myself experienced this error a number of times while using normally the app (not a result of stress test). Once the error occurred, apache failed to serve the page, of course. What I observed is that when I immediately reload the page, it loads up again very quickly (as normally the case). What this tells me is that the cause of this wsgi error is probably NOT because of the exhaustion of some type of resources (RAM, or opening files, etc.); because that lacking resources was the cause, then there would be some time for the resources to be recovered before the page would quickly be served again. This issue is annoying. Crashing like this is not pleasant from users' point of view. It's clearly related to scalability of web2py. I hope there's an answer to this soon. Here's a typical output of ab with 20 concurrent connections showing failed requests. Finished 254 requests Server Software:Apache/2.2.9 Server Port:80 Document Path: / Document Length:10133 bytes Concurrency Level: 20 Time taken for tests: 10.045 seconds Complete requests: 254 Failed requests:15 (Connect: 0, Receive: 0, Length: 15, Exceptions: 0) Write errors: 0 Non-2xx responses: 15 Keep-Alive requests:239 Total transferred: 2558415 bytes HTML transferred: 2448532 bytes Requests per second:25.29 [#/sec] (mean) Time per request: 790.914 [ms] (mean) Time per request: 39.546 [ms] (mean, across all concurrent requests) Transfer rate: 248.74 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:06 16.1 0 56 Processing: 121 748 592.96314312 Waiting: 120 688 589.95664243 Total:121 755 603.86314365 Percentage of the requests served within a certain time (ms) 50%631 66%710 75%772 80%804 90% 1670 95% 1893 98% 2788 99% 3929 100% 4365 (longest request)
Re: [web2py] Re: Apache, Wsgi problem
I can´t be sure which application it is. But Iĺl do some stresstests
later today and report more.
Kenneth
could this be a timeout issue? Could it be the app causes a deadlock
in the db? What app is this?
On Jan 11, 10:58 am, Kenneth Lundströmkenneth.t.lundst...@gmail.com
wrote:
I'm getting the same Premature end of script on a CentOS 5.?, MySQL, WSGI and
Apache.
Never done any stresstests, I'll try later to day.
Kenneth
- Ursprungsmeddelande -
Update:
+ There seems to be no web2py error. Thus, no traceback.
+ I don't remember if I got this error Premature end of script
headers: wsgihandler.py ) with sqlite, but having just tested again
with sqlite, I didn't get this error. At the moment, this error only
occurred with postgres (8.6.3 as I'm on Debian Lenny).
+ This error occurred sporadically. Testing with ab ranging from 50
to 200 concurrent connections, I couldn't always duplicate the error.
Sometimes, it occurred. Sometimes, it didn't.
PS: Someone asked about apache bench. It's included in apache2-utils
(which can be installed by apt). Pretty easy to use. I used this: ab
-kc 200 -t 10http://yourdomain.com/to open 200 concurrent requests
in 10 seconds.
On Jan 11, 8:58 am, Massimo Di Pierromassimo.dipie...@gmail.com
wrote:
Can I see the error? Is there a traceback? Is the db server on the
machine?
Massimo
On Jan 9, 11:30 am, VPvtp2...@gmail.com wrote:
It is indeed the case that there was a segmentation fault, reported
in apache error log.
Perhaps, it's not clear, but this problem occurs under posgres under
debian lenny, not sqlite. I am not running web2py as a CGI script.
I am using the web2py deployment script (for setting up apache and
web2py):
VirtualHost *:80
WSGIDaemonProcess web2py user=myusername group=myusername \
display-name=%{GROUP}
WSGIProcessGroup web2py
WSGIScriptAlias / /home/myusername/web2py/wsgihandler.py
On Jan 8, 9:59 pm, Graham Dumpletongraham.dumple...@gmail.com
wrote:
You were possibly using a an old version of sqlite which isn't
safe to use in a multithreaded configuration.
The MPM settings are not going to help in this case as that error
could only come about because you are using mod_wsgi daemon mode
and so application is running in distinct process and not those
affected by the MPM or its settings.
The only other way you could get that error is that you are
actually running web2py as a CGI script.
Overall, that specific error message means your daemon mode
process that is running web2py crashed. You would likely find that
there is a segmentation fault messages in main Apache error log as
well at that time.
Crashing could be because of sqlite thread problems, but could
also be because you are forcing web2py to run in main interpreter
of daemon processes and at the same time are using a third party C
extension module for Python that is not safe for use in sub
interpreters.
So, ensure sqlite is up to date. And ensure that you have:
WSGIApplicationGroup %{GLOBAL}
in configuration to force use of main interpreter.
Graham
On Sunday, January 9, 2011 6:44:14 AM UTC+11, VP wrote:
We occasionally got an Apache error so the page didn't get
displayed. So I decided to stress test using Apache Bench (ab).
It seems the site suffered failure up to 50-100 concurrent
connection.
Apache error log showed this error:
Premature end of script headers: wsgihandler.py
After digging around, I found similar discussions and change
apache2.conf like this:
# prefork MPM
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 256
MaxRequestsPerChild 500
ServerLimit 256
Didn't seem to help.
A few notes:
+ It appears when I switched to sqlite instead of posgres, I
didn't have the problem. (Sqlite had other problems, such as
occasional database locking, which is more serious)
+ I am on a VPS with 768MB with 1GB burstable. While I'm doing
the stress test with Apache Bench (ab), using free on the server
revealed memory usage was about 450MB. (Which is a lot, but is
still under limit).
=
In summary, memory was available. But we got this wsgi error in
Apache with multiple requests.
Any idea please?
Thanks.
[web2py] Re: Apache, Wsgi problem
Can I see the error? Is there a traceback? Is the db server on the
machine?
Massimo
On Jan 9, 11:30 am, VP vtp2...@gmail.com wrote:
It is indeed the case that there was a segmentation fault, reported in
apache error log.
Perhaps, it's not clear, but this problem occurs under posgres under
debian lenny, not sqlite. I am not running web2py as a CGI script. I
am using the web2py deployment script (for setting up apache and
web2py):
VirtualHost *:80
WSGIDaemonProcess web2py user=myusername group=myusername \
display-name=%{GROUP}
WSGIProcessGroup web2py
WSGIScriptAlias / /home/myusername/web2py/wsgihandler.py
On Jan 8, 9:59 pm, Graham Dumpleton graham.dumple...@gmail.com
wrote:
You were possibly using a an old version of sqlite which isn't safe to use
in a multithreaded configuration.
The MPM settings are not going to help in this case as that error could only
come about because you are using mod_wsgi daemon mode and so application is
running in distinct process and not those affected by the MPM or its
settings.
The only other way you could get that error is that you are actually running
web2py as a CGI script.
Overall, that specific error message means your daemon mode process that is
running web2py crashed. You would likely find that there is a segmentation
fault messages in main Apache error log as well at that time.
Crashing could be because of sqlite thread problems, but could also be
because you are forcing web2py to run in main interpreter of daemon
processes and at the same time are using a third party C extension module
for Python that is not safe for use in sub interpreters.
So, ensure sqlite is up to date. And ensure that you have:
WSGIApplicationGroup %{GLOBAL}
in configuration to force use of main interpreter.
Graham
On Sunday, January 9, 2011 6:44:14 AM UTC+11, VP wrote:
We occasionally got an Apache error so the page didn't get displayed.
So I decided to stress test using Apache Bench (ab). It seems the
site suffered failure up to 50-100 concurrent connection.
Apache error log showed this error:
Premature end of script headers: wsgihandler.py
After digging around, I found similar discussions and change
apache2.conf like this:
# prefork MPM
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 256
MaxRequestsPerChild 500
ServerLimit 256
Didn't seem to help.
A few notes:
+ It appears when I switched to sqlite instead of posgres, I didn't
have the problem. (Sqlite had other problems, such as occasional
database locking, which is more serious)
+ I am on a VPS with 768MB with 1GB burstable. While I'm doing the
stress test with Apache Bench (ab), using free on the server revealed
memory usage was about 450MB. (Which is a lot, but is still under
limit).
=
In summary, memory was available. But we got this wsgi error in
Apache with multiple requests.
Any idea please?
Thanks.
[web2py] Re: Apache, Wsgi problem
Update:
+ There seems to be no web2py error. Thus, no traceback.
+ I don't remember if I got this error Premature end of script
headers: wsgihandler.py ) with sqlite, but having just tested again
with sqlite, I didn't get this error. At the moment, this error only
occurred with postgres (8.6.3 as I'm on Debian Lenny).
+ This error occurred sporadically. Testing with ab ranging from 50
to 200 concurrent connections, I couldn't always duplicate the error.
Sometimes, it occurred. Sometimes, it didn't.
PS: Someone asked about apache bench. It's included in apache2-utils
(which can be installed by apt). Pretty easy to use. I used this: ab
-kc 200 -t 10 http://yourdomain.com/ to open 200 concurrent requests
in 10 seconds.
On Jan 11, 8:58 am, Massimo Di Pierro massimo.dipie...@gmail.com
wrote:
Can I see the error? Is there a traceback? Is the db server on the
machine?
Massimo
On Jan 9, 11:30 am, VP vtp2...@gmail.com wrote:
It is indeed the case that there was a segmentation fault, reported in
apache error log.
Perhaps, it's not clear, but this problem occurs under posgres under
debian lenny, not sqlite. I am not running web2py as a CGI script. I
am using the web2py deployment script (for setting up apache and
web2py):
VirtualHost *:80
WSGIDaemonProcess web2py user=myusername group=myusername \
display-name=%{GROUP}
WSGIProcessGroup web2py
WSGIScriptAlias / /home/myusername/web2py/wsgihandler.py
On Jan 8, 9:59 pm, Graham Dumpleton graham.dumple...@gmail.com
wrote:
You were possibly using a an old version of sqlite which isn't safe to use
in a multithreaded configuration.
The MPM settings are not going to help in this case as that error could
only
come about because you are using mod_wsgi daemon mode and so application
is
running in distinct process and not those affected by the MPM or its
settings.
The only other way you could get that error is that you are actually
running
web2py as a CGI script.
Overall, that specific error message means your daemon mode process that
is
running web2py crashed. You would likely find that there is a segmentation
fault messages in main Apache error log as well at that time.
Crashing could be because of sqlite thread problems, but could also be
because you are forcing web2py to run in main interpreter of daemon
processes and at the same time are using a third party C extension module
for Python that is not safe for use in sub interpreters.
So, ensure sqlite is up to date. And ensure that you have:
WSGIApplicationGroup %{GLOBAL}
in configuration to force use of main interpreter.
Graham
On Sunday, January 9, 2011 6:44:14 AM UTC+11, VP wrote:
We occasionally got an Apache error so the page didn't get displayed.
So I decided to stress test using Apache Bench (ab). It seems the
site suffered failure up to 50-100 concurrent connection.
Apache error log showed this error:
Premature end of script headers: wsgihandler.py
After digging around, I found similar discussions and change
apache2.conf like this:
# prefork MPM
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 256
MaxRequestsPerChild 500
ServerLimit 256
Didn't seem to help.
A few notes:
+ It appears when I switched to sqlite instead of posgres, I didn't
have the problem. (Sqlite had other problems, such as occasional
database locking, which is more serious)
+ I am on a VPS with 768MB with 1GB burstable. While I'm doing the
stress test with Apache Bench (ab), using free on the server revealed
memory usage was about 450MB. (Which is a lot, but is still under
limit).
=
In summary, memory was available. But we got this wsgi error in
Apache with multiple requests.
Any idea please?
Thanks.
Re: [web2py] Re: Apache, Wsgi problem
I'm getting the same Premature end of script on a CentOS 5.?, MySQL, WSGI and
Apache.
Never done any stresstests, I'll try later to day.
Kenneth
- Ursprungsmeddelande -
Update:
+ There seems to be no web2py error. Thus, no traceback.
+ I don't remember if I got this error Premature end of script
headers: wsgihandler.py ) with sqlite, but having just tested again
with sqlite, I didn't get this error. At the moment, this error only
occurred with postgres (8.6.3 as I'm on Debian Lenny).
+ This error occurred sporadically. Testing with ab ranging from 50
to 200 concurrent connections, I couldn't always duplicate the error.
Sometimes, it occurred. Sometimes, it didn't.
PS: Someone asked about apache bench. It's included in apache2-utils
(which can be installed by apt). Pretty easy to use. I used this: ab
-kc 200 -t 10 http://yourdomain.com/ to open 200 concurrent requests
in 10 seconds.
On Jan 11, 8:58 am, Massimo Di Pierro massimo.dipie...@gmail.com
wrote:
Can I see the error? Is there a traceback? Is the db server on the
machine?
Massimo
On Jan 9, 11:30 am, VP vtp2...@gmail.com wrote:
It is indeed the case that there was a segmentation fault, reported
in apache error log.
Perhaps, it's not clear, but this problem occurs under posgres under
debian lenny, not sqlite. I am not running web2py as a CGI script.
I am using the web2py deployment script (for setting up apache and
web2py):
VirtualHost *:80
WSGIDaemonProcess web2py user=myusername group=myusername \
display-name=%{GROUP}
WSGIProcessGroup web2py
WSGIScriptAlias / /home/myusername/web2py/wsgihandler.py
On Jan 8, 9:59 pm, Graham Dumpleton graham.dumple...@gmail.com
wrote:
You were possibly using a an old version of sqlite which isn't
safe to use in a multithreaded configuration.
The MPM settings are not going to help in this case as that error
could only come about because you are using mod_wsgi daemon mode
and so application is running in distinct process and not those
affected by the MPM or its settings.
The only other way you could get that error is that you are
actually running web2py as a CGI script.
Overall, that specific error message means your daemon mode
process that is running web2py crashed. You would likely find that
there is a segmentation fault messages in main Apache error log as
well at that time.
Crashing could be because of sqlite thread problems, but could
also be because you are forcing web2py to run in main interpreter
of daemon processes and at the same time are using a third party C
extension module for Python that is not safe for use in sub
interpreters.
So, ensure sqlite is up to date. And ensure that you have:
WSGIApplicationGroup %{GLOBAL}
in configuration to force use of main interpreter.
Graham
On Sunday, January 9, 2011 6:44:14 AM UTC+11, VP wrote:
We occasionally got an Apache error so the page didn't get
displayed. So I decided to stress test using Apache Bench (ab).
It seems the site suffered failure up to 50-100 concurrent
connection.
Apache error log showed this error:
Premature end of script headers: wsgihandler.py
After digging around, I found similar discussions and change
apache2.conf like this:
# prefork MPM
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 256
MaxRequestsPerChild 500
ServerLimit 256
Didn't seem to help.
A few notes:
+ It appears when I switched to sqlite instead of posgres, I
didn't have the problem. (Sqlite had other problems, such as
occasional database locking, which is more serious)
+ I am on a VPS with 768MB with 1GB burstable. While I'm doing
the stress test with Apache Bench (ab), using free on the server
revealed memory usage was about 450MB. (Which is a lot, but is
still under limit).
=
In summary, memory was available. But we got this wsgi error in
Apache with multiple requests.
Any idea please?
Thanks.
[web2py] Re: Apache, Wsgi problem
could this be a timeout issue? Could it be the app causes a deadlock
in the db? What app is this?
On Jan 11, 10:58 am, Kenneth Lundström kenneth.t.lundst...@gmail.com
wrote:
I'm getting the same Premature end of script on a CentOS 5.?, MySQL, WSGI and
Apache.
Never done any stresstests, I'll try later to day.
Kenneth
- Ursprungsmeddelande -
Update:
+ There seems to be no web2py error. Thus, no traceback.
+ I don't remember if I got this error Premature end of script
headers: wsgihandler.py ) with sqlite, but having just tested again
with sqlite, I didn't get this error. At the moment, this error only
occurred with postgres (8.6.3 as I'm on Debian Lenny).
+ This error occurred sporadically. Testing with ab ranging from 50
to 200 concurrent connections, I couldn't always duplicate the error.
Sometimes, it occurred. Sometimes, it didn't.
PS: Someone asked about apache bench. It's included in apache2-utils
(which can be installed by apt). Pretty easy to use. I used this: ab
-kc 200 -t 10http://yourdomain.com/to open 200 concurrent requests
in 10 seconds.
On Jan 11, 8:58 am, Massimo Di Pierro massimo.dipie...@gmail.com
wrote:
Can I see the error? Is there a traceback? Is the db server on the
machine?
Massimo
On Jan 9, 11:30 am, VP vtp2...@gmail.com wrote:
It is indeed the case that there was a segmentation fault, reported
in apache error log.
Perhaps, it's not clear, but this problem occurs under posgres under
debian lenny, not sqlite. I am not running web2py as a CGI script.
I am using the web2py deployment script (for setting up apache and
web2py):
VirtualHost *:80
WSGIDaemonProcess web2py user=myusername group=myusername \
display-name=%{GROUP}
WSGIProcessGroup web2py
WSGIScriptAlias / /home/myusername/web2py/wsgihandler.py
On Jan 8, 9:59 pm, Graham Dumpleton graham.dumple...@gmail.com
wrote:
You were possibly using a an old version of sqlite which isn't
safe to use in a multithreaded configuration.
The MPM settings are not going to help in this case as that error
could only come about because you are using mod_wsgi daemon mode
and so application is running in distinct process and not those
affected by the MPM or its settings.
The only other way you could get that error is that you are
actually running web2py as a CGI script.
Overall, that specific error message means your daemon mode
process that is running web2py crashed. You would likely find that
there is a segmentation fault messages in main Apache error log as
well at that time.
Crashing could be because of sqlite thread problems, but could
also be because you are forcing web2py to run in main interpreter
of daemon processes and at the same time are using a third party C
extension module for Python that is not safe for use in sub
interpreters.
So, ensure sqlite is up to date. And ensure that you have:
WSGIApplicationGroup %{GLOBAL}
in configuration to force use of main interpreter.
Graham
On Sunday, January 9, 2011 6:44:14 AM UTC+11, VP wrote:
We occasionally got an Apache error so the page didn't get
displayed. So I decided to stress test using Apache Bench (ab).
It seems the site suffered failure up to 50-100 concurrent
connection.
Apache error log showed this error:
Premature end of script headers: wsgihandler.py
After digging around, I found similar discussions and change
apache2.conf like this:
# prefork MPM
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 256
MaxRequestsPerChild 500
ServerLimit 256
Didn't seem to help.
A few notes:
+ It appears when I switched to sqlite instead of posgres, I
didn't have the problem. (Sqlite had other problems, such as
occasional database locking, which is more serious)
+ I am on a VPS with 768MB with 1GB burstable. While I'm doing
the stress test with Apache Bench (ab), using free on the server
revealed memory usage was about 450MB. (Which is a lot, but is
still under limit).
=
In summary, memory was available. But we got this wsgi error in
Apache with multiple requests.
Any idea please?
Thanks.
[web2py] Re: Apache, Wsgi problem
I'm still having this problem too (previous posts linked below). I
would love to find a solution. I'm not sure how to debug.
VP: Can you provide instructions for reproducing this bug using ab? I
had trouble using ab in the past. I am also on a VPS.
Since my last post (linked below), I have tried the following:
1) setting migrate=False
2) compiling the app
These did not fix the problem.
On Jan 9, 9:30 am, VP vtp2...@gmail.com wrote:
It is indeed the case that there was a segmentation fault, reported in
apache error log.
Perhaps, it's not clear, but this problem occurs under posgres under
debian lenny, not sqlite. I am not running web2py as a CGIscript. I
am using the web2py deploymentscript(for setting up apache and
web2py):
VirtualHost *:80
WSGIDaemonProcess web2py user=myusername group=myusername \
display-name=%{GROUP}
WSGIProcessGroup web2py
WSGIScriptAlias / /home/myusername/web2py/wsgihandler.py
On Jan 8, 9:59 pm, Graham Dumpleton graham.dumple...@gmail.com
wrote:
You were possibly using a an old version of sqlite which isn't safe to use
in a multithreaded configuration.
The MPM settings are not going to help in this case as that error could only
come about because you are using mod_wsgi daemon mode and so application is
running in distinct process and not those affected by the MPM or its
settings.
The only other way you could get that error is that you are actually running
web2py as a CGIscript.
Overall, that specific error message means your daemon mode process that is
running web2py crashed. You would likely find that there is a segmentation
fault messages in main Apache error log as well at that time.
Crashing could be because of sqlite thread problems, but could also be
because you are forcing web2py to run in main interpreter of daemon
processes and at the same time are using a third party C extension module
for Python that is not safe for use in sub interpreters.
So, ensure sqlite is up to date. And ensure that you have:
WSGIApplicationGroup %{GLOBAL}
in configuration to force use of main interpreter.
Graham
On Sunday, January 9, 2011 6:44:14 AM UTC+11, VP wrote:
We occasionally got an Apache error so the page didn't get displayed.
So I decided to stress test using Apache Bench (ab). It seems the
site suffered failure up to 50-100 concurrent connection.
Apache error log showed this error:
Prematureendofscriptheaders: wsgihandler.py
After digging around, I found similar discussions and change
apache2.conf like this:
# prefork MPM
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 256
MaxRequestsPerChild 500
ServerLimit 256
Didn't seem to help.
A few notes:
+ It appears when I switched to sqlite instead of posgres, I didn't
have the problem. (Sqlite had other problems, such as occasional
database locking, which is more serious)
+ I am on a VPS with 768MB with 1GB burstable. While I'm doing the
stress test with Apache Bench (ab), using free on the server revealed
memory usage was about 450MB. (Which is a lot, but is still under
limit).
=
In summary, memory was available. But we got this wsgi error in
Apache with multiple requests.
Any idea please?
Thanks.
[web2py] Re: Apache, Wsgi problem
I don't know if it is worth mentioning but as a Ubuntu 10.04.1 user, I noticed the 8.4.6 update to PostgreSQL was pushed by Ubuntu near the end of December. http://www.postgresql.org/docs/8.4/static/release-8-4-6.html first item of the release notes mention the database can fail on some file systems notably ext4 with data=journal mount option. I mention it only because your problem appears with the use of postgres.
[web2py] Re: Apache, Wsgi problem
Thanks, I just investigated this, but it looks like it did not fix the problem. In 8.4.6 Postgres changed the default wal_sync_method to fdatasync, because the old default open_datasync failed on ext4. I use ext3 (on ubuntu 9.10), but I tried changing this option in my postgres database anyway. I changed it, restarted postgres and apache, but still get the error.
[web2py] Re: Apache, Wsgi problem
For the record. I do not think I never said my viewpoint is the best. I do not recall other people of this list doing it either. I just state my viewpoint. I usually limit myself to one of the follow: - mention web2py in a thread related to web frameworks - provide an example of how web2py does things vs how others do things - defend the web2py way against attacks by competitors Personally, I never find annoying when people talk about their own work and I do expect people to be enthusiastic and promote their work. I want to know what other people do because I can learn from it. I also do not find annoying when other people provide a technical argument of why something is - in their view - wrong. For example mitsuhiko showing that exec can lead to a memory leak allows us to address the issue in a technical. He is not diplomatic (should post bug reports in the appropriate channels) but I have learned from him posts in the past. What I do find annoying is people claiming that program X - open source - if harmful to the python community because it is different. I guess Python is harmful to the C# community and by rule of majority we should not use it. Massimo On Jan 9, 1:21 am, Graham Dumpleton graham.dumple...@gmail.com wrote: On Sunday, January 9, 2011 6:05:16 PM UTC+11, Massimo Di Pierro wrote: Graham. We all appreciate your expertise here and appreciate people who say what they think. Perhaps diplomacy is not our strength. You do not need to justify your tweet. I though it was funny because - in my view - it would apply equally well to some of the other frameworks. Yes, there is at least one other web framework which I find a bit annoying as far as people associated with it pushing that their viewpoint is the best. They generally keep to themselves though. And no that other framework isn't Django. I have always found people in Django community quite open and pleasant to deal with and I wouldn't take the momentary frustrated outbursts of one individual as saying too much about the community as a whole. No matter what the framework, one can always come across specific individuals which are quite extreme in their views and try to monopolise a situation and so can be difficult to deal with. I am sure you have possibly seen that at times on the Python WEB-SIG. I am even guilty of it some times, especially when I have got frustrated because of stuff being hijacked all the time. :-( Graham On Jan 9, 12:24 am, Graham Dumpleton graham.d...@gmail.com wrote: On Sunday, January 9, 2011 3:54:37 PM UTC+11, Anthony wrote: Graham, welcome. Careful, though -- we might convert http://twitter.com/GrahamDumpleton/status/23120780938190848you. ;-) http://twitter.com/GrahamDumpleton/status/23120780938190848 I have been posting here for quite a long time actually and tried a few times to get Massimo to temper his ways but even though he is more restrained now, sorry to say, he does still seem to rub people up the wrong way as do the reactions of others in the web2py community. You guys really just got to learn to do your own thing and not treating it like a crusade where you have to convert the world. Django became popular on technical merit and because of easily accessible good documentation, not because its community went around like insert name of church group knocking on doors trying to convert people or arguing every little minor technical point to try and win people other. That sort of behaviour really reminds me of some new age christians I have known in the past who just could not shut up in trying to make you believe what they believed. It gets tiresome and sometimes one cant take any more and gets angry about having your time wasted. I can therefore appreciate the reactions of others associated with other web frameworks, and if you think the latest outburst about imports was some reaction about web2py getting a tutorial accepted in PyCon as some like to think, you are very sadly mistaken. They are more likely just a manifestation of this frustration of having web2py shoved in your face one too many times especially when muds gets thrown in your direction. So, quiet it down and you will go a lot further with less angst against you. FWIW, although I find some of the ways web2py does things an interesting approach and would be appealing to certain classes of users, I do have concerns about how it does other things, especially in conjunction with how it tries to label itself an enterprise framework. So, it definitely isn't in my mind suitable for all situations. As to the 'import' issue, I don't give a stuff about it either way so please don't drag me into that one. Graham On Saturday, January 8, 2011 10:59:37 PM UTC-5, Graham Dumpleton wrote: You were possibly using a an old version of sqlite which isn't safe to use in a multithreaded
[web2py] Re: Apache, Wsgi problem
It is indeed the case that there was a segmentation fault, reported in
apache error log.
Perhaps, it's not clear, but this problem occurs under posgres under
debian lenny, not sqlite. I am not running web2py as a CGI script. I
am using the web2py deployment script (for setting up apache and
web2py):
VirtualHost *:80
WSGIDaemonProcess web2py user=myusername group=myusername \
display-name=%{GROUP}
WSGIProcessGroup web2py
WSGIScriptAlias / /home/myusername/web2py/wsgihandler.py
On Jan 8, 9:59 pm, Graham Dumpleton graham.dumple...@gmail.com
wrote:
You were possibly using a an old version of sqlite which isn't safe to use
in a multithreaded configuration.
The MPM settings are not going to help in this case as that error could only
come about because you are using mod_wsgi daemon mode and so application is
running in distinct process and not those affected by the MPM or its
settings.
The only other way you could get that error is that you are actually running
web2py as a CGI script.
Overall, that specific error message means your daemon mode process that is
running web2py crashed. You would likely find that there is a segmentation
fault messages in main Apache error log as well at that time.
Crashing could be because of sqlite thread problems, but could also be
because you are forcing web2py to run in main interpreter of daemon
processes and at the same time are using a third party C extension module
for Python that is not safe for use in sub interpreters.
So, ensure sqlite is up to date. And ensure that you have:
WSGIApplicationGroup %{GLOBAL}
in configuration to force use of main interpreter.
Graham
On Sunday, January 9, 2011 6:44:14 AM UTC+11, VP wrote:
We occasionally got an Apache error so the page didn't get displayed.
So I decided to stress test using Apache Bench (ab). It seems the
site suffered failure up to 50-100 concurrent connection.
Apache error log showed this error:
Premature end of script headers: wsgihandler.py
After digging around, I found similar discussions and change
apache2.conf like this:
# prefork MPM
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 256
MaxRequestsPerChild 500
ServerLimit 256
Didn't seem to help.
A few notes:
+ It appears when I switched to sqlite instead of posgres, I didn't
have the problem. (Sqlite had other problems, such as occasional
database locking, which is more serious)
+ I am on a VPS with 768MB with 1GB burstable. While I'm doing the
stress test with Apache Bench (ab), using free on the server revealed
memory usage was about 450MB. (Which is a lot, but is still under
limit).
=
In summary, memory was available. But we got this wsgi error in
Apache with multiple requests.
Any idea please?
Thanks.
[web2py] Re: Apache, Wsgi problem
On Sunday, January 9, 2011 1:24:02 AM UTC-5, Graham Dumpleton wrote: You guys really just got to learn to do your own thing and not treating it like a crusade where you have to convert the world. But that's exactly the problem -- web2py is trying to do its own thing, and there is a vocal clique of Pythonistas who don't like that thing and want to stop it. They say things like web2py is *seriously harmful* and join the rest of us and claim by fiat that web2py is not really a Python framework. They are the ones on a crusade -- to expunge the diversity that web2py brings to the Python web framework world. web2py is not trying to say Django, Pyramid, and Flask are doing things wrong and everyone should do things the web2py way -- it's just saying, hey, here's another way to do things, and if it suits you, use it. The reddit gang, on the other hand, is in fact trying to convert the world to the one true way of doing things. The reddit gang wants to impose purity and homogeneity -- web2py users just want to get things done (and are happy to let other people get things done however they want). Django became popular on technical merit and because of easily accessible good documentation, not because its community went around like insert name of church group knocking on doors trying to convert people or arguing every little minor technical point to try and win people other. That sort of behaviour really reminds me of some new age christians I have known in the past who just could not shut up in trying to make you believe what they believed. Perhaps I'm looking in the wrong places, but I haven't seen anyone pushing web2py in this way. In your tweet, you said, web2py is appearing more and more look like a religious cult. Mere promotion is not what characterizes a cult. Companies advertise their products, and (I assume) you do not consider them to be religious cults simply because they're trying to convince people to buy those products. Massimo actively promotes web2py because he wants people to become aware of it, consider it, and benefit from it if it suits their needs and preferences. Otherwise, what was the point of creating it? When he promotes web2py, he provides detailed information and code samples. When people ask questions, he answers. When people make challenges, he engages them in constructive dialogue. You may think he does this too much, or not always in the most relevant places, but this is not cult-like behavior. In fact, it could be argued that the reddit Pythonistas are the ones engaged in more cult-like behavior. Their creeds include import this and PEP 8. Sometimes they can be prone to extremism (explicit is *better* than implicit becomes explicitness at all cost) -- often forgetting that their own creeds advise practicality beats purity and a foolish consistency is the hobgoblin of little minds. They call for wayward frameworks to join us. When challenged or pressed for evidence, they retreat to the safety of their like-minded peers, re-directing their frustrations to Twitter, where they know they cannot be challenged. Of course, they're not really a cult -- the point is, calling web2py a cult is a bit of a pot and kettle situation. Anyway, I'm not sure, but I suspect the problem may simply be one of perspective. It appears the reddit guys are for the most part hard-core, committed, expert Python web programmers with a fair amount of personal investment (and identity) in Python web frameworks and the Python community. They have a particular way of doing things and a particular set of needs, and web2py just isn't their cup of tea. That's fine. The problem is, they may not be considering that there are other market segments (besides people like themselves) for Python web frameworks. They may not appreciate the diversity of needs, backgrounds, and preferences to which web frameworks may be targeted. They also spend a lot of time on reddit, so they end up seeing every mention of web2py. This may seem annoying to *them*, but again, they may not be considering that they're not the target of the message and that the reddit audience is much broarder than just them. ...if you think the latest outburst about imports was some reaction about web2py getting a tutorial accepted in PyCon as some like to think, you are very sadly mistaken. They are more likely just a manifestation of this frustration of having web2py shoved in your face one too many times especially when muds gets thrown in your direction. I wasn't aware anyone was thinking the latest outburst was a reaction to the web2py tutorial -- who is saying that? Also, in this particular case, the reddit OP specifically asked about web2py, so certainly nothing was being shoved in anyone's face. And I don't think I've seen web2py folks throwing mud at anyone (unless mere disagreement automatically equals mud). So, it definitely isn't in my mind
[web2py] Re: Apache, Wsgi problem
I came to Web2py recently and I can definitely see how the promotion of Web2py in places like Reddit would rub people the wrong way. Much of the hostility seems related to how Web2py is promoted, not to its technical merits.
[web2py] Re: Apache, Wsgi problem
You were possibly using a an old version of sqlite which isn't safe to use
in a multithreaded configuration.
The MPM settings are not going to help in this case as that error could only
come about because you are using mod_wsgi daemon mode and so application is
running in distinct process and not those affected by the MPM or its
settings.
The only other way you could get that error is that you are actually running
web2py as a CGI script.
Overall, that specific error message means your daemon mode process that is
running web2py crashed. You would likely find that there is a segmentation
fault messages in main Apache error log as well at that time.
Crashing could be because of sqlite thread problems, but could also be
because you are forcing web2py to run in main interpreter of daemon
processes and at the same time are using a third party C extension module
for Python that is not safe for use in sub interpreters.
So, ensure sqlite is up to date. And ensure that you have:
WSGIApplicationGroup %{GLOBAL}
in configuration to force use of main interpreter.
Graham
On Sunday, January 9, 2011 6:44:14 AM UTC+11, VP wrote:
We occasionally got an Apache error so the page didn't get displayed.
So I decided to stress test using Apache Bench (ab). It seems the
site suffered failure up to 50-100 concurrent connection.
Apache error log showed this error:
Premature end of script headers: wsgihandler.py
After digging around, I found similar discussions and change
apache2.conf like this:
# prefork MPM
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 256
MaxRequestsPerChild 500
ServerLimit 256
Didn't seem to help.
A few notes:
+ It appears when I switched to sqlite instead of posgres, I didn't
have the problem. (Sqlite had other problems, such as occasional
database locking, which is more serious)
+ I am on a VPS with 768MB with 1GB burstable. While I'm doing the
stress test with Apache Bench (ab), using free on the server revealed
memory usage was about 450MB. (Which is a lot, but is still under
limit).
=
In summary, memory was available. But we got this wsgi error in
Apache with multiple requests.
Any idea please?
Thanks.
[web2py] Re: Apache, Wsgi problem
Graham, welcome. Careful, though -- we might
converthttp://twitter.com/GrahamDumpleton/status/23120780938190848you. ;-)
http://twitter.com/GrahamDumpleton/status/23120780938190848
On Saturday, January 8, 2011 10:59:37 PM UTC-5, Graham Dumpleton wrote:
You were possibly using a an old version of sqlite which isn't safe to use
in a multithreaded configuration.
The MPM settings are not going to help in this case as that error could
only come about because you are using mod_wsgi daemon mode and so
application is running in distinct process and not those affected by the MPM
or its settings.
The only other way you could get that error is that you are actually
running web2py as a CGI script.
Overall, that specific error message means your daemon mode process that is
running web2py crashed. You would likely find that there is a segmentation
fault messages in main Apache error log as well at that time.
Crashing could be because of sqlite thread problems, but could also be
because you are forcing web2py to run in main interpreter of daemon
processes and at the same time are using a third party C extension module
for Python that is not safe for use in sub interpreters.
So, ensure sqlite is up to date. And ensure that you have:
WSGIApplicationGroup %{GLOBAL}
in configuration to force use of main interpreter.
Graham
On Sunday, January 9, 2011 6:44:14 AM UTC+11, VP wrote:
We occasionally got an Apache error so the page didn't get displayed.
So I decided to stress test using Apache Bench (ab). It seems the
site suffered failure up to 50-100 concurrent connection.
Apache error log showed this error:
Premature end of script headers: wsgihandler.py
After digging around, I found similar discussions and change
apache2.conf like this:
# prefork MPM
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 256
MaxRequestsPerChild 500
ServerLimit 256
Didn't seem to help.
A few notes:
+ It appears when I switched to sqlite instead of posgres, I didn't
have the problem. (Sqlite had other problems, such as occasional
database locking, which is more serious)
+ I am on a VPS with 768MB with 1GB burstable. While I'm doing the
stress test with Apache Bench (ab), using free on the server revealed
memory usage was about 450MB. (Which is a lot, but is still under
limit).
=
In summary, memory was available. But we got this wsgi error in
Apache with multiple requests.
Any idea please?
Thanks.
[web2py] Re: Apache, Wsgi problem
On Sunday, January 9, 2011 3:54:37 PM UTC+11, Anthony wrote:
Graham, welcome. Careful, though -- we might
converthttp://twitter.com/GrahamDumpleton/status/23120780938190848you. ;-)
http://twitter.com/GrahamDumpleton/status/23120780938190848
I have been posting here for quite a long time actually and tried a few
times to get Massimo to temper his ways but even though he is more
restrained now, sorry to say, he does still seem to rub people up the wrong
way as do the reactions of others in the web2py community. You guys really
just got to learn to do your own thing and not treating it like a crusade
where you have to convert the world. Django became popular on technical
merit and because of easily accessible good documentation, not because its
community went around like insert name of church group knocking on doors
trying to convert people or arguing every little minor technical point to
try and win people other. That sort of behaviour really reminds me of some
new age christians I have known in the past who just could not shut up in
trying to make you believe what they believed. It gets tiresome and
sometimes one cant take any more and gets angry about having your time
wasted. I can therefore appreciate the reactions of others associated with
other web frameworks, and if you think the latest outburst about imports was
some reaction about web2py getting a tutorial accepted in PyCon as some like
to think, you are very sadly mistaken. They are more likely just a
manifestation of this frustration of having web2py shoved in your face one
too many times especially when muds gets thrown in your direction. So, quiet
it down and you will go a lot further with less angst against you.
FWIW, although I find some of the ways web2py does things an interesting
approach and would be appealing to certain classes of users, I do have
concerns about how it does other things, especially in conjunction with how
it tries to label itself an enterprise framework. So, it definitely isn't in
my mind suitable for all situations. As to the 'import' issue, I don't give
a stuff about it either way so please don't drag me into that one.
Graham
On Saturday, January 8, 2011 10:59:37 PM UTC-5, Graham Dumpleton wrote:
You were possibly using a an old version of sqlite which isn't safe to use
in a multithreaded configuration.
The MPM settings are not going to help in this case as that error could
only come about because you are using mod_wsgi daemon mode and so
application is running in distinct process and not those affected by the MPM
or its settings.
The only other way you could get that error is that you are actually
running web2py as a CGI script.
Overall, that specific error message means your daemon mode process that
is running web2py crashed. You would likely find that there is a
segmentation fault messages in main Apache error log as well at that time.
Crashing could be because of sqlite thread problems, but could also be
because you are forcing web2py to run in main interpreter of daemon
processes and at the same time are using a third party C extension module
for Python that is not safe for use in sub interpreters.
So, ensure sqlite is up to date. And ensure that you have:
WSGIApplicationGroup %{GLOBAL}
in configuration to force use of main interpreter.
Graham
On Sunday, January 9, 2011 6:44:14 AM UTC+11, VP wrote:
We occasionally got an Apache error so the page didn't get displayed.
So I decided to stress test using Apache Bench (ab). It seems the
site suffered failure up to 50-100 concurrent connection.
Apache error log showed this error:
Premature end of script headers: wsgihandler.py
After digging around, I found similar discussions and change
apache2.conf like this:
# prefork MPM
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 256
MaxRequestsPerChild 500
ServerLimit 256
Didn't seem to help.
A few notes:
+ It appears when I switched to sqlite instead of posgres, I didn't
have the problem. (Sqlite had other problems, such as occasional
database locking, which is more serious)
+ I am on a VPS with 768MB with 1GB burstable. While I'm doing the
stress test with Apache Bench (ab), using free on the server revealed
memory usage was about 450MB. (Which is a lot, but is still under
limit).
=
In summary, memory was available. But we got this wsgi error in
Apache with multiple requests.
Any idea please?
Thanks.
[web2py] Re: Apache, Wsgi problem
Graham. We all appreciate your expertise here and appreciate people
who say what they think. Perhaps diplomacy is not our strength. You do
not need to justify your tweet. I though it was funny because - in my
view - it would apply equally well to some of the other frameworks.
On Jan 9, 12:24 am, Graham Dumpleton graham.dumple...@gmail.com
wrote:
On Sunday, January 9, 2011 3:54:37 PM UTC+11, Anthony wrote:
Graham, welcome. Careful, though -- we might
converthttp://twitter.com/GrahamDumpleton/status/23120780938190848you. ;-)
http://twitter.com/GrahamDumpleton/status/23120780938190848
I have been posting here for quite a long time actually and tried a few
times to get Massimo to temper his ways but even though he is more
restrained now, sorry to say, he does still seem to rub people up the wrong
way as do the reactions of others in the web2py community. You guys really
just got to learn to do your own thing and not treating it like a crusade
where you have to convert the world. Django became popular on technical
merit and because of easily accessible good documentation, not because its
community went around like insert name of church group knocking on doors
trying to convert people or arguing every little minor technical point to
try and win people other. That sort of behaviour really reminds me of some
new age christians I have known in the past who just could not shut up in
trying to make you believe what they believed. It gets tiresome and
sometimes one cant take any more and gets angry about having your time
wasted. I can therefore appreciate the reactions of others associated with
other web frameworks, and if you think the latest outburst about imports was
some reaction about web2py getting a tutorial accepted in PyCon as some like
to think, you are very sadly mistaken. They are more likely just a
manifestation of this frustration of having web2py shoved in your face one
too many times especially when muds gets thrown in your direction. So, quiet
it down and you will go a lot further with less angst against you.
FWIW, although I find some of the ways web2py does things an interesting
approach and would be appealing to certain classes of users, I do have
concerns about how it does other things, especially in conjunction with how
it tries to label itself an enterprise framework. So, it definitely isn't in
my mind suitable for all situations. As to the 'import' issue, I don't give
a stuff about it either way so please don't drag me into that one.
Graham
On Saturday, January 8, 2011 10:59:37 PM UTC-5, Graham Dumpleton wrote:
You were possibly using a an old version of sqlite which isn't safe to use
in a multithreaded configuration.
The MPM settings are not going to help in this case as that error could
only come about because you are using mod_wsgi daemon mode and so
application is running in distinct process and not those affected by the
MPM
or its settings.
The only other way you could get that error is that you are actually
running web2py as a CGI script.
Overall, that specific error message means your daemon mode process that
is running web2py crashed. You would likely find that there is a
segmentation fault messages in main Apache error log as well at that time.
Crashing could be because of sqlite thread problems, but could also be
because you are forcing web2py to run in main interpreter of daemon
processes and at the same time are using a third party C extension module
for Python that is not safe for use in sub interpreters.
So, ensure sqlite is up to date. And ensure that you have:
WSGIApplicationGroup %{GLOBAL}
in configuration to force use of main interpreter.
Graham
On Sunday, January 9, 2011 6:44:14 AM UTC+11, VP wrote:
We occasionally got an Apache error so the page didn't get displayed.
So I decided to stress test using Apache Bench (ab). It seems the
site suffered failure up to 50-100 concurrent connection.
Apache error log showed this error:
Premature end of script headers: wsgihandler.py
After digging around, I found similar discussions and change
apache2.conf like this:
# prefork MPM
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 256
MaxRequestsPerChild 500
ServerLimit 256
Didn't seem to help.
A few notes:
+ It appears when I switched to sqlite instead of posgres, I didn't
have the problem. (Sqlite had other problems, such as occasional
database locking, which is more serious)
+ I am on a VPS with 768MB with 1GB burstable. While I'm doing the
stress test with Apache Bench (ab), using free on the server revealed
memory usage was about 450MB. (Which is a lot, but is still under
limit).
=
In summary, memory was available. But we got this wsgi error in
Apache with multiple requests.
Any idea please?
Thanks.
[web2py] Re: Apache, Wsgi problem
On Sunday, January 9, 2011 6:05:16 PM UTC+11, Massimo Di Pierro wrote:
Graham. We all appreciate your expertise here and appreciate people
who say what they think. Perhaps diplomacy is not our strength. You do
not need to justify your tweet. I though it was funny because - in my
view - it would apply equally well to some of the other frameworks.
Yes, there is at least one other web framework which I find a bit annoying
as far as people associated with it pushing that their viewpoint is the
best. They generally keep to themselves though. And no that other framework
isn't Django. I have always found people in Django community quite open and
pleasant to deal with and I wouldn't take the momentary frustrated outbursts
of one individual as saying too much about the community as a whole.
No matter what the framework, one can always come across specific
individuals which are quite extreme in their views and try to monopolise a
situation and so can be difficult to deal with. I am sure you have possibly
seen that at times on the Python WEB-SIG. I am even guilty of it some times,
especially when I have got frustrated because of stuff being hijacked all
the time. :-(
Graham
On Jan 9, 12:24 am, Graham Dumpleton graham.d...@gmail.com
wrote:
On Sunday, January 9, 2011 3:54:37 PM UTC+11, Anthony wrote:
Graham, welcome. Careful, though -- we might convert
http://twitter.com/GrahamDumpleton/status/23120780938190848you. ;-)
http://twitter.com/GrahamDumpleton/status/23120780938190848
I have been posting here for quite a long time actually and tried a few
times to get Massimo to temper his ways but even though he is more
restrained now, sorry to say, he does still seem to rub people up the
wrong
way as do the reactions of others in the web2py community. You guys
really
just got to learn to do your own thing and not treating it like a crusade
where you have to convert the world. Django became popular on technical
merit and because of easily accessible good documentation, not because
its
community went around like insert name of church group knocking on
doors
trying to convert people or arguing every little minor technical point to
try and win people other. That sort of behaviour really reminds me of
some
new age christians I have known in the past who just could not shut up in
trying to make you believe what they believed. It gets tiresome and
sometimes one cant take any more and gets angry about having your time
wasted. I can therefore appreciate the reactions of others associated
with
other web frameworks, and if you think the latest outburst about imports
was
some reaction about web2py getting a tutorial accepted in PyCon as some
like
to think, you are very sadly mistaken. They are more likely just a
manifestation of this frustration of having web2py shoved in your face
one
too many times especially when muds gets thrown in your direction. So,
quiet
it down and you will go a lot further with less angst against you.
FWIW, although I find some of the ways web2py does things an interesting
approach and would be appealing to certain classes of users, I do have
concerns about how it does other things, especially in conjunction with
how
it tries to label itself an enterprise framework. So, it definitely isn't
in
my mind suitable for all situations. As to the 'import' issue, I don't
give
a stuff about it either way so please don't drag me into that one.
Graham
On Saturday, January 8, 2011 10:59:37 PM UTC-5, Graham Dumpleton wrote:
You were possibly using a an old version of sqlite which isn't safe to
use
in a multithreaded configuration.
The MPM settings are not going to help in this case as that error
could
only come about because you are using mod_wsgi daemon mode and so
application is running in distinct process and not those affected by
the MPM
or its settings.
The only other way you could get that error is that you are actually
running web2py as a CGI script.
Overall, that specific error message means your daemon mode process
that
is running web2py crashed. You would likely find that there is a
segmentation fault messages in main Apache error log as well at that
time.
Crashing could be because of sqlite thread problems, but could also be
because you are forcing web2py to run in main interpreter of daemon
processes and at the same time are using a third party C extension
module
for Python that is not safe for use in sub interpreters.
So, ensure sqlite is up to date. And ensure that you have:
WSGIApplicationGroup %{GLOBAL}
in configuration to force use of main interpreter.
Graham
On Sunday, January 9, 2011 6:44:14 AM UTC+11, VP wrote:
We occasionally got an Apache error so the page didn't get
