Pylons memory leak
I'm running pylons with paster in daemon mode. At startup the paster process is using 10% (100mb) memory. Paster is serving about 500k requests per day, 350k are pages, 150k are ajax requests. Static files (an extra 800k requests) are being served directly with nginx which also acts as the revproxy for paster. When using CherryPy as the server, within 4-5 hours the paster process reaches 90% memory usage and then dies. Paste#http takes longer, about a couple days, I suppose as it recycles threads every 100 requests, but it too leaks memory. No in-memory caches are being used and the only modules that use c extensions are psycopg2 and PIL. The global (g) is not being used at all. What can I do to find the problem? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Bulk inserts
On Wed, Jun 11, 2008 at 7:20 PM, Contact 42 [EMAIL PROTECTED] wrote: Jonathan Vanasco wrote: write to a .txt log in some sort of standardized format , when it hits 10k lines run a batch query what happens if an insert fails ? It continues with the rest of the batch. It'll tell you at the end how many succeeded and how many failed. You can read the MySQL documentation to figure out how to look up the error messages. My schema is setup so that I don't get failures unless something really awful happens. -jj -- I, for one, welcome our new Facebook overlords! http://jjinux.blogspot.com/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Pylons memory leak
On Wed, Jun 11, 2008 at 10:28 PM, zepolen [EMAIL PROTECTED] wrote: I'm running pylons with paster in daemon mode. At startup the paster process is using 10% (100mb) memory. Paster is serving about 500k requests per day, 350k are pages, 150k are ajax requests. Static files (an extra 800k requests) are being served directly with nginx which also acts as the revproxy for paster. When using CherryPy as the server, within 4-5 hours the paster process reaches 90% memory usage and then dies. Paste#http takes longer, about a couple days, I suppose as it recycles threads every 100 requests, but it too leaks memory. No in-memory caches are being used and the only modules that use c extensions are psycopg2 and PIL. The global (g) is not being used at all. What can I do to find the problem? I've had some luck in the past finding problems using the gc module. You might want to google for solutions to this problem since it's something most Python programmers eventually run across. -jj -- I, for one, welcome our new Facebook overlords! http://jjinux.blogspot.com/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Truncating an html string safely
On Sat, Jun 7, 2008 at 7:24 AM, Matt Feifarek [EMAIL PROTECTED] wrote: Oops; replied from the wrong address. -- Forwarded message -- On Thu, Jun 5, 2008 at 2:36 PM, Ian Bicking [EMAIL PROTECTED] wrote: Well... it's hard to truncate exactly, as there's all that annoying nesting stuff. An untested attempt with lxml: Exactly. Thanks for the lead. I'm not sure I'm up to the challenge, but if I do get it working, I'll get it back to you, in case it's good enough to be added to lxml (or whatever). Mike: Seems like if we have the truncate function in webhelpers, a truncate that handles html would be wise... since we're, err, making html, usually, with Pylons. Since the Django code doesn't seem to depend on anything (but some Django cruft, which seems to be frosting really) MAYBE it would be better to start with. But I'll poke around a bit today. It would be fun to write a SAX handler that permits all tags, and counts all characters. It would stop permitting additional characters once it reached a certain limit. -jj -- I, for one, welcome our new Facebook overlords! http://jjinux.blogspot.com/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: DistributionNotFound: decorator problem using PyISAPIe
On Wed, Jun 4, 2008 at 3:12 PM, rcs_comp [EMAIL PROTECTED] wrote:
I am trying to get Paste running using the PyISAPIe WSGI server on
IIS. However, I am getting this error and I can't figure out why:
The same application succesfully runs with the paste HTTP server.
Traceback (most recent call last):
File C:\Python25\lib\site-packages\Http\Isapi.py, line 128, in
Request
imp.load_source(Name, Env.APPL_PHYSICAL_PATH +
'\pyisapie_default.py').Request
File D:\websites\pylonshw\python\\pyisapie_default.py, line 7, in
pylonsapp = loadapp('config:' + appdir + '/development.ini')
File C:\Python25\lib\site-packages\pastedeploy-1.3.1-py2.5.egg\paste
\deploy\loadwsgi.py, line 193, in loadapp
return loadobj(APP, uri, name=name, **kw)
File C:\Python25\lib\site-packages\pastedeploy-1.3.1-py2.5.egg\paste
\deploy\loadwsgi.py, line 213, in loadobj
global_conf=global_conf)
File C:\Python25\lib\site-packages\pastedeploy-1.3.1-py2.5.egg\paste
\deploy\loadwsgi.py, line 237, in loadcontext
global_conf=global_conf)
File C:\Python25\lib\site-packages\pastedeploy-1.3.1-py2.5.egg\paste
\deploy\loadwsgi.py, line 267, in _loadconfig
return loader.get_context(object_type, name, global_conf)
File C:\Python25\lib\site-packages\pastedeploy-1.3.1-py2.5.egg\paste
\deploy\loadwsgi.py, line 393, in get_context
section)
File C:\Python25\lib\site-packages\pastedeploy-1.3.1-py2.5.egg\paste
\deploy\loadwsgi.py, line 415, in _context_from_use
object_type, name=use, global_conf=global_conf)
File C:\Python25\lib\site-packages\pastedeploy-1.3.1-py2.5.egg\paste
\deploy\loadwsgi.py, line 345, in get_context
global_conf=global_conf)
File C:\Python25\lib\site-packages\pastedeploy-1.3.1-py2.5.egg\paste
\deploy\loadwsgi.py, line 237, in loadcontext
global_conf=global_conf)
File C:\Python25\lib\site-packages\pastedeploy-1.3.1-py2.5.egg\paste
\deploy\loadwsgi.py, line 274, in _loadegg
return loader.get_context(object_type, name, global_conf)
File C:\Python25\lib\site-packages\pastedeploy-1.3.1-py2.5.egg\paste
\deploy\loadwsgi.py, line 542, in get_context
object_type, name=name)
File C:\Python25\lib\site-packages\pastedeploy-1.3.1-py2.5.egg\paste
\deploy\loadwsgi.py, line 562, in find_egg_entry_point
pkg_resources.require(self.spec)
File build/bdist.linux-i686/egg/pkg_resources.py, line 626, in
require
File build/bdist.linux-i686/egg/pkg_resources.py, line 524, in
resolve
pkg_resources.DistributionNotFound: decorator=2.1.0
Are there multiple versions of Python in the mix, and if so, is one of
them missing the decorator egg?
-jj
--
I, for one, welcome our new Facebook overlords!
http://jjinux.blogspot.com/
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---
AW: per request template path modification
-Ursprüngliche Nachricht- Von: pylons-discuss@googlegroups.com [mailto:[EMAIL PROTECTED] Im Auftrag von kai Gesendet: Mittwoch, 11. Juni 2008 14:40 An: pylons-discuss@googlegroups.com Betreff: per request template path modification Hi All, How do I modify the mako template path per request. I'm serving up multiple languages and and I have the i18n stuff working. However some of the markup needs to also vary depending on the language. so during the request cycle I want to change the path to something like path = ['/myapp/templates/fr','/myapp/templates/default'] in order to pickup a language customized template if it is there and take the default if it is not. I had the same idea, and I had to patch the mako file name resolution sources to get it working. There are more issues (inheritance resolution etc) which need to be considered. The lang_code + templatename isn't really useful, either. AFAIK there is no really beatiful directory based focused solution for supporting multiple languages inside of MAKO. Unfortunatly. Andrew --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: initialise session in functional test
Previously Shannon -jj Behrens wrote: On Sun, Jun 8, 2008 at 3:36 AM, Wichert Akkerman [EMAIL PROTECTED] wrote: Does anyone have tips or example code for getting some test data into a (beaker) session so a controller that's being tested can pick that up? If I were in your shoes, I would do some variation of the following: * Put a password in your .ini file. * Create a controller method that checks whether the md5 of some query parameter matches the md5 of the password. If so, it loads the session. I don't want to add controller methods just to be able to run a unittest. I want to either get some real data in or add some mocks somewhere so I can test my controller in isolation. I'm not trying to be difficult - I just don't want to turn a unittest into an integration test. Wichert. -- Wichert Akkerman [EMAIL PROTECTED]It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Exception email reporting in production
Hi, I'm having trouble getting pylons to report errors via email where debug = false - basically nothing ever turns up. In production.ini there are some lines like: email_to = [EMAIL PROTECTED] smtp_server = localhost error_email_from = [EMAIL PROTECTED] I can manually set up smtplib and send emails from localhost via controllers without a problem so I'm a bit mystified why this is failing. No errors appear on the console either. On my development site which does not have a mail server, setting debug to false and causing an exception dumps some output to the console to do with smtplib failing: Additionally an error occurred while sending the lt;paste.exceptions.reporter.EmailReporter object... File /System/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/smtplib.py, line 310, in connect raise socket.error, msg error: (49, Can't assign requested address) so I'm doubly confused why at least something isn't appearing anywhere. Any hints as to where to start looking? Cheers Ben --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Bulk inserts
Depending on the locking behaviour of the bulk loader it could make sense to load the data into simple preprocessing tables which don't have any validation logic or primary key constraints. From these tables the final processing is then done. This would move the error handling from the bulk loader into the procedures inside the database, and solve bulk loading specific issues. We have also used this method to process ~10K record batch imports from a CSV file that is uploaded through a web interface. We were using MSSQL, but the theory would be the same and has worked very well for us. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Truncating an html string safely
On Jun 12, 5:13 am, Mike Orr [EMAIL PROTECTED] wrote: Although again, we have two issues. One is HTML-to-text (essentially lynx-as-a-function). The other is truncating an HTML string while keeping it well-formed (which means not stopping in the middle of a tag and closing any open tags). Here is another sanitizer (I think from something having to do with Zope): http://www.koders.com/python/fidFB51F4D2D89CC1397608213E09F11404D9B21059.aspx --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Truncating an html string safely
On Jun 12, 5:13 am, Mike Orr [EMAIL PROTECTED] wrote: Although again, we have two issues. One is HTML-to-text (essentially lynx-as-a-function). The other is truncating an HTML string while keeping it well-formed (which means not stopping in the middle of a tag and closing any open tags). Actually, I think we may have four issues...? 1) truncate HTML and end up with well-formed HTML. 2) strip all HTML tags (without an interest in text formatting) 3) html2text (trying to keep text formatting with p, block, etc.) 4) sanitizing HTML (not directly discussed here, but a good implementation of this will be helpful, increase security, and should be able to be extended trivially to provide #2, striping all HTML tags). --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Mako migration strategies
On Jun 11, 10:09 pm, Mike Orr [EMAIL PROTECTED] wrote: Dispatch to another function via a keyword argument? I'm not sure what that means. That means... The render function takes an engine name or uses the default engine, and invokes the corresponding plugin. which is where my confusion is... i thought the current functionality was default_engine = 'mako' def render( template , engine=None ): if not engine: engine = default_engine return 'render using the engine plugin / function / method / whatever' but the new functionality is no longer having a render() that wraps all the plugins/etc like that? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Bulk inserts
one of the things i HATE about mysql, is the 5.0 branch , nothing failed. they specifically wanted everything to go though, and valued that behavior over integrity. in terms of what happens when something does fail now? I'm pretty sure the mysql and postgres docs have workarounds/solutions for that. i've never had to code for that , as i've only dealt with data dumps. i meant to add in my original post- you can use a log rotation system to just change the 'write to' file every set interval, and then work off that. i think there is a python class that handles that automagically. i didn't think of the threads issues myself. locking would need to be done as mentioned above. two things i'd bring up: - doing something similar in mod_perl, i've had the logging done in a 'cleanup' handler that runs after the connection closes. this can often free up some blocking issues. - IIRC, a friend tried something similar where his apps were posting the data to an central twisted daemon on his lan. that daemon then handled all of the logging , rotating, and misc management. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Application keeps 'stalling'
this is an ab bug actually, see: https://issues.apache.org/bugzilla/show_bug.cgi?id=21495 you might want to update ab or use httperf for benchmarking On Jun 4, 7:44 pm, Jonathan Vanasco [EMAIL PROTECTED] wrote: An application i have running keeps randomly quitting or stalling. Quitting- I'm tracing a ton of stuff, I can't seem to figure out why its dying. I'm hoping monit can at least restart the next time. Stalling- Ive been running ab to bench and test it, hoping to find the quit. Running ab -n 1 -c 10 http://site.com/contact; I will always get this message around 4220 requests if i go through nginx , 8440 requests if i hit paster directly ( +- 3 requests over 30 tests -- very weird. i've never seen a bunching of results like this before ) apr_poll: The timeout specified has expired (70007) The site will be frozen for a minute or so, then be responsive again. Looking at the paster logs, no issues exist. Looking at pg_stat activity, there's no transaction - we're idle. anyone have a clue? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Application keeps 'stalling'
On Jun 12, 6:22 pm, ionel [EMAIL PROTECTED] wrote: this is an ab bug actually, see:https://issues.apache.org/bugzilla/show_bug.cgi?id=21495 you might want to update ab or use httperf for benchmarking I thought that ab might be at fault, but the website was unreachable by other means too. Maybe ab was maxing out my connecitons? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Exception email reporting in production
On Thu, Jun 12, 2008 at 3:48 AM, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Hi, I'm having trouble getting pylons to report errors via email where debug = false - basically nothing ever turns up. In production.ini there are some lines like: email_to = [EMAIL PROTECTED] smtp_server = localhost error_email_from = [EMAIL PROTECTED] I can manually set up smtplib and send emails from localhost via controllers without a problem so I'm a bit mystified why this is failing. No errors appear on the console either. On my development site which does not have a mail server, setting debug to false and causing an exception dumps some output to the console to do with smtplib failing: Additionally an error occurred while sending the lt;paste.exceptions.reporter.EmailReporter object... File /System/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/smtplib.py, line 310, in connect raise socket.error, msg error: (49, Can't assign requested address) so I'm doubly confused why at least something isn't appearing anywhere. Any hints as to where to start looking? If I were in your shoes, I'd try a few things: * Try using smtplib from within a controller. Does it work? * Try using pdb to debug your problem, per my instructions here: http://jjinux.blogspot.com/2007/08/python-coding-in-debugger-for-beginners.html Happy Hacking! -jj -- I, for one, welcome our new Facebook overlords! http://jjinux.blogspot.com/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
