Re: Pylons in production
Hello, I would like to reply to this post from my situation. On Jan 2, 12:50 pm, Ian Bicking [EMAIL PROTECTED] wrote: Matt Haggard wrote: I'm (in the process of) using pylons in a production environment and still have a few questions: 1) Do I really have to restart the server every time I make a change (controller/model)? I can foresee making many changes and some which require immediate attention, but restarting the server in the middle of the day would not be an option. So, do I have to make changes at night? Is there no way for the server to adopt the changes without killing all connections? When restarting any connections in-progress will be completed, and new connections will be rejected. If you are using a frontend like Apache that has something to retry connections, no actual requests will fail during a restart. In my situation, Apache is not an option I am able to utilize with Pylons as the current infrastructure caters more to being a Microsoft shop (of which I am fortunate using Python is even an option!). I believe I'm left with either one of two options: 1) A Python wrapper around paster that could perform the 'retry connections' (as I see this as a possible concern point for application users); or 2) Use IIS, which as I've found from prior posts, seems far more problematic with numerous issues that still need to be worked out (i.e. no reload option, must function as a Virtual Directory and for some reason does not correctly work with AuthKit -- another show- stopper). What I find very troubling about Pylons on Windows -- though I'm not sure if its Windows in particular or paster as well is even when the the source code does not change and simply some changes are done on a web form, after clicking the submit button for the form sometimes returns a 'Page Cannot Be Displayed' message when it should actually be showing the requested page as it does exist -- though the update seems to have occurred from the form. As far as I can see, this seems to be part of the same issue with the 'retry connections'. Thanks. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Pylons in production
Thank you Graham, I'll look into mod_wsgi -- that looks very promising. I have another follow-up question about multiple-app design -- and I'll give the whole story so it makes sense where I'm coming from: Currently, we have one big database with many tables, then a mess of pages that do various functions (I think this is pretty typical of many websites). We're moving to a MVC approach because it seems to make more sense (routes/orm/etc). Now I'm having trouble deciding what to split into separate apps, what to split into controllers, how to share models/templates/static files between the apps. I've just gotten to the point that I have (working) common directories for templates, template filters, models and static files. But upon reaching that point, I step back and wonder, Does it make more sense to combine these apps into a single app with several controllers? Am I completely misusing the Pylons model? For instance... 1) Every app will require authentication - which is based on the users table in the (common) database... and I import user.py (from the common model dir) for that purpose. 2) Every app will use the same db. That means that I can't run (or can I?) an app by itself, because it requires a db that resides up a level. I guess my question boils down to (at least in part): Should an app be a standalone app? Thanks, Matt Haggard On Jan 2, 8:47 pm, Graham Dumpleton [EMAIL PROTECTED] wrote: On Jan 3, 5:14 am, Matt Haggard [EMAIL PROTECTED] wrote: I'm (in the process of) using pylons in a production environment and still have a few questions: 1) Do I really have to restart the server every time I make a change (controller/model)? I can foresee making many changes and some which require immediate attention, but restarting the server in the middle of the day would not be an option. So, do I have to make changes at night? Is there no way for the server to adopt the changes without killing all connections? 2) The system will be composed of multiple apps... (perhaps as many as 20 or more) so would it be better (if I need to restart after every change) to have all the apps served by their own servers? I mean, is it better to have 20 instances of the paster server if uptime is such a priority? 3) In serving several apps (thanks to Ian Bicking for the help to get that going!) like this: [composite:main] use = egg:Paste#urlmap / = config:home_app/development.ini /app2 = config:app2/development.ini ... how do I make app2 aware that all of his urls are based off of /app2? well... actually, it is already aware of that. I'm really asking: Is there a way for app2 to know that the home directory is just plain / ? This comes up because I use a shared repository for images and when I reference images from app2 with src=images/something.jpg it looks for the image at the urlhttp://127.0.0.1:5000/app2/images/something.jpg instead ofhttp://127.0.0.1:5000/images/something.jpg. Also, I'd like for app2 to be able to link back to / without explicitly hard-coding that url (I can see in the future bumping everything up a directory... making / into /section and /app2 into /section/app2) Am I just going about this the wrong way? Many Pylons folks have an aversion to mod_wsgi and will refuse to use it on religious grounds without even trying it, but the problems you are trying to solve are exactly the sort of the things that mod_wsgi is really good for. This is because mod_wsgi handles both the restart case as well as the problem of splitting up a monolithic application to run across multiple distinct processes. The steps required to do this are relative easy and don't require having to setup lots of distinct server instances each with a supervisor system as Apache/ mod_wsgi will handle that all for you automatically. Starting with separation of applications into distinct processes to allow them to be individually restart, one would take your composite application and create separate .ini files for each application. One would just then mount them as separate WSGI applications. For example: WSGIScriptAlias /app2 /some/path/app2.wsgi WSGIScriptAlias / /some/path/home_app.wsgi Leave it like that and all you have done is separated the applications so that they run within distinct sub interpreters in the same processes. Add daemon processes to the mix though and you can delegate each to run in a separate process and/or processes. WSGIDaemonProcess app2 threads=25 WSGIScriptAlias /app2 /some/path/app2.wsgi Location /app2 WSGIProcessGroup app2 WSGIReloadMechanism Process /Location WSGIDaemonProcess home_app threads=25 WSGIScriptAlias / /some/path/home_app.wsgi Location / WSGIProcessGroup home_app WSGIReloadMechanism Process /Location Each application now runs in a separate process with 25 threads. Having enabled process reload mechanism available with mod_wsgi 2.0
Re: Pylons in production
On Jan 4, 6:14 am, Matt Haggard [EMAIL PROTECTED] wrote: Thank you Graham, I'll look into mod_wsgi -- that looks very promising. I have another follow-up question about multiple-app design -- and I'll give the whole story so it makes sense where I'm coming from: Currently, we have one big database with many tables, then a mess of pages that do various functions (I think this is pretty typical of many websites). We're moving to a MVC approach because it seems to make more sense (routes/orm/etc). Now I'm having trouble deciding what to split into separate apps, what to split into controllers, how to share models/templates/static files between the apps. I've just gotten to the point that I have (working) common directories for templates, template filters, models and static files. But upon reaching that point, I step back and wonder, Does it make more sense to combine these apps into a single app with several controllers? Am I completely misusing the Pylons model? For instance... 1) Every app will require authentication - which is based on the users table in the (common) database... and I import user.py (from the common model dir) for that purpose. 2) Every app will use the same db. That means that I can't run (or can I?) an app by itself, because it requires a db that resides up a level. I guess my question boils down to (at least in part): Should an app be a standalone app? If you want process separation for parts of your application but don't quite know if they should be split into actual separate applications, mod_wsgi still allows you to delegate just parts of an application to distinct daemon processes without breaking apart the code for the application. For example, you might have: WSGIScriptAlias / /some/path/thewholeapplication.py which maps to single .ini file for Pylons which may or may not use composition to binding multiple applications together. Using WSGIDaemonProcess/WSGIProcessGroup/Location directives as shown previously, you can arbitrarily delegate certain URL or subsets of URLs to different processes. For example, irrespective of application lines, imagine that you had certain URLs which did a lot of PDF file generation on the fly using reportlab and because of what was being done it would take up lots of memory whereas other parts of the application don't. You could delegate just those URLs to be handled in their own daemon process and then set a low number of maximum requests for that process so that it is recycled on a regular basis to reclaim memory. You could also set an inactivity timeout so that it is recycled if no requests received for a period. Thus: # Run main Pylons application in Apache child process for best performance. # The default configuration results in that happening so nothing to do except # mount the application on root. WSGIScriptAlias / /some/path/thewholeapplication.py # Create a daemon process for memory hungry PDF generation code. WSGIDaemonProcess memory-hungry-pdfs threads=5 maximum-requests=50 inactivity-timeout=30 # Delegate the memory hungry PDF URLs to the daemon process. Location /memory/hungry/urls/root WSGIProcessGroup memory-hungry-pdfs /Location Thus, rather than all processes being used for your main application being bloated out by the memory hungry parts of an application, you can isolate that stuff an keep overall memory down. The only issue in respect of your original goals about code reloading is that because there aren't separate application script files you can't just touch a script file for a specific application and have it be restarted. In the example above I put the main part of the application in the Apache child processes anyway, which ould required a full Apache restart to reload main part of application. To force reloading of memory hungry URL process, then you need to work out the pid and send it a SIGINT. The pid would be identified from Apache error log messages provided that LogLevel set to info. Reason pid is found in that way is that it still shows as 'httpd' process and so if all running as Apache user you can tell it is the daemon process from 'ps'. One feature of daemon processes is that one can make process run as distinct user. This would allow it to be easily identified in ps output, but more importantly you could run different parts of your application code as different users as well if they have special access requirements. Anyway, few more ideas for you there. :-) Graham On Jan 2, 8:47 pm, Graham Dumpleton [EMAIL PROTECTED] wrote: On Jan 3, 5:14 am, Matt Haggard [EMAIL PROTECTED] wrote: I'm (in the process of) using pylons in a production environment and still have a few questions: 1) Do I really have to restart the server every time I make a change (controller/model)? I can foresee making many changes and some which require immediate attention, but restarting the server in the middle of the day would not be an option
Re: Pylons in production
huh... it seems like it is finding the image after all (Question 3)... weird On Jan 2, 11:14 am, Matt Haggard [EMAIL PROTECTED] wrote: I'm (in the process of) using pylons in a production environment and still have a few questions: 1) Do I really have to restart the server every time I make a change (controller/model)? I can foresee making many changes and some which require immediate attention, but restarting the server in the middle of the day would not be an option. So, do I have to make changes at night? Is there no way for the server to adopt the changes without killing all connections? 2) The system will be composed of multiple apps... (perhaps as many as 20 or more) so would it be better (if I need to restart after every change) to have all the apps served by their own servers? I mean, is it better to have 20 instances of the paster server if uptime is such a priority? 3) In serving several apps (thanks to Ian Bicking for the help to get that going!) like this: [composite:main] use = egg:Paste#urlmap / = config:home_app/development.ini /app2 = config:app2/development.ini ... how do I make app2 aware that all of his urls are based off of /app2? well... actually, it is already aware of that. I'm really asking: Is there a way for app2 to know that the home directory is just plain / ? This comes up because I use a shared repository for images and when I reference images from app2 with src=images/something.jpg it looks for the image at the urlhttp://127.0.0.1:5000/app2/images/something.jpg instead ofhttp://127.0.0.1:5000/images/something.jpg. Also, I'd like for app2 to be able to link back to / without explicitly hard-coding that url (I can see in the future bumping everything up a directory... making / into /section and /app2 into /section/app2) Am I just going about this the wrong way? I appreciate any help you could give me... Matt Haggard --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Pylons in production
Matt Haggard wrote: I'm (in the process of) using pylons in a production environment and still have a few questions: 1) Do I really have to restart the server every time I make a change (controller/model)? I can foresee making many changes and some which require immediate attention, but restarting the server in the middle of the day would not be an option. So, do I have to make changes at night? Is there no way for the server to adopt the changes without killing all connections? When restarting any connections in-progress will be completed, and new connections will be rejected. If you are using a frontend like Apache that has something to retry connections, no actual requests will fail during a restart. 2) The system will be composed of multiple apps... (perhaps as many as 20 or more) so would it be better (if I need to restart after every change) to have all the apps served by their own servers? I mean, is it better to have 20 instances of the paster server if uptime is such a priority? You might want those apps separate to make them more maintainable (e.g., avoid breaking app A when you upgrade app B). I'd try to get a setup where restarts don't lose any connections, then for uptime it's not an issue. If you want to test how it's working, probably you could fire up something like ab (the apache benchmark app) and see if everything works right when you restart your app during the run. -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Pylons in production
Thanks! I have a follow-up on question 3: Is there a way for the separate apps to be aware of the urls of the other apps? Or for the server to provide a list of the apps/urls to each app through the config? On Jan 2, 11:50 am, Ian Bicking [EMAIL PROTECTED] wrote: Matt Haggard wrote: I'm (in the process of) using pylons in a production environment and still have a few questions: 1) Do I really have to restart the server every time I make a change (controller/model)? I can foresee making many changes and some which require immediate attention, but restarting the server in the middle of the day would not be an option. So, do I have to make changes at night? Is there no way for the server to adopt the changes without killing all connections? When restarting any connections in-progress will be completed, and new connections will be rejected. If you are using a frontend like Apache that has something to retry connections, no actual requests will fail during a restart. 2) The system will be composed of multiple apps... (perhaps as many as 20 or more) so would it be better (if I need to restart after every change) to have all the apps served by their own servers? I mean, is it better to have 20 instances of the paster server if uptime is such a priority? You might want those apps separate to make them more maintainable (e.g., avoid breaking app A when you upgrade app B). I'd try to get a setup where restarts don't lose any connections, then for uptime it's not an issue. If you want to test how it's working, probably you could fire up something like ab (the apache benchmark app) and see if everything works right when you restart your app during the run. -- Ian Bicking : [EMAIL PROTECTED] :http://blog.ianbicking.org --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Pylons in production
Matt Haggard wrote: Thanks! I have a follow-up on question 3: Is there a way for the separate apps to be aware of the urls of the other apps? Or for the server to provide a list of the apps/urls to each app through the config? No, you'll just have to tell each app about where the other app is. -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Pylons in production
On Jan 3, 5:14 am, Matt Haggard [EMAIL PROTECTED] wrote:
I'm (in the process of) using pylons in a production environment and
still have a few questions:
1) Do I really have to restart the server every time I make a change
(controller/model)? I can foresee making many changes and some which
require immediate attention, but restarting the server in the middle
of the day would not be an option. So, do I have to make changes at
night? Is there no way for the server to adopt the changes without
killing all connections?
2) The system will be composed of multiple apps... (perhaps as many as
20 or more) so would it be better (if I need to restart after every
change) to have all the apps served by their own servers? I mean, is
it better to have 20 instances of the paster server if uptime is such
a priority?
3) In serving several apps (thanks to Ian Bicking for the help to get
that going!) like this:
[composite:main]
use = egg:Paste#urlmap
/ = config:home_app/development.ini
/app2 = config:app2/development.ini
...
how do I make app2 aware that all of his urls are based off of /app2?
well... actually, it is already aware of that. I'm really asking: Is
there a way for app2 to know that the home directory is just
plain / ? This comes up because I use a shared repository for images
and when I reference images from app2 with src=images/something.jpg
it looks for the image at the
urlhttp://127.0.0.1:5000/app2/images/something.jpg
instead ofhttp://127.0.0.1:5000/images/something.jpg. Also, I'd like
for app2 to be able to link back to / without explicitly hard-coding
that url (I can see in the future bumping everything up a directory...
making / into /section and /app2 into /section/app2)
Am I just going about this the wrong way?
Many Pylons folks have an aversion to mod_wsgi and will refuse to use
it on religious grounds without even trying it, but the problems you
are trying to solve are exactly the sort of the things that mod_wsgi
is really good for. This is because mod_wsgi handles both the restart
case as well as the problem of splitting up a monolithic application
to run across multiple distinct processes. The steps required to do
this are relative easy and don't require having to setup lots of
distinct server instances each with a supervisor system as Apache/
mod_wsgi will handle that all for you automatically.
Starting with separation of applications into distinct processes to
allow them to be individually restart, one would take your composite
application and create separate .ini files for each application. One
would just then mount them as separate WSGI applications. For example:
WSGIScriptAlias /app2 /some/path/app2.wsgi
WSGIScriptAlias / /some/path/home_app.wsgi
Leave it like that and all you have done is separated the applications
so that they run within distinct sub interpreters in the same
processes. Add daemon processes to the mix though and you can delegate
each to run in a separate process and/or processes.
WSGIDaemonProcess app2 threads=25
WSGIScriptAlias /app2 /some/path/app2.wsgi
Location /app2
WSGIProcessGroup app2
WSGIReloadMechanism Process
/Location
WSGIDaemonProcess home_app threads=25
WSGIScriptAlias / /some/path/home_app.wsgi
Location /
WSGIProcessGroup home_app
WSGIReloadMechanism Process
/Location
Each application now runs in a separate process with 25 threads.
Having enabled process reload mechanism available with mod_wsgi 2.0,
to restart an application all you need to do is touch (ie., change
modification time) of the WSGI script file for that application and it
wll be automatically restarted prior to next request it receives. The
restart of that application doesn't affect other applications not
require Apache as a whole to be restarted.
The contents of those WSGI script files is pretty trivial and just
setup the WSGI application entry point.
import os, sys
sys.path.append('/usr/local/pylons/home_app')
from paste.deploy import loadapp
application = loadapp('config:/usr/local/pylons/home_app/
development.ini')
Although relatively easy to setup, it should do pretty well what you
want.
Graham
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
pylons-discuss group.
To post to this group, send email to pylons-discuss@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/pylons-discuss?hl=en
-~--~~~~--~~--~--~---
