[OT] backend processing of large jobs
I have what is probably a pretty common situation. The user submits a file for processing to my application. It takes many minutes to process this file, so I would like to submit this file as a job to a backend server that can queue up the files as they come in, process them, maintain status information on the queue and individual jobs, and finally deposit the results in a database. I would like to be able to query this backend job server for status on a job (by a job_id) to allow for the pylons web application to check ad lib for job status/completion. I have looked at constructing a standalone XML-RPC server or using twisted, but there are other ways, I am sure. Any suggestions? Thanks, Sean --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to [EMAIL PROTECTED] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: [OT] backend processing of large jobs
On 4/12/07, Alberto Valverde [EMAIL PROTECTED] wrote: On Apr 12, 2007, at 2:33 PM, Sean Davis wrote: I have what is probably a pretty common situation. The user submits a file for processing to my application. It takes many minutes to process this file, so I would like to submit this file as a job to a backend server that can queue up the files as they come in, process them, maintain status information on the queue and individual jobs, and finally deposit the results in a database. I would like to be able to query this backend job server for status on a job (by a job_id) to allow for the pylons web application to check ad lib for job status/completion. I have looked at constructing a standalone XML-RPC server or using twisted, but there are other ways, I am sure. Any suggestions? I've been using http://cheeseshop.python.org/pypi/threadpool/1.2.2 with Pylons' XML-RPC server for something similar succesfully. Thanks, Alberto. That is what I was thinking. Just to followup, if I may, but where do you keep your threadpool so that all requests have access to it? In a global variable? Sean --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to [EMAIL PROTECTED] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Pylon with Apache
Hi! We have used Apache with modpython3. We have used our owned python code as framework, and we used DBISAM databases with ODBC connections. Now I read about Pylons and Django. As I see the DJANGO not supports the ODBC connection. So I want to ask about Pylons: Can I use Pylons under Apache? We have PHP, Python sites (8-10 virtual hosts) and we want to replace only one or two projects with frameworks. But we don't want to use another port, we must use the port 80. So if we want to use Pylons, we need to transport these packages, or need to make Pylons to works under apache. Can we do these things? And: is Pylons supports the ODBC connections? Thanks for your help: dd --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to [EMAIL PROTECTED] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: [OT] backend processing of large jobs
On Apr 12, 2007, at 4:25 PM, Sean Davis wrote: Thanks, Alberto. That is what I was thinking. Just to followup, if I may, but where do you keep your threadpool so that all requests have access to it? In a global variable? place it at yourapp/lib/app_globals:Globals binding it to self inside __init__. You'll have it availbale at pylons.g Alberto --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to [EMAIL PROTECTED] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Response postprocessing
I found two use cases where it would be nice to postprocess the Response object in the base controller: 1) To do the Tidy validation in my previous email. 2) To apply a site template independent of the controller templates. (Useful if the regular templates use multiple engines, so that you don't have to maintain the same site template for every engine.) However, the base controller just does this: return WSGIController.__call__(self, environ, start_response) This means the Response comes and goes before you can see it. I got it to work by overriding the superclass method: === # start_response and routes_dict bla bla bla # I don't use .__before__ or .__after__ response = self._dispatch_call() # My code here if isinstance(response, basestring): c.content = response response = render(genshi, site_genshi) else: c.content = response.content response.content = render(genshi, site_genshi) # End my code if hasattr(response, 'wsgi_response'): # Copy the response object into the testing vars if we're testing if 'paste.testing_variables' in environ: environ['paste.testing_variables']['response'] = response return response(environ, start_response) elif isinstance(response, basestring): resp = pylons.Response(response) return resp(environ, start_response) return response === This works but it's a little annoying that I have to check for both string and Response, because the other place it's tested is inside an 'elif'. Also I don't understand, why do the two special cases call the response before returning it, while the last case just returns the response. I'd suggest cleaning this up so there's a point where there's a Response object exists in all cases, and then calling: self.__postprocess_response(self, resp) -- Mike Orr [EMAIL PROTECTED] --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to [EMAIL PROTECTED] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Response postprocessing
On Apr 12, 2007, at 3:24 PM, Mike Orr wrote: I found two use cases where it would be nice to postprocess the Response object in the base controller: There was a ticket to add something like this too: http://pylonshq.com/project/pylonshq/ticket/201 So that the Response object is available in __after__ for post- processing. Would that work? Cheers, Ben --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to [EMAIL PROTECTED] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: ANN: Pylons 0.9.5 Released
Awesome. Great work to all! As a newbie to Pylons, I can see that the folks in the IRC channel have been a great help and give me much to be excited about in my coming work with Pylons. Specifically, I appreciate the quick work on paster in Windows. Thanks for all the hard work! /alex On 4/12/07, Ben Bangert [EMAIL PROTECTED] wrote: I'm thrilled to announce a new release of Pylons, with one of the largest ticket fix counts of any Pylons release thus far. This release and its associated updates have only been possible with the growing and active Pylons community. A big shout-out to the many contributions by Phil Jenvey, Ian Bicking, James Gardner, Mike Orr, Shannon -jj Behrens, David Smith, Graham Higgins, and Wyatt Baldwin for their work on Pylons, Routes the mail list, and the documentation. Many thanks to the numerous others that have reported bugs and helped out on IRC and the mail list over the past few months. Also, as Pylons uses so many contributed projects, there's work going on to better integrate documentation across the smaller projects that make up the whole. A first stab at this is being taken with the Python Web Documentation effort which we realize not every single 'python web' project will be using; but its our hope that smaller WSGI based web components will put their docs up so they can all be utilized in a single place. The new documentation wiki: http://docs.pythonweb.org/ And onto update news for 0.9.5! There's big change logs for both Pylons and Routes, with some important updates in WebHelpers and Paste as well. First, the big improvements in 0.9.5, Unicode and i18n: - Routes now translates utf-8 characters during both URL generation, and URL recognition, many thanks to David Smith for numerous patches - The Request now has an option so that all GET/POST arguments come in as unicode, with the decoding you prefer - i18n now uses code contributed from Aquarium by Shannon -jj Behrens for translation fall-backs and has browser language matching code - Optional lazy translation options contributed by David Smith - Routes implicit defaults (action=index, id=None) can be turned off per route with _explicit=True - Routes memory can also be disabled by setting the mapper to explicit=True, which will also disable the implicit defaults Next, some backward compatibility changes: Webhelpers: * WARNING: paginate now takes arguments intended for the collection object as query_args. This could affect backwards compatibility. This fixes a common issue that non-keyword arguments passed into paginate get eaten by paginate's keyword arguments instead of being in *args to go on to the collection. * WARNING: Due to a typo, the Text helper highlight function no longer highlights text with the CSS class name 'hilight' by default: it now uses the CSS class name 'highlight' instead. The function's 'hilighter' keyword argument has also been deprecated, use 'highlighter' instead. Pylons: * WARNING: Pylons now requires the decorator module: it no longer packages it as pylons.decorator. Code relying on the pylons.decorator.decorator function will trigger a deprecation warning and should be changed to use decorator.decorator. * WARNING: pylons.h was deprecated for using projects' lib.helpers module directly in 0.9.3. pylons.h is now formally deprecated (emits DeprecationWarnings). Projects still accessing pylons.h must change the following import: from pylons import h to: import MYPROJ.lib.helpers as h And finally, the full changelog of the various parts that make Pylons what it is: PYLONS * Fixed paster shell breaking for projects where the base package was not the first package listed in top_level.txt. Patch from Alberto Valverde. Fixes #229. * Fixed doc references to config['app_conf']. Fixes #116. * Changed `get_engine_conf` to properly evaluate sqlalchemy echo statement when its 'debug'. Fixes #226. * make_session and create_engine now accept keyword arguments to pass to SQLAlchemy's create_engine. * make_session now accepts the keyword argument 'session_kwargs' to pass to SQLAlchemy's create_session. * Fixed _inspect_call to call function with keyword arguments instead of list args. Corrects issue with action defaults that caused the value for the latter args to be in the wrong spots. Spotted by Topher. Fixes #223. * Added the allow_none option (passed to xmlrpc.dumps) to XMLRPCController. Suggested by Jaroslaw Zabiello. * Updated XMLRPC Controller with patch for name lookup and additional unit tests for the patch. Fixes #216. * Updated docs for validate decorator to more clearly illustrate what the post_only args apply to. Fixes #221. * Added ability to return strings in the WSGIController. Fixes #218. * Added lazy i18n translation functions. Patch from David Smith. Fixes #181. * Added fix for
