> Date: Tue, 3 Mar 2009 15:26:50 -0800
> From: [email protected]
> To: [email protected]
> Subject: Re: Pylons, HTTP 201 Accepted, Task Queues and Background threads
>
>
> chris van wrote:
> > > Date: Tue, 3 Mar 2009 12:57:39 -0800
> > > Subject: Pylons, HTTP 201 Accepted, Task Queues and Background threads
> > > From: [email protected]
> > > To: [email protected]
> > >
> > > Hi everyone,
> > >
> > > I'm trying to find some docs or perhaps old discussions about
> > > implementing a task queue within a pylons application. The scenario
> > > I'm trying to support involves a request coming into the app server to
> > > perform an action which takes a long time to complete, such as
> > > rebuilding an index or updating a value across hundreds of thousands
> > > of objects.
> > >
> > > My thought was to create a processing thread when the app is loaded.
> > > When I get the request I add the item to a synchronized queue (which
> > > the processing thread blocks on) and return a HTTP 201 Accepted to the
> > > client. The processing thread picks up tasks from the queue and they
> > > are completed in the order received. The 201 response also has an
> > > additional Location header to poll the status of the task.
> > >
> > > The question that remained was how to create and manage processing
> > > thread. I've read a couple of threads on this subject, and hunted
> > > around google a bit and found a couple of options:
> > > -
> > >
> > http://groups.google.com/group/pylons-discuss/browse_thread/thread/e30fb912ca79b000/7cc1d4a6b1d9919d?lnk=gst&q=background#7cc1d4a6b1d9919d
> > > -
> > >
> > http://groups.google.com/group/pylons-discuss/browse_thread/thread/3e9dfda05af50634/bc914b96e2b96a1b?lnk=gst&q=background#bc914b96e2b96a1b
> > >
> > > Now I'm leaning towards creating a process using the python
> > > multiprocessing module which interfaces like a thread but skips issues
> > > with the GIL and pylons thread management. However, I didn't find any
> > > information about how to manage the process lifecycle and allow it to
> > > shutdown gracefully when the server is stopped.
> > >
> > > I'd appreciate feedback on this approach and any pointers to resources
> > > that will allow me to hook into the app lifecycle and manage my
> > > subprocess as well. Hopefully I can get a working recipe out of this
> > > and put it all together in the pylons cook book for future reference.
> > >
> > > Thanks.
> >
> > G'day,
> >
> > I know nothing about how to solve your problem, though it does sound
> > like something that interests me and which I would like to do myself. I
> > currently have a controller feeding data into a shared-memory circular
> > buffer, from which an entirely separate application reads and executes
> > in it's own time. I return a 400 OK to the client as soon as the
> > controller is finished validating the request and stores it in the
> > buffer; the client must then poll the server to get updates on the
> > status of the request once it is in the buffer. Returning 201 Accepted
> > sounds like a better solution, could you please let me know how
> > successful you are in implementing it under pylons and how it works for you?
> >
> > Thanks a lot,
>
> Chris,
>
> Two questions? What happens to the external application when the pylons
> application is shut down? Does it empty the buffer and block for new input or
> do
> you have a way to shut down the external application from pylons? Secondly,
> do
> clients poll the pylons server or the external application for status?
>
> I'm guessing you meant you return 200 OK not 400. Whether you choose to
> return a
> 200 or 201 status is mostly an aesthetic choice, though 201 is a clearer
> indication of the result. If you're trying to change the response from 200 to
> 201 you can manipulate the pylons.response to set its status_code to 201.
> Hope
> this helps you out.
>
> Kochhar
G'day Kochar,
In my ideal implementation, neither pylons application or the external
application will ever be shut down ;-) In reality, if the external application
is still running then it will empty the buffer and wait for new input. If the
external application is shutdown and pylons is still running the buffer will
fill as requests are received and then reject new input once it is full. Most
of the time, if only one application is running, then the other should be too
(or will be shortly). I guess this differs somewhat from the original message,
where it is desired to shut-down the background process if the server is
stopped. It wouldn't be wrong for my system to behave this way I guess, but I
prefer to keep the two seperate (which is why the external application does not
start/is not started by Pylons).
You got me on the 200/400, I've coded a lot more 400 responses using the abort
command for when new input requests are found to be syntactically invalid. 202
Accepted is the code I would like to be returning, though I am not 100% sure
what to make of the w3c recommendation "The entity returned with this
response SHOULD include an indication of the request's current status
and either a pointer to a status monitor or some estimate of when the
user can expect the request to be fulfilled." I can give a time estimate
easily enough (average processing time per input X number of inputs in buffer)
but I can't see this being of much practical use. What is meant by a "pointer
to a status monitor"? The original message in this thread mentions "The 201
response also has an additional Location header to poll the status of the task"
which I take to be refering to the same thing, and it just means including a
location field in the http header according to
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.30
Clients poll the pylons server which returns information about the
state of the buffer and (with limited accuracy) the state of the
external application. Reading more about this 202 response code, it
looks like I should provide a little more detail in the status monitor
about the progress of the specific request that generated the 202
message in relation to other items in the buffer.
Any suggestions about what format the status monitor should be in? At the
moment it is just going to be an XHTML page with fields identifying the number
of items in the buffer, position in the buffer and expected time in seconds
till execution (assuming the external application is running properly).
This is my first real web application, and as I am trying to control a physical
device, it is just a little different from the standard database-based web
applications most documentation is geared to, but I'm trying to make it as
standards complient and RESTful as I can manage, whilst struggling to
comprehend the somewhat esoteric concepts involved.
Chris.
_________________________________________________________________
Looking to change your car this year? Find car news, reviews and more
http://a.ninemsn.com.au/b.aspx?URL=http%3A%2F%2Fsecure%2Dau%2Eimrworldwide%2Ecom%2Fcgi%2Dbin%2Fa%2Fci%5F450304%2Fet%5F2%2Fcg%5F801459%2Fpi%5F1004813%2Fai%5F859641&_t=762955845&_r=tig_OCT07&_m=EXT
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"pylons-discuss" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/pylons-discuss?hl=en
-~----------~----~----~----~------~----~------~--~---
