Re: [Zope-dev] Early processing of request body (was: Proposal for optimized Blob handling)
Hi, Am Samstag, den 10.03.2007, 07:36 +0100 schrieb Dieter Maurer: Christian Theune wrote at 2007-3-7 22:05 +0100: ... If 5 seconds are spend in the application thread to untangle mime data which has nothing application-specific about it and then only 100ms or so in the application itself, I'd say there is a major overhead problem. But if the IO thread spends 5 seconds, then Zope will be unresponsive for 5 seconds -- for me (and hopefully others, too) a far more critical situation than a (single) worker spending 5 seconds... The IO (ZServer) thread should only perform minimal work in its asyncore callbacks -- each callback should return within a few milliseconds. My argument does not argue against different threads between the IO thread and the worker threads, just against giving the IO thread significant work (whether or not you consider it application specific). Ah. Maybe I didn't point this out enough. I was thinking about switching to a chunk-based approach on processing the request. That way I want to avoid having to process large file data twice. IMHO a variation of the FieldStorage could be implemented that processes the data line by line as it comes in. That would avoid the 5 sec. delay if you process it at once and should not be a problem in the IO thread. Christian -- gocept gmbh co. kg - forsterstraße 29 - 06112 halle/saale - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development signature.asc Description: Dies ist ein digital signierter Nachrichtenteil ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Early processing of request body (was: Proposal for optimized Blob handling)
Christian Theune wrote at 2007-3-7 22:05 +0100: ... If 5 seconds are spend in the application thread to untangle mime data which has nothing application-specific about it and then only 100ms or so in the application itself, I'd say there is a major overhead problem. But if the IO thread spends 5 seconds, then Zope will be unresponsive for 5 seconds -- for me (and hopefully others, too) a far more critical situation than a (single) worker spending 5 seconds... The IO (ZServer) thread should only perform minimal work in its asyncore callbacks -- each callback should return within a few milliseconds. My argument does not argue against different threads between the IO thread and the worker threads, just against giving the IO thread significant work (whether or not you consider it application specific). -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Early processing of request body (was: Proposal for optimized Blob handling)
This use-case is already covered by implementing ZPublisher.Iterators.IStreamIterator. You can return a stream iterator to Medusa and free the worker thread immediately. Stefan On 8. Mär 2007, at 05:40, Sidnei da Silva wrote: Hopefully something similar could be done for files being sent *out* of the application when they don't need any application processing anymore (ie, Blobs!). -- Anything that, in happening, causes something else to happen, causes something else to happen. --Douglas Adams ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Early processing of request body (was: Proposal for optimized Blob handling)
Hi, Am Donnerstag, den 08.03.2007, 01:40 -0300 schrieb Sidnei da Silva: On 3/7/07, Christian Theune [EMAIL PROTECTED] wrote: Right. This optimization is about leveraging the fact that in many situations the upstream bandwith is *much* lower than the IO bandwith to the disk. Another condition that I have (and I think this is the general pattern) is that application threads should be given back to the pool as quickly as possible. If 5 seconds are spend in the application thread to untangle mime data which has nothing application-specific about it and then only 100ms or so in the application itself, I'd say there is a major overhead problem. I had came to the same conclusion a couple weeks ago, somehow *wink*. Maybe we've been influenced by the same person. :) So if the uploaded file shouldn't be handled by the application thread, neither by the IO layer, then I guess we need a 'upload handling thread pool' of some sorts, whose sole purpose is to handle incoming requests that are large before it gets to the application thread while still outside the async IO layer. I really wonder whether that's necessary. Actually. I'll take a look around the other web frameworks and check how they do their REQUEST processing. Maybe I can learn something from that. Hopefully something similar could be done for files being sent *out* of the application when they don't need any application processing anymore (ie, Blobs!). We already have that. The FileStreamIterator allows you to hand out an iterable that will be used outside the application thread to stream data from. Christian -- gocept gmbh co. kg - forsterstraße 29 - 06112 halle/saale - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development signature.asc Description: Dies ist ein digital signierter Nachrichtenteil ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Early processing of request body (was: Proposal for optimized Blob handling)
On 3/8/07, Christian Theune [EMAIL PROTECTED] wrote: I really wonder whether that's necessary. Yeah, after re-reading Dieter's reply, I sort of wonder how big of a deal that is. Actually. I'll take a look around the other web frameworks and check how they do their REQUEST processing. Maybe I can learn something from that. I suspect you will find that they don't do anything special. You could look at how Apache or Squid does it though. Hopefully something similar could be done for files being sent *out* of the application when they don't need any application processing anymore (ie, Blobs!). We already have that. The FileStreamIterator allows you to hand out an iterable that will be used outside the application thread to stream data from. Right, I had something else in mind which is serving large data from the ZODB when you're done with the application logic. But that's another story. -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Early processing of request body (was: Proposal for optimized Blob handling)
I was looking through some publisher code and found that the `process_request` method which takes the request body as a file-like object and processes it as a FieldStorage happens within the application thread. This would be better if it happened beforehand because it can takes up time while a transaction is running and a thread is used although it doesn't require any application-specific code. AFAICT a modified version of FieldStorage would be required to allow line-wise consumption and parsing of data while it is being uploaded and then hand this into the request instead of stdin. However, the FieldStorage implementation is recursive and wasn't obvious to me at a first glance how much work it would be to replace this. Are there similar feelings it would be a good idea to do this kind of early line-wise processing of request bodies? Christian Am Mittwoch, den 07.03.2007, 17:44 +0100 schrieb Christian Theune: Hi, [modified slightly from a similar proposal to zope3-dev to match Zope 2's publisher] I'm writing up a proposal for the ZODB to make even more efficient Blob handling possible. This includes not copying the data from an uploaded file, but using a `link` operation when possible. However, the HTTPRequest class currently uses the default implementation of the cgi module's FieldStorage. I propose to create a small subclass to override the `make_file` method to use `NamedTemporaryFile` instead of `TemporaryFile` to allow the file being accessible from a filename so I can apply a `link` operation. Notice: The FieldStorage explicitly provides the `make_file` method to allow overriding in this sense. Does anybody feel like this would be a bad idea? Christian ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) -- gocept gmbh co. kg - forsterstraße 29 - 06112 halle/saale - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development signature.asc Description: Dies ist ein digital signierter Nachrichtenteil ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Early processing of request body (was: Proposal for optimized Blob handling)
Christian Theune wrote at 2007-3-7 20:09 +0100: I was looking through some publisher code and found that the `process_request` method which takes the request body as a file-like object and processes it as a FieldStorage happens within the application thread. This would be better if it happened beforehand because it can takes up time while a transaction is running and a thread is used although it doesn't require any application-specific code. In my view, it already now happens far too early, because it may raise exceptions and those exceptions are not handled by the standard_error_message usually used for error processing depending on the url. Therefore, if you move out things, you should take care that you move out only parts that cannot raise exceptions. Furthermore, you seem to propose to move work from a worker thread to the IO (i.e. ZServer) thread. I do not think that it is a good idea to put significant work on the IO thread. Note, that the IO thread is responsible to handle all IO. When you keep it busy with other tasks, it will not handle IO... -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Early processing of request body (was: Proposal for optimized Blob handling)
Hi, Am Mittwoch, den 07.03.2007, 21:48 +0100 schrieb Dieter Maurer: Christian Theune wrote at 2007-3-7 20:09 +0100: I was looking through some publisher code and found that the `process_request` method which takes the request body as a file-like object and processes it as a FieldStorage happens within the application thread. This would be better if it happened beforehand because it can takes up time while a transaction is running and a thread is used although it doesn't require any application-specific code. In my view, it already now happens far too early, because it may raise exceptions and those exceptions are not handled by the standard_error_message usually used for error processing depending on the url. Therefore, if you move out things, you should take care that you move out only parts that cannot raise exceptions. Ah. Interesting point! Furthermore, you seem to propose to move work from a worker thread to the IO (i.e. ZServer) thread. I do not think that it is a good idea to put significant work on the IO thread. Note, that the IO thread is responsible to handle all IO. When you keep it busy with other tasks, it will not handle IO... Right. This optimization is about leveraging the fact that in many situations the upstream bandwith is *much* lower than the IO bandwith to the disk. Another condition that I have (and I think this is the general pattern) is that application threads should be given back to the pool as quickly as possible. If 5 seconds are spend in the application thread to untangle mime data which has nothing application-specific about it and then only 100ms or so in the application itself, I'd say there is a major overhead problem. Christiabn -- gocept gmbh co. kg - forsterstraße 29 - 06112 halle/saale - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development signature.asc Description: Dies ist ein digital signierter Nachrichtenteil ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Early processing of request body (was: Proposal for optimized Blob handling)
On 3/7/07, Christian Theune [EMAIL PROTECTED] wrote: Right. This optimization is about leveraging the fact that in many situations the upstream bandwith is *much* lower than the IO bandwith to the disk. Another condition that I have (and I think this is the general pattern) is that application threads should be given back to the pool as quickly as possible. If 5 seconds are spend in the application thread to untangle mime data which has nothing application-specific about it and then only 100ms or so in the application itself, I'd say there is a major overhead problem. I had came to the same conclusion a couple weeks ago, somehow *wink*. Maybe we've been influenced by the same person. :) So if the uploaded file shouldn't be handled by the application thread, neither by the IO layer, then I guess we need a 'upload handling thread pool' of some sorts, whose sole purpose is to handle incoming requests that are large before it gets to the application thread while still outside the async IO layer. Hopefully something similar could be done for files being sent *out* of the application when they don't need any application processing anymore (ie, Blobs!). -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
