Re: [Zope3-dev] Re: URGENT RFC: Is anyone using response.write in Zope 3?
Jim Fulton wrote: I could certainly find evidence that you tried, but the implementation was actually buffering data in a string buffer until the request was finished. This was the case at least as early as spring of 2004. Even with more than 105 bytes output over a slow connection? That's the default threshold; see zope.server.adjustments.Adjustments.outbuf_overflow. It certainly worked at one time. In any case, it appears to be unnecessary now. Shane ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: URGENT RFC: Is anyone using response.write in Zope 3?
Shane Hathaway wrote:
Jim Fulton wrote:
I could certainly find evidence that you tried, but the implementation
was
actually buffering data in a string buffer until the request was
finished.
This was the case at least as early as spring of 2004.
Even with more than 105 bytes output over a slow connection? That's
the default threshold; see
zope.server.adjustments.Adjustments.outbuf_overflow. It certainly
worked at one time. In any case, it appears to be unnecessary now.
Christian Theune and I looked at this in early 2004 and came to the conclusion
that data were being stored in memory. I think we used the debugger to
determine this.
We weren't using a slow connection. A casual scan of the code suggests the
data *should* be stored in a temporary file, which made me doubt my assertions.
:)
I just verified though with the following view on X3.0:
class Big:
def __init__(self, context, request):
self.request = request
def big(self):
response = self.request.response
response.setHeader('content-type', 'text/plain')
response.setHeader('content-length', str(51*20*1000*100))
for i in range(20*1000*100):
response.write('x'*50+'\n')
That outputting lots of data causes memory to baloon.
In this case, on my machine, the virtual memory jumped
about 150 megs and the in-use memory jumped about 100 megs.
shrug
I then tried the following view with the trunk, whis is about the same
as 3.2:
import tempfile
class Big:
def __init__(self, context, request):
self.request = request
def big(self):
response = self.request.response
response.setHeader('content-type', 'text/plain')
f = tempfile.TemporaryFile()
for i in range(20*1000*100):
f.write('x'*50+'\n')
return f
Here the memory temporarily jumped about 7 megs and the output
came back about an order of magnitude faster.
I tried this with both the Twisted and ZServer servers and
got about the same result (although I got an error in the log
for ZServer that I need to look into).
Jim
--
Jim Fulton mailto:[EMAIL PROTECTED] Python Powered!
CTO (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: URGENT RFC: Is anyone using response.write in Zope 3?
Jim Fulton wrote: Shane Hathaway wrote: A pull strategy will be efficient for a lot more people. I don't know what you mean by this. I mean that the new strategy of sending open files to the publisher, which I call a pull strategy, will work better than pushing to temporary files. Which is why you did it. ;-) The idea is this: run a thread whose only responsibility is to stream image and file data out of the object database to browsers. The thread will maintain a permanently open database connection. Other threads will send open HTTP connections to this thread, asking it to finish the response. The special thread will get the binary data from the database using its own connection, which would play well with the ZODB cache, since we'd only ever have one copy of the image/file data in memory at a time. Contrast this with Zope 2's image and file serving, which loads image/file data into arbitrary database connections and immediately evicts them from the cache. It seems like having a special connection just for serving images/files would be a lot more efficient, especially when concurrency is high. This sounds too complicated to me. Well, ok. It seems easy to me. But you're the boss. Have you followed the ZODB blob work? I think this will provide great benefits for people who choose to store large (megabytes, not gigabytes or terabytes) files in the database. I think the people working on it could use some help finishing it, if you'd like to help. Ok. I just finished a big project and now I'm considering what to do next, so I'll put that on my list. Thanks. Shane ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: URGENT RFC: Is anyone using response.write in Zope 3?
Philipp von Weitershausen wrote: page 204, Example 12.24, line 17: Using the ``write()`` method of HTTP-based responses does not provide a performance advantage in Zope X3 3.0 and 3.1 and is not supported anymore in Zope 3.2 and higher. I would like to point out that response.write() originally did provide an advantage by transparently putting large output into a temporary file rather than holding it in RAM. This is a performance advantage for highly concurrent sites sending multiple megabytes over relatively slow links, since it allows precious database threads to be freed while the kernel optionally pages out the response until the client finishes downloading. However, the original response.write() does not provide a performance advantage in most situations because the original write() API pushes data; the data has nowhere to go except some temporary storage until the client is ready for it. Temporary storage is somewhat expensive. Pulling data instead is likely to provide a performance benefit in more situations. So I fully agree that the original write() should go (in fact I suppose it's gone already), but to say there was no performance advantage is imprecise. I spent a fair amount of time making write() fast, with some success. Shane ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: URGENT RFC: Is anyone using response.write in Zope 3?
Philipp von Weitershausen wrote: Shane Hathaway wrote: So I fully agree that the original write() should go (in fact I suppose it's gone already), but to say there was no performance advantage is imprecise. I spent a fair amount of time making write() fast, with some success. Interesting. Where exactly did this effort of making write() fast go into? Looking at X3 3.0's response.write() method, I basically see: def write(self, string): self._outstream.write(string) So using write() once doesn't at all seem like an advantage over simply returning the data... The interesting part is behind the scenes. If the response is large enough (it's an adjustable threshold), the response transparently gets sent to a temporary file. If the load on the server doesn't allow all the responses to fit in RAM, temporary files are an advantage because they allow the kernel to help manage the memory. (CPython's manner of using memory makes it hard for the kernel to page most memory held by Python processes.) However, as I said earlier, there was only advantage if a lot of conditions were met. Those conditions were often met by Zope Corporation customers. Shane ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: URGENT RFC: Is anyone using response.write in Zope 3?
Shane Hathaway wrote: Philipp von Weitershausen wrote: So using write() once doesn't at all seem like an advantage over simply returning the data... The interesting part is behind the scenes. If the response is large enough (it's an adjustable threshold), the response transparently gets sent to a temporary file. If the load on the server doesn't allow all the responses to fit in RAM, temporary files are an advantage because they allow the kernel to help manage the memory. (CPython's manner of using memory makes it hard for the kernel to page most memory held by Python processes.) Now that I've brought more of this issue back into my head, I remember that the advantage I just spoke of is independent of response.write(). However, there was still an advantage if a response is large enough that it doesn't comfortably fit in RAM. response.write() allowed the application to write the response in pieces, giving the kernel an opportunity to swap out the response data immediately rather than thrash. If Zope 2's ZServer didn't have this feature, zope.org would crawl even worse than it does now. I know this because of some zope.org debugging sessions. Shane ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: URGENT RFC: Is anyone using response.write in Zope 3?
Shane Hathaway wrote: Shane Hathaway wrote: Philipp von Weitershausen wrote: So using write() once doesn't at all seem like an advantage over simply returning the data... The interesting part is behind the scenes. If the response is large enough (it's an adjustable threshold), the response transparently gets sent to a temporary file. If the load on the server doesn't allow all the responses to fit in RAM, temporary files are an advantage because they allow the kernel to help manage the memory. (CPython's manner of using memory makes it hard for the kernel to page most memory held by Python processes.) Now that I've brought more of this issue back into my head, I remember that the advantage I just spoke of is independent of response.write(). However, there was still an advantage if a response is large enough that it doesn't comfortably fit in RAM. response.write() allowed the application to write the response in pieces, giving the kernel an opportunity to swap out the response data immediately rather than thrash. If Zope 2's ZServer didn't have this feature, zope.org would crawl even worse than it does now. I know this because of some zope.org debugging sessions. You are confusing Zope 2 and Zope 3. Zope 2's response.write does handle large output effciently, Zope 3's did not. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: URGENT RFC: Is anyone using response.write in Zope 3?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: When we refactored the Zope 3 pubisher to work more closely with WSGI, we decided to remove the response.write method. We should have written a proposal for this, but we failed to do so. Over the last few weeks there has been much discussion of this in which I asserted many times that I didn't think any applications outside of Zope 3 itself used this method. No one has disagreed with me. I want to double check this. Does anyone have any Zope 3 applications that are using response.write? Assuming that the answer is no (or that no one answers today), I'm going to more clearly document how to return long output and I'm going to add a method that generates a hopefully helpful error. Note that we may add this method (or something like it) back in the future to support chunked streaming output, IIRC, Christian Theune wrote that he had been using 'response.write'; I think the note was in the Twisted Publisher thread about a week and a half back. CC'ing him to ensure he knows to respond today. Tres. - -- === Tres Seaver +1 202-558-7113 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDptN9+gerLs4ltQ4RAnhiAJ9TvhSSBDPj6HmbNkL/otDduioBFACdF0VP m5iJH+iy7meJBSrlXyPLQZI= =94K7 -END PGP SIGNATURE- ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: URGENT RFC: Is anyone using response.write in Zope 3?
Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: When we refactored the Zope 3 pubisher to work more closely with WSGI, we decided to remove the response.write method. We should have written a proposal for this, but we failed to do so. Over the last few weeks there has been much discussion of this in which I asserted many times that I didn't think any applications outside of Zope 3 itself used this method. No one has disagreed with me. I want to double check this. Does anyone have any Zope 3 applications that are using response.write? Assuming that the answer is no (or that no one answers today), I'm going to more clearly document how to return long output and I'm going to add a method that generates a hopefully helpful error. Note that we may add this method (or something like it) back in the future to support chunked streaming output, IIRC, Christian Theune wrote that he had been using 'response.write'; I think the note was in the Twisted Publisher thread about a week and a half back. Thanks, for the reminder, but this was internal Zope code that had inadequate tests. It is also code that isn't being used in production at this time because it is incomplete. (This is sad, because I think zsync has tremendous potential. We should probably move it to a development branch and remove it from the main tree until it's ready. Sigh.) I think he stumbled accross this while addressing a collector issue. I really want to know if any Zope 3 applications outside of Zope 3 itself are using this API. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: URGENT RFC: Is anyone using response.write in Zope 3?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jeff Shell wrote: Yes, it's hurry.file. What's Tramline? We're using hurry.file for small images, generally, and it's been working fine. We've recently written a cache manager that writes the images out to the file system where Apache can serve them. Heh, that would be tramline, or half of it, anyway. ;) http://faassen.n--tree.net/blog/view/weblog/2005/11/11/0 Tres. - -- === Tres Seaver +1 202-558-7113 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDpv5Z+gerLs4ltQ4RAtK4AJ9wnZuLJjyH/Ahfhpi5Sk4AN1Qi9gCeNSgo MsRgJsQRrAhsnmFWUwPPUHs= =wrZ2 -END PGP SIGNATURE- ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
