Tres Seaver wrote:
After seeing Chris McDonough's excellent paper at the Plone conference on optimizing content delivery using the new IStreamIterator iterface, I began experimenting with implementing such an optimization in ZopePageTemplates.
I played around with having 'ZopePageTemplate._exec' request that it receive an iterator, instead of the usual big string. I hoped that such a change might enable greater concurrency and memory footprint, by avoiding creation of the big string at all; instead, medusa to push out the "chunk stream" represented by the StringIO buflist, while the appserver would be free to handle a new request without needing to malloc / copy the data.
Here are the timings I have seen so far, using 'zopectl debug', with the following template:
--------------- Template source -------------------------------- <html> <body> <div tal:repeat="item python:[x for x in range(1000)]" tal:content="item">ITEM</div> </body> </html> ----------------------------------------------------------------
--------------- Before the patch -------------------------------
Zope.debug('/test_iter', t=1)
250.2 milliseconds
Zope.debug('/test_iter', t=1)
106.7 milliseconds
Zope.debug('/test_iter', t=1)
106.5 milliseconds
Zope.debug('/test_iter', t=1)
124.6 milliseconds ----------------------------------------------------------------
--------------- After the patch -------------------------------- >>> Zope.debug('/test_iter', t=1) 249.2 milliseconds >>> Zope.debug('/test_iter', t=1) 107.2 milliseconds >>> Zope.debug('/test_iter', t=1) 125.0 milliseconds >>> Zope.debug('/test_iter', t=1) 162.1 milliseconds ----------------------------------------------------------------
Given that the performance looks similar in this context, which doesn't benefit from the medusa / concurrency intent of the patch, it seems as though it might be a win (of *course* there aren't any tests for it!)
I am attaching the patch I have so far for review and comment.
Tres.
------------------------------------------------------------------------
This patch causes ZopePageTemplates to return an IStreamIterator when called (published), potentially allowing medusa to return the same response payload (via the iterator) without needing to have the appserver thread join it into a big string.
--- PageTemplate.py 2004-09-25 21:39:50.595938847 -0400
+++ PageTemplate.py.new 2004-09-25 21:39:31.983326567 -0400
@@ -83,7 +84,7 @@
c['root'] = self
return c
- def pt_render(self, source=0, extra_context={}):
+ def pt_render(self, source=0, extra_context={}, iter_handler=None):
"""Render this Page Template"""
if not self._v_cooked:
self._cook()
@@ -100,7 +101,10 @@
getEngine().getContext(c),
output,
tal=not source, strictinsert=0)()
- return output.getvalue()
+ if iter_handler is not None:
+ return iter_handler(output)
+ else:
+ return output.getvalue()
def __call__(self, *args, **kwargs):
if not kwargs.has_key('args'):
--- ZopePageTemplate.py 2004-09-25 21:37:07.956803011 -0400
+++ ZopePageTemplate.py.new 2004-09-25 21:37:23.024870003 -0400
@@ -34,6 +34,8 @@
from OFS.Cache import Cacheable
from OFS.Traversable import Traversable
from OFS.PropertyManager import PropertyManager
+from ZPublisher.Iterators import IStreamIterator
+
from PageTemplate import PageTemplate
from Expressions import SecureModuleImporter
from PageTemplateFile import PageTemplateFile
@@ -253,7 +255,8 @@
# Execute the template in a new security context.
security.addContext(self)
try:
- result = self.pt_render(extra_context=bound_names)
+ result = self.pt_render(extra_context=bound_names,
+ iter_handler=StringIOIterator)
if keyset is not None:
# Store the result in the cache.
self.ZCacheable_set(result, keywords=keyset)
@@ -331,6 +334,26 @@
setattr(ZopePageTemplate, 'source.xml', ZopePageTemplate.source_dot_xml)
setattr(ZopePageTemplate, 'source.html', ZopePageTemplate.source_dot_xml)
+
+class StringIOIterator:
+ """ Adapt a StringIO object to IStreamIterator.
+ """
+
+ __implements__ = (IStreamIterator,)
+
+ def __init__(self, stringio):
+ self._buflist = stringio.buflist
+ self._index = 0
+
+ def next(self):
+
+ if self._index >= len(self._buflist):
+ raise StopIteration
+
+ data, self._index = self._buflist[self._index], self._index + 1
+
+ return data
+
# Product registration and Add support
manage_addPageTemplateForm = PageTemplateFile(
'www/ptAdd', globals(), __name__='manage_addPageTemplateForm')
------------------------------------------------------------------------
_______________________________________________
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists - http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
_______________________________________________
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists - http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
