After seeing Chris McDonough's excellent paper at the Plone conference on optimizing content delivery using the new IStreamIterator iterface, I began experimenting with implementing such an optimization in ZopePageTemplates.

I played around with having 'ZopePageTemplate._exec' request that it receive an iterator, instead of the usual big string. I hoped that such a change might enable greater concurrency and memory footprint, by avoiding creation of the big string at all; instead, medusa to push out the "chunk stream" represented by the StringIO buflist, while the appserver would be free to handle a new request without needing to malloc / copy the data.

Here are the timings I have seen so far, using 'zopectl debug', with the following template:

--------------- Template source --------------------------------
<html>
<body>
<div tal:repeat="item python:[x for x in range(1000)]"
     tal:content="item">ITEM</div>
</body>
</html>
----------------------------------------------------------------

--------------- Before the patch -------------------------------
Zope.debug('/test_iter', t=1)
250.2 milliseconds
Zope.debug('/test_iter', t=1)
106.7 milliseconds
Zope.debug('/test_iter', t=1)
106.5 milliseconds
Zope.debug('/test_iter', t=1)
124.6 milliseconds
----------------------------------------------------------------

--------------- After the patch --------------------------------
>>> Zope.debug('/test_iter', t=1)
249.2 milliseconds
>>> Zope.debug('/test_iter', t=1)
107.2 milliseconds
>>> Zope.debug('/test_iter', t=1)
125.0 milliseconds
>>> Zope.debug('/test_iter', t=1)
162.1 milliseconds
----------------------------------------------------------------

Given that the performance looks similar in this context, which doesn't benefit from the medusa / concurrency intent of the patch, it seems as though it might be a win (of *course* there aren't any tests for it!)

I am attaching the patch I have so far for review and comment.

Tres.
--
===============================================================
Tres Seaver                                [EMAIL PROTECTED]
Zope Corporation      "Zope Dealers"       http://www.zope.com
    This patch causes ZopePageTemplates to return an IStreamIterator
    when called (published), potentially allowing medusa to return the
    same response payload (via the iterator) without needing to have the
    appserver thread join it into a big string.

--- PageTemplate.py	2004-09-25 21:39:50.595938847 -0400
+++ PageTemplate.py.new	2004-09-25 21:39:31.983326567 -0400
@@ -83,7 +84,7 @@
             c['root'] = self
         return c
 
-    def pt_render(self, source=0, extra_context={}):
+    def pt_render(self, source=0, extra_context={}, iter_handler=None):
         """Render this Page Template"""
         if not self._v_cooked:
             self._cook()
@@ -100,7 +101,10 @@
                        getEngine().getContext(c),
                        output,
                        tal=not source, strictinsert=0)()
-        return output.getvalue()
+        if iter_handler is not None:
+            return iter_handler(output)
+        else:
+            return output.getvalue()
 
     def __call__(self, *args, **kwargs):
         if not kwargs.has_key('args'):
--- ZopePageTemplate.py	2004-09-25 21:37:07.956803011 -0400
+++ ZopePageTemplate.py.new	2004-09-25 21:37:23.024870003 -0400
@@ -34,6 +34,8 @@
 from OFS.Cache import Cacheable
 from OFS.Traversable import Traversable
 from OFS.PropertyManager import PropertyManager
+from ZPublisher.Iterators import IStreamIterator
+
 from PageTemplate import PageTemplate
 from Expressions import SecureModuleImporter
 from PageTemplateFile import PageTemplateFile
@@ -253,7 +255,8 @@
         # Execute the template in a new security context.
         security.addContext(self)
         try:
-            result = self.pt_render(extra_context=bound_names)
+            result = self.pt_render(extra_context=bound_names,
+                                    iter_handler=StringIOIterator)
             if keyset is not None:
                 # Store the result in the cache.
                 self.ZCacheable_set(result, keywords=keyset)
@@ -331,6 +334,26 @@
 setattr(ZopePageTemplate, 'source.xml',  ZopePageTemplate.source_dot_xml)
 setattr(ZopePageTemplate, 'source.html', ZopePageTemplate.source_dot_xml)
 
+
+class StringIOIterator:
+    """ Adapt a StringIO object to IStreamIterator.
+    """
+
+    __implements__ = (IStreamIterator,)
+
+    def __init__(self, stringio):
+        self._buflist = stringio.buflist
+        self._index = 0
+
+    def next(self):
+
+        if self._index >= len(self._buflist):
+            raise StopIteration
+
+        data, self._index = self._buflist[self._index], self._index + 1
+
+        return data
+
 # Product registration and Add support
 manage_addPageTemplateForm = PageTemplateFile(
     'www/ptAdd', globals(), __name__='manage_addPageTemplateForm')
_______________________________________________
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )

Reply via email to