Re: [Zope-dev] Re: [Zope3-dev] proposal: serving static content faster
I'm pretty sure that the tempfile penalty is unrelated to the results Paul sees in his tests, at least for smaller files. If the content-length header indicates that the body of the item is smaller than 128k, it does not appear to be spooled from a tempfile at all. This also may be why there isn't much of a difference between "normal" product results and Paul's "patched" product results at his 12K file size as opposed to the roughly 60% difference in speed at at 324K and 1216K (I cannot explain the large difference in results at 38322k across products; it appears to be product-specific). Also, for this file size, wrapping the data in a producer is actually just more overhead; products would be better off to do a direct RESPONSE.write if the total file size was under 128k (as evidenced by the fact that the iterator-based products are slower at small file sizes). It's a bit of a mystery about why retrieving data over ZEO is an order of magnitude slower than retrieving it from a local file storage when the ZEO server is actually on the same physical machine. I'm sure there are lots of optimization strategies for making this faster that involve a lot of heavy lifting and hard thinking in a dark room. ;-) While that might be fun, in the meantime, for the hoi-polloi who just need to get stuff done, I believe that one simple answer to this is, as usual, to do very aggressive caching and avoid ZODB access altogether for large blobs. The ZEO caching algorithm is largely agnostic about what kinds of objects it caches and doesn't allow for setting a cache-eviction policy other than whatever its implementors consider optimal for the most general case. This would be fairly difficult to change in the best-case circumstance for someone with deep ZODB knowledge. A more effective strategy for "normal" people might be to contribute to the "hardening" of something like FileCacheManager, which prevents ZODB access in the first place and allows for fine-grained policies on an application level by caching blobs on client filesystems. - C On Thu, 2004-04-08 at 12:16, Paul Winkler wrote: > On Wed, Mar 24, 2004 at 01:32:18PM -0500, Shane Hathaway wrote: > > Jeremy has suggested that object pre-fetching could be added to ZODB. > > This is much on my mind currently. > Any thoughts on what an API for pre-fetching might look like? > > The use case that most concerns me is: > If you have an Image or File object with a very long Pdata chain, > you're likely to A) thrash the ZEO client cache if it's not big enough, > and B) spend a long time waiting for all the objects in the chain > to load. At least, this is my theory on what's slowing things > down - I will be trying to verify this today. See the bottom of this page: > http://www.slinkp.com/code/zopestuff/blobnotes ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Re: [Zope3-dev] proposal: serving static content faster
On Thu, 8 Apr 2004 [EMAIL PROTECTED] wrote: > I'm working on a product which serves files from the filesystem. The > data retrieval method is the usual: > > def pushData(self, f, outstream): > finished = False > while not finished: > block = f.read(blocksize) > if len(block) < blocksize: > finished = True > outstream.write(block) > > f is the file on the filesystem, outstream is the request object. > > Testing with a 1Mbyte file (ab -n 12 -c 4), I get ~4.4 req/sec - > ~4.7Mbyte/sec after a few iterations (the os caches the file). Zope reads the file from a cache but then forces its entire contents into dirty buffers. If the file is sent before the OS decides the flush the buffers to disk, and the OS is somehow smart enough to cancel the write of those buffers, you're in luck--not much penalty. You might even decide to put these files on a RAM disk. Most network connections aren't that fast, though, so you have to expect a concurrency level much higher than 4. The penalty comes when the OS has to write all those concurrent copies of the same data to disk, then delete each of them when the download is finished. Zope could make a good file server if it just didn't make so many temporary copies. > It seems from these results, that ZServer's tempfile strategy causes > some (~20% if everything is cached) performance hit, but I think that > there should other bottleneck(s) beside this. Your test is too optimistic. Try a concurrency level of 200. Another bottleneck is the asyncore select loop, which has an O(n) delay for each I/O operation, where n is the number of connections currently open. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Re: [Zope3-dev] proposal: serving static content faster
> In fact, Zope puts large files (the threshold is around 256K - 512K) > into a temporary file before serving them, to free up application > threads. It's a tremendous handicap. I'm working on a product which serves files from the filesystem. The data retrieval method is the usual: def pushData(self, f, outstream): finished = False while not finished: block = f.read(blocksize) if len(block) < blocksize: finished = True outstream.write(block) f is the file on the filesystem, outstream is the request object. Testing with a 1Mbyte file (ab -n 12 -c 4), I get ~4.4 req/sec - ~4.7Mbyte/sec after a few iterations (the os caches the file). Now, I change this method to the following: def pushData(self, f, outstream, data='0'*65536): for i in range(16): outstream.write(data) The test results are the same. Now, if I disable the temporary file thing in ZServer/HTTPResponse.py the performance goes up to around 6.9 req/sec - ~7Mbyte/sec. If I restore my pushData method to it's original form it can still do ~6.2 req/sec - ~6.6Mbyte/sec. In this case, practically every disk operation was served from the cache. It seems from these results, that ZServer's tempfile strategy causes some (~20% if everything is cached) performance hit, but I think that there should other bottleneck(s) beside this. Regards, Sandor ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Fwd: [ZPT] Making PageTemplate's edit pages Unicode aware
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
I havn't received much feedback on the ZPT mailing list, so I
thought I'd bring it over here to a wider audience (thread is at
http://mail.zope.org/pipermail/zpt/2004-March/005218.html ).
Begin forwarded message:
From: Stuart Bishop <[EMAIL PROTECTED]>
Date: 29 March 2004 6:13:06 PM
To: Dieter Maurer <[EMAIL PROTECTED]>
Cc: [EMAIL PROTECTED]
Subject: Re: [ZPT] Makeing PageTemplate's edit pages Unicode aware
On 27/03/2004, at 9:57 PM, Dieter Maurer wrote:
Stuart Bishop wrote at 2004-3-25 12:27 +1100:
Currently, if you enter non-ascii text into the title or contents
fields on a PageTemplate's edit page, the data ends up stored as
an encoded string (using management_page_charset, if it is set.
Unknown
encoding if it is not).
This should be easy to fix using the foo:charset:ustring notation
to have Zope convert the encoded strings to Unicode. However, the
file upload feature is more problematic. Should the file upload
try converting the file to Unicode from UTF-8 and raise an exception
if this is not possible? I personally feel this is preferable to
ending up with arbitrarily enncoded document source, with no idea
of the character set used.
I do not think that Zope should convert when it does not know the
encoding. I am unaware that a missing "management_page_charset"
can be interpreted as "UTF-8". If this were the case, converstion
to unicode might be correct. By the way: the HTML specification
says that uploaded files should come with a "content-type"
declaration.
In this case, the charset specified there (if any) should be used
to determine the encoding.
Yes - A missing management_page_charset should probably be interpreted
as either US-ASCII or ISO-8859-1. US-ASCII is probably more correct,
but I would guess that most browsers will be configured to use
ISO-8859-1 as their default (and this might be specified in the HTML
spec?)
I guess using the charset type the browser tells us for file uploads
means we can blame the browser. I don't know how this could be reliable
(since text files themselves don't encode their character set unless
they happen to be UTF-16 or have a BOM). I am wondering if having a
file upload function is incompatible with a Unicode aware page
templates product.
If management_page_charset is not set, it is unknown what charset
is being used. The only way of knowing the character set of data that
has been submitted is to know the character set of the form that it
was submitted from. All other mechanisms do not work due to
incompatibilities in how the browsers work.
Currently, if you create a page template that contains non-ASCII
characters, any tal:content or tal:replace expressions that return
Unicode will now raise a Unicode error. This can be demonstrated
simply:
My 2¢
My 2¢
These are the things I think need to be fixed in Zope's Page Templates
implementation to make them Unicode aware. There may be more (?):
- It should be possible for the actual page template source to
be stored as a Unicode string. Currently, there is an assert
ensuring it is a traditional string.
- The title property should be a Unicode string.
- PageTemplateFile should grow an optional charset parameter,
defaulting to US-ASCII.
- PageTemplate.write(text) should raise an exception if text
is not either a Unicode string or an ASCII string.
- The ZopePageTemplate edit page should use Zope's
:charset:ustring notation so Unicode strings get passed
to its handler.
- The file upload widget needs to either be removed, or grow
a charset box. I don't think either of these solutions are
ideal :-(
Note that when I say 'Unicode string', we can still store ASCII
text using a traditional string to save space.
My application is currently using a ZopePageTemplate subclass that
has been modified to use Unicode strings for the document source
and title, and it seems to be functioning just fine. Does anyone
know if that "assert type(text) == type('')" in PageTemplate.write
is there for a reason?
- --
Stuart Bishop <[EMAIL PROTECTED]>
http://www.stuartbishop.net/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.2.3 (Darwin)
iD8DBQFAdfPfAfqZj7rGN0oRAkBuAJ0WLSC3V2eL+zNzkQqBqjJ2bl5degCfe2SB
DlT7NTsieQlDhVgEnHYaXp8=
=6XPE
-END PGP SIGNATURE-
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [Zope3-dev] proposal: serving static content faster
On Thu, Apr 08, 2004 at 08:02:05PM -0400, Shane Hathaway wrote: > On 04/08/04 12:16, Paul Winkler wrote: > >On Wed, Mar 24, 2004 at 01:32:18PM -0500, Shane Hathaway wrote: > > > >>Jeremy has suggested that object pre-fetching could be added to ZODB. > > > > > >This is much on my mind currently. > >Any thoughts on what an API for pre-fetching might look like? > > Well, thinking about it some more, it seems like it might be just as > easy to run a special thread just for fetching blobs. One queue should > tell the blob thread what to get, and another queue should return the > results. I think that would be a nice design. ok... i'll mull that over. > >The use case that most concerns me is: > >If you have an Image or File object with a very long Pdata chain, > >you're likely to A) thrash the ZEO client cache if it's not big enough, > >and B) spend a long time waiting for all the objects in the chain > >to load. At least, this is my theory on what's slowing things > >down - I will be trying to verify this today. See the bottom of this page: > >http://www.slinkp.com/code/zopestuff/blobnotes > > In fact, Zope puts large files (the threshold is around 256K - 512K) > into a temporary file before serving them, to free up application > threads. It's a tremendous handicap. If you're referring to the code in ZServerHTTPResponse.write: AFAICT, since Image.File calls RESPONSE.write() for each Pdata in its chain, we'll actually get a temporary file for each Pdata rather than one big one for the whole file. There may be (probably are) performance issues in there, but right now I'd be really happy to get ClientStorage a bit closer to the speed of FileStorage for this Pdata gunk. -- Paul Winkler http://www.slinkp.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [Zope3-dev] proposal: serving static content faster
On 04/08/04 12:16, Paul Winkler wrote: On Wed, Mar 24, 2004 at 01:32:18PM -0500, Shane Hathaway wrote: Jeremy has suggested that object pre-fetching could be added to ZODB. This is much on my mind currently. Any thoughts on what an API for pre-fetching might look like? Well, thinking about it some more, it seems like it might be just as easy to run a special thread just for fetching blobs. One queue should tell the blob thread what to get, and another queue should return the results. I think that would be a nice design. The use case that most concerns me is: If you have an Image or File object with a very long Pdata chain, you're likely to A) thrash the ZEO client cache if it's not big enough, and B) spend a long time waiting for all the objects in the chain to load. At least, this is my theory on what's slowing things down - I will be trying to verify this today. See the bottom of this page: http://www.slinkp.com/code/zopestuff/blobnotes In fact, Zope puts large files (the threshold is around 256K - 512K) into a temporary file before serving them, to free up application threads. It's a tremendous handicap. Here's a relevant tangent. Twisted and asyncore like all I/O to be event-driven, but ZODB is fundamentally not event-driven. In ZODB, any attribute access can incur I/O time, and you can't control that. This is why Stackless is interesting for ZODB. With Stackless, you might be able to switch stack frames at the moment ZODB needs to do some I/O and get some useful work done. Without Stackless, ZODB can not guarantee any particular response time and therefore should not operate inside a time-critical event loop. Threads can also solve this problem, but the global interpreter lock hurts performance. There's also POSH. With POSH, you can take advantage of multiple processors (which you can't do with Stackless nor threads)... that seems like a really good thing. Some careful coding might make Zope + POSH scream. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [Zope3-dev] proposal: serving static content faster
On Wed, Mar 24, 2004 at 01:32:18PM -0500, Shane Hathaway wrote: > Jeremy has suggested that object pre-fetching could be added to ZODB. This is much on my mind currently. Any thoughts on what an API for pre-fetching might look like? The use case that most concerns me is: If you have an Image or File object with a very long Pdata chain, you're likely to A) thrash the ZEO client cache if it's not big enough, and B) spend a long time waiting for all the objects in the chain to load. At least, this is my theory on what's slowing things down - I will be trying to verify this today. See the bottom of this page: http://www.slinkp.com/code/zopestuff/blobnotes -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's RETRO-NEWBIE Z! (random hero from isometric.spaceninja.com) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [Zope3-dev] proposal: serving static content faster
On Wed, Mar 24, 2004 at 12:57:00PM -0500, Chris McDonough wrote: > > On Wed, 2004-03-24 at 09:28, Shane Hathaway wrote: > > This sounds useful for serving content from the filesystem. > > > > However, I'm a little concerned about this because producers must not > > read from the object database. FWIW, I've added your warning to the IStreamIterator docstring. If I hadn't vaguely remembered this thread, I probably would have tripped over it sooner or later! -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's COBRA DAMN LAGOMORPH! (random hero from isometric.spaceninja.com) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Using a truely revision based storage for Zope ?
[EMAIL PROTECTED] wrote: I've started looking at the ZODB and APE packages to try and get some understanding of how the whole storage interaction works, but it'll take me some time to figure it all out ... So I thought I'd get feedback on the idea first ... Sounds great! If I were you, I would start by replacing Ape/lib/apelib/fs/fileops.py with something that uses the Subversion module. It's a very simple module and you should be able to get pretty far that way. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: manage_permission from script
Erik A.Dahl wrote:
Thanks for the suggestion but no joy here. Here is the full trace...
Python 2.3.3 (#1, Jan 27 2004, 09:17:28)
[GCC 3.2 20020903 (Red Hat Linux 8.0 3.2-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import Zope
>>> app = Zope.app()
>>> app.testfolder
>>> tf = app.testfolder
>>> tf.getPhysicalRoot().manage_permission("View", ["Owner",])
Traceback (most recent call last):
File "", line 1, in ?
File "/home/edahl/zope-2.7.0/lib/python/AccessControl/Role.py", line
164, in manage_permission
for p in self.ac_inherited_permissions(1):
File "/home/edahl/zope-2.7.0/lib/python/AccessControl/Role.py", line
82, in ac_inherited_permissions
for p in self._subobject_permissions():
File "/home/edahl/zope-2.7.0/lib/python/OFS/ObjectManager.py", line
212, in _subobject_permissions
return (Products.__ac_permissions__+
AttributeError: aq_acquire
'manage_permission' (really, the methods it calls) make the assumption
(unwarranted in your case) that they are called on an
acquisition-wrapped object. In your case, the root object is *not*
wrapped, and therefore does not have the 'aq_acquire' method. Normally,
when called via the publisher, the root object *is* wrapped (in a
RequestContainer).
A workaround would be to wrap the root first:
zopectl> debug
Starting debugger (the name "app" is bound to the top-level Zope object)
>>> app
>>> app.aq_chain # not wrapped
Traceback (most recent call last):
File "", line 1, in ?
AttributeError: aq_chain
>>> app.manage_addFolder( 'testfolder' )
>>> tf = app.testfolder
>>> pr = tf.getPhysicalRoot()
>>> pr.aq_acquire
Traceback (most recent call last):
File "", line 1, in ?
AttributeError: aq_acquire
>>> pr = pr.__of__(tf)
>>> pr.aq_chain
[, ,
]
>>> pr.aq_acquire
>>> manage_permission("View", ["Owner"])
>>>
The real fix would be to rip out the use *anywhere* of
'self.aq_acquire'; the 'aq_aquire' function from Acquisition will
always do the Right Thing (TM) if the object is not yet wrapped (the
difference parallels using 'aq_base', 'aq_parent', 'aq_inner' as
functions rather than methods).
Tres.
--
===
Tres Seaver[EMAIL PROTECTED]
Zope Corporation "Zope Dealers" http://www.zope.com
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Zope-Coders] Re: CVS: Zope3/src/zope/tal - talparser.py:1.6
On Thursday 08 April 2004 10:00 am, Philipp von Weitershausen wrote: > I would like to backport this patch (including tests) to Zope 2, since I > need to i18n XML generated by ZPTs. None here. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: CVS: Zope3/src/zope/tal - talparser.py:1.6
Gintautas Miliauskas wrote:
Update of /cvs-repository/Zope3/src/zope/tal
In directory cvs.zope.org:/tmp/cvs-serv27951/src/zope/tal
Modified Files:
talparser.py
Log Message:
Removed an assertion which disallows usage of Zope3 i18n in XML markup.
Added a few test cases which run the new code path.
=== Zope3/src/zope/tal/talparser.py 1.5 => 1.6 ===
--- Zope3/src/zope/tal/talparser.py:1.5 Fri Mar 19 16:42:04 2004
+++ Zope3/src/zope/tal/talparser.py Wed Apr 7 11:13:00 2004
@@ -81,7 +81,6 @@
taldict[keybase] = value
item = item + ("tal",)
elif ns == 'i18n':
-assert 0, "dealing with i18n: " + `(keybase, value)`
i18ndict[keybase] = value
item = item + ('i18n',)
fixedattrlist.append(item)
Hello,
I would like to backport this patch (including tests) to Zope 2, since I
need to i18n XML generated by ZPTs.
Any objections?
Philipp
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] manage_permission from script
Thanks for the suggestion but no joy here. Here is the full trace...
Python 2.3.3 (#1, Jan 27 2004, 09:17:28)
[GCC 3.2 20020903 (Red Hat Linux 8.0 3.2-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import Zope
>>> app = Zope.app()
>>> app.testfolder
>>> tf = app.testfolder
>>> tf.getPhysicalRoot().manage_permission("View", ["Owner",])
Traceback (most recent call last):
File "", line 1, in ?
File "/home/edahl/zope-2.7.0/lib/python/AccessControl/Role.py", line
164, in manage_permission
for p in self.ac_inherited_permissions(1):
File "/home/edahl/zope-2.7.0/lib/python/AccessControl/Role.py", line
82, in ac_inherited_permissions
for p in self._subobject_permissions():
File "/home/edahl/zope-2.7.0/lib/python/OFS/ObjectManager.py", line
212, in _subobject_permissions
return (Products.__ac_permissions__+
AttributeError: aq_acquire
On Apr 7, 2004, at 7:30 PM, <[EMAIL PROTECTED]> wrote:
For setting permission on the root application object, use
setPermissionDefault of SecurityInfo.
Eh, this only works for my own defined permissions. However I just
tried
this from my product and it does what it supposed to do:
self.getPhysicalRoot().manage_permission('View', ['Owner'])
Sandor
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
