Re: [Zope-dev] New version of BTreeFolder
On Mon, 28 Aug 2000 20:41:05 -0700, Kapil Thangavelu [EMAIL PROTECTED] wrote: i ran some basic tests comparing a btree folder against a folder. i'm not sure about my testing methodology but here are the results. the test code is at the end of this email. Im not sure this benchmark is measuring the things that a BTreeFolder should excel at One advantage of a BTreeFolder is that you can access *one* if its contained items without having to load *all* of the others. All of your tests iterate through the whole content. Secondly, it should be possible to add two items simultaneously without causing a conflict, however your creation test has no concurrency. Toby Dickenson [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] New version of BTreeFolder
Kapil Thangavelu wrote: Shane Hathaway wrote: BTW has anyone done any semi-formal speed tests? It's rumored that a custom getattr in Python can cause a bit of a slowdown. We could write a custom getattr in C to solve the problem. But we don't want to do any premature optimization. Shane I'll work on some tests. Kapil semi-formal test i ran some basic tests comparing a btree folder against a folder. i'm not sure about my testing methodology but here are the results. the test code is at the end of this email. the first pair of numbers is a timed creation test for size xxx of DTML Documents. the second pair of numbers is a timed test for len(objectIds()) the third pair of numbers is a timed test for attribute access with a call to int(dtmldoc.title_or_id()) # all the ids are stringed numbers my test box is running mandrake 7.1 with 512 M ram, dual proc celerons 550s (oc'd) some quick conclusions. the btree folder is slower on average for creation by anywhere from 15-25% andt btree's __getattr is on average almost twice as slow as the folder's getattr. order folder followed by btreefolder for size 100 0.0974419116974 :::0.126843929291 0.000378012657166 :::0.000354051589966 0.0101770162582 :::0.0186030864716 for size 100 0.0978569984436 :::0.125252962112 0.000376105308533 :::0.000353932380676 0.0101470947266 :::0.0185579061508 for size 500 0.52112698555 :::0.657492995262 0.00154399871826 :::0.00154197216034 0.0524510145187 :::0.0956329107285 for size 500 0.53660595417 :::0.714686989784 0.00154197216034 :::0.00157594680786 0.0532569885254 :::0.100463986397 for size 1000 1.11411702633 :::1.4010369 0.00302505493164 :::0.00306105613708 0.105547070503 :::0.193196058273 for size 1000 1.12388503551 :::1.58762395382 0.00370097160339 :::0.00381696224213 0.106634020805 :::0.194367051125 for size 2000 2.60056698322 :::3.20675897598 0.007483959198 :::0.00741696357727 0.216197967529 :::0.396993994713 for size 2000 2.64964604378 :::3.26421093941 0.0074850320816 :::0.00761103630066 0.214147925377 :::0.40997505188 for size 5000 9.50434195995 :::11.0002789497 0.0184400081635 :::0.018413901329 0.533676981926 :::0.997251987457 just to be fair i reversed the positions and did a second set of runs with only 1 run per size as the results were consistent with the first set. for size 100 0.128319978714 :::0.10124707222 0.000413060188293 :::0.000383019447327 0.018816113472 :::0.0104240179062 for size 100 0.12813603878 :::0.101204037666 0.00040602684021 :::0.000380039215088 0.0188159942627 :::0.0105849504471 for size 500 0.681550979614 :::0.546862006187 0.00152194499969 :::0.00160706043243 0.0983099937439 :::0.0540620088577 for size 1000 1.47808301449 :::1.17385303974 0.00308799743652 :::0.00319290161133 0.197283029556 :::0.108732938766 for size 5000 11.1204429865 :::9.15097498894 0.015163064003 :::0.0150960683823 1.00954806805 :::0.535046935081 Despite this evidence. at size 5000, looking at the btreefolder contents in the management screen was near instaneous while the folder took almost a minute. based on the above evidence though that has more to do with the interface than the underlying speed of the folder methods. night. Kapil test code to follow. external method import time import os import Products.BTreeFolder def btest(self, REQUEST, =None): ''' ''' if not : =100 results = [] bf = Products.BTreeFolder.BTreeFolder.BTreeFolder() bf.id = 'btree_test_b' self._setObject(bf.id, bf) id = 'btree_test_f' rf = self.manage_addFolder(id) # creation test for completeness stime = time.time() for num in xrange(): self.btree_test_b.manage_addDTMLDocument(str(num)) etime = time.time() result = etime-stime #self.get_transaction().commit() stime = time.time() for num in xrange(): self.btree_test_f.manage_addDTMLDocument(str(num)) etime = time.time() results.append(result, etime-stime) # access test stime = time.time() len(self.btree_test_b.objectIds()) etime = time.time() result = etime-stime stime = time.time() len(self.btree_test_f.objectIds()) etime = time.time() results.append(result, etime-stime) # iteration test stime = time.time() for oid, item in self.btree_test_b.objectItems(): int(item.title_or_id()) etime = time.time() result=etime-stime stime = time.time() for oid, item in self.btree_test_f.objectItems(): int(item.title_or_id()) etime = time.time() results.append(result, etime-stime) return results ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts
Re: [Zope-dev] New version of BTreeFolder
Kapil Thangavelu wrote: semi-formal test i ran some basic tests comparing a btree folder against a folder. i'm not sure about my testing methodology but here are the results. the test code is at the end of this email. (snip) some quick conclusions. the btree folder is slower on average for creation by anywhere from 15-25% andt btree's __getattr is on average almost twice as slow as the folder's getattr. I must say thank you! Your results look very conclusive. It would appear, then, that storing the objects in a BTree actually slows down reads and writes considerably. And the only thing that needs to be improved about folders is the user interface. The problem with the current Folder UI, you see, is that each object is loaded from the database every time you look at the folder contents view. Additionally, browsers have trouble rendering large tables. BTreeFolder gets the contents listing from the _objects attribute and generates a simple list. It's just lucky that all platforms I know of can handle thousands of items in a simple list without any slowdown. This provided us the data we needed for improving folders. Thanks again! Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] New version of BTreeFolder
Hi gang, This version just fixes a bug with the folder contents tab on Windows. For some reason, on Windows 95 (and perhaps other platforms) the Python interpreter behaves differently enough that if you use a class attribute from a different module and put it in your own class, you may in fact be modifying its global context. It's probably a bug in Python. As a workaround, this version of BTreeFolders doesn't have the "details" tab anymore. http://www.zope.org/Members/hathawsh/BTreeFolder/BTreeFolder-0.1.1.tar.gz BTW has anyone done any semi-formal speed tests? It's rumored that a custom getattr in Python can cause a bit of a slowdown. We could write a custom getattr in C to solve the problem. But we don't want to do any premature optimization. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] New version of BTreeFolder. Just a Newbie Question
Oh, well, I'd bet this is a silly question. But I can not resist nor find answer by myself. Q: Why don't you BTree-fy all the subclasses of ObjectManager? Is there a reason that anyone but can a newbie see? LEE Kwan Soo. ps. I only hope to be able to make clear statement in English, but i am already too sleepy now. sorry. - Original Message - From: "Shane Hathaway" [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Monday, August 28, 2000 10:26 PM Subject: [Zope-dev] New version of BTreeFolder Hi gang, This version just fixes a bug with the folder contents tab on Windows. For some reason, on Windows 95 (and perhaps other platforms) the Python interpreter behaves differently enough that if you use a class attribute from a different module and put it in your own class, you may in fact be modifying its global context. It's probably a bug in Python. As a workaround, this version of BTreeFolders doesn't have the "details" tab anymore. http://www.zope.org/Members/hathawsh/BTreeFolder/BTreeFolder-0.1.1.tar.gz BTW has anyone done any semi-formal speed tests? It's rumored that a custom getattr in Python can cause a bit of a slowdown. We could write a custom getattr in C to solve the problem. But we don't want to do any premature optimization. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] New version of BTreeFolder. Just a Newbie Question
LEE Kwan Soo wrote: Oh, well, I'd bet this is a silly question. But I can not resist nor find answer by myself. Q: Why don't you BTree-fy all the subclasses of ObjectManager? Is there a reason that anyone but can a newbie see? Obviously what BTreeFolder does should be available in standard folders. That is a long-term goal for Zope. BTreeFolder only exists to scope out the need for big folders as well as provide the capability in the short term. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] New version of BTreeFolder
Shane Hathaway wrote: Hi gang, This version just fixes a bug with the folder contents tab on Windows. For some reason, on Windows 95 (and perhaps other platforms) the Python interpreter behaves differently enough that if you use a class attribute from a different module and put it in your own class, you may in fact be modifying its global context. It's probably a bug in Python. As a workaround, this version of BTreeFolders doesn't have the "details" tab anymore. http://www.zope.org/Members/hathawsh/BTreeFolder/BTreeFolder-0.1.1.tar.gz BTW has anyone done any semi-formal speed tests? It's rumored that a custom getattr in Python can cause a bit of a slowdown. We could write a custom getattr in C to solve the problem. But we don't want to do any premature optimization. Shane I'll work on some tests. Kapil ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] New version of BTreeFolder. Just a Newbie Question
Shane Hathaway wrote: LEE Kwan Soo wrote: Oh, well, I'd bet this is a silly question. But I can not resist nor find answer by myself. Q: Why don't you BTree-fy all the subclasses of ObjectManager? Is there a reason that anyone but can a newbie see? Obviously what BTreeFolder does should be available in standard folders. That is a long-term goal for Zope. BTreeFolder only exists to scope out the need for big folders as well as provide the capability in the short term. Any chance at a BTree Folder w/customizer suppor tin the near future? :) -- Do not meddle in the affairs of sysadmins, for they are easy to annoy, and have the root password. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] New version of BTreeFolder. Just a Newbie Question
Bill Anderson wrote: Any chance at a BTree Folder w/customizer support in the near future? :) It should be quite easy to do. I vaguely recall Steve A or Steve S might have done it. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] New version of BTreeFolder. Just a Newbie Question
Bill Anderson wrote: Any chance at a BTree Folder w/customizer suppor tin the near future? :) I did some work towards this. However, I didn't continue. I realised that I could get the same effect by having a BTreeFolder inside a Customizer Folder. The only downside is a slightly longer URL for objects in the BTreeFolder. -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] New version of BTreeFolder
Steve Alexander wrote: BTreeFolder doesn't do the from Globals import default__class_init__ default__class_init__(BTreeFolder) thing. Does that matter at all? As none of the classes it is derived from have a __init__ method, I guess not. Or, does default__class_init__ do anything else? It converts __ac_permissions__ into usable permissions declarations (__roles__ attributes.) default__class_init__ really ought to be there; it's only recently that Brian started saying we need to use it everywhere. I forgot. :-) Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
