[Zope-dev] Re: any limits on object number?
Hello all, thanks to all the responses. I will test with BTreeFolder2, and subfoldering with first character of id as another subfolder2, so that working with ZMI will also be possible without further scripting or customized views. i will report the results back to the list. just an additional info: I run into this problem recently... I've got something around 40k objects to insert into ZODB. I started doing some tests with normal Folders, but I discarded it completely in favor of BTreeFolders. I got information that BTreeFolder was already tested with more than 400k objects... and that made me happy! :-) But (there's always a 'but'!)... Even using BTreeFolder to store all objects I was getting 300s delay to show a single object (ok, it's an archetypes-based one, containing 50 fields, splitted into 7 schematas, with lots of fancy stuff...). So I made a directory hash structure based on UID from each object (an AT UID is md5, so we have a hex base). Using this hash structure, with 16 divisions, I got a better performance (but still far from acceptable): 'only' ~100s to show a single object... :-( So I decided to create another level for the hash structure: now each folder has another 16 folder inside it. This time I was getting a delay of ~20s... As you're already thinking... it's time for another hash level. Now, with 16^3 additional BTreeFolders to split all my objects I got an acceptable performance: ~3s... The path is uggly as hell, f/f0/f0a/f0a5aac38aeff101b3168f2592dd879b, but at least the system is usable... Resuming what I've learned: don't abuse of BTreeFolder, hash your content and live happy forever... ;-) PS: The server is a modest PIII 1.2GHz, 1GB RAM with a 160MB/s SCSI controller running only one instance of Zope 2.7.1 without ZEO. HTH, -- Dorneles Treméa Caxias do Sul - RS - Brasil +55 54 9114 9312 - UIN: 2413568 X3ng Web Technology http://www.x3ng.com.br -BEGIN GEEK CODE BLOCK- Version: 3.12 GCS/IT d- s:-: a25 C+++ UBL$ P--- L++ E-- W+++ N++ o? K? w+ O M+ V-- PS+ PE- Y-- PGP++ t+ 5 X++ R+ tv+ b(++) DI+ D++ G e++ h r+++ y+++** --END GEEK CODE BLOCK-- ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Making a ZSQL.DA fully multi-threaded?
Brad Clements wrote: What's the actual problem you're experiencing? I have set the SAPDB timeout to 32400 seconds. Unless you have an extremely heavily loaded Zope, that's too low, and may be causing you problems... Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Making a ZSQL.DA fully multi-threaded?
Brad Clements wrote: That does suck, but it doesn't seem like an insurmountable problem for a DA. You just need a per-DA instance mutex which you acquire before opening a connection release afterwards. Someone else said that every thread has it's own copy of ZODB, hence their own copies of each DA. So putting the mutex on the DA won't help this, will it? That someone else was Jim, I'm just quoting him ;-) Have you looked at my ZOracleDA branch yet? I just use a module level Lock... Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: any limits on object number?
On Mon, 2004-07-19 at 11:38, Dorneles Trema wrote: Resuming what I've learned: don't abuse of BTreeFolder, hash your content and live happy forever... ;-) PS: The server is a modest PIII 1.2GHz, 1GB RAM with a 160MB/s SCSI controller running only one instance of Zope 2.7.1 without ZEO. Thanks a lot. Your report that performance enhances due to hashing is encouraging the decision we took. In our case we already began using BTreeFolder2, havent tried creating the large object base yet (will do this week), but we doing something like what you did. we are creating alphabetized subfolders, by taking the first char of each id. Are there any good popular hashing algorithms? If we have such a algo then further subfoldering could be made automatic after a certain number of objects increase in each folder. This way large scalable databases can be created in Zope. Nagarjuna ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Making a ZSQL.DA fully multi-threaded?
On 19 Jul 2004 at 7:44, Chris Withers wrote: Brad Clements wrote: What's the actual problem you're experiencing? I have set the SAPDB timeout to 32400 seconds. Unless you have an extremely heavily loaded Zope, that's too low, and may be causing you problems... That is the maximum allowed value in dbmgui. Anyway, after setting Zope to be single threaded, I have not had any segfaults. Perhaps I'm wrong on the 'exact' value. I set it to the maximum allowed, perhaps its 324000 seconds. I know it's more than a day so the first number above must be a typo. -- Brad Clements,[EMAIL PROTECTED] (315)268-1000 http://www.murkworks.com (315)268-9812 Fax http://www.wecanstopspam.org/ AOL-IM: BKClements ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: any limits on object number?
Even using BTreeFolder to store all objects I was getting 300s delay to show a single object (ok, it's an archetypes-based one, containing 50 fields, splitted into 7 schematas, with lots of fancy stuff...). So I made a directory hash structure based on UID from each object (an AT UID is md5, so we have a hex base). My own experience with BTreeFolder2 does not support your diagnosis. I do not believe BTreeFolder2 is the problem here. On a CMF-based CMS that I helped develop the largest BTreeFolder2-based containers held ca. 60,000 items at the top level, meaning we did not use a directory structure to restrict the number of items per folder. There was zero delay retrieving singe items and even stepping into the ZMI where it shows 1000 ids at a time was sub-2 second response time. jens ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: any limits on object number?
thanks to all the responses. I will test with BTreeFolder2, and subfoldering with first character of id as another subfolder2, so that working with ZMI will also be possible without further scripting or customized views. i will report the results back to the list. just an additional info: I run into this problem recently... I've got something around 40k objects to insert into ZODB. I started doing some tests with normal Folders, but I discarded it completely in favor of BTreeFolders. I got information that BTreeFolder was already tested with more than 400k objects... and that made me happy! :-) But (there's always a 'but'!)... Even using BTreeFolder to store all objects I was getting 300s delay to show a single object (ok, it's an archetypes-based one, containing 50 fields, splitted into 7 schematas, with lots of fancy stuff...). So I made a directory hash structure based on UID from each object (an AT UID is md5, so we have a hex base). Using this hash structure, with 16 divisions, I got a better performance (but still far from acceptable): 'only' ~100s to show a single object... :-( So I decided to create another level for the hash structure: now each folder has another 16 folder inside it. This time I was getting a delay of ~20s... As you're already thinking... it's time for another hash level. Now, with 16^3 additional BTreeFolders to split all my objects I got an acceptable performance: ~3s... The path is uggly as hell, f/f0/f0a/f0a5aac38aeff101b3168f2592dd879b, but at least the system is usable... Resuming what I've learned: don't abuse of BTreeFolder, hash your content and live happy forever... ;-) I very much doubt you solved the problem you think you solved. Access time to a single object in the hundred of seconds is not a BTreeFolder problem. BTreeFolder is designed to not be a bottleneck for concurrent access and large number of objects. You should have benched (using ZopeProfiler for instance) to find out where time really is spent. Maybe some of your/AT's code does a stupid loop on folder.objectIds() or something. Florent -- Florent Guillaume, Nuxeo (Paris, France) +33 1 40 33 79 87 http://nuxeo.com mailto:[EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] UnpickleableError using SESSION in 2.7.2RC1
Can sessions store instance objects created by an external method?
I'm running Python 2.3.4 (also happens with 2.3.2)
I also get this error with Zope 2.7.0
I'm trying to save a simple instance in a session. The instance is created in an
external
method. I can't figure out why I get this error at the end of the transaction:
Traceback (innermost last):
Module ZPublisher.Publish, line 107, in publish
Module Zope.App.startup, line 222, in commit
Module ZODB.Transaction, line 236, in commit
Module ZODB.Transaction, line 351, in _commit_objects
Module ZODB.Connection, line 416, in commit
- __traceback_info__: (('Products.Transience.TransientObject', 'TransientObject'),
'\x00\x00\x00\x00\x00\x00\x00%', '')
UnpickleableError: Cannot pickle extension class Acquisition.ImplicitAcquirerWrapper
at
409d7800 objects
My external method looks like this::
def CreateInstanceItem(**kw):
Generate an instance item
return InstanceItem(**kw)
And the InstanceItem is::
class InstanceItem(object):
__allow_access_to_unprotected_subobjects__=1
def __init__(self,**kw):
Initialize
self.__dict__ = kw
print Created instance, self
def __repr__(self):
return repr(self.__dict__)
I get output like this from bin/runzope
Created instance {'dn': 'cn=brad,ou=sfiusa,ou=Public,dc=strader-ferris,dc=com',
'isManager':
1, 'preferences': None, 'firstname': 'Brad', 'orgname': 'Ross Video Ltd.', 'lastname':
'Clements', 'userid': '[EMAIL PROTECTED]', 'emailaddress': '[EMAIL PROTECTED]',
'user':
[EMAIL PROTECTED], 'orgid': 'rv', 'isCustomer': 1, 'uid': 3.0}
And my python script that calls the CreateInstanceItem looks like this::
I = container.CreateInstanceItem(userid=userid,
orgid=orgid,
dn=dn,
firstname=firstname,
lastname=lastname,
user=user, # zope user
uid=uid,# our internal uid
preferences=preferences and None,
emailaddress=emailaddress,
orgname=orgname,
isManager=isManager,
isCustomer=isCustomer)
SESSION['user_details'] = I
It seems like the 'I' object is wrapped somehow. From this script I tried
SESSION['user_details'] = I.aq_base
But I got an attribute error, possibly that's a security thing hiding aq_base.
Is there any way to store an instance in a SESSION?
--
Brad Clements,[EMAIL PROTECTED] (315)268-1000
http://www.murkworks.com (315)268-9812 Fax
http://www.wecanstopspam.org/ AOL-IM: BKClements
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] APE and .svn folders, broke ZMI
Hi, I've run into a problem with Ape and I cannot determine if it's caused by a change in Python version, Zope version or Ape version. I have this problem with Ape 0.71, 0.8, and 0.81 And, Zope 2.7.1 and 2.7.2RC1 and Python 2.3.4 Basically, I use subversion to control my file-system files, so each folder that Ape uses also has a .svn subfolder. Sometime, probably when upgrading from Zope 2.7.0 to 2.7.1, the folders (which I think used to show up as .svn) are now showing up w/o a name. So, clicking on the folder (which corresponds to the modification time of the .svn folder), would send me to this URL http://server3.strader-ferris.com:7180/Strader/P/FS/Customer//manage_workspace (note the double slash at the end) I don't need to access .svn folders. But, now I cannot use ZMI to create objects in any Ape hosted folder. I get an error. Error Type: FSWriteError Error Value: Not a legal object name: '' Traceback (innermost last): Module ZPublisher.Publish, line 107, in publish Module Zope.App.startup, line 222, in commit Module ZODB.Transaction, line 236, in commit Module ZODB.Transaction, line 351, in _commit_objects Module apelib.zodb3.connection, line 294, in commit Module apelib.zodb3.storage, line 162, in store Module apelib.core.io, line 110, in store Module apelib.core.gateways, line 85, in store Module apelib.fs.structure, line 124, in store Module apelib.fs.connection, line 270, in write_directory FSWriteError: Not a legal object name: The above traceback occured, for example, when trying to create a file object called 'xyz'. -- Brad Clements,[EMAIL PROTECTED] (315)268-1000 http://www.murkworks.com (315)268-9812 Fax http://www.wecanstopspam.org/ AOL-IM: BKClements ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Troubles with hotfix20040714
On Mon, 19 Jul 2004 09:48:00 -0700 (PDT), C. Olmsted [EMAIL PROTECTED] wrote: Not sure if this is quite the correct list to post to, but I'm having trouble with Hotfix20040714. We're running zope 2.7, plone 2.0.3, and zwiki 0.32.0. Can you test with the 2.7.2 release candidate? This very much sounds similar, but all occurances of TAL_VERSION in the Zope 2.7.0 and 2.7.1 sources are corrected in the 20040714 hotfix. The only ways I can think of to trigger this problem with the hotfix installed is to import compile templates containing macros before the hotfix is loaded during product initialization (reasonably possible), or for someone else to import TAL_VERSION before the hotfix is loaded and generate TAL bytecode themselves (highly unlikely!). It would be interesting to know if the macros are from a PageTemplateFile instance. -Fred -- Fred L. Drake, Jr.fdrake at gmail.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
