Re: [Zope-dev] Folderish or SimpleItem object types for structural content
Martijn Jacobs wrote at 2005-12-1 01:06 +0100: ... If some conflict errors occure or for some reason the objects are not indexed (correctly) or not updated, some important information is not available for the user. I have experienced alot of problems with unindexed objects, or not reindexed objects due to 'random' conflict errors'. If random conflict errors lead to inconsistencies (object modified but not reindexed), then they are very badly handled: Any exception that bubbles up to ZPublisher causes the complete transaction to be aborted: the effect will be neither an object modification nor a reindexing. If the object is modified but not reindexed in the context of a ConflictError, then something must have caught the ConflictError -- a very bad thing... Unfortunately, unrestricted try: ... except: ... is not too uncommon. Zope some problems in this respect but meanwhile almost all places have been cleaned up. But usually, when an object is modified but not reindexed, it is the application (and not a ConflictError) that made the error -- it often forgets to call the reindex. By making a complete hierarchical structure using 'Object Managers', you can always assure that data is accessible, and if the ZCatalog is not up to date, only the search results will not represent the actual structure. A much more important reason for this structure is acquisition (which you will loose when you follow my proposal). 30.000 is not yet very impressing. That's good to know. It's hard to say offcourse, but what is in some way a 'limit' of the number of objects, for instance, if they all have to be indexed? Indexing objects uses ALOT of CPU time for example... In which amount of objects should you reconsider your design? (Speaking of a 'general' guideline) Usually, you index incrementally (one object at a time). The data structures involved when indexing a single object behave logarithmically in the total number of objects. Thus, I am not worried to have lots of objects indexed in the catalog. -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Folderish or SimpleItem object types for structural content
Martijn Jacobs wrote: catalog. The estimation about the amount of objects, with only the leave nodes as 'SimpleItem' objects will be 30.000. 30,000 is nothing. The production catalog on one of my projects has 220,000 objects in it, and I still wouldn't class that as huge. cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Folderish or SimpleItem object types for structural content
Martijn Jacobs wrote at 2005-11-29 22:09 +0100: ... That will be true, but I was wondering where to draw the line, for example with perfomance issues. All the objects will be in a ZCatalog, to allow searching for example. But one 'ObjectManager' step less in the choice what to do with the products results in much less objects in the catalog. This must be a funny classification scheme... But anyway: I have a completely different strategy for you: lets see whether you will like it. You do not materialize the classification scheme at all but you have only your leaves (let's call them objects). Each object has its classification as an attribute, say category. A category thus has the form topic/subtopic/subsubtopc/ You index the category with a PathIndex (I would recommend my Managable PathIndex) and use canned searches (so called topics) the recreate the classification structure via searches (if needed). If you do the last step (use topics for the complete recreation of the classification structure), then the number of objects will not decrease (what formerly was a classification folder is now a topic). But the new structure is much more flexible. You can now assign different categories (say along different dimensions) to your objects and will get different hierarchical (topical) views on your object set. The estimation about the amount of objects, with only the leave nodes as 'SimpleItem' objects will be 30.000. I never had so many objects in a ZODB, that's why I'm asking this, but as your answer suggests it's still the preferred way. 30.000 is not yet very impressing. And, how large will be your classification scheme? Usually, such a scheme is *much* smaller than the classified items (if the lowest classification level had one (or very few) element, if would be superfluous). Adding it would give a few more thousand objects in addition -- you would get 31.000 (or maybe 35.000) instead of 30.000. Do not be worried about this... You should take care to use the correct folder implementation, however. As soon as a folder is likely to have more than a few dozen of objects, you should use a BTreeFolder2 rather than a normal folder. The total number of ZODB objects will go up by this choice (but you already learned that you need not to worry about this) but access is much more efficient (if the folder contains large numbers of children). -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Folderish or SimpleItem object types for structural content
Hello Dieter. This must be a funny classification scheme... Well, I wanted to have the discussion to be general about this thing because I can imagine that this issue has been some point of discussion before, among others, but I will be more specific to explain : - The structure will have categories within categories. (hierarchical categories) - Each category can have custom properties which can be 'inherited' by the aqcuisition feature of zope. - Within the categories you have object's. - Each object can derive his properties through acquisition, or have their own properties (or override the acquisiton properties) - Each object will have item groups, with their own properties / files / comments / etc. - Each item group will contain items, each item will have properties, but also can have properties derived through acquisition from categories and objects : so these items will all have different properties, depending in which category they exist. With this rules I was thinking about the next structure : Category (Object Manager, within other Categories) - Object (Object Manager) - Item Group (Object Manager) - Item (Simple Item) But anyway: I have a completely different strategy for you: lets see whether you will like it. You do not materialize the classification scheme at all but you have only your leaves (let's call them objects). Each object has its classification as an attribute, say category. A category thus has the form topic/subtopic/subsubtopc/ You index the category with a PathIndex (I would recommend my Managable PathIndex) and use canned searches (so called topics) the recreate the classification structure via searches (if needed). If you do the last step (use topics for the complete recreation of the classification structure), then the number of objects will not decrease (what formerly was a classification folder is now a topic). But the new structure is much more flexible. You can now assign different categories (say along different dimensions) to your objects and will get different hierarchical (topical) views on your object set. I have been thinking about this kind of structure to use for this specific project. But there is a problem that occures (too) often (in my experience) with this approach. By doing this you really are depending on the ZCatalog : If some conflict errors occure or for some reason the objects are not indexed (correctly) or not updated, some important information is not available for the user. I have experienced alot of problems with unindexed objects, or not reindexed objects due to 'random' conflict errors'. By making a complete hierarchical structure using 'Object Managers', you can always assure that data is accessible, and if the ZCatalog is not up to date, only the search results will not represent the actual structure. 30.000 is not yet very impressing. That's good to know. It's hard to say offcourse, but what is in some way a 'limit' of the number of objects, for instance, if they all have to be indexed? Indexing objects uses ALOT of CPU time for example... In which amount of objects should you reconsider your design? (Speaking of a 'general' guideline) You should take care to use the correct folder implementation, however. As soon as a folder is likely to have more than a few dozen of objects, you should use a BTreeFolder2 rather than a normal folder. I've used the BTreeFolder2 before for a single folder containing large number of objects, and indeed, it's really boosting the perfomance a lot... Well thanks again for your thoughts, it helps in breaking patterns I'm used too, let's reconsider it using some external input :) Kind regards, Martijn. -- Martijn Jacobs Four Digits, internet solutions e-mail: [EMAIL PROTECTED] | web: http://www.fourdigits.nl tel: +31 (0)26 44 22 700 | fax: +31 (0)84 22 06 117 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Folderish or SimpleItem object types for structural content
Martijn Jacobs wrote at 2005-11-29 01:02 +0100: All those products are different 'types', with different functionality and different 'roles' as what they represent, so I write different products for every type. For every object that can contain subobjects you could derive it from 'Folder' (or Object Manager). But when you derive all these product types from 'Folder' you end up with ALOT of objects in your database. It that a problem? It might become one once you get millions of objects... Usually, I would expect that most objects come from you leaves and not from the classification structure... In a lot of real world cases you have categories and items, that makes things easy, but what about the structure explained above? I would use ObjectManagers to implement the classification structure and SimpleItems for the leaves. -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Folderish or SimpleItem object types for structural content
Hello Dieter. ...For every object that can contain subobjects you could derive it from 'Folder' (or Object Manager). But when you derive all these product types from 'Folder' you end up with ALOT of objects in your database. It that a problem? It might become one once you get millions of objects... Usually, I would expect that most objects come from you leaves and not from the classification structure... That will be true, but I was wondering where to draw the line, for example with perfomance issues. All the objects will be in a ZCatalog, to allow searching for example. But one 'ObjectManager' step less in the choice what to do with the products results in much less objects in the catalog. The estimation about the amount of objects, with only the leave nodes as 'SimpleItem' objects will be 30.000. I never had so many objects in a ZODB, that's why I'm asking this, but as your answer suggests it's still the preferred way. Thanks for the comments! -- Martijn Jacobs Four Digits, internet solutions e-mail: [EMAIL PROTECTED] | web: http://www.fourdigits.nl tel: +31 (0)26 44 22 700 | fax: +31 (0)84 22 06 117 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Folderish or SimpleItem object types for structural content
Hello guys! I have a simple question about the design of hierarchical zope product structures in Zope 2. For example, you have categories, within categories you have sub-categories, within sub-categories you have sub-sub categories, within that you have content items, within content items you haveetcetera. All those products are different 'types', with different functionality and different 'roles' as what they represent, so I write different products for every type. For every object that can contain subobjects you could derive it from 'Folder' (or Object Manager). But when you derive all these product types from 'Folder' you end up with ALOT of objects in your database. In a lot of real world cases you have categories and items, that makes things easy, but what about the structure explained above? My question for you guys is : When do you decide to make it a Folderish object which contains sub-objects, or to make it a Simple Item object with, for example, some structured XML content, or dictionary / list properties containing the 'sub-structure'? The last option is more work, because you have to write your add/edit/delete/etc code yourself, writing it with Folders makes it more easily, but then you have an explosion of objects in the ZODB. I would like to have some fresh thoughts about this, so maybe you can help. Thank you in advance, kind regards, Martijn. -- Martijn Jacobs Four Digits, internet solutions e-mail: [EMAIL PROTECTED] | web: http://www.fourdigits.nl tel: +31 (0)26 44 22 700 | fax: +31 (0)84 22 06 117 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
