Re: [Zope-dev] Massive scalability
Michael Bernstein wrote: I am currently planning two separate 'Archive' type projects/Products. In both cases, I need to make sure that my implementation will scale to hundreds of thousands or even millions of objects. In one project the objects are very simple ZClasses with a few attributes, in the other project, the objects will be instances of the Photo Product, and considerably larger. Do you mean "instances of the Photo Product" or "instances of class FooBar from the Photo Product" ? One implementation I'm considering is a simple Specialist with a Rack. Does anyone know if there are any inherent limitations on the number of objects that can be stored in a Rack? are there any performance limitations at the scale that I'm talking about? Seeing as a Rack can provide data from absolutely anywhere, I can't see a problem with this. If you're talking about the BTree implementation that Racks use when they store data in the ZODB, well, I've never stored more than a few thousand objects in one of those. There certainly aren't the same limitations that you get with the default ObjectManager, as that uses a python dict to hold its sub-objects. The performance limitations will more likely be to do with searching and indexing the data, adding the data in bulk (if you need to do this), and retrieving the data if you have a vast number of clients wanting it all at once. The other implementation I'm considering is to create a ZClass that inherits from ZCatalog and Btree Folder. I can't think why you'd want to do that. What role would instances of this class play in your application? Would this approach run into any scalability problems with the number and type of objects I'm talking about? I think other aspects of your application will determine whether it will scale. Scalabillity is an emergent property of a system. You only get to know about it when you consider the system holisticly. With Zope, where you store objects and how you plan to find objects, is more significant than what the objects you're storing are. -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Tip: Skinscript Debugging
When debugging skinscript, especially tricky dependencies between what
gets computed when during state changes, do the following:
1: Run zope in debug mode (-D)
2: Put an external method called print_ in the root of your ZODB, with
the code:
def print_(s):
print s
3: In your SkinScript, if you want to know when a trigger is called, or
when a set of attributes get computed, use this kind of thing:
WITH QUERY foo(id=some_id) COMPUTE
thing=RESULT,
bar=baz,
_printout_foo=self.print_('query foo called, id=%s' % some_id)
OTHERWISE LET
thing=_.None,
bar='no bar',
_printout_foo=self.print_('query foo called, otherwise clause')
WHEN OBJECT CHANGED CALL
print_('CHANGED start'),
Catalog
.uncatalog_object(_.string.join(self.getPhysicalPath(),'/')),
print_('CHANGED middle'),
Catalog
.catalog_object(self, _.string.join(self.getPhysicalPath(),'/')),
print_('CHANGED end')
4: Read the debug printout, and work out that you're computing
attributes for cataloging before changing the attributes they depend on.
5: Amend code and skinscript, test, document code, sleep.
--
Steve Alexander
Software Engineer
Cat-Box limited
http://www.cat-box.net
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] SkinScript enhancement
While debugging some dependencies in some of my SkinScript, I had an idea.
SkinScript could be extended so that COMPUTE statements can have a
RECOMPUTE FOR clause. This clause would tell a dataskin to invalidate
its cache for the attributes in the COMPUTE statement if any of the
attributes in the RECOMPUTE FOR clause changed.
So,
WITH QUERY some_method(primary_key=ms_vars_id) COMPUTE
foo,bar,baz
OTHERWISE LET
foo='no foo',
bar='no bar',
baz='no baz'
RECOMPUTE FOR
ms_vars_id
This is equivalent to the above statement, without the RECOMPUTE FOR
clause, followed by a trigger:
WHEN OBJECT ADDED, CHANGED CALL
HAS_CHANGED('ms_vars_id')
and self._uncache_attrs(['foo', 'bar', 'baz'])
This assumes a method of DataSkins.DataSkin _uncache_attrs:
def _uncache_attrs(self, names):
v=self._v_attrCache
for name in names:
del v[name]
I haven't tested any of this yet.
--
Steve Alexander
Software Engineer
Cat-Box limited
http://www.cat-box.net
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: SkinScript enhancement
Steve Alexander wrote: This assumes a method of DataSkins.DataSkin _uncache_attrs: def _uncache_attrs(self, names): v=self._v_attrCache for name in names: del v[name] I haven't tested any of this yet. And I realized just after I sent it that the method should be more like: def _uncache_attrs(self, names): v=self._v_attrCache for name in names: if v.has_key(name): del v[name] -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Massive scalability
Steve Alexander wrote: Michael Bernstein wrote: I am currently planning two separate 'Archive' type projects/Products. In both cases, I need to make sure that my implementation will scale to hundreds of thousands or even millions of objects. In one project the objects are very simple ZClasses with a few attributes, in the other project, the objects will be instances of the Photo Product, and considerably larger. Do you mean "instances of the Photo Product" or "instances of class FooBar from the Photo Product" ? Sorry. I meant instances of class Photo from the Photo Product. Does anyone know if there are any inherent limitations on the number of objects that can be stored in a Rack? are there any performance limitations at the scale that I'm talking about? If you're talking about the BTree implementation that Racks use when they store data in the ZODB, well, I've never stored more than a few thousand objects in one of those. There certainly aren't the same limitations that you get with the default ObjectManager, as that uses a python dict to hold its sub-objects. The performance limitations will more likely be to do with searching and indexing the data, adding the data in bulk (if you need to do this), and retrieving the data if you have a vast number of clients wanting it all at once. Yes, I was referring to the way a Rack stores data in the ZODB. Photo instances store several sizes of the same image as attributes of the object, as well as various meta-data fields. I anticipate indexing the meta-data in a ZCatalog. Data will not be added in bulk, but several people may want to retreive the data at the same time (if the site becomes popular). The other implementation I'm considering is to create a ZClass that inherits from ZCatalog and Btree Folder. I can't think why you'd want to do that. What role would instances of this class play in your application? The same as the Rack. A single archive of indexed objects. It seems that this would scale better than creating a ZClass that inherits from ZCatalog and ObjectManager as described here: http://www.zope.org/Members/tseaver/inherit_ZCatalog Would this approach run into any scalability problems with the number and type of objects I'm talking about? I think other aspects of your application will determine whether it will scale. Scalabillity is an emergent property of a system. You only get to know about it when you consider the system holisticly. The system is fairly simple: I want to store a large number of objects in a single location, I want to index them in a ZCatalog, I want to find objects by either searching for them, or by browsing a ZTopics based heirarchy (that is populated with ZCatalog searches as well). The search time (whether a user or ZTopic initiates it) should happen fairly fast, regardless of the number of objects (potentially hundreds of thousands), and direct object retreivals (or rendering) should also happen quickly, without major penalties for the number of objects. With Zope, where you store objects and how you plan to find objects, is more significant than what the objects you're storing are. I hope I've explained myself better this time. Thanks for the help, Michael Bernstein. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
