Re: [Zope-dev] Re: any limits on object number?

[Florent Guillaume]

> > thanks to all the responses.  I will test with BTreeFolder2, and
> > subfoldering with first character of id as another subfolder2, so that
> > working with ZMI will also be possible without further scripting or
> > customized views.  i will report the results back to the list.
> 
> just an additional info: I run into this problem recently...
> 
> I've got something around 40k objects to insert into ZODB.
> 
> I started doing some tests with normal Folders, but I discarded
> it completely in favor of BTreeFolders. I got information that
> BTreeFolder was already tested with more than 400k objects...
> and that made me happy! :-)
> 
> But (there's always a 'but'!)...
> 
> Even using BTreeFolder to store all objects I was getting >300s
> delay to show a single object (ok, it's an archetypes-based one,
> containing >50 fields, splitted into 7 schematas, with lots of
> fancy stuff...). So I made a directory hash structure based on
> UID from each object (an AT UID is md5, so we have a hex base).
> 
> Using this hash structure, with 16 divisions, I got a better
> performance (but still far from acceptable): 'only' ~100s to
> show a single object... :-(
> 
> So I decided to create another level for the hash structure:
> now each folder has another 16 folder inside it. This time I
> was getting a delay of ~20s...
> 
> As you're already thinking... it's time for another hash level.
> Now, with 16^3 additional BTreeFolders to split all my objects
> I got an acceptable performance: ~3s...
> 
> The path is uggly as hell, f/f0/f0a/f0a5aac38aeff101b3168f2592dd879b,
> but at least the system is usable...
> 
> Resuming what I've learned: don't abuse of BTreeFolder, hash
> your content and live happy forever... ;-)

I very much doubt you solved the problem you think you solved. Access
time to a single object in the hundred of seconds is not a BTreeFolder
problem. BTreeFolder is designed to not be a bottleneck for concurrent
access and large number of objects.

You should have benched (using ZopeProfiler for instance) to find out
where time really is spent. Maybe some of your/AT's code does a stupid
loop on folder.objectIds() or something.

Florent

-- 
Florent Guillaume, Nuxeo (Paris, France)
+33 1 40 33 79 87  http://nuxeo.com  mailto:[EMAIL PROTECTED]
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] Re: any limits on object number?

[Jens Vagelpohl]

Even using BTreeFolder to store all objects I was getting >300s
delay to show a single object (ok, it's an archetypes-based one,
containing >50 fields, splitted into 7 schematas, with lots of
fancy stuff...). So I made a directory hash structure based on
UID from each object (an AT UID is md5, so we have a hex base).
My own experience with BTreeFolder2 does not support your diagnosis. I 
do not believe BTreeFolder2 is the problem here.

On a CMF-based CMS that I helped develop the largest BTreeFolder2-based 
containers held ca. 60,000 items at the top level, meaning we did not 
use a directory structure to restrict the number of items per folder. 
There was zero delay retrieving singe items and even stepping into the 
ZMI where it shows 1000 ids at a time was sub-2 second response time.

jens
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

[Zope-dev] Re: any limits on object number?

[Dorneles Treméa]

Hi Nagarjuna,
Thanks a lot.  Your report that performance enhances due to hashing is
encouraging the decision we took. In our case we already began using
BTreeFolder2, havent tried creating the large object base yet (will do
this week), but we doing something like what you did.  we are creating
alphabetized subfolders, by taking the first char of each id.  Are there
any good popular hashing algorithms?  If we have such a algo then
further subfoldering could be made automatic after a certain number of
objects increase in each folder.  This way large scalable databases can
be created in Zope.
well, a hash is a... hash! :-) You can use any deterministic function
to populate your buckets... Just choose one who fits your needs.
The current code that I'm using is available[1], just keep in mind
that it was written after 2 badly slept nights... ;-)
[1]http://cvs.x3ng.com.br/cgi-bin/viewcvs.cgi/Recria/utils.py?rev=HEAD&view=auto
Regards,
--
Dorneles TremÃa
Caxias do Sul - RS - Brasil
+55 54 9114 9312 - UIN: 2413568
X3ng Web Technology 
-BEGIN GEEK CODE BLOCK-
Version: 3.12
GCS/IT d- s:->: a25 C+++ UBL$ P--- L++ E-- W+++
N++ o? K? w+ O M+ V-- PS+ PE- Y-- PGP++ t+ 5 X++ R+
tv+ b(++) DI+ D++ G+>+++ e++> h r+++ y+++**
--END GEEK CODE BLOCK--
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] Re: any limits on object number?

[Nagarjuna G.]

On Mon, 2004-07-19 at 11:38, Dorneles TremÃa wrote:

> Resuming what I've learned: don't abuse of BTreeFolder, hash
> your content and live happy forever... ;-)
> 
> PS: The server is a modest PIII 1.2GHz, 1GB RAM with a 160MB/s
> SCSI controller running only one instance of Zope 2.7.1 without
> ZEO.

Thanks a lot.  Your report that performance enhances due to hashing is
encouraging the decision we took. In our case we already began using
BTreeFolder2, havent tried creating the large object base yet (will do
this week), but we doing something like what you did.  we are creating
alphabetized subfolders, by taking the first char of each id.  Are there
any good popular hashing algorithms?  If we have such a algo then
further subfoldering could be made automatic after a certain number of
objects increase in each folder.  This way large scalable databases can
be created in Zope.

Nagarjuna

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )

[Zope-dev] Re: any limits on object number?

[Dorneles Treméa]

Hello all,
thanks to all the responses.  I will test with BTreeFolder2, and
subfoldering with first character of id as another subfolder2, so that
working with ZMI will also be possible without further scripting or
customized views.  i will report the results back to the list.
just an additional info: I run into this problem recently...
I've got something around 40k objects to insert into ZODB.
I started doing some tests with normal Folders, but I discarded
it completely in favor of BTreeFolders. I got information that
BTreeFolder was already tested with more than 400k objects...
and that made me happy! :-)
But (there's always a 'but'!)...
Even using BTreeFolder to store all objects I was getting >300s
delay to show a single object (ok, it's an archetypes-based one,
containing >50 fields, splitted into 7 schematas, with lots of
fancy stuff...). So I made a directory hash structure based on
UID from each object (an AT UID is md5, so we have a hex base).
Using this hash structure, with 16 divisions, I got a better
performance (but still far from acceptable): 'only' ~100s to
show a single object... :-(
So I decided to create another level for the hash structure:
now each folder has another 16 folder inside it. This time I
was getting a delay of ~20s...
As you're already thinking... it's time for another hash level.
Now, with 16^3 additional BTreeFolders to split all my objects
I got an acceptable performance: ~3s...
The path is uggly as hell, f/f0/f0a/f0a5aac38aeff101b3168f2592dd879b,
but at least the system is usable...
Resuming what I've learned: don't abuse of BTreeFolder, hash
your content and live happy forever... ;-)
PS: The server is a modest PIII 1.2GHz, 1GB RAM with a 160MB/s
SCSI controller running only one instance of Zope 2.7.1 without
ZEO.
HTH,
--
Dorneles Treméa
Caxias do Sul - RS - Brasil
+55 54 9114 9312 - UIN: 2413568
X3ng Web Technology 
-BEGIN GEEK CODE BLOCK-
Version: 3.12
GCS/IT d- s:->: a25 C+++ UBL$ P--- L++ E-- W+++
N++ o? K? w+ O M+ V-- PS+ PE- Y-- PGP++ t+ 5 X++ R+
tv+ b(++) DI+ D++ G+>+++ e++> h r+++ y+++**
--END GEEK CODE BLOCK--
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] Re: any limits on object number?

[Nagarjuna G.]

On Wed, 2004-07-14 at 18:55, Casey Duncan wrote:
> On Wed, 14 Jul 2004 13:46:57 +0530
> "Nagarjuna G." <[EMAIL PROTECTED]> wrote:
> 
> > We are builiding a large portal using Zope.  We need to create a large
> > number of objects.  The data component of the objects is small, but
> > each object carries lots of metadata.  My question is: Is there any
> > limit on the number of objects in a given folder?  I am not planning
> > to use any external RDBMS. Are there any known performace isssues when
> > the numbeer of objects increase, particularly when we store them in
> > the same folder?
> 
> Normal zope folders should probably not be used to hold more than a few
> dozen items. They store a list of their children in a single ZODB
> record, and as their numbers increase so do the size of the transactions
> that change the folder. Also normal Zope folders do not handle
> concurrent updates and will thus not perform well when multiple users
> are adding items to the folder.
> 
> Shane Hathaway's BTreeFolder2 product solves these problems. It is the
> thing to use when you want to store large numbers of objects in a single
> folder. It also handles concurrency much better.
> 
> hth,

thanks to all the responses.  I will test with BTreeFolder2, and
subfoldering with first character of id as another subfolder2, so that
working with ZMI will also be possible without further scripting or
customized views.  i will report the results back to the list.

Nagarjuna

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )

[Zope-dev] Re: any limits on object number?

[Casey Duncan]

On Wed, 14 Jul 2004 13:46:57 +0530
"Nagarjuna G." <[EMAIL PROTECTED]> wrote:

> We are builiding a large portal using Zope.  We need to create a large
> number of objects.  The data component of the objects is small, but
> each object carries lots of metadata.  My question is: Is there any
> limit on the number of objects in a given folder?  I am not planning
> to use any external RDBMS. Are there any known performace isssues when
> the numbeer of objects increase, particularly when we store them in
> the same folder?

Normal zope folders should probably not be used to hold more than a few
dozen items. They store a list of their children in a single ZODB
record, and as their numbers increase so do the size of the transactions
that change the folder. Also normal Zope folders do not handle
concurrent updates and will thus not perform well when multiple users
are adding items to the folder.

Shane Hathaway's BTreeFolder2 product solves these problems. It is the
thing to use when you want to store large numbers of objects in a single
folder. It also handles concurrency much better.

hth,

-Casey

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )