[ZODB-Dev] zeodb limitation with zope 2.12
Hi We are new to zope, We are planning a web application that saves the data of up to 1,000,000 users. I would like to know the limitation of zodb. Currently we are using ClientStorage with zeo, And zeo uses filestorage. What are the limitation of using filestorage? Is there a limit on the number of objects ? Is there a limitation of the global size of the database? Is there an easy way to split the data using several fileStorage? Do you recommend using DirectoryStorage or any other type of storage. I know I send many question together, I will aspirate any answer to one of question. Cheers Dganit ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zeodb limitation with zope 2.12
On 14.05.09 08:29, Dganit David wrote: Hi We are new to zope, We are planning a web application that saves the data of up to 1,000,000 users. This is a pointless number. The estimated total number of objects is relevant. I would like to know the limitation of zodb. Currently we are using ClientStorage with zeo, And zeo uses filestorage. What are the limitation of using filestorage? Basically one - expect limitations given by the OS. *Practical* limitations are introduced when it comes to aspects of packing and doing backups. A single filestorage should perhaps not become bigger than around 100GB in order to deal with packing and backup in a sane way. Is there a limit on the number of objects ? Basically no. Zope maintains an index of all objects in RAM (whose size is linear to the #objects). Is there a limitation of the global size of the database? no Is there an easy way to split the data using several fileStorage? Multiple storages can be mounted . Do you recommend using DirectoryStorage or any other type of storage. Consider consulting a Zope consultant with expertise in building large scale Zope apps. Building an application for 1M users requires careful planning. Nothing you can accomplish with basic Zope skills. Especially have in mind that the ZODB is not the best solution for heavy concurrent write operations. -aj begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:i...@zopyx.com title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zeodb limitation with zope 2.12
2009/5/14 Dganit David dgani...@puresight.com: We are planning a web application that saves the data of up to 1,000,000 users. I would like to know the limitation of zodb. Currently we are using ClientStorage with zeo, In my experience building a large scale application, ZEO did not scale with my application in the way that I had hoped. It worked great with a few hosts accessing the ZEO server in parallel, but when scaled up the ZEO server itself quickly became a bottleneck which had to be designed around. The thing that surprised me was that the server itself became the bottleneck with only 4 hosts active - it obviously depends on the access patterns of your application. A single ZEO server may not serve the needs of 1 million users if you have a large number accessing the application concurrently, so I suggest that you think about multiple servers, replication, and dividing the data itself up to enabling parallel distributed access before starting to code the application. And ensure that you have a test suite that will simulate the large scale requirements from the beginning. The only other limitation was packing - when the database grew to several GB it became a lengthy process to pack, and eventually packing became impossible due to the large memory requirements (I posted to the list at the time, and discovered that this was expected behaviour). Once you understand that ZEO is not some magic solution, and that you still need to think about all the usual database problems, then things will fall into place quite easily ;-) ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zeodb limitation with zope 2.12
On 14.05.09 11:59, Chris Bainbridge wrote: 2009/5/14 Dganit David dgani...@puresight.com: We are planning a web application that saves the data of up to 1,000,000 users. I would like to know the limitation of zodb. Currently we are using ClientStorage with zeo, In my experience building a large scale application, ZEO did not scale with my application in the way that I had hoped. It worked great with a few hosts accessing the ZEO server in parallel, but when scaled up the ZEO server itself quickly became a bottleneck which had to be designed around. The thing that surprised me was that the server itself became the bottleneck with only 4 hosts active - it obviously depends on the access patterns of your application. A single ZEO server may not serve the needs of 1 million users if you have a large number accessing the application concurrently, so I suggest that you think about multiple servers, replication, and dividing the data itself up to enabling parallel distributed access before starting to code the application. And ensure that you have a test suite that will simulate the large scale requirements from the beginning. A large scale-application like this always requires careful planning - independent of using the ZODB or a RDBMS. Assuming that this is a big project, there should be a budget for architecture :-) Andreas -- ZOPYX Ltd. Co. KG - Charlottenstr. 37/1 - 72070 Tübingen - Germany Web: www.zopyx.com - Email: i...@zopyx.com - Phone +49 - 7071 - 793376 Registergericht: Amtsgericht Stuttgart, Handelsregister A 381535 Geschäftsführer/Gesellschafter: ZOPYX Limited, Birmingham, UK E-Publishing, Python, Zope Plone development, Consulting begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:i...@zopyx.com title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
