Re: [Zope] When to use a relational database
On Thu, Feb 07, 2008 at 07:14:46PM -0800, Tim Nash wrote: > Why does storing a single document lead to close to 120 objects being > inserted? I'd bet almost all of that is updates to catalog indexes and metadata. > Is it something related to plone or is this the case for all > ZPT storing? Is it similarly true for the storing of DTML documents? You're confused. The relative speed of ZPT and DTML have only to do with rendering time. As far as the ZODB is concerned, they are equivalent. -- Paul Winkler http://www.slinkp.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] When to use a relational database
On Friday 08 February 2008, Dieter Maurer wrote: > Jaroslav Lukesh wrote at 2008-2-7 22:40 +0100: > >JAvascript nor CSS cannot be dynamically generated from ZPT. > > They can be generated by ZPTs -- though it is not very natural. Have a look at z3c.zrtresource, which is a simple package allowing some variable substitution. It was designed for CSS and JS. Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. "Zope Stephan Richter" ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] When to use a relational database
Thanks Dieter. I will look into using the zope 3 views in 2.10 and compare their performance with DTML. BTW, I hope plone people understand that I am a big fan of plone, zope 3 and zpt. They just don't fit my current application. @quizzical: As others have pointed out, zope really excels at handling semi-structured data. In addition, zope 2 (and maybe 3) is easier to administer than a relational database so it is a database that you can distribute to power users. Tim On Feb 8, 2008 1:32 PM, Dieter Maurer <[EMAIL PROTECTED]> wrote: > Tim Nash wrote at 2008-2-7 19:14 -0800: > > ... > >Why does storing a single document lead to close to 120 objects being > >inserted? > > Almost surely due to full text indexing. > > > > -- > Dieter > ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] When to use a relational database
Tim Nash wrote at 2008-2-7 19:14 -0800: > ... >Why does storing a single document lead to close to 120 objects being >inserted? Almost surely due to full text indexing. -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] When to use a relational database
Jaroslav Lukesh wrote at 2008-2-7 22:40 +0100: >JAvascript nor CSS cannot be dynamically generated from ZPT. They can be generated by ZPTs -- though it is not very natural. -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] When to use a relational database
Tim Nash wrote at 2008-2-7 13:21 -0800: >>From what I have read online, the things that slow zope down are >primarily the numerous security checks In modern Zope versions you can use Zope 3 views. The corresponding templates are trusted. For earlier Zope versions, there is "TrustedExecutables" (they do not yet work for Zope 2.10). >and the large number of objects >that need to be written to the zodb when you save a document. I doubt that the large number of objects is the real reason. The large number of objects result from full text indexing: the object is inserted into the document list for each word it contains. However, full text indexing is slow and expensive - independent from the insertion into the document lists. Obtain a profile and check whether this is really true > ... >Has anyone rated the >performance of say DTML vs ZPT? DTML is likely to be faster: it has a C implementation and is simpler than ZPT. I expect a factor of about 4. However, very often the total time is not dominated by the rendering itself but e.g. determining the values that should be rendered. -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] When to use a relational database
In this article: http://www.upfrontsystems.co.za/Members/roche/where-im-calling-from/zodb-benchmarks Roché Compaan says the following: "For an application like Plone an insertion rate of 250 objects per second results in terrible performance if you consider that storing a single document instance leads to close to 120 objects being inserted in a single transaction. And this explains why you can hardly insert more than 2 or 3 documents per second if you are uploading them into a Plone site." Why does storing a single document lead to close to 120 objects being inserted? Is it something related to plone or is this the case for all ZPT storing? Is it similarly true for the storing of DTML documents? And how can I reduce the number of inserts in my non-plone mostly DTML application? Thanks, Tim On 2/7/08, Jaroslav Lukesh <[EMAIL PROTECTED]> wrote: > JAvascript nor CSS cannot be dynamically generated from ZPT. ZPT is thing > for HTML and XML only. But with DTML you able to generate any binary data, > same as with python script or external method. > > Please read archives few months back, here are many discussions about DTML > vs. ZPT. > > In shorthand - ZPT is academic and Zope core developers language due some > ideas about extralarge projects like CNN which are sublimed in normal > praxis. > > DTML is faster to learn, faster to run, in shorthand is better than ZPT. > > FOr speed recignition, here are this mix of 1:4:10 from time of Zope 2.5.1: > > products or ext. methods are the fastest. DTML is 4-times slower and ZPT is > 10 times slower. Python script are slightly slower than products. > > It is in general of course, real situations may vary. > > JL. > > - Original Message - > From: "Tim Nash" <[EMAIL PROTECTED]> > > > > >From what I have read online, the things that slow zope down are > > primarily the numerous security checks and the large number of objects > > that need to be written to the zodb when you save a document. Is there > > a way we can manage these bottlenecks? Has anyone rated the > > performance of say DTML vs ZPT? For a developer like myself, I derive > > no benefits from zpt because the majority of my design elements are in > > javascript and css. So I will continue to use dtml unless zpt actually > > has better performance. Have performance/scalability studies been done > > for any of the popular zope products? I'd like to reduce the number of > > objects saved to the zodb if I knew how. > > > ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] When to use a relational database
JAvascript nor CSS cannot be dynamically generated from ZPT. ZPT is thing for HTML and XML only. But with DTML you able to generate any binary data, same as with python script or external method. Please read archives few months back, here are many discussions about DTML vs. ZPT. In shorthand - ZPT is academic and Zope core developers language due some ideas about extralarge projects like CNN which are sublimed in normal praxis. DTML is faster to learn, faster to run, in shorthand is better than ZPT. FOr speed recignition, here are this mix of 1:4:10 from time of Zope 2.5.1: products or ext. methods are the fastest. DTML is 4-times slower and ZPT is 10 times slower. Python script are slightly slower than products. It is in general of course, real situations may vary. JL. - Original Message - From: "Tim Nash" <[EMAIL PROTECTED]> >From what I have read online, the things that slow zope down are primarily the numerous security checks and the large number of objects that need to be written to the zodb when you save a document. Is there a way we can manage these bottlenecks? Has anyone rated the performance of say DTML vs ZPT? For a developer like myself, I derive no benefits from zpt because the majority of my design elements are in javascript and css. So I will continue to use dtml unless zpt actually has better performance. Have performance/scalability studies been done for any of the popular zope products? I'd like to reduce the number of objects saved to the zodb if I knew how. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] When to use a relational database
>From what I have read online, the things that slow zope down are primarily the numerous security checks and the large number of objects that need to be written to the zodb when you save a document. Is there a way we can manage these bottlenecks? Has anyone rated the performance of say DTML vs ZPT? For a developer like myself, I derive no benefits from zpt because the majority of my design elements are in javascript and css. So I will continue to use dtml unless zpt actually has better performance. Have performance/scalability studies been done for any of the popular zope products? I'd like to reduce the number of objects saved to the zodb if I knew how. Thanks, Tim On 2/7/08, Dieter Maurer <[EMAIL PROTECTED]> wrote: > quizzical wrote at 2008-2-6 16:03 -0800: > > ... > >Apologies if this is a dumb question, I have had a good look through the > >archives and on the web. My question is basically about how to design a > >website. I really like the ease with which you can build applications in > >zope but I often see comments implying that in order to scale well one may > >want to use a backend relational datbase. I have also seen several comments > >to the effect that one should keep the state of a user session out of the > >ZODB. > > > >I am slightly confused by this, I can't find any information which would > >help me decide what data would be better off in a relational database. > > A relational database is good when you have large amounts > of strictly structured data -- e.g. millions of user records > where each user record has a fews fields with atomic datatypes. > > The ZODB is good when you have weakly structured or unstructured data > and your write rate is not too high. > > The ZODB uses an almost self describing format. Therefore, it can > store virtually everything, without any data model fixed beforehand. > However, this leads to high redundancy and efficiency loss. > This is not so problematic for unstructured or semi structured > content -- as there is no much structure to exploit. > It is also mot problematic for small amounts of data (then optimality > is not an issue). But for large amounts of highly structured data, > you can gain a lot with respect to space and time efficiency from > a relational database. > > > > -- > Dieter > ___ > Zope maillist - Zope@zope.org > http://mail.zope.org/mailman/listinfo/zope > ** No cross posts or HTML encoding! ** > (Related lists - > http://mail.zope.org/mailman/listinfo/zope-announce > http://mail.zope.org/mailman/listinfo/zope-dev ) > ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] When to use a relational database
- Original Message - From: "Dieter Maurer" <[EMAIL PROTECTED]> quizzical wrote at 2008-2-6 16:03 -0800: ... Apologies if this is a dumb question, I have had a good look through the archives and on the web. My question is basically about how to design a website. I really like the ease with which you can build applications in zope but I often see comments implying that in order to scale well one may want to use a backend relational datbase. I have also seen several comments to the effect that one should keep the state of a user session out of the ZODB. I am slightly confused by this, I can't find any information which would help me decide what data would be better off in a relational database. A relational database is good when you have large amounts of strictly structured data -- e.g. millions of user records where each user record has a fews fields with atomic datatypes. For example pricelists etc. But you need to make join with ZODB - with hyperlink (URL1) or ZODB folder object property The ZODB is good when you have weakly structured or unstructured data and your write rate is not too high. ZODB effectivity is very high at tree structured data, as catalogues etc I use ZODB as catalogue container and RDB as pricelist at one travel agency site. If you want to know which, I will send you ULR to private mail and make some source view method for you. But it is written in pure DTML, few computing-intensive parts are in Python (tree structure, etc) and ZSQL to MSSQL. No sql-inject is possible thanks to DTML ;-) Near all other my webs are pure ZODB and uses SQL to store forms data only. Pure SQL driven web I use only for our own ERP, at ZODB are only DTML and bunch of ZSQL. The ZODB uses an almost self describing format. Therefore, it can store virtually everything, without any data model fixed beforehand. However, this leads to high redundancy and efficiency loss. It is possible to use some products like linux symlink to avoid data duplicates. Regards, JL. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] When to use a relational database
Dieter Maurer wrote: > > quizzical wrote at 2008-2-6 16:03 -0800: >> ... >>Apologies if this is a dumb question, I have had a good look through the >>archives and on the web. My question is basically about how to design a >>website. I really like the ease with which you can build applications in >>zope but I often see comments implying that in order to scale well one may >>want to use a backend relational datbase. I have also seen several comments >>to the effect that one should keep the state of a user session out of the >>ZODB. >> >>I am slightly confused by this, I can't find any information which would >>help me decide what data would be better off in a relational database. > > A relational database is good when you have large amounts > of strictly structured data -- e.g. millions of user records > where each user record has a fews fields with atomic datatypes. > > The ZODB is good when you have weakly structured or unstructured data > and your write rate is not too high. > > The ZODB uses an almost self describing format. Therefore, it can > store virtually everything, without any data model fixed beforehand. > However, this leads to high redundancy and efficiency loss. > This is not so problematic for unstructured or semi structured > content -- as there is no much structure to exploit. > It is also mot problematic for small amounts of data (then optimality > is not an issue). But for large amounts of highly structured data, > you can gain a lot with respect to space and time efficiency from > a relational database. > > > > -- > Dieter > ___ > Zope maillist - Zope@zope.org > http://mail.zope.org/mailman/listinfo/zope > ** No cross posts or HTML encoding! ** > (Related lists - > http://mail.zope.org/mailman/listinfo/zope-announce > http://mail.zope.org/mailman/listinfo/zope-dev ) > > Thanks everyone, thats cleared up my confusion and I can get on with Zope-ing (can Zope be a verb) :) Cheers -- View this message in context: http://www.nabble.com/When-to-use-a-relational-database-tp15324690p15341765.html Sent from the Zope - General mailing list archive at Nabble.com. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] When to use a relational database
quizzical wrote at 2008-2-6 16:03 -0800: > ... >Apologies if this is a dumb question, I have had a good look through the >archives and on the web. My question is basically about how to design a >website. I really like the ease with which you can build applications in >zope but I often see comments implying that in order to scale well one may >want to use a backend relational datbase. I have also seen several comments >to the effect that one should keep the state of a user session out of the >ZODB. > >I am slightly confused by this, I can't find any information which would >help me decide what data would be better off in a relational database. A relational database is good when you have large amounts of strictly structured data -- e.g. millions of user records where each user record has a fews fields with atomic datatypes. The ZODB is good when you have weakly structured or unstructured data and your write rate is not too high. The ZODB uses an almost self describing format. Therefore, it can store virtually everything, without any data model fixed beforehand. However, this leads to high redundancy and efficiency loss. This is not so problematic for unstructured or semi structured content -- as there is no much structure to exploit. It is also mot problematic for small amounts of data (then optimality is not an issue). But for large amounts of highly structured data, you can gain a lot with respect to space and time efficiency from a relational database. -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] When to use a relational database
On 2/7/08, quizzical <[EMAIL PROTECTED]> wrote: > > Hi everyone, > > Apologies if this is a dumb question, I have had a good look through the > archives and on the web. My question is basically about how to design a It will be a good idea to look at the discusion on this topic on plone list too. Some hints here http://www.nabble.com/When-to-use-a-relational-database-to15301512s6742.html Following link was suggested in this thread http://www.upfrontsystems.co.za/Members/roche/where-im-calling-from/zodb-benchmarks Chetan ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
