Re: [Zope] When to use a relational database

2008-02-09 Thread Tim Nash
Thanks Dieter. I will look into using the zope 3 views in 2.10 and
compare their performance with DTML. BTW, I hope plone people
understand that I am a big fan of plone, zope 3 and zpt. They just
don't fit my current application.

@quizzical:  As others have pointed out, zope really excels at
handling semi-structured data.  In addition, zope 2 (and maybe 3) is
easier to administer than a relational database so it is a database
that you can distribute to power users.

Tim

On Feb 8, 2008 1:32 PM, Dieter Maurer [EMAIL PROTECTED] wrote:
 Tim Nash wrote at 2008-2-7 19:14 -0800:
  ...
 Why does storing a single document lead to close to 120 objects being
 inserted?

 Almost surely due to full text indexing.



 --
 Dieter

___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] When to use a relational database

2008-02-09 Thread Stephan Richter
On Friday 08 February 2008, Dieter Maurer wrote:
 Jaroslav Lukesh wrote at 2008-2-7 22:40 +0100:
 JAvascript nor CSS cannot be dynamically generated from ZPT.

 They can be generated by ZPTs -- though it is not very natural.

Have a look at z3c.zrtresource, which is a simple package allowing some 
variable substitution. It was designed for CSS and JS.

Regards,
Stephan
-- 
Stephan Richter
Web Software Design, Development and Training
Google me. Zope Stephan Richter
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] When to use a relational database

2008-02-09 Thread Paul Winkler
On Thu, Feb 07, 2008 at 07:14:46PM -0800, Tim Nash wrote:
 Why does storing a single document lead to close to 120 objects being
 inserted?

I'd bet almost all of that is updates to catalog indexes and metadata.

 Is it something related to plone or is this the case for all
 ZPT storing? Is it similarly true for the storing of DTML documents?

You're confused. The relative speed of ZPT and DTML have only to do
with rendering time.  As far as the ZODB is concerned, they are
equivalent.


-- 

Paul Winkler
http://www.slinkp.com
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] When to use a relational database

2008-02-08 Thread Dieter Maurer
Tim Nash wrote at 2008-2-7 19:14 -0800:
 ...
Why does storing a single document lead to close to 120 objects being
inserted?

Almost surely due to full text indexing.



-- 
Dieter
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] When to use a relational database

2008-02-08 Thread Dieter Maurer
Jaroslav Lukesh wrote at 2008-2-7 22:40 +0100:
JAvascript nor CSS cannot be dynamically generated from ZPT.

They can be generated by ZPTs -- though it is not very natural.



-- 
Dieter
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] When to use a relational database

2008-02-08 Thread Dieter Maurer
Tim Nash wrote at 2008-2-7 13:21 -0800:
From what I have read online, the things that slow zope down are
primarily the numerous security checks

In modern Zope versions you can use Zope 3 views.
The corresponding templates are trusted.

For earlier Zope versions, there is TrustedExecutables (they do not
yet work for Zope 2.10).

and the large number of objects
that need to be written to the zodb when you save a document.

I doubt that the large number of objects is the real reason.
The large number of objects result from full text indexing:
the object is inserted into the document list for each word it
contains. However, full text indexing is slow and expensive - independent
from the insertion into the document lists.

Obtain a profile and check whether this is really true

 ...
Has anyone rated the
performance of say DTML vs ZPT?

DTML is likely to be faster: it has a C implementation and is
simpler than ZPT. I expect a factor of about 4.

However, very often the total time is not dominated by the
rendering itself but e.g. determining the values that should be rendered.



-- 
Dieter
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] When to use a relational database

2008-02-07 Thread quizzical



Dieter Maurer wrote:
 
 quizzical wrote at 2008-2-6 16:03 -0800:
 ...
Apologies if this is a dumb question, I have had a good look through the
archives and on the web. My question is basically about how to design a
website. I really like the ease with which you can build applications in
zope but I often see comments implying that in order to scale well one may
want to use a backend relational datbase. I have also seen several
comments
to the effect that one should keep the state of a user session out of the
ZODB. 

I am slightly confused by this, I can't find any information which would
help me decide what data would be better off in a relational database.
 
 A relational database is good when you have large amounts
 of strictly structured data -- e.g. millions of user records
 where each user record has a fews fields with atomic datatypes.
 
 The ZODB is good when you have weakly structured or unstructured data
 and your write rate is not too high.
 
 The ZODB uses an almost self describing format. Therefore, it can
 store virtually everything, without any data model fixed beforehand.
 However, this leads to high redundancy and efficiency loss.
 This is not so problematic for unstructured or semi structured
 content -- as there is no much structure to exploit.
 It is also mot problematic for small amounts of data (then optimality
 is not an issue). But for large amounts of highly structured data,
 you can gain a lot with respect to space and time efficiency from
 a relational database.
 
 
 
 -- 
 Dieter
 ___
 Zope maillist  -  Zope@zope.org
 http://mail.zope.org/mailman/listinfo/zope
 **   No cross posts or HTML encoding!  **
 (Related lists - 
  http://mail.zope.org/mailman/listinfo/zope-announce
  http://mail.zope.org/mailman/listinfo/zope-dev )
 
 

Thanks everyone, thats cleared up my confusion and I can get on with
Zope-ing (can Zope be a verb) :)
Cheers

-- 
View this message in context: 
http://www.nabble.com/When-to-use-a-relational-database-tp15324690p15341765.html
Sent from the Zope - General mailing list archive at Nabble.com.

___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] When to use a relational database

2008-02-07 Thread Jaroslav Lukesh
- Original Message - 
From: Dieter Maurer [EMAIL PROTECTED]

quizzical wrote at 2008-2-6 16:03 -0800:

...
Apologies if this is a dumb question, I have had a good look through the
archives and on the web. My question is basically about how to design a
website. I really like the ease with which you can build applications in
zope but I often see comments implying that in order to scale well one may
want to use a backend relational datbase. I have also seen several 
comments

to the effect that one should keep the state of a user session out of the
ZODB.

I am slightly confused by this, I can't find any information which would
help me decide what data would be better off in a relational database.


A relational database is good when you have large amounts
of strictly structured data -- e.g. millions of user records
where each user record has a fews fields with atomic datatypes.


For example pricelists etc. But you need to make join with ZODB - with 
hyperlink (URL1) or ZODB folder object property



The ZODB is good when you have weakly structured or unstructured data
and your write rate is not too high.


ZODB effectivity is very high at tree structured data, as catalogues etc

I use ZODB as catalogue container and RDB as pricelist at one travel agency 
site. If you want to know which, I will send you ULR to private mail and 
make some source view method for you. But it is written in pure DTML, few 
computing-intensive parts are in Python (tree structure, etc) and ZSQL to 
MSSQL. No sql-inject is possible thanks to DTML ;-)


Near all other my webs are pure ZODB and uses SQL to store forms data only.

Pure SQL driven web I use only for our own ERP, at ZODB are only DTML and 
bunch of ZSQL.



The ZODB uses an almost self describing format. Therefore, it can
store virtually everything, without any data model fixed beforehand.
However, this leads to high redundancy and efficiency loss.



It is possible to use some products like linux symlink to avoid data 
duplicates.


Regards, JL. 


___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] When to use a relational database

2008-02-07 Thread Jaroslav Lukesh
JAvascript nor CSS cannot be dynamically generated from ZPT. ZPT is thing 
for HTML and XML only. But with DTML you able to generate any binary data, 
same as with python script or external method.


Please read archives few months back, here are many discussions about DTML 
vs. ZPT.


In shorthand - ZPT is academic and Zope core developers language due some 
ideas about extralarge projects like CNN which are sublimed in normal 
praxis.


DTML is faster to learn, faster to run, in shorthand is better than ZPT.

FOr speed recignition, here are this mix of 1:4:10 from time of Zope 2.5.1:

products or ext. methods are the fastest. DTML is 4-times slower and ZPT is 
10 times slower. Python script are slightly slower than products.


It is in general of course, real situations may vary.

JL.

- Original Message - 
From: Tim Nash [EMAIL PROTECTED]




From what I have read online, the things that slow zope down are
primarily the numerous security checks and the large number of objects
that need to be written to the zodb when you save a document. Is there
a way we can manage these bottlenecks? Has anyone rated the
performance of say DTML vs ZPT? For a developer like myself, I derive
no benefits from zpt because the majority of my design elements are in
javascript and css. So I will continue to use dtml unless zpt actually
has better performance. Have performance/scalability studies been done
for any of the popular zope products? I'd like to reduce the number of
objects saved to the zodb if I knew how.


___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] When to use a relational database

2008-02-07 Thread Tim Nash
From what I have read online, the things that slow zope down are
primarily the numerous security checks and the large number of objects
that need to be written to the zodb when you save a document. Is there
a way we can manage these bottlenecks? Has anyone rated the
performance of say DTML vs ZPT? For a developer like myself, I derive
no benefits from zpt because the majority of my design elements are in
javascript and css. So I will continue to use dtml unless zpt actually
has better performance. Have performance/scalability studies been done
for any of the popular zope products? I'd like to reduce the number of
objects saved to the zodb if I knew how.

Thanks,
Tim

On 2/7/08, Dieter Maurer [EMAIL PROTECTED] wrote:
 quizzical wrote at 2008-2-6 16:03 -0800:
  ...
 Apologies if this is a dumb question, I have had a good look through the
 archives and on the web. My question is basically about how to design a
 website. I really like the ease with which you can build applications in
 zope but I often see comments implying that in order to scale well one may
 want to use a backend relational datbase. I have also seen several comments
 to the effect that one should keep the state of a user session out of the
 ZODB.
 
 I am slightly confused by this, I can't find any information which would
 help me decide what data would be better off in a relational database.

 A relational database is good when you have large amounts
 of strictly structured data -- e.g. millions of user records
 where each user record has a fews fields with atomic datatypes.

 The ZODB is good when you have weakly structured or unstructured data
 and your write rate is not too high.

 The ZODB uses an almost self describing format. Therefore, it can
 store virtually everything, without any data model fixed beforehand.
 However, this leads to high redundancy and efficiency loss.
 This is not so problematic for unstructured or semi structured
 content -- as there is no much structure to exploit.
 It is also mot problematic for small amounts of data (then optimality
 is not an issue). But for large amounts of highly structured data,
 you can gain a lot with respect to space and time efficiency from
 a relational database.



 --
 Dieter
 ___
 Zope maillist  -  Zope@zope.org
 http://mail.zope.org/mailman/listinfo/zope
 **   No cross posts or HTML encoding!  **
 (Related lists -
  http://mail.zope.org/mailman/listinfo/zope-announce
  http://mail.zope.org/mailman/listinfo/zope-dev )

___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] When to use a relational database

2008-02-07 Thread Dieter Maurer
quizzical wrote at 2008-2-6 16:03 -0800:
 ...
Apologies if this is a dumb question, I have had a good look through the
archives and on the web. My question is basically about how to design a
website. I really like the ease with which you can build applications in
zope but I often see comments implying that in order to scale well one may
want to use a backend relational datbase. I have also seen several comments
to the effect that one should keep the state of a user session out of the
ZODB. 

I am slightly confused by this, I can't find any information which would
help me decide what data would be better off in a relational database.

A relational database is good when you have large amounts
of strictly structured data -- e.g. millions of user records
where each user record has a fews fields with atomic datatypes.

The ZODB is good when you have weakly structured or unstructured data
and your write rate is not too high.

The ZODB uses an almost self describing format. Therefore, it can
store virtually everything, without any data model fixed beforehand.
However, this leads to high redundancy and efficiency loss.
This is not so problematic for unstructured or semi structured
content -- as there is no much structure to exploit.
It is also mot problematic for small amounts of data (then optimality
is not an issue). But for large amounts of highly structured data,
you can gain a lot with respect to space and time efficiency from
a relational database.



-- 
Dieter
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] When to use a relational database

2008-02-07 Thread Tim Nash
In this article:
http://www.upfrontsystems.co.za/Members/roche/where-im-calling-from/zodb-benchmarks

Roché Compaan says the following:

For an application like Plone an insertion rate of 250 objects per
second results in terrible performance if you consider that storing a
single document instance leads to close to 120 objects being inserted
in a single transaction. And this explains why you can hardly insert
more than 2 or 3 documents per second if you are uploading them into a
Plone site.

Why does storing a single document lead to close to 120 objects being
inserted? Is it something related to plone or is this the case for all
ZPT storing? Is it similarly true for the storing of DTML documents?
And how can I reduce the number of inserts in my non-plone mostly DTML
application?

Thanks,
Tim


On 2/7/08, Jaroslav Lukesh [EMAIL PROTECTED] wrote:
 JAvascript nor CSS cannot be dynamically generated from ZPT. ZPT is thing
 for HTML and XML only. But with DTML you able to generate any binary data,
 same as with python script or external method.

 Please read archives few months back, here are many discussions about DTML
 vs. ZPT.

 In shorthand - ZPT is academic and Zope core developers language due some
 ideas about extralarge projects like CNN which are sublimed in normal
 praxis.

 DTML is faster to learn, faster to run, in shorthand is better than ZPT.

 FOr speed recignition, here are this mix of 1:4:10 from time of Zope 2.5.1:

 products or ext. methods are the fastest. DTML is 4-times slower and ZPT is
 10 times slower. Python script are slightly slower than products.

 It is in general of course, real situations may vary.

 JL.

 - Original Message -
 From: Tim Nash [EMAIL PROTECTED]


  From what I have read online, the things that slow zope down are
  primarily the numerous security checks and the large number of objects
  that need to be written to the zodb when you save a document. Is there
  a way we can manage these bottlenecks? Has anyone rated the
  performance of say DTML vs ZPT? For a developer like myself, I derive
  no benefits from zpt because the majority of my design elements are in
  javascript and css. So I will continue to use dtml unless zpt actually
  has better performance. Have performance/scalability studies been done
  for any of the popular zope products? I'd like to reduce the number of
  objects saved to the zodb if I knew how.
 

___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] When to use a relational database

2008-02-06 Thread Chetan Kumar
On 2/7/08, quizzical [EMAIL PROTECTED] wrote:

 Hi everyone,

 Apologies if this is a dumb question, I have had a good look through the
 archives and on the web. My question is basically about how to design a

It will be a good idea to look at the discusion on this topic on plone list too.
Some hints here
http://www.nabble.com/When-to-use-a-relational-database-to15301512s6742.html
Following link was suggested in this thread
http://www.upfrontsystems.co.za/Members/roche/where-im-calling-from/zodb-benchmarks
Chetan
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )