RE: [Zope] How can I get a listing of all objects in the Zope 2.62 ZODB?

[Dieter Maurer]

Mark, Jonathan (Integic) wrote at 2006-9-13 11:06 -0400:
> ...
>To convert Squishdot to use an RDBMS I would have to modify the Posting and 
>other product scripts in Squishdot. If I could look at the Squishdot-related 
>objects in the ZODB I would have a better understanding of what Squishdot is 
>doing when it posts and retrieves a message. I could then duplicate that 
>behavior with Gadfly.

The "Gadfly" implementation included in Zope is bad -- do not use it
in earnest.

*If* you really want to go with a relational database, use a real one
(Postgres, MySQL (with transactional tables), Oracle, ...).

The alternative is to learn what make Squishdot slow and fix that.
A profile (e.g. with "ZopeProfiler") may help you to understand what
happens.



-- 
Dieter
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] How can I get a listing of all objects in the Zope 2.62 ZODB?

[Chris Withers]

Mark, Jonathan (Integic) wrote:

has been a positive one. I am grateful to Chris for writing it and am
glad that he has moved on to other cutting edge Zope projects.


What gives you the idea I've "moved on" anywhere? I still use Squishdot 
heavily and deployed a new instance for a customer a couple of months ago...



However, there is one problem. As the number of posts gets past 2000
the application slows to a crawl.


That's not been my experience at all..

But, it does highlight one thing: you must be a developer - coming with 
a solution rather than a problem and asking a question that has little 
to do with either ;-)


The question I believe you meant to ask is "My squishdot site is slow, 
how can I speed it up?" rather than this stuff about zodb internals.


You could have even asked on the Squishdot mailing list ;-)


You can observe the problem by comparing
http://jonathanmark.com/Blog/1128110364/index_html, which is in a
Squishdot instance with 3000 posts and is slow, and
http://goodbyejim.com, which has 800 posts. The front page of
jonathanmark.com/Blog is cached, hence one must visit an interior page
to observe the problem. 


Caching can be a legitimate solution to these problems, though I've 
never needed it myself...



All of these Squishdot instances run in the same Zope instance.


Okay, in this order:

- what version of Squishdot are you using?

- what version of Zope?

- what customisations have you done to the various templates?
  (I'd put money on something in here inadvertantly trying to drag all
   the posts into memory...)

- what size is your Data.fs?

- How much memory does the box have?

- What configuration changes have you made to Zope to ensure the object
  cache is of a useful size?


So I decided that I would modify Squishdot so that it uses a
relational database to store posts. 


"hey, this hammer's broken, gimme that wrench and I'll see if I can bash 
this nail in"


This is an awful lot of work for very little gain, why don't you 
actually try and solve the problem you're having rather than rewriting 
the entire app/framework/whatever?



I am certain that through proper
use of RDBMS indexes Squishdot would be scalable into the tens of
thousands of messages. 


I'm certain that proper use of ZCatalog indexes, which modern Squishdots 
already use, will scale just as well ;-)



For a small blog such as mine with several
thousand posts a year even Gadfly might be good enough.


Hahahaha... ok, you made me laugh. Seriously though, don't even go 
anywhere near gadflu...



To convert Squishdot to use an RDBMS I would have to modify the
Posting and other product scripts in Squishdot. 


...which will take about 100 times more effort than figuring out what's 
slowing down your sites ;-)



If I could look at the
Squishdot-related objects in the ZODB I would have a better
understanding of what Squishdot is doing when it posts and retrieves a
message. 


No, you really wouldn't.


If I may add my two cents on a related matter, what I like best about
Zope 2 is the VHM, undo, and content management.


You loose undo with your plan, as well as incurring an awful lot more 
work...



I work with RDBMSs
every day and understand them far better than I do the ZODB. 


Really? It doesn't show ;-) In all fairness, yes, if you understand RDB 
more, then your temptation will naturally be to run to it, but that is a 
mistake here...



more complex for DTML scripters like me. If one were to ask why Ruby
on Rails and Django get the attention that Zope does not, I feel that
a reliance on using the ZODB for textual transactional data is one
reason.


I rather strongly disagree with this statement, but this isn't the right 
time or place to debate it...



It sounds more difficult to set up an RDBMS app in Zope than in Django
or Ruby on Rails.


Indeed, whereas it is trivial to set up a ZODB-based app with indexing 
done using ZCatalog.



the hype. Since users prefer RDBMSs (that is what we know) new
developers are going to gravitate towards Django or RoR.


They're welcome to it them...


Wouldn't it be great if there could be a Zope app that would set up
RDBMS tables the way Django does, or generate code to access tables
the way RoR does? 


No. For me, it would suck slimy rocks through straws.
Now, a really good ORM that worked transparently with Zope's transaction 
management and allowed interaction in a similar style to zodb _would_ be 
of interest, but Zope 3 is your best bet for that...



But for lack of a better idea in the meantime it is
best for me to migrate my version of Squishdot to use an RDBMS.


I'm afraid I won't waste my time or yours helping you with that. If you 
want help with getting your sites faster, quickly, just answer the 
questions above...


Chris

--
Simplistix - Content Management, Zope & Python Consulting
   - http://www.simplistix.co.uk
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/list

RE: [Zope] How can I get a listing of all objects in the Zope 2.62 ZODB?

[Mark, Jonathan (Integic)]

Chris Withers, the developer of Squishdot, asks: 

>>OK. Can I ask why you're interested?

>>(the answer to that question will help give you more sensible answers to 
>> your first question.)

My experience using Squishdot to run goodbyejim.com for several years has been 
a positive one. I am grateful to Chris for writing it and am glad that he has 
moved on to other cutting edge Zope projects. 

However, there is one problem. As the number of posts gets past 2000 the 
application slows to a crawl.

You can observe the problem by comparing 
http://jonathanmark.com/Blog/1128110364/index_html, which is in a Squishdot 
instance with 3000 posts and is slow, and http://goodbyejim.com, which has 800 
posts. The front page of jonathanmark.com/Blog is cached, hence one must visit 
an interior page to observe the problem. A second Squishdot instance at 
http://jonathanmark.com/2005Blog/1147676261/index_html also seemed to be 
slowing down after 2000 posts.

All of these Squishdot instances run in the same Zope instance. 

So I decided that I would modify Squishdot so that it uses a relational 
database to store posts. I am certain that through proper use of RDBMS indexes 
Squishdot would be scalable into the tens of thousands of messages. For a small 
blog such as mine with several thousand posts a year even Gadfly might be good 
enough.

To convert Squishdot to use an RDBMS I would have to modify the Posting and 
other product scripts in Squishdot. If I could look at the Squishdot-related 
objects in the ZODB I would have a better understanding of what Squishdot is 
doing when it posts and retrieves a message. I could then duplicate that 
behavior with Gadfly.

If I may add my two cents on a related matter, what I like best about Zope 2 is 
the VHM, undo, and content management. I work with RDBMSs every day and 
understand them far better than I do the ZODB. Thus storing textual 
transactional data in the ZODB makes the system far more complex for DTML 
scripters like me. If one were to ask why Ruby on Rails and Django get the 
attention that Zope does not, I feel that a reliance on using the ZODB for 
textual transactional data is one reason. 

It sounds more difficult to set up an RDBMS app in Zope than in Django or Ruby 
on Rails. I have never done any of those things, but that is the hype. Since 
users prefer RDBMSs (that is what we know) new developers are going to 
gravitate towards Django or RoR.

Wouldn't it be great if there could be a Zope app that would set up RDBMS 
tables the way Django does, or generate code to access tables the way RoR does? 
But for lack of a better idea in the meantime it is best for me to migrate my 
version of Squishdot to use an RDBMS.
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] How can I get a listing of all objects in the Zope 2.62 ZODB?

[Chris Withers]

Mark, Jonathan (Integic) wrote:

Is there anything similar for the Zope 2.62 ZODB? I
have been using Squishdot at goodbyejim.com for years.
I still don't understand it. I would love to be able
to see everything that it dumps into the ZODB.


OK. Can I ask why you're interested?

(the answer to that question will help give you more sensible answers to 
your first question.)



to find out. But it ought to be possible to query the
ZODB directly and observe all of the objects that are
in it.


It is, but, as with your Oracle example, that's rarely very helpful...

Chris

--
Simplistix - Content Management, Zope & Python Consulting
   - http://www.simplistix.co.uk
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] How can I get a listing of all objects in the Zope 2.62 ZODB?

[Lennart Regebro]

On 9/12/06, Mark, Jonathan (Integic) <[EMAIL PROTECTED]> wrote:

In Oracle it is easy to get a listing of everything
that is in the Oracle database. One simply queries the
system view dba_objects.

Is there anything similar for the Zope 2.62 ZODB?


ZODB is an hierarchical object database. So, yes, there is and no there isn't.
You can't just make a query and get all the data, the data is stored as objects.

On the other hand, getting tha data is dead easy, although you don't
make a query. You just write python. For example, to get all the
subobjects for a particular object, you write

subobjects = the_object.objectValues()

Tada!

Do that recursively, and you pretty much get all the objects you would
be interested in. Then you need to understand each object type, but
that's another thing. ;-)
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] How can I get a listing of all objects in the Zope 2.62 ZODB?

[Dieter Maurer]

Mark, Jonathan (Integic) wrote at 2006-9-12 11:26 -0400:
>In Oracle it is easy to get a listing of everything
>that is in the Oracle database. One simply queries the
>system view dba_objects.
>
>Is there anything similar for the Zope 2.62 ZODB? I
>have been using Squishdot at goodbyejim.com for years.
>I still don't understand it. I would love to be able
>to see everything that it dumps into the ZODB.

In modern ZODB versions (I think ZODB 3.6 and above),
there is an iterator which allows you to iterate over
all objects in the ZODB.

It main use case is to perform migrations between major updates.
But, if you know how to "list" your objects intelligently (this is much more
difficult than with Oracle where all objects have a strict schema),
you definitely can do this as well (but I doubt, you will be happy).

>I understand that one can look at the Squishdot source
>to find out. But it ought to be possible to query the
>ZODB directly and observe all of the objects that are
>in it.

The ZODB level is *very* *very* low!
You do *NOT* want to work on this level when you have an alternative...



-- 
Dieter
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] How can I get a listing of all objects in the Zope 2.62 ZODB?

[Andreas Jung]

--On 12. September 2006 11:26:13 -0400 "Mark, Jonathan (Integic)" 
<[EMAIL PROTECTED]> wrote:



In Oracle it is easy to get a listing of everything
that is in the Oracle database. One simply queries the
system view dba_objects.

Is there anything similar for the Zope 2.62 ZODB? I
have been using Squishdot at goodbyejim.com for years.
I still don't understand it. I would love to be able
to see everything that it dumps into the ZODB.





-aj

pgpBwS4J3qsBa.pgp
Description: PGP signature
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )