[ZODB-Dev] Data.fs size grows non-stop

2009-12-07 Thread Jose Benito Gonzalez Lopez
Dear ZODB developers, Since some time ago (not sure since when) our database has passed from 15GB to 65GB so fast, and it keeps growing little by little (2 to 5 GB per day). It is clear that something is not correct in it. We would like to check which objects are taking most of the space or just

Re: [ZODB-Dev] Data.fs size grows non-stop

2009-12-07 Thread Laurence Rowe
2009/12/7 Jose Benito Gonzalez Lopez jose.benito.gonza...@cern.ch: Dear ZODB developers, Since some time ago (not sure since when) our database has passed from 15GB to 65GB so fast, and it keeps growing little by little (2 to 5 GB per day). It is clear that something is not correct in it.

Re: [ZODB-Dev] Data.fs size grows non-stop

2009-12-07 Thread Jim Fulton
On Mon, Dec 7, 2009 at 8:31 AM, Jose Benito Gonzalez Lopez jose.benito.gonza...@cern.ch wrote: Since some time ago (not sure since when) our database has passed from 15GB to 65GB so fast, and it keeps growing little by little (2 to 5 GB per day). It is clear that something is not correct in

Re: [ZODB-Dev] Data.fs size grows non-stop

2009-12-07 Thread Pedro Ferreira
Hello all, Since some time ago (not sure since when) our database has passed from 15GB to 65GB so fast, and it keeps growing little by little (2 to 5 GB per day). It is clear that something is not correct in it. I'd just like to add that there's some changes that can be related to this: -

Re: [ZODB-Dev] Data.fs size grows non-stop

2009-12-07 Thread Alan Runyan
I'd just like to add that there's some changes that can be related to this: - we had some classes inheriting from Persistent that now inherit from something else as well (but no extra arguments are being added, AFAIK); - we added some zope.interface definitions to some Persistent classes;

Re: [ZODB-Dev] Making ZODB / ZEO faster

2009-12-07 Thread Erik Dahl
Guys, Thanks for all the great feedback. Still processing it but here are somethings we will try. RelStorage - in our app context to see if there it helps / hurts. will report back results. Quick tests show some improvement. We will also look at tuning out current ZEO setup. Last time

[ZODB-Dev] big list (was Re: Making ZODB / ZEO faster)

2009-12-07 Thread Gary Poster
On Dec 7, 2009, at 11:08 AM, Erik Dahl wrote: ... Our slow loading object was a persistent with a regular list inside of the main pickle. Objects that the list pointed to were persistent which I believe means that hey will load separately. In general we have tried to make our

Re: [ZODB-Dev] Making ZODB / ZEO faster

2009-12-07 Thread Jim Fulton
On Mon, Dec 7, 2009 at 11:08 AM, Erik Dahl ed...@zenoss.com wrote: Guys, Thanks for all the great feedback.  Still processing it but here are somethings we will try.  We will also look at tuning out current ZEO setup.  Last time I looked there was only the invalidation queue.  I poked

Re: [ZODB-Dev] Data.fs size grows non-stop

2009-12-07 Thread Jim Fulton
On Mon, Dec 7, 2009 at 11:00 AM, Alan Runyan runy...@gmail.com wrote: ... A design pattern for RDBMS is to have 2 pools. READ pool and WRITE pool. Often the READ pool comes from some replica and WRITE is to the master. I'm unsure this pattern would work for ZODB.  I know Malthe was thinking

Re: [ZODB-Dev] Data.fs size grows non-stop

2009-12-07 Thread Christian Theune
On 12/07/2009 05:00 PM, Alan Runyan wrote: I'd just like to add that there's some changes that can be related to this: - we had some classes inheriting from Persistent that now inherit from something else as well (but no extra arguments are being added, AFAIK); - we added some zope.interface

Re: [ZODB-Dev] Data.fs size grows non-stop

2009-12-07 Thread Wichert Akkerman
On 2009-12-7 17:34, Jim Fulton wrote: On Mon, Dec 7, 2009 at 11:00 AM, Alan Runyanruny...@gmail.com wrote: ... A design pattern for RDBMS is to have 2 pools. READ pool and WRITE pool. Often the READ pool comes from some replica and WRITE is to the master. I'm unsure this pattern would work

Re: [ZODB-Dev] Making ZODB / ZEO faster

2009-12-07 Thread Erik Dahl
Biggest DBs we see are in the 10GB range. Most are significantly smaller (100s of MB) so I don't think size is an issue. We typically run big persistent caches so I don't think there is much help to be had there. It would only make reads faster anyway right? -EAD On Dec 7, 2009, at

Re: [ZODB-Dev] Data.fs size grows non-stop

2009-12-07 Thread Jose Benito Gonzalez Lopez
Hi, Actually, we have two different versions (current and beta) of the software, working against the same ZODB. We are just developing new features on top of the beta version. Fearing that it was due to some changes done by a programmer we have disabled the beta version but the growing of the DB

Re: [ZODB-Dev] Data.fs size grows non-stop

2009-12-07 Thread Jim Fulton
On Mon, Dec 7, 2009 at 3:08 PM, Jose Benito Gonzalez Lopez jose.benito.gonza...@cern.ch wrote: Hi, Actually, we have two different versions (current and beta) of the software, working against  the same ZODB. We are just developing new features on top of the beta version. Fearing that it was

Re: [ZODB-Dev] Making ZODB / ZEO faster

2009-12-07 Thread Jim Fulton
On Mon, Dec 7, 2009 at 1:46 PM, Erik Dahl ed...@zenoss.com wrote: Biggest DBs we see are in the 10GB range.  Most are significantly smaller (100s of MB) so I don't think size is an issue.  We typically run big persistent caches so I don't think there is much help to be had there.  It would

Re: [ZODB-Dev] Data.fs size grows non-stop

2009-12-07 Thread Shane Hathaway
Jose Benito Gonzalez Lopez wrote: Dear ZODB developers, Since some time ago (not sure since when) our database has passed from 15GB to 65GB so fast, and it keeps growing little by little (2 to 5 GB per day). It is clear that something is not correct in it. We would like to check which