Re: [ZODB-Dev] Data.fs size grows non-stop

[Roger]

Hi Pedro 

> Betreff: Re: AW: [ZODB-Dev] Data.fs size grows non-stop
> 
> 
> > If you need performance, take a look at the z3c.indexer 
> package which 
> > offers a much faster indexing concept. I implemented this package 
> > because my app coudn't handle the implicit default indexing 
> handling. 
> > The README.txt explains what is different and how to use it. It 
> > defently needs some work to implement a z3c.indexer based indexing 
> > concept but if you do it right, it can handle a lot more indexing 
> > tasks then the generic zope or zc.catalog concept.
> >
> >   
> Thanks a lot. But is it easy to use this from within a 
> non-Zope application? In the examples I could see references 
> to several zope.site classes, such as LocalSiteManager... 
> since this is a standalone app, I wonder if it won't force us 
> to import some other zope packages...

The site is only used as a sample setup and for provide
a local persistent IIntIds utility.

Every zope catalog implementation needs an object id
normaly an id given from IIntIds.

But if your app provides another unique integer id
for each object, you can use the z3c.indexer package 
with small modifications.

But of corse it will bring in some dependencies,
but not more then the zope or zc.catalog will.

Let me know if you could use the package with less
dependencies then we have now and we can try to cut
the dependencies down or probably defined them as test
dependencies

Regards
Roger Ineichen 

> Pedro
> >> Pedro
> >> ___
> >> For more information about ZODB, see the ZODB Wiki:
> >> http://www.zope.org/Wikis/ZODB/
> >>
> >> ZODB-Dev mailing list  -  ZODB-Dev@zope.org 
> >> https://mail.zope.org/mailman/listinfo/zodb-dev
> >>
> >> 
> >
> >   
> 
> 

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Jim Fulton]

On Thu, Dec 10, 2009 at 4:06 AM, Pedro Ferreira
 wrote:
> Jim Fulton wrote:
>>
>> On Wed, Dec 9, 2009 at 10:54 AM, Pedro Ferreira
>>  wrote:
>> ...
>>
>>>
>>> Well, at least we won't have to rewrite the whole bucket... but still, it
>>> would be much nicer to fragment the list
>>> in smaller chunks. We could use an OOBTree instead... but something less
>>> complex would suffice... any suggestions?
>>>
>>
>> It's hard for people to suggest something without knowing your usage
>> pattern.
>> Do you want something that is effectively a set?
>
> The situation is simple: we have events (objects) that have to be indexed by
> date. We currently use an OOBTree with the days as keys and lists of objects
> as values. The objective is to avoid rewriting these lists each time
> something changes, since we can have tens of thousands of events in the same
> day. Some kind of btree-based list or set would work, I guess.
>>
>> Is integer indexing important?
>>
>
> Well, it's an index of dates, so, I guess we could use integers as keys
> instead of strings, as we are now.

I wasn't referring to indexing at that level. You were using lists
rather than sets
and I was wondering whether there was a reason for that.

It sounds like a data structure with minimal container semantics
would be fine to store the events for a day.

If you can arrange that the events have a stable ordering such that
no 2 events are equal, then you can use OOTreeSets rather than
lists.

Jim

-- 
Jim Fulton
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Pedro Ferreira]

Hello,
> Are the values in your BTree implemented as persistent lists or some
> other subclass of Persistent?  That makes a big difference in the size
> of each commit to the BTree.
>   
Yes, that is exactly the problem. We are using simple Python lists, and 
that implies rewriting the whole bucket.
Still, rewriting only the list won't be the best solution, since they 
can be pretty big sometimes. That's why I was looking for something like 
a BTree-based list (that actually seems to exist).

Regards,

Pedro

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Pedro Ferreira]

> If you need performance, take a look at the z3c.indexer package
> which offers a much faster indexing concept. I implemented
> this package because my app coudn't handle the implicit default
> indexing handling. The README.txt explains what is different and how 
> to use it. It defently needs some work to implement a z3c.indexer
> based indexing concept but if you do it right, it can handle a
> lot more indexing tasks then the generic zope or zc.catalog
> concept.
>
>   
Thanks a lot. But is it easy to use this from within a non-Zope 
application? In the examples I could see references to several zope.site 
classes, such as LocalSiteManager... since this is a standalone app, I 
wonder if it won't force us to import some other zope packages...

Pedro
>> Pedro
>> ___
>> For more information about ZODB, see the ZODB Wiki:
>> http://www.zope.org/Wikis/ZODB/
>>
>> ZODB-Dev mailing list  -  ZODB-Dev@zope.org 
>> https://mail.zope.org/mailman/listinfo/zodb-dev
>>
>> 
>
>   

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Roger]

Hi Pedro 

> Betreff: Re: [ZODB-Dev] Data.fs size grows non-stop
> 
> 
> > Why not store the events on some BTree or similar structure with a 
> > simple unique integer id, and use a separate BTree to map dates to 
> > these events? You can even use zc.catalog or similar tools 
> to create 
> > that mapping - you would get a lot of query options for 
> free with those as well.
> >
> >   
> Yeah, that would require a lot of changes, but it would be 
> definitely great. What does zc.catalog have that zope.catalog 
> doesn't, by the way?

If you need performance, take a look at the z3c.indexer package
which offers a much faster indexing concept. I implemented
this package because my app coudn't handle the implicit default
indexing handling. The README.txt explains what is different and how 
to use it. It defently needs some work to implement a z3c.indexer
based indexing concept but if you do it right, it can handle a
lot more indexing tasks then the generic zope or zc.catalog
concept.

Here are some performance data:

z3c.indexer
indexer based indexing time:  2.47 s

zope.catalog
catalog based indexing time:  14.41 s

btw,
You can run the performance test by yourself
located in the z3c.indexer package.

Regards
Roger Ineichen

> Pedro
> ___
> For more information about ZODB, see the ZODB Wiki:
> http://www.zope.org/Wikis/ZODB/
> 
> ZODB-Dev mailing list  -  ZODB-Dev@zope.org 
> https://mail.zope.org/mailman/listinfo/zodb-dev
> 

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Wichert Akkerman]

On 2009-12-10 10:48, Pedro Ferreira wrote:
>
>> Why not store the events on some BTree or similar structure with a
>> simple unique integer id, and use a separate BTree to map dates to
>> these events? You can even use zc.catalog or similar tools to create
>> that mapping - you would get a lot of query options for free with
>> those as well.
>>
> Yeah, that would require a lot of changes, but it would be definitely
> great. What does zc.catalog have that zope.catalog doesn't, by the way?

I'm not sure to be honest. There is a whole plethora of options: 
zope.catalog, zc.catalog, repoze.catalog, hurry.query, and possibly 
others as well.

Wichert.

-- 
Wichert AkkermanIt is simple to make things.
http://www.wiggy.net/  It is hard to make things simple.
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Pedro Ferreira]

> Why not store the events on some BTree or similar structure with a 
> simple unique integer id, and use a separate BTree to map dates to these 
> events? You can even use zc.catalog or similar tools to create that 
> mapping - you would get a lot of query options for free with those as well.
>
>   
Yeah, that would require a lot of changes, but it would be definitely 
great. What does zc.catalog have that zope.catalog doesn't, by the way?

Pedro
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Pedro Ferreira]

Thanks!
But I still get an error:

pferreir 
/usr/lib/python2.6/site-packages/ZODB3-3.9.0-py2.6-linux-i686.egg$ 
zodbbrowser ~/indico/db/last.fs
Traceback (most recent call last):
  File "/usr/bin/zodbbrowser", line 8, in 
load_entry_point('zodbbrowser==0.6.1', 'console_scripts', 
'zodbbrowser')()
  File 
"/usr/lib/python2.6/site-packages/zodbbrowser-0.6.1-py2.6.egg/zodbbrowser/standalone.py",
 
line 179, in main
configure(options)
  File 
"/usr/lib/python2.6/site-packages/zodbbrowser-0.6.1-py2.6.egg/zodbbrowser/standalone.py",
 
line 87, in configure
context = xmlconfig.string(options.site_definition, context=context)
  File 
"/usr/lib/python2.6/site-packages/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py",
 
line 664, in string
processxmlfile(f, context)
  File 
"/usr/lib/python2.6/site-packages/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py",
 
line 378, in processxmlfile
parser.parse(src)
  File "/usr/lib/python2.6/xml/sax/expatreader.py", line 107, in parse
xmlreader.IncrementalParser.parse(self, source)
  File "/usr/lib/python2.6/xml/sax/xmlreader.py", line 123, in parse
self.feed(buffer)
  File "/usr/lib/python2.6/xml/sax/expatreader.py", line 207, in feed
self._parser.Parse(data, isFinal)
  File "/usr/lib/python2.6/xml/sax/expatreader.py", line 349, in 
end_element_ns
self._cont_handler.endElementNS(pair, None)
  File 
"/usr/lib/python2.6/site-packages/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py",
 
line 357, in endElementNS
self.context.end()
  File 
"/usr/lib/python2.6/site-packages/zope.configuration-3.6.0-py2.6.egg/zope/configuration/config.py",
 
line 537, in end
self.stack.pop().finish()
  File 
"/usr/lib/python2.6/site-packages/zope.configuration-3.6.0-py2.6.egg/zope/configuration/config.py",
 
line 684, in finish
args = toargs(context, *self.argdata)
  File 
"/usr/lib/python2.6/site-packages/zope.configuration-3.6.0-py2.6.egg/zope/configuration/config.py",
 
line 1376, in toargs
args[str(name)] = field.fromUnicode(s)
  File 
"/usr/lib/python2.6/site-packages/zope.configuration-3.6.0-py2.6.egg/zope/configuration/fields.py",
 
line 141, in fromUnicode
raise schema.ValidationError(v)
zope.configuration.xmlconfig.ZopeXMLConfigurationError: File "", 
line 4.10-4.72
ConfigurationError: ('Invalid value for', 'package', 'ImportError: 
Module zope.app has no global securitypolicy')

weird...

Marius Gedminas wrote:
> On Wed, Dec 09, 2009 at 06:09:36PM +0200, Marius Gedminas wrote:
>   
>> On Wed, Dec 09, 2009 at 02:42:58PM +0100, Pedro Ferreira wrote:
>> 
>>> Hello,
>>>   
 Just zodbbrowser with no prefix:

   http://pypi.python.org/pypi/zodbbrowser
   https://launchpad.net/zodbbrowser

 It's a web-app: it can connect to your ZEO server so you can inspect the
 DB while it's being used.
   
 
>>> We tried this, but we currently get an error related with the general 
>>> security policy for zope.app. Maybe we need to install Zope?
>>>   
>> No, it should be self-contained.
>>
>> I'm usually testing with the Zope 3.4 KGS.
>> 
>
> I've just released zodbbrowser 0.6.1 which works with ZODB 3.9.x and
> latest versions of other packages too.
>
> (Supporting both ZODB 3.8 and 3.9 is kinda tricky, but with some very
> ugly hacks I managed.)
>
> Marius Gedminas
>   

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Wichert Akkerman]

On 2009-12-10 10:06, Pedro Ferreira wrote:
> The situation is simple: we have events (objects) that have to be
> indexed by date. We currently use an OOBTree with the days as keys and
> lists of objects as values. The objective is to avoid rewriting these
> lists each time something changes, since we can have tens of thousands
> of events in the same day. Some kind of btree-based list or set would
> work, I guess.

Why not store the events on some BTree or similar structure with a 
simple unique integer id, and use a separate BTree to map dates to these 
events? You can even use zc.catalog or similar tools to create that 
mapping - you would get a lot of query options for free with those as well.

Wichert.

-- 
Wichert AkkermanIt is simple to make things.
http://www.wiggy.net/  It is hard to make things simple.
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Pedro Ferreira]

Jim Fulton wrote:
> On Wed, Dec 9, 2009 at 10:54 AM, Pedro Ferreira
>  wrote:
> ...
>   
>> Well, at least we won't have to rewrite the whole bucket... but still, it
>> would be much nicer to fragment the list
>> in smaller chunks. We could use an OOBTree instead... but something less
>> complex would suffice... any suggestions?
>> 
>
> It's hard for people to suggest something without knowing your usage pattern.
> Do you want something that is effectively a set?
The situation is simple: we have events (objects) that have to be 
indexed by date. We currently use an OOBTree with the days as keys and 
lists of objects as values. The objective is to avoid rewriting these 
lists each time something changes, since we can have tens of thousands 
of events in the same day. Some kind of btree-based list or set would 
work, I guess.
> Is integer indexing important?
>   
Well, it's an index of dates, so, I guess we could use integers as keys 
instead of strings, as we are now.

Thanks,

Pedro
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Marius Gedminas]

On Wed, Dec 09, 2009 at 06:09:36PM +0200, Marius Gedminas wrote:
> On Wed, Dec 09, 2009 at 02:42:58PM +0100, Pedro Ferreira wrote:
> > Hello,
> > > Just zodbbrowser with no prefix:
> > >
> > >   http://pypi.python.org/pypi/zodbbrowser
> > >   https://launchpad.net/zodbbrowser
> > >
> > > It's a web-app: it can connect to your ZEO server so you can inspect the
> > > DB while it's being used.
> > >   
> > We tried this, but we currently get an error related with the general 
> > security policy for zope.app. Maybe we need to install Zope?
> 
> No, it should be self-contained.
> 
> I'm usually testing with the Zope 3.4 KGS.

I've just released zodbbrowser 0.6.1 which works with ZODB 3.9.x and
latest versions of other packages too.

(Supporting both ZODB 3.8 and 3.9 is kinda tricky, but with some very
ugly hacks I managed.)

Marius Gedminas
-- 
Any sufficiently advanced Operating System is indistinguishable from Linux.
-- Jim Dennis


signature.asc
Description: Digital signature
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Jim Fulton]

On Wed, Dec 9, 2009 at 10:54 AM, Pedro Ferreira
 wrote:
...
> Well, at least we won't have to rewrite the whole bucket... but still, it
> would be much nicer to fragment the list
> in smaller chunks. We could use an OOBTree instead... but something less
> complex would suffice... any suggestions?

It's hard for people to suggest something without knowing your usage pattern.
Do you want something that is effectively a set? Is integer indexing important?
What are the items in the list?

> We are really considering replacing all our indexes with something more
> efficient and ZODB-friendly... do you thing zope.catalog would be a good
> choice?

A catalog is simply an organization of indexes, typically implemented
with BTrees.

Jim

-- 
Jim Fulton
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Christian Theune]

On 12/09/2009 04:58 PM, Pedro Ferreira wrote:
>
>> OOBTrees are complex?
>>
> No, I meant something list-like, rather than key-value.

zc.blist implements a BTree-based list.

-- 
Christian Theune · c...@gocept.com
gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany
http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1
Zope and Plone consulting and development

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Marius Gedminas]

On Wed, Dec 09, 2009 at 02:42:58PM +0100, Pedro Ferreira wrote:
> Hello,
> > Just zodbbrowser with no prefix:
> >
> >   http://pypi.python.org/pypi/zodbbrowser
> >   https://launchpad.net/zodbbrowser
> >
> > It's a web-app: it can connect to your ZEO server so you can inspect the
> > DB while it's being used.
> >   
> We tried this, but we currently get an error related with the general 
> security policy for zope.app. Maybe we need to install Zope?

No, it should be self-contained.

I'm usually testing with the Zope 3.4 KGS.  Easiest way to do that is to
check out the sources and use buildout:

  bzr get lp:zodbbrowser
  cd zodbbrowser
  python bootstrap.py
  bin/buildout
  bin/zodbbrowser --zeo localhost:1234 

I'll test it with the latest zope.* packages from PyPI and see if I can
reproduce the error.  Feel free to report a bug (and attach the
traceback you get) here: https://bugs.launchpad.net/zodbbrowser

> This would be a very handy tool.
> > I'd suggest dumping the last few transactions with one of the ZODB
> > scripts (fsdump.py perhaps) and seeing what objects get modified.
> >   
> That's what we've being doing, and we got some clues. We've modified 
> Jim's script in order to find out which OIDs are being rewritten, and 
> how much space they are taking, and this is a fragment of it:
> 
> OID class_name total_size percent_size n_pickles min_size avg_size max_size
> '\x00\x00\x00\x00%T\x89{' BTrees.OOBTree.OOBucket 17402831841 30% 8683 
> 1977885 2004241 2026518
> '\x00\x00\x00\x00%T\x89|' BTrees.OOBTree.OOBucket 14204430890 24% 8683 
> 1616904 1635889 1651956
> '\x00\x00\x00\x00\x04dUH' MaKaC.common.indexes.StatusIndex 11955954522 
> 20% 28513 418230 419315 420294
> '\x00\x00\x00\x00%\xa0%\x7f' BTrees.OOBTree.OOBucket 3532998238 6% 11238 
> 307112 314379 320647
> '\x00\x00\x00\x00%\xa0%\x80' BTrees.OOBTree.OOBucket 2193843302 3% 11238 
> 190816 195216 199007
> '\x00\x00\x00\x00\x04\x8e\xb6\x04' BTrees.OOBTree.OOBucket 1728216003 3% 
> 1953 880615 884903 887285
> [...]
> 
> As you can see, we have an OOBucket occupying more than 2MB (!) per 
> write.

Ouch.

> That's almost 17GB only considering the last 1M transactions of 
> the DB (we get ~3M transactions per week). We believe this bucket 
> belongs to some OOBTree-based index that we are using, whose values are 
> Python lists (maybe that was a bad choice to start with?). In any case, 
> how do OOBuckets work? Is it a simple key space segmentation strategy, 
> or are the values taken into account as well?
> Our theory is that an OOBTree simply divides the N keys in K buckets, 
> and doesn't care about the contents. So, since we are adding very large 
> lists as values, the tree remains unbalanced, and since new contents 
> will be added to this last bucket, each rewrite will imply the addition 
> of ~2MB to the file storage.
> Will the replacement of these lists with a persistent structure such as 
> a PersistentList solve the issue?

It should definitely help.  Now, if modify one of the lists in a bucket,
it needs to append the whole 2 megs of data to the Data.fs.  If you'd
used a PersistentList, it would only need to append as much data as your
persistent list contains.  Assuming there are ~30 lists in a bucket,
you'd get 30x space savings.

Are those lists long?  Are they modified often?  What do they contain?
I'm sure you can get better space efficiency by redesigning the data
structure.

Marius Gedminas
-- 
Dijkstra probably hates me
-- Linus Torvalds, in kernel/sched.c


signature.asc
Description: Digital signature
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Pedro Ferreira]

> OOBTrees are complex?
>   
No, I meant something list-like, rather than key-value.
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Andreas Jung]

Am 09.12.09 16:54, schrieb Pedro Ferreira:
> We could use an OOBTree instead... but something less
> complex would suffice... any suggestions?

OOBTrees are complex?

-aj
<>___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Pedro Ferreira]

Hello,
>
> In the case of BTrees, yes.  I assume your OOBuckets are used within
> OOBTrees. (?)
>
>   
Yes. I was not sure whether OOBTrees behaved exactly like a BTree.
>> So, since we are adding very large
>> lists as values, the tree remains unbalanced,
>> 
>
> No, they trees tend to stay fairly well balenced wrt keys.
>   
Well, I mean, they are balanced, but the contents are not distributed in 
an homogeneous way... which is the expected behavior since they don't 
care about the values.
>   
>> and since new contents
>> will be added to this last bucket,
>> 
>
> Why would the contents only be added to one bucket?
>   
Because we're using this OOBTree as a date index and each time we add 
new contents they're added to the entry that concerns the current day. 
Currently, the bucket that contains the current day also contains some 
days that themselves contain huge lists of contents. This will probably 
change as the buckets split, but we want to correct this once and for all...
>
>> Will the replacement of these lists with a persistent structure such as
>> a PersistentList solve the issue?
>> 
>
> It might help, but if the lists are very large, you'll still have a problem
> because a persistent list is still stored in one database record.
>   
Well, at least we won't have to rewrite the whole bucket... but still, 
it would be much nicer to fragment the list
in smaller chunks. We could use an OOBTree instead... but something less 
complex would suffice... any suggestions?

We are really considering replacing all our indexes with something more 
efficient and ZODB-friendly... do you thing zope.catalog would be a good 
choice?

Thanks once again,

Pedro
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Jim Fulton]

On Wed, Dec 9, 2009 at 8:42 AM, Pedro Ferreira
 wrote:
...
> We've modified
> Jim's script

Cool. Storage iterators are so simple and allow a wide variety of analyses.

> in order to find out which OIDs are being rewritten, and
> how much space they are taking, and this is a fragment of it:
>
> OID class_name total_size percent_size n_pickles min_size avg_size max_size
> '\x00\x00\x00\x00%T\x89{' BTrees.OOBTree.OOBucket 17402831841 30% 8683
> 1977885 2004241 2026518
> '\x00\x00\x00\x00%T\x89|' BTrees.OOBTree.OOBucket 14204430890 24% 8683
> 1616904 1635889 1651956
> '\x00\x00\x00\x00\x04dUH' MaKaC.common.indexes.StatusIndex 11955954522
> 20% 28513 418230 419315 420294
> '\x00\x00\x00\x00%\xa0%\x7f' BTrees.OOBTree.OOBucket 3532998238 6% 11238
> 307112 314379 320647
> '\x00\x00\x00\x00%\xa0%\x80' BTrees.OOBTree.OOBucket 2193843302 3% 11238
> 190816 195216 199007
> '\x00\x00\x00\x00\x04\x8e\xb6\x04' BTrees.OOBTree.OOBucket 1728216003 3%
> 1953 880615 884903 887285
> [...]
>
> As you can see, we have an OOBucket occupying more than 2MB (!) per
> write. That's almost 17GB only considering the last 1M transactions of
> the DB (we get ~3M transactions per week). We believe this bucket
> belongs to some OOBTree-based index that we are using, whose values are
> Python lists (maybe that was a bad choice to start with?). In any case,
> how do OOBuckets work?

Buckets themselves are essentially just sorted lists of key-value pairs.

> Is it a simple key space segmentation strategy,

In the case of BTrees, yes.  I assume your OOBuckets are used within
OOBTrees. (?)

> or are the values taken into account as well?

No.

> Our theory is that an OOBTree simply divides the N keys in K buckets,
> and doesn't care about the contents.

Right.

> So, since we are adding very large
> lists as values, the tree remains unbalanced,

No, they trees tend to stay fairly well balenced wrt keys.

> and since new contents
> will be added to this last bucket,

Why would the contents only be added to one bucket?


> each rewrite will imply the addition
> of ~2MB to the file storage.

That's definately a problem.

> Will the replacement of these lists with a persistent structure such as
> a PersistentList solve the issue?

It might help, but if the lists are very large, you'll still have a problem
because a persistent list is still stored in one database record.

Jim

-- 
Jim Fulton
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Laurence Rowe]

2009/12/9 Pedro Ferreira :
> Hello,
>> Just zodbbrowser with no prefix:
>>
>>   http://pypi.python.org/pypi/zodbbrowser
>>   https://launchpad.net/zodbbrowser
>>
>> It's a web-app: it can connect to your ZEO server so you can inspect the
>> DB while it's being used.
>>
> We tried this, but we currently get an error related with the general
> security policy for zope.app. Maybe we need to install Zope?
> This would be a very handy tool.
>> I'd suggest dumping the last few transactions with one of the ZODB
>> scripts (fsdump.py perhaps) and seeing what objects get modified.
>>
> That's what we've being doing, and we got some clues. We've modified
> Jim's script in order to find out which OIDs are being rewritten, and
> how much space they are taking, and this is a fragment of it:
>
> OID class_name total_size percent_size n_pickles min_size avg_size max_size
> '\x00\x00\x00\x00%T\x89{' BTrees.OOBTree.OOBucket 17402831841 30% 8683
> 1977885 2004241 2026518
> '\x00\x00\x00\x00%T\x89|' BTrees.OOBTree.OOBucket 14204430890 24% 8683
> 1616904 1635889 1651956
> '\x00\x00\x00\x00\x04dUH' MaKaC.common.indexes.StatusIndex 11955954522
> 20% 28513 418230 419315 420294
> '\x00\x00\x00\x00%\xa0%\x7f' BTrees.OOBTree.OOBucket 3532998238 6% 11238
> 307112 314379 320647
> '\x00\x00\x00\x00%\xa0%\x80' BTrees.OOBTree.OOBucket 2193843302 3% 11238
> 190816 195216 199007
> '\x00\x00\x00\x00\x04\x8e\xb6\x04' BTrees.OOBTree.OOBucket 1728216003 3%
> 1953 880615 884903 887285
> [...]
>
> As you can see, we have an OOBucket occupying more than 2MB (!) per
> write. That's almost 17GB only considering the last 1M transactions of
> the DB (we get ~3M transactions per week). We believe this bucket
> belongs to some OOBTree-based index that we are using, whose values are
> Python lists (maybe that was a bad choice to start with?). In any case,
> how do OOBuckets work? Is it a simple key space segmentation strategy,
> or are the values taken into account as well?
> Our theory is that an OOBTree simply divides the N keys in K buckets,
> and doesn't care about the contents. So, since we are adding very large
> lists as values, the tree remains unbalanced, and since new contents
> will be added to this last bucket, each rewrite will imply the addition
> of ~2MB to the file storage.

BTree buckets have no concept of the size of their contents, they
split when their number of keys reaches a threshold (30 for OOBTrees).

> Will the replacement of these lists with a persistent structure such as
> a PersistentList solve the issue?

The list would then be stored as a separate persistent object, so
changes to the bucket would not rewrite the entire list object. The
downside of this is that your application may become slower as reading
the contents of the index will incur additional object loads. Zope2's
ZCatalog stores index data as tuples in BTrees, but only a small
amount of metadata is stored (so the buckets are maybe 30-60KB). It
sounds like you are storing a large amount of metadata in the index,
or perhaps inadvertently indexing something. I've seen similar
problems caused by binary data ending up in a text index (where a
'word' ended up being several megabytes). Load the object to check the
problem is large values, rather than large keys.

Laurence
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Pedro Ferreira]

Hello,
> Just zodbbrowser with no prefix:
>
>   http://pypi.python.org/pypi/zodbbrowser
>   https://launchpad.net/zodbbrowser
>
> It's a web-app: it can connect to your ZEO server so you can inspect the
> DB while it's being used.
>   
We tried this, but we currently get an error related with the general 
security policy for zope.app. Maybe we need to install Zope?
This would be a very handy tool.
> I'd suggest dumping the last few transactions with one of the ZODB
> scripts (fsdump.py perhaps) and seeing what objects get modified.
>   
That's what we've being doing, and we got some clues. We've modified 
Jim's script in order to find out which OIDs are being rewritten, and 
how much space they are taking, and this is a fragment of it:

OID class_name total_size percent_size n_pickles min_size avg_size max_size
'\x00\x00\x00\x00%T\x89{' BTrees.OOBTree.OOBucket 17402831841 30% 8683 
1977885 2004241 2026518
'\x00\x00\x00\x00%T\x89|' BTrees.OOBTree.OOBucket 14204430890 24% 8683 
1616904 1635889 1651956
'\x00\x00\x00\x00\x04dUH' MaKaC.common.indexes.StatusIndex 11955954522 
20% 28513 418230 419315 420294
'\x00\x00\x00\x00%\xa0%\x7f' BTrees.OOBTree.OOBucket 3532998238 6% 11238 
307112 314379 320647
'\x00\x00\x00\x00%\xa0%\x80' BTrees.OOBTree.OOBucket 2193843302 3% 11238 
190816 195216 199007
'\x00\x00\x00\x00\x04\x8e\xb6\x04' BTrees.OOBTree.OOBucket 1728216003 3% 
1953 880615 884903 887285
[...]

As you can see, we have an OOBucket occupying more than 2MB (!) per 
write. That's almost 17GB only considering the last 1M transactions of 
the DB (we get ~3M transactions per week). We believe this bucket 
belongs to some OOBTree-based index that we are using, whose values are 
Python lists (maybe that was a bad choice to start with?). In any case, 
how do OOBuckets work? Is it a simple key space segmentation strategy, 
or are the values taken into account as well?
Our theory is that an OOBTree simply divides the N keys in K buckets, 
and doesn't care about the contents. So, since we are adding very large 
lists as values, the tree remains unbalanced, and since new contents 
will be added to this last bucket, each rewrite will imply the addition 
of ~2MB to the file storage.
Will the replacement of these lists with a persistent structure such as 
a PersistentList solve the issue?

Thanks for all the help everyone,

Regards,

Pedro
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Marius Gedminas]

On Mon, Dec 07, 2009 at 10:58:20PM +0100, Roger wrote:
> what I whould todo as next,
> 
> try z3c.zodbbrowser or the other ZODB introspector 
> tool from mgedmin, don't remember the name of the tool.

Just zodbbrowser with no prefix:

  http://pypi.python.org/pypi/zodbbrowser
  https://launchpad.net/zodbbrowser

It's a web-app: it can connect to your ZEO server so you can inspect the
DB while it's being used.

I'd suggest dumping the last few transactions with one of the ZODB
scripts (fsdump.py perhaps) and seeing what objects get modified.
If you know an OID of an object, you can look at it with ZODBBrowser by
entering it in the URL, e.g. @@zodbbrowser?oid=0x1234.  Seeing the
object might give some clue where it comes from.  Or, if you see a long
history list for an object, you'll see what URLs were accessed
during the requests that made modifications here.

Sadly, currently ZODBBrowser doesn't have the ability to look for
backreferences (planned in my mental TODO list---but it'll be a very
expensive operation, especially on a large DB, since ZODB itself doesn't
keep track), and it doesn't let you view transactions (also on my mental
TODO list, also a bit expensive).

> the z3c.zoodbbrowser can open a ZODB with a pythin GUI.
> There you can iterate over each objects in the ZODB.
> You will probably see more then you see in the UI,
> especialy if some objects get created which you 
> don't expect.

Marius Gedminas
-- 
There's a special hell 4 people who replace words with numbers.
-- Ben "Yahtzee" Croshaw


signature.asc
Description: Digital signature
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Shane Hathaway]

Jose Benito Gonzalez Lopez wrote:
> Dear ZODB developers,
> 
> Since some time ago (not sure since when) our database
> has passed from 15GB to 65GB so fast, and it keeps growing
> little by little (2 to 5 GB per day). It is clear that something is not
> correct in it.
> 
> We would like to check which objects are taking most of the space
> or just try to find out what is going on,...
> 
> Any help or suggestions would be much appreciated.

FWIW, RelStorage provides a history-free option which helps control 
database growth.  However, you should still find out why you're writing 
too much data.

Shane

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Jose Benito Gonzalez Lopez]

> Have you tried packing the database?
>   
Pack has just finished: 17GB.
Still a lot, some months ago was 6GB
> I recommend buying a bigger disk -- seriously, before you run out of space. :)
>   
yes, but by the time the bigger disk arrives...we will be done :)

BTW Jim, your class_stats has just finished and this is the top of the list:

BTrees.OOBTree.OOBucket 42727142198 68% 346317 51 123375 2026518
MaKaC.common.indexes.StatusIndex 11955954522 19% 28513 418230 419315 420294
MaKaC.conference.Contribution 660280324 1% 494295 954 1335 8873
MaKaC.user.Avatar 614486693 0% 67819 306 9060 589941
MaKaC.webinterface.displayMgr.SystemLink 535415312 0% 2504186 157 213 365
MaKaC.accessControl.AccessController 433480829 0% 1838673 159 235 1694


OOBucket is the greedy one. It seemsnormal?

Thanks everyone for the tools.  Tomorrow we will test them against our DB.

Good night from Switzerland (0h15)

Regards,
Jose


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Kapil Thangavelu]

i would think it would be more useful immediately to account for the rate of
growth rather than trying to account for all objects in the zodb. i've
attached a quick script you can use to get some introspection on the last
few transactions of a large data.fs file. it operates in read only mode,
just pass the Data.fs file as an argument.  For each of the last 20
(default) transactions it will report the objects (by class name) in the
transaction and their sizes aggregated by class. hopefully based on the
classes being written you can identify where the large writes are
happening.

cheers,

kapil



On Mon, Dec 7, 2009 at 3:08 PM, Jose Benito Gonzalez Lopez <
jose.benito.gonza...@cern.ch> wrote:

> Hi,
>
> Actually, we have two different versions (current and beta) of the
> software,
> working against  the same ZODB. We are just developing new features
> on top of the beta version.
> Fearing that it was due to some changes done by a programmer
> we have disabled the "beta version" but the growing of the DB is still
> there...and since our last email we've got 5GB more (71GB now)  We
> are a bit worried
> since the disk space is 102GB and  the application is widely used here
> at CERN and
> for outsiders too.
>
> So, up-to-now we are afraid that it is not a problem with any
> modifications in the codeand
> no clue about it.
> We are trying to pack to see the difference with the data.fs file.
>
> Jim, I am still running your script "class_stats.py".and it seems it
> is going to take a while yet
>
> Thank you so much for your help. More ideas appreciated!
>
> Best regards,
> Jose
>
>
>
>
> Alan Runyan wrote:
> >> I'd just like to add that there's some changes that can be related to
> this:
> >>
> >> - we had some classes inheriting from Persistent that now inherit from
> >> something else as well (but no extra arguments are being added, AFAIK);
> >> - we added some zope.interface definitions to some Persistent classes;
> >>
> >> maybe this causes some kind of behavior that we were not aware of?
> >>
> >
> > I doubt it.  Thousnads of people are doing this and do not report the
> same
> > behavior.
> >
> > What is more likely is that a programmer is changing a persistent object
> > very often.  One of the downsides of the ZODB is that it is so
> transparent
> > it is possible to unwillingly make database changes.
> >
> > A design pattern for RDBMS is to have 2 pools. READ pool and WRITE pool.
> > Often the READ pool comes from some replica and WRITE is to the master.
> > I'm unsure this pattern would work for ZODB.  I know Malthe was thinking
> > about this but unsure if he had anything concrete.
> >
> > cheers
> > alan
> >
>
> ___
> For more information about ZODB, see the ZODB Wiki:
> http://www.zope.org/Wikis/ZODB/
>
> ZODB-Dev mailing list  -  ZODB-Dev@zope.org
> https://mail.zope.org/mailman/listinfo/zodb-dev
>


fstail2.py
Description: Binary data
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Roger]

Hi Jose

> Betreff: Re: [ZODB-Dev] Data.fs size grows non-stop
> 
> Hi,
> 
> Actually, we have two different versions (current and beta) 
> of the software, working against  the same ZODB. We are just 
> developing new features on top of the beta version.
> Fearing that it was due to some changes done by a programmer 
> we have disabled the "beta version" but the growing of the DB 
> is still there...and since our last email we've got 5GB more 
> (71GB now)  We are a bit worried since the disk space is 
> 102GB and  the application is widely used here at CERN and 
> for outsiders too.

Can you give us some traffic numbers? How many page hints
does the app server have?

Are you using a single zope server or ZEO setup?

Are you sure you don't have write access on each access.
E.g. each session will force a write access because of
some changes in the code?

Or soemthing different, is there something going on 
in the organisation which uses the applications. e.g.
the download large documents and update and upload them again?
Which means the applicaton usage has really changed.

Did you compare the Webserver Logs?

> So, up-to-now we are afraid that it is not a problem with any 
> modifications in the codeand no clue about it.
> We are trying to pack to see the difference with the data.fs file.
> 
> Jim, I am still running your script "class_stats.py".and 
> it seems it is going to take a while yet
> 
> Thank you so much for your help. More ideas appreciated!

what I whould todo as next,

try z3c.zodbbrowser or the other ZODB introspector 
tool from mgedmin, don't remember the name of the tool.

the z3c.zoodbbrowser can open a ZODB with a pythin GUI.
There you can iterate over each objects in the ZODB.
You will probably see more then you see in the UI,
especialy if some objects get created which you 
don't expect.

Regards
Roger Ineichen

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Jim Fulton]

On Mon, Dec 7, 2009 at 3:08 PM, Jose Benito Gonzalez Lopez
 wrote:
> Hi,
>
> Actually, we have two different versions (current and beta) of the software,
> working against  the same ZODB. We are just developing new features
> on top of the beta version.
> Fearing that it was due to some changes done by a programmer
> we have disabled the "beta version" but the growing of the DB is still
> there...and since our last email we've got 5GB more (71GB now)  We
> are a bit worried
> since the disk space is 102GB and  the application is widely used here
> at CERN and
> for outsiders too.
>
> So, up-to-now we are afraid that it is not a problem with any
> modifications in the codeand
> no clue about it.
> We are trying to pack to see the difference with the data.fs file.
>
> Jim, I am still running your script "class_stats.py".and it seems it
> is going to take a while yet
>
> Thank you so much for your help. More ideas appreciated!

Have you tried packing the database?

I recommend buying a bigger disk -- seriously, before you run out of space. :)

Jim

-- 
Jim Fulton
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Jose Benito Gonzalez Lopez]

Hi,

Actually, we have two different versions (current and beta) of the software,
working against  the same ZODB. We are just developing new features
on top of the beta version.
Fearing that it was due to some changes done by a programmer
we have disabled the "beta version" but the growing of the DB is still
there...and since our last email we've got 5GB more (71GB now)  We 
are a bit worried
since the disk space is 102GB and  the application is widely used here 
at CERN and
for outsiders too.

So, up-to-now we are afraid that it is not a problem with any 
modifications in the codeand
no clue about it.
We are trying to pack to see the difference with the data.fs file.

Jim, I am still running your script "class_stats.py".and it seems it 
is going to take a while yet

Thank you so much for your help. More ideas appreciated!

Best regards,
Jose




Alan Runyan wrote:
>> I'd just like to add that there's some changes that can be related to this:
>>
>> - we had some classes inheriting from Persistent that now inherit from
>> something else as well (but no extra arguments are being added, AFAIK);
>> - we added some zope.interface definitions to some Persistent classes;
>>
>> maybe this causes some kind of behavior that we were not aware of?
>> 
>
> I doubt it.  Thousnads of people are doing this and do not report the same
> behavior.
>
> What is more likely is that a programmer is changing a persistent object
> very often.  One of the downsides of the ZODB is that it is so transparent
> it is possible to unwillingly make database changes.
>
> A design pattern for RDBMS is to have 2 pools. READ pool and WRITE pool.
> Often the READ pool comes from some replica and WRITE is to the master.
> I'm unsure this pattern would work for ZODB.  I know Malthe was thinking
> about this but unsure if he had anything concrete.
>
> cheers
> alan
>   

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Wichert Akkerman]

On 2009-12-7 17:34, Jim Fulton wrote:
> On Mon, Dec 7, 2009 at 11:00 AM, Alan Runyan  wrote:
> ...
>> A design pattern for RDBMS is to have 2 pools. READ pool and WRITE pool.
>> Often the READ pool comes from some replica and WRITE is to the master.
>> I'm unsure this pattern would work for ZODB.  I know Malthe was thinking
>> about this but unsure if he had anything concrete.
>
> Would both pools be used by a single process?

 From what I gather on high-load sites it is common to use one pool for 
queries, and a second pool for updates. Both are used in the same 
process(es)/threads.

Wichert.

-- 
Wichert AkkermanIt is simple to make things.
http://www.wiggy.net/  It is hard to make things simple.
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Christian Theune]

On 12/07/2009 05:00 PM, Alan Runyan wrote:
>> I'd just like to add that there's some changes that can be related to this:
>>
>> - we had some classes inheriting from Persistent that now inherit from
>> something else as well (but no extra arguments are being added, AFAIK);
>> - we added some zope.interface definitions to some Persistent classes;
>>
>> maybe this causes some kind of behavior that we were not aware of?
>
> I doubt it.  Thousnads of people are doing this and do not report the same
> behavior.
>
> What is more likely is that a programmer is changing a persistent object
> very often.  One of the downsides of the ZODB is that it is so transparent
> it is possible to unwillingly make database changes.
>
> A design pattern for RDBMS is to have 2 pools. READ pool and WRITE pool.
> Often the READ pool comes from some replica and WRITE is to the master.
> I'm unsure this pattern would work for ZODB.  I know Malthe was thinking
> about this but unsure if he had anything concrete.

My guess: have a ZEO fan-out setup and configure the read pool ZEO 
servers as read-only. Any transactions writing will fail miserably right 
away.

-- 
Christian Theune · c...@gocept.com
gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany
http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1
Zope and Plone consulting and development

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Jim Fulton]

On Mon, Dec 7, 2009 at 11:00 AM, Alan Runyan  wrote:
...
> A design pattern for RDBMS is to have 2 pools. READ pool and WRITE pool.
> Often the READ pool comes from some replica and WRITE is to the master.
> I'm unsure this pattern would work for ZODB.  I know Malthe was thinking
> about this but unsure if he had anything concrete.

Would both pools be used by a single process? Or is the assumption that
some applications would have a read-only view of the database and some
have a read-write view.

We've had customers that used read-only replicas.  We've thought of
doing the ourselves, but so far, the benefit hasn't outweighed the hassle
of separating our applications into read-only and read-write instances.

Jim

-- 
Jim Fulton
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Alan Runyan]

> I'd just like to add that there's some changes that can be related to this:
>
> - we had some classes inheriting from Persistent that now inherit from
> something else as well (but no extra arguments are being added, AFAIK);
> - we added some zope.interface definitions to some Persistent classes;
>
> maybe this causes some kind of behavior that we were not aware of?

I doubt it.  Thousnads of people are doing this and do not report the same
behavior.

What is more likely is that a programmer is changing a persistent object
very often.  One of the downsides of the ZODB is that it is so transparent
it is possible to unwillingly make database changes.

A design pattern for RDBMS is to have 2 pools. READ pool and WRITE pool.
Often the READ pool comes from some replica and WRITE is to the master.
I'm unsure this pattern would work for ZODB.  I know Malthe was thinking
about this but unsure if he had anything concrete.

cheers
alan
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Pedro Ferreira]

Hello all,
> Since some time ago (not sure since when) our database
> has passed from 15GB to 65GB so fast, and it keeps growing
> little by little (2 to 5 GB per day). It is clear that something is not
> correct in it.
>   
I'd just like to add that there's some changes that can be related to this:

- we had some classes inheriting from Persistent that now inherit from 
something else as well (but no extra arguments are being added, AFAIK);
- we added some zope.interface definitions to some Persistent classes;

maybe this causes some kind of behavior that we were not aware of?

Does anyone have any clue on whether this can cause troubles?

Thanks in advance,
Regards,

Pedro
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Jim Fulton]

On Mon, Dec 7, 2009 at 8:31 AM, Jose Benito Gonzalez Lopez
 wrote:
> Since some time ago (not sure since when) our database
> has passed from 15GB to 65GB so fast, and it keeps growing
> little by little (2 to 5 GB per day). It is clear that something is not
> correct in it.

Are you packing it regularly?

> We would like to check which objects are taking most of the space
> or just try to find out what is going on,...
>
> Any help or suggestions would be much appreciated.

I typically write simple scripts using the file storage iterator for
this.  (I tend to prefer
one-off scripts, mainly because I want to script mummies lile we have in
ZODB/scripts. :)

I've attached a script that computes statistics on class usage in a
file-storage database.

Hopefully, it is useful in its own right and useful as a starting
point for similar scripts.

Jim

-- 
Jim Fulton


class_stats.py
Description: Binary data
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Data.fs size grows non-stop

[Laurence Rowe]

2009/12/7 Jose Benito Gonzalez Lopez :
> Dear ZODB developers,
>
> Since some time ago (not sure since when) our database
> has passed from 15GB to 65GB so fast, and it keeps growing
> little by little (2 to 5 GB per day). It is clear that something is not
> correct in it.
>
> We would like to check which objects are taking most of the space
> or just try to find out what is going on,...
>
> Any help or suggestions would be much appreciated.

Take a look at my write up here:
http://plone.org/documentation/kb/debug-zodb-bloat

You will want analyze.py from the latest ZODB release (or download it
from http://svn.zope.org/ZODB/trunk/src/ZODB/scripts/) the version
that ships with Zope 2.10.9 is broken.

Laurence
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

[ZODB-Dev] Data.fs size grows non-stop

[Jose Benito Gonzalez Lopez]

Dear ZODB developers,

Since some time ago (not sure since when) our database
has passed from 15GB to 65GB so fast, and it keeps growing
little by little (2 to 5 GB per day). It is clear that something is not
correct in it.

We would like to check which objects are taking most of the space
or just try to find out what is going on,...

Any help or suggestions would be much appreciated.

Thank you so much in advance!

Best regards,
Jose

-- Jose Benito Gonzalez Lopez Voice: (+412276)74345 Fax: (+412276)68471 
CERN IT-UDS-AVC Indico 



___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev