Hi Jim,

On 08/08/2010, Jim Fulton <j...@zope.com> wrote:
> On Thu, Aug 5, 2010 at 2:36 AM, Martin Aspeli <optilude+li...@gmail.com>
> wrote:
> ...
>> I have a package (plone.registry) that currently has a persistent
>> structure like this:
>>
>> Registry(Persistent)
>> |
>> +--> Records(Persistent)
>>       |
>>       +--> BTree of Record(Persistent)
>>              |
>>              +--> PersistentField(Persistent)
>>
>> That is, a Registry is a persistent object containing a persistent
>> Records object that in turn contains a BTree of persistent Record
>
> Since BTrees are mapping, I assume that you mean the values are
> records and that the keys are something boring like strings or
> integers.

Yes. The keys are strings.

> I like to use mathematical notation when talking about BTrees and
> sets, as in:
>
>   Registry
>      BTree {? -> Record}
>
>
>> objects that contain a persistent PersistentField and a primitive
>> value.
>>
>> This is quite inefficient, though, because it results in a lot of
>> object loads. On any given request, some of our projects load a dozen
>> or more values from the registry. Each is just a simple primitive, but
>> we need to load the full shebang to make it work.
>
> Not sure what you mean by "full shebang".

The Registry, Records object, the relevant Record in the relevant
BTree, and possibly the PersistentField object.

In the "new" branch it just looks up the value in the relevant BTree.

>> Now, I'd like to move to this structure:
>>
>>  Registry(Persistent)
>>  |
>>  +--> Records
>>       |
>>       +--> BTree of Field
>>       |
>>       +--> BTree of values
>
> I'm foggy on what "field" and "value" are here or what your queries
> are doing. Maybe this is just a distraction.

Somewhat, unless you've worked with plone.registry. The point is to
allow the "get a value" API to just look at self.values[key], which is
a fast lookup and doesn't load anything except the relevant BTree
bucket + the registry itself.

>> Here, there's only one Persistent object, plus the two BTrees: one
>> holding all the fields and one holding all the values. Records no
>> longer needs to be persistent (its attributes become part of the
>> parent Registry's _p_jar).
>
> I wonder what role "Records" plays independent of the "Registry".

None, really. The main reason to have it is to be able to have an API
like registry.records with dict-like notation (there's also
__getitem__ on the registry, which returns the value of a given key,
not the Record). I made ``records`` an attribute of type Records, and
Records derives from Persistent. I wish I hadn't, since it can just
live in its parent's _p_jar.

> I also wonder why it matters whether it is persistent or not.

It's better if it isn't (one fewer object to load/fill up the cache),
though the real culprits are the many Record objects each being
persistent and loaded separately.

On a given request, we can end up loading a dozen or more values from
the registry, which means a dozen or more objects in the cache and
associated overhead.

>> Fields no longer need to be persistent
>> either, since they are in effect immutable objects. Values are
>> primitives anyway.
>>
>> I've done this (in a branch) and it works for new sites. However, I'm
>> having a nightmare trying to migrate old sites. As soon as I access
>> anything that uses the registry, I get ZODB errors, because the
>> persistent structure is now different. In particular, it's trying to
>> read a value into e.g. a Records object that used to derive from
>> Persistent, but now no longer does.
>
> What savings do you get by making Records non-persistent?

One fewer persistent object.

I think the real saving is in making the Record object non-persistent,
especially since the "read" use case can just read from the ``values``
BTree with the structure above.

>> What is the best way to manage this type of migration?
>
> Today, it probably makes the most sense to make new classes for the
> non-persistemnt objects.  You'll then need to write a script to
> rebuild the data structures.

Okay. So there's no way to get at the data if I take Persistent out of
the base classes for Records / Record.

Martin
_______________________________________________
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )

Reply via email to