-----------------------------------------------------------------
Proposal: A Logical Format API for the Chandler Sharing Framework
-----------------------------------------------------------------
(nicely formatted HTML version can be found at
http://peak.telecommunity.com/DevCenter/ChandlerSharingModel
)
Overview
========
This document describes a proposed API for the Chandler sharing framework
to allow individual parcels to support backward and forward-compatible
sharing, even when their domain model changes between parcel versions and
the clients doing the sharing do not all have the same version of the
parcel installed.
The proposed API does this by allowing parcel developers to specify
"sharing schemas" for their items. A sharing schema is a kind of logical
transmission format, that breaks items down into simple records containing
elementary data types that are easy to store or transmit for use by other
programs.
Sharing schemas defined using this API will also be used to implement "dump
and reload" including schema evolution during upgrades or downgrades. As a
parcel's item schema changes, its sharing schema(s) must be modified so
that data produced by previous versions of the parcel can still be
imported. A parcel can also optionally provide support for exporting data
in such a way that it can be read by older versions.
Typically, a parcel will provide its own sharing schema for the Kinds and
Annotations it contains. However, it's also possible for a parcel to
define one or more sharing schemas for other parcels that it depends on.
Parcel developers define a sharing schema by defining one or more record
types (using the [EMAIL PROTECTED] decorator), and one or more
``sharing.Schema`` subclasses. The record types define the format of the
data to be shared, and the ``sharing.Schema`` classes provide code that
convert items to records and vice versa. The ``sharing.Schema`` base class
will provide many utility methods to automatically handle common mapping
patterns, so that most schemas will include relatively little code.
Records and Record Types
========================
The API treats all data as "records", similar to the rows of a table in a
relational database. Each record is of some "record type", and contains a
fixed number of fields. As in a relational database, each field can hold
at most one value of one elementary data type, such as a number, string,
date/time value, etc. A field may also hold a value of ``None``, which is
conceptually similar to the "null" value in a relational database. There
is also a second kind of "null" value, called ``sharing.NoChange``, that
can be used to create "diff" or "delta" records that indicate only certain
parts of the record are changed.
To define a record type, a parcel developer will write a short function
using the [EMAIL PROTECTED] decorator. For example::
@sharing.recordtype("http://schemas.osafoundation.org/pim/contentitem";)
def itemrecord(itsUUID, title, body, createdOn, description,
lastModifiedBy):
"""Record type for content items; note lastModifiedBy is a UUID"""
The above defines a record type with 6 fields, named by the arguments to
the function. The string passed to ``recordtype()`` must be a unique URI,
and will be used to allow other programs (such as Cosmo) to identify
whether a particular record type is known to or understood by it.
(Note that any unique URI is acceptable, including URIs of the form
"uuid:...". That is, you need not have control of a domain name in order
to create your own unique URI, as you can use a UUID to create one.)
Type Checking and Conversion
----------------------------
In the simplest case, a recordtype function need not contain any code or
return any value. In such a case, the argument names -- and default
values, if any -- are sufficient to describe how the resulting record type
should behave. However, if you wish to provide type checking or conversion
of arguments, you will need to write a bit more code in a record type. For
example, here's a new version of the example above, that does a bit more
work to ensure it is used correctly::
@sharing.recordtype("http://schemas.osafoundation.org/pim/contentitem";)
def itemrecord(itsUUID, title, body, createdOn, description,
lastModifiedBy):
"""Record type for content items"""
if isinstance(lastModifiedBy, schema.Item):
lastModifiedBy = lastModifiedBy.itsUUID
if not isinstance(lastModifiedBy, UUID):
raise TypeError("lastModifiedBy must be an item or a UUID")
return itsUUID, title, body, createdOn, description, lastModifiedBy
Note, however, that although a recordtype function can accept items as
input, it *cannot* return items as output. They must be converted to
UUIDs, strings, numbers, or other elementary values. The EIM API is
record-oriented, not object-oriented.
Inter-Record Dependencies
-------------------------
Just as in a relational database, records may contain references to other
records. For example, let's suppose that we want to have a record type to
record "tags" associated with a content item. And, we want tags to be a
kind of content item themselves. Here's what we would do::
@sharing.recordtype("http://schemas.osafoundation.org/pim/tag";,
itsUUID = contentitem.itsUUID
)
def tag(itsUUID, tagname):
"""Record type for a tag; "inherits" from itemrecord"""
@sharing.recordtype("http://schemas.osafoundation.org/pim/contentitem/tags";,
item = itemrecord.itsUUID, tag = tag.itsUUID
)
def tagging(item, tag):
"""Record type linking tags to items"""
if isinstance(item, schema.Item):
item = item.itsUUID
if isinstance(tag, schema.Item):
tag = tag.itsUUID
if not isinstance(item, UUID):
raise TypeError("must be an item or a UUID", item)
if not isinstance(tag, UUID):
raise TypeError("must be an item or a UUID", tag)
return item, tag
Keyword arguments passed to the ``recordtype()`` decorator allow you to
define relationships between the fields in the record type being defined,
and the fields of existing record types. As you can see above, we use
``itemrecord.itsUUID`` and ``tag.itsUUID`` to refer to the ``itsUUID``
fields of the ``itemrecord`` and ``tag`` record types. This creates a
dependency between the record types, and affects the order in which records
will be imported or exported.
In the examples above, the order of record processing will always begin
with ``itemrecord``, followed by ``tag`` and ``tagging`` records. More
specifically, before a ``tagging`` record is processed, any ``tag`` and
``itemrecord`` records that have matching ``itsUUID`` fields will be
processed. And before a ``tag`` record is processed, any ``itemrecord``
with the same ``itsUUID`` will be processed first.
Recordtype Evolution
--------------------
As an application's schema changes, it may be necessary to add new fields
to existing record types. This can be done, as long as:
1. New fields are added to the end of the existing fields in the record
type function.
2. New fields must have a default value defined, and import code for the
record type must be able to handle a value of ``sharing.NoChange``. (This
allows two Chandlers with different versions of a parcel to interoperate,
even if one supports fields that the other does not.)
3. The record type's URI must not change, and all existing fields's names
must remain the same and in the same order.
In other words, if you want to change the name, meaning, or position of an
existing field (or remove fields), you *must* create a new recordtype with
a new URI to replace the old one. Such replacement also means that you
must create a new ``sharing.Schema`` in order to retain backward
compatibility with older sharing clients. (This topic will be covered in
more detail below, since we haven't talked about ``Schema`` classes yet.)
Defining a Sharing Schema
=========================
By themselves, record types only define a *format* for sharing and
import/export. To complete a parcel's sharing definition, it must also
define how to convert between items and records, by creating a
``sharing.Schema`` subclass. At minimum, such a subclass must include a
unique URI, a version number, and a user-visible description::
class ContentSchema(sharing.Schema):
uri = "http://schemas.osafoundation.org/pim";
version = 1
description = _("Core content items")
The sharing system will use these attributes to determine what formats it
"understands", and to allow users to select what version of a particular
format should be used for a particular "share", if applicable. (This is so
that users can choose an older version in order to collaborate with users
who don't have the latest version.)
It's important to note that unlike the Chandler application schema, not
every change to parcel's schema will require a change in its schema version
number. A ``sharing.Schema`` version number *only* needs to change when a
record type is to be replaced. That is, as long as you are only *adding*
new record types, or adding new fields to existing record types (as
described in the previous section), there is no need for the version number
to change. That's because older code will still be able to read the
records and fields that it understands, and ignore the new record types and
fields that it does not.
When a schema gets a new version number, you will often want to create a
second ``sharing.Schema`` subclass, to keep backward compatibility. For
example, we might have::
class OldContentSchema(sharing.Schema):
uri = "http://schemas.osafoundation.org/pim";
version = 1
description = _("Core content items")
#
# code to read/write old format here
# ...
class NewContentSchema(OldContentSchema):
version = 2
#
# code to read/write new format here
# ...
This allows the parcel to support sharing (or import/export and
dump/reload) of older formats. Any aspects of the old schema that are
retained by the new one can potentially be inherited, eliminating the need
for duplicate code. (Notice that in the above example we're also
inheriting the ``uri`` and ``description`` attributes.)
Export Methods
--------------
In order to function, a ``sharing.Schema`` subclass must define "exporter"
and "importer" methods. Continuing our simple item/tags example, let's
look at some exporters::
class ContentSchema(sharing.Schema):
uri = "http://schemas.osafoundation.org/pim";
version = 1
description = _("Core content items")
@sharing.exporter(pim.ContentItem)
def export_contentitem(self, item):
yield itemrecord(
item.itsUUID, item.title, item.body, item.createdOn,
item.description, item.lastModifiedBy
)
for t in item.tags:
yield tagging(item, t)
@sharing.exporter(pim.Tag)
def export_tag(self, item):
yield tag(item.itsUUID, item.tagname)
An exporter method is declared using [EMAIL PROTECTED](cls, ...)``, to
indicate what class or classes of items are handled by that
method. Methods may be generators that yield records, or they can just
return a list or other iterable object that yields records.
More than one exporter can be called for the same item. In the example
above, assuming that ``pim.Tag`` is a subclass of ``pim.ContentItem``, then
the ``export_contentitem()`` method will be called before ``export_tag()``
for each ``pim.Tag`` item being exported. The same principle applies for
export methods that apply to annotation classes; the export method for each
applicable annotation class will be called. All of the records supplied by
the various export methods are then output.
Notice that this means that export methods must be written in such a way
that they do not produce duplicate records. Each export method should
therefore confine itself to writing records specific to the class(es) it is
registered for, and allowing the base class export methods to handle the
base classes' data.
If you subclass your ``sharing.Schema``, the subclass inherits all of the
export methods defined by the base class. If you wish to redefine the
export handling for some particular item or annotation class, you must do
so by explicitly using
a new [EMAIL PROTECTED]()`` decoration; it is *not* sufficient to just
override a method with the same name. (This is because for performance
reasons, the lookup mechanism is not based on method names.)
Finally, you can declare more than one exporter for the same type in the
same ``sharing.Schema`` class; both will be called for items they apply to.
Importer Methods
----------------
Each ``sharing.Schema`` must declare "importer" methods to handle each
record type that it outputs. Here are some importers for the record types
we defined previously::
class ContentSchema(sharing.Schema):
# ...
@itemrecord.importer
def import_contentitem(self, record):
self.loadItemByUUID(
record.itsUUID, pim.ContentItem,
title = record.title,
body = record.body,
createdOn = record.createdOn,
description = record.description,
lastModifiedBy = self.loadItemByUUID(record.lastModifiedBy)
)
@tag.importer
def import_tag(self, record):
self.loadItemByUUID(record.itsUUID, pim.Tag,
tagname=record.tagname)
@tagging.importer
def import_tagging(self, record):
the_item = self.loadItemByUUID(record.item)
the_tag = self.loadItemByUUID(record.tag)
the_item.tags.add(the_tag)
Notice that importer methods do not need to return a value; their sole
purpose is to do whatever processing is required for the received records.
Only one importer can be registered for a given record type in a particular
``Schema`` subclass. Importers registered by base classes are inherited in
subclasses, unless overridden using the appropriate decorator in the
subclass. If you don't want to inherit *or* override support for a
particular record type, the record type can be listed in the
``do_not_import`` attribute of the class, e.g.::
do_not_import = sometype, othertype, ...
Utility Methods
---------------
The ``loadItemByUUID()`` method shown in the importer examples above is a
utility method provided by the ``sharing.Schema`` base class. It takes a
UUID, an optional item or annotation class, and keyword arguments for
attributes to set. The return value is an item of the specified class, or
a plain ``schema.Item`` if no class was specified and the item didn't
already exist.
If an item with the given UUID already exists, it's returned. If a class
was specified, the item's kind is upgraded if necessary. For example, the
importer for the ``tag`` recordtype above invokes it like this::
self.loadItemByUUID(record.itsUUID, pim.Tag, tagname=record.tagname)
If a ``pim.ContentItem`` of the right UUID exists, its kind is upgraded to
``pim.Tag``. If it does not exist, it is created as a ``pim.Tag``. If an
item exists, and it has a kind that is a subclass of ``pim.Tag``, its kind
will not be changed. This algorithm allows items' types to be upgraded
"just in time" as information becomes available.
If any of the attribute values supplied to ``loadItemByUUID()`` are
``sharing.NoChange``, no change is made to the attribute. Similarly, if
the UUID supplied to ``loadItemByUUID()`` is ``sharing.NoChange``,
``sharing.NoChange`` is returned instead of an item.
Over time, there will be additional utility methods added to
``sharing.Schema`` as common usage patterns are identified, to help reduce
the amount of boilerplate code that needs to be written.
The Sharing Interface
---------------------
For each import or export operation to be performed, the sharing framework
will create instances of the appropriate ``sharing.Schema`` subclasses,
passing in a repository view. So in our running example, the sharing
framework would invoke ``ContentSchema(rv)`` to get a ``ContentSchema``
instance with an ``itsView`` of ``rv``.
Then, depending on the operation(s) to be performed, the sharing framework
will call some of the following methods, which all have reasonable default
implementations provided by ``sharing.Schema``:
startExport()
Called before an export process begins, to allow the ``Schema``
instance to do any pre-export setup operations. The default implementation
does nothing, but can be overridden to initialize any data structures that
might be needed during the export operation.
exportItem(`item`)
Called to export an individual item, it should return a sequence or be
a generator yielding the relevant records for the supplied `item`. The
default implementation automatically looks up the registered export methods
and calls them, combining their results for the return value. This method
can be overridden if you have a sufficiently complex special case to need
it, or if you want to create a different way of registering
exporters. Note also that it's okay for this method to return an empty
sequence.
(Note: the sharing framework must not make any assumptions about a
relationship between the records returned, and the item passed in, since
some of the records may be for *related* items. Also, a schema can choose
not to export records for individual items, but instead just track which
items are to be exported and then provide all of the records when
``finishExport()`` is called.)
finishExport()
Called after an export operation is completed, this method should
return a sequence or be a generator yielding records. These records will
be exported along with any that were yielded by calls to
``exportItem()``. The default implementation of this method just returns
an empty list, but can be overridden to return or yield records, and
perhaps to tear down any temporary data structures created by
``beginExport()`` or ``exportItem()``.
startImport()
Called before an import operation begins. The default implementation
does nothing, but can be overridden to initialize any data structures that
might be needed during the import operation.
importRecord(`record`)
Called for each record to be imported, in an order determined
automatically by the declared inter-recordtype dependencies. (That is,
this method will not be passed a record until all the records it depends on
have been imported first.) The default implementation of this method
simply looks up and calls the relevant importer method.
finishImport()
Called after an import operation is completed. The default
implementation does nothing, but can be overridden to do any necessary
cleanup or finish-out of the import process.
Notice that for both ``importRecord()`` and ``exportItem()``, there is no
requirement that all processing for the given item or record take place
immediately. Some complex schema changes (or complex schemas) may need or
want to simply keep track of what items are being exported or what records
are being imported, and then do the actual importing or exporting in
``finishImport()`` or ``finishExport()``.
Thus, the sharing framework must not assume that it has seen all records
until all ``finishExport()`` methods (for each schema being exported) have
been called. Similarly, it cannot assume that items in the repository are
in their finished state until all of the active schemas' ``finishImport()``
methods have been called.
Implementation Details and Open Issues
======================================
Processing "Diffs"
------------------
Most of the API and examples above are written in terms that assume a
more-or-less "complete" and "additive" transfer of records, rather than
being difference-oriented.
It is assumed that ``sharing.NoChange`` will be used in record fields to
indicate that the field's value has not changed, and that the sharing
framework will be responsible for replacing records appropriately. Record
objects will probably support subtraction to produce diffs, e.g.
``diffRecord = newRecord - oldRecord``. It's possible that the sharing API
will do this by exporting both old and new versions of the same collection,
and then differencing the records that are in common, and perhaps creating
some kind of "deletion" record for records found in the old, but not the new.
At present, however, the API as designed has no support for deletion as
such. For well-defined collections (such as the ``.tags`` attribute in the
examples), this could be handled by clearing the collection when the first
record is received, at the cost of re-transmitting all members of the
collection. The alternative possibility is to never delete items from
collections, only add. (Which is what the above examples do; i.e., tags
are always added, and items are always created or updated, but nothing is
ever deleted.)
Key Management
--------------
The proposed API doesn't have a way to specify what fields of a record are
"keys" or are expected to be unique, except indirectly. Inter-record
dependencies define some keys by implication, in that the depended-on field
must be unique in order for a dependency to have meaning.
However, producing diffs for a record requires that the record know of one
or more fields that produce a "primary key" in database terminology,
because a difference record must always contain enough information for the
receiver to identify what the difference is to be applied to!
At this point, it's not clear to me if we will need some special way to
designate a primary key. One obvious way to do it would be to assume that
the first field is always the primary key, except that this doesn't work
for records like the ``tagging`` example, which effectively have *all*
their fields as part of the primary key.
Type Information
----------------
Currently, there is no way to define or look up what types are used in what
fields, nor is there any formal definition of what types are
acceptable. This is a big gaping hole in the current proposal that must be
remedied before we can expect any sort of dependable interoperability (e.g.
w/Cosmo). For now, we are punting on this until we get a better idea of
what's actually needed.
This gap in the proposal also means that we aren't in a position to e.g.
define a bunch of record types to describe other record types. This kind
of meta-description is important for being able to define an
extensible/discoverable sharing format between Chandler and Cosmo.
Multiple Inheritance
--------------------
There are a few quirks regarding multiple inheritance. First, I think that
we're going to have to prohibit a ``sharing.Schema`` class from inheriting
from more than one other ``sharing.Schema`` class, in order to avoid
possible ambiguities as to what inherited importers or exporters should be
invoked when both base classes have different ones defined, and the
subclass doesn't override them.
Second, there is a peculiar corner case that can arise when sharing data
between two machines, when multiple parcels and multiple inheritance are
involved. Suppose that there are two parcels "a" and "b" containing
classes "A" and "B" respectively, both of which are subclasses of
``pim.Item``. And then there is a parcel "c", containing class "C", which
inherits from both "A" and "B".
Let us further say that machine 1 has all three parcels installed, but
machine 2 has only parcels "a" and "b". As long as these two machines are
only sharing instances of "A" and "B", everything will be fine, but if
machine 1 transmits a "C" instance to machine 2 there will be a problem.
When machine 2 tries to process the records related to ``pim.Item`` or to
"A" instances, everything will work correctly. However, the "C" instance
will have created both "A" and "B" records, making it impossible for
``loadItemByUUID()`` to find a suitable kind. Morgen and I discussed the
possibility of having it simply synthesize one, but this could produce some
problems of its own, in that the Chandler UI might not know how to
correctly display this peculiar A/B hybrid, without additional information
that can only be found in parcel "c" -- which machine 2 does not have.
For the first version, we will probably have to have some kind of kludge to
detect this situation and handle it -- but precisely *how* we will handle
it is still open to investigation. We may have to create the problem first
in order to get a better handle on it.
Schema Registry and Selection
-----------------------------
``sharing.Schema`` classes and [EMAIL PROTECTED] objects will have to
be part of a parcel's persistent data, stored in the repository at the same
time that the parcel's kinds and annotations and so forth are
initialized. The sharing parcel will probably have some kind of persistent
object(s) stored in the repository that reference schemas and index them by
their supported record types and kinds, so that the sharing framework can
look them up.
The exact nature of these data structures is currently undefined. The data
structures needed are dependent on how schemas will need to be selected by
the sharing framework, so it's likely that a first cut implementation of
the API won't actually create any, and rely on the sharing framework to
just explicitly select what schema(s) to use for a particular share.
The selection strategy is further complicated by the possibility that more
than one schema might be offering to produce or consume records of the same
record type.
And last, but not least, due to the persistent nature of schema classes and
recordtype objects, it's likely that the Chandler application will need to
either set aside another parcel to contain the core types' sharing schema,
or else define that schema within the sharing parcel itself. (Otherwise,
we would be introducing circular parcel dependencies between the core types
and the sharing parcel.)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "chandler-dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/chandler-dev
