For an html version of the following, see
<http://people.osafoundation.org/grant/rearchitecture.html>
--Grant
Introduction
------------
Now that Chandler Desktop 1.0 has shipped, the time has come to deal
with
more long-term issues with the Chandler code base, as we shift our
energies
to attracting a volunteer developer community to maintain the app and
build
new features.
When Chandler Desktop shipped its Preview release in 2007, we had
extensive
discussions on this list about how to deal with issues of performance,
scalability, extensibility, stability and code maintenance. For a
summary,
see `<http://markmail.org/message/bxk2acxyouz2tcp2>`_. Although OSAF's
organizational goals are somewhat different now, I still believe it is
true that the rearchitecture project is the only way to make Chandler
sustainable by a volunteer developer community.
This missive is intended to describe how we can utilize our remaining
months of paid developer time to move forward on the rearchitecture. Of
course, with any substantial undertaking, there are higher-level issues,
like assessing risks and non-technical goals: those will be addressed in
a separate email (or possibly blog post).
In what follows, I'll recap some of OSAF's thinking last year regarding
technical goals of the rearchitecture, and then follow up with a summary
of work we did on a pilot project (before focusing on releasing 1.0).
I'll
then outline the major of areas of work a complete rearchitecture would
involve. Lastly, I'll throw out what subset we think is achievable with
remaining OSAF paid developer time. However, once the basic
infrastructure
pieces are in place, there is considerable flexibility as to what we
tackle
afterwards, so I'll mention some alternate strategies (or ideas for
strategies).
.. _toc:
.. contents:: **Table of Contents**
Rearchitecture
--------------
Ideally, a complete rearchitecture involves creating a codebase that:
1. Has separate layers for storage, domain model, interaction and
presentation.
This helps parallel development by allowing, for example, development
of interaction components for a new feature before the domain model
is
ready.
It also enables trying different storage back ends for performance
investigations, since it implies there would be a well-defined and
limited
set of APIs a new storage layer would have to implement.
Similarly, different presentations (via different UI libraries) are
possible, even though this wasn't (and still isn't) a goal for the
rearchitecture project.
2. Is "clean room" unit tested.
By this, I mean PJE's goal of having every line of code in the
project be
covered by at least one unit test. The main idea here is to reduce
the
risk of widespread changes (for instance, the kind of work that
often needs
to be done to fix performance problems).
Together with #1, this also implies that unit tests should test a
single
layer at a time, which should make tests fast. In the Chandler 1.0
situation,
if you write a test of the osaf.app package, you end up having to
run code
to create repository objects for the entire UI, leading to a set-up
time of
several seconds for that single test case.
3. Is thoroughly documented.
This decreases the ramp-up barrier for both internal and external
developers,
of course.
The Architecture Pilot Project (APP)
------------------------------------
In November/December of 2007, Phillip, Jeffrey and I spent some time
on a
pilot project for rearchitecting Chandler. The proposed scope for the
project
can be found in `this thread
<http://markmail.org/message/mh5thlwns6tmxx2x>`_, while some of the
technical
details can be found in the various README files in the chandler
rearchitecture
branch, viz.
#. `Top Level <http://svn.osafoundation.org/chandler/branches/rearchitecture/README.txt
>`_
#. `Platform <http://svn.osafoundation.org/chandler/branches/rearchitecture/Chandler-Platform/README.txt
>`_
#. `Application <http://svn.osafoundation.org/chandler/branches/rearchitecture/Chandler-App/README.txt
>`_
#. `Debugging <http://svn.osafoundation.org/chandler/branches/rearchitecture/Chandler-Debugging/README.txt
>`_
At a high level, the primary goals of the pilot project were to
demonstrate
feasibility of rearchitecture, and also to drive adoption of its ideas
amongst OSAF developers. (Ultimately, the plan was to have individual
developers move over from the old code base to the new as we ported over
people's areas of expertise). As a result, we spent more time on visibly
demoable features than if we were trying to implement 1--3 above in a
vacuum.
Although the pilot never really ran to completion, as a result of OSAF's
organizational restructuring, we did accomplish the following:
- We adopted the `Trellis API
<http://peak.telecommunity.com/DevCenter/Trellis>`_ for dependency
management (i.e. to have changes between different parts of the
app's object graph propagate correctly).
- The demo contains a pluggable architecture for the platform,
(incorporating
wx/twisted), for the Chandler-specific pieces and for debugging via
PyCrust.
- There is demo UI for sidebar, calendar, preview area and
minicalendar, but
not detail view.
- The domain model layer was intentionally left very minimal (i.e. just
enough to support demoing the GUI). However, propagation of domain-
level
changes to the running UI was demoable by using the debugger.
- The code contained the beginnings of a calendar data model, as
originally
`outlined <http://osdir.com/ml/python.peak/2007-07/msg00030.html>`_
by Phillip.
- There were some tests of presentation-level wx code via the `Python
Mocker
<http://labix.org/mocker>`_ library. This was partially done as an
investigation of what it takes to write "clean room" unit tests for
wx
code.
What we need to implement
-------------------------
Here, I have a broad breakdown of the areas we can contemplate working
on. This list is roughly in an implementation order, where we build a
complete application from the
"bottom up". However, once the basic infrastructure is built (i.e. the
first
three items) we can envision changing the order, as well as precisely
which
user features we want to implement in each area.
So, as you move down the list, the descriptions become more vague. More
details will be forthcoming as we undertake earlier items, and plan new
ones! For completeness, I've included some parts I doubt we'll get to.
#. **[Domain]: Infrastructure/Metamodel**
Basics for bootstrapping plugins and for having persistable
objects. In
other words, support for adding types via plugin entry points, for
some
notion of what is currently called ``Kind``, for extending existing
objects (akin to today's ``Annotation`` and ``Stamp`` concepts),
and for
making references to persistable objects by name. (This doesn't
include
support for actually persisting objects, but does support a kind
of core
schema that the Storage layer can implement).
#. **[Domain]: Item and DashboardEntry components**
Phillip and I sketched out a design for separating out what is
currently
the ``Item`` class (i.e. basic persistable object) from what is now
``ContentItem``. The latter currently encompasses several different
concepts ("Item created by the domain model" is one), but the one
we wanted
to focus on is "Item viewable in the Detail View". (Possibly
``ViewableEntry`` or something similar is a better name than what
we came
up with at the time). The main idea here is to have each
``DashboardEntry``
object be generated dynamically by an ``Item``, and thereby have a
way to
deal with cases like recurring events.
There still remain some design questions at this level that need
to be
resolved. For example, in the plugin-centric implementation of the
rearchitecture pilot, we see core Items as being composed of
"extensions" or "parts" (coming from various plugins), but it's
unclear
as to whether you can have more than one part for a given item
type (or
whether items even really have a type). However, these questions
can be
resolved after implementing things like DashboardEntries (and
Reminders, below).
`Requires: [Domain]: Infrastructure/Metamodel`
#. **[Domain]: Reminders**
At its core, a ``Reminder`` will be a separate type (i.e. database
table)
indexed by fire date. With a flexible storage layer, it would be
possible
to have this "table" be implemented outside the core database
(e.g. via a
directory full of entries), allowing for a small, long-running
reminder
daemon.
The ``Reminder`` system will work a little more like the current
chandler ``osaf.startup`` module: A single entry will point at a
piece of
code to be run when the reminder is due. This will allow both user-
level
reminders (i.e. ticklers or event-relative alarms) as well as tasks
scheduled by plugins (including the Chandler application).
`Requires: [Domain]: Infrastructure/Metamodel`
#. **[Interaction]: Basic infrastructure**
The basic design can be found `here <http://www.eby-sarna.com/pipermail/peak/2007-November/002799.html
>`_. This design is still incomplete, and so
will need to be fleshed out with the usual round of doctests and
implementation use cases.
`Requires: (None)`
#. **[Domain]: Calendar**
In the past, we spent some time looking at a scheme for implementing
calendar queries in reasonable time (i.e. so that most of the work
can
be done by database indexes, without too much time being spent at
the
Python level). In the rearchitecture pilot, Jeffrey had written
some code
at the interaction layer where ``Month`` objects cache events for a
calendar.
In practice, things will be complicated by factors such as
"floating"
timezones. Probably the best approach will be to have slightly
"fuzzy"
system, where we store and index event _dates_, but have queries
expand the
date range by a day on each end, and filter out events outside the
range.
Effectively, we are making the "display time" of events be
dynamic, while
the stored value is static. This does require extra computation to
retrieve any event, but some database backends (e.g. sqlite) can be
configured to add a date conversion function, and thereby do the
filtering/sorting themselves. (This function could be written in
C, and
therefore be "fast").
`Requires: [Domain]: Item and DashboardEntry components`
#. **[Storage]**
In the intervening months, Phillip has spent some
time investigating how to bind a Trellis-based object system onto
a SQL
back-end. After investigating and discarding a SQLAlchemy-based
approach,
he's settled on something that allows python expressions to be
used for
queries, and which will translate themselves as needed to SQL
queries. This
is flexible enough to allow you to write higher-layer tests that run
entirely in memory. (Clearly, there will eventually be a need to
write
functional tests that operate on a "real" database).
#. **[Interaction]: Table model**
During the pilot project, we started developing an interaction model
for Tables, in particular, the user-sortable kind like the Dashboard
in Chandler. In fact, Phillip added some support for this to the
Trellis
(via the `SubSet <http://peak.telecommunity.com/DevCenter/TrellisCollections#observing
>`_ and `Observing <http://peak.telecommunity.com/DevCenter/TrellisCollections#observing
>`_ types), but I (Grant) had yet to
adopt them.
`Requires: [Domain]: Basic infrastructure`
#. **[Interaction]: Detail view**
Some kind of "editing context" ("save buffer" is another term) is
needed
here, to model how changes make their way from the UI to lower
layers. In
particular, there would need to be a way to hook into the save
mechanism to
handle confirmation of recurrence changes. A save buffer can also
be used
to detect sharing conflicts in the case where you are busy editing a
value that is changed remotely.
`Requires: [Domain]: Basic infrastructure, [Interaction]: Basic
infrastructure`
#. **[Domain] EIM Import/Export**
The difficulty with porting the current Chandler EIM layer
is that it is tied to Chandler's data model (in particular, with
respect to
recurrence). On the plus side, it has a lot of unit tests to ease
the
transition. In favor of porting EIM are two other factors:
- EIM (or, more correctly, .chex backups) is the path
for moving users away from Chandler 1.0 to the rearchitected
project.
- As evidenced by the current Chandler ICalendar import/export
code, it
is also a useful bridge for supporting other formats.
#. **[Presentation] Styling framework**
For details, see this `PEAK list posting <http://www.eby-sarna.com/pipermail/peak/2007-December/002814.html
>`_. This is the layer that supports the "P" in
what is currently "CPIA" in Chandler. While it is something we want
long-term, it's not a strict requirement for implementing demo-level
user interfaces, which can be wired up directly as before.
#. **[Domain] Sharing**
The current sharing code base has two drawbacks so far as porting
over to the new architecture:
- The basic classes are not thoroughly unit tested; mostly there
are functional tests whose coverage of the code base is unclear
at the moment.
- The API isn't well-suited to using non-blocking (twisted) APIs
for the network pieces.
As a result, it will be a considerable amount of work to port over
all the different kinds of shares currently supported by Chandler.
However, it may well be possible to implement one particular kind
of conduit.
`Requires: [Domain] EIM Import/Export`
#. **[Interaction] Sharing**
Mostly, this involves modelling interactions around how conflicts
are
presented to the user (assuming we even want to implement sharing
conflicts at first), as well as issues like visibility of elements
that show syncing status.
#. **[Domain] ICalendar Import/Export**
We do have good unit tests for this functionality in current
Chandler,
and the strategy of implementing this on top of EIM has worked out
well. However, it's unclear whether full ICalendar support is
necessary
for the "core" app we are trying to build, so this might be on
opportuntiy for outside development.
#. **[Presentation] Remaining tasks**
While it is nice to have demos (even to attract volunteer
developers),
it's unlikely we'll have time to make a primary focus of reproducing
Chandler's UI. There is some new calendar code that is demo-
worthy, and
in fact represents a good area for developers interested in
helping out
with the rearchitecture project.
#. **[All] Email Support**
The existing mail service layer in Chandler is implemented in a
fairly
well-tested way, but attempting to support the gamut of IMAP/POP/
SMTP
servers is a testing and developer effort that I don't think
we have resources for at the moment.
#. **[All] Printing support**
Phillip recently mentioned that we could possibly use the
`PythonReports
<http://pythonreports.sourceforge.net/>`_ project as a means for
implementing some specialized printing layouts for Chandler 1.0
that could
be relatively straightforwardly ported to the rearchitecture
branch, but
this remains a project for non-core OSAF developers.
1.0: What can we achieve from here?
-------------------------------------------
Basically, continuing down the pilot project path (i.e. filling in the
pieces of a lookalike of the current app) doesn't really make sense,
because
it would lead to an improved (i.e. more tested/testable) version of the
Chandler 1.0 app, but without #2 in particular, we would still face
similar
reliability challenges as before.
Another consideration is that our organization and target audience have
changed considerably since last year. Rather than having a large paid
staff
to port the code, we will be relying on more limited resources, and
ultimately the goal is to have the code live on as a volunteer-run
project.
So, it makes sense to implement the original rearchitecture goals, (i.e.
1--3 above), while not necessarily focusing on a complete
reimplementation
of the current Chandler. In fact, leaving out features, or implementing
them minimally can be seen as creating opportunities for volunteer
developers. For example, only supporting sharing via the Hub leaves it
open to others to add support for other Cosmo servers, or for CalDAV in
general, or for particular Atompub-based servers.
With this in mind, a goal that Jeffrey, Phillip and I agreed would be
achievable using OSAF's remaining paid staff time is:
Achievable Goal
~
~
~
~
~
~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Create fully-tested interaction, domain and storage layers for
Chandler:**
This would include support for list and calendar views, stamping and the
detail view, but would not include support for email/sharing.
Besides achievability, this will will also provide promise that the
full rearchitecture is feasible. It will also leave room for outside
developers to
volunteer, and/or build custom applications.
As mentioned earlier, there is flexibility as to what we implement
once the first three or four items above. As a result, we can envision:
Possible Different Goals
~
~
~
~
~
~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Here are a couple of possible different routes we could take:
**Sharing/"Chandler Lite":** Cut back, say, on some calendar features,
but
on a full application (i.e. up to presentation layer) that can sync a
task list
with Cosmo.
**Calendar UI:** Implement enough of a calendar presentation layer to
allow
other developers to work on a small, self-contained calendar
(including a detail view).
There is also a parallel view of the rearchitecture,
where we can look at user-level features that can be omitted in
a first pass at either the interaction or domain layer. (Examples
might be the minicalendar or preview area in current Chandler). This
should obviously be discussed separately, especially once the basic
infrastructure is in place.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "chandler-dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/chandler-dev
