Now that we're wrapping up Preview, it is time to have the conversation
about where to go next with Chandler Desktop. My goal with this email is
not to communicate final decisions; my goal is to show you how I'm
thinking about next steps and to give strategic context for making
decisions from here. I want to name open problems and list out possible
feature directions.
*Big Picture Goals*
The goal for Preview was to establish a baseline working application
that is complete enough to demonstrate the vision and be useful to real
users. We are very very close -- yay.
The next goal is to build a user base and a working community.
(I wrote a similar note in response to Ted's thread on the Cosmo team's
next steps -- at a high level the goals are similar for both teams.)
All decisions should be evaluated in the light of getting users and
building a community around Chandler.
Users and a community are key to funding continued work on the project.
Users and a community also potentially enable the project to live on in
the absence of funding, or with significantly reduced funding. Perhaps
most importantly, users and a community are the measure of succeeding at
what we set out to do.
*Shorter release cycles*
I think we have broad agreement that the desktop should be shooting for
shorter release cycles. The motivations are basically the same as with
the server team:
- support users by fixing blocking problems quickly
- be able to react to user feedback and opportunities to get new users
- display a sense of momentum on the project
I know the team has been discussing shorter release cycles already --
Philippe will continue to drive the specifics on the changes to the
release process.
*Automated testing*
A challenges to shorter release cycles: one of the problems we face is
incomplete/insufficient automated tests. We need to step back and take
stock of what we have now and what we need to do to be able to be
effective in writing automated tests.
I'll call out some specific problems:
- Test coverage. We could use much better code coverage (unit tests) and
feature/use case coverage (functional tests).
- Writing tests is currently painful, as is maintaining them. I know
that people hesitate to write tests when the tests impose big upfront
costs as well as ongoing costs.
- There is some confusion about multiple frameworks, what we are going
to support in the long run, and how to use the new framework effectively.
I know we all know this, but perhaps it helps to remind ourselves of the
benefits of a project culture with better test coverage and the habit of
writing tests as we add features and fix bugs:
- Manually doing all of those test suites every cycle is not really
feasible. We've all felt the pain last week running through the test
specs! Certainly those recurrence test cases beg for automation.
- A good regression test suite gives us confidence when making changes
to the system. It allows us to continually refactor code without
worrying about destabilizing the system. Continually refactoring code
leads to shinier happier code -- easier to add features, easier to fix
bugs, more fun to work with all around.
Shorter release cycles and the ability to make changes to the
application while maintaining a stable application are going to demand
better/more automated tests.
I'm looking to Aparna and Philippe to drive the solution to this problem.
*Feature roadmap*
We want to use feedback from Preview to inform what we need to do to get
to a solid 1.0 release of the Desktop. That said, we have a pretty good
idea of what the realistic possibilities are. (Well, doing all of these
at once is not realistic!) Themes/tenets:
- Fix usability problems. Preview is essentially a "public beta". We've
focused on getting something solid out the door, but we know it has
corners that people bump into. We need to fix problems that might block
initial uptake. We also need to fix problems that really annoy people in
day to day usage. (Mimi has identified her "Top 20" -- this is a good
starting point for 0.7.1:
http://chandlerproject.org/Notes/PostPreviewPlanning)
- Interoperate with other calendars (and pims). Work we do here should
be motivated by whether or not it gets us new users. Can users move
their existing data from other pims/calendars into Chandler? Will users
be more likely to experiment with Chandler if they can share calendars
with people using other clients and servers?
- Iterate on the dashboard/triage table. Preview is our first release of
this functionality -- we know we have work to do to get this right. This
includes how the sidebar and dashboard/triage table work together. It
could also include new features like tagging or the ability to make some
customizations.
- Complete basic features, particularly basic features for calendaring.
This includes adding a month view and printing features, in addition to
smaller fixes like slightly more complicated recurrence patterns.
- Add contacts. We've had many end user requests already. Also, some
core contact features (like a basic schema) will be a baseline for
people to extend Chandler with plugins.
- Add enough email features to pass the threshold where Chandler is
usable as an email client. One of our bigger decisions is whether or not
we can effectively tackle this given our resources. (In particular
engineering and QA resources are the constraints I'm worried about, not
design -- though it would require design work as well).
- Add scheduling features. Ultimately this will be something of a
"business" decision: would we grow the user base by adding scheduling
features? Does this make sense with the resources we have? I don't think
we're going to tackle this without a motivating reason (like a good shot
at new users and possibly funding), but it needs to be on the table.
- Plugin support. I actually think the plugin support we already have is
pretty good. We don't need gold plating here -- we need to try and
remove obstacles people find when writing plugins. The gambit is that
the ability to write plugins will be a foundation for building a
developer community.
- Internationalization/localization. Again, our support here is pretty
good already for the stage we are in. We should do what we can to
support localizers, fixing problems that block them.
- Sync with mobile devices. (In particular, platform based sync).
- Sync across multiple desktops (and its close cousin, hub ui features
supporting the remote access case). Some users already do this, but the
design is not optimized for this case.
Feature decisions (for both desktop and server) is where I am going to
put much of my energy and attention. I don't know yet what the priority
should be for the desktop feature possibilities -- we want user feedback
to help inform that. I'll start a separate thread for 0.7.1 planning,
how to get started in the face of some uncertainty. The list above does
sort of map out a good part of the space of possibilities, and gives
context for architecture decisions.
*Architecture decisions*
One of the elephants in the room is the desktop architecture and its
current limitations that might interfere with our goals. I believe these
are the most pressing problems with the architecture:
- Testability (the architecture plays a role in the difficulty of
writing unit and functional tests)
- Performance, broadly speaking. We made good progress at the end of
0.7, and performance is pretty good for day to day Preview usage on
adequate hardware. Still, we have some lingering problems:
- large memory usage for the amount of data stored
- performance, stability, scalability for long lived chandlers
- performance of larger (more complex) data sets on less capable
configurations
- support for full email (can we handle users' full set of IMAP
folders?)
- API complexity, brittle architecture leading to buggy application code
- defered delete is an unusual/unexpected paradigm for python coders
- complex collection logic with unenforced limitations leads to
problems with indexes (and other bugs)
- complexity of notification types makes it hard to write the detail
view (and other views) so that it updates correctly
- Easy for plugin developer (or any application level developer) to
cause data corruption. UI level code shouldn't be able to mess up the
integrity of the data.
- Hard for plugin developer to understand application structure. Partly
a documentation issue, but partly an issue with complexity of APIs for
collections, notifications, etc.
It is perhaps worth stepping back and acknowledging that the goals for
the architecture have changed somewhat from the beginning of the
project. It is going to be important to let go of some of those goals,
accept sunk costs, and make forward looking decisions that will support
the existing feature set and new features mentioned above. Given that we
started writing this code with over ambitious, often rather fuzzy goals
it is perhaps not surprising that we can see with hindsight many
problems with the existing code. The good news is that our picture of
what we want (from an end user feature perspective) is now much more
clear than it was 5 years ago.
What goals or assumptions have changed? I'll call out a few:
- Everything should be an Item. Well, maybe not.
We originally wanted *everything* to be an Item: ui elements, schema,
etc. Among other reasons, there was this idea that you'd be able to edit
anything from a general UI. We've dropped this in a big way. We have no
"general editor" and I don't think we ever will.
- We've often imagined a very fast, very scalable repository in thinking
about how the repository should be used by the application in an ideal
world. Realistically, we need to design the application given repository
constraints.
Fallout?
- Perhaps we shouldn't be storing UI items in the repository. Yes,
transparently persisting UI state is in theory a cool property, but in
practice I don't know that we are effectively getting the persistence
right and it perhaps comes at too great a cost.
- Can we do better at separating out the domain schema from UI and lower
level details?
Again, I'm sure I'm missing others (both assumptions and possible
changes enabled by challenging old assumptions). My intention here is to
start the conversation.
I've had individual or small group conversations with several of you
about the problems and challenges with the architecture, and options we
have for addressing them -- in particular with Phillip Eby and Grant,
and separately with Andi. I'm sure others of you have had side
conversations as well, especially as we were doing performance work.
Phillip Eby and Grant have been exploring some options (at my request),
I'll let them introduce that work. I'm looking to Grant to look at the
big picture of the whole architecture and to drive some proposals for
making architecture changes. (Proposals from others welcome of course).
*Decisions*
Can we pull off architecture changes, better testing and implement all
of those features? Clearly we have to make some tough choices about what
is feasible. I don't have answers at this point. We need concrete
proposals. We need to clarify requirements. We need to narrow feature
decisions.
How will we make decisions? I'm going to come back to the point made at
the beginning of this message -- we want to build a user base and a
community. To build a user base we need to fix bugs and features where
we see user energy. To build the community we need to remove obstacles
to developer participation. To keep users we need to maintain
application stability. To add features maintaining application stability
we need better regression testing. And so on.
Time and resource constraints are the other forcing function of course.
Proposals for projects that span more than 6 months might not be very
helpful at this point.
I know this message has been long (thank you for reading this far!) and
the full set of work spelled out here is daunting. Now that we have
Preview (almost) in hand, we're in a really good position to iterate and
start getting users. Lets make some clear eyed decisions from here.
Cheers,
Katie
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "chandler-dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/chandler-dev
