On Fri, 2012-10-26 at 21:04 +0200, Patrick Ohly wrote:
> On Mon, 2012-09-17 at 11:32 +0200, Patrick Ohly wrote:
> > On Wed, 2012-09-12 at 14:51 +0200, Patrick Ohly wrote:
> > > I have some partial implementation of the API ready and will push it
> > > shortly to the new SyncEvolution git repo on freedesktop.org
> >
> > The code is now in the for-master/pim branch, see the
> > src/dbus/server/manager directory. The API text is in
> > src/dbus/server/pim/pim-manager-api.txt. I decided to avoid using IVI in
> > the code because it might turn out to be useful elsewhere.
>
> The code now fully implements the proposed API and has been tested by me
> for a while. The nightly testing includes the testpim.py tests. Manual
> testing with a Nokia N97 Mini also passed (with the caveats below). This
> is a good time to make it available to people who do not want to compile
> source from a git repo.
>
> Therefore I have merged the code into the master branch, tagged it as
> 1.3.99.1 and prepared the usual source .tar.gz:
> http://downloads.syncevolution.org/syncevolution/sources/syncevolution-1.3.99.1.tar.gz
>
> The benefits for traditional users are fairly small (except perhaps for
> the new caching mode) and the binaries would not contain the IVI
> features anyway, because the binaries target EDS < 3.6 and older
> distros. For these reasons I am not publishing binaries of 1.3.99.1.
>
> There are a few known issues regarding the PIM Manager in this release:
>
> - updating a contact is inefficient and can cause deadlocks in EDS:
> http://bugs.freedesktop.org/show_bug.cgi?id=55925
>
> - some minor memory leaks:
> http://bugs.freedesktop.org/show_bug.cgi?id=56396
>
> - in violation of the API spec, colon and upper case characters in the
> peer UID are not properly supported and should be avoided for now; I'm
> undecided whether I should fix the API to match the implementation or
> make the implementation more complicated to support the proposed API:
> http://bugs.freedesktop.org/show_bug.cgi?id=56436
All of these issues are resolved in SyncEvolution 1.3.99.2, which is
available from git and syncevolution.org now.
> A full overview of issues is here:
> https://bugs.freedesktop.org/showdependencygraph.cgi?id=55916&display=web&rankdir=TB
>
> I'm attaching the current API revision, a README which gives
> SyncEvolution specific information and two example scripts. All of this
> is also in the release and in the git repo.
Again I'm attaching the new API and SyncEvolution README for its
implementation. Instead of the examples (which you can get from git or
the source archive) I'm attaching the NEWS entry for 1.3.99.2.
> Regarding documentation, I'm still undecided whether specific peer
> config options, search capabilities and ordering really should be
> specified in the API documentation. I'm inclined to make this completely
> implementation specific and thus at best give examples in the API
> documentation. App developers need to consult their system documentation
> anyway to learn which version of the API is supported by the system.
As hinted above, I've restructured the documentation so that the
abstract API is covered by one document and the implementation by
another.
--
Best Regards
Patrick Ohly
Senior Software Engineer
Intel GmbH
Open Source Technology Center
Pützstr. 5 Phone: +49-228-2493652
53129 Bonn
Germany
SyncEvolution 1.3.99.2, xxxxxxxxxx
==================================
Another development snapshot. Includes all fixes that went into 1.3.2
and several improvements to the PIM Manager. Documentation was updated
and extended considerably. The pim-manager-api.txt now describes the
abstract API while src/dbus/server/pim/README explains how
SyncEvolution implements it.
Details:
* PIM Manager searches for a caller ID ('phone' search) in EDS
directly while folks still starts up. No unification is done of
these results. Intermediate results are replaced by the final ones
from folks once those are ready.
* PIM Manager: allow configuration of session directories (part of FDO #55921)
Useful for moving the session directories to a temporary file system.
They are essentially just useful for debugging when used as part of
PIM Manager.
- "logdir" - a directory in which directories are created with
debug information about sync session
- "maxsessions" - number of sessions that are allowed to exist
after a sync (>= 0): 0 is special and means unlimited,
1 for just the latest, etc.;
old sessions are pruned heuristically (for example,
keep sessions where something changed instead of
some where nothing changed), so there is no hard
guarantee that the last n sessions are present.
* PIM Manager: write less data to disk (part of FDO #55921)
Avoid writing config file changes to disk by enabling a new
"ephemeral" mode for syncing via the PIM Manager. In this mode,
config file changes are not flushed resp. discarded directly.
This prevents writing to .ini files in ~/.config.
The "synthesis" binfile client files are still written, but they get
redirected into the session directory, which can (and should) be set
to a temp file system and get deleted again quickly.
Data dumps are turned off now in the configs created by the PIM
Manager.
* syncevo-dbus-server: use syslog instead of standard output by default
* syncevo-dbus-server: command line options for controlling
output and startup
-d, --duration=seconds/'unlimited' Shut down automatically
when idle for this duration (default
300 seconds)
-v, --verbosity=level Choose amount of output, 0 = no output,
1 = errors, 2 = info, 3 = debug;
default is 1.
-o, --stdout Enable printing to stdout (result of
operations)
and stderr (errors/info/debug).
-s, --no-syslog Disable printing to syslog.
-p, --start-pim Activate the PIM Manager (= unified
address book)
immediately.
* PIM Manager: store set of active address books persistently (FDO #56334)
Together with storing the sort order persistently, this allows
restarting the daemon and have it create the same unified address book
again.
* PIM Manager: remove colon from valid peer UID character set (FDO #56436)
Using the UID as part of file names gets more problematic when
allowing colons. Remove that character from the API and enforce
the format in the source code.
* PIM Manager API: introduce contact ID and use it for reading
This makes it easier for a client to fully polulate its view with
contact data. Previously it could happen that due to concurrent
changes in the server, a client was returned data for the same
contact multiple times. A client had to detect that and re-issue
read requests.
* PIM Manager API: optional ViewAgent.Quiescent() (FDO #56428)
The callback is guaranteed to be invoked once when a search has
finished sending its initial results, and not sooner. This makes it
possible to check whether the current data contains some contact or
not.
* PIM Manager: limit number of search results (FDO #56142)
A 'limit' search term with a number as parameter (formatted as string)
can be added to a 'phone' or 'any-contains' search term to truncate the
search results after a certain number of contacts. Example:
Search([['any-contains', 'Joe'], ['limit', '10']])
=> return the first 10 Joes.
As with any other search, the resulting view will be updated if
contact data changes.
The limit must not be changed in a RefineSearch(). A 'limit' term may
(but doesn't have to) be given. If it is given, its value must match
the value set when creating the search. This limitation simplifies the
implementation and its testing. The limitation could be removed if
there is sufficient demand.
* PIM Manager: fix refining a search
Due to not mapping the local index in the view to the parent's index,
refining only worked in views where parent and child had the same
index for the contacts in the search view.
* PIM Manager: fix starting when done via search
When the unified address book (= FullView) was not running yet at the
time when a client wanted to search it, the unified address book was
not started and thus the search never returned results.
* PIM Manager: fix writing contact, support photo and notes
folks and EDS do not support writing properties in parallel
(https://bugzilla.gnome.org/show_bug.cgi?id=652659). Must serialize
setting of modified properties.
* PIM Manager: fix incorrect contact removal signals in filtered view
The filtered view did not check whether a parent's removed contact was
really part of the view before sending a removal signal for it.
* D-Bus: missing out parameters in D-Bus introspection XML (FDO #57292)
The problem was in the C++ D-Bus binding. If the method that gets bound
to D-Bus returns a value, that value was ignored in the signature:
int foo() => no out parameter
It works when the method was declared as having a retval:
void foo (int &result) => integer out parameter
This problem existed for both the libdbus and the GIO D-Bus
bindings. In SyncEvolution it affected methods like GetVersions().
* PIM Manager performance: pre-compute normalized telephone numbers
Looking up by phone number spends most of its cycles in normalizing of
the phone numbers in the unified address book. Instead of doing that
work over and over again during the search, do it once while loading.
Looking up a phone number only once does not gain from this change, it
even gets slower (more memory intensive, less cache locality). Only
searching multiple times becomes faster.
Ultimately it would be best to store the normalized strings together
with the telephone number inside EDS when the contact gets
created. Work on that is in progress.
* PIM Manager: improve performance of FullView sorting
This fixes the hotspot during populating the FullView content: moving
contacts around required copying IndividualData and thus copying
complex C++ structs and strings. Storing pointers and moving those
avoids that, with no lack of convenience thanks to boost::ptr_vector.
Reordering also becomes faster, because the intermediate copy only
needs to be of the pointers instead of the full content.
* PIM Manager example: add benchmarking
The new "checkpoints" split up the whole script run into pieces which
are timed separately, with duration printed to stdout. In addition,
tools like "perf" can be started for the duration of one phase.
* EDS: fix creating databases
--create-database was broken in combination with the final code in EDS
3.6 because it passed NULL for the UID to e_source_new_with_uid(),
which is considered an error by the implementation of that
method. Must use e_source_new() if we don't have a UID.
* fixed some memory leaks, extended tests to cover new features and bugs
Preamble
========
This text describes a D-Bus API. The API implements in-vehicle infotainment
(IVI)
use cases around contacts:
- cache address books from peers (primarily phones connected via Bluetooth)
in local address books
- provide a unified address book that combines a configurable (and changing)
subset of the local address books
- fast phone number lookup
- browsing and searching in the unified address book
Tasks that are expected to be done by the user of this API:
- identify peers and their capabilities
- decide how and when peer data should be cached
- define which data goes into the unified address book
In other words, the API provides the mechanisms and the user the
policy.
Several aspects of this API may differ depending on the
implementation. For example, searching for contacts and syncing with
peers are not described in the API. Consult the documentation of the
API implementation to learn what it supports. For SyncEvolution, that
documentation is the src/dbus/server/pim/README file.
Datatypes
=========
Peers
-----
A peer is an entity which has exactly one address book that is meant
to be cached locally. Typically a peer is a phone connected via
Bluetooth and accessed via PBAP, but it could also be a web service
that supports CardDAV or a phone with SyncML support.
Peers are identified by a unique string ID. That ID needs to be
assigned by the user of this API. The string must not be empty and may
only contain characters a-z, 0-9 and hyphen. No other assumptions
about its content are made. For example, the phone's Bluetooth MAC
address could be used after removing or replacing the colon and using
lower case hex characters.
For an entity that has more than one address book, multiple peers must
be configured.
For each peer, enough information must be provided to access its
address book. That information is passed via D-Bus as a
string-to-string dict, with keys and their values being defined by the
implementation.
Address books
-------------
Address books which mirror data from a specific peer use the string
"peer-<uid>" as ID, where <uid> is the unique ID of that peer.
In addition, there is a system address book which is independent of
any particular phone. Its ID is the empty string.
This naming scheme can be extended later on, to support other kinds
of address books.
Contact
-------
A single contact is transferred via D-Bus as a string->variant dict
where the keys are predefined property names and the values represent
simple values (a string for "full-name") or more complex structures
(list of phone numbers for "phone-numbers", with each list entry
itself being a combination of type flags and the actual value).
[comment: this mirrors the properties of a libfolks Individual:
http://telepathy.freedesktop.org/doc/folks/c/FolksIndividual.html]
Some properties of a FolksIndividual only make sense locally and are
not transmitted, for example the personas it is derived from.
Some other properties provide information not found that way in
FolksIndividual:
- "source" = list of string pairs, where each pair is a combination
of address book ID and local contact ID inside that address book
(not necessarily the same as the vCard UID of a contact!)
- "id" = an opaque string which identifies the contact while it
exists inside any PIM Manager view. See ContactsAdded and ReadContacts.
Property values which are large (like photos) are not sent via
D-Bus. Instead a link to a local file is sent.
For a full definition of contact properties see the implementation
documentation.
Search results
--------------
The goal is to support a UI which:
- displays an ordered list of the search result,
- can show the initial results with minimal delay,
- can load actual content for the display as needed (only
load the parts which are visible or will be visible soon).
The content of the unified address book can change at any time. The
API design takes that into account by using a model/view/controller
model.
The model is the complete list of contacts, sorted according to the
currently configured sort order. Sorting is part of the model to
simplify generating views.
The view is the subset of the data that a user of the API has
requested. In the most extreme case, all contacts are part of the
view. Therefore contact data has to be requested explicitly. Contacts
are numbered 0 to n-1 in each view, where n is the number of contacts
in the view. Sort order is the same as in the underlying model. Change
notifications with these index numbers are sent as contacts are added,
modified or removed.
The controller is the part of the API which allows changing contacts
in the system address book, changing the sort order, enabling or disabling
address books, etc.
Note that removing or adding a contact changes the numbers assigned to
other contacts. Example:
- A view containing 10 contacts is created.
- A notification about "contacts #0 to #9 added" is sent
(given as pair of first index and count, not list of numbers).
- The five contacts starting with #5 to are read via their ID.
- Contact #4 gets removed. The user needs to remember that
the data that it has now corresponds to contacts #4 to #8.
- Contact #5 gets added, before the contact which had that
number before. The user now has contacts #4 and #6 to #9.
It should request contact #5 if (or once) it is needed to
provide a complete list to the user.
[comment: using a view could be simplified by including contact data
in the change notifications. This is not planned at the moment because
it would not work well for large views. When adding it, there should
be an API to restrict which properties of a contact get sent.]
Error handling
==============
D-Bus error messages are not localized. They are meant for debugging,
not for displaying to the user. In cases where the caller may be able
to do something about an error, specific error codes are defined as
part of the API. However, typically errors are generic and the caller
simply has to assume that the PIM storage is currently unusable.
Unless noted otherwise, calls return when the requested operation is
complete.
The following errors are defined. In addition to the D-Bus name of the
error they provide a textual error description.
org._01.pim.contacts.Manager.Aborted
Some operation was intentionally aborted instead of letting it
complete. Typically not an error.
org._01.pim.contacts.Manager.BadStatus
A generic error report. The error description is a string
which gives further information for debugging.
API
===
PIM Manager
-----------
The PIM manager is used to hold the unified address book in memory,
create views on it, change configuration and control data transfers
from phones.
Service: org._01.pim.contacts
Interface: org._01.pim.contacts.Manager
Object path: /org/01/pim/contacts
Methods:
void Start()
The PIM manager does not start loading contact data right
away. That allows setting the options like sort order first
and/or delaying the loading until it is needed. After
Start(), changing options that affect the unified address
book will take effect immediately.
Calling Start() is optional, any method asking for data will
automatically do that.
void Stop()
Explicitly tells the PIM manager to discard the unified address
book and free up the memory if possible (= not currently in use).
Primarily useful for testing.
void SetSortOrder(string mode)
"mode" must be one of the values supported by the implementation.
string GetSortOrder()
Returns the current sort order.
list of strings GetActiveAddressBooks()
Returns the IDs of the address books which currently
contribute to the unified address book.
void SetActiveAddressBooks(list of strings)
Sets the address books which contribute to the unified
address book.
void SetPeer(string uid, dict properties)
Adds or modifies a peer. Modifying a peer does *not* affect
any contact data which might be cached for it.
void RemovePeer(string uid)
Removes a peer and all its cached data. If that data was
part of the active address books, it will be removed
automatically.
void SyncPeer(string uid)
Retrieve contacts from the peer and ensure that the local
cache is identical to the address book of the peer. The call
returns once the operation is complete. Only if there was no
error can the caller assume that the cache is up-to-date.
Otherwise it is in an undefined state.
void StopSync(string uid)
Stop any running sync for the given peer. The SyncPeer() method
which started such a sync will return with an "aborted" error
once the sync was stopped.
dict of UID to string key/value dict GetAllPeers()
Returns information about all currently configured peers.
object Search(list filter, object agent)
Creates a new view which contains all contacts matching the
filter. The call returns the object path of a view object
after validating parameters and starting the result
gathering, and before completing the search. The view object
can be used to control the view via the
org._01.pim.contacts.ViewControl interface.
The content of the filter is defined by the implementation.
Notifications for the view are sent back to the caller by
invoking methods from the org._01.pim.contacts.ViewAgent
interface on the object whose path is given in the "view"
parameter. If any of these method calls fail, the view will
automatically be destroyed.
In other words, the caller first needs to get ready to process
results by registering an object on the bus before calling
Search().
[comment: this allows sending results to just one recipient,
something that cannot be done easily with the use of signals as in,
for example, obexd. In obexd, the initiator of a transfer
has to subscribe to org.bluez.obex.Transfer on the object path
returned to it when starting the transfer, then check the current
status before waiting for signals, because the "Completed" signal
might have been sent before it could register for it.]
string AddContact(string addressbook, dict contact)
Adds a new contact to the given address book. Typically
only the system address book is writable. Contact properties
which are unknown or cannot be stored are silently ignored.
Returns the local ID of the new contact in the address book.
Photo data that is sent inline in the dict will be split out
into a file that gets associated with the contact. A photo
file that gets linked will continue to be owned by the
caller; the contact storage may or may not make a copy of it,
depending on which storage is used.
void ModifyContact(string addressbook, string localid, dict contact)
Updates an existing contact.
void RemoveContact(string addressbook, string localid)
Remove the contact and all of its associated data (like the
photo, if the photo file is owned by the contact storage).
Service: org._01.pim.contacts
Interface: org._01.pim.contacts.ViewControl
Object path: [variable prefix]/{view0,view1,....}
Methods:
list of (int index, contact dicts) pairs ReadContacts(array ids)
Requests the data of the contacts idenfified via their IDs.
Only the data of contacts that are still part of the view
can be returned.
The returned list contains the current index of the
requested contact plus its data. -1 and an empty
dictionary are returned for contacts which can no longer
be read, for example because they were removed from the
view in the meantime or because the ID was simply
invalid.
Note that the caller must process the call response after
all events via the ViewAgent interface. Otherwise the
index numbers are potentially out of sync and thus
unreliable. Doing this call asynchronously and dealing
with the response as part of the main event loop will do
the right thing automatically, because D-Bus guarantees
ordering of messages.
Making this explicit by returning data via another
org._01.pim.contacts.ViewAgent method was considered and
rejected a) for the sake of keeping this API simple and
b) to allow simple synchronous calls where it makes sense
(testing, for example).
void Close()
Closes the view and all resources associated with it.
Pending ReadContacts() calls will return without any
data and no error.
void RefineSearch(list filter)
Replaces the current filter of the view with a new one.
The new filter must be stricter than the old one. Contacts
which were already filtered out will not be added back
to the view when setting a less restrictive filter (simplifies
the implementation and improves performance).
Service: [user of the PIM Manager]
Interface: org._01.pim.contacts.ViewAgent
Object path: [as chosen by user of PIM Manager]
Methods:
void ContactsModified(object view, int start, array ids)
Contacts #start till #start + count (inclusive) have
changed. Data that the recipient of the call might have
cached became invalid and should be reloaded.
It is possible that a contact gets replaced by another
with a single "contact modified" signal. In other words,
the ID at each position may change and thus the IDs
are sent as part of the signal.
In cases where a contact changes its position in the
view, both a combination of "contact removed" + "contact
added" (single contact changes) as well as several
"contact modified" signals are possible (contacts swap
position, for example when reordering).
In the later case, a contact will temporarily appear
at two different positions.
void ContactsAdded(object view, int start, array ids)
New contacts were added to the view. The number
of new contacts is given via the size of the ids array.
The ID of each new contact is guaranteed to be the same
in all views. IDs may get reused after their contact got
removed from the last view it was contained in. In
particular there is no guarantee that it is persistent
across restarts of the PIM manager.
The contact which previously had index #start now
has index #start + count, etc.
void ContactsRemoved(object view, int start, array ids)
Some contacts were removed from the view.
The contact which previous had index #start + count
(if there was one) now has index #start, etc.
void Quiescent(object view)
The current content of the view is complete. No further
updates are expected until something changes again
(underlying data, ordering, active address books,
filter).
Changing data (directly or via syncing) can trigger
multiple "Quiescent" signals, depending on when these
changes are reported by the underlying storage.
Changing multiple settings will trigger one "Quiescent"
per change.
Implementing the Quiescent() method in a ViewAgent
is optional.
Overview
========
The code in this directory only gets compiled when SyncEvolution is
configured with --enable-dbus-service-pim. It uses libfolks and the
PBAP backend to implement a unified address book. This additional
functionality is provided via the org._01.pim D-Bus API.
That API is independent of libfolks and SyncEvolution. That it is
implemented inside syncevo-dbus-server merely simplifies the
implementation, by reusing some code provided by SyncEvolution
and the core Server class:
* C++ D-Bus bindings
* logging
* D-Bus server life cycle control: delay shut down while
clients make calls, inhibit shutdown while clients are
registered, restart when files change on disk during system
update, ...
* direct access to SyncEvolution instead having to go through
the http://api.syncevolution.org D-Bus API
Compilation
===========
Use --enable-dbus-service-pim --enable-pbap --enable-ebook.
--enable-ebook is already the default at the moment. It must not be
disabled.
The PBAP backend can be disabled. However, then fetching data from
phones via PBAP obviously does not work.
More information
================
Public discussion started here, comments can be sent via the gmane
"reply" feature:
http://comments.gmane.org/gmane.comp.mobile.syncevolution/4009
Issues specific to PIM Manager can be found in this graph:
https://bugs.freedesktop.org/showdependencygraph.cgi?id=55916&display=web&rankdir=TB
Dependencies
============
A fairly recent libfolks > 0.7.4 is required. The 837a88 commit
is required and several other pending changes are recommended:
https://bugzilla.gnome.org/show_bug.cgi?id=686693
writing birthday lacks conversion from UTC
https://bugzilla.gnome.org/show_bug.cgi?id=685401
linking by email
https://bugzilla.gnome.org/show_bug.cgi?id=686695
support nickname in add_persona_from_details
Other requirements:
* Evolution Data Server >= 3.6
* boost::locale and libphonenumber (when using the default
sorting and searching)
* glib >= 2.30
* Python >= 2.7 (only for testing)
Known issues
============
None at the moment.
Extending the implementation
============================
Sorting and searching can be replaced with a different implementation
at compile time via --enable-dbus-service-pim[=<locale>]: it uses the
file src/dbus/server/pim/locale-factory-<locale>.cpp to implement
sorting and searching.
Any additional library dependencies for that file need to be added to
the main configure.ac.
The default implementation is based on boost::locale and
libphonenumber and is described below.
Sorting
=======
The sort order can be "first/last", "last/first", "full name".
"first/last" sorts based on the first name stored in the "name"
property, with the last name used to break ties between equal
first names.
"last/first" reverts that comparison.
"full name" sorts based on the full name chosen for the contact if
there is such a string, otherwise it uses the concatenation of the
individual name componts without prefix (= "[<first name>]
[<middle name>] [<last name>] [<suffix>])" as fallback. In other
words, it sorts correctly when either all contacts have a full
name explicitly set or the full names that were set following the
same pattern as the fallback.
Sorting is case-insensitive. The default is "last/first" if not set
earlier.
Searching
=========
Supported searches:
[ ] - An empty list matches all contacts.
[ ['phone', '<number>'] ] - Look up a valid phone number (= "caller ID").
The country code for the current locale is added if no country
code was given in the number. Phone numbers in the unified address
book must start with the resulting full number, after being normalized
the same way.
In other words:
- Formatting does not matter.
- Alpha characters are aliases for numbers on the keypad and match
their corresponding number.
- Additional digits in the address book are ignored, only
the prefix must match (extensions may or may not be included in
<number>).
- Phone numbers in the address book which cannot be normalized
cannot be matched.
[ ['any-contains', '<text>', <flags>] ] - Sub-string search for <text>
in the following contact values: first, middle or last name,
formatted name, nick name, phone number (sub-string search which ignores
formatting and treats alpha characters as aliases, in contrast
to prefix match in 'phone') or email address. Optional flags include:
'case-insensitive' (the default), 'case-sensitive'.
Using a list will allow extending the search capabilities later,
for example by allowing multiple terms which all must match
and/or adding recursive queries like this:
['and',
['or', ['any-contains', 'joe'], ['any-contains', 'joan']],
['phone', '1234']
]
Lookup and search are different: the former is based on a valid
number, the later on user input.
A 'phone' lookup can compare normalized numbers including the country
code, to ensure that the lookup is exact and does not mismatch numbers
from different countries. Heuristics like suffix matching do not do
this correctly in all cases.
An 'any-contains' search is based on user input, which might contain
just some digits in the middle of the phone number. The search ignores
formatting in both input and address book.
In both cases, alpha characters are treated as aliases for their
corresponding digit on a keypad when matching phone numbers.
In addition, an empty string as sort order picks a simple, ASCII-based
"last/first" sorting. This is used for testing.
A 'limit' search term with a number as parameter (formatted as string)
can be added to a 'phone' or 'any-contains' search term to truncate the
search results after a certain number of contacts. Example:
Search([['any-contains', 'Joe'], ['limit', '10']])
=> return the first 10 Joes.
As with any other search, the resulting view will be updated if
contact data changes.
The limit must not be changed in a RefineSearch(). A 'limit' term may
(but doesn't have to) be given. If it is given, its value must match
the value set when creating the search. This limitation simplifies the
implementation and its testing. The limitation could be removed if
there is sufficient demand.
Phone number lookup
===================
A "phone" search must return results quickly (<30ms with 10000
contacts) under all circumstances, including the period of time where
the unified address book is still getting assembled in memory. To
achieve this, SyncEvolution searches directly in the active address
books and presents these results until the ones from the unified
address book are ready.
A quiescence signal will be sent when:
1. results from EDS are complete before the ones from the unified
address book
2. results from the unified address book are complete.
The first signal will be skipped and the EDS results discarded if EDS
turns out to be slower than the unified address book.
Results from different EDS address books are not unified, for the sake
of simplicity. They get sorted according to the sort order that was
active when starting the search. Changing the sort order while the
search runs will only affect the final results from the unified
address book.
Refining such a search is not supported because refining a phone
number lookup is not useful.
Peers
=====
The following keys are supported for the configuration of a peer:
- "protocol" - defines how to access the address book. "PBAB" (phone
book access protocol) and "file" (read vCard files from
directory) are implemented. "SyncML" and "CardDAV"
could be added easily.
- "transport" - defines how to establish the connection. The only
supported value is "Bluetooth" (for protocol=PBAP or
SyncML), which is also the default if not given
explicitly.
- "address" - the Bluetooth MAC address in the aa:bb:cc:dd:ee:ff
format (for transport=Bluetooth).
- "database" - empty or unset for the internal address book
(protocol=PBAP), or the path to the directory
(protocol=file).
- "logdir" - a directory in which directories are created with debug
information about sync session.
- "maxsessions" - number of sessions that are allowed to exist
after a sync (>= 0): 0 is special and means unlimited,
1 for just the latest, etc.;
old sessions are pruned heuristically (for example,
keep sessions where something changed instead of
some where nothing changed), so there is no hard
guarantee that the last n sessions are present.
Not supported via the API at the moment:
- selecting a specific phone address book
- selecting which vCard properties get cached
Contact Data
============
A contact dictionary has the following key/value pairs. More properties
may be added later.
contact dictionary:
"id" - string, see API description
"source" - string, see API description
"full-name" - string, formatted by the user or automatically (vCard FN)
"nickname" - string
"structured-name" - name dictionary (vCard N)
"photo" - string, the URL (usually of a local file; EDS strips all inline photo
data in vCards and puts them into local files, leading to URLs like:
file:///home/user/.local/share/evolution/addressbook/system/photos/pas_id_5012983E0000065A_photo-file0.image%2Fjpeg)
"birthday" - birthday tuple
"emails" - value list with strings as value
"phones" - value list with strings as value
"urls" - value list with strings as value
"notes" - list of strings; in practice only one entry is supported
"addresses" - value list with an address dictionary as value
"roles" - list of role dictionaries
name dictionary:
"family" - string, the last name
"given" - string, the fist name
"additional" - string, middle names
"prefixes" - string, name prefix like "Mr."
"suffixes" - string, name suffix like "Sr."
birthday tuple:
(yy, mm, dd) - integer values for year, month, and day
role dictionary:
"organisation" - main organization ("Foo ACME")
"title" - title inside that organization ("vice president")
"role" - role as part of that origanization ("adviser")
value list:
[ (value, [parameter, ...]), (value, ...) ]
value - depends on the property
parameter - a string, with values again depending on the property;
a value may have zero to n different parameters
phone parameters:
"voice"
"fax"
"car"
"cell"
"pager"
"pref"
"home"
"work"
"other" (might not be set explicitly)
address parameters:
"home"
"work"
"other" (might not be set explicitly)
url parameters:
"x-home-page"
"x-blog"
"x-free-busy" - public calendar free/busy URL
"x-video" - video chat URL
address dictionary:
"po-box" - string, post office box
"extension" - string, address extension
"street" - string, street name
"locality" - string, city name
"region" - string, area name
"postal-code" - string
"country" - string
Note: all of the strings and values above are defined by SyncEvolution
in individual-traits.cpp, except for parameter strings. Those come
directly from folks, more specifically AbstractFieldDetails:
http://telepathy.freedesktop.org/doc/folks/vala/Folks.AbstractFieldDetails.html
Here is an example contact dictionary, using Python syntax:
{
'full-name': 'John Doe',
'nickname': 'Johnny',
'structured-name': {'given': 'John', 'family': 'Doe'},
'birthday': (2006, 1, 8),
'photo': 'file:///home/user/file.png',
'roles': [
{
'organisation': 'Test Inc.',
'role': 'professional test case',
'title': 'Senior Tester',
},
],
'source': [
('test-dbus-foo', luids[0])
],
'id': 'xyz',
'notes': [
'This is a test case which uses almost all Evolution fields.',
],
'emails': [
('[email protected]', ['home']),
('[email protected]', ['other']),
('[email protected]', ['work']),
('[email protected]', ['other']),
],
'phones': [
('business 1', ['voice', 'work']),
('businessfax 4', ['fax', 'work']),
('car 7', ['car']),
('home 2', ['home', 'voice']),
('homefax 5', ['fax', 'home']),
('mobile 3', ['cell']),
('pager 6', ['pager']),
('primary 8', ['pref']),
],
'addresses': [
({'country': 'New Testonia',
'locality': 'Test Megacity',
'po-box': 'Test Box #3',
'postal-code': '12347',
'region': 'Test County',
'street': 'Test Drive 3'},
[]),
({'country': 'Old Testovia',
'locality': 'Test Town',
'po-box': 'Test Box #2',
'postal-code': '12346',
'region': 'Upper Test County',
'street': 'Test Drive 2'},
['work']),
({'country': 'Testovia',
'locality': 'Test Village',
'po-box': 'Test Box #1',
'postal-code': '12345',
'region': 'Lower Test County',
'street': 'Test Drive 1'},
['home']),
],
'urls': [
('chat', ['x-video']),
('free/busy', ['x-free-busy']),
('http://john.doe.com', ['x-home-page']),
('web log', ['x-blog']),
],
}
Configuration and data handling
===============================
The configuration of peers is mapped to normal SyncEvolution
configurations. Every peer gets its own context, which contains both
the local source definition for the peer and sync plus target configs.
The special name space prefix "pim-manager-" used to identify them and
avoid potential conflicts with normal SyncEvolution configurations.
Look in ~/.config/syncevolution/pim-manager-* to find the
configuration files.
Local EDS databases are created with a fixed UID and name that also
have that prefix. One could go one step further than it is currently
done and set these databases to "disabled" in the ESourceRegistry,
which would hide them in Evolution.
The SyncEvolution command line can be used to view or manipulate
these databases:
https://syncevolution.org/wiki/item-operations
The sort order and set of active address books are stored persistently
in ~/.config/syncevolution/pim-manager.ini as:
- "sort" = same value as in the API
- "active" = space, comma or tab separated EDS source UUIDs
Usage
=====
The PIM manager is part of the syncevo-dbus-server. It can be started
automatically via D-Bus or explicitly. When started automatically, it
logs error messages to syslog.
The PIM manager starts as idle and needs to be started via the D-Bus
Start() API before it begins assembling the unified address book.
SyncEvolution installs an XDG autostart file
(/etc/xdg/autostart/syncevo-dbus-server.desktop) which, on systems
supporting that mechanism, ensures that syncevo-dbus-server is started
once. This is used for automatic syncing, which is not active when
using only the PIM Manager. Therefore syncevo-dbus-server will
terminate again after some idle period.
It's recommended to disable this mechanism or patch
syncevo-dbus-server.desktop to start the server with different
options. In particular the "--start-pim" option may be useful to
start up the UI and the server in parallel.
Here is a list of supported options:
Help Options:
-h, --help Show help options
Application Options:
-d, --duration=seconds/'unlimited' Shut down automatically
when idle for this duration (default
300 seconds)
-v, --verbosity=level Choose amount of output, 0 = no output,
1 = errors, 2 = info, 3 = debug;
default is 1.
-o, --stdout Enable printing to stdout (result of
operations)
and stderr (errors/info/debug).
-s, --no-syslog Disable printing to syslog.
-p, --start-pim Activate the PIM Manager (= unified
address book)
immediately.
Limitations
===========
There are no hard-coded limits. In practice the number of contacts,
parallel searches, peers, etc. is limited by CPU performance, amount
of RAM and disk space.
Functionality
=============
Caching
-------
Caching of contacts is done roughly like this:
1. The PBAP backend reads all contact data. See src/backends/pbap.
2. As part of a local sync (see main SyncEvolution README), a local-cache-slow
sync is done. In that mode all local data is compared against the incoming
data. Matches are found according to the properties that are defined
as compare="always" or compare="slowsync" in
src/syncevo/configs/datatypes/00vcard-fieldlist.xml.
In the default file, these are the first, middle and last name and the
organization.
3. Local entries which have a match are updated with data from that match,
unmatched local entries are removed and unmatched incoming ones are
created.
This means that changing "less important" properties will be turned
into local updates. This has the desirable effect of not changing the
local ID of the existing contact, which might become important when
attaching local meta data to that contact via its local ID (not
supported at the moment; folks itself has a favorite flag which is
stored like that). It also reduces disk IO.
If matching fails, a contact will get deleted and recreated. The end result
in the unified address book is still the same, because folks does not
rely on the ID for linking.
Supported fields
----------------
The PBAP backend always downloads the entire vCard. It supports restricting
the vCard to specific properties, see src/backend/pbap/README. However, this
is not used by the PIM Manager.
In particular, PHOTO data is always included in the download right
away. A mode where caching only works without PHOTO in the initial
pass and then adds that in a second step could be added.
Internally a vCard is represented as a Synthesis field list (again,
see 00vcard-fieldlist.xml). X- extensions in the incoming vCard are
preserved.
Unified address book
--------------------
The unified address book is assembled by folks based on properties
that it considers "linkable". Linkable properties must be unique for a
person. A phone number is not linkable, because different persons
living at the same place might share a phone. For EDS, the linkable
properties are defined in folk's backends/eds/lib/edsf-persona.vala
and currently are:
- instant messaging handles
- email addresses
- local IDs (not used in this context)
- web service addresses (not used in this context)
folks supports heuristics for identifying persons which are likely to
be the same. This is not used by the PIM Manager. If it was, then links
would have to be established based on the local IDs and keeping those
stable became more important.
Localization
------------
The usual environment variables are used to determine the locale:
LC_CTYPE, LC_ALL, and LANG in that order (i.e. LC_CTYPE first and LANG
last). There is no support for changing that once syncevo-dbus-server
runs.
boost::locale is used for collation while sorting contacts. The
secondary collation level is used, which means that case, punctuation
are ignored while accents are relevant. This is hard-coded in
locale-factory-boost.cpp as DEFAULT_COLLATION_LEVEL.
Searching is either done with a strict text comparison (case
sensitive) or after using case folding (case insensitive, see
http://www.boost.org/doc/libs/1_51_0/libs/locale/doc/html/glossary.html#term_case_folding).
A 'phone' lookup uses libphonenumber to compare the free-form text
field for telephone numbers against a caller ID. The text field must
be in a format that libphonenumber understands, otherwise it will be
ignored. A country code is optional. If missing, the country code of
the current locale will be added before the comparison.
Testing
=======
Tests are mostly written in Python. See test/test-dbus.py (base
classes and testing of the normal SyncEvolution D-Bus API) and
src/dbus/server/pim/testpim.py.
These operations manipulate the system address book. To avoid
unexpected data loss when a developer runs the script in his
normal environment, check that testpim.py gets run like this:
XDG_CONFIG_HOME=`pwd`/temp-testpim/config \
XDG_DATA_HOME=`pwd`/temp-testpim/local/cache \
PATH=<SyncEvolution install path>/bin:<SyncEvolution install
path>/libexec:$PATH \
<path to SyncEvolution source>/test/dbus-session.sh \
<path to SyncEvolution source>/src/dbus/server/pim/testpim.py
This will use temp-testpim in addition to temp-testdbus in the current
directory for temporary files.
If TEST_DBUS_PBAP_PHONE is set to the Bluetooth MAC address (like
A0:4E:04:1E:AD:30) of a phone, PBAP syncing with that phone is tested.
The phone must be paired, connected and support SyncML in addition to
PBAP. The test hard-codes Nokia SyncML settings to keep it simpe and
because Nokia phones are most likely to be used.
_______________________________________________
SyncEvolution mailing list
[email protected]
http://lists.syncevolution.org/listinfo/syncevolution
