About 1.4.99.2
==============
1.4.99.2 is a new development snapshot. It enhances interoperability
with CardDAV servers and in particular Google Contacts considerably.
Contact data gets converted to and from the format typically used by
CardDAV servers, so now anniversary, spouse, manager, assistant and
instant message information are exchanged properly.
Categories are not supported by Google CardDAV and thus still get lost.
Custom labels get stored in EDS as extensions and no longer get lost
when updating some other aspects of a contact. However, Evolution does
not show custom labels and removes them when editing a property which
has a custom label (BGO #730636).
Scanning for CardDAV/CalDAV resources was enhanced. It now finds
additional calendars with Google CalDAV and works with iCloud.
However, syncing with iCloud ran into a server bug (reported as
17001498 "CalDAV REPORT drops calendar data") and needs further work.
The Ubuntu Online Accounts backend was added to syncevolution.org
binaries, targeting Ubuntu Saucy and later.
Details:
* vcard profile: avoid data loss during merging
When resolving a merge conflict, repeating properties were taken
wholesale from the winning side (for example, all email addresses). If
a new email address had been added on the loosing side, it got lost.
Arguably it is better to preserve as much data as possible during a
conflict. SyncEvolution now does that in a merge script by checking
which properties in the loosing side do not exist in the winning side
and copying those entries.
Typically only the main value (email address, phone number) is checked
and not the additional meta data (like the type). Otherwise minor
differences (for example, both sides have same email address, but with
different types) would lead to duplicates.
Only addresses are treated differently: for them all attributes
(street, country, city, etc.) are compared, because there is no single
main value.
* engine: UID support in contact data
Before, the UID property in a vCard was ignored by the engine.
Backends were responsible for ensuring that the property is
set if required by the underlying storage. This turned out to be
handled incompletely in the WebDAV backend.
This change moves this into the engine:
- UID is now field. It does not get used for matching
because the engine cannot rely on it being stored
by both sides.
- It gets parsed if present, but only generated if
explicitly enabled (because that is the traditional
behavior).
- It is never shown in the DevInf's CtCap
because the Synthesis engine would always show it
regardless whether a rule enabled the property.
That's because rules normally only get triggered
after exchanging DevInf and thus DevInf has to
be rule-independent. We don't want it shown because
then merging the incoming item during a local sync
would use the incoming UID, even if it is empty.
- Before writing, ensure that UID is set.
When updating an existing item, the Synthesis engine reads
the existing item, preserves the existing UID unless the peer
claims to support UID, and then updates with the existing UID.
This works for local sync (where SyncEvolution never claims
to support UID when talking to the other side). It will break
with peers which have UID in their CtCap although they
rewrite the UID and backends whose underlying storage cannot
handle UID changes during an update (for example, CardDAV).
* CardDAV: use Apple/Google/CardDAV vCard flavor
In principle, CardDAV servers support arbitrary vCard 3.0
data. Extensions can be different and need to be preserved. However,
when multiple different clients or the server's Web UI interpret the
vCards, they need to agree on the semantic of these vCard extensions.
In practice, CardDAV was pushed by Apple and Apple clients are
probably the most common clients of CardDAV services. When the Google
Contacts Web UI creates or edits a contact, Google CardDAV will
send that data using the vCard flavor used by Apple.
Therefore it makes sense to exchange contacts with *all* CardDAV
servers using that format. This format could be made configurable in
SyncEvolution on a case-by-case basis; at the moment, it is
hard-coded.
During syncing, SyncEvolution takes care to translate between the
vCard flavor used internally (based on Evolution) and the CardDAV
vCard flavor. This mapping includes:
X-AIM/JABBER/... <-> IMPP + X-SERVICE-TYPE
Any IMPP property declared as X-SERVICE-TYPE=AIM will get
mapped to X-AIM. Same for others. Some IMPP service types
have no known X- property extension; they are stored in
EDS as IMPP. X- property extensions without a known X-SERVICE-TYPE
(for example, GaduGadu and Groupwise) are stored with
X-SERVICE-TYPE values chosen by SyncEvolution so that
Google CardDAV preserves them (GroupWise with mixed case
got translated by Google into Groupwise, so the latter is used).
Google always sends an X-ABLabel:Other for IMPP. This is ignored
because the service type overrides it.
The value itself also gets transformed during the mapping. IMPP uses
an URI as value, with a chat protocol (like "aim" or "xmpp") and
some protocol specific identifier. For each X- extension the
protocol is determined by the property name and the value is the
protocol specific identifier without URL encoding.
X-SPOUSE/MANAGER/ASSISTANT <-> X-ABRELATEDNAMES + X-ABLabel
The mapping is based on the X-ABLabel property attached to
the X-ABRELATEDNAMES property. This depends on the English
words "Spouse", "Manager", "Assistant" that Google CardDAV
and Apple devices seem to use regardless of the configured
language.
As with IMPP, only the subset of related names which have
a corresponding X- property extension get mapped. The rest
is stored in EDS using the X-ABRELATEDNAMES property.
X-ANNIVERSARY <-> X-ABDATE
Same here, with X-ABLabel:Anniversary as the special case
which gets mapped.
X-ABLabel parameter <-> property
CardDAV vCards have labels attached to arbitrary other properties
(TEL, ADR, X-ABDATE, X-ABRELATEDNAMES, ...) via vCard group tags:
item1.X-ABDATE:2010-01-01
item1.X-ABLabel:Anniversary
The advantage is that property values can contain arbitrary
characters, including line breaks and double quotation marks,
which is not possible in property parameters.
Neither EDS nor KDE (judging from the lack of responses on the
KDE-PIM mailing list) support custom labels. SyncEvolution could
have used grouping as it is done in CardDAV, but grouping is not
used much (not at all?) by the UIs working with the vCards in EDS
and KDE. It seemed easier to use a new X-ABLabel parameter.
Characters which cannot be stored in a parameter get converted
(double space to single space, line break to space, etc.) during
syncing. In practice, these characters don't appear in X-ABLabel
properties anyway because neither Apple nor Google UIs allow entering
them for custom labels.
The "Other" label is used by Google even in case where it adds no
information. For example, all XMPP properties have an associated
X-ABLabel=Other although the Web UI does not provide a means to edit
or show such a label. Editing the text before the value in the UI
changes the X-SERVICE-TYPE parameter value, not the X-ABLabel as for
other fields.
Therefore the "Other" label is ignored by removing it during syncing.
X-EVOLUTION-UI-SLOT (the parameter used in Evolution to determine the
order of properties in the UI) gets stored in CardDAV. The only exception
is Google CardDAV which got confused when an IMPP property had both
X-SERVICE-TYPE and X-EVOLUTION-UI-SLOT parameters set. For Google,
X-EVOLUTION-UI-SLOT is only sent on other properties and thus ordering
of chat information can get lost when syncing with Google.
* synccompare: support grouping and quoted parameter strings
Grouped properties are sorted first according to the actual property
name, then related properties are moved to the place where their group
tag appears first. The first grouped property gets a "- " prefix, all
following ones are just indended with " ". The actual group tag is not
part of the normalized output, because its value is irrelevant:
BDAY:19701230
- EMAIL:[email protected]
X-ABLabel:custom-label2
...
FN:Mr. John 1 Doe Sr.
- IMPP;X-SERVICE-TYPE=AIM:aim:aim
X-ABLabel:Other
...
- X-ABDATE:19710101
X-ABLabel:Anniversary
Redundant tags (those set for only a single property, X-ABLabel:Other)
get removed as part of normalizing an item.
* WebDAV: use server's order when listing collections
When doing a recursive scan of the home set, preserve the order of
entries as reported by the server and check the first one first. The
server knows better which entries are more relevant for the user (and
thus should be the default) or may have some other relevant
order. Previously, SyncEvolution replaced that order with sorting by
URL, which led to a predictable, but rather meaningless order.
For example, Google lists the users own calendar first, followed by
the shared calendars sorted alphabetical by their name. Now
SyncEvolution picks the main calendar as default correctly when
scanning from https://www.google.com/calendar/dav/.
* WebDAV: improved database search (Google, Zimbra)
Zimbra has a principal URL that also serves as home set. When using it
as start URL, SyncEvolution only looked the URL once, without listing
its content, and thus did not find the databases.
When following the Zimbra principal URL indirectly, SyncEvolution did
check all of the collections there recursively. Unfortunately that
also includes many mail folders, causing the scan to abort after
checking 1000 collections (an internal safe guard).
The solution for both includes tracking what to do with a URL. For the
initial URL, only meta data about the URL itself gets
checked. Recursive scanning is only done for the home set. If that
home set contains many collections, scanning is still slow and may run
into the internal safe guard limit. This cannot be avoided because the
CalDAV spec explicitly states that the home set may contain normal
collections which contain other collections, so a client has to do the
recursive scan.
When looking at a specific calendar, Google CalDAV does not report
what the current principal or the home set is and therefore
SyncEvolution stopped after finding just the initial calendar. Now it
detects the lack of meta information and adds all parents also as
candidates that need to be looked at. The downside of this is that it
doesn't know anything about which parents are relevant, so it ends up
checking https://www.google.com/calendar/ and
https://www.google.com/.
In both cases Basic Auth gets rejected with a temporary redirect to
the Google login page, which is something that SyncEvolution must
ignore immediately during scanning without applying the resend
workaround for "temporary rejection of valid credentials" that can
happen for valid Google CalDAV URLs.
* WebDAV: enhanced database search (Google Calendar)
Additional databases where not found for several
reasons. SyncEvolution ignored all shared calendars
(http://calendarserver.org/ns/shared) and Google marks the additional
calendars that way. The other problem was that the check for leaf
collections (= collections which cannot contain other desired
collections) incorrectly excluded those collections instead of only
preventing listing of their content.
With this change,
https://www.google.com/calendar/dav/?SyncEvolution=Google can be used
as starting point for Google Calendar.
* WebDAV: fix database scan on iCloud
The calendar home set URL on iCloud (the one ending in /calendars/) is
declared as containing calendar data. That was enough for
SyncEvolution to accept it incorrectly as calendar. However, the home
set only contains calendar data indirectly.
* WebDAV: support redirects between hosts and DNS SRV lookup based on URL
When finding a new URL, we must be prepared to reinitialize the Neon
session with the new host settings.
iCloud does not have .well-known support on its www.icloud.com
server. To support lookup with a non-icloudd.com email address, we
must do DNS SRV lookup when access to .well-known URLs fails. We do
this without a www prefix on the host first, because that is what happens
to work for icloud.com.
With these changes it becomes possible to do database scans on Apple
iCloud, using syncURL=https://www.icloud.com or
syncURL=https://icloud.com. Giving the syncURL like this is only
necessary for a username that does not end in @icloud.com. When
the syncURL is not set, the domain for DNS SRV lookup is taken
from the username.
* WebDAV: more efficient item creation
PUT has the disadvantage that a client needs to choose a name and then
figure out what the real name on the server is. With Google CardDAV that
requires sending another request and only works because the server happens
to remember the original name (which is not guaranteed!).
POST works for new items without a name and happens to be implemented
by Google such that the response already includes all required
information (new name and revision string).
POST is checked for as described in RFC 5995 once before creating a new
item. Servers which don't support it continue to get a PUT.
* WebDAV: send "User-Agent: SyncEvolution"
Apple iCloud servers reject requests unless they contain a User-Agent
header. The exact value doesn't seem to matter. Making the string
configurable might be better, but can still be done later when it
is more certain whether and for what it is needed.
* WebDAV: refactor and fix DNS SRV lookup
The syncevo-webdav-lookup script was not packaged. It did not report
"not found" DNS results correctly and the caller did not check for
this either, so when looking up the information for a domain which
does not have DNS SRV entries, SyncEvolution ended up retrying for
while as if there had been a temporary lookup problem.
* signon: make Accounts optional
The new "signon" provider only depends on lib[g]signon-glib. It uses
gSSO if found, else UOA. Instead of pulling parameters and the
identity via libaccounts-glib, the user of SyncEvolution now has to
ensure that the identity exists and pass all relevant parameters
in the "signon:" username.
* gSSO: adapt to gSSO >= 2.0
* config templates: Funambol URLs
Funambol turned of the URL redirect from my.funambol.com to
onemedia.com. The Funambol template now uses the current URL. Users
with existing Funambol configs must updated the syncURL property
manually to https://onemediahub.com/sync
Kudos to Daniel Clement for reporting the change.
* command line: fix --update from directory
The "--update <dir name>" operation was supposed to take the
item luids from the file names inside the directory. That part
had not been implemented, turning the operation accidentally
into an "--import".
Also missing was the escaping/unescaping of luids. Now the
same escaping is done as in command line output and command
line parsing to make the luids safe for use as file name.
* testing: added server-specific tests for CardDAV covering
remote item formats and edit conflicts.
Upgrading from releases <= 1.3.99.4:
------------------------------------
If the value of "username/databaseUser/proxyUser" contains a colon,
the "user:" prefix must be added to the value, to continue treating it
like a plain user name and not some reference to an unknown identity
provider (like "id:", "goa:", "signon:", etc.).
The lookup of passwords in GNOME Keyring was updated slightly in
1.3.99.5. It may be necessary to set passwords anew if the old one is
no longer found.
Upgrading from release 1.2.x:
-----------------------------
The sync format of existing configurations for Mobical (aka Everdroid)
must be updated manually, because the server has encoding problems when
using vCard 3.0 (now the default for Evolution contacts):
syncevolution --configure \
syncFormat=text/x-vcard \
mobical addressbook
The Funambol template explicitly enables usage of the
"refresh-from-server" sync mode to avoid getting throttled with 417
'retry later' errors. The same must be added to existing configs
manually:
syncevolution --configure \
enableRefreshSync=TRUE \
funambol
Upgrading from releases before 1.2:
-----------------------------------
Old configurations can still be read. But writing, as it happens
during a sync, must migrate the configuration first. Releases >= 1.2
automatically migrates configurations. The old configurations
will still be available (see "syncevolution --print-configs") but must
be renamed manually to use them again under their original names with
older SyncEvolution releases.
Source, Installation, Further information
=========================================
http://syncevolution.org/blogs/pohly/2014/syncevolution-14992-released
Source code bundles for users are available in
http://downloads.syncevolution.org/syncevolution/sources
and the original source is in the git repositories
http://cgit.freedesktop.org/SyncEvolution/
i386, lpia and amd64 binaries for Debian-based distributions are
available via the "unstable" syncevolution.org repository. Add the
following entry to your /apt/source.list:
deb http://downloads.syncevolution.org/apt unstable main
Then install "syncevolution-evolution", "syncevolution-kde" and/or
"syncevolution-activesync".
These binaries include the "sync-ui" GTK GUI and were compiled for
Ubuntu 10.04 LTS (Lucid), except for ActiveSync binaries which were
compiled for Debian Wheezy, Ubuntu Saucy and Ubuntu Trusty. A backend
for Ubuntu Online Accounts was compiled on Ubuntu Saucy. The packages
mentioned above are meta-packages which pull in suitable packages
matching the distro during installation.
Older distributions like Debian 4.0 (Etch) can no longer be supported
with precompiled binaries because of missing libraries, but the source
still compiles when not enabling the GUI (the default).
The same binaries are also available as .tar.gz and .rpm archives in
http://downloads.syncevolution.org/syncevolution/. In contrast
to 0.8.x archives, the 1.x .tar.gz archives have to be unpacked and the
content must be moved to /usr, because several files would not be found
otherwise.
After installation, follow the
http://syncevolution.org/documentation/getting-started steps.
--
Patrick Ohly, on behalf of everyone who has helped
to make SyncEvolution possible:
http://syncevolution.org/about/contributors
_______________________________________________
SyncEvolution mailing list
[email protected]
https://lists.syncevolution.org/mailman/listinfo/syncevolution
