[os-libsynthesis] avoiding disk writes during sync without changes
Hello!
For IVI, I am working on cloud syncing. When syncing against a WebDAV
server, SyncEvolution runs a two-way sync with binfile client on the
WebDAV side and a server with admin data handled by SyncEvolution on the
other side.
One goal for IVI is to minimize or better, avoid disk writes, because
flash storage must last as long as possible.
The most common case is that nothing changed on either side. In this
case, libsynthesis unnecessarily updates nonce (even if not used; I've
already patched that (1)), sync anchors (again, I have a patch for this:
skip writing of admin data after detecting the special case (2)) and the
change log in the binfile client.
This last write happens in TBinfileImplDS::changeLogPreflight(). The
changes are minor, just a few bytes change. I suspect that these are the
time stamps and modcount embedded in the log.
Would it be possible to check in changeLogPreflight() how significant
the changes are? If there were no item changes, what would be the effect
of not updating the header?
There are two cases where that can happen with the attached patch:
- the processes crashes
- SyncEvolution skips the session shutdown, see (2).
(1) For local sync, requestedauthnone/requestedauth and
requiredauthnone/requiredauth are used. Patch attached. Okay?
(2) This happens in SyncEvolution: the SaveAdminData implementation
detects that nothing happened during the sync, then returns an error to
libsynthesis and tells the SyncEvolution event loop to stop the sync
without sending the final reply message to the client. The server side
then finishes the sync successfully while the client side simply aborts
without doing its own session shutdown. This also eliminates
SaveAdminData on the binfile client side.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
From 4683fceb72cef0e7301b735a23b1f40323da52ba Mon Sep 17 00:00:00 2001
From: Patrick Ohly patrick.o...@intel.com
Date: Fri, 29 Aug 2014 10:35:47 +0200
Subject: [PATCH 1/2] syncsession: avoid unnecessary nonce update
When the configure auth type needs no nonce, don't generate one
and return NULL immediately, like newChallenge() would do.
That avoids unnecessary disk writes for storing the new nonce, which
is important for local syncs in IVI. It also avoids debug output about
a challenge that would not get sent.
---
src/sysync/syncsession.cpp | 11 +--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/src/sysync/syncsession.cpp b/src/sysync/syncsession.cpp
index 54c119c..f7c7a11 100644
--- a/src/sysync/syncsession.cpp
+++ b/src/sysync/syncsession.cpp
@@ -4061,15 +4061,22 @@ bool TSyncSession::processSyncOpItem(
// generate challenge for session
SmlChalPtr_t TSyncSession::newSessionChallenge(void)
{
+ sysync::TAuthTypes auth = requestedAuthType();
+
+ // Avoid misleading debug output (there is no challenge)
+ // and more importantly, creating a new nonce that is not
+ // going to be used. That causes unnecessary disk writes.
+ if (auth == sysync::auth_none) return NULL;
+
string nonce;
getNextNonce(fRemoteURI.c_str(),nonce);
PDEBUGPRINTFX(DBG_PROTO,(
Challenge for next auth: AuthType=%s, Nonce='%s', binary %sallowed,
-authTypeSyncMLNames[requestedAuthType()],
+authTypeSyncMLNames[auth],
nonce.c_str(),
getEncoding()==SML_WBXML ? : NOT
));
- return newChallenge(requestedAuthType(),nonce,getEncoding()==SML_WBXML);
+ return newChallenge(auth,nonce,getEncoding()==SML_WBXML);
} // TSyncSession::newSessionChallenge
--
2.1.0.rc1
From 115cef8a0ec1823b58808bbb9820f7cb82c2a313 Mon Sep 17 00:00:00 2001
From: Patrick Ohly patrick.o...@intel.com
Date: Fri, 29 Aug 2014 10:39:17 +0200
Subject: [PATCH 2/2] binfileclient: avoid disk writes in changeLogPreflight()
During a normal sync where nothing changed, only the header gets
updated. This change is not critical and thus does not have to be
flushed to disk unless also some entries get added or updated.
The advantage is that when SyncEvolution detects a sync where nothing
changed on either side and skips the client's session shutdown, the
.bfi is left unchanged, which reduces flash wearout.
To detect item changes, a brute-force byte comparison is used. This
requires less changes to the code and is less error-prone than adding
modified=true to all places where existingentries gets modified.
---
src/sysync/binfileimplds.cpp | 24 +++-
1 file changed, 19 insertions(+), 5 deletions(-)
diff --git a/src/sysync/binfileimplds.cpp b/src/sysync/binfileimplds.cpp
index 314ccc7..947fa0c 100755
--- a/src/sysync/binfileimplds.cpp
+++ b/src/sysync/binfileimplds.cpp
@@ -741,9 +741,9 @@ localstatus TBinfileImplDS::changeLogPreflight(bool aValidChangelog)
lsd.c_str()
));
#endif
- // - save header
Re: [os-libsynthesis] comparison of PHOTO data
On Mon, 2014-07-14 at 18:15 +0200, Lukas Zeller wrote:
Hello Patrick,
On 11 Jul 2014, at 15:27, Patrick Ohly patrick.o...@intel.com wrote:
I'm currently investigating why a comparison of two PHOTO fields of
different length returns field equal. PHOTO is defined as:
field name=PHOTO type=string compare=conflict
merge=fillempty/
My first question would be why PHOTO was defined string here, not blob?
I changed that in this commit:
commit 6d3b1cf64b09cf1603b813137d14d529a31dda8b
Author: Patrick Ohly patrick.o...@intel.com
Date: Fri Jul 5 09:45:15 2013 +0200
EDS: update PHOTO+GEO during slow sync, avoid rewriting PHOTO file
If PHOTO and/or GEO were the only modified properties during a slow
sync, the updated item was not written into local storage because
they were marked as compare=never = not relevant.
For PHOTO this was intentional in the sample config, with the
rationale that local storages often don't store the data exactly as
requested. When that happens, comparing the data would lead to
unnecessary writes. But EDS and probably all other local SyncEvolution
storages (KDE, file) store the photo exactly as requested, so not
considering changes had the undesirable effect of not always writing
new photo data.
For GEO, ignoring it was accidental.
A special merge script handles EDS file:// photo URIs. When the
loosing item has the data in a file and the winning item has binary
data, the data in the file may still be up-to-date, so before allowing
MERGEFIELDS() to overwrite the file reference with binary data and
thus forcing EDS to write a new file, check the content. If it
matches, use the file reference in both items.
Unfortunately I did not write down why I changed the type, instead of
merely changing the compare setting.
Does it matter at all?
IMHO The real bug is that TBlobField does not override compareWith.
TBlobField's compareWith could ignore aCaseInsensitive and use memcmp,
for the case of actually comparing two blobs, and fall back to
TStringField for comparison with other types.
We have std::string as value and therefore can store null bytes as part
of the data, but the actual comparison falls back to C-string
operations, which only work for null bytes at the end.
There's to much C strings throughout the entire project, so for text,
C string semantics are assumed pretty much everywhere.
TBlobField however makes use of the true binary byte string capability of
std::string exactly this way.
That's also true for TStringField, except that it occasionally falls
back to plain C operations, which is only correct for real strings.
I think the strcmp() needs to be replaced with something that also looks
at the rest of the string if no difference is found. Agreed?
A fully std::string compatible comparison would be nicer here, and fix the
problem.
Here's the patch that I ended up using:
commit 9278e054e9a9a2aa8c73aed98cb42bf1f9bfd0fe
Author: Patrick Ohly patrick.o...@intel.com
Date: Mon Jul 14 05:00:54 2014 -0700
string fields: full compare
String fields also get used for arbitrary binary data, like
photos. In that case we need to compare the entire std::string,
not just the part before any embedded null byte.
This gets done using std::string::compare. Case-insensitive
comparison still uses C strucmp() and thus should never be used
for non-string data.
diff --git a/src/sysync/itemfield.cpp b/src/sysync/itemfield.cpp
index e6f7d71..64867e0 100644
--- a/src/sysync/itemfield.cpp
+++ b/src/sysync/itemfield.cpp
@@ -801,7 +801,7 @@ bool TStringField::merge(TItemField aItemField, const char
aSep)
// Note: Both fields must be assigned. NO TEST IS DONE HERE!
sInt16 TStringField::compareWith(TItemField aItemField, bool aCaseInsensitive)
{
- sInt16 result;
+ int result;
PULLFROMPROXY;
if (aItemField.isBasedOn(fty_string)) {
TStringField *sfP = static_castTStringField *(aItemField);
@@ -812,7 +812,7 @@ sInt16 TStringField::compareWith(TItemField aItemField,
bool aCaseInsensitive)
if (aCaseInsensitive)
result=strucmp(fString.c_str(),sfP-fString.c_str());
else
- result=strcmp(fString.c_str(),sfP-fString.c_str());
+ result=fString.compare(sfP-fString);
}
else {
// convert other field to string
@@ -821,7 +821,7 @@ sInt16 TStringField::compareWith(TItemField aItemField,
bool aCaseInsensitive)
if (aCaseInsensitive)
result=strucmp(fString.c_str(),s.c_str());
else
- result=strcmp(fString.c_str(),s.c_str());
+ result=fString.compare(s);
}
return result 0 ? 1 : (result0 ? -1 : 0);
} // TStringField::compareWith
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
[os-libsynthesis] comparison of PHOTO data
Hello!
I'm currently investigating why a comparison of two PHOTO fields of
different length returns field equal. PHOTO is defined as:
field name=PHOTO type=string compare=conflict merge=fillempty/
I think the specific situation is that the two values contain a null
byte somewhere in the middle, and the part before that is equal.
I think the following code does the comparison, doesn't it?
sInt16 TStringField::compareWith(TItemField aItemField, bool aCaseInsensitive)
{
sInt16 result;
PULLFROMPROXY;
if (aItemField.isBasedOn(fty_string)) {
TStringField *sfP = static_castTStringField *(aItemField);
#ifdef STREAMFIELD_SUPPORT
sfP-pullFromProxy(); // make sure we have all chars
#endif
// direct compare possible, return strcmp
if (aCaseInsensitive)
result=strucmp(fString.c_str(),sfP-fString.c_str());
else
result=strcmp(fString.c_str(),sfP-fString.c_str());
}
...
We have std::string as value and therefore can store null bytes as part
of the data, but the actual comparison falls back to C-string
operations, which only work for null bytes at the end.
I think the strcmp() needs to be replaced with something that also looks
at the rest of the string if no difference is found. Agreed?
Or do we need a real binary type? Using string type for PHOTO also has
the risk that during a merge operation, the two values will get
concatenated (thus breaking the images) unless some custom merge script
resolves the conflict differently first.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] How to run syncevolution at bare minimum processing.
On Mon, 2014-05-19 at 18:46 +0530, anuj chauhan wrote: I have a quad-core machine with 8gb ram and these processes are consuming about 90-94% of the cpu usage.I want to tune syncevolution so that it could run on bare minimum processing.For this i have done following modification : 1. Commented out the code which gets the Server DevInf. I suspect, this will reduce load on Server and client as well. 2. Removed the file synccompare. During these concurrent runs, top command showed high CPU usage for Perl just after syncevolution completed its exercise. By removing this file, this reduced the client load and processing. Instead it would have been better to use printChanges=0. This disables the invocation of synccompare. 3. Reduced the log level to 0. Number of IO went down along with the disk usage. You can decrease it further by also using dumpData=0. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] mixing properties with and without group tag (was: Re: suppressempty + arrays)
On Wed, 2014-05-14 at 17:44 +0200, Lukas Zeller wrote: Hello Patrick, On 12.05.2014, at 18:25, Patrick Ohly patrick.o...@intel.com wrote: [...] and a property which has a group tag must not reuse any of these unassigned group tag values. Actually, a property which has a group *field* - it doesn't matter whether the current property has a group tag value. This check was missing. Attached a patch adding it, in a brute-force manner. Does that look right? Yes, it looks right to me. I guess you've found out in the meantime if it also *works* right :-) Yes, it works for me. But it's good that you had a look at it anyway, because my vCard profile certainly isn't representative. I'll include this and the other patches (including the removal of backslash escaping in parameters!) in the master branch of libsynthesis on freedesktop.org once it passed all my tests. There's also a memory leak fix for SWAP(). -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] mixing properties with and without group tag (was: Re: suppressempty + arrays)
On Mon, 2014-05-12 at 12:55 +0200, Patrick Ohly wrote: The X-ABRELATEDNAMES properties were not generated. The labels should be redundant, but some peers get confused. Google preserves them as stand-alone X-ABLabel without tag. DAViCal preserves them with tag, which then happened to confuse SyncEvolution's conversion code (separate issue). This second issue actually is in the groupfield support of libsynthesis: [2014-05-12 15:02:45.501] Parsing: * [2014-05-12 15:02:45.501] BEGIN:VCARD VERSION:3.0 PRODID:-//Synthesis AG//NONSGML SyncML Engine V3.4.0.47//EN REV:20140512T150240Z UID:syuid974165.212266710163478 N:Doe;John;;; FN:John Doe X-EVOLUTION-FILE-AS:Doe\, John TITLE:tester TEL;TYPE=WORK,VOICE:business 1 X-MOZILLA-HTML:FALSE item3.X-ABLabel:Spouse item2.X-ABLabel:Manager item1.X-ABLabel:Assistant END:VCARD * [2014-05-12 15:02:45.501] Successfully parsed: * [2014-05-12 15:02:45.501] Item LocalID='syuid974165.212266710163478.vcf', RemoteID='', operation=wants-add, size: [maxlocal,maxremote,actual] * [2014-05-12 15:02:45.501] - 0 :integer SYNCLVL [ 0, n/a, 0] : unassigned - 1 : timestamp REV [ 0, 0, 0] : 2014-05-12T15:02:40Z (TZ: UTC) - 2 : string UID [ 0, n/a,27] : syuid974165.212266710163478 - 3 : string GROUP_TAG [ 0, n/a, 0] : array with 3 elements -- element0 : item3 -- element1 : item2 -- element2 : item1 - 4 : string N_LAST [ 0, 0, 3] : Doe - 5 : string N_FIRST [ 0, 0, 4] : John ... - 23 : telephone TEL [ 0, 0, 0] : array with 1 elements -- element0 : business 1 - 24 :integer TEL_FLAGS [ 0, 0, 0] : array with 1 elements -- element0 : 10 - 25 :integer TEL_SLOT[ 0, 0, 0] : array with 0 elements ... - 83 : string LABEL [ 0, 0, 0] : array with 3 elements -- element0 : Spouse -- element1 : Manager -- element2 : Assistant - 84 : string XPROPS [ 0, 0, 0] : array with 0 elements This field list makes it look like TEL business 1 at index #0 had the same group tag as LABEL Spouse, thus adding a label to a TEL which had no label. I think the code which deals with group tags must use the same logic that I introduced for sharedfield: a property which has a group field array, but no group tag, must set an unassigned value in the group field array, and a property which has a group tag must not reuse any of these unassigned group tag values. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] mixing properties with and without group tag (was: Re: suppressempty + arrays)
On Mon, 2014-05-12 at 17:29 +0200, Patrick Ohly wrote:
I think the code which deals with group tags must use the same logic
that I introduced for sharedfield: a property which has a group field
array, but no group tag, must set an unassigned value in the group field
array,
That was already done:
if (aPropP-groupFieldID!=FID_NOT_SUPPORTED) {
TItemField *g_fldP =
aItem.getArrayFieldAdjusted(aPropP-groupFieldID+baseoffset,repoffset,false);
if (g_fldP)
g_fldP-setAsString(aGroupName,aGroupNameLen); // store the group name
(aGroupName might be NULL, that's ok)
}
and a property which has a group tag must not reuse any of these
unassigned group tag values.
Actually, a property which has a group *field* - it doesn't matter
whether the current property has a group tag value. This check was
missing. Attached a patch adding it, in a brute-force manner. Does that
look right?
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
From f0396958e9311db0bcf44a1810ee89afc99b Mon Sep 17 00:00:00 2001
From: Patrick Ohly patrick.o...@intel.com
Date: Mon, 12 May 2014 09:03:56 -0700
Subject: [PATCH] mimedir parser: avoid reusing group tag
When two properties (TEL and X-ABLabel in this example) share the same group
field, then storing the property where a group tag was used for the first time
must not reuse unassigned group field entries.
There was some logic for that already (see someGroups), but it failed to
handle this case. This approach here uses brute-force instead of trying to me
smart: the reuse check in the nmaxrep for loop will fail repeatedly at the
beginning until reaching the end of the group field array.
---
src/sysync/mimedirprofile.cpp |7 +--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/src/sysync/mimedirprofile.cpp b/src/sysync/mimedirprofile.cpp
index 418c5d0..e618f80 100644
--- a/src/sysync/mimedirprofile.cpp
+++ b/src/sysync/mimedirprofile.cpp
@@ -4168,8 +4168,11 @@ bool TMimeDirProfileHandler::parseProperty(
TItemField *e_fldP = NULL;
if (e_basefldP)
e_fldP=e_basefldP-getArrayField(e_rep,true); // get leaf field, if it exists
- if (!e_basefldP || (e_fldP e_fldP-isAssigned())) {
-// base field of one of the main fields does not exist or leaf field is already assigned
+ if (!e_basefldP || (e_fldP e_fldP-isAssigned()) ||
+ (aPropP-groupFieldID!=FID_NOT_SUPPORTED !valuelist
+ aItem.getArrayFieldAdjusted(aPropP-groupFieldID+baseoffset,e_rep,true))) {
+// base field of one of the main fields does not exist or leaf field is already assigned,
+// or the group field entry is already in use (doesn't matter whether it is empty)
// - skip that repetition
dostore = false;
break;
--
1.7.10.4
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] parameter value escaping (was: Re: vCard group tags)
On Mon, 2014-05-05 at 14:42 +0200, Patrick Ohly wrote: On Fri, 2014-05-02 at 11:59 +0200, Lukas Zeller wrote: On 02.05.2014, at 10:38, Patrick Ohly patrick.o...@intel.com wrote: I noticed another problem with the use X-ABLabel parameter approach: storing complex strings (spaces, quotation marks) in a parameter value is harder. That's probably why Apple chose the X-ABLabel property approach. An unparseable parameter could ruin the real data, a unknown property is less dangerous. The EDS vCard parser gets it wrong and fails to parse: X-ABRELATEDNAMES;X-ABLabel=domestic partner:domestic partner That is valid according to http://tools.ietf.org/html/rfc2425#page-5 because the space is a SAFE-CHAR and thus may appear in a ptext, but the EDS parser does not expect the space. To work around this, we could voluntarily quote the string even though it is not required. Now, the conceptual problem with X-ABLabel parameter is that a quoted string cannot contain double quotes. It is probably rare that a user enters double quotes as part of a label, but it cannot be ruled out either. Line breaks are also only allowed in property values and not parameter values. I've looked into TMimeDirProfileHandler::generateValue() some more to understand under which circumstances libsynthesis uses quoted strings (with double quotes at start and end) as parameter value. At first glance it doesn't seem to do that at all. Instead special values are escaped with backslash. item29.X-ABLabel:custom-label5\nUmlaut-ä\nSemicolon\; - X-ABRELATEDNAMES;X-ABLabel=custom-label5\nUmlaut-ä\nSemicolon\;:custom relationship Where is it specified that the backslash escape mechanism can be used in parameter values? http://tools.ietf.org/html/rfc2425#page-5 says: param= param-name = param-value *(, param-value) param-name = x-name / iana-token param-value = ptext / quoted-string ptext = *SAFE-CHAR SAFE-CHAR= WSP / %x21 / %x23-2B / %x2D-39 / %x3C-7E / NON-ASCII ; Any character except CTLs, DQUOTE, ;, :, , My reading of that is that special characters must not appear in a ptext at all, not even when escaped with backslash. One has to use a quoted string, which (unfortunately) cannot hold all characters either. IMHO libsynthesis is currently producing broken vCards. I consider changing the code so that it uses quoted strings if it detects unsafe characters in the value and filters out invalid ones. unsafe would be more conservative than in the RFC itself and also include spaces, to work around the EDS parser bug. Attached is a patch which makes the libsynthesis parser and generator behave according to my current understanding of the RFCs. The risk of course is that there are cases where backslashes are used in parameter values and the peer (unpatched libsynthesis, other implementations) expect that backslash escaping is used. The Evolution Data Server parser does not use backslash escaping for parameters. To exchange values containing backslashes, the patch is needed. This becomes relevant in the context of the X-ABLabel parameter. My hope is that all other parameters are simple enough that the ambiguity never arises. Or do we need an on/off switch for backslash escaping depending on the peer? -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. From 8a7356c1518aca405eb29ddf12a2d1052375da68 Mon Sep 17 00:00:00 2001 From: Patrick Ohly patrick.o...@intel.com Date: Tue, 6 May 2014 12:13:19 +0200 Subject: [PATCH 2/2] MIME parser + encoder: no backslash quoting in parameter values MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit RFC 2425 (MIME DIR) and RFC 6350 (vCard) do not describe backslash escaping for special characters in parameter values. Only the more limited quoted-string (double quotes at start and end, no line breaks, no double quotes inside value) is specified. The following two examples both contain a literal backslash: URL;X-ABLabel=Custom-label6 Backslash \:http://custom.com X-ABRELATEDNAMES;X-ABLabel=custom-label5 Umlaut ä Semicolon ; Backslash \ end of label:custom relationship This commit limits backslash escaping to parsing and generating property values. Backslashs in parameters are stored literally during parsing. A quoted string parameter value is used for every value that is more complex than alphanumeric plus underscore and hyphen. In particular spaces (while allowed in unquoted values) are only generated as part of quoted strings because the Evolution Data Server parser had problems with them (fixed in EDS 3.10). --- src/sysync/mimedirprofile.cpp | 60 - 1 file changed, 42
[os-libsynthesis] folding parameter values (was: Re: vCard group tags)
On Mon, 2014-05-05 at 14:42 +0200, Patrick Ohly wrote: On Fri, 2014-05-02 at 11:59 +0200, Lukas Zeller wrote: On 02.05.2014, at 10:38, Patrick Ohly patrick.o...@intel.com wrote: I noticed another problem with the use X-ABLabel parameter approach: storing complex strings (spaces, quotation marks) in a parameter value is harder. That's probably why Apple chose the X-ABLabel property approach. An unparseable parameter could ruin the real data, a unknown property is less dangerous. The EDS vCard parser gets it wrong and fails to parse: X-ABRELATEDNAMES;X-ABLabel=domestic partner:domestic partner That is valid according to http://tools.ietf.org/html/rfc2425#page-5 because the space is a SAFE-CHAR and thus may appear in a ptext, but the EDS parser does not expect the space. To work around this, we could voluntarily quote the string even though it is not required. Now, the conceptual problem with X-ABLabel parameter is that a quoted string cannot contain double quotes. It is probably rare that a user enters double quotes as part of a label, but it cannot be ruled out either. Line breaks are also only allowed in property values and not parameter values. I've looked into TMimeDirProfileHandler::generateValue() some more to understand under which circumstances libsynthesis uses quoted strings (with double quotes at start and end) as parameter value. At first glance it doesn't seem to do that at all. Instead special values are escaped with backslash. item29.X-ABLabel:custom-label5\nUmlaut-ä\nSemicolon\; - X-ABRELATEDNAMES;X-ABLabel=custom-label5\nUmlaut-ä\nSemicolon\;:custom relationship Where is it specified that the backslash escape mechanism can be used in parameter values? http://tools.ietf.org/html/rfc2425#page-5 says: param= param-name = param-value *(, param-value) param-name = x-name / iana-token param-value = ptext / quoted-string ptext = *SAFE-CHAR SAFE-CHAR= WSP / %x21 / %x23-2B / %x2D-39 / %x3C-7E / NON-ASCII ; Any character except CTLs, DQUOTE, ;, :, , My reading of that is that special characters must not appear in a ptext at all, not even when escaped with backslash. One has to use a quoted string, which (unfortunately) cannot hold all characters either. Furthermore, folding is not described for parameter values, is it? X-ABRELATEDNAMES;X-ABLabel=custom-label5 Umlaut ä Semicolon ; Backslash \ newline tab \t end of label:custom relationship This is what libsynthesis produces for a long parameter value. The \t was part of the original value. With the revised parser/generator it just gets passed through. I think the generator should be changed to not fold a line unless the property value has started. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] folding parameter values (was: Re: vCard group tags)
On Tue, 2014-05-06 at 13:29 +0200, Patrick Ohly wrote:
On Mon, 2014-05-05 at 14:42 +0200, Patrick Ohly wrote:
On Fri, 2014-05-02 at 11:59 +0200, Lukas Zeller wrote:
On 02.05.2014, at 10:38, Patrick Ohly patrick.o...@intel.com wrote:
I noticed another problem with the use X-ABLabel parameter approach:
storing complex strings (spaces, quotation marks) in a parameter value
is harder.
That's probably why Apple chose the X-ABLabel property approach. An
unparseable parameter could ruin the real data, a unknown property is
less dangerous.
The EDS vCard parser gets it wrong and fails to parse:
X-ABRELATEDNAMES;X-ABLabel=domestic partner:domestic partner
That is valid according to http://tools.ietf.org/html/rfc2425#page-5
because the space is a SAFE-CHAR and thus may appear in a ptext, but the
EDS parser does not expect the space. To work around this, we could
voluntarily quote the string even though it is not required.
Now, the conceptual problem with X-ABLabel parameter is that a quoted
string cannot contain double quotes. It is probably rare that a user
enters double quotes as part of a label, but it cannot be ruled out
either. Line breaks are also only allowed in property values and not
parameter values.
I've looked into TMimeDirProfileHandler::generateValue() some more to
understand under which circumstances libsynthesis uses quoted strings
(with double quotes at start and end) as parameter value. At first
glance it doesn't seem to do that at all. Instead special values are
escaped with backslash.
item29.X-ABLabel:custom-label5\nUmlaut-ä\nSemicolon\;
-
X-ABRELATEDNAMES;X-ABLabel=custom-label5\nUmlaut-ä\nSemicolon\;:custom
relationship
Where is it specified that the backslash escape mechanism can be used in
parameter values?
http://tools.ietf.org/html/rfc2425#page-5 says:
param= param-name = param-value *(, param-value)
param-name = x-name / iana-token
param-value = ptext / quoted-string
ptext = *SAFE-CHAR
SAFE-CHAR= WSP / %x21 / %x23-2B / %x2D-39 / %x3C-7E / NON-ASCII
; Any character except CTLs, DQUOTE, ;, :, ,
My reading of that is that special characters must not appear in a ptext
at all, not even when escaped with backslash. One has to use a quoted
string, which (unfortunately) cannot hold all characters either.
Furthermore, folding is not described for parameter values, is it?
X-ABRELATEDNAMES;X-ABLabel=custom-label5 Umlaut ä Semicolon ; Backslash \
newline tab \t end of label:custom relationship
This is what libsynthesis produces for a long parameter value. The \t
was part of the original value. With the revised parser/generator it
just gets passed through.
I think the generator should be changed to not fold a line unless the
property value has started.
I'm not sure about that anymore. The EDS parser/generator has no
problems with folding inside parameters.
Attached is a patch which does what I had in mind, but we can probably
ignore it unless folding inside parameters really turns out to be wrong
and/or cause problems.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
diff --git a/src/sysync/mimedirprofile.cpp b/src/sysync/mimedirprofile.cpp
index 83acbca..929ef5c 100644
--- a/src/sysync/mimedirprofile.cpp
+++ b/src/sysync/mimedirprofile.cpp
@@ -2101,6 +2101,7 @@ static void finalizeProperty(
const char *proptext,
string aString,
TMimeDirMode aMimeMode, // MIME mode (older or newer vXXX format compatibility)
+ size_t foldStartingAt, // First byte offset where folding is allowed.
bool aDoNotFold, // set to prevent folding
bool aDoSoftBreak // set to insert QP-softbreaks when \r is encountered, otherwise do a full hard break (which essentially inserts a space for mimo_old)
)
@@ -2112,6 +2113,7 @@ static void finalizeProperty(
ssize_t foldLoc = -1; // possible break location - linear white space or explicit break indicator
bool explf;
cAppCharP firstunwritten=proptext; // none written yet
+ const char *start = proptext;
while (proptext (c=*proptext)!=0) {
// remember position of last lwsp (space or TAB)
if (aMimeMode==mimo_old (c==' ' || c==0x09))
@@ -2142,8 +2144,8 @@ static void finalizeProperty(
llen++;
// explicit linefeed flag
explf=(c=='\n' || c=='\r');
-if (aDoNotFold) {
- // prohibit folding for ugly devices like V3i
+if (aDoNotFold || (proptext (proptext - start) foldStartingAt)) {
+ // prohibit folding for ugly devices like V3i or in the middle of the parameters
if (explf) {
// append what we have until here
n--; // explicit \n or \r is ignored
Re: [os-libsynthesis] vCard group tags
On Fri, 2014-05-02 at 11:59 +0200, Lukas Zeller wrote: On 02.05.2014, at 10:38, Patrick Ohly patrick.o...@intel.com wrote: I noticed another problem with the use X-ABLabel parameter approach: storing complex strings (spaces, quotation marks) in a parameter value is harder. That's probably why Apple chose the X-ABLabel property approach. An unparseable parameter could ruin the real data, a unknown property is less dangerous. The EDS vCard parser gets it wrong and fails to parse: X-ABRELATEDNAMES;X-ABLabel=domestic partner:domestic partner That is valid according to http://tools.ietf.org/html/rfc2425#page-5 because the space is a SAFE-CHAR and thus may appear in a ptext, but the EDS parser does not expect the space. To work around this, we could voluntarily quote the string even though it is not required. Now, the conceptual problem with X-ABLabel parameter is that a quoted string cannot contain double quotes. It is probably rare that a user enters double quotes as part of a label, but it cannot be ruled out either. Line breaks are also only allowed in property values and not parameter values. I've looked into TMimeDirProfileHandler::generateValue() some more to understand under which circumstances libsynthesis uses quoted strings (with double quotes at start and end) as parameter value. At first glance it doesn't seem to do that at all. Instead special values are escaped with backslash. item29.X-ABLabel:custom-label5\nUmlaut-ä\nSemicolon\; - X-ABRELATEDNAMES;X-ABLabel=custom-label5\nUmlaut-ä\nSemicolon\;:custom relationship Where is it specified that the backslash escape mechanism can be used in parameter values? http://tools.ietf.org/html/rfc2425#page-5 says: param= param-name = param-value *(, param-value) param-name = x-name / iana-token param-value = ptext / quoted-string ptext = *SAFE-CHAR SAFE-CHAR= WSP / %x21 / %x23-2B / %x2D-39 / %x3C-7E / NON-ASCII ; Any character except CTLs, DQUOTE, ;, :, , My reading of that is that special characters must not appear in a ptext at all, not even when escaped with backslash. One has to use a quoted string, which (unfortunately) cannot hold all characters either. IMHO libsynthesis is currently producing broken vCards. I consider changing the code so that it uses quoted strings if it detects unsafe characters in the value and filters out invalid ones. unsafe would be more conservative than in the RFC itself and also include spaces, to work around the EDS parser bug. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] vCard group tags
On Fri, 2014-05-02 at 11:59 +0200, Lukas Zeller wrote:
Hello Patrick,
On 02.05.2014, at 10:38, Patrick Ohly patrick.o...@intel.com wrote:
What do you mean by independent properties?
For example, ADR and TEL are independent in the sense that their values
are stored in different field arrays. Sharing the LABEL field for the
new parameter creates a dependency (or risk of collision) that did not
exist before.
But how can this work at all? The index in the field arrays is what
relates the components (e.g. TEL, TEL_FLAGS and LABEL) to each other.
This would mean all of the field arrays would need to have rows for
the *sum* of all the TEL+ADR+EMAIL+URL+xxx, with most rows empty, to
be able to share a single LABEL array.
Now I see why you want to check the availability of LABEL row even if
assigned via a property parameter.
Hmmm - indeed this might work, at the expense of a lot of empty
elements in the actual property field arrays.
Exactly, that's what I need and get with the patch.
I suspect that a groupfield would cause less empty elements if (and only
if) no groups are used, because then values from different properties
can use the same UNASSIGNED entry in the group field. But I haven't
checked whether that's really how it works; it is not very relevant with
Google because Google uses many group tags, even if the label is just
Other.
If different properties end up sharing the same UNASSIGNED group tag,
then adding labels is harder in scripts (must move entries before
setting the tag), but that's not necessary at the moment.
I'm just not sure right now if the generator part of MimeProfile is
prepared for properly skipping entirely empty rows to avoid generating
empty property lines in the vCard output. But that would be fixable as
well.
That part already works.
That's because the code which checks where to store the ADR value does
not check whether the LABEL at the position is available.
There's another problem:
TEL:123456789
ADR;X-ABLabel=labelAdr:xxx
A simplistic check LABEL[0] empty will accept position #0 for ADR. But
doing so will add the labelAdr also to the TEL, which was previously
created at position #0.
That means that the check has to be LABEL[0] does not exist.
Yes. Fortunately, TItemField differentiates between empty and unassigned; the
check needs to be for the latter.
Unassigned is not good enough. If the LABEL array gets extended at the
end (for example, because the third ADR had a X-ABLabel parameter), then
I have unassigned entries at the beginning of the array which must not
be used, because there are already other properties associated with
them.
Here's the patch:
diff --git a/src/sysync/mimedirprofile.cpp b/src/sysync/mimedirprofile.cpp
index 5b0ecc7..3699ad0 100644
--- a/src/sysync/mimedirprofile.cpp
+++ b/src/sysync/mimedirprofile.cpp
@@ -4154,6 +4154,35 @@ bool TMimeDirProfileHandler::parseProperty(
else
dostore = true; // at least one field exists, we might
store
}
+// - check if fields used for parameters are empty
+const TParameterDefinition *paramP = aPropP-parameterDefs;
+while (paramP) {
+ if (mimeModeMatch(paramP-modeDependency)
+#ifndef NO_REMOTE_RULES
+ (!paramP-ruleDependency ||
isActiveRule(paramP-ruleDependency))
+#endif
+ ) {
+if (paramP-convdef.fieldid==FID_NOT_SUPPORTED)
+ continue; // no field, no need to check it
+sInt16 e_fid = paramP-convdef.fieldid /* +baseoffset */;
+sInt16 e_rep = repoffset;
+aItem.adjustFidAndIndex(e_fid,e_rep);
+// - get base field
+TItemField *e_basefldP = aItem.getField(e_fid);
+TItemField *e_fldP = NULL;
+if (e_basefldP)
+ e_fldP=e_basefldP-getArrayField(e_rep,true); // get
leaf field, if it exists
+if (!e_basefldP || e_fldP) {
+ // base field of one of the main fields does not exist
or leaf field is already in use
+ // (unassigned is not good enough, otherwise we might
end up adding information
+ // to some other, previously parsed property using the
same array field)
+ // - skip that repetition
+ dostore=false;
+ break;
+}
+ }
+ paramP=paramP-next;
+}
// check if we can test more repetitions
if (!dostore) {
if (aRepArray[repid]+1maxrep || maxrep==REP_ARRAY) {
I'm undecided now how to proceed. Simplify X-ABLabel property values so
that they can be stored as parameter? Use the more complex X-ABLabel
property and grouping
Re: [os-libsynthesis] vCard group tags
On Tue, 2014-04-29 at 18:00 +0200, Lukas Zeller wrote: Hi Patrick, you are faster in understanding the details than I could look them up :-) Live and learn. Or use the source, Luke ;-) However, it helps to spell out observations and thoughts occasionally, so let me continue with a related problem. I decided that a separate X-ABLabel for every TEL, ADR, X-ABDate, X-ABRELATEDNAMES, etc. is too verbose and too different from traditional usage of vCard. Recipients of such a vCard must be able to understand and support groups, which is not guaranteed. For example, the Evolution vCard parser supports groups, but the higher level of abstraction does not (or not easily). I also don't see what advantage X-ABLabel as property has over X-ABLabel as parameter. I understand that a property could itself have parameters or complex values, but that's not the case here. A simple string might just as well be attached as parameter. So that's what I am trying to do: 1. Parse with groups: X-ABLabel property enabled. 2. Throw away group tags. 3. Generate with parameter: X-ABLabel parameter enabled. This direction works. What fails is reading such a generated vCard, because different properties store their X-ABLabel parameter in the same LABEL field array, overwriting each others' values. The logic for choosing a position must now also (or instead?!) check whether the LABEL position is unused when adding a new ADR. In other words, when parsing an ADR, look at LABEL to determine the position. I thought I could achieve that with: property name=X-ABLabel suppressempty=yes groupfield=GROUP_TAG rule=HAVE-ABLABEL-PROPERTY value field=LABEL repeat=array increment=1 minshow=0/ position field=LABEL repeat=array increment=1 minshow=1/ /property property name=ADR values=7 groupfield=GROUP_TAG value index=0 field=ADR_POBOX/ value index=1 field=ADR_ADDTL/ value index=2 field=ADR_STREET/ value index=3 field=ADR_CITY/ value index=4 field=ADR_REG/ value index=5 field=ADR_ZIP/ value index=6 field=ADR_COUNTRY/ position field=LABEL repeat=array increment=1 minshow=1/ parameter name=TYPE default=yes positional=no show=yes value field=ADR_STREET_FLAGS conversion=multimix combine=, enum name=HOME value=B0/ enum name=WORK value=B1/ enum mode=ignore value=B2/ !-- OTHER -- !-- enum mode=prefix name=X-CustomLabel- value=1.L/ -- !-- enum mode=prefix name=X-Synthesis-Ref value=2.L/ -- /value /parameter parameter name=X-ABLabel rule=HAVE-ABLABEL-PARAMETER value field=LABEL/ /parameter /property However, when parsing a vCard with HAVE-ABLABEL-PARAMETER unset and HAVE-ABLABEL-PROPERTY set, sysync::TMimeDirProfileHandler::parseProperty() goes into an endless loop involving sysync::TMultiFieldItem::adjustFidAndIndex() directly on the first ADR: ADR;TYPE=HOME:PO;neighborhood;home address\n;City;State;ZIP;Country It's permanently increasing repoffset. I've not looked further, but I suspect that this is because of using LABEL twice, once indirectly via the group field and once via the position field. I'll try applying different property elements, even if that means more duplication in the profile. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] vCard group tags
: 1
-- element1 : 2
- 75 : string ADR_STREET_LABEL [ n/a, 0, 0] : unassigned
- 76 :integer ADR_STREET_ID [ n/a, 0, 0] : unassigned
- 77 : multiline ADR_POBOX [ 0, 0, 0] : array with 3 elements
-- element0 : PO
-- element1 : unassigned
-- element2 : unassigned
- 78 : multiline ADR_CITY[ 0, 0, 0] : array with 3 elements
-- element0 : City
-- element1 : unassigned
-- element2 : unassigned
- 79 : multiline ADR_REG [ 0, 0, 0] : array with 3 elements
-- element0 : State
-- element1 : unassigned
-- element2 : unassigned
- 80 : multiline ADR_ZIP [ 0, 0, 0] : array with 3 elements
-- element0 : ZIP
-- element1 : unassigned
-- element2 : unassigned
- 81 : multiline ADR_COUNTRY [ 0, 0, 0] : array with 3 elements
-- element0 : Country
-- element1 : unassigned
-- element2 : unassigned
- 82 : multiline NOTE[ 0, 0,15] : A test contact.
- 83 : string PHOTO [ 0, 0, 516] : �PNG
- 84 : string PHOTO_TYPE [ 0, 0, 0] : empty
- 85 : string PHOTO_VALUE [ 0, 0, 0] : empty
- 86 : string GEO_LAT [ 0, 0, 0] : empty
- 87 : string GEO_LONG[ 0, 0, 0] : empty
- 88 : string CRYPTOENCRYPTPREF [ 0, 0, 0] : empty
- 89 : string CRYPTOPROTOPREF [ 0, 0, 0] : array with 0 elements
- 90 : string CRYPTOSIGNPREF [ 0, 0, 0] : empty
- 91 : string OPENPGPFP [ 0, 0, 0] : empty
- 92 : string XPROPS [ 0, 0, 0] : array with 2 elements
-- element0 :
X-PHONETIC-FIRST-NAME:John
-- element1 :
X-PHONETIC-LAST-NAME:Doe
Do I need to modify the source code to ensure that parameters do not
exist yet?
Similar to for (sInt16 e=0; eaPropP-numValues; e++) { the code would
also need to iterate over aPropP-parameterDefs.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] vCard group tags
On Wed, 2014-04-16 at 17:34 +0200, Lukas Zeller wrote: Hello Patrick, On 11.04.2014, at 12:18, Patrick Ohly patrick.o...@intel.com wrote: Google CardDAV and iOS/OS X use group tags to represent custom labels in vCards. Attached is an example. Note that the Google vCard format used by CardDAV is not the same as the one used by Gmail's export feature. The example is from CardDAV. I am trying to understand how I can support that using libsynthesis. The doc mentions: groupfield [...] Is there an example using this feature? I implemented that feature back at Synthesis in 2010 for Beat's Android client, and what is shown in the docs (the TITLE/ORG case) is in fact taken from that client's config. Is the following field list and profile correct? Field list: field name=TITLE array=yes type=string/ field name=ORG_NAME array=yes type=string/ field name=ORG_DIVISION array=yes type=string/ field name=ORG_OFFICE tarray=yes ype=string/ field name=ORG_TEAM array=yes type=string/ field name=ROLE array=yes type=string/ field name=GROUP-TAG-TITLE-ORG array=yes type=string/ Profile: property name=TITLE grouptag=GROUP-TAG-TITLE-ORG value field=TITLE/ /property property name=ORG values=4 grouptag=GROUP-TAG-TITLE-ORG value index=0 field=ORG_NAME/ value index=1 field=ORG_DIVISION/ value index=2 field=ORG_OFFICE/ value index=3 field=ORG_TEAM/ /property My understanding is that parsing the example from the documentation will lead to: ORG = [ myOwnCompany, myEmployer ] TITLE = [ boss, employee ] GROUP-TAG-TITLE-ORG = [ A, B ] Yes, that's excactly what is supposed to happen. Unfortunately I am not getting the group-tag field populated. I'm attaching my field list and profile. When parsing, I get: [2014-04-28 19:31:37.575] Parsing: * [2014-04-28 19:31:37.575] BEGIN:VCARD VERSION:3.0 N:Doe;John;1;Mr.;Sr. FN:Mr. John 1 Doe Sr. NICKNAME:Johnny TITLE:tester ORG:at company REV:2014-04-28T17:31:23Z UID:6a5bd79f8b1cdd58 BDAY;VALUE=DATE:1970-12-30 ADR;TYPE=HOME:PO;neighborhood;home address\n;City;State;ZIP;Country ADR;TYPE=WORK:;;work address ADR;TYPE=OTHER:;;custom address TEL;TYPE=WORK:business 1 TEL;TYPE=CELL:mobile TEL;TYPE=HOME:home TEL:main TEL;TYPE=FAX,WORK:work fax TEL;TYPE=FAX,HOME:home fax item13.TEL:google voice TEL;TYPE=PAGER:pager item14.TEL:custom EMAIL;TYPE=HOME,PREF:john@home.com EMAIL;TYPE=WORK:d...@work.com item1.EMAIL:j...@custom.com NOTE:A test contact. PHOTO;ENCODING=B:iVBORw0KGgoNSUhEUgAAACQXCAYAAABj7u2bBmJLR0QA/w D/AP+gvaeTCXBIWXMAAAsTAAALEwEAmpwYB3RJTUUH1gEICjgdiWkBOQAAAB10RVh0Q 29tbWVudABDcmVhdGVkIHdpdGggVGhlIEdJTVDvZCVuAAABaElEQVRIx+3Wu0tcURAG8F98gRKT YGORRqwksJV/QOqFFIFgKgsRYbHV1larDQQCKQxpUscyhUmXJuCSNpYWPsAU6wPxHW6aWbgsu+v e3RUs7geHc+fON3O+M4c5HHLkyHG/eISkg5heIGmUr++hVWigyY6THlejbWSt0Bv8QBXX2MF7jK U4IyjjJ45xg31sYKZuw7Xv9Gh6vvXO9QbBtbGNJ8Ert+AlTURkFjQX9g5e4ykGUcBm+FaDexx2M UQOYhIL2Lpj09oV9CvsQgPuePj+hP037BL6M6yRSdDZHWVOcBHcEv7FvyN8xxqmeynovA1Baf4U VvANhyn/Uq8E/Q57ssNufhvx1QZrDHfS9p9i3sQsnscdNowXWEQlOBXMYyI4j3EavqFUzpOYl4O TqUJ9+NzmkbXyb6Ryfumm7Wso4it2cYXL6K6PeBmcV8E5iEvxPDjv8CyVaxQfsIfbqGIlf17k6B b/Ae0cnahfg6KuAElFTkSuQmCC item4.IMPP;X-SERVICE-TYPE=GoogleTalk:xmpp:google%20talk item5.IMPP;X-SERVICE-TYPE=AIM:aim:aim item6.IMPP;X-SERVICE-TYPE=Yahoo:ymsgr:yahoo item7.IMPP;X-SERVICE-TYPE=Skype:skype:skype item8.IMPP;X-SERVICE-TYPE=QQ:x-apple:QQ item9.IMPP;X-SERVICE-TYPE=MSN:msnim:MSN item10.IMPP;X-SERVICE-TYPE=ICQ:aim:ICQ item11.IMPP;X-SERVICE-TYPE=Jabber:xmpp:Jabber item12.IMPP;X-SERVICE-TYPE=Chat-label:x-apple:custom%20chat X-EVOLUTION-FILE-AS:Doe\, John X-MOZILLA-HTML:FALSE X-ABLABEL:custom-label X-PHONETIC-FIRST-NAME:John X-PHONETIC-LAST-NAME:Doe item1.X-ABLabel:custom-label item2.X-ABDATE:1971-01-01 item2.X-ABLabel:Anniversary item3.X-ABDATE:2000-02-01 item3.X-ABLabel:custom-label item4.X-ABLabel:Other item5.X-ABLabel:Other item6.X-ABLabel:Other item7.X-ABLabel:Other item8.X-ABLabel:Other item9.X-ABLabel:Other item10.X-ABLabel:Other item11.X-ABLabel:Other item12.X-ABLabel:Other item13.X-ABLabel:Google Voice item14.X-ABLabel:custom-label item15.X-ABRELATEDNAMES:spouse item15.X-ABLabel:Spouse item16.X-ABRELATEDNAMES:child item16.X-ABLabel:Child item17.X-ABRELATEDNAMES:mother item17.X-ABLabel:Mother item18.X-ABRELATEDNAMES:father item18.X-ABLabel:Father item19.X-ABRELATEDNAMES:parent item19.X-ABLabel:Parent item20.X-ABRELATEDNAMES:brother item20.X-ABLabel:Brother item21.X-ABRELATEDNAMES:sister item21.X-ABLabel:Sister item22.X-ABRELATEDNAMES:friend item22.X-ABLabel:Friend item23.X-ABRELATEDNAMES:relative item23.X-ABLabel:relative item24.X-ABRELATEDNAMES:manager item24.X-ABLabel:Manager item25.X-ABRELATEDNAMES:assistant item25.X-ABLabel:Assistant item26.X-ABRELATEDNAMES:referred-by item26.X-ABLabel:referred by item27.X-ABRELATEDNAMES:partner item27.X
Re: [os-libsynthesis] vCard group tags
: Country -- element1 : unassigned -- element2 : unassigned - 82 : multiline NOTE[ 0, 0,15] : A test contact. - 83 : string PHOTO [ 0, 0, 516] : �PNG - 84 : string PHOTO_TYPE [ 0, 0, 0] : empty - 85 : string PHOTO_VALUE [ 0, 0, 0] : empty - 86 : string GEO_LAT [ 0, 0, 0] : empty - 87 : string GEO_LONG[ 0, 0, 0] : empty - 88 : string CRYPTOENCRYPTPREF [ 0, 0, 0] : empty - 89 : string CRYPTOPROTOPREF [ 0, 0, 0] : array with 0 elements - 90 : string CRYPTOSIGNPREF [ 0, 0, 0] : empty - 91 : string OPENPGPFP [ 0, 0, 0] : empty - 92 : string XPROPS [ 0, 0, 0] : array with 2 elements -- element0 : X-PHONETIC-FIRST-NAME:John -- element1 : X-PHONETIC-LAST-NAME:Doe -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. 00vcard-fieldlist.xml Description: XML document 01vcard-profile.xml Description: XML document ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] vCard group tags
On Tue, 2014-04-29 at 09:50 +0200, Patrick Ohly wrote:
On Mon, 2014-04-28 at 23:18 +0200, Lukas Zeller wrote:
Hi Patrick,
Unfortunately I am not getting the group-tag field populated. I'm
attaching my field list and profile.
small oversight on my and your part :-)
The attribute to be added to property is called groupfield, not
grouptag...
I fixed that (see attached profile + field list). It fills the GROUP_TAG
array now, but the LABEL array only has one entry, apparently the last
one. The array positions also don't match:
item34.URL:http\://custom.com
item34.X-ABLabel:Custom-label
- 3 : string GROUP_TAG [ 0, n/a, 0] : array with 19
elements
-- element0 : item34
- 4 : string LABEL [ 0, 0, 0] : array with 1 elements
-- element0 : Custom-label
- 34 : string WEB [ 0, 0, 0] : array with 19
elements
-- element0 : unassigned
...
-- element 18 : http://custom.com;
I've stepped through TMimeDirProfileHandler::parseProperty() when
parsing a simpler example:
BEGIN:VCARD
VERSION:3.0
N:Doe;John;1;Mr.;Sr.
FN:Mr. John 1 Doe Sr.
UID:6f354d698b7ccd22
item1.URL:http\://company.com
item1.X-ABLabel:Work
item2.URL:http\://custom.com
item2.X-ABLabel:Custom-label
END:VCARD
When parsing item1.X-ABLabel:Work,
TMimeDirProfileHandler::parseProperty() immediately skips over the
parameter parsing because there is none and fieldoffsetfound was set to
true in
fieldoffsetfound = (aPropP-nameExts==NULL); // no first pass needed at all
w/o nameExts, just use offs=0
It then has repoffset == 0 when storing the group tag:
// parameters are all processed by now, decision made to store data (if
!dostore, routine exits above)
// - store the group tag value if we have one
if (aPropP-groupFieldID!=FID_NOT_SUPPORTED) {
TItemField *g_fldP =
aItem.getArrayFieldAdjusted(aPropP-groupFieldID+baseoffset,repoffset,false);
if (g_fldP)
g_fldP-setAsString(aGroupName,aGroupNameLen); // store the group name
(aGroupName might be NULL, that's ok)
}
This happens to be correct (accidentally) for item1.X-ABLabel:Work.
When repeating this for item2.X-ABLabel, the setAsString() above
overwrites item1 with item2 at offset 0, leading to:
- 3 : string GROUP_TAG [ 0, n/a, 0] : array with 2
elements
-- element0 : item2
-- element1 : item2
The second element here came from item2.URL.
The group tag matching is buried in the parameter parsing loop. Skipping
it via the (aPropP-nameExts==NULL) is incorrect if the property has a
group name.
I'm not sure what the correct way of executing that code in this
particular case is. The following patch does not work:
diff --git a/src/sysync/mimedirprofile.cpp b/src/sysync/mimedirprofile.cpp
index 5b0ecc7..16a5aac 100644
--- a/src/sysync/mimedirprofile.cpp
+++ b/src/sysync/mimedirprofile.cpp
@@ -3835,7 +3835,7 @@ bool TMimeDirProfileHandler::parseProperty(
encoding = enc_none; // no encoding by default
charset = aMimeMode==mimo_standard ? chs_utf8 : fDefaultInCharset; // always
UTF8 for real MIME-DIR (same as enclosing SyncML doc), for mimo_old depends on
inputcharset remote rule option (normally UTF-8)
nameextmap = 0; // no name extensions detected so far
- fieldoffsetfound = (aPropP-nameExts==NULL); // no first pass needed at all
w/o nameExts, just use offs=0
+ fieldoffsetfound = (aPropP-nameExts==NULL) !aGroupNameLen; // no first
pass needed at all w/o nameExts, just use offs=0, except when we have to match
a group name
valuelist = aPropP-valuelist; // cache flag
// prepare storage as unprocessed value
if (aPropP-unprocessed) {
It loops twice, but still never reaches the group name matching because
that is inside a while (propnameextP) loop that never gets entered.
Should that code be copied out for the simpler case of a property with
no aPropP-nameExts?
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] vCard group tags
On Tue, 2014-04-29 at 17:18 +0200, Patrick Ohly wrote:
On Tue, 2014-04-29 at 09:50 +0200, Patrick Ohly wrote:
On Mon, 2014-04-28 at 23:18 +0200, Lukas Zeller wrote:
Hi Patrick,
Unfortunately I am not getting the group-tag field populated. I'm
attaching my field list and profile.
small oversight on my and your part :-)
The attribute to be added to property is called groupfield, not
grouptag...
I fixed that (see attached profile + field list). It fills the GROUP_TAG
array now, but the LABEL array only has one entry, apparently the last
one. The array positions also don't match:
item34.URL:http\://custom.com
item34.X-ABLabel:Custom-label
- 3 : string GROUP_TAG [ 0, n/a, 0] : array with 19
elements
-- element0 : item34
- 4 : string LABEL [ 0, 0, 0] : array with 1
elements
-- element0 : Custom-label
- 34 : string WEB [ 0, 0, 0] : array with 19
elements
-- element0 : unassigned
...
-- element 18 : http://custom.com;
I've stepped through TMimeDirProfileHandler::parseProperty() when
parsing a simpler example:
BEGIN:VCARD
VERSION:3.0
N:Doe;John;1;Mr.;Sr.
FN:Mr. John 1 Doe Sr.
UID:6f354d698b7ccd22
item1.URL:http\://company.com
item1.X-ABLabel:Work
item2.URL:http\://custom.com
item2.X-ABLabel:Custom-label
END:VCARD
When parsing item1.X-ABLabel:Work,
TMimeDirProfileHandler::parseProperty() immediately skips over the
parameter parsing because there is none and fieldoffsetfound was set to
true in
fieldoffsetfound = (aPropP-nameExts==NULL); // no first pass needed at
all w/o nameExts, just use offs=0
It then has repoffset == 0 when storing the group tag:
// parameters are all processed by now, decision made to store data (if
!dostore, routine exits above)
// - store the group tag value if we have one
if (aPropP-groupFieldID!=FID_NOT_SUPPORTED) {
TItemField *g_fldP =
aItem.getArrayFieldAdjusted(aPropP-groupFieldID+baseoffset,repoffset,false);
if (g_fldP)
g_fldP-setAsString(aGroupName,aGroupNameLen); // store the group name
(aGroupName might be NULL, that's ok)
}
This happens to be correct (accidentally) for item1.X-ABLabel:Work.
Looking further at aPropP-nameExts it seems that this what relates to
position in the profile config. I did not have that for my X-ABLabel.
Adding it seems to fix the problem. So it seems that groupfield can
only be used reliably with a property which has a position, correct?
property name=X-ABLabel suppressempty=yes groupfield=GROUP_TAG
value field=LABEL repeat=array increment=1 minshow=0/
position field=LABEL repeat=array increment=1 minshow=1/
/property
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] SyncML Server Performance using Syncevolution
On Tue, 2014-04-29 at 23:22 +0530, Sachin Gupta wrote: one more thing. everytime i invoke syncevolution, it has to load all the xmls, build up its data structures, and all the other stuff. Can i implement threads in it and run it as a daemon. This will i can save time from doing all this. From within, can i then implement threads and launch multiple sync operations? Multithreading is not going to work. You should be able to run a sync repeatedly in the same process, but reusing the XML loading will require putting the loop fairly deeply into the SyncEvolution stack (see SyncContext.cpp). I'm still not sure why you want to do this when you can simply use more client machines. It might safe you some time for setting up a cluster if (and only if!) you can manage to run all clients from the same machine, but I have my doubts whether that will be possible. -- Best Regards Patrick Ohly Senior Software Engineer Intel GmbH Open Source Technology Center Usenerstr. 5a Phone: +49-228-2493652 53129 Bonn Germany ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] Duplication of data during a merge scenario
On Mon, 2014-04-28 at 18:23 +0530, Rajesh Kumar Pawar wrote: I've been trying to work on a merging scenario using syncevolution as a client. The steps that I followed in this scenario are as follows 1. Created a contact with TEL;CELL:12456399 2. Perform slow sync with server 3. Modify the field in client vCard as TEL;CELL:12456300 4. Wait for 5min to ensure sufficient difference between the modification time stamps. 5. Modified the field as TEL;CELL:12456391 on server. 6. Perform a two-way sync. In this scenario, the server decides how to handle the conflict. What SyncML implementation are you using as server? You only mention that you use SyncEvolution as client. If the server is not SyncEvolution or libsynthesis based, then you need to talk to the server implementer; there's not much that SyncEvolution or libsynthesis can do. At least not when sticking to a standard SyncML sync. There was an idea a while back to do a SyncML sync in two phases: in the first phase pretend that the client has no changes, and just retrieve changes from the server. This way conflict resolution can be done in the client. Then in the second phase, send the client changes (including merge results) back to the server, hoping that no further changes had been made there in the meantime. This never got implemented, though. In SyncEvolution there is a similar (but not quite the same) data loss issue when adding an address on one side and a telephone number on the other side. I'm currently looking at that. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] SyncML Server Performance using Syncevolution
On Sat, 2014-04-26 at 23:42 +0530, Sachin Gupta wrote: On Fri, Apr 25, 2014 at 3:29 PM, Patrick Ohly patrick.o...@intel.com wrote: On Fri, 2014-04-25 at 15:03 +0530, Sachin Gupta wrote: Can you suggest how i can test SyncML Server performance and have 2500 users/syncevolutions connecting simultaneously? There's no ready-made solution. You'll have to write your own scripts for configuring SyncEvolution and running the desired benchmark. Note that each context in SyncEvolution gets its own device ID. So if you want to simulate n different devices, use: syncevolution --configure ... client-1@client-1 ... syncevolution --configure ... client-n@client-n I figured so. So wrote scripts which will launch syncevo each with unique device ids and seperate user accounts. But the concern is managing these number of syncs through a time period. Exploring if JMETER can help me out in this.\ Also being a process, it would not be possible to launch so many processes in parallel. Memory and CPU would be issues, right? Would need very high end systems for this? I have not measured this. Try it and you'll see. My expection is that you will need multiple client machines, though. Would it be possible doing this launching processes or shall i look into creating threads within the Syncevolution launching and controlling sessions from there? I would just use multiple client machines. Much simpler and scales perfectly. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] mapping between individual X- extensions and grouped extensions
On Sat, 2014-04-26 at 12:30 +0200, Lukas Zeller wrote: The afterread/beforewrite script could do such a conversion as well, however for normalizing data these are executed too late on the server side for normalized data to be used in slow sync matching, so it'll be more complicated to correctly match and merge records. What do you mean with that? The afterread script gets called after reading and before using the field list, right? So whatever transformation is necessary (for example, X-JABBER - IMPP) can be done in time before the engine processes the IMPP field. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] SyncML Server Performance using Syncevolution
On Fri, 2014-04-25 at 15:03 +0530, Sachin Gupta wrote: Hi Lukas/Patrick, I have to run some test cases using syncevolution as client to test a SyncML server performance. Expectation is to put load of around 2500 users syncing concurrently for an hour. That's a bit vague. What kind of changes are supposed to be synced during that hour? The load will depend a lot on that. At the low end you just have 2500 users connecting repeatedly without data changing on client or server. The high end is open-ended; lots of items and slow syncs will be more expensive than few items and incremental changes. Can you suggest how i can test SyncML Server performance and have 2500 users/syncevolutions connecting simultaneously? There's no ready-made solution. You'll have to write your own scripts for configuring SyncEvolution and running the desired benchmark. Note that each context in SyncEvolution gets its own device ID. So if you want to simulate n different devices, use: syncevolution --configure ... client-1@client-1 ... syncevolution --configure ... client-n@client-n -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] mapping between individual X- extensions and grouped extensions
Hello! I am trying to figure out how I can translate between Evolution/SyncML vCard format and CardDAV (in particular, Google Contacts). Google Contacts seems to follow the format as used by Apple, which is not surprising considering that Apple iOS and OS X are probably the main clients using CardDAV. The key difference is that CardDAV uses IMPP (rfc 4770) for all kinds of chat protocols, whereas Evolution and several SyncML implementations use one kind of X- extension per protocol (X-AIM, X-JABBER, ...). Evolution: X-JABBER;TYPE=HOME;X-EVOLUTION-UI-SLOT=2:JABBER DOE Google CardDAV: item5.IMPP;X-SERVICE-TYPE=GoogleTalk:xmpp:google%20talk item5.X-ABLabel:Other Note that X-SERVICE-TYPE=GoogleTalk is the provider of the service. xmpp: is part of the URL value and defines the chat protocol used by the service; the rest of the value uses URL encoding of special characters (a space in this case). The label is just that, a label chosen by the user. Evolution doesn't have support for custom labels, so there is nothing exactly corresponding to X-ABLabel yet. TYPE comes close. Same for some dates (X-EVOLUTION-ANNIVERSARY vs. X-ABDate): Evolution: X-EVOLUTION-ANNIVERSARY:2006-01-09 Google CardDAV: item3.X-ABDATE:1971-01-01 item3.X-ABLabel:Anniversary item4.X-ABDATE:2000-02-01 item4.X-ABLabel:custom-label The exception is BDAY, which is used by both sides. Finally, the same difference exists for related persons: Evolution: X-EVOLUTION-SPOUSE:Joan Doe Google CardDAV: item16.X-ABRELATEDNAMES:spouse item16.X-ABLabel:Spouse At the moment, SyncEvolution uses separate fields for AIM (AIM_HANDLE, an array) and spouse (a single value). I'm leaning towards changing that into a more generic field list where there is one array for IMPP, one array for DATE, and one array for related NAMES. I think that can be mapped fairly directly to the vCard format used by Google CardDAV. For example: field name=IMPP array=yes type=string compare=conflict/ !-- includes protocol: xmpp:google%20talk -- field name=IMPP_SERVICE array=yes type=string compare=conflict/ !-- GoogleTalk -- field name=IMPP_LABEL array=yes type=string compare=conflict/ !-- Other -- field name=GROUP-TAG-IMPP array=yes type=string compare=conflict/ !-- group tag connecting IMPP_LABEL = X-ABLabel with IMPP and IMPP_SERVICE -- The bigger problem will be on the Evolution side. I don't see how I can teach libsynthesis that a IMPP entry whose protocol (encoded as part of the value!) is xmpp maps to X-JABBER. Should I keep the traditional JABBER_HANDLE array and move entries back and forth between it and the IMPP array? This could be done with incoming/outgoing resp. afterread/beforewrite scripts. Then the traditional profile will only use the JABBER_HANDLE, as before, whereas the new profile only uses IMPP. For SyncEvolution-SyncEvolution syncing the new profile should be used because it will be more complete. Perhaps I can achieve that by offering multiple datatypes and then letting the sync engines negotiate the most suitable one. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] UID + CardDAV
On Fri, 2014-04-11 at 16:52 +0200, Patrick Ohly wrote: [property name=UID rule=HAVE-VCARD-UID shows up in CtCap] Could it be a bug that the disable property shows up in the CtCap? Darn, probably a circular dependency again: we have to be ready to send CtCap before we know the peer, so the rule mechanism (which depends on knowing the peer) can't be used to influence the CtCap. Right? A closer look at the CtCap seems to confirm that. For example, SyncEvolution knows that KDE uses X-KADDRESSBOOK-X-Profession instead of ROLE, so it has: !-- X-KADDRESSBOOK-X-Profession property for KDE, ROLE for everyone else -- property name=ROLE rule=KDE/ property name=ROLE rule=other value field=ROLE/ /property property name=X-KADDRESSBOOK-X-Profession suppressempty=yes show=no rule=KDE value field=ROLE/ /property This is a somewhat convoluted way of saying that vCard ROLE is not active for KDE (no value specied) and that X-KADDRESSBOOK-X-Profession is to be used instead. In the CtCap, both properties show up: PropertyPropNameROLE/PropNameMaxOccur1/MaxOccur/Property PropertyPropNameX-KADDRESSBOOK-X-Profession/PropNameMaxOccur1/MaxOccur/Property This means that I can't put properties that are never meant to be sent over SyncML into the profile used for syncing. I need to maintain a separate profile with UID enabled that gets used for MAKETEXTWITHPROFILE/PARSETEXTWITHPROFILE. I wonder whether it makes sense to have X-KADDRESSBOOK-X-Profession appear in the CtCap. Probably not, because SyncEvolution will not use it during syncing, only during storing in the datastore. I probably need to change the way how SyncEvolution adapts the main profile for specific storages: instead of setting rules in MAKETEXTWITHPROFILE or PARSETEXTWITHPROFILE, it needs to do its own custom pre-processing to turn the main profile into a KDE profile or an EDS profile. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] vCard group tags
Hello! Google CardDAV and iOS/OS X use group tags to represent custom labels in vCards. Attached is an example. Note that the Google vCard format used by CardDAV is not the same as the one used by Gmail's export feature. The example is from CardDAV. I am trying to understand how I can support that using libsynthesis. The doc mentions: groupfield New in 3.4: This optional attribute can be used to specify a string field (usually a array or a repeating field) which represents the vCard or vCalendar group tag (a prefix to the property name, separated by a dot). The group tag can be used to link properties together which can occur multiple times. For example, some vCards might contain more than one ORG and TITLE. Now each title belongs to a particular organisation, so the group tag is used to represent that: A.ORG:myOwnCompany B.ORG:myEmployer B.TITLE:employee A.TITLE:boss The groupfield mechanism makes sure that TITLE and ORG repetitions will be stored in the same repetition index (array position if ORG and TITLE are mapped to arrays) according to their group tag, even if occuring out of order in the incoming vCard. Is there an example using this feature? Is the following field list and profile correct? Field list: field name=TITLE array=yes type=string/ field name=ORG_NAME array=yes type=string/ field name=ORG_DIVISION array=yes type=string/ field name=ORG_OFFICE tarray=yes ype=string/ field name=ORG_TEAM array=yes type=string/ field name=ROLE array=yes type=string/ field name=GROUP-TAG-TITLE-ORG array=yes type=string/ Profile: property name=TITLE grouptag=GROUP-TAG-TITLE-ORG value field=TITLE/ /property property name=ORG values=4 grouptag=GROUP-TAG-TITLE-ORG value index=0 field=ORG_NAME/ value index=1 field=ORG_DIVISION/ value index=2 field=ORG_OFFICE/ value index=3 field=ORG_TEAM/ /property My understanding is that parsing the example from the documentation will lead to: ORG = [ myOwnCompany, myEmployer ] TITLE = [ boss, employee ] GROUP-TAG-TITLE-ORG = [ A, B ] The challenge will be to convert between item6.IMPP;X-SERVICE-TYPE=AIM:aim:aim item6.X-ABLabel:Other and X-AIM:aim Custom labels will also be fun, if that's what Evolution decides to use (currently it is unsupported). -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. BEGIN:VCARD VERSION:3.0 N:Doe;John;1;Mr.;Sr. FN:Mr. John 1 Doe Sr. NICKNAME:Johnny TITLE:tester ORG:at company REV:2014-04-11T08:43:42Z UID:6f354d698b7ccd22 BDAY;VALUE=DATE:1970-12-30 ADR;TYPE=HOME:PO;neighborhood;home address\n;City;State;ZIP;Country ADR;TYPE=WORK:;;work address item1.ADR:;;custom address TEL;TYPE=WORK:business 1 TEL;TYPE=CELL:mobile TEL;TYPE=HOME:home TEL:main TEL;TYPE=FAX,WORK:work fax TEL;TYPE=FAX,HOME:home fax item14.TEL:google voice TEL;TYPE=PAGER:pager item15.TEL:custom EMAIL;TYPE=HOME,PREF:john@home.com EMAIL;TYPE=WORK:d...@work.com item2.EMAIL:j...@custom.com NOTE:A test contact. PHOTO;ENCODING=B:/9j/4AAQSkZJRgABAQAAAQABAAD/4QDSRXhpZgAASUkqAAgCADEBAg AHJgAAAGmHBAABLgBQaWNhc2EAAAIAAJAHAAQwMjIwhpIHAH0AAABMA AAAICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA gICAgICAgICAgICAgICAgICAgAP/bAIQAAwICCAgICAgICAgICAgICAgICAgICAgICAgICAgIBw gIBwcHBwcHBwcHBwcHCgcHBwgJCQkHBwwMCggMBwgJCAEDBAQGBQYKBgYKDQwMDA0MDAwMDAgIC AgIDAgIDAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgI/8AAEQgCgAKAAwEiAAIR AQMRAf/EAB0AAAICAwEBAQIDBAUBBgcACAn/xABFEAACAQMDAwIEBAQEBAMHBQE AAQIDBBEFITEGEkFRYQcTInEIMoGRFEKhsSNSwfAVYtHhM3LxFhgkQ0SCkgk0U2PSF//EABsBAA IDAQEBAAACAQMEBQYH/8QAKxEBAQACAQQDAAICAQQDAAECEQMEEiExBRNBI lEyYRQjQoGxUpGh/9oADAMBAAIRAxEAPwD5IyEjOAkj1rmMIZFHkgoocMoykZSCigKzBDFEwojH EEUAaPYMpExErMUGonohJDDbyPSQUUZcQGwxQ1I9BBpAXYcBJBJBKIDbBlIJRM9oxWIoOKMxiMh EAxFBRRlRDjEDb09FDInoxCSDRbRQPJYCiYk/ATwJXnPJ6nAFQwZVX6sDWoo0tsHp1CJ/E/U0Rq txnZC2T2skXtlVzBnu/CyI0uf0uOdyJe3vY8N7Ln7Fdz0jVWDnwZUe2e7xng1266uoucYKaxjd5 Nc1L4kU4N9zz2vZ+wt5sf03ZXSatfH3FT1KK5a/7nGKvxpeW8ZXCNX6h+JlSrtH6UZ8+q456XTp 9u+aj1nSpLum1+5RT+Mdo0/D+5843GqVpbym5L38EaKT3Zhz62z0unTvoH//ALPa5xLj1JVP42W 8WllYfufOPYm8I9s+FuimdfyG+h9SaD8VqEqm81hm+2+vUJ7xqxa+58QUrlLjZr3LGy6krQbcak kvuaMevv6S8D7dtLiEt4tE7HufGmkfE+6pbxm2l4Z0rRPxEvEfnRx9tzbxdbjldVVl076CUMLKe 56Fxl5xn1OSW/x1t6jSb7Y+pc1/iDSh2zp1FKL533Nc6nCqvprplX8ufPgfaSTjnG65OZ6L8VaN RuDlzsmb1ouqR7eUWTmwpbx1bSiNoLBFp105LcbVvI96SaLcbFdx0mxQyKFjEWUWsNBRR7AcYi7
Re: [os-libsynthesis] Photo Data getting truncated
On Wed, 2014-04-09 at 19:13 +0530, anuj chauhan wrote: Hi Patrick, I am trying to syncronize image data with funambol server but while sending the data to server syncevolution is truncating the image data to somewhere 200-300 bytes.The actual image data was 2423 bytes. Below is snippet of what am trying to send(length of image data is 2423 bytes) : BEGIN:VCARD N:lastname3;firstname3;Kumar;Mr;Phd EMAIL;WORK;INTERNET:wo...@hotmail.com EMAIL;HOME;INTERNET:ho...@hotmail.com URL:http://www.care1.com ADR;HOME:;;Caprio;Demalio;Romania;RD3258;USA ADR;WORK:;;Caprio;Demalio;Romania;RD3258;USA PHOTO;ENCODING=b;TYPE=JPEG:/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAkGBxMTEhQUExQWFRQXGBUUFxUUFxQUFRUUFBQXFhUVFBQYHCggGBwlHBQUITEhJSkrLi4uFx8zODMsNygtLisBCgoKDg0OGhAQGywkHxwsLCwsLCwsLCwsLCwsLCwsLCwsLCwsLCwsLCwsLCwsLCwsLCwsLCwsLCwsLCwsLCwsLP/AABEIALcBEwMBIgACEQEDEQH/xAAcAAABB...remainder of B encoded binary data... END:VCARD There is no VERSION:3.0 in this example. Are you sure that the engine is parsing it as vCard 3.0? Have you checked that base64 decoding worked as expected? Look at the field list dump at loglevel=4. If all of that worked, can you attach the full test data? -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] How to synchronise image with server
On Mon, 2014-04-07 at 18:00 +0530, Anuj wrote: Hi Patrick, Thanks for replying , I have a doubt on where to define value =uri In mime profile or somewhere else.please specify. Please follow http://www.ietf.org/rfc/rfc2426.txt for formatting your PHOTO data. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] How to synchronise image with server
On Fri, 2014-04-04 at 05:28 -0500, anuj chauhan wrote: Hi Patrick, I created a photo field as mentioned on https://tools.ietf.org/html/rfc6350#page-30.Below is sample for that PHOTO:file:///home/droot/evol_store1/photos/Blue hills.jpg. but when i tracked the outgoing log I found that syncevolution has converted the string file:///home/droot/evol_store1/photos/Blue hills.jpg. into base64 format and sent the url for syncronization, A full log at loglevel=4 would be useful in cases like this. However, after looking at 04vcard-photo-inlining.xml I have a hunch: you don't specify VALUE=uri explicitly, and the script checks for that. That follows http://www.ietf.org/rfc/rfc2426.txt which says The default is binary value. You used vCard 4.0, which is not used or supported by SyncEvolution. vCard 4.0 has a different definition of the PHOTO property (value is always a URI and inline data is represented as a special URI). -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] How to synchronise image with server
On Wed, 2014-04-02 at 13:32 +0530, Anuj wrote: Hi , I want to synchronise image with server but have no clue of its representation in vCard.Do I need to put URL of image in vCard photo field or entire image data need to be put in vCard.kindly guide me on this issue I am using file based backend for syncevolution. You can either base64 encode the photo data as specified by https://tools.ietf.org/html/rfc6350#page-30 or you can use a file URL. SyncEvolution then will read the file data and inline it before sending to the server. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] How to build a installation package for syncevoltion client for ubuntu
On Tue, 2014-03-04 at 16:33 +0530, anuj chauhan wrote: w.r.t previous mail, i want to ask that how do i build a .deb package from the source compiled locally? I can see that there are options to make or make install, but could not find any command to build a .deb package. normally there are options to build a package using buildpkg command, but i could not find the same here. It is not normal for autotool's based projects to have make targets for packaging in distro-specific formats like .rpm or .deb. This is something that distros add when taking upstream source from a git repo or source tar balls. In the case of SyncEvolution, there is a make deb target. It's meant to be used as part of the nightly testing and depends on CheckInstall being installed. Your mileage may vary. You are probably better off learning how to build a .deb package and doing that the normal way - see Debian HOWTOs. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] how to configure libsynthesis syncronizing a group of contactcs.
On Mon, 2014-01-27 at 07:16 -0600, anuj chauhan wrote: I want to synchronize a group of contacts .Is there any group-filed tag in libsynthesis for contacts.or any subsection in xml configuraton for group-synchronization.I am completly blank on how to achieve this through synthesis. Defining a group of contacts is not standardized in vCard. If you want to synchronize it, you probably have at least one system where groups of contacts are supported. Which system is that and how does it exchange a group? -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] PHOTO + compare=never
On Fri, 2013-07-05 at 14:05 +0200, Lukas Zeller wrote: On 05.07.2013, at 11:58, Patrick Ohly patrick.o...@intel.com wrote: [...] I broke MERGEFIELDS() when introducing that mode, see attached patch. It is now still not backward-compatible. Is there a way to have a builtin function with an optional parameter? There is, see the OPTVAL() in the param definitions, for example param_find in scriptcontext.cpp. In the function implementation, you can test for isAssigned() to see if the parameter was specified in the call or not. I've implemented that now in the patch that'll appear in the FDO master git branch soon. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] temporary local ID + FinalizeLocalID
On Wed, 2013-06-26 at 16:10 +0200, Patrick Ohly wrote:
On Mon, 2013-06-10 at 12:35 +0200, Lukas Zeller wrote:
On 10.06.2013, at 11:57, Patrick Ohly patrick.o...@intel.com wrote:
I'll probably try something else: if commands were delayed in a
message which is marked Final, then force execution of the commands at
the end of processing that message and issue their Status commands, just
as syncing normally would.
Sounds like a good compromise to me.
I've implemented that. I've run this through a full test against
different peers, without problems. Or rather, there were problems
earlier, which shows that the tests covered relevant corner cases ;-)
These problems have been addressed.
After adding more tests, I found one more problem: the combination of
Moredata and queuing preceding items failed. Patch attached. I intend
to squash it into the preserve status order patch before including FDO
master branch.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
From 1f01b69ee1aa597924f90c48a299db254a0cd5d9 Mon Sep 17 00:00:00 2001
From: Patrick Ohly patrick.o...@intel.com
Date: Mon, 1 Jul 2013 08:52:47 +0200
Subject: [PATCH 1/4] merge: ordered status
Add/Replace commands which contain incomplete items are special: they
must trigger a Status 213 response in the same message, which cannot
be done without processing the previously queued commands first.
We could try to queue the incomplete command without processing it at
all, but that would change the response in a way that might be
unexpected by the peer (although legal). Better play it safe and
finish all queued commands, then process the command.
---
src/sysync/synccommand.cpp | 18 ++
src/sysync/synccommand.h |4
src/sysync/syncsession.cpp |7 +++
3 files changed, 29 insertions(+)
diff --git a/src/sysync/synccommand.cpp b/src/sysync/synccommand.cpp
index ef363d1..3b27d63 100755
--- a/src/sysync/synccommand.cpp
+++ b/src/sysync/synccommand.cpp
@@ -2289,6 +2289,24 @@ missingeoc:
} // TSyncOpCommand::AddNextChunk
+// SyncOp commands can execute out of order except when they
+// contain chunked items, because then we would have to issue
+// a 213 Status immediately, which would violate the ordering
+// of Status replies.
+bool TSyncOpCommand::canExecuteOutOfOrder()
+{
+ SmlItemListPtr_t *itemnodePP=(fSyncOpElementP-itemList);
+ while (*itemnodePP) {
+SmlItemListPtr_t thisitemnode = *itemnodePP;
+if (thisitemnode-item
+thisitemnode-item-flags SmlMoreData_f) {
+ return false;
+}
+itemnodePP = (thisitemnode-next);
+ }
+ return true;
+}
+
// execute command (perform real actions, generate status)
// returns true if command has executed and can be deleted
bool TSyncOpCommand::execute(void)
diff --git a/src/sysync/synccommand.h b/src/sysync/synccommand.h
index bc5869c..c070905 100755
--- a/src/sysync/synccommand.h
+++ b/src/sysync/synccommand.h
@@ -90,6 +90,9 @@ public:
#endif
// - analyze command (but do not yet execute)
virtual bool analyze(TPackageStates aPackageState) { fPackageState=aPackageState; return true; }; // returns false if command is bad and cannot be executed
+ // - execute() can be called even if there are other already queued commands.
+ // True by default, exceptions must be defined explicitly.
+ virtual bool canExecuteOutOfOrder() { return true; }
// - execute command (perform real actions, generate status)
virtual bool execute(void); // returns true if command could execute, false if it must be queued for later finishing (next message)
// - get number of bytes that will be still available in the workspace after
@@ -413,6 +416,7 @@ public:
virtual ~TSyncOpCommand();
virtual bool isSyncOp() { return true; };
virtual bool analyze(TPackageStates aPackageState);
+ virtual bool canExecuteOutOfOrder();
virtual bool execute(void);
#ifndef USE_SML_EVALUATION
// - get (approximated) message size required for sending it
diff --git a/src/sysync/syncsession.cpp b/src/sysync/syncsession.cpp
index 870342a..a394bed 100644
--- a/src/sysync/syncsession.cpp
+++ b/src/sysync/syncsession.cpp
@@ -2549,6 +2549,13 @@ Ret_t TSyncSession::process(TSmlCommand *aSyncCommandP)
delayExecUntilNextRequest(aSyncCommandP);
}
else {
+if (fDelayedExecutionCommands.size()0
+!aSyncCommandP-canExecuteOutOfOrder()) {
+ PDEBUGPRINTFX(DBG_SESSION,(%s: command cannot be executed with other commands already delayed - flush queue,aSyncCommandP-getName()));
+ fCmdIncoming = NULL;
+ tryDelayedExecutionCommands();
+}
+
// command is ok, execute it
fCmdIncomingState=aSyncCommandP-getPackageState
[os-libsynthesis] PHOTO + compare=never
Hello! I just noticed one aspect of the example configs that I wasn't aware of: syncclient_sample_config.xml: field name=PHOTO type=blob compare=never merge=fillempty/ What I see in a slow is that if all fields are equal except for the photo, the modified photo is not stored because merging considers it not relevant (compare=never = eqm_none). According to the doc: never: field is not compared at all. This is for fields that do not contain user data, such as REV in vCard. It would not make sense to compare these fields, as they are not rele- vant for finding out if two objects have the same data or not. What was the rationale for using that mode for PHOTO? Is it for storages which store photo data after re-encoding it? With such a storage, the comparison would yield a false field is different, causing unnecessary writes. But with a storage that stores the data as-is, comparing it is better (IMHO), because modified data actually gets stored. The downside is that comparisons become more expensive. Speaking of comparisons, with EDS this is slightly tricky. Inlined data gets replaced with a file reference, so what the comparison really needs to do is not comparing file:///tmp/testing/temp-testpim/data/evolution/addressbook/pim-manager-testsync-testcontacts-foo/photos/pas_id_51D53D180001_photo-file1.image%252Fjpeg against the binary data in the incoming item, but rather the content of that file. I probably need to write a comparescript for that, right? In that case compare=never may be the right thing to do again, but I am not sure. Won't it lead to the situation again where MERGEFIELDS() incorrectly not marks an item as changed even though it was? -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] PHOTO + compare=never
On Thu, 2013-07-04 at 13:01 +0300, Patrick Ohly wrote: Hello! I just noticed one aspect of the example configs that I wasn't aware of: syncclient_sample_config.xml: field name=PHOTO type=blob compare=never merge=fillempty/ What I see in a slow is that if all fields are equal except for the photo, the modified photo is not stored because merging considers it not relevant (compare=never = eqm_none). According to the doc: never: field is not compared at all. This is for fields that do not contain user data, such as REV in vCard. It would not make sense to compare these fields, as they are not rele- vant for finding out if two objects have the same data or not. What was the rationale for using that mode for PHOTO? Is it for storages which store photo data after re-encoding it? With such a storage, the comparison would yield a false field is different, causing unnecessary writes. But with a storage that stores the data as-is, comparing it is better (IMHO), because modified data actually gets stored. The downside is that comparisons become more expensive. When using compare=conflict fot PHOTO, I get [2013-07-04 14:35:45.739] Winning and loosing Field 'PHOTO' not equal: 'BLOB size=0' 'BLOB size=0' [2013-07-04 14:35:45.739] - updated fields such that both have same value 'BLOB size=0' and PHOTO == EMPTY in a script also does not work. Shouldn't a blob comparison compare the content (= byte string) and treat a blob of size 0 as empty? -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] redundant DB updates - script for processing incoming items from SyncML peer?
Hello! When storing contacts in Evolution Data Server, SyncEvolution applies a beforewritescript which massages the contact such that it meets Evolution conventions. For example, the VOICE flag must be set for TEL, because otherwise Evolution ignores the number. This leads to the following situation: * Contact with TEL;TYPE=WORK:1234 is received. * It gets processed and before writing, the flag is set = TEL;TYPE=WORK,VOICE:1234 * The next sync is a slow sync, so the incoming item (absolutely unmodified!) gets compared against the one from the datastore. * Because the datastores' and the incoming items TEL flags are different, the engine concludes that the DB item must be merged and updated in the DB. * Because the same beforewritescript is again applied after the merge, the exact same item gets sent as update to the DB, without changing anything. This leads to one unnecessary database operation, which is undesirable when slow syncs are frequent and the storage resides on flash storage. This is the case for syncing with PBAP in IVI. IMHO the adapt item to what is expected by data store step should be applied to the incoming item *before* processing it, in other words, in the incomingscript. If processing then doesn't break the item again, the script does not need to run again before writing. But the SySync_config_reference.pdf explicitly warns against that: Note that this is not the place to implement database specific conversions, because this is better done in the beforewritescript and afterread-script in the datastore section. Is that because normally, datatypes are shared between different data stores? I could avoid that by defining datatypes such that they only get used in exactly one data store. The putting data store specific transformations into the datatype's incomingscript would be okay, wouldn't it? -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] temporary local ID + FinalizeLocalID
On Mon, 2013-06-10 at 12:35 +0200, Lukas Zeller wrote:
On 10.06.2013, at 11:57, Patrick Ohly patrick.o...@intel.com wrote:
I'll probably try something else: if commands were delayed in a
message which is marked Final, then force execution of the commands at
the end of processing that message and issue their Status commands, just
as syncing normally would.
Sounds like a good compromise to me.
It seems to be working, too :-)
The effect will be that commands only get delayed across messages if
there will be more of them coming in anyway. The hope is that this will
avoid or at least minimize state change issues.
Note that the problem that I am running into also exists in the
unmodified code: I expect similar issues when the server delays
processing because of timing.
Probably not, because the server can only delay processing at the very
beginning of the sync, while it loads the sync set in the background.
So far, this was the only (but well tested with very long lasting sync
set loads) case of delaying command execution. Delaying execution
after sync has actually started is a new case.
Ah, I see. I had skipped over the
fLocalSyncDatastoreP-engIsStarted(false) check in the relevant code:
TSyncSession::processSyncOpItem()
...
// check if we can process it now
// Note: request time limit is active in server only.
if (!fLocalSyncDatastoreP-engIsStarted(false) || RemainingRequestTime()0) {
aQueueForLater=true; // re-execute later...
return true; // ...but otherwise ok
}
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] temporary local ID + FinalizeLocalID
On Mon, 2013-06-03 at 22:17 +0200, Lukas Zeller wrote: How about yet another approach: the store is allowed to return a new error code, LOCERR_AGAIN, for add/update/delete operations. The engine then remembers all operations with that return code and calls them again (with the exact same parameters and without freeing resources!) at a suitable time, for example the end of the current message. In the second call, the store must execute or finish the operation. The engine must not call an operation on an item for which it has already something pending. If that happens, it has to complete the first operation first. That way the store can keep a list of pending operations with the ItemID as key. Sounds like a good solution indeed. I like it better than late-handling return codes especially because it avoids any risk of breaking existing datastores (and possible unsafe assumptions in their implementations), because for those nothing would change outside *and* inside the engine. It's also somewhat similar to the finalisationscript mechanism, which also keeps items (or parts of them) in memory for using them again at the end of the session, usually to resolve cross references (parent/child tasks for example). Maybe some of the mechanisms can be re-used for LOCERR_AGAIN. I found a different mechanism: TSyncSession::processSyncOpItem() can decide to delay execution of the command by setting a flag. Currently this is done when processing the message already took too long. I have changed the command and item processing call chain and my backend so that it can kick of the operation and continue when the chain is invoked again. For that, I am setting the aQueueForLater=true to use the existing queuing mechanism for SyncML commands. Now I found one problem with that: after the first of several Add or Update commands got queued, all following commands are also queued. What I'd like to see instead is that they all get processed. Then in the level above the engine, right before sending the response, I would gather all pending operations and combine them into a batched, asynchronous add or update operation. The batching is expected to be much more efficient with EDS. It also allows overlapping local processing in the PIM storage with network IO. But right now, I always only get one item to be batched because everything else is still in the queue for later processing. Is the command received after other commands needed to be delayed - must be delayed, too something which is imposed by SyncML? I tried to think of situations where the engine needs to enforce completion of the pending operation before triggering another one, but couldn't think of such a situation. My expectation is that either the backend will properly serialize the item changes or the item changes are independent (update foo, delete bar). But what could happen is that update foo gets started, does not complete, and then delete bar gets processed right away. That would re-order the status messages such that the status for the later command gets sent first. That's because my backend currently can only do insert/update asynchronously, but not deletes. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] temporary local ID + FinalizeLocalID
Hello! I'm trying to batch local database adds together to increase performance in SyncEvolution (https://bugs.freedesktop.org/show_bug.cgi?id=52669 and https://bugs.freedesktop.org/show_bug.cgi?id=64838#c5). My hope was that I can postpone the actual modification and return a temporary local ID. Then later or in parallel, combine multiple items together in a batch add. Finally, the temporary ID was meant to be replaced by the final one in FinalizeLocalID. But I hit the first snag pretty early: FinalizeLocalID() doesn't seem to get called on the server side where I need this feature primarily (for PBAP, CalDAV, ActiveSync the server side is the one with the local database). dsFinalizeLocalID, the wrapper around the store's FinalizeLocalID, gets called in three functions: (gdb) b sysync::TBinfileImplDS::changeLogPostflight Breakpoint 2 at 0x106f956: file /home/pohly/syncevolution/libsynthesis/src/sysync/binfileimplds.cpp, line 462. (gdb) b sysync::TBinfileImplDS::SaveAdminData Breakpoint 3 at 0x10739fd: file /home/pohly/syncevolution/libsynthesis/src/sysync/binfileimplds.cpp, line 2389. (gdb) b sysync::TLocalEngineDS::engGenerateMapItems Breakpoint 4 at 0x109c79c: file /home/pohly/syncevolution/libsynthesis/src/sysync/localengineds.cpp, line 6635. None of these get called on the server side during a refresh-from-client, slow or normal two-way sync. That makes sense, map items are only relevant on the client, and the server is not using the binfile class. But shouldn't the server also call FinalizeLocalID somewhere? Where would be the right place for it? Regarding FinalizeLocalID() in the client, I see some conceptual issues with it, at least for what I am trying to achieve. My main concern is about error handling. FinalizeLocalID() can return an error code, but dsFinalizeLocalID does not check for errors. It merely checks for ID modified. So in case of an error, the local store itself has to remember that writing failed and do something. What that something could be is not clear. It could return an error in EndDataWrite(), in which case I would expect the engine to enforce a slow sync in the next sync session. I can't think of other, realistic options right now. I think a better solution would be to delay handling the result of a database write until the entire SyncML message has been processed. Then give the store a chance to finalize the ID of each item that was touched as part of that SyncML message. The store can use that opportunity to flush the pending operations and return error codes for each item, just as it would normally do in the synchronous write operations. Then the engine continues with the same error handling that it already does. The advantage, besides better error handling, is that the store has a natural place to flush pending operations. Without the intermediate FinalizeLocalID() calls a new max pending operations setting independent of the SyncML message size would be needed. Comments? -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] Sync mode extensions must be optional
On Thu, 2012-10-11 at 12:27 +0200, Lukas Zeller wrote: Hi Patrick, I found that some servers don't like the sync mode extensions in the devInf you introduced for SyncEvolution a while ago (crash syncs with misleading errors as apparently their wbxml decoders fail or something). As I see no way to automatically control this, I added a session level config flag syncmodeextensions that must be set explicitly to enable the extensions. The default is off, to make sure current libsynthesis (master branch) users will no see any difference in behaviour once they update. For SyncEvolution, you need to add this flag to your config. For SyncEvolution 1.3.99.3, I've extended the SyncEvolution SyncMLVersion setting to control whether the extensions are enabled. They are enabled by default because I don't expect many users to use affected servers, whereas more user will want it on for use with SyncEvolution as server. I had already done this a while ago, but then noticed a problem during release testing which caused me to back out the changes again because I didn't have time to investigate. It turned out that I didn't enable the feature on the server side. Strictly speaking, that shouldn't be necessary, because the server will only add the extensions when the client already sent his, so there should not be any problem. But it makes sense to have this configurable, just in case. The master branch of libsynthesis on freedesktop.org is now based on the latest gitorious master branch. Lukas, it contains two patches which you might want to include: commit 9b0c01cf2fff6701bdb026518048fc4b06a7072e Author: Patrick Ohly patrick.o...@intel.com Date: Thu Feb 21 00:51:53 2013 -0800 caching mode: fix memory leak When deleting unmatched local sync items, the TSyncItem instance was leaked after removing it from fItems. commit 99159e0991664f8c8319e634598ea6c9bd73fcc2 Author: Patrick Ohly patrick.o...@intel.com Date: Mon Sep 10 09:15:20 2012 +0200 autotools: bumped minor version Bumped minor version so that SyncEvolution can ensure that the version it is compiled against really fixes the VJOURNAL-plain text conversion problem. BTW, did you tag 3.4.0.47? There is a comment about the version, but not the tag itself. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] one-way sync + sync tokens not updated
Hello!
SyncEvolution's ActiveSync backend is the first one which uses the
string tokens in StartDataRead() and EndDataWrite(). The backend always
runs inside a binfile-based SyncML client.
In one particular test I am using a one-way-from-server sync,
initiated by the client, and noticed a problem: the new token created by
the backend in EndDataWrite() was not passed to the StartDataRead() in
the next sync.
The backend cannot handle that, because the tokens come directly from
ActiveSync, which only allows reusing old tokens in a few limited cases.
In this case, the server rejected the obsolete token, causing an
unexpected slow sync.
Full log is here:
http://syncev.meego.com/2012-08-16-17-03_all/testing-amd64/14-exchange/Client_Sync_eds_contact_testOneWayFromLocal.send.client.B/syncevolution-log.html
It has no debug output explaining the problem. I tracked it down to
this:
// save end of session state
localstatus TCustomImplDS::implSaveEndOfSession(bool aUpdateAnchors)
{
localstatus sta=LOCERR_OK;
PDEBUGBLOCKCOLL(SaveEndOfSession);
// update TCustomImplDS dsSavedAdmin variables (other levels have already
updated their variables
if (aUpdateAnchors) {
if (!fRefreshOnly || fSlowSync) {
...
// also update opaque reference string possibly needed in DS API
implementations
fPreviousToRemoteSyncIdentifier = fCurrentSyncIdentifier;
=PDEBUGPRINTFX(DBG_ADMIN+DBG_DBAPI+DBG_EXOTIC,(updating sync token
(fPreviousToRemoteSyncIdentifier) from %s to current sync token
%s,fPreviousToRemoteSyncIdentifier.c_str(),fCurrentSyncIdentifier.c_str()));
} else {
=PDEBUGPRINTFX(DBG_ADMIN+DBG_DBAPI+DBG_EXOTIC,(keeping old sync token
(fPreviousToRemoteSyncIdentifier) %s instead of updating to current sync token
%s,fPreviousToRemoteSyncIdentifier.c_str(),fCurrentSyncIdentifier.c_str()));
}
I added that debug output. It confirms that the keeping old sync token
branch is taken.
What is the rationale here? Is the goal perhaps that if the client
switches back to a two-way sync, all changes since the last two-way sync
get sent, despite the intermediate one-way sync?
Those changes include the changes made on behalf of the server during
the one-way-from-server sync. Is that filtered out?
My test is probably broken. IMHO one-way sync only makes sense into a
storage which never changes on the receiving side. In the ActiveSync
case, that cannot be guaranteed, so I might as well just skip the test.
OTOH, a user might decide to use an ActiveSync server as remote backup,
in which case one-way syncing makes sense again. Would it be acceptable
to always take the updating sync token branch above?
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] PBAP: one-way sync modes
Hello! For the PBAP caching mechanism in SyncEvolution I'd like to use the Synthesis engine. I think that makes sense because the engine does a few things which would have to be done manually otherwise (translate between formats, finding pairs). The improvements below would also make sense in other use cases. At the moment the engine (or rather, SyncML) lacks a few things which I need to change. I'm focusing on one-way-from-client syncing because the server has a bit more control over what it does and because it matches the SyncEvolution use case. In SyncML, incremental one-way syncs fall back to a two-way slow sync after a failure. If the server has an item which the client no longer has, then the item will be recreated on the client, despite the intention to only transfer data in one direction. Right? This is undesirable in general (IMHO), and an absolute no-no for PBAP as the client, because it cannot write the changes. As a first step I would disable sending changes to the client if the client asked for a one-way-from-client sync. The server's datastore can already be marked read-only (readonly config option, SETREADONLY()); something similar for the peer's datastore would make sense. Then I could leave the default behavior as it is and use a script function to trigger the desired behavior. Actually, there is SETREFRESHONLY(). According to the docs, it is meant to be used for turning a two-way sync requested by the client into a refresh-from-remote. I need to check whether I can use that or still need to add/modify something. Suppose a slow sync was done in that modified refresh-only mode. Any item that the server has which are not on the client need to be removed if the server's storage is meant to mirror the client. At the moment, the engine cannot know whether it is meant to mirror the data and thus it will leave the extra items on the server unchanged. I intend to add a mirror config option similar to readonly. If set, the engine will not only avoid sending changes to the client, it will also remove those extra local items. Finally, during a slow sync where a match was found, extra properties in the server's copy of an item must be removed to avoid keeping stale data. This can already be done by configuring the merging of items accordingly (instead of preserving as much data as possible, ensure that the client's item is always considered the winning one and that its version completely overwrites the server's). -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] [SyncEvolution] Recurrences with Nth weekday of the month
On Thu, 2012-06-21 at 10:51 +1000, Mathew McBride wrote: Hi, I'm testing SyncEvolution against my Funambol connector (GroupDAV) at the moment. I've been testing some recurrence combinations and it seems recurrences of type 'N' weekday of the month (i.e second tuesday of the month) aren't supported? [...] The original Evolution event has this: RRULE:BYDAY=TU;BYSETPOS=2;COUNT=2;FREQ=MONTHLY;WKST=SU The environment is Ubuntu 12.04 with the provided Evolution and SyncEvolution versions (1.2.2). Are these type of recurrences supported with Evolution? In Evolution, yes, in libsynthesis, no. I can reproduce the problem and noticed that the code (src/sysync/rrules.cpp, RRULE2toInternal()) bails out because it doesn't recognize the BYSETPOS keyword. Lukas, was that a conscious decision, perhaps because it cannot be supported in combination with vCalendar 1.0, or just an oversight? What would it take to support that? -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] Funambol and large SyncML messages?
On Mon, 2012-06-18 at 09:30 +0200, Lukas Zeller wrote: occasionally, I get error reports from libsynthesis (iOS client) users which experience SyncML Error 511 (server error) with Funambol/OneMediaHub servers. Analyzing these, it always boils down to a big (99kBytes) client message that the server apparently cannot handle. As the server does not send a MaxMsgSize, libsynthesis makes the message as big as fits into it's local buffer which is 100k by default. My question - did you experience this large message size problem with Funambol as well? Or are you using a smaller message size than the default in SyncEvolution anyway? The default message size is 15 bytes. I vaguely remember such issues for myFunambol, but chalked it up to server isn't reliable instead of investigating. Testing against OneMedia has not been successful at all, as in most tests don't run. First there was the issue with slow syncs causing the server to report the 411 try again later error, then it started throwing XML parser errors for a (as far as I can tell so far) correct DevInf. I'm just wondering if reducing the message size could reliably fix that Error 511 problem - and if so how small the message size should be. I don't know. I guess you could try asking Funambol (their mailing list is now on SourceForge), but be prepared to be told to install a Funambol client and emulate its behavior (that was the response that I got for the DevInf issue). -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] [PATCH] Improve readability of SyncML TK error messages
On Thu, 2012-06-07 at 12:31 +0200, Lukas Zeller wrote: Hi Andris, thanks for the patch. However, the SMLERRPRINTFX() macro, like all debug output macros across the entire library, is supposed to output a linefeed by itself. It also actually does so in my builds of libsynthesis, so if the LFs are missing in your output, it must be something along the path from SMLERRPRINTFX() to the actual output function. I think the problem is in the CONSOLEINFO_LIBC shortcut output mode Patrick added this February (13ff1e4149 (logging + Linux: enable console output)), where CONSOLEPRINTF() is mapped to fprintf(stderr,...) without adding a LF. In my (iOS) build this new mode is not used, but output ends in a plain puts(), which does add the LF. Indeed. I wasn't aware of that convention when adding the code. diff --git a/src/sysync/sysync_debug.h b/src/sysync/sysync_debug.h index 61d4cdd..68dce5e 100755 --- a/src/sysync/sysync_debug.h +++ b/src/sysync/sysync_debug.h @@ -110,7 +110,7 @@ TDebugLogger *getDbgLogger(void); // Because a lot of libs log to stderr, include a unique prefix. // Assumes that all printf format strings are plain strings. #define CONSOLEPUTS(m) CONSOLE_PRINTF_VARARGS(%s, (m)) -#define CONSOLE_PRINTF_VARARGS(_m, _args...) fprintf(stderr, SYSYNC _m, ##_args) +#define CONSOLE_PRINTF_VARARGS(_m, _args...) fprintf(stderr, SYSYNC _m \n, ##_args) #define CONSOLEPRINTF(m) CONSOLE_PRINTF_VARARGS m # else // CONSOLEINFO_LIBC #define CONSOLEPUTS(m) ConsolePuts(m) I've added that to my repo and also refined the CONSOLEINFO_LIBC code a bit: instead of always using stdio, it lets the app override the actual function. I'm adding that because I get quite a bit of error output from the SyncML TK which seems to be harmless. Some of reporting inside libsynthesis duplicates my own reporting. So I'd rather disable console printing dynamically and only enable it when the log level in SyncEvolution is sufficiently high. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] scripts + datastore
On Fri, 2012-04-27 at 16:27 +0200, Lukas Zeller wrote: On Apr 25, 2012, at 15:11 , Patrick Ohly wrote: But in practice that pointer is always NULL. I wasn't sure anyway whether I would get the pointer to the local or remote datastore. from fDsP you'd get the pointer to the local datastore (TLocalEngineDS *). So to reach the remote datastore from a given TMultiFieldItem (which you certainly have at comparescript), I'd rather use yourMultiFieldItem-getRemoteItemType-getRelatedDatastore() to get to the datastore that has defined that item's remote type. Thanks, that worked. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] scripts + datastore
Hello! I am trying to solve one issue in SyncEvolution: when trying to find pairs, it needs to know on a per-datastore basis whether both remote and local storage have truly unique UID/RECURRENCE-ID that can be relied upon (iCalendar 2.0 semantic). So far, I am using a compare script for that, but it has to make assumptions about the peer. To overcome that I added code that allows clients to add SyncCap entries to the CTCap (similar to the can restart flag). This information is stored at the receiving end in the TSyncDataStore base class by the TRemoteDataStore while parsing the SyncCap (again, very similar to fCanRestart). But now my problem is: how can the compare script access that information? It runs inside the datatype context. Does that mean that all datastores sharing the same type also share the same context and that the initscript for the type is only invoked once? The script functions in multifielditemtype.cpp (like SYNCOP()) looked promising. It's possible to get a pointer to some kind of datastore: TSyncDataStore *related = static_castTMultiFieldItemType *(aFuncContextP-getCallerContext())-getRelatedDatastore(); But in practice that pointer is always NULL. I wasn't sure anyway whether I would get the pointer to the local or remote datastore. Any hints? -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] Sync server with oracle database
On Mon, 2012-04-23 at 22:23 +0200, Lukas Zeller wrote: To make a server out of libsynthesis, what needs to be done outside libsynthesis is the connection to the HTTP transport, and session handling (SyncML session objects must be kept around between HTTP requests until the entire session completes or is aborted). The opensource SyncEvolution project has done this, however SyncEvolution has quite some complexity of its own so I don't know how easy it would be to use the HTTP transport code from SyncEvolution. It would be pretty hard. The HTTP server in SyncEvolution is written in Python using the Twisted framework and depends on the D-Bus API provided by the SyncEvolution core, so one also inherits all the rest of SyncEvolution (own configuration system, backends, etc.). For writing one's own server, the Synthesis SDK and server will be easier. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] 409 item merged in client + multiple sync cycles in a single session
On Tue, 2012-03-06 at 14:50 +0100, Patrick Ohly wrote: I haven't look into this yet, but still have it on my radar. Done in the meego.gitorious.org master branch. I found that checking for collisions is hard (not all records are in memory), so I settled for making the chance of collisions smaller in the string case by including a running count. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] 409 item merged in client + multiple sync cycles in a single session
On Tue, 2012-02-07 at 16:05 +0100, Patrick Ohly wrote: On Mon, 2012-02-06 at 21:29 +0100, Patrick Ohly wrote: I'm currently experimenting with a different approach for handling the 409 in the binfile client: when an Add fails with 409, catch it as it is done at the moment, but then tell the server that it was mapped to a dummy LUID. Mark that LUID as deleted in the change log. Then in the next session, delete the redundant item on the server. I'm combining that with running multiple SyncML sessions in the same context. Will post code soon... The result is in the internal-sync branch in the meego.gitorious.org repo of libsynthesis. It's called internal-sync because I am working on an extension of the SyncML protocol that is only understand when SyncEvolution talks to SyncEvolution (either in the SyncEvolution local sync mode - see the http://syncevolution.org/blogs/pohly/2012/fosdem-2012 talk for a diagram illustrating that) or in SyncEvolution client to SyncEvolution server mode. Instead of repeating myself, let me quote the commit messages of the relevant commits on that branch below. There are a few more commits in that branch. Lukas, what do you think? This is not done yet, but if possible I'd like to get feedback before going down an entirely wrong path. Some tests for this new code are in SyncEvolution (not pushed yet) and pass as expected. I'll keep working on this, in particular the temporary problem part and add more data in second cycle cases. I've also pushed the internal-sync branch for SyncEvolution. It has tests for the 409 in client of dumb SyncML server problem (testAddBothSides when configured to use SyncEvolution server with file backend), restarting a two-way sync with SyncContext::requestAnotherSync() (testTwoWayRestart) and the extended SyncSource semantic. All of that works fine. It took a bit longer than expected because writing these tests required quite a bit of code refactoring. Speaking of the extended semantic that beginSync() may be called multiple times: right now this is mandatory. My hope is that backend implementers will adapt to that change before releasing 1.3, or that no such changes are necessary. EDS and file backends worked fine without changes. I prefer that approach because it avoids special cases. Mikel, most relevant for you is the new SyncSource/SyncContext::requestAnotherSync() API call. You might be able to implement the two-phase transfer of contacts like this: 1. In first beginSync() call: * only retrieve contacts without PHOTO data * ask for another cycle with requestAnotherSync() 2. In the second beginSync(): * also retrieve PHOTO data * mark all contacts with new or updated PHOTO data as updated I also implemented the backend is read-only part. The Synthesis engine already had that, it was just a matter of making that available in SyncEvolution. You can rebase the PBAP branch onto internal-sync. I'll refrain from rebasing internal-sync from now on. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] 409 item merged in client
On Mon, 2012-01-30 at 18:04 +0100, Patrick Ohly wrote: Should it have removed one item while processing the Map? IMHO it can't, because it doesn't have enough information to determine which of its two items are up-to-date. It only knows that the client must have merged two items, but not yet which one won. In the second session, the server gets an updated item with client ID foo. Now it could update one copy and delete the other, but because it only kept a link between its item 0 and foo, it only does the update part. Would something break if the server allowed the same client item to map to multiple items on the server and then did a remove in its database when receiving an update? I suspect that the client is simply doing something not expected by typical SyncML servers. I'm currently experimenting with a different approach for handling the 409 in the binfile client: when an Add fails with 409, catch it as it is done at the moment, but then tell the server that it was mapped to a dummy LUID. Mark that LUID as deleted in the change log. Then in the next session, delete the redundant item on the server. I'm combining that with running multiple SyncML sessions in the same context. Will post code soon... -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] super data store: trying to read to check for existance
On Tue, 2011-10-25 at 09:24 +0200, Lukas Zeller wrote:
On Oct 24, 2011, at 16:55 , Patrick Ohly wrote:
Why is it necessary to read before trying to delete? If the item exists,
then reading it is a fairly expensive test.
Unfortunately, with some some backends, this is the only test that reliably
works. For example, some SQL databases (or their ODBC driver) can't return a
correct number of affected rows for DELETE statements. So reading before
deleting was the only way to detect if any of the subdatastores really had
that item and correctly return 404 or 200.
So far, my backends were written with the expectation that they have to
cope with delete requests for items which are already deleted. This
follows from the inherent race condition between syncing and some other
process which might delete items while a sync runs.
Are all backends expected to return 404 when an item doesn't exist?
Not all backends (see above, built-in ODBC can't), but yes, for plugin
backends, returning proper error codes is specified, also for delete.
Still, if a plugin would not conform to that in its implementation of
delete, that would probably have gone unnoticed so far.
I checked the API docs and did not see it mentioned explicitly.
In SyncEvolution I added some remarks about 404 (will probably not be
noticed) and also added automated Client::Source tests for it (which is
harder to ignore if someone runs the tests ;-)
I also changed the error logging so that 404 is reported to users in
ReadItem and not logged in DeleteItem (while still returned to the
engine).
Of course, the test is a bit expensive - if that's a concern, one
could differentiate between plugin datastores and others (all the
builtin ones, including SQL/ODBC), and use the expensive read test
only for the latter. Like a virtual dsDeleteDetectsItemPresence()
which returns false by default, but can be overridden in plugin
datastores to return true.
The patch below implements that idea (from the for-master/bmc23744
branch). SyncEvolution relies on that patch to avoid the (harmless)
ERROR messages for item not found in ReadItem(), although everything
should work okay also without the patch.
Does that look right?
From fb5cc0dc19fb60e40a4ca2dcfe44a52a75ec7354 Mon Sep 17 00:00:00 2001
From: Patrick Ohly patrick.o...@intel.com
Date: Fri, 28 Oct 2011 15:42:57 +0200
Subject: [PATCH] server engine: more efficient deletion in superdatastore
Attempting to read an item just to check whether it exists is expensive.
It also may trigger error messages when the item does not exist (done by
SyncEvolution).
Therefore, if a data store is able to properly report whether it found
the item which is to be deleted, a different logic is used for
deletion in the superdatastore:
- attempt to delete until it succeeds in a subdatastore
- recreate the sync item from the copy after a failed delete attempt
(which deletes the sync item), if there is another loop iteration
Deleting the copy of the sync item was moved to the function exit code
to avoid code duplication.
By default all data stores are assumed to not support 404 in its
delete operation. The only exception is the plugin API. The (somewhat
undocumented) assumption is that all methods properly report 404 when
the requested item does not exist. The attempt to read code already
relied on that. Now the code relies on that for the attempt to
delete.
Probably it even works when the plugin returns some other error
code (the regular value will be false in that case) or no error code
at all: the translation code between remote ID and local ID will
already detect that the item is not in the mapping table if it is not
in the subdatastore. So the actual plugin will not get called at all
in the case where its expected behavior would matter.
---
src/DB_interfaces/api_db/pluginapids.h |4 ++
src/sysync/superdatastore.cpp | 80 ++--
src/sysync/syncdatastore.h |2 +
3 files changed, 62 insertions(+), 24 deletions(-)
diff --git a/src/DB_interfaces/api_db/pluginapids.h
b/src/DB_interfaces/api_db/pluginapids.h
index 4fbfc79..f215094 100755
--- a/src/DB_interfaces/api_db/pluginapids.h
+++ b/src/DB_interfaces/api_db/pluginapids.h
@@ -172,6 +172,10 @@ public:
virtual void dsResetDataStore(void) { InternalResetDataStore();
inherited::dsResetDataStore(); };
virtual ~TPluginApiDS();
+ // override TSyncDataStore: the plugin must be able to return 404
+ // when an item is not found during delete
+ virtual bool dsDeleteDetectsItemPresence() const { return true; }
+
#ifndef BINFILE_ALWAYS_ACTIVE
/// @name api methods defining the interface from TCustomImplDS to
TXXXApi actual API implementations
/// @{
diff --git a/src/sysync/superdatastore.cpp b/src/sysync/superdatastore.cpp
index e70d7bb..5c0a2b5 100755
--- a/src/sysync/superdatastore.cpp
+++ b/src/sysync/superdatastore.cpp
@@ -530,7 +530,7 @@ bool TSuperDataStore
Re: [os-libsynthesis] super data store: trying to read to check for existance
On Tue, 2011-10-25 at 09:24 +0200, Lukas Zeller wrote: Hello Patrick, On Oct 24, 2011, at 16:55 , Patrick Ohly wrote: Why is it necessary to read before trying to delete? If the item exists, then reading it is a fairly expensive test. Unfortunately, with some some backends, this is the only test that reliably works. For example, some SQL databases (or their ODBC driver) can't return a correct number of affected rows for DELETE statements. So reading before deleting was the only way to detect if any of the subdatastores really had that item and correctly return 404 or 200. So far, my backends were written with the expectation that they have to cope with delete requests for items which are already deleted. This follows from the inherent race condition between syncing and some other process which might delete items while a sync runs. Are all backends expected to return 404 when an item doesn't exist? Not all backends (see above, built-in ODBC can't), but yes, for plugin backends, returning proper error codes is specified, also for delete. Are these expectations documented somewhere? I'm still a bit fuzzy about which error codes are expected. For example, should a delete of a non-existent item succeed (200) or fail (404)? Still, if a plugin would not conform to that in its implementation of delete, that would probably have gone unnoticed so far. Right. SyncEvolution has hardly ever (never?) returned 404 and that has not been an issue until now ;-) Of course, the test is a bit expensive - if that's a concern, one could differentiate between plugin datastores and others (all the builtin ones, including SQL/ODBC), and use the expensive read test only for the latter. Like a virtual dsDeleteDetectsItemPresence() which returns false by default, but can be overridden in plugin datastores to return true. That sounds like the right compromise. I'm much more comfortable returning 404 in a delete and not reporting that to the user as an error, compared to not reporting a 404 in a ReadItem call (as I have to do now). -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] Google Calendar + per-item permission errors
On Di, 2011-08-02 at 15:39 +0200, Lukas Zeller wrote: Hello Patrick, On Aug 1, 2011, at 20:39 , Patrick Ohly wrote: In a local slow sync between Evolution and Google Calendar I see the following problem: [...] * The server logs Status: 403: originator exception and WARNING: Aborting Session with Reason Status 403 (REMOTE problem). That says it all. The session ends here without finishing the slow sync. This is a bit too drastic. [...] Should it be turned into a local error along the were somewhere? I remember vaguely that we discussed something like that and came to the conclusion that a backend can return errors in the range 500 to the Synthesis engine. That's what SyncEvolution is doing here. There's a place in localengineds.cpp (around line 3337) where all the error codes are catched in a switch/case statement that should NOT abort the session, but rather mark the item for resend later, which means in the next session. 403 is not among these, because usually a auth problem is really fatal and retry does not help. For your case at hand, I'd recommend to translate that kind of 403 error to 417 (retry later) in the backend (or in the sentitemstatusscript), which means that the backend has some hope that a retry in the next session might help. I followed that approach. It means that the problematic item will be sent again and again, but that's better than the alternatives (abort session or only logging the issue). I still need to understand better how I arrived at this situation, though. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] Google Calendar + per-item permission errors
Hello! In a local slow sync between Evolution and Google Calendar I see the following problem: * Both sides have a meeting event in almost the same state. * The Synthesis engine decides that the event in Google Calendar needs to be updated. * The PUT command fails with 403 You don't have access to change that event. * This error code is returned as SyncML status for Replace from SyncML client to server. * The server logs Status: 403: originator exception and WARNING: Aborting Session with Reason Status 403 (REMOTE problem). That says it all. The session ends here without finishing the slow sync. This is a bit too drastic. I'd prefer to continue the session and merely report that this one item couldn't be updated. Never mind that this is my own calendar and that Google should allow me to modify all of the events in it... I wonder whether the 403 HTTP error really should be passed through like that. It gets returned by the Synthesis backend: * [2011-08-01 18:20:56.785] PUT: bad HTTP status: status 1.1, code 403, class 4, You don't have access to change that event., must not retry * [2011-08-01 18:20:56.786] Running destroy hooks. * [2011-08-01 18:20:56.786] Request ends. * [2011-08-01 18:20:56.786] exception thrown at /home/pohly/syncevolution/syncevolution/src/backends/webdav/NeonCXX.cpp:685 * [2011-08-01 18:20:56.786] error code from SyncEvolution access denied (remote, status 403): PUT: bad HTTP status: status 1.1, code 403, class 4, You don't have access to change that event. * [2011-08-01 18:20:56.786] aID=(04008200E00074C5B7101A82E008B0884FC7A4E8CB01159F761E3FE944408F8932B4E44FFBFD.ics/20110428T15Z,) res=403 * [2011-08-01 18:20:56.786] cannot update record in database (sta=403) * [2011-08-01 18:20:56.786] Database Error -- SyncML status 403 * [2011-08-01 18:20:56.786] - Operation replace failed with SyncML status=403 –[2011-08-01 18:20:56.786] End of 'Process_Item' [-top] [-enclosing] * [2011-08-01 18:20:56.786] processSyncOpItem: Error while processing item, status=403 Should it be turned into a local error along the were somewhere? I remember vaguely that we discussed something like that and came to the conclusion that a backend can return errors in the range 500 to the Synthesis engine. That's what SyncEvolution is doing here. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] dealing with PHOTO:file:// (BMC #19661)
On Fr, 2011-07-22 at 15:30 +0200, Lukas Zeller wrote: Hello Patrick, thanks for the READ() script function! This is sure a useful extension of the script toolset. I'm still tempted to embed Python as scripting engine into libsynthesis (optional, of course). The main advantage would be that we get such generic operations like file read/write and string manipulations for free. But in the current situation adding a normal function certainly was easier :-) Next on my TODO list is a WRITE() function: it'll be the inverse operation for inlining data and will become useful once files can be attached to contacts in Evolution Data Server. However, I'd like to mention in this context that the engine for a long time contains a mechanism for the general problem with large fields. I know, but it didn't seem usable here, as you said. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] uri + binary PHOTO values (was: Re: field merging + parameters)
On Do, 2011-07-21 at 12:21 +0200, Patrick Ohly wrote:
Now there is only one other problem with PHOTO uris. They get encoded as
binary data:
PHOTO;VALUE=uri:http://example.com/photo.jpg
=
PHOTO;ENCODING=B;VALUE=uri:aHR0cDovL2V4YW1wbGUuY29tL3Bob3RvLmpwZw==
Is that because PHOTO is defined as blob?
Would it make sense to be selective in the encoder for blob and only
use binary encoding if the content contains non-printable characters?
This patch has the desired effect:
index 4105d03..616cb68 100644
--- a/src/sysync/mimedirprofile.cpp
+++ b/src/sysync/mimedirprofile.cpp
@@ -23,6 +23,7 @@
#include syncagent.h
+#include ctype.h
using namespace sysync;
@@ -2274,9 +2275,17 @@ sInt16 TMimeDirProfileHandler::generateValue(
}
// append to existing string
fldP-appendToString(outval,maxSiz);
- // force B64 encoding
- aEncoding=enc_base64;
- aNonASCII=false;
+ // force B64 encoding if non-printable or non-ASCII characters
+ // are in the value
+ size_t len = outval.size();
+ for (size_t i = 0; i len; i++) {
+char c = outval[i];
+if (!isascii(c) || !isprint(c)) {
+ aEncoding=enc_base64;
+ aNonASCII=false;
+ break;
+}
+ }
}
else {
// apply custom field(s)-to-string translation if needed
I tried it with both binary PHOTO and VALUE=uri.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] adding or overriding CtCap
On Mo, 2011-06-20 at 20:20 +0100, Patrick Ohly wrote:
Verified with debugging output. I've not looked into setfieldoptions()
for the exact logic which enables the field.
This has the effect that my X-FOOBAR-EXTENSION and X-TEST extensions
(stored locally in XPROPS) are not getting preserved when importing the
updated contact from Google.
What would be a proper fix for this?
I've come to the conclusion that wildcard properties should not be
marked as available by default. With the following commit I get what I
need, is this the right approach?
commit 92d2f367e72eccedb2cd1e103ec7770b6a426ef2
Author: Patrick Ohly patrick.o...@intel.com
Date: Tue Jun 21 09:35:49 2011 +0200
engine: updated implementation of unprocessed wildcard properties
The 3529d3cb commit which introduced support for unprocessed wildcard
properties added the logic that such properties are always considered
available regardless of the actual CtCap of the peer.
This has the undesirable effect that the content of these properties
are removed when receiving an update from a peer which doesn't
support them.
Therefore this commit reverts that part of 3529d3cb: setfieldoptions()
with NULL as first parameter now only enables mandatory properties,
as it did before the introduction of unprocessed properties.
unprocessed properties need to be listed explicitly in the CtCap
like any other property. This is currently untested.
diff --git a/src/sysync/mimedirprofile.cpp b/src/sysync/mimedirprofile.cpp
index 27b1abb..4105d03 100644
--- a/src/sysync/mimedirprofile.cpp
+++ b/src/sysync/mimedirprofile.cpp
@@ -5139,7 +5139,7 @@ bool
TMimeDirProfileHandler::analyzeCTCap(SmlDevInfCTCapPtr_t aCTCapP, TSyncItem
for (sInt16 i=0; iitemTypeP-fFieldDefinitionsP-numFields(); i++) {
itemTypeP-getFieldOptions(i)-available=false;
}
- // force mandatory+unprocessed properties to be always available
+ // force mandatory properties to be always available
setfieldoptions(NULL,fProfileDefinitionP,itemTypeP);
}
// now we have received fields
@@ -5174,9 +5174,9 @@ bool TMimeDirProfileHandler::setLevelOptions(const char
*aLevelName, bool aEnabl
// enable fields related to aPropP property in profiles recursively
-// or (if aPropP is NULL), enable fields of all mandatory (or wildcard)
properties
+// or (if aPropP is NULL), enable fields of all mandatory properties
void TMimeDirProfileHandler::setfieldoptions(
- const SmlDevInfCTDataPtr_t aPropP, // property to enable fields for, NULL if
all mandatory+unprocessed properties
+ const SmlDevInfCTDataPtr_t aPropP, // property to enable fields for, NULL if
all mandatory properties should be en
const TProfileDefinition *aProfileP,
TMimeDirItemType *aItemTypeP
)
@@ -5233,7 +5233,7 @@ void TMimeDirProfileHandler::setfieldoptions(
while (propdefP) {
// compare
if (
- (propname==NULL (propdefP-mandatory || propdefP-unprocessed)) ||
+ (propname==NULL propdefP-mandatory) ||
(propname (strucmp(propname,TCFG_CSTR(propdefP-propname))==0))
) {
// match (or enabling mandatory) - enable all fields that are related
to this property
--
Best Regards
Patrick Ohly
Senior Software Engineer
Intel GmbH
Open Source Technology Center
Pützstr. 5 Phone: +49-228-2493652
53129 Bonn
Germany
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] adding or overriding CtCap
On Mi, 2011-06-15 at 14:02 +0200, Patrick Ohly wrote: Hello! One of the problems that we have in SyncEvolution is the loss of properties not supported by SyncML servers, like Google: https://bugs.meego.com/show_bug.cgi?id=15029 For example, BDAY is not supported and gets lost in a round-trip sync because the client doesn't know that the server didn't store the field. That's because the Google server does not provide CtCap as part of its DevInf. The attached patches address this problem by allowing a RemoteRule to provide CtCap information. It's not currently in the branch used by SyncEvolution, but I need to add something like it soon. Comments welcome. I'm a bit undecided whether the RemoteRule should provide a DevInf or only CtCap. Right now, only CtCap is used and the rest is ignored, but perhaps that might change someday? Any comments about this patch and/or the approach? Thanks to Lukas' hint about the updateallfields option I continued testing. It almost works as intended, except for the chat information. They are defined as fields like this: field name=AIM_HANDLEarray=yes type=string compare=conflict/ field name=AIM_SLOT array=yes type=string compare=conflict/ And in the profile: property name=X-AIM suppressempty=yes rule=KDE/ property name=X-AIM suppressempty=yes rule=other value field=AIM_HANDLE/ parameter name=X-EVOLUTION-UI-SLOT positional=no show=no rule=HAVE-EVOLUTION-UI-SLOT value field=AIM_SLOT/ /parameter /property property name=X-messaging/aim-All suppressempty=yes rule=KDE value field=AIM_HANDLE/ /property The logic in our profile is this: when talking to KDE in the Akonadi backend, avoid using X-AIM and enable X-messaging/aim-All instead. When talking to a SyncML peer or Evolution, use X-AIM. Now when parsing a CtCap which does not mentioned PropertyPropNameX-AIM/PropName/Property at all, the engine concludes that: - AIM_HANDLE : AVAILABLEmaxoccur=1, maxsize=0 (unlimited) - AIM_SLOT : n/a maxoccur=0, maxsize=0 (unlimited) The AIM_HANDLE = AVAILABLE is wrong here. I also tried it with the simpler profile: property name=X-AIM suppressempty=yes value field=AIM_HANDLE/ parameter name=X-EVOLUTION-UI-SLOT positional=no show=no value field=AIM_SLOT/ /parameter /property Then it works. So apparently something in the rule parameter confuses the CtCap parser. Any idea? -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] compare scripts
On Fr, 2011-06-17 at 14:41 +0100, Lukas Zeller wrote:
See the attached patch for a new script function COMPARISONMODE(), compiles
but is UNTESTED.
I can confirm that it works. Thanks a lot. I'm going to add it as a
commit to the master branch on meego.gitorious.org.
With this, your comparescript could probably look like:
if (COMPARISONMODE!=age
TARGET.UID == REFERENCE.UID
TARGET.ORIGSTART == REFERENCE.ORIGSTART) {
RES = 0;
} else {
RES = COMPARE(TARGET.DMODIFIED, REFERENCE.DMODIFIED);
}
Are you sure that the else clause for UID or RECURRENCE-ID not matching
should be another content comparison? If the comparison mode is not
age, then I want to rely exclusively on UID/RECURRENCE-ID for
equal/not equal, and thus I have the following logic:
INTEGER RES;
if (COMPARISONMODE() != age SESSIONVAR(VCALENDAR_COMPARE_UID) ) {
if (TARGET.UID == REFERENCE.UID
TARGET.ORIGSTART == REFERENCE.ORIGSTART) {
RES = 0;
} else {
RES = -999;
}
} else {
RES = COMPAREFIELDS();
}
The VCALENDAR_COMPARE_UID session variable will be false when I want the
default behavior. Does this look right?
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] adding or overriding CtCap
On Mo, 2011-06-20 at 15:03 +0100, Lukas Zeller wrote: Hello, I don't see why it should work in the simple profile but not work in the complex one. However, maybe this is a red herring and the problem lies in the fReceivedFieldDefs (in TMimeDirItemType) flag. This is a per-type flag which is set once a property list from devInf was parsed with TMimeDirProfileHandler::analyzeCTCap(). That function is only called once during a sync session. The updated or added CtCap is inserted into TSyncSession::analyzeRemoteDevInf() by the checkRemoteSpecifics() call, which runs before the analyzeCTCap() call later in analyzeRemoteDevInf(). Its effect is that when parsing a CTCap again (which is still possible), the available status of the fields are NOT reset before parsing the CTCap, such that possibly more fields get enabled, but those that are already enabled from a previous CTCap scan remain enabled. This is needed e.g. for combining vCalendar subtypes VEVENT/VTODO. Could it be that the new OverrideDevInf somehow causes this flag to be reset in an early stage, such that parsing the without X-AIM will still result in AIM_HANDLE enabled, because it was set somehow previously? So that doesn't seem to be it. I'm now looking at the actual parser code... gdb to the rescue. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] adding or overriding CtCap
On Mo, 2011-06-20 at 17:27 +0100, Patrick Ohly wrote:
Problem solved...
Except for one field:
!-- store extensions that don't match any of the other fields --
field name=XPROPS array=yes type=string compare=never/
Profile:
property name=X-* suppressempty=yes show=false
value field=XPROPS/
position field=XPROPS repeat=array increment=1 minshow=0/
/property
This is getting set to AVAILABLE here:
if (proplistP) {
if (!itemTypeP-fReceivedFieldDefs) {
// there is a propList, and we haven't scanned one already for this type
// (could be the case for DS 1.2 vCalendar where we get events tasks
separately)
// so disable all non-mandatory fields first (available ones will be
re-enabled)
for (sInt16 i=0; iitemTypeP-fFieldDefinitionsP-numFields(); i++) {
itemTypeP-getFieldOptions(i)-available=false;
}
// force mandatory+unprocessed properties to be always available
=setfieldoptions(NULL,fProfileDefinitionP,itemTypeP);
}
// now we have received fields
itemTypeP-fReceivedFieldDefs=true;
}
Verified with debugging output. I've not looked into setfieldoptions()
for the exact logic which enables the field.
This has the effect that my X-FOOBAR-EXTENSION and X-TEST extensions
(stored locally in XPROPS) are not getting preserved when importing the
updated contact from Google.
What would be a proper fix for this?
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] temporary UID mapping
On Fr, 2010-12-17 at 14:08 +, Lukas Zeller wrote: On Dec 17, 2010, at 8:47 , Patrick Ohly wrote: Now it turns out that there's a bad bug here, which probably explains all this (and maybe other) weird behaviour. I'm glad that you got to the bottom of this. Much better than pampering over the symptoms. So my question now is: should the code that I added to detect collisions remain in the code? The loop which detects non-unique temporary IDs is harmless. But the one which iterates over all mappings to find a previous mapping to the same local ID has a real performance impact. My preference would be this: - turn the messages in these loops into real errors - keep them for a while - remove after the real fix has proven to be effective I fully agree with this, just go ahead! Okay, working on it. I wondered why this bug did not surface in my interrupted sync tests. SyncEvolution's client-test simulates a whole range of different scenarios (items added/removed/modified) and a loss of network connection at any point in the sync session. I'm 100% certain that this passed when I declared SyncEvolution's server support ready for users. Looking at my test setup, I see that the server side was configured to use the file data stores. Those use very simple numbers as ID, and thus the whole temporary ID mapping code wasn't exercised. Rerunning the tests with the Evolution calendar store (which uses UID as local ID and thus depends on the mapping code) shows that the mapping problems *do* occur. My workarounds are triggered (visible in the logs), so the tests themselves finish successfully. With your patch applied, the are no longer triggered. IMHO this confirms that you found the root cause. An additional benefit is that the old meta data seems to be erased reliably. I don't think there's any further need for the erase mapping on refresh-sync feature that I was proposing. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] temporary UID mapping
On Do, 2010-12-16 at 17:35 +, Lukas Zeller wrote: Hello Patrick, On Dec 9, 2010, at 17:47 , Patrick Ohly wrote: I might have found it. Scenario: - SERVER is the ID on the server, CLIENT on the client - server has a mapping from SERVER to #35 (ident 2) and to CLIENT (ident 1) Where is this from? I assume it is from a previous session (that has not completed, otherwise it should not be there). Anecdotes indicate that it indeed happens when a previous session did not complete correctly. Now it turns out that there's a bad bug here, which probably explains all this (and maybe other) weird behaviour. I'm glad that you got to the bottom of this. Much better than pampering over the symptoms. So my question now is: should the code that I added to detect collisions remain in the code? The loop which detects non-unique temporary IDs is harmless. But the one which iterates over all mappings to find a previous mapping to the same local ID has a real performance impact. My preference would be this: - turn the messages in these loops into real errors - keep them for a while - remove after the real fix has proven to be effective -- Best Regards Patrick Ohly Senior Software Engineer Intel GmbH Open Source Technology Center Pützstr. 5 Phone: +49-228-2493652 53129 Bonn Germany ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] libsynthesissdk.a built without -fPIC
On Do, 2010-12-09 at 06:47 +, Andris Pavenis wrote: The patch is not sufficient. One must also set CFLAGS, not only CXXFLAGS as libsynthesissdk contains also C cources. Oh, I thought it was C++ only. Sorry! See attached patch for details (complete updated patch, not incremental). Merged into the meego.gitorious.org repo. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] sending iCalendar 2.0 recurrence rule to Nokia phone: rule ignored (BMC #11241)
On Di, 2010-12-14 at 11:43 +, Lukas Zeller wrote: Hello Patrick, On Dec 13, 2010, at 13:22 , Patrick Ohly wrote: From http://bugs.meego.com/show_bug.cgi?id=11241. Problem summary below, patch attached. I'm a bit unsure about hard-coding UTC as time context in that code, please suggest how this might be improved. [... - stored in iCalendar 2.0 format as:] RRULE:FREQ=DAILY;INTERVAL=1;UNTIL=20111231T10 [... - rendered by libsynthesis as] RRULE:D1 20111231 [...] The spec says that the end date must contain a time: enddate::= ISO 8601_date_time value(e.g., 19940712T101530Z) I agree. There are no date-only values in vCalendar 1.0 at all. However what I don't understand is why and where the the original timestamp as represented by UNTIL=20111231T10 becomes a date-only value as shown in the vCalendar 1.0? IMHO, this should be a date+time long before it gets into mimedirprofile. So I'd like to understand why this happens in this case. I see however a case that needs some treatment - which is when a date-only iCalendar 2.0 is the input (which then also can have a date-only UNTIL= part in the RRULE). Such a record would be rendered incorrectly in vCalendar 1.0 (as date-only). That's exactly the situation here. The until date/time should IMHO be handled exactly like dates with the CONVMODE_AUTOENDDATE conversion mode, which renders date-only values as floating date+time in mimo_old (vCalendar xxx), but as date-only values in MIME-DIR (iCalendar 2.0). You are not suggesting to use convmode=auto[end]date as it is, right? If I read the documentation right, it will turn 20101231 into either 20101231T235959 (autoenddateinclusive) or 20101231T00. In both cases it would cut off the last occurrence on day 20101231. But I can see how converting to 20101231T235959 might work. My goal in the patch was to recreate exactly the same RRULE as sent by the Nokia phone, with the right time embedded in it. Not sure whether that is really necessary, perhaps something larger would also work. What time it should insert as fallback might be a bit tricky, but I think the start time of the event should do. My suggestion (I'll propose a patch soon) would be to make RR_END behave like CONVMODE_AUTOENDDATE to handle the case of an input timestamp which is a date-only. Okay. Let me know when it is ready and I'll redo my tests. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] sending iCalendar 2.0 recurrence rule to Nokia phone: rule ignored (BMC #11241)
Hello! From http://bugs.meego.com/show_bug.cgi?id=11241. Problem summary below, patch attached. I'm a bit unsure about hard-coding UTC as time context in that code, please suggest how this might be improved. - This was sent to us by the phone: BEGIN:VCALENDAR VERSION:1.0 TZ:+01 DAYLIGHT:TRUE;+02;20100328T01Z;20101031T01Z;; DAYLIGHT:TRUE;+02;20110327T01Z;20111030T01Z;; BEGIN:VEVENT SUMMARY:Phone DTSTART:20101212T09Z DTEND:20101212T09Z CLASS:PRIVATE RRULE:D1 20111231T10 AALARM;TYPE=X-EPOCSOUND:20101212T084500Z;;; LAST-MODIFIED:20101213T081626Z PRIORITY:2 END:VEVENT END:VCALENDAR Translated into: BEGIN:VCALENDAR VERSION:2.0 PRODID:-//Synthesis AG//NONSGML SyncML Engine V3.4.0.8//EN BEGIN:VTIMEZONE TZID:Europe/Zurich BEGIN:STANDARD DTSTART:19671029T03 RRULE:FREQ=MONTHLY;INTERVAL=12;BYDAY=-1SU TZOFFSETFROM:+0200 TZOFFSETTO:+0100 TZNAME:CET END:STANDARD BEGIN:DAYLIGHT DTSTART:19870329T02 RRULE:FREQ=MONTHLY;INTERVAL=12;BYDAY=-1SU TZOFFSETFROM:+0100 TZOFFSETTO:+0200 TZNAME:CEST END:DAYLIGHT END:VTIMEZONE BEGIN:VEVENT LAST-MODIFIED:20101213T081626Z CLASS:PRIVATE PRIORITY:2 SUMMARY:Phone DESCRIPTION:Phone DTSTART;TZID=Europe/Zurich:20101212T10 RRULE:FREQ=DAILY;INTERVAL=1;UNTIL=20111231T10 DTEND;TZID=Europe/Zurich:20101212T10 BEGIN:VALARM TRIGGER;VALUE=DATE-TIME:20101212T084500Z ACTION:DISPLAY END:VALARM END:VEVENT END:VCALENDAR After changing the summary, it is sent back to the phone as an update as: BEGIN:VCALENDAR VERSION:1.0 TZ:+01:00 DAYLIGHT;ENCODING=QUOTED-PRINTABLE:TRUE;+02;20110327T01Z;20111= 030T01Z;CET;CEST BEGIN:VEVENT LAST-MODIFIED:20101213T091201Z CLASS:PRIVATE PRIORITY:2 SUMMARY:Phone mod DESCRIPTION:Phone DTSTART:20101212T09Z RRULE:D1 20111231 DTEND:20101212T09Z AALARM:20101212T084500Z;;;Phone mod DALARM:20101212T084500Z;;;Phone mod END:VEVENT END:VCALENDAR One relevant difference might be the explicit time in the RRULE. The phone includes it, the Synthesis engine doesn't. Yep, that seems to be it. Both forever (RRULE:D1 #0) and fixed number of times work. The later one works because the Synthesis engine internally converts it into a fixed end date/time. The spec says that the end date must contain a time: enddate::= ISO 8601_date_time value(e.g., 19940712T101530Z) It does not say whether that end datetime is included in the recurrence, but experiments show that this is the interpretation on the phone (vCalendar 1.0) and in Evolution (iCalendar 2.0). Given that vCalendar does not allow floating end dates, I think the encoder for vCalendar RRULE should be fixed to not generate such broken rules. What time it should insert as fallback might be a bit tricky, but I think the start time of the event should do. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. From 400b471a43d02dbcdf2c6affdde7ac1b1d7171a8 Mon Sep 17 00:00:00 2001 From: Patrick Ohly patrick.o...@intel.com Date: Mon, 13 Dec 2010 13:08:40 +0100 Subject: [PATCH] vCalendar 1.0: avoid RRULE end date without time When parsing an iCalendar 2.0 event, it is possible to arrive at an rrule until date/time which contains no time value. vCalendar 1.0 requires such a time. Not sending it, as the engine did before, causes Nokia phones to ignore the recurrence rule (bugs.meego.com #11241). This patch avoids this case by looking at the time of the event start and adds that time to the recurrence until date. The comment in the default config says: !-- recurrence rule block, fields must be in that order, including DTSTART as last field !! -- But it doesn't seem like DTSTART was really used before. Therefore this patch tries to deal with not finding a timestamp after the recurrence rule block by clearing the TCTX_DATEONLY, without actually setting a time. Probably not quite correct, but perhaps better than not changing anything. Another problematic aspect is to match the time zone context of the until clause with the start date/time of the event. Currently the patch assumes that the start time is going to be encoded as UTC. --- src/sysync/mimedirprofile.cpp | 30 ++ 1 files changed, 30 insertions(+), 0 deletions(-) diff --git a/src/sysync/mimedirprofile.cpp b/src/sysync/mimedirprofile.cpp index 2cbce65..1314240 100644 --- a/src/sysync/mimedirprofile.cpp +++ b/src/sysync/mimedirprofile.cpp @@ -1700,6 +1700,12 @@ bool TMimeDirProfileHandler::fieldToMIMEString( // - until if (!(tsFldP = ITEMFIELD_DYNAMIC_CAST_PTR(TTimestampField,fty_timestamp,aItem.getArrayField(aFid,aArrIndex,true return false; aFid++; // do NOT INCREMENT in macro, as it would get incremented twice + // - start time (according to comment for recurrence rule block, this must
[os-libsynthesis] importing time zone definitions + TZID
Hello Beat, Lukas!
While debugging an issue (http://bugs.meego.com/show_bug.cgi?id=9600) I
noticed that SyncEvolution was sending broken vCalendar 1.0 DAYLIGHT
properties.
I tracked it down to the assumption that tz_entry::name is composed of
standard and daylight saving zone names (CET/CEST), which I broken in
the libical timezone import code on Linux.
I'd like to suggest that this assumption gets relaxed such that the
original TZID can be stored together with these two names in a tz_entry.
See attached patches, which I also committed to master on
meego.gitorious.org.
Is that okay and suitable for merging back?
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
From d9f9af69176d776d09f99d85789973fd5d3b10f4 Mon Sep 17 00:00:00 2001
From: Patrick Ohly patrick.o...@intel.com
Date: Sun, 12 Dec 2010 21:19:16 +0100
Subject: [PATCH 1/2] vtimezones: explicitly store std and dst TZNAME
In vCalendar 1.0, the DAYLIGHT property must list standard and daylight
saving zone names, for example CET;CEST.
ContextToTzDaylight() used to extract these two names from the
tz_entry::name field. This was rather implicit and not documented, so
the Linux timezone import code got it wrong and stored the original
TZID (like
/freeassociation.sourceforge.net/Tzfile/Australia/Melbourne), leading
to an invalid DAYLIGHT property with too many components. Such an
entry is accepted by Nokia phones in an ADD, but not an UPDATE
(bugs.meego.com #9600).
The code would also have failed for a (hypothetical) VTTIMEZONE
definition with TZNAMEs that contain slashes.
To make the code more robust and easier to understand, this patch
introduces explicit dstName and stdName fields in
tz_entry. VTIMEZONEtoTZEntry() sets them directly, thus allowing the
removal of two parameters in that function.
ContextToTzDaylight() still falls back to splitting the tz_entry::name
field if dstName or stdName are not set when needed, so these new
fields are strictly optional. In particular the code which creates
tz_entry from the builtin table can work as before.
The code which replaces the original TZID in tz_entry::name with a
combination of its TZNAMEs is still active, although it is no longer
needed. It probably was added in the first place to preserve the two
names so that they can be exported again.
---
src/platform_adapters/linux/platform_timezones.cpp |3 -
src/sysync/timezones.h | 14 +++-
src/sysync/vtimezone.cpp | 83 +++-
src/sysync/vtimezone.h |2 -
4 files changed, 76 insertions(+), 26 deletions(-)
diff --git a/src/platform_adapters/linux/platform_timezones.cpp b/src/platform_adapters/linux/platform_timezones.cpp
index d65f868..3fdc15b 100644
--- a/src/platform_adapters/linux/platform_timezones.cpp
+++ b/src/platform_adapters/linux/platform_timezones.cpp
@@ -191,12 +191,9 @@ void finalizeSystemZoneDefinitions(GZones* aGZones)
continue;
PLOGDEBUGPUTSX(aGZones-getDbgLogger, DBG_PARSE+DBG_EXOTIC, vtimezone);
tz_entry t;
-string dstName, stdName;
if (VTIMEZONEtoTZEntry(
vtimezone,
t,
- stdName,
- dstName,
#ifdef SYDEBUG
aGZones-getDbgLogger
#endif
diff --git a/src/sysync/timezones.h b/src/sysync/timezones.h
index d3f30d4..6d16e90 100755
--- a/src/sysync/timezones.h
+++ b/src/sysync/timezones.h
@@ -66,7 +66,19 @@ typedef struct tChangeStruct {
class tz_entry {
public:
- std::string name; /** name, same as TZID in VTIMEZONE, e.g. CET/CEST or /softwarestudio.org/Tzfile/Europe/Berlin */
+ std::string name; /** name, same as TZID in
+VTIMEZONE, e.g. CET/CEST or
+/softwarestudio.org/Tzfile/Europe/Berlin;
+see also dst/stdName */
+ std::string stdName; /** optional standard time name, e.g. CEST; must be
+set if name is not a concatenation of
+standard and daylight name (CET/CEST);
+the vCalendar 1.0 code relies on that */
+ std::string dstName; /** used instead of splitting
+name if (and only if)
+stdName is set; may be empty
+in zones without daylight
+saving */
std::string location; /** location string as used in Olson TZID, e.g. Europe/Berlin */
short bias; /** minutes difference to UTC (west negative, east positive */
short
Re: [os-libsynthesis] temporary UID mapping
On Do, 2010-12-09 at 11:26 +, Lukas Zeller wrote: Still, the thing is that there's never a deletion of a single tempGUID map entry (of course, there ARE deletions of map entries with other types, but not of mapentry_tempidmap). Does that type map to sysync::cMapID::ident? In my mapping on disk I currently see only entries with ident == 2 and a a remote ID. So you are saying that entries with ident == 2 are never meant to be removed from the mapping? Won't that overflow the disk after (some admittedly rather long) usage with many new items transferred? If you see the number of mapentry_tempidmaps decreasing during a sync session (counted from that anchorpoint at line 5852 in localengineds.cpp), there must be a problem with making these persistent and loading them later on (or some other not yet discovered bug - in the engine or the DB plugins). There definitely were missing entries. I'll keep watching :-/ -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] temporary UID mapping
On Do, 2010-12-09 at 13:18 +, Lukas Zeller wrote: So you are saying that entries with ident == 2 are never meant to be removed from the mapping? Not exactly that - what I'm saying is that these are (or should) always be removed all together. So either the list is still growing, or all of its items should go away at the same time, which cleans up the ID space. I might have found it. Scenario: - SERVER is the ID on the server, CLIENT on the client - server has a mapping from SERVER to #35 (ident 2) and to CLIENT (ident 1) - item is deleted on server - two-way sync What I now see in the log is this: fTempGUIDMap: restore mapping from #35 to SERVER Item LocalID='SERVER', RemoteID='CLIENT', operation=delete SyncOp=delete: LocalID=SERVER RemoteID=CLIENT fTempGUIDMap: translated realLocalID='SERVER' to tempLocalID='#63' The fTempGUIDMap is a map tempLocalID-ID. It now contains #65-SERVER and #35-SERVER. But the map in the DB AP is a map from (ID, ident) - ([remote ID], flags). It cannot hold both #65-SERVER and #35-SERVER. So what I see next is the saving of these two entries: ModifyMap called: aEntryType=tempidmap, aLocalID='SERVER, aRemoteid='#35', aMapflags=0x0, aDelete=0 found entry by entrytype/localID='SERVER' - remoteid='#35', mapflags=0x0, changed=0, deleted=1, added=0, markforresume=0, savedmark=0 ModifyMap called: aEntryType=tempidmap, aLocalID='SERVER, aRemoteid='#63', aMapflags=0x0, aDelete=0 found entry by entrytype/localID='SERVER' - remoteid='#35', mapflags=0x0, changed=0, deleted=0, added=0, markforresume=0, savedmark=0 apiSaveAdminData: entryType=normal, localid='SERVER', remoteID='CLIENT', mapflags=0x8, changed=0, deleted=0, added=0, markforresume=1, savedmark=0 UpdateMapItem 'SERVER' + 1 = 'CLIENT' + 9, res=0 apiSaveAdminData: entryType=tempidmap, localid='SERVER', remoteID='#63', mapflags=0x0, changed=1, deleted=0, added=0, markforresume=0, savedmark=0 UpdateMapItem 'SERVER' + 2 = '#63' + 0, res=0 The last UpdateMapItem overwrites the entry for #35, because it uses the same ID + ident. The next sync then populates fTempGUIDMap with entry #63, but has no #35 and thus the size() + 1 assumption no longer holds. I remember that I was confused about the map item DB calls. Originally I implemented that as (ID) - (ident, [remote ID], flags). But after we discussed it, I changed that to (ID, ident) - ([remote ID], flags), which IMHO is what the API description says: /*! Map table handling: Update a map item of this context * * @param aContext The datastore context * @param mIDMapID ( with \localID,\remoteID, \flags and \ident ). * If there is already a MapID element with \localID and \ident, * it will be updated, else created. * * @return error code, if this MapID can't be updated (e.g. not yet existing). * * USED ONLY WITH \plugin_datastoredadmin */ _ENTRY_ TSyError UpdateMapItem( CContext aContext, cMapID mID ); So, what is the solution to this problem? Do I still misunderstand something or is there a genuine problem? ;-/ FWIW, there was one DeleteMapItem for item SERVER in the sessions above, but it was for ident 1 (after deleting it). -- Best Regards Patrick Ohly Senior Software Engineer Intel GmbH Open Source Technology Center Pützstr. 5 Phone: +49-228-2493652 53129 Bonn Germany ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] temporary UID mapping - maximum ID size
On Mi, 2010-12-08 at 12:22 +0100, Patrick Ohly wrote: I'm currently (involuntarily ;-) stress-testing this code by running SyncEvolution-SyncEvolution syncs with lots of iCalendar 2.0 items, which happen to have very long IDs. Related to this: is there some way to increase the maximum ID size advertised by the Synthesis engine? Does that make sense - tradeoff between sending long IDs in messages and having to deal with the mapping? It seems that currently the size is hard-coded, with different values for binfile client and server. Server: libsynthesis/src/sysync/localengineds.cpp: fMaxGUIDSize = 64; Client: libsynthesis/src/sysync/binfileimplds.cpp:fMaxGUIDSize=BINFILE_MAXGUIDSIZE; libsynthesis/src/sysync/binfileimplds.h:#define BINFILE_MAXGUIDSIZE 63 -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] temporary UID mapping
Hello! The Synthesis engine has the feature that its backends are allowed to use IDs of arbitrary length. The engine will translate into IDs shorter than the maximum ID size supported by the peer. TLocalEngineDS::adjustLocalIDforSize() creates these temporary IDs, using: fTempGUIDMap.size() + 1 // as list only grows, we have unique tempuids for sure I'm currently (involuntarily ;-) stress-testing this code by running SyncEvolution-SyncEvolution syncs with lots of iCalendar 2.0 items, which happen to have very long IDs. I see failures where the server assigns the same temporary ID to different items in the same sync. I've added debug logging. It shows that the following happens: 1. fTempGUIDMap is restored from the map file such that it has 105 entries, *but* these are non-contiguous (from #1 to #124). 2. fTempGUIDMap.size() + 1 is #106, which already exists in fTempGUIDMap. 3. Overwriting that entry does not increase fTempGUIDMap.size(). 4. A second item is assigned the same #106 temp ID. I don't know exactly how I arrived at this state. Is the on-disk dump of fTempGUIDMap really meant to preserve all temporary IDs in perpetuity? I don't think so, because that would fill up the disk and there is a delete map item operation. Unless there is some additional constraint, then the assumption that the list only grows is wrong. I have added a workaround which checks for ID collisions in TLocalEngineDS::adjustLocalIDforSize(). Is that the right solution or do I need to search for the reason why the mapping has gaps? Also note that I end up with a temporary ID which must have been used before in an older sync session. Could that be a problem? -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] libsynthesissdk.a built without -fPIC
On Mi, 2010-11-24 at 14:07 +, Andris Pavenis wrote: libsynthesissdk.a can be linked into external libsynthesis plugins, but is built without -fPIC. It works OK for 32 bit Linux but not 64 bit Linux. I'm getting error message: /usr/bin/ld: /usr/lib64/libsynthesissdk.a(libsynthesissdk_la-SDK_support.o): relocation R_X86_64_32 against `a local symbol' can not be used when making a shared object; recompile with -fPIC /usr/lib64/libsynthesissdk.a: could not read symbols: Bad value collect2: ld returned 1 exit status I'm aware of the need for -fPIC on 64 bit. What I don't quite understand is that I'm not seeing that problem when compiling on it myself and that we haven't heard of the problem elsewhere before. libsyncevolution.so links fine against a libsynthesissdk.a. What I see is that libtool invokes g++ with -fPIC -DPIC when compiling libsynthesissdk.a source files. This is with libtool 2.2.6b on Debian Testing, with --enable-shared --enable-static. Also works in other combinations, expect with an explicit --disable-shared, but then also libsynthesis.a is compiled without PIC mode. In that case I wouldn't expect libsynthesissdk.a to work inside a shared library, although I don't know whether that is formally defined anywhere in libtool or what the common way of handling these configure flags is. PIC mode causes a slight performance degradation, so it makes sense to not use it in static libs which are meant to be used in executables. I added following patch when building own RPM package (libsynthesis-3.4.0.16) --- libsynthesis_3.4.0.16/src/Makefile.am.in.orig 2010-11-24 15:52:52.0 +0200 +++ libsynthesis_3.4.0.16/src/Makefile.am.in2010-11-24 15:53:24.0 +0200 @@ -115,6 +115,7 @@ else libsynthesissdk_la_SOURCES += @LIBSYNTHESISSDK_SOURCES_BOTH@ endif libsynthesissdk_la_CPPFLAGS = \ + -fPIC \ -D_GNU_SOURCE=1 \ -include $(top_builddir)/config.h \ -I$(srcdir)/Targets/ReleasedProducts/SDK \ I'm a bit reluctant to hard-code -fPIC in the makefile - what if the compiler doesn't support it? Unlikely, but not impossible. Does the attached patch work for you? If it does, then I can include it in the next Linux libsynthesis update (which usually happens via SyncEvolution) and it trickle into upstream libsynthesis the next time Lukas merges back changes. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. From 8c8c2b243de4e1a3f013881eef48369c437156eb Mon Sep 17 00:00:00 2001 From: Patrick Ohly patrick.o...@intel.com Date: Mon, 6 Dec 2010 10:43:49 +0100 Subject: [PATCH] autotools: always enforce PIC mode for libsynthesissdk.a Some versions of libtool use PIC mode automatically for static libraries, others don't (?!). See the [os-libsynthesis] libsynthesissdk.a built without -fPIC for a problem report. This patch extracts PIC flags from libtool configure flags and uses them for the two static libraries (libsynthesissdk.a and libsynthesisstubs.a). This is a bit more portable than hard-coding the -fPIC flag, albeit making an assumption about the naming of libtool internal variables. --- configure.in |7 +++ src/Makefile.am.in |2 ++ 2 files changed, 9 insertions(+), 0 deletions(-) diff --git a/configure.in b/configure.in index b0ee64a..e2a0c14 100644 --- a/configure.in +++ b/configure.in @@ -12,6 +12,13 @@ AC_PROG_LIBTOOL AC_PROG_MAKE_SET AM_PROG_CC_C_O +dnl Extract PIC flags from libtool configure for libsynthesissdk.a +dnl (makes assumptions about libtool var naming!). Some versions +dnl of libtool use PIC mode automatically for static libraries, +dnl others don't (?!). See [os-libsynthesis] libsynthesissdk.a built without -fPIC. +PIC_CXXFLAGS=$lt_prog_compiler_pic_CXX +AC_SUBST(PIC_CXXFLAGS) + AC_ARG_ENABLE(debug-logs, AS_HELP_STRING([--debug-logs], [For developers: add links to call location to HTML log files. Depends on Doxygen (for HTML version of source) and g++ (for __PRETTY_FUNCTION__).]), diff --git a/src/Makefile.am.in b/src/Makefile.am.in index e412171..7be4a00 100644 --- a/src/Makefile.am.in +++ b/src/Makefile.am.in @@ -108,6 +108,7 @@ XMLPARSE_CFLAGS_BUILTIN = -I$(srcdir)/Targets/ReleasedProducts/clientEngine_auto endif libsynthesissdk_la_LDFLAGS = -static +libsynthesissdk_la_CXXFLAGS = $(PIC_CXXFLAGS) libsynthesissdk_la_SOURCES = @LIBSYNTHESISSDK_HEADERS@ if COND_STATIC libsynthesissdk_la_SOURCES += @LIBSYNTHESISSDK_SOURCES_SDK_ONLY@ @@ -128,6 +129,7 @@ libsynthesissdk_la_CPPFLAGS = \ -I$(srcdir)/syncml_tk/src/sml/mgr/inc/ libsynthesisstubs_la_LDFLAGS = -static +libsynthesisstubs_la_CXXFLAGS = $(libsynthesissdk_la_CXXFLAGS) libsynthesisstubs_la_SOURCES = sysync_SDK/Sources/enginestubs.c
Re: [os-libsynthesis] slow-sync matching
On Mo, 2010-11-15 at 16:50 +, Patrick Ohly wrote: Now that I had that sorted out, I was running into another configuration issue: slowsyncstrategyduplicate/slowsyncstrategy is set in the syncserv_sample_config.xml and SyncEvolution copied that. Therefore the engine ended up trying to duplicate events despite a UID match. This is not possible in the backend, which will always overwrite the existing event. So what I need in this case is newer-wins with server-wins or client-wins as fallback instead of duplicate - such a setting does not exist, does it? After some more thinking I came to the conclusion that the fallback is not needed for my iCalendar 2.0 case: because UID/RECURRENCE-ID will always find pairs (if they exist) and LAST-MODIFIED is always there, there will always be a newer item, so the duplicate fallback will never trigger. Finally, remember that I had intentionally created differences in the SUMMARY of my two events. My expectation was that this difference would be resolved during syncing. [...] Note that I made another change for my iCalendar 2.0 case: all fields have compare=never, only UID/RECURRENCE-ID has compare=always. Could that have an effect on merging once pairs are found? After looking at the code it seems so. compareWith() returns that the items are identical enough, despite the change in SUMMARY. Yes, that was it. I had the idea of introducing a new compare level between never and conflict to mark fields which are relevant for the user (and thus changes must be stored), without using them to find pairs (which levels = conflict would do). It turned out that Lukas already was ahead of me, as usual ;-) The new level already exists, although in this case the purpose was a bit different: commit d96d05611cf7d35459927328e18b7f0f0a74571e Author: Lukas Zeller l...@synthesis.ch Date: Tue Aug 24 17:41:53 2010 +0200 engine: added new compare mode for fields: scripted These are NOT included in standard comparisons, so the comparison can be done in a script, but they are considered relevant otherwise, in particular these are included when calculating CRC for change detection. I'm happy to report that using compare=scripted solves the SUMMARY not updated problem. Solved for me. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] slow-sync matching
On Do, 2010-11-11 at 14:52 +, Patrick Ohly wrote: Hello! I have a question about matching items. Here's my situation: * Client and server both have the same iCalendar 2.0 VEVENT (UID is identical). * SUMMARY is changed on server, one line is added to description. * SUMMARY is changed on client. * Slow sync. I had changed the calendar field list so that all fields have compare=never, except for UID and ORIGSTART, which have compare=always. The rationale is that if it is known that both sides support UID, that alone should be used to find matches. What I expect in this case is that: 1. Synthesis engine finds the match. 2. The more recently modified SUMMARY from the client is preserved (DMODIFIED = LAST-MODIFIED is marked as age=yes). 3. The additional line of the description is preserved (DESCRIPTION has merge=lines). What happens instead is that the engine finds the match, but then skips merging the items and updating them on server and client, leaving them unsynchronized. Is that the desired behavior? Do I have to configure the engine differently? Yes. Lukas told me that there are configuration options, and then I also found them in the documentation: 11.36.6 updateclientinslowsync: update client records during slowsync Contained in: remoterule or server (settings in server are defaults for sessions with- out a remote rule applied or with a remote rule applied that does not specify a value for this particular option) Can contain: boolean value Attributes:none Default: off If this option is set in a remoterule, the SyncML server will try to update client records with server data if comparison shows that the server has additional data that the client does not have during non-first time slow sync. Note that this updating always takes place in first-time sync, regardless of this options's setting. It is off by default because many clients cannot store some data but also cannot inform the server what they can store exactly, so turning this option on will cause much un- needed client updates at slow sync. For clients that store everything they report in devinf how- ever, this feature can be switched on resulting in better data consistency after a slow sync. See also corresponding function in 6.14.6. 11.36.7 updateserverinslowsync: update server records during slowsync Contained in: remoterule or server (settings in server are defaults for sessions with- out a remote rule applied or with a remote rule applied that does not specify a value for this particular option) Can contain: boolean value Attributes:none Default: off If this option is set in a remoterule, the SyncML server will try to update server records with cli- ent data if comparison shows that the client has additional data that the server does not have during non-first time slow sync. Note that this updating always takes place in first-time sync, regardless of this options's setting. It is off by default because differences in client and server database layouts often cause uneeded updates which will cause other clients to be updated on the next sync as well. For clients that store everything they report in devinf however, this feature can be switched on result- ing in better data consistency after a slow sync. See also corresponding function in 6.14.6. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] slow-sync matching
Hello! I have a question about matching items. Here's my situation: * Client and server both have the same iCalendar 2.0 VEVENT (UID is identical). * SUMMARY is changed on server, one line is added to description. * SUMMARY is changed on client. * Slow sync. I had changed the calendar field list so that all fields have compare=never, except for UID and ORIGSTART, which have compare=always. The rationale is that if it is known that both sides support UID, that alone should be used to find matches. What I expect in this case is that: 1. Synthesis engine finds the match. 2. The more recently modified SUMMARY from the client is preserved (DMODIFIED = LAST-MODIFIED is marked as age=yes). 3. The additional line of the description is preserved (DESCRIPTION has merge=lines). What happens instead is that the engine finds the match, but then skips merging the items and updating them on server and client, leaving them unsynchronized. Is that the desired behavior? Do I have to configure the engine differently? From the log file: # [2010-11-11 15:40:44.964] Testing filter '' against item: # [2010-11-11 15:40:44.964] Filter test result is TRUE # [2010-11-11 15:40:44.964] comparing (this) local item localID='2010t135424z-8821-33500-1...@pohly-mobl1-rid' with incoming (other) item remoteID='2010t135424z-8821-33500-1...@pohly-mobl1.ics' # [2010-11-11 15:40:44.964] Compared [loc=2010t135424z-8821-33500-1...@pohly-mobl1-rid,REM=] with [LOC=,rem=2010t135424z-8821-33500-1...@pohly-mobl1.ics] (eqMode=1), cmpres=0 # [2010-11-11 15:40:44.964] TStdLogicDS::getMatchingItem, found remoteID='2010t135424z-8821-33500-1...@pohly-mobl1.ics' is equal in content with localID='2010t135424z-8821-33500-1...@pohly-mobl1-rid' # [2010-11-11 15:40:44.964] Slow Sync Match detected - localID='2010t135424z-8821-33500-1...@pohly-mobl1-rid' matches incoming item # [2010-11-11 15:40:44.964] Compared [loc=2010t135424z-8821-33500-1...@pohly-mobl1-rid,REM=] with [LOC=,rem=2010t135424z-8821-33500-1...@pohly-mobl1.ics] (eqMode=2), cmpres=0 # [2010-11-11 15:40:44.964] calendar: testState=TRUE - expected state='sync_mode_stable', found state=='sync_set_ready' # [2010-11-11 15:40:44.964] ModifyMap called: aEntryType=normal, aLocalID='2010t135424z-8821-33500-1...@pohly-mobl1-rid, aRemoteid='2010t135424z-8821-33500-1...@pohly-mobl1.ics', aMapflags=0x0, aDelete=0 # [2010-11-11 15:40:44.964] - found entry by entrytype/localID='2010t135424z-8821-33500-1...@pohly-mobl1-rid' - remoteid='2010t135424z-8821-33500-1...@pohly-mobl1.ics', mapflags=0x0, changed=0, deleted=1, added=0, markforresume=0, savedmark=0 # [2010-11-11 15:40:44.964] - matching entry found - re-activating deleted and/or updating contents if needed # [2010-11-11 15:40:44.964] Map entry updated: LocalID='2010t135424z-8821-33500-1...@pohly-mobl1-rid', RemoteID='2010t135424z-8821-33500-1...@pohly-mobl1.ics' # [2010-11-11 15:40:44.964] Preventing localID='2010t135424z-8821-33500-1...@pohly-mobl1-rid' to be sent to client # [2010-11-11 15:40:44.964] Item with localID='2010t135424z-8821-33500-1...@pohly-mobl1-rid' will NOT be sent to client (usually due to slowsync match) # [2010-11-11 15:40:44.964] Deleted command 'Status' (outgoing MsgID=0, CmdID=0) -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] libsynthesis update - 3.0.4.16 (was: Re: optional property parameters, sub/remoterules)
On Do, 2010-09-23 at 02:30 -0700, Lukas Zeller wrote: I owe you an answer regarding commits 0e139ee311 and 4275e77001: [...] But altough going back is possible here, the binfile store generally does not provide going back (that is, without loosing all settings), only forward. As soon as something changes in the data format, the DB version field in the binfile is incremented, and it will no longer be compatible with the previous version. In general that approach is fine, I wasn't expecting anything else. It's just that I'd like to warn users and whenever possible, give them the choice of going back. That increases the number of people who can try out an experimental snapshot, because they can try something and then fall back to the stable version more easily. Given that these two changes are fairly intrusive, I think I'll pass for SyncEvolution 1.1. Instead I'll release with some specific fixes backported, and then combine the libsynthesis update with some other non-backward compatible changes in SyncEvolution itself. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] sending photo to Nokia phone (N97 mini): wrong CTCap
On Thu, 2010-08-26 at 23:00 +0100, Lukas Zeller wrote: it's a known bug in many Nokia phones that they have a maxSize of 255 on all fields, including the PHOTO. And yes, there is a remoterule option called ignoredevinfmaxsize for that. It applies to all fields, however I agree that those constant 255 maxsizes are probably meaningless anyway. Ah, excellent. I was sure there would be something, and I wasn't disappointed ;-) Speaking of CtCap, do you happen to know what Nokia's involvement was in the definition of that part of the spec? Or speaking more generally, what was the desired usage of CtCap? I've just been in another discussion around that, where the Synthesis use of CtCap to determine unsupported properties was questioned. One interpretation of CtCap is limitations for some properties. In that interpretation, unlisted properties might still be supported. The other interpretation is that CtCap has to be complete and accurate, and thus anything not covered by it (value too long, unknown property) is not stored by the device. This is the interpretation used by Synthesis. FWIW, it sounds more plausible to me. But is there anything in the standard which supports one or the other interpretation? Or perhaps it was part of meeting minutes? -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] DB API + simple type
Hello! I'm working on a modification of the engine which bypasses the obligatory data conversion via field lists and profiles. The use case for this are client and server which perfectly agree on some proprietary item data and thus need no conversion, or plugins which want to handle the data conversion themselves. So far I have hooked up the existing TSimpleItem with the config parsing (added type and config for it). Now I try to figure out how to read and write the item data and possibly meta data via the plugin API. My initial thought was to expose certain pre-defined field names and use the *AsKey variants of the API, but that seems to depend on inheriting from TMultiFieldItem (pluginapids.cpp needs to down-cast to that if fPluginDSConfigP-fItemAsKey is true). Now I'm looking at the traditional API instead (the #ifdef DBAPI_TEXTITEMS part). In that context I have a question about the API as defined in sync_dbapi.h: * NOTE: By default, the SyncML engine asks for \aID only. * \aItemData can be returned, if anyway available or * \aItemData must be returned, if the engine asks for it * (when calling ReadNextItem:allfields at 'ContextSupport' with \allfields). */ _ENTRY_ TSyError ReadNextItem ( CContext aContext, ItemID aID, appCharP *aItemData, sInt32 *aStatus,bool aFirst ); The sentence in the NOTE is a bit long and I don't understand the if the engine asks for it part. How does it call ReadNextItem:allfields? After reading the documentation of ContextSupport() it becomes a bit clearer, but not much. Where is the complete list of support rules that the plugin might have to handle? If I understand this right, the plugin can choose to support the initial n rules, but not some specific ones. I assume the ordering of rules is chosen so that there's never a need to not support, say rule #1 but support rules #2 and #3? -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] DB API + simple type
On Fri, 2010-05-14 at 10:46 +0100, Beat Forster wrote: Hello Patrick I'm working on a modification of the engine which bypasses the obligatory data conversion via field lists and profiles. The use case for this are client and server which perfectly agree on some proprietary item data and thus need no conversion, or plugins which want to handle the data conversion themselves. You're talking about LoadAdminData/SaveAdminData ? No, I'm thinking about the actual item content. After reading the documentation of ContextSupport() it becomes a bit clearer, but not much. Where is the complete list of support rules that the plugin might have to handle? There is currently only this rule supported by the engine. Originally we thought there would be much more of them, but this wasn't the case until today. Ah, okay. So what exactly is the meaning of ReadNextItem:allfields? What effect does it have when the plugin refuses to acknowledge that rule? -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] DB API + simple type
On Fri, 2010-05-14 at 17:28 +0100, Lukas Zeller wrote: Hello Patrick, I think by far the simplest method would be NOT to try to bypass MultifieldItem, but derive it with an implementation that puts/gets data (and maybe metadata) into/from well-known string or BLOB fields. The entire DB API thing was developed long after we more or less settled for TMultiFieldItem as a common denomiantor for all data types. Yeah, I noticed. I have a sync starting based on the TSimpleItem and the string-based API, but now I run into more places where a TMultiFieldItem is expected, like for example in customimplds.cpp: 3118 // - create dummy item 3119 TStatusCommand dummy(getSession()); 3120 TMultiFieldItem *delitemP = 3121static_castTMultiFieldItem *(newItemForRemote(ity_multifield)); 3122 delitemP-setSyncOp(sop_delete); 3123 PDEBUGPRINTFX(DBG_DATA,( 3124Zapping datastore: deleting %ld items from database, 3125(long)fSyncSetList.size() The TField mechanism has proven so useful in various contexts, like the API itself, but also for scripting etc. that while deriving from beyond TMultiFieldItem would still be possible, I see no benefit (and a lot of drawbacks) actually doing so. Initializing a TMultiFieldItem(Type) seemed more complex than necessary, but I agree, it probably is easier to set it up right and just use it. If it's of any help for you, I could offer to add such a thing real soon (this weekend?). That would be a big help. I'm fairly confident that I would get it done myself (in particular after having spent some time with the code), but there might be snags that'll delay me that you know how to avoid. My goal was to have something rudimentary next Monday, so if it is not too much bother for you, then I'll punt the problem to you. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] logging the source code location of debug messages
On Wed, 2010-05-12 at 17:43 +0100, Lukas Zeller wrote: that's a very cool addition indeed :-) Thanks! Glad you like it :-) I'm about to integrate it here, and I have added config options for it, such that I can choose to create txmt:// links (which directly open TextMate or BBEdit code editors on my Mac with the cursor on the right line) instead of doxygen. A second config option allows to override the default base path (the one you define with SYDEBUG_LOCATION). Something similar might also be useful on Linux. These Doxygen runs are fairly slow. Something for later... I'll post the changes soon together with the other pending changes on my branch. Note that I updated master and doxygen once more today. Please pull again before merging. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] wrong interpretation of timezone information
On Mon, 2010-05-10 at 16:48 +0100, Lukas Zeller wrote: Hi Patrick, On May 10, 2010, at 9:04 , Patrick Ohly wrote: Confirmed, works for me. Thanks! I should have mentioned that I had already made similar changes as you on the moblin.org master branch (check for NULL obj in debug macros, enable debug logging inside vtimezone code when invoked by mimedir profile parser). My own debug logging patches go a bit further: - log an error when parsing a VTIMEZONE fails - logging of libical timezone conversion - logging of timezone matching I'm trying to track down an unexpected timezone failure with this which occurs when normal Evolution events are involved. Do you agree with these changes? Would you mind merging them? I rebased them on your luz branch. The attempt to init the time zones after reading config (fc000b0f45 (TSyncAppBase: moved system time zone parsing after config reading)) creates a chicken and egg problem. The config reading process relies itself on the time zone stuff being initialized, most prominently for definetimezone, but also for other tags that have ISO8601 timestamp values. I see. Maybe the dilemma can be solved by adding an optional, second initialize run after config reading that can be switched on specifically to track down timezone init problems. That would imply redoing some of the work. What about the following (code in master, as I need to fix that anyway): commit 7bbec4fd20cde059973d4d1c840fcee85d947e60 Author: Patrick Ohly patrick.o...@intel.com Date: Tue May 11 16:57:35 2010 +0200 time zone init and logging: split into part before and after config parsing This patch reverts the move system time zone parsing after config reading part of commit fc000b0f45. As Lukas pointed out, config parsing already depends on time zones, for example for definetimezone. But this patch keeps the finishConfig() callback and uses it to invoke a second global time zone function. On Linux, that function is used to add the libical time zone definitions and logs them at the same time. Builtin time zones are added at the original time, before parsing the config. This is a compromise between having no time zones while parsing the config and not logging the libical time zones. It works for SyncEvolution, because it doesn't reference the additional time zones in the config. Another option would be to use the ConferrPuts() channel, which is ready before config is read by definition. That won't help, because I need this information in the normal log file to simplify remote debugging. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] UID in iCalendar 2.0
Hello! I tracked down a problem caused by Evolution not being able to store two iCalendar 2.0 items with the same UID. The root cause is that the server doesn't know this, leading to http://bugzilla.moblin.org/show_bug.cgi?id=9682 - clear all items on a server which uses the file backend - slow sync with one item on client A - modify item on client A so that a key element differs (like SUMMARY in task) - slow sync again Expected result: - client and server in sync, either with one item or two Actual result: - server has two items, client one (the unmodified one from the initial sync) The root cause is that the server treats these two items as different when using the file backend, because neither that backend nor the engine know about the significance of the UID property, which happens to be identical here. In the second slow sync, it sends an Add command with the item that it had from the initial sync. The client then turns the Add into a Replace of the item that it just sent to the server, overwriting the local, more recent data with the older copy from the server. The client uses the EDS backend, which detects that the new item from the server has the same UID as a local item, and therefore it *replaces* that item instead of adding it anew. IMHO the correct solution would be for the server to detect that two conflicting items have the same UID. The UID of the item on the server then must be replaced with something which satisfies the iCalendar 2.0 globally unique criteria before it is sent to the client in an Add. Or the property must be cleared, which will cause the client to create a value. I have no idea how to implement either one or the other, though. Suggestions? Solutions which don't work: - Compare only the UID property. A) we don't know whether the client really provides a stable, globally unique UID and b) we would end up with only one item even when both items were modified and thus should both be preserved. - Don't store UID on the server. We want it there for backup/restore of iCalendar 2.0 clients. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] X-Synthesis-Ref and X-CustomLabel- TYPE extensions
On Wed, 2010-04-07 at 13:45 +0100, Patrick Ohly wrote: Hello! This is related to http://bugzilla.moblin.org/show_bug.cgi?id=10462 [...] I suppose for SyncEvolution 1.0 we should simply remove the extension from the XML config used by us. There's a slight loss of functionality when SyncEvolution is used as server for a Synthesis (iPhone) client, but that is not its primary role anyway. I've implemented this. I'm still interested in hearing about how these extensions should be handled by a server. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] X-Synthesis-Ref and X-CustomLabel- TYPE extensions
Hello! This is related to http://bugzilla.moblin.org/show_bug.cgi?id=10462 Steps to reproduce: - sync Synthesis iPhone client, SyncEvolution server, Evolution - modify contact on iPhone - send to server and Evolution - TYPE=X-Synthesis-Ref0 is added to ADR, EMAIL, TEL Outcome: - Evolution shows email as other (might be right, there was no TYPE=HOME/WORK) - ADR is not shown (NOT right!) Our XML configuration contains the TYPE=X-Synthesis-x extensions. We inherited those from the Synthesis example config. Having them is useful when operating as server with the file backend, because the iPhone client can store and later retrieve its extensions. But when storing an item in Evolution, we should not add these extensions. Implementing that is tricky, because we don't want to maintain two different sets of vcard profiles. Not sure how to do this yet. I have a few questions about this. First, X-Synthesis-Refx maps to the field ADR_STREET_ID with value x. What is the semantic of this? Same for ADR_STREET_LABEL, which would show up as X-CustomLabel-label. I'm asking to figure out whether this might be worth mapping to something in Evolution or elsewhere. Second, how does the Synthesis server handle synchronization between clients which support this extensions and those that don't? The DevInf does not contain entries for these extensions, so the server cannot tell whether a client supports them or not. Third, are there perhaps other clients which get confused by these extensions? At least ScheduleWorld might preserve them (not entirely sure). Regarding solving this problem, the obvious choice would be to make X-Synthesis-Ref depend on a specific rule: when encoding for Evolution, don't enable the extension. But this is not possible at the moment. I suppose for SyncEvolution 1.0 we should simply remove the extension from the XML config used by us. There's a slight loss of functionality when SyncEvolution is used as server for a Synthesis (iPhone) client, but that is not its primary role anyway. Then later on we should either enhance the rule mechanism or go for some dynamic pre-processing of profiles. That way we can turn the general purpose vCard mimeprofile into a vCard-Evolution mimeprofile which matches exactly what Evolution supports. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] suspend failure
On Tue, 2010-03-16 at 14:40 +, Lukas Zeller wrote: Most probably, there should be an extra check for that too late suspend. Your patch in the luz branch works, thanks a lot. Now I just have the inverse problem: the tests pass even if the suspend requests were ignored entirely. Manually checking the next log shows that we did resume and we get the progress event which tells us that the session was resumed. But it would be nice if this could be checked already before running that next sync. Is there a possibility to retrieve the datastore is suspended information from a binfile-based client? Is it possible in a server? -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] suspend failure
On Thu, 2010-03-18 at 14:05 +, Lukas Zeller wrote: Hello Patrick, On Mar 18, 2010, at 14:00 , Patrick Ohly wrote: On Tue, 2010-03-16 at 14:40 +, Lukas Zeller wrote: Most probably, there should be an extra check for that too late suspend. Your patch in the luz branch works, thanks a lot. Thanks for the feedback. After pushing I suddenly had the impression it could still be wrong for another case, I have to follow that thought first to make sure. So that's why I did not announce the patch out loud... It has passed all of my tests, so I'll go ahead and use this in the next SyncEvolution snapshot. If you have more thoughts on what I should test, please let me know. But it would be nice if this could be checked already before running that next sync. Is there a possibility to retrieve the datastore is suspended information from a binfile-based client? Is it possible in a server? In both cases, it's the resumeAlertCode target field. If it is not zero, the datastore is resumable - i.e. a client will try to resume and a server will accept a resume. In binfile based setups, you can use the /profiles/xx/targets/dbid/resumeAlertCode settings key to access that value. In both cases - there are no targets in a server, so how would that work there? I suppose it is part of the admin data, but that is internal and and peeking inside it would bypass the official API. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] suspend failure
On Thu, 2010-03-18 at 15:36 +, Lukas Zeller wrote: [server admin data] Yes, it would bypass the API, but unlike with binfiles, in the server case YOU are the provider of the storage for that admin data, and how the fields are stored is under YOUR control, and not a secret of the SyncML engine. So I'd guess it is ok to have a look at them (or even modify them - such as setting resumeAlertCode to zero to prevent a resume, or blank out the anchor to force a slow sync). So the format is part of the API? I guess it has to be even though it is not documented (or is it?), because changes to it would break server software upgrades. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] suspend/resume + datastore + tokens
On Fr, 2010-02-26 at 15:12 +, Patrick Ohly wrote: Hello! When we started with SyncEvolution, we discussed the StartDataRead() resumeToken parameter. At that time, we came to the conclusion that a binfile based client doesn't have to distinguish between different tokens. Always reporting the changes since the last sync is good enough, the binfile layer takes care of merging the recent changes with those that haven't been sent. Now, on the server side I think tokens are relevant, right? SyncEvolution as server doesn't support multiple tokens yet, and I was hoping that one of the existing tests would fail because of that, but no luck so far. I found two problems (see other emails), but none that I could attribute to SyncEvolution ;-} In which situation on the server side is the distinction between lastToken and resumeToken relevant? I discussed this with Lukas. Here's my write up of the answer: http://bugzilla.moblin.org/show_bug.cgi?id=2425#c5 (In reply to comment #4) True. However, I learned in the meantime that the binfile support is not enabled for SyncML servers. Once we turn SyncEvolution also into a server, we need to revisit this issue. ... so now we need to solve this issue. Without it, our suspend/resume tests against our own server should fail. Only in some very constructed situation, and one which is not currently covered by any of our tests. Here's where the two anchors make a difference: 1. client and server in sync 2. item added on server 3. client and server sync 4. suspend after sending that new item to client 5. a second item added on server 5.1 resume = send changes since step 4 5.2 two-way sync instead of resume, using anchors from step 1 = send changes since step 1 Our server fails in 5.2, because it will always only send changes since the last sync (step 4 in this example). The relevance of this is debatable. It depends on clients actually being able to roll back to a previous state. Our own client can't do that, so it would have to do a slow sync in 5.2 instead of resuming. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] LibSynthesis internal data structure and conversion...
On Di, 2010-03-09 at 22:28 +, paul wrote: LibSynthesis uses an internal data structure formatted as multiline aa:bbCRLF... The internal representation is the field list. This aa:bbCRLF format is an external format for it, as far as I remember. I'm not entirely sure where exactly it is used. I never had to use it in SyncEvolution. Is there any API available in LibSynthesis to convert that to vCard format? I see a function recently added DataConversion, but I'm not sure if that is just used for testing or if it will work. It won't work. The DataConversion() function will convert from one format into another. The format is defined by a datatype in the XML configuration. The aa:bbCRLF format is not such a datatype. Can you describe what you are trying to accomplish? Perhaps there are other ways to do it, or it is useful enough to add something. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] LibSynthesis internal data structure and conversion...
On Mo, 2010-03-15 at 10:11 +, paul wrote: Well I wrote my database plugin in copying the example of SDK_demodb, because for whatever reason I couldn't get the SDK_textdb example to work. So now I have a plugin written in C for a client. Now the the libsynthesis engine calls this plugin for example with an InsertItem(), I get data in the aa:bbCRLF format. However I have no why of interpreting that data and I need it in vcard format without rewriting code to convert it based on the configuration xml file. This is exactly what SyncEvolution does. Here's an outline how it works, for details see the SyncEvolution source code: * in the plugin's Module_Capabilities, set CA_ItemAsKey to :yes (SynthesisDBPlugin.cpp) * instead of the normal Item() callbacks implement the ItemAsKey() variants (SynthesisDBPlugin.cpp) * in the initscript of the data store, create a variable (itemdata in SyncSource.cpp) and map it so that it can be accessed from the AsKey() code (map name='data' references='itemdata' type='string'/) * in the before/afterwritescript, call the MAKE/PARSETEXTWITHPROFILE() macros to convert to/from the field list (SyncSource.cpp) * in your AsKey() implementations, read/write the data field (SyncSourceSerialize::readItemAsKey, SyncSourceSerialize::insertItemAsKey) -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] resume Add second half
On So, 2010-03-07 at 01:56 +0100, Patrick Ohly wrote:
Anyway, it fails in one case when the item was split:
1. client has a new big item
2. client sends Add with MoreData/
3. server buffers the item
4. client sends second half
5. server processes the complete item, removes the PIStored blob,
sends reply
5.5. admin data is stored with lastitemstatus=0, in contrast to the 201
status stored there when the item is not split across messages.
6. reply never reaches client
7. client and server suspend
8. upon resume, client resends the second half
(ItemSourceLocURI2273/LocURI/Source
MetaEMIdatapos=19241/EMI/MetaData...)
9. server attempts to parse the item, fails and returns 415
The logs don't tell me what the relevant difference is, so I'll have to
dive into the debugger.
I found something and got it working with the following patch. Lukas,
does this make sense?
commit ce704f6f521e9153d801f47d750a9a45fcc6f507
Author: Patrick Ohly patrick.o...@intel.com
Date: Mon Mar 15 16:16:32 2010 +0100
resume + pending item fix in SyncML server
When an item that was split among messages was finally reassembled
and processed by the server, the status sent to the client was not
properly stored in the admin data. As a result of that, when the
session got suspended directly after sending that status, the client
would resume, send the last item chunk again, but then not get
the previous status. Instead the server attempted to parse the
incomplete item and failed.
The root cause was that TSyncSession::process() was called
recursively. The inner one for the incomplete command saved the
status in fLastItemStatus. The outer one then later didn't have
a status pointer, fell back to status 0 and overwrote the valid
status.
This patch solves this by suppressing that second updating of
fLastItemStatus and all related fields if the item was handled
by an inner process() call.
diff --git a/src/sysync/synccommand.cpp b/src/sysync/synccommand.cpp
index 1892b24..73560cc 100755
--- a/src/sysync/synccommand.cpp
+++ b/src/sysync/synccommand.cpp
@@ -2270,6 +2270,7 @@ bool TSyncOpCommand::execute(void)
localstatus sta;
TSyncOpCommand *incompleteCmdP;
bool queueforlater,processitem;
+ bool nostatus;
SmlItemListPtr_t tobequeueditems=NULL;
SYSYNC_TRY {
@@ -2292,6 +2293,7 @@ bool TSyncOpCommand::execute(void)
while (*itemnodePP) {
queueforlater=false; // do no queue by default
processitem=true; // process by default
+ nostatus=false; // set to true if someone else is responsible for
updating fLastItemStatus
thisitemnode = *itemnodePP;
// no result nor status so far
statusCmdP=NULL;
@@ -2531,7 +2533,10 @@ bool TSyncOpCommand::execute(void)
// - first remove global link to it to avoid recursion
fSessionP-fIncompleteDataCommandP=NULL;
// - execute now (and pass ownership)
- // issues appropriate statuses
+ // issues appropriate statuses, so we don't need to deal with
it;
+ // in fact, we must not touch fLastItemStatus because we don't
+ // know the status
+ nostatus=true;
fSessionP-process(incompleteCmdP);
// - this item is processed now, continue in loop if there are
more items
processitem=false; // do not process the item normally
@@ -2628,7 +2633,7 @@ bool TSyncOpCommand::execute(void)
// item processed
// - remember status (final only) for possible suspend and resume
sta= statusCmdP ? statusCmdP-getStatusCode() : 0;
-if (sta!=213) {
+if (!nostatus sta!=213) {
// final status received, save it for possible resend
fDataStoreP-fLastItemStatus = sta;
// but forget data stored at DS level
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] [PATCH] SAN 1.1 generation support: Moved utility code from engine to sdk
On Mon, 2010-03-08 at 08:27 -0800, Beat Forster wrote: Our SDK is using stringutil.h/stringutil.cpp for standalone application, so you can't just remove them, because they are missing then for these projects. But if these files were kept and stringutils.h/cpp were moved into the SDK, the change was okay? As before, because moving files into the SDK implies a relicensing, this change would have to be made in the synthesis.ch git repo by Synthesis. Can we make that fairly soon so that we can remove the hack which exposes these functions in the libsynthesis.so? -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] resend Delete
Hello! Another resume problem, probably caused by SyncEvolution: 1. item deleted on server 2. Delete sent to client 3. client processes it, its next message to server gets lost 4. client and server suspend 5. after resume, Delete is sent again 6. client backend returns 10500 in DeleteItem (sysync::LOCAL_STATUS_CODE + 500) 7. 10500 is sent to server 8. server shuts down session *without* reply 9. client encounters network problem due to missing reply The client should not treat removal of an already removed item as error. This happens here because I was using the file backend, not the Evolution backends, which already work like that (step 6). I suppose I'm still a bit uncertain about local status codes. Is the plugin allowed to return an error marked as local to the Synthesis engine? If yes, then the engine should have sent 500 to the server, not 10500 (step 7). If no, then I need to be more careful in the exception handling code which creates the status code. If the consumer of the status is the engine, we must not mark the error as local. If the consumer is SyncEvolution above the engine, then they should be marked as local, just as if they came out of the engine. Client and server don't agree on whether the session should continue (steps 8 and 9). Not sure who is right here. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] resume Add second half
Hello! Lukas, your fix with not processing a resent Add works when the item fits into one message. I'm not entirely sure why, though. What I see in the log file is: [2010-03-06 07:30:58.009] Started processing Command 'Add' (incoming MsgID=2, CmdID=5) * +–[2010-03-06 07:30:58.009] 'processCmd' - Processing incoming command, Cmd=Add, IncomingMsgID=2, CmdID=5 [--][++] [-end] [-enclosing] * [2010-03-06 07:30:58.009] command started processing * [2010-03-06 07:30:58.009] Received last chunk for already processed item - just resending status 201 * [2010-03-06 07:30:58.009] Created command 'Status' (outgoing) The already processed message is not from your recent patch. Anyway, it fails in one case when the item was split: 1. client has a new big item 2. client sends Add with MoreData/ 3. server buffers the item 4. client sends second half 5. server processes the complete item, removes the PIStored blob, sends reply 6. reply never reaches client 7. client and server suspend 8. upon resume, client resends the second half (ItemSourceLocURI2273/LocURI/Source MetaEMIdatapos=19241/EMI/MetaData...) 9. server attempts to parse the item, fails and returns 415 The logs don't tell me what the relevant difference is, so I'll have to dive into the debugger. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] resume + double Add
On Fri, 2010-02-26 at 15:05 +, Patrick Ohly wrote: [...] So it seems that the engine does check for the remote ID, but only after having already added the item again. I think this was due to not enabling resumesupport. After setting that, the same test passes. Sorry for the confusion - I hadn't quite realized that suspend/resume was optional. Shouldn't the server have refused to resume with resumesupport off? Just wondering, we'll certainly turn it on now... -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] resume + double Add
On Thu, 2010-03-04 at 23:49 +, Lukas Zeller wrote: On Mar 4, 2010, at 18:49 , Patrick Ohly wrote: On Fri, 2010-02-26 at 15:05 +, Patrick Ohly wrote: However, while tracking through the code I found that indeed in the server, the case of a client sending duplicate Adds was not handled at all. So even with resumesupport a repeated add from the client IMHO would have created a duplicate in the server DB. As most clients always send Replace (including libsynthesis not so long ago), this case did not surface. I now added a check for that in 3e687ddfca (server engine: added missing check for re-sent Add during resume.) Then my testing wasn't all in vain, even though it doesn't seem to be 100% reliable - see below. I'll continue testing with your patch included. ...and resumesupport is irrelevant (so what I said in the previous message is not entirely correct - old plugins will AUTOMATICALLY disable resume; resumesupport is only relevant for SQL backends). So it's even stranger that enabling resumesupport should have made the tests to pass now. ??? I also enable blob support, but that shouldn't be relevant for these tests. But I might have been wrong about it working: rerunning it shows that the results are not always reproducable. I need to look into that. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] Newbie question on setting the target locuri...
On Mon, 2010-03-01 at 08:36 +, paul wrote: It seems like I should be able to set the target locuri in the configuration file, but I don't see an example for that. Can I set the target locuri in the config xml file and if yes what is the form? I don't think you can. URIs on the client side are set via a key in the profile that corresponds to the data source. I'm sure this is in the documentation somewhere. Here's how we do it in SyncEvolution: http://git.moblin.org/cgit.cgi/syncevolution/tree/src/syncevo/SyncContext.cpp?id=syncevolution-0-9-2#n1646 -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
