Re: [os-libsynthesis] unified log file
Hello Patrick, Before going into details, i.e. reviewing the patch I'd like to make sure I understand what is the goal: - you want a single log file for ALL logging, which means: - both global and session log of libsynthesis (including messages generated by the DB plugins) - PLUS messages generated by your application Right? I don't see why you need two engines for that. You can initialize the same engine twice, once at the beginning with basically only the log options, and a second time when your generated config stream is ready. Within the engine, you need: - a new flag to have the session logger redirect its output to the global logger, and also forwarding the appendtoexisting flag to the global logger such that it won't overwrite the beginning of the log. Maybe some changes are needed to prevent re-instantiation of the global logger at the second init, I'll have to check that. - a way to send debug message from the outside (the app) into the global log. Luckily, these already exist (Beat uses them exactly this way in his test suite). See DB_DebugPuts, DB_DebugBlock, DB_DebugEndBlock and DB_DebugEndThread entries in the UI_Call_In record, or the ready-to-use wrappers for these in SDK_utils (DEBUG_xxx). Just let me know if your goal is something entirely different I haven't understood so far. But if I'm correct, I think we have do a much less invasive solution - and especially can avoid EVIL globals :-) Best Regards, Lukas Zeller On Jun 30, 2009, at 20:04 , Patrick Ohly wrote: Hello! My goal since switching to the Synthesis engine was to have one single log file again. There are other ideas for logging improvements, but the single log is the most important one: http://bugzilla.moblin.org/show_bug.cgi?id=3474 I've been dragging my feet a bit with it and I knew it would be kind of complicated, but now I have something which works. Beware, I didn't say it is nice ;-} It's also not final yet, but if possible, I'd like to get it done this week. I've pushed the code into the moblin.org logging branch of libsynthesis and SyncEvolution. Lukas, Beat, can you have a look at the code and tell me whether this is okay in principle and what might have to be changed to make it suitable for upstream? The main issues I see are: * had to use a global variable to pass information (TSyncAppBase::fGlobalAppLoggerP); ideally I'd pass the handle for the first engine (with the active logger) to the creator of the second engine (the one with the datastores) * can only log plain text via DB_DebugPuts_Func(); I'd like to include more information (severity level, source file/line) -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] statistics about deleted items in refresh-from-server sync
Hi Patrick, just my comment: Usually the engine knows about the number of elements in the sync set, therefore no additional information is needed to get this count. In case of a refresh-from-server I assume the sync set will not be read, but the plugin could return 20030 if no ReadNextItem sequence has been done before. So the engine will remove element by element then and get the number of elements this way. I'm not sure I follow here. Suppose a database plugin implements the current DeleteSyncSet(). How does the engine know how many items were deleted? Is it possible without causing overhead (the main reason for the existance of DeleteSyncSync(), I suppose)? Efficiency is one reason, yes. The main reason however was the need for wiping metadata not directly visible in the sync when the entire sync set is being refreshed, vs. a incremental update. In particular, groups in the iPhone address book are added when first used, but are not refcounted (not deleted when empty). But when I do a reload device, I don't want such metadata to persist. As a plugin writer you can decide not to implement DeleteSyncSet(), and then the engine will read the sync set (IDs, modified status) as for any normal sync, and then issue DeleteItem() in a loop. You still can make use of the call to DeleteSyncSet() as an indicator the entire sync set will be deleted and wipe metadata - if you return LOCERR_NOTIMP the engine will still do a one-by-one delete loop. Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] message resending over OBEX
Hello, the experiments where devices were disconnected while a session ran and it continued seamlessly after reconnecting were with HTTP (e.g. devices connected to the internet via WLAN, and reconnecting over GPRS when walking out of WLAN coverage). I must admit that I don't have a lot of practical OBEX experience, but IMHO OBEX is used only in scenarios where the connection cannot change as with HTTP. I'd say if someone pulls the USB cable during a sync, the session should abort/suspend. Similarily if the device goes out of BT coverage. The lower protocol levels of these OBEX connections provide some degree of data integrity by themselves already to compensate for intermittent interference etc. If these aren't sufficient, I doubt we can do much on the higher level. Furthermore, If I remember correctly, a sync session in OBEX runs within a single OBEX session anyway (unlike HTTP), so if that transport level session breaks, a new SyncML session must be started anyway. Best Regards, Lukas On Jul 30, 2009, at 17:59 , Patrick Ohly wrote: Hello! As discussed here in an earlier thread, when clients don't get a reply to their HTTP POST, they can resend that message and that way work around half-open TCP connections. It may also be a good idea in case of other, detected network problems. Over on the SyncEvolution list we we are now discussing the integration of SyncML with OBEX as part of a Linux desktop. Here the question of message resending came up again. With HTTP, clients can be redirected to a special URL which encodes a session ID. That means messages resent after a loss of connection can still be matched with the session before parsing the message content. With OBEX connections, the same thing is not possible: it is not possible to include meta information when reconnecting, so the SyncML recipient would have to parse the message before it can determine the session ID. The other problem is the question who reconnects. In the normal OBEX use case, the desktop runs as SyncML server and connects to the client. But the current resend message by client error handling would then depend on the client being able to connect to the server. Finally, OBEX would work over USB, Bluetooth and IrMC. A server which has connected via Bluetooth might come back via USB. I remember that Lukas mentioned experiments where devices were disconnected while a session ran and it continued seamlessly after reconnecting. Was that using HTTP as transport protocol or OBEX? If it was HTTP, any thoughts on doing the same with OBEX? Hopeless or doable? -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] About 'autodate' and 'autoenddate'
Hi Patrick, On Aug 11, 2009, at 14:19 , Patrick Ohly wrote: autoenddate is significant only for vCalendar 1.0 rendering, rendering = parsing? handling would have been the right word - it affects both parsing and generating... as it will turn on detecting 23:59:59 as an end-of-day marker and convert it to 0:00 the next day and when generating vCalendar 1.0, it will represent all-day-events according to the autoenddateinclusive setting. For iCalendar 2.0, autodate and autoenddate behave the same. mobical.net uses vCalendar 1.0, whereas all other servers we test with use iCalendar 2.0. So we could change the setting without affecting any of the other servers. However, I'm still a bit fuzzy about how autoenddate interacts with the detect alldays in vCalendar 1.0 (0:00-0:00 or 23:59 localtime) part of VCALENDAR_INCOMING_SCRIPT. Can someone walk me through the sequence of conversions that happen when applying autoenddate to the mobical.net example DTSTART/END and why MAKEALLDAY() then fails? See the other mail - there was indeed a bug in the autoenddate handling when parsing. The idea of the (hardwired) autoenddate mechanism is to abstract the ugly details of vCalendar 23:59:59 DTENDs which really mean 0:00 of the next day, such that subsequent steps (incomingscript, or the DB plugin code directly) don't need extra checks for that special case, i.e. don't need any case handling for vCalendar 1.0. So what happens is: - incoming timestamps for autoenddate fields will be converted to item time zone (usually system localtime), and then checked for 23:59:59. - if check returns true, timestamp will be made 0:00 of the next day (here was a bug, resulting TZ was wrong when input was not floating). The next step, in your case incomingscript, can now rely on all-day events spanning entire days, and base further examinations on this. This is not so important if this is handled in incomingscript, because there you'll probably use ALLDAYCOUNT() to detect alldays, and this function will (for historical reasons, before autoenddate was available) also recognize the 23:59:59 case. However it IS important when detecting all-day is done in a DB plugin, without an incomingscript in place (that's how all our WM and PALM clients work), because we absolutely don't want any vCalendar 1.0 format dependent code in DB adapter implementations. Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] Your client+statistics workaround patch (f51c0bc3a77d29b5af33c5003b4752173ed35ddb in moblin git)
Hi, thanks for the patch to fix the client+statistics problem with counting client adds as updates when the server returns 200 for them instead of 201. There is one basic problem with your solution: The fact that a entry in the database already has a change log entry does NOT mean that the entry has been added to the server which is addressed in this session (i.e. it does NOT mean that the operation will be a normal replace). The reason for this is that the changelog is shared between multiple profiles (multiple servers), and thus the changelog entries are created when ANY of the profiles first sees the entry. Consider the following: 1. You add a new record X 2. You sync to server A (with profile A) 3. changelogPreflight detects X as new and creates a changelog entry for it 4. You sync to server B (with profile B) 5. changelogPreflight will find the changelog entry created in 3 and not know that X is new for server B That's why binfileds based clients only send Replace - the current changelog mechanism can't really detect adds. The standard would allow clients to send Add, but does not require it. I am aware that your solution works with the current SyncEvolution setup, as it does not use multiple profiles at that time (or implements them by switching the entire binfile store, including changelogs). However I can't add it to libsynthesis master as it would break multi-profile. Now, to fix the problem in a generic way such that multi-profile will still work the following could be done: - extend the changelog to have a modcount_created field, and set it with the current modcount instead of setting the chgl_newadd flag. - in implGetItem(), compare the changelog entry's modcount_created with the current modcount to safely determine if the record is an add or a replace. Furthermore, I see no need for sop_soft_add, because if we safely know which items are really adds, the client can send them as Add commands to the server. Most probably this will also fix the statistics problem with google and zyb without any other code change (because I suspect that these simply return 201 for every Add and 200 for every Replace, without checking if replaces are implicit adds). Sending Adds openly has another advantage - IMHO statistics should show what the server actually reports, and not what we think it should have reported. There might be valid cases where a server can join incoming adds with existing data and reporting those as added would not be correct either. Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] empty anchor or dummy string
Hi Patrick, I just saw your patch: slow sync: avoid empty anchors, that confuses ScheduleWorld This patch fixes http://bugzilla.moblin.org/show_bug.cgi?id=4703, ScheduleWorld: delete on server can not sync to client (after doing slow sync). Somehow the ScheduleWorld server has problems with last/last. Saving the mapping on the server fails in that case. Mark called an empty anchor string invalid. I don't agree with that and think this issue should be fixed on the server, but as it can be worked around easily (and perhaps more quickly) on the client side, here's the patch for it. Instead of it now uses no local anchor. For symmetry reasons (not required to solve this problem) the last remote anchor is set to no remote anchor. I'm a bit confused, because I don't see how an empty last/last can be created at all. The last tag is optional within anchor, and it is therefore created using newPCDataOptString() in newMetaAnchor(), which does not create the last element at all when the input string is missing or empty. So IMHO the engine NEVER sends a last/last. Still, if the problem is that Scheduleworld also cannot handle a missing last tag, I'd favor a patch that would insert a dummy last just when creating the actual SyncML command (i.e. at the points where newMetaAnchor() is called), and not touch fLastLocalAnchor or fLastRemoteAnchor. Reason is that these variables have the semantic meaning of no anchor = empty string. Changing that could mess up code testing these for emptyness e.g. to detect first-time-sync status (I haven't checked if that is really used somewhere, but it would be OK to do so). Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] vCard 2.1 and quoted parameter values
Hi Yongsheng, I have a question regarding your patch to allow vCard 2.1 parameters to have quoted values. IMHO the vCard 2.1 specs don't allow this: param = “TYPE” [ws] “=“ [ws] ptypeval / “VALUE” [ws] “=“ [ws] pvalueval / “ENCODING” [ws] “=“ [ws] pencodingval / “CHARSET” [ws] “=“ [ws] charsetval / “LANGUAGE” [ws] “=“ [ws] langval / “X-”word [ws] “=“ [ws]word / knowntype with ptypeval = knowntype / “X-” word and word = any printable 7bit us-ascii except []=:., Your example TEL;TYPE=X-EVOLUTION-COMPANY:12345 would be a typeval The definition of word allows the doublequote to be part of the word, so interpreting it as a quote around the value is wrong. I admit a case where a string beginning with a doublequote does not mean quoting the value is highly unlikely. But I wanted to check with you what is the reason for this change - do you see that type of invalid quoting in an existing server? Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] session security
On Sep 15, 2009, at 11:41 , Patrick Ohly wrote: One thing you can do (as a server) is to request auth not only for the first message, but again for every subsequent one (by responding 200 in the SyncHdr status, and not 212 as usually done). Together with MD5 auth and https, I think this is waterproof, because in the non- interceptable SSL response you have the nonce you need to succeed with MD5 auth for the next message. Yes, looks much better than the current approach. But will all clients support it? From what I remember about the Funambol library, I very much doubt that it would cope with such a server behavior. I think it is a MUST in the standard, and if I correctly remember it was even tested by SCTS (the SyncML initiative's test suite tool of the old days - every participant in a SyncFest had to pass SCTS as part of the registration). But I admit I haven't tested this recently, neither with our own implementation nor foreign ones. It's good to have this in mind though for Synthesis-Synthesis syncs. Does the Synthesis engine create the number itself? Yes. Then the next question is: how is encoding the number inside the RespURL handled? What I'm aiming at is, how does the server know that it should append ?sessionid=number to the base URL? This might work for HTTP, but other transports might require different methods to tell the client about the session ID. Correct. The way this is formatted (or added at all - AFAIK other transports don't need it anyway, because they have a transport level session concept) is transport specific. Therefore, in libsynthesis_srv, providing the RespURI (or the parts of it that are transport specific, I'm working on the details) will be something the app will need to do. Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] client-side check of server anchors (ZYB)
Hello Patrick, On Sep 15, 2009, at 15:37 , Patrick Ohly wrote: Is it possible to disable checking of the anchors provided by a SyncML server? No. Our testing with ZYB.com (Vodafone) is stuck because the server sends incorrect anchors (apparently based on time, but sometimes with offset against GMT and sometimes without). They have acknowledged the issue, but a fix is slow in coming. Indeed. I have correspondence with them for this very issue dating back a year. However it seems that they had it working in between and now it surfaced again - it was some problem with the GMT offset back in 2008, and apparently still (or again) is. Disabling the client-side check of the anchors would allow us to do normal incremental syncs. ZYB needs legacy mode anyway, we could also disable anchor checking in legacy mode. Or add a new option flag to the profile for that. I would not want to generally disable client side anchor checking, of course. What do you think? Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] libsynthesis_srv
Hi Patrick, I just pushed the files required to build libsynthesis_srv (luz branch). I also created a simple makefile (src/server_engine_linux.mk) from the existing, but mostly abandoned client_engine_linux.mk. I just verifed I can cd src make -f server_engine_linux.mk to make sure all files needed are there and listed in that makefile. I hope this is enough information to integrate this into the automake environment as an additional target. For now, this will create a separate library with the same API (signature wise, semantics are slightly different, of course) as the client version of libsynthesis. To make both server and client libraries linkable into the same app, libsynthesis_srv defines a different prefix for the entry points: SySync_srv_x (defined in serverengine_opensource_linux_prefix.h), as opposed to SySync_x for the client variant. I haven't checked the SDK support code side of things yet, as I'm not too familar with that (my own use of libsynthesis all uses different wrappers in ObjC and even Delphi which in turn directly use the C API). Next I'll work on the code itself to make it actually functional and push stuff to the luz branch incrementally. Just a first overview how it will work: - connecting to libsynthesis_srv and initializing the engine with config is identical to the client variant. As a starting point for an XML server config I'd take the syncserv_demo.xml file which is in all of our free demo server downloads. Of course, I'll push a sample config for the server libary later into the OS repository. - all the settings configuration stuff (/profiles/xxx settings keys) is non-existant in the server. Everything that must be configured is in the XML config. - opening a session works the same as in the client, just use OpenSession(). You can pass the sessionID (the one that will be used later to identify messages of that session at the transport level, e.g. using CGI like ?sessionID=xx) as aSessionName parameter; if you leave it empty, the engine will create a unique numeric ID of it's own. - running the session works similar than in client - but the order of STEPCMD_xxx is of course a bit different. I'm working on that right now and will provide more details when it's ready. - unlike the client library, which can only run one session per connected library instance, the server engine can run more than one session in the same instance at a time, and these sessions may also be run in different threads. It is even possible to switch threads between processing of each SyncML request in the core engine, but I'll have to check if this makes sense with the libsynthesis_srv API. Note however that the engine has a single XML configuration for all sessions that is parsed in once and then the in-memory tree is accessed strictly read-only by all sessions. Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] 222 endless loop detection
Hello Patrick, On Oct 16, 2009, at 18:16 , Patrick Ohly wrote: I'm afraid the 222 loop detection that Congwu added for the message resend situation has negative side effects in other cases. Negative like aborting a sync prematurely. I'm glad I found this before releasing SyncEvolution 0.9.1 next week - now we just need a fix... ;-} Indeed. Here's what I ran into: * get client and ScheduleWorld server into sync with several hundred items * delete all items on the client * run two-way sync with a smallish message size (10KB) * client aborts with Warning: More than 5 consecutive Alert 222 within 20 seconds- looks like endless loop, abort session I'm not attaching the log, but can provide it on request. Instead I extended client-test in SyncEvolution's master branch so that the problem can be reproduced: CLIENT_TEST_SERVER=scheduleworld \ CLIENT_TEST_NUM_ITEMS=1000 \ ./client-test Client::Sync::vcard30::testManyDeletes As far as I can tell, what happened is this: * client sends a message to the server with a very compact list of deleted items * server responds with several ( 5) messages that contain the Status messages for the deletes * soon the client has no changes left while the server is still sending his Status replies, so the client starts sending Alert 222 I agree. I wonder however how the status messages can be that much bigger than the deletes, as the client sends a separate delete command for each item (does not make use of theoretically possible multiple items per command), so I'd expect it to be of similar size than the matching status. Congwu, Lukas, you are both more familiar with this code than I am. Could the loop detection be changed so that the counter is reset each time genuine progress is made, like receiving status messages? Yes, I'll have a look. Resetting the counter for receiving status for a Sync op command sounds good to me as a criteria to reset the counter. Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] failure to add item in DB, but sync successful?!
Hello Patrick, On Nov 3, 2009, at 14:09 , Patrick Ohly wrote: I have checked, the InsertItemAsKey() implementation returns a 510 error code. This error code is passed up until it reaches sysync::TBinfileImplDS::implProcessItem(), where it is handled like this: 1962error: 1963 // report OS specific error codes as item text back to the originator 1964 ok=false; 1965 PDEBUGPRINTFX(DBG_ERROR,(Database Error -- SyncML status %ld, dberr=%ld,(long)statuscode,(long)lastDBError())); 1966 aStatusCommand.addErrorCodeString(lastDBError()); 1967done: 1968 aStatusCommand.setStatusCode(statuscode); 1969 return ok; 1970} // TBinfileImplDS::implProcessItem First observation: lastDBError() is provided by sysync::TCustomImplDS::lastDBError() and always returns 0. This means a rather uninformative Err = 0 is added to the outgoing Status. Is there a way to provide a more meaningful error string or code? By pure coincidence we just ran into that as well today (confused by the Err = 0 info). I have changed that (and will push as soon as I have a stable state again, other work in progress :-) such that both debug log and Status text will only show a message if there IS in fact a DB-specific error code. Note that for DB Api at this time the net effect will be that NO error code will be shown. Second observation: the failure is not recorded and thus doesn't show up in the sync statistics or the overall sync result. Is this expected? Not really. I'd expect it to get recorded as rejected. I'll have a look. Should this be handled by the application sitting on top of libsynthesis, by communicating with the DB backends directly? No That would be possible in SyncEvolution, but IMHO it would violate the layering principle in libsynthesis. Yes :-) Just for the record, the log shows the problem like this: [...] [2009-11-03 14:05:14.384] ical20: Internal error: attempt to reduce state from 'client_maps_sent' to 'data_access_done' - aborting [2009-11-03 14:05:14.384] ical20: testState=FALSE - expected state=='data_access_done', found state=='client_maps_sent' –[2009-11-03 14:05:14.384] End of 'SyncHdr' [-top] [-enclosing] The extracting event error is something that I hard-coded in the calendar backend. Note the Internal error. I'm not sure whether it is related. Probably not, but I'm not absolutely sure. Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] server login without credential check
if (!isAuthTypeAllowed(authtype)) {
aStatusCommand.setStatusCode(401); // we need another auth
type, tell client which one
authok=false; // anyway, reject
PDEBUGPRINTFX(DBG_ERROR,(Authorization failed (wrong type of
creds), sending 401 + chal));
}
else if (!authok) {
// auth type allowed, but auth itself not ok
aStatusCommand.setStatusCode(401); // unauthorized, bad
credentials
PDEBUGPRINTFX(DBG_ERROR,(Authorization failed (invalid
credentials) sending 401 + chal));
}
if (!authok) {
// - add challenge
aStatusCommand.setChallenge(newSessionChallenge());
}
Would it be okay to add something like this before else if (!
authok)?
It solves the problem for me.
if (!authok isAuthTypeAllowed(auth_none)) {
authok = true;
}
That changes the meaning of requiredauthnone/requiredauth from
all
kinds of authentication methods allowed, including none to all kinds
of credentials allowed, including invalid ones. Because a client with
invalid credentials was already allowed to connect by not sending
them,
this doesn't make a server setup less secure. It has the advantage
that
the client setup becomes easier.
I added my patch to the master branch on moblin.org, because it is
needed for SyncEvolution. Please merge if you agree with this change.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Lukas Zeller (l...@synthesis.ch)
-
Synthesis AG, SyncML Solutions Sustainable Software Concepts
i...@synthesis.ch, http://www.synthesis.ch
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] failure to add item in DB, but sync successful?!
Hello Patrick, On Nov 10, 2009, at 19:31 , Patrick Ohly wrote: On Tue, 2009-11-03 at 18:23 +, Lukas Zeller wrote: On Nov 3, 2009, at 14:09 , Patrick Ohly wrote: Second observation: the failure is not recorded and thus doesn't show up in the sync statistics or the overall sync result. Is this expected? Not really. I'd expect it to get recorded as rejected. I'll have a look. Did you find out something about this? Hm - looking at your log snippet... Just for the record, the log shows the problem like this: –[2009-11-03 14:05:14.381] End of 'ScriptExecute' [-top] [-enclosing] [2009-11-03 14:05:14.382] # SyncEvolution (LNK): InsertItemAsKey: ical20 025B6F50 [2009-11-03 14:05:14.382] adding fake timezone with daylight starting in May [2009-11-03 14:05:14.383] ical20: extracting event [2009-11-03 14:05:14.383] cannot create record in database (sta=510) [2009-11-03 14:05:14.383] Database Error -- SyncML status 510, dberr=0 [2009-11-03 14:05:14.383] - Operation add failed with SyncML status=510 –[2009-11-03 14:05:14.383] End of 'Process_Item' [-top] [-enclosing] [2009-11-03 14:05:14.383] processSyncOpItem: Error while processing item, status=510 ...I see in the code (syncsession.cpp) that when processSyncOpItem: Error while processing item is logged, this inevitably also increases the fLocalItemsError counter (the one you get reported with the PEV_DSSTATS_E progress event at the end of the session. One more question about the problem: what if the DB has temporary problems, such that retrying the operation in the next sync has a chance to succeed again? Is that something that could be done by returning a suitable status to the peer, a) in the general case and b) when the peer is also the Synthesis engine? You could of course return something indicating a temporary problem rather than just 510. This might make a difference with some servers, but I don't know. In my experience a implementation that does not just return 500 for all sorts of errors already counts as advanced :-( For a Synthesis based peer, all items that receive an error status () will be flagged and resent in the next session by default. You can switch this behaviour on the datastore level using resendfailing, and on a per-item level in the sentitemstatusscript using SETRESEND(), where you can also check the status code and do other things like aborting a session or silently accept errors. Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] HTML logs and appending
Hi Patrick, On Nov 25, 2009, at 8:38 , Patrick Ohly wrote: [...] we now instantiate one engine for logging, destruct it, then construct the final engine. The resulting log is okay, but not perfect. It contains the htmlhead bodyh2Start of log - Synthesis SyncML Engine 3.3.0.0/h2 preamble twice. The indention of the second engine is two deep. I thought HTML logs were kind of appendable? Basically they should be appendable, yes - the question however is why the first log is not properly closed in terms of ul etc. Would it be possible to not write the header when the log file is in append mode and not empty? Probably, but that would cause wrong indentation when the first bit of the log is actually properly closed. Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] Server Config changes to enable events+todo superdatastore
Hello Congwu, On Nov 25, 2009, at 2:19 , Chen Congwu wrote: While I was enabling superdatastore for SyncEvolution server usage, I found the config file syncserv_sample_config.xml needs some minor fixing: It was using a bool variable 'ISEVENT' to differeciate events and todos, while for events+todo dispath we actually need a string variable 'KIND' for the sub-datastore dispatch process. Yes, indeed. That's wrong in the superdatastore definition. Thanks! I am changing 'ISEVENT' to 'KIND' in SyncEvolution, do you think it is reasonable? Yes, should work but probably it is easier (less changes) to do the opposite, i.e. change the dispatchfilter in the superdatastore definitions to check ISEVENT instead: contains datastore=events dispatchfilterF.ISEVENT:=1/dispatchfilter guidprefixe/guidprefix /contains contains datastore=tasks dispatchfilterF.ISEVENT:=0/dispatchfilter guidprefixt/guidprefix /contains I'll do that for the sample config - for SyncEvolution it's up to you to decide what makes more sense. BTW, looking at that superdatastore definition, I realized the type definitions only include vCalendar 1.0 - I guess it should be extended to include iCalendar 2.0 as well: typesupport use datatype=iCalendar20 mode=rw preferred=yes/ use datatype=vcalendar10 mode=rw preferred=legacy/ /typesupport Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] Any observations of mobical.net having problems with client Add?
Hi Patrick, Thanks for the feedback! We haven't heard back from that customer since, which usually means the problem has gone away (for whatever reason). I'm inclined to put this into the random failures (at mobical) department - it might well be that they had a temporary problem with adding records on the database layer, independently of Add vs. Replace. Best Regards, Lukas On Nov 30, 2009, at 14:11 , Patrick Ohly wrote: On Tue, 2009-11-17 at 19:11 +, Lukas Zeller wrote: I have a customer report indicating that the change made with c3ad24c31d (engine: statistics workaround for servers not sending status 201 for completed adds) sometimes confuses mobical.net and causes it to return status 400 for attempts to add items to the server. Not always, but sometimes. So far I was not aware of Add from client being a potential issue with mobical - so my question: did anybody experience spurious status 400 rejects while testing with mobical? I could not reproduce the case so far (havent tried too hard, either). We have tried with both Add and Replace for new items and have not been able to reproduce such spurious rejects. We had some nightly test failures with Mobical.com in one-way-from-client mode (server doesn't handle its own anchors correctly, need lenient mode) and random failures (server returns plain text instead of (WB)XML). But this is independent of the Google statistics workaround. For details, see: http://bugzilla.moblin.org/show_bug.cgi?id=8121 -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] merging your patches - va_list question
Hello Patrick,
I'm about to review merge the patches from the moblin/master branch into our
libsynthesis repo - thanks to your earlier cleanup of the history it's a really
straightforward task, thanks for that!
The only change that I'm not sure about is:
71aef21545 (TDebugLoggerBase::DebugOpenBlock(): unused va_list)
Doesn't the static prefix make that argument a global scope variable, and
thus make the routine non-reentrant and non-threadsafe?
Just because in another commit, you point out that a va_list can't be re-used,
which IMHO implies it is changed when it is processed by vsprintf. Of course,
in this case with the list being empty we can be reasonably safe it is not
changed or a change would not have any effect, even if multiple threads should
try it simultaneously. Still, is that clean enough?
Second - I don't really see why declaring it static would mean anything for
initialisation (apart from the fact that gcc seems not to be able to figure out
it should complain about it then) - C-ish stuff is not implicitly initialized,
neither in global scope nor on the stack.
I tried to initialize va by using va_start() on it, however gcc rejects this in
a function which has no ... in the declaration. We could add a ... just to make
gcc happy. Not nice either.
So my proposal would be just to call
DebugOpenBlock(blockname,title,collapsed,fmt,...) with fmt=NULL and no optional
arguments instead of DebugVOpenBlock:
// open new Block without attribute list
void TDebugLoggerBase::DebugOpenBlock(cAppCharP aBlockName, cAppCharP
aBlockTitle, bool aCollapsed)
{
// we need a format and debug not completely off
if (getMask() aBlockName) {
DebugOpenBlock(aBlockName,aBlockTitle,aCollapsed,NULL);
}
} // TDebugLoggerBase::DebugOpenBlock
This is completely safe and we don't have to mess with the fact that apparently
there's no proper way to initialize an empty va_list.
Lukas Zeller (l...@synthesis.ch)
-
Synthesis AG, SyncML Solutions Sustainable Software Concepts
i...@synthesis.ch, http://www.synthesis.ch
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] merging your patches - va_list question
Hi Patrick, thanks for the comments. On Jan 27, 2010, at 12:24 , Patrick Ohly wrote: Second - I don't really see why declaring it static would mean anything for initialisation (apart from the fact that gcc seems not to be able to figure out it should complain about it then) - C-ish stuff is not implicitly initialized, neither in global scope nor on the stack. In global scope it is filled with zero. Only on the stack is the content undefined. I know that most environments do that - but from a puristic C point of view: is this really something C specification demands? IMHO in Windows even all mallocs are really callocs and probably even stack vars are nulled. Some say it's good because it makes a reproducible environment, some say it's bad because it prevents missing initialisations are ever detected. Anyway, in small embedded C environments I've used, there was no auto-nulling of static vars. This is completely safe and we don't have to mess with the fact that apparently there's no proper way to initialize an empty va_list. Yes, that looks like a cleaner solution, with slightly higher overhead. The overhead is why I left the check in, to prevent that the change costs any time in non-debug mode. With debug enabled, it causes one function call more, but looking at all the stuff that happens inside DebugVOpenBlock I think we can afford it :-) Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] detecting slow syncs
Hello Patrick, On Jan 26, 2010, at 17:38 , Patrick Ohly wrote: As I might have mentioned before, in SyncEvolution we try to prevent unwanted slow syncs to give the user the chance to choose some other way of recovery (refresh sync, restore from backup): http://bugzilla.moblin.org/show_bug.cgi?id=2416 My current solution only partly works in a SyncML client: in the alertscript, the client checks the mode suggested by the server. The anchor check of the client happens after that, so if the client engine switches to a slow sync because the server's anchor is wrong, the slow sync is not prevented. In a SyncML server, alertscript is called before any kind of anchor checking and thus not useful for this purpose at all. What about datastoreinitscript or maybe initscript? These are called before user data is accessed the first time, which would be early enough to stop the whole thing if needed, but late enough to be sure the sync mode is known. BTW: I realized that a useful document was missing in the docs so far - see my latest commit 9dc1e5c1a3 (Doc: useful table showing the call flow of scripts within a server session) on the luz branch. What's currently missing would be some DS-level functions to query the sync mode, but that'll be simple to add (probably in customimplds.cpp, within TCustomDSfuncs). To differentiate unwanted from client-requested slow sync, we'd probably need two of them, like ALERTEDMODE and ACTUALMODE. Some functions along this line already exist, I'd have to check to make sure we don't duplicate stuff. What I don't really get is why the server would want to modify the client behaviour other than switching to slow sync (for two-way alert) or to refresh (for a one-way), which it already does. Here's a different idea. Instead of doing the checking via script, could the StartDataRead() in the datastore plugin do the checking? First problem: is this really called after the sync mode has been determined, in both client and server? Second problem: how can the plugin find out what the mode is? I think StartDataRead() is called at the right time. Yes, I would say so (without checking bit by bit). The mode could be passed via a key, except that a) a server has no profiles and thus no per-datastore keys (right?) Yes, the entire /profiles/ hierarchy does not exist in the server. Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] Outlook Recurring Event mapped to wrong Timezone after sync
Hi Raji, On Jan 27, 2010, at 19:18 , Bommaraju, Rajyalakshmi wrote: Timezone mapping is behaving little weird , For the same Buenos Aire meeting which I created logs and sent last week was mapped to Africa/Harare, now same event getting mapped to Standard Time/Daylight Savings Time 1. This is definitely better. Africa/Harare was a completely false match (for whatever reason). Standard Time/Daylight Savings Time means that there is no match for that VTIMEZONE, so libsynthesis creates a custom TZ definition from the VTIMEZONE. I stepped through the code and in the TIMEZONEtoInternal function has GZOnes-matchTZ function to match the Timezone to loaded timezone definitions (tzcmp compares name,ident,bias,dst,std) and if that fails, replaces the TZID with TZNAME created from VTIMEZONE STANDARD/TZNAME which is Standard Time in this case and DAYLIGHT/TZNAME which is Daylight Savings Time. It would help if the VTIMEZONE TZNAME labels were more significant. Many VTIMEZONE generators put the time zone abbreviations like PST/PDT, CET/CEST or in your case ART/ARST, which results in a more meaningful time zone name. In the matchTZ This time just the Timezone is wrong string but the event was displayed at the correct time in my evolution calendar. Yes, the name is technically irrelevant when the actual time zone parameters in the VTIMEZONE are correct. The question is why the list of timezones read from libical does not have a definition for ART/ARST (or maybe it has one, but with different parameters from those in the VTIMEZONE). We found that a definition for ART/ARST was missing in the built-in time zones table of libsynthesis itself (a patch is on the way). However in the SyncEvolution context, this list is not used but superseded by the list imported from libical, as far as I know (Patrick, correct me if I'm wrong). So to find the reason for false matches (like the Africa/Harare case) and the missing match for ART/ARST that imported data is the key. Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] detecting slow syncs
Hello Patrick, On Jan 29, 2010, at 19:49 , Patrick Ohly wrote: Another peculiarity: on the server, after calling ABORTDATASTORE(), 'ReadSyncSet' is done and items are read from the database as part of 'GetItems'. On the client, only 'ReadSyncSet' is done. Server log below. Shouldn't the engine skip these steps because the store has already failed? Yes, it should but a check for isAborted() was missing after calling the datastoreinitscript. I have added it now (bc0e54ca12 (Added check such that if datastoreinitscript calls ABORTDATASTORE(), the abort gets propagated)), but I'm not sure it will improve the situation, as it will cause the entire sync to stop, not just the datastore (because a problem in initializing the sync is considered a fatal error and will lead to a AbortSession() (at syncagent.cpp:1171). On the other hand, I don't think just suppressing operations on that single datastore would help a lot - most servers will expect a full sync cycle or then get confused. We could check at syncagent.cpp:1171 if the abort reason status is zero (i.e. ABORTDATASTORE(0) called), and if so behave differently, i.e. just muting that datastore, but leave the sessing running. But as said, I'm not sure that will help with many servers. There is SLOWSYNC(), documented as returns true if session is a slow sync. I found that it also returns true on a client if the session is a refresh-from-client. Is that intentional? Yes. Within the engine, the term slow sync no only covers the two-way slow sync, but the refresh syncs as well, as these are conceptually slows syncs as well. I think the SyncML standard docs used the term like this at some point. So: sync mode Normal variantSlow variant - -- - two-way normal two-way slow sync with data match from client update from client (update server)refresh from client (reload server) from server update from server (update client)refresh from server (reload device) After all - a full refresh is as slow as a two-way slow sync; all data has to travel from one side to the other :-) Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] detecting slow syncs
Hi Patrick, On Feb 4, 2010, at 22:16 , Patrick Ohly wrote: On Di, 2010-02-02 at 11:23 +, Lukas Zeller wrote: We could check at syncagent.cpp:1171 if the abort reason status is zero (i.e. ABORTDATASTORE(0) called), and if so behave differently, i.e. just muting that datastore, but leave the sessing running. But as said, I'm not sure that will help with many servers. ABORTDATASTORE(0) would lead to the engine picking some other, generic status (514). I solved this by introducing a specific status code that tells the engine abort this store, but continue with the rest. The SyncEvolution takes care of actually aborting the session, as before. Not pushed to git yet. Patch attched. If that approach looks right, I'll merge the Synthesis master branch with the fix for ABORTDATASTORE() and add my own patch on top (or you include it). Looks fine. I'll push the extended remote rule stuff we discussed today, so once the XML fragment merge is done (see other mail) that can be added as well. Now, for the server side things are bit more tricky. First of all, is there any way how the caller of the engine can determine the status of each source, as it can for a client via the progress events? The lack of progress events in the server is already a big problem for interactive monitoring of a running server session, but not getting information about per-source failures affects our error handling and reporting. Ok, I see that the progress events for the server must get more priority on my todo list! Second, datastoreinitscript is apparently called when the server processes the Sync command. There is only one Sync command in that message, so the client-side trick of processing the whole message to determine the sync mode for all stores doesn't work. Hooking into the processing of the Status response of the client for the server's Alert won't work either, because someone client and server also only send Alert and Status for one store per message. When the client alerts the server, there are two Alerts in the message. Then the server replies with an Alert for only one of the stores. Why is that? Couldn't it include two Alerts in the same reply? Somehow I lost track now. In what situation do you see that happen? Basically however, the *message* is not a relevant unit, only the *package* is. Implementations are free to split up any package into as many messages as they like. For example I've seen implementations which split the updates from server package into one message per data item, probably to be able to detect data items that make the client crash precisely. Finally, on the server side we typically don't know whether the user has chosen a slow sync mode intentionally. He might have. So we can only enable the slow sync detection for server-initiated syncs were we control the sync mode. This problem can be solved, for example by only enabling the slow sync detection in configs where the server initiates the connection or by making it conditional on having sent a SAN. Makes sense. However, genereally thinking about the whole topic, I'm not sure if trying to continue some syncs and aborting others is not asking for potentially massive trouble with confused peers (222 loops etc.), while gaining not much. What is the cost of aborting the other datastores in case of an unexpected slow sync? A bit of additional traffic, but that's it. As the abort happens before any actual data is exchanged, there's very little danger of impact on the sync in the other datastores. On the other hand, my experience is that to get the SyncML state machine completely right and stable enough to survive things like partial syncs is very hard. Even after 10 years with ours there were still recent cases where it did behave wrong, and issues with final/ and 222 loops is certainly the most common problem on the SyncML protocol level with any new or immature implementation I've seen. The outcome of trying to run a partial sync is IMHO very much unpredictable. I think it would be safer to just abort the entire session on a unexpected slow sync, in both client and server cases. Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] XML config fragments
Hello Patrick, On Feb 5, 2010, at 11:45 , Patrick Ohly wrote: Lukas, can you merge our master branch? I cannot rebase because the scripts mentioned above depend on the commit ID. Ok, but are you sure these are pushed? No, I forgot. Please look again now. done now. I also added the discussed enhancements regarding remote rules now (but not yet on master, only on luz). Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] superdatastore in SyncML client
Hi Patrick, I can just say, if you come to the point where a superdatastore seems to need any admin data (profiles, maps, anchors, whatever) you're off track :-) So, regarding engPrepareClientSyncAlert(): The problem is that this, as a client-only routine, has never been made superdatastore aware, i.e. it's not a virtual (SUPERDS_VIRTUAL) and TSuperDataStore does not have a derived implementation for it (yet). We need an implementation of engPrepareClientSyncAlert() in TSuperDataStore which avoids calling its own inherited (TLocalEngineDS), but instead loops trough its subdatastores and calls engPrepareClientSyncAlert() on those. Have a look at the server case counterpart TSuperDatastore::engProcessSyncAlert() to get the idea. engGenerateClientSyncAlert() and engInitForClientSync() are hot candidates for requiring a similar derived implementation in TSuperDatastore as well... How should anchor handling work for a superdatastore? TSuperDatastore::engInitSyncAnchors(), which is a server client routine, already takes care of that (calculates a common anchor value from all subdatastore's anchors, and a common fFirstTimeSync as well). For processing superdatastore's data items, the code is already in place that dispatches incoming Sync Ops to the correct subdatastore - by ID prefix for Replace/Delete and by subdatastore filter for adds (see TSuperDatastore::engProcessRemoteItem()). Same for generating Sync Ops for the remote, this is handled by TSuperDatastore::generateSyncCommands() which iterates through all its subdatastores to get the items to send and prefixes their localID according to the subdatastore they originate from. Finally, I apparently implemented engGenerateMapItems() already, despite the fact this is a client-only routine (That's usually my way of working - if I stumble upon something obviously missing, I add a dummy with a warning output or if it's a no-brainer to code, I just write it, even if there are additional things missing for the feature to work. What I forgot is to mark that code, and everything else related to client-side superdatastores certainly is: UNTESTED). Please let me know if this kind of guidance to the missing links is ok - or if you think it'll be something you'd rather want me to do myself. If the latter, it'll take some time because next week is quite busy and the week after that is snow vacation :-) Best Regards, Lukas Zeller On Feb 12, 2010, at 14:40 , Patrick Ohly wrote: On Do, 2010-02-11 at 19:50 +, Patrick Ohly wrote: [no target for superdatastore] The only solution I see is to make the subdatastore URI's more complex: instead of foo, let's use foo:uri where foo is the name of the superdatastore and uri its remote URI. I'm going with that and have syncs running, albeit with some problems. One problem is that the client always forces a slow sync for the superdatastore. That's because of the following line in TLocalEngineDS::engPrepareClientSyncAlert(): // check if we are forced to slowsync (otherwise, fSlowSync is pre-set from dsSetClientSyncParams() fSlowSync = fSlowSync || fLastLocalAnchor.empty() || fFirstTimeSync; fSlowSync is false, the anchor empty, fFirstTimeSync false. How should anchor handling work for a superdatastore? -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] superdatastore in SyncML client
Hi Patrick, There can't be a a target nor a TBinfileImplDS for the superdatastore. This still holds true... You almost had me convinced and I already changed quite a bit of code, until I came to the point where I wanted to call dsSetClientSyncParams() for the superdatastore. If that store has no target associated with it, how can the client set the URI? Yes, you're right, I forgot about the remote URI The only solution I see is to make the subdatastore URI's more complex: instead of foo, let's use foo:uri where foo is the name of the superdatastore and uri its remote URI. Makes sense. Just a cosmetic thought: why not foouri, so the real remote URI would still be on the same syntax level as for other datastores, whereas the angle bracket serves as a prefix container for options to treat that URI differently locally. Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] [SyncEvolution] Nokia N85 sync status
Hello Patrick, On Feb 18, 2010, at 23:52, I wrote: [...] So I'm trying to move your additions for client-side superdatastore up to the abstract level (i.e. localengineDS and syncagent) before I merge with master. I'll work on that right now, so maybe I can commit it today (if not, it might take longer as I'm in vacation next week). committed to luz/synthesis-indefero :-) Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] resent messages on server
Hello Patrick, the caching of the last server response is pretty generic and should be available in libsynthesis as far as I remember (no sources here at 1400m in the Swiss mountains :-) However it's an option and might be disabled or accidentally not compiled in. Lukas l...@synthesis.ch On 25.02.2010, at 21:18, Patrick Ohly patrick.o...@intel.com wrote: Hello! How does the server handle message resends by a client? I mean the situation where a client sends message A, server responds with B, B gets lost, clients sends A again after a timeout. I think Lukas once told me that a server has to remember what data it received by the client (A) and the reply (B), then when receiving a message check whether it is the same as A. If so, resend the cached reply B. This is something that only makes sense with HTTP. Does that mean that the implementation of this also has to be in the HTTP server transport? Or is there a common implementation of this feature in libsynthesis? I'm asking because I am running our resend tests against SyncEvolution as HTTP server and see that tests are failing. Our syncevo-http-server.py daemon has no special logic for detecting resends. If the functionality must be implemented there, then this explains the failure. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] SAN 1.0/1.1 support for server mode
Hello Congwu, On Feb 23, 2010, at 8:40 , Chen, Congwu wrote: I am trying to enabling SAN 1.0/1.1 support for SyncML server mode so that we can support syncing with older phones. It seems current code in san.cpp only supports parsing SAN 1.0/1.1 package not generating it. Is it possible/acceptable to add this part of logic inside san.cpp with (#ifndef WITHOUT_SAN_1_1)? Yes, of course, that would be a nice addition! BTW: we have a command line utility to create SAN messages (and pack them for SMS delivery) - at this time it can only generate DS 1.2 SAN, but if you enhance san.cpp with a DS 1.1 generator, it could be updated to do that as well. So far, pushalerter was not in opensource, but we decided to LGPL it and add it (see pushalerter branch). At this time, it's only source, w/o any makefile, so I keep it in a separate branch for now. Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] timezone handling in vcalendar 1.0
Hello Congwu, On Mar 4, 2010, at 9:04 , Chen Congwu wrote: I am a bit confused about timezone handling in vcalendar 1.0 format, if I have a VEVENT with a local timezone say 'Asia/Shanghai', how does it expressed in DTSTART field? libsynthesis uses UTC for all points in time, because then even if the time zone detection does not work, at least the point in time will usually be correct (unless it's something as complex as a recurrence over DST boundaries). As far as I remember, all phone clients I saw use UTC as well (except for very old ones which have a localtime-only calendar, and can't do any time zone calculations). In icalendar 2.0, there is a parameter 'TZID' for DTSTART, so I can add it there, in vcalendar 1.0, how does it supposed to work? Will it use the item level 'TZ' property? I suspect so. Yes, that's how it works. But as TZ/DAYLIGHT support is something rarely found working correctly (or implemented at all), I'd always use UTC and treat the TZ information as a nice supplementary information, if present. If no TZ is there, the item time zone defaults to the user time zone. What I found with libsynthesis/SyncEvolution is it always send time in UTC format with absolution timezone if remote can handle UTC. (While in icalendar we will send time in UTC format with timezone specified by 'TZID' parameter). This does not have negative effect but I am curious whether the different handling is intentional? Yes, definitely :-) Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] resumesupport + binfile clients
Hello, On Mar 4, 2010, at 17:53 , Patrick Ohly wrote: I noticed that suspend/resume support has to be enabled explicitly via datastoreresumesupport. Is that really necessary in a binfile client? No. If binfiles are active, dsResumeSupportedInDB() always returns true. Doesn't the binfile layer provide everything which is needed and thus suspend/resume could be enabled by default? Yes, it does. So it should be enabled by default. However, when not using the binfile layer (like in the server), the admin part of the DB might not support the extra fields needed for resume, so it's disabled by default for backward compatibility with old plugins or old SQL-based database backends. Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] resume + double Add
Hi Patrick,
On Mar 4, 2010, at 18:49 , Patrick Ohly wrote:
On Fri, 2010-02-26 at 15:05 +, Patrick Ohly wrote:
[...]
So it seems that the engine does check for the remote ID, but only after
having already added the item again.
I think this was due to not enabling resumesupport. After setting
that, the same test passes. Sorry for the confusion - I hadn't quite
realized that suspend/resume was optional.
As said in the previous message, yes, for the server it is optional.
However, while tracking through the code I found that indeed in the server, the
case of a client sending duplicate Adds was not handled at all. So even with
resumesupport a repeated add from the client IMHO would have created a
duplicate in the server DB.
As most clients always send Replace (including libsynthesis not so long ago),
this case did not surface.
I now added a check for that in 3e687ddfca (server engine: added missing check
for re-sent Add during resume.)
Shouldn't the server have refused to resume with resumesupport off?
Yes. However, when you use a admin data plugin, dsResumeSupportedInDB() is
checking the API version of the plugin:
pluginapids.cpp:941:
/// returns true if DB implementation supports resume (saving of resume
marks, alert code, pending maps, tempGUIDs)
bool TPluginApiDS::dsResumeSupportedInDB(void)
{
if (fPluginDSConfigP-fDBApiConfig_Admin.Connected()) {
// we can do resume if plugin supports it
return
fPluginDSConfigP-fDBApiConfig_Admin.Version()=sInt32(VE_InsertMapItem);
}
return inherited::dsResumeSupportedInDB();
} // TPluginApiDS::dsResumeSupportedInDB
...and resumesupport is irrelevant (so what I said in the previous message is
not entirely correct - old plugins will AUTOMATICALLY disable resume;
resumesupport is only relevant for SQL backends).
So it's even stranger that enabling resumesupport should have made the tests
to pass now. ???
Lukas Zeller (l...@synthesis.ch)
-
Synthesis AG, SyncML Solutions Sustainable Software Concepts
i...@synthesis.ch, http://www.synthesis.ch
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] saving suspend state in the middle of a sync
Hi Patrick, On Mar 5, 2010, at 8:01 , Patrick Ohly wrote: On Thu, 2010-03-04 at 22:17 +, Lukas Zeller wrote: a server session can end at any time simply by the fact that the client does not answer any more within the timeout. But the next session (trying to resume) can start long before the previous one actually HAS timed out. This is not something that can happen for us, but I see what you mean. Yes, in the bluetooth context, that's can't happen. But I hope one day client-initiated local HTTP sync in the (W)LAN will also be supported and then you'll need it :-) Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] resend Delete
Hello Patrick, On Mar 6, 2010, at 7:10 , Patrick Ohly wrote: Another resume problem, probably caused by SyncEvolution: 1. item deleted on server 2. Delete sent to client 3. client processes it, its next message to server gets lost 4. client and server suspend 5. after resume, Delete is sent again 6. client backend returns 10500 in DeleteItem (sysync::LOCAL_STATUS_CODE + 500) Yes, the backend should return a 404. 500 is considered fatal and completely unspecific and will abort any session. 7. 10500 is sent to server 8. server shuts down session *without* reply Hm, I'd expect a reply here, too. Would need to analyze logs in detail to find out why it does not here. 9. client encounters network problem due to missing reply The client should not treat removal of an already removed item as error. This happens here because I was using the file backend, not the Evolution backends, which already work like that (step 6). I suppose I'm still a bit uncertain about local status codes. Is the plugin allowed to return an error marked as local to the Synthesis engine? If yes, then the engine should have sent 500 to the server, not 10500 (step 7). No. The 1 offset is something internal to the engine. The plugins should return 3-digit SyncML error codes or in some special cases 20xxx ones. All plugin errors are local anyway. If no, then I need to be more careful in the exception handling code which creates the status code. If the consumer of the status is the engine, we must not mark the error as local. If the consumer is SyncEvolution above the engine, then they should be marked as local, just as if they came out of the engine. As plugin errors are local by definition, we could make the calling code more robust against plugins adding the 10xxx offset, so if you have routines used in plugins and outside, they could both use 10xxx. Client and server don't agree on whether the session should continue (steps 8 and 9). Not sure who is right here. Basically, the server should send a response for every request, so that's probably a bug. Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] [PATCH] SAN 1.1 generation support: Moved utility code from engine to sdk
Hello, As before, because moving files into the SDK implies a relicensing, this change would have to be made in the synthesis.ch git repo by Synthesis. Can we make that fairly soon so that we can remove the hack which exposes these functions in the libsynthesis.so? Lukas has to check if there are other dependencies for theses moved files. Lukas, if it's ok, can you update the synthesis.ch repo accordingly ? There will be some dependencies in OUR non-public tree, especially old project files referring to the sysync/* location for some of the source and header files. But we can avoid them on the short term by just keeping the sysync/* copies around for now, and fix the projects one by one later. So Congwu and Patrick, please go ahead and move those files until your builds work ok, and put these in a branch in your repo. I'll then merge these into synthesis.ch master to make the license change official. Ok? Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] suspend failure
:26.074] after SessionStep: STEPCMD_OK * [2010-02-26 14:51:26.074] before SessionStep: STEPCMD_STEP * [2010-02-26 14:51:26.074] after SessionStep: STEPCMD_PROGRESS * [2010-02-26 14:51:26.074] before SessionStep: STEPCMD_STEP * [2010-02-26 14:51:26.074] after SessionStep: STEPCMD_OK * [2010-02-26 14:51:26.074] before SessionStep: STEPCMD_STEP * [2010-02-26 14:51:26.074] Calling smlProcessData(NEXT_COMMAND) ... For the record, to reproduce the problem, set up client-test against a Synthesis server and run: CLIENT_TEST_SERVER=syncevolution CLIENT_TEST_RETRY=t CLIENT_TEST_RESEND=t CLIENT_TEST_SUSPEND=t CLIENT_TEST_INTERRUPT_AT=2 CLIENT_TEST_EVOLUTION_PREFIX=file:///tmp/test_ ./client-test Client::Sync::file_vcard30::Suspend::testUserSuspendClientAdd ... Test name: Client::Sync::file_vcard30::Suspend::testUserSuspendClientAdd equality assertion failed - Expected: no error (remote, status 0) - Actual : unexpected slow sync (local, status 22000) -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. syncevolution-log.htmlsyncevolution-log_trm004_007_outgoing.xml___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] [PATCH] SAN 1.1 generation support: Moved utility code from engine to sdk
Hello Congwu, I just merged your SAN 1.1/1.0 changes from the moblin/san11 branch into our master (2dd107df40c551a5517beb4a751c02e9277890a6). Also merged are some other changes from moblin/master. I'm still in the process merging these with our internal codebase and push the (few) updates from there to libsynthesis. But the current master has all the moves needed to authorize the LGPL-BSD move for the files needed outside the engine for SAN 1.1. Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] suspend failure
Hello Patrick, On Mar 18, 2010, at 14:00 , Patrick Ohly wrote: On Tue, 2010-03-16 at 14:40 +, Lukas Zeller wrote: Most probably, there should be an extra check for that too late suspend. Your patch in the luz branch works, thanks a lot. Thanks for the feedback. After pushing I suddenly had the impression it could still be wrong for another case, I have to follow that thought first to make sure. So that's why I did not announce the patch out loud... Now I just have the inverse problem: the tests pass even if the suspend requests were ignored entirely. Manually checking the next log shows that we did resume and we get the progress event which tells us that the session was resumed. But it would be nice if this could be checked already before running that next sync. Is there a possibility to retrieve the datastore is suspended information from a binfile-based client? Is it possible in a server? In both cases, it's the resumeAlertCode target field. If it is not zero, the datastore is resumable - i.e. a client will try to resume and a server will accept a resume. In binfile based setups, you can use the /profiles/xx/targets/dbid/resumeAlertCode settings key to access that value. Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] suspend failure
Hello Patrick, On Mar 18, 2010, at 16:22 , Patrick Ohly wrote: Thanks for the feedback. After pushing I suddenly had the impression it could still be wrong for another case, I have to follow that thought first to make sure. So that's why I did not announce the patch out loud... It has passed all of my tests, so I'll go ahead and use this in the next SyncEvolution snapshot. If you have more thoughts on what I should test, please let me know. I will. It might be just a ghost, but I'll track it down :-) In both cases, it's the resumeAlertCode target field. If it is not zero, the datastore is resumable - i.e. a client will try to resume and a server will accept a resume. In binfile based setups, you can use the /profiles/xx/targets/dbid/resumeAlertCode settings key to access that value. In both cases - there are no targets in a server, so how would that work there? I suppose it is part of the admin data, but that is internal and and peeking inside it would bypass the official API. Yes, it would bypass the API, but unlike with binfiles, in the server case YOU are the provider of the storage for that admin data, and how the fields are stored is under YOUR control, and not a secret of the SyncML engine. So I'd guess it is ok to have a look at them (or even modify them - such as setting resumeAlertCode to zero to prevent a resume, or blank out the anchor to force a slow sync). Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] suspend failure
On Mar 18, 2010, at 16:55 , Patrick Ohly wrote: On Thu, 2010-03-18 at 15:36 +, Lukas Zeller wrote: [server admin data] So the format is part of the API? I guess it has to be even though it is not documented (or is it?), because changes to it would break server software upgrades. Probably the plugin API variant is not explicitly documented, probably because it is possible (and fully functional) to look at the admin data text format as an opaque string, and just save/retrieve it as-is, without bothering what's in the fields. However, on the SQL/ODBC level, it was always documented in the config reference in 12.20.2 synctargetgetsql, synctargetnewsql, synctargetupdatesql, synctargetdeletesql: Sync target management. That's only semi-useful for plugins, as in the SQL case there are no field names. But it describes the semantics. Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] suspend failure
Hi Patrick, On Mar 18, 2010, at 16:22 , Patrick Ohly wrote: On Thu, 2010-03-18 at 14:05 +, Lukas Zeller wrote: On Mar 18, 2010, at 14:00 , Patrick Ohly wrote: Your patch in the luz branch works, thanks a lot. Thanks for the feedback. After pushing I suddenly had the impression it could still be wrong for another case, I have to follow that thought first to make sure. So that's why I did not announce the patch out loud... It has passed all of my tests, so I'll go ahead and use this in the next SyncEvolution snapshot. If you have more thoughts on what I should test, please let me know. I followed it through and I came to the conclusion that the case I had in mind is ok. What I was not sure about was what would happen if that response from the server would also contain the final map status, and what if not. But both cases are ok - if it does contain the status, it will be processed despite the pending resume, if it does not (unlikely, but in theory the server could request another message exchange to deliver it) it would be ok because then the maps would be saved pending and re-sent at the beginning of the next session (resumed or not). Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] [PATCH] fix/workarounds in smltk DevInf decoding
Hi Congwu,
thanks for these fixes! I merged them with my luz branch in our git repo.
Lukas
On Mar 24, 2010, at 5:26 , Chen Congwu wrote:
Hello,
We found some problems in smltk DevInf decoding for some Nokia
phones[1], here are the two patches:
commit 1077b831ecf4dcc7e780bc4d4ec0f7b4326ceb94
Author: Chen Congwu congwu.c...@intel.com
Date: Wed Mar 24 12:00:42 2010 +0800
smltk+DevInf decoding: avoid a possible infinite loop
For DevInf 1.1 parsing, DevInf decoding will cause an infinite loop
if there is an empty PropName tag (found from Nokia N900,
Nokia 7210c):
eg:
CTCap
CTTypetext/plain/CTType
PropName/
/CTCap
buildDevInfProperty will return OK without further processing if
encountering an empty tag (in this case PropName) and the caller
buildDevInfCtCap will re-evaluate the tag without advancing the
parser, the re-evaluation can only end if current tag is an end tag
or an unknown tag, otherwize it will call buildDevInfProperty with
exactly the same environment again.
diff --git a/src/syncml_tk/src/sml/xlt/all/xltdevinf.c
b/src/syncml_tk/src/sml/xlt/all/xltdevinf.c
index 28b2aea..5ad9e4c 100755
--- a/src/syncml_tk/src/sml/xlt/all/xltdevinf.c
+++ b/src/syncml_tk/src/sml/xlt/all/xltdevinf.c
@@ -651,7 +651,9 @@ static Ret_t buildDevInfProperty(XltDecoderPtr_t
pDecoder, VoidPtr_t *ppElem, in
XltDecScannerPtr_tpScanner = pDecoder-scanner;
Ret_t rc;
-if (IS_EMPTY(pScanner-curtok)) {
+/* Do not return immediately for 1.2 style, the outer loop ends only
meeting an end tag
+ * which will lead to an infinite loop*/
+if (datastoreLocal IS_EMPTY(pScanner-curtok)) {
return SML_ERR_OK;
}
@@ -825,6 +827,7 @@ Ret_t buildDevInfCtcap(XltDecoderPtr_t pDecoder,
VoidPtr_t *ppElem, Boolean_t da
rc = buildDevInfProperty(pDecoder,
(VoidPtr_t)pCtcap-data-prop,datastoreLocal);
if (rc==SML_ERR_OK)
continue; // re-evaluate current tag (tag that caused
buildDevInfProperty() to end, either unknown or closing /CTCap
+ // this means do not return SML_ERR_OK unless this
is an unknow tag or closing /CTCcap, otherwise it will tri
}
break;
commit 277d35dc82f814e86da26020a17eeea65f7d9634
Author: Chen Congwu congwu.c...@intel.com
Date: Wed Mar 24 09:58:25 2010 +0800
smltk: DevInf decoding
Some Nokia phones (N900) sent a nor-wellformed DevInf:
The DevInf version was declared as 1.2, while CTCAP was not
inside Datastore(which is the DevInf 1.1 behaviour). In addition,
the content inside CTCAP was in DevInf 1.2 format.
The original approach detects whether this is a DevInf 1.2 or 1.1
by testing whether CTCAP was inside datastore, thus will think
this is a DevInf 1.1 when decoding the CTCAP which fails.
The patch worked around this, when decoding CTCAP (precisely the
Property or PropName inside CTCAP), use 1.2 style if we are
decoding Property and 1.1 stype when decoding PropName
diff --git a/src/syncml_tk/src/sml/xlt/all/xltdevinf.c
b/src/syncml_tk/src/sml/xlt/all/xltdevinf.c
index b589209..28b2aea 100755
--- a/src/syncml_tk/src/sml/xlt/all/xltdevinf.c
+++ b/src/syncml_tk/src/sml/xlt/all/xltdevinf.c
@@ -812,7 +812,10 @@ Ret_t buildDevInfCtcap(XltDecoderPtr_t pDecoder,
VoidPtr_t *ppElem, Boolean_t da
break;
case TN_DEVINF_PROPERTY:
// DS 1.2 case: only /Property ends the property building
process, next token must be read first
-rc = buildDevInfProperty(pDecoder,
(VoidPtr_t)pCtcap-data-prop,datastoreLocal);
+// If there is a Property tag, let's take it as DS 1.2 Property
Decoding even if the CTCAP is globally
+// This is found from some Nokia phones (eg. N900, which will
send a DevInf v1.2 but the CTCAP
+// was not inside the datastore as DevInf v1.1)
+rc = buildDevInfProperty(pDecoder,
(VoidPtr_t)pCtcap-data-prop,TRUE);
break;
case TN_DEVINF_PROPNAME:
// DS 1.2 case: current token TN_DEVINF_PROPNAME is processed by
builder, next occurence of TN_DEVINF_PROPNAME ends property
[1] http://bugzilla.moblin.org/show_bug.cgi?id=10324
--
Regards,
Chen Congwu
Moblin China Development
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Lukas Zeller (l...@synthesis.ch)
-
Synthesis AG, SyncML Solutions Sustainable Software Concepts
i...@synthesis.ch, http://www.synthesis.ch
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] Server progress events
Hello Patrick, as promised I added server progress events now to the engine. See updates in luz branch (in particular b15e49512c (engine 3.4.0.7: Progress events now session local and available for server sessions)). It works exactly the same as in clients - SessionStep returns with stepCmd==STEPCMD_PROGRESS when there is a progress event to show. Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] wrong interpretation of timezone information
Hello Patrick,
Thanks for the analysis. I changed the rrule parsing routine to allow returning
a adjusted DTSTART so the engine now correctly detects this iCalendar's
timezone as GMT. I just pushed these changes into the luz branch on synthesis
indefero.
Best Regards,
Lukas
On May 4, 2010, at 22:40 , Patrick Ohly wrote:
On Mon, 2010-04-26 at 13:40 +0100, Patrick Ohly wrote:
On Wed, 2010-04-21 at 09:39 +0200, Patrick Ohly wrote:
I have a meeting in my Evolution calendar which I synchronize with
SyncEvolution. I noticed that the outgoing VEVENT has start and end
time
converted to UTC times which are one hour off.
Here's the stripped down event:
BEGIN:VCALENDAR
PRODID:-//Ximian//NONSGML Evolution Calendar//EN
VERSION:2.0
METHOD:PUBLISH
BEGIN:VTIMEZONE
TZID:GMT Standard Time
BEGIN:STANDARD
DTSTART:16010101T02
TZOFFSETFROM:+0100
TZOFFSETTO:+
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:16010101T01
TZOFFSETFROM:+
TZOFFSETTO:+0100
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=3
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
SUMMARY:test event
DTSTART;TZID=GMT Standard Time:20100421T15
DTEND;TZID=GMT Standard Time:20100421T16
UID:04008200E00074C5B7101A82E00800B5A586D0A4CA0110
CLASS:PUBLIC
PRIORITY:5
DTSTAMP:20100415T035521Z
TRANSP:OPAQUE
STATUS:CONFIRMED
SEQUENCE:4
END:VEVENT
END:VCALENDAR
This fails because RRULE2toInternal() in rrule.cpp fails to convert the
RRULE properties into the internal format. It bombs out here:
switch (freq)
{
...
case 'Y' :
if (byday == ||
(byday.length() == 2 byday[0] ==
RRULE_weekdays[startwday][0]
byday[1] ==
RRULE_weekdays[startwday][1]))
{
...
}
else
{
goto incompat;
}
It reaches the goto incompat because byday == -1SU.
Recurrence rules with last day in month ... are common for VTIMEZONEs.
Beat, is that really something that is not handled by the Synthesis code
at the moment? What would be necessary to add support for it?
I'm still wondering about this.
In the meantime I looked into VTIMEZONEs which parse successfully. What
I found is that FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10 (as in the example
above) is supported, by converting it to the equivalent
FREQ=MONTHLY;INTERVAL=12;BYDAY=-1SU.
However, there is a sanity check that compares the start month:
if (DayMonthCombi( freq, byday,bymonth,startmonth )) {
freq= 'M'; // a different model will be chosen for this case,
interval= 12*interval; // which is in fact aequivalent
bymonth = ;
} // if
In the example above, DTSTART=16010101T02 and thus startmonth !=
bymonth. DayMonthCombi() then returns false.
Does the month in DTSTART really matter in this case? I would expect
that the first recurrence will be in the month specified in BYMONTH
which is = than the month in DTSTART, so one could apply the conversion
to FREQ=MONTHLY;INTERVAL=12;BYDAY=-1SU by setting the month in DTSTART
to the value from BYMONTH.
Does that make sense?
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Lukas Zeller (l...@synthesis.ch)
-
Synthesis AG, SyncML Solutions Sustainable Software Concepts
i...@synthesis.ch, http://www.synthesis.ch
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] wrong interpretation of timezone information
Hi Patrick, On May 10, 2010, at 9:04 , Patrick Ohly wrote: Confirmed, works for me. Thanks! I should have mentioned that I had already made similar changes as you on the moblin.org master branch (check for NULL obj in debug macros, enable debug logging inside vtimezone code when invoked by mimedir profile parser). My own debug logging patches go a bit further: - log an error when parsing a VTIMEZONE fails - logging of libical timezone conversion - logging of timezone matching I'm trying to track down an unexpected timezone failure with this which occurs when normal Evolution events are involved. Do you agree with these changes? Would you mind merging them? I rebased them on your luz branch. The attempt to init the time zones after reading config (fc000b0f45 (TSyncAppBase: moved system time zone parsing after config reading)) creates a chicken and egg problem. The config reading process relies itself on the time zone stuff being initialized, most prominently for definetimezone, but also for other tags that have ISO8601 timestamp values. Maybe the dilemma can be solved by adding an optional, second initialize run after config reading that can be switched on specifically to track down timezone init problems. Another option would be to use the ConferrPuts() channel, which is ready before config is read by definition. Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] logging the source code location of debug messages
Hello Patrick, that's a very cool addition indeed :-) Thanks! I'm about to integrate it here, and I have added config options for it, such that I can choose to create txmt:// links (which directly open TextMate or BBEdit code editors on my Mac with the cursor on the right line) instead of doxygen. A second config option allows to override the default base path (the one you define with SYDEBUG_LOCATION). I'll post the changes soon together with the other pending changes on my branch. Best Regards, Lukas On May 11, 2010, at 16:22 , Patrick Ohly wrote: Hello! I've just pushed some experimental code to a doxygen branch on moblin.org: commit 7d9ecc9316d12412cc72c1c41a0889dbeeeb06a2 Author: Patrick Ohly patrick.o...@intel.com Date: Tue May 11 14:12:42 2010 +0200 log call location of debug messages For someone unfamiliar with the source code, it is sometimes hard to locate the code which produced a certain message. This patch solves this issue like this: - produce a fully cross-referenced HTML version of the source with Doxygen - extend the logging functions and macros so that each message comes with file name, line number and function name information - turn the time stamp in the HTML output into a link to the source code line, with the function name as title The reason for picking the time stamp was that making other text a link would interfere with the visual markup of it (color, bold, etc.). In such an enriched log one can: - hover over the time stamp to see function name in a bubble help and the URL (and thus source code location) at the bottom of the browser window (at least in Firefox) - click on the time stamp to jump to the source code Because this feature makes both the library and the logs larger, it is turned off by default. Configure with --enable-debug-logs to enable it when using autotools. Macro tricks are used to produce the same code as before when the feature is off. Limitations: - Without time stamp logging, no links are recorded. - Each change in the Synthesis source code forces a complete new run of Doxygen. This makes compilation quite a bit slower, but the only alternative would be to accept that logs and source dump are inconsistent in the changed parts. - The links contain the absolute file path (making it a bit harder to share logs and source code dumps). - The TClientEngineInterface::debugPuts() API introduced for and used by SyncEvolution leads to links which contain the wrong absolute path to the Doxygen HTML files (inside libsynthesis instead of SyncEvolution). - The prefix parameter in that same API is still ignored. You can an example of an extended log and the corresponding Doxygen files here: http://downloads.syncevolution.org/tmp/synthesis-html.tar.gz Unpack it in /tmp, then point your web browser to file:///tmp/home/pohly/.cache/syncevolution/scheduleworld...@client_+test_+1-2010-05-11-14-04/syncevolution-log.html Is that something that would be acceptable for core libsynthesis? I find it useful as it is, despite the limitations. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] DB API + simple type
Hello Patrick, I think by far the simplest method would be NOT to try to bypass MultifieldItem, but derive it with an implementation that puts/gets data (and maybe metadata) into/from well-known string or BLOB fields. The entire DB API thing was developed long after we more or less settled for TMultiFieldItem as a common denomiantor for all data types. The TField mechanism has proven so useful in various contexts, like the API itself, but also for scripting etc. that while deriving from beyond TMultiFieldItem would still be possible, I see no benefit (and a lot of drawbacks) actually doing so. Sorry for misguiding you with mentioning TSimpleItem in our phone call. But I really would suggest creating a TDirectDataItemType on top of TMultiFieldItemType. That item type could have a hard-coded list of TFields (no need to configure them) representing data and metadata. And it would implement internalFillInData and internalSetItemData which would directly pass data between the SyncML item data and predefined text/BLOB fields. This way, there's zero changes towards the DBAPI, all options retained (scripts), and no parallel implementations of similar concepts. If it's of any help for you, I could offer to add such a thing real soon (this weekend?). Best Regards, Lukas Zeller On May 14, 2010, at 9:58 , Patrick Ohly wrote: Hello! I'm working on a modification of the engine which bypasses the obligatory data conversion via field lists and profiles. The use case for this are client and server which perfectly agree on some proprietary item data and thus need no conversion, or plugins which want to handle the data conversion themselves. So far I have hooked up the existing TSimpleItem with the config parsing (added type and config for it). Now I try to figure out how to read and write the item data and possibly meta data via the plugin API. My initial thought was to expose certain pre-defined field names and use the *AsKey variants of the API, but that seems to depend on inheriting from TMultiFieldItem (pluginapids.cpp needs to down-cast to that if fPluginDSConfigP-fItemAsKey is true). Now I'm looking at the traditional API instead (the #ifdef DBAPI_TEXTITEMS part). In that context I have a question about the API as defined in sync_dbapi.h: * NOTE: By default, the SyncML engine asks for \aID only. * \aItemData can be returned, if anyway available or * \aItemData must be returned, if the engine asks for it * (when calling ReadNextItem:allfields at 'ContextSupport' with \allfields). */ _ENTRY_ TSyError ReadNextItem ( CContext aContext, ItemID aID, appCharP *aItemData, sInt32 *aStatus,bool aFirst ); The sentence in the NOTE is a bit long and I don't understand the if the engine asks for it part. How does it call ReadNextItem:allfields? After reading the documentation of ContextSupport() it becomes a bit clearer, but not much. Where is the complete list of support rules that the plugin might have to handle? If I understand this right, the plugin can choose to support the initial n rules, but not some specific ones. I assume the ordering of rules is chosen so that there's never a need to not support, say rule #1 but support rules #2 and #3? -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] DB API + simple type
Hello Patrick, On May 14, 2010, at 19:56 , Patrick Ohly wrote: On Fri, 2010-05-14 at 17:28 +0100, Lukas Zeller wrote: Hello Patrick, I think by far the simplest method would be NOT to try to bypass MultifieldItem, but derive it with an implementation that puts/gets data (and maybe metadata) into/from well-known string or BLOB fields. The entire DB API thing was developed long after we more or less settled for TMultiFieldItem as a common denomiantor for all data types. Yeah, I noticed. I have a sync starting based on the TSimpleItem and the string-based API, but now I run into more places where a TMultiFieldItem is expected, like for example in customimplds.cpp: 3118// - create dummy item 3119TStatusCommand dummy(getSession()); 3120TMultiFieldItem *delitemP = 3121 static_castTMultiFieldItem *(newItemForRemote(ity_multifield)); 3122delitemP-setSyncOp(sop_delete); 3123PDEBUGPRINTFX(DBG_DATA,( 3124 Zapping datastore: deleting %ld items from database, 3125 (long)fSyncSetList.size() Yes, customimplds.cpp already makes assumptions. It's not totally generic in the original concept of the engine; that's where it's name comes from, it means something like customisable datastore and was implemented for multifield items only. The original concept did not make any assumptions about what an item would be. The TField mechanism has proven so useful in various contexts, like the API itself, but also for scripting etc. that while deriving from beyond TMultiFieldItem would still be possible, I see no benefit (and a lot of drawbacks) actually doing so. Initializing a TMultiFieldItem(Type) seemed more complex than necessary, but I agree, it probably is easier to set it up right and just use it. If it's of any help for you, I could offer to add such a thing real soon (this weekend?). That would be a big help. I'm fairly confident that I would get it done myself (in particular after having spent some time with the code), No doubt! but there might be snags that'll delay me that you know how to avoid. That's why I offer it - I often automatically get an uneasy feeling when I'm about to do something wrong, simply because it rings a faint bell of dead-ends I encountered (and created) myself long time ago :-) That saves some time. My goal was to have something rudimentary next Monday, so if it is not too much bother for you, then I'll punt the problem to you. So let's hack :-) I'll see that I have something for you to test this weekend (on a simpletype branch). Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] sending photo to Nokia phone (N97 mini): wrong CTCap
Hi Patrick, it's a known bug in many Nokia phones that they have a maxSize of 255 on all fields, including the PHOTO. And yes, there is a remoterule option called ignoredevinfmaxsize for that. It applies to all fields, however I agree that those constant 255 maxsizes are probably meaningless anyway. Lukas On Aug 26, 2010, at 17:24 , Patrick Ohly wrote: Hello! I'm investigating why a photo sent to a Nokia device does not show properly on the device. It was originally reported for a 5800 (http://bugs.meego.com/show_bug.cgi?id=5860); I can also reproduce it with a N97 mini. One problem is that the device lies about its capabilities. CTCap contains 256 as maximum length for PHOTO. When encoding, the Synthesis library dutifully obliges and truncates the field. The device clearly is capable of handling larger photos - the photo was originally attached to the contact *on* the device itself and then copied to the server. Is there a remote rule that I can use to ignore the length limitation of the PHOTO property? Should I ignore the whole CTCap, as it is clearly not based on reality? I think there is an option for that, but won't that have undesirable effects like not preserving local extensions that are really not supported by the phone? Another problem might be the TYPE. Evolution doesn't set JPEG by itself. We may have to add that to the outgoing PHOTO. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] sending photo to Nokia phone (N97 mini): wrong CTCap
Hi Patrick, On Aug 30, 2010, at 10:43 , Patrick Ohly wrote: Speaking of CtCap, do you happen to know what Nokia's involvement was in the definition of that part of the spec? Or speaking more generally, what was the desired usage of CtCap? I've just been in another discussion around that, where the Synthesis use of CtCap to determine unsupported properties was questioned. One interpretation of CtCap is limitations for some properties. In that interpretation, unlisted properties might still be supported. The other interpretation is that CtCap has to be complete and accurate, and thus anything not covered by it (value too long, unknown property) is not stored by the device. This is the interpretation used by Synthesis. FWIW, it sounds more plausible to me. The real world devices that most prominently and strictly supported our interpretation were Nokia's. We had a lot of trouble of getting all data out of them without entirely omitting CTCap in the beginning, until we found that not only all properties needed to be listed (that was the case already in very early versions of our server), but also the complete list of possible property parameters. For example, a Nokia phone would not send any telephone number unless the CTCap had not only TEL but also the possible property parameters WORK, HOME, CELL... And because their clients were for a long time pretty much the only ones looking at CTCap at all apart from ours, I'd at least say that this interpretation is a de facto standard. But I have no insight into the involvement of Nokia in that part of the specs. But is there anything in the standard which supports one or the other interpretation? If there is, I am not aware of it. Or perhaps it was part of meeting minutes? Maybe, I don't know. Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] optional property parameters, sub/remoterules
Hello Patrick, thanks a lot for implementing this! I just reviewed the changes and found no objections. So I updated our git master and luz branches with it. I'm now in the process of merging some updates on our side and will push them onto luz in the next hour or so (but leave master at its current position for now). Best Regards, Lukas On Sep 3, 2010, at 18:08 , Patrick Ohly wrote: On Thu, 2010-09-02 at 11:48 +0200, Patrick Ohly wrote: Now I would go one step further and suppress generating this parameter for a peer, after identifying the peer. I thought that this should be possible with the remoterule config option, but parameter doesn't support that according to the documentation. BTW, the config parser doesn't complain when it is specified, it just doesn't do anything. Is adding support for it a good idea? The semantic would be during parsing and generating, ignore parameter unless no remoterule set or rule is active. I'm a bit uncertain whether this should apply during parsing *and* generating, because parsing something that was sent to us shouldn't do any harm. But it wouldn't be consistent. I've implemented this and verified that it does what we need by adapting SyncEvolution. Attached are the patches, also pushed to meego.gitorious.org in the parameter-rule branch. Does that look acceptable? I'd prefer to use them in SyncEvolution 1.1 only after a review by Synthesis. MAKETEXTWITHPROFILE(... EVOLUTION) Except that the last step doesn't work yet either. Would it make sense to extend TMimeDirProfileHandler::setRemoteRule() such that setting a rule also sets all included rules? This is what I implemented. It has the advantage that the choice about full recursion can be made differently in different situations. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. 0001-MIME-Profile-added-parameter-rule-something.patch0002-MIME-Profile-check-parameter-rule-something-when-gen.patch0003-MIME-Profile-check-parameter-rule-something-when-par.patch0004-MIME-Profile-setting-one-rule-also-activates-all-inc.patch___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] libsynthesis update - 3.0.4.16 (was: Re: optional property parameters, sub/remoterules)
Hello Patrick, I owe you an answer regarding commits 0e139ee311 and 4275e77001: On Sep 13, 2010, at 16:42 , Patrick Ohly wrote: 1) 0e139ee311 (engine 3.4.0.10: separate changelogs, pendingmaps and pendingitem for each profile to allow for different syncsets in profiles (as needed by SyncML PRO iOS)): This change is transparent as long as the app using libsynthesis does not mess with the *.bfi files directly, especially make assumptions about their names. It is possible to stay with the old unified changelog for applications not using more than one profile (by explicitly turning off the new separate changelogs in the config), but I would recommend to use the new way because the old way is deprecated and might be removed entirely later. SyncEvolution has already worked the new way by using separate bin directories for each peer. The commit message says that transition to the new format is done automatically. Doesn't this prevent downgrading? Not entirely, but changelogs will be lost when going back which will force a slow sync. But altough going back is possible here, the binfile store generally does not provide going back (that is, without loosing all settings), only forward. As soon as something changes in the data format, the DB version field in the binfile is incremented, and it will no longer be compatible with the previous version. While is not always possible, we try to support that in SyncEvolution as often as possible. Is there a way to avoid the automatic conversion and continue writing in the old format? Yes, you can set separatechangelogs (within datastore) to false to keep using the old method. However, I would recommend that only as a temporary solution, for example to retain backward compatibility until the next major release (where breaking it is probably more acceptable than between minor revisions). In the long term perspective, I'd like to remove the old method, because it simply does not work correctly with multiple profiles, and multiple profiles is something the engine does have and should work. 2) 4275e77001 (engine: refined folding for pre-MIME-DIR and MIME-DIR - new foldbetween property attribute.): This changes the way folding occurs in pre-MIME-DIR formats (vCard 2.1, vCalendar 1.0), as it no longer inserts extra spaces to break long multi-value properties with no spaces in the values, but instead uses QP (as it already did for single-value properties with no spaces in the value). However, as the new (correct) behaviour probably causes more troubles for poor implementation with multi-values like EXDATES, there's now a new property-level config option to allow inserting a space between value and value separator when needed to wrap long lines. With this, EXDATES is rendered similar as before (before the spaces were inserted AFTER the separator, which caused a extra LEADING space, while now it will be an extra TRAILING space). So the recommendation is to add foldbetween=yes in the config for EXDATES (and possibly other semantically similar properties). What are such other properties? Anything which has a separator in the value? I'd say those where the probability for correct parsing by cheap implementations is better with an extra space, but unchopped individual values. In my setups, the only candidate is EXDATES. Other multi-values like ADR suffer more from the extra space, and for these parsers more likely can handle unfolding correctly. Best Regards, Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] libsynthesis update - 3.0.4.16 (was: Re: optional property parameters, sub/remoterules)
Makes perfect sense to me to postpone these updates for later. Best Regards, Lukas On Sep 29, 2010, at 18:28 , Patrick Ohly wrote: On Do, 2010-09-23 at 02:30 -0700, Lukas Zeller wrote: I owe you an answer regarding commits 0e139ee311 and 4275e77001: [...] But altough going back is possible here, the binfile store generally does not provide going back (that is, without loosing all settings), only forward. As soon as something changes in the data format, the DB version field in the binfile is incremented, and it will no longer be compatible with the previous version. In general that approach is fine, I wasn't expecting anything else. It's just that I'd like to warn users and whenever possible, give them the choice of going back. That increases the number of people who can try out an experimental snapshot, because they can try something and then fall back to the stable version more easily. Given that these two changes are fairly intrusive, I think I'll pass for SyncEvolution 1.1. Instead I'll release with some specific fixes backported, and then combine the libsynthesis update with some other non-backward compatible changes in SyncEvolution itself. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. Lukas Zeller (l...@synthesis.ch) - Synthesis AG, SyncML Solutions Sustainable Software Concepts i...@synthesis.ch, http://www.synthesis.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] temporary UID mapping
Hello Patrick, On Dec 9, 2010, at 13:24 , Patrick Ohly wrote: On Do, 2010-12-09 at 11:26 +, Lukas Zeller wrote: Still, the thing is that there's never a deletion of a single tempGUID map entry (of course, there ARE deletions of map entries with other types, but not of mapentry_tempidmap). Does that type map to sysync::cMapID::ident? Yes. In my mapping on disk I currently see only entries with ident == 2 and a a remote ID. 2 = is mapentry_tempidmap. After a successful sync to a non-empty client, I'd expect some ident==1 entries (= mapentry_normal = real SyncML remoteID-localID map entries) as well. ident==3 (= mapentry_pendingmap) entries only exist on a client after a suspended or aborted sync which had Adds from the server. So you are saying that entries with ident == 2 are never meant to be removed from the mapping? Not exactly that - what I'm saying is that these are (or should) always be removed all together. So either the list is still growing, or all of its items should go away at the same time, which cleans up the ID space. From a DB plugin perspective, you should always see a row of DeleteMapItem operations, and after these are executed no ident==2 items should be left in the database. I checked through the code that issues these DeleteMapItem operations, and I don't see how something else could happen. The mechanism is not trivial because there's a level in between fTempGUIDMap and the actual load/save (fMapTable) to optimize DB accesses (prevent deleting and re-adding entries in the DB if mappings just get temporarily deleted and re-added between load and next save). Maybe there's a condition that can cause some, but not all ident==2 entries to become deleted. Won't that overflow the disk after (some admittedly rather long) usage with many new items transferred? That would defintely be a bad thing ;-) Lukas ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] sending iCalendar 2.0 recurrence rule to Nokia phone: rule ignored (BMC #11241)
Hello Patrick,
On Dec 14, 2010, at 12:59 , Patrick Ohly wrote:
The until date/time should IMHO be handled exactly like dates with the
CONVMODE_AUTOENDDATE conversion mode, which renders date-only values
as floating date+time in mimo_old (vCalendar xxx), but as date-only
values in MIME-DIR (iCalendar 2.0).
You are not suggesting to use convmode=auto[end]date as it is, right?
No, just similar in concept. Such that you can think more or less in iCalendar
2.0 (UNTIL can be date-only or date-time, depending whether the master is an
all-day or an appointment), and the engine will take care of representing it in
vCalendar 1.0.
If I read the documentation right, it will turn 20101231 into either
20101231T235959 (autoenddateinclusive) or 20101231T00.
20101230T235959, not 31st (typo assumed :-)
In both cases it would cut off the last occurrence on day 20101231.
Yes, that can't be applied 1:1 to UNTIL. I'd suggest that for an all-day the
vCalendar 1.0 representation would be 23:59:59 on that day (and not on the day
before as for DTEND). With this, the last occurrence will be included for
certain.
But I can see how converting to 20101231T235959 might work. My goal in
the patch was to recreate exactly the same RRULE as sent by the Nokia
phone, with the right time embedded in it. Not sure whether that is
really necessary, perhaps something larger would also work.
In my experience it works better with a generous recurrence end. I haven't met
a SyncML client having any problems with an recurrence end anywhere between the
last occurrence and the next date produced by the rule. But in a few cases I've
seen the opposite, implementations cutting off the last occurrence even if the
end date correctly matched DTSTART. So I think it's safe and most compatible to
send 23:59:59 for a vCalendar 1.0 allday.
My suggestion (I'll propose a patch soon) would be to make RR_END
behave like CONVMODE_AUTOENDDATE to handle the case of an input
timestamp which is a date-only.
Okay. Let me know when it is ready and I'll redo my tests.
My suggestion (not tested yet) would be:
From 08858608eeb252eef84b340ccc76f0b4f890a19d Mon Sep 17 00:00:00 2001
From: Lukas Zeller l...@synthesis.ch
Date: Tue, 14 Dec 2010 15:53:28 +0100
Subject: [PATCH] engine: prevent generating invalid recurrence end
specifications in vCalendar 1.0 RRULE (date-only not allowed, must be date+time)
For all-day entries, RR_END can become a date-only value
(for example when read in via iCalendar 2.0). Up to now,
rendering these as vCalendar 1.0 resulted in an invalid
RRULE containing a date-only recurrence end specification.
This patch makes sure the recurrence end specification
is always shown as a datetime (23:59:59 on the last
occurrence)
---
sysync/mimedirprofile.cpp |9 +
1 files changed, 9 insertions(+), 0 deletions(-)
diff --git a/sysync/mimedirprofile.cpp b/sysync/mimedirprofile.cpp
index 5d461f7..7db894a 100644
--- a/sysync/mimedirprofile.cpp
+++ b/sysync/mimedirprofile.cpp
@@ -1399,6 +1399,7 @@ bool TMimeDirProfileHandler::fieldToMIMEString(
case CONVMODE_AUTOENDDATE:
case CONVMODE_AUTODATE: // show date-only as date in iCal 2.0
(mimo_standard), but always as timestamp for vCal 1.0 (mimo_old)
if (fMimeDirMode==mimo_standard) goto timestamp; // use autodate if
MIME-DIR format is not vCal 1.0 style
+ // for vCal 1.0 style, always renders as timestamp (as date-only values
are not allowed there)
case CONVMODE_TIMESTAMP: // always show as timestamp
// get explictly as timestamp (even if field or field contents is date)
autodate = false; // do not show as date, even if it is a date-only
@@ -1715,8 +1716,16 @@ bool TMimeDirProfileHandler::fieldToMIMEString(
lineartime_t tzend = until;
// A RRULE with no end extends at least into current time (for tz range
update, see below)
if (until==noLinearTime) {
+ // no end, but we still need a range to generate time zones for
tzend = getSession()-getSystemNowAs(TCTX_UTC);
}
+ else {
+// Treat RR_END similar to CONVMODE_AUTODATE, i.e. prevent rendering a
date-only value in mimo_old (which is not correct according to the standard)
+if (TCTX_IS_DATEONLY(untilcontext) fMimeDirMode==mimo_old) {
+ // there are no date-only recurrence ends in vCalendar 1.0
+ until =
lineartime2dateonlyTime(until)+secondToLinearTimeFactor*SecsPerHour*24-1 ; //
make time part 23:59:59.999 of this day
+}
+ }
// Now do the conversion
bool ok;
if (fMimeDirMode==mimo_old) {
--
1.7.2.3+GitX
Best Regards,
Lukas
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] sending iCalendar 2.0 recurrence rule to Nokia phone: rule ignored (BMC #11241)
Hello Patrick, On Dec 14, 2010, at 16:38 , Patrick Ohly wrote: My suggestion (not tested yet) would be: From 08858608eeb252eef84b340ccc76f0b4f890a19d Mon Sep 17 00:00:00 2001 From: Lukas Zeller l...@synthesis.ch Date: Tue, 14 Dec 2010 15:53:28 +0100 Subject: [PATCH] engine: prevent generating invalid recurrence end specifications in vCalendar 1.0 RRULE (date-only not allowed, must be date+time) Almost works, except that the added time is truncated because the untilcontext still says date-only for the conversion. I've added one line (patch below) and commit an amended commit to the meego.gitorious.org repo. [... patch ...] Oops. Thanks for the fix! Lukas PS: regarding the map issues - I'm still investigating (and getting distracted by all sorts of things :-( ) ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] temporary UID mapping
Hello Patrick,
On Dec 9, 2010, at 17:47 , Patrick Ohly wrote:
I might have found it.
Scenario:
- SERVER is the ID on the server, CLIENT on the client
- server has a mapping from SERVER to #35 (ident 2) and to CLIENT (ident 1)
Where is this from? I assume it is from a previous session (that has not
completed, otherwise it should not be there). That would be ok.
Strictly following the SyncML specs, sending the SERVER ID (or a temp ID
representing it) to the client is only needed for Add (to allow mapping
back). However, because the engine has the alwayssendlocalid config option,
it always calculates the temp ID, even if the option is off. This is something
that could be optimized (but it would hide the bug we're after, so that's an
exercise for later :-)
- item is deleted on server
- two-way sync
What I now see in the log is this:
fTempGUIDMap: restore mapping from #35 to SERVER
Whatever origin that is and whether it makes sense here or not, I agree that
this can happen and should not cause problems.
Item LocalID='SERVER', RemoteID='CLIENT', operation=delete
SyncOp=delete: LocalID=SERVER RemoteID=CLIENT
fTempGUIDMap: translated realLocalID='SERVER' to tempLocalID='#63'
By itself, this is correct. The temp mapping #35-SERVER that was loaded at the
beginning of the sync is invalid by now (transferring data for this session has
begun). So basically it is ok that the engine generates a new temp mapping.
What surprised me is that the old #35-SERVER mapping still exists in
fTempGUIDMap at this point. As I said earlier, fTempGUIDMap is cleared once per
session before items are exchanged. But not at the very beginning of the
session, because a client might send pending maps left over from the previous
session.
Now it turns out that there's a bad bug here, which probably explains all this
(and maybe other) weird behaviour. Fact is that fTempGUIDMap gets cleared way
too early, that is BEFORE apiLoadAdminData is called. Which means that old and
new tempGUIDs get mixed here. And that fTempGUIDMap can become a more or less
ever-growing list (growth however limited a lot by the very collisions we are
talking about).
The clearing of fTempGUIDMap must happen at the point where the server proceses
the Sync command, that is when exchange of this session's items starts. This
is before the datastore changes state from dssta_syncmodestable to
dssta_dataaccessstarted.
That's what I did now (again not tested yet because I have to run out of the
office now). But I wanted to share the finding already, so here's a patch:
From a5e1bd5241380d6f2e19e1affdc1002d71a638f4 Mon Sep 17 00:00:00 2001
From: Lukas Zeller l...@plan44.ch
Date: Thu, 16 Dec 2010 18:32:02 +0100
Subject: [PATCH] engine: server case: fixed bad bug that could mess up
tempGUIDs. These must be cleared when first Sync is received.
Until now, the tempGUIDs were cleared only once when Alert was received.
This is ok to create a clean starting point for loading previous session's
tempGUIDs, which might be needed to resolve early maps also related to
the previous session.
However, as soon as the current session starts Syncing, that is,
exchanging new items, these temporary IDs become invalid and
fTempGUIDMap must be cleared once again.
The missing cleanup has led to collisions in the tempGUID name space,
because the tempGUIDs are created based on the fact that the list
is empty at the beginning of Sync. As it could have entries
from (possibly much!) earlier sessions, all sort of weird things
could and did happen.
Thanks Patrick for the extensive investigation of this!
---
sysync/localengineds.cpp | 11 ---
1 files changed, 8 insertions(+), 3 deletions(-)
diff --git a/sysync/localengineds.cpp b/sysync/localengineds.cpp
index ba8307f..b084cb4 100755
--- a/sysync/localengineds.cpp
+++ b/sysync/localengineds.cpp
@@ -2424,8 +2424,11 @@ TAlertCommand *TLocalEngineDS::engProcessSyncAlert(
// if we process a sync alert now, we haven't started sync or map
generation
#ifdef SYSYNC_SERVER
if (IS_SERVER) {
- // server case: forget Temp GUID mapping
- fTempGUIDMap.clear(); // forget all previous temp GUID mappings
+ // make sure we are not carrying forward any left-overs. Last sessions's
tempGUID mappings that are
+ // needed for early map resolution might be loaded by the call to
engInitSyncAnchors below.
+ // IMPORTANT NOTE: the tempGUIDs that might get loaded will become
invalid as soon as Sync
+ // starts - so fTempGUIDMap needs to be cleared again as soon as the
first Sync command arrives from the client.
+ fTempGUIDMap.clear();
}
#endif
// save remote's next anchor for saving at end of session
@@ -3902,7 +3905,9 @@ bool TLocalEngineDS::engProcessSyncCmd(
startingNow = true; // initiating start now
#ifdef SYSYNC_SERVER
if (IS_SERVER) {
- // - let local datastore (derived DB-specific class) prepare
Re: [os-libsynthesis] dealing with PHOTO:file:// (BMC #19661)
Hello Patrick, thanks for the READ() script function! This is sure a useful extension of the script toolset. However, I'd like to mention in this context that the engine for a long time contains a mechanism for the general problem with large fields. Some data (usually opaque binary data like the PHOTO or email attachments, but also possibly very large text fields like NOTE) should be loaded on-demand only, and not with with the syncset and the fields that are needed for ID and content matching. So string fields can have a proxy object (a better term would probably be data provider), which is called not before the contents of the field is actually needed - usually when encoding for the remote peer. It is the p mode flag in the datastore maps which controls the use of field proxies. In the ODBC/SQL-backend, these proxies are configured with their own SQL statement which loads the field's data. In the plugin backend, which is used in SyncEvolution, the single-field-pull mechanism is mapped onto the ReadBLOB/WriteBLOB Api. The proxy mechanism was even designed with the idea that really huge data should never be loaded as a block, but only streamed through the engine. However, that was never implemented on the encoding side, as the current SyncML item chunking mechanism is not ready for streamed generation (total size must be known in advance). But for a SQL based server like our IOT server, it already helps a lot if contact images are NOT loaded as part of the syncset loading, but only when actually needed. This is JFYI - for the problem at hand the READ() script solution is sure a clean and efficient way to go. Best Regards, Lukas On Jul 22, 2011, at 9:37 , Patrick Ohly wrote: Hello! I'm currently working on https://bugs.meego.com/show_bug.cgi?id=19661: like N900/Maemo 5 before, MeeGo apps prefer to store URIs of local photo files in the PIM storage instead of storing the photo data in EDS (better for performance). When such a contact is synced to a peer, the photo data must be included in the vCard. Ove solved that in his N900 port by inlining the data when reading from EDS, before handing the data to SyncEvolution/Synthesis. This has the downside that the data must be loaded in all cases, including those where it is not needed (slow sync comparison of the other properties in server mode) and remains in memory much longer. I'd like to propose a more efficient solution that'll work with all backends. Ove, if this goes in, then you might be able to remove the special handling of photos in your port. The idea is that a) the data model gets extended to allow both URI and data in PHOTO and b) a file URI gets replaced by the actual file content right before sending (but not sooner). Lukas, can you review the libsynthesis changes? See below. I pushed to the bmc19661 branch in meego.gitorious.org. Some other fixes are also included. --- $ git log --reverse -p 92d2f367..bmc19661 commit 01c6ff4f7136d2c72b520818ee1ba89dc53c71f0 Author: Patrick Ohly patrick.o...@intel.com Date: Fri Jul 22 08:12:05 2011 +0200 SMLTK: fixed g++ 4.6 compiler warning g++ 4.6 warns that the rc variable is getting assigned but never used. smlEndEvaluation() must have been meant to return that error code instead of always returning SML_ERR_OK - fixed. diff --git a/src/syncml_tk/src/sml/mgr/all/mgrcmdbuilder.c b/src/syncml_tk/src/sml/mgr/all/mgrcmdbuilder.c index ae040a4..601b530 100755 --- a/src/syncml_tk/src/sml/mgr/all/mgrcmdbuilder.c +++ b/src/syncml_tk/src/sml/mgr/all/mgrcmdbuilder.c @@ -698,7 +698,7 @@ SML_API Ret_t smlEndEvaluation(InstanceID_t id, MemSize_t *freemem) return SML_ERR_WRONG_USAGE; rc = xltEndEvaluation(id, (XltEncoderPtr_t)(pInstanceInfo-encoderState), freemem); - return SML_ERR_OK; + return rc; } #endif commit 8d5cce896dcc5dba028d1cfa18f08e31adcc6e73 Author: Patrick Ohly patrick.o...@intel.com Date: Fri Jul 22 08:36:22 2011 +0200 blob fields: avoid binary encoding if possible This change is meant for the PHOTO value, which can contain both binary data and plain text URIs. Other binary data fields might also benefit when their content turns out to be plain text (shorter encoding). The change is that base64 encoding is not enforced if all characters are ASCII and printable. That allows special characters like colon, comma, and semicolon to appear unchanged in the content. Regardless whether the check succeeds, the result is guaranteed to contain only ASCII characters, either because it only contains those to start with or because of the base64 encoding. diff --git a/src/sysync/mimedirprofile.cpp b/src/sysync/mimedirprofile.cpp index 4105d03..1499876 100644 --- a/src/sysync/mimedirprofile.cpp +++ b/src/sysync/mimedirprofile.cpp @@ -23,6 +23,7 @@ #include syncagent.h +#include ctype.h using namespace sysync;
Re: [os-libsynthesis] uri + binary PHOTO values (was: Re: field merging + parameters)
On Jul 22, 2011, at 16:08 , Patrick Ohly wrote: On Fr, 2011-07-22 at 15:41 +0200, Lukas Zeller wrote: Hi Patrick, On Jul 22, 2011, at 15:00 , Patrick Ohly wrote: I couldn't think of cases where avoiding binary encoding might have a negative effect, but you are right, it is not according to spec and it makes sense to not change existing behavior. Although I guess many clients would bail out when encountering a PHOTO (non-URI type) with non-B64 content, Why? Because they (incorrectly) assume that PHOTO must be B64 encoded, without checking the parameters to ensure that it really is? Just wondering. I can imagine that there are such clients. I can't imagine photo data possibly be ASCII so I admit this cannot happen. There are pure-ASCII image formats. Okay, not very common in vCards I guess ;-) Indeed! OOZZZ$$$Z$$Z$ZZZOOOZZZZZ OOZZZZZZ$$ZZ$$$ZZZ$$$ZZZZOZZOOOZZZOZ ZZZZOOZ$ZZZZZZZ$$ZZOOOOOZZOZOZOO ZZZOOZO$$$$$$$ZZ OOZZOOOZZ$Z$$$ZZOZZZOZZOOZOO OZOOOZZOOZO$Z$ZZZOOOZZZOOOZOOO88 OOOZZOOZOOOZZZ$ZZ$$Z$ZZ$ZOOZZZOO ZZOZZZZ$$$Z$$$Z$$$ZZ$OOO $$ZZOOZOOOOZZZZZZOZZZ$77$ZZOZ$ZZ$ZZOOOO8ZOOO $$Z$$Z$ZZOOOZZ$$$ZOOOZ7III77$ZZOZZOZZZOZZZO8Z$$$ZZOO $ZOOOZ$$777$OOZ$77I77I???II7$Z$ZZ$$ZZOO88OOZZOOZZZOOOZ$$ $Z$$$ZZOZZ7$77$ZZZ$ZZZ$7I==+??III77$$$$$ZZZO8OZZZ$ZOOZ$$$ZZZ$$Z$$$ZZZ$ZZ $$$$ZZ$$$$ZOZ$I+===++=+??III777$$$77Z$77$ZZZOZOOOZZZOOZ$$ZZZ$$ZZ ZZ$$$7ZOZZZ$I?~~=+$$7III7$$$Z$ZZOOOZOOO$7$$$Z$ZZ $Z$$$$ZOZ$$I+=+??I7II???I77$$$Z$ZZO$7$$Z $$$7$$ZZ$ZZO$7+===~:~=+III?++?III?II++???I7$$$OZ 77$$77Z$$ZZ$$777III777$OI+=++?III??I??+=~~===++==~==+???+===+III?I77$ZZOOZ$Z 777$77$$777$$7$$$Z$Z++???+==?II?+::,:~~~::~===++===+++==~~:~=++?I?II7$$$ZOO$ 77$$777$+=~=I7I+~::~~:,,:~::~~:~~++=~~==+++=~~:~=+==+III77$$$ZOO 7777$ZZOOZZ$$77II777I7?++=?77??7$?=~~:,,:::,,,::,,,::==+=~~~=+++=~~=+?+==+III77$$$ZO8OZ$ II???II7$ZZ$Z$ZZ$77II7III77I77II??77?IOD8O+:,..,::,,.,,~=::=+=~~==+++==++===+~=?777$77ZZ8D8O$777 III7$$7$$$$7$I7$77Z778D$+$7~:,,,::,..,:=+:,,,~==~=:::~:=?$77$77$7$$Z8DDDO777 $$$777II777$$$ZZI7ZO$$$7ZDD8O77?~,...,,:~??=:,,~===++?~~~==::~::=?77$$$Z8Z77 77II777$$$OZI$88OZ$I$8DDDZ$?~::,,,..,,:=?OO$?~,:+?=:~==~:+I7I777$$$ODZ77 777III???II78DD8OZI++?II?+=::::,~?7Z877O8Z+:~~=+??=:,:+?+~==~=??I77$$ZO8DO77 III??I??I77II7III7ZOD88OZ7+~:::,:::~?$Z$$:=$88Z+::=+??+~::~+II???+?+I7$$ZZ8DDZ$77I77 I7I??7III?77II$8O88O$I+~,:~:~::,,,:~+?7$OZODOO87:=+~::~=?III7OZ?I$$ZZODDD$77 77?7III?IZOZZOO$7I~:~~===~~:~~==~::,::,,:~~?7OOZ$$7?+7?=~~~==?ZDDDOI7$$Z8DD8Z777 II77I?I?III7$OZZZOZ$7I~:==++=~~~::,,,:,,,:,:~?II+++=?7III7I=~~~=+IODDD8$7$ZZ8DDO$77I III777II?IIIIII7Z$$77?~++??=~~~~,,...:==++??I77??8DZ7$ZO8D8ZI777 II??II??IIII?II7ZZZ$7?+??+===~~:,:.,:~=+=+++?I77?+?I$DDOI$Z8D8ZZ7I77I?I7 I???III??I???I7$$ZZOOZ$7II777??+=~~~::::,,,,,,,,:=+==+++II$Z$77ODDDO$$OO8DDD8ZI7$?III?II ??I??I???I$$$ZZOZ$7III77I?+=~~~=~:::,,:...,,:=+==++?IIZZ$$$ODD8$ZOOZODDDO7II$??II?II II??7ZOZ$7I???~?O8D88OZZ7I=:,,~:,:,,.,,:~++==+?IIIZ$I78DDO$?ODD8ZIII7I?I I???++???++7$$777ZO$7I
Re: [os-libsynthesis] Problems developing libsynthesis based server
Hello Andris, I haven't fully understood your scenario yet. I see that your server has 4 items which apparently, at some time, have existed on the client (otherwise, there would be no RemoteID known at the server's end). In the sync you are running however, these items for some reason don't exist any more in the client, so the result is a 404. Could you perhaps send a complete set of logs, starting with the initial slow sync and then the subsequent to-client-only sync that causes the problems? From the log snippets below I'd assume the real problem happened in the session before that one, and the 404 are just consequential errors. Best Regards, Lukas PS: Don't make your DB plugin insert a new item when no item for update is found. libsynthesis contains code to do that when appropriate - if the plugin did that by itself that could severly mess up things. libsynthesis relies on plugins to return 404 when an item addressed does not exist in the DB. On Jul 29, 2011, at 13:39 , Andris Pavenis wrote: I have some problems with server to client only synchronization. I used our own libsynthesis based server and 2 different clients for testing (Nokia E65 mobile phone and own libsynthesis based test client). Currently contacts data-store is marked read/only in server side configuration. On client side data-store is emptyed. Below are fragments of both server and client side logs. Removing DS admin data before sync causes slow sync and in that case this problem does not appear. Tried also to remove DS admin data directly after reading them from LoadAdminData, but it does not help. I could of course insert a new item when requested item is not found in UpdateItem and return its ID through newID, but it would fix problem only for my test client. Server must be able to work also with client available for example in mobile phones. What could be wrong? Andris Server side: – [2011-07-29 12:37:59.766] 'SyncSet' - Items involved in Sync, datastore=contacts [--][++] [-end] [-enclosing] • [2011-07-29 12:37:59.766] SyncOp=wants-replace : LocalID= 1 RemoteID= 1 • [2011-07-29 12:37:59.766] SyncOp=wants-replace : LocalID= 2 RemoteID= 2 • [2011-07-29 12:37:59.766] SyncOp=wants-replace : LocalID= 3 RemoteID= 3 • [2011-07-29 12:37:59.766] SyncOp=wants-replace : LocalID= 4 RemoteID= 4 – [2011-07-29 12:37:59.767] 'issue' - issuing command, Cmd=Sync [--][++] [-end] [-enclosing] • – [2011-07-29 12:37:59.767] 'sync' - Opened Sync command bracket, Reopen=no, SourceURI=contacts, TargetURI=./contacts, IncomingMsgID=2, CmdID=3 [--][++] [-end] [-enclosing] • [2011-07-29 12:37:59.768] Opened Sync bracket, Source='contacts', Target='./contacts' as (outgoing MsgID=2, CmdID=3) • – [2011-07-29 12:37:59.768] 'SyncGen' - Now generating sync commands, datastore=contacts [--][++] [-end] [-enclosing] • [2011-07-29 12:37:59.768] Created command '[unknown]' (outgoing) • – [2011-07-29 12:37:59.768] 'Item_Generate' - generating SyncML item, SyncOp=wants-replace, RemoteID=1 [--][++] [-end] [-enclosing] • [2011-07-29 12:37:59.768] Generating • [2011-07-29 12:37:59.768] Item LocalID='', RemoteID='1', operation=wants-replace • [2011-07-29 12:37:59.768] *** field data not shown because userdata log is disabled *** –[2011-07-29 12:37:59.768] End of 'Item_Generate' [-top] [-enclosing] Client side (item is not found in data-store and as result code 404 is generated). • – [2011-07-29 12:38:03.692] 'processCmd' - Processing incoming command, Cmd=Replace, IncomingMsgID=2, CmdID=70 [--][++] [-end] [-enclosing] • command started processing • Created command 'Status' (outgoing) • Item (syncop: replace) started processing, remoteID='', localID='67' • Remote sent replace-operation: • - Source: remoteID ='', remoteName='' • - Target: localID ='67', remoteName='' • Explicit type 'text/vcard' specified in command or item meta • Version '3.0' obtained from item data • – [2011-07-29 12:38:03.692] 'Item_Parse' - parsing SyncML item, SyncOp=replace, format=plain-text, LocalID=67 [--][++] [-end] [-enclosing] –[2011-07-29 12:38:03.693] End of 'Item_Parse' [-top] [-enclosing] • – [2011-07-29 12:38:03.693] 'Process_Item' - processing remote item, SyncOp=replace, LocalID=67 [--][++] [-end] [-enclosing] • replace item operation received • startDataWrite called, status=0 • TStdLogicDS::logicProcessRemoteItem starting, SyncOp=replace, RemoteID='', LocalID='67' • cannot update record in database (sta=404)
Re: [os-libsynthesis] Google Calendar + per-item permission errors
Hello Patrick, On Aug 1, 2011, at 20:39 , Patrick Ohly wrote: In a local slow sync between Evolution and Google Calendar I see the following problem: [...] * The server logs Status: 403: originator exception and WARNING: Aborting Session with Reason Status 403 (REMOTE problem). That says it all. The session ends here without finishing the slow sync. This is a bit too drastic. [...] Should it be turned into a local error along the were somewhere? I remember vaguely that we discussed something like that and came to the conclusion that a backend can return errors in the range 500 to the Synthesis engine. That's what SyncEvolution is doing here. There's a place in localengineds.cpp (around line 3337) where all the error codes are catched in a switch/case statement that should NOT abort the session, but rather mark the item for resend later, which means in the next session. 403 is not among these, because usually a auth problem is really fatal and retry does not help. For your case at hand, I'd recommend to translate that kind of 403 error to 417 (retry later) in the backend (or in the sentitemstatusscript), which means that the backend has some hope that a retry in the next session might help. Here's the excerpt from the switch statement which lists the error codes that lead to retry-later behaviour, along with comments that describe the rationale to catch these codes here and not let them abort the session: case 424: // size mismatch (e.g. due to faild partial item resume attempt - retry will help) case 417: // retry later (remote says that retry will probably work) case 506: // processing error, retry later (remote says that retry will probably work) case 404: // not found (retry is not likely to help, but does not harm too much, either) case 408: // timeout (if that happens on a single item, retry probably helps) case 415: // bad type (retry is not likely to help, but does not harm too much, either) case 510: // datastore failure (too unspecific to know if retry might help, but why not?) case 500: // general failure (too unspecific to know if retry might help, but why not?) Note that the retry feature only works with DB backends that support resume (in particular, the flag field in the ID mapping table), which was always the case for SyncEvolution, IMHO. Best Regards, Lukas Zeller, plan44.ch l...@plan44.ch - www.plan44.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] Plain text (B64) authentication when plugin uses Password_MD5_Nonce_IN
Hello Andris, On Aug 4, 2011, at 15:22 , Andris Pavenis wrote: There is note in TPluginApiAgent::CheckLogin() that plain text authentication is impossible when plugin uses password verifucation type Password_MD5_Nonce_IN. I guess it is still possible as one have access to - user name - password - nonce value (it may however be queried in the begin of method) From these data one can generate MD5(MD5(user:pwd):nonce) to provide to the plugin for authentication. This situation is useful when real authentication check is provided by other server outside the libsynthesis plugin and this server does not provide autside access neither to password nor MD5(user:pwd). Makes perfect sense. Thanks for the patch! Would you mind entering the contributor agreement (see explanation and link at http://www.synthesis.ch/indefero/index.php/p/libsynthesis/), so I can include the patch into the libsynthesis repo? I don't recall why I decided to reject plain text auth with a DB that can check MD5+nonce when I wrote that code. Probably I didn't believe that clients would actually use plain text auth any more. But as you point out, using it is not completely insane if transport is always SSL (I hope it is, in case of that Fumabol client?). Still, I see no reason why any client should still use plain text. It's simply bad practice to transmit original secrets. Of course you can't intercept them from the SSL connection, but all layers of the software at the server end of the SSL connection will see and pass that secret around. That does not speak against your patch, of course! Attached is some very preliminary patch to include support for plain text login in this case. It does not check config setting whether plain text authorisation should be allowed at all (plain text login is not so bad evil in case of using SSL). There's no need to check that in the plugin. Checking if plain text auth is allowed at all in the config is done in SyncSession for all types of DB backends (after asking the plugin - so even if the plugin says ok, but config disallows plain text, the login will be rejected). With this patch authentication of Funambol Android client from Android Market (which uses plain text login and XML) suceeded even if this cĺient still does not working with libsynthesis based server (and seems that I'll have to give up in attempts to get it working). :-( What kind of other obstacles did you run into with that client? Best Regards, Lukas Lukas Zeller, plan44.ch l...@plan44.ch - www.plan44.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] Contact Synchronization problem between libsynthesis based server and S60 3rd Edition phones
Hello Andris, On Sep 12, 2011, at 11:59 , Andris Pavenis wrote: There are following problems synchronizing contacts between libsynthesis based server (our own implementation) and S60 3rd edition mobile phones (I used Nokia E65 and E50 for testing). I'm giving an example to illustrate the problem: - let us assume that contact on both server and in mobile phone address book contain one or several phone numbers of the same type (for example TEL_FLAGS[xx]:8): TEL[0]: TEL_FLAGS[0]:8 TEL[1]: TEL_FLAGS[1]:8 TEL[2]: TEL_FLAGS[2]:8 TEL[3]: TEL_FLAGS[3]:8 - a part of phone numbers are deleted on server side (not from the end): TEL[0]: TEL_FLAGS[0]:8 TEL[1]: TEL_FLAGS[1]:8 - An attempt to synchronize device causes SyncML Replace command to be generated for updating contact on client side. First 2 numbers on client device are replaced with 2 numbers send from server but 3rd and 4th number remains unchanged TEL[0]: TEL_FLAGS[0]:8 TEL[1]: TEL_FLAGS[1]:8 TEL[2]: TEL_FLAGS[2]:8 TEL[3]: TEL_FLAGS[3]:8 The same problem appears also when there is only one number of each type present and some of them are deleted on server side. After synchronization it is still present on client side (none replaced, so the first remains where it was) As far as I checked Funambol server tries to workaround this problem by sending empty phone number after I remove phone number using Funambol WebGui demo version (it supports however only one number of each type so test possibilities are rather limited) There is a config option for libsynthesis for repeating properties. You can set the minshow attribute on position (see SySync_config_reference 10.3.7, approx page 84) to force the engine to produce empty fields in excess of the number of stored values. The problem is that the handling of empty properties is highly client specific (from crashing, to ignoring them, to interpreting them to actually wipe existing fields). Note that the suppressempty attribute of property as well as the noemptyproperties in remoterule can both overrule a minshow specification. Best Regards, Lukas Lukas Zeller, plan44.ch l...@plan44.ch - www.plan44.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] super data store: trying to read to check for existance
Hello Patrick, On Oct 24, 2011, at 16:55 , Patrick Ohly wrote: Why is it necessary to read before trying to delete? If the item exists, then reading it is a fairly expensive test. Unfortunately, with some some backends, this is the only test that reliably works. For example, some SQL databases (or their ODBC driver) can't return a correct number of affected rows for DELETE statements. So reading before deleting was the only way to detect if any of the subdatastores really had that item and correctly return 404 or 200. So far, my backends were written with the expectation that they have to cope with delete requests for items which are already deleted. This follows from the inherent race condition between syncing and some other process which might delete items while a sync runs. Are all backends expected to return 404 when an item doesn't exist? Not all backends (see above, built-in ODBC can't), but yes, for plugin backends, returning proper error codes is specified, also for delete. Still, if a plugin would not conform to that in its implementation of delete, that would probably have gone unnoticed so far. Of course, the test is a bit expensive - if that's a concern, one could differentiate between plugin datastores and others (all the builtin ones, including SQL/ODBC), and use the expensive read test only for the latter. Like a virtual dsDeleteDetectsItemPresence() which returns false by default, but can be overridden in plugin datastores to return true. Best Regards, Lukas Zeller, plan44.ch l...@plan44.ch - www.plan44.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] 409 item merged in client
Hello Patrick, sorry for the late answer. I've been on a business in Germany last week and as usual, stuff has piled up in my inbox. Digging though the case for a while, I suddenly realized that in client case the implProcessItem() method used is the one in binfileimplds.cpp and not the one in customimplds.cpp. However, we added the 409 merge trick in customimplds.cpp only - no wonder it goes south in the client case. I'll check if I can easily add the same functionality to binfileimpds. The reason why it is a separate method with seemingly duplicate functionality is that binfileimpl is integrated tightly with the binfile-based change log mechanism (even if you don't use it) - which makes the implementation differ quite a bit even if functionality provided is (i.e. in this case: *should be*) the same. Best Regards, Lukas PS: I'm almost trough with detangling my repositories such that I can now work exclusively with the opensource libsynthesis repo, which will help a lot keeping everything up-to-date there and integrate patches more quickly. As soon as the transition is complete, I'll also move the public libsynthesis repository to gitorious. On Dec 14, 2011, at 16:10 , Patrick Ohly wrote: I've started to test SyncEvolution client - SyncEvolution server automatically. The server is currently using the plain file backend and thus cannot detect add-add UID conflicts. In the test of that aspect I noticed the following problem: * client and server both have a new item with UID=foo * client sends an Add UID=foo to the server, which accepts the new item, leading to a duplication on the server * in the same session, the server sends his version of the UID=foo item * the client's backend detects the duplicate and returns 409 to the engine Here's the detailed log: http://syncev.meego.com/2011-12-14-12-24_testing_syncevohttp/testing-amd64/15-syncevohttp/Client_Sync_eds_event_testAddBothSides.send-update.client.B/syncevolution-log.html * [2011-12-14 12:48:53.896] InsertItemAsKey res=409 * [2011-12-14 12:48:53.897] cannot create record in database (sta=409) * [2011-12-14 12:48:53.898] Database Error -- SyncML status 409 * [2011-12-14 12:48:53.899] - Operation add failed with SyncML status=409 –[2011-12-14 12:48:53.900] End of 'Process_Item' [-top] [-enclosing] * [2011-12-14 12:48:53.901] processSyncOpItem: Error while processing item, status=409 * [2011-12-14 12:48:53.902] Irregularity in execution of item, status=409 – [2011-12-14 12:48:53.903] 'issue' - issuing command, Cmd=Status [--][++] [-end] [-enclosing] * [2011-12-14 12:48:53.903] Command 'Status': is 1-th counted cmd, cmdsize(+tags needed to end msg)=38, available=149687 (maxfree=260907, freeaftersend=260869, notUsableBufferBytes()=111220) * [2011-12-14 12:48:53.904] WARNING: Non-OK Status 409 returned to remote! From there on it all goes south ;-) The next sync in my testing is an intentional slow sync. The server still has the two items with the same UID, the client adds one, then rejects the second = slow sync fails, test aborts. Lukas, can you remind me how this was meant to work? -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] 409 item merged in client + multiple sync cycles in a single session
Hi Patrick, On Mar 9, 2012, at 14:30 , Patrick Ohly wrote: On Tue, 2012-03-06 at 14:50 +0100, Patrick Ohly wrote: I haven't look into this yet, but still have it on my radar. Done in the meego.gitorious.org master branch. I found that checking for collisions is hard (not all records are in memory), so I settled for making the chance of collisions smaller in the string case by including a running count. I guess this is way safe enough. The worst that can happen is that two (at that time by definition already obsolete) server items will get a mapping to the same client ID when a fake-ID generation collision should occur (which now can only happen with suspendresume where libsynthesis is reinstantiated in between). If so, the client will send two deletes for the same clientID to the server in a subsequent sync. Depending on the server implementation, either the first delete wipes all items mapped to that ID and the second delete will get a 404, which is fine. Or the first delete wipes just the first item that maps to that item, and the second delete then wipes the second - correct as well. Even if a super-smart server would merge the two items upon receiving a map to the same clientID, the end result would be correct (altough the merged item would likely make no sense - but as it is doomed at the time of merge already that would be an acceptable intermediate state for a case that is extremely unlikely to occur at all). Best Regards, Lukas Lukas Zeller, plan44.ch l...@plan44.ch - www.plan44.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] About the db plugin in the ios
Hi Zhoulei, On Mar 16, 2012, at 4:46 , zhou lei wrote: I am a newer of the syncML, and try this on ios for a half month. And finally find out that the libsynthesis is best one, it have demo and docs, and guys works on this are all very kind. And I even do not know how the syncML protocol, but I want to know how the libsynthesis works, especially the lib and the db plugin. I just know how the db plugin would be called, but i got more questions, and some of them is quite easy for you but important for me. There are the questions: 1.Is that the libsynthesis should let the db underversion controle, i mean if the version is not the same the syncML would sync with server or if the data is not the same the syncML would sync with the server. libsynthesis normally relies on the plugin to report which records have changed since the last sync. Usually, plugins use a last modified date mechanism to detect which records have changed. For clients, there is also the possibility to use CRC based change detection (see description of crcchangedetection in doc/SySync_config_reference.pdf). With this, the engine always reads all data items and calculates a CRC checksum over the actual data to find out if a item has changed or not. This might be ok or not, depending on how slow or fast the database backend is, and how large the sync set (total number of items). In any case, it's more an emergency method for databases that cannot be changed and don't have a usable last modified timestamp. When designing new databases, it is recommended to implement a stable modified-date. 2.Why the Target Dependencies in the project of SyncMLClientSample is without the static_combiEngine_opensource_ios, when i got the demo, first time I even could not combile this demo. Thats quite easy to solve, but make me think this demo may not work at the first. SyncMLClientSample does have a target dependency on static_clientEngine_opensource_ios (because it's a client-only sample), not static_combiEngine_opensource_ios. But you are right, there is a problem, which is that the SYNCMLLIB build setting was wrong and did point to the combi lib. I fixed this now (on the luz branch in gitorious. But if you need both SyncML client and server, you can use static_combiEngine_opensource_ios instead (change the target dependency and change the SYNCMLLIB setting) 3.Why the docs did not maintioned about the ios db plugin, in the doc i found out the a e.g. named snowwhite, but not for ios, and the doc for the xml is quite simple, I even dont know whats the relationship between the xml and the db plugins, the only way is set a break point and step by step gointo the calls, and there are so many classes, you should take time to find out who is whose father and whose is the child. You are right, the iOS things were not decribed so well, mainly because the iOS version of libsynthesis was open sourced only recently. In particular, I forgot to update and post the readme file for the sample app. I did that now, see d82dbde2b8 (iOS: added readme and fixed SYNCMMLIB build setting which was pointing to the wrong library (sample uses the client lib, not the combi lib)) 4.In the demo client of ios, i found a define of SYNCML_TARGET_DBID 2001, and in the MainViewController.m there is a targetkey for 1001, 2001 is defined in the xml, but where is the 1001, and when get the 1001, it returns NULL, for what purpose? And i say the note in the appdelegate.h, 1003 is maintioned here, are they define by the ios? Or are they defined by the syncML? i mean that for the address boot we only could define it to 1001 or what. Thanks for your precise investigation! That's indeed a bug in the sample. I corrected it now, see 7220cbd400 (iOS sample app: fixed inconsistencies with dbtypeid, now using SYNCML_TARGET_DBID definition everywhere.). The dbtypeid is the key to link the datastore in the config to the app. So for the sample app which uses TextDB based contacts, it must be 2001 everywhere. The 1001 used in MainViewController.m was a bug, sorry for that. Best Regards, Lukas Lukas Zeller, plan44.ch l...@plan44.ch - www.plan44.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] what's deference between dbplugin for cocoa and for cpp
Hi Zhoulei, On Mar 20, 2012, at 10:31 , zhou lei wrote: We can find snowwhite in the doc of the libsynthesis, and in the file named dbapi.cpp we can find it too. This is the C++ branch of things, it is not relevant for iOS (except maybe for general understanding of how the library works on other platforms). And from the demo for IOS, We can find out that, in cocoa, the lib would using the code in dbplugin_cocoa.m. dbplugin_cocoa.m is the wrapper code that handles the somewhat tricky interface between C and ObjC for you. No need to edit this. The actual plugin code to write will go into sample_dbplugin1.mm (2,3,4...). All you need to do is implementing the following methods: - (TSyError)startDataReadWithLastToken:(cAppCharP)aLastToken andResumeToken:(cAppCharP)aResumeToken; - (TSyError)readNextItemAsKey:(SettingsKey *)aItemKey itemIdP:(NSString **)aItemIdP parentIdP:(NSString **)aParentIdP statusP:(sInt32 *)aStatusP isFirst:(BOOL)aFirst; - (TSyError)readItemAsKey:(SettingsKey *)aItemKey itemID:(NSString *)aItemID parentID:(NSString *)aParentID; - (TSyError)endDataRead; - (TSyError)startDataWrite; - (TSyError)insertItemAsKey:(SettingsKey *)aItemKey parentID:(NSString *)aParentID newItemIdP:(NSString **)aNewItemIdP; - (TSyError)updateItemAsKey:(SettingsKey *)aItemKey itemIdP:(NSString **)aItemIdP parentIdP:(NSString **)aParentIdP; - (TSyError)deleteItem:(NSString *)aItemID parentID:(NSString *)aParentID; - (TSyError)endDataWriteWithSuccess:(BOOL)aSuccess andNewToken:(NSString **)aNewTokenP; ...plus maybe some of the optional methods if you have advanced features like filtering (but probably not to start with). These are semantically 1:1 representations in ObjC of the functions you find described in more detail in doc/SDK_manual.pdf and also in the C++ samples. Because of the work the wrapper code does for you, it is only a small subset of what the C plugin ABI consists of. Things like module/db context creation/deletion is done in the wrapper. One important note: the Cocoa wrapper code only uses the AsKey variants from the plugin API. This means that rather than passing items as text strings, a SettingsKey object is passed, on which you can call methods like stringValueByName: / setStringValueByName:toValue: to access individual fields (see SettingsKey.h for all available access methods). Note that this is the same mechanism as for accessing settings, and you'll find examples for SettingsKey usage in the sample iOS app. Best Regards, Lukas Zeller, plan44.ch l...@plan44.ch - www.plan44.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] os-libsynthesis Digest, Vol 33, Issue 6
Hi Zhoulei, On Mar 19, 2012, at 3:57 , zhou lei wrote: In my satuation, I want to sync the address book from the iphone to the sync server, and when needed the client would update the address book from the server. And for now, I just want to make the client works, and would using the syncML server of the google sync. Be careful and don't expect too much, Google contact sync is one of the less stellar SyncML implementations :-( For the local address book, it may not been the iphone's but could be a db from sqlite. If you use SQLite, you don't need to write your own plugin. libsynthesis has built-in SQLite3 interface, all you need is creating a so-called mapping between the fields in the fieldlist and your actual database fields. This is done in the fieldmap section of the datastore in the XML config. Have a look at /src/sysync_SDK/configs/sunbird_client.xml, this is a sample of an interface to a SQLite3 backend. The sample looks a bit complicated because of the large scripts handling all the special cases with exceptions etc., but for a plain contact database you probably will not need any scripts at all, so just a few lines of map definitions will do. And when I set the dbplugin_module to iPhone_plugin1 or iPhone_plugin4, and try to sync, it will do nothing, off cause, there is not any usefull code in the iPhone_plugin1, but the function there is just been call ed when install the module. Must i using the session? All you need to do is filling actual DB access code into the methods in sample_dbplugin1.mm that have a comment with //%%% todo, starting with startDataReadWithLastToken. The wrapper code handles all the rest, instantiates the module and database contexts, and calls the methods you find in sample_dbplugin1.mm. Best Regards, Lukas Zeller, plan44.ch l...@plan44.ch - www.plan44.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] os-libsynthesis Digest, Vol 33, Issue 9
Hello, On Mar 28, 2012, at 5:47 , zhou lei wrote: If the google sync is not the good one, which sync server would be better? It depends on what you want to do. Of course, if you need to get Google contacts, there's no choice. But otherwise, there are many services and software that support SyncML - have a look at the wikipedia article for a extensive list: SyncML Wikipedia article If i set the plugin_module to iPhone_dbplugin1 the client get a 20010 error code.why? 20010 is a XML parsing error (see doc/SySync_config_reference.pdf, page 200). You told me that just modify the code in the sample_dbplugin1.mm.,like readItemAsKey. But I found out that, the functions would never been called in the class of SamplePluginDB1, even the class SamplePluginDB1 never init by the engine. Probably because of the XML error, the engine never gets as far to instantiate the plugins. Best Regards, Lukas Zeller, plan44.ch l...@plan44.ch - www.plan44.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] os-libsynthesis Digest, Vol 33, Issue 9
Hi Zhoulei,
On Mar 31, 2012, at 11:12 , zhou lei wrote:
I try to modify ReadItemAsKey. Using SetValue to set the contacts, and i
just guess that read mean read the local contacts and send them to server.
Exactly. The terms in the plugin are all meant towards the database.
But with the struct fCB in the sample_dbplugin1.mm, just get a error named
EXC_BAD_ACCESS. And I use it like this fCB-ui.SetValue( fCB, aItemKey,
N_FIRST ,VALTYPE_TEXT, 123131 , strlen(123131)); , the fCB was set
when init the SamplePluginDB1.
I would really discourage using a mixture of ObjC and parts from the C++
examples. The ObjC wrapper was done exactly because dealing with the fCB and
it's members directly is difficult, and you need to do all the memory
management on your own.
EXC_BAD_ACCESS is a bus error which is hard to debug, especially as two
completely different memory management systems need to cooperate here (ObjC
refcounted world with C++/C traditional malloc/free world).
The ObjC wrapper classes however are really simple to use from the ObjC world.
I made a skeleton for the two read* methods to show how to easily access fields
from ObjC, without any messing with fCB, and with automatic conversion from
NSString (the SettingsKey also gives you convenience methods to access dates as
NSDate and integers).
- (TSyError)readNextItemAsKey:(SettingsKey *)aItemKey
itemIdP:(NSString **)aItemIdP parentIdP:(NSString **)aParentIdP
statusP:(sInt32 *)aStatusP
isFirst:(BOOL)aFirst
{
TSyError sta = LOCERR_OK;
[self debugOut:[NSString stringWithFormat:@readNextItemAsKey: aFirst=%d,
aFirst]];
// read next item from DB
if (aFirst) {
// %%% reset iterator over dataset here
}
// get next item
BOOL found = NO;
do {
if (/* %%% no more data items */) {
*aStatusP = ReadNextItem_EOF;
break;
}
else {
found = YES;
// %%% get IDs of records from the database
// Note: data is not required at this point, however, if the database
is such
// that it always reads the data anyway when getting the IDs,
the data
// CAN be OPTIONALLY read here already (see below)
NSString *localID = /* item ID as string */ @dummyID;
*aItemIdP = localID;
// %%% find out if it has modified since last time
if (/* %%% modified since last suspend */)
*aStatusP = ReadNextItem_Resumed; // changed since last resume (implies
changed since last sync also)
else if (/* %%% modified since last completed sync session */)
*aStatusP = ReadNextItem_Changed; // changed since last sync (but NOT
since last resume)
else
*aStatusP = ReadNextItem_Unchanged; // not changed, but caller wants to
see all records
// %%% optionally, you can ready actual item data from the DB here and
return it. If you don't deliver
// data here, the engine will later call readItemAsKey to get the data
// %%% just set some sample properties
// - simple strings
[aItemKey setStringValueByName:N_FIRST toValue:/* dummy first name */
@Theodor];
[aItemKey setStringValueByName:N_LAST toValue:/* dummy last name */
@Tester];
// - strings in an array
sInt32 telFieldID = [aItemKey valueIDByName:TEL];
for (int i=0; i3; i++) {
[aItemKey setStringValueByID:telFieldID arrayIndex:i toValue:/* dummy
TEL */ @12345 78 90];
}
// %%% advance iterator to next item (such that next item will be
delivered on next call to reasNextItemAsKey
}
} while (!found);
// done
return sta;
}
- (TSyError)readItemAsKey:(SettingsKey *)aItemKey
itemID:(NSString *)aItemID parentID:(NSString *)aParentID
{
TSyError sta = LOCERR_OK;
[self debugOut:[NSString stringWithFormat:
@readItemAsKey: itemID=%@ parentID=%@,
aItemID, aParentID
]];
// %%% read item identified by NSString in aItemID from DB
if (/* %%% found item with ID aItemID*/) {
// %%% just set some sample properties
// - simple strings
[aItemKey setStringValueByName:N_FIRST toValue:/* dummy first name */
@Theodor];
[aItemKey setStringValueByName:N_LAST toValue:/* dummy last name */
@Tester];
// - strings in an array
sInt32 telFieldID = [aItemKey valueIDByName:TEL];
for (int i=0; i3; i++) {
[aItemKey setStringValueByID:telFieldID arrayIndex:i toValue:/* dummy TEL
*/ @12345 78 90];
}
}
else {
// item not found
sta = DB_NotFound;
}
// done
return sta;
}
Best Regards,
Lukas
PS: please make sure to Cc: your replies to the mailing list (Synthesis
os-libsynthesis@synthesis.ch) as well, and not only to me, so everyone on the
list can follow the conversation.
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] The sync works now. But some other problems about the set value.
Hi Zhoulei,
On Apr 1, 2012, at 4:16 , zhou lei wrote:
I just try it as what to told me to do, and the client works. Thank you very
much.
Glad to hear it works now!
For property TEL in the vcard, it just has too many enum values, how could
i set the TEL value with its enum flags. If only set it by the
setstringvaluebyname, It may use the default one, but i want to set it myself.
The TEL_FLAGS is just an integer field used as a bitmap to create the TYPE
parameter. So bit 0 set will add HOME, bit 1 WORK etc.
So to make a TEL;TYPE=WORK:123456789 telephone number (all three the same in
this sample), you would something like:
// - strings in an array
sInt32 telFieldID = [aItemKey valueIDByName:TEL];
sInt32 telFlagsID = [aItemKey valueIDByName:TEL_FLAGS];
for (int i=0; i3; i++) {
[aItemKey setStringValueByID:telFieldID arrayIndex:i
toValue:@123456789];
[aItemKey setIntValueByID:telFieldID arrayIndex:i toValue:(11)]; //
set bit 1 to make it a WORK number
}
The other fields, TEL_LABEL and TEL_ID are to support the custom label
feature of the iOS address book, and also to expose the iOS ID of the array
elements. If you have your own address book database, you probably don't need
these and you could remove these extra fields from the XML config as well.
Please also have a look at the conversion modes and options around page 81 of
the doc/SySync_config_reference.pdf. Also, on page 86 there is a much simpler
example of how to map telephone numbers to a few fixed fields (instead of
arrays). Depending on your address book, that might make more sense to
implement, rather than using the complex mapping needed for the iOS address
book which can hold a variable number of TELs for each contact and has custom
labels and IDs.
Best Regards,
Lukas Zeller
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] scripts + datastore
Hello Patrick, On Apr 25, 2012, at 15:11 , Patrick Ohly wrote: I am trying to solve one issue in SyncEvolution: when trying to find pairs, it needs to know on a per-datastore basis whether both remote and local storage have truly unique UID/RECURRENCE-ID that can be relied upon (iCalendar 2.0 semantic). So far, I am using a compare script for that, but it has to make assumptions about the peer. To overcome that I added code that allows clients to add SyncCap entries to the CTCap (similar to the can restart flag). This information is stored at the receiving end in the TSyncDataStore base class by the TRemoteDataStore while parsing the SyncCap (again, very similar to fCanRestart). The problem is that CTCap is not necessarily tied to a datastore. It is since SyncML 1.2, but wasn't before. That's also the reason why getRelatedDatastore() can be NULL. It is also NULL when using a type from a script with the MAKETEXTWITHPROFILE et. al. functions. But now my problem is: how can the compare script access that information? It runs inside the datatype context. Does that mean that all datastores sharing the same type also share the same context and that the initscript for the type is only invoked once? insert English word for Jein. The datatype context indeed exists only once per type, so the script variables exist only once. However, the init script might be called more than once if multiple datastores are using the same type. The TScriptContext object will be the same, but script functions that use fDsP (see below) can return different information. So using these in the initscript usually makes no sense. The script functions in multifielditemtype.cpp (like SYNCOP()) looked promising. It's possible to get a pointer to some kind of datastore: TSyncDataStore *related = static_castTMultiFieldItemType *(aFuncContextP-getCallerContext())-getRelatedDatastore(); For getting the local datastore related to a particular script invocation, there is TLocalEngineDS *related = static_castTMultiFieldItemType *(aFuncContextP-getCallerContext())-fDsP; fDsP is one of several variables I'd call current type usage context, fCurrentSyncOp which feeds the SYNCOP() script function is another one of them (See multifielitemtype.h Line 202 ff). But in practice that pointer is always NULL. I wasn't sure anyway whether I would get the pointer to the local or remote datastore. from fDsP you'd get the pointer to the local datastore (TLocalEngineDS *). So to reach the remote datastore from a given TMultiFieldItem (which you certainly have at comparescript), I'd rather use yourMultiFieldItem-getRemoteItemType-getRelatedDatastore() to get to the datastore that has defined that item's remote type. As said, in SyncML 1.1/1.0 cases you'd get NULL back because the devInf does not associate types with datastores at all. But I'd assume your extension does not have to work below 1.2, so that'll be no problem. Best Regards, Lukas ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] libsynthesis and SonyEricsson Xperia S own SyncML client
Hi Andris, On 21.05.2012, at 15:43, Andris Pavenis wrote: Tried to test whether synchronization works with Xperia-S own SyncML client (comes with Android 2.3.7 installed on phone, software version 6.0.A.3.73) using custom libsynthesis based SyncML server. - the first problem was chunked transport encoding use by phone software initially not supported by mentioned above server software. It was however easy enough to fix. - the next problem is failure of libsynthesis to get authentication information sent by phone extracted from messages. Also decoded incoming WBXML messages saved into XML format are corrupt. I found a bug which was causing this, and commited a fix in the luz branch (see https://gitorious.org/libsynthesis). The problem was that the XML encoder in the SyncML toolkit could not handle empty tags, but the SonyEricsson devInf contains a OEM/OEM empty tag. So now the XML conversion of the messages you captured from the Xperia-S (aka LT26i) now work ok. However, this does not explain the auth problems. For unknown reason the client tries to synchronize at first without authentication even when both user name and password are specified in its configuration. Authentication is first provided in second attempt of 4 totally. 3 of 4 incoming WBXML format packets seems to contain authentication information but also wbxml2xml does not show it I don't see any authentication in the incoming messages, not in the XML translation and also not when looking at hexdumps of the WBXML. The only thing I see is the LocName test, but that's not enough for an auth attempt, there must be some credentials. It is valid (or even recommended) for a device to try first without any credentials to see what auth method the server requests. So the first of these attempts is perfectly ok, and the server responds with an authentication challenge for MD5 auth, which is the secure method which does not require user/pw to be transmitted over the line, but only a hash. However, for some reason the LT26i does not respond to that challenge, but retries the probe again. One reason for this could be that the device does not actually get the response, but just some transport level error, and thus just resends it's first message serveral times. You said that there was some problem with chunked transport encoding - maybe there's a similar problem in the other direction such that the server result arrives garbled at the phone's end? [...] Last tests was done with current GIT version of libsynthesis from master branch. I recommend using the luz branch for testing - as it is always ahead of master and contains the latest patches before they are in master. Best Regards, Lukas ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] libsynthesis and SonyEricsson Xperia S own SyncML client
Hi Andris, On 24.05.2012, at 08:33, Andris Pavenis wrote: I don't see any authentication in the incoming messages, not in the XML translation and also not when looking at hexdumps of the WBXML. The only thing I see is theLocName test, but that's not enough for an auth attempt, there must be some credentials. It responds according with authentication to WireShark. Try using WireShark on capture file and decode port 9321 as HTTP. Yes, I see that, but I can't find the matching wbxml incoming message dump in the *.wbxml you sent. All those 4 don't have the Cred element, IMHO. It seems however that public ID //SYNCML//DTD SyncML 1.2//ENmlq\0031.2 instead of //SYNCML//DTD SyncML 1.2//EN also causes some problems. For example WireShark shows WBXML tag numbers instead of interpretting them as SyncML WBXML tags. That's wrong on SE's part, and is certainly the reason why Wireshark does not recognize it as SyncML, but apparently is not be a problem for the SyncML TK decoder. If it was a problem, it would not decode anything in that message. Best Regards, Lukas ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] [PATCH] Improve readability of SyncML TK error messages
Hi Andris, thanks for the patch. However, the SMLERRPRINTFX() macro, like all debug output macros across the entire library, is supposed to output a linefeed by itself. It also actually does so in my builds of libsynthesis, so if the LFs are missing in your output, it must be something along the path from SMLERRPRINTFX() to the actual output function. I think the problem is in the CONSOLEINFO_LIBC shortcut output mode Patrick added this February (13ff1e4149 (logging + Linux: enable console output)), where CONSOLEPRINTF() is mapped to fprintf(stderr,...) without adding a LF. In my (iOS) build this new mode is not used, but output ends in a plain puts(), which does add the LF. So my suggestion would be this: diff --git a/src/sysync/sysync_debug.h b/src/sysync/sysync_debug.h index 61d4cdd..68dce5e 100755 --- a/src/sysync/sysync_debug.h +++ b/src/sysync/sysync_debug.h @@ -110,7 +110,7 @@ TDebugLogger *getDbgLogger(void); // Because a lot of libs log to stderr, include a unique prefix. // Assumes that all printf format strings are plain strings. #define CONSOLEPUTS(m) CONSOLE_PRINTF_VARARGS(%s, (m)) -#define CONSOLE_PRINTF_VARARGS(_m, _args...) fprintf(stderr, SYSYNC _m, ##_args) +#define CONSOLE_PRINTF_VARARGS(_m, _args...) fprintf(stderr, SYSYNC _m \n, ##_args) #define CONSOLEPRINTF(m) CONSOLE_PRINTF_VARARGS m # else // CONSOLEINFO_LIBC #define CONSOLEPUTS(m) ConsolePuts(m) Can you please check if this solves the problem in your build? Lukas On 05.06.2012, at 14:41, Andris Pavenis wrote: Patch attached. Andris 0001-Improve-output-of-SyncML-TK-error-messages.patch___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] Funambol + refresh-from-server = 417
Hi Patrick,
Just changing getSyncStateAlertCode() call with the second argument set to
false does the same trick. Because that second parameter switches between
simplified (true) and standard conformant (false) client alert codes
already.
uInt16 alertCode = getSyncStateAlertCode(fServerAlerted,true);
Just because support for refresh-from sync modes was so bad in the early SyncML
days, this parameter remained a constant.
The true part needs to be replaced with a configuration option that
chooses between old (use slow sync) and new behavior (use refresh
from server).
That now applies to that second parameter of getSyncStateAlertCode()
Or has enough time passed that we no longer have to care
about servers which implement slow sync, but not refresh from server?
The benefit of SyncML in general and libsynthesis in particular is that it
works with real legacy systems. In fact, the more time passes, the more the
main reason for using libsynthesis will become smooth legacy support. So I
wouldn't want to switch standard behaviour here, and opt for a config flag.
Best Regards,
Lukas
On 06.06.2012, at 15:49, Patrick Ohly wrote:
Hello!
It seems that Funambol has implemented some kind of protection against
excessive slow syncs: I can do a slow sync once fine. Doing it again
shortly afterwards (say, a few seconds later, as in automated testing)
results in a 417 retry later status for the Alert command (in the
SyncML response message).
They have not responded to my question about this behavior, so I don't
have an official confirmation that my interpretation is correct.
This server behavior makes one (otherwise useful) feature of
libsynthesis a bit troublesome: when an app requests a refresh from
server (forceslow = 1, syncmode = 1 in the binfile client API), then
the engine will ask the server for a slow sync instead of telling it to
do a refresh-from-server. In other words, the Funambol server has to
assume the worst (a slow sync) and applies its throttling even though
the client only wants to do a relatively benign refresh from server.
I've checked that the Funambol server does many refresh from server
syncs without 417. For that I patched libsynthesis:
diff --git a/src/sysync/localengineds.cpp b/src/sysync/localengineds.cpp
index ce3d034..98f15dc 100644
--- a/src/sysync/localengineds.cpp
+++ b/src/sysync/localengineds.cpp
@@ -3481,6 +3481,11 @@ localstatus TLocalEngineDS::engGenerateClientSyncAlert(
));
// create appropriate initial alert command
uInt16 alertCode = getSyncStateAlertCode(fServerAlerted,true);
+ if (alertCode == 201
+ fSyncMode == smo_fromserver
+ true) {
+alertCode = 205;
+ }
// check for resume
if (fResuming) {
// check if what we resume is same as what we wanted to do
Does that patch look right in principle?
The true part needs to be replaced with a configuration option that
chooses between old (use slow sync) and new behavior (use refresh
from server). Or has enough time passed that we no longer have to care
about servers which implement slow sync, but not refresh from server?
Then we could switch to the new behavior unconditionally.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
[os-libsynthesis] Funambol and large SyncML messages?
Hi Patrick, occasionally, I get error reports from libsynthesis (iOS client) users which experience SyncML Error 511 (server error) with Funambol/OneMediaHub servers. Analyzing these, it always boils down to a big (99kBytes) client message that the server apparently cannot handle. As the server does not send a MaxMsgSize, libsynthesis makes the message as big as fits into it's local buffer which is 100k by default. My question - did you experience this large message size problem with Funambol as well? Or are you using a smaller message size than the default in SyncEvolution anyway? I'm just wondering if reducing the message size could reliably fix that Error 511 problem - and if so how small the message size should be. Best Regards, Lukas Lukas Zeller, plan44.ch l...@plan44.ch - www.plan44.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] one-way sync + sync tokens not updated
Hello Patrick, On 21.08.2012, at 09:28, Patrick Ohly patrick.o...@intel.com wrote: SyncEvolution's ActiveSync backend is the first one which uses the string tokens in StartDataRead() and EndDataWrite(). The backend always runs inside a binfile-based SyncML client. In one particular test I am using a one-way-from-server sync, initiated by the client, and noticed a problem: the new token created by the backend in EndDataWrite() was not passed to the StartDataRead() in the next sync. The backend cannot handle that, because the tokens come directly from ActiveSync, which only allows reusing old tokens in a few limited cases. In this case, the server rejected the obsolete token, causing an unexpected slow sync. [...] I added that debug output. It confirms that the keeping old sync token branch is taken. What is the rationale here? Is the goal perhaps that if the client switches back to a two-way sync, all changes since the last two-way sync get sent, despite the intermediate one-way sync? Exactly! Changes made on the updated-only side must not be forgotten, so the reference point for syncing BACK data is the last sync where data is sent to the remote (hence fPreviousTOREMOTESyncIdentifier). A update-from-remote is not a To-Remote-Sync, so the identifier is not updated. Those changes include the changes made on behalf of the server during the one-way-from-server sync. Is that filtered out? The binfile implementation does some filtering of changes done by sync vs real changes (chgl_modbysync flag), although I'm not 100% sure it would cover this case correctly; it was implemented for suspendresume which has a similar challenge (partial syncs bring in new data, but the token cannot be updated until the sync has completed). But that particular code you were looking at long predates binfile. The original idea was that it is better to reflect back too many changes to the server than loosing client data or getting out of sync at a switch from update device back to two way. OTOH, a user might decide to use an ActiveSync server as remote backup, in which case one-way syncing makes sense again. Would it be acceptable to always take the updating sync token branch above? Yes, I guess that could be added as an option. To make it clean, a application using that options should make sure a switch back to two way also forces a slow sync; otherwise the resulting state could be out of sync (changes left on one side alone indefinitely even after a seemingly complete and successful two-way sync). Best Regards, Lukas Zeller, plan44.ch l...@plan44.ch - www.plan44.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] PBAP: one-way sync modes
Hello Patrick, On 31.08.2012, at 15:49, Patrick Ohly patrick.o...@intel.com wrote: [...] let's focus on one-way syncing for PBAP. At the moment, the engine cannot know whether it is meant to mirror the data and thus it will leave the extra items on the server unchanged. I intend to add a mirror config option similar to readonly. If set, the engine will not only avoid sending changes to the client, it will also remove those extra local items. I've done it a bit differently: instead of adding a config option, there's now only a macro call. The full code is in the pbap branch. I also pushed the patches which use this feature into the pbap SyncEvolution branch. I'm attaching the libsynthesis patches for review. Lukas, does this make sense? Yes. I looked through the patches and everything looks ok to me. Can it get included upstream? Yes. As I thought it was about time to clean up and update master as well, I merged everything into my working branch and rebased it on luz_rebased, as a release candidate for updating master real soon now... If yes, then I would rebase onto your latest code first. I held back with that because I wanted to release 1.3 with minimal changes. luz_rebased is a candidate for master, you can take that, or create your own rebased version, whatever fits better. Once this has settled, I'll see how to update master. Best Regards, Lukas Zeller, plan44.ch l...@plan44.ch - www.plan44.ch ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] Def between iPhone and IOS simulator
Hi Zhoulei, On 13.12.2012, at 04:57, zhou lei zhou...@globalroam.com wrote: You may have a try on this. Find out the log file name is quite hard there. I try to send the file, but can only find the tmp/sysynclogs, and this is a file path, i have no permission to do things on it. If can compress the log file, and send it by mail on the app, thats great. I just could not find the log file name. And i found !--logfile$(logpath)/synclog.txt/logfile-- in file ios_syncclient_app_sample.xml, means i should set it my self? You can set it yourself, yes. But note that synclog.txt is not the logfile you want to see (that's only one line per sync). The extended debug logfiles are automatically named after date and time and session ID and put into where logpath points. If you can't access /tmp/sysynclogs, you can change the definition of the logpath config variable on line 9 of ios_syncclient_app_sample.xml For example, to change the log directory to the app's document folder (which you can access from you Mac/PC using for example DiskAid http://www.digidna.net/products/diskaid, free functionality sufficient), change line 9 to: configvar name=logpath value=$(sandbox)/Documents/ This will make the app write the full logs to the documents folder. Best Regards, Lukas ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] Sync mode extensions must be optional
Hello Patrick, On 11.03.2013, at 01:43, Patrick Ohly patrick.o...@intel.com wrote: The master branch of libsynthesis on freedesktop.org is now based on the latest gitorious master branch. Lukas, it contains two patches which you might want to include [...]: Thanks! I merged them into luz branch, and will update master eventually. BTW, did you tag 3.4.0.47? There is a comment about the version, but not the tag itself. I tagged it, but forgot to push the tag to gitorious, sorry for that. Done now. Best Regards, Lukas Zeller ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] temporary local ID + FinalizeLocalID
Hello Patrick, FinalizeLocalID came into existence as a workaround for the iOS client datastores, in particular the address book which did not generate proper IDs until a very expensive save all method was called. I tried to implement it as generically as possible, but avoiding any extra risk (and development time I didn't have then) beyond what was needed to fix that problem. Which was for binfile based clients only. For the same reason (less risk to break something) the FinalizeLocalID implementation must be safe against being called with a already finalized ID. The idea is that the calls to FinalizeLocalID can be placed wherever a final ID is required, without need to track if the same item will get finalized again later. So making the FinalizeLocalID mechanism work for non-binfile and server should be not much more than placing the call at a few more places. Probably, for the server it's only when processing Map items, and before persisting them at the end of the request. How about yet another approach: the store is allowed to return a new error code, LOCERR_AGAIN, for add/update/delete operations. The engine then remembers all operations with that return code and calls them again (with the exact same parameters and without freeing resources!) at a suitable time, for example the end of the current message. In the second call, the store must execute or finish the operation. The engine must not call an operation on an item for which it has already something pending. If that happens, it has to complete the first operation first. That way the store can keep a list of pending operations with the ItemID as key. Sounds like a good solution indeed. I like it better than late-handling return codes especially because it avoids any risk of breaking existing datastores (and possible unsafe assumptions in their implementations), because for those nothing would change outside *and* inside the engine. It's also somewhat similar to the finalisationscript mechanism, which also keeps items (or parts of them) in memory for using them again at the end of the session, usually to resolve cross references (parent/child tasks for example). Maybe some of the mechanisms can be re-used for LOCERR_AGAIN. So, from my side: just go ahead :-) Best Regards, Lukas On 03.06.2013, at 15:12, Patrick Ohly patrick.o...@intel.com wrote: Hello! I'm trying to batch local database adds together to increase performance in SyncEvolution (https://bugs.freedesktop.org/show_bug.cgi?id=52669 and https://bugs.freedesktop.org/show_bug.cgi?id=64838#c5). My hope was that I can postpone the actual modification and return a temporary local ID. Then later or in parallel, combine multiple items together in a batch add. Finally, the temporary ID was meant to be replaced by the final one in FinalizeLocalID. But I hit the first snag pretty early: FinalizeLocalID() doesn't seem to get called on the server side where I need this feature primarily (for PBAP, CalDAV, ActiveSync the server side is the one with the local database). dsFinalizeLocalID, the wrapper around the store's FinalizeLocalID, gets called in three functions: (gdb) b sysync::TBinfileImplDS::changeLogPostflight Breakpoint 2 at 0x106f956: file /home/pohly/syncevolution/libsynthesis/src/sysync/binfileimplds.cpp, line 462. (gdb) b sysync::TBinfileImplDS::SaveAdminData Breakpoint 3 at 0x10739fd: file /home/pohly/syncevolution/libsynthesis/src/sysync/binfileimplds.cpp, line 2389. (gdb) b sysync::TLocalEngineDS::engGenerateMapItems Breakpoint 4 at 0x109c79c: file /home/pohly/syncevolution/libsynthesis/src/sysync/localengineds.cpp, line 6635. None of these get called on the server side during a refresh-from-client, slow or normal two-way sync. That makes sense, map items are only relevant on the client, and the server is not using the binfile class. But shouldn't the server also call FinalizeLocalID somewhere? Where would be the right place for it? Regarding FinalizeLocalID() in the client, I see some conceptual issues with it, at least for what I am trying to achieve. My main concern is about error handling. FinalizeLocalID() can return an error code, but dsFinalizeLocalID does not check for errors. It merely checks for ID modified. So in case of an error, the local store itself has to remember that writing failed and do something. What that something could be is not clear. It could return an error in EndDataWrite(), in which case I would expect the engine to enforce a slow sync in the next sync session. I can't think of other, realistic options right now. I think a better solution would be to delay handling the result of a database write until the entire SyncML message has been processed. Then give the store a chance to finalize the ID of each item that was touched as part of that SyncML message. The store can use that opportunity to flush the pending operations and return error codes
Re: [os-libsynthesis] temporary local ID + FinalizeLocalID
Hello Patrick, On 10.06.2013, at 11:57, Patrick Ohly patrick.o...@intel.com wrote: Looking at it like that, the question becomes: how does the server decided that it still needs an answer from the client? Perhaps the problem is on the client side after all: in message msgID=2, the client client sent the Add command with a Final flag. In msgID=3, it acknowledges the server's SyncCmd with a Status, *without* a Final flag. The server therefore (in it's MessageEnded checks) continues with the session, whereas the client quits it. Should the client continue to set the Final flag as long as it is waiting for the server to finish? This is one of the things the specs did not nail and actual implementations varied. Some severely misbehaved when receiving extra final/ (more than one per phase), and others were tolerant. Things are complicated further because final/ is session scope, whereas the actual phase is datastore scope. This can lead to really strange overlaps when multiple datastores are involved. It'd didn't do that because the server didn't set it either. My feeling is that untangling all these state changes is going to be tricky. Undoubtedly. Untangling the code that hasn't become prettier over time would be one thing. However, the real problem is that there's no clear specs to even draw a state diagram in the first place. I'll probably try something else: if commands were delayed in a message which is marked Final, then force execution of the commands at the end of processing that message and issue their Status commands, just as syncing normally would. Sounds like a good compromise to me. The effect will be that commands only get delayed across messages if there will be more of them coming in anyway. The hope is that this will avoid or at least minimize state change issues. Note that the problem that I am running into also exists in the unmodified code: I expect similar issues when the server delays processing because of timing. Probably not, because the server can only delay processing at the very beginning of the sync, while it loads the sync set in the background. So far, this was the only (but well tested with very long lasting sync set loads) case of delaying command execution. Delaying execution after sync has actually started is a new case. Best Regards, Lukas Zeller ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] PHOTO + compare=never
Hello Patrick, On 04.07.2013, at 12:01, Patrick Ohly patrick.o...@intel.com wrote: I just noticed one aspect of the example configs that I wasn't aware of: syncclient_sample_config.xml: field name=PHOTO type=blob compare=never merge=fillempty/ [...] What was the rationale for using that mode for PHOTO? Is it for storages which store photo data after re-encoding it? Exactly. For the mobile platforms I wrote clients, I never had an address book which did not somehow mangle (re-encode, resize, etc.) the data. With such a storage, the comparison would yield a false field is different, causing unnecessary writes. Yes. But with a storage that stores the data as-is, comparing it is better (IMHO), because modified data actually gets stored. The downside is that comparisons become more expensive. That's another reason. Comparison causes the BLOB contents to be pulled from storage (server side), which can be quite expensive. Speaking of comparisons, with EDS this is slightly tricky. Inlined data gets replaced with a file reference, so what the comparison really needs to do is not comparing file:///tmp/testing/temp-testpim/data/evolution/addressbook/pim-manager-testsync-testcontacts-foo/photos/pas_id_51D53D180001_photo-file1.image%252Fjpeg against the binary data in the incoming item, but rather the content of that file. I probably need to write a comparescript for that, right? In that case compare=never may be the right thing to do again, but I am not sure. There's a compare=script for that (might not be in the docs because that was added later, if I correctly remember somehow related to SyncEvolution...) Won't it lead to the situation again where MERGEFIELDS() incorrectly not marks an item as changed even though it was? I haven't verified in the code right now, but I think that's what the script mode was supposed to avoid. Best Regards, Lukas ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] NumberOfChanges + binfileclient
Hello Patrick, your analysis is (as always :-) straight to the point - a circular dependency. I first thought maybe you could run the preflight earlier, but then I found the note in binfileimplds saying Don't call before types are ok (we need TSyncItems). That's the real problem, because checking for changes and calculating data CRCs, the preflight needs TSyncItems, and these in turn need to be related to a TSyncItemType and these are subject to negotiation during sync init. So I see no easy way to run the preflight earlier :-( Best Regards, Lukas On 21.01.2014, at 16:27, Patrick Ohly patrick.o...@intel.com wrote: I am wondering about NumberOfChanges information sent from a Synthesis client to a Synthesis server. Normally this is not really interesting, but in my case the client and server are both running locally and the user is actually controlling the server side, so having the client's information would be useful in that case. What I see is that the client side calls TBinfileImplDS::getNumberOfChanges() inside TLocalEngineDS::engClientStartOfSyncMessage(). If getNumberOfChanges() returned a valid number here, it would get added to the Sync header. But TBinfileImplDS does not have a value yet. The value gets computed only later, in TBinfileImplDS::changeLogPreflight(). The engine only calls TBinfileImplDS::getNumberOfChanges() once more for local progress events. Is there a chance to make the SyncML NumberOfChanges feature work for binfile clients? Any idea what change might be needed? To me it looks like a circular dependency: DB initialization doesn't start unless syncing started, and having the information available when syncing starts depends on initialization. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. signature.asc Description: Message signed with OpenPGP using GPGMail ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] How to supress a tag for a property
Hello Anuj, On 27.02.2014, at 13:20, anuj chauhan acd...@gmail.com wrote: I am trying to send a higher value for Max Occour for Note field, May I ask for the context of this? I've never seen vCards with more than one NOTE field in the wild, so I'm curious why you'd want a MaxOccur of 200? It might be easier to help when I understand what you are trying to archieve with your devInf modifications. while doing so my database gets currupted i.e the values in vcard get stored in jumbled sequence. Below is the code snippet and how the value are being stored while running sync-evolution on this Code for output Dev Inf sent from client And the client is Syncevolution? Property PropNameNOTE/PropName MaxSize1000/MaxSize MaxOccur200/MaxOccur /Property Property So how did you make Syncevolution show MaxOccur 200 here? Without seeing the libsynthesis config that leads to this MaxOccur it is difficult for me to analyze what might be going wrong when parsing the vCard below... Response of Server What Server? Also SyncEvolution or some third-party? Data BEGIN:VCARD VERSION:2.1 N:chauhan;anuj;pratap singh;anuj01;CHAUHAN FN:pratap singh anuj01 anuj ADR:HI ALL;; BDAY:1987-08-19 EMAIL;INTERNET:acd...@gmail.com.asdasdasdadd URL:helloall.anuj.com TITLE:Sr Softeare engineer ORG:ABC;AWS TEL;TYPE=VOICE:0011234568 NOTE:HI ALL PHOTO;ENCODING=BASE64: MDA0NjM3NDk4OQ== END:VCARD /Data Apart from the CRLFs which are not visible here (I assume this is due to copypaste into the mail) this looks ok so far. Data Dumped in Client Database BEGIN:VCARD N:chauhan;anuj;pratap singh;anuj01;CHAUHAN FN:pratap singh anuj01 anuj TEL:0011234568 ADR:HI ALL\;\;\;\;\;\; EMAIL;TYPE=INTERNET:acd...@gmail.com PHOTO;ENCODING=BASE64;ENCODING=B:MDA0NjM3NDk4OQ== VERSION:3.0 ORG:ABC\; TITLE:Sr Softeare engineer BDAY:19870819 URL:helloall.anuj.com NOTE:HI ALL NOTE:4 NOTE:BASE64 END:VCARD So you have an array field for NOTE here? When i change MaxOccur value to a lower value (1-10) syncevolution works fine.but on higher numbers or setting repeat=array ,values in dumped vcard(database) get currupted. Note that if you set repeat in position to a value, you must make sure the underlying field is an array or there are subsequent fields that can take the value of the repeated occurrences. Best Regards, Lukas signature.asc Description: Message signed with OpenPGP using GPGMail ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] issues with combination of TYPE
Hi Rajesh, On 12.03.2014, at 11:00, rajesh kumar pawar pawar.rajesh...@gmail.com wrote: I am working to customize the Syncevolution and am facing issues with combination of TYPE parameters. The problem I am facing is that the syncevolution is not producing FLAGS or the Bitmap values for the instances where there is a combination of TYPES like HOME,INTERNET etc. The idea between the bitmap and multimix mapping modes are that each TYPE is mapped to a bit. value field=EMAIL_FLAGS combine=, conversion=multimix enum name=WORK,INTERNET value=B0/ enum name=HOME,INTERNET value=B1/ enum name=INTERNET value=B2/ enum name=PREF value=B3/ /value The first two enums don't work because their name part already contains combined names. Just convert these to: value field=EMAIL_FLAGS combine=, conversion=multimix enum name=WORK value=B0/ enum name=HOME value=B1/ enum name=INTERNET value=B2/ enum name=PREF value=B3/ /value With this, you map WORK to bit 0, HOME to bit 1, INTERNET to bit 2 etc. So when an incoming vCard has TYPE=WORK,INTERNET (or INTERNET,WORK), you'll get EMAIL_FLAGS with bit 0 and bit 2 set (= 2^0 + 2^2 = 1 + 4 = 5 = 0b101) Similarily, for TYPE=WORK,PREF (or PREF,WORK) you'll get bit 0 and bit 3 set (=2^0 + 2^3 = 1 + 8 = 9 = 0b1001) This way, you can easily parse and generate any combination of the basic TYPEs. Best Regards, Lukas signature.asc Description: Message signed with OpenPGP using GPGMail ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] Varying Property types values representation in xml
Hi Sachin, please have a look at my answer to Rajesh (just a few minutes ago on the mailing list), this is closely related. Basically, HOME,VOICE is *not* a single enum value, but a list of two separate enums, HOME and VOICE. The bitmap and multimix mapping modes in libsynthesis support mapping such enums into a bitfield value. However in the devInf, you'll always have the basic enum values, not combinations. Best Regards, Lukas On 08.03.2014, at 17:10, Sachin Gupta ching...@gmail.com wrote: Hi Lukas, I am trying to modify the capabilities for a set of TEL values. Normally we have the representation as: Property PropNameTEL/PropName PropParam ParamNameTYPE/ParamName ValEnumCELL/ValEnum ValEnumVOICE/ValEnum ValEnumFAX/ValEnum ValEnumPAGER/ValEnum ValEnumOTHER/ValEnum ValEnumPREF/ValEnum /PropParam /Property In this case the represenation is quite easy as explained in the doc and the sample config. But how do i represent a format in the config xml for: Property PropNameTEL/PropName PropParam ParamNameTYPE/ParamName ValEnumCELL/ValEnum ValEnumHOME,VOICE/ValEnum ValEnumWORK,VOICE/ValEnum ValEnumWORK,FAX/ValEnum ValEnumHOME,FAX/ValEnum ValEnumPAGER/ValEnum ValEnumOTHER/ValEnum ValEnumPREF/ValEnum /PropParam /Property Also what will be the expected properties in the file based vcard for these highlighted ones? Regards Sachin signature.asc Description: Message signed with OpenPGP using GPGMail ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] vCard group tags
Hello Patrick, On 11.04.2014, at 12:18, Patrick Ohly patrick.o...@intel.com wrote: Google CardDAV and iOS/OS X use group tags to represent custom labels in vCards. Attached is an example. Note that the Google vCard format used by CardDAV is not the same as the one used by Gmail's export feature. The example is from CardDAV. I am trying to understand how I can support that using libsynthesis. The doc mentions: groupfield [...] Is there an example using this feature? I implemented that feature back at Synthesis in 2010 for Beat's Android client, and what is shown in the docs (the TITLE/ORG case) is in fact taken from that client's config. Is the following field list and profile correct? Field list: field name=TITLE array=yes type=string/ field name=ORG_NAME array=yes type=string/ field name=ORG_DIVISION array=yes type=string/ field name=ORG_OFFICE tarray=yes ype=string/ field name=ORG_TEAM array=yes type=string/ field name=ROLE array=yes type=string/ field name=GROUP-TAG-TITLE-ORG array=yes type=string/ Profile: property name=TITLE grouptag=GROUP-TAG-TITLE-ORG value field=TITLE/ /property property name=ORG values=4 grouptag=GROUP-TAG-TITLE-ORG value index=0 field=ORG_NAME/ value index=1 field=ORG_DIVISION/ value index=2 field=ORG_OFFICE/ value index=3 field=ORG_TEAM/ /property My understanding is that parsing the example from the documentation will lead to: ORG = [ myOwnCompany, myEmployer ] TITLE = [ boss, employee ] GROUP-TAG-TITLE-ORG = [ A, B ] Yes, that's excactly what is supposed to happen. The challenge will be to convert between item6.IMPP;X-SERVICE-TYPE=AIM:aim:aim item6.X-ABLabel:Other and X-AIM:aim Custom labels will also be fun, if that's what Evolution decides to use (currently it is unsupported). As always, vCard details are fun, and X-stuff even more so ;-) Best Regards, Lukas signature.asc Description: Message signed with OpenPGP using GPGMail ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] How to stop asking for server DevInf
Hi Anuj, On 25.04.2014, at 11:12, Anuj acd...@gmail.com wrote: While doing sync with server I observed that server is sending a dev inf and my evolution client is changing the behaviour according to servers dev inf for further sync sessions. I guess so, because that's the whole point in devInf - allow the sync process to adapt behaviour to the endpoint's capabilities. What benefit would you expect from suppressing that exchange of capabilities? How do I make my client stop asking for servers dev inf? There's no official way in libsynthesis to achieve that, because all of libsysnthesis is designed to make use of devInf as much as possible. Furthermore, even if you could suppress sending a Get command asking for devInf (see syncagent.cpp, lines 1244 ff), this does not necessarily mean the server does not send devInf by itself (using the Put command). Best Regards, Lukas signature.asc Description: Message signed with OpenPGP using GPGMail ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] [SyncEvolution] mapping between individual X- extensions and grouped extensions
Hi Emiliano, On 26.04.2014, at 12:31, Emiliano Heyns emiliano.he...@iris-advies.com wrote: Hello! Where do I find more on these scripts? This could be very interesting for me. The script language is documented in the libsynthesis config reference, see http://cgit.freedesktop.org/SyncEvolution/libsynthesis/tree/doc/SySync_config_reference.pdf To see what scripts are available and when they are run within a sync session, http://cgit.freedesktop.org/SyncEvolution/libsynthesis/tree/doc/SySync_script_call_flow.pdf is helpful. It documents the server side only, but the client flow of calls is not fundamentally different. Note that SyncEvolution already makes use of these scripts for a long time, it's nothing new. Lukas signature.asc Description: Message signed with OpenPGP using GPGMail ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] vCard group tags
Hi Patrick, you are faster in understanding the details than I could look them up :-) Yes, again I should have noticed, but I lack the daily sync practice... Every property that can occur multiple times, i.e. maps to a series of side-by-side fields or array fields, needs the position tag to specify how the multiple instances are to be filled into fields. This has nothing to do with groupfield directly, but of course groupfield usually does not make sense for non recurring properties. On 29.04.2014, at 17:48, Patrick Ohly patrick.o...@intel.com wrote: Looking further at aPropP-nameExts it seems that this what relates to position in the profile config. Exactly. (the reason why it is called nameExts internally is because in the early days, the primary usage was to direct TELs into one of multiple TEL fields based on WORK, HOME, FAX, so these worked as field name extensions. Like having fields TEL_WORK and TEL_HOME, and two positions that routed incoming TELs according to TYPE using the has and hasnot attributes) I did not have that for my X-ABLabel. Adding it seems to fix the problem. So it seems that groupfield can only be used reliably with a property which has a position, correct? Correct, see above. Best Regards, Lukas signature.asc Description: Message signed with OpenPGP using GPGMail ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] SyncML Server Performance using Syncevolution
Hi Sachin, if what you need is just a lot of pure SyncML clients to run against your server, you might consider using my former employer's free SyncML DEMO client (see http://www.synthesis.ch/dl_client.php?bp=CLIpv=vtag=13). Apparently it hasn't been touched since I left there 2010, and it's not opensource, but is based on the same SyncML engine as used in SyncEvolution, with the same XML config. It only has a simplistic TAB separated text file backend. But it is small, simple and can be started from the command line, including injecting parameters into the XML config (such as different user names, device ids) via the -D command line option. Other people have used this client for creating test loads, so that might work for you as well. However, that was long time ago, don't ask me about more details :-) Best Regards, Lukas On 29.04.2014, at 19:52, Sachin Gupta ching...@gmail.com wrote: one more thing. everytime i invoke syncevolution, it has to load all the xmls, build up its data structures, and all the other stuff. Can i implement threads in it and run it as a daemon. This will i can save time from doing all this. From within, can i then implement threads and launch multiple sync operations? Can this be done in Syncevo or you foresee issues in this ? Kindly suggest Regards On Mon, Apr 28, 2014 at 11:48 PM, Sachin Gupta ching...@gmail.com wrote: yes :) doing some profiling. Will update with the results. How do i maintain the load on the server; 2400 users for an hour? Can JMeter or locust help me in this or i will need custom built scripts for this? Any suggestions? Regards On Sun, Apr 27, 2014 at 12:27 AM, Patrick Ohly patrick.o...@intel.com wrote: On Sat, 2014-04-26 at 23:42 +0530, Sachin Gupta wrote: On Fri, Apr 25, 2014 at 3:29 PM, Patrick Ohly patrick.o...@intel.com wrote: On Fri, 2014-04-25 at 15:03 +0530, Sachin Gupta wrote: Can you suggest how i can test SyncML Server performance and have 2500 users/syncevolutions connecting simultaneously? There's no ready-made solution. You'll have to write your own scripts for configuring SyncEvolution and running the desired benchmark. Note that each context in SyncEvolution gets its own device ID. So if you want to simulate n different devices, use: syncevolution --configure ... client-1@client-1 ... syncevolution --configure ... client-n@client-n I figured so. So wrote scripts which will launch syncevo each with unique device ids and seperate user accounts. But the concern is managing these number of syncs through a time period. Exploring if JMETER can help me out in this.\ Also being a process, it would not be possible to launch so many processes in parallel. Memory and CPU would be issues, right? Would need very high end systems for this? I have not measured this. Try it and you'll see. My expection is that you will need multiple client machines, though. Would it be possible doing this launching processes or shall i look into creating threads within the Syncevolution launching and controlling sessions from there? I would just use multiple client machines. Much simpler and scales perfectly. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. signature.asc Description: Message signed with OpenPGP using GPGMail ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] vCard group tags
Hello Patrick,
On 02.05.2014, at 10:38, Patrick Ohly patrick.o...@intel.com wrote:
What do you mean by independent properties?
For example, ADR and TEL are independent in the sense that their values
are stored in different field arrays. Sharing the LABEL field for the
new parameter creates a dependency (or risk of collision) that did not
exist before.
But how can this work at all? The index in the field arrays is what relates the
components (e.g. TEL, TEL_FLAGS and LABEL) to each other.
This would mean all of the field arrays would need to have rows for the *sum*
of all the TEL+ADR+EMAIL+URL+xxx, with most rows empty, to be able to share a
single LABEL array.
Now I see why you want to check the availability of LABEL row even if assigned
via a property parameter.
Hmmm - indeed this might work, at the expense of a lot of empty elements in the
actual property field arrays. I'm just not sure right now if the generator part
of MimeProfile is prepared for properly skipping entirely empty rows to avoid
generating empty property lines in the vCard output. But that would be fixable
as well.
[...]
It's simpler.
TEL;X-ABLabel=labelTel:123456789
ADR;X-ABLabel=labelAdr:xxx
stores only the second label at position #0
= LABEL = [ 'labelAdr' ]
Understood now. Yes, because the only thing TEL and ADR have in common is the
LABEL, and this is not checked.
That's because the code which checks where to store the ADR value does
not check whether the LABEL at the position is available.
There's another problem:
TEL:123456789
ADR;X-ABLabel=labelAdr:xxx
A simplistic check LABEL[0] empty will accept position #0 for ADR. But
doing so will add the labelAdr also to the TEL, which was previously
created at position #0.
That means that the check has to be LABEL[0] does not exist.
Yes. Fortunately, TItemField differentiates between empty and unassigned; the
check needs to be for the latter.
Similar to for (sInt16 e=0; eaPropP-numValues; e++) { the code would
also need to iterate over aPropP-parameterDefs.
Yes.
This helped, but so far I have only implemented the too simplistic
check.
I noticed another problem with the use X-ABLabel parameter approach:
storing complex strings (spaces, quotation marks) in a parameter value
is harder.
That's probably why Apple chose the X-ABLabel property approach. An unparseable
parameter could ruin the real data, a unknown property is less dangerous.
The EDS vCard parser gets it wrong and fails to parse:
X-ABRELATEDNAMES;X-ABLabel=domestic partner:domestic partner
That is valid according to http://tools.ietf.org/html/rfc2425#page-5
because the space is a SAFE-CHAR and thus may appear in a ptext, but the
EDS parser does not expect the space. To work around this, we could
voluntarily quote the string even though it is not required.
Now, the conceptual problem with X-ABLabel parameter is that a quoted
string cannot contain double quotes. It is probably rare that a user
enters double quotes as part of a label, but it cannot be ruled out
either. Line breaks are also only allowed in property values and not
parameter values.
I'm undecided now how to proceed. Simplify X-ABLabel property values so
that they can be stored as parameter? Use the more complex X-ABLabel
property and grouping? Bummer.
Really hard to say :-(
I'd say it depends on how important these ABLabels are (outside of Apple
devices, that is. In iOS's Addressbook (AB), the Label IS important and
actually represents the standard TYPE of a TEL, EMAIL or ADR as well, by using
cryptic internal string constants like _$!Work!$_ or _$!HomeFAX!$_).
Best Regards,
Lukas
signature.asc
Description: Message signed with OpenPGP using GPGMail
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] comparison of PHOTO data
Hello Patrick,
On 11 Jul 2014, at 15:27, Patrick Ohly patrick.o...@intel.com wrote:
I'm currently investigating why a comparison of two PHOTO fields of
different length returns field equal. PHOTO is defined as:
field name=PHOTO type=string compare=conflict merge=fillempty/
My first question would be why PHOTO was defined string here, not blob?
I think the specific situation is that the two values contain a null
byte somewhere in the middle, and the part before that is equal.
I think the following code does the comparison, doesn't it?
sInt16 TStringField::compareWith(TItemField aItemField, bool
aCaseInsensitive)
{
sInt16 result;
PULLFROMPROXY;
if (aItemField.isBasedOn(fty_string)) {
TStringField *sfP = static_castTStringField *(aItemField);
#ifdef STREAMFIELD_SUPPORT
sfP-pullFromProxy(); // make sure we have all chars
#endif
// direct compare possible, return strcmp
if (aCaseInsensitive)
result=strucmp(fString.c_str(),sfP-fString.c_str());
else
result=strcmp(fString.c_str(),sfP-fString.c_str());
}
...
Yes, but this was written for text strings. I admit it is very c-ish style, but
correct enough for text strings.
IMHO The real bug is that TBlobField does not override compareWith.
TBlobField's compareWith could ignore aCaseInsensitive and use memcmp, for the
case of actually comparing two blobs, and fall back to TStringField for
comparison with other types.
We have std::string as value and therefore can store null bytes as part
of the data, but the actual comparison falls back to C-string
operations, which only work for null bytes at the end.
There's to much C strings throughout the entire project, so for text, C string
semantics are assumed pretty much everywhere.
TBlobField however makes use of the true binary byte string capability of
std::string exactly this way.
I think the strcmp() needs to be replaced with something that also looks
at the rest of the string if no difference is found. Agreed?
A fully std::string compatible comparison would be nicer here, and fix the
problem.
Or do we need a real binary type?
We do have it :-) It's called BLOB :-)
Using string type for PHOTO also has
the risk that during a merge operation, the two values will get
concatenated (thus breaking the images) unless some custom merge script
resolves the conflict differently first.
Probably others, because TStringField was certainly never designed to carry
binary data, altough it can do it to some extent.
Best Regards,
Lukas
signature.asc
Description: Message signed with OpenPGP using GPGMail
___
os-libsynthesis mailing list
os-libsynthesis@synthesis.ch
http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] comparison of PHOTO data
Hello Patrick, the patch is fine, anyway! Regarding differences between blob and string when it comes to compare On 14 Jul 2014, at 20:10, Patrick Ohly patrick.o...@intel.com wrote: My first question would be why PHOTO was defined string here, not blob? I changed that in this commit: [...] Unfortunately I did not write down why I changed the type, instead of merely changing the compare setting. Does it matter at all? Not any more today, I'd guess. But much of that code is dated from 2003, when we had way slower environments and at the same time we were working in a project for using SyncML for E-Mail sync, where attachment blobs could get multi-megabytes. I don't remember the details, but it was all about trying to avoid touching those fat attachment blobs and cause them expensively pulled from disk until absolutely needed. I think there is is code somewhere in the script engine that deliberately restricts comparing blobs for that reason. But as said, with contact photos and todays infrastructure, these considerations are slightly outdated today ;-) Palm Pilots with 4Meg storage total (disk+ram) are no longer the target... Best Regards, Lukas signature.asc Description: Message signed with OpenPGP using GPGMail ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] aborting a sync - slow sync
Hi Patrick, On 24.09.2014, at 17:33, Patrick Ohly patrick.o...@intel.com wrote: Let me pick your brain for a second... :-) It has gotten a bit rusty as far as sync is concerned. But I'll try :-) When SyncEvolution runs a local sync (i.e. both client and server are provided by SyncEvolution) and the sync gets aborted (= STEPCMD_ABORT is passed to SessionStep()), I noticed that the next sync starts again as if nothing happened. For example, aborting a normal two-way sync and then syncing again does another normal sync using the same sync anchors because the datastores's SaveAdminData has never been called during the aborted sync. What I *remember* (=not verified by looking at the code recently) is that the engine tries to find out if anything has happened already when the sync is aborted. If so, it would save state. Otherwise, it would just stop and next sync would start as if nothing happened (because it thinks in fact nothing of importance *has* happened, and avoid the relatively expensive process of saving state). The result is that the server can end up adding the same items twice, once during the aborted sync and again during the next one. This causes duplicates, because the client side has no means of detecting the duplicate. Did you actually see such problems happeing or do you just anticipate them because they *would* occur if the engine *did* abort without saving state in all cases? Now I could not resist looking at the code - two of these early abort checks are in TSyncAgent::NextMessage(), look for happened in the comments, but these only apply to early suspends, not abort. I guess another relevant check is in TLocalEngineDS::engAbortDataStoreSync(), look for preventResuming. I don't have the complete picture however from just this quick look up... How is this case supposed to be handled? Is SyncEvolution doing something wrong in its change or meta data handling? The only thing I could imagine going wrong would be not to continue calling SessionStep() after the STEPCMD_ABORT until SessionStep() actually signals session done, because it might take another step to completely clean up the session, which is needed to make all datastores call saveAdminData. Best Regards, Lukas signature.asc Description: Message signed with OpenPGP using GPGMail ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
Re: [os-libsynthesis] aborting a sync - slow sync
On 25.09.2014, at 08:43, Patrick Ohly patrick.o...@intel.com wrote: [...] Would this save state include resetting the sync anchor? No. I can't think of any other way how the server could avoid duplicates, because it cannot know whether the client has added the items and if so, under which ID... except that there is also suspend/resume. Even without suspend/resume, the client should remember unsent map commands and send these at the beginning of the next session (early maps) in all cases. With these, the server will know of items actually added vs those only sent and not added. AFAIK this early map even predates SyncML 1.2, but few implementations actually used it. It's different on the client side because the client can detect the duplicate item based on the server's ID. Yes. Did you actually see such problems happeing or do you just anticipate them because they *would* occur if the engine *did* abort without saving state in all cases? I've seen it happen in the the case that I described (message sent with several Adds inside, server aborts before receiving a response). Ok, so it does not work as *intended* :-/ So the expected result is that a suspend state needs to be written on both sides and then the next sync must resume? With SyncML 1.2 and both sides capable (DB-wise) of SuspendResume, yes. The only thing I could imagine going wrong would be not to continue calling SessionStep() after the STEPCMD_ABORT until SessionStep() actually signals session done, because it might take another step to completely clean up the session, which is needed to make all datastores call saveAdminData. I need to check this on both server and client side. I suspect that SyncEvolution is too aggressive on the client side and doesn't let it finish the session normally. If STEPCMD_ABORT leads to suspend state being written and resuming the sync in the next session, why is STEPCMD_SUSPEND needed? My understanding is that STEPCMD_SUSPEND will take longer because it relies on further SyncML message exchanges. Does this perhaps make it more reliable? STEPCMD_SUSPEND causes the client to send a suspend alert (explicit suspend), whereas abort just saves the state and hopes for the other side to detect the abort and save its state as well (implicit suspend). So STEPCMD_SUSPEND is the more graceful way to stop a sync. In my mobile clients the stop button issues a STEPCMD_SUSPEND when pressed the first time, and a STEPCMD_ABORT when pressed the second time (e.g. for when network is stalled and the suspend message does not get through). Best Regards, Lukas signature.asc Description: Message signed with OpenPGP using GPGMail ___ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis
