Re: [Evolution-hackers] cleaning up the timezone handling mess
is anyone else getting duplicates of this message every few days? This has been going on for months and is a bit annoying. Is the mailing-list server looping or what? Jeff On Thu, 2008-04-03 at 19:45 +0200, Patrick Ohly wrote: Hello, did the slightly inflammatory subject catch your attention? Good, please keep reading... ;-) Details can be found in multiple Bugzilla entries, the most important one apparently being this one: http://bugzilla.gnome.org/show_bug.cgi?id=301363 Let me summarize the current situation: since 2.12, Evolution uses the host's timezone information. This is only a partial solution. For events created by other programs the incomplete and sometimes obsolete VTIMEZONE information is still used. As an example, take the two attached meeting invitations. The 2006 one is a real stripped down invitation from Outlook, created in 2006 using the old DST rules. The newer 2008 one uses the current rules. When importing 2006 into a local calendar file, the GMT -0600 (Standard) / GMT -0500 (Daylight) timezone definition is added to the calendar. When importing 2008, the timezone definition is not updated. This has the effect that the new meeting is not displayed correctly in the calendar. But replacing the timezone definition would not be correct either, because it would break the mapping of past events of the old event before the DST change in 2007. This is a conceptual problem of how timezone definitions are handled: they should be attached to events, not to calendars. The Itip formatter plugin demonstrates that: it uses the timezone definition included in the meeting invitation and correctly calculates the local times for both events. One could argue that the sending program is at fault: it could have used a different TZID after changing the timezone definition. Alternatively it could have sent a more complex VTIMEZONE which also includes the historic rules. I believe both causes other problems in practice. Speculating about it is futile anyway and won't change the meeting invitations that Evolution has to deal with. Besides, don't blame Outlook too much: Evolution = 1.12 now does exactly the same thing. For example, a meeting invitation now uses /softwarestudio.org/Tzfile/America/Los_Angeles and only includes the current rules for daylight saving transitions. The 2006 event demonstrates another problem: the recurrences during those weeks in 2007 and 2008 where old and new rules differ are not calculated correctly. A user of SyncEvolution and ScheduleWorld reported timezone issues here: http://www.scheduleworld.com/jforum/posts/list/1543.page In the ensuing discussion Mark Swanson from ScheduleWorld suggested that all clients should use the same TZIDs to reference a complete local Olsen database. I don't think that this is doable in all cases, but I think Evolution should at least try it. Without further ado, here's my proposal how importing events into Evolution should work. If you agree, then I can try to create a patch. For each TZID used in an event, Evolution should try to find the corresponding system timezone via a fuzzy search. If the TZID already follows the pseudo-standard set by the Olsen database, then the comparison could be based on the local part, e.g. America/Los_Angeles. This solves the problem of importing events generated by ScheduleWorld (which currently uses a /scheduleworld.com/revision date/ prefix and thus is not mapped to system time zones). If the TZID is in some other format (like the ones used by Outlook), then a hard-coded table could do the mapping. This would solve the problem with the 2006 example. This step is kind of ugly and optional: I myself would argue that for such recurring meetings the sender is responsible for sending an update with the current VTIMEZONE. In my experience this really happened in many cases beginning of 2007. If a match is found, then the event is stored with the original TZIDs replaced by the matching system ones. The VTIMEZONE is discarded. If no match is found, then the custom VTIMEZONE definition must be stored in the calendar. I don't think that the EDS infrastructure should be changed. It would break APIs and be a lot of work to implement, plus it would store multiple redundant copies of identical VTIMEZONE definitions, leading to worse performance. What I suggest instead is the following scheme: check whether the calendar already has a VTIMEZONE with a given TZID. If one is found, check whether the timezone definition is really identical. If it is, all is well. If not, then rename the new timezone by attaching number. Repeat the check and renaming until a 100% match is found or the timezone can be stored without colliding with an existing timezone. Then proceed with importing the event with renamed TZIDs. I believe that this could be implemented without changing anything in libecal. I could implement it as part of
Re: [Evolution-hackers] cleaning up the timezone handling mess
On Tue, 2008-07-01 at 14:12 +, Jeffrey Stedfast wrote: is anyone else getting duplicates of this message every few days? This has been going on for months and is a bit annoying. Not me. Maybe it's your local mail server? Use C-u in Evo (or select View - Message Source) and look at the Received: headers. You should be able to determine which server is resending the message by looking at the timestamps. ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] cleaning up the timezone handling mess
On Mi, 2008-04-16 at 13:59 +0530, chenthill palanisamy wrote: I'll create a new bug report. Should we move the discussion away from this list? I think if we discuss this on IRC, we can conclude on the design faster. It would be better to do the patch reviews in bugzilla. Chenthill and I tried to meet on IRC, but despite his generous offer to meet outside his working hours we haven't found a slot yet. As I wanted to get something working before leaving for a two-week business trip where I cannot do anything on the code itself, I completed the code/patch as I had suggested earlier. It's still incomplete (no Outlook TZID table, memory handling of ical strings missing), but it solves the problem of importing conflicting time zone definitions already and maps TZIDs which contain a location at the end. As suggested by Chenthill I have moved the discussion to a new issue in the tracker. Please watch it and reply there if you are interested: http://bugzilla.gnome.org/show_bug.cgi?id=528902 time zone handling insufficient -- Bye, Patrick Ohly -- [EMAIL PROTECTED] http://www.estamos.de/ ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] cleaning up the timezone handling mess
On Tue, 2008-04-15 at 18:07 +0200, Patrick Ohly wrote: On Tue, 2008-04-15 at 13:59 +0530, chenthill wrote: To preserve the timezone history, the best way would be to store the old rules in VTIMEZONES which should be done in libical. If I understand you correctly, you are suggesting to merge a VTIMEZONE with TZID=FOO that comes with a new meeting invitation with a previously stored VTIMEZONE with the same TZID, so that the merged VTIMEZONE has the rules from both the original VTIMEZONE. The problem is that incoming VTIMEZONE definitions from Exchange do not contain information to which year the rules apply. In the examples that I quoted, both use DTSTART:16010101T02 and thus are mutually exclusive. Nope I don't mean merging here. The VTIMEZONE from libical currently do not provide the history of timezone changes (older + current rrules) for the system timezones. Libical should be modified so that the VTIMEZONE's generated stores the timezone history. The Tzfile would have the history stored on the system, libical just does not store it in VTIMEZONE. Say if a meeting is received from exchange server with some Tzid for America/New_York. As we discussed below, this needs to be mapped to system timezone and libical should provide the VTIMEZONE which has the timezone history. This way, we need not use the VTIMEZONE sent by exchange and still display the older/current events properly. I do not recommend having another set of tzid's with versions in it for maintaining the old rules. This would trigger the comparison of rules between different versions of timezones. I don't get this part. Can you elaborate what you mean? Are you saying that storing a VTIMEZONE with TZID=FOO as TZID=FOO 2 when it conflicts with an existing VTIMEZONE should be avoided? yes, if libical is modified to return VTIMEZONE with the history and once mapping between foreign timezones to system timezones is done at the backend, this is would not be required. All the older events would be properly displayed. Once the outlook timezones are mapped with libical ones, the libical timezones which would have the old rules can be used. Mapping foreign TZIDs to system timezones always should be attempted first. Storing a VTIMEZONE under a different name is only the fallback. [VTIMEZONE and VEVENT are added separately] I don't think it works like that. iirc the whole VCALENDAR (used as top-level component in itip-formatter) which has events and timezone gets passed to the backend, the backend adds the timezone and events to the cache, notifies the UI. After looking at the Evolution source code I already came to the same conclusion. However, clients like SyncEvolution do add VTIMEZONE and VEVENT separately. That is necessary because the UID of each new VEVENT is required immediately, which is not possible (or at least very awkward) via ecal_receive_objects(). I hope SyncEvolution is the only backend which does it in this way. SyncEvolution is not a backend. It is a client program using the synchronous libecal API. Ah ok. I misunderstood that it has a corresponding external backend as well. I believe OpenSync works the same way, although I haven't looked at it during the last two years. I am not sure if we would need a libecal API for this. Is it not possible to handle it inside e_cal_backend_add_timezone? No, because the call cannot return the information to which TZID the timezone was mapped. Modifying the icaltimezone *zone would be an API change. The tzid need not be changed for the VEVENT. The following would work fine even if VTIMEZONE and VEVENT are handled separately, e_cal_backend_add_timezone - Add the timezone to backend's cache if the timezone cannot be matched with the system timezone else do not add. e_cal_backend_get_timezone - use the mapping and provide proper system timezone or if its unable to map, it needs to get the timezone from the cache. The clients can now use the corresponding libecal API's to get the right timezone. So, I feel this can be done commonly to all backends at EDS. Yeah you can get the code after two weeks, not a problem. Please file a bug on this since the old bug is about, outdated evolution timezones which is obsolete now as we pickup the timezones from the system. The bug also has a too many comments already. Probably we can also schedule a meeting on irc and have a discussion on this. Please let me know the time which would be suitable for you. I'll create a new bug report. Should we move the discussion away from this list? I think if we discuss this on IRC, we can conclude on the design faster. It would be better to do the patch reviews in bugzilla. Usually we could meet on IRC during the evening CEST, but there's not much time this week. I may not be available over next week. Let me know a suitable time during this week or after April 28th. - Chenthill.
Re: [Evolution-hackers] cleaning up the timezone handling mess
On Wed, 2008-04-16 at 13:59 +0530, chenthill palanisamy wrote: The VTIMEZONE from libical currently do not provide the history of timezone changes (older + current rrules) for the system timezones. Libical should be modified so that the VTIMEZONE's generated stores the timezone history. The Tzfile would have the history stored on the system, libical just does not store it in VTIMEZONE. Wow, in that case the situation is much worse than I thought. My impression was that the patch to use system timezone definitions would solve the problem of using out-dated, incomplete timezone definitions shipped with Evolution. The truth is that it only eases the maintenance pain regarding out-dated definitions, but doesn't address the timezones change over time problem at all, right? If this was a known limitation, why was it not already filed as bug a long time ago? Evolution users running into timezone problems were instead told to update to the latest Evolution release and update their system timezone definitions, which in many cases didn't help at all. Say if a meeting is received from exchange server with some Tzid for America/New_York. As we discussed below, this needs to be mapped to system timezone and libical should provide the VTIMEZONE which has the timezone history. This way, we need not use the VTIMEZONE sent by exchange and still display the older/current events properly. I agree that libical absolutely must use the complete timezone history stored on the system. This is independent from mapping foreign TZIDs to native ones; both is necessary, but can be implemented separately. Do you have an estimate how much work improvements for libical would be? Where are the places which have to be changed? How can we distinguish between extracting a VTIMEZONE which is to be sent to some other program and extracting a VTIMEZONE for internal use? Finally, one organizational question: should a patch be prepared against the libical in the GNOME or the SF repository? According to this post, the SF repository is trying to consolidate the various versions floating around: http://sourceforge.net/forum/forum.php?thread_id=1912252forum_id=51011 How does the Evolution team stand with regards to that effort? I do not recommend having another set of tzid's with versions in it for maintaining the old rules. This would trigger the comparison of rules between different versions of timezones. I don't get this part. Can you elaborate what you mean? Are you saying that storing a VTIMEZONE with TZID=FOO as TZID=FOO 2 when it conflicts with an existing VTIMEZONE should be avoided? yes, if libical is modified to return VTIMEZONE with the history and once mapping between foreign timezones to system timezones is done at the backend, this is would not be required. All the older events would be properly displayed. You assume that the mapping works in all cases. I don't think this is realistic. There will always be a program FOO somewhere, somewhen using a TZID=BAR which is unknown to Evolution and thus cannot be mapped. Even getting this right just for Outlook alone will be challenging and require permanent maintenance. I hope SyncEvolution is the only backend which does it in this way. SyncEvolution is not a backend. It is a client program using the synchronous libecal API. Ah ok. I misunderstood that it has a corresponding external backend as well. Have a look at http://www.estamos.de/projects/SyncML/index.html - calling a SyncML server that VEVENTs are exchanged with external backend kind of fits, but not quite. I am not sure if we would need a libecal API for this. Is it not possible to handle it inside e_cal_backend_add_timezone? No, because the call cannot return the information to which TZID the timezone was mapped. Modifying the icaltimezone *zone would be an API change. The tzid need not be changed for the VEVENT. The following would work fine even if VTIMEZONE and VEVENT are handled separately, e_cal_backend_add_timezone - Add the timezone to backend's cache if the timezone cannot be matched with the system timezone else do not add. e_cal_backend_get_timezone - use the mapping and provide proper system timezone or if its unable to map, it needs to get the timezone from the cache. The clients can now use the corresponding libecal API's to get the right timezone. So, I feel this can be done commonly to all backends at EDS. This fails to handle VTIMEZONE definition collisions in those cases where no mapping to system timezones is found. In that case the VTIMEZONE needs to be stored with a different TZID and the VEVENT needs to be patched. -- Bye, Patrick Ohly -- [EMAIL PROTECTED] http://www.estamos.de/ ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] cleaning up the timezone handling mess
On Wed, Apr 16, 2008 at 10:00 PM, Patrick Ohly [EMAIL PROTECTED] wrote: On Wed, 2008-04-16 at 13:59 +0530, chenthill palanisamy wrote: I don't get this part. Can you elaborate what you mean? Are you saying that storing a VTIMEZONE with TZID=FOO as TZID=FOO 2 when it conflicts with an existing VTIMEZONE should be avoided? yes, if libical is modified to return VTIMEZONE with the history and once mapping between foreign timezones to system timezones is done at the backend, this is would not be required. All the older events would be properly displayed. You assume that the mapping works in all cases. I don't think this is realistic. There will always be a program FOO somewhere, somewhen using a TZID=BAR which is unknown to Evolution and thus cannot be mapped. Even getting this right just for Outlook alone will be challenging and require permanent maintenance. Very true.. it was/is a serious PITA while I was figuring out the details for the MAPI provider. On the brighter side, Exchange/Outlook 2007 has got this sorted out to an extent. They now store the historical rules in the timezone blob. See [1]. (The MAPI provider does not yet makes use of these rules, it only identifies the timezone - maps it to one of the system timezones - then uses the system timezone information to generate the start-end times of the event.. its a todo on my list to make use of the stored information :-) ) And, like you have mentioned, the mapping needs constant maintenance despite publications like [2] or [3] :-( -Suman [1] http://blogs.msdn.com/stephen_griffin/archive/2006/12/06/outlook-2007-timezone-structures.aspx [2] http://msdn2.microsoft.com/en-us/library/ms912053.aspx [3] http://technet2.microsoft.com/WindowsVista/en/library/31f49a21-cfed-4b63-b420-58a9eabbb04e1033.mspx?mfr=true ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] cleaning up the timezone handling mess
On Fri, 2008-04-11 at 23:31 +0200, Patrick Ohly wrote: On Fri, 2008-04-11 at 00:55 -0600, P Chenthill wrote: [Exchange calendar backend patch] In cases of exchange the old VTIMEZONE and the new VTIMEZONE would have the same tzid, only the rules would be different. All events old and new will just use the latest timezone from the backend since they would have the same tzid in their start/end dates. And that would be wrong for the old events which are truly in the past. Consider the 2006 example that I sent: when viewing the year 2006 in the calendar, the system timezone definition is used and thus Evolution correctly uses the summer saving rules from 2006. Using the updated VTIMEZONE for events in that year would lead to a one hour shift during those weeks where old and new rules differ. It's arguably the lesser evil for recurring events because the current, still relevant recurrences would be displayed correctly, so there is some value in it. But for old events which are archived for reference purposes it would be just wrong. What I am suggesting would solve this problem by preserving both the old and the new VTIMEZONE. For recurring events the mapping to the more complex system timezone definitions helps. To preserve the timezone history, the best way would be to store the old rules in VTIMEZONES which should be done in libical. Since exchange does not like it, the older rules should be stripped off while sending the timezones to the exchange server at the backend and libical should facilitate this. I do not recommend having another set of tzid's with versions in it for maintaining the old rules. This would trigger the comparison of rules between different versions of timezones. Once the outlook timezones are mapped with libical ones, the libical timezones which would have the old rules can be used. [VTIMEZONE and VEVENT are added separately] I don't think it works like that. iirc the whole VCALENDAR (used as top-level component in itip-formatter) which has events and timezone gets passed to the backend, the backend adds the timezone and events to the cache, notifies the UI. After looking at the Evolution source code I already came to the same conclusion. However, clients like SyncEvolution do add VTIMEZONE and VEVENT separately. That is necessary because the UID of each new VEVENT is required immediately, which is not possible (or at least very awkward) via ecal_receive_objects(). I hope SyncEvolution is the only backend which does it in this way. I am not sure if we would need a libecal API for this. Is it not possible to handle it inside e_cal_backend_add_timezone ? I don't know much about how SyncEvolution handles it. While considering with all the other backend's in mind which are under EDS, exchange, I don't feel the need. This makes the new function a bit more compleyoutube.com/TuxSkyNetx, but I still think it is doable to come up with a solution which can be used by backends and clients. Attached is a proposal. In SyncEvolution I would lookup existing VTIMEZONE in the calendar via the e_cal_tzlookup_ecal() utility function, which then uses e_cal_get_timezone(). I checked the code of the file backend. The e_cal_check_timezone() call would fit right before the icalcomponent_merge_component() call and would use the other tzlookup function which checks against local VTIMEZONE components. All the other backends would have to be adapted in a similar way. I considered modifying the general e_cal_backend_file_receive_objects(), but decided against it because individual backends might have better ways of dealing with the problem. I also don't want to touch code which I cannot test. I will be out of town for three days. If there are no objections, then I'll try to get working code together when I come back, hopefully before I go on a business trip for two weeks the week after. Time, anyone got some time? Yeah you can get the code after two weeks, not a problem. Please file a bug on this since the old bug is about, outdated evolution timezones which is obsolete now as we pickup the timezones from the system. The bug also has a too many comments already. Probably we can also schedule a meeting on irc and have a discussion on this. Please let me know the time which would be suitable for you. thanks, Chenthill. ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] cleaning up the timezone handling mess
On Tue, 2008-04-15 at 13:59 +0530, chenthill wrote: To preserve the timezone history, the best way would be to store the old rules in VTIMEZONES which should be done in libical. If I understand you correctly, you are suggesting to merge a VTIMEZONE with TZID=FOO that comes with a new meeting invitation with a previously stored VTIMEZONE with the same TZID, so that the merged VTIMEZONE has the rules from both the original VTIMEZONE. The problem is that incoming VTIMEZONE definitions from Exchange do not contain information to which year the rules apply. In the examples that I quoted, both use DTSTART:16010101T02 and thus are mutually exclusive. I do not recommend having another set of tzid's with versions in it for maintaining the old rules. This would trigger the comparison of rules between different versions of timezones. I don't get this part. Can you elaborate what you mean? Are you saying that storing a VTIMEZONE with TZID=FOO as TZID=FOO 2 when it conflicts with an existing VTIMEZONE should be avoided? Once the outlook timezones are mapped with libical ones, the libical timezones which would have the old rules can be used. Mapping foreign TZIDs to system timezones always should be attempted first. Storing a VTIMEZONE under a different name is only the fallback. [VTIMEZONE and VEVENT are added separately] I don't think it works like that. iirc the whole VCALENDAR (used as top-level component in itip-formatter) which has events and timezone gets passed to the backend, the backend adds the timezone and events to the cache, notifies the UI. After looking at the Evolution source code I already came to the same conclusion. However, clients like SyncEvolution do add VTIMEZONE and VEVENT separately. That is necessary because the UID of each new VEVENT is required immediately, which is not possible (or at least very awkward) via ecal_receive_objects(). I hope SyncEvolution is the only backend which does it in this way. SyncEvolution is not a backend. It is a client program using the synchronous libecal API. I believe OpenSync works the same way, although I haven't looked at it during the last two years. I am not sure if we would need a libecal API for this. Is it not possible to handle it inside e_cal_backend_add_timezone? No, because the call cannot return the information to which TZID the timezone was mapped. Modifying the icaltimezone *zone would be an API change. Yeah you can get the code after two weeks, not a problem. Please file a bug on this since the old bug is about, outdated evolution timezones which is obsolete now as we pickup the timezones from the system. The bug also has a too many comments already. Probably we can also schedule a meeting on irc and have a discussion on this. Please let me know the time which would be suitable for you. I'll create a new bug report. Should we move the discussion away from this list? Usually we could meet on IRC during the evening CEST, but there's not much time this week. -- Bye, Patrick Ohly -- [EMAIL PROTECTED] http://www.estamos.de/ ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] cleaning up the timezone handling mess
Hello, attached a first revision of the new API call which implements the mapping of TZIDs to internal ones. I'm currently using (and testing) the code in SyncEvolution and it works already for some simple test cases. Some parts are marked as TODO; those will be filled in later. Right now I just want to illustrate the idea with a bit more code. Note also that I slightly rewrote the API documentation. There are likely memory leaks of libical strings. It is pretty hard to tell in advance which strings and entities need to be freed; IMHO the only reliable way to get this right is to have test cases and use valgrind. I'll do that before the final revision of the patch. Now, one specific question: how should I create a GError in the out-of-memory situation of e_cal_check_timezones()? I checked some of the other Evolution code to get an idea, but at least the few places I looked at didn't seem to check for out-of-memory. -- Bye, Patrick Ohly -- [EMAIL PROTECTED] http://www.estamos.de/ /* * Copyright (C) 2008 Patrick Ohly * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or * (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY, TITLE, NONINFRINGEMENT or FITNESS FOR A PARTICULAR * PURPOSE. See the GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA * 02111-1307 USA */ #ifndef ECAL_EXTENSION_H #define ECAL_EXTENSION_H #include libical/ical.h #include glib.h G_BEGIN_DECLS /** * This function cleans up VEVENT, VJOURNAL, VTODO and VTIMEZONE * items which are to be imported into Evolution. * * Using VTIMEZONE definitions is problematic because they cannot be * updated properly when timezone definitions change. They are also * incomplete (for compatibility reason only one set of rules for * summer saving changes can be included, even if different rules * apply in different years). This function looks for matches of the * used TZIDs against system timezones and replaces such TZIDs with * the corresponding system timezone. This works for TZIDs containing * a location (found via a fuzzy string search) and for Outlook TZIDs * (via a hard-coded lookup table). * * Some programs generate broken meeting invitations with TZID, but * without including the corresponding VTIMEZONE. Importing such * invitations unchanged causes problems later on (meeting displayed * incorrectly, e_cal_get_component_as_string() fails). The situation * where this occurred in the past (found by a SyncEvolution user) is * now handled via the location based mapping. * * If this mapping fails, this function also deals with VTIMEZONE * conflicts: such conflicts occur when the calendar already contains * an old VTIMEZONE definition with the same TZID, but different * summer saving rules. Replacing the VTIMEZONE potentially breaks * displaying of old events, whereas not replacing it breaks the new * events (the behavior in Evolution = 2.22.1). * * The way this problem is resolved is by renaming the new VTIMEZONE * definition until the TZID is unique. A running count is appended to * the TZID. All items referencing the renamed TZID are adapted * accordingly. * * @param comp a VCALENDAR containing a list of * VTIMEZONE and arbitrary other components, in * arbitrary order: these other components are * modified by this call * @param tzlookup a callback function which is called to retrieve * a calendar's VTIMEZONE definition; the returned * definition is freed by e_cal_check_timezones(); * NULL indicates that no such timezone exists * or an error occurred * @param customan arbitrary pointer which is passed through to * the tzlookup function * @retval erroran error description in case of a failure * @return TRUE if successful, FALSE otherwise. */ gboolean e_cal_check_timezones(icalcomponent *comp, icaltimezone *(*tzlookup)(const char *tzid, const void *custom, GError **error), const void *custom, GError **error); /** * An implementation of the tzlookup callback which clients * can use. Calls e_cal_get_timezone(). * * @param custom must be a valid ECal pointer */ icaltimezone *e_cal_tzlookup_ecal(const char *tzid, const void
Re: [Evolution-hackers] cleaning up the timezone handling mess
On Wed, 2008-04-09 at 00:42 +0200, Patrick Ohly wrote: On Mo, 2008-04-07 at 23:40 -0600, P Chenthill wrote: On Thu, 2008-04-03 at 19:45 +0200, Patrick Ohly wrote: One could argue that the sending program is at fault: it could have used a different TZID after changing the timezone definition. Alternatively it could have sent a more complex VTIMEZONE which also includes the historic rules. I believe both causes other problems in practice. Speculating about it is futile anyway and won't change the meeting invitations that Evolution has to deal with. Besides, don't blame Outlook too much: Evolution = 1.12 now does exactly the same thing. For example, a meeting invitation now uses /softwarestudio.org/Tzfile/America/Los_Angeles and only includes the current rules for daylight saving transitions. It just includes the current rules for outlook compatibility. Yep, that's the I believe both causes other problems in practice part ;-) I am just back after a small vacation. Ah, that explains the silence. I'll be away over the weekend and Monday myself and don't know whether I'll have time before I came back. I dont think anyone is work on this issue currently. I made a patch for exchange backend for solving this issue long time back. But unfortunately its not committed in trunk. Why? Technical issues that I should be aware of? I think it was postponed to commit after some freeze and missed later. I don remember the exact reasons. I am sure, its not due to any technical reasons :-) It uses a timestamp in the VTIMEZONE file to store the latest one always. By this way we would also know the last time period when the timezone has been changed for that location. I can't claim to understand the patch or the code that it applies to. When you say that only the latest VTIMEZONE is stored, that implies that only one is stored, right? How does that solve the problem of having an event in the calendar which depends on an old revision of the VTIMEZONE and another one which needs the current revision? The cache would have only one version of the timezone at any point since the tzid is used as a uid for the VTIMEZONE components for storing them. The X-EVOLUTION-TZ-TIMESTAMP would be used to ensure that only the latest VTIMEZONE is stored. Since the UI would pick the timezone from the backend, it would get the right timezone. In cases of exchange the old VTIMEZONE and the new VTIMEZONE would have the same tzid, only the rules would be different. All events old and new will just use the latest timezone from the backend since they would have the same tzid in their start/end dates. I am ok with handling using the sequence numbers. I am not sure whether a new libecal API would be needed to solve this, since the timezones would be handled at the backend. The client adds the VTIMEZONE and some time later the corresponding event. At that time the backend won't be able to correlate the two, but that is necessary to change the TZID in the event if the VTIMEZONE was stored with a different TZID. Only the client can do that. I don't think it works like that. iirc the whole VCALENDAR (used as top-level component in itip-formatter) which has events and timezone gets passed to the backend, the backend adds the timezone and events to the cache, notifies the UI. This is the same behavior while accepting and importing events. Another reason why this has to be handled in backend is, there can be cases when the user can accept events using some other client and receive the events through the respective backends. So handling it in the backend would be better. Atleast above is the behavior w.r.t exchange, and other backends under EDS. It would be better to handle all the timezones in a common place inside EDS. The reason for that is, same timezones can exist in multiple calendars which could not be matched to either system timezone or through the fuzzy logic. Currently it is handled separately for every calendar and the same information is stored redundantly. Another issue which should be taken care of is, the system timezones should not be stored in the cache. They should always be taken from the libical builtin timezone. Please consider these too while implementing the solution. That's way beyond what I have time for, sorry. Besides, I want to keep my changes as small as possible to increase the chance of getting them into the stable branch. - Chenthill. ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] cleaning up the timezone handling mess
On Fri, 2008-04-11 at 00:55 -0600, P Chenthill wrote: [Exchange calendar backend patch] In cases of exchange the old VTIMEZONE and the new VTIMEZONE would have the same tzid, only the rules would be different. All events old and new will just use the latest timezone from the backend since they would have the same tzid in their start/end dates. And that would be wrong for the old events which are truly in the past. Consider the 2006 example that I sent: when viewing the year 2006 in the calendar, the system timezone definition is used and thus Evolution correctly uses the summer saving rules from 2006. Using the updated VTIMEZONE for events in that year would lead to a one hour shift during those weeks where old and new rules differ. It's arguably the lesser evil for recurring events because the current, still relevant recurrences would be displayed correctly, so there is some value in it. But for old events which are archived for reference purposes it would be just wrong. What I am suggesting would solve this problem by preserving both the old and the new VTIMEZONE. For recurring events the mapping to the more complex system timezone definitions helps. [VTIMEZONE and VEVENT are added separately] I don't think it works like that. iirc the whole VCALENDAR (used as top-level component in itip-formatter) which has events and timezone gets passed to the backend, the backend adds the timezone and events to the cache, notifies the UI. After looking at the Evolution source code I already came to the same conclusion. However, clients like SyncEvolution do add VTIMEZONE and VEVENT separately. That is necessary because the UID of each new VEVENT is required immediately, which is not possible (or at least very awkward) via ecal_receive_objects(). This makes the new function a bit more complex, but I still think it is doable to come up with a solution which can be used by backends and clients. Attached is a proposal. In SyncEvolution I would lookup existing VTIMEZONE in the calendar via the e_cal_tzlookup_ecal() utility function, which then uses e_cal_get_timezone(). I checked the code of the file backend. The e_cal_check_timezone() call would fit right before the icalcomponent_merge_component() call and would use the other tzlookup function which checks against local VTIMEZONE components. All the other backends would have to be adapted in a similar way. I considered modifying the general e_cal_backend_file_receive_objects(), but decided against it because individual backends might have better ways of dealing with the problem. I also don't want to touch code which I cannot test. I will be out of town for three days. If there are no objections, then I'll try to get working code together when I come back, hopefully before I go on a business trip for two weeks the week after. Time, anyone got some time? -- Bye, Patrick Ohly -- [EMAIL PROTECTED] http://www.estamos.de/ /** * This function cleans up VEVENT, VJOURNAL, VTODO and VTIMEZONE * items which are to be imported into Evolution. * * It deals with VTIMEZONE conflicts which occur when the calendar * already contains an old VTIMEZONE definition with the same TZID, * but different summer saving rules. Replacing the VTIMEZONE * potentially breaks displaying of old events, whereas not replacing * it breaks the new events (the behavior in Evolution = 2.22.1). * * The way this problem is resolved is by renaming the new VTIMEZONE * definition until the TZID is unique. A running count is appended to * the TZID. All items referencing the renamed TZID are adapted * accordingly. * * Using VTIMEZONE definitions is problematic because they cannot be * updated properly when timezone definitions change and are * incomplete (for compatibility reason only one set of rules for * summer saving changes can be included, even if different rules * apply to different years). This function also looks for matches * of the used TZIDs against system timezones and replaces such TZIDs * with the corresponding system timezone. This works for TZIDs * following the Olson naming scheme by comparing the country/city * part and for Outlook TZIDs (via a hard-coded lookup table). * * @param comp either a VCALENDAR containing a list of * VTIMEZONE/VEVENT/VTODO/VJOURNAL items (in * arbitrary order) or such a list directly; * these are the items which are modified * @param tzlookup a callback function which is called to retrieve * a calendar's VTIMEZONE definition; the returned * string is freed by e_cal_check_timezones(); * NULL indicates that no such timezone exists * or an error occurred * @param customan arbitrary pointer which is passed through to * the tzlookup function * @retval erroran error description in case of a failure * @return TRUE if successful, FALSE
Re: [Evolution-hackers] cleaning up the timezone handling mess
On Mo, 2008-04-07 at 23:40 -0600, P Chenthill wrote: On Thu, 2008-04-03 at 19:45 +0200, Patrick Ohly wrote: One could argue that the sending program is at fault: it could have used a different TZID after changing the timezone definition. Alternatively it could have sent a more complex VTIMEZONE which also includes the historic rules. I believe both causes other problems in practice. Speculating about it is futile anyway and won't change the meeting invitations that Evolution has to deal with. Besides, don't blame Outlook too much: Evolution = 1.12 now does exactly the same thing. For example, a meeting invitation now uses /softwarestudio.org/Tzfile/America/Los_Angeles and only includes the current rules for daylight saving transitions. It just includes the current rules for outlook compatibility. Yep, that's the I believe both causes other problems in practice part ;-) I am just back after a small vacation. Ah, that explains the silence. I'll be away over the weekend and Monday myself and don't know whether I'll have time before I came back. I dont think anyone is work on this issue currently. I made a patch for exchange backend for solving this issue long time back. But unfortunately its not committed in trunk. Why? Technical issues that I should be aware of? It uses a timestamp in the VTIMEZONE file to store the latest one always. By this way we would also know the last time period when the timezone has been changed for that location. I can't claim to understand the patch or the code that it applies to. When you say that only the latest VTIMEZONE is stored, that implies that only one is stored, right? How does that solve the problem of having an event in the calendar which depends on an old revision of the VTIMEZONE and another one which needs the current revision? I am ok with handling using the sequence numbers. I am not sure whether a new libecal API would be needed to solve this, since the timezones would be handled at the backend. The client adds the VTIMEZONE and some time later the corresponding event. At that time the backend won't be able to correlate the two, but that is necessary to change the TZID in the event if the VTIMEZONE was stored with a different TZID. Only the client can do that. It would be better to handle all the timezones in a common place inside EDS. The reason for that is, same timezones can exist in multiple calendars which could not be matched to either system timezone or through the fuzzy logic. Currently it is handled separately for every calendar and the same information is stored redundantly. Another issue which should be taken care of is, the system timezones should not be stored in the cache. They should always be taken from the libical builtin timezone. Please consider these too while implementing the solution. That's way beyond what I have time for, sorry. Besides, I want to keep my changes as small as possible to increase the chance of getting them into the stable branch. -- Bye, Patrick Ohly -- [EMAIL PROTECTED] http://www.estamos.de/ ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] cleaning up the timezone handling mess
On Do, 2008-04-03 at 19:45 +0200, Patrick Ohly wrote: did the slightly inflammatory subject catch your attention? Hmm, it seems didn't, or the people who maintain the Evolution calendar are not reading this list/on vacation/busy. Should I rather lead this discussion in the bug tracker? -- Bye, Patrick Ohly -- [EMAIL PROTECTED] http://www.estamos.de/ ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] cleaning up the timezone handling mess
On Thu, 2008-04-03 at 19:45 +0200, Patrick Ohly wrote: Hello, did the slightly inflammatory subject catch your attention? Good, please keep reading... ;-) Details can be found in multiple Bugzilla entries, the most important one apparently being this one: http://bugzilla.gnome.org/show_bug.cgi?id=301363 Let me summarize the current situation: since 2.12, Evolution uses the host's timezone information. This is only a partial solution. For events created by other programs the incomplete and sometimes obsolete VTIMEZONE information is still used. As an example, take the two attached meeting invitations. The 2006 one is a real stripped down invitation from Outlook, created in 2006 using the old DST rules. The newer 2008 one uses the current rules. When importing 2006 into a local calendar file, the GMT -0600 (Standard) / GMT -0500 (Daylight) timezone definition is added to the calendar. When importing 2008, the timezone definition is not updated. This has the effect that the new meeting is not displayed correctly in the calendar. But replacing the timezone definition would not be correct either, because it would break the mapping of past events of the old event before the DST change in 2007. This is a conceptual problem of how timezone definitions are handled: they should be attached to events, not to calendars. The Itip formatter plugin demonstrates that: it uses the timezone definition included in the meeting invitation and correctly calculates the local times for both events. One could argue that the sending program is at fault: it could have used a different TZID after changing the timezone definition. Alternatively it could have sent a more complex VTIMEZONE which also includes the historic rules. I believe both causes other problems in practice. Speculating about it is futile anyway and won't change the meeting invitations that Evolution has to deal with. Besides, don't blame Outlook too much: Evolution = 1.12 now does exactly the same thing. For example, a meeting invitation now uses /softwarestudio.org/Tzfile/America/Los_Angeles and only includes the current rules for daylight saving transitions. It just includes the current rules for outlook compatibility. The 2006 event demonstrates another problem: the recurrences during those weeks in 2007 and 2008 where old and new rules differ are not calculated correctly. A user of SyncEvolution and ScheduleWorld reported timezone issues here: http://www.scheduleworld.com/jforum/posts/list/1543.page In the ensuing discussion Mark Swanson from ScheduleWorld suggested that all clients should use the same TZIDs to reference a complete local Olsen database. I don't think that this is doable in all cases, but I think Evolution should at least try it. Without further ado, here's my proposal how importing events into Evolution should work. If you agree, then I can try to create a patch. For each TZID used in an event, Evolution should try to find the corresponding system timezone via a fuzzy search. If the TZID already follows the pseudo-standard set by the Olsen database, then the comparison could be based on the local part, e.g. America/Los_Angeles. This solves the problem of importing events generated by ScheduleWorld (which currently uses a /scheduleworld.com/revision date/ prefix and thus is not mapped to system time zones). If the TZID is in some other format (like the ones used by Outlook), then a hard-coded table could do the mapping. This would solve the problem with the 2006 example. This step is kind of ugly and optional: I myself would argue that for such recurring meetings the sender is responsible for sending an update with the current VTIMEZONE. In my experience this really happened in many cases beginning of 2007. If a match is found, then the event is stored with the original TZIDs replaced by the matching system ones. The VTIMEZONE is discarded. If no match is found, then the custom VTIMEZONE definition must be stored in the calendar. I don't think that the EDS infrastructure should be changed. It would break APIs and be a lot of work to implement, plus it would store multiple redundant copies of identical VTIMEZONE definitions, leading to worse performance. What I suggest instead is the following scheme: check whether the calendar already has a VTIMEZONE with a given TZID. If one is found, check whether the timezone definition is really identical. If it is, all is well. If not, then rename the new timezone by attaching number. Repeat the check and renaming until a 100% match is found or the timezone can be stored without colliding with an existing timezone. Then proceed with importing the event with renamed TZIDs. I believe that this could be implemented without changing anything in libecal. I could implement it as part of SyncEvolution, but that won't solve the problem when Evolution itself imports events. A new API call in libecal would
Re: [Evolution-hackers] cleaning up the timezone handling mess
On Thu, 2008-04-03 at 19:45 +0200, Patrick Ohly wrote: did the slightly inflammatory subject catch your attention? Good, please keep reading... ;-) I will read your email as soon as I get a chance, but I have to add my voice to those saying please, please, PLEASE someone fix this complete and utter disaster! I was going to post a much more inflammatory rant about this than yours (although without any of your useful suggestions) today. Yesterday I had the excruciatingly embarrassing experience of walking into a conference room for an important meeting with important people at my company... about 3 minutes before everyone packed up their stuff to leave. I was an hour late!!! My Evo calendar said the meeting started at 3pm EDT, but everyone else's calendar said it started at 2pm. I went back and logged into Exchange OWA, and sure enough THAT calendar said 2pm as well. So, it's hard to understand how OWA knows the right date/time and Evolution, which uses data retrieved from OWA, doesn't. But, I've not really sat down to puzzle out how timezones are handled. It's really a brown-paper-bag situation that this should still be happening in Evolution well over a year after the timezone changes that caused the problems (in the US anyway) were approved. We run into this twice a year and it's still broken. I have many examples of Exchange meeting requests that do the wrong thing in Evo, if anyone needs them. -- - Paul D. Smith [EMAIL PROTECTED] http://make.mad-scientist.us Please remain calm--I may be mad, but I am a professional.--Mad Scientist ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
[Evolution-hackers] cleaning up the timezone handling mess
Hello, did the slightly inflammatory subject catch your attention? Good, please keep reading... ;-) Details can be found in multiple Bugzilla entries, the most important one apparently being this one: http://bugzilla.gnome.org/show_bug.cgi?id=301363 Let me summarize the current situation: since 2.12, Evolution uses the host's timezone information. This is only a partial solution. For events created by other programs the incomplete and sometimes obsolete VTIMEZONE information is still used. As an example, take the two attached meeting invitations. The 2006 one is a real stripped down invitation from Outlook, created in 2006 using the old DST rules. The newer 2008 one uses the current rules. When importing 2006 into a local calendar file, the GMT -0600 (Standard) / GMT -0500 (Daylight) timezone definition is added to the calendar. When importing 2008, the timezone definition is not updated. This has the effect that the new meeting is not displayed correctly in the calendar. But replacing the timezone definition would not be correct either, because it would break the mapping of past events of the old event before the DST change in 2007. This is a conceptual problem of how timezone definitions are handled: they should be attached to events, not to calendars. The Itip formatter plugin demonstrates that: it uses the timezone definition included in the meeting invitation and correctly calculates the local times for both events. One could argue that the sending program is at fault: it could have used a different TZID after changing the timezone definition. Alternatively it could have sent a more complex VTIMEZONE which also includes the historic rules. I believe both causes other problems in practice. Speculating about it is futile anyway and won't change the meeting invitations that Evolution has to deal with. Besides, don't blame Outlook too much: Evolution = 1.12 now does exactly the same thing. For example, a meeting invitation now uses /softwarestudio.org/Tzfile/America/Los_Angeles and only includes the current rules for daylight saving transitions. The 2006 event demonstrates another problem: the recurrences during those weeks in 2007 and 2008 where old and new rules differ are not calculated correctly. A user of SyncEvolution and ScheduleWorld reported timezone issues here: http://www.scheduleworld.com/jforum/posts/list/1543.page In the ensuing discussion Mark Swanson from ScheduleWorld suggested that all clients should use the same TZIDs to reference a complete local Olsen database. I don't think that this is doable in all cases, but I think Evolution should at least try it. Without further ado, here's my proposal how importing events into Evolution should work. If you agree, then I can try to create a patch. For each TZID used in an event, Evolution should try to find the corresponding system timezone via a fuzzy search. If the TZID already follows the pseudo-standard set by the Olsen database, then the comparison could be based on the local part, e.g. America/Los_Angeles. This solves the problem of importing events generated by ScheduleWorld (which currently uses a /scheduleworld.com/revision date/ prefix and thus is not mapped to system time zones). If the TZID is in some other format (like the ones used by Outlook), then a hard-coded table could do the mapping. This would solve the problem with the 2006 example. This step is kind of ugly and optional: I myself would argue that for such recurring meetings the sender is responsible for sending an update with the current VTIMEZONE. In my experience this really happened in many cases beginning of 2007. If a match is found, then the event is stored with the original TZIDs replaced by the matching system ones. The VTIMEZONE is discarded. If no match is found, then the custom VTIMEZONE definition must be stored in the calendar. I don't think that the EDS infrastructure should be changed. It would break APIs and be a lot of work to implement, plus it would store multiple redundant copies of identical VTIMEZONE definitions, leading to worse performance. What I suggest instead is the following scheme: check whether the calendar already has a VTIMEZONE with a given TZID. If one is found, check whether the timezone definition is really identical. If it is, all is well. If not, then rename the new timezone by attaching number. Repeat the check and renaming until a 100% match is found or the timezone can be stored without colliding with an existing timezone. Then proceed with importing the event with renamed TZIDs. I believe that this could be implemented without changing anything in libecal. I could implement it as part of SyncEvolution, but that won't solve the problem when Evolution itself imports events. A new API call in libecal would be a better solution. Before I proceed, let me ask: are there other plans to tackle the problem? Is someone working on it? Similar questions in #301363 were not answered. Do you agree with the outlined plan in
