Re: [Evolution-hackers] Camel Manifesto (update)
On Sun, 2011-03-13 at 21:32 +, David Woodhouse wrote: > Ug, and now I've found that that workaround doesn't work either. If we > try to rename a folder, we end up blocking in the main thread, waiting > for the soup request that we deliberately put into an idle function so > that it could run in the main thread... > > Thread 1 (Thread 0x77d93980 (LWP 9874)): > #0 pthread_cond_wait@@GLIBC_2.3.2 () at > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 > #1 0x0034ff00ee2d in e_flag_wait (flag=0x2002220) at e-flag.c:130 > #2 0x7fffe387e934 in ews_get_folder_info_sync (store=0x6e42a0 > [CamelEwsStore], top=0x1480880 "asd", flags=1, error=0x0) at > camel-ews-store.c:500 > #3 0x0035020508ae in camel_store_get_folder_info (store=0x6e42a0 > [CamelEwsStore], top=0x1480880 "asd", flags=1, error=0x0) at > camel-store.c:1122 > #4 0x003505460ce8 in folder_tree_cell_edited_cb (folder_tree= optimized out>, path_string=, new_name=0x1ca6460 "asd") > at em-folder-tree.c:624 > > As before, I would prefer to *fix* the broken locking, so that we can > call the soup functions from any thread. But if I'm still not allowed to > do that, what's the best way to fix this deadlock? We talked about this on IRC already, but for the record EMFolderTree is at fault here for calling a blocking Camel function. folder_tree_cell_edited_cb() instead needs to fetch folder info asynchronously and then do the actual folder rename asynchronously. I've already peppered the function with FIXME comments to this effect. I'll try to follow up on that this week before the code freeze. ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Camel Manifesto (update)
On Thu, 2011-02-17 at 18:37 +, David Woodhouse wrote: > I was told today that the GIO (and libsoup) async methods may not be > called from a thread other than the one running the mainloop. I found a > stupid race in libsoup¹ and filed a fix, but the bug was closed INVALID > because apparently it's not *supposed* to be thread-safe. > > That's a serious issue for using it from Evolution; it means we have to > jump through stupid hoops like using idle callbacks to call it from the > main thread, just as we do for gconf. Ug, and now I've found that that workaround doesn't work either. If we try to rename a folder, we end up blocking in the main thread, waiting for the soup request that we deliberately put into an idle function so that it could run in the main thread... Thread 1 (Thread 0x77d93980 (LWP 9874)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0034ff00ee2d in e_flag_wait (flag=0x2002220) at e-flag.c:130 #2 0x7fffe387e934 in ews_get_folder_info_sync (store=0x6e42a0 [CamelEwsStore], top=0x1480880 "asd", flags=1, error=0x0) at camel-ews-store.c:500 #3 0x0035020508ae in camel_store_get_folder_info (store=0x6e42a0 [CamelEwsStore], top=0x1480880 "asd", flags=1, error=0x0) at camel-store.c:1122 #4 0x003505460ce8 in folder_tree_cell_edited_cb (folder_tree=, path_string=, new_name=0x1ca6460 "asd") at em-folder-tree.c:624 As before, I would prefer to *fix* the broken locking, so that we can call the soup functions from any thread. But if I'm still not allowed to do that, what's the best way to fix this deadlock? -- dwmw2 ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Camel Manifesto (update)
On Thu, 2011-02-17 at 18:37 +, David Woodhouse wrote: > I assume that your intention is that the Camel async methods would *not* > be similarly broken, and that you should be able to call them from *any* > thread and expect them not to break? > > If so, we need be *very* careful about calling into gio. No, that's not my intention. My intention is that asynchronous functions should only ever be called from the main loop thread. ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Camel Manifesto (update)
On Mon, 2010-09-27 at 12:52 -0400, Matthew Barnes wrote: > 3. Asynchronous Methods > --- > > All the upgrades I've been making to Camel over the past year, and all > the changes described above were in preparation for this: Camel now has > an asynchronous public API. > > Let me immediately point out that there is ZERO IMPACT TO EXISTING CAMEL > PROVIDERS beyond the API changes I mentioned above. Camel providers > need not do anything special to support the new asynchronous methods. > As in GIO, all async methods default to simply calling the corresponding > synchronous method from a thread pool, which is all Evolution does. I was told today that the GIO (and libsoup) async methods may not be called from a thread other than the one running the mainloop. I found a stupid race in libsoup¹ and filed a fix, but the bug was closed INVALID because apparently it's not *supposed* to be thread-safe. That's a serious issue for using it from Evolution; it means we have to jump through stupid hoops like using idle callbacks to call it from the main thread, just as we do for gconf. Apparently, gio suffers this brain-damage too. I assume that your intention is that the Camel async methods would *not* be similarly broken, and that you should be able to call them from *any* thread and expect them not to break? If so, we need be *very* careful about calling into gio. I think perhaps we need to reset expectations about what a sane API looks like in the 21st century. We are all set to stop using gconf, print out its source code, and ritually urinate on it and then burn it. In part because it's not thread-safe and we have to do stupid things to ensure we call into it from the main thread. And yet people are still implementing new APIs with the *same* problem... -- dwmw2 ¹ https://bugzilla.gnome.org/show_bug.cgi?id=642573 ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Camel Manifesto (update)
On Mon, 2010-09-27 at 12:52 -0400, Matthew Barnes wrote: > Transient operations are now implicit: if you push a new status > message onto a non-empty message stack, the message is treated as > transient. A transient message just means there's a longer delay > before the message is shown in Evolution's status bar, since transient > operations tend to come and go quickly. If the transient operation > finishes before the timer expires, the transient message is never > shown at all. This helps filter out messages that would otherwise > flash by too quickly for users to read. Hi, I do not like this change. Its result is that I do not see which folder is getting updated and what percentage is done on it, which I consider as one of the most valuable information being on the status bar. I only see there "Checking for new mail" now, and I have no idea about actual progress of the operation behind it. Bye, Milan ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Camel Manifesto (update)
Here's another update on how the Camel API upgrades are coming along. My initial roadmap from 2009 is referenced in [1], and my last update was in May [2]. This next round of API changes will be fairly disruptive, so I'd like to coordinate with anyone finishing up their own Camel branch so my changes don't stomp on your work. I know Chen is finishing his Maildir branch, so unless I hear from someone else I'll plan to commit as soon as Chen's changes are in. The changes are substantial, but the result is pretty straight-forward and boring. Most of these changes were presented as future goals in my May update [2]. There's three major changes: 1. Redefine CamelOperation -- I'm removing the whole thread-private CamelOperation concept, along with the register/unregister functions. Thread-private CamelOperations seem convenient but actually have a detrimental effect on the public API and is one of the major problems I have with Camel: you can't tell from the API which functions block and which ones don't. As a result, Evolution is still rife with Camel calls that block the main loop and hang the UI. The forthcoming changes will finally solve this design flaw. As in GIO, all blocking functions will take a GCancellable pointer. Since CamelOperation is now a subclass of GCancellable, any GCancellable pointer in Camel may be a CamelOperation, a plain GCancellable, or NULL. I've made some further simplifications to CamelOperation both internally and externally. The internal message stack is simpler now, and as such camel_operation_start() and camel_operation_end() have been renamed to camel_operation_push_message() and camel_operation_pop_message(), which I think are more intuitive. Transient operations are now implicit: if you push a new status message onto a non-empty message stack, the message is treated as transient. A transient message just means there's a longer delay before the message is shown in Evolution's status bar, since transient operations tend to come and go quickly. If the transient operation finishes before the timer expires, the transient message is never shown at all. This helps filter out messages that would otherwise flash by too quickly for users to read. Finally, passing NULL to any CamelOperation function is now a no-op. 2. Rename Blocking Methods -- Almost[*] all blocking method names have grown a "_sync" suffix. This has a couple advantages: - It's pretty damn obvious now when you're making a blocking Camel call from Evolution or some other client program. - Blocking Camel calls are now pretty easy to grep for (camel_.*_sync). We can finally track down those blocking the main loop and file bugs. Methods already using the word "sync" I've expanded to "synchronize" so the name isn't too weird. So for example, camel_folder_sync() is now camel_folder_synchronize_sync(). [*] I didn't bother renaming CamelStream methods, since I plan to replace then with GIO streams in the near future and clients generally don't call stream methods directly anyway. 3. Asynchronous Methods --- All the upgrades I've been making to Camel over the past year, and all the changes described above were in preparation for this: Camel now has an asynchronous public API. Let me immediately point out that there is ZERO IMPACT TO EXISTING CAMEL PROVIDERS beyond the API changes I mentioned above. Camel providers need not do anything special to support the new asynchronous methods. As in GIO, all async methods default to simply calling the corresponding synchronous method from a thread pool, which is all Evolution does. The asynchronous methods will take the old blocking method names in order to emphasize to clients that this is now the recommended way to use Camel. (In case it wasn't already obvious, no I don't care about Camel's backward-compatibility right now.) Here's a before and after example to help make this all concrete. Before: gboolean camel_folder_append_message(CamelFolder *folder, CamelMimeMessage *message, const CamelMessageInfo *info, gchar **appended_uid, GError **error) After: void camel_folder_append_message(CamelFolder *folder, CamelMimeMessage *message, CamelMessageInfo *info, gint io_priority, GCancellable *cancellable, GAsyncReadyCallback callback, gpointer user_data) gboolean camel_folder_append_message_finish (CamelFolder *folder, GAsyncResult *result, gchar **appended_ui
Re: [Evolution-hackers] Camel Manifesto (update)
I wanted to give a status update on how this is progressing, since some of you may have noticed me breaking Camel's API with gleeful abandon since the 2.31 development cycle began. My initial roadmap is here: http://mail.gnome.org/archives/evolution-hackers/2009-November/msg00019.html What's Accomplished --- - Only may be included from outside of Camel's core. - The API is slowly being sealed up, with public instance data being moved to private structs, along with new accessor functions and in some cases GObject properties. I'm working on this incrementally. - CamelObject is now a GObject subclass, and Camel's old type system is completely gone. - The CamelArg API has been replaced with GObject properties. Persistent properties (whose values are saved to the object's binary state file) are implemented by extending GParamFlags with a new flag (CAMEL_PARAM_PERSISTENT) which you can specify when registering the GObject property. camel_object_state_read/write() query the object class for properties with this flag. We currently only use state files to save a handful of booleans, so I've dropped support for other types until we actually need more. In the distant future I may kill off binary state files completely and delegate the job to EShellView's "state" key file. - CamelObject events have been replaced by GObject signals. Short Term Goals - Replace CamelException with GError. I already have this mostly done on a branch. It's straight-forward busywork except for one issue: Poor error handling often leads to multiple errors piling up on a single exception instance. I consider this a programming error regardless of how the exception mechanism handles it. GError handles this by accepting the first error and rejecting subsequent attempts to set an error with a run-time warning (unless the GError is explicitly cleared). CamelException just silently overwrites previous errors with new errors. In the coming days I'll be committing a patch to make CamelException behave like GError in this respect. I'm hoping this will expose some of more egregious error handling issues in Camel providers, but the truth is we'll probably be hunting these down for some time. Once things are stable under the new CamelException semantics, the GError transition should proceed with minimal disruption. - I would like for all blocking methods in Camel to explicitly take a GCancellable or CamelOperation argument, similar to GIO. (Turning CamelOperation into a GCancellable subclass seems like a good fit.) This will happen on a branch for two reasons: 1) not exactly sure how to get there from here yet, so I'll need to experiment a bit, and 2) I may be blocked on the TLS support for GIO since SSL TCP streams need a secure file descriptor to support cancellations, which GCancellable does not yet offer. Long-Term Goals --- - All blocking methods in Camel will have corresponding "async" and "finish" methods, whose default implementation will spawn a thread, run the synchronous method, and pass the result to the user-supplied GAsyncReadyCallback in the main thread. This closely follows the asynchronous pattern in GIO. This approach avoids the need to rewrite all Camel providers to be asynchronous, but they will be free to override the async methods if they so choose. There may be cases where it's more natural to do an operation asynchronously than synchronously. - Once all of this is in place and TLS support has landed in GIO, GIO streams will replace CamelStreams. - In the far distant future I'd like to convert all the MIME filters to the new GConverter data conversion interface for GIO streams, and maybe offer up a few of them to GIO itself. The short-term goals are a priority for 3.0, the long-term goals are just stuff I'm planning for some future Evolution 3.x release. Matthew Barnes ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Camel Manifesto
Hi Matthew, On Fri, 2009-11-27 at 11:58 -0500, Matthew Barnes wrote: > In fact the mail-to-eds effort is part of what's motivating all this. Ah ! - ok :-) sounds good. > To make the discussion more concrete, these are my current plans for > Camel's extreme makeover. The final API will hopefully feel like a > natural extension of GIO. So - I'm well up for that :-) reducing the incidence of wheel re-invention, reducing LOC, increasing familiarity and reducing the skill-set required to hack on the code are all excellent work; I'm really eager to see the good results. > Evolution will then be free to call Camel's async methods from the main > thread. I'll still have to deal with queuing up operations in order on > the Evolution side, but everything will be cancellable (and actually > cancel when you tell it to, especially when the network is flaky) and > we'll hopefully have far fewer issues with frozen UIs. > > Does that sound like a more down-to-earth approach? Yes - very sensible; of course, I love step-wise re-factoring, so we don't end up with these massive branches that take forever to merge; ie. merging just the GObjectification, then merging the GIO-ification [ or whatever ]: I guess that could conflict with keeping Camel API stable, but is that even a useful goal ? The only thing that slightly concerns me, is that I am not certain (knowing ~nothing about it) that the Camel API matches well what the Evolution mailer uses, and/or what works in terms of IPC. When I last did the analysis of a good place to cut here, the line went through some of the Evo mailer code, as well as camel. If the idea is to use GObject introspection to provide a useful & usable D-BUS interface, I rather suspect that the result will be an over-granular, and round-trippy disaster :-) nevermind exposing lifecycle counting over the wire (which is a total disaster) [ or if not that, then the GObject API will be somewhat unpleasantly chunky to use in-process (if that is a goal) ]. As such, presumably it might be interesting to use the mail / d-bus interfaces to provide the initial bulk of the asynchronicity, to clean up the Evolution UI process. Then, in future - if people want to use the backend API itself in an async fashion from other pieces of code, that can be done as needed (?) It just seems to me that since this is D-BUS and we have to implement these awful in-lined object adaptors, we can at least use the DBusMessage encapsulation to throw all the parameters to a slow operation into it's own thread, before unpacking all them all. That would contrast with the method of implementing all slow D-BUS methods using async calls that then spawn threads - which would require unpacking all arguments, and then carefully re-packing them into various async closures. But I speculate, as always :-) HTH, Michael. -- michael.me...@novell.com <><, Pseudo Engineer, itinerant idiot ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Camel Manifesto
On Thu, 2009-11-26 at 14:24 +, Michael Meeks wrote: > So - I'm well up for hiding complexity behind an asynchronous API in > general; that's a great goal. I guess there is also the mail-to-e-d-s > red herring to consider in the mix - that (potentially) adds a layer > of asynchronicity to the equation in the form of remote dbus calls; > perhaps worth considering that in parallel - though it would be cut at a > different place (potentially). In fact the mail-to-eds effort is part of what's motivating all this. Your point about the backends devolving into complex, hard-to-follow state machines is well taken. Evolution itself is what I'm really aiming to de-thread, and having thought about it further I think I can accomplish that without having to disrupt the Camel backends too much. To make the discussion more concrete, these are my current plans for Camel's extreme makeover. The final API will hopefully feel like a natural extension of GIO. - Convert Camel to GObject and discard the redundant parts of its custom type system. (That's mostly done now on my branch.) - Replace CamelException with GError, and make sure all methods that take a GError return -something- so failures can be detected without having to test the GError directly. (I'm slogging through all that now -- over the holiday weekend, no less!) - Gradually add GObject introspection annotations to the public API. - Once GIO gets SSL support, replace all the CamelStreams with stock GIO (and GNIO) streams plus our own GIO stream subclasses for things like filtered streams, "NULL" streams, etc. [1] - Every method that can block will take a GCancellable argument. This will address one of my major problems with Camel's current API, which is that you can't easily tell which functions block and which ones do not. This will make it obvious and explicit. - All (or at least most) of the POSIX or GLib-wrapped POSIX calls for file I/O will be replaced with cancellable GIO equivalents. - And finally the async aspect: taking a cue from GIO, blocking methods will have asynchronous variants whose default behavior is to simply call the corresponding synchronous method from an isolated thread. Subclasses (including the backends) can override the default async methods if they want to. Evolution will then be free to call Camel's async methods from the main thread. I'll still have to deal with queuing up operations in order on the Evolution side, but everything will be cancellable (and actually cancel when you tell it to, especially when the network is flaky) and we'll hopefully have far fewer issues with frozen UIs. Does that sound like a more down-to-earth approach? Matthew Barnes [1] Alex's new GConverter interface also caught my eye as a potential replacement for CamelMimeFilter sometime in the distant future. ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Camel Manifesto
Hi Matthew,
On Fri, 2009-11-20 at 10:46 -0500, Matthew Barnes wrote:
> There may be isolated cases internally to Camel where it can exploit
> parallelism in CPU-intensive tasks with threading or where threads are
> necessary for interacting with synchronous-only libraries, but it should
> be used sparingly and hidden behind a fully asynchronous API.
So - I'm well up for hiding complexity behind an asynchronous API in
general; that's a great goal. I guess there is also the mail-to-e-d-s
red herring to consider in the mix - that (potentially) adds a layer of
asynchronicity to the equation in the form of remote dbus calls; perhaps
worth considering that in parallel - though it would be cut at a
different place (potentially).
> It should not be central to the design of the entire mail application,
> as it is currently. Basically I want the mail front-end in Evolution 3
> to be single threaded, or as close to that as possible.
Sounds reasonable.
> The first is what I think is a very insightful paper on the inherent
> problems with threads:
> http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf
Sure - I read it carefully many moons hence; and it's good - though
IMHO it over-states the case somewhat, or at least - it seems to me that
sometimes the alternative is worse.
The problem IMHO comes when there is a multi-step process; eg.
DN lookup
socket connect
N way ssh handshake
none of which the application cares about - really; the 'async' way is
to whack this all as atomised pieces of code, into some state machine:
switch (state) {
case DNS_CONNECTING:
if (not failed) {
state = DNS_CONNECTED;
pollstate = IO_OUT|ERR;
}
case DNS_CONNECTED:
send (lookup_msg);
pollstate = IO_IN|ERR;
state = DNS_WAIT_REPLY;
case DNS_WAIT_REPLY:
read () ...
if (!short_read)
state = SOCKET_NEW
else
continue in this state
case SOCKET_NEW:
...
case SOCKET_CONNECTED:
pollstate = IO_OUT|ERR;
case SOCKET_CAN_WRITE:
...
pollstate = IO_IN|ERR;
case SOCKET_WAIT_RESULT:
...
}
etc. etc. etc. This is basically what ORBit2 / linc does - although, of
course we get lazy when eg. Windows demands more round-trips, and we
probably do DNS synchronously (that stuff never worked well anyway), and
so on. It is not particularly awful - though, some tricks such as
checking for 'IO_IN' before HUP etc. to avoid loosing the end of a
message are worth not forgetting ;-)
Of course - as the number of the steps in the handshake grows the scope
for error and confusion grow - nevermind the debugging problem: when it
locks up, what went wrong ? :-) how do you even see the state of the
umpteen state machines that are ticking away behind the scenes ?
Of course - some large 'state' structure is required - replicating an
equivalent thread's stack (but on the heap), and that has to be
lifecycle managed and so on in a similar way to threads I guess - with
some extra function overhead.
The threaded version with async callback API I guess has the same
initial closure creation overhead; but then the code is fairly easy to
read:
host_addr = do_blocking_lookup (name);
if (cancelled || !host_addr)
goto emit_error;
fd = socket();
connect (host_addr);
if (cancelled || !connect_error)
goto emit_error;
write_request (fd);
read_reply (fd);
emit_success_callback () etc.
it is also rather easy to debug - as soon as anything fails - with
'bug-buddy' or other conventional debugging tools it is easy to see who
was causing the blocking / dead-locking, or what synchronous calls were
not responsive.
It is also far easier and clearer to re-use code via blocking calls (I
suspect) - than to cobble other sets of states into your state machine.
And of course, none of this is news to anyone I'm sure. Clearly though
- the simpler the locking, and the closer to clean & simple message
passing - the easier and safer the threading becomes.
Anyhow - to me at least ( thankfully shielded from the pain that is
suffered by users of camel ) it seems like a large influx of re-writing
everything as async is unlikely to give substantial reliability wins
(beyond those intrinsic to having a great hacker re-read, and test the
code).
But - since I'm not doing it, I can only write long & silly mails to
try to persuade :-)
ATB,
Michael.
--
michael.me...@novell.com <><, Pseudo Engineer, itinerant idiot
___
Evolution-
Re: [Evolution-hackers] Camel Manifesto
On Sat, 2009-11-21 at 14:57 +0530, Sankar P wrote: > On Sat, Nov 21, 2009 at 12:51 AM, Jeffrey Stedfast > wrote: > > Matthew Barnes wrote: > >> With work on Bonobo removal wrapping up, I've finally started > taking a > >> closer look at Camel (Evolution's mail storage and networking > library) > >> and laying out plans for where I'd like it to go over the short and > long > >> term, with the ultimate goal of splitting it off as a useful > standalone > >> GNOME library (but we're a long way from that). > >> > >> As usual I'm taking a bottom-up approach, starting with basic > cleanup > >> chores (both code and development policies) and building up from > there. > >> Here's some of my thoughts: > >> > >> > >> Backward Compatibility Policy > >> - > >> > >> A reverse dependency search in Debian and Fedora reveals the only > >> external projects currently linking to Camel are mail-notification, > >> evolution-jescs and Anjal (please correct me if I've missed any). > >> > >> That tells me that until Camel moves out of its parents' basement > and > >> matures into an real, honest-to-goodness library, fixing its > interface > >> is more important than maintaining backward compatibility. > Deprecation > >> periods for obsolete API are not necessary, in my opinion. The > few > >> external projects linking to Camel will just have to keep up with > the > >> changes every six months. > >> > > > > I think it's a worthy goal to separate Camel out of Evolution. > > > >> That's not a license to go hog wild though. Some caveats: > >> > >> 1) The soname -must- be kept accurate. If you break the API or > ABI, > >>increment the soname when you commit the break. It doesn't > matter if > >>the break doesn't affect external projects, nor even if it's > already > >>been incremented since the last point release. Bump it > anyway. > >>"Always bump" is an easy policy to remember. It makes our > own daily > >>development run smoother, and helps ensure a release doesn't > slip out > >>with an inaccurate soname. > >> > >>If you're not sure if your patch requires a soname increment, > please > >>ask in IRC or Bugzilla. Patch reviewers should try to > remember too. > >> > >> 2) If you must break the API, try to do so in a way that things > will > >>fail noisily at build time rather than mysteriously at run > time. For > >>example, if you want to change the behavior of an existing > function, > >>it's better to rename the function or change its parameter > list so > >>that stale Evolution code will fail to build. > >> > >> 3) Camel started life as a general purpose mail library and I'd > like to > >>try to get back to that. Camel has become too > Evolution-centric in > >>my view, with too many quick-fix hacks for Evolution bugs that > would > >>not be appropriate for a general purpose mail library. I > will clean > >>these up as I find them, but try to keep that in mind when > altering > >>the API yourself. > >> > > > > I'm not sure what Evolution-specific quick-fix hacks you mean? > > > >> > >> Migrate to GObject > >> -- > >> > >> Camel's homegrown type system will be replaced with GObject so that > >> introspection and D-Bus + language bindings are possible. > CamelObject > >> will remain (for now), but as a subclass of GObject. The > redundant > >> parts of CamelObject will be removed. > >> > > > > Worthy goals... there are some things like CamelObjectBag and such > that > > don't have an equivalent in GObject. > > > >> I'm also following GLib and GTK+'s example of sealing up public > instance > >> data in private sections and enforcing that only its top-level > header > >> file be included outside of Camel (including the providers). > Unlike > >> GLib and GTK+, there will be no transition period. > >> > > Okay. > > > >> This will give us a lot more freedom to reorganize the library and > >> refactor code without disturbing the ABI. Debugging is also > easier when > >> you can trap data accesses through "get" and "set" functions. > >> > > > > Sure, but I don't think any code actually goes behind any get/set > > method's back at the moment (or at least didn't back when I worked > on > > Camel). Regardless, I'm cool with the proposed changes... > > > >> I've been chipping away at this as a side project for the past year > (it > >> was a good mindless activity when I got burned out on Bonobo > removal), > >> and I recently published my results to git.gnome.org as a branch > named > >> "camel-gobject". The CamelObject conversion is finished -- > including > >> all the boilerplate changes in the subclasses -- but I haven't > finished > >> sealing up the API. > >> > >> The branch probably won't land until 2.31 at the earliest. The > backward > >> compatibility policies I described above would be in effect > thereafter. > >> > >> > >> Kill CamelStream > >> > >> > >> This is a distant future goal and
Re: [Evolution-hackers] Camel Manifesto
On Fri, 2009-11-20 at 14:21 -0500, Jeffrey Stedfast wrote: > I agree with Michael Meeks' concerns here. I also think there are much > more important fish to fry which are also far easier to tackle. Sure. I should have been clearer that this is by no means a high priority task. It will happen gradually, in stages, as time permits. For now I'm blocked anyway waiting for SSL support to land in GIO. Once that happens, CamelStreams will be replaced by GIO streams. Existing providers can continue using synchronous stream operations for awhile longer, while the parts of Evolution that are already asynchronous or will soon be transitioning to that won't have to double-buffer streamed data from Camel any longer. Meanwhile, once the aftershocks from the Bonobo removal are sufficiently squared away, I'll be turning my attention to the summary database and other higher priority issues like you mentioned. The GObject transition will mainly happen in the background, during my off hours. Matthew Barnes ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Camel Manifesto
On Sat, Nov 21, 2009 at 12:51 AM, Jeffrey Stedfast wrote: > Matthew Barnes wrote: >> With work on Bonobo removal wrapping up, I've finally started taking a >> closer look at Camel (Evolution's mail storage and networking library) >> and laying out plans for where I'd like it to go over the short and long >> term, with the ultimate goal of splitting it off as a useful standalone >> GNOME library (but we're a long way from that). >> >> As usual I'm taking a bottom-up approach, starting with basic cleanup >> chores (both code and development policies) and building up from there. >> Here's some of my thoughts: >> >> >> Backward Compatibility Policy >> - >> >> A reverse dependency search in Debian and Fedora reveals the only >> external projects currently linking to Camel are mail-notification, >> evolution-jescs and Anjal (please correct me if I've missed any). >> >> That tells me that until Camel moves out of its parents' basement and >> matures into an real, honest-to-goodness library, fixing its interface >> is more important than maintaining backward compatibility. Deprecation >> periods for obsolete API are not necessary, in my opinion. The few >> external projects linking to Camel will just have to keep up with the >> changes every six months. >> > > I think it's a worthy goal to separate Camel out of Evolution. > >> That's not a license to go hog wild though. Some caveats: >> >> 1) The soname -must- be kept accurate. If you break the API or ABI, >> increment the soname when you commit the break. It doesn't matter if >> the break doesn't affect external projects, nor even if it's already >> been incremented since the last point release. Bump it anyway. >> "Always bump" is an easy policy to remember. It makes our own daily >> development run smoother, and helps ensure a release doesn't slip out >> with an inaccurate soname. >> >> If you're not sure if your patch requires a soname increment, please >> ask in IRC or Bugzilla. Patch reviewers should try to remember too. >> >> 2) If you must break the API, try to do so in a way that things will >> fail noisily at build time rather than mysteriously at run time. For >> example, if you want to change the behavior of an existing function, >> it's better to rename the function or change its parameter list so >> that stale Evolution code will fail to build. >> >> 3) Camel started life as a general purpose mail library and I'd like to >> try to get back to that. Camel has become too Evolution-centric in >> my view, with too many quick-fix hacks for Evolution bugs that would >> not be appropriate for a general purpose mail library. I will clean >> these up as I find them, but try to keep that in mind when altering >> the API yourself. >> > > I'm not sure what Evolution-specific quick-fix hacks you mean? > >> >> Migrate to GObject >> -- >> >> Camel's homegrown type system will be replaced with GObject so that >> introspection and D-Bus + language bindings are possible. CamelObject >> will remain (for now), but as a subclass of GObject. The redundant >> parts of CamelObject will be removed. >> > > Worthy goals... there are some things like CamelObjectBag and such that > don't have an equivalent in GObject. > >> I'm also following GLib and GTK+'s example of sealing up public instance >> data in private sections and enforcing that only its top-level header >> file be included outside of Camel (including the providers). Unlike >> GLib and GTK+, there will be no transition period. >> > Okay. > >> This will give us a lot more freedom to reorganize the library and >> refactor code without disturbing the ABI. Debugging is also easier when >> you can trap data accesses through "get" and "set" functions. >> > > Sure, but I don't think any code actually goes behind any get/set > method's back at the moment (or at least didn't back when I worked on > Camel). Regardless, I'm cool with the proposed changes... > >> I've been chipping away at this as a side project for the past year (it >> was a good mindless activity when I got burned out on Bonobo removal), >> and I recently published my results to git.gnome.org as a branch named >> "camel-gobject". The CamelObject conversion is finished -- including >> all the boilerplate changes in the subclasses -- but I haven't finished >> sealing up the API. >> >> The branch probably won't land until 2.31 at the earliest. The backward >> compatibility policies I described above would be in effect thereafter. >> >> >> Kill CamelStream >> >> >> This is a distant future goal and will have to happen gradually, but I >> would like Camel to shift to a single-threaded design where all file and >> network operations directly use or are derived from GIO's asynchronous >> file and stream APIs. SSL support is currently under development for >> GIO, and that's the only missing piece I see at the platform layer. >> >> I realiz
Re: [Evolution-hackers] Camel Manifesto
On Fri, 2009-11-20 at 14:21 -0500, Jeffrey Stedfast wrote: > The sqlite backend stuff could also use some work. As far as I'm > aware, the tables are non-optimal. I really think it would be worthwhile engaging someone who has "SQL guru" on their resume and asking them for help on this. Maybe just an informative query to the sqlite mailing list will get some interest and useful responses. I know just enough SQL to know that I don't know nearly enough to write a robust, efficient SQL schema. As with security protocols, I like to leave this to the experts and just plead for their help :-) ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Camel Manifesto
Matthew Barnes wrote: > With work on Bonobo removal wrapping up, I've finally started taking a > closer look at Camel (Evolution's mail storage and networking library) > and laying out plans for where I'd like it to go over the short and long > term, with the ultimate goal of splitting it off as a useful standalone > GNOME library (but we're a long way from that). > > As usual I'm taking a bottom-up approach, starting with basic cleanup > chores (both code and development policies) and building up from there. > Here's some of my thoughts: > > > Backward Compatibility Policy > - > > A reverse dependency search in Debian and Fedora reveals the only > external projects currently linking to Camel are mail-notification, > evolution-jescs and Anjal (please correct me if I've missed any). > > That tells me that until Camel moves out of its parents' basement and > matures into an real, honest-to-goodness library, fixing its interface > is more important than maintaining backward compatibility. Deprecation > periods for obsolete API are not necessary, in my opinion. The few > external projects linking to Camel will just have to keep up with the > changes every six months. > I think it's a worthy goal to separate Camel out of Evolution. > That's not a license to go hog wild though. Some caveats: > > 1) The soname -must- be kept accurate. If you break the API or ABI, >increment the soname when you commit the break. It doesn't matter if >the break doesn't affect external projects, nor even if it's already >been incremented since the last point release. Bump it anyway. >"Always bump" is an easy policy to remember. It makes our own daily >development run smoother, and helps ensure a release doesn't slip out >with an inaccurate soname. > >If you're not sure if your patch requires a soname increment, please >ask in IRC or Bugzilla. Patch reviewers should try to remember too. > > 2) If you must break the API, try to do so in a way that things will >fail noisily at build time rather than mysteriously at run time. For >example, if you want to change the behavior of an existing function, >it's better to rename the function or change its parameter list so >that stale Evolution code will fail to build. > > 3) Camel started life as a general purpose mail library and I'd like to >try to get back to that. Camel has become too Evolution-centric in >my view, with too many quick-fix hacks for Evolution bugs that would >not be appropriate for a general purpose mail library. I will clean >these up as I find them, but try to keep that in mind when altering >the API yourself. > I'm not sure what Evolution-specific quick-fix hacks you mean? > > Migrate to GObject > -- > > Camel's homegrown type system will be replaced with GObject so that > introspection and D-Bus + language bindings are possible. CamelObject > will remain (for now), but as a subclass of GObject. The redundant > parts of CamelObject will be removed. > Worthy goals... there are some things like CamelObjectBag and such that don't have an equivalent in GObject. > I'm also following GLib and GTK+'s example of sealing up public instance > data in private sections and enforcing that only its top-level header > file be included outside of Camel (including the providers). Unlike > GLib and GTK+, there will be no transition period. > Okay. > This will give us a lot more freedom to reorganize the library and > refactor code without disturbing the ABI. Debugging is also easier when > you can trap data accesses through "get" and "set" functions. > Sure, but I don't think any code actually goes behind any get/set method's back at the moment (or at least didn't back when I worked on Camel). Regardless, I'm cool with the proposed changes... > I've been chipping away at this as a side project for the past year (it > was a good mindless activity when I got burned out on Bonobo removal), > and I recently published my results to git.gnome.org as a branch named > "camel-gobject". The CamelObject conversion is finished -- including > all the boilerplate changes in the subclasses -- but I haven't finished > sealing up the API. > > The branch probably won't land until 2.31 at the earliest. The backward > compatibility policies I described above would be in effect thereafter. > > > Kill CamelStream > > > This is a distant future goal and will have to happen gradually, but I > would like Camel to shift to a single-threaded design where all file and > network operations directly use or are derived from GIO's asynchronous > file and stream APIs. SSL support is currently under development for > GIO, and that's the only missing piece I see at the platform layer. > > I realize this is a drastic course correction and will require rewriting > all the providers and much of the mailer code in Evolution, but I firmly > believe that the o
Re: [Evolution-hackers] Camel Manifesto
On Fri, 2009-11-20 at 10:36 +, Michael Meeks wrote: > Hmm; you really propose to remove all threading from camel's > implementation ? or just from it's API ? a full removal might be > problematic. There may be isolated cases internally to Camel where it can exploit parallelism in CPU-intensive tasks with threading or where threads are necessary for interacting with synchronous-only libraries, but it should be used sparingly and hidden behind a fully asynchronous API. It should not be central to the design of the entire mail application, as it is currently. Basically I want the mail front-end in Evolution 3 to be single threaded, or as close to that as possible. My motivation for the manifesto stems from two sources: The first is what I think is a very insightful paper on the inherent problems with threads: http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf The second is Peter William's design document on Camel, which has been sitting in our source tree for nearly a decade: http://git.gnome.org/cgit/evolution/tree/mail/README.async Setting aside the outdated implementation details he talks about, it's clear that many of Camel's early design decisions that were made during the 1.x era (or perhaps earlier?) are essentially still in place today. Reading through that document now, the rationale just doesn't stand up anymore, in my opinion. CamelObject should have been killed off long ago, and the arrival of GIO and now GNIO totally changes the game for us. Peter even hints at -- as what was then a distant future goal -- basically what I'm proposing now, here in the distant future. Standing on the cusp of a new era for GNOME, I feel like the time is right to reevaluate what's working and what's not and at least -attempt- a course correction. I think an asynchronous design that takes full advantage of our increasingly capable platform libraries is the way forward. I just don't want to spend the duration of GNOME 3 struggling with the same tired old issues that we struggled with for much of GNOME 2. The community's patience is wearing thin and so is mine. Matthew Barnes ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Camel Manifesto
Hi Matthew, On Wed, 2009-11-18 at 14:07 -0500, Matthew Barnes wrote: > With work on Bonobo removal wrapping up, I've finally started taking a > closer look at Camel (Evolution's mail storage and networking library) Ah - another life-time of cleaning up, and polishing code: the goal sounds really nice. > This is a distant future goal and will have to happen gradually, but I > would like Camel to shift to a single-threaded design where all file and > network operations directly use or are derived from GIO's asynchronous > file and stream APIs. Hmm; you really propose to remove all threading from camel's implementation ? or just from it's API ? a full removal might be problematic. While clearly threading, if done in an un-constrained way, has it's own peculiar problems - there are a couple of obvious advantages: First debugging - while both async and threaded programming loose determinism due to event re-ordering - the debugger at least understands threads - and can hopefully show you your state in a reasonably follow-able way - you can step through slow/blocking calls with 'next', 'finish' etc.; I worry about a world packed with highly granular asynchronous state-machines, all chained together - with no good way to see what is happening, what state everything is in, and (of course) why the app is serenely inactive in it's mainloop suddenly ;-) Secondly - of course, a fully async mode potentially would loose any (possible) benefit of parallelism that in theory threads can provide - in this world of dual-core laptop CPUs, and even hyper-threaded Atoms. OTOH - I'm sure you know what you're doing ;-) ATB, Michael. -- michael.me...@novell.com <><, Pseudo Engineer, itinerant idiot ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
