Re: [Evolution-hackers] Camel Manifesto (update)

2011-03-13 Thread David Woodhouse
On Thu, 2011-02-17 at 18:37 +, David Woodhouse wrote:
 I was told today that the GIO (and libsoup) async methods may not be
 called from a thread other than the one running the mainloop. I found a
 stupid race in libsoup¹ and filed a fix, but the bug was closed INVALID
 because apparently it's not *supposed* to be thread-safe.
 
 That's a serious issue for using it from Evolution; it means we have to
 jump through stupid hoops like using idle callbacks to call it from the
 main thread, just as we do for gconf. 

Ug, and now I've found that that workaround doesn't work either. If we
try to rename a folder, we end up blocking in the main thread, waiting
for the soup request that we deliberately put into an idle function so
that it could run in the main thread...

Thread 1 (Thread 0x77d93980 (LWP 9874)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0034ff00ee2d in e_flag_wait (flag=0x2002220) at e-flag.c:130
#2  0x7fffe387e934 in ews_get_folder_info_sync (store=0x6e42a0 
[CamelEwsStore], top=0x1480880 asd, flags=1, error=0x0) at 
camel-ews-store.c:500
#3  0x0035020508ae in camel_store_get_folder_info (store=0x6e42a0 
[CamelEwsStore], top=0x1480880 asd, flags=1, error=0x0) at camel-store.c:1122
#4  0x003505460ce8 in folder_tree_cell_edited_cb (folder_tree=value 
optimized out, path_string=value optimized out, new_name=0x1ca6460 asd) at 
em-folder-tree.c:624

As before, I would prefer to *fix* the broken locking, so that we can
call the soup functions from any thread. But if I'm still not allowed to
do that, what's the best way to fix this deadlock?

-- 
dwmw2

___
evolution-hackers mailing list
evolution-hackers@gnome.org
To change your list options or unsubscribe, visit ...
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Camel Manifesto (update)

2011-03-13 Thread Matthew Barnes
On Sun, 2011-03-13 at 21:32 +, David Woodhouse wrote: 
 Ug, and now I've found that that workaround doesn't work either. If we
 try to rename a folder, we end up blocking in the main thread, waiting
 for the soup request that we deliberately put into an idle function so
 that it could run in the main thread...
 
 Thread 1 (Thread 0x77d93980 (LWP 9874)):
 #0  pthread_cond_wait@@GLIBC_2.3.2 () at 
 ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
 #1  0x0034ff00ee2d in e_flag_wait (flag=0x2002220) at e-flag.c:130
 #2  0x7fffe387e934 in ews_get_folder_info_sync (store=0x6e42a0 
 [CamelEwsStore], top=0x1480880 asd, flags=1, error=0x0) at 
 camel-ews-store.c:500
 #3  0x0035020508ae in camel_store_get_folder_info (store=0x6e42a0 
 [CamelEwsStore], top=0x1480880 asd, flags=1, error=0x0) at 
 camel-store.c:1122
 #4  0x003505460ce8 in folder_tree_cell_edited_cb (folder_tree=value 
 optimized out, path_string=value optimized out, new_name=0x1ca6460 asd) 
 at em-folder-tree.c:624
 
 As before, I would prefer to *fix* the broken locking, so that we can
 call the soup functions from any thread. But if I'm still not allowed to
 do that, what's the best way to fix this deadlock?

We talked about this on IRC already, but for the record EMFolderTree is
at fault here for calling a blocking Camel function.

folder_tree_cell_edited_cb() instead needs to fetch folder info
asynchronously and then do the actual folder rename asynchronously.

I've already peppered the function with FIXME comments to this effect.
I'll try to follow up on that this week before the code freeze.


___
evolution-hackers mailing list
evolution-hackers@gnome.org
To change your list options or unsubscribe, visit ...
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Camel Manifesto (update)

2011-02-17 Thread Matthew Barnes
On Thu, 2011-02-17 at 18:37 +, David Woodhouse wrote:
 I assume that your intention is that the Camel async methods would *not*
 be similarly broken, and that you should be able to call them from *any*
 thread and expect them not to break?
 
 If so, we need be *very* careful about calling into gio.

No, that's not my intention.  My intention is that asynchronous
functions should only ever be called from the main loop thread.

___
evolution-hackers mailing list
evolution-hackers@gnome.org
To change your list options or unsubscribe, visit ...
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Camel Manifesto (update)

2010-09-29 Thread Milan Crha
On Mon, 2010-09-27 at 12:52 -0400, Matthew Barnes wrote:
 Transient operations are now implicit: if you push a new status
 message onto a non-empty message stack, the message is treated as
 transient.  A transient message just means there's a longer delay
 before the message is shown in Evolution's status bar, since transient
 operations tend to come and go quickly.  If the transient operation
 finishes before the timer expires, the transient message is never
 shown at all.  This helps filter out messages that would otherwise
 flash by too quickly for users to read. 

Hi,
I do not like this change. Its result is that I do not see which folder
is getting updated and what percentage is done on it, which I consider
as one of the most valuable information being on the status bar.

I only see there Checking for new mail now, and I have no idea about
actual progress of the operation behind it.
Bye,
Milan

___
evolution-hackers mailing list
evolution-hackers@gnome.org
To change your list options or unsubscribe, visit ...
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Camel Manifesto (update)

2010-09-27 Thread Matthew Barnes
Here's another update on how the Camel API upgrades are coming along.
My initial roadmap from 2009 is referenced in [1], and my last update
was in May [2].

This next round of API changes will be fairly disruptive, so I'd like to
coordinate with anyone finishing up their own Camel branch so my changes
don't stomp on your work.  I know Chen is finishing his Maildir branch,
so unless I hear from someone else I'll plan to commit as soon as Chen's
changes are in.

The changes are substantial, but the result is pretty straight-forward
and boring.  Most of these changes were presented as future goals in my
May update [2].  There's three major changes:


1. Redefine CamelOperation
--

I'm removing the whole thread-private CamelOperation concept, along with
the register/unregister functions.  Thread-private CamelOperations seem
convenient but actually have a detrimental effect on the public API and
is one of the major problems I have with Camel: you can't tell from the
API which functions block and which ones don't.  As a result, Evolution
is still rife with Camel calls that block the main loop and hang the UI.

The forthcoming changes will finally solve this design flaw.  As in GIO,
all blocking functions will take a GCancellable pointer.

Since CamelOperation is now a subclass of GCancellable, any GCancellable
pointer in Camel may be a CamelOperation, a plain GCancellable, or NULL.

I've made some further simplifications to CamelOperation both internally
and externally.  The internal message stack is simpler now, and as such
camel_operation_start() and camel_operation_end() have been renamed to
camel_operation_push_message() and camel_operation_pop_message(), which
I think are more intuitive.

Transient operations are now implicit: if you push a new status message
onto a non-empty message stack, the message is treated as transient.  A
transient message just means there's a longer delay before the message
is shown in Evolution's status bar, since transient operations tend to
come and go quickly.  If the transient operation finishes before the
timer expires, the transient message is never shown at all.  This helps
filter out messages that would otherwise flash by too quickly for users
to read.

Finally, passing NULL to any CamelOperation function is now a no-op.


2. Rename Blocking Methods
--

Almost[*] all blocking method names have grown a _sync suffix.  This
has a couple advantages:

- It's pretty damn obvious now when you're making a blocking Camel call
  from Evolution or some other client program.

- Blocking Camel calls are now pretty easy to grep for (camel_.*_sync).
  We can finally track down those blocking the main loop and file bugs.

Methods already using the word sync I've expanded to synchronize so
the name isn't too weird.  So for example, camel_folder_sync() is now
camel_folder_synchronize_sync().


[*] I didn't bother renaming CamelStream methods, since I plan to
replace then with GIO streams in the near future and clients
generally don't call stream methods directly anyway.


3. Asynchronous Methods
---

All the upgrades I've been making to Camel over the past year, and all
the changes described above were in preparation for this: Camel now has
an asynchronous public API.

Let me immediately point out that there is ZERO IMPACT TO EXISTING CAMEL
PROVIDERS beyond the API changes I mentioned above.  Camel providers
need not do anything special to support the new asynchronous methods.
As in GIO, all async methods default to simply calling the corresponding
synchronous method from a thread pool, which is all Evolution does.

The asynchronous methods will take the old blocking method names in
order to emphasize to clients that this is now the recommended way to
use Camel.  (In case it wasn't already obvious, no I don't care about
Camel's backward-compatibility right now.)


Here's a before and after example to help make this all concrete.

Before:

gboolean
camel_folder_append_message(CamelFolder *folder,
CamelMimeMessage *message,
const CamelMessageInfo *info,
gchar **appended_uid,
GError **error)

After:

void
camel_folder_append_message(CamelFolder *folder,
CamelMimeMessage *message,
CamelMessageInfo *info,
gint io_priority,
GCancellable *cancellable,
GAsyncReadyCallback callback,
gpointer user_data)

gboolean
camel_folder_append_message_finish (CamelFolder *folder,
GAsyncResult *result,
gchar **appended_uid,
   

Re: [Evolution-hackers] Camel Manifesto

2009-11-30 Thread Michael Meeks
Hi Matthew,

On Fri, 2009-11-27 at 11:58 -0500, Matthew Barnes wrote:
 In fact the mail-to-eds effort is part of what's motivating all this.

Ah ! - ok :-) sounds good.

 To make the discussion more concrete, these are my current plans for
 Camel's extreme makeover.  The final API will hopefully feel like a
 natural extension of GIO.

So - I'm well up for that :-) reducing the incidence of wheel
re-invention, reducing LOC, increasing familiarity and reducing the
skill-set required to hack on the code are all excellent work; I'm
really eager to see the good results.

 Evolution will then be free to call Camel's async methods from the main
 thread.  I'll still have to deal with queuing up operations in order on
 the Evolution side, but everything will be cancellable (and actually
 cancel when you tell it to, especially when the network is flaky) and
 we'll hopefully have far fewer issues with frozen UIs.
 
 Does that sound like a more down-to-earth approach?

Yes - very sensible; of course, I love step-wise re-factoring, so we
don't end up with these massive branches that take forever to merge; ie.
merging just the GObjectification, then merging the GIO-ification [ or
whatever ]: I guess that could conflict with keeping Camel API stable,
but is that even a useful goal ?

The only thing that slightly concerns me, is that I am not certain
(knowing ~nothing about it) that the Camel API matches well what the
Evolution mailer uses, and/or what works in terms of IPC. When I last
did the analysis of a good place to cut here, the line went through some
of the Evo mailer code, as well as camel.

If the idea is to use GObject introspection to provide a useful 
usable D-BUS interface, I rather suspect that the result will be an
over-granular, and round-trippy disaster :-) nevermind exposing
lifecycle counting over the wire (which is a total disaster) [ or if not
that, then the GObject API will be somewhat unpleasantly chunky to use
in-process (if that is a goal) ].

As such, presumably it might be interesting to use the mail / d-bus
interfaces to provide the initial bulk of the asynchronicity, to clean
up the Evolution UI process. Then, in future - if people want to use the
backend API itself in an async fashion from other pieces of code, that
can be done as needed (?)

It just seems to me that since this is D-BUS and we have to implement
these awful in-lined object adaptors, we can at least use the
DBusMessage encapsulation to throw all the parameters to a slow
operation into it's own thread, before unpacking all them all. That
would contrast with the method of implementing all slow D-BUS methods
using async calls that then spawn threads - which would require
unpacking all arguments, and then carefully re-packing them into various
async closures.

But I speculate, as always :-)

HTH,

Michael.

-- 
 michael.me...@novell.com  , Pseudo Engineer, itinerant idiot

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Camel Manifesto

2009-11-27 Thread Matthew Barnes
On Thu, 2009-11-26 at 14:24 +, Michael Meeks wrote:
   So - I'm well up for hiding complexity behind an asynchronous API in
 general; that's a great goal. I guess there is also the mail-to-e-d-s
 red herring to consider in the mix - that (potentially) adds a layer
 of asynchronicity to the equation in the form of remote dbus calls;
 perhaps worth considering that in parallel - though it would be cut at a
 different place (potentially).

In fact the mail-to-eds effort is part of what's motivating all this.

Your point about the backends devolving into complex, hard-to-follow
state machines is well taken.  Evolution itself is what I'm really
aiming to de-thread, and having thought about it further I think I can
accomplish that without having to disrupt the Camel backends too much.

To make the discussion more concrete, these are my current plans for
Camel's extreme makeover.  The final API will hopefully feel like a
natural extension of GIO.

- Convert Camel to GObject and discard the redundant parts of its
  custom type system.  (That's mostly done now on my branch.) 

- Replace CamelException with GError, and make sure all methods that
  take a GError return -something- so failures can be detected without
  having to test the GError directly.  (I'm slogging through all that
  now -- over the holiday weekend, no less!)

- Gradually add GObject introspection annotations to the public API.

- Once GIO gets SSL support, replace all the CamelStreams with stock
  GIO (and GNIO) streams plus our own GIO stream subclasses for things
  like filtered streams, NULL streams, etc. [1]

- Every method that can block will take a GCancellable argument.  This
  will address one of my major problems with Camel's current API, which
  is that you can't easily tell which functions block and which ones do
  not.  This will make it obvious and explicit.

- All (or at least most) of the POSIX or GLib-wrapped POSIX calls for
  file I/O will be replaced with cancellable GIO equivalents.

- And finally the async aspect: taking a cue from GIO, blocking methods
  will have asynchronous variants whose default behavior is to simply
  call the corresponding synchronous method from an isolated thread.
  Subclasses (including the backends) can override the default async
  methods if they want to.

Evolution will then be free to call Camel's async methods from the main
thread.  I'll still have to deal with queuing up operations in order on
the Evolution side, but everything will be cancellable (and actually
cancel when you tell it to, especially when the network is flaky) and
we'll hopefully have far fewer issues with frozen UIs.

Does that sound like a more down-to-earth approach?

Matthew Barnes


[1] Alex's new GConverter interface also caught my eye as a potential
replacement for CamelMimeFilter sometime in the distant future.

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Camel Manifesto

2009-11-26 Thread Michael Meeks
Hi Matthew,

On Fri, 2009-11-20 at 10:46 -0500, Matthew Barnes wrote:
 There may be isolated cases internally to Camel where it can exploit
 parallelism in CPU-intensive tasks with threading or where threads are
 necessary for interacting with synchronous-only libraries, but it should
 be used sparingly and hidden behind a fully asynchronous API.

So - I'm well up for hiding complexity behind an asynchronous API in
general; that's a great goal. I guess there is also the mail-to-e-d-s
red herring to consider in the mix - that (potentially) adds a layer of
asynchronicity to the equation in the form of remote dbus calls; perhaps
worth considering that in parallel - though it would be cut at a
different place (potentially).

   It should not be central to the design of the entire mail application,
 as it is currently.  Basically I want the mail front-end in Evolution 3
 to be single threaded, or as close to that as possible.

Sounds reasonable.

 The first is what I think is a very insightful paper on the inherent
 problems with threads:
 http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf

Sure - I read it carefully many moons hence; and it's good - though
IMHO it over-states the case somewhat, or at least - it seems to me that
sometimes the alternative is worse.

The problem IMHO comes when there is a multi-step process; eg.

DN lookup
socket connect
N way ssh handshake 

none of which the application cares about - really; the 'async' way is
to whack this all as atomised pieces of code, into some state machine:

switch (state) {
case DNS_CONNECTING:
if (not failed) {
state = DNS_CONNECTED;
pollstate = IO_OUT|ERR;
}
case DNS_CONNECTED:
send (lookup_msg);
pollstate = IO_IN|ERR;
state = DNS_WAIT_REPLY;
case DNS_WAIT_REPLY:
read () ...
if (!short_read)
state = SOCKET_NEW
else
continue in this state
case SOCKET_NEW:
...
case SOCKET_CONNECTED:
pollstate = IO_OUT|ERR;
case SOCKET_CAN_WRITE:
...
pollstate = IO_IN|ERR;
case SOCKET_WAIT_RESULT:
...
}

etc. etc. etc. This is basically what ORBit2 / linc does - although, of
course we get lazy when eg. Windows demands more round-trips, and we
probably do DNS synchronously (that stuff never worked well anyway), and
so on. It is not particularly awful - though, some tricks such as
checking for 'IO_IN' before HUP etc. to avoid loosing the end of a
message are worth not forgetting ;-)

Of course - as the number of the steps in the handshake grows the scope
for error and confusion grow - nevermind the debugging problem: when it
locks up, what went wrong ? :-) how do you even see the state of the
umpteen state machines that are ticking away behind the scenes ?

Of course - some large 'state' structure is required - replicating an
equivalent thread's stack (but on the heap), and that has to be
lifecycle managed and so on in a similar way to threads I guess - with
some extra function overhead.

The threaded version with async callback API I guess has the same
initial closure creation overhead; but then the code is fairly easy to
read:

host_addr = do_blocking_lookup (name);
if (cancelled || !host_addr)
goto emit_error;
fd = socket();
connect (host_addr);
if (cancelled || !connect_error)
goto emit_error;
write_request (fd);
read_reply (fd);
emit_success_callback () etc.

it is also rather easy to debug - as soon as anything fails - with
'bug-buddy' or other conventional debugging tools it is easy to see who
was causing the blocking / dead-locking, or what synchronous calls were
not responsive.

It is also far easier and clearer to re-use code via blocking calls (I
suspect) - than to cobble other sets of states into your state machine.

And of course, none of this is news to anyone I'm sure. Clearly though
- the simpler the locking, and the closer to clean  simple message
passing - the easier and safer the threading becomes.

Anyhow - to me at least ( thankfully shielded from the pain that is
suffered by users of camel ) it seems like a large influx of re-writing
everything as async is unlikely to give substantial reliability wins
(beyond those intrinsic to having a great hacker re-read, and test the
code).

But - since I'm not doing it, I can only write long  silly mails to
try to persuade :-)

ATB,

Michael.

-- 
 michael.me...@novell.com  , Pseudo Engineer, itinerant idiot

___
Evolution-hackers 

Re: [Evolution-hackers] Camel Manifesto

2009-11-22 Thread Chenthill
On Sat, 2009-11-21 at 14:57 +0530, Sankar P wrote:
 On Sat, Nov 21, 2009 at 12:51 AM, Jeffrey Stedfast f...@novell.com
 wrote:
  Matthew Barnes wrote:
  With work on Bonobo removal wrapping up, I've finally started
 taking a
  closer look at Camel (Evolution's mail storage and networking
 library)
  and laying out plans for where I'd like it to go over the short and
 long
  term, with the ultimate goal of splitting it off as a useful
 standalone
  GNOME library (but we're a long way from that).
 
  As usual I'm taking a bottom-up approach, starting with basic
 cleanup
  chores (both code and development policies) and building up from
 there.
  Here's some of my thoughts:
 
 
  Backward Compatibility Policy
  -
 
  A reverse dependency search in Debian and Fedora reveals the only
  external projects currently linking to Camel are mail-notification,
  evolution-jescs and Anjal (please correct me if I've missed any).
 
  That tells me that until Camel moves out of its parents' basement
 and
  matures into an real, honest-to-goodness library, fixing its
 interface
  is more important than maintaining backward compatibility.
  Deprecation
  periods for obsolete API are not necessary, in my opinion.  The
 few
  external projects linking to Camel will just have to keep up with
 the
  changes every six months.
 
 
  I think it's a worthy goal to separate Camel out of Evolution.
 
  That's not a license to go hog wild though.  Some caveats:
 
  1) The soname -must- be kept accurate.  If you break the API or
 ABI,
 increment the soname when you commit the break.  It doesn't
 matter if
 the break doesn't affect external projects, nor even if it's
 already
 been incremented since the last point release.  Bump it
 anyway.
 Always bump is an easy policy to remember.  It makes our
 own daily
 development run smoother, and helps ensure a release doesn't
 slip out
 with an inaccurate soname.
 
 If you're not sure if your patch requires a soname increment,
 please
 ask in IRC or Bugzilla.  Patch reviewers should try to
 remember too.
 
  2) If you must break the API, try to do so in a way that things
 will
 fail noisily at build time rather than mysteriously at run
 time.  For
 example, if you want to change the behavior of an existing
 function,
 it's better to rename the function or change its parameter
 list so
 that stale Evolution code will fail to build.
 
  3) Camel started life as a general purpose mail library and I'd
 like to
 try to get back to that.  Camel has become too
 Evolution-centric in
 my view, with too many quick-fix hacks for Evolution bugs that
 would
 not be appropriate for a general purpose mail library.  I
 will clean
 these up as I find them, but try to keep that in mind when
 altering
 the API yourself.
 
 
  I'm not sure what Evolution-specific quick-fix hacks you mean?
 
 
  Migrate to GObject
  --
 
  Camel's homegrown type system will be replaced with GObject so that
  introspection and D-Bus + language bindings are possible.
  CamelObject
  will remain (for now), but as a subclass of GObject.  The
 redundant
  parts of CamelObject will be removed.
 
 
  Worthy goals... there are some things like CamelObjectBag and such
 that
  don't have an equivalent in GObject.
 
  I'm also following GLib and GTK+'s example of sealing up public
 instance
  data in private sections and enforcing that only its top-level
 header
  file be included outside of Camel (including the providers).
  Unlike
  GLib and GTK+, there will be no transition period.
 
  Okay.
 
  This will give us a lot more freedom to reorganize the library and
  refactor code without disturbing the ABI.  Debugging is also
 easier when
  you can trap data accesses through get and set functions.
 
 
  Sure, but I don't think any code actually goes behind any get/set
  method's back at the moment (or at least didn't back when I worked
 on
  Camel). Regardless, I'm cool with the proposed changes...
 
  I've been chipping away at this as a side project for the past year
 (it
  was a good mindless activity when I got burned out on Bonobo
 removal),
  and I recently published my results to git.gnome.org as a branch
 named
  camel-gobject.  The CamelObject conversion is finished --
 including
  all the boilerplate changes in the subclasses -- but I haven't
 finished
  sealing up the API.
 
  The branch probably won't land until 2.31 at the earliest.  The
 backward
  compatibility policies I described above would be in effect
 thereafter.
 
 
  Kill CamelStream
  
 
  This is a distant future goal and will have to happen gradually,
 but I
  would like Camel to shift to a single-threaded design where all
 file and
  network operations directly use or are derived from GIO's
 asynchronous
  file and stream APIs.  SSL support is currently under development
 for
  GIO, and that's the only missing piece I see at the platform 

Re: [Evolution-hackers] Camel Manifesto

2009-11-21 Thread Sankar P
On Sat, Nov 21, 2009 at 12:51 AM, Jeffrey Stedfast f...@novell.com wrote:
 Matthew Barnes wrote:
 With work on Bonobo removal wrapping up, I've finally started taking a
 closer look at Camel (Evolution's mail storage and networking library)
 and laying out plans for where I'd like it to go over the short and long
 term, with the ultimate goal of splitting it off as a useful standalone
 GNOME library (but we're a long way from that).

 As usual I'm taking a bottom-up approach, starting with basic cleanup
 chores (both code and development policies) and building up from there.
 Here's some of my thoughts:


 Backward Compatibility Policy
 -

 A reverse dependency search in Debian and Fedora reveals the only
 external projects currently linking to Camel are mail-notification,
 evolution-jescs and Anjal (please correct me if I've missed any).

 That tells me that until Camel moves out of its parents' basement and
 matures into an real, honest-to-goodness library, fixing its interface
 is more important than maintaining backward compatibility.  Deprecation
 periods for obsolete API are not necessary, in my opinion.  The few
 external projects linking to Camel will just have to keep up with the
 changes every six months.


 I think it's a worthy goal to separate Camel out of Evolution.

 That's not a license to go hog wild though.  Some caveats:

 1) The soname -must- be kept accurate.  If you break the API or ABI,
    increment the soname when you commit the break.  It doesn't matter if
    the break doesn't affect external projects, nor even if it's already
    been incremented since the last point release.  Bump it anyway.
    Always bump is an easy policy to remember.  It makes our own daily
    development run smoother, and helps ensure a release doesn't slip out
    with an inaccurate soname.

    If you're not sure if your patch requires a soname increment, please
    ask in IRC or Bugzilla.  Patch reviewers should try to remember too.

 2) If you must break the API, try to do so in a way that things will
    fail noisily at build time rather than mysteriously at run time.  For
    example, if you want to change the behavior of an existing function,
    it's better to rename the function or change its parameter list so
    that stale Evolution code will fail to build.

 3) Camel started life as a general purpose mail library and I'd like to
    try to get back to that.  Camel has become too Evolution-centric in
    my view, with too many quick-fix hacks for Evolution bugs that would
    not be appropriate for a general purpose mail library.  I will clean
    these up as I find them, but try to keep that in mind when altering
    the API yourself.


 I'm not sure what Evolution-specific quick-fix hacks you mean?


 Migrate to GObject
 --

 Camel's homegrown type system will be replaced with GObject so that
 introspection and D-Bus + language bindings are possible.  CamelObject
 will remain (for now), but as a subclass of GObject.  The redundant
 parts of CamelObject will be removed.


 Worthy goals... there are some things like CamelObjectBag and such that
 don't have an equivalent in GObject.

 I'm also following GLib and GTK+'s example of sealing up public instance
 data in private sections and enforcing that only its top-level header
 file be included outside of Camel (including the providers).  Unlike
 GLib and GTK+, there will be no transition period.

 Okay.

 This will give us a lot more freedom to reorganize the library and
 refactor code without disturbing the ABI.  Debugging is also easier when
 you can trap data accesses through get and set functions.


 Sure, but I don't think any code actually goes behind any get/set
 method's back at the moment (or at least didn't back when I worked on
 Camel). Regardless, I'm cool with the proposed changes...

 I've been chipping away at this as a side project for the past year (it
 was a good mindless activity when I got burned out on Bonobo removal),
 and I recently published my results to git.gnome.org as a branch named
 camel-gobject.  The CamelObject conversion is finished -- including
 all the boilerplate changes in the subclasses -- but I haven't finished
 sealing up the API.

 The branch probably won't land until 2.31 at the earliest.  The backward
 compatibility policies I described above would be in effect thereafter.


 Kill CamelStream
 

 This is a distant future goal and will have to happen gradually, but I
 would like Camel to shift to a single-threaded design where all file and
 network operations directly use or are derived from GIO's asynchronous
 file and stream APIs.  SSL support is currently under development for
 GIO, and that's the only missing piece I see at the platform layer.

 I realize this is a drastic course correction and will require rewriting
 all the providers and much of the mailer code in Evolution, but I firmly
 believe that the overuse of threads carries much 

Re: [Evolution-hackers] Camel Manifesto

2009-11-21 Thread Matthew Barnes
On Fri, 2009-11-20 at 14:21 -0500, Jeffrey Stedfast wrote:
 I agree with Michael Meeks' concerns here. I also think there are much
 more important fish to fry which are also far easier to tackle.

Sure.  I should have been clearer that this is by no means a high
priority task.  It will happen gradually, in stages, as time permits.
For now I'm blocked anyway waiting for SSL support to land in GIO.  Once
that happens, CamelStreams will be replaced by GIO streams.  Existing
providers can continue using synchronous stream operations for awhile
longer, while the parts of Evolution that are already asynchronous or
will soon be transitioning to that won't have to double-buffer streamed
data from Camel any longer.

Meanwhile, once the aftershocks from the Bonobo removal are sufficiently
squared away, I'll be turning my attention to the summary database and
other higher priority issues like you mentioned.  The GObject transition
will mainly happen in the background, during my off hours.

Matthew Barnes

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Camel Manifesto

2009-11-20 Thread Michael Meeks
Hi Matthew,

On Wed, 2009-11-18 at 14:07 -0500, Matthew Barnes wrote:
 With work on Bonobo removal wrapping up, I've finally started taking a
 closer look at Camel (Evolution's mail storage and networking library)

Ah - another life-time of cleaning up, and polishing code: the goal
sounds really nice.

 This is a distant future goal and will have to happen gradually, but I
 would like Camel to shift to a single-threaded design where all file and
 network operations directly use or are derived from GIO's asynchronous
 file and stream APIs.

Hmm; you really propose to remove all threading from camel's
implementation ? or just from it's API ? a full removal might be
problematic.

While clearly threading, if done in an un-constrained way, has it's own
peculiar problems - there are a couple of obvious advantages:

First debugging - while both async and threaded programming loose
determinism due to event re-ordering - the debugger at least understands
threads - and can hopefully show you your state in a reasonably
follow-able way - you can step through slow/blocking calls with 'next',
'finish' etc.; I worry about a world packed with highly granular
asynchronous state-machines, all chained together - with no good way to
see what is happening, what state everything is in, and (of course) why
the app is serenely inactive in it's mainloop suddenly ;-)

Secondly - of course, a fully async mode potentially would loose any
(possible) benefit of parallelism that in theory threads can provide -
in this world of dual-core laptop CPUs, and even hyper-threaded Atoms.

OTOH - I'm sure you know what you're doing ;-)

ATB,

Michael.

-- 
 michael.me...@novell.com  , Pseudo Engineer, itinerant idiot

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Camel Manifesto

2009-11-20 Thread Matthew Barnes
On Fri, 2009-11-20 at 10:36 +, Michael Meeks wrote:
   Hmm; you really propose to remove all threading from camel's
 implementation ? or just from it's API ? a full removal might be
 problematic.

There may be isolated cases internally to Camel where it can exploit
parallelism in CPU-intensive tasks with threading or where threads are
necessary for interacting with synchronous-only libraries, but it should
be used sparingly and hidden behind a fully asynchronous API.  It should
not be central to the design of the entire mail application, as it is
currently.  Basically I want the mail front-end in Evolution 3 to be
single threaded, or as close to that as possible.

My motivation for the manifesto stems from two sources:

The first is what I think is a very insightful paper on the inherent
problems with threads:
http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf

The second is Peter William's design document on Camel, which has been
sitting in our source tree for nearly a decade:
http://git.gnome.org/cgit/evolution/tree/mail/README.async

Setting aside the outdated implementation details he talks about, it's
clear that many of Camel's early design decisions that were made during
the 1.x era (or perhaps earlier?) are essentially still in place today.

Reading through that document now, the rationale just doesn't stand up
anymore, in my opinion.  CamelObject should have been killed off long
ago, and the arrival of GIO and now GNIO totally changes the game for
us.  Peter even hints at -- as what was then a distant future goal --
basically what I'm proposing now, here in the distant future.

Standing on the cusp of a new era for GNOME, I feel like the time is
right to reevaluate what's working and what's not and at least -attempt-
a course correction.  I think an asynchronous design that takes full
advantage of our increasingly capable platform libraries is the way
forward.

I just don't want to spend the duration of GNOME 3 struggling with the
same tired old issues that we struggled with for much of GNOME 2.  The
community's patience is wearing thin and so is mine.

Matthew Barnes

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Camel Manifesto

2009-11-20 Thread Paul Smith
On Fri, 2009-11-20 at 14:21 -0500, Jeffrey Stedfast wrote:
 The sqlite backend stuff could also use some work. As far as I'm
 aware, the tables are non-optimal.

I really think it would be worthwhile engaging someone who has SQL
guru on their resume and asking them for help on this.  Maybe just an
informative query to the sqlite mailing list will get some interest and
useful responses.

I know just enough SQL to know that I don't know nearly enough to write
a robust, efficient SQL schema.  As with security protocols, I like to
leave this to the experts and just plead for their help :-)

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers