Re: [Evolution-hackers] Overview of camel-lite

2007-06-11 Thread Gilles Dartiguelongue
Le lundi 11 juin 2007 à 10:57 +0300, Philip Van Hoof a écrit :
  * All compilation warnings fixed (we usually compile with -Wall and
-Werror). Fixing this caused four major bug fixes in initialisation
 of variables in Camel. 

ok, right away, let me just say evo team is interested in fixing as much
compilation warnings as possible. In last week's meeting, Matthew told
me he was especially interested in fixes for camel and I was about to
start to work on it but if you have it all done already... :) 
-- 
Gilles Dartiguelongue [EMAIL PROTECTED]

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Overview of camel-lite

2007-06-11 Thread Philip Van Hoof
On Mon, 2007-06-11 at 10:54 +0200, Gilles Dartiguelongue wrote:
 Le lundi 11 juin 2007 à 10:57 +0300, Philip Van Hoof a écrit :
   * All compilation warnings fixed (we usually compile with -Wall and
 -Werror). Fixing this caused four major bug fixes in initialisation
  of variables in Camel. 
 
 ok, right away, let me just say evo team is interested in fixing as much
 compilation warnings as possible. In last week's meeting, Matthew told
 me he was especially interested in fixes for camel and I was about to
 start to work on it but if you have it all done already... :) 

I posted quite a lot of patches a few months ago with all these
compilation warning fixes.

You can search for compilation warnings in camel


-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Overview of camel-lite

2007-06-11 Thread Srinivasa Ragavan
On Mon, 2007-06-11 at 12:47 +0300, Philip Van Hoof wrote:
 On Mon, 2007-06-11 at 14:48 +0530, Srinivasa Ragavan wrote:
  On Mon, 2007-06-11 at 10:57 +0300, Philip Van Hoof wrote:
   I'm going to make an overview of camel-lite so that if people read the
   discussions here, that they know what it's all about.
   
   Camel-lite is the fork of Camel that Tinymail is using internally right
   now. Camel-lite's code is in my opinion not really in a good shape, it
   more or less implements certain features that are necessary for
   Tinymail.
   
   Bad things:
   
* Some of the features have been done in a hackish way. 
   
* Mostly this had to do with non-blocked reads not being very well
  supported within the code and design of upstream camel
   
* Removes support for a few for Evolution very important but for
  Tinymail unused features
   
* It's a fork (effectively, yes)
   
   Neutral:
   
* I am trying to get some of the changes in upstream Camel. Although
  I'm selective because I don't want to bore the Evolution team with
  features that they'll most likely not consider for inclusion anyway.
   
  My point of view is that if they show interested, I'll bump the
  priority of trying to clean it up and getting it upstream
  
  Of course, we are open for patches that are clean and doesn't break
  existing functionality.
  
  Im fine to push some of your IMAP features from camel-lite, provided
  that there is a safe fallback for servers which do not implement them
  and I don't really want Evolution to break badly for those servers.
 
 All those features are already checked for in the CAPABILITY flags. 
 
 There is one bandwidth improvement in camel-lite that might not work
 with some servers: Upstream camel uses UID FETCH to get a list of
 UIDs. Camel-lite uses UID SEARCH for this. Although I will most likely
 implement a fall back situation based on the EXISTS count, it might not
 be as-nice on a server that incorrectly implements UID
 SEARCH (although they fixed it, Citadel's IMAP server didn't give a
 correct answer to UID SEARCH).
 
 Important note about the current Camel code, because you just said that
 this IMAP code works well with all IMAP servers :-) : According to the
 RFC of IMAP, you MUST disregard the CAPABILITY flags after entering
 STARTTLS *and* after any change in authentication.
 
 Starting TLS can (and on some servers, like exchange's IMAP, can) change
 the capabilities. Authentication can also influence these capabilities.
 
 Upstream Camel's IMAP code does not discard the capabilities at all
 required times. It doesn't surprise me a lot that there are so few bug
 reports about this, because the majority of 'new features' that you'll
 get after either STARTTLS or authentication, only improve the
 possibilities of an IMAP server.
 
 For example IDLE on Exchange IMAP servers, will only be in the
 CAPABILITY line after authentication (at least some Exchange servers).
 
 ps. Note that this is indeed one of my patches that I've sent in the
 past to the patches mailing list (I think there's a bug for it too).
 
  I think the biggest thing with the current IMAP implementation is that it
  has become stable over a period of time and works fine with most of the
  existing servers and falls back to alternative pat(c)h in case of any
  server issues. I think we need to carry this. 
  
  I would appreciate you for the things you have done in camel-lite and
  nice to see things going in a constructive way :)
 
 I think (opinion) that the most easy way to get things in a clean way
 into an upstream version, is to split Camel away from EDS. I noticed
 that Jeffrey wouldn't object and that Matthew more or less agrees with
 this point of view (although, don't let me speak for them).
 

Every time we discuss the first road-block is the split. Why don't we do
some progressive work together before we discuss on this? It would have
made real sense if there are many projects that depends on camel. For
now, it is just Evolution and tinymail. I dont see a great point in
splitting it. I remember that we had discussed it too many times at
various forums. (Guadec/Mail/bugs/...). I'm not in favor of it atm.

-Srini.


 From that point, I can more easily move functionality from camel-lite
 into this camel project (that might have several different types of
 versions and implementations).
 
 For me it's important to have a small dependency: I see no reason to
 depend on the entire EDS suite, just for Camel (which is a component
 that is technically 100% disconnected from all the rest of EDS anyway).
 
 Also important is that the Evolution team can then watch the changes
 into this Camel version mature, and Evolution can then use an older
 version of Camel (so that things get tested in projects like Tinymail).
 
 It will take a lot of time to expell camel-lite out of Tinymail: I had
 to change some types and structures that would have very few to do with
 how Evolution 

Re: [Evolution-hackers] Introduction and Questions

2007-06-11 Thread Ross Boylan
I'm preserving the exchange for context; my responses are sprinkled
below.
On Sun, 2007-06-10 at 23:47 -0400, Jeffrey Stedfast wrote:
 On Sun, 2007-06-10 at 15:55 -0700, Ross Boylan wrote:
  On Fri, 2007-06-08 at 20:22 -0400, Jeffrey Stedfast wrote:

Second question: even if it creates a folder, does it need to stick
around for the folder creation to finish?  I think I remember seeing
that camel was single-threaded
   
   not true...
 
 I should have been clearer here:
 
 yes, it needs to wait for the folder object instantiation to finish (the
 function returns a pointer to the instantiated object, afterall).
And your proposal with open on that object is designed to decouple
creation of the object from creation of the summary, right?  When I said
folder creation I was thinking of the whole get headers and make index
process.
 
 not true that camel is single-threaded - it has a few of its own threads
 going - most notably is the thread for filtering incoming mail for
 providers like imap and maildir (they share the same thread).
 
   
, relying on the client app to do
threading.  Would there be a way to multi-thread this somewhere
   (either
in camel or in the app)?  Obviously doing so would complicate
   things,
because at some point one might need to block (e.g., if I move a
   message
from folder A to B and then switch the UI to look at B).
   
   okay, I think you need to familiarize yourself with Camel's API before
   we discuss anything further :)
   
  http://www.go-evolution.org/Mail_Threading begins
  The Camel API is blocking and synchronous
  although it then goes on to qualify this:
  Custom Threads
  Two tasks inside Camel use threads internally in order to implement a
  cancellable api ontop of a non-cancellable one.
  
  Further
  Mail-OPS
  mail-ops.h contains asynchronous versions of varous Camel api's
  It sounds as if mail-ops is outside of Camel, however.
 
 it is external to camel, yes.
 
  
  So it sounds as if Camel could (in principle) respond to a move request
  by issuing the appropriate IMAP command and then, starting a thread to
  do the other activities (indexing the target folder and deleting the the
  message from the  source folder) and return.
 
 it could, yes, but it'd need a way to report an error unless it waited
 for the operations to finish before returning. For moving mail, you
 typically want to know that it succeeded :)
 
 all of the current camel APIs are designed such that the caller expects
 that when the function returns, the operation has either succeeded or
 failed (where the failure would be set on the CamelException argument).
 
It would then block on
  operations that attempted to access the target folder until the other
  operations completed.
 
 yes, this is true... well, the way folders are implemented at this time
 anyway...
 
  
  I think this could be called a syncronous API, though perhaps that's a
  stretch.
 
 it is indeed a synchronous API :)
Syncronous, but it fails the you know if you've succeeded when the
function returns test.
 
  
  On the other hand, http://www.go-evolution.org/Camel.Operation does not
  sound like a bocking syncronous API at all, so maybe the statement
  quoted at the top is just obsolete?
 
 Camel code is written under the assumption that it is likely it is being
 used in a thread, so it has CamelOperation as a means to:
 
 1. report the progress
 2. cancel operations (typically i/o)
 
 what happens is Evolution has a thread-pool with a CamelOperation object
 for each of the threads the mailer creates so that it can cancel
 operations and/or get progress updates.
This part of the camel API is not syncronous, right?
 
  
  So, first of all I'm confused about the nature of Camel's API and
  operation as far as threading and syncronicity.  
  
  Second, I don't have a sense of whether its features are historical
  accidents (camel was implemented in a simple way and evo then used it as
  it was)
 
 evolution mailer was originally not multi-threaded if my recollection
 serves correctly, although around the time of the first 0.0 release
 threading was added (about 7 years ago).
 
 camel and evolution were developed together, so their designs evolved
 together.
Why was camel separated out at all?

Side note: when I first was trying to figure out what was going on, I
looked in the evo sources and discovered the code wasn't there, but was
in the evo-data-server.  This was confusing because, at least on debian,
the latter was described as providing calendar and address book
functions.  I think the Debian split mirrors the what I see in your
repository.  My current guess is that some of the code in
evolution-data-server (the directory) is really for evolution, and not
for the evo-data-server application.
 
   or the result of some deliberate design decisions.  Blocking
  syncronous operations are simpler to implement,
 
 yep
 
   to use,
 
 yep
 
   to debug,
 
 yep, altho with threads thrown 

Re: [Evolution-hackers] Introduction and Questions

2007-06-11 Thread Ross Boylan
On Mon, 2007-06-11 at 09:46 -0700, Ross Boylan wrote:
 Speaking of complexity, evolution/camel (and maybe Gnome?) do a kind
 of
 objects with C.  Why not just use C++?  Was the C++ toolchain too
 unreliable at the start, or was there some other reason? 
I didn't mean to exclude other alternatives.  The general question was
if you're doing objects anyway, why not use a language that already
supports them?
-- 
Ross Boylan  wk:  (415) 514-8146
185 Berry St #5700   [EMAIL PROTECTED]
Dept of Epidemiology and Biostatistics   fax: (415) 514-8150
University of California, San Francisco
San Francisco, CA 94107-1739 hm:  (415) 550-1062

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] Camel Data Cache mechanism

2007-06-11 Thread Sebastien Tandel
Hi,


   I am looking at the way CAMEL is working and have some questions
about the cache implementation.

First, let's see if I've understood the mechanism basics (and, of
course, don't hesitate to correct me if I'm wrong :))

Camel has a generic cache system which stores objects in Camel Bags.
This camel data cache uses CamelStreamFs to store the real objects we
want to cache. According to the way it is implemented now :
- It is not possible to bound the size of a cache. There only exists
timer for expiration/invalidation data.
- And each of these CamelStreamFs uses a path build by the IMAP provider
(in my case). This path is built with session-storage_path and
camel_service_get_path() but I was unable to determine the value of
session-storage_path. I made here the hypothesis all the objects are
stored on the disk ...

Based on these assumptions, here are the questions :
1) To what I've seen, the cache is on a fs ... Is there any project
which intends to have some cache in the main memory? or is there a
possibility to use a tmpfs?
2) Could we imagine to be able to limit the size of the cache?

I was wondering whether it could be possible to have a mechanism which
would be like a two-level cache. The first level is the main memory of a
size X, in which are stored the more recent accessed objects and the
second level the hard disk with as well a bounded size if possible and
finally access the network if the information is not cached.
Is that a crazy idea? Or are there already some mechanisms implemented
I've not seen?



Regards,
Sebastien Tandel
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Introduction and Questions

2007-06-11 Thread Jeffrey Stedfast
On Mon, 2007-06-11 at 09:46 -0700, Ross Boylan wrote:
 I'm preserving the exchange for context; my responses are sprinkled
 below.
 On Sun, 2007-06-10 at 23:47 -0400, Jeffrey Stedfast wrote:
  On Sun, 2007-06-10 at 15:55 -0700, Ross Boylan wrote:
   On Fri, 2007-06-08 at 20:22 -0400, Jeffrey Stedfast wrote:
 
 Second question: even if it creates a folder, does it need to stick
 around for the folder creation to finish?  I think I remember seeing
 that camel was single-threaded

not true...
  
  I should have been clearer here:
  
  yes, it needs to wait for the folder object instantiation to finish (the
  function returns a pointer to the instantiated object, afterall).
 And your proposal with open on that object is designed to decouple
 creation of the object from creation of the summary, right?

correct, the ::open() would load the summary from disk and (if the store
is network based), sync state with the server (find out about messages
which were deleted since last session, flag changes, new messages, etc)

   When I said
 folder creation I was thinking of the whole get headers and make index
 process.

ok

  
  not true that camel is single-threaded - it has a few of its own threads
  going - most notably is the thread for filtering incoming mail for
  providers like imap and maildir (they share the same thread).
  

 , relying on the client app to do
 threading.  Would there be a way to multi-thread this somewhere
(either
 in camel or in the app)?  Obviously doing so would complicate
things,
 because at some point one might need to block (e.g., if I move a
message
 from folder A to B and then switch the UI to look at B).

okay, I think you need to familiarize yourself with Camel's API before
we discuss anything further :)

   http://www.go-evolution.org/Mail_Threading begins
   The Camel API is blocking and synchronous
   although it then goes on to qualify this:
   Custom Threads
   Two tasks inside Camel use threads internally in order to implement a
   cancellable api ontop of a non-cancellable one.
   
   Further
   Mail-OPS
   mail-ops.h contains asynchronous versions of varous Camel api's
   It sounds as if mail-ops is outside of Camel, however.
  
  it is external to camel, yes.
  
   
   So it sounds as if Camel could (in principle) respond to a move request
   by issuing the appropriate IMAP command and then, starting a thread to
   do the other activities (indexing the target folder and deleting the the
   message from the  source folder) and return.
  
  it could, yes, but it'd need a way to report an error unless it waited
  for the operations to finish before returning. For moving mail, you
  typically want to know that it succeeded :)
  
  all of the current camel APIs are designed such that the caller expects
  that when the function returns, the operation has either succeeded or
  failed (where the failure would be set on the CamelException argument).
  
 It would then block on
   operations that attempted to access the target folder until the other
   operations completed.
  
  yes, this is true... well, the way folders are implemented at this time
  anyway...
  
   
   I think this could be called a syncronous API, though perhaps that's a
   stretch.
  
  it is indeed a synchronous API :)
 Syncronous, but it fails the you know if you've succeeded when the
 function returns test.

most of the camel APIs don't fail that test

  
   
   On the other hand, http://www.go-evolution.org/Camel.Operation does not
   sound like a bocking syncronous API at all, so maybe the statement
   quoted at the top is just obsolete?
  
  Camel code is written under the assumption that it is likely it is being
  used in a thread, so it has CamelOperation as a means to:
  
  1. report the progress
  2. cancel operations (typically i/o)
  
  what happens is Evolution has a thread-pool with a CamelOperation object
  for each of the threads the mailer creates so that it can cancel
  operations and/or get progress updates.
 This part of the camel API is not syncronous, right?

in so much as doing camel_operation_cancel() returning doesn't mean that
the other thread has stopped, correct. Basically it just means that
you've sent a command along a pipe to the thread to cancel at its
earliest convenience.

  
   
   So, first of all I'm confused about the nature of Camel's API and
   operation as far as threading and syncronicity.  
   
   Second, I don't have a sense of whether its features are historical
   accidents (camel was implemented in a simple way and evo then used it as
   it was)
  
  evolution mailer was originally not multi-threaded if my recollection
  serves correctly, although around the time of the first 0.0 release
  threading was added (about 7 years ago).
  
  camel and evolution were developed together, so their designs evolved
  together.
 Why was camel separated out at all?

good model/view split :) you 

Re: [Evolution-hackers] Camel Data Cache mechanism

2007-06-11 Thread Jeffrey Stedfast
On Mon, 2007-06-11 at 15:27 -0300, Sebastien Tandel wrote:
 Hi,
 
 
I am looking at the way CAMEL is working and have some questions
 about the cache implementation.
 
 First, let's see if I've understood the mechanism basics (and, of
 course, don't hesitate to correct me if I'm wrong :))
 
 Camel has a generic cache system which stores objects in Camel Bags.
 This camel data cache uses CamelStreamFs to store the real objects we
 want to cache. According to the way it is implemented now :
 - It is not possible to bound the size of a cache. There only exists
 timer for expiration/invalidation data.
 - And each of these CamelStreamFs uses a path build by the IMAP provider
 (in my case). This path is built with session-storage_path and
 camel_service_get_path() but I was unable to determine the value of
 session-storage_path. I made here the hypothesis all the objects are
 stored on the disk ...
 
 Based on these assumptions, here are the questions :
 1) To what I've seen, the cache is on a fs ... Is there any project
 which intends to have some cache in the main memory? or is there a
 possibility to use a tmpfs?

not afaik.

 2) Could we imagine to be able to limit the size of the cache?

I don't see any reason why this couldn't be implemented :)

 
 I was wondering whether it could be possible to have a mechanism which
 would be like a two-level cache. The first level is the main memory of a
 size X, in which are stored the more recent accessed objects and the
 second level the hard disk with as well a bounded size if possible and
 finally access the network if the information is not cached.
 Is that a crazy idea? Or are there already some mechanisms implemented
 I've not seen?

Is there any reason to do this? What problem would you be solving?

The cache is typically only used for messages from a remote host (in
fact, I can't think of anything else it's used for). The problem with
trying to keep an in-memory cache is that users don't typically keep
hitting the same message, so there seems to be little point in having an
in-memory cache on top of the disk cache.


Jeff


___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Introduction and Questions

2007-06-11 Thread Ross Boylan
On Mon, 2007-06-11 at 14:39 -0400, Jeffrey Stedfast wrote:
So it sounds as if Camel could (in principle) respond to a move
 request
by issuing the appropriate IMAP command and then, starting a
 thread to
do the other activities (indexing the target folder and deleting
 the the
message from the  source folder) and return.
   
   it could, yes, but it'd need a way to report an error unless it
 waited
   for the operations to finish before returning. For moving mail,
 you
   typically want to know that it succeeded :)
   
   all of the current camel APIs are designed such that the caller
 expects
   that when the function returns, the operation has either succeeded
 or
   failed (where the failure would be set on the CamelException
 argument).
   
  It would then block on
operations that attempted to access the target folder until the
 other
operations completed.
   
   yes, this is true... well, the way folders are implemented at this
 time
   anyway...
   

I think this could be called a syncronous API, though perhaps
 that's a
stretch.
   
   it is indeed a synchronous API :)
  Syncronous, but it fails the you know if you've succeeded when the
  function returns test.
 
 most of the camel APIs don't fail that test
 
This was in the context of something I was thinking of (So it sounds as
if Camel could ).  The feature/change I was contemplating sounds as
if it would be a bad idea because it violates the usual model for camel
calls.  Probably either a smaller change (keeping the
blocking/syncronous/useful return values in place) or a bigger one
(general asyncronicity) would be a better idea--at least in a perfect
world of infinite coding time.:)  In fact, your idea with folder open
preserves the expected features of the camel API.
-- 
Ross Boylan  wk:  (415) 514-8146
185 Berry St #5700   [EMAIL PROTECTED]
Dept of Epidemiology and Biostatistics   fax: (415) 514-8150
University of California, San Francisco
San Francisco, CA 94107-1739 hm:  (415) 550-1062

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers