Re: [Evolution-hackers] PATCH: Storing and fetching bodystructure, the patch

2009-08-17 Thread Philip Van Hoof

Please take a look at the new and reworked patch for adding
bodystructure retrieval and storage support. 

http://bugzilla.gnome.org/show_bug.cgi?id=588557#c7

I reworked the patch to make it store using a separate table, as
requested in the comments by Srinivasa.


On Tue, 2009-07-14 at 12:57 +0200, Philip Van Hoof wrote:
 Hi there,
 
 I would like Evolution to store IMAP's bodystructure as often as
 possible. The reason for this is storing RDF graphs. Let me explain.
 
 I wrote a EPlugin for Evolution that is compiled and distributed by the
 Tracker project. This plugin fetches as much metadata about E-mails as
 possible and pushes it into Tracker's RDF store.
 
 Tracker's RDF store is a Nepomuk-ontology based RDF store that offers
 SPARQL as query language, and SPARQL Update as storage language.
 
 We have enhanced the Nepomuk message ontology so that it can store the
 entire structure of an E-mail. This means the structure that you can
 request using the BODY or BODYSTRUCTURE FETCH requests. These return you
 the MIME structure of the message in a pre-parsed skiplist format.
 
 Using this info our plugin can reconstruct the message's skeleton, but
 then in RDF as a graph.
 
 I don't just talk. I wrote the patch to do this too. It's attached.
 
 I tried to keep the changes small, and I implemented the migration code
 so that the tables' schemas will automatically be converted. The patch
 needs a good review, though. And testing.
 
 This is the point of the patch (a stored bodystructure string):
 
 sqlite select bodystructure from 'INBOX/100' LIMIT 2;
 (TEXT PLAIN (CHARSET ISO-8859-1) NIL NIL 8BIT 2304 53 NIL NIL NIL 
 NIL)
 (TEXT PLAIN NIL NIL NIL 7BIT 3829 80 NIL NIL NIL NIL)
 sqlite 
 
 Note about adding BODYSTRUCTURE to the IMAP FETCH query that IMAP
 servers usually cache the body-structures. It wont slowdown an IMAP
 server, one that is worth being called an IMAP server, much.
 
 It will add some bandwidth. But apparently isn't Evolution trying to
 save bandwidth, otherwise camel-imap-folder.c would use ENVELOPE instead
 of BODY.PEEK[HEADER.FIELDS(-a long list-)] (right?).
 
 
 Here are some pointers about things I just wrote:
 
 The Nemomuk message ontology
 http://git.gnome.org/cgit/tracker/tree/data/ontologies/34-nmo.ontology
 
 How we want this stuff in Tracker, how it could be used
 http://live.gnome.org/Tracker/Documentation/EmailSparql
 
 The Evolution plugin:
 http://git.gnome.org/cgit/tracker/tree/src/plugins/evolution/tracker-evolution-plugin.c
 
 
 Let me know!
 
 
 ___
 Evolution-hackers mailing list
 Evolution-hackers@gnome.org
 http://mail.gnome.org/mailman/listinfo/evolution-hackers
-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] PATCH: Storing and fetching bodystructure, the patch

2009-07-23 Thread Philip Van Hoof

Ping, do we have a decision on this?

On Tue, 2009-07-14 at 12:57 +0200, Philip Van Hoof wrote:
 Hi there,
 
 I would like Evolution to store IMAP's bodystructure as often as
 possible. The reason for this is storing RDF graphs. Let me explain.
 
 I wrote a EPlugin for Evolution that is compiled and distributed by the
 Tracker project. This plugin fetches as much metadata about E-mails as
 possible and pushes it into Tracker's RDF store.
 
 Tracker's RDF store is a Nepomuk-ontology based RDF store that offers
 SPARQL as query language, and SPARQL Update as storage language.
 
 We have enhanced the Nepomuk message ontology so that it can store the
 entire structure of an E-mail. This means the structure that you can
 request using the BODY or BODYSTRUCTURE FETCH requests. These return you
 the MIME structure of the message in a pre-parsed skiplist format.
 
 Using this info our plugin can reconstruct the message's skeleton, but
 then in RDF as a graph.
 
 I don't just talk. I wrote the patch to do this too. It's attached.
 
 I tried to keep the changes small, and I implemented the migration code
 so that the tables' schemas will automatically be converted. The patch
 needs a good review, though. And testing.
 
 This is the point of the patch (a stored bodystructure string):
 
 sqlite select bodystructure from 'INBOX/100' LIMIT 2;
 (TEXT PLAIN (CHARSET ISO-8859-1) NIL NIL 8BIT 2304 53 NIL NIL NIL 
 NIL)
 (TEXT PLAIN NIL NIL NIL 7BIT 3829 80 NIL NIL NIL NIL)
 sqlite 
 
 Note about adding BODYSTRUCTURE to the IMAP FETCH query that IMAP
 servers usually cache the body-structures. It wont slowdown an IMAP
 server, one that is worth being called an IMAP server, much.
 
 It will add some bandwidth. But apparently isn't Evolution trying to
 save bandwidth, otherwise camel-imap-folder.c would use ENVELOPE instead
 of BODY.PEEK[HEADER.FIELDS(-a long list-)] (right?).
 
 
 Here are some pointers about things I just wrote:
 
 The Nemomuk message ontology
 http://git.gnome.org/cgit/tracker/tree/data/ontologies/34-nmo.ontology
 
 How we want this stuff in Tracker, how it could be used
 http://live.gnome.org/Tracker/Documentation/EmailSparql
 
 The Evolution plugin:
 http://git.gnome.org/cgit/tracker/tree/src/plugins/evolution/tracker-evolution-plugin.c
 
 
 Let me know!
 
 
 ___
 Evolution-hackers mailing list
 Evolution-hackers@gnome.org
 http://mail.gnome.org/mailman/listinfo/evolution-hackers
-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] A Camel API to get the filename of the cache, also a proposal to have one format to rule them all

2009-01-06 Thread Philip Van Hoof
On Mon, 2009-01-05 at 09:41 -0500, Jeffrey Stedfast wrote:
 Philip Van Hoof wrote:
  On Mon, 2009-01-05 at 08:25 -0500, Jeffrey Stedfast wrote:
 
 
  Maildir doesn't store individual MIME parts separately. So Mailbox is
  equally hard to handle for metadata engines as MBox is. Only difference
  with MBox is that we need to seek() to some location.
 
  So Maildir doesn't make it possible for us to let app developers
  implement indexing plugins easily, like a typical exif extractor.

 
 I guess, but they could just link with gmime or camel :p

Which is what Tracker is doing at this moment. But for various reasons
we still end up copying the E-mail's decoded attachments to /tmp, then
scan them with the indexer's plugins, and then unlink() the files.

Suffice to say that this ain't ideal when scanning 10.000 E-mails that
way. Much more efficient for us would be to simply enter evo's caches
and read the MIME parts as normal already decoded files.

I also think such a format would improve some of Evolution's own
features:

o. For example a making a thumbnail of an image could use the platform's
   infrastructure, and see it being cached using the thumbnail-spec.

   Less code

o. Another feature is the Save as feature for attachments. Instead of
   having to open a GFile and using CamelStream converted to a
   GOutputStream and decode-streaming it to that stream to save the
   attachment on the filesystem, you just copy the file.

   Less code

o. Inline image viewers: Instead of having to plug the decoded memory of
   the attachment into a blob of memory, you just use any image viewer. 

   Less code

o. Inline attached images for text/html MIME part viewers: right now
   migrating GtkHTML to WebKit or GtkMozEmbed is hard because GtkHTML
   had implemented some special thing that allows it to get itself a
   blob of memory fed as pixmap buffer for images whom src attribute
   start with cid.

   Less code


I'm not even sure if WebKit and GtkMozEmbed support rendering blobs of
memory. Although I have been asking the developers of the respective
components at nearly each conference I meet them about this. They all
promised to at least offer some sort of infrastructure for this.

Lot's of promises ;)

After thinking about it very hard, and quite a lot, I didn't find any
good reason to store attachments in Base64 encoding. I only found
reasons why you would want to store it decoded: Less code, same features

The only exception why storing in Base64 encoding could be the feature:
View the source of this E-mail. You can perform the Base64 encode as
the E-mail becomes visible in the E-mail source viewer, it's not a good
reason (let's say this introduces 5 lines of camel_stream_* code).

You could say: because we want to use a standard for our storage:

 - Mailbox can't work on Windows because the author of the spec refuses
   to change the character ':' into '!' for the filenames. Which renders
   his entire specification completely useless. Windows is not
   irrelevant, it's being used a lot. Ignoring it is like carving the
   word stupid on your head with a knife.

   But fine, let him. We are free to ignore his spec, right?

   Maildir also doesn't specify storing MIME parts as separate files.

 - MBox is just broken. You can't put 3Gigs of data in one file, require
   a rewrite of that file each time you want to remove 1kb of data out
   of it and have no index on it (this, at least, is something Maildir
   got right by letting the kernel's FS take care of that: atomic
   renames and DIR is quite good as an index).

   An MBox file is a ticking timebomb waiting to get corrupted.

   MBox also doesn't specify storing MIME parts as separate files.

 - What other formats do we have? Is there one so called standard
   format that stores MIME parts as individual decoded files?

   Because if not then just like the Maildir-guy I'll quickly make a
   website and give it a name. And then let's all start calling it a
   standard. Problem solved? It's not that Maildir is really that much
   more than that. A website that describes a broken way of storing
   E-mails.

   Well, ok, a few IMAP server guys decided to use that specification to
   shut up people who say that IMAP servers that store in a binary
   format are not compatible with their freedom religions. Of course
   that's an ill-educated point of view, but who cares. Freedom! *sigh*

-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be


___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] A Camel API to get the filename of the cache, also a proposal to have one format to rule them all

2009-01-05 Thread Philip Van Hoof
On Mon, 2009-01-05 at 08:25 -0500, Jeffrey Stedfast wrote:

 migrating away from the IMAP specific data cache would be good.

Yes. I think IMAP and the local providers are the only ones that are
still using a specialized datacache.

The IMAP4 one, for example, ain't using a specialized one.

  b) migrate away the mbox data cache (the all-in-one file crap)
  
  I'm all for it. Once I thought of doing this, but the options were like
  Maildir or a format of one mbox file per mail in a distributed folder
  [CamelDataCache sort of format, like imap4/GW/Exchange]. But IIRC Fejj,
  had some concern like, Local still might be good to be held in a
  'standards' way. I know it hurts us on expunge/mailbox rewrite etc.

 
 what mbox data cache? CamelDataCache would probably be the best cache to
 use for IMAP.

Although I would change CamelDataCache to store individual MIME parts as
separate files instead of files that look like a single-mail MBox file.

I would also decode the separate MIME parts before storing if the
original E-mail had them encoded (which is usually the case, and always
for binary attachments). This to make it more easy for metadata engines
to index the MIME parts, and to allow such to do this efficiently. 

Perhaps also to reduce disk-space, as encoded consumes more disk-space,
but that is for me just a nice side-effect.

So my format would create a directory foreach E-mail, or prefix each
MIME part with the uid. Perhaps

INBOX/subfolders/temp/1.  // headers+multipart container
INBOX/subfolders/temp/1.1 // multipart container
INBOX/subfolders/temp/1.1.1   // text/plain
INBOX/subfolders/temp/1.1.2   // text/html
INBOX/subfolders/temp/1.2.1   // inline JPeg attachment
INBOX/subfolders/temp/1.BODYSTRUCTURE // Bodystructure of the E-mail
INBOX/subfolders/temp/1.ENVELOPE  // Top envelope of the E-mail

ps. Perhaps I would store 1.BODYSTRUCTURE in the database instead. I
would probably store 1.ENVELOPE in the database (like how it is now).

I would probably on top of storing BODYSTRUCTURE and ENVELOPE in the
database also store them in separate files. Even if most filesystems
will consume 4k or more (sector or block size) for those mini files.

To get the JPeg attachment:

$ cp INBOX/subfolders/temp/1.2.1 ~/mommy.jpeg

$ exif INBOX/subfolders/temp/1.2.1
EXIF tags in 'INBOX/subfolders/temp/1.2.1' ('Intel' byte order):
+--
Tag |Value 
+--
Image Description   |Mommy with cake at birthday 
Manufacturer|SONY  
Model   |DSC-T33   
...

$ tracker-search -s EMails birthday
Results:
  email://u...@server/INBOX/temp/1
  email://u...@server/INBOX/temp/1#2.1
  ~/mommy.jpeg


[CUT]

 this can cause problems if you need to verify signed parts because
 re-encoding them might not result in the same output.

Ok, for signatures I guess we can make an exception and keep then
encoded in their original format then.

  For Maildir I recommend wasting diskspace by storing both the original
  Maildir format and in parallel store the attachments separately.
 
  Maildir ain't accessible by current Evolution's UI, by the way.
 
  For MBox I recommend TO STOP USING THIS BROKEN FORMAT. It's insane with
  today's mailboxes that easily grow to 3 gigabytes in size per user.
  
  I second your thoughts for MBox stuff. 

 
 Eh, I think mbox works fine but I can understand wanting to move to
 Maildir which is also fine :-)

Maildir doesn't store individual MIME parts separately. So Mailbox is
equally hard to handle for metadata engines as MBox is. Only difference
with MBox is that we need to seek() to some location.

So Maildir doesn't make it possible for us to let app developers
implement indexing plugins easily, like a typical exif extractor.

We would have to Base64 decode image attachments before extracting exif,
for example. Instead of just saying: here's a stream, or here's a FILE*,
go ahead and extract the info you want. (with a stream we could make it
relatively easy to auto-base64 decode, but often are these extractors
still FILE* based, not stream based).

There's IMO not really a good reason to keep the attachments stored in
their encoded version. Except the signatures, perhaps, but we don't
really need those in decoded form anyway. So it would be fine to have an
exception on signatures (to keep them encoded-stored).

Hmmaybe someday having the fingerprint information about a person might
be useful to verify the identify of an individual before linking the
person with a contact in our RDF triple store.


-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be

[Evolution-hackers] A Camel API to get the filename of the cache, also a proposal to have one format to rule them all

2009-01-02 Thread Philip Van Hoof
Hi there evos,

For an EPlugin that I'm working on I will need a Camel API to get the
filename of the cache.

I will attach a patch that adds this API. The EPlugin that I'm developing is
available at Bug# 565091 and more information about it can be found at

http://live.gnome.org/Evolution/Metadata.


I added a bug for tracking this request:

http://bugzilla.gnome.org/show_bug.cgi?id=566279

I know that for maildir (cur, tmp, new) and mbox (seek position) it's a
little bit controversial to return a filename. For maildir I always use
the cur-file one and for mbox I added /!seek_pos to the end of the
returned filename. 

The reason why I need this is that for indexing already cached E-mails,
Tracker will MIME parse what we can MIME parse. For example filenames
and Exif data of attached images is stolen out of the cached items, to
be made searchable.

We don't want to require Evolution to eat all the code involved in
indexing massive amounts of file formats. Best thing we can do right now
is to simply pass the filenames over IPC.

We STRONGLY recommend to the Evolution team to:

a) migrate away the IMAP specific data cache (see c to store separate parts)
b) migrate away the mbox data cache (the all-in-one file crap)

And to

c) invent a better storage format that doesn't store the attachments in
server's (usually) Base64 encoding. The one format to rule them all.

Instead store the encoded attachments in decoded format (original file
format). This will reduce diskspace (encoding increases diskspace usage)
and will make it more easy to scan the original file for XMP and Exif
information. Don't try to gzip or whatever anything. None of that makes
any sense (original files are usually compressed ideally already).

For example: devices that want to compress have filesystems that do this
for you. Don't be silly trying to do this yourself.

By storing the encoded version the only thing you currently gain is that
the feature view E-mail source doesn't need to recode the attachments.

This ain't a much-used feature. It doesn't have to be fast, at all.

No it doesn't. Really it doesn't.

For Maildir I recommend wasting diskspace by storing both the original
Maildir format and in parallel store the attachments separately.

Maildir ain't accessible by current Evolution's UI, by the way.

For MBox I recommend TO STOP USING THIS BROKEN FORMAT. It's insane with
today's mailboxes that easily grow to 3 gigabytes in size per user.


Once all start using the CamelDataCache API, implementing that new
format and implementing converters wont be very hard. 

For existing CamelDataCache users it's just one format to convert. For
IMAP, mbox, Maildir and mh it's indeed a few extra formats to handle
using a conversion. Wont kill you to implement that, and,  I'll help.


I know c) is a controversial proposal. But the current situation really
makes NO sense. Just go look at the ugly ugly code in Camel, think about
it for a second or two, and you too will see that it just ain't making
any sense.

Not for indexing-engines like either Beagle or Tracker nor is the
needless redundancy in implementations making any sense for Evolution
itself. It's rather a maintenance burden and it's making Evolution far
less agile for supporting new capabilities (like getting its cached data
indexed by softwares that focus on search capabilities).


-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be
Index: camel/providers/nntp/camel-nntp-folder.c
===
--- camel/providers/nntp/camel-nntp-folder.c	(revision 9848)
+++ camel/providers/nntp/camel-nntp-folder.c	(working copy)
@@ -123,6 +123,25 @@
 return ((CamelFolderClass *) folder_class)-set_message_flags (folder, uid, flags, set);
 }
 
+static char*
+nntp_get_filename (CamelFolder *folder, const char *uid, CamelException *ex)
+{
+	CamelNNTPStore *nntp_store = (CamelNNTPStore *) folder-parent_store;
+	char *article, *msgid;
+
+	article = alloca(strlen(uid)+1);
+	strcpy(article, uid);
+	msgid = strchr (article, ',');
+	if (msgid == NULL) {
+		camel_exception_setv (ex, CAMEL_EXCEPTION_SYSTEM,
+  _(Internal error: UID in invalid format: %s), uid);
+		return NULL;
+	}
+	*msgid++ = 0;
+
+	return camel_data_cache_get_filename (nntp_store-cache, cache, msgid, ex);
+}
+
 static CamelStream *
 nntp_folder_download_message (CamelNNTPFolder *nntp_folder, const char *id, const char *msgid, CamelException *ex)
 {
@@ -483,6 +502,7 @@
 	camel_folder_class-count_by_expression = nntp_folder_count_by_expression;
 	camel_folder_class-search_by_uids = nntp_folder_search_by_uids;
 	camel_folder_class-search_free = nntp_folder_search_free;
+	camel_folder_class-get_filename = nntp_get_filename;
 }
 
 CamelType
Index: camel/providers/pop3/camel-pop3-folder.c
===
--- camel/providers/pop3

Re: [Evolution-hackers] [Tracker] Crossover projects (Was: Metadata and Evolution)

2008-12-24 Thread Philip Van Hoof
On Tue, 2008-12-23 at 13:25 +0100, Philip Van Hoof wrote:

Hi evos and trakkies,

  On Fri, 2008-12-19 at 10:55 +0100, Philip Van Hoof wrote:
  
   I will be tracking a patch that is related to the Evolution metadata
   proposal (x) at this bug in GNOME's bugzilla.
   
   http://bugzilla.gnome.org/show_bug.cgi?id=565082
   
   It contains a patch that Evolution's most recent Camel needs in order to
   support the EPlugin that will be written to implement (x).
   
   
   #x http://live.gnome.org/Evolution/Metadata
 
 I attached two new patches. One for Tracker and one for Evolution's data
 server. The one for Tracker is highly unfinished and untested but has
 most of the skeleton that I mentioned before now implemented.

The version I attached this time has a few things that are already
working.

It starts to make sense to review this one, Srag. Although it's a patch
for Tracker, the Evolution plugin could or might some day be ported to
be generic and usable in the repo of Evolution itself (the only thing
that would have to be changed is a little bit of build-environment and
rename tracker to evolution in two or three symbols).

The EPlugin is 

src/tracker-evolution-plugin/tracker-evolution-plugin.h and
src/tracker-evolution-plugin/tracker-evolution-plugin.c

And of course

src/tracker-evolution-plugin/liborg-freedesktop-Tracker-evolution-plugin.eplug.xml

The tracker-evolution-registrar.c (which is unfinished, as I'm testing
with a Vala sample as client - attached too -) is what will run in
Tracker's processes.


-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be


tracker-evo-plugin.diff.gz
Description: GNU Zip compressed data
using DBus;

// [DBus (name = org.gnome.evolution.metadata.Manager)]
// public interface Manager : GLib.Object {
//public abstract void Register (DBus.ObjectPath registrar_path, int last_checkout);
// }

[DBus (name = org.gnome.evolution.metadata.Registrar)]
public class Registrar: GLib.Object {

public void Set (string subject, string[] predicates, string[] values) {
	message (set: + subject + \n);
}

public void Cleanup () {
	message (cleanup\n);
}

public void SetMany (string[] subjects, string[][] predicates, string[][] values) {

	message (setmany\n);

	foreach (weak string subject in subjects) {
		message (setmany: + subject + \n);
	}
}

public void UnsetMany (string[] subjects) {
	message (unsetmany\n);

	foreach (weak string subject in subjects) {
		message (unsetmany: + subject + \n);
	}
}

public void Unset (string subject) {
	message (unset: + subject + \n);
}
}

public class MyApplication : GLib.Object {

private DBus.Connection conn;
private Registrar registrar;

private void on_reply (GLib.Error e) {
  message (done);
}

public void setup () throws DBus.Error, GLib.Error {

	dynamic DBus.Object obj;
	DBus.ObjectPath path;
	int stored_time = 0;

conn = DBus.Bus.get (DBus.BusType.SESSION);

	registrar = new Registrar ();
	path = new DBus.ObjectPath (/my/application/evolution_registrar);

	obj = conn.get_object (org.gnome.evolution,
   /org/gnome/evolution/metadata/Manager,
   org.gnome.evolution.metadata.Manager);

	conn.register_object (path, registrar);

	obj.Register (path, stored_time, on_reply);

}

static int main (string[] args) {
var loop = new MainLoop (null, false);

var app = new MyApplication ();

try {

app.setup ();

} catch (DBus.Error e) {
stderr.printf (Failed to initialise);
return 1;
} catch {
stderr.printf (Dynamic method failure);
return 1;
}

loop.run ();

return 0;
}
}
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] [Tracker] Crossover projects (Was: Metadata and Evolution)

2008-12-23 Thread Philip Van Hoof
On Fri, 2008-12-19 at 14:37 +0100, Philip Van Hoof wrote:

Hi evos and trakkies,

 On Fri, 2008-12-19 at 10:55 +0100, Philip Van Hoof wrote:
 
  I will be tracking a patch that is related to the Evolution metadata
  proposal (x) at this bug in GNOME's bugzilla.
  
  http://bugzilla.gnome.org/show_bug.cgi?id=565082
  
  It contains a patch that Evolution's most recent Camel needs in order to
  support the EPlugin that will be written to implement (x).
  
  
  #x http://live.gnome.org/Evolution/Metadata

I attached two new patches. One for Tracker and one for Evolution's data
server. The one for Tracker is highly unfinished and untested but has
most of the skeleton that I mentioned before now implemented.

The one for Evolution's data server adds logging deletes to the database
operations. Actually is also this unfinished as me and the Evolution
team should discuss how we will keep this log trimmed to a certain size.

The patches have also been attached to the two bugs being used to track
all this.

 I attached a patch that illustrates the skeleton of the project that
 will implement support for this Metadata format in Tracker, as a EPlugin
 that'll be installed by the Tracker package (at least initially).
 
 This is a bug for tracking the progression on this. 
 
 http://bugzilla.gnome.org/show_bug.cgi?id=565091
 
 ps. Bug# 565082 (above) will of course block Bug #565091. It's also
 already marked as such.

-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be


evo-schema-migrate-add-columns-and-log-deletes.diff.gz
Description: GNU Zip compressed data


tracker-evo-plugin.diff.gz
Description: GNU Zip compressed data
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Semantic Desktop Evolution Plugin

2008-12-23 Thread Philip Van Hoof
On Tue, 2008-12-23 at 14:42 +0100, Thomas Franz wrote:

 I'm doing research on personal information management
 and have been working on a semantic desktop implementation
 and extensions for the KDE Dolphin file manager, Gnome Nautilus,
 and Thunberbird Email (see [1] for a video of what I'm talking about).


You might be interested in this:

http://live.gnome.org/Evolution/Metadata

It's being developed as we speak

-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Adding the created and modified columns to summary tables

2008-12-19 Thread Philip Van Hoof
I have created a bug for this:

http://bugzilla.gnome.org/show_bug.cgi?id=565082

I have attached a cleaner version of the patch too.

Please comment and/or review

On Wed, 2008-12-17 at 12:41 +0100, Philip Van Hoof wrote:
 Hi there,
 
 This patch deals with converting the old SQLite tables to a new format
 and adds created and modified.
 
 It's a first rough idea, so please let me know what I should change.
 
-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be
Index: camel/camel-db.c
===
--- camel/camel-db.c	(revision 9841)
+++ camel/camel-db.c	(working copy)
@@ -767,15 +767,137 @@
 	return ((camel_db_command (cdb, query, ex)));
 }
 
+static int
+camel_db_migrate_folder_prepare (CamelDB *cdb, const char *folder_name, gint version, CamelException *ex)
+{
+	int ret = 0;
+	sqlite3_stmt *stmt = NULL;
+	char *table_creation_query;
+
+	/* Migration stage one: storing the old data */
+
+	if (version  1) {
+		table_creation_query = sqlite3_mprintf (DROP TABLE 'temp_%q', folder_name);
+		stmt = NULL;
+		ret = sqlite3_prepare_v2 (cdb-db, table_creation_query, -1, stmt, NULL);
+		if (ret == SQLITE_OK)
+			ret = sqlite3_step (stmt);
+		sqlite3_finalize (stmt);
+		sqlite3_free (table_creation_query);
+
+		table_creation_query = sqlite3_mprintf (CREATE TEMP TABLE IF NOT EXISTS 'temp_%q' (  uid TEXT PRIMARY KEY , flags INTEGER , msg_type INTEGER , read INTEGER , deleted INTEGER , replied INTEGER , important INTEGER , junk INTEGER , attachment INTEGER , msg_security INTEGER , size INTEGER , dsent NUMERIC , dreceived NUMERIC , subject TEXT , mail_from TEXT , mail_to TEXT , mail_cc TEXT , mlist TEXT , followup_flag TEXT , followup_completed_on TEXT , followup_due_by TEXT , part TEXT , labels TEXT , usertags TEXT , cinfo TEXT , bdata TEXT, created TEXT, modified TEXT ), folder_name);
+		stmt = NULL;
+		ret = sqlite3_prepare_v2 (cdb-db, table_creation_query, -1, stmt, NULL);
+		if (ret == SQLITE_OK)
+			ret = sqlite3_step (stmt);
+		sqlite3_finalize (stmt);
+		sqlite3_free (table_creation_query);
+
+		table_creation_query = sqlite3_mprintf (INSERT INTO 'temp_%q' SELECT uid , flags , msg_type , read , deleted , replied , important , junk , attachment , msg_security , size , dsent , dreceived , subject , mail_from , mail_to , mail_cc , mlist , followup_flag , followup_completed_on , followup_due_by , part , labels , usertags , cinfo , bdata , strftime(\%%s\, 'now'), strftime(\%%s\, 'now') FROM %Q, folder_name, folder_name);
+		stmt = NULL;
+		ret = sqlite3_prepare_v2 (cdb-db, table_creation_query, -1, stmt, NULL);
+		if (ret == SQLITE_OK)
+			ret = sqlite3_step (stmt);
+		sqlite3_finalize (stmt);
+		sqlite3_free (table_creation_query);
+
+		table_creation_query = sqlite3_mprintf (DROP TABLE %Q, folder_name);
+		stmt = NULL;
+		ret = sqlite3_prepare_v2 (cdb-db, table_creation_query, -1, stmt, NULL);
+		if (ret == SQLITE_OK)
+			ret = sqlite3_step (stmt);
+		sqlite3_finalize (stmt);
+		sqlite3_free (table_creation_query);
+	}
+
+	/* Add later version migrations here */
+
+	return ret;
+}
+
+static int
+camel_db_migrate_folder_recreate (CamelDB *cdb, const char *folder_name, gint version, CamelException *ex)
+{
+	int ret = 0;
+	char *table_creation_query;
+
+	/* Migration stage two: writing back the old data */
+
+	if (version  1) {
+		table_creation_query = sqlite3_mprintf (INSERT INTO %Q SELECT uid , flags , msg_type , read , deleted , replied , important , junk , attachment , msg_security , size , dsent , dreceived , subject , mail_from , mail_to , mail_cc , mlist , followup_flag , followup_completed_on , followup_due_by , part , labels , usertags , cinfo , bdata, created, modified FROM 'temp_%q', folder_name, folder_name);
+		ret = camel_db_add_to_transaction (cdb, table_creation_query, ex);
+		sqlite3_free (table_creation_query);
+
+		table_creation_query = sqlite3_mprintf (DROP TABLE 'temp_%q', folder_name);
+		ret = camel_db_add_to_transaction (cdb, table_creation_query, ex);
+		sqlite3_free (table_creation_query);
+	}
+
+	/* Add later version migrations here */
+
+	return ret;
+}
+
+static int
+camel_db_write_folder_version (CamelDB *cdb, const char *folder_name, int old_version, CamelException *ex)
+{
+	int ret = 0;
+	char *version_creation_query;
+	char *version_insert_query;
+
+	version_creation_query = sqlite3_mprintf (CREATE TABLE IF NOT EXISTS '%q_version' ( version TEXT ), folder_name);
+
+	if (old_version == -1)
+		version_insert_query = sqlite3_mprintf (INSERT INTO '%q_version' VALUES ('1'), folder_name);
+	else
+		version_insert_query = sqlite3_mprintf (UPDATE '%q_version' SET version='1', folder_name);
+
+	ret = camel_db_add_to_transaction (cdb, version_creation_query, ex);
+	ret = camel_db_add_to_transaction (cdb, version_insert_query, ex);
+
+	sqlite3_free (version_creation_query);
+	sqlite3_free (version_insert_query);
+
+	return ret

[Evolution-hackers] Metadata and Evolution

2008-12-19 Thread Philip Van Hoof
Hi guys (evos and trakkies),

I will be tracking a patch that is related to the Evolution metadata
proposal (x) at this bug in GNOME's bugzilla.

http://bugzilla.gnome.org/show_bug.cgi?id=565082

It contains a patch that Evolution's most recent Camel needs in order to
support the EPlugin that will be written to implement (x).


#x http://live.gnome.org/Evolution/Metadata

-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] Crossover projects (Was: Metadata and Evolution)

2008-12-19 Thread Philip Van Hoof
Hi evos and trakkies,

On Fri, 2008-12-19 at 10:55 +0100, Philip Van Hoof wrote:

 I will be tracking a patch that is related to the Evolution metadata
 proposal (x) at this bug in GNOME's bugzilla.
 
 http://bugzilla.gnome.org/show_bug.cgi?id=565082
 
 It contains a patch that Evolution's most recent Camel needs in order to
 support the EPlugin that will be written to implement (x).
 
 
 #x http://live.gnome.org/Evolution/Metadata

I attached a patch that illustrates the skeleton of the project that
will implement support for this Metadata format in Tracker, as a EPlugin
that'll be installed by the Tracker package (at least initially).

This is a bug for tracking the progression on this. 

http://bugzilla.gnome.org/show_bug.cgi?id=565091

ps. Bug# 565082 (above) will of course block Bug #565091. It's also
already marked as such.


-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be


eplugin-metadata-tracker.diff.gz
Description: GNU Zip compressed data
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] Adding the created and modified columns to summary tables

2008-12-17 Thread Philip Van Hoof
Hi there,

This patch deals with converting the old SQLite tables to a new format
and adds created and modified.

It's a first rough idea, so please let me know what I should change.

-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be
Index: camel/camel-db.c
===
--- camel/camel-db.c	(revision 9841)
+++ camel/camel-db.c	(working copy)
@@ -770,12 +770,65 @@
 int 
 camel_db_prepare_message_info_table (CamelDB *cdb, const char *folder_name, CamelException *ex)
 {
-	int ret;
+	sqlite3_stmt *stmt = NULL;
+	int ret, version = 0, recreate = 0;
 	char *table_creation_query, *safe_index;
+	char *version_creation_query;
+	char *version_insert_query;
 
+	table_creation_query = sqlite3_mprintf (SELECT version FROM '%q_version', folder_name);
+
+	ret = sqlite3_prepare_v2 (cdb-db, table_creation_query, -1, stmt, NULL);
+
+	if (ret == SQLITE_OK)
+		ret = sqlite3_step (stmt);
+
+	if (ret == SQLITE_OK)
+		version = sqlite3_column_int (stmt, 0);
+	else
+		recreate = 1;
+
+	sqlite3_finalize (stmt);
+
+	sqlite3_free (table_creation_query);
+
+	if (version  1) {
+		table_creation_query = sqlite3_mprintf (DROP TABLE 'temp_%q', folder_name);
+		stmt = NULL;
+		ret = sqlite3_prepare_v2 (cdb-db, table_creation_query, -1, stmt, NULL);
+		if (ret == SQLITE_OK)
+			ret = sqlite3_step (stmt);
+		sqlite3_finalize (stmt);
+		sqlite3_free (table_creation_query);
+
+		table_creation_query = sqlite3_mprintf (CREATE TEMP TABLE IF NOT EXISTS 'temp_%q' (  uid TEXT PRIMARY KEY , flags INTEGER , msg_type INTEGER , read INTEGER , deleted INTEGER , replied INTEGER , important INTEGER , junk INTEGER , attachment INTEGER , msg_security INTEGER , size INTEGER , dsent NUMERIC , dreceived NUMERIC , subject TEXT , mail_from TEXT , mail_to TEXT , mail_cc TEXT , mlist TEXT , followup_flag TEXT , followup_completed_on TEXT , followup_due_by TEXT , part TEXT , labels TEXT , usertags TEXT , cinfo TEXT , bdata TEXT, created TEXT, modified TEXT ), folder_name);
+		stmt = NULL;
+		ret = sqlite3_prepare_v2 (cdb-db, table_creation_query, -1, stmt, NULL);
+		if (ret == SQLITE_OK)
+			ret = sqlite3_step (stmt);
+		sqlite3_finalize (stmt);
+		sqlite3_free (table_creation_query);
+
+		table_creation_query = sqlite3_mprintf (INSERT INTO 'temp_%q' SELECT uid , flags , msg_type , read , deleted , replied , important , junk , attachment , msg_security , size , dsent , dreceived , subject , mail_from , mail_to , mail_cc , mlist , followup_flag , followup_completed_on , followup_due_by , part , labels , usertags , cinfo , bdata , strftime(\%%s\, 'now'), strftime(\%%s\, 'now') FROM %Q, folder_name, folder_name);
+		stmt = NULL;
+		ret = sqlite3_prepare_v2 (cdb-db, table_creation_query, -1, stmt, NULL);
+		if (ret == SQLITE_OK)
+			ret = sqlite3_step (stmt);
+		sqlite3_finalize (stmt);
+		sqlite3_free (table_creation_query);
+
+		table_creation_query = sqlite3_mprintf (DROP TABLE %Q, folder_name);
+		stmt = NULL;
+		ret = sqlite3_prepare_v2 (cdb-db, table_creation_query, -1, stmt, NULL);
+		if (ret == SQLITE_OK)
+			ret = sqlite3_step (stmt);
+		sqlite3_finalize (stmt);
+		sqlite3_free (table_creation_query);
+	}
+
 	/* README: It is possible to compress all system flags into a single column and use just as userflags but that makes querying for other applications difficult an d bloats the parsing code. Instead, it is better to bloat the tables. Sqlite should have some optimizations for sparse columns etc. */
 
-	table_creation_query = sqlite3_mprintf (CREATE TABLE IF NOT EXISTS %Q (  uid TEXT PRIMARY KEY , flags INTEGER , msg_type INTEGER , read INTEGER , deleted INTEGER , replied INTEGER , important INTEGER , junk INTEGER , attachment INTEGER , msg_security INTEGER , size INTEGER , dsent NUMERIC , dreceived NUMERIC , subject TEXT , mail_from TEXT , mail_to TEXT , mail_cc TEXT , mlist TEXT , followup_flag TEXT , followup_completed_on TEXT , followup_due_by TEXT , part TEXT , labels TEXT , usertags TEXT , cinfo TEXT , bdata TEXT ), folder_name);
+	table_creation_query = sqlite3_mprintf (CREATE TABLE IF NOT EXISTS %Q (  uid TEXT PRIMARY KEY , flags INTEGER , msg_type INTEGER , read INTEGER , deleted INTEGER , replied INTEGER , important INTEGER , junk INTEGER , attachment INTEGER , msg_security INTEGER , size INTEGER , dsent NUMERIC , dreceived NUMERIC , subject TEXT , mail_from TEXT , mail_to TEXT , mail_cc TEXT , mlist TEXT , followup_flag TEXT , followup_completed_on TEXT , followup_due_by TEXT , part TEXT , labels TEXT , usertags TEXT , cinfo TEXT , bdata TEXT, created TEXT, modified TEXT ), folder_name);
 
 	ret = camel_db_add_to_transaction (cdb, table_creation_query, ex);
 
@@ -808,7 +861,29 @@
 	ret = camel_db_add_to_transaction (cdb, table_creation_query, ex);
 	g_free (safe_index);
 	sqlite3_free (table_creation_query);
-	
+
+	if (version  1) {
+		table_creation_query = sqlite3_mprintf (INSERT

Re: [Evolution-hackers] [Evolution] Beagle and Tracker, letting Evolution feed those beasts RDF triples instead

2008-12-15 Thread Philip Van Hoof
On Wed, 2008-12-10 at 16:37 +, Michael Meeks wrote:

[CUT]

   So there is at least some bound to the growth of the deleted UUID
 log ;-) which is the size / likelyhood of re-use in the UUID space.
 
   It's hard to think of solutions that are that satisfying; but - perhaps
 something like cropping the deletion log-size at a percentage of stored
 mail size, with some log overflow type message to flag that; or having
 some arbitrary size bound on it, or more carefully disabling logging
 when search services are disabled, or ... having only a single client,
 or warning the user that they should run their search service some more,
 or perhaps even coupling the indexing piece more closely to the mailer
 itself somehow.

After some discussion on IRC we decided to add a Cleanup method to the
registrar's interface. This method will be called whenever Evolution has
a reason to believe that the `last_checkout` date as passed during the
registry of the registrar has become (or is) too old.

After the Cleanup Evolution will do a re-import using mostly SetMany.

I have updated http://live.gnome.org/Evolution/Metadata for this.


-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] [Tracker] Support in Tracker for ultra-new Evolution installs that use SQLite for the summary format

2008-12-12 Thread Philip Van Hoof
On Fri, 2008-12-12 at 01:59 +0100, Philip Van Hoof wrote:
 On Thu, 2008-12-11 at 23:58 +0200, Ivan Frade wrote:
  hi philip,
  
   short comment (the tablet is not the best device to review patches):
  can we call this module evolution224 and keep also the old module?
  we could choose between one or other with a flag in compilation time
  (the distributions will choose the correct for them)
 
 With this patch the code will detect the format and automatically choose
 the right way to parse both.
 

This version of the patch adds support for IMAP subfolders and fixes the
fact that in the last patch the email:// URI was generated wrong (it
still contained the folders.db string-piece).


   regards,
  
  Ivan
  
  On 12/11/08, Philip Van Hoof s...@pvanhoof.be wrote:
   This patch makes ultra-new Evolution installs work again with Tracker.
  
   There's one problem and that is that the query will only find E-mails in
   the INBOX folder. You can easily find the Query and figure out what the
   problem is:
  
   The design that Carlos made assumes that for each folder there's a
   summary file. In the new Evolution cache format there's just one
   folders.db for each account.
  
   I could do a generated UNION select after first doing select * from
   folders on folders.db and then generating a query that includes all
   folders. I just have not done this for now and instead I'm just using
   INBOX and I'm neglecting the other folders.
  
   This is NOT the same as the proposal that I am doing at (a). This is
   instead a ad-hoc solution for the new situation (Evolution using SQLite
   for the summaries). I find this solution rather nasty, to be honest.
  
   (a) http://live.gnome.org/Evolution/Metadata
  
   For Carlos: I have also fixed a serious problem in evolution-pop.c,
   which is by the way unaffected by Evolution's changes (and works, if you
   just apply the patch that I included in this larger patch). The POP
   support's get_message_metadata was not returning metadata.
  
   This was crashing my tracker-indexer (as seemingly my compiler was
   putting return 0x2 where the return was omitted, and the memory I have
   at 0x2 didn't dereference TrackerModuleMetadata's members very well).
  
   Please review and/or rework the patch.
  
   --
   Philip Van Hoof, freelance software developer
   home: me at pvanhoof dot be
   gnome: pvanhoof at gnome dot org
   http://pvanhoof.be/blog
   http://codeminded.be
  
  
-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be
Index: src/tracker-indexer/modules/evolution-pop.c
===
--- src/tracker-indexer/modules/evolution-pop.c	(revision 2701)
+++ src/tracker-indexer/modules/evolution-pop.c	(working copy)
@@ -360,6 +360,8 @@
 	}
 
 	g_list_free (list);
+
+	return metadata;
 }
 
 static TrackerModuleMetadata *
Index: src/tracker-indexer/modules/evolution.c
===
--- src/tracker-indexer/modules/evolution.c	(revision 2701)
+++ src/tracker-indexer/modules/evolution.c	(working copy)
@@ -70,7 +70,9 @@
 	strchr (basename, '.') == NULL) {
 		type = MAIL_STORAGE_LOCAL;
 	} else if (g_str_has_prefix (path, imap_dir) 
-		   strcmp (basename, summary) == 0) {
+		   (strcmp (basename, summary) == 0 ||
+		strcmp (basename, folders.db) == 0) ||
+		strcmp (basename, cmeta) == 0) {
 		type = MAIL_STORAGE_IMAP;
 	}
 
Index: src/tracker-indexer/modules/Makefile.am
===
--- src/tracker-indexer/modules/Makefile.am	(revision 2701)
+++ src/tracker-indexer/modules/Makefile.am	(working copy)
@@ -16,7 +16,8 @@
 	$(GIO_CFLAGS)			\
 	$(GLIB2_CFLAGS)			\
 	$(GCONF_CFLAGS)			\
-	$(GMIME_CFLAGS)
+	$(GMIME_CFLAGS)			\
+	$(SQLITE3_CFLAGS)
 
 indexer_modules_LTLIBRARIES = 		\
 	libtracker-module-applications.la\
@@ -67,6 +68,7 @@
 	$(GMODULE_LIBS)			\
 	$(GLIB2_LIBS)			\
 	$(GCONF_LIBS)			\
-	$(GMIME_LIBS)
+	$(GMIME_LIBS)			\
+	$(SQLITE3_LIBS)
 
 endif
Index: src/tracker-indexer/modules/evolution-imap.c
===
--- src/tracker-indexer/modules/evolution-imap.c	(revision 2701)
+++ src/tracker-indexer/modules/evolution-imap.c	(working copy)
@@ -128,7 +128,18 @@
 g_free (file-imap_dir);
 g_free (file-cur_message_uid);
 
-fclose (file-summary);
+	if (file-db) {
+		sqlite3_close (file-db);
+	}
+
+	if (file-stmt) {
+		sqlite3_finalize (file-stmt);
+	}
+
+	if (file-summary) {
+		fclose (file-summary);
+	}
+
 	close (file-fd);
 
 G_OBJECT_CLASS (tracker_evolution_imap_file_parent_class)-finalize (object);
@@ -482,10 +493,88 @@
 }
 
 static void
+get_folder_info (const gchar *path, const gchar *imap_dir, gchar **out_foldersdb, gchar **out_mailbox)
+{
+	gchar *mailbox = NULL

Re: [Evolution-hackers] [Tracker] Support in Tracker for ultra-new Evolution installs that use SQLite for the summary format

2008-12-12 Thread Philip Van Hoof

I just posted a new version of the patch which now opens the DB in
read-only mode.

Thanks for the suggestion

On Fri, 2008-12-12 at 15:12 +0800, Jerry Tan wrote:
 Hi, philip
 
 How about to open folders.db in readonly mode?
 
 I just check sqlite's API. 
 sqlite3_open_v2 support a para with  readonly flag
 
 
 
 On 2008ǯ12·î12Æü 00:34, Philip Van Hoof wrote: 
  This patch makes ultra-new Evolution installs work again with Tracker.
  
  There's one problem and that is that the query will only find E-mails in
  the INBOX folder. You can easily find the Query and figure out what the
  problem is:
  
  The design that Carlos made assumes that for each folder there's a
  summary file. In the new Evolution cache format there's just one
  folders.db for each account.
  
  I could do a generated UNION select after first doing select * from
  folders on folders.db and then generating a query that includes all
  folders. I just have not done this for now and instead I'm just using
  INBOX and I'm neglecting the other folders.
  
  This is NOT the same as the proposal that I am doing at (a). This is
  instead a ad-hoc solution for the new situation (Evolution using SQLite
  for the summaries). I find this solution rather nasty, to be honest.
  
  (a) http://live.gnome.org/Evolution/Metadata
  
  For Carlos: I have also fixed a serious problem in evolution-pop.c,
  which is by the way unaffected by Evolution's changes (and works, if you
  just apply the patch that I included in this larger patch). The POP
  support's get_message_metadata was not returning metadata. 
  
  This was crashing my tracker-indexer (as seemingly my compiler was
  putting return 0x2 where the return was omitted, and the memory I have
  at 0x2 didn't dereference TrackerModuleMetadata's members very well).
  
  Please review and/or rework the patch.
  

  
  
  
  
  ___
  tracker-list mailing list
  tracker-l...@gnome.org
  http://mail.gnome.org/mailman/listinfo/tracker-list
 
-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] [Tracker] Support in Tracker for ultra-new Evolution installs that use SQLite for the summary format

2008-12-11 Thread Philip Van Hoof
On Thu, 2008-12-11 at 23:58 +0200, Ivan Frade wrote:
 hi philip,
 
  short comment (the tablet is not the best device to review patches):
 can we call this module evolution224 and keep also the old module?
 we could choose between one or other with a flag in compilation time
 (the distributions will choose the correct for them)

With this patch the code will detect the format and automatically choose
the right way to parse both.


  regards,
 
 Ivan
 
 On 12/11/08, Philip Van Hoof s...@pvanhoof.be wrote:
  This patch makes ultra-new Evolution installs work again with Tracker.
 
  There's one problem and that is that the query will only find E-mails in
  the INBOX folder. You can easily find the Query and figure out what the
  problem is:
 
  The design that Carlos made assumes that for each folder there's a
  summary file. In the new Evolution cache format there's just one
  folders.db for each account.
 
  I could do a generated UNION select after first doing select * from
  folders on folders.db and then generating a query that includes all
  folders. I just have not done this for now and instead I'm just using
  INBOX and I'm neglecting the other folders.
 
  This is NOT the same as the proposal that I am doing at (a). This is
  instead a ad-hoc solution for the new situation (Evolution using SQLite
  for the summaries). I find this solution rather nasty, to be honest.
 
  (a) http://live.gnome.org/Evolution/Metadata
 
  For Carlos: I have also fixed a serious problem in evolution-pop.c,
  which is by the way unaffected by Evolution's changes (and works, if you
  just apply the patch that I included in this larger patch). The POP
  support's get_message_metadata was not returning metadata.
 
  This was crashing my tracker-indexer (as seemingly my compiler was
  putting return 0x2 where the return was omitted, and the memory I have
  at 0x2 didn't dereference TrackerModuleMetadata's members very well).
 
  Please review and/or rework the patch.
 
  --
  Philip Van Hoof, freelance software developer
  home: me at pvanhoof dot be
  gnome: pvanhoof at gnome dot org
  http://pvanhoof.be/blog
  http://codeminded.be
 
 
-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] [Evolution] Beagle and Tracker, letting Evolution feed those beasts RDF triples instead

2008-12-10 Thread Philip Van Hoof
On Wed, 2008-12-10 at 11:12 +, Michael Meeks wrote:
 Hi Philip,
 
 On Tue, 2008-12-09 at 19:59 +0100, Philip Van Hoof wrote:
   http://live.gnome.org/Evolution/Metadata
  
  For early visitors of that page, refresh because I have added/changed
  quite a lot of it already.
 
   Looks really good.
 
   The only thing that I don't quite understand (the perennial problem
 with asynchronous interfaces), is the memory issue: it seems we need to
 store all Unset information on deleted mails somewhere [ unless you are
 a womble like me that keeps ~all mail forever ;-].
 
   What does the lifecycle for the data in that Unset store look like ?
 [ I assume that as/when you re-connect to the service you're as much
 likely to get an UnsetMany as a SetMany ]. What if that data starts to
 grow larger than the remaining data it describes ? ;-) [ depending on
 how we do Junk mail filtering of course that might be quite a common
 occurrence for some ].

I think the LifeCycle is best described by this document:

http://live.gnome.org/MetadataOnRemovableDevices

It specifies a metadata cache format for removable devices in Turtle
format. 

For your information when reading the document: The removal of a
resource as a special notation using blank resources  , and the
removal of a predicate (of a field of a resource) uses the notation
pfx:predicate .

Although cached metadata on a removable device is not the exact same
use-case, the life-cycle of what the RDF store (or the metadata engine)
wants is the same:

- When a new resource is created or one of its predicates (one of its
  fields) is being updated, it just wants to know about these updates or
  creates. An update is the same as a create if the resource didn't
  exist before.

  For a cache it's important to know the modified timestamp so that
  you know whether your copy of the metadata is most recent, or the
  cache is about the resource is most recent.

  For Evolution (for E-mail clients) we can simplify this as whenever a
  Set or a SetMany happens, we assume time() to be that date. That's
  because we can assume the E-mail client to have top-most priority in
  all cases (being the benevolent dictator about metadata about E-mails,
  it knows best what we should swallow and when we should swallow its
  updates - we should not make up our own minds and decisions about it)

- When a resource got deleted then the RDF store wants to know about
  this as soon as possible. Asynchronously (like if the RDF store,
  being a subscriber, joins the subscription after the deletion took
  place) this also counts: as soon as possible. Preferably immediately
  after the subscription.

  Right now I don't think Evolution is keeping state about deleted UIDs

  With IMAP there's a trick that you can do: you can assume that a hole
  in the UIDSET meant that some sort of deleting occurred. That's
  because IMAP is ~ specified that the server can't reuse UIDs (some
  IMAP servers might not respect this, and those are also broken in
  Evolution afaik - or at least require a workaround that makes
  Evolution basically perform like a POP client for IMAP when
  synchronizing -)

  With POP I don't think you can make any such assumptions.

- Removing the predicate from a resource (the field of a resource) ain't
  needed for E-mail. Luckily E-mail is a mostly read-only storage. With
  exception of fields like nmo:isRead. Maybe if we want to support
  removing a flag or a custom-flag at some point we might need to add
  something to the API to indicate the removal of a field of a resource.

  For example it's not possible that the CC or the TO list of an E-mail
  changes. Because E-mails, once stored, are read-only in that aspect.


I think, anyway, that it would make sense for Evolution to start doing
two things in the CamelDB:

  * Log all deletions (just the UID should suffice), if the service
reuses UIDs then upon effective reuse of the UID, this log's UID
deletion should be removed from the log. Else you loose the E-mail
at whoever depends on this log for knowing about effective
deletions.

 * Record the timestamp for each record in the summary table. This
   timestamp would store the time() when the record got added and maybe
   would also store the time() (preferably separately) when the last
   time the E-mail's flags got changed was.

With those two additions to the schema of the CamelDB it would I think
be possible to make a plugin that implements the service as proposed on
the wiki page.

Matthew Barnes replied on IRC that we should start storing those
timestamps anyhow. I also think it's a good idea. I was planning to
discuss this with psankar and srag too.

If we'd change the schema then we will also need to implement a
migration path from the old schema to the new.

Using virtual tables you can simulate MySQL's ALTER TABLE in SQLite.

TRANSACTION 

SELECT * FROM orig_table INTO virtual_table;

DROP orig_table;

CREATE orig_table (
  ...
  created

Re: [Evolution-hackers] [Evolution] Beagle and Tracker, letting Evolution feed those beasts RDF triples instead

2008-12-09 Thread Philip Van Hoof
On Tue, 2008-12-09 at 18:00 +0530, Sankar wrote:

Hey Sankar,

I'm writing a plugin that will implement the Manager class as
described here. Tracker will then implement being a Registrar.

http://live.gnome.org/Evolution/Metadata

I will be using camel-db.h as you hinted me on IRC to implement the
features in a well performing way (direct SQLite access).

I will start working on this plugin tomorrow or next week. At the same
time I will be implementing support for it in Tracker, which will serve
as a prototype for other metadata engines.

I hope to inspire people from the Evolution team, and from the different
metadata engines, to comment on the proposed D-Bus API.

Let's try to get this valuable metadata out of those damn E-mail clients
and let's try to get it right this time. Not ad-hoc, but right.

If the namespace should be translated from org.gnome to org.freedesktop
we can of course do this afterwards. The metadata.Manager part would
also have to be renamed to a better name. But in the end implementation
would not be affected a lot by such renames. Meanwhile we can prototype
it in GNOME's Evolution D-Bus namespace.

The reason for all this prototyping is that we wouldn't like to release
a Tracker that doesn't support Evolution's new summary format.

This time we're into getting it right so just hacking around the new
summary format by fixing something that wrongfully interpreted
Evolution's cache by itself instead of letting Evolution tell us about
it ... 

* Well we could do this, but really ... let's just get it right now that
  I can spend time on this. At least that's my point of view on this.

* Other apps trying to read Evolution's caches externally just isn't
  ever going to be generic for all E-mail clients, and is not really
  right. For example file locking and (now that it's SQLite based)
  caring about transactions being held by Evolution and all that stuff.
  Caring about the possibility of Evolution changing the database
  schema.

* It's just not very nice to do it that way in my opinion: It adds a
  unasked for burden on the Evolution team too: having to negotiate with
  us when you want to change the schema of the database. Else you will
  break a lot of people's desktops unannounced. Evolution would need to
  make a mechanism for us to tell us about the version of the schema,
  for example. And we would have to implement things in Tracker that
  deal with all versions of Evolution's cache versions.

  One big spaghetti mess distributed over multiple projects.

  So, let's just do it right 


 On Mon, 2008-12-08 at 18:59 +0100, Philip Van Hoof wrote:
  All metadata engines are nowadays working on a method to let them get
  their metadata fed by external applications.
  Such APIs come down to storing RDF triples. A RDF triple comes down to a
  URI, a property and a value.
  
  For example (in Turtle format, which is SparQL's inline format and the
  typical w3's RDF storage format):
  We'd like to make an Evolution plugin that does this for Tracker. 
  
  Obviously would it be as easy as letting software like Beagle become an
  implementer of prox's InsertRDFTriples to start supporting Beagle with
  the same code and Evolution plugin, this way.
  
  I just don't know which EPlugin hooks I should use. Iterating all
  accounts and foreach account all folders and foreach folder all
  CamelMessageInfo instances is trivial and I know how to do this.
  
  What I don't know is what reliable hooks are for:
  
* Application started
 
 org.gnome.evolution.shell.events:1.0 - es-event.c - 
 
 sample plugin:
 groupwise-account-setup/org-gnome-gw-account-setup.eplug.xml 
 
 
* Account added
 
 org.gnome.evolution.mail.config:1.0 
 
 sample plugin:
 groupwise-account-setup/org-gnome-gw-account-setup.eplug.xml 
 
 For account-added: id = org.gnome.evolution.mail.config.accountDruid
 For account-edited: id = org.gnome.evolution.mail.config.accountEditor
 
* Account removed
 
 You may have to write a new hook
 
* Folder created
* Folder deleted
* Folder moved
* Message deleted (expunged)
* Message flagged for removal 
* Message flagged as Read and as Unread
* Message flagged (generic)
* Message moved (ie. deleted + created)
* New message received
  * Full message 
  * Just the ENVELOPE
  
 
 If you try to update your metadata for every of the above operations, it
 may be a overkill in terms of performance (and I believe more disk
 access as well for updating your metadata store). You can add a new hook
 while any change is made to the summary DB and listen to that. All the
 above changes will have to eventually come to summary DB for them to be
 valid.
 
 
 However, I personally believe:
 
 More and more applications are using sqlite (firefox and evolution my
 two most used apps.)  So, it may be a better idea to directly map the
 tables in an sqlite database into the search applications' data-store
 (beagle, tracker etc.) instead of depending

Re: [Evolution-hackers] [Evolution] Beagle and Tracker, letting Evolution feed those beasts RDF triples instead

2008-12-09 Thread Philip Van Hoof
On Tue, 2008-12-09 at 13:59 +0100, Philip Van Hoof wrote:
 On Tue, 2008-12-09 at 18:00 +0530, Sankar wrote:
 
 Hey Sankar,
 
 I'm writing a plugin that will implement the Manager class as
 described here. Tracker will then implement being a Registrar.
 
 http://live.gnome.org/Evolution/Metadata

For early visitors of that page, refresh because I have added/changed
quite a lot of it already.

This wiki page also serves as the description of the proposal. 

A experience developer should get a quite good idea of what will be
needed in Evolution:

Keeping timestamps around foreach message so that I can do a variation
of camel_db_read_message_info_records that accepts a since timestamp.

For example:

camel_db_message_infos_that_changed_since (db, since, callback, userd)

Something less easy is keeping track of deleted ones too. This would be
needed for the Unset and UnsetMany calls. Only thing that has to be
kept around is the UID. With direct access to IMAP I could implement
this in IMAP by searching for holes in the UID sets. For POP there's no
real other way than to just store all the UIDs or long-uids that ever
got expunged/popped/deleted and then locally deleted.

This is of course important for accurately cleaning up metadata engines
that want to accurately be aware of removed resources (removed E-mails).

This was also a painful part back when we manually parsed the summary
files: we had to scan all existing items to check if it's not in the
original summary file any longer. This meant having to parse-all,
scan-all, process-all each time we start up.

Not very nice for desktop-startup time :-(

Especially on mobile you want fast startup time and as few things as
possible to do before becoming operational. A metadata engine is of
course not good if it still has inaccuracies like metadata about data
that has long been removed in its stores.

Anyway, Evolution can easily log this as it's either updated by IMAP's
IDLE, unsolicited EXPUNGE events or NOTIFY, or by its synchronization
with POP or Evolution was the responsible for deleting the E-mail (or
apply this logic to E-mail protocol super X and E-mail protocol mega Y).


Let me know what you guys think ...

-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] [Fwd: [Evolution] Beagle and Tracker, letting Evolution feed those beasts RDF triples instead]

2008-12-09 Thread Philip Van Hoof
Forwarding this E-mail to the hackers mailing list.

I just noticed that I mailed this to the Evolution user's mailing list
by accident instead of the Hackers mailing list.

-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be
---BeginMessage---
All metadata engines are nowadays working on a method to let them get
their metadata fed by external applications.

For example the Xesam team at Freedesktop are putting together an API
like this: http://xesam.org/main/XesamMetadataAPI

Such APIs come down to storing RDF triples. A RDF triple comes down to a
URI, a property and a value.

For example (in Turtle format, which is SparQL's inline format and the
typical w3's RDF storage format):

imap://[EMAIL PROTECTED]/INBOX/001 nmo:to [EMAIL PROTECTED] ;
nmo:messageSubject RE: Hello World ;
... ;
nmo:from [EMAIL PROTECTED] .

Usually the API goes like this:

void
InsertRDFTriples (const char *subject, 
 const char **predicates, 
 const char **values);

For example:

static void
my_code (void) {
char **predicates = malloc0 (sizeof (char*) * 4);
char **values = malloc0 (sizeof (char*) * 4);
char *subject = imap://[EMAIL PROTECTED]/INBOX/001;

predicates[0] = nmo:to;
values[0] = [EMAIL PROTECTED];

predicates[1] = nmo:messageSubject;
values[1] = \RE: Hello World\;

predicates[2] = nmo:from;
values[2] = [EMAIL PROTECTED];

dbus_g_proxy_begin_call (prox,
 InsertRDFTriples,
 on_finished,
 NULL,
 NULL,
 G_TYPE_STRING, subject,
 G_TYPE_STRV, predicates,
 G_TYPE_STRV, values,
 G_TYPE_INVALID);

free (predicates);
free (values);
}

We'd like to make an Evolution plugin that does this for Tracker. 

Obviously would it be as easy as letting software like Beagle become an
implementer of prox's InsertRDFTriples to start supporting Beagle with
the same code and Evolution plugin, this way.

I just don't know which EPlugin hooks I should use. Iterating all
accounts and foreach account all folders and foreach folder all
CamelMessageInfo instances is trivial and I know how to do this.

What I don't know is what reliable hooks are for:

  * Application started
  * Account added
  * Account removed
  * Folder created
  * Folder deleted
  * Folder moved
  * Message deleted (expunged)
  * Message flagged for removal 
  * Message flagged as Read and as Unread
  * Message flagged (generic)
  * Message moved (ie. deleted + created)
  * New message received
* Full message 
* Just the ENVELOPE

Note that the EPlugin documentation is by far not detailed enough to
distill these hooks out of that. 

Who can help me with this? Where can I find a list? What source code
files do I have to crunch to know this? Will I have to read all of
the .c files in mail/ ?

-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be

___
Evolution-list mailing list
[EMAIL PROTECTED]
http://mail.gnome.org/mailman/listinfo/evolution-list

---End Message---
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] camel_write's IO_TIMOUT, sounds strange to me

2008-05-21 Thread Philip Van Hoof
Hi there, 

It looks like for each write, Evolution just waits IO_TIMEOUT to decide
whether or not it succeeded. IO_TIMEOUT is (60*4) (camel-file-utils.c)
or specific per kind of CamelTcpStream (usually similar values seem to
be used).

Any transfer that takes more than four minutes seems to be treated as a
timeout. Or, that is the behaviour that we observed. Over slow bandwidth
connections like GPRS, that doesn't make a lot of sense. I don't think
an arbitrary amount of seconds is never a good estimate if you send
entire messages with this timeout:

  Let's take for example IMAP's APPEND.

  Evolution's default IMAP code's do_append sends the entire ba-data
  for the entire ba-len to the IMAP continuation, which results in one
  single camel_stream_write, which results in one single camel_write
  which results in one single write() of ba-len. This ba-len is the
  entire size of the entire E-mail (all its MIME parts, including its
  large attachments). 

  That write() is non-blocking and a select() guards over this arbitrary
  IO_TIMEOUT to decide whether or not the (large) write timed-out.

Or, I misunderstood something about the camel_write() function ;-)
(which is not unlikely, I frequently misunderstand code).

Anyway ... with this patch we write in chunks of maximum 64 bytes with a
timeout of 15 seconds per chunk. We decided that 64 bytes doesn't add
too much latency, that 64 bytes is more than the usual IMAP command's
size (so no splitting up in chunks occurs for typical IMAP commands) and
that if 64 bytes didn't succeed in 15 seconds ... something is wrong
anyway (even over GPRS, although it's not impossible that it's really
just that slow).

But I'm not sure ... whether this is the right thing to do.

http://tinymail.org/trac/tinymail/changeset/3678

So I'm adding Camel experts in CC, in the hopes that they will de or
un-obfuscate camel_write() for me ;-). Just waiting 4 minutes and then
making a blunt decision that a write (fd, data, 8MB) is timed out,
if it's not finished yet, sounds strange to me.


-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Error storing summary.

2008-05-10 Thread Philip Van Hoof

And I said before many times that 600MB files is pure insanity to begin
with. Writing them takes a huge amount of time, disk-space (if you want
to use an atomic rename) and the smallest corruption in the filesystem
will destroy an entire mailbox (instead of just one file).

Evolution should switch to using Maildir for its local folders.

A filesystem managed by the kernel is a better kind of database than any
kind of one-file based database Evolution developers are going to come
up with, including their MBox code, unless they switch to SQLite or
MySQL Embedded or something like that (but then still, that's pointless
if you will only store file-like blobs anyway).

For example wouldn't deleting (expunging) items require a rewrite of
your 600MB file. Just one unlink() per expunged item.

If tml finds problems using Maildir on FAT32, use in stead of ':' the
character '!' in the filenames of items in tmp, cur and new. This has
been tested with several E-mail clients and solutions already, indeed.

I have voiced this a few times already by the way.


On Fri, 2008-05-09 at 23:57 +0200, Andre Klapper wrote:
 OK, so i managed to fuck up my POP Inbox for the third time in a few
 weeks, means: every time i switch to another folder i now get an Error
 storing summary (or something like that). i've seen some reports about
 this in bugzilla, e.g.
 http://bugzilla.gnome.org/show_bug.cgi?id=532049 . the last time i ran
 into this psankar told me to manually edit the mbox file and remove the
 offending email that misses the From line. this is not fun anymore when
 having a 600MB inbox file. a normal user is not willing to learn vim and
 emacs, me neither. if nobody's working on a fix i should consider
 switching to thunderbird.
 
 andre
-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] [Fwd: (Patch) Add support for LibCST to tinymail camel]

2008-04-09 Thread Philip Van Hoof
Hey Jeffrey, Matthew,

And Jose 

This (forwarded message) might be interesting for you guys too: it's
another implementation of camel-certdb.c, but this time for Maemo's
libcst.

I think we should consider thinking about abstracting this a bit better,
so that it's more convenient to provide an all new implementation or to
plug it in.

There is, for example, a GSoc student interesting in doing Seahorse
integration in Evolution. Guessing his work too will overlap.

Mozilla has its own key store, Maemo has its own key store, Seahorse has
a key store, Devices X and Devices Y will probably also have a key store
that is controlled by their specialised UI and that comes with default
installed fingerprints. For example for GMail IMAP, etc.

In my opinion is key signing and storing is more than just putting files
in $HOME/.camel_cert, it's something that depends on the platform a lot.

I would like to see Camel becoming less glued to Evolution, and this is
one of those items that does glue it at this moment.

My personal opinion on this ...

.. is that not Camel but the layer above Camel should solve it (with
perhaps a default solution). For example an Evolution plugin would solve
it in Evolution, or if there's no such plugin default code would.

In Tinymail it would be implementing a TnyCertificateHandler and doing
something like:

TnyAccount *acc = ...

tny_account_set_certificate_handler ( acc, 
tny_seahorse_certificate_handler_new ());

Any unmodified TnyAccount would be constructed with:

priv-cert_handler = new tny_default_certificate_handler_new ();

This way an E-mail client like Modest could either go for one of
Tinymail's featured certificate handlers, subclass those, or implement a
new one from scratch:

TnyAccount *acc = tny_..._modest_account_new ( ... )

tny_account_set_certificate_handler ( acc, 
modest_cst_certificate_handler_new ());

Modest's developer would implement a ModestCstCertificateHandler class
in Modest's layer.

For Camel(-lite) this would mean slashing CamelCertDb and in stead
making an implementation that expects the layer above to fulfil the
current functions (callbacks or a GInterface).

I would of course prefer this to go in Camel upstream rather than yet
again maintaining a large delta of changes between camel-lite and camel
upstream.

Ideas, thoughts?

-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be



---BeginMessage---
Hi,

This patch adds support for using CST as certificate repository in
camel (tinymail camel backend). CST is the common certificate repository
available in Nokia maemo-based internet tablets.

It's implemented as a fast hack to avoid a big rework of the currently
working implementation.

Of course, it seems all the certdb stuff should be refactored to allow
multiple backends (CST, file certdb, mozilla, whatever). But this adds
provisional support.

About autotools stuff, it only enables the cst support if available and
you use nss ssl implementation (I didn't implement the openssl side
support for camel-certdb-cst, so I'm not sure if it's working).

Changelog entry:
* Added libtinymail-camel/camel-lite/camel/camel-certdb-cst.c.
  Implementation of certificates database using maemo libcst. This
  makes you can manage certificates using the maemo tablet certificate
  manager.
* libtinymail-camel/camel-lite/camel/camel-certdb.h,
  libtinymail-camel/camel-lite/camel/camel-private.h: modified to
  have a pointer to the CST object in certdb and certificate id in
  certificate object in case we compile with CST support.
* libtinymail-camel/camel-lite/camel/camel-tcp-stream-ssl.c:
  Modified a bit the code for proper integration with cst (mainly
  removing direct access to internal fields).
* libtinymail-camel/camel-lite/configure.ac,
  libtinymail-camel/camel-lite/camel/Makefile.am:
  Added support for detection and configuration of libcst.

-- 
José Dapena Paz [EMAIL PROTECTED]
Igalia
Index: m4/gtk-doc.m4
===
--- m4/gtk-doc.m4	(revision 3558)
+++ m4/gtk-doc.m4	(working copy)
@@ -9,22 +9,22 @@
   AC_BEFORE([AC_PROG_LIBTOOL],[$0])dnl setup libtool first
   AC_BEFORE([AM_PROG_LIBTOOL],[$0])dnl setup libtool first
   dnl for overriding the documentation installation directory
-  AC_ARG_WITH([html-dir],
-AS_HELP_STRING([--with-html-dir=PATH], [path to installed docs]),,
+  AC_ARG_WITH(html-dir,
+AC_HELP_STRING([--with-html-dir=PATH], [path to installed docs]),,
 [with_html_dir='${datadir}/gtk-doc/html'])
   HTML_DIR=$with_html_dir
-  AC_SUBST([HTML_DIR])
+  AC_SUBST(HTML_DIR)
 
   dnl enable/disable documentation building
-  AC_ARG_ENABLE([gtk-doc],
-AS_HELP_STRING([--enable-gtk-doc],
-   [use gtk-doc to build documentation [[default=no]]]),,
-[enable_gtk_doc=no])
+  AC_ARG_ENABLE(gtk-doc,
+AC_HELP_STRING

[Evolution-hackers] Content-Disposition for images in the signature

2008-03-16 Thread Philip Van Hoof
Hi there,

When you add an image to your HTML signature, it'll make the
content-disposition attachment and it'll set the filename header.

Both actions are incorrect: the content-disposition is inline and
there's no need to set the filename header. Both will make E-mail
clients like Outlook, but also Evolution itself, think that the E-mail
contains an attachment (a file attachment).

I have seen Evolution do this wrong for all kinds of inline embedded
images, whenever you insert this into your HTML document. This is
incorrect behaviour and not conform MIME.

ps. For a free software E-mail client, I think the better option is to
go with the specifications. That Outlook gets things wrong is not a good
excuse. Although I think modern E-mail clients like Outlook are getting
this right nowadays.


-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] The -- separator for signatures

2008-03-16 Thread Philip Van Hoof
Hi there,

Although perhaps not a written standard, although I'm confident somebody
once wrote this down, are E-mail clients who do it right adding -- \n
in front of the signature. This way can receiving E-mail clients and
mailing lists do something with the signature (for example, moving the
personal signature lower, adding advertising or legal warnings).

Evolution is getting this right for text/plain signatures, but not for
text/html ones. For the text/plain signature you don't need to add any
such characters, Evolution will do this for you. For the text/html
signature, you do (inconsistency).

For E-mails where you send both the text/plain and the text/html MIME
parts, you therefore can't really define it for your E-mails. Unless you
change the text/html signature to contain this -- \n sequence and not
add it to the text/plain. Or if you don't want it ... well you then
can't really define the desired behaviour for the text/plain one
(inconsistency++ + confusion).

In any case, it's not consistent. Therefore confusing, and in my opinion
therefore incorrect behaviour.

ps. Since in HTML \n is not visible, I'd use -- \nbr/

-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] The -- separator for signatures

2008-03-16 Thread Philip Van Hoof
Done

http://bugzilla.gnome.org/show_bug.cgi?id=522784
http://bugzilla.gnome.org/show_bug.cgi?id=522783

On Sun, 2008-03-16 at 10:43 -0400, Matthew Barnes wrote:
 On Sun, 2008-03-16 at 14:46 +0100, Philip Van Hoof wrote:
  Evolution is getting this right for text/plain signatures, but not for
  text/html ones. For the text/plain signature you don't need to add any
  such characters, Evolution will do this for you. For the text/html
  signature, you do (inconsistency).
 
 Philip, please file bugs for this and the content-disposition issue you
 described in your last post.  It's really hard to track issues on a
 mailing list.
 
 Matthew Barnes
 
 ___
 Evolution-hackers mailing list
 Evolution-hackers@gnome.org
 http://mail.gnome.org/mailman/listinfo/evolution-hackers
-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Loading really large E-mails on devices with not enough Vm

2008-03-04 Thread Philip Van Hoof

On Sat, 2008-01-26 at 23:22 -0500, Jeffrey Stedfast wrote:
 Something like the attached patch might work, tho it is untested.

I had to change 

else if (!CAMEL_IS_SEEKABLE_SUBSTREAM (stream))

into

else if (!CAMEL_IS_SEEKABLE_STREAM (stream))

I don't know why you where testing for substream, as substream provides
no extra functionality that seems to be related here ...

 So my guess is that this will break the parser :(
 
 It might break in the stream case as well, you'd have to follow the code
 paths a bit to know for sure. For instance, even if creating the
 seekable substream doesn't perform an underlying seek on the original
 stream, setting it in a data wrapper might call camel_stream_reset()
 which /might/ do an lseek() on the source fs stream.

The problem with the patch is that it makes each MIME part's data start
at the headers, in stead of at the actual content.

I tried determining the start right after the first call to
camel_mime_parser_step but that just resulted in start == end.


 Not an insurmountable problem to solve, but it does make things a little
 more difficult and possibly touchy.

 
 
 On Sat, 2008-01-26 at 22:48 -0500, Jeffrey Stedfast wrote:
  On Sat, 2008-01-26 at 22:12 -0500, Jeffrey Stedfast wrote:
   On Sat, 2008-01-26 at 13:44 +0100, Philip Van Hoof wrote:
This is what happens if you try to open a truly large E-mail on a device
that has not as much memory available:

Is there something we can do about this? Can we change the MIME parsing
algorithm to be less memory demanding for example?

Note that GArray is not really very sparse with memory once you start
having a really large array. Perhaps we can in stead change this to a
normal pointer array of a fixed size (do we know the size before we
start parsing, so that we can allocate an exact size in stead, perhaps?)
   
   eh, why would you change it to a GPtrArray? It doesn't hold pointers, it
   holds message part content.
   
   Unfortunately we don't know the size ahead of time.
   
   I suppose you could use a custom byte array allocator so that you can
   force it to grow by larger chunks or something, dunno.
  
  
   The way GMime handles this is by not loading content into RAM, but 
   that may be harder to do with Camel, especially in the mbox case.
  
  er, I should probably explain this:
  
  - writing the code should be relatively easy to do, but in the mbox
  case, the mbox may end up getting expunged or rewritten for some other
  reason which may cause problems, not sure how that would work.
  
  I think in Maildir, as long as the fd remains open, the file won't
  actually disappear after an unlink() until the fd gets closed, so that
  might work out ok assuming you can spare the fd (which might be the
  other problem with Evolution?).
  
  Jeff
  
  
  ___
  Evolution-hackers mailing list
  Evolution-hackers@gnome.org
  http://mail.gnome.org/mailman/listinfo/evolution-hackers
 ___
 Evolution-hackers mailing list
 Evolution-hackers@gnome.org
 http://mail.gnome.org/mailman/listinfo/evolution-hackers
-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Loading really large E-mails on devices with not enough Vm

2008-03-04 Thread Philip Van Hoof

Hey Jeffrey,

I did some experimenting and after this it seems to work:

http://tinymail.org/trac/tinymail/changeset/3462

I had to get the value of folder_tell at the exact location where the
state is at CAMEL_MIME_PARSER_STATE_HEADER.

Then it worked.

I tested this against for example a testing E-mail that is floating
around the Lemonade test servers of 32 MB, various simpler E-mails and
one of 7 MB. Both big E-mails mostly had image attachments.

Tinymail has a mimepart viewer that uses a pixbuf loader, and it
succeeded just fine in loading the images.

I have a message open that is 40246320 bytes in size, this is my VmRSS
for Tinymail's demoui. 

VmRSS: 16428 kB

Those 16Mb is probably data in the GtkPixbuf and the summary.

lemonade.andrew.cmu.edu:143, testuser1, pass1 and pick the largest mail
on that server (~40 MB).


On Tue, 2008-03-04 at 18:40 +0100, Philip Van Hoof wrote:
 On Sat, 2008-01-26 at 23:22 -0500, Jeffrey Stedfast wrote:
  Something like the attached patch might work, tho it is untested.
 
 I had to change 
 
   else if (!CAMEL_IS_SEEKABLE_SUBSTREAM (stream))
 
 into
 
   else if (!CAMEL_IS_SEEKABLE_STREAM (stream))
 
 I don't know why you where testing for substream, as substream provides
 no extra functionality that seems to be related here ...
 
  So my guess is that this will break the parser :(
  
  It might break in the stream case as well, you'd have to follow the code
  paths a bit to know for sure. For instance, even if creating the
  seekable substream doesn't perform an underlying seek on the original
  stream, setting it in a data wrapper might call camel_stream_reset()
  which /might/ do an lseek() on the source fs stream.
 
 The problem with the patch is that it makes each MIME part's data start
 at the headers, in stead of at the actual content.
 
 I tried determining the start right after the first call to
 camel_mime_parser_step but that just resulted in start == end.
 
 
  Not an insurmountable problem to solve, but it does make things a little
  more difficult and possibly touchy.
 
  
  
  On Sat, 2008-01-26 at 22:48 -0500, Jeffrey Stedfast wrote:
   On Sat, 2008-01-26 at 22:12 -0500, Jeffrey Stedfast wrote:
On Sat, 2008-01-26 at 13:44 +0100, Philip Van Hoof wrote:
 This is what happens if you try to open a truly large E-mail on a 
 device
 that has not as much memory available:
 
 Is there something we can do about this? Can we change the MIME 
 parsing
 algorithm to be less memory demanding for example?
 
 Note that GArray is not really very sparse with memory once you start
 having a really large array. Perhaps we can in stead change this to a
 normal pointer array of a fixed size (do we know the size before we
 start parsing, so that we can allocate an exact size in stead, 
 perhaps?)

eh, why would you change it to a GPtrArray? It doesn't hold pointers, it
holds message part content.

Unfortunately we don't know the size ahead of time.

I suppose you could use a custom byte array allocator so that you can
force it to grow by larger chunks or something, dunno.
   
   
The way GMime handles this is by not loading content into RAM, but 
that may be harder to do with Camel, especially in the mbox case.
   
   er, I should probably explain this:
   
   - writing the code should be relatively easy to do, but in the mbox
   case, the mbox may end up getting expunged or rewritten for some other
   reason which may cause problems, not sure how that would work.
   
   I think in Maildir, as long as the fd remains open, the file won't
   actually disappear after an unlink() until the fd gets closed, so that
   might work out ok assuming you can spare the fd (which might be the
   other problem with Evolution?).
   
   Jeff
   
   
   ___
   Evolution-hackers mailing list
   Evolution-hackers@gnome.org
   http://mail.gnome.org/mailman/listinfo/evolution-hackers
  ___
  Evolution-hackers mailing list
  Evolution-hackers@gnome.org
  http://mail.gnome.org/mailman/listinfo/evolution-hackers
-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Loading really large E-mails on devices with not enough Vm

2008-01-27 Thread Philip Van Hoof
We'll try this, and if it works for all mails that we wanted to test,
I'll let you know.

Thanks a lot! Adding Modest's project manager in CC

On Sat, 2008-01-26 at 23:22 -0500, Jeffrey Stedfast wrote:
 Something like the attached patch might work, tho it is untested.
 
 If this doesn't work, then I suspect the problem is that the seek
 position might get changed out from under the mime parser (assuming it
 is using either a CamelStreamFs or an fd).
 
 Note that camel_stream_fs_new_with_fd[_and_bounds]() calls lseek() on
 the fd passed in.
 
 From the dup() man page:
 
After  a  successful  return from dup() or dup2(), the old and new file
descriptors may be used interchangeably.  They refer to the  same  open
file description (see open(2)) and thus share file offset and file sta‐
tus flags; for example,  if  the  file  offset  is  modified  by  using
lseek(2)  on one of the descriptors, the offset is also changed for the
other.
 
 So my guess is that this will break the parser :(
 
 It might break in the stream case as well, you'd have to follow the code
 paths a bit to know for sure. For instance, even if creating the
 seekable substream doesn't perform an underlying seek on the original
 stream, setting it in a data wrapper might call camel_stream_reset()
 which /might/ do an lseek() on the source fs stream.
 
 Not an insurmountable problem to solve, but it does make things a little
 more difficult and possibly touchy.
 
 Jeff
 
 
 
 On Sat, 2008-01-26 at 22:48 -0500, Jeffrey Stedfast wrote:
  On Sat, 2008-01-26 at 22:12 -0500, Jeffrey Stedfast wrote:
   On Sat, 2008-01-26 at 13:44 +0100, Philip Van Hoof wrote:
This is what happens if you try to open a truly large E-mail on a device
that has not as much memory available:

Is there something we can do about this? Can we change the MIME parsing
algorithm to be less memory demanding for example?

Note that GArray is not really very sparse with memory once you start
having a really large array. Perhaps we can in stead change this to a
normal pointer array of a fixed size (do we know the size before we
start parsing, so that we can allocate an exact size in stead, perhaps?)
   
   eh, why would you change it to a GPtrArray? It doesn't hold pointers, it
   holds message part content.
   
   Unfortunately we don't know the size ahead of time.
   
   I suppose you could use a custom byte array allocator so that you can
   force it to grow by larger chunks or something, dunno.
  
  
   The way GMime handles this is by not loading content into RAM, but 
   that may be harder to do with Camel, especially in the mbox case.
  
  er, I should probably explain this:
  
  - writing the code should be relatively easy to do, but in the mbox
  case, the mbox may end up getting expunged or rewritten for some other
  reason which may cause problems, not sure how that would work.
  
  I think in Maildir, as long as the fd remains open, the file won't
  actually disappear after an unlink() until the fd gets closed, so that
  might work out ok assuming you can spare the fd (which might be the
  other problem with Evolution?).
  

-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Loading really large E-mails on devices with not enough Vm

2008-01-27 Thread Philip Van Hoof
This is very strange, though. It looks like stream=0x0 but the
mime-parser's stream ain't NULL.

(gdb) print buffer
$1 = (GByteArray *) 0x80e4dc0
(gdb) print stream
$2 = (CamelStream *) 0x0
(gdb) print *mp
$3 = {parent = {klass = 0x80def80, hooks = 0x0, ref_count = 1, flags = 0}, priv 
= 0x8272770}
(gdb) print *mp-priv
$4 = {state = CAMEL_MIME_PARSER_STATE_BODY, outbuf = 0x827e800 
Content-Transfer-Encoding: quoted-printable, 
  outptr = 0x827e800 Content-Transfer-Encoding: quoted-printable, outend = 
0x827ec00 , fd = -1, stream = 0x826ab10, ioerrno = 0, 
  realbuf = 0x827ec08 , 
  inbuf = 0x827ec88 
ش\222�\177��\034\\\004�\225=�L\2365gke�-�\037p\024\233��\023\213~LJP~\225�/���O\002�Vtc\235gǦ�\215�\206\025-\231\*ӱ\232Nz\205\036\n�\223�2U�A\237%Qn,
 inptr = 0x827fc5c 
�~y[\017ʶ���\204\037�\213�l�Z�`Qh9\235\f�+�\224\024\\p���\n\226\y�5��\220\n, 
  inend = 0x827fc88 \n, atleast = 0, seek = 413944216, unstep = 0, midline = 
1, scan_from = 0, scan_pre_from = 0, eof = 0, 
  start_of_from = -1, start_of_boundary = 11048, start_of_headers = 11092, 
header_start = -1, filterid = 1, filters = 0x0, 
  parts = 0x828c210}
(gdb) print *mp-priv-stream
$5 = {parent_object = {klass = 0x80ffdc8, hooks = 0x0, ref_count = 2, flags = 
0}, eos = 0}
(gdb) 



#define _PRIVATE(o) (((CamelMimeParser *)(o))-priv)
CamelStream *
camel_mime_parser_stream (CamelMimeParser *parser)
{
struct _header_scan_state *s = _PRIVATE (parser);

return s-stream;
}

Maybe it's not a CamelSeekableSubstream? Else would parent_stream not be F?

(gdb) print * (CamelSeekableSubstream *)mp-priv-stream
$7 = {parent_object = {parent_object = {parent_object = {klass = 0x80ffdc8, 
hooks = 0x0, ref_count = 2, flags = 0}, eos = 0}, 
position = 413948312, bound_start = 0, bound_end = -1, some_stack = '\0' 
repeats 49 times}, parent_stream = 0x16}
(gdb) 


On Sun, 2008-01-27 at 13:38 +0100, Philip Van Hoof wrote:
 Looks like the GByteArray is still being created.
 
 (gdb) break camel-mime-part-utils.c:82
 Breakpoint 2 at 0xb6dd541e: file camel-mime-part-utils.c, line 82.
 (gdb) delete 1
 (gdb) cont
 Continuing.
 
 Breakpoint 2, simple_data_wrapper_construct_from_parser (dw=0xb3f02800, 
 mp=0x827bcd0) at camel-mime-part-utils.c:82
 82  if (buffer != NULL) {
 (gdb) print buffer
 $1 = (GByteArray *) 0x80e4dc0
 (gdb) 
 
 
 Breakpoint 1, camel_mime_parser_step (parser=0x827bcd0, 
 databuffer=0xb4882f3c, datalength=0xb4882f40) at camel-mime-parser.c:610
 610 struct _header_scan_state *s = _PRIVATE (parser);
 (gdb) bt
 #0  camel_mime_parser_step (parser=0x827bcd0, databuffer=0xb4882f3c, 
 datalength=0xb4882f40) at camel-mime-parser.c:610
 #1  0xb6dd5456 in simple_data_wrapper_construct_from_parser (dw=0xb3f02800, 
 mp=0x827bcd0) at camel-mime-part-utils.c:81
 #2  0xb6dd55e9 in camel_mime_part_construct_content_from_parser 
 (dw=0x824e8c8, mp=0x827bcd0) at camel-mime-part-utils.c:127
 #3  0xb6dd6ff1 in construct_from_parser (mime_part=0x824e8c8, mp=0x827bcd0) 
 at camel-mime-part.c:968
 #4  0xb6dd70af in camel_mime_part_construct_from_parser (mime_part=0x824e8c8, 
 mp=0x827bcd0) at camel-mime-part.c:996
 #5  0xb6de1aab in construct_from_parser (multipart=0x8246f80, mp=0x827bcd0) 
 at camel-multipart.c:577
 #6  0xb6de1bea in camel_multipart_construct_from_parser (multipart=0x8246f80, 
 mp=0x827bcd0) at camel-multipart.c:609
 #7  0xb6dd5681 in camel_mime_part_construct_content_from_parser 
 (dw=0x8254570, mp=0x827bcd0) at camel-mime-part-utils.c:144
 #8  0xb6dd6ff1 in construct_from_parser (mime_part=0x8254570, mp=0x827bcd0) 
 at camel-mime-part.c:968
 #9  0xb6dd1de4 in construct_from_parser (dw=0x8254570, mp=0x827bcd0) at 
 camel-mime-message.c:597
 #10 0xb6dd70af in camel_mime_part_construct_from_parser (mime_part=0x8254570, 
 mp=0x827bcd0) at camel-mime-part.c:996
 #11 0xb6dd7122 in construct_from_stream (dw=0x8254570, s=0x826ab10) at 
 camel-mime-part.c:1012
 #12 0xb6dc2f63 in camel_data_wrapper_construct_from_stream 
 (data_wrapper=0x8254570, stream=0x826ab10) at camel-data-wrapper.c:270
 #13 0xb60fbd97 in maildir_get_message (folder=0x80def28, uid=0x8269dd0 
 1192085835.11467_1.evergrey, 
 
 
 [EMAIL PROTECTED]:~/Current/mailtests/md/spam1/cur$ ls -alh 
 1192085835.11467_1.evergrey\!2\,SH 
 -rw-r--r-- 1 pvanhoof pvanhoof 401M 2008-01-27 13:28 
 1192085835.11467_1.evergrey!2,SH
 [EMAIL PROTECTED]:~/Current/mailtests/md/spam1/cur$ 
 
 
 
 On Sat, 2008-01-26 at 23:22 -0500, Jeffrey Stedfast wrote:
  Something like the attached patch might work, tho it is untested.
  
  If this doesn't work, then I suspect the problem is that the seek
  position might get changed out from under the mime parser (assuming it
  is using either a CamelStreamFs or an fd).
  
  Note that camel_stream_fs_new_with_fd[_and_bounds]() calls lseek() on
  the fd passed in.
  
  From the dup() man page:
  
 After  a  successful  return from dup() or dup2(), the old and new 
  file
 descriptors may be used interchangeably

Re: [Evolution-hackers] Loading really large E-mails on devices with not enough Vm

2008-01-27 Thread Philip Van Hoof
Looks like the GByteArray is still being created.

(gdb) break camel-mime-part-utils.c:82
Breakpoint 2 at 0xb6dd541e: file camel-mime-part-utils.c, line 82.
(gdb) delete 1
(gdb) cont
Continuing.

Breakpoint 2, simple_data_wrapper_construct_from_parser (dw=0xb3f02800, 
mp=0x827bcd0) at camel-mime-part-utils.c:82
82  if (buffer != NULL) {
(gdb) print buffer
$1 = (GByteArray *) 0x80e4dc0
(gdb) 


Breakpoint 1, camel_mime_parser_step (parser=0x827bcd0, databuffer=0xb4882f3c, 
datalength=0xb4882f40) at camel-mime-parser.c:610
610 struct _header_scan_state *s = _PRIVATE (parser);
(gdb) bt
#0  camel_mime_parser_step (parser=0x827bcd0, databuffer=0xb4882f3c, 
datalength=0xb4882f40) at camel-mime-parser.c:610
#1  0xb6dd5456 in simple_data_wrapper_construct_from_parser (dw=0xb3f02800, 
mp=0x827bcd0) at camel-mime-part-utils.c:81
#2  0xb6dd55e9 in camel_mime_part_construct_content_from_parser (dw=0x824e8c8, 
mp=0x827bcd0) at camel-mime-part-utils.c:127
#3  0xb6dd6ff1 in construct_from_parser (mime_part=0x824e8c8, mp=0x827bcd0) at 
camel-mime-part.c:968
#4  0xb6dd70af in camel_mime_part_construct_from_parser (mime_part=0x824e8c8, 
mp=0x827bcd0) at camel-mime-part.c:996
#5  0xb6de1aab in construct_from_parser (multipart=0x8246f80, mp=0x827bcd0) at 
camel-multipart.c:577
#6  0xb6de1bea in camel_multipart_construct_from_parser (multipart=0x8246f80, 
mp=0x827bcd0) at camel-multipart.c:609
#7  0xb6dd5681 in camel_mime_part_construct_content_from_parser (dw=0x8254570, 
mp=0x827bcd0) at camel-mime-part-utils.c:144
#8  0xb6dd6ff1 in construct_from_parser (mime_part=0x8254570, mp=0x827bcd0) at 
camel-mime-part.c:968
#9  0xb6dd1de4 in construct_from_parser (dw=0x8254570, mp=0x827bcd0) at 
camel-mime-message.c:597
#10 0xb6dd70af in camel_mime_part_construct_from_parser (mime_part=0x8254570, 
mp=0x827bcd0) at camel-mime-part.c:996
#11 0xb6dd7122 in construct_from_stream (dw=0x8254570, s=0x826ab10) at 
camel-mime-part.c:1012
#12 0xb6dc2f63 in camel_data_wrapper_construct_from_stream 
(data_wrapper=0x8254570, stream=0x826ab10) at camel-data-wrapper.c:270
#13 0xb60fbd97 in maildir_get_message (folder=0x80def28, uid=0x8269dd0 
1192085835.11467_1.evergrey, 


[EMAIL PROTECTED]:~/Current/mailtests/md/spam1/cur$ ls -alh 
1192085835.11467_1.evergrey\!2\,SH 
-rw-r--r-- 1 pvanhoof pvanhoof 401M 2008-01-27 13:28 
1192085835.11467_1.evergrey!2,SH
[EMAIL PROTECTED]:~/Current/mailtests/md/spam1/cur$ 



On Sat, 2008-01-26 at 23:22 -0500, Jeffrey Stedfast wrote:
 Something like the attached patch might work, tho it is untested.
 
 If this doesn't work, then I suspect the problem is that the seek
 position might get changed out from under the mime parser (assuming it
 is using either a CamelStreamFs or an fd).
 
 Note that camel_stream_fs_new_with_fd[_and_bounds]() calls lseek() on
 the fd passed in.
 
 From the dup() man page:
 
After  a  successful  return from dup() or dup2(), the old and new file
descriptors may be used interchangeably.  They refer to the  same  open
file description (see open(2)) and thus share file offset and file sta‐
tus flags; for example,  if  the  file  offset  is  modified  by  using
lseek(2)  on one of the descriptors, the offset is also changed for the
other.
 
 So my guess is that this will break the parser :(
 
 It might break in the stream case as well, you'd have to follow the code
 paths a bit to know for sure. For instance, even if creating the
 seekable substream doesn't perform an underlying seek on the original
 stream, setting it in a data wrapper might call camel_stream_reset()
 which /might/ do an lseek() on the source fs stream.
 
 Not an insurmountable problem to solve, but it does make things a little
 more difficult and possibly touchy.
 
 Jeff
 
 
 
 On Sat, 2008-01-26 at 22:48 -0500, Jeffrey Stedfast wrote:
  On Sat, 2008-01-26 at 22:12 -0500, Jeffrey Stedfast wrote:
   On Sat, 2008-01-26 at 13:44 +0100, Philip Van Hoof wrote:
This is what happens if you try to open a truly large E-mail on a device
that has not as much memory available:

Is there something we can do about this? Can we change the MIME parsing
algorithm to be less memory demanding for example?

Note that GArray is not really very sparse with memory once you start
having a really large array. Perhaps we can in stead change this to a
normal pointer array of a fixed size (do we know the size before we
start parsing, so that we can allocate an exact size in stead, perhaps?)
   
   eh, why would you change it to a GPtrArray? It doesn't hold pointers, it
   holds message part content.
   
   Unfortunately we don't know the size ahead of time.
   
   I suppose you could use a custom byte array allocator so that you can
   force it to grow by larger chunks or something, dunno.
  
  
   The way GMime handles this is by not loading content into RAM, but 
   that may be harder to do with Camel, especially

Re: [Evolution-hackers] Loading really large E-mails on devices with not enough Vm

2008-01-27 Thread Philip Van Hoof

On Sun, 2008-01-27 at 11:27 -0500, Jeffrey Stedfast wrote:
 On Sun, 2008-01-27 at 13:44 +0100, Philip Van Hoof wrote:
  This is very strange, though. It looks like stream=0x0 but the
  mime-parser's stream ain't NULL.
 
 that just means the stream the parser is using is not a subclass of
 CamelSeekableSubstream

The parser was parsing one that was opened by the maildir implementation
of camel_folder_get_message. So it's a file in a maildir, so I guess we
can make a stream for that file inherit CamelSeekableSubstream, right?


-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] A different implementation of imap_rescan

2008-01-26 Thread Philip Van Hoof
Hi there,

The imap_rescan function of the default IMAP provider of Evolution will
in case at the front of the IMAP Mailbox items got expunged, more or
less start marking all of the locally cached summary info as invalid.

(note. summary in this context means the envelope information of each
item that we have cached locally in a file called summary and that,
when loaded, loads into a integer-based index. The sequence number on
the IMAP server minus one must correspond to that index for each item)

As a result will the entire summary be re-retrieved. This is one of the
many reasons why Evolution's IMAP feels slow for the users: they removed
something and suddenly the next operation takes a very long time. That's
the summary being re-retrieved ... sometimes.

Luckily wont most people delete old items, so it wont occur often for
most people's IMAP operations. Nevertheless it can occur.

I figured it's a bit too-aggressive about this (it = the code). It can
also assume that only a few items got removed, keep how many got
removed, and shift the table's indexes according to the amount of
removed items.

If then we still see a significant mismatch, we can still nonetheless
mark as invalid, remove and perform a re-retrieval of the summary.

I do this in Tinymail's camel-lite code, and now ported it to upstream
camel in the patch that I attached. I have not tested this porting and
note that I do quite a lot of things different in Tinymail (it supports
condstore, which is heavily related to this code).

But for code review this might be interesting to post to you guys, so
here it goes ...



-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be
Index: camel-imap-folder.c
===
--- camel-imap-folder.c	(revision 8395)
+++ camel-imap-folder.c	(working copy)
@@ -664,6 +664,7 @@
 	GArray *removed;
 	gboolean ok;
 	CamelFolderChangeInfo *changes = NULL;
+	gint tr = 0;
 
 	imap_folder-need_rescan = FALSE;
 
@@ -732,16 +733,17 @@
 	 * from the summary.
 	 */
 	removed = g_array_new (FALSE, FALSE, sizeof (int));
-	for (i = 0; i  summary_len  new[i].uid; i++) {
-		info = camel_folder_summary_index (folder-summary, i);
+	for (i = 0; i + tr  summary_len  new[i].uid; i++) {
+		info = camel_folder_summary_index (folder-summary, i + tr);
 		iinfo = (CamelImapMessageInfo *)info;
 
 		if (strcmp (camel_message_info_uid (info), new[i].uid) != 0) {
+			tr++;
 			camel_message_info_free(info);
 			seq = i + 1;
 			g_array_append_val (removed, seq);
 			i--;
-			summary_len--;
+			/* summary_len--; */
 			continue;
 		}
 
@@ -762,6 +764,7 @@
 
 		camel_message_info_free(info);
 		g_free (new[i].uid);
+		new[i].uid = NULL;
 	}
 
 	if (changes) {
@@ -772,6 +775,7 @@
 	seq = i + 1;
 
 	/* Free remaining memory. */
+	i = 0;
 	while (i  summary_len  new[i].uid)
 		g_free (new[i++].uid);
 	g_free (new);
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Diary replaying on IMAP accounts

2008-01-21 Thread Philip Van Hoof

On Sat, 2008-01-19 at 10:48 -0500, Jeffrey Stedfast wrote:
 This is why I started looking at dropping CamelDisco* and replacing all
 instances with CamelOffline* - tri-state is awful, dual-state is ftw.
 
 anyways, if your fix works, go for it.

Hey Jeffrey,

Do you think that this should be committed in upstream Camel too?

I'll pass the question to Matthew too. I'm not sure whether the function
should indeed always return TRUE in case of RESYNCING state. Perhaps
it's better to adapt the return value of the function and replace all
uses of it? Perhaps TRUE in case of RESYNCING is fine?

There are indeed multiple solutions to solve this.

Note that once the diary starts working, you'll have one or two small
bugs in the diary code too (a variable diary-folders becoming NULL at
some point, yet still being used somewhere).

I wonder whether these diaries ever worked? So suddenly making it work
might introduce new bugs in Evolution too.

Note that the behaviour is that the APPEND command will fail, Camel will
react to that failure by reconnecting and forgetting about the diary.

The result can be called data loss .. since the message that was to be
appended is now only locally available, and not appended remotely.

It's kinda hard to spot this bug, as it requires working with another
E-mail client (if you open the message, it'll just work since it's
cached locally, and it'll also be in the summary since the summary
supports temporary UIDs).

(sounds like severe to me)


 On Sat, 2008-01-19 at 15:13 +0100, Philip Van Hoof wrote:
  Hi there,
  
  When we connect with an IMAP service if we have moved messages while we
  where offline, the diary's replay function will be utilised.
  
  While this takes place, the store's disco connection state is
  CAMEL_DISCO_STORE_RESYNCING.
  
  The camel_disco_store_check_online seems to only care about this state
  being ONLINE. RESYNCING is not seen as being online. It seems that
  therefore commands like the APPEND one are failing:
  
  
  void
  camel_disco_diary_replay (CamelDiscoDiary *diary, CamelException *ex)
  {
  ...
  camel_folder_append (...)
  ... or ...
  camel_folder_transfer_messages_to (...)
  ...
  }
  
  
  imap_append_online (CamelFolder *folder, CamelMimeMessage *message,
  const CamelMessageInfo *info, char **appended_uid,
  CamelException *ex)
  {
  ...
  do_append (...)
  ...
  }
  
  static CamelImapResponse *
  do_append (CamelFolder *folder, CamelMimeMessage *message,
 const CamelMessageInfo *info, char **uid,
 CamelException *ex)
  {
  ...
  response = camel_imap_command (store, NULL, ex, APPEND %F%s%s {%d},
 folder-full_name, flagstr ?   : ,
 flagstr ? flagstr : , ba-len);
  g_free (flagstr);
  
  if (!response) {
  ...
  retry ... (but eventually)
  return NULL;
  ...
  }
  
  ...
  }
  
  
  CamelImapResponse *
  imap_read_response (CamelImapStore *store, CamelException *ex)
  {
  ...
  response-untagged = g_ptr_array_new ();
  while ((type = camel_imap_command_response (store, respbuf, ex))
 == CAMEL_IMAP_RESPONSE_UNTAGGED)
  g_ptr_array_add (response-untagged, respbuf);
  
  if (type == CAMEL_IMAP_RESPONSE_ERROR) {
  camel_imap_response_free_without_processing (store, response);
     return NULL;   -
  }
  ...
  }
  
  
  CamelImapResponseType
  camel_imap_command_response (CamelImapStore *store, char **response,
   CamelException *ex)
  {
  CamelImapResponseType type;
  char *respbuf;
  int len = -1;
  
  if (camel_imap_store_readline (store, respbuf, ex)  0) {
  CAMEL_SERVICE_REC_UNLOCK (store, connect_lock);
  ---return CAMEL_IMAP_RESPONSE_ERROR; --- this happens
  }
  
  ...
  }
  
  
  /* FIXME: please god, when will the hurting stop? Thus function is so
 fucking broken it's not even funny. */
  ssize_t
  camel_imap_store_readline (CamelImapStore *store, char **dest, 
  CamelException *ex)
  {
  CamelStreamBuffer *stream;
  char linebuf[1024] = {0};
  GByteArray *ba;
  ssize_t nread;
  
  g_return_val_if_fail (CAMEL_IS_IMAP_STORE (store), -1);
  g_return_val_if_fail (dest, -1);
  
  *dest = NULL;
  
  /* Check for connectedness. Failed (or cancelled) operations will
   * close the connection. We can't expect a read to have any
   * meaning if we reconnect, so always set an exception.
   */
  
  if (!camel_imap_store_connected (store, ex))
  -return -1;  ---
  
  ...
  }
  
  
  gboolean
  camel_imap_store_connected (CamelImapStore *store, CamelException *ex)
  {
  ... (lot's of completely funny looking code, but eventually

Re: [Evolution-hackers] Diary replaying on IMAP accounts

2008-01-21 Thread Philip Van Hoof

On Mon, 2008-01-21 at 10:45 -0500, Jeffrey Stedfast wrote:
 On Mon, 2008-01-21 at 13:17 +0100, Philip Van Hoof wrote:

[CUT]
 
  (sounds like severe to me)
  
 
 To be honest, I have no idea what the side effects of changing that code
 to return TRUE for the RESYNCING state are. That's one of the problems
 with a tri-state that isn't really a tri-state :\

Agree

 As you mentioned, having it return FALSE is clearly not correct and
 returning TRUE may introduce some bugs :\
 
 As far as I know, there are some bugs regarding offline usage not
 resyncing properly when going back online, so you may have found the
 cause.

Perhaps, yes.

 I'll let the current maintainers make a judgment call on whether to
 accept this into mainline Camel or not (looks like either way, there's
 gonna have to be some attention given to this area of code)

So Matthew, the ball is in your camp now :-)!

-- I'll ping Srinivasa by adding in CC too. --

 Sorry that I can't be any more concrete than that :(

No problem, I understand. It surprised me quite a bit too as I found
this one. I didn't expect the problem to be in that area of the code.

The bug report leading to this one was also vague :), our bug tester was
telling us that it's not right that he could move a message while the
client was offline. He expected the UI to block his action in stead.

So we defended that this is actually supported and that the message
should be stored remotely as soon as the client goes online.

We tested this to be sure of our claim, and we saw that the message was
not visible for other E-mail clients ... oeps


-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] Diary replaying on IMAP accounts

2008-01-19 Thread Philip Van Hoof
Hi there,

When we connect with an IMAP service if we have moved messages while we
where offline, the diary's replay function will be utilised.

While this takes place, the store's disco connection state is
CAMEL_DISCO_STORE_RESYNCING.

The camel_disco_store_check_online seems to only care about this state
being ONLINE. RESYNCING is not seen as being online. It seems that
therefore commands like the APPEND one are failing:


void
camel_disco_diary_replay (CamelDiscoDiary *diary, CamelException *ex)
{
...
camel_folder_append (...)
... or ...
camel_folder_transfer_messages_to (...)
...
}


imap_append_online (CamelFolder *folder, CamelMimeMessage *message,
const CamelMessageInfo *info, char **appended_uid,
CamelException *ex)
{
...
do_append (...)
...
}

static CamelImapResponse *
do_append (CamelFolder *folder, CamelMimeMessage *message,
   const CamelMessageInfo *info, char **uid,
   CamelException *ex)
{
...
response = camel_imap_command (store, NULL, ex, APPEND %F%s%s {%d},
   folder-full_name, flagstr ?   : ,
   flagstr ? flagstr : , ba-len);
g_free (flagstr);

if (!response) {
...
retry ... (but eventually)
return NULL;
...
}

...
}


CamelImapResponse *
imap_read_response (CamelImapStore *store, CamelException *ex)
{
...
response-untagged = g_ptr_array_new ();
while ((type = camel_imap_command_response (store, respbuf, ex))
   == CAMEL_IMAP_RESPONSE_UNTAGGED)
g_ptr_array_add (response-untagged, respbuf);

if (type == CAMEL_IMAP_RESPONSE_ERROR) {
camel_imap_response_free_without_processing (store, response);
   return NULL;   -
}
...
}


CamelImapResponseType
camel_imap_command_response (CamelImapStore *store, char **response,
 CamelException *ex)
{
CamelImapResponseType type;
char *respbuf;
int len = -1;

if (camel_imap_store_readline (store, respbuf, ex)  0) {
CAMEL_SERVICE_REC_UNLOCK (store, connect_lock);
---return CAMEL_IMAP_RESPONSE_ERROR; --- this happens
}

...
}


/* FIXME: please god, when will the hurting stop? Thus function is so
   fucking broken it's not even funny. */
ssize_t
camel_imap_store_readline (CamelImapStore *store, char **dest, CamelException 
*ex)
{
CamelStreamBuffer *stream;
char linebuf[1024] = {0};
GByteArray *ba;
ssize_t nread;

g_return_val_if_fail (CAMEL_IS_IMAP_STORE (store), -1);
g_return_val_if_fail (dest, -1);

*dest = NULL;

/* Check for connectedness. Failed (or cancelled) operations will
 * close the connection. We can't expect a read to have any
 * meaning if we reconnect, so always set an exception.
 */

if (!camel_imap_store_connected (store, ex))
-return -1;  ---

...
}


gboolean
camel_imap_store_connected (CamelImapStore *store, CamelException *ex)
{
... (lot's of completely funny looking code, but eventually): ...
camel_disco_store_check_online
...
}

gboolean
camel_disco_store_check_online (CamelDiscoStore *store, CamelException *ex)
{
-- Nothing here seems to accept the RESYNCING state as being online 
---

if (camel_disco_store_status (store) != CAMEL_DISCO_STORE_ONLINE) {
camel_exception_set (ex, CAMEL_EXCEPTION_SERVICE_UNAVAILABLE,
 _(You must be working online to 
   complete this operation));
return FALSE;
}

return TRUE;
}



note: I have fixed this in my version by doing this:


gboolean
camel_disco_store_check_online (CamelDiscoStore *store, CamelException *ex)
{

if (camel_disco_store_status (store) == CAMEL_DISCO_STORE_RESYNCING)
return TRUE;

if (camel_disco_store_status (store) != CAMEL_DISCO_STORE_ONLINE) {
camel_exception_set (ex, CAMEL_EXCEPTION_SERVICE_UNAVAILABLE,
 _(You must be working online to 
   complete this operation));
return FALSE;
}

return TRUE;
}



-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] Disabled code in Evolution has a bug

2008-01-15 Thread Philip Van Hoof
Hi there,

In Evolution this code is not used (imap_parse_namespace_response -
imap_namespace_decode), nonetheless it contains a bug.

If the delimiter for a namespace is NIL, the imap_next_word will search
for the first space. This ain't correct if there are no parameters to
parse. For example ((MyShared NIL)).

http://tinymail.org/trac/tinymail/changeset/3263#file2

I'm adding Jeffrey in CC because I think he once intended to use this
code to improve namespace support?

Camel-lite has gotten too specific in this area of the code: it supports
the shared and personal namespace by only asking for LSUB recursively
for those two namespaces, and LIST and LSUB for the personal namespace.

The namespace-line parsing, however, is recovered from things that I
found in existing (but not used) camel's imap code.

ps. I commented out the original line in camel-lite because the += 3
might not be the right fix (perhaps is fixing imap_next_word better).


Hf.


-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Memory leak in camel-imap-message-cache.c

2008-01-14 Thread Philip Van Hoof

On Mon, 2008-01-14 at 16:02 +0530, Srinivasa Ragavan wrote:
 Look at the free_part function being called at 
 g_hash_table_foreach (cache-parts, free_part, cache);
 
 It frees the key.

Aha. Okay thanks for the clarification.

I wonder, wouldn't it be better to use the standard infrastructure of
GHashTable to clearup keys? The full version of the GHashTable's
constructor has a callback function pointer parameter that'll be used
for freeing up keys like this.

I of course also wonder howcome my memory analysis tool pointed me to
this memory leak. Although it might be explained by one CamelMessage
reference still being open when the exit(0) took place.


 On Mon, 2008-01-14 at 02:58 +0100, Philip Van Hoof wrote:
  Hi there,
  
  The cache-parts = g_hash_table_new (g_str_hash, g_str_equal); of
  camel-imap-message-cache.c does not free its keys (it's the default way
  of creating a hashtable, so there's no freeup function provided for the
  keys).
  
  Yet at (or around) line 118 we see this:
  
  g_hash_table_insert (cache-parts, g_strdup (uid), subparts);
  
  That uid is string-copied. So either the hashtable needs a freeup
  function or the string should not be copied or .. this is wrong.
  
  Because I don't know how important the string copying is, what do you
  guys think we should do here?
  
  
 
-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Possible memory leak

2008-01-14 Thread Philip Van Hoof

I think I've found the leak ...

g_slist_remove_link:

Removes an element from a GSList, without freeing the element. The
removed element's next link is set to NULL, so that it becomes a
self-contained list with one element.

Notice the without freeing the element

Yet 

void
camel_operation_end (CamelOperation *cc)
{
...

g_free(s);
cc-status_stack = g_slist_remove_link(cc-status_stack,
 cc-status_stack);
UNLOCK();

if (msg) {
cc-status(cc, msg, sofar, oftotal, cc-status_data);
g_free(msg);
}
}

I think this needs to be something like this:

GSList *item = cc-status_stack;
cc-status_stack = g_slist_remove_link(cc-status_stack, item);
g_slist_free (item);


Can somebody with GSList know-how acknowledge this?



On Mon, 2008-01-14 at 02:28 +0100, Philip Van Hoof wrote:
 On Mon, 2008-01-14 at 01:47 +0100, Philip Van Hoof wrote:
 
  I have this memory analysis tool that I'm willing to believe that tells
  me this line in camel-folder.c causes 381 times a leaked allocation of
  in total 5.95 kb. (I opened one folder of 1000 items). It seems to be
  the g_slice_alloc (which does a malloc since I disabled gslice) of the
  g_hash_node_new.
 
 I found it, this was my own mistake in camel-lite's adaptations.
 
 I have found another leak in camel-operation.c:
 
 The cc-status_stack = g_slist_prepend(cc-status_stack, s); in
 camel_operation_start seems to be leaked each time it's called.
 
 I have 23 calls leaking 184 bytes. Various callers of
 camel_operation_start (all of them, it seems) seem to create this leak.
 
 My version of camel-operation.c adds two fields (a total and a current
 position in stead of a percentage, because my requirements where to give
 accurate to-the-byte progress info, not rounded percentages). Those
 changes are closely reviewed once more and don't seem to be causing this
 leak.
 
 ps. I attached the differences that I have. I basically replaced pc
 with a sofar and a oftotal.
 
 
 ___
 Evolution-hackers mailing list
 Evolution-hackers@gnome.org
 http://mail.gnome.org/mailman/listinfo/evolution-hackers
-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] Possible memory leak

2008-01-13 Thread Philip Van Hoof

Hi there,

I have this memory analysis tool that I'm willing to believe that tells
me this line in camel-folder.c causes 381 times a leaked allocation of
in total 5.95 kb. (I opened one folder of 1000 items). It seems to be
the g_slice_alloc (which does a malloc since I disabled gslice) of the
g_hash_node_new.

camel-folder.c:2255 (last line of camel_folder_change_info_add_uid)

g_hash_table_insert(p-uid_stored, olduid, info-uid_added);

I'm not immediately seeing it myself (it looks like p-uid_stored is
destroyed in the finalise indeed). But just FYI ... it might be
interesting to monitor this one.

If each new mail causes a GHashNode leak .. that might be quite a lot of
memory in Evolution.

ps. The caller for the camelfolderchange was imap_update_summary, but I
already checked that the change instance is finalised and that seems to
be the case indeed.


-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Possible memory leak

2008-01-13 Thread Philip Van Hoof

On Mon, 2008-01-14 at 01:47 +0100, Philip Van Hoof wrote:

 I have this memory analysis tool that I'm willing to believe that tells
 me this line in camel-folder.c causes 381 times a leaked allocation of
 in total 5.95 kb. (I opened one folder of 1000 items). It seems to be
 the g_slice_alloc (which does a malloc since I disabled gslice) of the
 g_hash_node_new.

I found it, this was my own mistake in camel-lite's adaptations.

I have found another leak in camel-operation.c:

The cc-status_stack = g_slist_prepend(cc-status_stack, s); in
camel_operation_start seems to be leaked each time it's called.

I have 23 calls leaking 184 bytes. Various callers of
camel_operation_start (all of them, it seems) seem to create this leak.

My version of camel-operation.c adds two fields (a total and a current
position in stead of a percentage, because my requirements where to give
accurate to-the-byte progress info, not rounded percentages). Those
changes are closely reviewed once more and don't seem to be causing this
leak.

ps. I attached the differences that I have. I basically replaced pc
with a sofar and a oftotal.


-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be
--- camel-operation.c	2007-12-23 18:12:33.0 +0100
+++ /home/pvanhoof/repos/tinymail/trunk/libtinymail-camel/camel-lite/camel/camel-operation.c	2007-11-15 10:42:00.0 +0100
@@ -46,7 +46,7 @@
 struct _status_stack {
 	guint32 flags;
 	char *msg;
-	int pc;/* last pc reported */
+	int sofar, oftotal;
 	unsigned int stamp;		/* last stamp reported */
 };
 
@@ -514,12 +514,14 @@
 	s = g_malloc0(sizeof(*s));
 	s-msg = msg;
 	s-flags = 0;
+	s-sofar = 0;
+	s-oftotal = 100;
 	cc-lastreport = s;
 	cc-status_stack = g_slist_prepend(cc-status_stack, s);
 
 	UNLOCK();
 
-	cc-status(cc, msg, CAMEL_OPERATION_START, cc-status_data);
+	cc-status(cc, msg, CAMEL_OPERATION_START, 100, cc-status_data);
 
 	d(printf(start '%s'\n, msg, pc));
 }
@@ -578,17 +580,17 @@
 /**
  * camel_operation_progress:
  * @cc: Operation to report to.
- * @pc: Percent complete, 0 to 100.
+ * @sofar: Amount complete
+ * @oftotal: Max amount (or, amount when completed)
  *
  * Report progress on the current operation.  If @cc is NULL, then the
- * currently registered operation is used.  @pc reports the current
- * percentage of completion, which should be in the range of 0 to 100.
+ * currently registered operation is used.  @
  *
  * If the total percentage is not know, then use
  * camel_operation_progress_count().
  **/
 void
-camel_operation_progress (CamelOperation *cc, int pc)
+camel_operation_progress (CamelOperation *cc, int sofar, int oftotal)
 {
 	unsigned int now;
 	struct _status_stack *s;
@@ -608,7 +610,8 @@
 	}
 
 	s = cc-status_stack-data;
-	s-pc = pc;
+	s-sofar = sofar;
+	s-oftotal = oftotal;
 
 	/* Transient messages dont start updating till 4 seconds after
 	   they started, then they update every second */
@@ -632,7 +635,7 @@
 	UNLOCK();
 
 	if (cc) {
-		cc-status(cc, msg, pc, cc-status_data);
+		cc-status(cc, msg, sofar, oftotal, cc-status_data);
 		g_free(msg);
 	}
 }
@@ -646,7 +649,7 @@
 void
 camel_operation_progress_count (CamelOperation *cc, int sofar)
 {
-	camel_operation_progress(cc, sofar);
+	camel_operation_progress(cc, sofar, 100);
 }
 
 /**
@@ -664,7 +667,8 @@
 	struct _status_stack *s, *p;
 	unsigned int now;
 	char *msg = NULL;
-	int pc = 0;
+	int sofar = 0;
+	int oftotal = 100;
 
 	if (cc == NULL)
 		cc = co_getcc();
@@ -693,13 +697,15 @@
 if (p-flags  CAMEL_OPERATION_TRANSIENT) {
 	if (p-stamp + CAMEL_OPERATION_TRANSIENT_DELAY  now) {
 		msg = g_strdup(p-msg);
-		pc = p-pc;
+		sofar = p-sofar;
+		oftotal = p-oftotal;
 		cc-lastreport = p;
 		break;
 	}
 } else {
 	msg = g_strdup(p-msg);
-	pc = p-pc;
+	sofar = p-sofar;
+	oftotal = p-oftotal;
 	cc-lastreport = p;
 	break;
 }
@@ -709,7 +715,8 @@
 		g_free(s-msg);
 	} else {
 		msg = s-msg;
-		pc = CAMEL_OPERATION_END;
+		sofar = CAMEL_OPERATION_END;
+		oftotal = 100;
 		cc-lastreport = s;
 	}
 	g_free(s);
@@ -718,7 +725,7 @@
 	UNLOCK();
 
 	if (msg) {
-		cc-status(cc, msg, pc, cc-status_data);
+		cc-status(cc, msg, sofar, oftotal, cc-status_data);
 		g_free(msg);
 	}
 }
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] Memory leak in camel-imap-message-cache.c

2008-01-13 Thread Philip Van Hoof

Hi there,

The cache-parts = g_hash_table_new (g_str_hash, g_str_equal); of
camel-imap-message-cache.c does not free its keys (it's the default way
of creating a hashtable, so there's no freeup function provided for the
keys).

Yet at (or around) line 118 we see this:

g_hash_table_insert (cache-parts, g_strdup (uid), subparts);

That uid is string-copied. So either the hashtable needs a freeup
function or the string should not be copied or .. this is wrong.

Because I don't know how important the string copying is, what do you
guys think we should do here?


-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] improved rfc2047 decode patch

2007-12-27 Thread Philip Van Hoof
 this kind of
encoded string:
   
  
  =?gb2312?b?any-encoded-text?==?gb2312?b?any-encoded-text?=
Two encoded-words are not separated by any
   character.
   
   Are you sure? I wrote the code to be able to handle
   this case and I just
   tested it again (noticed that I didn't have a test
   case like this in my
   test suite so added one) and it works fine.
   
   Do you have an example subject/whatever header for
   me to test against?
   
  
  I make my conclusion too hastiness. Yes, your patch
  support this kind of email,
 
 ok ;-)
 
   but it didn't support the
  email that break a single multi-byte character across
  multiple encoded-word tokens, and when it decode the
  header that break a encoded-word token across two
  lines, there is no result display on evolution, for
  example, the Subject is empty.
 
 ok, just fixed this in svn. I had tested a broken UTF-8 header earlier
 and so didn't see a slight bug in my code.
 
  I'll use Camle with your patch to check all email on
  my mbox  and use gmime to decode all email header to
  find out it's capacity.
 
 
 Ok, awesome.
 
 Jeff
 
 ___
 Evolution-hackers mailing list
 Evolution-hackers@gnome.org
 http://mail.gnome.org/mailman/listinfo/evolution-hackers
-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] improved rfc2047 decode patch

2007-12-25 Thread Philip Van Hoof

Awesome! In the afternoon I started with the exact same port, but had to
pause because of family visiting, I'm back home and you have it
finished :). Thanks a lot!

Brought it to tny's camel. FYI:
http://tinymail.org/trac/tinymail/changeset/3203


On Tue, 2007-12-25 at 19:28 -0500, Jeffrey Stedfast wrote:
 This patch is a port of my GMime rfc2047 decoder which is even more
 liberal in what it accepts than Thunderbird and is what I will be
 committing to svn.
 
 closing bugs:
 
 #302991
 #315513
 #502178
 
 Jeff
 
 ___
 Evolution-hackers mailing list
 Evolution-hackers@gnome.org
 http://mail.gnome.org/mailman/listinfo/evolution-hackers
-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] Merging camel-mime-utils.c, question

2007-12-24 Thread Philip Van Hoof
While I was merging camel-lite with camel upstream I noticed this one in
camel-mime-utils.c:

@@ -2110,7 +2105,7 @@
domain = g_string_append_c(domain, '[');
inptr++;
header_decode_lwsp(inptr);
-   while (*inptr  camel_mime_is_dtext(*inptr)) {
+   while (camel_mime_is_dtext(*inptr)  *inptr) {
domain = g_string_append_c(domain, *inptr);
inptr++;
}

I wonder which one is the most correct. It looks to me that first doing
something with *inptr and then testing whether it was NULL doesn't make
as much sense as first testing and then doing something.

So I kinda forgot whether I made this change as a result of a bugfix or
whether somebody upstream made the swap (I think I did it as a bugfix).


-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] Porting the namespace stuff to upstream camel

2007-12-24 Thread Philip Van Hoof
Hi there,

I'm doing a shot at trying to port the namespace things that I did for
Tny's Camel-lite to Evolution.

It's not working yet and I have not tested it, there are probably quite
a few things still wrong with the patch too ...

Nonetheless, I'm sending it already to get ideas/opinions from the other
developers.

Some remarks so far ...

I think in the end we'll have to bite the bullet and remove those 
imap_store-dir_sep and imap_store-namespace things. This is where most
of the current ugly things come from.

Perhaps just rewriting get_folders_sync completely in stead of having
three different versions of it (like what I did) might also cleanup the
patch (and make it look smaller).

I'm not a big fan of how refresh_refresh vs. get_folder_info_online
works. I don't think Evolution should periodically update the list, but
should always update it the first time after the startup of the
application (no matter what) and perhaps start supporting NOTIFY to get
notifications about mailbox changes more early while running.

Perhaps for users who run Evolution for years without restarting
(unrealistic).

Imho is refresh_refresh very error prone, probably very untested, that
wont update the UI screens with folder lists anyway, ...

But anyway ...

(my point is, that I don't see the point of that piece of code)


Once you have the full_name strings at SELECT context, it looks like all
folder operations just work (no need to prefix them with something that
comes from the namespace's name).

I also adapted CamelImapSummaryStore to cope with multiple namespaces.
Also this part of the patch was quickly done without a lot of error
checking (to get something working quickly).



-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be



Index: camel-imap-store.c
===
--- camel-imap-store.c	(revision 8315)
+++ camel-imap-store.c	(working copy)
@@ -210,6 +210,9 @@
 		disco-diary = NULL;
 	}
 
+	if (imap_store-namespaces)
+		imap_namespaces_destroy (imap_store-namespaces);
+
 	g_free (imap_store-custom_headers);
 }
 
@@ -1452,6 +1455,10 @@
 	/* Get namespace and hierarchy separator */
 	if ((store-capabilities  IMAP_CAPABILITY_NAMESPACE) 
 	!(store-parameters  IMAP_PARAM_OVERRIDE_NAMESPACE)) {
+
+
+		struct _namespaces *namespaces;
+
 		response = camel_imap_command (store, NULL, ex, NAMESPACE);
 		if (!response)
 			goto done;
@@ -1460,28 +1467,41 @@
 		if (!result)
 			goto done;
 
-#if 0
-		/* new code... */
-		namespaces = imap_parse_namespace_response (result);
-		imap_namespaces_destroy (namespaces);
-		/* end new code */
-#endif
+		store-namespaces = imap_parse_namespace_response (result);
+		namespaces = store-namespaces;
 
-		name = camel_strstrcase (result, NAMESPACE (();
-		if (name) {
-			char *sep;
+		if (namespaces  namespaces-personal) {
+			store-namespace = namespaces-personal-prefix;
+			store-dir_sep = namespaces-personal-delim;
+		} else {
+			store-namespace = NULL;
+			store-dir_sep = 0;
+		}
 
-			name += 12;
-			store-namespace = imap_parse_string ((const char **) name, len);
-			if (name  *name++ == ' ') {
-sep = imap_parse_string ((const char **) name, len);
-if (sep) {
-	store-dir_sep = *sep;
-	g_free (sep);
-}
-			}
+		if (namespaces  namespaces-personal) {
+			ns = camel_imap_store_summary_namespace_new(store-summary, namespaces-personal-prefix, namespaces-personal-delim);
+			camel_imap_store_summary_namespace_add(store-summary,ns);
+			camel_imap_store_summary_namespace_set(store-summary, ns);
 		}
+
+		if (namespaces  namespaces-other) {
+			ns = camel_imap_store_summary_namespace_new(store-summary, namespaces-other-prefix, namespaces-other-delim);
+			camel_imap_store_summary_namespace_add(store-summary,ns);
+		}
+
+		if (namespaces  namespaces-shared) {
+			ns = camel_imap_store_summary_namespace_new(store-summary, namespaces-shared-prefix, namespaces-shared-delim);
+			camel_imap_store_summary_namespace_add(store-summary,ns);
+		}
+
+
 		g_free (result);
+
+
+	} else {
+		ns = camel_imap_store_summary_namespace_new(store-summary, store-namespace, store-dir_sep);
+		camel_imap_store_summary_namespace_add(store-summary,ns);
+		camel_imap_store_summary_namespace_set(store-summary, ns);
 	}
 
 	if (!store-namespace)
@@ -2591,6 +2611,241 @@
 	g_hash_table_destroy(present);
 }
 
+
+
+static void
+get_folders_sync_ns(CamelImapStore *imap_store, struct _namespace *namespace, gboolean has_other, gboolean has_shared, CamelException *ex)
+{
+	CamelImapResponse *response;
+	CamelFolderInfo *fi = NULL, *first = NULL, *hfi;
+	char *list;
+	int i, count, j;
+	GHashTable *present;
+	CamelStoreInfo *si;
+	int loops = 2;
+	char *prefix, *lst = *;
+	gboolean bah = FALSE;
+
+	if (!namespace-prefix || strlen (namespace-prefix) == 0) {
+
+		/**
+		 * NAMESPACE (( .) (x y) (x y) ... 
+		 * In this case the IMAP server

Re: [Evolution-hackers] [patch] fixed incorrect rfc2047 decode for CJK header

2007-12-23 Thread Philip Van Hoof
Hey Jacky,

This is a port of your patch to Tinymail's camel-lite

On Sun, 2007-12-23 at 23:09 +0800, jacky wrote:
 Hi, all.
 
 The rfc2047 decoder in libcamel can not decode some
 CJK header correctly. Although some of them are not
 correspond to RFC, but I need to decode it correctly
 and I thought if evolution can display there email
 correctly more people like it.
 
 So I write a new rfc2047 decoder, and it's in the
 patch. With the patch, libcamel can decode CJK header
 correctly and evolution can display CJK header
 correctly now. I had test it in my mailbox. My mailbox
 has 2000 emails which were sent by evolution,
 thunderbird, outlook, outlook express, foxmail, open
 webmail, yahoo, gmail, lotus notes, etc. Without this
 patch, almost 20% of there emails can't be decoded and
 displayed correctly, with this patch, 99% of there
 emails can be decoded and displayed correctly.
 
 And I found that the attachment with CJK name can't be
 recognised and displayed by outlook / outlook express
 / foxmail. This is because there email clients do not
 support RFC2184. Evolution always use RFC2184 encode
 mothod to encode attachment name, so the email with
 CJK named attachment can't display in outlook /
 outlook express / foxmail. In thunderbird, you can set
 the option mail.strictly_mime.parm_folding to 0 or 1
 for using RFC2047 encode mothod to encode attachment
 name. Can we add a similar option?
 
 Best regards.
 
 
   ___ 
 雅虎邮箱传递新年祝福,个性贺卡送亲朋! 
 http://cn.mail.yahoo.com/gc/index.html?entry=5souce=mail_mailletter_tagline
 ___ Evolution-hackers mailing 
 list Evolution-hackers@gnome.org 
 http://mail.gnome.org/mailman/listinfo/evolution-hackers
-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be



Index: libtinymail-camel/camel-lite/camel/camel-mime-utils.c
===
--- libtinymail-camel/camel-lite/camel/camel-mime-utils.c	(revision 3190)
+++ libtinymail-camel/camel-lite/camel/camel-mime-utils.c	(working copy)
@@ -821,125 +821,207 @@
 	*in = inptr;
 }
 
+static void
+print_hex (unsigned char *data, size_t len)
+{
+	size_t i, x;
+	unsigned char *p = data;
+	char high, low;
+
+	x = 0;
+	printf (%04u, x);
+	for (i = 0; i  len; i++) {
+		high = *p  4;
+		high = (high10) ? high + '0' : high + 'a' - 10;
+
+		low = *p  0x0f;
+		low = (low10) ? low + '0' : low + 'a' - 10;
+
+		printf (0x%c%c  , high, low);
+
+		p++;
+		x++;
+		if (i % 8 == 7) {
+			printf (\n%04u, x);
+		}
+	}
+	printf (\n);
+}
+
+static size_t
+conv_to_utf8 (const char *encname, char *in, size_t inlen, char *out, size_t outlen)
+{
+	char *charset, *inbuf, *outbuf;
+	iconv_t ic;
+	size_t inbuf_len, outbuf_len, ret;
+
+	charset = (char *) e_iconv_charset_name (encname);
+
+	ic = e_iconv_open (UTF-8, charset);
+	if (ic == (iconv_t) -1) {
+		printf (e_iconv_open() error\n);
+		return (size_t)-1;
+	}
+
+	inbuf = in;
+	inbuf_len = inlen;
+
+	outbuf = out;
+	outbuf_len = outlen;
+
+	ret = e_iconv (ic, (const char **) inbuf, inbuf_len, outbuf, outbuf_len);
+	if (ret == (size_t)-1) {
+		printf (e_iconv() error! source charset is %s, target charset is %s\n, charset, UTF-8);
+		printf (converted %u bytes, but last %u bytes can't convert!!\n, inlen - inbuf_len, inbuf_len);
+		printf (source data:\n);
+		print_hex (in, inlen);
+
+		*outbuf = '\0';
+		printf (target string is \%s\\n, out);
+
+		return (size_t)-1;
+	}
+
+	ret = outlen - outbuf_len;
+	out[ret] = '\0';
+
+	e_iconv_close (ic);
+
+	return ret;
+}
+
 /* decode rfc 2047 encoded string segment */
+#define DECWORD_LEN 1024
+#define UTF8_DECWORD_LEN 2048
+
 static char *
 rfc2047_decode_word(const char *in, size_t len)
 {
-	const char *inptr = in+2;
-	const char *inend = in+len-2;
-	const char *inbuf;
-	const char *charset;
-	char *encname, *p;
-	int tmplen;
-	size_t ret;
-	char *decword = NULL;
-	char *decoded = NULL;
-	char *outbase = NULL;
-	char *outbuf;
-	size_t inlen, outlen;
-	gboolean retried = FALSE;
-	iconv_t ic;
-	int idx = 0;
+	char prev_charset[32], curr_charset[32];
+	char encode;
+	char *start, *inptr, *inend;
+	char decword[DECWORD_LEN], utf8_decword[UTF8_DECWORD_LEN];
+	char *decword_ptr, *utf8_decword_ptr;
+	size_t inlen, outlen, ret;
 
 	d(printf(rfc2047: decoding '%.*s'\n, len, in));
 
+	prev_charset[0] = curr_charset[0] = '\0';
+
+	decword_ptr = decword;
+	utf8_decword_ptr = utf8_decword;
+
 	/* quick check to see if this could possibly be a real encoded word */
-
-	if (len  8 || !(in[0] == '='  in[1] == '?')) {
+	if (len  8
+	|| !(in[0] == '='  in[1] == '?'
+		  in[len-1] == '='  in[len-2] == '?')) {
 		d(printf(invalid\n));
 		return NULL;
 	}
 
-	/* skip past the charset to the encoding type */
-	inptr = memchr (inptr, '?', inend-inptr);
-	if (inptr != NULL  inptr  inend + 2  inptr[2] == '?') {
-		d

Re: [Evolution-hackers] [patch] fixed incorrect rfc2047 decode for CJKheader

2007-12-23 Thread Philip Van Hoof

On Sun, 2007-12-23 at 14:51 -0700, Jeff Stedfast wrote:

 What types of bugs are you actually trying to fix? What is it about CJK
 messages in particular that are not getting decoded properly? Your email
 was overly vague.

Looks like he wants to support both 'B' and 'b' and 'Q' and 'q' in stead
of just 'B' and 'Q' for the first characters for Base64 or Quoted
strings, for example. 

The rfc2047_decode_word implementation indeed has a few serious
potential buffer overflows. For example the charset buffer will be
overflowed if larger than 32 bytes.



  jacky [EMAIL PROTECTED] 12/23/07 10:09 AM 
 Hi, all.
 
 The rfc2047 decoder in libcamel can not decode some
 CJK header correctly. Although some of them are not
 correspond to RFC, but I need to decode it correctly
 and I thought if evolution can display there email
 correctly more people like it.
 
 So I write a new rfc2047 decoder, and it's in the
 patch. With the patch, libcamel can decode CJK header
 correctly and evolution can display CJK header
 correctly now. I had test it in my mailbox. My mailbox
 has 2000 emails which were sent by evolution,
 thunderbird, outlook, outlook express, foxmail, open
 webmail, yahoo, gmail, lotus notes, etc. Without this
 patch, almost 20% of there emails can't be decoded and
 displayed correctly, with this patch, 99% of there
 emails can be decoded and displayed correctly.
 
 And I found that the attachment with CJK name can't be
 recognised and displayed by outlook / outlook express
 / foxmail. This is because there email clients do not
 support RFC2184. Evolution always use RFC2184 encode
 mothod to encode attachment name, so the email with
 CJK named attachment can't display in outlook /
 outlook express / foxmail. In thunderbird, you can set
 the option mail.strictly_mime.parm_folding to 0 or 1
 for using RFC2047 encode mothod to encode attachment
 name. Can we add a similar option?
 
 Best regards.
 
 
   ___ 
 雅虎邮箱传递新年祝福,个性贺卡送亲朋! 
 http://cn.mail.yahoo.com/gc/index.html?entry=5souce=mail_mailletter_taglin
 ___
 Evolution-hackers mailing list
 Evolution-hackers@gnome.org
 http://mail.gnome.org/mailman/listinfo/evolution-hackers
-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] First attempt at support for NAMESPACE

2007-12-21 Thread Philip Van Hoof

* My current protocol related code does this for LIST, LSUB and
  NAMESPACE:

if capabilities contains NAMESPACE {

  personal, other, shared = get namespaces

/* We do both LIST and LSUB of the personal namespace
 * or we use LIST-EXTENDED and therefore don't need to
 * do the LSUB */

  if (personal-prefix.length  0) {

if capabilities contains LIST-EXTENDED
  LIST (SUBSCRIBED)  personal-prefix *   
else {
  LIST personal-prefix *   
  LSUB personal-prefix *   
}

  } else {

/* If the namespace is , this would otherwise result
 * in a LIST  *, which would not be a good idea if 
 * the IMAP server has a huge list in shared and other */

if capabilities contains LIST-EXTENDED
  LIST (SUBSCRIBED) personal-prefix %   
else {
  LIST personal-prefix %   
  LSUB personal-prefix %   
}
  }

/* We dont' get LIST of non-personal namespaces as that 
 * could lead to IMAP servers returning huge folder lists,
 * for example when the IMAP server proxies a news server */

  if (other-prefix.length  0)
LSUB other-prefix *   
  else
LSUB other-prefix %   

  if (shared-prefix.length  0)
LSUB shared-prefix *   
  else
LSUB shared-prefix %   

} else {

  /* We don't have namespaces, so we're pretty limited in what we
   * can ask for. */

  if capabilities contains LIST-EXTENDED
LIST (SUBSCRIBED)  * or %
  else {
LIST  * or %
LSUB  * or %
 }

}

* I changed the folder management related code to cope with folders
  under namespaces

* I changed the CamelImapStoreSummary format and code to cope with
  namespaces

* I have this patch for camel-lite, if the Evolution team is interested
  in changing quite a lot of their LIST LSUB and NAMESPACE related code,
  this can be ported to Evolution too.


-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Support for LIST-EXTENDED's SUBSCRIBED parameter

2007-12-20 Thread Philip Van Hoof
I further changed Camel-lite in such a way that only the following
protocol conversation takes place:

if CAPABILITIES contains NAMESPACE
  - NAMESPACE
  - NAMESPACE ((namespace that matters dir_sep_char) ... )
  # Camel ignores the other namespaces (this is bad!)
  item = namespace that matters + dir_sep_char + * or %
else
  namespace = false
  item = *
endif

Note that usually namespace that matters is simply INBOX. Note that
Camel ignores all not-first namespaces (and that supporting all
namespaces implies a format change of the CamelStoreSummary's caching).

Then this is done to know the subscription state of INBOX (and counts as
a roundtrip, I'm still figuring out how I can avoid this one as I think
it's unnecessary):

(now, this only happens if namespace != )

if capabilities contains LIST-EXTENDED
  - LIST (SUBSCRIBED)  INBOX
  - LIST (\Subscribed) INBOX ...
else
  -  LIST  INBOX
  - LIST INBOX ...
  -  LSUB  INBOX
  - LSUB INBOX ...
endif

This is getting the list of folders. Either item is namespace.*' or
it's *. Note that requesting * is a very bad idea for very large
folder lists (it's recursive, not all IMAP servers will like you for
doing this, and it'll be a lot slower than requesting % in stead).

In case you don't request a recursive list, Camel puts % in stead of
*, but note that most of Evolution's code requests a recursive one (so
it'll at least once occur that a recursive is requested, and therefore
it'll request it as * anyway, or at least that's what I saw happening
when analysing the traffic).

Better would indeed be to request a new list each time you click-open a
tree node, and request % in stead. This might not be practical for the
UI developer, though.

Here the LIST-EXTENDED changes kick into action.

if capabilities contains LIST-EXTENDED
  - LIST (SUBSCRIBED)  item
  - LIST (\NonExistent \Subscribed) ...
  - LIST (\Subscribed) ...
  - ...
else
  - LIST  item
  - LIST ...
  - LSUB  item
  - LSUB ...
endif

Now. What Evolution does (it doesn't support the LIST-EXTENDED case) is
merge the LIST and LSUB together while marking items in LSUB as
CAMEL_FOLDER_SUBSCRIBED. Marking them is fine, of course, but merging
them together is utterly wrong. 

Courier is one of the IMAP server that don't do housekeeping of LSUB
when you for example delete a folder yet don't unsubscribed it first.
The net result is that with Evolution you get folder-nodes that don't
exist at the server.

For the user this means that when he clicks on that folder-node, that he
gets an error about the folder not existing.

Of course in Courier you can have subfolders of folders that don't
exist. Oh the joys of IMAP :) (and the joys of server people disagreeing
on the interpretation of the standard).

As how I see it should the IMAP server give in its LIST response a
\NonExistant and/or \NoSelect flag for such folders. But should not omit
them in LIST. But merging LSUB and LIST together is even more wrong. At
least in my opinion (you end up showing folders to the user that simply
don't exist on the server).

Note: A greyish tree node that presents a parent for child-folders when
it ain't selectable ... would be a more correct UI presentation of such
a \NoSelect parent folder (or, placeholder in the folder tree).

Note. An example is the [GMail] folder of Google IMAP. Google's IMAP
server is one of the first times that I saw \NoSelect being actively
used by an IMAP server in a default-ish way.

Evolution will soon get a lot of GMail IMAP users ...

Sounds like something to correctly handle from now on.




On Wed, 2007-12-19 at 21:32 +0100, Philip Van Hoof wrote:
 The net result of this patch is less round trips when connecting with an
 IMAP server that has support for LIST-EXTENDED.
 
 http://tinymail.org/trac/tinymail/changeset/3155
 
 Note that Evolution merges LIST and LSUB together. This is wrong
 behaviour. An item not in LIST yet in LSUB simply doesn't exist.
 
 
-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Support for LIST-EXTENDED's SUBSCRIBED parameter

2007-12-20 Thread Philip Van Hoof

This is some first piece of code to parse the other namespaces

In: imap_connect_online 


result = camel_imap_response_extract (store, response, 
NAMESPACE, ex);
if (!result)
goto done;

#if 0
/* new code... */
namespaces = imap_parse_namespace_response (result);
imap_namespaces_destroy (namespaces);
/* end new code */
#endif

name = camel_strstrcase (result, NAMESPACE (();
if (name) {
gboolean first = TRUE;
gboolean cont = TRUE;
name += 10;

while (cont) {
char *sep, *str = NULL;

while (*name == ' ')
name++;

if (*name == '('  *(name+1) == '(') {
name += 2;
if (first)
store-namespace = 
imap_parse_string ((const char **) name, len);
else
str = imap_parse_string ((const 
char **) name, len);

if (name  *name++ == ' ') {
sep = imap_parse_string ((const 
char **) name, len);
if (sep) {
if (first)
store-dir_sep 
= *sep;
g_free (sep);
}
if (*name != ')' || *(name+1) 
!= ')') {
cont = FALSE;
break;
}
/* The )) */
name += 2;
}
} else if (!strncmp (name, NIL, 3))
name += 3;

if (!name)
cont = FALSE;

if (str) {
printf (Other namespace:%s\n, str);
g_free (str);
}
first = FALSE;
}
}
g_free (result);
}

On Thu, 2007-12-20 at 14:06 +0100, Philip Van Hoof wrote:
 I further changed Camel-lite in such a way that only the following
 protocol conversation takes place:
 
 if CAPABILITIES contains NAMESPACE
   - NAMESPACE
   - NAMESPACE ((namespace that matters dir_sep_char) ... )
   # Camel ignores the other namespaces (this is bad!)
   item = namespace that matters + dir_sep_char + * or %
 else
   namespace = false
   item = *
 endif
 
 Note that usually namespace that matters is simply INBOX. Note that
 Camel ignores all not-first namespaces (and that supporting all
 namespaces implies a format change of the CamelStoreSummary's caching).
 
 Then this is done to know the subscription state of INBOX (and counts as
 a roundtrip, I'm still figuring out how I can avoid this one as I think
 it's unnecessary):
 
 (now, this only happens if namespace != )
 
 if capabilities contains LIST-EXTENDED
   - LIST (SUBSCRIBED)  INBOX
   - LIST (\Subscribed) INBOX ...
 else
   -  LIST  INBOX
   - LIST INBOX ...
   -  LSUB  INBOX
   - LSUB INBOX ...
 endif
 
 This is getting the list of folders. Either item is namespace.*' or
 it's *. Note that requesting * is a very bad idea for very large
 folder lists (it's recursive, not all IMAP servers will like you for
 doing this, and it'll be a lot slower than requesting % in stead).
 
 In case you don't request a recursive list, Camel puts % in stead of
 *, but note that most of Evolution's code requests a recursive one (so
 it'll at least once occur that a recursive is requested, and therefore
 it'll request it as * anyway, or at least that's what I saw happening
 when analysing the traffic).
 
 Better would indeed be to request a new list each time you click-open a
 tree node, and request % in stead. This might not be practical for the
 UI developer, though.
 
 Here the LIST-EXTENDED changes kick into action.
 
 if capabilities contains LIST-EXTENDED
   - LIST (SUBSCRIBED)  item
   - LIST (\NonExistent \Subscribed) ...
   - LIST (\Subscribed) ...
   - ...
 else
   - LIST  item
   - LIST ...
   - LSUB  item
   - LSUB ...
 endif
 
 Now. What Evolution does (it doesn't support the LIST-EXTENDED case) is
 merge the LIST and LSUB together while marking items in LSUB

Re: [Evolution-hackers] Support for LIST-EXTENDED's SUBSCRIBED parameter

2007-12-20 Thread Philip Van Hoof

Oh, and it's wrong.

It seems that an IMAP server can send an multiple parameters after the
directory separator for a namespace. So after the directory separator,
the ptr should be read skipping each \string\ until )) is seen.

And it seems that it's a skiplist, so a parameter can be a
SOMETHING (Some setting Some more). hrm ... that's more work :)


On Thu, 2007-12-20 at 21:46 +0100, Philip Van Hoof wrote:
 This is some first piece of code to parse the other namespaces
 
 In: imap_connect_online 
 
 
   result = camel_imap_response_extract (store, response, 
 NAMESPACE, ex);
   if (!result)
   goto done;
 
 #if 0
   /* new code... */
   namespaces = imap_parse_namespace_response (result);
   imap_namespaces_destroy (namespaces);
   /* end new code */
 #endif
 
   name = camel_strstrcase (result, NAMESPACE (();
   if (name) {
   gboolean first = TRUE;
   gboolean cont = TRUE;
   name += 10;
 
   while (cont) {
   char *sep, *str = NULL;
 
   while (*name == ' ')
   name++;
 
   if (*name == '('  *(name+1) == '(') {
   name += 2;
   if (first)
   store-namespace = 
 imap_parse_string ((const char **) name, len);
   else
   str = imap_parse_string ((const 
 char **) name, len);
 
   if (name  *name++ == ' ') {
   sep = imap_parse_string ((const 
 char **) name, len);
   if (sep) {
   if (first)
   store-dir_sep 
 = *sep;
   g_free (sep);
   }
   if (*name != ')' || *(name+1) 
 != ')') {
   cont = FALSE;
   break;
   }
   /* The )) */
   name += 2;
   }
   } else if (!strncmp (name, NIL, 3))
   name += 3;
 
   if (!name)
   cont = FALSE;
 
   if (str) {
 printf (Other namespace:%s\n, str);
   g_free (str);
   }
   first = FALSE;
   }
   }
   g_free (result);
   }
 
 On Thu, 2007-12-20 at 14:06 +0100, Philip Van Hoof wrote:
  I further changed Camel-lite in such a way that only the following
  protocol conversation takes place:
  
  if CAPABILITIES contains NAMESPACE
- NAMESPACE
- NAMESPACE ((namespace that matters dir_sep_char) ... )
# Camel ignores the other namespaces (this is bad!)
item = namespace that matters + dir_sep_char + * or %
  else
namespace = false
item = *
  endif
  
  Note that usually namespace that matters is simply INBOX. Note that
  Camel ignores all not-first namespaces (and that supporting all
  namespaces implies a format change of the CamelStoreSummary's caching).
  
  Then this is done to know the subscription state of INBOX (and counts as
  a roundtrip, I'm still figuring out how I can avoid this one as I think
  it's unnecessary):
  
  (now, this only happens if namespace != )
  
  if capabilities contains LIST-EXTENDED
- LIST (SUBSCRIBED)  INBOX
- LIST (\Subscribed) INBOX ...
  else
-  LIST  INBOX
- LIST INBOX ...
-  LSUB  INBOX
- LSUB INBOX ...
  endif
  
  This is getting the list of folders. Either item is namespace.*' or
  it's *. Note that requesting * is a very bad idea for very large
  folder lists (it's recursive, not all IMAP servers will like you for
  doing this, and it'll be a lot slower than requesting % in stead).
  
  In case you don't request a recursive list, Camel puts % in stead of
  *, but note that most of Evolution's code requests a recursive one (so
  it'll at least once occur that a recursive is requested, and therefore
  it'll request it as * anyway, or at least that's what I saw happening
  when analysing the traffic).
  
  Better would indeed be to request a new list each time you click-open a
  tree node, and request % in stead. This might not be practical for the
  UI

Re: [Evolution-hackers] Support for LIST-EXTENDED's SUBSCRIBED parameter

2007-12-20 Thread Philip Van Hoof

On Thu, 2007-12-20 at 21:46 +0100, Philip Van Hoof wrote:
 This is some first piece of code to parse the other namespaces

Forget that, imap_parse_namespace_response implements this already and
looks like a fine starting point.


-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] First attempt at support for NAMESPACE

2007-12-20 Thread Philip Van Hoof
Here's a first attempt. This is not a official patch proposal because it
ain't finished yet (I made quite a few shortcuts in the code, and I need
to cleanup some stuff too).

http://pvanhoof.be/files/camel-lite-shared-folders.diff

ps. NAMESPACE means: support for company-wide shared folders.
ps. I'm using `LIST  *', this is indeed not a good idea. I also have
the list of namespaces in a parsed way, so I can ask the folder lists in
a better way too .. (as I said, shortcuts in the code).


-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] Support for LIST-EXTENDED's SUBSCRIBED parameter

2007-12-19 Thread Philip Van Hoof
The net result of this patch is less round trips when connecting with an
IMAP server that has support for LIST-EXTENDED.

http://tinymail.org/trac/tinymail/changeset/3155

Note that Evolution merges LIST and LSUB together. This is wrong
behaviour. An item not in LIST yet in LSUB simply doesn't exist.


-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Closing a NSS connection

2007-12-15 Thread Philip Van Hoof

On Sat, 2007-12-15 at 11:36 -0500, Matthew Barnes wrote:

 Looks like there's a couple more PR_Close() calls in e_msgport_destroy()
 (libedataserver/e-msgport.c).  I'm assuming these need shutdown too?

No, as far as I know those are named pipes. Not socket filedescriptors.

But feel free to verify my claims, I don't know all such details of
Camel by heart ;)


-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Let the porting begin

2007-11-20 Thread Philip Van Hoof
On Wed, 2007-10-24 at 11:58 -0400, Jeffrey Stedfast wrote:
 I took a look at the IDLE implementation last night and felt it went
 about it the wrong way.

Yes, you are right. I think the right fix is to create a new API called
camel_tcp_stream_wait that works like this:


int bytes = camel_tcp_stream_wait (tcp_stream, stop_fd,
 store-idle_sleep, line_buffer, 1023);

I need the amount of read-read bytes, to know where the buffer stops, so
I return it. I need a stop_fd to get the select() to early-stop. I need
a timeout for in case of connection problems. I need the buffer to put
the newly read data in and I need a size to tell it how large my buffer
max is.


.. idle_thread () ..
{

  start_idle (store);
  line_buffer = malloc (1024)
  memset (line_buffer, 0, sizeof (*line_buffer));
  ...

  while (store-idle_cont)  {
char *lasts = NULL;
int bytes = camel_tcp_stream_wait (tcp_stream, stop_fd, 
store-idle_sleep, line_buffer, 1023);

if (bytes  0) {
char *resp;
line_buffer[bytes+1] = '\0';

for (resp = strtok_r (line_buffer, \n, lasts); resp; resp = 
strtok_r (NULL, \n, lasts)) {
if (resp  strlen (resp)  1  resp[0] == '*') {
if (!idle_resp)
idle_resp = idle_response_new (folder);
consume_idle_line (store, folder, resp, 
idle_resp);
lines++;
}
}
printf (IDLE: read %d bytes %s\n, bytes, line_buffer);
}

if (!store-idle_cont || (idle_resp  idle_resp-exists_happened)) {
idle_done (store, folder, idle_resp);
stopped = TRUE;
if (!store-idle_cont)
retval = idle_resp;
} 

if (store-idle_cont) {
if (idle_resp) {
process_idle_response (idle_resp);
idle_response_free (idle_resp);
idle_resp = NULL;
printf (Processed %d lines\n, lines);
lines = 0;
}
if (stopped)
idle_start (store);
}
memset (line_buffer, 0, sizeof (*line_buffer));
  }
}

That wait is implemented like this:


static ssize_t
stream_wait (CamelTcpStream *stream, int stop_fd, int timeout, char *buffer, 
size_t max_len)
{
CamelTcpStreamRaw *raw = CAMEL_TCP_STREAM_RAW (stream);
int fd = raw-sockfd;
ssize_t nread;
int fdmax;
fd_set rdset;
u_long yes = 1;
struct timeval tv;
int res;

//ioctlsocket (fd, FIONBIO, yes);
fdmax = MAX (fd, stop_fd) + 1;

FD_ZERO (rdset);
FD_SET (fd, rdset);
FD_SET (stop_fd, rdset);

tv.tv_sec = timeout;
tv.tv_usec = 0;
nread = -1;

res = select (fdmax, rdset, 0, 0, tv);
if (res  0  !FD_ISSET (stop_fd, rdset))
nread = recv (fd, buffer, max_len, 0);
else
nread = -1;

return nread;
}


Afaics It's not true that you don't need to change any API:

The camel_stream_read() will block until all n bytes are read. This is
something you don't know during IDLE (how many bytes you can expect to
receive), and on top of that you want to continue immediately after
select returns, not keep retrying until you have n bytes (to simulate a
blocking read). You also want a dedicated stop fildescriptor, not the
standard cancel_fd as then any cancel will interfere with this (while
you want strict control over this to let other threads start and stop
IDLE).

Well ... let me put it this way: I tried it from a few angles, I didn't
succeed with the normal camel_stream_read :-(

I have reworked the IDLE implementation. You can apply this patch to
Tinymail's upstream (trunk). You most likely can't apply it to upstream
Camel. (http://svn.tinymail.org/svn/tinymail/trunk)

There's still one problem: sometimes camel_imap_folder_stop_idle is not
returning late enough (the socket is still in IDLE state when it
returns).

Other than that, it usually works.

I'm trying to fix this last problem right now, and after that there will
most likely be a few days of testing before I put this in Tinymail's
repository.


 You should simply poll() on the socket descriptors (and a pipe fd used
 for telling the IDLE thread's I/O loop to finish and send DONE).

Yep


 Jeff
 
 On Mon, 2007-10-08 at 00:41 +0200, Philip Van Hoof wrote:
  On Sun, 2007-10-07 at 14:15 +0200, Philip Van Hoof wrote:
   Hi there,
  
   Using this changeset you can follow the changes to camel-lite:
   
 http://tinymail.org/trac/tinymail/changeset/2823
   
  
  This changeset are a bunch of compilation warnings for Matthew's Base64
  patch to Camel: http://tinymail.org/trac/tinymail/changeset/2827
  
  
  ps. Adding Matthew

[Evolution-hackers] Merging camel-lite to camel

2007-11-15 Thread Philip Van Hoof
Hi there,

This is a note to notify the fabulous developers at Evolution that
Matthew Barnes's preparation patch for camel-lite has been committed to
Tinymail's trunk.

This means that each difference that you'll find between camel-lite and
uptream camel is now a pure feature or functional change and that code-
style and whitespace usage are synchronised between the two softwares.

Matthew also told me he's going to work on getting pieces upstream.

Note that the IDLE work in Camel-lite can use some rewriting. I recently
posted a re-implementation of the IDLE support here. This implementation
used select() as proposed by Jeffrey Stedfast. It didn't yet work 100%
correct as I had a bunch of problems related to reading a buffer of data
and detecting when a line is finished (and other such problems). It did
sometimes work though... (that's not good enough)

Right now IDLE in Camel-lite is implemented by using an ugly loop in a
thread that usleep()s some time and does a non-blocking attempt to read
some data from the socket. Indeed, select(), poll() or epoll() are a lot
better and much more suitable for this (you don't have to convince me).

I'm right now also working on support for NOTIFY. With NOTIFY you
specify which exact notifications you want to get. You'll get them
either during IDLE mode, or as an unsolicited event during your
messages. My patch always uses IDLE mode for this.

With NOTIFY you can also know about count updates (STATUS, which gives
you the amount of messages and the amount of UNSEEN items) of folders
that are not currently selected, about remote subscription changes of
folders, about remote folder deletions, remote folder renames and remote
folder creations.

Combine all those together, and you can effectively make an E-mail
client that will always show an uptodate view of an IMAP account while
you edit and work with the account from another machine. And you'll get
instant updates as they happen.

This is indeed a feature that you want on a mobile device, as a mobile
device's E-mail client is usually used primarily as a secondary E-mail
client. It also reduces the possibility that the user ends up in a
erroneous situation: he wants to put things in a folder that got removed
remotely.

If we can make the E-mail client update itself instantly, the user will
have to cope with the remote folder removal (or rename) because his view
with instantly update to this new situation.

With your cellphone in your pocket, the E-mail client on it opened and
active, and meanwhile editing your IMAP account using your laptop ..

 this is a far more realistic situation that one might think.


-- Note that NOTIFY's RFC is currently in proposal state. No IMAP
servers currently implement it (perhaps Sun's beta version does).



-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] Memory leak question in CamelImapCommand

2007-10-25 Thread Philip Van Hoof
Hi there,

t almost sounds impossible but still, and that's why I ask. I noticed
that the variable args in imap_command_strdup_vprintf is never freed.

That would be a rather large memory leak (almost every IMAP command is
created this way).

So I'm a bit stunned that nobody else ever saw this one and I wonder
whether I'm just not looking very good, or being confused or something
(this happens often to me, so it wouldn't surprise me).

The patch is of course a simple + g_ptr_array_free (args, TRUE); right
before the return out;, right?

ps. Adding Jeffrey in CC as I think he has a good idea how this function
should work.

-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] GMail IMAP support in Evolution

2007-10-25 Thread Philip Van Hoof
On Wed, 2007-10-24 at 17:01 +0530, Srinivasa Ragavan wrote:
 Hi Øystein
 
 On Wed, 2007-10-24 at 13:11 +0200, Øystein Gisnås wrote:
  Google seem to be in the process of introducing IMAP support to GMail
  [1]. Personally I think GMail offers an extremely attractive email
  solution by now. Evolution does already support integration with GMail
  via SMTP and POP, and now also via IMAP. In addition to following the
  IMAP standards as closely as possible, 

I agree here. I think we (Evolution team, although I'm not part of that,
strictly speaking) should put a lot more effort in getting the best out
of the latest IMAP standards.

A lot of IMAP servers are also improving a lot lately, and are
implementing these newer things. 

CONDSTORE, QRESYNC, IDLE, NOTIFY are samples of very interesting IMAP
enhancements that we should support in my opinion.

I also think that taking a look at spruce and GMime is a good idea for
this. Although improving Camel's IMAP4 provider is not a bad idea
either.

The current Camel IMAP provider is limited in what we can make it do. I
would, for example, love to have a solid client-side library that does
pipelining with the IMAP server.

Marrying the MIME handling with the server, for things like CATENATE, is
also something I would love to have.

Oh ... I have a lot of ideas :)

 I think we should do explicit
  QA on Evolution-GMail IMAP integration, to make sure our users'
  experience is as good as possible. One of the slashdot comments has
  already commented that Outlook works better with GMail IMAP than
  Evolution. That should change!
 
 It will change with 2.22. We are bringing down nice things down from
 camel-lite to upstream camel and also doing nice things here as well. 

Note that I will attend a Lemonade meeting in Munich next month. If
there are any items that I should raise for Evolution, you can let me
know about them.

I of course have a small list myself already.


-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Memory leak question in CamelImapCommand

2007-10-25 Thread Philip Van Hoof
On Thu, 2007-10-25 at 11:02 -0400, Matthew Barnes wrote:
 On Thu, 2007-10-25 at 16:45 +0200, Philip Van Hoof wrote:
  
  The patch is of course a simple + g_ptr_array_free (args, TRUE); right
  before the return out;, right?
  
  ps. Adding Jeffrey in CC as I think he has a good idea how this function
  should work.
 
 Hi Philip,
 
 I actually fixed that a few months ago.
 
 http://bugzilla.gnome.org/show_bug.cgi?id=447753
 

Oh, strange. Must have been a merge problem of last week then ...

Ok well, then I guess we both saw this :). Good.

-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Let the porting begin

2007-10-24 Thread Philip Van Hoof
On Wed, 2007-10-24 at 11:58 -0400, Jeffrey Stedfast wrote:
 I took a look at the IDLE implementation last night and felt it went
 about it the wrong way.

Note that the IDLE implementation got changed (reimplemented) this
morning.

Although the same problems that you mention here still apply.

I also added some comments. The poll() idea is indeed something that at
some point I should refactor the loop to.

 Firstly, the added camel_stream_[read,write]_nb() and
 camel_stream_read_idle() functionality is totally unnecessary and just
 makes the camel stream API gross (not to mention duplicating a lot of
 code as there's no real difference from the normal read/write
 implementations other than the timeout).
 
 You should simply poll() on the socket descriptors (and a pipe fd used
 for telling the IDLE thread's I/O loop to finish and send DONE).
 
 Jeff
 
 On Mon, 2007-10-08 at 00:41 +0200, Philip Van Hoof wrote:
  On Sun, 2007-10-07 at 14:15 +0200, Philip Van Hoof wrote:
   Hi there,
  
   Using this changeset you can follow the changes to camel-lite:
   
 http://tinymail.org/trac/tinymail/changeset/2823
   
  
  This changeset are a bunch of compilation warnings for Matthew's Base64
  patch to Camel: http://tinymail.org/trac/tinymail/changeset/2827
  
  
  ps. Adding Matthew in CC.
  
 
-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] QRESYNC implementation for Camel-lite

2007-10-18 Thread Philip Van Hoof
On Thu, 2007-10-18 at 10:46 +0100, Dave Cridland wrote:
 On Thu Oct 18 00:41:39 2007, Philip Van Hoof wrote:
  I implemented support for QRESYNC in Tinymail's camel-lite. I never
  tested this because somehow the MBox copy that the friendly guys at
  Isode gave me, is not starting up and I know of no other IMAP server
  that already offers the QRESYNC capability. 

 Thanks for the bragging-by-proxy. :-)

Well, you guys make an IMAP server that follows the latest IMAP protocol
enhancements. Even the proposals that are still in draft like QRESYNC.

And some of the people who work on these standards are employed by
Isode. It's just fair to mention that.

With your help I managed to get my hands on one that does things like
QRESYNC, CONDSTORE and ENABLE (which I also had to add, didn't knew
about having to ENABLE them too). So I can now start trying to get it
actually working too.

 I think we're the first to have a released product, but I'm not  
 certain we're the only ones who've implemented it. Most of the other  
 closed-source companies have much longer release cycles that we do.  
 (We made 12 releases this year, and we're not slowing yet).

Good

 We're certainly not the only ones to have things like CONDSTORE,  
 which is much harder work, and gets both client and server 90% of the  
 way there, so QRESYNC will probably start to appear soon.

Awesome! And let's convince the latecomers to hurry up with it. It does
make quite a lot of things more easy. Especially the VANISHED response
is very useful indeed.

No more fiddling with sequence numbers and single line EXPUNGE
responses, and all immediately in the SELECT's response packed together
with the flag updates.

  I'll retry tomorrow after actually reading the documentation that  
  came
  with it.
 
 Reading the docs? You're weird...

I wish certain IMAP developers, who's name(s) shall not be pronounced,
would read the specs too :-\. That would make the lives of client
developers far less frustrating.

  I'm for example not sure about the VANISHED reply (and the exact  
  meaning
  of that earlier parameter).
  
  
 EARLIER indicates that the VANISHED response you're looking at  
 doesn't refer to an event that's just occured, but to one (or lots)  
 that occured at some unspecified point in time, probably when you  
 weren't looking.

 To put it another way, these EARLIER ones are resynchronization data,  
 and not event notifications.

Aha okay (so .. I can just remove the items with that UID) :)

  ps. I added the nice folks at Evolution in CC, as some of these code
  warriors are planning to migrate some of newer features in  
  camel-lite to
  upstream camel.
 
 Hoorah - I know several people within Isode using Evolution who'll be  
 happy at that. Feel free to bug me if there's anything I can do to  
 help (test accounts, etc).

Will do. Although the porting of all this to Evolution will not be for
tomorrow I think. 

Well, CONDSTORE and QRESYNC are more easy for them to backport to
upstream Camel. IDLE is a bit more difficult as this involves dealing
with the events in their ui pieces too (and the patch for IDLE is a
little bit ugly as I had to do certain things that are not possible with
how the stream-related code is structured in upstream Camel).


-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] QRESYNC implementation for Camel-lite

2007-10-17 Thread Philip Van Hoof
Hi there fellow hackers and other insane people, like me, who spend all
night coding on proposed IMAP protocol enhancements.

I implemented support for QRESYNC in Tinymail's camel-lite. I never
tested this because somehow the MBox copy that the friendly guys at
Isode gave me, is not starting up and I know of no other IMAP server
that already offers the QRESYNC capability.

I'll retry tomorrow after actually reading the documentation that came
with it.

Therefore, is this implementation based on what I've found on Draft #6
of the IMAP4 Extensions for Quick Mailbox Resynchronization which is
available here:

http://www3.tools.ietf.org/html/draft-ietf-lemonade-reconnect-client-06

Especially for Dave and Alexey: you'd be my heroes if you could take a
look at the things I'm changing in the patch. Especially those things
that are related to the protocol.

I'm for example not sure about the VANISHED reply (and the exact meaning
of that earlier parameter).

ps. I added the nice folks at Evolution in CC, as some of these code
warriors are planning to migrate some of newer features in camel-lite to
upstream camel.

-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog



Index: libtinymail-camel/camel-lite/camel/providers/imap/camel-imap-store.c
===
--- libtinymail-camel/camel-lite/camel/providers/imap/camel-imap-store.c	(revision 2841)
+++ libtinymail-camel/camel-lite/camel/providers/imap/camel-imap-store.c	(working copy)
@@ -702,6 +702,8 @@
 	{ CONDSTORE,  IMAP_CAPABILITY_CONDSTORE },
 	{ IDLE, 	IMAP_CAPABILITY_IDLE },	
 	{ BINARY, 	IMAP_CAPABILITY_BINARY },
+	{ QRESYNC, 	IMAP_CAPABILITY_QRESYNC },
+
 	{ NULL, 0 }
 };
 
Index: libtinymail-camel/camel-lite/camel/providers/imap/camel-imap-store.h
===
--- libtinymail-camel/camel-lite/camel/providers/imap/camel-imap-store.h	(revision 2841)
+++ libtinymail-camel/camel-lite/camel/providers/imap/camel-imap-store.h	(working copy)
@@ -119,6 +119,7 @@
 #define IMAP_CAPABILITY_CONDSTORE		(1  12)
 #define IMAP_CAPABILITY_IDLE			(1  13)
 #define IMAP_CAPABILITY_BINARY			(1  14)
+#define IMAP_CAPABILITY_QRESYNC			(1  15)
 
 #define IMAP_PARAM_OVERRIDE_NAMESPACE		(1  0)
 #define IMAP_PARAM_CHECK_ALL			(1  1)
Index: libtinymail-camel/camel-lite/camel/providers/imap/camel-imap-folder.c
===
--- libtinymail-camel/camel-lite/camel/providers/imap/camel-imap-folder.c	(revision 2841)
+++ libtinymail-camel/camel-lite/camel/providers/imap/camel-imap-folder.c	(working copy)
@@ -120,6 +120,9 @@
 static void imap_rename (CamelFolder *folder, const char *new);
 static void imap_set_push_email (CamelFolder *folder, gboolean setting);
 
+static int process_condstore_line (CamelImapFolder *imap_folder, char *resp, CamelFolderChangeInfo *changes);
+
+
 /* message manipulation */
 static CamelMimeMessage *imap_get_message (CamelFolder *folder, const gchar *uid,
 	   CamelFolderReceiveType type, gint param, CamelException *ex);
@@ -151,9 +154,10 @@
 static GPtrArray *imap_search_by_uids	(CamelFolder *folder, const char *expression, GPtrArray *uids, CamelException *ex);
 static void   imap_search_free  (CamelFolder *folder, GPtrArray *uids);
 static int imap_get_local_size (CamelFolder *folder);
-
 static void imap_thaw (CamelFolder *folder);
 
+static void camel_imap_folder_changed_for_uids (CamelFolder *folder, GPtrArray *expunged, CamelFolderChangeInfo *changes);
+
 static CamelObjectClass *parent_class;
 
 static GData *parse_fetch_response (CamelImapFolder *imap_folder, char *msg_att);
@@ -406,6 +410,12 @@
 	return retval;
 }
 
+char*
+camel_imap_folder_get_highestmodseq (CamelImapFolder *imap_folder)
+{
+	return get_highestmodseq (imap_folder);
+}
+
 /* Called with the store's connect_lock locked */
 
 
@@ -444,6 +454,7 @@
 	char *resp, *phighestmodseq = NULL, *highestmodseq = NULL;
 	CamelImapStore *store = CAMEL_IMAP_STORE (folder-parent_store);
 	gboolean removals = FALSE, condstore = FALSE, needtoput=FALSE, suc=FALSE;
+	CamelFolderChangeInfo *changes = NULL;
 
 	count = camel_folder_summary_count (folder-summary);
 
@@ -522,7 +533,38 @@
  * else tries to interpret it. */
 g_free (response-untagged-pdata[i]);
 g_ptr_array_remove_index (response-untagged, i--);
+			} else if (!g_ascii_strncasecmp (resp,  FETCH, 6)) {
+/* QRESYNC! Sanity! 
+ * Making the copy sucks a little bit, but I'd
+ * having to rewrite the parsing thing of this
+ * Camel IMAP crap :-) */
+char *nresp = g_strdup_printf (* %s, resp);
+if (changes == NULL)
+	changes = camel_folder_change_info_new();
+if (process_condstore_line (imap_folder, nresp, changes) == -1)
+	g_warning (Invalid QRESYNC response: (%s), nresp);
+g_free (nresp);
 			}
+
+		} else

Re: [Evolution-hackers] There's no need to be so hard on iconv

2007-10-12 Thread Philip Van Hoof
On Thu, 2007-10-11 at 12:14 +0200, Philip Van Hoof wrote:
 I also have this one, for example
 
 Subject: =?ISO-2022-JP?B?GyRCJSIlaiUsLUUtRRsoQg==?= (
 =?ISO-2022-JP?B?GyRCITchJiZYISYbKEI=?=)o 
 =?ISO-2022-JP?B?GyRCIlwbKEI=?=
 =?ISO-2022-JP?B?GyRCIiglUSE8JXMbKEI=?= !!
 =?ISO-2022-JP?B?GyRCISMhJhsoQg==?= :*: =?ISO-2022-JP?B?GyRCISYbKEI=?=
 =?ISO-2022-JP?B?GyRCISwheRsoQg==?=
 
 I know that the characters  (, )o ,  !! and  :*:  should not be
 there (at least, I think they shouldn't), I guess this is done by spam
 bots to confuse spamassassin (not sure, though).
 
 This check in the code (line 842 in camel-mime-utils.c) makes any such
 string become the base64 one. This is of course really not readable for
 normal human beings (although, it depends on what you call a normal
 one).
 
 842: .. /* quick check to see if this could possibly be a real encoded word */
 843: ..  if (len  8 || !(in[0] == '='  in[1] == '?'  in[len-1] == '='  
 in[len-2] == '?')) {
 844: ..   d(printf(invalid\n));
 845: ..   return NULL;
 846: ..  }
 
 When just trying to decode the string, ignoring the check, it does work
 quite well. At least for this case.
 
 I'm attaching yet another patch that ignores this check.


The entire check should not be ignored. The len8 check, for example, is
important. So don't commit this patch :-)



 On Thu, 2007-10-11 at 11:52 +0200, Philip Van Hoof wrote:
  In case iconv does not succeed in decoding for example the Subject
  header, it returns the base64 encoded one. That is is obviously not
  readable at all. The decoded one after base64 decoding (which did
  succeed in my test case) or whatever iconv could recover from it, sounds
  like a better option.
  
  This changeset (patch) on camel-mime-utils.c deals with the error
  situation (in case iconv did not return -1) by returning what(ever)
  iconv could recover from the string:
  
  http://tinymail.org/trac/tinymail/changeset/2830#file2
  
  I attached:
  
  svn diff libtinymail-camel/camel-lite/camel/camel-mime-utils.c -r 2829  
  /home/pvanhoof/diff.diff
  
  This is the Subject line of my test target:
  
  Subject: =?ISO-2022-JP?B?GyRCM048QiRLOkdEY0Z8NWsbKEI=?=
  
  =?ISO-2022-JP?B?GyRCIzJLfDFfIUFGfEonJCQkRyQqRU8kN0NXJDckXiQ5ISMlYSE8GyhC?=
  =?ISO-2022-JP?B?GyRCJWslXiUsJTglcxsoQg==?=
  
  
  I also opened a bug for this one:
  http://bugzilla.gnome.org/show_bug.cgi?id=485677
  
  
  ___
  Evolution-hackers mailing list
  Evolution-hackers@gnome.org
  http://mail.gnome.org/mailman/listinfo/evolution-hackers
 ___
 Evolution-hackers mailing list
 Evolution-hackers@gnome.org
 http://mail.gnome.org/mailman/listinfo/evolution-hackers
-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] There's no need to be so hard on iconv

2007-10-11 Thread Philip Van Hoof
In case iconv does not succeed in decoding for example the Subject
header, it returns the base64 encoded one. That is is obviously not
readable at all. The decoded one after base64 decoding (which did
succeed in my test case) or whatever iconv could recover from it, sounds
like a better option.

This changeset (patch) on camel-mime-utils.c deals with the error
situation (in case iconv did not return -1) by returning what(ever)
iconv could recover from the string:

http://tinymail.org/trac/tinymail/changeset/2830#file2

I attached:

svn diff libtinymail-camel/camel-lite/camel/camel-mime-utils.c -r 2829  
/home/pvanhoof/diff.diff

This is the Subject line of my test target:

Subject: =?ISO-2022-JP?B?GyRCM048QiRLOkdEY0Z8NWsbKEI=?=

=?ISO-2022-JP?B?GyRCIzJLfDFfIUFGfEonJCQkRyQqRU8kN0NXJDckXiQ5ISMlYSE8GyhC?=
=?ISO-2022-JP?B?GyRCJWslXiUsJTglcxsoQg==?=


I also opened a bug for this one:
http://bugzilla.gnome.org/show_bug.cgi?id=485677


-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog



Index: libtinymail-camel/camel-lite/camel/camel-mime-utils.c
===
--- libtinymail-camel/camel-lite/camel/camel-mime-utils.c	(revision 2829)
+++ libtinymail-camel/camel-lite/camel/camel-mime-utils.c	(working copy)
@@ -904,7 +904,14 @@
 	e_iconv (ic, NULL, 0, outbuf, outlen);
 	*outbuf = 0;
 	decoded = g_strdup (outbase);
+} else {
+	perror (iconv);
+	e_iconv (ic, NULL, 0, outbuf, outlen);
+	*outbuf = 0;
+	decoded = g_strdup (outbase);
+	/* decoded = g_strdup (inbuf); */
 }
+
 e_iconv_close (ic);
 			} else {
 w(g_warning (Cannot decode charset, header display may be corrupt: %s: %s,
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] There's no need to be so hard on iconv

2007-10-11 Thread Philip Van Hoof

I also have this one, for example

Subject: =?ISO-2022-JP?B?GyRCJSIlaiUsLUUtRRsoQg==?= (
=?ISO-2022-JP?B?GyRCITchJiZYISYbKEI=?=)o =?ISO-2022-JP?B?GyRCIlwbKEI=?=
=?ISO-2022-JP?B?GyRCIiglUSE8JXMbKEI=?= !!
=?ISO-2022-JP?B?GyRCISMhJhsoQg==?= :*: =?ISO-2022-JP?B?GyRCISYbKEI=?=
=?ISO-2022-JP?B?GyRCISwheRsoQg==?=

I know that the characters  (, )o ,  !! and  :*:  should not be
there (at least, I think they shouldn't), I guess this is done by spam
bots to confuse spamassassin (not sure, though).

This check in the code (line 842 in camel-mime-utils.c) makes any such
string become the base64 one. This is of course really not readable for
normal human beings (although, it depends on what you call a normal
one).

842: .. /* quick check to see if this could possibly be a real encoded word */
843: ..  if (len  8 || !(in[0] == '='  in[1] == '?'  in[len-1] == '='  
in[len-2] == '?')) {
844: .. d(printf(invalid\n));
845: .. return NULL;
846: ..  }

When just trying to decode the string, ignoring the check, it does work
quite well. At least for this case.

I'm attaching yet another patch that ignores this check.


On Thu, 2007-10-11 at 11:52 +0200, Philip Van Hoof wrote:
 In case iconv does not succeed in decoding for example the Subject
 header, it returns the base64 encoded one. That is is obviously not
 readable at all. The decoded one after base64 decoding (which did
 succeed in my test case) or whatever iconv could recover from it, sounds
 like a better option.
 
 This changeset (patch) on camel-mime-utils.c deals with the error
 situation (in case iconv did not return -1) by returning what(ever)
 iconv could recover from the string:
 
 http://tinymail.org/trac/tinymail/changeset/2830#file2
 
 I attached:
 
 svn diff libtinymail-camel/camel-lite/camel/camel-mime-utils.c -r 2829  
 /home/pvanhoof/diff.diff
 
 This is the Subject line of my test target:
 
 Subject: =?ISO-2022-JP?B?GyRCM048QiRLOkdEY0Z8NWsbKEI=?=
 
 =?ISO-2022-JP?B?GyRCIzJLfDFfIUFGfEonJCQkRyQqRU8kN0NXJDckXiQ5ISMlYSE8GyhC?=
 =?ISO-2022-JP?B?GyRCJWslXiUsJTglcxsoQg==?=
 
 
 I also opened a bug for this one:
 http://bugzilla.gnome.org/show_bug.cgi?id=485677
 
 
 ___
 Evolution-hackers mailing list
 Evolution-hackers@gnome.org
 http://mail.gnome.org/mailman/listinfo/evolution-hackers
-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog



Index: libtinymail-camel/camel-lite/camel/camel-mime-utils.c
===
--- libtinymail-camel/camel-lite/camel/camel-mime-utils.c	(revision 2830)
+++ libtinymail-camel/camel-lite/camel/camel-mime-utils.c	(working copy)
@@ -842,7 +842,7 @@
 	/* quick check to see if this could possibly be a real encoded word */
 	if (len  8 || !(in[0] == '='  in[1] == '?'  in[len-1] == '='  in[len-2] == '?')) {
 		d(printf(invalid\n));
-		return NULL;
+		/* return NULL; */
 	}
 	
 	/* skip past the charset to the encoding type */
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] There's no need to be so hard on iconv

2007-10-11 Thread Philip Van Hoof
On Thu, 2007-10-11 at 13:10 -0400, Jeffrey Stedfast wrote:
 On Thu, 2007-10-11 at 17:14 +0200, Philip Van Hoof wrote:
  On Thu, 2007-10-11 at 10:52 -0400, Jeffrey Stedfast wrote:
   I have far better fixes in GMime that need to be ported to Camel.
   
  
  Porting GMime to Camel would be an interesting effort indeed. Perhaps
  just replacing CamelMimePart with GMime.
 
 Well... I hadn't exactly meant that it would be a good idea to
 necessarily replace Camel's MIME parser/objects with GMime. I had simply
 meant for Camel to lift GMime's rfc2047 decoder logic which does more
 with charset fallback than Camel currently does... plus it also is a bit
 more liberal in decoding malformed rfc2047 encoded-word tokens (well,
 assuming ENABLE_RFC2047_WORKAROUNDS is defined...).

Aha, so we can just look at what we find in the  #ifdef
ENABLE_RFC2047_WORKAROUNDS #endif blocks and learn for your effort :)

Cool, thanks. (note. isn't spruce GPL and Camel LGPL?)

-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Copyright of Camel's individual source files

2007-10-09 Thread Philip Van Hoof
On Tue, 2007-10-09 at 10:48 -0400, Jeffrey Stedfast wrote:
 It was supposed to be GPLv2 or LGPLv2 (forget which), but without the
 or later clause.

For what it's worth, it would be more easy for projects like OpenChange
and Tinymail if the work would either be dual licensed as LGPL v2 and
LGPL v3 or with the or later clause.

The problem would be that otherwise if the authors of these libraries
would want to move their work to a newer version of the LGPL license,
Camel's license might turn out to be incompatible with this.

Which is something to avoid, I think.


 On Tue, 2007-10-09 at 16:19 +0530, Srinivasa Ragavan wrote:
  Philip,
  
  This is observed in Evolution also. The OpenChange hackers brought to
  our notice and I'm with the Novell legal team to get this resolved
  altogether. But that process seems like taking time and I have to wait a
  but before doing anything.
  
  -Srini.
  
  On Mon, 2007-10-08 at 12:08 +0200, Philip Van Hoof wrote:
   Hi there,
   
   The README.COPYRIGHT of EDS's Camel states:
   
* This program is free software; you can redistribute it and/or 
* modify it under the terms of the GNU General Public License as 
* published by the Free Software Foundation; either version 2 of the
* License, or (at your option) any later version.
   
   Whereas a lot of files (like, camel-address.c, to pick one example) state:
   
* This program is free software; you can redistribute it and/or
* modify it under the terms of version 2 of the GNU Lesser General Public
* License as published by the Free Software Foundation.
   
   It looks like EDS's COPYING file also uses the any later version
   version of the GPL v2.
   
   I'm not sure whether it's a good idea to have mixed licenses for one
   piece of code (being Camel). Would it be possible to change the license
   of all of EDS's files to be the same?
   
   Note that Novell/Ximian seems to be the copyright holder of all files,
   that of course means this organisation makes this decision.
   
   
   Thanks!
   
  
  ___
  Evolution-hackers mailing list
  Evolution-hackers@gnome.org
  http://mail.gnome.org/mailman/listinfo/evolution-hackers
 
 ___
 Evolution-hackers mailing list
 Evolution-hackers@gnome.org
 http://mail.gnome.org/mailman/listinfo/evolution-hackers
-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] [Fwd: Re: Merging work] was: Let the porting

2007-10-08 Thread Philip Van Hoof
Hi there Evo hackers,

My fellow Tinymail coder Jose Dapena Paz hinted me which pieces of
camel-lite's camel-mime-utils.c needed patching work and which pieces
could come from upstream to camel-lite.

This was one of the few remaining files that needed a synchronisation
between upstream Camel and camel-lite. You can see a changeset here:

http://tinymail.org/trac/tinymail/changeset/2828

All other files in the root directory of Camel's sources now only
contain for camel-lite necessary changes and now have all recent
upstream-Camel changes included. You can find this root dir here:

https://svn.tinymail.org/svn/tinymail/trunk/libtinymail-camel/camel-lite/camel/ 

versus

http://svn.gnome.org/viewcvs/evolution-data-server/trunk/camel/


I don't think a lot in upstream-Camel's providers has since changed, but
the code in the providers/ directory has not yet been synchronised with
upstream. Most of the changes in that directory are specific and related
to the new features in camel-lite.


-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog



---BeginMessage---

El lun, 08-10-2007 a las 00:15 +0200, Philip Van Hoof escribió:
 Hey Jose,
 
 Would it be possible for you to check the differences of upstream
 Camel's camel-mime-utils.c with camel-lite's. I refrained from
 synchronising our camel-lite's camel-mime-utils.c with upstream Camel
 because there might have been changes that you made to support the
 attachment purging. I've found some changes that were unknown to me.

I've been reviewing the svn log, and it seems I didn't make any change
in camel mime utils (and after reviewing the differences between
upstream camel and tinymail camel I still don't find anything I can
remember).

Anyway, briefly, the changes are:
* A lot of compilation warning fixes, using proper typecasts. They
should go upstream.
* The change in parenthesis and conditions in line 795 present in
upstream camel seems to be interesting for us, as it seems an obvious
error.
* Our fix for escaping ? seems ok at lines 1155 and 1168. Should go
upstream.
* You changed CamelContentType instantiation to use g slice. Should be
upstream unless it breaks something there for backwards compatibility.
* We should get the change at line 2097 upstream. But it seems even
upstream is not ok, as the *inptr validation should be 
(*inptr  camel_mime_is_dtext(*inptr))
instead of 
(camel_mime_is_dtext(*inptr)  *inptr)

In general there are no traumatic changes, and they are all more or
less wise both in our side and on camel upstream side. So I would say we
should make these changes go upstream in official camel (the only doubt
would be the GSlice issue).

-- 
Jose Dapena Paz [EMAIL PROTECTED]
Igalia
___
tinymail-devel-list mailing list
[EMAIL PROTECTED]
http://mail.gnome.org/mailman/listinfo/tinymail-devel-list
---End Message---
---BeginMessage---
Hey Jose,

Would it be possible for you to check the differences of upstream
Camel's camel-mime-utils.c with camel-lite's. I refrained from
synchronising our camel-lite's camel-mime-utils.c with upstream Camel
because there might have been changes that you made to support the
attachment purging. I've found some changes that were unknown to me.

This is the upstream version:
http://svn.gnome.org/viewcvs/evolution-data-server/trunk/camel/camel-mime-utils.c?view=log

I can recommend KDiff3 for this, but you can use any tool you like of
course.

Obviously would it be nice to keep your changes and merge any possible
new changes applied to the upstream version to our version. The idea is
to make it possible to get a clean and nice patch using just diff -r.
That patch should then show which exact changes that we made to create
our features (like your attachment purger). That way upstream Evolution
can filter out what they like and bring it to Evolution too.


ps. I'm adding the Tinymail mailing list in CC to keep the rest of the
world updated on this.


-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog



Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
tinymail-devel-list mailing list
[EMAIL PROTECTED]
http://mail.gnome.org/mailman/listinfo/tinymail-devel-list
---End Message---
---BeginMessage---

Thanks

On Mon, 2007-10-08 at 08:15 +0200, Jose Dapena Paz wrote:
 El lun, 08-10-2007 a las 00:15 +0200, Philip Van Hoof escribió:
  Hey Jose,
  
  Would it be possible for you to check the differences of upstream
  Camel's camel-mime-utils.c with camel-lite's. I refrained from
  synchronising our camel-lite's camel-mime-utils.c with upstream Camel
  because there might have been changes that you

[Evolution-hackers] Copyright of Camel's individual source files

2007-10-08 Thread Philip Van Hoof
Hi there,

The README.COPYRIGHT of EDS's Camel states:

 * This program is free software; you can redistribute it and/or 
 * modify it under the terms of the GNU General Public License as 
 * published by the Free Software Foundation; either version 2 of the
 * License, or (at your option) any later version.

Whereas a lot of files (like, camel-address.c, to pick one example) state:

 * This program is free software; you can redistribute it and/or
 * modify it under the terms of version 2 of the GNU Lesser General Public
 * License as published by the Free Software Foundation.

It looks like EDS's COPYING file also uses the any later version
version of the GPL v2.

I'm not sure whether it's a good idea to have mixed licenses for one
piece of code (being Camel). Would it be possible to change the license
of all of EDS's files to be the same?

Note that Novell/Ximian seems to be the copyright holder of all files,
that of course means this organisation makes this decision.


Thanks!

-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] Let the porting begin

2007-10-07 Thread Philip Van Hoof
Hi there,

I got notified that some Evolution hackers are interested in bringing
the features of camel-lite to camel. This is of course excellent news!

To ease this process, I pushed all the recent changes that happened to
camel, to camel-lite. These changes include the copyright's address
changes, the changes Matthew Barnes did on the BASE64 decoding, most of
the compiler-warning changes (although a lot of them had already been
put in camel-lite, of course), Kjartan Maraas's 0 vs. NULL changes,
Milan Crha's changes (set-label, compiler warnings, etc).

Using this changeset you can follow the changes to camel-lite:

http://tinymail.org/trac/tinymail/changeset/2823


Note about compiler warnings: camel-lite contains fixes for -pedantic
warnings. When making differences, you might find compiler-warning fixes
that you wouldn't expect with -Wall. Camel-lite right now has two or
three things that are problematic and not yet fixed for -pedantic:

 - Enumerations that exceed the size of an int, which are not allowed in
   ANSI C (or C99, I don't remember exactly).

 - Casting function pointers to (void*) which also isn't allowed (but I
   have no idea how else I can store them in a GHashTable, which is what
   Camel is doing at some locations)


Some of the changes to camel-lite might not be welcomed to camel. Among
them I expect the changes to camel-object not to be interested (some of
these changes cut one or two bytes from the CamelObject's size and put
some more and also less locking-on-signal-emissions in place). Others
are more clear like features that are just not interesting. And yet
others are just not done in a clean way.

On future plans for camel-lite: I'm interested in implementing QRSYNC
next to the CONDSTORE support. Although only one IMAP server already
supports this (Isode's Mbox).


-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Let the porting begin

2007-10-07 Thread Philip Van Hoof
On Sun, 2007-10-07 at 14:15 +0200, Philip Van Hoof wrote:
 Hi there,

 Using this changeset you can follow the changes to camel-lite:
 
   http://tinymail.org/trac/tinymail/changeset/2823

More such commits to Tinymail's camel-lite happened today. Nearly all of
those are merging work in action. After today, using KDiff3 or any other
tool that can recursively compare two directories should be painless and
straight forward on ./libtinymail-camel/camel-lite/camel
vs. ./evolution-data-server/camel


-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] Patch that fixes compilation warnings on iconv-detect.c

2007-10-04 Thread Philip Van Hoof
These compilation warnings prevent -Wall -Werror as the configure stage
fails if CFLAGS is set that way.

http://bugzilla.gnome.org/show_bug.cgi?id=483301

-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] Memory leak in Camel's CamelDiscoFolder's getv

2007-09-26 Thread Philip Van Hoof

There's a memory leak in camel-disco-folder.c

static int
disco_getv(CamelObject *object, CamelException *ex, CamelArgGetV *args)
{
int i, count=0;
guint32 tag;

for (i=0;iargs-argc;i++) {
CamelArgGet *arg = args-argv[i];

tag = arg-tag;

switch (tag  CAMEL_ARG_TAG) {
case CAMEL_OBJECT_ARG_PERSISTENT_PROPERTIES:
case CAMEL_FOLDER_ARG_PROPERTIES: {
CamelArgGetV props;

props.argc = 1;
props.argv[0] = *arg;
((CamelObjectClass *)parent_class)-getv(object, ex, 
props);
---*arg-ca_ptr = g_slist_concat(*arg-ca_ptr, 
g_slist_copy(disco_folder_properties));
break; }
/* disco args */
case CAMEL_DISCO_FOLDER_ARG_OFFLINE_SYNC:
*arg-ca_int = ((CamelDiscoFolder 
*)object)-offline_sync;
break;
default:
count++;
continue;
}

arg-tag = (tag  CAMEL_ARG_TYPE) | CAMEL_ARG_IGNORE;
}

if (count)
return ((CamelObjectClass *)parent_class)-getv(object, ex, 
args);

return 0;
}


==15153== 53,610 (11,989 direct, 41,621 indirect) bytes in 323 blocks are 
definitely lost in loss record 184 of 195
==15153==at 0x4020626: malloc (vg_replace_malloc.c:149)
==15153==by 0x4CECE9C: g_malloc (in 
/targets/w37/usr/lib/libglib-2.0.so.0.1200.12)
==15153==by 0x4CFD38A: g_slice_alloc (in 
/targets/w37/usr/lib/libglib-2.0.so.0.1200.12)
==15153==by 0x4CFDC51: g_slist_copy (in 
/targets/w37/usr/lib/libglib-2.0.so.0.1200.12)
==15153==by 0x4126F53: disco_getv (camel-disco-folder.c:205)
==15153==by 0x684ECFD: imap_getv (camel-imap-folder.c:761)
==15153==by 0x419E298: camel_object_get (camel-object.c:1574)
==15153==by 0x419E392: cobject_state_write (camel-object.c:517)
==15153==by 0x419EA69: camel_object_state_write (camel-object.c:1834)
==15153==by 0x4127459: disco_sync (camel-disco-folder.c:282)
==15153==by 0x413663B: camel_folder_sync (camel-folder.c:311)
==15153==by 0x414EB44: store_sync (camel-store.c:690)


-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] Unexepected blocking reads in Camel

2007-07-02 Thread Philip Van Hoof
Hey Camelers,

I'm not sure but it looks like if any Camel API call that causes a read
or a write that isn't wrapped by a CamelOperation, does a blocking read
or write that don't respect the timeouts.

For example (I'll comment inline with '##' characters):

ssize_t
camel_read (int fd, char *buf, size_t n)
{
ssize_t nread;
int cancel_fd;

if (camel_operation_cancel_check (NULL)) {
errno = EINTR;
return -1;
}
#ifndef G_OS_WIN32
cancel_fd = camel_operation_cancel_fd (NULL);
#else
cancel_fd = -1;
#endif
if (cancel_fd == -1) {

##
## So in case the cancel_fd is not present, then do a normal read
##

do {
nread = read (fd, buf, n);
} while (nread == -1  (errno == EINTR || errno == EAGAIN || 
errno == EWOULDBLOCK));
} else {
#ifndef G_OS_WIN32
int errnosav, flags, fdmax;
fd_set rdset;

flags = fcntl (fd, F_GETFL);
fcntl (fd, F_SETFL, flags | O_NONBLOCK);

do {

##
## Else add the cancel_fd to a select, and either wait for a timeout, a
## successful read or the cancel_fd being set (right?)
##

struct timeval tv;
int res;

FD_ZERO (rdset);
FD_SET (fd, rdset);
FD_SET (cancel_fd, rdset);
fdmax = MAX (fd, cancel_fd) + 1;

###
### The thing is the IO_TIMEOUT. It's respected in case of cancel_fd being
### set. It's not looked at in case cancel_fd was -1. Which is significantly
### different, of course (and probably unexpected behaviour too).
###

tv.tv_sec = IO_TIMEOUT;
tv.tv_usec = 0;
nread = -1;

res = select(fdmax, rdset, 0, 0, tv);
if (res == -1)
;
else if (res == 0)
errno = ETIMEDOUT;
else if (FD_ISSET (cancel_fd, rdset)) {
errno = EINTR;
goto failed;
} else {
do {
nread = read (fd, buf, n);
} while (nread == -1  errno == EINTR);
}
} while (nread == -1  (errno == EINTR || errno == EAGAIN || 
errno == EWOULDBLOCK));
failed:
errnosav = errno;
fcntl (fd, F_SETFL, flags);
errno = errnosav;
#endif
}

return nread;
}



I created a patch for camel-lite that does the (what I think is)
expected behaviour.

http://tinymail.org/trac/tinymail/changeset/2349?format=diff


-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Unexepected blocking reads in Camel

2007-07-02 Thread Philip Van Hoof
On Mon, 2007-07-02 at 12:53 -0600, Veerapuram Varadhan wrote:
 On Mon, 2007-07-02 at 20:40 +0200, Philip Van Hoof wrote:

  I'm not sure but it looks like if any Camel API call that causes a read
  or a write that isn't wrapped by a CamelOperation, does a blocking read
  or write that don't respect the timeouts.
  
  I created a patch for camel-lite that does the (what I think is)
  expected behaviour.
  
 IIRC, Camel calls are blocking and they are not supposed to be called in
 main thread - First lesson in Camel.  So, it is the expected behavior of
 Camel and Yes, if you have CamelOperation specified, they are
 cancellable.

It's not the fact that they are blocking that disturbs me, what is
unexpected, I think, is the fact that the IO_TIMEOUT is only respected
in case of cancel_fd being available (in case the operation is wrapped
by a CamelOperation).

In the other case, the IO_TIMEOUT isn't respected and therefore it's
possible for a thread to for ever block and wait (for example in case
the connection with the IMAP server unexpectedly disappears).

At least that is what I observed, and what the patch fixes

ps. The patch doesn't make the read and writes non-blocking, as all of
the non blocked reads and writes are being looped until what we wanted
to read, has been read or in case the IO_TIMEOUT occurs (just like when
the cancel_fd is available, only without the cancel_fd in FD_SET).


-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Overview of camel-lite

2007-06-14 Thread Philip Van Hoof

On top of all these it will be very interesting to see what the latest
bugfix-style patches to camel-lite were.

The reason is that we are in group coding on Tinymail (and a product
that works on top of Tinymail) for a few weeks.

We have found a lot of problems (while valgrinding a lot of situations)
in both the POP3 and IMAP code of camel-lite. These where not only
problems that are in the code that implements one of these new features.

Of the more serious ones where a complete lack of locking in the POP3
store and folder, whereas this is very much needed for multiple of the
struct members of the types being used.


I have also found a security bug (actually a few, but one is very
serious) a few weeks ago. I first contacted Jeffrey and I think Matthew
too, but I just looked and the bug has not yet been fixed nor have I
seen a lot of action.

Therefore I'm stopping the silence (for security) and making a bug,
marking it as a security one that is critical (as it can be used to
remotely execute code on a victim's computer).

http://bugzilla.gnome.org/show_bug.cgi?id=447414

Please fix this asap and contact distributors of Evolution about it.



On Mon, 2007-06-11 at 10:57 +0300, Philip Van Hoof wrote:
 I'm going to make an overview of camel-lite so that if people read the
 discussions here, that they know what it's all about.
 
 Camel-lite is the fork of Camel that Tinymail is using internally right
 now. Camel-lite's code is in my opinion not really in a good shape, it
 more or less implements certain features that are necessary for
 Tinymail.
 
 Bad things:
 
  * Some of the features have been done in a hackish way. 
 
  * Mostly this had to do with non-blocked reads not being very well
supported within the code and design of upstream camel
 
  * Removes support for a few for Evolution very important but for
Tinymail unused features
 
  * It's a fork (effectively, yes)
 
 Neutral:
 
  * I am trying to get some of the changes in upstream Camel. Although
I'm selective because I don't want to bore the Evolution team with
features that they'll most likely not consider for inclusion anyway.
 
My point of view is that if they show interested, I'll bump the
priority of trying to clean it up and getting it upstream
 
 Support:
 
  * Push E-mail through IMAP IDLE, IMAP commands shutdown and restart the
IDLE state of the connection. A non-blocking read handles EXISTS,
FETCH and EXPUNGE events (you see messages being added almost
instantly when they either get added or received)
 
  * Better detecting of EXPUNGE through / partly caused by IMAP IDLE (you
see your messages getting removed almost instantly if you remove
messages from the folder from another E-mail client)
 
  * Better detecting and handling of FETCH through / party caused by IMAP
IDLE (flag changes are effectively working instantly with for example
Cyrus: you see your view changing almost instantly if you work from
another E-mail client with your folder)
 
  * Better progress indicating. Camel-lite doesn't use percentages but n
of nth values. For example bytes of a message or items of a summary
download
 
  * Support for BINARY fetching messages, which is more bandwidth
efficient in some situations 
 
  * Support for CONDSTORE which is far more bandwidth efficient when
synchronising folders
 
  * Support for receiving a message while summary is being downloaded.
Camel-lite creates for this single-use-case a new socket (this is a
very new feature that is being bug-fixed these weeks)
 
  * Support for displaying the recently received headers early while
folder summary is being downloaded (very neat, and extremely useful
with the parallel receiving of a message being functional now)
 
  * Stores already-received summary information very early, recovers from
already stored information (saves bandwidth in case of a connection
failure)
 
  * Makes the selection (the amount of columns or fields) of the summary
download a lot smaller
 
  * Doesn't store a message temporarily in memory while retrieving it:
immediately streams from the socket to the file stream in the cache.
This is obviously an important memory-usage fix for large messages on
mobile devices.
 
  * Implements summary support for POP3, makes the POP3 provider of Camel
a full and real CamelStore that can be used just like the IMAP one.
This, by the way, uses TOP for retrieving and generating the summary
 
  * Implements partial message retrieval (only retrieving the readable
body mime part of a message) for both IMAP and POP3 (with POP3 the
connection is forcefully cut, which might not work well with all POP3
servers. Although this is regretfully one of the few only solutions
for this type of feature with POP3)
 
  * A lot of bug fixes and locking improvements (also a lot of new
situations to support and lock correctly, so it can also be seen as a
non

Re: [Evolution-hackers] Overview of camel-lite

2007-06-11 Thread Philip Van Hoof
On Mon, 2007-06-11 at 10:54 +0200, Gilles Dartiguelongue wrote:
 Le lundi 11 juin 2007 à 10:57 +0300, Philip Van Hoof a écrit :
   * All compilation warnings fixed (we usually compile with -Wall and
 -Werror). Fixing this caused four major bug fixes in initialisation
  of variables in Camel. 
 
 ok, right away, let me just say evo team is interested in fixing as much
 compilation warnings as possible. In last week's meeting, Matthew told
 me he was especially interested in fixes for camel and I was about to
 start to work on it but if you have it all done already... :) 

I posted quite a lot of patches a few months ago with all these
compilation warning fixes.

You can search for compilation warnings in camel


-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Introduction and Questions

2007-06-09 Thread Philip Van Hoof
On Fri, 2007-06-08 at 20:45 -0400, Jeffrey Stedfast wrote:
 On Fri, 2007-06-08 at 18:34 +0300, Philip Van Hoof wrote:
  The imap4 project is making things look better, though all in all it's
  still much of the same (blocking and waiting for results, in stead of
  letting the server do most the work and do pipelining).
 
 adding pipelining support would not be difficult to do, actually.

I noticed that, indeed. Last time I was reading it, I did notice a few
things in its code that would make it difficult.

All in all, the ideas behind the 'imap4' way of working (with the IMAP
server) are indeed a lot better than the 'imap' code.

Maybe it would be a good idea to bring features like condstore and idle
to it? If there are some more people interested in this, I would
probably help out too. Unlike the stuff that I have put in the 'imap'
code, this should be done in a proper non-hackish way then, imo.

What I disliked most about Camel's 'imap' code, though, is the fact that
the sequences have to correspond to the array indexes of the
CamelFolderSummary. It sounds like it would have been more easy if that
was a key in a hashtable.

For example if a message got expunged that had a sequence value in the
beginning of the folder, it right now more or less means that the entire
folder has to be re-synchronised. While the majority of the local data
can be perfectly corrected in stead too.

Luckily very few people often remove things in the beginning of their
mail folders (since those E-mails are usually the older emails).

I have some ideas on having blocks of mmap data for CamelFolderSummary,
that for example contain ~ 1000 items per block, and having reference
counting per block.

That way I could create a vfolder-like feature that keeps only the
blocks alive that contain summary items that matched the query. In stead
of having to keep all the memory of each folder in the vfolder alive.

I have a prototype of this somewhat working, although done extremely
hackish too. So at some point I'm going to rewrite this first.

Another benefit is that the blocks will never have to be rewritten:
removals are simply marked until  50% of the block is removed (and then
the entire block is rewritten), flag changes are stored in a different
file and on push-email events or a summary download, the latest headers
are kept in normal memory until there are enough of them to start a new
block.

Rewriting the summary file truly is a hit on performance on some
devices. On Flash it also introduces some level wearing. Which is why
it's another benefit.



-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Introduction and Questions

2007-06-08 Thread Philip Van Hoof
On Thu, 2007-06-07 at 09:25 -0400, Jeffrey Stedfast wrote:

 On Thu, 2007-06-07 at 10:56 +0300, Philip Van Hoof wrote:
 
  The best way is to ask for the ENVELOPE and the remaining info using the
  normal BODY.PEEK method.
 
 Have you actually ever tested this theory? or did you just pull this out
 of your proverbial?

I have. The ENVELOPE-way is slightly better because words like From:
and To: are not repeated each time. Although with COMPRESS and/or
support for deflate in the TLS/SSL library will probably compress that
difference away mostly.

Note I didn't test this with compression. I saw a difference of 5% to
10% depending on several factors in bandwidth saving. I usually test all
this with wireshark, which doesn't help a lot with compressed data. It
would be nice, indeed, to get some analysis of how data is being
compressed when compression by either COMPRESS or the SSL library is
being done.

Note that I haven't yet implemented COMPRESS in camel-lite, but I will
do this very soon (I have a CamelStreamGzip thingy ready for this).

On top of bandwidth, on pure speed a lot IMAP servers will return your
result faster if you only use ENVELOPE. That's because they optimise on
the fields that are in ENVELOPE (indexes, etc etc).

But then again, there's not enough in ENVELOPE to support all of
Evolution's features, so you'll still have to use BODY.PEEK which will
most likely slowdown everything back again.

 You should note that Thunderbird does not use ENVELOPE, probably because
 they found it was LESS efficient to fetch the ENVELOPE +
 BODY.PEEK[HEADER.FIELDS (REFERENCES CONTENT-TYPE)] on many existing IMAP
 servers in the wild.

Probably, indeed.

 Having myself tested this in the past, I don't recall it making much
 difference one way or the other.

On bandwidth, it does. Especially when you have a lot of messages in
some folders and the server is not compressing things (no COMPRESS nor
TLS/SSL compressing things for you).

 Sadly using ENVELOPE breaks on at least one proprietary IMAP server
 which occasionally likes to send back responses with NIL as each and
 every item, at least until a client has issued a BODY.PEEK fetch
 request.

Sounds like a bug in that server implementation.

   It should be possible to specify per folder
  which of the headers are to be fetched, to make it really efficient.
 
 how do you propose calculating the list of headers each folder should
 request? which headers would you eliminate conditionally?

I'd probably start with the references in case the user doesn't want his
headers-view to be threaded, and fetch only the sequence number plus the
references header the first time threaded is enabled on that folder and
after that (while its enabled) for each summary-item update too.

A lot mobile devices's screens are simply not suitable for threaded view
anyway: introduces way to much wasted white space. Why fetch this
information and waste the user's GPRS bandwidth and therefore often his
money too? (for a lot people, GPRS bandwidth costs money per downloaded
megabyte)

In case the From is not displayed, there's no real need to fetch it
until the first time the column is added. Same for the dates (date sent
and date received, usually only one of both is used by the user). 

Etc etc etc.
 
  The imap4 implementation seems to have an ENVELOPE parser, so I guess
  either it does it already or it will use ENVELOPE in future.
 
 it fetches:
 
 ENVELOPE BODY.PEEK[HEADER.FIELDS (Content-Type References In-Reply-To)]
 
 and optionally the mailing-list headers
 
 I believe it fetches IN-REPLY-TO (which is included in ENVELOPE)
 explicitly because some IMAP servers tried to be smart and parse the
 IN-REPLY-TO header for us and didn't handle embedded phrases which
 certain broken free-software mailers like to send.
 
  
  For a mobile device, that works over GPRS, you for example are usually
  not interested (not at all) in the mailinglist specific headers.
 
 this is mobile-device specific tho, when I was implementing imap4 and
 people started trying to use it, it was their #1 complaint (which is why
 imap4 conditionally requests the mlist headers... tho that flag may be
 hard-coded to TRUE now, I'm not sure)

But some people don't use it. Although people more often complain about
missing features and not so much about that their E-mail client is
wasting their money on GPRS bandwidth, I still believe it's important to
support low bandwidth modes too. Also for Evolution, as Evolution is
often used with laptops.

I do agree, though, that Evolution has a focus difference compared to
what I'm doing with Camel. In my opinion is this why splitting Camel
from Evolution might help: we could then more easily introduce features
for letting Camel switch from low bandwidth mode, to Evolution mode and
back.

Right now any change that isn't absolutely 100% super pro and for
Evolution and Evolution only, gets ignored by you guys at Novell.

Usually the explanation goes like this: Yeah but Evolution

Re: [Evolution-hackers] Introduction and Questions

2007-06-08 Thread Philip Van Hoof
On Fri, 2007-06-08 at 09:48 -0400, Jeffrey Stedfast wrote:

 I think the laptop problem is solved with the basic headers feature,
 at least as far as collecting new summary info is concerned.
 
 Syncing flags is another story, and where the real
 slowness/user-frustration lies.
 
 I'm sure David Woodhouse would agree here.

ps. The condstore work in camel-lite is the most easy part to migrate.

  Right now any change that isn't absolutely 100% super pro and for
  Evolution and Evolution only, gets ignored by you guys at Novell.
 
 I would argue that not to be the case. I would instead argue that the
 reason many of your patches haven't been incorporated into Camel is
 that they are hackish and/or tailored explicitly for /your/ needs, and
 not for the good of the many.

I agree that a lot of the things are hackish, some of the changes are
this way because there are some design problems in Camel that make
certain things not possible without such hacks.

For example the fact that non-blocking read()s are not possible by the
default CamelStream things, and that none of the existing code really
copes very well with non-blocking reads.

The imap4 project is making things look better, though all in all it's
still much of the same (blocking and waiting for results, in stead of
letting the server do most the work and do pipelining).

 That's not the one I was thinking of, actually... I was more thinking
 of the BODY(STRUCTURE) stuff.

Camel-lite doesn't use BODYSTRUCTURE, it only uses body.peek[0] for
doing a partial retrieval. I'm, however, planning to let the
CamelMimePart late fetch not-yet retrieved parts when needed (in stead
of always downloading the full, or only the body part of the message).

That would conflict with IMAP servers that don't do mime parsing right.

 Then don't be so negative. I'd hardly say that imap4 has a majority of
 its design wrong.

 Try some constructive criticism for a change...

You are right, let's try to criticise constructive. My apologises for
the negative way of putting that I don't like how imap4 is designed.


-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Introduction and Questions

2007-06-07 Thread Philip Van Hoof
On Thu, 2007-06-07 at 08:25 +0200, Gilles Dartiguelongue wrote:
 Le jeudi 07 juin 2007 à 01:53 +0300, Philip Van Hoof a écrit :
  Without immediately writing to disk work, the download by itself will
  consume around 120 MB of RAM, and will most likely fail due to network
  timeouts and other such problems (it'll take a while, since Evolution
  fetches a ridiculous large amount of headers for each message for
  building its summary).
  
 isn't the imap-features plugin's goal to reduce/customize the amount of
 downloaded headers ? Or is it that it's still not enough ?

It improves the situation by setting your url-string to have the
basic_headers option. In the imap code of Camel, it will then ask for
less headers (but still way too much).

A major improvement it certainly isn't.

The best way is to ask for the ENVELOPE and the remaining info using the
normal BODY.PEEK method. It should be possible to specify per folder
which of the headers are to be fetched, to make it really efficient.

The imap4 implementation seems to have an ENVELOPE parser, so I guess
either it does it already or it will use ENVELOPE in future.

For a mobile device, that works over GPRS, you for example are usually
not interested (not at all) in the mailinglist specific headers.

It would also be possible to do it in multiple passes. For example get a
modest list of headers, and after that get more and more.

In any case, none of the current Evolution code implements consuming the
CONDSTORE capabilities of some modern IMAP servers (like MBox and
Cyrus).

CONDSTORE is really going to make an enormous difference in bandwidth
consumption with large folders. That's because you only need to fetch
the flags of the changed messages to synchronise flags.

Note that Tinymail's camel-lite implements all the needed solutions to
this bandwidth consumption problem. And that its code is, although a lot
of work because the Evolution maintainers didn't seem interested at the
time it was happening, definitely portable to upstream.

Its implementation include solutions for the headers, it immediately
saves the headers to disk and unlike Evolution's Camel it can resume
partial summary downloads, it wont peek memory allocation during
downloading, it implements Push E-mail using IMAP IDLE and it implements
CONDSTORE.

Although I'm definitely not satisfied with any of either Camel nor
camel-lite's IMAP code. The thing is that I'd much prefer a client-side
implementation that does proper pipelining. For example Infotrope,
Telomer and Polymer does this on an experimental basis (Infotrope is the
library).

ps. In my opinion is also the imap4 project getting the majority of its
design wrong. Although it looks better than the imap one, it's not the
best way to use an IMAP server. If a project is going to write an IMAP
client implementation 'today': they better do it right. A lot is
changing in IMAP today (Lemonade, MORG, etc). It's important to get it
right this time (the vast majority of E-mail clients totally suck at how
they use the IMAP server, including Evolution indeed).


-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Introduction and Questions

2007-06-07 Thread Philip Van Hoof
On Thu, 2007-06-07 at 03:05 -0600, Sankar P wrote:
 On Thu, 2007-06-07 at 10:56 +0300, Philip Van Hoof wrote:

  
  It improves the situation by setting your url-string to have the
  basic_headers option. In the imap code of Camel, it will then ask for
  less headers (but still way too much).
 
 May be way too much for a mobile client; but not for a desktop email
 client. Sure you dont want your desktop mail client to have threading or
 show mail-size or have such basic things ?

When I'm using Evolution on a laptop, I feel very mobile too.

btw. This is a quote that I have from Federico, I think (remember) he
said that to me last GUADEC. Although

Very often I'm indeed using my laptop at an airport or something, using
GPRS. Why isn't Evolution suitable for this use-case?

I agree, though, that Tinymail's camel-lite memory improvements are less
of a priority for Evolution. It's bandwidth work, though, is important
in my opinion. Same for the Push E-mail capabilities.

  A major improvement it certainly isn't.
 
 Try comparing the performance of fetching all headers and basic_headers
 only. Some crude data which I collected a year back is at
 http://psankar.blogspot.com/2006/05/imap-performance.html

Of course, you can look at the code to see what differs between the
basic_headers mode and the normal one. And that will obviously make a
big difference (the selection of the query is simply a lot smaller).

But if Evolution is causing 900% of its bandwidth to be unnecessary, and
the basic_headers mode saves 400% of that, there is still 500% of it
that is too much. Which is the situation with basic_headers.

The basic_headers option also doesn't implement condstore, which is
quite important to safe bandwidth (if supported by the IMAP server).

  The best way is to ask for the ENVELOPE and the remaining info using the
  normal BODY.PEEK method. It should be possible to specify per folder
  which of the headers are to be fetched, to make it really efficient.
 
 Do you really think any user on earth will really be interested in
 configuring what headers to fetch on a folder-basis ? 

The application should figure that list out automatically, obviously. 
 
  ps. In my opinion is also the imap4 project getting the majority of its
  design wrong. Although it looks better than the imap one, it's not the
  best way to use an IMAP server. If a project is going to write an IMAP
  client implementation 'today': they better do it right. A lot is
  changing in IMAP today (Lemonade, MORG, etc). It's important to get it
  right this time (the vast majority of E-mail clients totally suck at how
  they use the IMAP server, including Evolution indeed).
 
 Such a sweeping generalization !?

Maybe, but I've seen quite a lot of their code, and they are indeed very
badly programmed. Most of the custom Camel providers are in a very bad
shape too, by the way.

And ... camel-lite, camel upstream and its standard providers are also
in a bad shape :(

 As you mentioned, IMAP (like any other technology) grows at a very
 high-speed. What you consider as the right client implementation today
 may become obsolete tomorrow. When the initial IMAP provider was written
 there may not be a standard spec. for the PUSH/IDLE etc.

This is true. I agree.

 No application can have an intelligent design that will remain forever
 satisfying all new needs/changes. Application designs should just
 evolve, adapting to the new changes.

I also agree with this.

 IIRC, you had a branch on Evolution's Camel to work on whatever changes
 you wanted to make, so that you don't have to wait for the delay due to
 review/maintainer.  Would love to see some of the improvements that you
 have done ported there. So that it becomes easy for the maintainer to
 approve them and bring onto trunk.

I think the best way forward is what Matthew and I share an opinion on:

To split Camel away from Evolution, and start getting as much of
camel-lite back into that upstream version by making the new function-
ality more and more pluggable.

It would probably take a few releases to get things in upstream, though.

To do this adventure on my own, I have too much other priorities right
now. If I see somebody being interested, I will definitely help this
person and join his efforts as much as possible.


-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Introduction and Questions

2007-06-06 Thread Philip Van Hoof
On Thu, 2007-05-31 at 14:10 -0400, Jeffrey Stedfast wrote:
 On Thu, 2007-05-31 at 07:58 -0700, Ross Boylan wrote:
  Hi.  I've been getting into the code of evolution recently, and am thinking 
  of 
  doing a bit more to see if I can get it working OK for my situation.  I 
  have 
  an IMAP mailbox which is very large, both in terms of folders (over 100) 
  and 
  messages (the largest folder has 300,000 messages; my INBOX has about 
  22,000).
 
 the largest INBOX I've ever used was about ~100,000 messages, so you may
 get to have some fun :)

Without CONDSTORE support, Evolution needs ridiculous amounts of both
time and bandwidth to synchronise such a folder's flags. When online,
this happens each time you click on a folder in Evolution.
 
Without immediately writing to disk work, the download by itself will
consume around 120 MB of RAM, and will most likely fail due to network
timeouts and other such problems (it'll take a while, since Evolution
fetches a ridiculous large amount of headers for each message for
building its summary).


-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] IMAP preauthenticated

2007-03-16 Thread Philip Van Hoof

Does the PREAUTH happen before or after the STARTTLS, or aren't you
using STARTTLS?

ps. You can break on this line to check what is in buf. It's in
camel-imap-store.c around line 700 (mine is heavily patched, so the line
might differ).

static gboolean
connect_to_server (CamelService *service, struct addrinfo *ai, int
ssl_mode, CamelException *ex)
{
...
if (!strncmp(buf, * PREAUTH, 9))
store-preauthed = TRUE;
...
}


On Tue, 2007-03-13 at 14:12 -0700, David Shifflett wrote:
 How do I configure Evolution such that it doesn't
 try and send a username/password (via the LOGIN command)?
 
 My IMAP server is configured to have the users
 preauthenticated.
 The IMAP server sends PREAUTH as part of the initial
 message, but Evolution ignores this
 and tries the LOGIN command anyway,
 which of course causes IMAP to generate an error
 (it isn't expecting a LOGIN command).
 
 The IMAP server is part of a research project
 and the user has already been authenticated,
 so no login is required (or allowed).
 This IMAP server works fine with Mozilla, Thunderbird,
 and even Outlook Express.
 
 Thanks for any help,
 David Shifflett
 
 
 
 ___
 Evolution-hackers mailing list
 Evolution-hackers@gnome.org
 http://mail.gnome.org/mailman/listinfo/evolution-hackers
-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] The recent camel-lite improvements

2007-02-15 Thread Philip Van Hoof
On Wed, 2007-02-07 at 21:40 +0100, Gilles Dartiguelongue wrote:
 It'd be nice to see the work for IDLE make it to mainline too.

I would like to warn that the work for IDLE pulls nearly each and every
change to the IMAP code with it, as a dependency.

The IDLE work requires changes to dealing with expunges, changes to the
camelstream type (it needs non-blocked reading, and with the current
design of Camel there's also no other way to read the IDLE responses
correctly), changes to the imap-command type and really significant
changes to the locking of the IMAP provider's connection.

It also enjoys the condstore support, a lot. As well as that it enjoys
the rewrite of the imap_update_summary implementation. That's mostly
because the index of each summary item must be exactly correct with the
sequence on the IMAP mailbox.

Imagine that not being in sync and a push happening on that sequence. A
push like an expunge or a flags-update: the wrong message would be
changed locally.

That would be like a bug that the user experiences as data loss. So it
really really has to be absolutely correct. This is also why extra
checking has been put in place in both imap_summary_update, imap_rescan
and the new imap_rescan_condstore implementation: it *HAS* to be correct

As a developer myself, I'm totally pro pushing the changes to Evolu-
tion's Mailer component. You would indeed have instant changes and even
eventually incrementally filling up the folder-view while you are
downloading the folder(this is possible, yes. Like Polymer .. yes).

But  from a stability point of view, it *would* most likely
destabilize the product ... Evolution at this moment.

And that is must likely not interesting for Novell. Am I wrong or right
about this feeling of mine ? :-)

I would, however, aid the person that gets my work into a RD branch of
Evolution or whatever. Evolution, however, isn't *my* focus project. And
this would consume my entire own focus. I will help the person who makes
it his project. That's what these notes are about.



___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] The recent camel-lite improvements

2007-02-07 Thread Philip Van Hoof
I'm of course planning to get this stuff upstream. But I have to warn
the dreamers.

A lot of it is/was done as a hack. That's because I envision that
someday I actually will switch from using Camel to most likely fejj's
libspruce. 

With Camel this was for the last feature, IDLE support, almost undoable
too. That's because I needed non-blocking read support from the
CamelStream types. That change went all the way through a lot Camel
layers to the camel-file-utils and the three TCP implementations (raw,
ssl and openssl).

In other words: a total new design is yet again needed. Eventually I
rather see libspruce fill in that gap, than Camel.

I have no idea if the Evolution team is interested in the features and
changes. But I hereby offer my help to bring them upstream.

But please note that I will not do it 'automatically' myself anymore.

The very simple reason is lack of interest. Whenever I posted a patch in
the past, it took months before somebody even touched the status of the
bug, commented on it or even took a look at it. This has nothing to do
with time anymore, as far as I can see (weeks, okay. But months?)

Which is absolutely not my problem, nor should I condemn that or
whatever nor am I trying to make a statement other than: well then, I'm
also not going to loose *my* time with that.

I guess that sounds fair, or doesn't it?

(I have to note, though, that fejj very recently commented on some of
the patches, for which I'm grateful and I have been taking into account
his comments and will improve the stuff he mentioned soon).

Although I *AM* actively going to help the person who will port this
stuff to Evolution's Camel. By that I mean that I *will* make time for
this person. A lot if necessary. Entire evenings of free time if I have
to.

It's not so much about *time* itself, it's 100% about loosing it on
something that wont be acted upon anyway. Nobody enjoys that, I
certainly don't.


ps. I indeed also posted this comment to give the angry anti-fork people
of our great and knowledgeable community anti-flamewar food.


___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] The recent camel-lite improvements

2007-02-07 Thread Philip Van Hoof
On Wed, 2007-02-07 at 08:58 -0700, Veerapuram Varadhan wrote:
 On Wed, 2007-02-07 at 11:36 +0100, Philip Van Hoof wrote:
 
 Of all those features/implementations that you have mentioned so far, I
 could list 2 of them fit the interest of evolution - mmap (without
 summary format changes - adding len is okay) and memory-optimization in
 using tokens in camel-folder-summary.c. Rest of them are too focused
 and don't provide abstractions to extend for a desktop application.

I agree that most of the changes are indeed focused on mobile uses. I
mentioned the changes in the E-mail because there are some people who
privately and on IRC urge me to synchronize as much as possible of the
work with Evolution's Mailer code.

While I don't disagree with them, I too feel that it's up to the Mailer
maintainers -and people to decide about this. I share your opinion that
most of the changes are not focused on desktop applications.

 Also, I am still on the assumption that you are getting your mmap work
 committed to the mmap-branch, which would be validated and taken during
 the next development cycle and I am yet to see any updates on that.

I wrote this down, and will take a look at what I can extract from the
current camel-lite version into the current Camel version of Evolution.

If somebody wants to speed this up, assistance is welcomed :)

Note though that I'm planning to implement a very different idea on the
summary work. Something with mmap()ed segments (multiple summary files
per folder, on the disk) of 1,000 summary items per segment and putting
the flags in a separate file.

Might sound a little bit overkill too. So I might withdraw from the idea
too. It's just that I have certain performance issues when locally
searching, that could more easily be solved this way.

  Although I *AM* actively going to help the person who will port this
  stuff to Evolution's Camel. By that I mean that I *will* make time for
  this person. A lot if necessary. Entire evenings of free time if I have
  to.
  
  It's not so much about *time* itself, it's 100% about loosing it on
  something that wont be acted upon anyway. Nobody enjoys that, I
  certainly don't.
  
 We appreciate that and looking forward to get the good work merged soon.

Thanks.



___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] CamelImapFolder improvement that will make get_message stream immediately to the CamelImapMessageCache

2007-01-15 Thread Philip Van Hoof

http://bugzilla.gnome.org/show_bug.cgi?id=397130

Please review.

Thanks

-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] Camel's IMAP provider consumes more bandwidth than needed when retrieving new summary info

2007-01-15 Thread Philip Van Hoof
http://bugzilla.gnome.org/show_bug.cgi?id=397131

Please review.

Thanks
-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] Partial message retrieval for POP and IMAP copyright reassignment permission

2007-01-11 Thread Philip Van Hoof
I implemented this in camel-lite. Feel free to take a look at the code
in the tinymail's camel-lite repository.

Novell is hereby allowed to take these pieces out of camel-lite and give
themselves copyright ownership on their copy (maybe check with me how
that patch looks like, but so far nothing has gone in camel-lite that I
wouldn't allow copyright ownership reassignment for).

Most of these features are still in this is a hack status. And they
might not be really useful for a desktop E-mail client (they are crucial
for mobile devices).


-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Quick 'n dirty POP summary support

2007-01-10 Thread Philip Van Hoof

About this one and FYI:

On Wed, 2007-01-10 at 20:46 +0100, Philip Van Hoof wrote:
 @@ -386,6 +433,11 @@
 if (pop3_store-cache  fi-uid)
 camel_data_cache_remove(pop3_store-cache, cache, 
 fi-uid, NULL);
 }
 +
 +
 +   /* TNY TODO Shouldn't this remove message from memory? Oh 
 well, adding it */
 +   camel_object_unref (CAMEL_OBJECT (message)); 


I think this is a memory leak in the existing implementation of
camel_pop3_delete_old (message is never unreferenced, yet created).

If it is a memory leak, and it looks like it is, then this is actually a
huge one. No? (entire messages are leaked, or at least streams to files
being cached, keeping files open and stuff like that).




-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] New implementation of camel_imap_folder_fetch_data

2007-01-08 Thread Philip Van Hoof
);
isnextdone = FALSE;
}

camel_stream_write (stream, line, strlen (line));

linenum++;
}

g_free (tag);

camel_stream_reset (stream);

} 

CAMEL_SERVICE_REC_UNLOCK (store, connect_lock);
CAMEL_IMAP_FOLDER_REC_UNLOCK (imap_folder, cache_lock);

return stream;

errorhander:

CAMEL_SERVICE_REC_UNLOCK (store, connect_lock);
CAMEL_IMAP_FOLDER_REC_UNLOCK (imap_folder, cache_lock);

camel_exception_setv (ex, CAMEL_EXCEPTION_SERVICE_UNAVAILABLE,
_(Could not find message body in FETCH response.));

if (stream)
camel_object_unref (CAMEL_OBJECT (stream));

return NULL;
}



-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] New implementation of camel_imap_folder_fetch_data

2007-01-08 Thread Philip Van Hoof

Bweachrgg. It seems camel_imap_command_response performs an extra
CAMEL_SERVICE_REC_UNLOCK (store, connect_lock) (for some reason, I can't
figure out why that it is like that, but ... ).

And it looks that the code calling camel_imap_folder_fetch_data assumes
that this unlock happens (at least at some obscure places).

In other words: crazy code. Anyway, adding that extra unlock seems to
fix it (the current implementation below caused some races here).

Just add an extra CAMEL_SERVICE_REC_UNLOCK (store, connect_lock) before
the returning of the stream and in the errorhandler label.

b

ps. The locking code of the imap provider is really, really broken or
done in an absolutely insane way.


On Tue, 2007-01-09 at 02:18 +0100, Philip Van Hoof wrote:
 Hi there,
 
 I just finished this one, so there's probably a huge amount of bugs in
 it. I know I should first debug them away before posting on mailing
 lists ;). However, it would be nice if some (other) people would test
 this reimplementation (it's a method in camel-imap-folder.c, by the
 way).
 
 Just take a look at it, compare it with the memory consumption of
 copying everything into a GData after first copying everything into a
 GPtrArray of a CamelImapResponse structure (which is what the original
 does).
 
 ps. the if (TRUE) {} thing is because in the version for tinymail,
 there's support for partially retrieving messages. The else part
 implements that support (feel free to take a look at tinymail's
 camel-lite for the implementation, it's not a secret or something).
 
 
 CamelStream *
 camel_imap_folder_fetch_data (CamelImapFolder *imap_folder, const char *uid,
 const char *section_text, gboolean cache_only,
 CamelException *ex)
 {
   CamelFolder *folder = CAMEL_FOLDER (imap_folder);
   CamelImapStore *store = CAMEL_IMAP_STORE (folder-parent_store);
   CamelStream *stream;
   
   /* EXPUNGE responses have to modify the cache, which means
* they have to grab the cache_lock while holding the
* connect_lock.
 
* Because getting the service lock may cause MUCH unecessary
* delay when we already have the data locally, we do the
* locking separately.  This could cause a race
* getting the same data from the cache, but that is only
* an inefficiency, and bad luck.
*/
 
   CAMEL_IMAP_FOLDER_REC_LOCK (imap_folder, cache_lock);
   stream = camel_imap_message_cache_get (imap_folder-cache, uid, 
 section_text, ex);
 
   /* TNY TODO: if (full) Detect retrieval status (if partial refetch) */
 
   if (!stream  (!strcmp (section_text, HEADER) || !strcmp 
 (section_text, 0))) 
   {
   camel_exception_clear (ex);
   stream = camel_imap_message_cache_get (imap_folder-cache, uid, 
 , ex);
   }
   CAMEL_IMAP_FOLDER_REC_UNLOCK (imap_folder, cache_lock);
   
   if (stream || cache_only)
   return stream;
 
   camel_exception_clear(ex);
 
   CAMEL_SERVICE_REC_LOCK (store, connect_lock);
   CAMEL_IMAP_FOLDER_REC_LOCK (imap_folder, cache_lock);
 
   if (!camel_imap_store_connected(store, ex)) {
   camel_exception_set (ex, CAMEL_EXCEPTION_SERVICE_UNAVAILABLE,
_(This message is not currently 
 available));
   CAMEL_IMAP_FOLDER_REC_UNLOCK (imap_folder, cache_lock);
   CAMEL_SERVICE_REC_UNLOCK (store, connect_lock);
   return NULL;
   }
   
   camel_exception_clear (ex);
 
 stream = camel_imap_message_cache_insert (imap_folder-cache, 
   uid, full?section_text:, , 0, NULL);
   if (stream == NULL)
   stream = camel_stream_mem_new ();
 
   if (!stream)
   goto errorhander;
 
   if (TRUE)
   {
   gboolean first = TRUE;
   gchar line[512];
   guint linenum = 0;
   ssize_t nread; 
   CamelStreamBuffer *server_stream = CAMEL_STREAM_BUFFER 
 (store-istream);
   gchar *tag;
   guint taglen;
   gboolean isnextdone = FALSE;
 
   if (store-server_level  IMAP_LEVEL_IMAP4REV1  !*section_text)
   camel_imap_command_start (store, folder, ex,
   UID FETCH %s RFC822.PEEK,uid);
   else
   camel_imap_command_start (store, folder, ex,
   UID FETCH %s BODY.PEEK[%s],uid, section_text);
 
   tag = g_strdup_printf (%c%.5u, store-tag_prefix, 
 store-command-1);
   taglen = strlen (tag);
   store-command++;
 
   while (nread = camel_stream_buffer_gets (server_stream, line, 512) 
  0)
   {
 
   /* It might be the line before the last line */
   if (line[0] == ')'  (line[1] == '\n' || (line[1] == '\r' 
  line[2] == '\n')))
   {
   isnextdone = TRUE

Re: [Evolution-hackers] New implementation of camel_imap_folder_fetch_data

2007-01-08 Thread Philip Van Hoof

Ah, I figured it out. The camel_imap_command_start places the lock, and
the functions that are usually used with the results unlock it.

owk. 

* On success, the store's connect_lock will be locked. It will be freed
 * when you call camel_imap_response_free. (The lock is recursive, so
 * callers can grab and release it themselves if they need to run
 * multiple commands atomically.) 


Easy == different


On Tue, 2007-01-09 at 03:47 +0100, Philip Van Hoof wrote:
 Bweachrgg. It seems camel_imap_command_response performs an extra
 CAMEL_SERVICE_REC_UNLOCK (store, connect_lock) (for some reason, I can't
 figure out why that it is like that, but ... ).
 
 And it looks that the code calling camel_imap_folder_fetch_data assumes
 that this unlock happens (at least at some obscure places).
 
 In other words: crazy code. Anyway, adding that extra unlock seems to
 fix it (the current implementation below caused some races here).
 
 Just add an extra CAMEL_SERVICE_REC_UNLOCK (store, connect_lock) before
 the returning of the stream and in the errorhandler label.
 
 b
 
 ps. The locking code of the imap provider is really, really broken or
 done in an absolutely insane way.
 
 
 On Tue, 2007-01-09 at 02:18 +0100, Philip Van Hoof wrote:
  Hi there,
  
  I just finished this one, so there's probably a huge amount of bugs in
  it. I know I should first debug them away before posting on mailing
  lists ;). However, it would be nice if some (other) people would test
  this reimplementation (it's a method in camel-imap-folder.c, by the
  way).
  
  Just take a look at it, compare it with the memory consumption of
  copying everything into a GData after first copying everything into a
  GPtrArray of a CamelImapResponse structure (which is what the original
  does).
  
  ps. the if (TRUE) {} thing is because in the version for tinymail,
  there's support for partially retrieving messages. The else part
  implements that support (feel free to take a look at tinymail's
  camel-lite for the implementation, it's not a secret or something).
  
  
  CamelStream *
  camel_imap_folder_fetch_data (CamelImapFolder *imap_folder, const char *uid,
const char *section_text, gboolean cache_only,
CamelException *ex)
  {
  CamelFolder *folder = CAMEL_FOLDER (imap_folder);
  CamelImapStore *store = CAMEL_IMAP_STORE (folder-parent_store);
  CamelStream *stream;
  
  /* EXPUNGE responses have to modify the cache, which means
   * they have to grab the cache_lock while holding the
   * connect_lock.
  
   * Because getting the service lock may cause MUCH unecessary
   * delay when we already have the data locally, we do the
   * locking separately.  This could cause a race
   * getting the same data from the cache, but that is only
   * an inefficiency, and bad luck.
   */
  
  CAMEL_IMAP_FOLDER_REC_LOCK (imap_folder, cache_lock);
  stream = camel_imap_message_cache_get (imap_folder-cache, uid, 
  section_text, ex);
  
  /* TNY TODO: if (full) Detect retrieval status (if partial refetch) */
  
  if (!stream  (!strcmp (section_text, HEADER) || !strcmp 
  (section_text, 0))) 
  {
  camel_exception_clear (ex);
  stream = camel_imap_message_cache_get (imap_folder-cache, uid, 
  , ex);
  }
  CAMEL_IMAP_FOLDER_REC_UNLOCK (imap_folder, cache_lock);
  
  if (stream || cache_only)
  return stream;
  
  camel_exception_clear(ex);
  
  CAMEL_SERVICE_REC_LOCK (store, connect_lock);
  CAMEL_IMAP_FOLDER_REC_LOCK (imap_folder, cache_lock);
  
  if (!camel_imap_store_connected(store, ex)) {
  camel_exception_set (ex, CAMEL_EXCEPTION_SERVICE_UNAVAILABLE,
   _(This message is not currently 
  available));
  CAMEL_IMAP_FOLDER_REC_UNLOCK (imap_folder, cache_lock);
  CAMEL_SERVICE_REC_UNLOCK (store, connect_lock);
  return NULL;
  }
  
  camel_exception_clear (ex);
  
  stream = camel_imap_message_cache_insert (imap_folder-cache, 
  uid, full?section_text:, , 0, NULL);
  if (stream == NULL)
  stream = camel_stream_mem_new ();
  
  if (!stream)
  goto errorhander;
  
  if (TRUE)
  {
  gboolean first = TRUE;
  gchar line[512];
  guint linenum = 0;
  ssize_t nread; 
  CamelStreamBuffer *server_stream = CAMEL_STREAM_BUFFER 
  (store-istream);
  gchar *tag;
  guint taglen;
  gboolean isnextdone = FALSE;
  
  if (store-server_level  IMAP_LEVEL_IMAP4REV1  !*section_text)
  camel_imap_command_start (store, folder, ex,
  UID FETCH %s RFC822.PEEK,uid);
  else
  camel_imap_command_start (store, folder, ex,
  UID FETCH %s BODY.PEEK[%s],uid, section_text

Re: [Evolution-hackers] New implementation of camel_imap_folder_fetch_data

2007-01-08 Thread Philip Van Hoof
 (imap_folder, cache_lock);

return stream;

errorhander:

CAMEL_IMAP_FOLDER_REC_UNLOCK (imap_folder, cache_lock);

camel_exception_setv (ex, CAMEL_EXCEPTION_SERVICE_UNAVAILABLE,
_(Could not find message body in FETCH response.));

if (stream)
camel_object_unref (CAMEL_OBJECT (stream));

return NULL;
}


-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


[Evolution-hackers] Memory usage of CamelFolderSummary and CamelImapFolder

2006-11-16 Thread Philip Van Hoof
I did another round of checking where the memory of Camel is going to in
tinymail's Camel.

I have also tested its Camel with Evolution with success.

Tinymail's Camel has the following features:

  o. The CamelFolderSummary uses mmap. This significantly reduces memory
 usage because an mmap is on-demand paged.

  o. The CamelMessageInfoBase structure is significantly smaller (this
 has been put in #ifdef's so that it can easily be reversed for
 Evolution). This reduces memory usage of the memory that can't be
 mmap()ed.

  o. When new message-info arrives, each 1000th item the
 CamelFolderSummary is dumped to disk and effectively reloaded. The
 pstring_hashtable will also be updated (pstrings that no longer
 occur are unreferenced)

  o. The CamelImapFolder consumes much less bandwidth by asking for a
 lot less headers and by more efficiently forming IMAP commands,
 the procedure has been reimplemented into a cancellable one. By
 that I mean that if a cancellation happens, already received data
 can mostly be recovered (and will be recovered). This without the
 continuations which are only available in IMAP4rev1 (by simply
 storing data more quickly on disk, and starting from the previous
 store point)

  o. When the CamelFolderSummary instance is reloaded, it will reuse
 CamelMessageInfo instances. It will not destroy them unless they
 have been removed during an expunge request.

 If the message-info is available in the mmap, it will unreference
 the pstrings that might have been in use by the message-info and it
 will reassign the struct's char pointers to locations in the mmap.

 Because Camel has a property accessor, this even works on folders
 that are open (messages that are currently visible should not be a
 problem because the tree-view makes copies of the strings when they
 need to become visible -- the GtkTreeView does, I haven't checked
 Evolution's but Evolution didn't crash after I did a a lot basic
 removing, copying, scrolling and moving of messages).

  o. The this is a non-mmapped message-info instance flags have been
 put in the flags member rather than separate gbooleans which
 consumed another two ints in memory per instance

These four/five/six dots (together) solve all the remaining problems
that the original mmap patch had (the most important issue was that when
new messages arrived, those where not mmap()ed, they are now reloaded
periodically hence will be mmap()ed during such a reload quickly).

The speed of fetching new messages is actually much faster than the
original Evolution implementation. That's probably because a lot less
bandwidth is needed. If you would count without the bandwidth
optimisation (on a hypothetical extremely fast IMAP service) you would
see a performance hit compared to the original one. That's mostly
because each 1000th received message-info, a reload happens.


For a massif graph of downloading 3000, 800, 3000, 800, 3000, 800
headers you can check out this message on the tinymail mailing list:

http://mail.gnome.org/archives/tinymail-devel-list/2006-November/msg00111.html


The relevant code:
https://svn.tinymail.org/svn/tinymail/trunk/libtinymail-camel/camel-lite/camel/camel-folder-summary.c
https://svn.tinymail.org/svn/tinymail/trunk/libtinymail-camel/camel-lite/camel/providers/imap/camel-imap-folder.c

If I can assist people with bringing their Camel into this shape, then
please let me know.

-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
work: vanhoof at x-tend dot be
blog: http://pvanhoof.be/blog

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Memory usage of CamelFolderSummary and CamelImapFolder

2006-11-16 Thread Philip Van Hoof
On Thu, 2006-11-16 at 16:13 +0100, Philip Van Hoof wrote:
 I did another round of checking where the memory of Camel is going to in
 tinymail's Camel.
 
 I have also tested its Camel with Evolution with success.
 
 Tinymail's Camel has the following features:
 

Next to these features, tinymail's Camel also:

  o. Removes all compilation warnings
  o. Uses all of Matthew Barnes's very nice patches (except the
 GStringChunk one) (thanks a lot Matthew)
  o. Uses all my GSlice-and-other patches (check the patches mailing
 list of Evolution to get an overview)
  o. Renames all the library-names to avoid filename conflicts
  o. Removes the libedataserver dependency (a very small libedataserver
 is statically linked)
  o. Has correct linking (the provider's libraries have all incorrect
 LDFLAGS crap, and link with WAY to much libraries) -- I hate it
 when people have no clue about what they are doing in the
 Makefile.am or simply forget to remove old linking flags
  o. Has its own configure.ac and its own build environment, and is
 integrated with tinymail's build and repository
  o. Reimplements OpenSSL support (certificates are on todo)
  o. Still supports all typical features of the normal Camel
 (NSS/NSPR/SMIME/SSL/Kerberos/etc etc)
  o. Removes the folder meta-info patch (unused by Tinymail, and not
 really done very well -- consumes memory, and it looks like a lot,
 but in stead of testing it .. I simply removed it, so I don't know
 for sure)

  o. Eum ...
  o. Did I miss something?


And next to existing features will tinymail's Camel soon:

  o. Support partial message retrieval in IMAP (certain)
  o. Support partial message retrieval in POP (uncertain)
  o. Support summaries in the POP provider (most likely)
  o. Support merging and backup of local cache (certain)
  o. Have correct certificate management for the SSL support (low
 priority)
  o. . . . . (you already know I wont stop, no matter what)



   o. The CamelFolderSummary uses mmap. This significantly reduces memory
  usage because an mmap is on-demand paged.
 
   o. The CamelMessageInfoBase structure is significantly smaller (this
  has been put in #ifdef's so that it can easily be reversed for
  Evolution). This reduces memory usage of the memory that can't be
  mmap()ed.
 
   o. When new message-info arrives, each 1000th item the
  CamelFolderSummary is dumped to disk and effectively reloaded. The
  pstring_hashtable will also be updated (pstrings that no longer
  occur are unreferenced)
 
   o. The CamelImapFolder consumes much less bandwidth by asking for a
  lot less headers and by more efficiently forming IMAP commands,
  the procedure has been reimplemented into a cancellable one. By
  that I mean that if a cancellation happens, already received data
  can mostly be recovered (and will be recovered). This without the
  continuations which are only available in IMAP4rev1 (by simply
  storing data more quickly on disk, and starting from the previous
  store point)
 
   o. When the CamelFolderSummary instance is reloaded, it will reuse
  CamelMessageInfo instances. It will not destroy them unless they
  have been removed during an expunge request.
 
  If the message-info is available in the mmap, it will unreference
  the pstrings that might have been in use by the message-info and it
  will reassign the struct's char pointers to locations in the mmap.
 
  Because Camel has a property accessor, this even works on folders
  that are open (messages that are currently visible should not be a
  problem because the tree-view makes copies of the strings when they
  need to become visible -- the GtkTreeView does, I haven't checked
  Evolution's but Evolution didn't crash after I did a a lot basic
  removing, copying, scrolling and moving of messages).
 
   o. The this is a non-mmapped message-info instance flags have been
  put in the flags member rather than separate gbooleans which
  consumed another two ints in memory per instance
 
 These four/five/six dots (together) solve all the remaining problems
 that the original mmap patch had (the most important issue was that when
 new messages arrived, those where not mmap()ed, they are now reloaded
 periodically hence will be mmap()ed during such a reload quickly).
 
 The speed of fetching new messages is actually much faster than the
 original Evolution implementation. That's probably because a lot less
 bandwidth is needed. If you would count without the bandwidth
 optimisation (on a hypothetical extremely fast IMAP service) you would
 see a performance hit compared to the original one. That's mostly
 because each 1000th received message-info, a reload happens.
 
 
 For a massif graph of downloading 3000, 800, 3000, 800, 3000, 800
 headers you can check out this message on the tinymail mailing list:
 
 http://mail.gnome.org/archives/tinymail

Re: [Evolution-hackers] Memory usage of CamelFolderSummary and CamelImapFolder

2006-11-16 Thread Philip Van Hoof
On Thu, 2006-11-16 at 11:41 -0500, Joe Shaw wrote:

Hey Joe,

 On Thu, 2006-11-16 at 16:13 +0100, Philip Van Hoof wrote:
o. The CamelFolderSummary uses mmap. This significantly reduces memory
   usage because an mmap is on-demand paged.
 
 Does the on-disk format of the CamelFolderSummary change much or at all?
 In reading a summary from disk with Beagle, the main problem we've found
 is that it is entirely unsearchable, because records within the file are
 of variable length and there is no end-of-record marker, which means
 that you can't open the file, seek to some random location, and expect
 to find where the next (or previous) message begins.  This means that
 any time the summary changes, we have to walk the whole thing over again
 to see changes.

It does change. Now the files are mmap()able and all strings have end
markers (\0 characters). All strings are also data-padded to four bytes.

So you can mmap() the file and search for strings, once found you can
walk-back to the uid of the E-mail.

The variable-length is still the case. But I can, if necessary, adjust
the summary file format to have end-of-item markers (so that you can
walk the file back until you find that marker, and they you'll know the
exact location of for example the uid).

 There was some work a while back to do a metasummary, which was
 essentially a summary of the summary for easier searching, but I'm not
 sure what the end result of that was, or if it's in 2.8 or newer.

That is a patch that I have removed from tinymail's Camel (because I
dislike its implementation).


-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


  1   2   >