[Evolution-hackers] Avoiding a strdup in camel-folder-summar.c
This patch is unrelated to the mmap() patches that I've also been
sending.
It basically avoids an strdup() by implementing a smarter free_token().
ps. I'm keeping Varadhan and Jeffrey in CC because the mailinglist
aren't working (or very slow).
--
Philip Van Hoof, software developer at x-tend
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
work: vanhoof at x-tend dot be
http://www.pvanhoof.be - http://www.x-tend.be
? camel-mime-tables.c
Index: camel-folder-summary.c
===
RCS file: /cvs/gnome/evolution-data-server/camel/camel-folder-summary.c,v
retrieving revision 1.149
diff -u -r1.149 camel-folder-summary.c
--- camel-folder-summary.c 6 Jul 2006 19:43:46 - 1.149
+++ camel-folder-summary.c 9 Jul 2006 16:43:30 -
@@ -1339,7 +1339,7 @@
if (len = 0) {
ret = NULL;
} else if (len= tokens_len) {
- ret = g_strdup(tokens[len-1]);
+ ret = tokens[len-1];
} else {
io(printf (Invalid token encountered: %d, len));
*str = NULL;
@@ -1853,6 +1853,22 @@
return ci;
}
+static void
+free_token (gchar *token)
+{
+ gint i=0;
+ gboolean no=FALSE;
+
+ for (i=0; (i tokens_len); i++)
+ {
+ if (tokens[i] == token)
+ no = TRUE;
+ }
+
+ if (!no)
+ g_free (token);
+}
+
static CamelMessageContentInfo *
content_info_load(CamelFolderSummary *s, FILE *in)
{
@@ -1860,7 +1876,7 @@
char *type, *subtype;
guint32 count, i;
CamelContentType *ct;
-
+
io(printf(Loading content info\n));
ci = camel_folder_summary_content_info_new(s);
@@ -1868,8 +1884,11 @@
camel_folder_summary_decode_token(in, type);
camel_folder_summary_decode_token(in, subtype);
ct = camel_content_type_new(type, subtype);
- g_free(type); /* can this be removed? */
- g_free(subtype);
+
+
+ free_token (type); /* can this be removed? */
+ free_token (subtype);
+
if (camel_file_util_decode_uint32(in, count) == -1 || count 500)
goto error;
@@ -1882,8 +1901,8 @@
camel_content_type_set_param(ct, name, value);
/* TODO: do this so we dont have to double alloc/free */
- g_free(name);
- g_free(value);
+ free_token (name);
+ free_token (value);
}
ci-type = ct;
@@ -1937,9 +1956,9 @@
content_info_free(CamelFolderSummary *s, CamelMessageContentInfo *ci)
{
camel_content_type_unref(ci-type);
- g_free(ci-id);
- g_free(ci-description);
- g_free(ci-encoding);
+ free_token (ci-id);
+ free_token (ci-description);
+ free_token (ci-encoding);
e_memchunk_free(s-content_info_chunks, ci);
}
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers
[Evolution-hackers] Abnormal huge allocations happening when scanning new messages
The function imap_rescan (or a nearby function) allocates +60 MB of ram
to fetch 14,558 headers and then it suddenly clears everything.
I find it very hard to believe that you really need to allocate 60MB
of ram, just to write a summary file of ~1MB. Is there something not
being freed while processing happens?
Can't this be implemented with a simple .. ?
while (stuff = get_next_from_imap ())
{
add_to_summary (stuff)
free_that_next_stuff (stuff)
}
How can such an implementation ever need 60MB ram? I also don't think my
IMAP service is sending me more than a megabyte or two of information.
The only way how I *can* implement having to allocate 60MB just to store
a 1MB summary, is to do something like:
all_stuff = get_everything_from_imap()
copy (allstuff, extraallstuff)
copy (extraallstuff, extraextraallstuff)
copy (extraextraallstuff, extraextraextraallstuff)
foreach (stuff in extraextraextraallstuff)
{
add_to_summary (stuff)
}
free (allstuff)
free (extraallstuff)
free (extraextraallstuff)
free (extraextraextraallstuff)
I'm probably soon going to checkout what is really happening here.
Because allocating 60MB for fetching the headers of a small 14,558
messages folder is ... simply insane in my opinion. Note that the
allocation happens linear (in valgrind, you'll see a perfect triangle
with a 90 degrees angle when the deallocation happens).
I also noticed that by far not everything is getting freed after the
procedure. So it's also leaking very much (like, a 3 MB leak at least).
By quick-looking at the current code. I find it hard to believe that
somebody actually *really* used his common sense to implement this: the
code is really a mess. And while saying that, I don't care which famous
great hacker implemented it. It's not well-done imo.
The imap4 code at least *looks* nicer. But I haven't yet in depth
checked out that one (and it's often not really working).
ps. I have this grey feeling that I'm going to find a lot dead bodies in
camel, now that I started really looking into its code. Is libspruce
ready Fejj? ;-)
--
Philip Van Hoof, software developer at x-tend
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
work: vanhoof at x-tend dot be
http://www.pvanhoof.be - http://www.x-tend.be
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers
[Evolution-hackers] Message remove how-to?
Hello, I have managed to get a list of messages into a folder. Now when I delete a message I set a deleted flag. But the message still remains in place until I go to another folder and return, becouse the full list is then reloaded. Now the question is: How do I remove a message from the list and renew the list. Thank's in advance. If my question isn't clear please ask some more information. ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] camel-folder-summary.c with mmap
On Mon, 2006-07-10 at 09:57 +0200, Philip Van Hoof wrote: On Sun, 2006-07-09 at 18:08 +0200, Philip Van Hoof wrote: On Sun, 2006-07-09 at 14:59 +0200, Philip Van Hoof wrote: Note about this E-mail. I'm assuming people who will read this, have a technical in-depth knowledge of Camel and camel-folder-summary.c SomeExtraNotes There's still one or two small bugs that cause the file position to be *sometimes* two bytes incorrect. It's kinda hard to find why because of the many possibilities and rather difficult to interpret file format (with its token compression and added, imho mostly unneeded, information per specific provider implementation). So if you are going to test this, and you get funny hangs in decode_uint32: that's probably because the offset in the mmap() addr pointer got wrong, and it's now hanging in a 00 byte (decoding such a byte to a uint32 basically means that it'll loop forever over it). In gdb, use up and print *(char*)s-filepos to get an idea what it stored as the last interesting position. It should point at the byte right after the last thing that got read. The ideal situation would be developing a new summary format that is more suitable for mmap(), but not backward compatible. There was way to many things to fix (like strdup()ing things) for using fopen() in the new m mode. That mode also isn't platform independent, of course. I tried my new format on an old implementation, and it's not really working. So I'm soon going to check what's wrong here (can't be very hard to fix this, probably the pstring length of empty strings or something like that). The memory savings are, however, significant. The speed, ie. when sorting the summary view on tinymail, which causes a lot string-reads in the mmap, is very very good. Opening a lot file descriptors and seeing the amount of mmap()'s grow on a Evolution instance ... didn't slowdown my desktop (it did scale). Overall: I'm extremely satisfied with the results and I haven't even experimented with posix_madvise(). The load-speed is definitely faster than the current fread/malloc/strdup implementation. I'm soon going to provide cachegrind results on that. -- Philip Van Hoof, software developer at x-tend home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be http://www.pvanhoof.be - http://www.x-tend.be ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
