Re: [Evolution-hackers] Camel in evolution-data-server, a different proposal
On Wed, 2006-07-12 at 19:13 +0200, Philip Van Hoof wrote: > On Wed, 2006-07-12 at 17:00 +0100, Ross Burton wrote: > > On Wed, 2006-07-12 at 17:39 +0200, Philip Van Hoof wrote: > > So you are proposing the following library packages: > > > > libedataserver > > libedataserverui > > libebook > > libedata-book > > libecal > > libedata-cal > > libgroupwise > > > And then binary packages for the Groupwise helpers, the Exchange > > helpers, and the server binaries themselves. > > Yeah. Sounds perfect. Looks very clean. ... and fragments the platform. > At this moment, all those fall under the name of "evolution comma data > comma server". Some of these libraries (like Camel) don't necessarily > have "anything" to do with the Evolution data that is being managed by > the data server of Evolution. They have a lot to do with it. iMip for calendaring for example (really you want the imip direct mail fall back to happen in e-d-s, not the client). There was also an idea to tie in a unified account settings dialog so that exchange/groupwise/jecs could be configured in a unified manner. > The E-mails and, folder-summaries, state data and summaries of Camel are > *not at all* being managed by Evolutions data server. It's simply > totally unrelated to the Evolutions data server. It looks like even some > Novell employees don't know that, probably cause it's being marketed as > "one" package. There were plans to look at this. In fact I discussed such a scenario with Jeff last week. Speed was always a major concern however, but something like the disk summary branch might alleviate this. > The ideal situation would be that most of these components would be > reusable by application developers that don't have to use the Evolution > data server at all. > > Why glue it? I think you're only real example is camel, which shares code with the other pieces anyhow. -JP -- JP Rosevear <[EMAIL PROTECTED]> Novell, Inc. ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] The mmap stuff definitely needs more eyes!
Hi there,
This one works on the Nokia 770 device!
http://pvanhoof.be/files/camel_folder_summary_with_mmap_fixes11_data_alignment04.diff
So my real target has finally been reached. Using this stuff, it should
be possible to display huge folders using tinymail on a Nokia 770.
With huge I mean: I'm expecting to support more than 20,000 headers per
folder. I don't know .. in theory it should even work with 100,000 or
more headers. But there's theory and reality. Lets test?
ps. It's of course going to take a while to receive the summary of such
a folder over wifi.
ps. I only tested this (on the device) with the imap implementation.
ps. Other people, please test this?
On Mon, 2006-07-17 at 20:47 +0200, Philip Van Hoof wrote:
> Eeeek. Look at what I just found in my own patches!!
>
> static CamelMessageInfo *
> message_info_load (CamelFolderSummary *s, FILE *in)
> {
> CamelMessageInfo *info;
> CamelImapMessageInfo *iinfo;
>
> info = camel_imap_summary_parent->message_info_load (s, in);
>
> if (info) {
> unsigned char* ptrchr = s->filepos;
> ptrchr = camel_file_util_mmap_decode_uint32 (ptrchr,
> &iinfo->server_flags, FALSE);
> s->filepos = ptrchr;
> label_to_flags(iinfo);
> }
>
> return info;
> }
>
>
> The "iinfo" is simply never assigned, yet being used! Eeek! *scared
> by my own mistakes now*. Very strange that didn't crash and burn on x86
> but does crash on ARM.
--
Philip Van Hoof, software developer at x-tend
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
work: vanhoof at x-tend dot be
http://www.pvanhoof.be - http://www.x-tend.be
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers
[Evolution-hackers] The mmap stuff definitely needs more eyes!
Eeeek. Look at what I just found in my own patches!!
static CamelMessageInfo *
message_info_load (CamelFolderSummary *s, FILE *in)
{
CamelMessageInfo *info;
CamelImapMessageInfo *iinfo;
info = camel_imap_summary_parent->message_info_load (s, in);
if (info) {
unsigned char* ptrchr = s->filepos;
ptrchr = camel_file_util_mmap_decode_uint32 (ptrchr,
&iinfo->server_flags, FALSE);
s->filepos = ptrchr;
label_to_flags(iinfo);
}
return info;
}
The "iinfo" is simply never assigned, yet being used! Eeek! *scared
by my own mistakes now*. Very strange that didn't crash and burn on x86
but does crash on ARM.
--
Philip Van Hoof, software developer at x-tend
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
work: vanhoof at x-tend dot be
http://www.pvanhoof.be - http://www.x-tend.be
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] The fixes11 mmap patch doesn't respect data alignment on some architectures
On Fri, 2006-07-14 at 19:23 +0200, Philip Van Hoof wrote:
> While the x86 handles unaligned access, ARM doesn't. The patch will not
> work on architectures that don't handle unaligned access.
>
> I will try to fix it, but I don't have a lot experience in this field.
Second attempt to introduce data alignment. It's still getting Killed on
the Nokia 770 ARM device. Regretfully I don't have a debugger for that
device.
In my hexeditor it looks like all of the summary file is now aligned to
4 bytes (or at least something is aligned to 4 bytes).
There's also no dmesg message on the device when the kill happens.
Could use some assistance from an expert in this field ;-) (I'm not)
--
Philip Van Hoof, software developer at x-tend
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
work: vanhoof at x-tend dot be
http://www.pvanhoof.be - http://www.x-tend.be
? camel-mime-tables.c
? providers/groupwise/.deps
? providers/groupwise/.libs
? providers/groupwise/camel-groupwise-folder.lo
? providers/groupwise/camel-groupwise-journal.lo
? providers/groupwise/camel-groupwise-provider.lo
? providers/groupwise/camel-groupwise-store-summary.lo
? providers/groupwise/camel-groupwise-store.lo
? providers/groupwise/camel-groupwise-summary.lo
? providers/groupwise/camel-groupwise-transport.lo
? providers/groupwise/camel-groupwise-utils.lo
? providers/groupwise/libcamelgroupwise.la
? providers/hula/.deps
? providers/hula/.libs
? providers/hula/camel-hula-provider.lo
? providers/hula/libcamelhula.la
Index: camel-file-utils.c
===
RCS file: /cvs/gnome/evolution-data-server/camel/camel-file-utils.c,v
retrieving revision 1.17
diff -u -p -r1.17 camel-file-utils.c
--- camel-file-utils.c 2 Jun 2006 00:52:29 - 1.17
+++ camel-file-utils.c 15 Jul 2006 10:18:26 -
@@ -49,135 +49,120 @@
#define IO_TIMEOUT (60*4)
+
/**
- * camel_file_util_encode_uint32:
+ * camel_file_util_encode_fixed_int32:
* @out: file to output to
* @value: value to output
*
- * Utility function to save an uint32 to a file.
+ * Encode a gint32, performing no compression, but converting
+ * to network order.
*
* Return value: 0 on success, -1 on error.
**/
int
-camel_file_util_encode_uint32 (FILE *out, guint32 value)
+camel_file_util_encode_fixed_int32 (FILE *out, gint32 value)
{
- int i;
+ guint32 save;
- for (i = 28; i > 0; i -= 7) {
- if (value >= (1 << i)) {
- unsigned int c = (value >> i) & 0x7f;
- if (fputc (c, out) == -1)
-return -1;
- }
- }
- return fputc (value | 0x80, out);
+ save = g_htonl (value);
+ if (fwrite (&save, sizeof (save), 1, out) != 1)
+ return -1;
+ return 0;
}
/**
- * camel_file_util_decode_uint32:
+ * camel_file_util_decode_fixed_int32:
* @in: file to read from
* @dest: pointer to a variable to store the value in
*
- * Retrieve an encoded uint32 from a file.
+ * Retrieve a gint32.
*
- * Return value: 0 on success, -1 on error. @*dest will contain the
- * decoded value.
+ * Return value: 0 on success, -1 on error.
**/
int
-camel_file_util_decode_uint32 (FILE *in, guint32 *dest)
+camel_file_util_decode_fixed_int32 (FILE *in, gint32 *dest)
{
-guint32 value = 0;
- int v;
+ guint32 save;
-/* until we get the last byte, keep decoding 7 bits at a time */
-while ( ((v = fgetc (in)) & 0x80) == 0 && v!=EOF) {
-value |= v;
-value <<= 7;
-}
- if (v == EOF) {
- *dest = value >> 7;
+ if (fread (&save, sizeof (save), 1, in) == 1) {
+ *dest = g_ntohl (save);
+ return 0;
+ } else {
return -1;
}
- *dest = value | (v & 0x7f);
-
-return 0;
}
/**
- * camel_file_util_encode_fixed_int32:
+ * camel_file_util_encode_uint32:
* @out: file to output to
* @value: value to output
*
- * Encode a gint32, performing no compression, but converting
- * to network order.
+ * Utility function to save an uint32 to a file.
*
* Return value: 0 on success, -1 on error.
**/
int
-camel_file_util_encode_fixed_int32 (FILE *out, gint32 value)
+camel_file_util_encode_uint32 (FILE *out, guint32 value)
{
- guint32 save;
-
- save = g_htonl (value);
- if (fwrite (&save, sizeof (save), 1, out) != 1)
- return -1;
- return 0;
+ return camel_file_util_encode_fixed_int32 (out, value);
}
/**
- * camel_file_util_decode_fixed_int32:
+ * camel_file_util_decode_uint32:
* @in: file to read from
* @dest: pointer to a variable to store the value in
*
- * Retrieve a gint32.
+ * Retrieve an encoded uint32 from a file.
*
- * Return value: 0 on success, -1 on error.
+ * Return value: 0 on success, -1 on error. @*dest will contain the
+ * decoded value.
**/
int
-camel_file_util_decode_fixed_int32 (FILE *in, gint32 *dest)
+camel_file_util_decode_uint32 (FILE *in, guint32 *dest)
{
- guint32 save;
+ return camel_file_util_decode_fixed_int32 (in, (gint32*)dest);
+}
- if (fread (&save, sizeof (save), 1, in) == 1) {
- *dest = g_ntohl (save);
- return 0;
Re: [Evolution-hackers] Folder summaries with mmap() (version nine -- fixes a leak)
On Wed, 2006-07-12 at 09:12 +0530, Harish Krishnaswamy wrote:
> On Tue, 2006-07-11 at 20:33 +0200, Philip Van Hoof wrote:
> > On Tue, 2006-07-11 at 23:14 +0530, Harish Krishnaswamy wrote:
> > ps. I think we should do a in-depth tech-meeting about this stuff. What
> > do you think, Harish? I have a few ideas on reducing memory usage when
> > Evolution is online (if Evolution is online, summary information of new
> > messages is still stored in memory, and not immediately reloaded using
> > mmap) --> this makes Evolution consume (a lot) more memory when its
> > online, when using the patch (as much as the current implementation).
> Sure, Philip.
> We could discuss it on the weekly IRC meeting (Wednesday 1000 UTC -
> #evolution-meet) - though I do not think it is convenient for fejj, if
> we wishes to join. We can work out a separate meeting and post it to e-h
> later too.
>
> I am more positive about the *real* effectiveness of this meeting unlike
> the "gtk-mozembed" episode as we are talking with CODE and there are
> concrete achievables at an arm's length. :-). Count me in.
During such a meeting I'm going to propose actual Camel architectural
changes. They will affect mostly internal changes (like how to deal with
the CamelFolderSummary and CamelMessageInfo(Base) type from the
providers).
The external API will probably change but not afaics in mail/ something
that would affect the current Evolution code a lot.
I would mainly do the adding of messages to the CamelFolderSummary very
different. I wouldn't store it using malloc (nor strdup nor pstring_add)
in CamelMessageInfo(Base) instances. I would simply append it to the
summary file and at the end of scanning .. launch my mmap code to parse
the offsets (of also the new items) out of the mmap()ed memory region.
This is very different from the current technique, where all is simply
added and the data itself is stored in strdup, malloc and pstring_add
memory. And sometimes (when?) dumped to the summary file. But where the
CamelFolderSummary instance isn't very often reloaded (which would give
it the chance to use my mmap code).
This is why when being online, Evolution still consumes +220M. Yet when
being offline, Evolution consumes ~30M (I have 112 folders, a lot of
them with more than 10,000 headers -- mailing lists --).
ps. My original Evolution consumes ~380M
--
Philip Van Hoof, software developer at x-tend
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
work: vanhoof at x-tend dot be
http://www.pvanhoof.be - http://www.x-tend.be
? camel-mime-tables.c
Index: camel-file-utils.c
===
RCS file: /cvs/gnome/evolution-data-server/camel/camel-file-utils.c,v
retrieving revision 1.17
diff -p -u -r1.17 camel-file-utils.c
--- camel-file-utils.c 2 Jun 2006 00:52:29 - 1.17
+++ camel-file-utils.c 12 Jul 2006 00:16:11 -
@@ -105,6 +105,54 @@ camel_file_util_decode_uint32 (FILE *in,
}
+unsigned char*
+camel_file_util_mmap_decode_uint32 (unsigned char *start, guint32 *dest, gboolean is_string)
+{
+ guint32 value = 0, rlen;
+ int v;
+#if DEBUG
+ int a=0;
+#endif
+ unsigned char *retval;
+
+ /* until we get the last byte, keep decoding 7 bits at a time */
+
+while ( ((v = *start++) & 0x80) == 0 ) {
+value |= v;
+value <<= 7;
+#if DEBUG
+ a++;
+ if (a > 5)
+ return NULL;
+#endif
+}
+
+ rlen = value | (v & 0x7f);
+ retval = start;
+
+ if (is_string) {
+#ifdef DEVEL
+ if (rlen == 1) {
+ rlen = 0;
+ } else {
+ unsigned int slen = strlen ((char*)retval)+1;
+ if (slen != rlen) {
+printf ("Problem in fileformat. String was %d, should be %d for %s %02x at %d\n", rlen, slen, retval, *retval, retval);
+rlen = slen;
+ }
+ }
+#else
+ if (rlen == 1)
+ rlen = 0;
+#endif
+
+ }
+
+
+ *dest = rlen;
+ return retval;
+}
+
/**
* camel_file_util_encode_fixed_int32:
* @out: file to output to
@@ -167,7 +215,7 @@ int \
camel_file_util_decode_##type(FILE *in, type *dest) \
{ \
type save = 0; \
- int i = sizeof(type) - 1; \
+ int i = sizeof(type) -1; \
int v = EOF; \
\
while (i >= 0 && (v = fgetc (in)) != EOF) { \
@@ -181,6 +229,24 @@ camel_file_util_decode_##type(FILE *in,
}
+#define MMAP_DECODE_T(type)\
+unsigned char* \
+camel_file_util_mmap_decode_##type(unsigned char *start, type *dest) \
+{ \
+ type save = 0; \
+ int i = sizeof(type) - 1; \
+ int v; \
+ \
+while (i >= 0) {\
+ v = *start++;\
+ save |= ((type)v) << (i * 8); \
+ i--; \
+ } \
+ *dest = save; \
+ return start; \
+}
+
+
/**
* camel_file_util_encode_time_t:
* @out: file to output to
@@ -202,6 +268,7 @@ CFU_ENCODE_T(time_t)
* Return value: 0 on success, -1 on error.
**/
CFU_DECODE_T(time_t)
+MMAP_DECODE_T(time_t)
/**
* camel_file_util_encode_off_t:
@@ -225,6 +292,7 @@ CFU_ENCODE_T(off_t)
* Return value: 0 on success
Re: [Evolution-hackers] Folder summaries with mmap() (version nine -- fixes a leak)
On Tue, 2006-07-11 at 16:47 +0200, Philip Van Hoof wrote:
> On Tue, 2006-07-11 at 13:17 +0200, Philip Van Hoof wrote:
> > This is a version (version eight) that actually works (afaics).
>
> This version (version nine) fixes a leak when new messages arrive.
This version (version ten) fixes a few strdup()'s, free()'s and other
stuff. Carefully look at the comments of the providers.
I had to disable some things that would otherwise corrupt the mmap()ed
memory block.
These things probably need some refactoring thoughts.
However. I'm sending this using an Evolution that is running from
LD_LIBRARY_PATH=/opt/camel/lib. So far Evolution hasn't crashed and it's
still writing summary new files for all my folders ;-).
http://pvanhoof.be/files/camel_folder_summary_with_mmap_fixes10.diff
--
Philip Van Hoof, software developer at x-tend
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
work: vanhoof at x-tend dot be
http://www.pvanhoof.be - http://www.x-tend.be
? camel-mime-tables.c
Index: camel-file-utils.c
===
RCS file: /cvs/gnome/evolution-data-server/camel/camel-file-utils.c,v
retrieving revision 1.17
diff -u -p -r1.17 camel-file-utils.c
--- camel-file-utils.c 2 Jun 2006 00:52:29 - 1.17
+++ camel-file-utils.c 11 Jul 2006 19:50:11 -
@@ -105,6 +105,46 @@ camel_file_util_decode_uint32 (FILE *in,
}
+unsigned char*
+camel_file_util_mmap_decode_uint32 (unsigned char *start, guint32 *dest, gboolean is_string)
+{
+ guint32 value = 0, rlen;
+ int v;
+ unsigned char *retval;
+
+ /* until we get the last byte, keep decoding 7 bits at a time */
+
+while ( ((v = *start++) & 0x80) == 0 ) {
+value |= v;
+value <<= 7;
+}
+
+ rlen = value | (v & 0x7f);
+ retval = start;
+
+ if (is_string) {
+#ifdef DEVEL
+ if (rlen == 1) {
+ rlen = 0;
+ } else {
+ unsigned int slen = strlen ((char*)retval)+1;
+ if (slen != rlen) {
+printf ("Problem in fileformat. String was %d, should be %d for %s %02x at %d\n", rlen, slen, retval, *retval, retval);
+rlen = slen;
+ }
+ }
+#else
+ if (rlen == 1)
+ rlen = 0;
+#endif
+
+ }
+
+
+ *dest = rlen;
+ return retval;
+}
+
/**
* camel_file_util_encode_fixed_int32:
* @out: file to output to
@@ -167,7 +207,7 @@ int \
camel_file_util_decode_##type(FILE *in, type *dest) \
{ \
type save = 0; \
- int i = sizeof(type) - 1; \
+ int i = sizeof(type) -1; \
int v = EOF; \
\
while (i >= 0 && (v = fgetc (in)) != EOF) { \
@@ -181,6 +221,24 @@ camel_file_util_decode_##type(FILE *in,
}
+#define MMAP_DECODE_T(type)\
+unsigned char* \
+camel_file_util_mmap_decode_##type(unsigned char *start, type *dest) \
+{ \
+ type save = 0; \
+ int i = sizeof(type) - 1; \
+ int v; \
+ \
+while (i >= 0) {\
+ v = *start++;\
+ save |= ((type)v) << (i * 8); \
+ i--; \
+ } \
+ *dest = save; \
+ return start; \
+}
+
+
/**
* camel_file_util_encode_time_t:
* @out: file to output to
@@ -202,6 +260,7 @@ CFU_ENCODE_T(time_t)
* Return value: 0 on success, -1 on error.
**/
CFU_DECODE_T(time_t)
+MMAP_DECODE_T(time_t)
/**
* camel_file_util_encode_off_t:
@@ -225,6 +284,7 @@ CFU_ENCODE_T(off_t)
* Return value: 0 on success, -1 on failure.
**/
CFU_DECODE_T(off_t)
+MMAP_DECODE_T(off_t)
/**
* camel_file_util_encode_size_t:
@@ -248,6 +308,7 @@ CFU_ENCODE_T(size_t)
* Return value: 0 on success, -1 on failure.
**/
CFU_DECODE_T(size_t)
+MMAP_DECODE_T(size_t)
/**
@@ -269,11 +330,22 @@ camel_file_util_encode_string (FILE *out
if ((len = strlen (str)) > 65536)
len = 65536;
+
+ if (len==0) len=-1;
if (camel_file_util_encode_uint32 (out, len+1) == -1)
return -1;
- if (len == 0 || fwrite (str, len, 1, out) == 1)
- return 0;
+
+ if (len != 0)
+ {
+ if (fwrite (str, len, 1, out) == 1)
+ {
+ if (fputc ('\0', out) == -1)
+return -1;
+ return 0;
+ }
+ }
+
return -1;
}
@@ -291,7 +363,8 @@ int
camel_file_util_decode_string (FILE *in, char **str)
{
guint32 len;
- register char *ret;
+ register char *ret, c;
+ long a;
if (camel_file_util_decode_uint32 (in, &len) == -1) {
*str = NULL;
@@ -311,7 +384,14 @@ camel_file_util_decode_string (FILE *in,
return -1;
}
+ a = ftell (in);
+
+ if ((c = fgetc (in)) != '\0')
+ /* If this is the old format, oeps .. we are so sorry */
+ fseek (in, a, SEEK_SET);
+
ret[len] = 0;
+
*str = ret;
return 0;
}
Index: camel-file-utils.h
===
RCS file: /cvs/gnome/evolution-data-server/camel/camel-file-utils.h,v
retrieving revision 1.11
diff -u -p -r1.11 camel-file-utils.h
--- camel-file-utils.h 10 Jan 2006 07:56:46 - 1.11
+++ camel-file-utils.h 11 Jul 2006 19:50:12 -
@@ -46,6 +46,7 @@ int camel_file_util_encode_fixed_int32 (
int camel_file_util_decode_fixed_int32 (FILE *in, gint32 *);
int camel_file_u
Re: [Evolution-hackers] Folder summaries with mmap() (version nine -- fixes a leak)
On Tue, 2006-07-11 at 13:17 +0200, Philip Van Hoof wrote:
> This is a version (version eight) that actually works (afaics).
This version (version nine) fixes a leak when new messages arrive.
However. The implementation/architecture of Camel requires that new
messages get transformed into CamelMessageInfo instances and added to
the CamelSummaryInfo structure. I think it would be better to simply
a.s.a.p. write it to the summary file and at the end of getting the new
messages, create a new CamelSummaryInfo structure with the new message
info instances ... using the mmap technique.
Currently it's obviously consuming more memory when new messages arrive
(compared to the mmap technique). In Evolution, such a folder never gets
closed. So it will for ever keep using this memory structure (in stead
of the information using the mmap).
Doing this would require appending to the summary file, closing the
summary object instance (which closes the mmap), reopening it when
receiving new messages completed and letting the mmap code parse that
into message-info instances.
Note .. starting from "add_message_from_data" (in camel-imap-folder.c),
there's a leak.
I think the leak is at camel_folder_summary_info_new_from_message and
more specific the message_info_new_from_header. The leak is ~200 kb for
a folder of ~10,000 headers
http://pvanhoof.be/files/camel_folder_summary_with_mmap_fixes09.diff
There's lots of decoding and copying happening there. So it's not going
to be easy to spot it.
> Therefore I'm re-adding the evolution-patches mailing list. Because the
> mailing lists are slow, I'm also adding some people that might have a
> direct interest in this.
>
> Please review this patch carefully and multiple times. Please do ask me
> questions about it (if needed). Please do test it extensively. Please do
> launch your evolution with valgrind massif and also under gdb.
>
> In case a crash happens in the ..decode_uint32 method , 'up' twice and
> 'print *s' so that you know the filename and other interesting infor-
> mation. Also check s->filepos to know the current file position and for
> example check variables like 'count' and 'len' after doing one 'up'.
>
> You can also find a copy here. I will upload new versions to the same
> location (and will increase the version number in the filename).
>
> http://pvanhoof.be/files/camel_folder_summary_with_mmap_fixes08.diff
>
--
Philip Van Hoof, software developer at x-tend
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
work: vanhoof at x-tend dot be
http://www.pvanhoof.be - http://www.x-tend.be
? .broken-date-parser.c.swp
? camel-mime-tables.c
Index: camel-file-utils.c
===
RCS file: /cvs/gnome/evolution-data-server/camel/camel-file-utils.c,v
retrieving revision 1.17
diff -u -p -r1.17 camel-file-utils.c
--- camel-file-utils.c 2 Jun 2006 00:52:29 - 1.17
+++ camel-file-utils.c 11 Jul 2006 14:34:18 -
@@ -105,6 +105,46 @@ camel_file_util_decode_uint32 (FILE *in,
}
+unsigned char*
+camel_file_util_mmap_decode_uint32 (unsigned char *start, guint32 *dest, gboolean is_string)
+{
+ guint32 value = 0, rlen;
+ int v;
+ unsigned char *retval;
+
+ /* until we get the last byte, keep decoding 7 bits at a time */
+
+while ( ((v = *start++) & 0x80) == 0 ) {
+value |= v;
+value <<= 7;
+}
+
+ rlen = value | (v & 0x7f);
+ retval = start;
+
+ if (is_string) {
+#ifdef DEVEL
+ if (rlen == 1) {
+ rlen = 0;
+ } else {
+ unsigned int slen = strlen ((char*)retval)+1;
+ if (slen != rlen) {
+printf ("Problem in fileformat. String was %d, should be %d for %s %02x at %d\n", rlen, slen, retval, *retval, retval);
+rlen = slen;
+ }
+ }
+#else
+ if (rlen == 1)
+ rlen = 0;
+#endif
+
+ }
+
+
+ *dest = rlen;
+ return retval;
+}
+
/**
* camel_file_util_encode_fixed_int32:
* @out: file to output to
@@ -167,7 +207,7 @@ int \
camel_file_util_decode_##type(FILE *in, type *dest) \
{ \
type save = 0; \
- int i = sizeof(type) - 1; \
+ int i = sizeof(type) -1; \
int v = EOF; \
\
while (i >= 0 && (v = fgetc (in)) != EOF) { \
@@ -181,6 +221,24 @@ camel_file_util_decode_##type(FILE *in,
}
+#define MMAP_DECODE_T(type)\
+unsigned char* \
+camel_file_util_mmap_decode_##type(unsigned char *start, type *dest) \
+{ \
+ type save = 0; \
+ int i = sizeof(type) - 1; \
+ int v; \
+ \
+while (i >= 0) {\
+ v = *start++;\
+ save |= ((type)v) << (i * 8); \
+ i--; \
+ } \
+ *dest = save; \
+ return start; \
+}
+
+
/**
* camel_file_util_encode_time_t:
* @out: file to output to
@@ -202,6 +260,7 @@ CFU_ENCODE_T(time_t)
* Return value: 0 on success, -1 on error.
**/
CFU_DECODE_T(time_t)
+MMAP_DECODE_T(time_t)
/**
* camel_file_util_encode_off_t:
@@ -225,6 +284,7 @@ CFU_ENCODE_T(off_t)
* Return value: 0 on success, -1 on failure.
Re: [Evolution-hackers] This one works, the mmap() patch
On Mon, 2006-07-10 at 22:44 +0200, Philip Van Hoof wrote:
> Please test.
>
> It should consume *A LOT* less memory. Especially in offline mode
And I'll just continue with fixes06
- Implements it for nntp, imapp, imap4, groupwise and fixes some of the
problems in the local implementation.
Remark: Because the method camel_file_util_encode_string was modified to
write a \0 at the end of each string and add 1 to the length in front of
it (pstring), the CamelStoreSummary types will have to read these new
strings from the files (using a non-mmap method, because I haven't yet
adjusted those to use mmap).
I could implement the CamelStoreSummary also using mmap(), or I could
try to make sure that reading these works perfectly. I'm first going to
try the second. I think if there's thing crashing, that is mostly the
reason.
I can also search and replace all camel_file_util_encode_string in the
mmap()ed files with a camel_file_util_mmap_encode_string or something
like that: there's a lot things that try to use the encode_string util.
--
Philip Van Hoof, software developer at x-tend
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
work: vanhoof at x-tend dot be
http://www.pvanhoof.be - http://www.x-tend.be
? camel-mime-tables.c
Index: camel-file-utils.c
===
RCS file: /cvs/gnome/evolution-data-server/camel/camel-file-utils.c,v
retrieving revision 1.17
diff -u -p -r1.17 camel-file-utils.c
--- camel-file-utils.c 2 Jun 2006 00:52:29 - 1.17
+++ camel-file-utils.c 10 Jul 2006 23:02:32 -
@@ -105,6 +105,47 @@ camel_file_util_decode_uint32 (FILE *in,
}
+unsigned char*
+camel_file_util_mmap_decode_uint32 (unsigned char *start, guint32 *dest, gboolean is_string)
+{
+ guint32 value = 0, rlen;
+ int v;
+ unsigned char *retval;
+
+ /* until we get the last byte, keep decoding 7 bits at a time */
+
+while ( ((v = *start++) & 0x80) == 0 ) {
+value |= v;
+value <<= 7;
+}
+
+ rlen = value | (v & 0x7f);
+ retval = start;
+
+ if (is_string) {
+
+#ifdef DEVEL
+ if (rlen == 1) {
+ rlen = 0;
+ } else {
+ unsigned int slen = strlen ((char*)retval)+1;
+ if (slen != rlen) {
+printf ("Problem in fileformat. String was %d, should be %d for %s %02x at %d\n", rlen, slen, retval, *retval, retval);
+rlen = slen;
+ }
+ }
+#else
+ if (rlen == 1)
+ rlen = 0;
+#endif
+
+ }
+
+
+ *dest = rlen;
+ return retval;
+}
+
/**
* camel_file_util_encode_fixed_int32:
* @out: file to output to
@@ -167,7 +208,7 @@ int \
camel_file_util_decode_##type(FILE *in, type *dest) \
{ \
type save = 0; \
- int i = sizeof(type) - 1; \
+ int i = sizeof(type) -1; \
int v = EOF; \
\
while (i >= 0 && (v = fgetc (in)) != EOF) { \
@@ -181,6 +222,24 @@ camel_file_util_decode_##type(FILE *in,
}
+#define MMAP_DECODE_T(type)\
+unsigned char* \
+camel_file_util_mmap_decode_##type(unsigned char *start, type *dest) \
+{ \
+ type save = 0; \
+ int i = sizeof(type) - 1; \
+ int v; \
+ \
+while (i >= 0) {\
+ v = *start++;\
+ save |= ((type)v) << (i * 8); \
+ i--; \
+ } \
+ *dest = save; \
+ return start; \
+}
+
+
/**
* camel_file_util_encode_time_t:
* @out: file to output to
@@ -202,6 +261,7 @@ CFU_ENCODE_T(time_t)
* Return value: 0 on success, -1 on error.
**/
CFU_DECODE_T(time_t)
+MMAP_DECODE_T(time_t)
/**
* camel_file_util_encode_off_t:
@@ -225,6 +285,7 @@ CFU_ENCODE_T(off_t)
* Return value: 0 on success, -1 on failure.
**/
CFU_DECODE_T(off_t)
+MMAP_DECODE_T(off_t)
/**
* camel_file_util_encode_size_t:
@@ -248,6 +309,7 @@ CFU_ENCODE_T(size_t)
* Return value: 0 on success, -1 on failure.
**/
CFU_DECODE_T(size_t)
+MMAP_DECODE_T(size_t)
/**
@@ -269,11 +331,22 @@ camel_file_util_encode_string (FILE *out
if ((len = strlen (str)) > 65536)
len = 65536;
+
+ if (len==0) len=-1;
if (camel_file_util_encode_uint32 (out, len+1) == -1)
return -1;
- if (len == 0 || fwrite (str, len, 1, out) == 1)
- return 0;
+
+ if (len != 0)
+ {
+ if (fwrite (str, len, 1, out) == 1)
+ {
+ if (fputc ('\0', out) != 1)
+return -1;
+ return 0;
+ }
+ }
+
return -1;
}
Index: camel-file-utils.h
===
RCS file: /cvs/gnome/evolution-data-server/camel/camel-file-utils.h,v
retrieving revision 1.11
diff -u -p -r1.11 camel-file-utils.h
--- camel-file-utils.h 10 Jan 2006 07:56:46 - 1.11
+++ camel-file-utils.h 10 Jul 2006 23:02:32 -
@@ -46,6 +46,7 @@ int camel_file_util_encode_fixed_int32 (
int camel_file_util_decode_fixed_int32 (FILE *in, gint32 *);
int camel_file_util_encode_uint32 (FILE *out, guint32);
int camel_file_util_decode_uint32 (FILE *in, guint32 *);
+
int camel_file_util_encode_time_t (FILE *out, time_t);
int camel_file_util_decode_time_t (FILE *in, time_t *);
int camel_file_util_encode_off
Re: [Evolution-hackers] This one works, the mmap() patch
On Tue, 2006-07-11 at 01:14 +0200, Philip Van Hoof wrote:
> On Mon, 2006-07-10 at 22:44 +0200, Philip Van Hoof wrote:
> > Please test.
o. This fixes07 fixes the camel_file_util_decode_string method in such a
way that it still supports the old pstrings. This also fixes all issues
with the CamelStoreSummary support. The CamelStoreSummary isn't using
mmap because it's such a small file that it's not really worth using
mmap on it I think.
o. I forgot to mention that I removed the saving in the load method by
simply letting it return -1 and set the errno variable which will make a
lower level system retry building the summary file.
I think this will be the one before I go to bed and get myself some
sleep ;-). I might do a fixes08 tomorrow. hehe.
> And I'll just continue with fixes06
>
> - Implements it for nntp, imapp, imap4, groupwise and fixes some of the
> problems in the local implementation.
>
> Remark: Because the method camel_file_util_encode_string was modified to
> write a \0 at the end of each string and add 1 to the length in front of
> it (pstring), the CamelStoreSummary types will have to read these new
> strings from the files (using a non-mmap method, because I haven't yet
> adjusted those to use mmap).
>
> I could implement the CamelStoreSummary also using mmap(), or I could
> try to make sure that reading these works perfectly. I'm first going to
> try the second. I think if there's thing crashing, that is mostly the
> reason.
>
> I can also search and replace all camel_file_util_encode_string in the
> mmap()ed files with a camel_file_util_mmap_encode_string or something
> like that: there's a lot things that try to use the encode_string util.
--
Philip Van Hoof, software developer at x-tend
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
work: vanhoof at x-tend dot be
http://www.pvanhoof.be - http://www.x-tend.be
? camel-mime-tables.c
Index: camel-file-utils.c
===
RCS file: /cvs/gnome/evolution-data-server/camel/camel-file-utils.c,v
retrieving revision 1.17
diff -u -p -r1.17 camel-file-utils.c
--- camel-file-utils.c 2 Jun 2006 00:52:29 - 1.17
+++ camel-file-utils.c 11 Jul 2006 00:54:17 -
@@ -105,6 +105,46 @@ camel_file_util_decode_uint32 (FILE *in,
}
+unsigned char*
+camel_file_util_mmap_decode_uint32 (unsigned char *start, guint32 *dest, gboolean is_string)
+{
+ guint32 value = 0, rlen;
+ int v;
+ unsigned char *retval;
+
+ /* until we get the last byte, keep decoding 7 bits at a time */
+
+while ( ((v = *start++) & 0x80) == 0 ) {
+value |= v;
+value <<= 7;
+}
+
+ rlen = value | (v & 0x7f);
+ retval = start;
+
+ if (is_string) {
+#ifdef DEVEL
+ if (rlen == 1) {
+ rlen = 0;
+ } else {
+ unsigned int slen = strlen ((char*)retval)+1;
+ if (slen != rlen) {
+printf ("Problem in fileformat. String was %d, should be %d for %s %02x at %d\n", rlen, slen, retval, *retval, retval);
+rlen = slen;
+ }
+ }
+#else
+ if (rlen == 1)
+ rlen = 0;
+#endif
+
+ }
+
+
+ *dest = rlen;
+ return retval;
+}
+
/**
* camel_file_util_encode_fixed_int32:
* @out: file to output to
@@ -167,7 +207,7 @@ int \
camel_file_util_decode_##type(FILE *in, type *dest) \
{ \
type save = 0; \
- int i = sizeof(type) - 1; \
+ int i = sizeof(type) -1; \
int v = EOF; \
\
while (i >= 0 && (v = fgetc (in)) != EOF) { \
@@ -181,6 +221,24 @@ camel_file_util_decode_##type(FILE *in,
}
+#define MMAP_DECODE_T(type)\
+unsigned char* \
+camel_file_util_mmap_decode_##type(unsigned char *start, type *dest) \
+{ \
+ type save = 0; \
+ int i = sizeof(type) - 1; \
+ int v; \
+ \
+while (i >= 0) {\
+ v = *start++;\
+ save |= ((type)v) << (i * 8); \
+ i--; \
+ } \
+ *dest = save; \
+ return start; \
+}
+
+
/**
* camel_file_util_encode_time_t:
* @out: file to output to
@@ -202,6 +260,7 @@ CFU_ENCODE_T(time_t)
* Return value: 0 on success, -1 on error.
**/
CFU_DECODE_T(time_t)
+MMAP_DECODE_T(time_t)
/**
* camel_file_util_encode_off_t:
@@ -225,6 +284,7 @@ CFU_ENCODE_T(off_t)
* Return value: 0 on success, -1 on failure.
**/
CFU_DECODE_T(off_t)
+MMAP_DECODE_T(off_t)
/**
* camel_file_util_encode_size_t:
@@ -248,6 +308,7 @@ CFU_ENCODE_T(size_t)
* Return value: 0 on success, -1 on failure.
**/
CFU_DECODE_T(size_t)
+MMAP_DECODE_T(size_t)
/**
@@ -269,11 +330,22 @@ camel_file_util_encode_string (FILE *out
if ((len = strlen (str)) > 65536)
len = 65536;
+
+ if (len==0) len=-1;
if (camel_file_util_encode_uint32 (out, len+1) == -1)
return -1;
- if (len == 0 || fwrite (str, len, 1, out) == 1)
- return 0;
+
+ if (len != 0)
+ {
+ if (fwrite (str, len, 1, out) == 1)
+ {
+ if (fputc ('\0', out) == -1)
+return -1;
+ return 0;
+ }
+ }
+
return -1;
}
@@ -291,7 +363,8 @@ int
camel_file_util_decode_string (FILE *in, c
