Re: [Evolution-hackers] Reviewing imap_update_summary

2006-10-23 Thread Sankar P
On Sun, 2006-10-22 at 15:41 +0200, Philip Van Hoof wrote:
 Greetings,
 
 imap_update_summary is implemented in three or four steps:
 
   o. Getting (all) the headers/uids
   o. Finding the ones that we must still fetch
   o. Fetching those (x)
   o. Writing out the summary
 
 The steps each consume memory and reuse some of the memory of the
 previous step. Pointers to that memory is stored in GPtrArray's like
 fetch_data and messages.
 
 In the code I have found no real reason to why this was done in
 separated loops (steps) rather than one step and at the end of the loop,
 free the data already. Especially for the third step (x), which seem to
 consume most memory while it's happening.

I rewrote this behavior in camel-GroupWise to fetch data in iterations,
so that the memory requirement remains a constant k instead of O(n), n
being the number of messages. I expected it work better. (GW and IMAP
code are similar in this aspect)

However, when I tested it, as expected, the memory requirement came down
but the number of disk-access increased and hence it became slow. So I
reverted it to the old behavior. 

You can try rewriting this in IMAP and you will find out that the time
taken to complete the sync will increase in case of large folders.


 
 The current implementation requires the data being received from the
 IMAP service to be kept in memory until the entire folder has been
 received and all steps done. This consumes more than one entire kilobyte
 per message. Multiply that with for example 5,000 headers and you'll get
 5 MB memory consumption for fetching the new messages of a very small
 IMAP folder (in case no other messages had been received before you
 first started the procedure).
 
 Multiply that with 50,000 headers and you'll get 50 - 60 MB memory
 consumption for a not extremely big but relatively big IMAP folders.
 
 Which will be freed, yes, but nevertheless it's a slowly growing peak
 (the speed depends on the connection with your IMAP server) that only
 gets deallocated or when pressing cancel or when all messages are
 received (which can take a significant amount of time).

I tested the changes that I made (in camel-GW) and found that in actual
deployment scenario, people prefer having an occasional memory-shootup
(which will come down as soon as mail-fetch is complete) rather than
having so many disk-access, that will eventually make operations longer
to complete.


 
 The strange part is that if I measure the amount of bytes that I receive
 from the IMAP service; I measure far less bytes being transferred than
 bytes being consumed in memory. It not only stores all the received
 data, it also stores a lot more in memory (probably mostly 4 bytes
 pointers and GData stuff).
 
 I wonder whether there was a reason why it was implemented this way? If
 not, I'm planning to rewrite imap_update_summary in a different way. For
 example by immediately creating a CamelMessageInfo struct or burning it
 to the summary file instantly and freeing the GData created from the
 stream in-loop.

If you want the memory usage to be a constant value, the best solution
is to make the folder_sync function fetch summaries iteratively from a
database back-end (not from flat-files or mmap). Perhaps this should be
done at a higher (camel) level so that all the providers can make use of
it; Means rewriting parts of the camel folder, summary etc.

Anyway, all this is already covered under On-Disk-Summaries. If you
are so obsessed about memory usage, go ahead and give it a try. 

Some times, the customer's needs are different from what the hacker may
perceive to be the most important thing. Evolution's periodic memory
shootup (and subsequent coming down) is considered to be normal by the
customers than things like Proxy-authentication-failure (libsoup), EDS
crashes etc, that we have been working on.

It is an interesting work but we (the Evolution team) have got other
priorities driven based on actual customer deployment needs. These are
the most important things that Evolution (and indirectly Camel also)
should address to become a stable enterprise-level GNOME app. 

Sankar


___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Reviewing imap_update_summary

2006-10-23 Thread Philip Van Hoof
On Mon, 2006-10-23 at 04:02 -0600, Sankar P wrote:

Hey Sankar,

Thanks for the response.

 I rewrote this behavior in camel-GroupWise to fetch data in iterations,
 so that the memory requirement remains a constant k instead of O(n), n
 being the number of messages. I expected it work better. (GW and IMAP
 code are similar in this aspect)
 
 However, when I tested it, as expected, the memory requirement came down
 but the number of disk-access increased and hence it became slow. So I
 reverted it to the old behavior. 

I acknowledge that the time increases. For me not significant but it
does indeed take longer. Doing nothing is of course faster than dumping
something to disk.

 You can try rewriting this in IMAP and you will find out that the time
 taken to complete the sync will increase in case of large folders.

This is correct and the case. On a mobile device, however, the time it
takes is (imo) less important than the possibility to do it. Maybe I
shouldn't use the 1, 2, 4, 8, etc serie but do something like 1, 10,
100, 1000 to decrease the amount of to-disk dumps? What do you think?

At this moment it's the provider-code itself that determines this nth.

When talking about 'design', it might indeed be better if the camel-
folder-summary.c implements that decision. For now, the experiment
itself is implemented as a hack (I agree).

I would be interested in trying to improve it in a design-sense if
there's interest (and discussion) from the Evolution team in something
like this.

Being a hack, once supported by all providers, I am probably going to
use it in tinymail. I would of course prefer that Camel itself would
someday enjoy the same improvements too.

But it's not something I require nor will push for. I touch evolution,
if you want it..tell me :)

 I tested the changes that I made (in camel-GW) and found that in actual
 deployment scenario, people prefer having an occasional memory-shootup
 (which will come down as soon as mail-fetch is complete) rather than
 having so many disk-access, that will eventually make operations longer
 to complete.

For a desktop user I can imagine that, yes.

For a mobile device, the memory shootup means that it's impossible to
support such large folders and that during the download, the device
becomes unusable for almost-large folders (that I definitely do want to
support) sized around 20,000 hdrs.

This is perhaps why a different design would be a good idea?: delegate
the decision to a more specific plugin or piece of code that is global
for all providers (rather than a patch-per-provider for each target).
 
 If you want the memory usage to be a constant value, the best solution
 is to make the folder_sync function fetch summaries iteratively from a
 database back-end (not from flat-files or mmap). Perhaps this should be
 done at a higher (camel) level so that all the providers can make use of
 it; Means rewriting parts of the camel folder, summary etc.

I agree

 Anyway, all this is already covered under On-Disk-Summaries. If you
 are so obsessed about memory usage, go ahead and give it a try. 

I should. But I need something that works today ;).

I have already, however, repeatedly stated that I'm very interested in
the on-disk-summary concepts and that if a team would start with this
work, I'd definitely join that team.

I'm not naive to think that I could do it on my own. I ACK that
implementing the concepts takes a team of smart people. I think that
libspruce and GMime would be a nice (fresh) starting point for such an
implementation.

If it would happen, tinymail would probably become one of the first
E-mail framework(/client) to actively use the disk-summary concepts and
ideas.

Absolutely :)

 Some times, the customer's needs are different from what the hacker may
 perceive to be the most important thing. Evolution's periodic memory
 shootup (and subsequent coming down) is considered to be normal by the
 customers than things like Proxy-authentication-failure (libsoup), EDS
 crashes etc, that we have been working on.

 It is an interesting work but we (the Evolution team) have got other
 priorities driven based on actual customer deployment needs. These are
 the most important things that Evolution (and indirectly Camel also)
 should address to become a stable enterprise-level GNOME app. 

No problem. Yet I hope my experiments might someday be useful for the
Evolution project. Until then, I'll use tinymail as host for them.


-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
work: vanhoof at x-tend dot be
blog: http://pvanhoof.be/blog

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Evolution Data Server - Java or .Net bindings

2006-10-23 Thread Ritesh Khadgaray
On Fri, 2006-10-20 at 22:34 +0200, Maciej Piechotka wrote:
 Is there are any Java or .Net bindings?
 I'd like to write backends using library which are only on this two languages.

i have heard of evolution-sharp
http://ftp.gnome.org/pub/GNOME/sources/evolution-sharp/

 
 Regards
 ___
 Evolution-hackers mailing list
 Evolution-hackers@gnome.org
 http://mail.gnome.org/mailman/listinfo/evolution-hackers
-- 
Ritesh Khadgaray
LinuX N Stuff
Ph: +919822394463
Eat Right, Exercise, Die Anyway.

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Evolution Data Server - Java or .Net bindings

2006-10-23 Thread Maciej Piechotka
On 10/23/06, Ritesh Khadgaray [EMAIL PROTECTED] wrote:
 On Fri, 2006-10-20 at 22:34 +0200, Maciej Piechotka wrote:
  Is there are any Java or .Net bindings?
  I'd like to write backends using library which are only on this two 
  languages.

 i have heard of evolution-sharp
 http://ftp.gnome.org/pub/GNOME/sources/evolution-sharp/


I know - but it's rather frontend then backend - or I'm blind.

 
  Regards

Regards
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Evolution Data Server - Java or .Net bindings

2006-10-23 Thread Ritesh Khadgaray
On Mon, 2006-10-23 at 18:54 +0200, Maciej Piechotka wrote:
 On 10/23/06, Ritesh Khadgaray [EMAIL PROTECTED] wrote:
  On Fri, 2006-10-20 at 22:34 +0200, Maciej Piechotka wrote:
   Is there are any Java or .Net bindings?
   I'd like to write backends using library which are only on this two 
   languages.
 
  i have heard of evolution-sharp
  http://ftp.gnome.org/pub/GNOME/sources/evolution-sharp/
 
 
 I know - but it's rather frontend then backend - or I'm blind.
ouch. time for me to visit a optician 

http://www.thecentric.com/?cat=2


 
  
   Regards
 
 Regards
-- 
Ritesh Khadgaray
LinuX N Stuff
Ph: +919822394463
Eat Right, Exercise, Die Anyway.

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Evolution Data Server - Java or .Net bindings

2006-10-23 Thread Ritesh Khadgaray
On Mon, 2006-10-23 at 18:54 +0200, Maciej Piechotka wrote:
 On 10/23/06, Ritesh Khadgaray [EMAIL PROTECTED] wrote:
  On Fri, 2006-10-20 at 22:34 +0200, Maciej Piechotka wrote:
   Is there are any Java or .Net bindings?
   I'd like to write backends using library which are only on this two 
   languages.
 
  i have heard of evolution-sharp
  http://ftp.gnome.org/pub/GNOME/sources/evolution-sharp/
 
 
 I know - but it's rather frontend then backend - or I'm blind.

additionally, time for me to file a bug report ?

Name: evolution-sharp  Relocations: (not
relocatable)
Version : 0.11.1Vendor: Red Hat, Inc.
Release : 10.fc6Build Date: Tue 12 Sep 2006
11:06:33 AM IST
...
URL :
http://ftp.gnome.org/pub/GNOME/sources/evolution-sharp/0.10/
Summary : Evolution Data Server Mono Bindings
Description :
Mono/C# bindings for the Evolution addressbook.


 
  
   Regards
 
 Regards
-- 
Ritesh Khadgaray
LinuX N Stuff
Ph: +919822394463
Eat Right, Exercise, Die Anyway.

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Reviewing imap_update_summary

2006-10-23 Thread Philip Van Hoof

With UID FETCH %d:%d FLAGS being something not all IMAP servers don't
correctly support, I was a little bit forced to rewrite the
imap_update_summary function into these two ones:

I also simplified it a little bit. And removed one of the two/three
GPtrArrays (that are being synchronized and other funny stuff).

ps. I still need to test this one, and recheck for problems, etc etc ...


static guint32 
imap_get_uids (CamelFolder *folder, CamelImapStore *store, CamelException *ex, 
GPtrArray *needheaders, int size, int got)
{
char *resp;
CamelImapResponseType type;
guint32 cnt = 0;
CamelImapFolder *imap_folder = CAMEL_IMAP_FOLDER (folder);
GData *data;
 
camel_operation_start (NULL, _(Fetching summary information for new 
messages in %s), folder-name);
while ((type = camel_imap_command_response (store, resp, ex)) ==
CAMEL_IMAP_RESPONSE_UNTAGGED) 
{
cnt++;
data = parse_fetch_response (imap_folder, resp);
g_free (resp);
if (!data)
continue;
g_ptr_array_add (needheaders, g_datalist_get_data (data, 
UID));
if (size  0)
camel_operation_progress (NULL, got * 100 / size);
}
camel_operation_end (NULL);
g_free (resp);
return cnt;

}

static void
imap_update_summary (CamelFolder *folder, int exists,
 CamelFolderChangeInfo *changes,
 CamelException *ex)
{
   CamelImapStore *store = CAMEL_IMAP_STORE (folder-parent_store);
   CamelImapFolder *imap_folder = CAMEL_IMAP_FOLDER (folder);
   GPtrArray *fetch_data = NULL, *messages = NULL, *needheaders;
   guint32 flags, uidval;
   int i, seq, first, size, got;
   CamelImapResponseType type;
   const char *header_spec;
   CamelImapMessageInfo *mi, *info;
   CamelStream *stream;
   char *uid, *resp;
   GData *data;
   gboolean more = TRUE;
   unsigned int nextn = 1, cnt=0, tcnt=0;

   if (store-server_level = IMAP_LEVEL_IMAP4REV1)
header_spec = HEADER.FIELDS ( CAMEL_MESSAGE_INFO_HEADERS 
MAILING_LIST_HEADERS );
   else
header_spec = 0;

   /* Used as a way to fetch all Headers instead of the selective headers.
  Support for fetching custom headers could be done in a better way,
  using CamelURL and EPlugins. */

   if( g_getenv (EVO_IMAP_FETCH_ALL_HEADERS) )
header_spec = HEADER;

   tcnt = 0;
   while (more)
   {
seq = camel_folder_summary_count (folder-summary);
first = seq + 1;
if (seq  0) {
mi = (CamelImapMessageInfo *)camel_folder_summary_index 
(folder-summary, seq - 1);
uidval = strtoul(camel_message_info_uid (mi), NULL, 10);
camel_message_info_free(mi-info);
} else
uidval = 0;

size = (exists - seq) * (IMAP_PRETEND_SIZEOF_FLAGS + 
IMAP_PRETEND_SIZEOF_SIZE + IMAP_PRETEND_SIZEOF_HEADERS);
got = 0;

if (!camel_imap_command_start (store, folder, ex,
UID FETCH %d:%d FLAGS, uidval + 1, uidval + 1 + nextn))
return;

more = FALSE; 
needheaders = g_ptr_array_new ();
cnt = imap_get_uids (folder, store, ex, needheaders, size, got);
tcnt += cnt;

if (tcnt = (exists - seq))
more = FALSE;
else
more = TRUE;

if (more  (((exists - seq)  nextn)  (cnt  nextn)))
{
if (!camel_imap_command_start (store, folder, ex,
UID FETCH %d:* FLAGS, uidval + 1))
return;
cnt = imap_get_uids (folder, store, ex, needheaders, size, got);
tcnt += cnt;
more = FALSE;
}

if (nextn  1000)
nextn += (nextn+5);
else
nextn = 1000;

messages = g_ptr_array_new ();
if (needheaders-len) 
{
char *uidset;
int uid = 0;

qsort (needheaders-pdata, needheaders-len,
sizeof (void *), uid_compar);

camel_operation_start (NULL, _(Fetching summary information 
for new messages in %s), folder-name);
while (uid  needheaders-len) 
{
uidset = imap_uid_array_to_set (folder-summary, 
needheaders, uid, UID_SET_LIMIT, uid);
if (!camel_imap_command_start (store, folder, ex,
   UID FETCH %s (FLAGS 
INTERNALDATE BODY.PEEK[%s]),
   uidset, header_spec)) {
g_ptr_array_free (needheaders, TRUE);
camel_operation_end (NULL);
g_free (uidset);
goto lose;
}
   

Re: [Evolution-hackers] Reviewing imap_update_summary

2006-10-23 Thread Philip Van Hoof
On Tue, 2006-10-24 at 01:56 +0200, Philip Van Hoof wrote:

 static guint32 
 imap_get_uids (
 {
...
   while ((type = camel_imap_command_response (store, resp, ex)) ==
   CAMEL_IMAP_RESPONSE_UNTAGGED) 
   {
...
   if (size  0)
   camel_operation_progress (NULL, got * 100 / size);

Which obviously has a memory leak here.

   }
...
 }

And other problems.

Feel free to follow up at imap_update_summary in 
https://svn.tinymail.org/svn/tinymail/trunk/libtinymail-camel/camel-lite/camel/providers/imap/camel-imap-folder.c

:)



-- 
Philip Van Hoof, software developer at x-tend 
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
work: vanhoof at x-tend dot be 
http://www.pvanhoof.be - http://www.x-tend.be

___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] Evolution Data Server - Java or .Net bindings

2006-10-23 Thread Maciej Piechotka
On 10/23/06, Ritesh Khadgaray [EMAIL PROTECTED] wrote:
 On Mon, 2006-10-23 at 18:54 +0200, Maciej Piechotka wrote:
  On 10/23/06, Ritesh Khadgaray [EMAIL PROTECTED] wrote:
   On Fri, 2006-10-20 at 22:34 +0200, Maciej Piechotka wrote:
Is there are any Java or .Net bindings?
I'd like to write backends using library which are only on this two 
languages.
  
   i have heard of evolution-sharp
   http://ftp.gnome.org/pub/GNOME/sources/evolution-sharp/
  
 
  I know - but it's rather frontend then backend - or I'm blind.
 ouch. time for me to visit a optician

 http://www.thecentric.com/?cat=2


Sorry -  I couldn't find this site (or is it just a joke, which I
don't understand? English is not my native language).

Regards
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers