Re: [Evolution-hackers] Evolution Data Server - Java or .Net bindings
On 10/23/06, Ritesh Khadgaray <[EMAIL PROTECTED]> wrote: > On Mon, 2006-10-23 at 18:54 +0200, Maciej Piechotka wrote: > > On 10/23/06, Ritesh Khadgaray <[EMAIL PROTECTED]> wrote: > > > On Fri, 2006-10-20 at 22:34 +0200, Maciej Piechotka wrote: > > > > Is there are any Java or .Net bindings? > > > > I'd like to write backends using library which are only on this two > > > > languages. > > > > > > i have heard of evolution-sharp > > > http://ftp.gnome.org/pub/GNOME/sources/evolution-sharp/ > > > > > > > I know - but it's rather frontend then backend - or I'm blind. > ouch. time for me to visit a optician > > http://www.thecentric.com/?cat=2 > Sorry - I couldn't find this site (or is it just a joke, which I don't understand? English is not my native language). Regards ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Reviewing imap_update_summary
On Tue, 2006-10-24 at 01:56 +0200, Philip Van Hoof wrote:
> static guint32
> imap_get_uids (
> {
...
> while ((type = camel_imap_command_response (store, &resp, ex)) ==
> CAMEL_IMAP_RESPONSE_UNTAGGED)
> {
...
> if (size > 0)
> camel_operation_progress (NULL, got * 100 / size);
Which obviously has a memory leak here.
> }
...
> }
And other problems.
Feel free to follow up at "imap_update_summary" in
https://svn.tinymail.org/svn/tinymail/trunk/libtinymail-camel/camel-lite/camel/providers/imap/camel-imap-folder.c
:)
--
Philip Van Hoof, software developer at x-tend
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
work: vanhoof at x-tend dot be
http://www.pvanhoof.be - http://www.x-tend.be
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Reviewing imap_update_summary
With "UID FETCH %d:%d FLAGS" being something not all IMAP servers don't
correctly support, I was a little bit forced to rewrite the
imap_update_summary function into these two ones:
I also simplified it a little bit. And removed one of the two/three
GPtrArrays (that are being synchronized and other funny stuff).
ps. I still need to test this one, and recheck for problems, etc etc ...
static guint32
imap_get_uids (CamelFolder *folder, CamelImapStore *store, CamelException *ex,
GPtrArray *needheaders, int size, int got)
{
char *resp;
CamelImapResponseType type;
guint32 cnt = 0;
CamelImapFolder *imap_folder = CAMEL_IMAP_FOLDER (folder);
GData *data;
camel_operation_start (NULL, _("Fetching summary information for new
messages in %s"), folder->name);
while ((type = camel_imap_command_response (store, &resp, ex)) ==
CAMEL_IMAP_RESPONSE_UNTAGGED)
{
cnt++;
data = parse_fetch_response (imap_folder, resp);
g_free (resp);
if (!data)
continue;
g_ptr_array_add (needheaders, g_datalist_get_data (&data,
"UID"));
if (size > 0)
camel_operation_progress (NULL, got * 100 / size);
}
camel_operation_end (NULL);
g_free (resp);
return cnt;
}
static void
imap_update_summary (CamelFolder *folder, int exists,
CamelFolderChangeInfo *changes,
CamelException *ex)
{
CamelImapStore *store = CAMEL_IMAP_STORE (folder->parent_store);
CamelImapFolder *imap_folder = CAMEL_IMAP_FOLDER (folder);
GPtrArray *fetch_data = NULL, *messages = NULL, *needheaders;
guint32 flags, uidval;
int i, seq, first, size, got;
CamelImapResponseType type;
const char *header_spec;
CamelImapMessageInfo *mi, *info;
CamelStream *stream;
char *uid, *resp;
GData *data;
gboolean more = TRUE;
unsigned int nextn = 1, cnt=0, tcnt=0;
if (store->server_level >= IMAP_LEVEL_IMAP4REV1)
header_spec = "HEADER.FIELDS (" CAMEL_MESSAGE_INFO_HEADERS
MAILING_LIST_HEADERS ")";
else
header_spec = "0";
/* Used as a way to fetch all Headers instead of the selective headers.
Support for fetching custom headers could be done in a better way,
using CamelURL and EPlugins. */
if( g_getenv ("EVO_IMAP_FETCH_ALL_HEADERS") )
header_spec = "HEADER";
tcnt = 0;
while (more)
{
seq = camel_folder_summary_count (folder->summary);
first = seq + 1;
if (seq > 0) {
mi = (CamelImapMessageInfo *)camel_folder_summary_index
(folder->summary, seq - 1);
uidval = strtoul(camel_message_info_uid (mi), NULL, 10);
camel_message_info_free(&mi->info);
} else
uidval = 0;
size = (exists - seq) * (IMAP_PRETEND_SIZEOF_FLAGS +
IMAP_PRETEND_SIZEOF_SIZE + IMAP_PRETEND_SIZEOF_HEADERS);
got = 0;
if (!camel_imap_command_start (store, folder, ex,
"UID FETCH %d:%d FLAGS", uidval + 1, uidval + 1 + nextn))
return;
more = FALSE;
needheaders = g_ptr_array_new ();
cnt = imap_get_uids (folder, store, ex, needheaders, size, got);
tcnt += cnt;
if (tcnt >= (exists - seq))
more = FALSE;
else
more = TRUE;
if (more && (((exists - seq) > nextn) && (cnt < nextn)))
{
if (!camel_imap_command_start (store, folder, ex,
"UID FETCH %d:* FLAGS", uidval + 1))
return;
cnt = imap_get_uids (folder, store, ex, needheaders, size, got);
tcnt += cnt;
more = FALSE;
}
if (nextn < 1000)
nextn += (nextn+5);
else
nextn = 1000;
messages = g_ptr_array_new ();
if (needheaders->len)
{
char *uidset;
int uid = 0;
qsort (needheaders->pdata, needheaders->len,
sizeof (void *), uid_compar);
camel_operation_start (NULL, _("Fetching summary information
for new messages in %s"), folder->name);
while (uid < needheaders->len)
{
uidset = imap_uid_array_to_set (folder->summary,
needheaders, uid, UID_SET_LIMIT, &uid);
if (!camel_imap_command_start (store, folder, ex,
"UID FETCH %s (FLAGS
INTERNALDATE BODY.PEEK[%s])",
uidset, header_spec)) {
g_ptr_array_free (needheaders, TRUE);
camel_operation_end (NULL);
g_free (uidset);
Re: [Evolution-hackers] Evolution Data Server - Java or .Net bindings
On Mon, 2006-10-23 at 18:54 +0200, Maciej Piechotka wrote: > On 10/23/06, Ritesh Khadgaray <[EMAIL PROTECTED]> wrote: > > On Fri, 2006-10-20 at 22:34 +0200, Maciej Piechotka wrote: > > > Is there are any Java or .Net bindings? > > > I'd like to write backends using library which are only on this two > > > languages. > > > > i have heard of evolution-sharp > > http://ftp.gnome.org/pub/GNOME/sources/evolution-sharp/ > > > > I know - but it's rather frontend then backend - or I'm blind. additionally, time for me to file a bug report ? Name: evolution-sharp Relocations: (not relocatable) Version : 0.11.1Vendor: Red Hat, Inc. Release : 10.fc6Build Date: Tue 12 Sep 2006 11:06:33 AM IST ... URL : http://ftp.gnome.org/pub/GNOME/sources/evolution-sharp/0.10/ Summary : Evolution Data Server Mono Bindings Description : Mono/C# bindings for the Evolution addressbook. > > > > > > > Regards > > Regards -- Ritesh Khadgaray LinuX N Stuff Ph: +919822394463 Eat Right, Exercise, Die Anyway. ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Evolution Data Server - Java or .Net bindings
On Mon, 2006-10-23 at 18:54 +0200, Maciej Piechotka wrote: > On 10/23/06, Ritesh Khadgaray <[EMAIL PROTECTED]> wrote: > > On Fri, 2006-10-20 at 22:34 +0200, Maciej Piechotka wrote: > > > Is there are any Java or .Net bindings? > > > I'd like to write backends using library which are only on this two > > > languages. > > > > i have heard of evolution-sharp > > http://ftp.gnome.org/pub/GNOME/sources/evolution-sharp/ > > > > I know - but it's rather frontend then backend - or I'm blind. ouch. time for me to visit a optician http://www.thecentric.com/?cat=2 > > > > > > > Regards > > Regards -- Ritesh Khadgaray LinuX N Stuff Ph: +919822394463 Eat Right, Exercise, Die Anyway. ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Evolution Data Server - Java or .Net bindings
On 10/23/06, Ritesh Khadgaray <[EMAIL PROTECTED]> wrote: > On Fri, 2006-10-20 at 22:34 +0200, Maciej Piechotka wrote: > > Is there are any Java or .Net bindings? > > I'd like to write backends using library which are only on this two > > languages. > > i have heard of evolution-sharp > http://ftp.gnome.org/pub/GNOME/sources/evolution-sharp/ > I know - but it's rather frontend then backend - or I'm blind. > > > > Regards Regards ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Evolution Data Server - Java or .Net bindings
On Fri, 2006-10-20 at 22:34 +0200, Maciej Piechotka wrote: > Is there are any Java or .Net bindings? > I'd like to write backends using library which are only on this two languages. i have heard of evolution-sharp http://ftp.gnome.org/pub/GNOME/sources/evolution-sharp/ > > Regards > ___ > Evolution-hackers mailing list > Evolution-hackers@gnome.org > http://mail.gnome.org/mailman/listinfo/evolution-hackers -- Ritesh Khadgaray LinuX N Stuff Ph: +919822394463 Eat Right, Exercise, Die Anyway. ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Reviewing imap_update_summary
On Mon, 2006-10-23 at 04:02 -0600, Sankar P wrote: Hey Sankar, Thanks for the response. > I rewrote this behavior in camel-GroupWise to fetch data in iterations, > so that the memory requirement remains a constant k instead of O(n), n > being the number of messages. I expected it work better. (GW and IMAP > code are similar in this aspect) > > However, when I tested it, as expected, the memory requirement came down > but the number of disk-access increased and hence it became slow. So I > reverted it to the old behavior. I acknowledge that the time increases. For me not significant but it does indeed take longer. Doing nothing is of course faster than dumping something to disk. > You can try rewriting this in IMAP and you will find out that the time > taken to complete the sync will increase in case of large folders. This is correct and the case. On a mobile device, however, the time it takes is (imo) less important than the possibility to do it. Maybe I shouldn't use the "1, 2, 4, 8, etc" serie but do something like "1, 10, 100, 1000" to decrease the amount of to-disk dumps? What do you think? At this moment it's the provider-code itself that determines this nth. When talking about 'design', it might indeed be better if the camel- folder-summary.c implements that decision. For now, the experiment itself is implemented as a hack (I agree). I would be interested in trying to improve it in a design-sense if there's interest (and discussion) from the Evolution team in something like this. Being a hack, once supported by all providers, I am probably going to use it in tinymail. I would of course prefer that Camel itself would someday enjoy the same improvements too. But it's not something I require nor will push for. I "touch evolution", if you want it..tell me :) > I tested the changes that I made (in camel-GW) and found that in actual > deployment scenario, people prefer having an occasional memory-shootup > (which will come down as soon as mail-fetch is complete) rather than > having so many disk-access, that will eventually make operations longer > to complete. For a desktop user I can imagine that, yes. For a mobile device, the memory shootup means that it's impossible to support such large folders and that during the download, the device becomes unusable for almost-large folders (that I definitely do want to support) sized around 20,000 hdrs. This is perhaps why a different design would be a good idea?: delegate the decision to a more specific plugin or piece of code that is global for all providers (rather than a patch-per-provider for each target). > If you want the memory usage to be a constant value, the best solution > is to make the folder_sync function fetch summaries iteratively from a > database back-end (not from flat-files or mmap). Perhaps this should be > done at a higher (camel) level so that all the providers can make use of > it; Means rewriting parts of the camel folder, summary etc. I agree > Anyway, all this is already covered under "On-Disk-Summaries". If you > are so obsessed about memory usage, go ahead and give it a try. I should. But I need something that works today ;). I have already, however, repeatedly stated that I'm very interested in the on-disk-summary concepts and that if a team would start with this work, I'd definitely join that team. I'm not naive to think that I could do it on my own. I ACK that implementing the concepts takes a team of smart people. I think that libspruce and GMime would be a nice (fresh) starting point for such an implementation. If it would happen, tinymail would probably become one of the first E-mail framework(/client) to actively use the disk-summary concepts and ideas. Absolutely :) > Some times, the customer's needs are different from what the hacker may > perceive to be the most important thing. Evolution's periodic memory > shootup (and subsequent coming down) is considered to be normal by the > customers than things like Proxy-authentication-failure (libsoup), EDS > crashes etc, that we have been working on. > It is an interesting work but we (the Evolution team) have got other > priorities driven based on actual customer deployment needs. These are > the most important things that Evolution (and indirectly Camel also) > should address to become a stable enterprise-level GNOME app. No problem. Yet I hope my experiments might someday be useful for the Evolution project. Until then, I'll use tinymail as host for them. -- Philip Van Hoof, software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org work: vanhoof at x-tend dot be blog: http://pvanhoof.be/blog ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Reviewing imap_update_summary
On Sun, 2006-10-22 at 15:41 +0200, Philip Van Hoof wrote: > Greetings, > > imap_update_summary is implemented in three or four steps: > > o. Getting (all) the headers/uids > o. Finding the ones that we must still fetch > o. Fetching those (x) > o. Writing out the summary > > The steps each consume memory and reuse some of the memory of the > previous step. Pointers to that memory is stored in GPtrArray's like > fetch_data and messages. > > In the code I have found no real reason to why this was done in > separated loops (steps) rather than one step and at the end of the loop, > free the data already. Especially for the third step (x), which seem to > consume most memory while it's happening. I rewrote this behavior in camel-GroupWise to fetch data in iterations, so that the memory requirement remains a constant k instead of O(n), n being the number of messages. I expected it work better. (GW and IMAP code are similar in this aspect) However, when I tested it, as expected, the memory requirement came down but the number of disk-access increased and hence it became slow. So I reverted it to the old behavior. You can try rewriting this in IMAP and you will find out that the time taken to complete the sync will increase in case of large folders. > > The current implementation requires the data being received from the > IMAP service to be kept in memory until the entire folder has been > received and all steps done. This consumes more than one entire kilobyte > per message. Multiply that with for example 5,000 headers and you'll get > 5 MB memory consumption for fetching the new messages of a very small > IMAP folder (in case no other messages had been received before you > first started the procedure). > > Multiply that with 50,000 headers and you'll get 50 - 60 MB memory > consumption for a not extremely big but relatively big IMAP folders. > > Which will be freed, yes, but nevertheless it's a slowly growing peak > (the speed depends on the connection with your IMAP server) that only > gets deallocated or when pressing cancel or when all messages are > received (which can take a significant amount of time). I tested the changes that I made (in camel-GW) and found that in actual deployment scenario, people prefer having an occasional memory-shootup (which will come down as soon as mail-fetch is complete) rather than having so many disk-access, that will eventually make operations longer to complete. > > The strange part is that if I measure the amount of bytes that I receive > from the IMAP service; I measure far less bytes being transferred than > bytes being consumed in memory. It not only stores all the received > data, it also stores a lot more in memory (probably mostly 4 bytes > pointers and GData stuff). > > I wonder whether there was a reason why it was implemented this way? If > not, I'm planning to rewrite imap_update_summary in a different way. For > example by immediately creating a CamelMessageInfo struct or burning it > to the summary file instantly and freeing the GData created from the > stream in-loop. If you want the memory usage to be a constant value, the best solution is to make the folder_sync function fetch summaries iteratively from a database back-end (not from flat-files or mmap). Perhaps this should be done at a higher (camel) level so that all the providers can make use of it; Means rewriting parts of the camel folder, summary etc. Anyway, all this is already covered under "On-Disk-Summaries". If you are so obsessed about memory usage, go ahead and give it a try. Some times, the customer's needs are different from what the hacker may perceive to be the most important thing. Evolution's periodic memory shootup (and subsequent coming down) is considered to be normal by the customers than things like Proxy-authentication-failure (libsoup), EDS crashes etc, that we have been working on. It is an interesting work but we (the Evolution team) have got other priorities driven based on actual customer deployment needs. These are the most important things that Evolution (and indirectly Camel also) should address to become a stable enterprise-level GNOME app. Sankar ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
