Re: [Evolution-hackers] PATCH: Storing and fetching bodystructure, the patch
Please take a look at the new and reworked patch for adding
bodystructure retrieval and storage support.
http://bugzilla.gnome.org/show_bug.cgi?id=588557#c7
I reworked the patch to make it store using a separate table, as
requested in the comments by Srinivasa.
On Tue, 2009-07-14 at 12:57 +0200, Philip Van Hoof wrote:
> Hi there,
>
> I would like Evolution to store IMAP's bodystructure as often as
> possible. The reason for this is storing RDF graphs. Let me explain.
>
> I wrote a EPlugin for Evolution that is compiled and distributed by the
> Tracker project. This plugin fetches as much metadata about E-mails as
> possible and pushes it into Tracker's RDF store.
>
> Tracker's RDF store is a Nepomuk-ontology based RDF store that offers
> SPARQL as query language, and SPARQL Update as storage language.
>
> We have enhanced the Nepomuk message ontology so that it can store the
> entire structure of an E-mail. This means the structure that you can
> request using the BODY or BODYSTRUCTURE FETCH requests. These return you
> the MIME structure of the message in a pre-parsed skiplist format.
>
> Using this info our plugin can reconstruct the message's skeleton, but
> then in RDF as a graph.
>
> I don't just talk. I wrote the patch to do this too. It's attached.
>
> I tried to keep the changes small, and I implemented the migration code
> so that the tables' schemas will automatically be converted. The patch
> needs a good review, though. And testing.
>
> This is the point of the patch (a stored bodystructure string):
>
> sqlite> select bodystructure from 'INBOX/100' LIMIT 2;
> ("TEXT" "PLAIN" ("CHARSET" "ISO-8859-1") NIL NIL "8BIT" 2304 53 NIL NIL NIL
> NIL)
> ("TEXT" "PLAIN" NIL NIL NIL "7BIT" 3829 80 NIL NIL NIL NIL)
> sqlite>
>
> Note about adding BODYSTRUCTURE to the IMAP FETCH query that IMAP
> servers usually cache the body-structures. It wont slowdown an IMAP
> server, one that is worth being called "an IMAP server", much.
>
> It will add some bandwidth. But apparently isn't Evolution trying to
> save bandwidth, otherwise camel-imap-folder.c would use ENVELOPE instead
> of BODY.PEEK[HEADER.FIELDS(-a long list-)] (right?).
>
>
> Here are some pointers about things I just wrote:
>
> The Nemomuk message ontology
> http://git.gnome.org/cgit/tracker/tree/data/ontologies/34-nmo.ontology
>
> How we want this stuff in Tracker, how it could be used
> http://live.gnome.org/Tracker/Documentation/EmailSparql
>
> The Evolution plugin:
> http://git.gnome.org/cgit/tracker/tree/src/plugins/evolution/tracker-evolution-plugin.c
>
>
> Let me know!
>
>
> ___
> Evolution-hackers mailing list
> Evolution-hackers@gnome.org
> http://mail.gnome.org/mailman/listinfo/evolution-hackers
--
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
http://pvanhoof.be/blog
http://codeminded.be
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] PATCH: Storing and fetching bodystructure, the patch
Ping, do we have a decision on this?
On Tue, 2009-07-14 at 12:57 +0200, Philip Van Hoof wrote:
> Hi there,
>
> I would like Evolution to store IMAP's bodystructure as often as
> possible. The reason for this is storing RDF graphs. Let me explain.
>
> I wrote a EPlugin for Evolution that is compiled and distributed by the
> Tracker project. This plugin fetches as much metadata about E-mails as
> possible and pushes it into Tracker's RDF store.
>
> Tracker's RDF store is a Nepomuk-ontology based RDF store that offers
> SPARQL as query language, and SPARQL Update as storage language.
>
> We have enhanced the Nepomuk message ontology so that it can store the
> entire structure of an E-mail. This means the structure that you can
> request using the BODY or BODYSTRUCTURE FETCH requests. These return you
> the MIME structure of the message in a pre-parsed skiplist format.
>
> Using this info our plugin can reconstruct the message's skeleton, but
> then in RDF as a graph.
>
> I don't just talk. I wrote the patch to do this too. It's attached.
>
> I tried to keep the changes small, and I implemented the migration code
> so that the tables' schemas will automatically be converted. The patch
> needs a good review, though. And testing.
>
> This is the point of the patch (a stored bodystructure string):
>
> sqlite> select bodystructure from 'INBOX/100' LIMIT 2;
> ("TEXT" "PLAIN" ("CHARSET" "ISO-8859-1") NIL NIL "8BIT" 2304 53 NIL NIL NIL
> NIL)
> ("TEXT" "PLAIN" NIL NIL NIL "7BIT" 3829 80 NIL NIL NIL NIL)
> sqlite>
>
> Note about adding BODYSTRUCTURE to the IMAP FETCH query that IMAP
> servers usually cache the body-structures. It wont slowdown an IMAP
> server, one that is worth being called "an IMAP server", much.
>
> It will add some bandwidth. But apparently isn't Evolution trying to
> save bandwidth, otherwise camel-imap-folder.c would use ENVELOPE instead
> of BODY.PEEK[HEADER.FIELDS(-a long list-)] (right?).
>
>
> Here are some pointers about things I just wrote:
>
> The Nemomuk message ontology
> http://git.gnome.org/cgit/tracker/tree/data/ontologies/34-nmo.ontology
>
> How we want this stuff in Tracker, how it could be used
> http://live.gnome.org/Tracker/Documentation/EmailSparql
>
> The Evolution plugin:
> http://git.gnome.org/cgit/tracker/tree/src/plugins/evolution/tracker-evolution-plugin.c
>
>
> Let me know!
>
>
> ___
> Evolution-hackers mailing list
> Evolution-hackers@gnome.org
> http://mail.gnome.org/mailman/listinfo/evolution-hackers
--
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
http://pvanhoof.be/blog
http://codeminded.be
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] PATCH: Storing and fetching bodystructure, the patch
I created a tracking bug for this:
http://bugzilla.gnome.org/show_bug.cgi?id=588557
On Tue, 2009-07-14 at 12:57 +0200, Philip Van Hoof wrote:
> I would like Evolution to store IMAP's bodystructure as often as
> possible. The reason for this is storing RDF graphs. Let me explain.
>
> I wrote a EPlugin for Evolution that is compiled and distributed by
> the
> Tracker project. This plugin fetches as much metadata about E-mails as
> possible and pushes it into Tracker's RDF store.
>
> Tracker's RDF store is a Nepomuk-ontology based RDF store that offers
> SPARQL as query language, and SPARQL Update as storage language.
>
> We have enhanced the Nepomuk message ontology so that it can store the
> entire structure of an E-mail. This means the structure that you can
> request using the BODY or BODYSTRUCTURE FETCH requests. These return
> you
> the MIME structure of the message in a pre-parsed skiplist format.
>
> Using this info our plugin can reconstruct the message's skeleton, but
> then in RDF as a graph.
>
> I don't just talk. I wrote the patch to do this too. It's attached.
>
> I tried to keep the changes small, and I implemented the migration
> code
> so that the tables' schemas will automatically be converted. The patch
> needs a good review, though. And testing.
>
> This is the point of the patch (a stored bodystructure string):
>
> sqlite> select bodystructure from 'INBOX/100' LIMIT 2;
> ("TEXT" "PLAIN" ("CHARSET" "ISO-8859-1") NIL NIL "8BIT" 2304 53 NIL
> NIL NIL NIL)
> ("TEXT" "PLAIN" NIL NIL NIL "7BIT" 3829 80 NIL NIL NIL NIL)
> sqlite>
>
> Note about adding BODYSTRUCTURE to the IMAP FETCH query that IMAP
> servers usually cache the body-structures. It wont slowdown an IMAP
> server, one that is worth being called "an IMAP server", much.
>
> It will add some bandwidth. But apparently isn't Evolution trying to
> save bandwidth, otherwise camel-imap-folder.c would use ENVELOPE
> instead
> of BODY.PEEK[HEADER.FIELDS(-a long list-)] (right?).
>
>
> Here are some pointers about things I just wrote:
>
> The Nemomuk message ontology
> http://git.gnome.org/cgit/tracker/tree/data/ontologies/34-nmo.ontology
>
> How we want this stuff in Tracker, how it could be used
> http://live.gnome.org/Tracker/Documentation/EmailSparql
>
> The Evolution plugin:
> http://git.gnome.org/cgit/tracker/tree/src/plugins/evolution/tracker-evolution-plugin.c
>
>
> Let me know!
>
--
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
http://pvanhoof.be/blog
http://codeminded.be
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] PATCH: Storing and fetching bodystructure, the patch
This new version of the patch fixes memoryleak on mi->bodystructure in
message_info_free and fixes a "version < 1" in camel-db.c
On Tue, 2009-07-14 at 12:57 +0200, Philip Van Hoof wrote:
> Hi there,
>
> I would like Evolution to store IMAP's bodystructure as often as
> possible. The reason for this is storing RDF graphs. Let me explain.
>
> I wrote a EPlugin for Evolution that is compiled and distributed by the
> Tracker project. This plugin fetches as much metadata about E-mails as
> possible and pushes it into Tracker's RDF store.
>
> Tracker's RDF store is a Nepomuk-ontology based RDF store that offers
> SPARQL as query language, and SPARQL Update as storage language.
>
> We have enhanced the Nepomuk message ontology so that it can store the
> entire structure of an E-mail. This means the structure that you can
> request using the BODY or BODYSTRUCTURE FETCH requests. These return you
> the MIME structure of the message in a pre-parsed skiplist format.
>
> Using this info our plugin can reconstruct the message's skeleton, but
> then in RDF as a graph.
>
> I don't just talk. I wrote the patch to do this too. It's attached.
>
> I tried to keep the changes small, and I implemented the migration code
> so that the tables' schemas will automatically be converted. The patch
> needs a good review, though. And testing.
>
> This is the point of the patch (a stored bodystructure string):
>
> sqlite> select bodystructure from 'INBOX/100' LIMIT 2;
> ("TEXT" "PLAIN" ("CHARSET" "ISO-8859-1") NIL NIL "8BIT" 2304 53 NIL NIL NIL
> NIL)
> ("TEXT" "PLAIN" NIL NIL NIL "7BIT" 3829 80 NIL NIL NIL NIL)
> sqlite>
>
> Note about adding BODYSTRUCTURE to the IMAP FETCH query that IMAP
> servers usually cache the body-structures. It wont slowdown an IMAP
> server, one that is worth being called "an IMAP server", much.
>
> It will add some bandwidth. But apparently isn't Evolution trying to
> save bandwidth, otherwise camel-imap-folder.c would use ENVELOPE instead
> of BODY.PEEK[HEADER.FIELDS(-a long list-)] (right?).
>
>
> Here are some pointers about things I just wrote:
>
> The Nemomuk message ontology
> http://git.gnome.org/cgit/tracker/tree/data/ontologies/34-nmo.ontology
>
> How we want this stuff in Tracker, how it could be used
> http://live.gnome.org/Tracker/Documentation/EmailSparql
>
> The Evolution plugin:
> http://git.gnome.org/cgit/tracker/tree/src/plugins/evolution/tracker-evolution-plugin.c
>
>
> Let me know!
>
>
> ___
> Evolution-hackers mailing list
> Evolution-hackers@gnome.org
> http://mail.gnome.org/mailman/listinfo/evolution-hackers
--
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
http://pvanhoof.be/blog
http://codeminded.be
diff --git a/camel/camel-db.c b/camel/camel-db.c
index be63c5b..cb131f0 100644
--- a/camel/camel-db.c
+++ b/camel/camel-db.c
@@ -1138,7 +1138,7 @@ camel_db_create_message_info_table (CamelDB *cdb, const gchar *folder_name, Came
gchar *table_creation_query, *safe_index;
/* README: It is possible to compress all system flags into a single column and use just as userflags but that makes querying for other applications difficult an d bloats the parsing code. Instead, it is better to bloat the tables. Sqlite should have some optimizations for sparse columns etc. */
- table_creation_query = sqlite3_mprintf ("CREATE TABLE IF NOT EXISTS %Q ( uid TEXT PRIMARY KEY , flags INTEGER , msg_type INTEGER , read INTEGER , deleted INTEGER , replied INTEGER , important INTEGER , junk INTEGER , attachment INTEGER , dirty INTEGER , size INTEGER , dsent NUMERIC , dreceived NUMERIC , subject TEXT , mail_from TEXT , mail_to TEXT , mail_cc TEXT , mlist TEXT , followup_flag TEXT , followup_completed_on TEXT , followup_due_by TEXT , part TEXT , labels TEXT , usertags TEXT , cinfo TEXT , bdata TEXT, created TEXT, modified TEXT )", folder_name);
+ table_creation_query = sqlite3_mprintf ("CREATE TABLE IF NOT EXISTS %Q ( uid TEXT PRIMARY KEY , flags INTEGER , msg_type INTEGER , read INTEGER , deleted INTEGER , replied INTEGER , important INTEGER , junk INTEGER , attachment INTEGER , dirty INTEGER , size INTEGER , dsent NUMERIC , dreceived NUMERIC , subject TEXT , mail_from TEXT , mail_to TEXT , mail_cc TEXT , mlist TEXT , followup_flag TEXT , followup_completed_on TEXT , followup_due_by TEXT , part TEXT , labels TEXT , usertags TEXT , cinfo TEXT , bdata TEXT, created TEXT, modified TEXT, bodystructure TEXT )", folder_name);
ret = camel_db_add_to_transaction (cdb, table_creation_query, ex);
sqlite3_free (table_creation_query);
@@ -1193,7 +1193,12 @@ camel_db_migrate_folder_prepare (CamelDB *cdb, const gchar *folder_name, gint ve
/* Migration stage one: storing the old data */
- if (version < 1) {
+ if (version < 2) {
+
+ /* Between version 1-2 the following things are changed
+ * ADDED: bodystructure: text
+ * */
+
/* Between version 0-1 the following things are ch
[Evolution-hackers] PATCH: Storing and fetching bodystructure, the patch
Hi there,
I would like Evolution to store IMAP's bodystructure as often as
possible. The reason for this is storing RDF graphs. Let me explain.
I wrote a EPlugin for Evolution that is compiled and distributed by the
Tracker project. This plugin fetches as much metadata about E-mails as
possible and pushes it into Tracker's RDF store.
Tracker's RDF store is a Nepomuk-ontology based RDF store that offers
SPARQL as query language, and SPARQL Update as storage language.
We have enhanced the Nepomuk message ontology so that it can store the
entire structure of an E-mail. This means the structure that you can
request using the BODY or BODYSTRUCTURE FETCH requests. These return you
the MIME structure of the message in a pre-parsed skiplist format.
Using this info our plugin can reconstruct the message's skeleton, but
then in RDF as a graph.
I don't just talk. I wrote the patch to do this too. It's attached.
I tried to keep the changes small, and I implemented the migration code
so that the tables' schemas will automatically be converted. The patch
needs a good review, though. And testing.
This is the point of the patch (a stored bodystructure string):
sqlite> select bodystructure from 'INBOX/100' LIMIT 2;
("TEXT" "PLAIN" ("CHARSET" "ISO-8859-1") NIL NIL "8BIT" 2304 53 NIL NIL NIL NIL)
("TEXT" "PLAIN" NIL NIL NIL "7BIT" 3829 80 NIL NIL NIL NIL)
sqlite>
Note about adding BODYSTRUCTURE to the IMAP FETCH query that IMAP
servers usually cache the body-structures. It wont slowdown an IMAP
server, one that is worth being called "an IMAP server", much.
It will add some bandwidth. But apparently isn't Evolution trying to
save bandwidth, otherwise camel-imap-folder.c would use ENVELOPE instead
of BODY.PEEK[HEADER.FIELDS(-a long list-)] (right?).
Here are some pointers about things I just wrote:
The Nemomuk message ontology
http://git.gnome.org/cgit/tracker/tree/data/ontologies/34-nmo.ontology
How we want this stuff in Tracker, how it could be used
http://live.gnome.org/Tracker/Documentation/EmailSparql
The Evolution plugin:
http://git.gnome.org/cgit/tracker/tree/src/plugins/evolution/tracker-evolution-plugin.c
Let me know!
--
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
http://pvanhoof.be/blog
http://codeminded.be
diff --git a/camel/camel-db.c b/camel/camel-db.c
index be63c5b..504eea4 100644
--- a/camel/camel-db.c
+++ b/camel/camel-db.c
@@ -1138,7 +1138,7 @@ camel_db_create_message_info_table (CamelDB *cdb, const gchar *folder_name, Came
gchar *table_creation_query, *safe_index;
/* README: It is possible to compress all system flags into a single column and use just as userflags but that makes querying for other applications difficult an d bloats the parsing code. Instead, it is better to bloat the tables. Sqlite should have some optimizations for sparse columns etc. */
- table_creation_query = sqlite3_mprintf ("CREATE TABLE IF NOT EXISTS %Q ( uid TEXT PRIMARY KEY , flags INTEGER , msg_type INTEGER , read INTEGER , deleted INTEGER , replied INTEGER , important INTEGER , junk INTEGER , attachment INTEGER , dirty INTEGER , size INTEGER , dsent NUMERIC , dreceived NUMERIC , subject TEXT , mail_from TEXT , mail_to TEXT , mail_cc TEXT , mlist TEXT , followup_flag TEXT , followup_completed_on TEXT , followup_due_by TEXT , part TEXT , labels TEXT , usertags TEXT , cinfo TEXT , bdata TEXT, created TEXT, modified TEXT )", folder_name);
+ table_creation_query = sqlite3_mprintf ("CREATE TABLE IF NOT EXISTS %Q ( uid TEXT PRIMARY KEY , flags INTEGER , msg_type INTEGER , read INTEGER , deleted INTEGER , replied INTEGER , important INTEGER , junk INTEGER , attachment INTEGER , dirty INTEGER , size INTEGER , dsent NUMERIC , dreceived NUMERIC , subject TEXT , mail_from TEXT , mail_to TEXT , mail_cc TEXT , mlist TEXT , followup_flag TEXT , followup_completed_on TEXT , followup_due_by TEXT , part TEXT , labels TEXT , usertags TEXT , cinfo TEXT , bdata TEXT, created TEXT, modified TEXT, bodystructure TEXT )", folder_name);
ret = camel_db_add_to_transaction (cdb, table_creation_query, ex);
sqlite3_free (table_creation_query);
@@ -1193,7 +1193,12 @@ camel_db_migrate_folder_prepare (CamelDB *cdb, const gchar *folder_name, gint ve
/* Migration stage one: storing the old data */
- if (version < 1) {
+ if (version < 2) {
+
+ /* Between version 1-2 the following things are changed
+ * ADDED: bodystructure: text
+ * */
+
/* Between version 0-1 the following things are changed
* ADDED: created: time
* ADDED: modified: time
@@ -1208,7 +1213,7 @@ camel_db_migrate_folder_prepare (CamelDB *cdb, const gchar *folder_name, gint ve
ret = camel_db_add_to_transaction (cdb, table_creation_query, ex);
sqlite3_free (table_creation_query);
- table_creation_query = sqlite3_mprintf ("INSERT INTO 'mem.%q' SELECT uid , flags , msg_type , read , deleted , replied , important , junk , attachment , msg_security , size , dsent , dreceived , sub
