storing From and Subject in xapian

2011-05-16 Thread Sebastian Spaeth
On Sat, 14 May 2011 21:37:25 -0400, Austin Clements wrote: > I wonder if a better approach would be to use > notmuch_message_get_header everywhere, rather than introducing > _notmuch_message_get_header_value, and have it simply recognize > headers that can be retrieved directly from the database.

storing From and Subject in xapian

2011-05-16 Thread Istvan Marko
Austin Clements writes: > I wonder if a better approach would be to use > notmuch_message_get_header everywhere, rather than introducing > _notmuch_message_get_header_value, and have it simply recognize > headers that can be retrieved directly from the database. Then > library callers could

Re: storing From and Subject in xapian

2011-05-16 Thread Sebastian Spaeth
On Sat, 14 May 2011 21:37:25 -0400, Austin Clements amdra...@mit.edu wrote: I wonder if a better approach would be to use notmuch_message_get_header everywhere, rather than introducing _notmuch_message_get_header_value, and have it simply recognize headers that can be retrieved directly from

storing From and Subject in xapian

2011-05-15 Thread servilio
On 12 May 2011 04:39, Istvan Marko wrote: > Stewart Smith writes: > >> Would it be possible to progressively fill the DB with the new data? >> >> i.e. >> >> if Subject/From not in db for message >> ? ?add Subject/From for this message to DB. > > I started looking into this but then realized that

storing From and Subject in xapian

2011-05-14 Thread Austin Clements
I wonder if a better approach would be to use notmuch_message_get_header everywhere, rather than introducing _notmuch_message_get_header_value, and have it simply recognize headers that can be retrieved directly from the database. Then library callers could take advantage of this optimization and

Re: storing From and Subject in xapian

2011-05-14 Thread Austin Clements
I wonder if a better approach would be to use notmuch_message_get_header everywhere, rather than introducing _notmuch_message_get_header_value, and have it simply recognize headers that can be retrieved directly from the database. Then library callers could take advantage of this optimization and

Re: storing From and Subject in xapian

2011-05-14 Thread servilio
On 12 May 2011 04:39, Istvan Marko notm...@kismala.com wrote: Stewart Smith stew...@flamingspork.com writes: Would it be possible to progressively fill the DB with the new data? i.e. if Subject/From not in db for message    add Subject/From for this message to DB. I started looking into

storing From and Subject in xapian

2011-05-12 Thread Istvan Marko
Stewart Smith writes: > Would it be possible to progressively fill the DB with the new data? > > i.e. > > if Subject/From not in db for message >add Subject/From for this message to DB. I started looking into this but then realized that notmuch search opens the database in read-only mode so

Re: storing From and Subject in xapian

2011-05-12 Thread Istvan Marko
Stewart Smith stew...@flamingspork.com writes: Would it be possible to progressively fill the DB with the new data? i.e. if Subject/From not in db for message add Subject/From for this message to DB. I started looking into this but then realized that notmuch search opens the database in

storing From and Subject in xapian

2011-05-11 Thread Stewart Smith
On Sun, 08 May 2011 22:24:37 -0700, Istvan Marko wrote: > Jameson Graef Rollins writes: > > > Unless I hear a strong positive response I'll hold off on considering it > > for 0.6, and suggest instead targeting it for 0.7. > > I would say wait until 0.7 at least. > > An important thing missing

Re: storing From and Subject in xapian

2011-05-10 Thread Stewart Smith
On Sun, 08 May 2011 22:24:37 -0700, Istvan Marko notm...@kismala.com wrote: Jameson Graef Rollins jroll...@finestructure.net writes: Unless I hear a strong positive response I'll hold off on considering it for 0.6, and suggest instead targeting it for 0.7. I would say wait until 0.7 at

storing From and Subject in xapian

2011-05-08 Thread Istvan Marko
Jameson Graef Rollins writes: > Unless I hear a strong positive response I'll hold off on considering it > for 0.6, and suggest instead targeting it for 0.7. I would say wait until 0.7 at least. An important thing missing is fallback to the old method for messages where the Subject/From VALUE

storing From and Subject in xapian

2011-05-08 Thread Jameson Graef Rollins
On Wed, 4 May 2011 21:48:39 -0400, Austin Clements wrote: > This is awesome. What was your machine configuration? Does anyone else have an opinions about this patch? It seems reasonable to me (other than a couple errant comments that were left in and should be removed). It seems worth the

Re: storing From and Subject in xapian

2011-05-08 Thread Jameson Graef Rollins
On Wed, 4 May 2011 21:48:39 -0400, Austin Clements amdra...@mit.edu wrote: This is awesome. What was your machine configuration? Does anyone else have an opinions about this patch? It seems reasonable to me (other than a couple errant comments that were left in and should be removed). It

storing From and Subject in xapian

2011-05-05 Thread Istvan Marko
Austin Clements writes: > This is awesome. What was your machine configuration? Reasonably modern linux box, Core i5. Both the xapian db and the mail files are on the same 7200 RPM SATA drive, ext4 filesystem. I guess the SSD might explain why you your uncached results are not as bad as mine.

storing From and Subject in xapian

2011-05-04 Thread Austin Clements
On Wed, May 4, 2011 at 9:48 PM, Austin Clements wrote: > As another data point, with a probably very different configuration (8 > year old P4, new SSD), my test query was 1.9X faster uncached and 1.6X > faster cached. ?It also produced 60% fewer disk reads. ?I saw the same > 1% increase in

storing From and Subject in xapian

2011-05-04 Thread Austin Clements
This is awesome. What was your machine configuration? As another data point, with a probably very different configuration (8 year old P4, new SSD), my test query was 1.9X faster uncached and 1.6X faster cached. It also produced 60% fewer disk reads. I saw the same 1% increase in database size.

Re: storing From and Subject in xapian

2011-05-04 Thread Austin Clements
This is awesome. What was your machine configuration? As another data point, with a probably very different configuration (8 year old P4, new SSD), my test query was 1.9X faster uncached and 1.6X faster cached. It also produced 60% fewer disk reads. I saw the same 1% increase in database size.

Re: storing From and Subject in xapian

2011-05-04 Thread Austin Clements
On Wed, May 4, 2011 at 9:48 PM, Austin Clements amdra...@mit.edu wrote: As another data point, with a probably very different configuration (8 year old P4, new SSD), my test query was 1.9X faster uncached and 1.6X faster cached.  It also produced 60% fewer disk reads.  I saw the same 1%

storing From and Subject in xapian

2011-05-03 Thread Istvan Marko
I have been looking at the I/O patterns of "notmuch search" with the default output format and noticed that it has to parse the maildir file of every matched message to get the From and Subject headers. I figured that this must be slowing things down, especially when the files are not in the