Re: Provide an option to make thread summaries keep initial subject

2021-12-07 Thread Thomas Schwinge
Hi!

Regarding the following ideas -- from almost a decade ago ;-) -- is
anyone aware of any work in that area?

On 2012-09-25T15:31:37-0400, Austin Clements  wrote:
> Quoth Olivier Berger on Sep 25 at  6:03 pm:
>> Whenever a participant changes the subject in the middle of a thread,
>> the summary reported by notmuch search will change.
>> 
>> However, the result is that some mails tend to "disappear" from search
>> results, when (bad) participants reply instead of composing a new mail,
>> and change a subject (see
>> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=688699 for more details
>> on that problem, and a discussion). Of course, they're still there, but
>> their subject being masked, one may then grep or C-s for a particular
>> subject and miss them.
>> 
>> I think it would be interesting to allow notmuch to potentially keep the
>> original subject and not the most recent one for the thread summaries.
>> 
>> What do you think ?
>
> I think this would be fantastic.  I've proposed unconditionally
> showing the earliest subject before and it seems that people who
> correspond mostly with those who have good threading etiquette would
> prefer this change, but those who correspond with more people who use
> 'reply' like an address book prefer the current behavior.

And then, especially, the following one would be very useful for me:

> Another option, which I'd like to experiment with but haven't found
> the time, is to show *all* distinct subjects for matched messages in a
> thread (modulo "Re:", etc) in the summary buffer, probably on multiple
> lines.  Since most threads only have a single unique subject, they
> would appear just as they do now, but it would be clear when someone
> (or something, like git) changed the subject mid-thread.  This
> approach would be far more robust while retaining good usability, but
> it would require more code than just changing our subject-picking
> heuristic.

I'm aware of notmuch Emacs UI 'notmuch-tree' and 'notmuch-unthreaded',
but these are not quite what is desired here: too verbose, compared to
the concise display variant of 'notmuch-search'.


Grüße
 Thomas
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org


Re: notmuch vs. SIGPIPE

2020-01-21 Thread Thomas Schwinge
Hi!

On 2020-01-20T23:36:51+0100, I wrote:
> On 2020-01-20T12:55:28+0100, I wrote:
>> While looking a bit into the item raised in
>> id:87muamgspy@euler.schwinge.homeip.net I noticed the following
>> (mis?)behavior by notmuch.
>>
>> To set the stage:
>>
>> $ yes | head -n 1
>> y
>> $ echo "${PIPESTATUS[@]}"
>> 141 0
>>
>> As expected, the 'yes' process exits with SIGPIPE right after the 'head'
>> process terminated.
>
> See also , for example.
>
>> However:
>>
>> $ notmuch search \* | head -n 1 & sleep 22; jobs; ps -f
>> [1] 622009
>> thread:00032bb2   the future [1/1] Jenna Moss; Steve Burbon, 
>> Washington (hurd list spam)
>> [1]+  Running notmuch search \* | head -n 1 &
>> UID  PIDPPID  C STIME TTY  TIME CMD
>> thomas6218514297  0 12:38 pts/38   00:00:00 /bin/bash
>> thomas622008  621851 99 12:48 pts/38   00:00:22 
>> /home/thomas/command/notmuch.real search \*
>> thomas622013  621851  0 12:48 pts/38   00:00:00 ps -f
>>
>> Even after its "pipe-consumer" 'head' process has terminated, the
>> 'notmuch' process still keeps running, and running, and running...

> So, libgpgme via libgmime initialization is doing something with signals.
> Here, 'sig=13' is SIGPIPE

> Breakpoint 1, __GI___sigaction (sig=13, act=0x7fffd4e0, oact=0x0) at 
> ../nptl/sigaction.c:24

> (gdb) print *act
> $3 = {__sigaction_handler = {sa_handler = 0x1, sa_sigaction = 0x1}, 
> sa_mask = {__val = {0 }}, sa_flags = 0, sa_restorer = 0x0}
>
> A '0x1' '__sigaction_handler' means 'SIG_IGN', ignore any such signals.
> This is unexpected (to me, at least), not what I'd expect with notmuch?
>
> According to a (very quick) check/survey, this has apparently been the
> case "forever", and is documented on
> ,
> together with some rationale.  Aha, aha, OK, I see.
>
> So, assuming we want to keep it that way (given the gpgme rationale), is
> the problem then that we fail to handle appropriately in notmuch any
> EPIPE that we may be getting from 'write' etc.?  This remains to be
> explored another day.

Indeed.  Using a FIFO to simulate a pipe:

$ mkfifo f
$ head -n 1 < f &
[1] 775394

(This will trigger SIGPIPE/EPIPE after one line.)

$ gdb -q notmuch
Reading symbols from notmuch...
(gdb) break write
Breakpoint 1 at 0xd420
(gdb) run dump > f
Starting program: [...]/notmuch dump > f
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, __GI___libc_write (fd=8, buf=0x55683710, nbytes=8192) at 
../sysdeps/unix/sysv/linux/write.c:26
26  ../sysdeps/unix/sysv/linux/write.c: No such file or directory.
(gdb) bt
#0  0x7789a8f0 in __GI___libc_write (fd=8, buf=0x55683710, 
nbytes=8192) at ../sysdeps/unix/sysv/linux/write.c:26
#1  0x77d8ff99 in gz_comp (state=state@entry=0x55683620, 
flush=0) at gzwrite.c:89
#2  0x77d909ff in gzvprintf (file=0x55683620, format=, va=va@entry=0x7fffd190) at gzwrite.c:442
#3  0x77d90ad4 in gzprintf (file=file@entry=0x55683620, 
format=format@entry=0x5558f54e " -- %s\n") at gzwrite.c:457
#4  0x5556964e in dump_tags_message (size_p=0x7fffd2d0, 
buffer_p=0x7fffd2c8, output=0x55683620, output_format=1, 
message=0x556884e0, ctx=0x555d01b0) at notmuch-dump.c:203
#5  0x5556964e in database_dump_file (include=, 
output_format=1, query_str=, output=0x55683620, 
notmuch=0x555d01b0) at notmuch-dump.c:254
#6  0x5556964e in notmuch_database_dump (notmuch=0x555d01b0, 
output_file_name=0x0, query_str=, 
output_format=DUMP_FORMAT_BATCH_TAG, include=, 
gzip_output=) at notmuch-dump.c:314
#7  0x55569e4b in notmuch_dump_command (config=, 
argc=, argv=0x7fffd860) at notmuch-dump.c:413
#8  0x55565426 in main (argc=2, argv=0x7fffd858) at 
notmuch.c:505

This is the libz 'printf' pass-through code via 'notmuch dump'.

(gdb) finish
Run till exit from #0  __GI___libc_write (fd=8, buf=0x55683710, 
nbytes=8192) at ../sysdeps/unix/sysv/linux/write.c:26
#notmuch-dump batch-tag:3 config,properties,tags

First line got printed.

gz_comp (state=state@entry=0x55683620, flush=0) at gzwrite.c:90
90  gzwrite.c: No such file or directory.
Value returned is $1 = 8192

Supposedly 8192 bytes have been written, which seems a bit dubious, but I
have not studies the fine details of how FIFO buffering vs. SIGPIPE/EPIPE
work -- presumably indeed 8192 bytes were written into the FIFO buffer,
but then only line read, then the FIFO/buffer closed, so the other bytes
lost, and then only on the next 'write', we get...

(gdb) c
Continuing.

Breakpoint 1, __GI___libc_write (fd=8, 

Re: notmuch vs. SIGPIPE

2020-01-20 Thread Thomas Schwinge
Hi!

On 2020-01-20T12:55:28+0100, I wrote:
> While looking a bit into the item raised in
> id:87muamgspy@euler.schwinge.homeip.net I noticed the following
> (mis?)behavior by notmuch.
>
> To set the stage:
>
> $ yes | head -n 1
> y
> $ echo "${PIPESTATUS[@]}"
> 141 0
>
> As expected, the 'yes' process exits with SIGPIPE right after the 'head'
> process terminated.

See also , for example.

> However:
>
> $ notmuch search \* | head -n 1 & sleep 22; jobs; ps -f
> [1] 622009
> thread:00032bb2   the future [1/1] Jenna Moss; Steve Burbon, 
> Washington (hurd list spam)
> [1]+  Running notmuch search \* | head -n 1 &
> UID  PIDPPID  C STIME TTY  TIME CMD
> thomas6218514297  0 12:38 pts/38   00:00:00 /bin/bash
> thomas622008  621851 99 12:48 pts/38   00:00:22 
> /home/thomas/command/notmuch.real search \*
> thomas622013  621851  0 12:48 pts/38   00:00:00 ps -f
>
> Even after its "pipe-consumer" 'head' process has terminated, the
> 'notmuch' process still keeps running, and running, and running...  It
> has to be killed manually (unless it before exits because of concurrent
> database modification).  This doesn't seem expected behavior to me?
>
> Now, I do have a patch or two (actually dozensa; reverts, WIP etc.) on
> top of months-old notmuch sources, so I'll later try to reproduce that
> with clean sources.

$ gdb -q --args notmuch dump
Reading symbols from notmuch...
(gdb) break sigaction
Breakpoint 1 at 0xe130
(gdb) r
Starting program: [...]/notmuch dump
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, __GI___sigaction (sig=13, act=0x0, oact=0x7fffd4e0) at 
../nptl/sigaction.c:24
24  ../nptl/sigaction.c: No such file or directory.
(gdb) bt
#0  0x777e92f0 in __GI___sigaction (sig=13, act=0x0, 
oact=0x7fffd4e0) at ../nptl/sigaction.c:24
#1  0x775bfa8d in  () at /usr/lib/x86_64-linux-gnu/libgpgme.so.11
#2  0x775c64cd in gpgme_check_version () at 
/usr/lib/x86_64-linux-gnu/libgpgme.so.11
#3  0x775c6667 in gpgme_check_version_internal () at 
/usr/lib/x86_64-linux-gnu/libgpgme.so.11
#4  0x77f456b3 in g_mime_init () at 
/usr/lib/x86_64-linux-gnu/libgmime-3.0.so.0
#5  0x5556539d in main (argc=2, argv=0x7fffd828) at 
notmuch.c:469

So, libgpgme via libgmime initialization is doing something with signals.
Here, 'sig=13' is SIGPIPE, 'act=0x0' means to just query, and store
current handling into 'oact':

(gdb) finish
Run till exit from #0  __GI___sigaction (sig=13, act=0x0, 
oact=0x7fffd4e0) at ../nptl/sigaction.c:24
0x775bfa8d in ?? () from /usr/lib/x86_64-linux-gnu/libgpgme.so.11
Value returned is $1 = 0
(gdb) print *(struct sigaction *) 0x7fffd4e0
$2 = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask 
= {__val = {0, 8, 93824992565632, 0, 32, 88, 0, 22, 0, 140733193388034, 
93823560581120, 7, 93824992559120, 32, 5, 93824992694848}}, sa_flags = 0, 
sa_restorer = 0x0}

A '0x0' '__sigaction_handler' means 'SIG_DFL', which for SIGPIPE means to
terminate the process.  This is as expected.  However, then:

(gdb) continue 
Continuing.

Breakpoint 1, __GI___sigaction (sig=13, act=0x7fffd4e0, oact=0x0) at 
../nptl/sigaction.c:24
24  in ../nptl/sigaction.c
(gdb) bt
#0  0x777e92f0 in __GI___sigaction (sig=13, act=0x7fffd4e0, 
oact=0x0) at ../nptl/sigaction.c:24
#1  0x775bfadc in  () at /usr/lib/x86_64-linux-gnu/libgpgme.so.11
#2  0x775c64cd in gpgme_check_version () at 
/usr/lib/x86_64-linux-gnu/libgpgme.so.11
#3  0x775c6667 in gpgme_check_version_internal () at 
/usr/lib/x86_64-linux-gnu/libgpgme.so.11
#4  0x77f456b3 in g_mime_init () at 
/usr/lib/x86_64-linux-gnu/libgmime-3.0.so.0
#5  0x5556539d in main (argc=2, argv=0x7fffd828) at 
notmuch.c:469
(gdb) print *act
$3 = {__sigaction_handler = {sa_handler = 0x1, sa_sigaction = 0x1}, sa_mask 
= {__val = {0 }}, sa_flags = 0, sa_restorer = 0x0}

A '0x1' '__sigaction_handler' means 'SIG_IGN', ignore any such signals.
This is unexpected (to me, at least), not what I'd expect with notmuch?

According to a (very quick) check/survey, this has apparently been the
case "forever", and is documented on
,
together with some rationale.  Aha, aha, OK, I see.

So, assuming we want to keep it that way (given the gpgme rationale), is
the problem then that we fail to handle appropriately in notmuch any
EPIPE that we may be getting from 'write' etc.?  This remains to be
explored another day.


Grüße
 Thomas
___
notmuch mailing list

notmuch vs. SIGPIPE

2020-01-20 Thread Thomas Schwinge
Hi!

While looking a bit into the item raised in
id:87muamgspy@euler.schwinge.homeip.net I noticed the following
(mis?)behavior by notmuch.

To set the stage:

$ yes | head -n 1
y
$ echo "${PIPESTATUS[@]}"
141 0

As expected, the 'yes' process exits with SIGPIPE right after the 'head'
process terminated.  However:

$ notmuch search \* | head -n 1 & sleep 22; jobs; ps -f
[1] 622009
thread:00032bb2   the future [1/1] Jenna Moss; Steve Burbon, 
Washington (hurd list spam)
[1]+  Running notmuch search \* | head -n 1 &
UID  PIDPPID  C STIME TTY  TIME CMD
thomas6218514297  0 12:38 pts/38   00:00:00 /bin/bash
thomas622008  621851 99 12:48 pts/38   00:00:22 
/home/thomas/command/notmuch.real search \*
thomas622013  621851  0 12:48 pts/38   00:00:00 ps -f

Even after its "pipe-consumer" 'head' process has terminated, the
'notmuch' process still keeps running, and running, and running...  It
has to be killed manually (unless it before exits because of concurrent
database modification).  This doesn't seem expected behavior to me?

Now, I do have a patch or two (actually dozensa; reverts, WIP etc.) on
top of months-old notmuch sources, so I'll later try to reproduce that
with clean sources.


Grüße
 Thomas


signature.asc
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Feature idea: Emacs 'notmuch-search' "paged" display

2020-01-18 Thread Thomas Schwinge
Hi!

I had this following idea, but my Emacs foo is too limited to start
implementing this -- if reasonably possible at all.

Here is -- I suppose, not verified -- what happens if you do a
command-line 'notmuch search [...] | less': the 'notmuch' process creates
some output and the 'less' process reads some.  At some point, 'less'
stops reading (let's assume when the screen is full, for simplicity),
then the pipe gets filled up (some few KiB, I think?), and then 'notmuch'
blocks in 'write', so doesn't do any futher work until 'less' reads more
data.

Here is -- I suppose, not verified -- what happens if you do a Emacs
'notmuch-search [...]': the 'notmuch' process creates some output and the
Emacs process reads some.  And then some more, and again, and so on,
until 'EOF'.  So, 'notmuch' never blocks in 'write', because Emacs reads
it all.

Now, I have a number of generic search queries that produce a long list
of search results, so to avoid Emacs building up a huge 'notmuch-search'
buffer I usually quickly do "M-: (ignore-errors (interrupt-process))".
Oftentimes I'm only interested in the first few results, or I'm
immediately going to 'notmuch-search-filter' the results.

Is it possible in Emacs to emulared the "paged" display as explained in
the 'less' example?  That is, only read in "a screenful of data", then
let the pipe fill up, then 'notmuch' block, and only once the user moves
the point towards the end of the current Emacs buffer, continue to read
data from the pipe.

I'm willing to try implementing this, but I'd appreciate some
sanity-checking as well as guidance by somebody more familiar with Emacs
internals.


Grüße
 Thomas


signature.asc
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Transitioning notmuch/Xapian from 32-bit to 64-bit system

2019-07-09 Thread Thomas Schwinge
Hi!

Suppose you have a huge notmuch/Xapian database, built on a 32-bit system
(well, actually on x86_64-pc-linux-gnu, but using a years old 32-bit
notmuch binary; notmuch 0.9, Xapian 1.2.21 -- don't laugh), and suppose
you're finally going to update that years old notmuch installation
(release by release, forward-porting a bunch of patches).  Naturally, I'd
now do a native 64-bit build (unless somebody tells me a 32-bit build
would actually be preferable in terms of memory savings due to pointer
sizes?).  Doing some light (read-only!) testing, it seems to work fine to
access the old 32-bit built Xapian "chert" database with 64-bit
notmuch/Xapian.  Is that (a) generally safe and expected to work fine,
(b) there may be issues, or (c) don't do that, or don't know?

(Of course, I'll eventually want to rebuild the database, to take
advantage of several new features both on the notmuch and Xapian sides,
but I'd prefer to do that later.)


Grüße
 Thomas


signature.asc
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


header continuation issue in notmuch frontend/alot/pythons email module

2013-06-24 Thread Thomas Schwinge
Hi!

On Mon, 24 Jun 2013 10:57:10 +0200, Justus Winter <4winter at 
informatik.uni-hamburg.de> wrote:
> Quoting Austin Clements (2013-06-23 18:59:39)
> > Quoth Justus Winter on Jun 23 at  3:11 pm:
> > > I recently had a problem replying to a mail written by Thomas Schwinge
> > > using an oldish notmuch. Not sure if it has been fixed in more recent

"Oldish", yeah, yeah, I know...  (Mumbles someting about long TODO list.)

> > > versions, but I think notmuch could improve uppon its header
> > > generation (see below). Problematic part of the mail:
> > > 
> > > ~~~ snip ~~~
> > > [...]
> > > To: someone at example.org, "line
> > >  break" , someoneelse at example.org
> > > User-Agent: Notmuch/0.9-101-g81dad07 (http://notmuchmail.org) 
> > > Emacs/23.4.1 (i486-pc-linux-gnu)
> > > [...]
> > > ~~~ snap ~~~

> > Do you happen to know how the strangely folded "to" header was
> > produced for this message?

I just entered/copied all the addresses into one long To: line, and then
let message-mode do its thing.

> No, but Thomas might. Thomas, the problematic message is
> id:877ghpqckb.fsf at kepler.schwinge.homeip.net

Here is the header from the message as I sent it:

To: Samuel Thibault , Justus Winter
 <4winter at informatik.uni-hamburg.de>, fotis.koutoulakis at gmail.com, Ian
 Lance Taylor , toscano.pino at tiscali.it, Luis Machado
 , =?utf-8?B?6ZmG5bKz?=
 

And this is what I received from the bug-hurd mailing list:

To: Samuel Thibault , Justus Winter
<4winter at informatik.uni-hamburg.de>, , "Ian
Lance Taylor" , , Luis 
Machado
,
=?utf-8?B?6ZmG5bKz?= 

So the "corruption" (if it is declared as one; I don't have time right
now to follow your RFC interpretation) must have happened after sending
it off -- perhaps my company's Microsoft Exchange server (as Justus
received a direct copy from that one), or even msmtp used as the local
MTA.


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20130624/0ecdd369/attachment-0001.pgp>


Re: [PATCH] dump: Don't sort.

2011-11-29 Thread Thomas Schwinge
Hi!

First, thanks to David, Tomi, Tom for moving this forward.


On Sat, 19 Nov 2011 16:11:13 +0100, Petter Reinholdtsen p...@hungry.com wrote:
 [Thomas Schwinge]
  +/* This used to use NOTMUCH_SORT_MESSAGE_ID.  On 2011-10-29, a 
  measurement
  + * on a 372981 messages instance showed that wall time can be reduced 
  from
  + * 28 minutes (sorted by Message-ID) to 15 minutes (unsorted), the 
  latter
  + * being much more ``database-disk-layout-friendly''.  Subsequently 
  sorting
  + * the 25 MiB of data is a no-brainer, if required.  */

Here is the measurement re-done -- I discovered that while doing the
former, there had been parallel work been done in another Xen domU on
that system, disturbing the measurement.

Discard caches, every time before dumping:

$ sync; sleep 3; echo -n 3 | sudo dd of=/proc/sys/vm/drop_caches

Original (sorted by Message-ID):

$ \time notmuch dump  ~/tmp/Mail-notmuch_dump/dump
26.41user 16.56system 14:34.81elapsed 4%CPU (0avgtext+0avgdata 
167152maxresident)k
2994440inputs+55896outputs (41major+11627minor)pagefaults 0swaps

Unsorted:

$ \time notmuch dump | sort  ~/tmp/Mail-notmuch_dump/dump
24.79user 3.86system 12:00.22elapsed 3%CPU (0avgtext+0avgdata 
57216maxresident)k
2929192inputs+0outputs (40major+4942minor)pagefaults 0swaps

The difference is no longer as big as before, but still better than
nothing.

 This sound like a great idea for my use case.  Doing 'notmuch dump'
 with my 1.2 million emails take hours at the moment (not very fast
 encrypted file system), and result in a 90 MiB dump file.

... and you will gain most by putting the .notmuch directory onto a SSD,
as I have done by now:

Original (sorted by Message-ID), with .notmuch on SSD:

$ \time notmuch dump  ~/tmp/Mail-notmuch_dump/dump
24.86user 13.40system 1:06.01elapsed 57%CPU (0avgtext+0avgdata 
167200maxresident)k
2992184inputs+55920outputs (49major+11622minor)pagefaults 0swaps

Unsorted, with .notmuch on SSD:

$ \time notmuch dump  ~/tmp/Mail-notmuch_dump/dump
21.90user 2.68system 0:51.70elapsed 47%CPU (0avgtext+0avgdata 
57248maxresident)k
2926912inputs+55920outputs (50major+4934minor)pagefaults 0swaps

User and system time (roughly) remain the same, but the wall time drops
considerably -- a SSD at its best, obviously.


Generally speaking, I decided it was enough to just put the .notmuch
directory onto the SSD, and not the whole mail store: if new messages are
added (notmuch new), they're still in the page cache anyway (having been
retrieven via POP3 or whatever just before), and for regular message read
access, a HDD's seek time shouldn't matter too much (and I've taken
notice of Austin's patches which even retrieven Subject: etc. from the
DB), so what remains to be optimized is random access to the DB.


Grüße,
 Thomas


pgpWlrh1zMijp.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH] dump: Don't sort.

2011-11-28 Thread Thomas Schwinge
Hi!

First, thanks to David, Tomi, Tom for moving this forward.


On Sat, 19 Nov 2011 16:11:13 +0100, Petter Reinholdtsen  
wrote:
> [Thomas Schwinge]
> > +/* This used to use NOTMUCH_SORT_MESSAGE_ID.  On 2011-10-29, a 
> > measurement
> > + * on a 372981 messages instance showed that wall time can be reduced 
> > from
> > + * 28 minutes (sorted by Message-ID) to 15 minutes (unsorted), the 
> > latter
> > + * being much more ``database-disk-layout-friendly''.  Subsequently 
> > sorting
> > + * the 25 MiB of data is a no-brainer, if required.  */

Here is the measurement re-done -- I discovered that while doing the
former, there had been parallel work been done in another Xen domU on
that system, disturbing the measurement.

Discard caches, every time before dumping:

$ sync; sleep 3; echo -n 3 | sudo dd of=/proc/sys/vm/drop_caches

Original (sorted by Message-ID):

$ \time notmuch dump > ~/tmp/Mail-notmuch_dump/dump
26.41user 16.56system 14:34.81elapsed 4%CPU (0avgtext+0avgdata 
167152maxresident)k
2994440inputs+55896outputs (41major+11627minor)pagefaults 0swaps

Unsorted:

$ \time notmuch dump | sort > ~/tmp/Mail-notmuch_dump/dump
24.79user 3.86system 12:00.22elapsed 3%CPU (0avgtext+0avgdata 
57216maxresident)k
2929192inputs+0outputs (40major+4942minor)pagefaults 0swaps

The difference is no longer as big as before, but still better than
nothing.

> This sound like a great idea for my use case.  Doing 'notmuch dump'
> with my 1.2 million emails take hours at the moment (not very fast
> encrypted file system), and result in a 90 MiB dump file.

... and you will gain most by putting the .notmuch directory onto a SSD,
as I have done by now:

Original (sorted by Message-ID), with .notmuch on SSD:

$ \time notmuch dump > ~/tmp/Mail-notmuch_dump/dump
24.86user 13.40system 1:06.01elapsed 57%CPU (0avgtext+0avgdata 
167200maxresident)k
2992184inputs+55920outputs (49major+11622minor)pagefaults 0swaps

Unsorted, with .notmuch on SSD:

$ \time notmuch dump > ~/tmp/Mail-notmuch_dump/dump
21.90user 2.68system 0:51.70elapsed 47%CPU (0avgtext+0avgdata 
57248maxresident)k
2926912inputs+55920outputs (50major+4934minor)pagefaults 0swaps

User and system time (roughly) remain the same, but the wall time drops
considerably -- a SSD at its best, obviously.


Generally speaking, I decided it was enough to just put the .notmuch
directory onto the SSD, and not the whole mail store: if new messages are
added (notmuch new), they're still in the page cache anyway (having been
retrieven via POP3 or whatever just before), and for regular message read
access, a HDD's seek time shouldn't matter too much (and I've taken
notice of Austin's patches which even retrieven Subject: etc. from the
DB), so what remains to be optimized is random access to the DB.


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/2028/9b7e457c/attachment-0001.pgp>


[PATCH 2/2] notmuch.el:notmuch-search-process-filter: Rewritten. Cope with incomplete lines.

2011-11-15 Thread Thomas Schwinge
Hi!

On Thu, 10 Mar 2011 18:02:09 -0800, Carl Worth  wrote:
> On Thu,  3 Feb 2011 00:56:39 +0100, Thomas Schwinge  
> wrote:
> > This issue has been lying in ambush as of 2009-11-24's commit
> > 93af7b574598637c2766dd1f8ef343962c9a8efb.
> 
> Thanks very much for tracking down this bug, Thomas. What a nasty bug to
> have in notmuch!
> 
> Your fix seems to drop the last thread from the search results
> view. I've now committed a slightly modified fix that avoids that
> problem. I also made the test case provide slightly cleaner results.
> 
> Let me know if you see any problems.

That is much better, thanks!

But we're not there yet...  %-| That is, today I hit another issue that
appears to hide in the same elisp code.  See ``Error: Unexpected output


Google+ presence

2011-11-08 Thread Thomas Schwinge
Hi!

I created a page on Google+ for notmuch:
.  Come in and say hello!


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 



Google+ presence

2011-11-08 Thread Thomas Schwinge
Hi!

I created a page on Google+ for notmuch:
https://plus.google.com/103455990805261371148.  Come in and say hello!


Grüße,
 Thomas


pgpp536YbkJw6.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH] restore: Be more liberal in which data to accept.

2011-10-29 Thread Thomas Schwinge
From: Thomas Schwinge <tho...@schwinge.name>

There are ``Message-ID''s out in the wild that contain spaces.

---


Hi!

Carl, the main question for you is: does this break sup-import
operability?


Spammers are quite inventive for creating ``interesting Messages-ID''s.
Apparently, notmuch handles these fine internally, but it breaks a
dump/restore cycle:

$ notmuch restore < ~/tmp/Mail-notmuch_dump/dump
No filename given. Reading dump from stdin.
Warning: Ignoring invalid input line: 3791856948.991306994491 at m0.net 
Received:fromdialup-62.215.274.4.dial1.stamford([62.215.274.4]  ([...])
Warning: Ignoring invalid input line: PM200010:29:54 AM ([...])
Warning: Ignoring invalid input line: PM200010:51:48 AM ([...])
Warning: Ignoring invalid input line: PM200011:47:35 AM ([...])
Warning: Ignoring invalid input line: PM200011:48:46 AM ([...])
Warning: Ignoring invalid input line: PM200011:50:10 AM ([...])
Warning: Ignoring invalid input line: PM200012:21:05 AM ([...])
Warning: Ignoring invalid input line: PM200012:21:17 AM ([...])
Warning: Ignoring invalid input line: PM200012:21:18 AM ([...])
Warning: Ignoring invalid input line: PM200012:21:32 AM ([...])
Warning: Ignoring invalid input line: PM20001:48:38 PM ([...])
Warning: Ignoring invalid input line: PM20001:53:07 PM ([...])
Warning: Ignoring invalid input line: PM20004:01:48 AM ([...])
Warning: Ignoring invalid input line: PM20004:01:59 AM ([...])
Warning: Ignoring invalid input line: PM20004:10:44 AM ([...])
Warning: Ignoring invalid input line: PM20004:20:00 AM ([...])
Warning: Ignoring invalid input line: PM20005:06:50 PM ([...])
Warning: Ignoring invalid input line: PM20005:14:17 AM ([...])
Warning: Ignoring invalid input line: PM20005:32:15 PM ([...])
Warning: Ignoring invalid input line: PM20005:32:22 PM ([...])
Warning: Ignoring invalid input line: PM20005:33:05 PM ([...])
Warning: Ignoring invalid input line: PM20005:33:57 AM ([...])
Warning: Ignoring invalid input line: PM20006:24:12 AM ([...])
Warning: Ignoring invalid input line: PM20006:25:04 AM ([...])
Warning: Ignoring invalid input line: PM20006:25:49 AM ([...])
Warning: Ignoring invalid input line: PM20006:26:11 AM ([...])
Warning: Ignoring invalid input line: PM20007:05:34 PM ([...])
Warning: Ignoring invalid input line: PM2000PM 04:09:15 ([...])
Warning: Ignoring invalid input line: PM2000 11:07:41 ([...])
Warning: Ignoring invalid input line: PM2000 12:42:47 ([...])
Warning: Ignoring invalid input line: PM2000 12:42:48 ([...])
Warning: Ignoring invalid input line: PM2000 5:58:28 ([...])
Warning: Ignoring invalid input line: PM2000 6:30:51 ([...])
Warning: Ignoring invalid input line: Prospect Mailer 2:37:04 ([...])
Warning: Ignoring invalid input line: Prospect Mailer 2:37:09 ([...])
Warning: Ignoring invalid input line: Prospect Mailer 2:37:11 ([...])
Warning: Ignoring invalid input line: Prospect Mailer 2:37:12 ([...])
Warning: Ignoring invalid input line: Prospect Mailer 2:37:45 ([...])
Warning: Ignoring invalid input line: Prospect Mailer 2:38:10 ([...])

Thus, dump; remove all tags; restore is not nullipotent, which it should
be.

Especially noteworthy is probably the first one: it happens to have
gotten a Received line mangled into the Message-ID, and it ends with a
space character.

Some more from the freak show:

$MESSAGE_ID ([...])

%CUSTOM_CHAR[8-10]$%CUSTOM_CHAR[8-10]$%CUSTOM_CHAR[8-10]@%CUSTOM_DOMAIN.msn.com 
([...])
%RNDDIGIT1025.%RNDDIGIT15%RNDLCCHAR15%RNDDIGIT110%RNDLCCHAR13@ ([...])
%RNDDIGIT1025.%RNDDIGIT15%RNDLCCHAR15%RNDDIGIT110ucp at yahoo.com ([...])
%RNDDIGIT1025.%RNDDIGIT15%RNDLCCHAR15%RNDDIGIT110vs at yahoo.com ([...])
%RNDDIGIT27eq52md1$9rg57p%RNDDIGIT14$277ts40lsh@%RNDWORD13ivo4068 ([...])
%RNDDIGIT27g10u874$3cqh62f%RNDDIGIT14$7fgo121wnwt@%RNDWORD13quw32712 ([...])
%RNDDIGIT27mog75vx711$541xqm480xc%RNDDIGIT14$031nq1pk@%RNDWORD13av2979 
([...])
%RNDDIGIT27nqf761drk7$7l4mza%RNDDIGIT14$96ijq17zq@%RNDWORD13b1779 ([...])
%RNDDIGIT27q0tcg10$94pcn1mw%RNDDIGIT14$7x77pztx@%RNDWORD13ny7619 ([...])
%RNDDIGIT27uiw866tv49$5c3rg%RNDDIGIT14$6jl43vv@%RNDWORD13uwh17820 ([...])
%RNDDIGIT27x966lug3$0pr016r%RNDDIGIT14$8ye15k@%RNDWORD13qps90907 ([...])

%RNDDIGIT310%RNDLCCHAR15%RNDDIGIT15%RNDLCCHAR15$%RNDDIGIT17%RNDDIGIT13%RNDLCCHAR13%RNDDIGIT13$%RNDDIGIT15%RNDLCCHAR13%RNDDIGIT13%RNDLCCHAR13%RNDDIGIT13@
 ([...])

%RNDDIGIT310%RNDLCCHAR15%RNDDIGIT15%RNDLCCHAR15$%RNDDIGIT17%RNDDIGIT13%RNDLCCHAR13%RNDDIGIT13$%RNDDIGIT15%RNDLCCHAR13%RNDDIGIT13%RNDLCCHAR13%RNDDIGIT13
 at bambi ([...])

%RNDDIGIT310%RNDLCCHAR15%RNDDIGIT15%RNDLCCHAR15$%RNDDIGIT17%RNDDIGIT13%RNDLCCHAR13%RNDDIGIT13$%RNDDIGIT15%RNDLCCHAR13%RNDDIGIT13%RNDLCCHAR13%RNDDIGIT13
 at wheelchair ([...])
%RNDDIGIT520.%RNDDIGIT110.%RNDDIGIT110 at -%RN

[PATCH] dump: Don't sort.

2011-10-29 Thread Thomas Schwinge
From: Thomas Schwinge <tho...@schwinge.name>

This improves usage experience considerably in the given scenario.

---


Hi!

I decided that it'd be useful to put the reasoning and data right next to
the source code (as opposed to putting it into the commit message), for
the next guy to read this code has it all in one place.


Gr??e,
 Thomas


---

 notmuch-dump.c |7 ++-
 1 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/notmuch-dump.c b/notmuch-dump.c
index 7e7bc17..a431e23 100644
--- a/notmuch-dump.c
+++ b/notmuch-dump.c
@@ -45,7 +45,12 @@ notmuch_dump_command (unused (void *ctx), int argc, char 
*argv[])
fprintf (stderr, "Out of memory\n");
return 1;
 }
-notmuch_query_set_sort (query, NOTMUCH_SORT_MESSAGE_ID);
+/* This used to use NOTMUCH_SORT_MESSAGE_ID.  On 2011-10-29, a measurement
+ * on a 372981 messages instance showed that wall time can be reduced from
+ * 28 minutes (sorted by Message-ID) to 15 minutes (unsorted), the latter
+ * being much more ``database-disk-layout-friendly''.  Subsequently sorting
+ * the 25 MiB of data is a no-brainer, if required.  */
+notmuch_query_set_sort (query, NOTMUCH_SORT_UNSORTED);

 if (argc) {
output = fopen (argv[0], "w");
-- 
tg: (3bafdfc..) t/dump_unsorted (depends on: baseline)


[PATCH] dump: Don't sort.

2011-10-29 Thread Thomas Schwinge
From: Thomas Schwinge tho...@schwinge.name

This improves usage experience considerably in the given scenario.

---


Hi!

I decided that it'd be useful to put the reasoning and data right next to
the source code (as opposed to putting it into the commit message), for
the next guy to read this code has it all in one place.


Grüße,
 Thomas


---

 notmuch-dump.c |7 ++-
 1 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/notmuch-dump.c b/notmuch-dump.c
index 7e7bc17..a431e23 100644
--- a/notmuch-dump.c
+++ b/notmuch-dump.c
@@ -45,7 +45,12 @@ notmuch_dump_command (unused (void *ctx), int argc, char 
*argv[])
fprintf (stderr, Out of memory\n);
return 1;
 }
-notmuch_query_set_sort (query, NOTMUCH_SORT_MESSAGE_ID);
+/* This used to use NOTMUCH_SORT_MESSAGE_ID.  On 2011-10-29, a measurement
+ * on a 372981 messages instance showed that wall time can be reduced from
+ * 28 minutes (sorted by Message-ID) to 15 minutes (unsorted), the latter
+ * being much more ``database-disk-layout-friendly''.  Subsequently sorting
+ * the 25 MiB of data is a no-brainer, if required.  */
+notmuch_query_set_sort (query, NOTMUCH_SORT_UNSORTED);
 
 if (argc) {
output = fopen (argv[0], w);
-- 
tg: (3bafdfc..) t/dump_unsorted (depends on: baseline)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH] restore: Be more liberal in which data to accept.

2011-10-29 Thread Thomas Schwinge
From: Thomas Schwinge tho...@schwinge.name

There are ``Message-ID''s out in the wild that contain spaces.

---


Hi!

Carl, the main question for you is: does this break sup-import
operability?


Spammers are quite inventive for creating ``interesting Messages-ID''s.
Apparently, notmuch handles these fine internally, but it breaks a
dump/restore cycle:

$ notmuch restore  ~/tmp/Mail-notmuch_dump/dump
No filename given. Reading dump from stdin.
Warning: Ignoring invalid input line: 3791856948.991306994...@m0.net 
Received:fromdialup-62.215.274.4.dial1.stamford([62.215.274.4]  ([...])
Warning: Ignoring invalid input line: PM200010:29:54 AM ([...])
Warning: Ignoring invalid input line: PM200010:51:48 AM ([...])
Warning: Ignoring invalid input line: PM200011:47:35 AM ([...])
Warning: Ignoring invalid input line: PM200011:48:46 AM ([...])
Warning: Ignoring invalid input line: PM200011:50:10 AM ([...])
Warning: Ignoring invalid input line: PM200012:21:05 AM ([...])
Warning: Ignoring invalid input line: PM200012:21:17 AM ([...])
Warning: Ignoring invalid input line: PM200012:21:18 AM ([...])
Warning: Ignoring invalid input line: PM200012:21:32 AM ([...])
Warning: Ignoring invalid input line: PM20001:48:38 PM ([...])
Warning: Ignoring invalid input line: PM20001:53:07 PM ([...])
Warning: Ignoring invalid input line: PM20004:01:48 AM ([...])
Warning: Ignoring invalid input line: PM20004:01:59 AM ([...])
Warning: Ignoring invalid input line: PM20004:10:44 AM ([...])
Warning: Ignoring invalid input line: PM20004:20:00 AM ([...])
Warning: Ignoring invalid input line: PM20005:06:50 PM ([...])
Warning: Ignoring invalid input line: PM20005:14:17 AM ([...])
Warning: Ignoring invalid input line: PM20005:32:15 PM ([...])
Warning: Ignoring invalid input line: PM20005:32:22 PM ([...])
Warning: Ignoring invalid input line: PM20005:33:05 PM ([...])
Warning: Ignoring invalid input line: PM20005:33:57 AM ([...])
Warning: Ignoring invalid input line: PM20006:24:12 AM ([...])
Warning: Ignoring invalid input line: PM20006:25:04 AM ([...])
Warning: Ignoring invalid input line: PM20006:25:49 AM ([...])
Warning: Ignoring invalid input line: PM20006:26:11 AM ([...])
Warning: Ignoring invalid input line: PM20007:05:34 PM ([...])
Warning: Ignoring invalid input line: PM2000PM 04:09:15 ([...])
Warning: Ignoring invalid input line: PM2000¿ÀÀü 11:07:41 ([...])
Warning: Ignoring invalid input line: PM2000¿ÀÈÄ 12:42:47 ([...])
Warning: Ignoring invalid input line: PM2000¿ÀÈÄ 12:42:48 ([...])
Warning: Ignoring invalid input line: PM2000¿ÀÈÄ 5:58:28 ([...])
Warning: Ignoring invalid input line: PM2000¿ÀÈÄ 6:30:51 ([...])
Warning: Ignoring invalid input line: Prospect Mailer 2:37:04 ([...])
Warning: Ignoring invalid input line: Prospect Mailer 2:37:09 ([...])
Warning: Ignoring invalid input line: Prospect Mailer 2:37:11 ([...])
Warning: Ignoring invalid input line: Prospect Mailer 2:37:12 ([...])
Warning: Ignoring invalid input line: Prospect Mailer 2:37:45 ([...])
Warning: Ignoring invalid input line: Prospect Mailer 2:38:10 ([...])

Thus, dump; remove all tags; restore is not nullipotent, which it should
be.

Especially noteworthy is probably the first one: it happens to have
gotten a Received line mangled into the Message-ID, and it ends with a
space character.

Some more from the freak show:

$MESSAGE_ID ([...])

%CUSTOM_CHAR[8-10]$%CUSTOM_CHAR[8-10]$%CUSTOM_CHAR[8-10]@%CUSTOM_DOMAIN.msn.com 
([...])
%RNDDIGIT1025.%RNDDIGIT15%RNDLCCHAR15%RNDDIGIT110%RNDLCCHAR13@ ([...])
%rnddigit1025.%rnddigit15%rndlcchar15%rnddigit110...@yahoo.com ([...])
%rnddigit1025.%rnddigit15%rndlcchar15%rnddigit11...@yahoo.com ([...])
%RNDDIGIT27eq52md1$9rg57p%RNDDIGIT14$277ts40lsh@%RNDWORD13ivo4068 ([...])
%RNDDIGIT27g10u874$3cqh62f%RNDDIGIT14$7fgo121wnwt@%RNDWORD13quw32712 ([...])
%RNDDIGIT27mog75vx711$541xqm480xc%RNDDIGIT14$031nq1pk@%RNDWORD13av2979 
([...])
%RNDDIGIT27nqf761drk7$7l4mza%RNDDIGIT14$96ijq17zq@%RNDWORD13b1779 ([...])
%RNDDIGIT27q0tcg10$94pcn1mw%RNDDIGIT14$7x77pztx@%RNDWORD13ny7619 ([...])
%RNDDIGIT27uiw866tv49$5c3rg%RNDDIGIT14$6jl43vv@%RNDWORD13uwh17820 ([...])
%RNDDIGIT27x966lug3$0pr016r%RNDDIGIT14$8ye15k@%RNDWORD13qps90907 ([...])

%RNDDIGIT310%RNDLCCHAR15%RNDDIGIT15%RNDLCCHAR15$%RNDDIGIT17%RNDDIGIT13%RNDLCCHAR13%RNDDIGIT13$%RNDDIGIT15%RNDLCCHAR13%RNDDIGIT13%RNDLCCHAR13%RNDDIGIT13@
 ([...])

%RNDDIGIT310%RNDLCCHAR15%RNDDIGIT15%RNDLCCHAR15$%RNDDIGIT17%RNDDIGIT13%RNDLCCHAR13%RNDDIGIT13$%RNDDIGIT15%RNDLCCHAR13%RNDDIGIT13%RNDLCCHAR13%RNDDIGIT13@bambi
 ([...])

%RNDDIGIT310%RNDLCCHAR15%RNDDIGIT15%RNDLCCHAR15$%RNDDIGIT17%RNDDIGIT13%RNDLCCHAR13%RNDDIGIT13$%RNDDIGIT15%RNDLCCHAR13%RNDDIGIT13%RNDLCCHAR13%RNDDIGIT13@wheelchair
 ([...])
%RNDDIGIT520.%RNDDIGIT110.%RNDDIGIT110@-%RNDLCCHAR13%RNDDIGIT13

Austin's custom query parser: folder/directory searching, some numbers

2011-10-27 Thread Thomas Schwinge
Hi!

As I already told on IRC (and which I still have to polish and
publish...), I recently merged Austin's custom query parser into my local
tree, mainly (for now) for its exact folder/directory searching
capabilities.

Austin had published this work several months ago, and Carl in the mean
time had implemented his own folder: searches.  Now, there was a conflict
about which to use; they have different semantics, Carl's being
inadequate for my use case (not rooted, for example).  On IRC, Carl
recently had the most pragmatic solution for how to approach this: if we
can't agree on having either his folder: semantics, or Austin's strict
filename matching -- then just have both of them.  So I now have arranged
for having both Carl's folder: (with it's ``weak'' mail folder
semantics), and also Austin's directory: (with it's ``hard''
directory/filename matching semantics), and on top of the latter
implemented rdirectory: which extends directory: by recursive matching.
This works really nice.


IRC, freenode, #notmuch, 2011-09-30:

 tschwinge: Before you get in too deep I should point out
  that there's a (not unsurmountable) flaw in the folder handling.
  Because it expands to all of the desired dir-entry terms, it can
  chew up a huge amount of memory (~50K per matched file, IIRC).

After importing several GNU mailing lists' archives yesterday, I now did
some measurements, and it is in the 20s KiB per file, ranging from 26 KiB
for a 9000 files hierarchy to 21 KiB for a 23000 files hierarchy (the
reason for the non-linearity mostly being notmuch's regular resident
size, etc., I assume).

And, of course:

$ find ~/Mail-schwinge.name-thomas/import/GNU/2011-04-03/ -type f | wc -l
276010
$ notmuch search --output=files -- rdirectory:import/GNU/2011-04-03 | grep 
-F import/GNU/2011-04-03 | wc -l
0
$ echo "${PIPESTATUS[@]}"
137 1 0
$ dmesg | grep notmuch
[3797089.224252] notmuch invoked oom-killer: gfp_mask=0x200da, order=0, 
oom_adj=0, oom_score_adj=0
[3797089.224282] notmuch cpuset=/ mems_allowed=0
[3797089.224290] Pid: 586, comm: notmuch Not tainted 3.0.0-1-686-pae #1
[3797089.232081] [  586]  1000   586   310693   257874   0   0  
   0 notmuch
[3797089.232081] Out of memory: Kill process 586 (notmuch) score 697 or 
sacrifice child
[3797089.232081] Killed process 586 (notmuch) total-vm:1242772kB, 
anon-rss:1031492kB, file-rss:4kB

:-) (But this is no problem for me; I don't need to do such
coarse-grained matching.)

 tschwinge: The solution is probably to add folder terms to
  messages (but as one, unsplit term, unlike in cworth's approach)
  and expand on those so that the space is bounded by the number of
  matched folders, rather than files.  That would also make it quite
  easy to do arbitrary glob matching.

(These would now be directory terms.)  This suggestion still stands.
(But I'm not working on it at the moment.)


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 



Austin's custom query parser: folder/directory searching, some numbers

2011-10-27 Thread Thomas Schwinge
Hi!

As I already told on IRC (and which I still have to polish and
publish...), I recently merged Austin's custom query parser into my local
tree, mainly (for now) for its exact folder/directory searching
capabilities.

Austin had published this work several months ago, and Carl in the mean
time had implemented his own folder: searches.  Now, there was a conflict
about which to use; they have different semantics, Carl's being
inadequate for my use case (not rooted, for example).  On IRC, Carl
recently had the most pragmatic solution for how to approach this: if we
can't agree on having either his folder: semantics, or Austin's strict
filename matching -- then just have both of them.  So I now have arranged
for having both Carl's folder: (with it's ``weak'' mail folder
semantics), and also Austin's directory: (with it's ``hard''
directory/filename matching semantics), and on top of the latter
implemented rdirectory: which extends directory: by recursive matching.
This works really nice.


IRC, freenode, #notmuch, 2011-09-30:

amdragon tschwinge: Before you get in too deep I should point out
  that there's a (not unsurmountable) flaw in the folder handling.
  Because it expands to all of the desired dir-entry terms, it can
  chew up a huge amount of memory (~50K per matched file, IIRC).

After importing several GNU mailing lists' archives yesterday, I now did
some measurements, and it is in the 20s KiB per file, ranging from 26 KiB
for a 9000 files hierarchy to 21 KiB for a 23000 files hierarchy (the
reason for the non-linearity mostly being notmuch's regular resident
size, etc., I assume).

And, of course:

$ find ~/Mail-schwinge.name-thomas/import/GNU/2011-04-03/ -type f | wc -l
276010
$ notmuch search --output=files -- rdirectory:import/GNU/2011-04-03 | grep 
-F import/GNU/2011-04-03 | wc -l
0
$ echo ${PIPESTATUS[@]}
137 1 0
$ dmesg | grep notmuch
[3797089.224252] notmuch invoked oom-killer: gfp_mask=0x200da, order=0, 
oom_adj=0, oom_score_adj=0
[3797089.224282] notmuch cpuset=/ mems_allowed=0
[3797089.224290] Pid: 586, comm: notmuch Not tainted 3.0.0-1-686-pae #1
[3797089.232081] [  586]  1000   586   310693   257874   0   0  
   0 notmuch
[3797089.232081] Out of memory: Kill process 586 (notmuch) score 697 or 
sacrifice child
[3797089.232081] Killed process 586 (notmuch) total-vm:1242772kB, 
anon-rss:1031492kB, file-rss:4kB

:-) (But this is no problem for me; I don't need to do such
coarse-grained matching.)

amdragon tschwinge: The solution is probably to add folder terms to
  messages (but as one, unsplit term, unlike in cworth's approach)
  and expand on those so that the space is bounded by the number of
  matched folders, rather than files.  That would also make it quite
  easy to do arbitrary glob matching.

(These would now be directory terms.)  This suggestion still stands.
(But I'm not working on it at the moment.)


Grüße,
 Thomas


pgptbmAORgE44.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH] cli: change argument parsing convention for subcommands

2011-10-22 Thread Thomas Schwinge
Hi!

On Fri, 21 Oct 2011 09:19:17 -0300, david at tethera.net wrote:
> previously we deleted the subcommand name from argv before passing to
> the subcommand. In this version, the deletion is done in the actual
> subcommands. Although this causes some duplication of code, it allows
> us to be more flexible about how we parse command line arguments in
> the subcommand, including possibly using off-the-shelf routines like
> getopt_long that expect the name of the command in argv[0].

Ack.  Like when the C library startup passes control to the main
function, where argv[0] is the invoked executable.

It seems that notmuch.c:notmuch_help_command also needs to be adapted?

notmuch-setup.c:notmuch_setup_command does not need to be adapted (and
hasn't been) for it doesn't look at its argv.  (It should bail out if
there are any arguments passed, but that's for another patch.)


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 



Re: [PATCH] cli: change argument parsing convention for subcommands

2011-10-22 Thread Thomas Schwinge
Hi!

On Fri, 21 Oct 2011 09:19:17 -0300, da...@tethera.net wrote:
 previously we deleted the subcommand name from argv before passing to
 the subcommand. In this version, the deletion is done in the actual
 subcommands. Although this causes some duplication of code, it allows
 us to be more flexible about how we parse command line arguments in
 the subcommand, including possibly using off-the-shelf routines like
 getopt_long that expect the name of the command in argv[0].

Ack.  Like when the C library startup passes control to the main
function, where argv[0] is the invoked executable.

It seems that notmuch.c:notmuch_help_command also needs to be adapted?

notmuch-setup.c:notmuch_setup_command does not need to be adapted (and
hasn't been) for it doesn't look at its argv.  (It should bail out if
there are any arguments passed, but that's for another patch.)


Grüße,
 Thomas


pgpfvAtw3TObR.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH v2] emacs: Turn id:"" elements into buttons for notmuch searches

2011-10-17 Thread Thomas Schwinge
Hi!

On Mon, 17 Oct 2011 14:13:05 +0200, Daniel Schoepe  wrote:
> On Mon, 17 Oct 2011 11:16:45 +0200, Thomas Schwinge  
> wrote:
> > Is it permissible for message IDs to contain double quotes?  If not (and
> > I hope so), might id:\"[^\"]+\" be a better regexp?  (Untested.)  As it
> > appears to me, this would allow proper matching in text like this, too:
> > 
> > Bla bla, see id:"some at thing".  Bla bla.
> 
> My patch already works for this example, and I don't think spaces are
> allowed in message IDs, so it shouldn't make a difference.

Ah, of course, right...  Yet, won't yours -- id:\"[^ ]+\" -- match too
much in this example:

Bla, bla, see id:"some at thing"--"some famous quote" is bla.

How about this one?  id:\(\"?\)[^[:space:]"]+\1

Test:

  id:some at thing "something"
  id:"some at thing"--"something"
  id:"some at thing"--something
  id:some at thing "something else"
  id:"some at thing"--"something else"
  id:"some at thing"--something else
  id:some at thing
  id:"some at thing"


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20111017/93931cb8/attachment-0001.pgp>


[PATCH v2] emacs: Turn id:"" elements into buttons for notmuch searches

2011-10-17 Thread Thomas Schwinge
Hi!

Good idea!


On Wed,  6 Jul 2011 16:18:01 +0200, Daniel Schoepe  wrote:
> +(defun notmuch-show-buttonise-links (start end)
> +  "Buttonise URLs and mail addresses between START and END.
> +
> +This also turns id:\"\"-parts into buttons for
> +a corresponding notmuch search."
> +  (goto-address-fontify-region start end)
> +  (save-excursion
> +(goto-char start)
> +(while (re-search-forward "id:\"[^ ]+\"" end t)

Is it permissible for message IDs to contain double quotes?  If not (and
I hope so), might id:\"[^\"]+\" be a better regexp?  (Untested.)  As it
appears to me, this would allow proper matching in text like this, too:

Bla bla, see id:"some at thing".  Bla bla.


Even if not favorable, should a syntax without double quotes be
supported, too?  Here we'd really have to match on whitespace and line
end, as everything else is too ambiguous.


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 



Re: [PATCH v2] emacs: Turn id:message-id elements into buttons for notmuch searches

2011-10-17 Thread Thomas Schwinge
Hi!

Good idea!


On Wed,  6 Jul 2011 16:18:01 +0200, Daniel Schoepe 
daniel.scho...@googlemail.com wrote:
 +(defun notmuch-show-buttonise-links (start end)
 +  Buttonise URLs and mail addresses between START and END.
 +
 +This also turns id:\message id\-parts into buttons for
 +a corresponding notmuch search.
 +  (goto-address-fontify-region start end)
 +  (save-excursion
 +(goto-char start)
 +(while (re-search-forward id:\[^ ]+\ end t)

Is it permissible for message IDs to contain double quotes?  If not (and
I hope so), might id:\[^\]+\ be a better regexp?  (Untested.)  As it
appears to me, this would allow proper matching in text like this, too:

Bla bla, see id:some@thing.  Bla bla.


Even if not favorable, should a syntax without double quotes be
supported, too?  Here we'd really have to match on whitespace and line
end, as everything else is too ambiguous.


Grüße,
 Thomas


pgpDXb0sg539b.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH v2] emacs: Turn id:message-id elements into buttons for notmuch searches

2011-10-17 Thread Thomas Schwinge
Hi!

On Mon, 17 Oct 2011 14:13:05 +0200, Daniel Schoepe 
daniel.scho...@googlemail.com wrote:
 On Mon, 17 Oct 2011 11:16:45 +0200, Thomas Schwinge tho...@schwinge.name 
 wrote:
  Is it permissible for message IDs to contain double quotes?  If not (and
  I hope so), might id:\[^\]+\ be a better regexp?  (Untested.)  As it
  appears to me, this would allow proper matching in text like this, too:
  
  Bla bla, see id:some@thing.  Bla bla.
 
 My patch already works for this example, and I don't think spaces are
 allowed in message IDs, so it shouldn't make a difference.

Ah, of course, right...  Yet, won't yours -- id:\[^ ]+\ -- match too
much in this example:

Bla, bla, see id:some@thing--some famous quote is bla.

How about this one?  id:\(\?\)[^[:space:]]+\1

Test:

  id:some@thing something
  id:some@thing--something
  id:some@thing--something
  id:some@thing something else
  id:some@thing--something else
  id:some@thing--something else
  id:some@thing
  id:some@thing


Grüße,
 Thomas


pgpdGmuGTnwsh.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[no subject]

2011-10-16 Thread Thomas Schwinge
Hi!

On Mon, 10 Oct 2011 10:49:15 -0300, david at tethera.net wrote:
> OK, here is my proposal to add search terms to notmuch dump.

Having worked in the same area ;-), I felt competent to review this.  And
I definitely do like David's approach.  The patches look good, with the
following comments:

What's missing is adding (roughly) the same text to the notmuch manpage,
``notmuch help dump'', NEWS file.  These should be added to the
respective patches, for enhance functionality and deprecation of output
filename.

> 2b7781d test: all dump-restore tests should be working now
> 7a203d6 notmuch-dump: treat any remaining arguments after the filename as 
> search t

I would suggest to combine these two into one patch: enhance
implementation (7a203d6) and update the tests (2b7781d) is one unit.

> d6715d7 test: add tests for command line arguments to notmuch-dump

Specifically:

On Mon, 10 Oct 2011 10:49:17 -0300, david at tethera.net wrote:
> The plan is to add the possibility of search terms after the file name,
> and the use of -- to stop looking for an output file name.
> ---
>  test/dump-restore |   28 
>  1 files changed, 28 insertions(+), 0 deletions(-)
> 
> diff --git a/test/dump-restore b/test/dump-restore
> index 96c4f19..699337c 100755
> --- a/test/dump-restore
> +++ b/test/dump-restore
> @@ -8,6 +8,34 @@ test_expect_success "Dumping all tags" "generate_message &&
>  notmuch new &&
>  notmuch dump > dump.expected"
>  
> +test_begin_subtest "dump outfile"
> +notmuch dump dump-outfile.actual
> +test_expect_equal_file dump.expected dump-outfile.actual
> +
> +test_begin_subtest "dump outfile --"
> +notmuch dump dump-1-arg-dash.actual
> +test_expect_equal_file dump.expected dump-1-arg-dash.actual
> 
> [...]

I don't understand the purpose of the second test above.  Was this meant
to be ``notmuch dump dump-1-arg-dash.actual --'' (as suggested by the
description), or ``notmuch dump -- > dump-1-arg-dash.actual''?


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 



[python] set rpath in setup.cfg

2011-10-16 Thread Thomas Schwinge
Hi!

On Wed, 12 Oct 2011 23:34:28 -0700, Jameson Graef Rollins  wrote:
> On Thu, 13 Oct 2011 08:12:03 +0200, dtk  wrote:
> > in my experience, it tends to cause awkward side effects that are hard to
> > debug, the main problem being that it overrides all default paths and is
> > hard to target at a single problematic application.
> 
> I think it's fairly straightforward to prepend a library path to the ld
> library path without overriding all defaults with something like this
> (for bash):
> 
> LD_LIBRARY_PATH=/new/ld/path:$LD_LIBRARY_PATH

This is in fact not safe, and may cause subtle issues: if LD_LIBRARY_PATH
is empty initally, it'll evaluate to:

LD_LIBRARY_PATH=/new/ld/path:

Of this one, the last element is empty, and ld.so will thus look into the
current directory for libc.so, for example.  Now you cd /tmp/ and enter
ls...  :-)

Or, as it once bit me, you're on a system with such an LD_LIBRARY_PATH
set (without me knowing about it).  You do glibc development, and wonder
why some commands begin acting strangely when you're in the glibc build
directory...

Thus:

LD_LIBRARY_PATH=/new/ld/path${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 



Re:

2011-10-16 Thread Thomas Schwinge
Hi!

On Mon, 10 Oct 2011 10:49:15 -0300, da...@tethera.net wrote:
 OK, here is my proposal to add search terms to notmuch dump.

Having worked in the same area ;-), I felt competent to review this.  And
I definitely do like David's approach.  The patches look good, with the
following comments:

What's missing is adding (roughly) the same text to the notmuch manpage,
``notmuch help dump'', NEWS file.  These should be added to the
respective patches, for enhance functionality and deprecation of output
filename.

 2b7781d test: all dump-restore tests should be working now
 7a203d6 notmuch-dump: treat any remaining arguments after the filename as 
 search t

I would suggest to combine these two into one patch: enhance
implementation (7a203d6) and update the tests (2b7781d) is one unit.

 d6715d7 test: add tests for command line arguments to notmuch-dump

Specifically:

On Mon, 10 Oct 2011 10:49:17 -0300, da...@tethera.net wrote:
 The plan is to add the possibility of search terms after the file name,
 and the use of -- to stop looking for an output file name.
 ---
  test/dump-restore |   28 
  1 files changed, 28 insertions(+), 0 deletions(-)
 
 diff --git a/test/dump-restore b/test/dump-restore
 index 96c4f19..699337c 100755
 --- a/test/dump-restore
 +++ b/test/dump-restore
 @@ -8,6 +8,34 @@ test_expect_success Dumping all tags generate_message 
  notmuch new 
  notmuch dump  dump.expected
  
 +test_begin_subtest dump outfile
 +notmuch dump dump-outfile.actual
 +test_expect_equal_file dump.expected dump-outfile.actual
 +
 +test_begin_subtest dump outfile --
 +notmuch dump dump-1-arg-dash.actual
 +test_expect_equal_file dump.expected dump-1-arg-dash.actual
 
 [...]

I don't understand the purpose of the second test above.  Was this meant
to be ``notmuch dump dump-1-arg-dash.actual --'' (as suggested by the
description), or ``notmuch dump --  dump-1-arg-dash.actual''?


Grüße,
 Thomas


pgpIGD2Q47gtf.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH] emacs: Modify notmuch-show-get-message-id to return message-id unprefixed with "id:".

2011-10-11 Thread Thomas Schwinge
Hi!

On Sun,  9 Oct 2011 15:35:48 -0700, Jameson Graef Rollins  wrote:
>  (defun notmuch-show-get-message-id ()
>"Return the message id of the current message."
> -  (concat "id:\"" (notmuch-show-get-prop :id) "\""))
> +  (concat "\"" (notmuch-show-get-prop :id) "\""))

Shouldn't the double quotes be removed here, too?  (And be re-added in
the other places where id: is added.)


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 



Re: [PATCH] NEWS: add notes about emacs improvements and reply formating cleanup

2011-10-10 Thread Thomas Schwinge
Hi!

On Sat, 08 Oct 2011 20:55:59 -0700, Jameson Graef Rollins 
jroll...@finestructure.net wrote:
 Ok, I just realized why these messages aren't showing up in reply to the
 messages I intend: it's the id: prefix that's there when I use c i
 to copy the message ID to the kill-ring!  I'm then in-reply-to'ing
 
 id:87ipnzd4ds.fsf@zancas.localnet
 
 instead of just
 
 87ipnzd4ds.fsf@zancas.localnet

Well, yes.  :-)

 Patch to fix that to follow!

Hmm, don't know.  Of course, c f also doesn't stash From:Some One
someone@some.where, but just Some One someone@some.where.  On the
other hand, the id:[Message ID] format has the advantage that it's
directly usable as a notmuch search term.  I'm ambivalent.


Grüße,
 Thomas


pgpMjkbNatWNz.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH] emacs: Modify notmuch-show-get-message-id to return message-id unprefixed with id:.

2011-10-10 Thread Thomas Schwinge
Hi!

On Sun,  9 Oct 2011 15:35:48 -0700, Jameson Graef Rollins 
jroll...@finestructure.net wrote:
  (defun notmuch-show-get-message-id ()
Return the message id of the current message.
 -  (concat id:\ (notmuch-show-get-prop :id) \))
 +  (concat \ (notmuch-show-get-prop :id) \))

Shouldn't the double quotes be removed here, too?  (And be re-added in
the other places where id: is added.)


Grüße,
 Thomas


pgpvG4KkqPOly.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH] NEWS: add notes about emacs improvements and reply formating cleanup

2011-10-09 Thread Thomas Schwinge
Hi!

On Sat, 08 Oct 2011 20:55:59 -0700, Jameson Graef Rollins  wrote:
> Ok, I just realized why these messages aren't showing up in reply to the
> messages I intend: it's the "id:" prefix that's there when I use "c i"
> to copy the message ID to the kill-ring!  I'm then in-reply-to'ing
> 
> id:87ipnzd4ds.fsf at zancas.localnet
> 
> instead of just
> 
> 87ipnzd4ds.fsf at zancas.localnet

Well, yes.  :-)

> Patch to fix that to follow!

Hmm, don't know.  Of course, c f also doesn't stash From:"Some One
", but just Some One .  On the
other hand, the id:"[Message ID]" format has the advantage that it's
directly usable as a notmuch search term.  I'm ambivalent.


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 



[PATCH] emacs: Add callback functions to crypto sigstatus button.

2011-10-07 Thread Thomas Schwinge
Hi!

On Tue, 31 May 2011 10:07:13 -0700, Jameson Graef Rollins  wrote:
> This adds two callback functions to the sigstatus button.  If the sig

First, thanks for this!

> status is "good", then clicking the button displays the output of "gpg
> --list-keys" on the key fingerprint.  If the sigstatus is "bad", then
> clicking the button will retrieve the key from the keyserver, and
> redisplay the current buffer.

Now, what happens on redisplay is that (apparently) all the current state
is lost: the point jumps back to its original position (the beginning of
the buffer, for example), messages/parts view toggles are reset, etc.  I
guess that is just how the refresh function works -- but is it really
what we want, and how difficult would it be to preserve that state?


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 



Re: [PATCH] emacs: Add callback functions to crypto sigstatus button.

2011-10-07 Thread Thomas Schwinge
Hi!

On Tue, 31 May 2011 10:07:13 -0700, Jameson Graef Rollins 
jroll...@finestructure.net wrote:
 This adds two callback functions to the sigstatus button.  If the sig

First, thanks for this!

 status is good, then clicking the button displays the output of gpg
 --list-keys on the key fingerprint.  If the sigstatus is bad, then
 clicking the button will retrieve the key from the keyserver, and
 redisplay the current buffer.

Now, what happens on redisplay is that (apparently) all the current state
is lost: the point jumps back to its original position (the beginning of
the buffer, for example), messages/parts view toggles are reset, etc.  I
guess that is just how the refresh function works -- but is it really
what we want, and how difficult would it be to preserve that state?


Grüße,
 Thomas


pgprHUsMq78kv.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH] patchformatting: Test Suite Enhancements

2011-09-29 Thread Thomas Schwinge
Based on the emails starting at
id:"87liu2kcq6.fsf at servo.factory.finestructure.net".
---

Hi!

I applied the attached patch, based on this thread's discussion.


Gr??e,
 Thomas


---
 patchformatting.mdwn |   10 ++
 1 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/patchformatting.mdwn b/patchformatting.mdwn
index cf5bf81..914371d 100644
--- a/patchformatting.mdwn
+++ b/patchformatting.mdwn
@@ -41,6 +41,16 @@ Eric S. Raymond has written good
 [Software Release Practice 
HOWTO](http://tldp.org/HOWTO/Software-Release-Practice-HOWTO/).
 Check what he has to say about this issue. 

+### Test Suite Enhancements
+
+New features as well as bug fixes should typically come with test suite
+enhancements.  The test suite changes should be done first (tagged as *expected
+to fail*), and the feature implementation or bug fix should come second
+(removing the *expected to fail* tag).  This way, the test suite specifies the
+behavior you're trying to implement, be it a new feature or a bug fix.  By
+defining beforehand exactly what you expect to happen, everyone can confirm
+that your patch achieves what it is meant it to.
+
 ## Prepare patches for e-mail submission

 If you've made just one commit (containing just one bugfix or new feature)
-- 
1.7.6.3



[PATCH, v2] notmuch restore --accumulate

2011-09-29 Thread Thomas Schwinge
From: Thomas Schwinge <tho...@schwinge.name>

The --accumulate switch causes the union of the existing and new tags to be
applied, instead of replacing each message's tags as they are read in from the
dump file.

---

Hi!

Beware that I have not yet used this new functionality in the wild.  ;-)
(But I do plan to do so, soon.)  And, I think that the testsuite
enhancements cover quite a number of real-world scenarios.

This is v2; it addresses Jameson's and Austin's comments.


Gr??e,
 Thomas

---

 NEWS  |9 +
 notmuch-restore.c |   42 +-
 notmuch.1 |   14 ++
 notmuch.c |   10 +++---
 test/dump-restore |   10 --
 5 files changed, 59 insertions(+), 26 deletions(-)

diff --git a/NEWS b/NEWS
index ee84e9a..2a184ef 100644
--- a/NEWS
+++ b/NEWS
@@ -12,6 +12,15 @@ Correct handling of interruptions during "notmuch new"
   detect messages on resume, or leave the database in a state
   temporarily or permanently inconsistent with the mail store.

+New command-line features
+-
+
+Add "notmuch restore --accumulate" option
+
+  The --accumulate switch causes the union of the existing and new tags to be
+  applied, instead of replacing each message's tags as they are read in from
+  the dump file.
+
 Library changes
 ---

diff --git a/notmuch-restore.c b/notmuch-restore.c
index f095f64..5aad60c 100644
--- a/notmuch-restore.c
+++ b/notmuch-restore.c
@@ -31,7 +31,8 @@ notmuch_restore_command (unused (void *ctx), int argc, char 
*argv[])
 size_t line_size;
 ssize_t line_len;
 regex_t regex;
-int rerr;
+notmuch_bool_t accumulate;
+int i, rerr;

 config = notmuch_config_open (ctx, NULL, NULL);
 if (config == NULL)
@@ -44,14 +45,28 @@ notmuch_restore_command (unused (void *ctx), int argc, char 
*argv[])

 synchronize_flags = notmuch_config_get_maildir_synchronize_flags (config);

-if (argc) {
-   input = fopen (argv[0], "r");
-   if (input == NULL) {
-   fprintf (stderr, "Error opening %s for reading: %s\n",
-argv[0], strerror (errno));
-   return 1;
+accumulate = FALSE;
+input = NULL;
+for (i = 0; i < argc; i++) {
+   if (STRNCMP_LITERAL (argv[i], "--accumulate") == 0) {
+   accumulate = TRUE;
+   } else {
+   if (input == NULL) {
+   input = fopen (argv[i], "r");
+   if (input == NULL) {
+   fprintf (stderr, "Error opening %s for reading: %s\n",
+argv[i], strerror (errno));
+   return 1;
+   }
+   } else {
+   fprintf (stderr,
+"Cannot read dump from more than one file: %s\n",
+argv[i]);
+   return 1;
+   }
}
-} else {
+}
+if (input == NULL) {
printf ("No filename given. Reading dump from stdin.\n");
input = stdin;
 }
@@ -94,6 +109,13 @@ notmuch_restore_command (unused (void *ctx), int argc, char 
*argv[])
goto NEXT_LINE;
}

+   /* In order to detect missing messages, this check/optimization is
+* intentionally done *after* first finding the message.  */
+   if (accumulate && (file_tags == NULL || *file_tags == '\0'))
+   {
+   goto NEXT_LINE;
+   }
+
db_tags_str = NULL;
for (db_tags = notmuch_message_get_tags (message);
 notmuch_tags_valid (db_tags);
@@ -115,7 +137,9 @@ notmuch_restore_command (unused (void *ctx), int argc, char 
*argv[])
}

notmuch_message_freeze (message);
-   notmuch_message_remove_all_tags (message);
+
+   if (!accumulate)
+   notmuch_message_remove_all_tags (message);

next = file_tags;
while (next) {
diff --git a/notmuch.1 b/notmuch.1
index 5a8c83d..883371d 100644
--- a/notmuch.1
+++ b/notmuch.1
@@ -454,7 +454,7 @@ section below for details of the supported syntax for 
.
 The
 .BR dump " and " restore
 commands can be used to create a textual dump of email tags for backup
-purposes, and to restore from that dump
+purposes, and to restore from that dump.

 .RS 4
 .TP 4
@@ -462,17 +462,19 @@ purposes, and to restore from that dump

 Creates a plain-text dump of the tags of each message.

-The output is to the given filename, if any, or to stdout.
+The output is written to the given filename, if any, or to stdout.

 These tags are the only data in the notmuch database that can't be
 recreated from the messages themselves.  The output of notmuch dump is
 therefore the only critical thing to backup (and much more friendly to
 incremental backup than the native database files.)
 .TP
-.BR restore " "
+.BR restore " [--accumulate] []"

 Restores the tags from the giv

[PATCH, v2] notmuch restore --accumulate

2011-09-29 Thread Thomas Schwinge
From: Thomas Schwinge <tho...@schwinge.name>

Flesh out what ``notmuch restore --accumulate'' is supposed to do.  Its tests
are currently XFAILed; the functionality will be added in another patch.

Also generally enhance the dump-restore testsuite, and make it more
failure-proof.

---
 test/dump-restore |   77 ++---
 test/test-lib.sh  |3 +-
 2 files changed, 63 insertions(+), 17 deletions(-)

diff --git a/test/dump-restore b/test/dump-restore
index a4de370..d253756 100755
--- a/test/dump-restore
+++ b/test/dump-restore
@@ -4,21 +4,66 @@ test_description="\"notmuch dump\" and \"notmuch restore\""

 add_email_corpus

-test_expect_success "Dumping all tags" "generate_message &&
-notmuch new &&
-notmuch dump dump.expected"
-
-test_begin_subtest "Clearing all tags"
-sed -e "s/(\([^(]*\))$/()/" < dump.expected > clear.expected
-notmuch restore clear.expected
-notmuch dump clear.actual
-test_expect_equal "$(< clear.actual)" "$(< clear.expected)"
-
-test_begin_subtest "Restoring original tags"
-notmuch restore dump.expected
-notmuch dump dump.actual
-test_expect_equal "$(< dump.actual)" "$(< dump.expected)"
-
-test_expect_success "Restore with nothing to do" "notmuch restore 
dump.expected"
+test_expect_success 'Dumping all tags' \
+  'generate_message &&
+  notmuch new &&
+  notmuch dump dump.expected'
+
+# This is rather arbitrary: it matches some of the email corpus' messages, but
+# not all of them.
+search_term=from:worth
+
+test_expect_success 'Dumping all tags to stdout' \
+  'notmuch tag +ABC +DEF -- $search_term &&
+  notmuch dump > dump-ABC_DEF.expected &&
+  ! cmp dump.expected dump-ABC_DEF.expected'
+
+test_expect_success 'Clearing all tags' \
+  'sed -e "s/(\([^(]*\))$/()/" < dump.expected > clear.expected &&
+  notmuch restore clear.expected &&
+  notmuch dump clear.actual &&
+  test_cmp clear.expected clear.actual'
+
+# Missing notmuch restore --accumulate.
+test_subtest_known_broken
+test_expect_success 'Accumulate original tags' \
+  'notmuch tag +ABC +DEF -- $search_term &&
+  notmuch restore --accumulate < dump.expected &&
+  notmuch dump dump.actual &&
+  test_cmp dump-ABC_DEF.expected dump.actual'
+
+test_expect_success 'Restoring original tags' \
+  'notmuch restore dump.expected &&
+  notmuch dump dump.actual &&
+  test_cmp dump.expected dump.actual'
+
+test_expect_success 'Restore with nothing to do' \
+  'notmuch restore < dump.expected &&
+  notmuch dump > dump.actual &&
+  test_cmp dump.expected dump.actual'
+
+# Missing notmuch restore --accumulate.
+test_subtest_known_broken
+test_expect_success 'Restore with nothing to do, II' \
+  'notmuch restore --accumulate dump.expected &&
+  notmuch dump dump.actual &&
+  test_cmp dump.expected dump.actual'
+
+# Missing notmuch restore --accumulate.
+test_subtest_known_broken
+test_expect_success 'Restore with nothing to do, III' \
+  'notmuch restore --accumulate < clear.expected &&
+  notmuch dump dump.actual &&
+  test_cmp dump.expected dump.actual'
+
+# notmuch restore currently only considers the first argument.
+test_subtest_known_broken
+test_expect_success 'Invalid restore invocation' \
+  'test_must_fail notmuch restore dump.expected another_one'
+
+# The follwing test already succeeds due to notmuch restore currently only
+# considering the first argument.
+test_expect_success 'Invalid restore invocation, II' \
+  'test_must_fail notmuch restore --accumulate dump.expected another_one'

 test_done
diff --git a/test/test-lib.sh b/test/test-lib.sh
index f8df6a5..f524ebf 100755
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -462,6 +462,7 @@ test_expect_equal ()
 fi
 }

+# Like test_expect_equal, but takes two filenames.
 test_expect_equal_file ()
 {
exec 1>&6 2>&7  # Restore stdout and stderr
@@ -724,7 +725,7 @@ test_external_without_stderr () {
fi
 }

-# This is not among top-level (test_expect_success | test_expect_failure)
+# This is not among top-level (test_expect_success)
 # but is a prefix that can be used in the test script, like:
 #
 #  test_expect_success 'complain and die' '
-- 
tg: (f63d605..) t/restore-accumulate-test (depends on: master)


[PATCH, v2] notmuch restore --accumulate

2011-09-29 Thread Thomas Schwinge
From: Thomas Schwinge tho...@schwinge.name

Flesh out what ``notmuch restore --accumulate'' is supposed to do.  Its tests
are currently XFAILed; the functionality will be added in another patch.

Also generally enhance the dump-restore testsuite, and make it more
failure-proof.

---
 test/dump-restore |   77 ++---
 test/test-lib.sh  |3 +-
 2 files changed, 63 insertions(+), 17 deletions(-)

diff --git a/test/dump-restore b/test/dump-restore
index a4de370..d253756 100755
--- a/test/dump-restore
+++ b/test/dump-restore
@@ -4,21 +4,66 @@ test_description=\notmuch dump\ and \notmuch restore\
 
 add_email_corpus
 
-test_expect_success Dumping all tags generate_message 
-notmuch new 
-notmuch dump dump.expected
-
-test_begin_subtest Clearing all tags
-sed -e s/(\([^(]*\))$/()/  dump.expected  clear.expected
-notmuch restore clear.expected
-notmuch dump clear.actual
-test_expect_equal $( clear.actual) $( clear.expected)
-
-test_begin_subtest Restoring original tags
-notmuch restore dump.expected
-notmuch dump dump.actual
-test_expect_equal $( dump.actual) $( dump.expected)
-
-test_expect_success Restore with nothing to do notmuch restore 
dump.expected
+test_expect_success 'Dumping all tags' \
+  'generate_message 
+  notmuch new 
+  notmuch dump dump.expected'
+
+# This is rather arbitrary: it matches some of the email corpus' messages, but
+# not all of them.
+search_term=from:worth
+
+test_expect_success 'Dumping all tags to stdout' \
+  'notmuch tag +ABC +DEF -- $search_term 
+  notmuch dump  dump-ABC_DEF.expected 
+  ! cmp dump.expected dump-ABC_DEF.expected'
+
+test_expect_success 'Clearing all tags' \
+  'sed -e s/(\([^(]*\))$/()/  dump.expected  clear.expected 
+  notmuch restore clear.expected 
+  notmuch dump clear.actual 
+  test_cmp clear.expected clear.actual'
+
+# Missing notmuch restore --accumulate.
+test_subtest_known_broken
+test_expect_success 'Accumulate original tags' \
+  'notmuch tag +ABC +DEF -- $search_term 
+  notmuch restore --accumulate  dump.expected 
+  notmuch dump dump.actual 
+  test_cmp dump-ABC_DEF.expected dump.actual'
+
+test_expect_success 'Restoring original tags' \
+  'notmuch restore dump.expected 
+  notmuch dump dump.actual 
+  test_cmp dump.expected dump.actual'
+
+test_expect_success 'Restore with nothing to do' \
+  'notmuch restore  dump.expected 
+  notmuch dump  dump.actual 
+  test_cmp dump.expected dump.actual'
+
+# Missing notmuch restore --accumulate.
+test_subtest_known_broken
+test_expect_success 'Restore with nothing to do, II' \
+  'notmuch restore --accumulate dump.expected 
+  notmuch dump dump.actual 
+  test_cmp dump.expected dump.actual'
+
+# Missing notmuch restore --accumulate.
+test_subtest_known_broken
+test_expect_success 'Restore with nothing to do, III' \
+  'notmuch restore --accumulate  clear.expected 
+  notmuch dump dump.actual 
+  test_cmp dump.expected dump.actual'
+
+# notmuch restore currently only considers the first argument.
+test_subtest_known_broken
+test_expect_success 'Invalid restore invocation' \
+  'test_must_fail notmuch restore dump.expected another_one'
+
+# The follwing test already succeeds due to notmuch restore currently only
+# considering the first argument.
+test_expect_success 'Invalid restore invocation, II' \
+  'test_must_fail notmuch restore --accumulate dump.expected another_one'
 
 test_done
diff --git a/test/test-lib.sh b/test/test-lib.sh
index f8df6a5..f524ebf 100755
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -462,6 +462,7 @@ test_expect_equal ()
 fi
 }
 
+# Like test_expect_equal, but takes two filenames.
 test_expect_equal_file ()
 {
exec 16 27  # Restore stdout and stderr
@@ -724,7 +725,7 @@ test_external_without_stderr () {
fi
 }
 
-# This is not among top-level (test_expect_success | test_expect_failure)
+# This is not among top-level (test_expect_success)
 # but is a prefix that can be used in the test script, like:
 #
 #  test_expect_success 'complain and die' '
-- 
tg: (f63d605..) t/restore-accumulate-test (depends on: master)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH, v2] notmuch restore --accumulate

2011-09-29 Thread Thomas Schwinge
From: Thomas Schwinge tho...@schwinge.name

The --accumulate switch causes the union of the existing and new tags to be
applied, instead of replacing each message's tags as they are read in from the
dump file.

---

Hi!

Beware that I have not yet used this new functionality in the wild.  ;-)
(But I do plan to do so, soon.)  And, I think that the testsuite
enhancements cover quite a number of real-world scenarios.

This is v2; it addresses Jameson's and Austin's comments.


Grüße,
 Thomas

---

 NEWS  |9 +
 notmuch-restore.c |   42 +-
 notmuch.1 |   14 ++
 notmuch.c |   10 +++---
 test/dump-restore |   10 --
 5 files changed, 59 insertions(+), 26 deletions(-)

diff --git a/NEWS b/NEWS
index ee84e9a..2a184ef 100644
--- a/NEWS
+++ b/NEWS
@@ -12,6 +12,15 @@ Correct handling of interruptions during notmuch new
   detect messages on resume, or leave the database in a state
   temporarily or permanently inconsistent with the mail store.
 
+New command-line features
+-
+
+Add notmuch restore --accumulate option
+
+  The --accumulate switch causes the union of the existing and new tags to be
+  applied, instead of replacing each message's tags as they are read in from
+  the dump file.
+
 Library changes
 ---
 
diff --git a/notmuch-restore.c b/notmuch-restore.c
index f095f64..5aad60c 100644
--- a/notmuch-restore.c
+++ b/notmuch-restore.c
@@ -31,7 +31,8 @@ notmuch_restore_command (unused (void *ctx), int argc, char 
*argv[])
 size_t line_size;
 ssize_t line_len;
 regex_t regex;
-int rerr;
+notmuch_bool_t accumulate;
+int i, rerr;
 
 config = notmuch_config_open (ctx, NULL, NULL);
 if (config == NULL)
@@ -44,14 +45,28 @@ notmuch_restore_command (unused (void *ctx), int argc, char 
*argv[])
 
 synchronize_flags = notmuch_config_get_maildir_synchronize_flags (config);
 
-if (argc) {
-   input = fopen (argv[0], r);
-   if (input == NULL) {
-   fprintf (stderr, Error opening %s for reading: %s\n,
-argv[0], strerror (errno));
-   return 1;
+accumulate = FALSE;
+input = NULL;
+for (i = 0; i  argc; i++) {
+   if (STRNCMP_LITERAL (argv[i], --accumulate) == 0) {
+   accumulate = TRUE;
+   } else {
+   if (input == NULL) {
+   input = fopen (argv[i], r);
+   if (input == NULL) {
+   fprintf (stderr, Error opening %s for reading: %s\n,
+argv[i], strerror (errno));
+   return 1;
+   }
+   } else {
+   fprintf (stderr,
+Cannot read dump from more than one file: %s\n,
+argv[i]);
+   return 1;
+   }
}
-} else {
+}
+if (input == NULL) {
printf (No filename given. Reading dump from stdin.\n);
input = stdin;
 }
@@ -94,6 +109,13 @@ notmuch_restore_command (unused (void *ctx), int argc, char 
*argv[])
goto NEXT_LINE;
}
 
+   /* In order to detect missing messages, this check/optimization is
+* intentionally done *after* first finding the message.  */
+   if (accumulate  (file_tags == NULL || *file_tags == '\0'))
+   {
+   goto NEXT_LINE;
+   }
+
db_tags_str = NULL;
for (db_tags = notmuch_message_get_tags (message);
 notmuch_tags_valid (db_tags);
@@ -115,7 +137,9 @@ notmuch_restore_command (unused (void *ctx), int argc, char 
*argv[])
}
 
notmuch_message_freeze (message);
-   notmuch_message_remove_all_tags (message);
+
+   if (!accumulate)
+   notmuch_message_remove_all_tags (message);
 
next = file_tags;
while (next) {
diff --git a/notmuch.1 b/notmuch.1
index 5a8c83d..883371d 100644
--- a/notmuch.1
+++ b/notmuch.1
@@ -454,7 +454,7 @@ section below for details of the supported syntax for 
search-terms.
 The
 .BR dump  and  restore
 commands can be used to create a textual dump of email tags for backup
-purposes, and to restore from that dump
+purposes, and to restore from that dump.
 
 .RS 4
 .TP 4
@@ -462,17 +462,19 @@ purposes, and to restore from that dump
 
 Creates a plain-text dump of the tags of each message.
 
-The output is to the given filename, if any, or to stdout.
+The output is written to the given filename, if any, or to stdout.
 
 These tags are the only data in the notmuch database that can't be
 recreated from the messages themselves.  The output of notmuch dump is
 therefore the only critical thing to backup (and much more friendly to
 incremental backup than the native database files.)
 .TP
-.BR restore  filename
+.BR restore  [--accumulate] [filename]
 
 Restores the tags from the given file (see
-.BR notmuch dump .
+.BR notmuch dump ).
+
+The input is read from the given filename, if any, or from stdin.
 
 Note

[PATCH] patchformatting: Test Suite Enhancements

2011-09-29 Thread Thomas Schwinge
Based on the emails starting at
id:87liu2kcq6@servo.factory.finestructure.net.
---

Hi!

I applied the attached patch, based on this thread's discussion.


Grüße,
 Thomas


---
 patchformatting.mdwn |   10 ++
 1 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/patchformatting.mdwn b/patchformatting.mdwn
index cf5bf81..914371d 100644
--- a/patchformatting.mdwn
+++ b/patchformatting.mdwn
@@ -41,6 +41,16 @@ Eric S. Raymond has written good
 [Software Release Practice 
HOWTO](http://tldp.org/HOWTO/Software-Release-Practice-HOWTO/).
 Check what he has to say about this issue. 
 
+### Test Suite Enhancements
+
+New features as well as bug fixes should typically come with test suite
+enhancements.  The test suite changes should be done first (tagged as *expected
+to fail*), and the feature implementation or bug fix should come second
+(removing the *expected to fail* tag).  This way, the test suite specifies the
+behavior you're trying to implement, be it a new feature or a bug fix.  By
+defining beforehand exactly what you expect to happen, everyone can confirm
+that your patch achieves what it is meant it to.
+
 ## Prepare patches for e-mail submission
 
 If you've made just one commit (containing just one bugfix or new feature)
-- 
1.7.6.3

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH] Repeatability when copying a whole directory into a new one.

2011-09-29 Thread Thomas Schwinge
This new test currently fails -- but it shouldn't.
---

Hi!

I found this while manually copying directories and running notmuch new.

Am I just too sleepy at this time, or is it another DB vs. directory
mtime issue?

 BROKEN Repeatability when copying a whole directory into a new one
--- new.18.expected 2011-09-29 23:23:39.0 +
+++ new.18.output   2011-09-29 23:23:39.0 +
@@ -1,2 +1 @@
-Processed 51 total files in almost no time.
 No new mail.


Grüße,
 Thomas

---
 test/new |   21 +
 1 files changed, 21 insertions(+), 0 deletions(-)

diff --git a/test/new b/test/new
index 49f390d..0afb04c 100755
--- a/test/new
+++ b/test/new
@@ -153,4 +153,25 @@ rm -rf ${MAIL_DIR}/two
 output=$(NOTMUCH_NEW)
 test_expect_equal $output No new mail. Removed 3 messages.
 
+
+test_begin_subtest 'Repeatability when copying a whole directory into a new 
one'
+
+add_email_corpus
+
+mkdir $MAIL_DIR/2nd
+cp -a $MAIL_DIR/cur $MAIL_DIR/2nd/
+output1=$(notmuch new)
+
+rm -rf $MAIL_DIR/2nd
+notmuch new  /dev/null
+
+# This was quite enjoyable.  Let's do it again.
+mkdir $MAIL_DIR/2nd
+cp -a $MAIL_DIR/cur $MAIL_DIR/2nd/
+output2=$(notmuch new)
+
+test_subtest_known_broken
+test_expect_equal $output2 $output1
+
+
 test_done
-- 
1.7.5.4

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH] notmuch restore --accumulate

2011-09-09 Thread Thomas Schwinge
Hi!

On Fri, 9 Sep 2011 12:13:06 -0400, Austin Clements  wrote:
> The idea behind sending the test first is that people can see that it fails
> and that the subsequent patch indeed fixes it.  What I find works well is to
> submit the test case with the test marked as broken and then the main patch,
> including the change to un-mark it as broken.

Ah, that's indeed a good approach for bug fixes (and it also preserves
git bisect compatibility), but still: why separate patches for new
functionality?  (I'm not trying to be a pain here, but would like to
understand your rationale behind this.)


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 



[PATCH] notmuch restore --accumulate

2011-09-09 Thread Thomas Schwinge
Hi!

On Fri, 9 Sep 2011 11:06:34 +0200, Louis Rilling  wrote:
> On 05/09/11 12:31 -0700, Jameson Graef Rollins wrote:
> > Also, we generally prefer to have modifications to the test suite in
> > separate patches that precede the patches that add the features/fix the
> > bugs.
> > 
> 
> For new features, this does not look like 'git bisect'-safe.

Exactly my thoughts.  I can perhaps see the usefulness (for first
separately committing the testcase) for bugfixes, but why for new
features?

> I would say that
> the testsuite patch should follow the new feature patch, don't you?

I would keep them together; why separate them?


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 



Re: [PATCH] notmuch restore --accumulate

2011-09-09 Thread Thomas Schwinge
Hi!

On Fri, 9 Sep 2011 11:06:34 +0200, Louis Rilling l.rill...@av7.net wrote:
 On 05/09/11 12:31 -0700, Jameson Graef Rollins wrote:
  Also, we generally prefer to have modifications to the test suite in
  separate patches that precede the patches that add the features/fix the
  bugs.
  
 
 For new features, this does not look like 'git bisect'-safe.

Exactly my thoughts.  I can perhaps see the usefulness (for first
separately committing the testcase) for bugfixes, but why for new
features?

 I would say that
 the testsuite patch should follow the new feature patch, don't you?

I would keep them together; why separate them?


Grüße,
 Thomas


pgp7NEw98lD7v.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH] notmuch restore --accumulate

2011-09-05 Thread Thomas Schwinge
From: Thomas Schwinge <tho...@schwinge.name>

Also enhance the dump-restore testsuite, and make it generally more
failure-proof.

Signed-off-by: Thomas Schwinge 

---

Hi!

Beware that I have not yet used this new functionality in the wild.  ;-)
(But I do plan to do so, soon.)  And, I think that the testsuite
enhancements cover quite a number of real-world scenarios.


Gr??e,
 Thomas

---

 NEWS  |   13 ++
 notmuch-restore.c |   42 ++---
 notmuch.1 |   14 ---
 notmuch.c |   10 +--
 test/dump-restore |   67 
 test/test-lib.sh  |1 +
 6 files changed, 115 insertions(+), 32 deletions(-)

diff --git a/NEWS b/NEWS
index f715142..d2a788f 100644
--- a/NEWS
+++ b/NEWS
@@ -1,3 +1,16 @@
+Notmuch TODO (TODO)
+===
+
+New command-line features
+-
+
+Add "notmuch restore --accumulate" option
+
+  The --accumulate switch causes the union of the existing and new tags to be
+  applied, instead of replacing each message's tags as they are read in from
+  the dump file.
+
+
 Notmuch 0.7 (2011-08-01)
 

diff --git a/notmuch-restore.c b/notmuch-restore.c
index f095f64..5aad60c 100644
--- a/notmuch-restore.c
+++ b/notmuch-restore.c
@@ -31,7 +31,8 @@ notmuch_restore_command (unused (void *ctx), int argc, char 
*argv[])
 size_t line_size;
 ssize_t line_len;
 regex_t regex;
-int rerr;
+notmuch_bool_t accumulate;
+int i, rerr;

 config = notmuch_config_open (ctx, NULL, NULL);
 if (config == NULL)
@@ -44,14 +45,28 @@ notmuch_restore_command (unused (void *ctx), int argc, char 
*argv[])

 synchronize_flags = notmuch_config_get_maildir_synchronize_flags (config);

-if (argc) {
-   input = fopen (argv[0], "r");
-   if (input == NULL) {
-   fprintf (stderr, "Error opening %s for reading: %s\n",
-argv[0], strerror (errno));
-   return 1;
+accumulate = FALSE;
+input = NULL;
+for (i = 0; i < argc; i++) {
+   if (STRNCMP_LITERAL (argv[i], "--accumulate") == 0) {
+   accumulate = TRUE;
+   } else {
+   if (input == NULL) {
+   input = fopen (argv[i], "r");
+   if (input == NULL) {
+   fprintf (stderr, "Error opening %s for reading: %s\n",
+argv[i], strerror (errno));
+   return 1;
+   }
+   } else {
+   fprintf (stderr,
+"Cannot read dump from more than one file: %s\n",
+argv[i]);
+   return 1;
+   }
}
-} else {
+}
+if (input == NULL) {
printf ("No filename given. Reading dump from stdin.\n");
input = stdin;
 }
@@ -94,6 +109,13 @@ notmuch_restore_command (unused (void *ctx), int argc, char 
*argv[])
goto NEXT_LINE;
}

+   /* In order to detect missing messages, this check/optimization is
+* intentionally done *after* first finding the message.  */
+   if (accumulate && (file_tags == NULL || *file_tags == '\0'))
+   {
+   goto NEXT_LINE;
+   }
+
db_tags_str = NULL;
for (db_tags = notmuch_message_get_tags (message);
 notmuch_tags_valid (db_tags);
@@ -115,7 +137,9 @@ notmuch_restore_command (unused (void *ctx), int argc, char 
*argv[])
}

notmuch_message_freeze (message);
-   notmuch_message_remove_all_tags (message);
+
+   if (!accumulate)
+   notmuch_message_remove_all_tags (message);

next = file_tags;
while (next) {
diff --git a/notmuch.1 b/notmuch.1
index 5a8c83d..883371d 100644
--- a/notmuch.1
+++ b/notmuch.1
@@ -454,7 +454,7 @@ section below for details of the supported syntax for 
.
 The
 .BR dump " and " restore
 commands can be used to create a textual dump of email tags for backup
-purposes, and to restore from that dump
+purposes, and to restore from that dump.

 .RS 4
 .TP 4
@@ -462,17 +462,19 @@ purposes, and to restore from that dump

 Creates a plain-text dump of the tags of each message.

-The output is to the given filename, if any, or to stdout.
+The output is written to the given filename, if any, or to stdout.

 These tags are the only data in the notmuch database that can't be
 recreated from the messages themselves.  The output of notmuch dump is
 therefore the only critical thing to backup (and much more friendly to
 incremental backup than the native database files.)
 .TP
-.BR restore " "
+.BR restore " [--accumulate] []"

 Restores the tags from the given file (see
-.BR "notmuch dump" "."
+.BR "notmuch dump" ")."
+
+The input is read from the given filename, if any, or from stdin.

 Note: The dump f

Build system

2011-02-08 Thread Thomas Schwinge
Hallo!

On Sun, 30 Jan 2011 21:14:39 +1000, Carl Worth  wrote:
> On Sun, 30 Jan 2011 11:28:08 +0100, Thomas Schwinge  
> wrote:
> > Still, my point holds that (unless someone is willing to spend time on
> > this, of course) we shouldn't try to replicate the Autotools, but instead
> > keep our system as simple as it currently is, and thus just have it fail
> > if configured outside of the source tree.
> 
> Oh, I agree that if we don't support this then we should give the user a
> nice error message. But I think it will actually be very easy to add
> support for this. (And at this point, I think the notmuch build system
> is something that other projects could emulate if they want. I don't
> think it's too crazt).

Is the testsuite also easy to convert to VPATH style builds?  (I don't
know, but would expect some difficulties.)


And, another thing I just noticed: I had the source tree configured with
--prefix=[something].  Now I updated the sources, re-ran make, and saw
this:

$ make

Note: Calling ./configure with no command-line arguments. This is often 
fine,
  but if you want to specify any arguments (such as an alternate prefix
  into which to install), call ./configure explicitly and then make 
again.
  See "./configure --help" for more details.

./configure
Welcome to Notmuch, a system for indexing, searching and tagging your email.
[...]

My --prefix=[something] is gone.  (At least the build system warns about
this.)  But it's another issue that other build systems already have
solved.  (And, I probably wouldn't have expected that to do the right
thing if the configure script / build system wouldn't try to be like GNU
Autoconf.)


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110208/1484aa33/attachment.pgp>


Re: Build system

2011-02-08 Thread Thomas Schwinge
Hallo!

On Sun, 30 Jan 2011 21:14:39 +1000, Carl Worth cwo...@cworth.org wrote:
 On Sun, 30 Jan 2011 11:28:08 +0100, Thomas Schwinge tho...@schwinge.name 
 wrote:
  Still, my point holds that (unless someone is willing to spend time on
  this, of course) we shouldn't try to replicate the Autotools, but instead
  keep our system as simple as it currently is, and thus just have it fail
  if configured outside of the source tree.
 
 Oh, I agree that if we don't support this then we should give the user a
 nice error message. But I think it will actually be very easy to add
 support for this. (And at this point, I think the notmuch build system
 is something that other projects could emulate if they want. I don't
 think it's too crazt).

Is the testsuite also easy to convert to VPATH style builds?  (I don't
know, but would expect some difficulties.)


And, another thing I just noticed: I had the source tree configured with
--prefix=[something].  Now I updated the sources, re-ran make, and saw
this:

$ make

Note: Calling ./configure with no command-line arguments. This is often 
fine,
  but if you want to specify any arguments (such as an alternate prefix
  into which to install), call ./configure explicitly and then make 
again.
  See ./configure --help for more details.

./configure
Welcome to Notmuch, a system for indexing, searching and tagging your email.
[...]

My --prefix=[something] is gone.  (At least the build system warns about
this.)  But it's another issue that other build systems already have
solved.  (And, I probably wouldn't have expected that to do the right
thing if the configure script / build system wouldn't try to be like GNU
Autoconf.)


Grüße,
 Thomas


pgpBCNqZxy7LJ.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 2/2] notmuch.el:notmuch-search-process-filter: Rewritten. Cope with incomplete lines.

2011-02-03 Thread Thomas Schwinge
Hallo!

On Thu, 3 Feb 2011 12:06:20 -0500, Austin Clements  wrote:
> Is there a reason you keep the remaining data in a string instead of
> taking the more idiomatic elisp approach of leaving it in the process
> buffer?  In fact, the code would probably be simpler if you
> immediately appended the string to the process buffer like a normal
> process-filter and then peeled things away using buffer-oriented
> regexp functions like looking-at.  Elisp is a lot better at
> manipulating buffers than it is at manipulating strings.

Ha, I hear you -- this is what I meant to do originally.  But then, the
save-in-string approach (even though I always considered keeping state in
the string a bit ugly) seemed more simple to me.  As I said: writing
elisp code is not my primary profession...  :-) (Perhaps I should buy a
book about it, or something.)  Now that you confirmed my original idea,
I'll see about re-writing the code accordingly, so thanks for the input!


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 



[PATCH 2/2] notmuch.el:notmuch-search-process-filter: Rewritten. Cope with incomplete lines.

2011-02-03 Thread Thomas Schwinge
This issue has been lying in ambush as of 2009-11-24's commit
93af7b574598637c2766dd1f8ef343962c9a8efb.

Signed-off-by: Thomas Schwinge 
---
 emacs/notmuch.el |   70 +++--
 1 files changed, 41 insertions(+), 29 deletions(-)

diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index 3d82f0d..35ccee6 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -641,9 +641,6 @@ non-authors is found, assume that all of the authors match."
 (propertize authors 'face 'notmuch-search-matching-authors)))

 (defun notmuch-search-insert-authors (format-string authors)
-  ;; Save the match data to avoid interfering with
-  ;; `notmuch-search-process-filter'.
-  (save-match-data
 (let* ((formatted-authors (format format-string authors))
   (formatted-sample (format format-string ""))
   (visible-string formatted-authors)
@@ -709,7 +706,7 @@ non-authors is found, assume that all of the authors match."
  (setq overlay (make-overlay start (point)))
  (overlay-put overlay 'invisible invis-spec)
  (overlay-put overlay 'isearch-open-invisible 
#'notmuch-search-isearch-authors-show)))
-  (insert padding
+  (insert padding)))

 (defun notmuch-search-insert-field (field date count authors subject tags)
   (cond
@@ -736,6 +733,10 @@ non-authors is found, assume that all of the authors 
match."
  do (notmuch-search-insert-field field date count authors subject 
tags)))
   (insert "\n"))

+(defvar notmuch-search-process-filter-data nil
+  "Data that has not yet been processed.")
+(make-variable-buffer-local 'notmuch-search-process-filter-data)
+
 (defun notmuch-search-process-filter (proc string)
   "Process and filter the output of \"notmuch search\""
   (let ((buffer (process-buffer proc))
@@ -743,31 +744,41 @@ non-authors is found, assume that all of the authors 
match."
 (if (buffer-live-p buffer)
(with-current-buffer buffer
  (save-excursion
-   (let ((line 0)
- (more t)
- (inhibit-read-only t))
- (while more
-   (if (string-match "^\\(thread:[0-9A-Fa-f]*\\) \\([^][]*\\) 
\\(\\[[0-9/]*\\]\\) \\([^;]*\\); \\(.*\\) (\\([^()]*\\))$" string line)
-   (let* ((thread-id (match-string 1 string))
-  (date (match-string 2 string))
-  (count (match-string 3 string))
-  (authors (match-string 4 string))
-  (subject (match-string 5 string))
-  (tags (match-string 6 string))
-  (tag-list (if tags (save-match-data (split-string 
tags)
- (goto-char (point-max))
- (let ((beg (point-marker)))
-   (notmuch-search-show-result date count authors subject 
tags)
-   (notmuch-search-color-line beg (point-marker) tag-list)
-   (put-text-property beg (point-marker) 
'notmuch-search-thread-id thread-id)
-   (put-text-property beg (point-marker) 
'notmuch-search-authors authors)
-   (put-text-property beg (point-marker) 
'notmuch-search-subject subject)
-   (if (string= thread-id notmuch-search-target-thread)
-   (progn
- (set 'found-target beg)
- (set 'notmuch-search-target-thread "found"
- (set 'line (match-end 0)))
- (set 'more nil)
+   (let ((inhibit-read-only t)
+ ;; We may have a partial line saved from the last iteration.
+ (string (concat notmuch-search-process-filter-data string))
+ (start 0))
+ (goto-char (point-max))
+ ;; Split `string' into lines.
+ (while (string-match "\n" string start)
+   (let ((line (substring string start (match-beginning 0
+ ;; Save the beginning of the next line already here, so that
+ ;; we can mangle the match data later on.
+ (setq start (match-end 0))
+ (if (string-match
+  "^\\(thread:[0-9A-Fa-f]*\\) \\([^][]*\\) 
\\(\\[[0-9/]*\\]\\) \\([^;]*\\); \\(.*\\) (\\([^()]*\\))$"
+  line)
+ (let* ((thread-id (match-string 1 line))
+(date (match-string 2 line))
+(count (match-string 3 line))
+(authors (match-string 4 line))
+(subject (match-string 5 line))
+(tags (match-string 6 line))
+(tag-list (if tags (split-string tags
+   (let

[PATCH 1/2] New test: Emacs' forgetfulness.

2011-02-03 Thread Thomas Schwinge
Signed-off-by: Thomas Schwinge 
---
 test/emacs-forgetfulness |   38 ++
 test/notmuch-test|1 +
 2 files changed, 39 insertions(+), 0 deletions(-)
 create mode 100755 test/emacs-forgetfulness

diff --git a/test/emacs-forgetfulness b/test/emacs-forgetfulness
new file mode 100755
index 000..e17b26f
--- /dev/null
+++ b/test/emacs-forgetfulness
@@ -0,0 +1,38 @@
+#!/bin/bash
+
+test_description=Emacs\'\ forgetfulness
+
+. test-lib.sh
+
+# RFC822 imposes a 998 character limit per line.
+x=0123456789 # 10
+x=$x$x$x$x$x$x$x$x$x$x # 100
+x=$x$x$x$x$x$x$x$x$x # 900
+
+# If setting this ``too high'' (TODO: yet to be determined), Emacs will crash
+# with a segmentation fault.
+n=20
+for i in $(seq 1 $n); do
+  # Roughly 2 KiB per message.  That is, we need two messages in order to
+  # exceed the typical size of the pipe buffer (4 KiB on commodity systems).
+  generate_message [subject]=$i-$x [from]=$i-$x at x.x
+done
+# With 20 messages ?? 2 KiB, we have about 10 full pipe buffers, which should 
be
+# enough to trigger the erroneous behavior.
+
+notmuch new > /dev/null
+
+test_begin_subtest 'Search for all messages'
+output=$(exec 2>&1; \
+ diff -wu \
+   <(notmuch search \* \
+   | sed \
+   -e 's%^thread:[0-9a-f]*\ %%' \
+   -e 's%;%%'; \
+ echo 'End of search results.'; \
+ echo) \
+   <(test_emacs 2>&1 \
+   '(notmuch-search "*") (notmuch-test-wait) (message 
(buffer-string))'))
+test_expect_equal "$output" ''
+
+test_done
diff --git a/test/notmuch-test b/test/notmuch-test
index 9d77c0f..2f11eac 100755
--- a/test/notmuch-test
+++ b/test/notmuch-test
@@ -36,6 +36,7 @@ TESTS="
   encoding
   emacs
   maildir-sync
+  emacs-forgetfulness
 "

 # Clean up any results from a previous run
-- 
1.7.1



[BUG] Emacs UI dropping every 25th line, roughly

2011-02-03 Thread Thomas Schwinge
Hallo!

Here is the problem, as suspected.

On Wed, 02 Feb 2011 17:12:16 +0100, I wrote:
> At this / in the middle of this point, the 4 KiB size is hit.  After
> accumulating some more (notice the 3.6 seconds delay), Emacs read()s the
> first chunk (line breaks inserted for clarity):
> 
> 16:26:57.928798 read(8, "[first two dozen results]"
> "thread:000006c7   2009-12-16 [2/2] 
> Thomas Schwinge; [subject] ([flags])\n"
> "t", 4096) = 4095
> 
> The last result line is obviously incomplete, only the `t' so far.

notmuch.el:notmuch-search-process-filter will be called for processing
these lines.  It'll do fine up to the single `t' -- which simply isn't a
conforming line and thus will be dropped (which is questionable behavior
anyway, I would think?).

> There is more to be read, so Emacs quickly goes on with the next chunk:
> 
> 16:26:57.966247 read(8, "hread:0ac4   2009-12-16 [1/1] jsm28 
> at sourceware.org; [subject] ([flags])\n"
> "[more results]", 4096) = 4095

Same thing; this time ``hread:[...]'' isn't a confirming line, and will
be dropped.  After that, regular processing continues.


This problem has been around as of the beginning of the incremental /
asynchronous search results changes, as documented in
id:87aayatnw4.fsf at yoom.home.cworth.org on 2009-11-25; 2009-11-24's commit
93af7b574598637c2766dd1f8ef343962c9a8efb.


Emacs LISP is not my speciality, neither is any other LISP dialect, so
someone please carefully review this.


Gr??e (und gute Nacht...),
 Thomas



[PATCH] Shell programming: directories are ``executable'' (that is, searchable), too.

2011-02-03 Thread Thomas Schwinge
Signed-off-by: Thomas Schwinge tho...@schwinge.name
---
 test/test-lib.sh |5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/test/test-lib.sh b/test/test-lib.sh
index f536172..3471ead 100755
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -815,13 +815,14 @@ EOF
 }
 
 
+# Locate the directory containing the `notmuch' executable we are to use.
 find_notmuch_path ()
 {
 dir=$1
 
 while [ -n $dir ]; do
bin=$dir/notmuch
-   if [ -x $bin ]; then
+   if [ -f $bin -a -x $bin ]; then
echo $dir
return
fi
@@ -858,7 +859,7 @@ then
 
make_valgrind_symlink () {
# handle only executables
-   test -x $1 || return
+   test -f $1 -a -x $1 || return
 
base=$(basename $1)
symlink_target=$TEST_DIRECTORY/../$base
-- 
1.7.1

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 2/2] notmuch.el:notmuch-search-process-filter: Rewritten. Cope with incomplete lines.

2011-02-03 Thread Thomas Schwinge
This issue has been lying in ambush as of 2009-11-24's commit
93af7b574598637c2766dd1f8ef343962c9a8efb.

Signed-off-by: Thomas Schwinge tho...@schwinge.name
---
 emacs/notmuch.el |   70 +++--
 1 files changed, 41 insertions(+), 29 deletions(-)

diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index 3d82f0d..35ccee6 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -641,9 +641,6 @@ non-authors is found, assume that all of the authors match.
 (propertize authors 'face 'notmuch-search-matching-authors)))
 
 (defun notmuch-search-insert-authors (format-string authors)
-  ;; Save the match data to avoid interfering with
-  ;; `notmuch-search-process-filter'.
-  (save-match-data
 (let* ((formatted-authors (format format-string authors))
   (formatted-sample (format format-string ))
   (visible-string formatted-authors)
@@ -709,7 +706,7 @@ non-authors is found, assume that all of the authors match.
  (setq overlay (make-overlay start (point)))
  (overlay-put overlay 'invisible invis-spec)
  (overlay-put overlay 'isearch-open-invisible 
#'notmuch-search-isearch-authors-show)))
-  (insert padding
+  (insert padding)))
 
 (defun notmuch-search-insert-field (field date count authors subject tags)
   (cond
@@ -736,6 +733,10 @@ non-authors is found, assume that all of the authors 
match.
  do (notmuch-search-insert-field field date count authors subject 
tags)))
   (insert \n))
 
+(defvar notmuch-search-process-filter-data nil
+  Data that has not yet been processed.)
+(make-variable-buffer-local 'notmuch-search-process-filter-data)
+
 (defun notmuch-search-process-filter (proc string)
   Process and filter the output of \notmuch search\
   (let ((buffer (process-buffer proc))
@@ -743,31 +744,41 @@ non-authors is found, assume that all of the authors 
match.
 (if (buffer-live-p buffer)
(with-current-buffer buffer
  (save-excursion
-   (let ((line 0)
- (more t)
- (inhibit-read-only t))
- (while more
-   (if (string-match ^\\(thread:[0-9A-Fa-f]*\\) \\([^][]*\\) 
\\(\\[[0-9/]*\\]\\) \\([^;]*\\); \\(.*\\) (\\([^()]*\\))$ string line)
-   (let* ((thread-id (match-string 1 string))
-  (date (match-string 2 string))
-  (count (match-string 3 string))
-  (authors (match-string 4 string))
-  (subject (match-string 5 string))
-  (tags (match-string 6 string))
-  (tag-list (if tags (save-match-data (split-string 
tags)
- (goto-char (point-max))
- (let ((beg (point-marker)))
-   (notmuch-search-show-result date count authors subject 
tags)
-   (notmuch-search-color-line beg (point-marker) tag-list)
-   (put-text-property beg (point-marker) 
'notmuch-search-thread-id thread-id)
-   (put-text-property beg (point-marker) 
'notmuch-search-authors authors)
-   (put-text-property beg (point-marker) 
'notmuch-search-subject subject)
-   (if (string= thread-id notmuch-search-target-thread)
-   (progn
- (set 'found-target beg)
- (set 'notmuch-search-target-thread found
- (set 'line (match-end 0)))
- (set 'more nil)
+   (let ((inhibit-read-only t)
+ ;; We may have a partial line saved from the last iteration.
+ (string (concat notmuch-search-process-filter-data string))
+ (start 0))
+ (goto-char (point-max))
+ ;; Split `string' into lines.
+ (while (string-match \n string start)
+   (let ((line (substring string start (match-beginning 0
+ ;; Save the beginning of the next line already here, so that
+ ;; we can mangle the match data later on.
+ (setq start (match-end 0))
+ (if (string-match
+  ^\\(thread:[0-9A-Fa-f]*\\) \\([^][]*\\) 
\\(\\[[0-9/]*\\]\\) \\([^;]*\\); \\(.*\\) (\\([^()]*\\))$
+  line)
+ (let* ((thread-id (match-string 1 line))
+(date (match-string 2 line))
+(count (match-string 3 line))
+(authors (match-string 4 line))
+(subject (match-string 5 line))
+(tags (match-string 6 line))
+(tag-list (if tags (split-string tags
+   (let ((beg (point-marker)))
+ (notmuch-search-show-result date count authors 
subject tags

Re: [BUG] Emacs UI dropping every 25th line, roughly

2011-02-03 Thread Thomas Schwinge
Hallo!

Here is the problem, as suspected.

On Wed, 02 Feb 2011 17:12:16 +0100, I wrote:
 At this / in the middle of this point, the 4 KiB size is hit.  After
 accumulating some more (notice the 3.6 seconds delay), Emacs read()s the
 first chunk (line breaks inserted for clarity):
 
 16:26:57.928798 read(8, [first two dozen results]
 thread:06c7   2009-12-16 [2/2] 
 Thomas Schwinge; [subject] ([flags])\n
 t, 4096) = 4095
 
 The last result line is obviously incomplete, only the `t' so far.

notmuch.el:notmuch-search-process-filter will be called for processing
these lines.  It'll do fine up to the single `t' -- which simply isn't a
conforming line and thus will be dropped (which is questionable behavior
anyway, I would think?).

 There is more to be read, so Emacs quickly goes on with the next chunk:
 
 16:26:57.966247 read(8, hread:0ac4   2009-12-16 [1/1] 
 js...@sourceware.org; [subject] ([flags])\n
 [more results], 4096) = 4095

Same thing; this time ``hread:[...]'' isn't a confirming line, and will
be dropped.  After that, regular processing continues.


This problem has been around as of the beginning of the incremental /
asynchronous search results changes, as documented in
id:87aayatnw4@yoom.home.cworth.org on 2009-11-25; 2009-11-24's commit
93af7b574598637c2766dd1f8ef343962c9a8efb.


Emacs LISP is not my speciality, neither is any other LISP dialect, so
someone please carefully review this.


Grüße (und gute Nacht...),
 Thomas

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH 2/2] notmuch.el:notmuch-search-process-filter: Rewritten. Cope with incomplete lines.

2011-02-03 Thread Thomas Schwinge
Hallo!

On Thu, 3 Feb 2011 12:06:20 -0500, Austin Clements amdra...@mit.edu wrote:
 Is there a reason you keep the remaining data in a string instead of
 taking the more idiomatic elisp approach of leaving it in the process
 buffer?  In fact, the code would probably be simpler if you
 immediately appended the string to the process buffer like a normal
 process-filter and then peeled things away using buffer-oriented
 regexp functions like looking-at.  Elisp is a lot better at
 manipulating buffers than it is at manipulating strings.

Ha, I hear you -- this is what I meant to do originally.  But then, the
save-in-string approach (even though I always considered keeping state in
the string a bit ugly) seemed more simple to me.  As I said: writing
elisp code is not my primary profession...  :-) (Perhaps I should buy a
book about it, or something.)  Now that you confirmed my original idea,
I'll see about re-writing the code accordingly, so thanks for the input!


Grüße,
 Thomas


pgpkcG4W4yhJj.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH] Shell programming: directories are ``executable'' (that is, searchable), too.

2011-02-02 Thread Thomas Schwinge
Signed-off-by: Thomas Schwinge 
---
 test/test-lib.sh |5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/test/test-lib.sh b/test/test-lib.sh
index f536172..3471ead 100755
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -815,13 +815,14 @@ EOF
 }


+# Locate the directory containing the `notmuch' executable we are to use.
 find_notmuch_path ()
 {
 dir="$1"

 while [ -n "$dir" ]; do
bin="$dir/notmuch"
-   if [ -x "$bin" ]; then
+   if [ -f "$bin" -a -x "$bin" ]; then
echo "$dir"
return
fi
@@ -858,7 +859,7 @@ then

make_valgrind_symlink () {
# handle only executables
-   test -x "$1" || return
+   test -f "$1" -a -x "$1" || return

base=$(basename "$1")
symlink_target=$TEST_DIRECTORY/../$base
-- 
1.7.1



[BUG] Emacs UI dropping every 25th line, roughly

2011-02-02 Thread Thomas Schwinge
Hallo!

On Sun, 30 Jan 2011 22:02:03 +0100, I wrote:
> That is, roughly every 25th line is dropped in the Emacs notmuch-search
> buffer!  (Via setting notmuch-command to a shell script, I intercepted
> the notmuch / emacs pipeline with tee, and found that what comes out of
> ``notmuch | tee'' is still sane, so I'm fairly sure it's an Emacs /
> notmuch elisp code issue.)
> 
> And, 25 times the medium length of a ``notmuch search [...]'' output line
> is... 4 KiB, the standard page size.  Is this ``just'' a problem in the
> notmuch elisp code, or is Emacs doing something awful with (wild
> speculation...) short reads on buffer page boundaries (or whatever else)?

I began to analyze this.  Here is a dump of my steps.

I used strace on the (already running) Emacs process.  strace would log
to several files: one for the main (Emacs) process, and a new one for
each forked process.  I can see that a ``notmuch search [...]'' process
is fork()ed / exec()uted.  It does a lot of read()ing from the DB, and
write()s out the results record by record (which in our case means line
by line).  Up to the point where I examined, there have been no short
writes.  The log:

[some write()s for the first two dozen search results]
16:26:54.143464 write(1, "thread:000006c7   2009-12-16 [2/2] Thomas 
Schwinge; [subject] ([flags])\n", 135) = 135
[...]
16:26:54.266866 write(1, "thread:0ac4   2009-12-16 [1/1] jsm28 
at sourceware.org; [subject] ([flags])\n", 181) = 181

At this / in the middle of this point, the 4 KiB size is hit.  After
accumulating some more (notice the 3.6 seconds delay), Emacs read()s the
first chunk (line breaks inserted for clarity):

16:26:57.928798 read(8, "[first two dozen results]"
    "thread:06c7   2009-12-16 [2/2] Thomas 
Schwinge; [subject] ([flags])\n"
"t", 4096) = 4095

The last result line is obviously incomplete, only the `t' so far.

There is more to be read, so Emacs quickly goes on with the next chunk:

16:26:57.966247 read(8, "hread:0ac4   2009-12-16 [1/1] jsm28 at 
sourceware.org; [subject] ([flags])\n"
"[more results]", 4096) = 4095

That's the remainder of the partial line, plus further results.  This is
all fine.  It's now Emacs' job to re-assemble the two partial lines
(thread ac4).  I note that indeed this search result is missing in the
notmuch-search buffer.

(By the way, I wonder why read() returns 4095 bytes instead of 4096?)

I'll next move on to debugging what actually appears at the elisp layer.
I somehow don't think the Emacs core is faulty; probably rather our
asynchronous results presentation layer handles partial lines
incorrectly?  More to come later.


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110202/3a632b7e/attachment.pgp>


[PATCH 4/4] ruby: Add generated files to gitignore

2011-01-31 Thread Thomas Schwinge
Hallo!

On Mon, 10 Jan 2011 16:39:28 +0200, Ali Polatel  wrote:
> --- a/.gitignore
> +++ b/.gitignore
> @@ -13,3 +13,7 @@ libnotmuch.so*
>  .*.swp
>  *.elc
>  releases
> +
> +bindings/ruby/mkmf.log
> +bindings/ruby/notmuch.so
> +bindings/ruby/Makefile

These should rather be put into bindings/ruby/.gitignore, I'd say.


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 



Re: [PATCH 4/4] ruby: Add generated files to gitignore

2011-01-31 Thread Thomas Schwinge
Hallo!

On Mon, 10 Jan 2011 16:39:28 +0200, Ali Polatel a...@exherbo.org wrote:
 --- a/.gitignore
 +++ b/.gitignore
 @@ -13,3 +13,7 @@ libnotmuch.so*
  .*.swp
  *.elc
  releases
 +
 +bindings/ruby/mkmf.log
 +bindings/ruby/notmuch.so
 +bindings/ruby/Makefile

These should rather be put into bindings/ruby/.gitignore, I'd say.


Grüße,
 Thomas


pgp6hmBu7ufym.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[BUG] Emacs UI dropping every 25th line, roughly

2011-01-30 Thread Thomas Schwinge
Hallo!

I noticed this one on a system with my work emails (which is what I
recently reported on the IRC channel), but can reproduce it on another
system, too.  :-/

The first system is using Ubuntu's emacs23 23.1+1-4ubuntu7.1+maverick1,
the other Debian's emacs23 23.2+1-7.

In the Emacs UI, do a ``M-x notmuch-seach RET tag:notmuch RET'', wait for
it to finish and save the buffer's text to a file, called emacs.

(In my DB, the notmuch mailing list is tagged as `notmuch' -- but it
doesn't matter, just use a search term that matches some hundreds of
messages; ``*'' if you want.)

$ notmuch search tag:notmuch > stdout
$ diff -w -U1 \
<(sed < stdout -e 's%^thread:[0-9a-f]*\ %%' -e 's%|%,%' -e 's%;%%' ) \
emacs \
| cut -c -30
--- /dev/fd/63  2011-01-30 21:4
+++ emacs   2011-01-30 21:30:39.
@@ -76,3 +76,2 @@
  November 21 [1/1] Dmitry Kur
- November 21 [9/9] Tassilo Ho
  November 20 [5/5] Matthieu L
@@ -100,3 +99,2 @@
  November 12 [9/9] Jameson Ro
- November 12 [3/3] Jameson Ro
  November 12 [7/7] David Edmo
@@ -125,3 +123,2 @@
  November 03 [13/13] david at te
- November 02 [2/2] Gregor Kap
  November 01 [10/10] Felipe C
@@ -151,3 +148,2 @@
   October 13 [11/11] Michal S
-  October 13 [14/14] Felipe C
   October 11 [1/1] Kristoffer
@@ -177,3 +173,2 @@
   2010-07-06 [2/2] dbp at riseup
-  2010-07-05 [9/9] Dmitry Kur
   2010-06-27 [1/1] servilio at g
@@ -203,3 +198,2 @@
   2010-06-04 [2/2] Tomas Carn
-  2010-06-04 [2/2] dme at dme.or
   2010-06-04 [4/4] Sebastian 
[...]

That is, roughly every 25th line is dropped in the Emacs notmuch-search
buffer!  (Via setting notmuch-command to a shell script, I intercepted
the notmuch / emacs pipeline with tee, and found that what comes out of
``notmuch | tee'' is still sane, so I'm fairly sure it's an Emacs /
notmuch elisp code issue.)

And, 25 times the medium length of a ``notmuch search [...]'' output line
is... 4 KiB, the standard page size.  Is this ``just'' a problem in the
notmuch elisp code, or is Emacs doing something awful with (wild
speculation...) short reads on buffer page boundaries (or whatever else)?


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 



[PATCH] Add a few tests for searching LWN emails.

2011-01-30 Thread Thomas Schwinge
Hallo!

On Sat, 29 Jan 2011 05:59:38 +1000, Carl Worth  wrote:
> On Thu, 27 Jan 2011 03:31:49 -0700, Thomas Schwinge  
> wrote:
> > These tests should pass -- but they currently don't.

> Thanks for sending these test cases. This is actually my favorite way to
> receive bug reports. I really appreciate it!

I'm doing this for a simple resons: I won't manually have to remember (!)
and later re-run my testings in order to confirm that the issue has been
fixed.  :-)


> So I'm not sure what could reasonably be changed here in GMime or not. I
> definitely do want to fix notmuch so that it indexes all of the text
> here regardless if it's formatted in an RFC-compliant way or not.

Given Matthieu's comment, I wouldn't; as I explained in my reply to his
email.  Now, if this is indeed a GMIME issue -- do we want these tests in
notmuch at all, or should I rather (adapt and) submit them for GMIME?


> [*] Except for test VI which has a bug in that it searches for the word
> "mailing" in the subject header, but no such word exists in the
> message's subject header.

Ack.  Fixed locally.


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110130/478f379a/attachment.pgp>


[PATCH] Add a few tests for searching LWN emails.

2011-01-30 Thread Thomas Schwinge
Hallo!

On Sat, 29 Jan 2011 14:45:19 +0100, Matthieu Lemerre  wrote:
> I had already reported this problem in id:"87ipzvk2xh.fsf at free.fr".

Oh, I see!

> Recent versions of GMime perform more robust parsing that fix the
> problem, but unfortunately debian only ship old versions of the package.

In this case, I'm not eager to work around this issue in notmuch -- this
would only complicate the local code, and yet would be redundand in some
months.  (Your mileage may vary.)  That is, we already do rely on the
GMIME library for this parsing (etc.), so that's the place where this
issue needs to be fixed (and has been).  Hopefully, after the imminent
Debian release, the version of the library will be updated, or someone
(we, I guess) will propose backporting this patch, and get it accepted.
(Which may be non-trivial, as that wasn't a simple patch, but instead
sort of a parser re-write, as I understood it.)


> I don't believe we will be able to make all people from whom we receive
> email always send RFC2822-compliant email addresses :)

Indeed.  :-)


Gr??e,
 Thomas


PS: Matthieu, interesting how peoples' (that is, our) paths cross again.
:-)
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 



Build system

2011-01-30 Thread Thomas Schwinge
Hallo!

On Sun, 30 Jan 2011 11:12:42 +0100, I  wrote:
> On Sat, 29 Jan 2011 06:58:59 +1000, Carl Worth  wrote:
> > Rather than documenting a limitation here, why don't we do what people
> > actually want.
> 
> Well, not a big deal for me -- I'm just used to building projects
> containing a configure file outside of the source tree, and the notmuch
> build machinery at first accepted this, but then fell over.
> 
> 
> > What do other build systems generally do when running configure from
> > some other directory?
> 
> What Rob said -- but: why re-invent Autoconf / Automake if it's already
> there?  I totally admit that the GNU Autotools have their ugly corners
> (and indeed a lot of these), but on the other hand they do solve some
> issues quite nicely.
> 
> You surely had reasons to not use these tools, and that's fine with me.
> (And I'm not especially interested in working on build systems -- done
> that enough in the past.)  Have you looked at other tools before going
> for the straight-forward (and that is very fine!) Makefile-based
> solution?

OK -- I found the thread starting at
id:"1258897630-22282-1-git-send-email-jeff at ocjtech.us", where this has
been discussed already (as I should have expected).  ;-)

Still, my point holds that (unless someone is willing to spend time on
this, of course) we shouldn't try to replicate the Autotools, but instead
keep our system as simple as it currently is, and thus just have it fail
if configured outside of the source tree.


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 



Build system (was: [PATCH] Have to configure and build inside the source directory.)

2011-01-30 Thread Thomas Schwinge
Hallo!

On Sat, 29 Jan 2011 06:58:59 +1000, Carl Worth  wrote:
> Rather than documenting a limitation here, why don't we do what people
> actually want.

Well, not a big deal for me -- I'm just used to building projects
containing a configure file outside of the source tree, and the notmuch
build machinery at first accepted this, but then fell over.


> What do other build systems generally do when running configure from
> some other directory?

What Rob said -- but: why re-invent Autoconf / Automake if it's already
there?  I totally admit that the GNU Autotools have their ugly corners
(and indeed a lot of these), but on the other hand they do solve some
issues quite nicely.

You surely had reasons to not use these tools, and that's fine with me.
(And I'm not especially interested in working on build systems -- done
that enough in the past.)  Have you looked at other tools before going
for the straight-forward (and that is very fine!) Makefile-based
solution?


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 



Build system (was: [PATCH] Have to configure and build inside the source directory.)

2011-01-30 Thread Thomas Schwinge
Hallo!

On Sat, 29 Jan 2011 06:58:59 +1000, Carl Worth cwo...@cworth.org wrote:
 Rather than documenting a limitation here, why don't we do what people
 actually want.

Well, not a big deal for me -- I'm just used to building projects
containing a configure file outside of the source tree, and the notmuch
build machinery at first accepted this, but then fell over.


 What do other build systems generally do when running configure from
 some other directory?

What Rob said -- but: why re-invent Autoconf / Automake if it's already
there?  I totally admit that the GNU Autotools have their ugly corners
(and indeed a lot of these), but on the other hand they do solve some
issues quite nicely.

You surely had reasons to not use these tools, and that's fine with me.
(And I'm not especially interested in working on build systems -- done
that enough in the past.)  Have you looked at other tools before going
for the straight-forward (and that is very fine!) Makefile-based
solution?


Grüße,
 Thomas


pgpneqvZ4RMuS.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: Build system

2011-01-30 Thread Thomas Schwinge
Hallo!

On Sun, 30 Jan 2011 11:12:42 +0100, I tho...@schwinge.name wrote:
 On Sat, 29 Jan 2011 06:58:59 +1000, Carl Worth cwo...@cworth.org wrote:
  Rather than documenting a limitation here, why don't we do what people
  actually want.
 
 Well, not a big deal for me -- I'm just used to building projects
 containing a configure file outside of the source tree, and the notmuch
 build machinery at first accepted this, but then fell over.
 
 
  What do other build systems generally do when running configure from
  some other directory?
 
 What Rob said -- but: why re-invent Autoconf / Automake if it's already
 there?  I totally admit that the GNU Autotools have their ugly corners
 (and indeed a lot of these), but on the other hand they do solve some
 issues quite nicely.
 
 You surely had reasons to not use these tools, and that's fine with me.
 (And I'm not especially interested in working on build systems -- done
 that enough in the past.)  Have you looked at other tools before going
 for the straight-forward (and that is very fine!) Makefile-based
 solution?

OK -- I found the thread starting at
id:1258897630-22282-1-git-send-email-j...@ocjtech.us, where this has
been discussed already (as I should have expected).  ;-)

Still, my point holds that (unless someone is willing to spend time on
this, of course) we shouldn't try to replicate the Autotools, but instead
keep our system as simple as it currently is, and thus just have it fail
if configured outside of the source tree.


Grüße,
 Thomas


pgpUkEkTRKgbH.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH] Add a few tests for searching LWN emails.

2011-01-30 Thread Thomas Schwinge
Hallo!

On Sat, 29 Jan 2011 14:45:19 +0100, Matthieu Lemerre ra...@free.fr wrote:
 I had already reported this problem in id:87ipzvk2xh@free.fr.

Oh, I see!

 Recent versions of GMime perform more robust parsing that fix the
 problem, but unfortunately debian only ship old versions of the package.

In this case, I'm not eager to work around this issue in notmuch -- this
would only complicate the local code, and yet would be redundand in some
months.  (Your mileage may vary.)  That is, we already do rely on the
GMIME library for this parsing (etc.), so that's the place where this
issue needs to be fixed (and has been).  Hopefully, after the imminent
Debian release, the version of the library will be updated, or someone
(we, I guess) will propose backporting this patch, and get it accepted.
(Which may be non-trivial, as that wasn't a simple patch, but instead
sort of a parser re-write, as I understood it.)


 I don't believe we will be able to make all people from whom we receive
 email always send RFC2822-compliant email addresses :)

Indeed.  :-)


Grüße,
 Thomas


PS: Matthieu, interesting how peoples' (that is, our) paths cross again.
:-)


pgp91DSDscqtI.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[BUG] Emacs UI dropping every 25th line, roughly

2011-01-30 Thread Thomas Schwinge
Hallo!

I noticed this one on a system with my work emails (which is what I
recently reported on the IRC channel), but can reproduce it on another
system, too.  :-/

The first system is using Ubuntu's emacs23 23.1+1-4ubuntu7.1+maverick1,
the other Debian's emacs23 23.2+1-7.

In the Emacs UI, do a ``M-x notmuch-seach RET tag:notmuch RET'', wait for
it to finish and save the buffer's text to a file, called emacs.

(In my DB, the notmuch mailing list is tagged as `notmuch' -- but it
doesn't matter, just use a search term that matches some hundreds of
messages; ``*'' if you want.)

$ notmuch search tag:notmuch  stdout
$ diff -w -U1 \
(sed  stdout -e 's%^thread:[0-9a-f]*\ %%' -e 's%|%,%' -e 's%;%%' ) \
emacs \
| cut -c -30
--- /dev/fd/63  2011-01-30 21:4
+++ emacs   2011-01-30 21:30:39.
@@ -76,3 +76,2 @@
  November 21 [1/1] Dmitry Kur
- November 21 [9/9] Tassilo Ho
  November 20 [5/5] Matthieu L
@@ -100,3 +99,2 @@
  November 12 [9/9] Jameson Ro
- November 12 [3/3] Jameson Ro
  November 12 [7/7] David Edmo
@@ -125,3 +123,2 @@
  November 03 [13/13] david@te
- November 02 [2/2] Gregor Kap
  November 01 [10/10] Felipe C
@@ -151,3 +148,2 @@
   October 13 [11/11] Michal S
-  October 13 [14/14] Felipe C
   October 11 [1/1] Kristoffer
@@ -177,3 +173,2 @@
   2010-07-06 [2/2] dbp@riseup
-  2010-07-05 [9/9] Dmitry Kur
   2010-06-27 [1/1] servilio@g
@@ -203,3 +198,2 @@
   2010-06-04 [2/2] Tomas Carn
-  2010-06-04 [2/2] d...@dme.or
   2010-06-04 [4/4] Sebastian 
[...]

That is, roughly every 25th line is dropped in the Emacs notmuch-search
buffer!  (Via setting notmuch-command to a shell script, I intercepted
the notmuch / emacs pipeline with tee, and found that what comes out of
``notmuch | tee'' is still sane, so I'm fairly sure it's an Emacs /
notmuch elisp code issue.)

And, 25 times the medium length of a ``notmuch search [...]'' output line
is... 4 KiB, the standard page size.  Is this ``just'' a problem in the
notmuch elisp code, or is Emacs doing something awful with (wild
speculation...) short reads on buffer page boundaries (or whatever else)?


Grüße,
 Thomas


pgpuD6lHtwOxk.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


notmuch's idea of concurrency / failing an invocation

2011-01-28 Thread Thomas Schwinge
Hallo!

On Fri, 28 Jan 2011 15:10:01 +1000, Carl Worth  wrote:
> On Thu, 27 Jan 2011 17:20:21 -0500, Austin Clements  
> wrote:
> > I'm looking into breaking notmuch new up into small transactions.  It
> > wouldn't be much a leap from there to simply close and reopen the database
> > between transactions if another task wants to use it, which would release
> > the lock and let the queued notmuch task have the database for a bit.
> 
> That sounds like something very useful to pursue. Please continue!

Ack!  And actually -- I just wondered about that: what happens if
``notmuch new'' has executed notmuch_database_add_message for a new
message M, but then is killed before adding any tags and finishing up
(and supposing that the DB isn't in an invalid state now).  This process
of adding M to the DB and applying any tags isn't one single transaction
currently, right?  (And this is exactly what you're working on
chainging?)  Am I right that what currently happens is that upon the next
``notmuch new'' run, notmuch will not reconsider M (given that it already
is present in the DB), but continue with the next messages -- thus
leaving M without any tags?  This isn't a very likely scenario, but still
a possible one.

> > It seems silly to have a daemon when all of notmuch's state is already on 
> > disk
> > and queue on a lock is as good as a queue in a daemon, but without the
> > accompanying architectural shenanigans.

Ack, too.  A daemon seems one abstraction layer too much.  (But I'm not
actively opposed either, if someone has a valid use for such a scheme.)

> It would definitely be nice to avoid the complexity inherent in having a
> daemon, but how do you imagine "queue on a lock" to work? We don't have
> anything like that in place now.

I suppose what he means is trying to get the lock, and if that fails wait
a bit / wait until it is available again.

Actually, as a next step, wouldn't it also be possible to add some
heuristic to avoid ``notmuch new'' (being a low-priority task) blocking
some interactive user (UI; high-priority task)?  But we can pursue such
schemes as soon as the basic infrastructure is in place.

> Another advantage that can happen with queueing (wherever it occurs) is
> to allow a client to be very responsive without waiting for an operation
> to complete. Though that can of course be band if the operation isn't
> reliably committed.

(Obviously this can only work as long as we don't immediatelly need the
operation's result; think ``notmuch show''.)

So, if the DB has the functionality to internally queue and immediatelly
acknowledge transactions, and only later (reliably) commit them, wouldn't
that be fine indeed?  For example, ``notmuch tag'' then wouldn't have to
wait for the DB to be writable.  (And note that I have no idea whether
Xapian supports such things.)  But on the other hand we would like to
immediatelly display the requested change in the UI, right?

What notmuch-show.el:notmuch-show-remove-tag currently does is *not*
re-asking the DB for a message's current tags after having removed a
specific one, but instead it interprets the tag removal command itself --
which is easy enough in this case, and rather unlikely to ever yield
different results, at least unless there's another process operating on
the DB concurrently.

Otherwise, the other way round, the client could maintain a list of to-do
items, to which actions are added if the DB is currently busy, and this
list is periodically worked on in order to get it empty.  For example,
tag changes that are in this list, but not yet committed in the DB could
be displayed in another color in the UI.  But doing so would shift the
responsibility to the UI, which should be in the DB, in my humble
opinion.  (Actually, this issue feels similar to the one who should be
doing the re-trying in case the DB is busy: the UI, or the notmuch
process itself, which we're discussing in another thread.)


As you can guess I'm not very much into DBs, and neither too much into
concurrent systems, so if my ideas don't make sense, please feel free to
refer me to literature.


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: 



[PATCH] Clarify usage of `additional_headers' in test/test-lib.sh:generate_message.

2011-01-28 Thread Thomas Schwinge
Hallo!

On Fri, 28 Jan 2011 15:36:25 +1000, Carl Worth  wrote:
> On Thu, 27 Jan 2011 02:17:21 -0700, Thomas Schwinge  
> wrote:
> > +# Note that in the way we're setting it above and using it below,
> > +# `additional_headers' will also serve as the header / body separator
> > +# (empty line in between).

> I'd even prefer to have the newline explicitly in the HERE document, but
> it's awkward to avoid having the "extra" newline at the end of
> ${additional_headers} the way I'm constructing it incrementally. So just
> documenting the current approach is probably best for now.

Matches my thoughts :-) -- and as it occurs to me right now, doing it in
one here document should be possible like this, if additional_headers is
changed to have the newline *at the beginning* of the string:

cat <"$gen_msg_filename"
From: ${template[from]}
To: ${template[to]}
Message-Id: <${gen_msg_id}>
Subject: ${template[subject]}
Date: ${template[date]}${additional_headers}

${template[body]}
EOF

Or, of course, we could split the here document: base header,
conditionally (if set at all) additional_headers, new line, body.

If you'd like me to prepare (and test) any of these, please tell.


Gr??e,
 Thomas


PS: Didn't know you'd be doing a presentation of notmuch at LCA2011 -- I
saw your announcement on the IRC channel (re live stream) what it was too
late already.  But then, it would have been a rather inconvenient time /
timezone anyways, being based in Germany.  So, how has it been?
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110128/a7d49d4e/attachment.pgp>


notmuch's idea of concurrency / failing an invocation

2011-01-27 Thread Thomas Schwinge
Hallo!

Stepping away from the current code base -- what is notmuch's original
idea of concurrency?  That is, all of us probably know that one:

A Xapian exception occurred opening database: Unable to get write
  lock on /home/thomas/Mail-schwinge.name-thomas/.notmuch/xapian:
  already locked

I recently saw that one while using the Emacs UI (that one tried to
remove a unread tag or similar), and in parallel a delivery to the
notmuch DB was going on.

Apparently the DB we're using doesn't allow for simultaneous writing
(even though it can't even possibly have been dangerous in this case).

Which is the original idea here?  Is it that...

  * each and every client should catch these kinds of errors, and retry,
or eventually give up at some point, and report the status to the
user; or is it that...

  * notmuch internally should catch these concurrency cases, and retry,
or eventually give up at some point (``notmuch --maximum-wait=30s tag
[...]''), and fail as seen above?

This one is an obvious temporary error due to a concurrency situation.
Wouldn't the latter suggestion be preferable here?  I guess that in most
cases the DB isn't locked for long periods of time, and thus the
concurrency situation would decline quickly.


One difficulty I see is judging which errors are temporary and which are
permanent -- which is obvious in a lot of cases (concurrent DB access,
memory starved or any other OS resource), but may not be, for example in
case of I/O errors (is ``disk full'' a permanent error?).  And then, for
some of these cases, waiting does make sense (concurrent DB access, as
suggested above), and for other (temporary?) errors it doesn't make (a
lot of) sense (out of memory: only sensible thing is to abort, and have
the caller re-try, or disk full: waiting for some free space may be worth
it, or it may be not).


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: 



[PATCH] Add a few tests for searching LWN emails.

2011-01-27 Thread Thomas Schwinge
These tests should pass -- but they currently don't.

Signed-off-by: Thomas Schwinge 

---

Hallo!

I reported this on IRC some weeks ago; here is a more elaborate report.

What we get from these emails, is an author named ``LWN.net'', and the
``Weekly Notification'' / ``Mailing Lists'' bits are stripped away.  I
suspect this may be a misinterpretation in the notmuch address parser,
related to the dot in the name.  I have not yet looked at the relevant
code.

The same problem might exist for the To: parser, which I have not yet
checked.


Gre,
 Thomas

 test/search |   36 
 1 files changed, 36 insertions(+), 0 deletions(-)

diff --git a/test/search b/test/search
index b180c7f..2ae5640 100755
--- a/test/search
+++ b/test/search
@@ -3,6 +3,12 @@ test_description='"notmuch search" in several variations'
 . ./test-lib.sh

 add_email_corpus
+add_message \
+  '[from]="LWN.net Weekly Notification "' \
+  '[subject]="LWN.net Weekly Edition for January 27, 2011 available"'
+add_message \
+  '[from]="LWN.net Mailing Lists "' \
+  '[subject]="LWN.net newly freed content for January 27, 2011"'

 test_begin_subtest "Search body"
 add_message '[subject]="body search"' '[date]="Sat, 01 Jan 2000 12:00:00 
-"' [body]=bodysearchtest
@@ -57,6 +63,34 @@ add_message '[subject]="search by from (name)"' 
'[date]="Sat, 01 Jan 2000 12:00:
 output=$(notmuch search from:"Search By From Name" | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Search By From 
Name; search by from (name) (inbox unread)"

+test_begin_subtest "LWN, I:"
+output=$(notmuch search from:'lwn.net weekly notification' | 
notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] LWN.net Weekly 
Notification; LWN.net Weekly Edition for January 27, 2011 available (inbox 
unread)"
+
+test_begin_subtest "LWN, II:"
+output=$(notmuch search from:'lwn.net mailing lists' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] LWN.net Mailing 
Lists; LWN.net newly freed content for January 27, 2011 (inbox unread)"
+
+test_begin_subtest "LWN, III:"
+output=$(notmuch search from:lwn and from:weekly | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] LWN.net Weekly 
Notification; LWN.net Weekly Edition for January 27, 2011 available (inbox 
unread)"
+
+test_begin_subtest "LWN, IV:"
+output=$(notmuch search from:lwn and from:mailing | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] LWN.net Mailing 
Lists; LWN.net newly freed content for January 27, 2011 (inbox unread)"
+
+test_begin_subtest "LWN, V:"
+output=$(notmuch search from:lwn at lwn.net and subject:weekly | 
notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] LWN.net Weekly 
Notification; LWN.net Weekly Edition for January 27, 2011 available (inbox 
unread)"
+
+test_begin_subtest "LWN, VI:"
+output=$(notmuch search from:lwn at lwn.net and subject:mailing | 
notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] LWN.net Mailing 
Lists; LWN.net newly freed content for January 27, 2011 (inbox unread)"
+
+test_begin_subtest "LWN, VII:"
+output=$(notmuch count from:lwn at lwn.net)
+test_expect_equal "$output" 2
+
 test_begin_subtest "Search by to: (address)"
 add_message '[subject]="search by to (address)"' '[date]="Sat, 01 Jan 2000 
12:00:00 -"' [to]=searchbyto at example.com
 output=$(notmuch search to:searchbyto at example.com | notmuch_search_sanitize)
@@ -97,6 +131,8 @@ thread:XXX   2009-11-18 [1/1] Stewart Smith; [notmuch] 
[PATCH] Fix linking with
 thread:XXX   2009-11-18 [2/2] Lars Kellogg-Stedman; [notmuch] \"notmuch help\" 
outputs to stderr? (attachment inbox unread)
 thread:XXX   2009-11-17 [1/1] Mikhail Gusarov; [notmuch] [PATCH] Handle rename 
of message file (inbox unread)
 thread:XXX   2009-11-17 [2/2] Alex Botero-Lowry, Carl Worth; [notmuch] 
preliminary FreeBSD support (attachment inbox unread)
+thread:XXX   2001-01-05 [1/1] LWN.net Weekly Notification; LWN.net Weekly 
Edition for January 27, 2011 available (inbox unread)
+thread:XXX   2001-01-05 [1/1] LWN.net Mailing Lists; LWN.net newly freed 
content for January 27, 2011 (inbox unread)
 thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; body search (inbox unread)
 thread:XXX   2000-01-01 [1/1] searchbyfrom; search by from (inbox unread)
 thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; search by to (inbox unread)
-- 
tg: (74cb76a..) t/from-lwn (depends on: master)


[PATCH] Clarify usage of `additional_headers' in test/test-lib.sh:generate_message.

2011-01-27 Thread Thomas Schwinge
Signed-off-by: Thomas Schwinge 
---
 test/test-lib.sh |5 -
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/test/test-lib.sh b/test/test-lib.sh
index d179426..f536172 100755
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -351,8 +351,11 @@ ${additional_headers}"
 ${additional_headers}"
 fi

+# Note that in the way we're setting it above and using it below,
+# `additional_headers' will also serve as the header / body separator
+# (empty line in between).

-cat <"$gen_msg_filename"
+cat <"$gen_msg_filename"
 From: ${template[from]}
 To: ${template[to]}
 Message-Id: <${gen_msg_id}>
-- 
1.7.1



notmuch's idea of concurrency / failing an invocation

2011-01-27 Thread Thomas Schwinge
Hallo!

Stepping away from the current code base -- what is notmuch's original
idea of concurrency?  That is, all of us probably know that one:

A Xapian exception occurred opening database: Unable to get write
  lock on /home/thomas/Mail-schwinge.name-thomas/.notmuch/xapian:
  already locked

I recently saw that one while using the Emacs UI (that one tried to
remove a unread tag or similar), and in parallel a delivery to the
notmuch DB was going on.

Apparently the DB we're using doesn't allow for simultaneous writing
(even though it can't even possibly have been dangerous in this case).

Which is the original idea here?  Is it that...

  * each and every client should catch these kinds of errors, and retry,
or eventually give up at some point, and report the status to the
user; or is it that...

  * notmuch internally should catch these concurrency cases, and retry,
or eventually give up at some point (``notmuch --maximum-wait=30s tag
[...]''), and fail as seen above?

This one is an obvious temporary error due to a concurrency situation.
Wouldn't the latter suggestion be preferable here?  I guess that in most
cases the DB isn't locked for long periods of time, and thus the
concurrency situation would decline quickly.


One difficulty I see is judging which errors are temporary and which are
permanent -- which is obvious in a lot of cases (concurrent DB access,
memory starved or any other OS resource), but may not be, for example in
case of I/O errors (is ``disk full'' a permanent error?).  And then, for
some of these cases, waiting does make sense (concurrent DB access, as
suggested above), and for other (temporary?) errors it doesn't make (a
lot of) sense (out of memory: only sensible thing is to abort, and have
the caller re-try, or disk full: waiting for some free space may be worth
it, or it may be not).


Grüße,
 Thomas


pgpQ4jpU3xiMy.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH] Clarify usage of `additional_headers' in test/test-lib.sh:generate_message.

2011-01-27 Thread Thomas Schwinge
Hallo!

On Fri, 28 Jan 2011 15:36:25 +1000, Carl Worth cwo...@cworth.org wrote:
 On Thu, 27 Jan 2011 02:17:21 -0700, Thomas Schwinge tho...@schwinge.name 
 wrote:
  +# Note that in the way we're setting it above and using it below,
  +# `additional_headers' will also serve as the header / body separator
  +# (empty line in between).

 I'd even prefer to have the newline explicitly in the HERE document, but
 it's awkward to avoid having the extra newline at the end of
 ${additional_headers} the way I'm constructing it incrementally. So just
 documenting the current approach is probably best for now.

Matches my thoughts :-) -- and as it occurs to me right now, doing it in
one here document should be possible like this, if additional_headers is
changed to have the newline *at the beginning* of the string:

cat EOF $gen_msg_filename
From: ${template[from]}
To: ${template[to]}
Message-Id: ${gen_msg_id}
Subject: ${template[subject]}
Date: ${template[date]}${additional_headers}

${template[body]}
EOF

Or, of course, we could split the here document: base header,
conditionally (if set at all) additional_headers, new line, body.

If you'd like me to prepare (and test) any of these, please tell.


Grüße,
 Thomas


PS: Didn't know you'd be doing a presentation of notmuch at LCA2011 -- I
saw your announcement on the IRC channel (re live stream) what it was too
late already.  But then, it would have been a rather inconvenient time /
timezone anyways, being based in Germany.  So, how has it been?


pgpKO5IrlXe81.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 2/3] new: Add all initial tags at once

2011-01-26 Thread Thomas Schwinge
Hallo!

On Fri, 21 Jan 2011 10:59:36 +0100, Michal Sojka  wrote:
> --- a/notmuch-new.c
> +++ b/notmuch-new.c
> @@ -418,6 +418,7 @@ add_files_recursive (notmuch_database_t *notmuch,
>   /* success */
>   case NOTMUCH_STATUS_SUCCESS:
>   state->added_messages++;
> + notmuch_message_freeze (message);
>   for (tag=state->new_tags; *tag != NULL; tag++)
>   notmuch_message_add_tag (message, *tag);
>   if (state->synchronize_flags == TRUE) {
> @@ -433,6 +434,7 @@ add_files_recursive (notmuch_database_t *notmuch,
>   notmuch_message_maildir_flags_to_tags (message);
>   }
>   }
> + notmuch_message_thaw (message);
>   break;
>   /* Non-fatal issues (go on to next file) */
>   case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:

I do support the patch's idea (which was recently committed; and what
follows in this message is not at all directed towards Michal, who wrote
this patch) -- but what about return values checking?  This is one aspect
of the notmuch C code (which I generally consider to be nice to read and
of high quality, as I said before already), that I consider totally
lacking -- there are literally hundreds of C functions calls where the
return values are just discarded.  This is bad.  For example (simulating
a full disk):

$ notmuch dump > /dev/full
$ echo $?
0

The command returned ``success'' -- but no data has been saved.  This
could, in some extreme (but still likely) case mean: total tag data loss.
(This is likely due to printf / close or fprintf / fclose (yes, really,
(f)close! -- think of buffering) not getting their return values
checked.)  Here is what is expected:

$ echo foo > /dev/full
bash: echo: write error: No space left on device
$ echo $?
1

Then, in the few (!) lines quoted above, there are (exactly / only) calls
to notmuch_message_freeze, notmuch_message_add_tag,
notmuch_message_maildir_flags_to_tags, notmuch_message_thaw -- each of
which can fail, will return this via the notmuch_status_t return value
(whose possible values are documented, too), but none has their return
value checked.

Other languages have the concept of exceptions; C doesn't, so we're
supposed to put some ``ABORT_IF_NOT_NOTMUCH_STATUS_SUCCESS(ret)''
statements after each and every non-void (etc.) C function call.  Or make
the functions abort themselves (which is not a too good idea, as we
surely agree).  Or use a different programming language -- now, at the
present state, it wouldn't be too painful to switch, in my opinion.  (I
won't suggest any specific language, though.)  If staying with C (which I
don't object, either), then this needs a whole code audit, and a lot of
discipline in the future.

Comments?  (And I hope this doesn't sound too harsh :-) -- but it is a
serious programming issue.)


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: 



bug tracking

2011-01-13 Thread Thomas Schwinge
Hallo!

On Mon, 03 May 2010 15:32:09 -0400, Jesse Rosenthal  
wrote:
> Well, this is as good a time to make an announcement as any. I have a
> prototype, nm-remote, in its early stages available here:
> 
> git clone http://jkr.acm.jhu.edu/git/nm-remote.git

I wanted to have a look at this, but the repository is no longer
accessible.


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: 



Re: bug tracking

2011-01-13 Thread Thomas Schwinge
Hallo!

On Mon, 03 May 2010 15:32:09 -0400, Jesse Rosenthal jrosent...@jhu.edu wrote:
 Well, this is as good a time to make an announcement as any. I have a
 prototype, nm-remote, in its early stages available here:
 
 git clone http://jkr.acm.jhu.edu/git/nm-remote.git

I wanted to have a look at this, but the repository is no longer
accessible.


Grüße,
 Thomas


pgpqZ8TdEe9On.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH] Properly quote Emacs' notmuch-command.

2011-01-11 Thread Thomas Schwinge
From: Thomas Schwinge <thomas_schwi...@mentor.com>

It happens that my notmuch-command may look like this:

(setq notmuch-command "~/Mentor Graphics/command/notmuch")

Most of the times, notmuch-command is passed to call-process which will simply
do the right thing, but there are a few other cases where it is fed through the
shell and thus needs to be properly quoted / escaped.

I'm not entirely sure that shell-quote-wildcard-pattern is the correct function
to be used.  We can't use shell-quote-argument (which I first meant to use), as
that one will also escape the tilde (~).  Next idea was to first resolve the
tilde, and then use shell-quote-argument.  But I didn't find a suitable Emacs
function: we can't (unconditionally) use expand-file-name, as that one will
absolutize the filename: it will (wrongly) turn the default value `notmuch'
into `$CWD/notmuch'.

Signed-off-by: Thomas Schwinge 

---

Hallo!

As stated above, if someone has (more Emacs knowledge and) a better
solution, please shout.

An additional item to conider: this solves the issue with tilde
expansion, but what about other shell magic like filename expansion
(globbing; which is what shell-quote-wildcard-pattern explicitly does
*not* quote / escape)?


Once we have determined what we actually want, is it OK to send a
testsuite patch to make it use such a ``nonstandard'' path, for catching
regressions early in the future?


Gre,
 Thomas

 emacs/notmuch-lib.el  |4 ++--
 emacs/notmuch-show.el |4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
index dd180ee..fdeb32b 100644
--- a/emacs/notmuch-lib.el
+++ b/emacs/notmuch-lib.el
@@ -61,7 +61,7 @@ the user hasn't set this variable with the old or new value."
   (let ((long-string
 ;; Trim off the trailing newline.
 (substring (shell-command-to-string
-(concat notmuch-command " --version"))
+(concat (shell-quote-wildcard-pattern notmuch-command) " 
--version"))
0 -1)))
 (if (string-match "^notmuch\\( version\\)? \\(.*\\)$"
  long-string)
@@ -72,7 +72,7 @@ the user hasn't set this variable with the old or new value."
   "Return a value from the notmuch configuration."
   ;; Trim off the trailing newline
   (substring (shell-command-to-string
- (concat notmuch-command " config get " item))
+ (concat (shell-quote-wildcard-pattern notmuch-command) " config 
get " item))
  0 -1))

 (defun notmuch-database-path ()
diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index 3a60d43..820f90f 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -957,12 +957,12 @@ than only the current message."
   (let (shell-command)
 (if entire-thread
(setq shell-command 
- (concat notmuch-command " show --format=mbox "
+ (concat (shell-quote-wildcard-pattern notmuch-command) " show 
--format=mbox "
  (shell-quote-argument
   (mapconcat 'identity 
(notmuch-show-get-message-ids-for-open-messages) " OR "))
  " | " command))
   (setq shell-command
-   (concat notmuch-command " show --format=raw "
+   (concat (shell-quote-wildcard-pattern notmuch-command) " show 
--format=raw "
(shell-quote-argument (notmuch-show-get-message-id)) " | " 
command)))
 (start-process-shell-command "notmuch-pipe-command" "*notmuch-pipe*" 
shell-command)))

-- 
tg: (b3caef1..) t/emcs-quote_notmuch-command (depends on: master)


[ANN] notmuch-deliver

2011-01-11 Thread Thomas Schwinge
Hallo!

On Mon, 08 Nov 2010 09:50:46 -0800, Carl Worth  wrote:
> On Wed, 26 May 2010 17:01:34 +0300, Ali Polatel  wrote:
> > notmuch-deliver is a maildir delivery tool for notmuch mail indexer. It
> > reads from standard input, delivers the mail to the specified maildir
> > and adds it to the notmuch database. This is meant as a convenient
> > alternative to running notmuch new after mail delivery.
> 
> Thanks for sharing this, Ali.
> 
> What's the best way to advertise this to potential users?

I recently put a description and link onto the notmuch web pages.

> Should we include a separate utils directory in the notmuch repository
> with auxiliary programs like this?

I wouldn't do so.  But that is not a very strong opinion of mine.

In general, I like it if I see a repository containing one specific tool,
and that one cleanly interfaces through specified interfaces with another
tool.  These two things are no longer as cleanly visible once
notmuch-deliver was part of the notmuch repository.


> Or should we implement this functionality within the notmuch binary
> itself?

That's another option, of course.  (And a separate discussion.)


> I'm open to suggestions.
> 
> If nothing else, the notmuchmail.org web page should grow a section to
> point to auxiliary programs like this that users might find helpful.

I'm working on that (and other parts of the web pages) as I go on with
exploring the ``notmuch world''.

I'll also take the liberty to put stuff from the mailing list or IRC
discussions into web pages, for we have to document this notmuch beast
;-), and it's better to have a generic place to refer people to, instead
of discussing the same things more than once.


If someone disagrees with any of this, I'm open to discuss these items.


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: 



[PATCH] Have to configure and build inside the source directory.

2011-01-11 Thread Thomas Schwinge
From: Thomas Schwinge tho...@schwinge.name

Signed-off-by: Thomas Schwinge tho...@schwinge.name

---
 configure |   11 +++
 1 files changed, 11 insertions(+), 0 deletions(-)

diff --git a/configure b/configure
index c58dd0f..b5e446c 100755
--- a/configure
+++ b/configure
@@ -188,6 +188,17 @@ developers. Then, please email those details to the 
Notmuch list
 (notmuch@notmuchmail.org) so that we can hopefully make future
 versions of notmuch easier for you to use.
 
+EOF
+
+if ! { :  configure; } 2 /dev/null; then
+cat EOF
+*** Error: You have to configure and build in the source directory.
+
+EOF
+exit 1
+fi
+
+cat EOF
 We'll now investigate your system to verify that all required
 dependencies are available:
 
-- 
tg: (b3caef1..) srcdir (depends on: master)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Hallo!, and hooray!, and some first work, too

2011-01-09 Thread Thomas Schwinge
Hallo!

After having read about it in ..., in ..., (don't know anymore), and then
again over at LWN, I have had the plan to try (or rather: directly switch
to) notmuch like half a year ago, or longer.  I eventually was brave
enough to begin that process in the last days of 2010.  I'm still with
you, as you can see.  Hooray!  :-)

As I said on IRC already: there are, of course, still some rough edges,
but what else should I expect from such a young project.


I'm currently using getmail, then maildrop which invokes notmuch-deliver
for getting my emails into both maildir storage, and making them known to
notmuch.  In that process I can apply tags at will (based on List-Id: /
Mailing-List: headers, mostly, but also for possibly-spam, and so on),
and then use a script that operates on these to further massage them.
Thus, I don't really have a need for ``notmuch new'' anymore, and
notmuch-deliver is sort of an atomic / integrated ``put into maildir and
add tags'' tool for me.  I will post more details on my setup (or rather:
directly put it into the wiki), as soon as all this settles down some
more.

I have not yet switched all my email channels to notmuch, but will do so
soon, I think.

Likewise, there are some topics I'd like to discuss.  I'll also begin to
plow through notmuch's mailing list archive, and will gladly accept
id:xxx pointers that answer / discuss the questions I'll be sending.


I poked at notmuch-deliver's code two weeks ago already (Ali, beware,
there'll be some few further patches coming), and in the last hours, I
finally had a look at notmuch.h and some of the source code.  Here is a
diff containing some comments, or to-do items.  Not all are fully fledged
(I have neither been using talloc, nor Xapian before), but perhaps
someone nevertheless feels like commenting on them.  In general I can
say, that the notmuch code makes a solid impression.  :-)


diff --git a/emacs/notmuch-maildir-fcc.el b/emacs/notmuch-maildir-fcc.el
index 3f1c124..0d7017c 100644
--- a/emacs/notmuch-maildir-fcc.el
+++ b/emacs/notmuch-maildir-fcc.el
@@ -16,6 +16,14 @@
 ;; To use this as the fcc handler for message-mode,
 ;; customize the notmuch-fcc-dirs variable

+;; TODO.  A bunch of maildir code is not specific to fcc handling and should be
+;; factored out.
+
+;; TODO.  In fact, apart from using notmuch-database-path (which should
+;; probably be message-directory instead, anyways?), this whole unit is not
+;; specific to notmuch, and should perhaps be integrated into Emacs'
+;; message.el -- or is there even anything in Gnus that can directly be used?
+
 (eval-when-compile (require 'cl))
 (require 'message)

@@ -73,7 +81,7 @@ yet when sending a mail."
 (defun notmuch-fcc-header-setup ()
   "Add an Fcc header to the current message buffer.

-Can be added to `message-send-hook' and will set the Fcc header
+Is by default added to `message-header-setup-hook' and will set the Fcc header
 based on the values of `notmuch-fcc-dirs'. An existing Fcc header
 will NOT be removed or replaced."

@@ -163,7 +171,7 @@ will NOT be removed or replaced."
 (make-directory (concat path "/new/") t)
 (make-directory (concat path "/tmp/") t))
((file-regular-p path)
-(error "%s is a file. Can't creat maildir." path))
+(error "%s is a file. Can't create maildir." path))
(t
 (error "I don't know how to create a maildir here"

@@ -173,6 +181,8 @@ if successful, nil if not."
   (let ((msg-id (notmuch-maildir-fcc-make-uniq-maildir-id)))
 (while (file-exists-p (concat destdir "/tmp/" msg-id))
   (setq msg-id (notmuch-maildir-fcc-make-uniq-maildir-id)))
+;; TODO.  Race.  Should instead (try to) open (create) the file exclusively,
+;; until that succeeds.
 (cond ((notmuch-maildir-fcc-dir-is-maildir-p destdir)
   (write-file (concat destdir "/tmp/" msg-id))
   msg-id)
diff --git a/lib/database.cc b/lib/database.cc
index 7a00917..b08c429 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -551,7 +551,10 @@ notmuch_database_create (const char *path)

 notmuch = notmuch_database_open (path,
 NOTMUCH_DATABASE_MODE_READ_WRITE);
+/* XXX: should check 'notmuch'.  */
+
 notmuch_database_upgrade (notmuch, NULL, NULL);
+/* XXX: should check return value.  */

   DONE:
 if (notmuch_path)
@@ -760,18 +763,6 @@ handle_sigalrm (unused (int signal))
 do_progress_notify = 1;
 }

-/* Upgrade the current database.
- *
- * After opening a database in read-write mode, the client should
- * check if an upgrade is needed (notmuch_database_needs_upgrade) and
- * if so, upgrade with this function before making any modifications.
- *
- * The optional progress_notify callback can be used by the caller to
- * provide progress indication to the user. If non-NULL it will be
- * called periodically with 'count' as the number of messages upgraded
- * so far and 'total' the overall number of messages that will be
- * converted.
- */