[PATCH] Shell programming: directories are ``executable'' (that is, searchable), too.

2011-02-02 Thread Thomas Schwinge
Signed-off-by: Thomas Schwinge 
---
 test/test-lib.sh |5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/test/test-lib.sh b/test/test-lib.sh
index f536172..3471ead 100755
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -815,13 +815,14 @@ EOF
 }


+# Locate the directory containing the `notmuch' executable we are to use.
 find_notmuch_path ()
 {
 dir="$1"

 while [ -n "$dir" ]; do
bin="$dir/notmuch"
-   if [ -x "$bin" ]; then
+   if [ -f "$bin" -a -x "$bin" ]; then
echo "$dir"
return
fi
@@ -858,7 +859,7 @@ then

make_valgrind_symlink () {
# handle only executables
-   test -x "$1" || return
+   test -f "$1" -a -x "$1" || return

base=$(basename "$1")
symlink_target=$TEST_DIRECTORY/../$base
-- 
1.7.1



[PATCH] Have --format=mbox insert Status:

2011-02-02 Thread Peter John Hartman
Hi folks,

This is not a complete patch, but should give you an idea.  (It works well
enough for me, but I think it could do more).

diff --git a/notmuch-show.c b/notmuch-show.c
index ef421ec..dff66de 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -272,6 +272,12 @@ format_message_mbox (const void *ctx,

 printf ("From %s %s", from, date_asctime);

+if (strstr(_get_tags_as_string(ctx,message),"unread") != NULL) {
+  printf ("Status: O\n");
+} else {
+  printf ("Status: RO\n");
+}
+  
 while ((line_len = getline (, _size, file)) != -1 ) {
if (_is_from_line (line))
putchar ('>');

It could, for instance, insert X-Status: AFTD flags, maybe, and it could be
a little more sensitive about O.  Maybe if tag is "new" it won't put the O
there, but I wasn't sure.  

Anyway, this is chiefly useful for me and others who use the mbox format. 
For instance, I combine notmuch with mutt as follows:

notmuch show --format=mbox $(notmuch search thread:xxx | cut -d' ' -f1) >
tmp.mbox && mutt -f tmp.mbox && rm -f tmp.mbox

Best,
Peter


-- 
sic dicit magister P
PhD Candidate
Collaborative Programme in Ancient and Medieval Philosophy
University of Toronto
http://individual.utoronto.ca/peterjh


new "crypto" branch providing full PGP/MIME support

2011-02-02 Thread Jameson Rollins
Hi, all.  I have pushed a new branch called "crypto" to my notmuch
repository [0].  This branch provides full support for PGP/MIME signed
and encrypted messages, including emacs UI support.  It has been applied
on top of cworth's current master (21e97c50).  It includes the
following:

* David Edmondson's improved multipart handling patch series (cherry-picked)
* Daniel Gillmor's PGP/MIME signature verification patch series (cherry-picked)
* my PGP/MIME decryption+verification patch series
* a test suite for signature verification and decryption
* emacs support for the above

I have signed-off on all cherry-picked commits, as I have reviewed and
tested them extensively (I have been using them in my daily notmuch use
for many months now).  The multipart and sig-verification patches were
cherry-picked to get around a couple of accidental spurious commits in
their originating branches that would have needed to be reverted.

Please test and provide feedback.  I would really like to see this
series merged into the mainline for the next release, if at all
possible.

jamie.

[0] git://finestructure.net/notmuch
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110202/80ba1966/attachment.pgp>


[BUG] Emacs UI dropping every 25th line, roughly

2011-02-02 Thread Thomas Schwinge
Hallo!

On Sun, 30 Jan 2011 22:02:03 +0100, I wrote:
> That is, roughly every 25th line is dropped in the Emacs notmuch-search
> buffer!  (Via setting notmuch-command to a shell script, I intercepted
> the notmuch / emacs pipeline with tee, and found that what comes out of
> ``notmuch | tee'' is still sane, so I'm fairly sure it's an Emacs /
> notmuch elisp code issue.)
> 
> And, 25 times the medium length of a ``notmuch search [...]'' output line
> is... 4 KiB, the standard page size.  Is this ``just'' a problem in the
> notmuch elisp code, or is Emacs doing something awful with (wild
> speculation...) short reads on buffer page boundaries (or whatever else)?

I began to analyze this.  Here is a dump of my steps.

I used strace on the (already running) Emacs process.  strace would log
to several files: one for the main (Emacs) process, and a new one for
each forked process.  I can see that a ``notmuch search [...]'' process
is fork()ed / exec()uted.  It does a lot of read()ing from the DB, and
write()s out the results record by record (which in our case means line
by line).  Up to the point where I examined, there have been no short
writes.  The log:

[some write()s for the first two dozen search results]
16:26:54.143464 write(1, "thread:06c7   2009-12-16 [2/2] Thomas 
Schwinge; [subject] ([flags])\n", 135) = 135
[...]
16:26:54.266866 write(1, "thread:0ac4   2009-12-16 [1/1] jsm28 
at sourceware.org; [subject] ([flags])\n", 181) = 181

At this / in the middle of this point, the 4 KiB size is hit.  After
accumulating some more (notice the 3.6 seconds delay), Emacs read()s the
first chunk (line breaks inserted for clarity):

16:26:57.928798 read(8, "[first two dozen results]"
"thread:06c7   2009-12-16 [2/2] Thomas 
Schwinge; [subject] ([flags])\n"
"t", 4096) = 4095

The last result line is obviously incomplete, only the `t' so far.

There is more to be read, so Emacs quickly goes on with the next chunk:

16:26:57.966247 read(8, "hread:0ac4   2009-12-16 [1/1] jsm28 at 
sourceware.org; [subject] ([flags])\n"
"[more results]", 4096) = 4095

That's the remainder of the partial line, plus further results.  This is
all fine.  It's now Emacs' job to re-assemble the two partial lines
(thread ac4).  I note that indeed this search result is missing in the
notmuch-search buffer.

(By the way, I wonder why read() returns 4095 bytes instead of 4096?)

I'll next move on to debugging what actually appears at the elisp layer.
I somehow don't think the Emacs core is faulty; probably rather our
asynchronous results presentation layer handles partial lines
incorrectly?  More to come later.


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110202/3a632b7e/attachment.pgp>


[RFC PATCH v2 0/8] Custom query parser, date search, folder search, and more

2011-02-02 Thread Carl Worth
Restricting my reply to one tiny bit of your mail:

You wrote:
> non-recursive is the only thing that makes sense for Maildir++ folders

Either I'm not understanding Maildir++ folders, or I don't agree with
you.

I might have an email archive that looks like this:

  Maildir
.work
  .project1
  .project2
  .etc...
.family
  .dad
  .mom
  .brother
  .etc...

With the above setup, what would be unreasonable about wanting to search
for all work-related messages (across all projects, say) with a string
like "folder:work" ?

Now, a person might definitely want to search for messages in the
".work" folder directly, (not including the sub-folders), so we should
provide support for users to get at that behavior as well, (such as a
proposed "folder:work$" or so).

To me, both cases are perfectly legitimate, and I don't understand an
argument that claims that only one makes sense. (Or again, I may be
misunderstanding something.)

-Carl

-- 
carl.d.worth at intel.com
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110202/6dc377eb/attachment.pgp>


notmuch's idea of concurrency / failing an invocation

2011-02-02 Thread Stewart Smith
On Sat, 29 Jan 2011 19:14:27 -0500, Daniel Kahn Gillmor  wrote:
> On 01/28/2011 08:05 PM, Stewart Smith wrote:
> > I'm about at the point where I'm going to take my git mail store
> > experiments and get them really to work (and everyone will have to use
> > 'notmuch cat' or the like to access the messages)
> 
> Would this hypothetical git-based mail store retain the atomicity and
> lockless concurrent-access of a maildir?  That is, could it be used in a
> server environment?

My idea is that it would be... at least with the experiments conducted
so far.

> > which should provide
> > both great storage efficiency, much faster backups of your Maildir as
> > well as having way fewer paths to traverse checking for new mail.
> 
> when you say "backups of your Maildir" do you mean "backups of your
> git-based mail store" ?  or is this somehow a literal Maildir stored in git?

I'll write more "soon" when there is more code behind it... and I figure
out a good upgrade path to something that is also self-consistently sane.

-- 
Stewart Smith


[RFC PATCH v2 0/8] Custom query parser, date search, folder search, and more

2011-02-02 Thread Austin Clements
I rebased the query parser against current master.  It's on the
qparser-3 branch at
  http://awakening.csail.mit.edu/git/notmuch.git

At cworth's request, I've folded the database closing bug fix in to
the appropriate patch.  I also stripped out my implementation of
folder searching, since it obviously conflicts with cworth's [1].  I'm
not planning to resend the patches unless asked because there were no
actual code changes.

[1] I still assert the "correct" folder solution is somewhere between
mine and cworth's: rooted and non-recursive by default (non-recursive
is the only thing that makes sense for Maildir++ folders and it would
be silly for the default to depend on folder type), but leveraging the
flexibility of custom query transforms to support Maildir++ folders
and recursive (and maybe non-rooted) searches as well.  Non-rooted and
recursive searches even map to natural syntaxes that align with
Xapian's existing wildcard syntax.

Quoth myself on Jan 16 at  3:10 am:
> This is version 2 of the custom query parser.  It now supports date
> searches with sane syntax, folder searches (without any additions or
> changes to the database, unlike cworth's recent commit), and "tag:*"
> and "-tag:*" queries for finding tagged and untagged messages.  I used
> these features to guide changes to the original design and to validate
> the approach.  This is still RFC, but it's much less raw now.
> 
> In addition to the new features, the core query parser has a bunch of
> cleanups and changes, including completely redone NEAR and ADJ
> operators that now behave essentially the same as they do in Xapian's
> query parser.  I also split the implementation of these out into a
> separate patch for ease of review.
> 
> There's a notable lack of tests in this current series.  I do have a
> pile of tests for the lexer, parser, and generator, but the
> infrastructure for testing them needs cleanup before I send that out.
> 

-- 
Austin Clements  MIT/'06/PhD/CSAIL
amdragon at mit.edu   http://web.mit.edu/amdragon
   Somewhere in the dream we call reality you will find me,
  searching for the reality we call dreams.