[PATCH 1/1] NEWS: removal of backported support for Emacs version 22

[David Bremner]

Tomi Ollila  writes:

> ---
>  NEWS | 13 +
>  
> +Emacs Interface
> +---
> +
> +Removal of backported Emacs version 23 functionality for Emacs 22
> +
> +  Two years ago some Emacs 23 functionality that notmuch used were
> +  backported to notmuch to be conditionally used with Emacs version 22.
> +  That worked for a few months until new Emacs 23 features were taken
> +  into use and due to lack of interest no one continued to backport
> +  those. To clean up dead code and to avoid confusion in new
> +  developers these backports are now removed.

Hi Tomi;

What do you think about a more user-focussed item, something like 

,
| No Emacs 22 support
| ---
| 
| We started some Emacs 22 support, but it never really worked, and the
| last remnants have been removed from this release
`

but better, of course ;).

d

[PATCH 1/1] NEWS: removal of backported support for Emacs version 22

[Tomi Ollila]

---
 NEWS | 13 +
 1 file changed, 13 insertions(+)

diff --git a/NEWS b/NEWS
index c348068..8d22e1b 100644
--- a/NEWS
+++ b/NEWS
@@ -18,6 +18,19 @@ Bash command-line completion
   `notmuch config`. The new completion support depends on the
   bash-completion package.

+Emacs Interface
+---
+
+Removal of backported Emacs version 23 functionality for Emacs 22
+
+  Two years ago some Emacs 23 functionality that notmuch used were
+  backported to notmuch to be conditionally used with Emacs version 22.
+  That worked for a few months until new Emacs 23 features were taken
+  into use and due to lack of interest no one continued to backport
+  those. To clean up dead code and to avoid confusion in new
+  developers these backports are now removed.
+
+
 Vim Front-End
 -

-- 
1.8.0

[PATCH] emacs: Fix trimming regexp in notmuch-check-exit-status

[Tomi Ollila]

On Fri, May 17 2013, Austin Clements  wrote:

> For such a simple regexp, this was broken in a very complicated way.
>
> The intent was to strip the newline (and potentially other whitespace)
> off the end of the error string so there wasn't an extra newline in
> the error signal.  However, the regexp was deeply dependent on the
> active syntax table and the subtleties of $.  We didn't notice this
> because all notmuch major modes put ?\n in the whitespace class, which
> makes this behaved as intended: the "\\s " matches all newlines, but
> by matching the newline character, causes the $ *not* to match
> *except* where it matched the empty string at the very end of the
> string, which was not followed by a newline.
>
> However, if the syntax table declares ?\n to be non-whitespace
> (lisp-mode declares it as endcomment, and is likely to be the mode
> you're in when testing functions), then this regexp behaves completely
> differently, matching trailing spaces at the end of every line within
> the string.
>
> The solution is to say what we mean for whitespace *and* to switch
> from $ to \', which matches only the end of the string, rather than
> the end of each line.  Both are necessary or this will strip away
> interior newlines, which is not what we want.

LGTM.

Tomi


> ---
>  emacs/notmuch-lib.el |2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
> index 790136e..59b1ce3 100644
> --- a/emacs/notmuch-lib.el
> +++ b/emacs/notmuch-lib.el
> @@ -438,7 +438,7 @@ You may need to restart Emacs or upgrade your notmuch 
> package."))
>   (if err
>   ;; We have an error message straight from the CLI.
>   (notmuch-logged-error
> -  (replace-regexp-in-string "\\s $" "" err) extra)
> +  (replace-regexp-in-string "[ \n\r\t\f]*\\'" "" err) extra)
> ;; We only have combined output from the CLI; don't inundate
> ;; the user with it.  Mimic `process-lines'.
> (notmuch-logged-error (format "%s exited with status %s"
> -- 
> 1.7.10.4
>
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 3/5] emacs: Use async process helper for search

[Austin Clements]

Quoth Mark Walters on May 18 at  8:14 am:
> 
> I have only very briefly looked at this: it seems to not quite apply to
> master (one fix up see below)
> 
> Also, as far as I can see condition-case-unless-debug (used in patch
> 2/5) is emacs 24 only.

Indeed.  I'll just use condition-case in v2 (unless we want to
introduce a compatibility macro).

I really wish the Emacs documentation included what version functions
were introduced in.  I couldn't even *find* a copy of the non-latest
documentation; I wound up downloading a tarball of Emacs 23 and
building the documentation myself.

> Best wishes
> 
> Mark
> 
> On Sat, 18 May 2013, Austin Clements  wrote:
> > Previously, search started the async notmuch process directly.  Now,
> > it uses `notmuch-start-notmuch'.  This simplifies the process sentinel
> > a bit and means that we no longer have to worry about errors
> > interleaved with the JSON output.
> >
> > We also update the tests of Emacs error handling, since the error
> > output is now separated from the search results buffer.
> > ---
> >  emacs/notmuch.el |   19 +--
> >  test/emacs   |   36 
> >  2 files changed, 37 insertions(+), 18 deletions(-)
> >
> > diff --git a/emacs/notmuch.el b/emacs/notmuch.el
> > index 4c1a6ca..b8d9c44 100644
> > --- a/emacs/notmuch.el
> > +++ b/emacs/notmuch.el
> > @@ -653,15 +653,8 @@ of the result."
> > ;; For version mismatch, there's no point in
> > ;; showing the search buffer
> > (when (or (= exit-status 20) (= exit-status 21))
> > - (kill-buffer))
> > -   (condition-case err
> > -   (notmuch-check-async-exit-status proc msg)
> > - ;; Suppress the error signal since strange
> > - ;; things happen if a sentinel signals.  Mimic
> > - ;; the top-level's handling of error messages.
> > - (error
> > -  (message "%s" (error-message-string err))
> 
> This line is 
>  (message "%s" (second err))
> in master.

Sorry, I thought these patches were independent from the ones I'd sent
earlier, but obviously not.  It should apply now that David has
applied the other patch.

> > -  (throw 'return nil)))
> > + (kill-buffer)
> > + (throw 'return nil))
> > (if (and atbob
> >  (not (string= notmuch-search-target-thread 
> > "found")))
> > (set 'never-found-target-thread t)
> > @@ -938,10 +931,9 @@ Other optional parameters are used as follows:
> >(erase-buffer)
> >(goto-char (point-min))
> >(save-excursion
> > -   (let ((proc (start-process
> > -"notmuch-search" buffer
> > -notmuch-command "search"
> > -"--format=json" "--format-version=1"
> > +   (let ((proc (notmuch-start-notmuch
> > +"notmuch-search" buffer #'notmuch-search-process-sentinel
> > +"search" "--format=json" "--format-version=1"
> >  (if oldest-first
> >  "--sort=oldest-first"
> >"--sort=newest-first")
> > @@ -951,7 +943,6 @@ Other optional parameters are used as follows:
> >   ;; should be called no matter how the process dies.
> >   (parse-buf (generate-new-buffer " *notmuch search parse*")))
> >   (process-put proc 'parse-buf parse-buf)
> > - (set-process-sentinel proc 'notmuch-search-process-sentinel)
> >   (set-process-filter proc 'notmuch-search-process-filter)
> >   (set-process-query-on-exit-flag proc nil
> >  (run-hooks 'notmuch-search-hook)))
> > diff --git a/test/emacs b/test/emacs
> > index f033bdf..d38ae8c 100755
> > --- a/test/emacs
> > +++ b/test/emacs
> > @@ -853,11 +853,10 @@ test_expect_success "Rendering HTML mail with images" 
> > \
> >  'cat OUTPUT && grep -q smiley OUTPUT'
> >  
> >  
> > -test_begin_subtest "Search handles subprocess errors"
> > +test_begin_subtest "Search handles subprocess error exit codes"
> >  cat > notmuch_fail < >  #!/bin/sh
> >  echo This is output
> > -echo This is an error >&2
> >  exit 1
> >  EOF
> >  chmod a+x notmuch_fail
> > @@ -874,8 +873,6 @@ sed -i -e 's/^\[.*\]$/[XXX]/' ERROR
> >  test_expect_equal "$(cat OUTPUT; echo ---; cat MESSAGES; echo ---; cat 
> > ERROR)" "\
> >  Error: Unexpected output from notmuch search:
> >  This is output
> > -Error: Unexpected output from notmuch search:
> > -This is an error
> >  End of search results.
> >  ---
> >  $PWD/notmuch_fail exited with status 1 (see *Notmuch errors* for more 
> > details)
> > @@ -885,4 +882,35 @@ $PWD/notmuch_fail exited with status 1
> >  command: $PWD/notmuch_fail search --format\=json --format-version\=1 
> > --sort\=newest-first tag\:inbox
> >  exit status: 1"
> >  
> > +test_begin_subtest "Search handles subprocess warnings"
> > +cat > notmuch_fail 

Solaris support - missing or incompatible functions

[David Bremner]

Vladimir Marek  writes:

>> 
>> There may be some prerequisites to running tests that you're missing,
>> and the test setup fails to tell you about. A pastebin of the 'make
>> test' output might be helpful.
>
> Ok, in that case please let me take a look at the tests first.
>

For what it's worth, I can confirm no test failures for me either on
linux (Debian wheezy).

d

[PATCH 3/5] emacs: Use async process helper for search

[Mark Walters]

I have only very briefly looked at this: it seems to not quite apply to
master (one fix up see below)

Also, as far as I can see condition-case-unless-debug (used in patch
2/5) is emacs 24 only.

Best wishes

Mark

On Sat, 18 May 2013, Austin Clements  wrote:
> Previously, search started the async notmuch process directly.  Now,
> it uses `notmuch-start-notmuch'.  This simplifies the process sentinel
> a bit and means that we no longer have to worry about errors
> interleaved with the JSON output.
>
> We also update the tests of Emacs error handling, since the error
> output is now separated from the search results buffer.
> ---
>  emacs/notmuch.el |   19 +--
>  test/emacs   |   36 
>  2 files changed, 37 insertions(+), 18 deletions(-)
>
> diff --git a/emacs/notmuch.el b/emacs/notmuch.el
> index 4c1a6ca..b8d9c44 100644
> --- a/emacs/notmuch.el
> +++ b/emacs/notmuch.el
> @@ -653,15 +653,8 @@ of the result."
>   ;; For version mismatch, there's no point in
>   ;; showing the search buffer
>   (when (or (= exit-status 20) (= exit-status 21))
> -   (kill-buffer))
> - (condition-case err
> - (notmuch-check-async-exit-status proc msg)
> -   ;; Suppress the error signal since strange
> -   ;; things happen if a sentinel signals.  Mimic
> -   ;; the top-level's handling of error messages.
> -   (error
> -(message "%s" (error-message-string err))

This line is 
   (message "%s" (second err))
in master.


> -(throw 'return nil)))
> +   (kill-buffer)
> +   (throw 'return nil))
>   (if (and atbob
>(not (string= notmuch-search-target-thread 
> "found")))
>   (set 'never-found-target-thread t)
> @@ -938,10 +931,9 @@ Other optional parameters are used as follows:
>(erase-buffer)
>(goto-char (point-min))
>(save-excursion
> - (let ((proc (start-process
> -  "notmuch-search" buffer
> -  notmuch-command "search"
> -  "--format=json" "--format-version=1"
> + (let ((proc (notmuch-start-notmuch
> +  "notmuch-search" buffer #'notmuch-search-process-sentinel
> +  "search" "--format=json" "--format-version=1"
>(if oldest-first
>"--sort=oldest-first"
>  "--sort=newest-first")
> @@ -951,7 +943,6 @@ Other optional parameters are used as follows:
> ;; should be called no matter how the process dies.
> (parse-buf (generate-new-buffer " *notmuch search parse*")))
> (process-put proc 'parse-buf parse-buf)
> -   (set-process-sentinel proc 'notmuch-search-process-sentinel)
> (set-process-filter proc 'notmuch-search-process-filter)
> (set-process-query-on-exit-flag proc nil
>  (run-hooks 'notmuch-search-hook)))
> diff --git a/test/emacs b/test/emacs
> index f033bdf..d38ae8c 100755
> --- a/test/emacs
> +++ b/test/emacs
> @@ -853,11 +853,10 @@ test_expect_success "Rendering HTML mail with images" \
>  'cat OUTPUT && grep -q smiley OUTPUT'
>  
>  
> -test_begin_subtest "Search handles subprocess errors"
> +test_begin_subtest "Search handles subprocess error exit codes"
>  cat > notmuch_fail <  #!/bin/sh
>  echo This is output
> -echo This is an error >&2
>  exit 1
>  EOF
>  chmod a+x notmuch_fail
> @@ -874,8 +873,6 @@ sed -i -e 's/^\[.*\]$/[XXX]/' ERROR
>  test_expect_equal "$(cat OUTPUT; echo ---; cat MESSAGES; echo ---; cat 
> ERROR)" "\
>  Error: Unexpected output from notmuch search:
>  This is output
> -Error: Unexpected output from notmuch search:
> -This is an error
>  End of search results.
>  ---
>  $PWD/notmuch_fail exited with status 1 (see *Notmuch errors* for more 
> details)
> @@ -885,4 +882,35 @@ $PWD/notmuch_fail exited with status 1
>  command: $PWD/notmuch_fail search --format\=json --format-version\=1 
> --sort\=newest-first tag\:inbox
>  exit status: 1"
>  
> +test_begin_subtest "Search handles subprocess warnings"
> +cat > notmuch_fail < +#!/bin/sh
> +echo This is output
> +echo This is a warning >&2
> +echo This is another warning >&2
> +exit 0
> +EOF
> +chmod a+x notmuch_fail
> +test_emacs "(let ((notmuch-command \"$PWD/notmuch_fail\"))
> +(with-current-buffer \"*Messages*\" (erase-buffer))
> +(with-current-buffer \"*Notmuch errors*\" (erase-buffer))
> +(notmuch-search \"tag:inbox\")
> +(notmuch-test-wait)
> +(with-current-buffer \"*Messages*\"
> +   (test-output \"MESSAGES\"))
> +(with-current-buffer \"*Notmuch errors*\"
> +   (test-output \"ERROR\"))
> +(test-output))"
> +sed -i -e 's/^\[.*\]$/[XXX]/' 

[PATCH] emacs: Proper error string handling in search sentinel

[David Bremner]

Austin Clements  writes:

> Apparently Emacs provides a function to stringify errors properly.
> Use this in the search sentinel where we have to do our own error
> messaging, rather than assuming the first error argument will be the
> descriptive string.

Pushed.

Mark, this should fix the problem with applying the asynch parser
series you noticed.

d

[PATCH 3/5] emacs: Use async process helper for search

[David Bremner]

Mark Walters  writes:

>> -  (error
>> -   (message "%s" (error-message-string err))
>
> This line is 
>  (message "%s" (second err))
> in master.
>

This is a seperate patch sent to the list recently.

d

[PATCH 5/5] emacs: Use streaming S-expr parser for search

[Austin Clements]

In addition to being the Right Thing to do, this noticeably improves
the time taken to display the first page of search results, since it's
roughly an order of magnitude faster than the JSON parser.
Interestingly, it does *not* significantly improve the time to
completely fill a large search buffer because for large search
buffers, the cost of creating author invisibility overlays and
inserting text (which slows down with more overlays) dominates.
However, the time required to display the first page of results is
generally more important to the user experience.
---
 emacs/notmuch.el |   13 +++--
 test/emacs   |   10 +++---
 2 files changed, 6 insertions(+), 17 deletions(-)

diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index b8d9c44..5a8c957 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -58,6 +58,7 @@
 (require 'notmuch-hello)
 (require 'notmuch-maildir-fcc)
 (require 'notmuch-message)
+(require 'notmuch-parser)

 (defcustom notmuch-search-result-format
   `(("date" . "%12s ")
@@ -809,13 +810,6 @@ non-authors is found, assume that all of the authors 
match."
(setq notmuch-search-target-thread "found")
(goto-char beg)

-(defun notmuch-search-show-error (string  objects)
-  (save-excursion
-(goto-char (point-max))
-(insert "Error: Unexpected output from notmuch search:\n")
-(insert (apply #'format string objects))
-(insert "\n")))
-
 (defun notmuch-search-process-filter (proc string)
   "Process and filter the output of \"notmuch search\""
   (let ((results-buf (process-buffer proc))
@@ -829,8 +823,7 @@ non-authors is found, assume that all of the authors match."
(save-excursion
  (goto-char (point-max))
  (insert string))
-   (notmuch-json-parse-partial-list 'notmuch-search-show-result
-'notmuch-search-show-error
+   (notmuch-sexp-parse-partial-list 'notmuch-search-show-result
 results-buf)

 (defun notmuch-search-tag-all ( tag-changes)
@@ -933,7 +926,7 @@ Other optional parameters are used as follows:
   (save-excursion
(let ((proc (notmuch-start-notmuch
 "notmuch-search" buffer #'notmuch-search-process-sentinel
-"search" "--format=json" "--format-version=1"
+"search" "--format=sexp" "--format-version=1"
 (if oldest-first
 "--sort=oldest-first"
   "--sort=newest-first")
diff --git a/test/emacs b/test/emacs
index d38ae8c..7d42abf 100755
--- a/test/emacs
+++ b/test/emacs
@@ -856,7 +856,7 @@ test_expect_success "Rendering HTML mail with images" \
 test_begin_subtest "Search handles subprocess error exit codes"
 cat > notmuch_fail < notmuch_fail <&2
 echo This is another warning >&2
 exit 0
@@ -903,8 +901,6 @@ test_emacs "(let ((notmuch-command \"$PWD/notmuch_fail\"))
   (test-output))"
 sed -i -e 's/^\[.*\]$/[XXX]/' ERROR
 test_expect_equal "$(cat OUTPUT; echo ---; cat MESSAGES; echo ---; cat ERROR)" 
"\
-Error: Unexpected output from notmuch search:
-This is output
 End of search results.
 ---
 This is a warning (see *Notmuch errors* for more details)
-- 
1.7.10.4

[PATCH 4/5] emacs: Streaming S-expression parser

[Austin Clements]

This provides the same interface as the streaming JSON parser, but
reads S-expressions incrementally.  The only difference is that the
`notmuch-sexp-parse-partial-list' helper does not handle interleaved
error messages (since we now have the ability to separate these out at
the invocation level), so it no longer takes an error function and
does not need to do the horrible resynchronization that the JSON
parser had to.

Some implementation improvements have been made over the JSON parser.
This uses a vector instead of a list for the parser data structure,
since this allows faster access to elements (and modern versions of
Emacs handle storage of small vectors efficiently).  Private functions
follow the "prefix--name" convention.  And the implementation is much
simpler overall because S-expressions are much easier to parse.
---
 emacs/Makefile.local|1 +
 emacs/notmuch-parser.el |  212 +++
 2 files changed, 213 insertions(+)
 create mode 100644 emacs/notmuch-parser.el

diff --git a/emacs/Makefile.local b/emacs/Makefile.local
index 456700a..a910aff 100644
--- a/emacs/Makefile.local
+++ b/emacs/Makefile.local
@@ -3,6 +3,7 @@
 dir := emacs
 emacs_sources := \
$(dir)/notmuch-lib.el \
+   $(dir)/notmuch-parser.el \
$(dir)/notmuch.el \
$(dir)/notmuch-query.el \
$(dir)/notmuch-show.el \
diff --git a/emacs/notmuch-parser.el b/emacs/notmuch-parser.el
new file mode 100644
index 000..1b7cf64
--- /dev/null
+++ b/emacs/notmuch-parser.el
@@ -0,0 +1,212 @@
+;; notmuch-parser.el --- streaming S-expression parser
+;;
+;; Copyright ? Austin Clements
+;;
+;; This file is part of Notmuch.
+;;
+;; Notmuch is free software: you can redistribute it and/or modify it
+;; under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+;;
+;; Notmuch is distributed in the hope that it will be useful, but
+;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+;; General Public License for more details.
+;;
+;; You should have received a copy of the GNU General Public License
+;; along with Notmuch.  If not, see .
+;;
+;; Authors: Austin Clements 
+
+(require 'cl)
+
+(defun notmuch-sexp-create-parser (buffer)
+  "Return a streaming S-expression parser that reads from BUFFER.
+
+This parser is designed to incrementally read an S-expression
+whose structure is known to the caller.  Like a typical
+S-expression parsing interface, it provides a function to read a
+complete S-expression from the input.  However, it extends this
+with an additional function that requires the next value in the
+input to be a list and descends into it, allowing its elements to
+be read one at a time or further descended into.  Both functions
+can return 'retry to indicate that not enough input is available.
+
+The parser always consumes input from BUFFER's point.  Hence, the
+caller is allowed to delete any data before point and may
+resynchronize after an error by moving point."
+
+  (vector 'notmuch-sexp-parser
+ buffer
+ ;; List depth
+ 0
+ ;; Partial parse position marker
+ nil
+ ;; Partial parse state
+ nil))
+
+(defmacro notmuch-sexp--buffer (sp)`(aref ,sp 1))
+(defmacro notmuch-sexp--depth (sp) `(aref ,sp 2))
+(defmacro notmuch-sexp--partial-pos (sp)   `(aref ,sp 3))
+(defmacro notmuch-sexp--partial-state (sp) `(aref ,sp 4))
+
+(defun notmuch-sexp-read (sp)
+  "Consume and return the value at point in SP's buffer.
+
+Returns 'retry if there is insufficient input to parse a complete
+value (though it may still move point over whitespace).  If the
+parser is currently inside a list and the next token ends the
+list, this moves point just past the terminator and returns 'end.
+Otherwise, this moves point to just past the end of the value and
+returns the value."
+
+  (with-current-buffer (notmuch-sexp--buffer sp)
+(skip-chars-forward " \n\r\t")
+(cond ((eobp) 'retry)
+ ((= (char-after) ?\))
+  ;; We've reached the end of a list
+  (if (= (notmuch-sexp--depth sp) 0)
+  ;; .. but we weren't in a list.  Let read signal the
+  ;; error.
+  (read (current-buffer))
+;; Go up a level and return an end token
+(decf (notmuch-sexp--depth sp))
+(forward-char)
+'end))
+ ((= (char-after) ?\()
+  ;; We're at the beginning of a list.  If we haven't started
+  ;; a partial parse yet, attempt to read the list in its
+  ;; entirety.  If this fails, or we've started a partial
+  ;; parse, extend the partial parse to figure out when we
+  ;; have a complete list.
+  (catch 'return
+(when (null 

[PATCH 3/5] emacs: Use async process helper for search

[Austin Clements]

Previously, search started the async notmuch process directly.  Now,
it uses `notmuch-start-notmuch'.  This simplifies the process sentinel
a bit and means that we no longer have to worry about errors
interleaved with the JSON output.

We also update the tests of Emacs error handling, since the error
output is now separated from the search results buffer.
---
 emacs/notmuch.el |   19 +--
 test/emacs   |   36 
 2 files changed, 37 insertions(+), 18 deletions(-)

diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index 4c1a6ca..b8d9c44 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -653,15 +653,8 @@ of the result."
;; For version mismatch, there's no point in
;; showing the search buffer
(when (or (= exit-status 20) (= exit-status 21))
- (kill-buffer))
-   (condition-case err
-   (notmuch-check-async-exit-status proc msg)
- ;; Suppress the error signal since strange
- ;; things happen if a sentinel signals.  Mimic
- ;; the top-level's handling of error messages.
- (error
-  (message "%s" (error-message-string err))
-  (throw 'return nil)))
+ (kill-buffer)
+ (throw 'return nil))
(if (and atbob
 (not (string= notmuch-search-target-thread 
"found")))
(set 'never-found-target-thread t)
@@ -938,10 +931,9 @@ Other optional parameters are used as follows:
   (erase-buffer)
   (goto-char (point-min))
   (save-excursion
-   (let ((proc (start-process
-"notmuch-search" buffer
-notmuch-command "search"
-"--format=json" "--format-version=1"
+   (let ((proc (notmuch-start-notmuch
+"notmuch-search" buffer #'notmuch-search-process-sentinel
+"search" "--format=json" "--format-version=1"
 (if oldest-first
 "--sort=oldest-first"
   "--sort=newest-first")
@@ -951,7 +943,6 @@ Other optional parameters are used as follows:
  ;; should be called no matter how the process dies.
  (parse-buf (generate-new-buffer " *notmuch search parse*")))
  (process-put proc 'parse-buf parse-buf)
- (set-process-sentinel proc 'notmuch-search-process-sentinel)
  (set-process-filter proc 'notmuch-search-process-filter)
  (set-process-query-on-exit-flag proc nil
 (run-hooks 'notmuch-search-hook)))
diff --git a/test/emacs b/test/emacs
index f033bdf..d38ae8c 100755
--- a/test/emacs
+++ b/test/emacs
@@ -853,11 +853,10 @@ test_expect_success "Rendering HTML mail with images" \
 'cat OUTPUT && grep -q smiley OUTPUT'


-test_begin_subtest "Search handles subprocess errors"
+test_begin_subtest "Search handles subprocess error exit codes"
 cat > notmuch_fail <&2
 exit 1
 EOF
 chmod a+x notmuch_fail
@@ -874,8 +873,6 @@ sed -i -e 's/^\[.*\]$/[XXX]/' ERROR
 test_expect_equal "$(cat OUTPUT; echo ---; cat MESSAGES; echo ---; cat ERROR)" 
"\
 Error: Unexpected output from notmuch search:
 This is output
-Error: Unexpected output from notmuch search:
-This is an error
 End of search results.
 ---
 $PWD/notmuch_fail exited with status 1 (see *Notmuch errors* for more details)
@@ -885,4 +882,35 @@ $PWD/notmuch_fail exited with status 1
 command: $PWD/notmuch_fail search --format\=json --format-version\=1 
--sort\=newest-first tag\:inbox
 exit status: 1"

+test_begin_subtest "Search handles subprocess warnings"
+cat > notmuch_fail <&2
+echo This is another warning >&2
+exit 0
+EOF
+chmod a+x notmuch_fail
+test_emacs "(let ((notmuch-command \"$PWD/notmuch_fail\"))
+  (with-current-buffer \"*Messages*\" (erase-buffer))
+  (with-current-buffer \"*Notmuch errors*\" (erase-buffer))
+  (notmuch-search \"tag:inbox\")
+  (notmuch-test-wait)
+  (with-current-buffer \"*Messages*\"
+ (test-output \"MESSAGES\"))
+  (with-current-buffer \"*Notmuch errors*\"
+ (test-output \"ERROR\"))
+  (test-output))"
+sed -i -e 's/^\[.*\]$/[XXX]/' ERROR
+test_expect_equal "$(cat OUTPUT; echo ---; cat MESSAGES; echo ---; cat ERROR)" 
"\
+Error: Unexpected output from notmuch search:
+This is output
+End of search results.
+---
+This is a warning (see *Notmuch errors* for more details)
+---
+[XXX]
+This is a warning
+This is another warning"
+
 test_done
-- 
1.7.10.4

[PATCH 2/5] emacs: Utilities to manage asynchronous notmuch processes

[Austin Clements]

This provides a new notmuch-lib utility to start an asynchronous
notmuch process that handles redirecting of stderr and checking of the
exit status.  This is similar to `notmuch-call-notmuch-json', but for
asynchronous processes (and it leaves output processing to the
caller).
---
 emacs/notmuch-lib.el |   73 +++---
 1 file changed, 69 insertions(+), 4 deletions(-)

diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
index 59b1ce3..a543471 100644
--- a/emacs/notmuch-lib.el
+++ b/emacs/notmuch-lib.el
@@ -383,18 +383,21 @@ signaled error.  This function does not return."
   (error "%s" (concat msg (when extra
" (see *Notmuch errors* for more details)"

-(defun notmuch-check-async-exit-status (proc msg)
+(defun notmuch-check-async-exit-status (proc msg  command err-file)
   "If PROC exited abnormally, pop up an error buffer and signal an error.

 This is a wrapper around `notmuch-check-exit-status' for
 asynchronous process sentinels.  PROC and MSG must be the
-arguments passed to the sentinel."
+arguments passed to the sentinel.  COMMAND and ERR-FILE, if
+provided, are passed to `notmuch-check-exit-status'.  If COMMAND
+is not provided, it is taken from `process-command'."
   (let ((exit-status
 (case (process-status proc)
   ((exit) (process-exit-status proc))
   ((signal) msg
 (when exit-status
-  (notmuch-check-exit-status exit-status (process-command proc)
+  (notmuch-check-exit-status exit-status (or command (process-command 
proc))
+nil err-file

 (defun notmuch-check-exit-status (exit-status command  output 
err-file)
   "If EXIT-STATUS is non-zero, pop up an error buffer and signal an error.
@@ -448,7 +451,7 @@ You may need to restart Emacs or upgrade your notmuch 
package."))


 (defun notmuch-call-notmuch-json ( args)
-  "Invoke `notmuch-command' with `args' and return the parsed JSON output.
+  "Invoke `notmuch-command' with ARGS and return the parsed JSON output.

 The returned output will represent objects using property lists
 and arrays as lists.  If notmuch exits with a non-zero status,
@@ -469,6 +472,68 @@ an error."
  (json-read)))
(delete-file err-file)

+(defun notmuch-start-notmuch (name buffer sentinel  args)
+  "Start and return an asynchronous notmuch command.
+
+This starts and returns an asynchronous process running
+`notmuch-command' with ARGS.  The exit status is checked via
+`notmuch-check-async-exit-status'.  Output written to stderr is
+redirected and displayed when the process exits (even if the
+process exits successfully).  NAME and BUFFER are the same as in
+`start-process'.  SENTINEL is a process sentinel function to call
+when the process exits, or nil for none.  The caller must *not*
+invoke `set-process-sentinel' directly on the returned process,
+as that will interfere with the handling of stderr and the exit
+status."
+
+  ;; There is no way (as of Emacs 24.3) to capture stdout and stderr
+  ;; separately for asynchronous processes, or even to redirect stderr
+  ;; to a file, so we use a trivial shell wrapper to send stderr to a
+  ;; temporary file and clean things up in the sentinel.
+  (let* ((err-file (make-temp-file "nmerr"))
+;; Use a pipe
+(process-connection-type nil)
+(proc (apply #'start-process name buffer
+ "sh" "-c"
+ "ERR=\"$1\"; shift; exec \"$0\" \"$@\" 2>\"$ERR\""
+ notmuch-command err-file args)))
+(process-put proc 'err-file err-file)
+(process-put proc 'sub-sentinel sentinel)
+(process-put proc 'real-command (cons notmuch-command args))
+(set-process-sentinel proc #'notmuch-start-notmuch-sentinel)
+proc))
+
+(defun notmuch-start-notmuch-sentinel (proc event)
+  (let ((err-file (process-get proc 'err-file))
+   (sub-sentinel (process-get proc 'sub-sentinel))
+   (real-command (process-get proc 'real-command)))
+(condition-case-unless-debug err
+   (progn
+ ;; Invoke the sub-sentinel, if any
+ (when sub-sentinel
+   (funcall sub-sentinel proc event))
+ ;; Check the exit status.  This will signal an error if the
+ ;; exit status is non-zero.
+ (notmuch-check-async-exit-status proc event real-command err-file)
+ ;; If that didn't signal an error, then any error output was
+ ;; really warning output.  Show warnings, if any.
+ (let ((warnings
+(with-temp-buffer
+  (unless (= (second (insert-file-contents err-file)) 0)
+(end-of-line)
+;; Show first line; stuff remaining lines in the
+;; errors buffer.
+(let ((l1 (buffer-substring (point-min) (point
+  (skip-chars-forward "\n")
+  (cons l1 (unless (eobp)
+

[PATCH 1/5] test: Remove extraneous Emacs error handling test

[Austin Clements]

We now check error handling more carefully in the last test in
test/emacs and we're about to add more error handling tests.  (This
was also a strange place for this test, since it had nothing to do
with large search buffers.)
---
 test/emacs-large-search-buffer |   11 ---
 1 file changed, 11 deletions(-)

diff --git a/test/emacs-large-search-buffer b/test/emacs-large-search-buffer
index 9dcbef5..8b1251f 100755
--- a/test/emacs-large-search-buffer
+++ b/test/emacs-large-search-buffer
@@ -29,15 +29,4 @@ test_emacs '(notmuch-search "*")
 sed -i -e s',  *, ,g' -e 's/xxx*/[BLOB]/g' OUTPUT
 test_expect_equal_file OUTPUT EXPECTED

-test_begin_subtest "Ensure that emacs doesn't drop error messages"
-test_emacs '(notmuch-search "--this-option-does-not-exist")
-   (notmuch-test-wait)
-   (test-output)'
-cat 

[PATCH 0/5] Make Emacs search use sexp format

[Austin Clements]

This series implements an incremental S-expression parser and switches
search over to it.  To simplify things, it also implements better
handing of stderr for asynchronous processes so we don't have to
handle errors embedded in the S-expression stream.

Re: [PATCH 3/5] emacs: Use async process helper for search

[Mark Walters]

I have only very briefly looked at this: it seems to not quite apply to
master (one fix up see below)

Also, as far as I can see condition-case-unless-debug (used in patch
2/5) is emacs 24 only.

Best wishes

Mark

On Sat, 18 May 2013, Austin Clements amdra...@mit.edu wrote:
 Previously, search started the async notmuch process directly.  Now,
 it uses `notmuch-start-notmuch'.  This simplifies the process sentinel
 a bit and means that we no longer have to worry about errors
 interleaved with the JSON output.

 We also update the tests of Emacs error handling, since the error
 output is now separated from the search results buffer.
 ---
  emacs/notmuch.el |   19 +--
  test/emacs   |   36 
  2 files changed, 37 insertions(+), 18 deletions(-)

 diff --git a/emacs/notmuch.el b/emacs/notmuch.el
 index 4c1a6ca..b8d9c44 100644
 --- a/emacs/notmuch.el
 +++ b/emacs/notmuch.el
 @@ -653,15 +653,8 @@ of the result.
   ;; For version mismatch, there's no point in
   ;; showing the search buffer
   (when (or (= exit-status 20) (= exit-status 21))
 -   (kill-buffer))
 - (condition-case err
 - (notmuch-check-async-exit-status proc msg)
 -   ;; Suppress the error signal since strange
 -   ;; things happen if a sentinel signals.  Mimic
 -   ;; the top-level's handling of error messages.
 -   (error
 -(message %s (error-message-string err))

This line is 
   (message %s (second err))
in master.


 -(throw 'return nil)))
 +   (kill-buffer)
 +   (throw 'return nil))
   (if (and atbob
(not (string= notmuch-search-target-thread 
 found)))
   (set 'never-found-target-thread t)
 @@ -938,10 +931,9 @@ Other optional parameters are used as follows:
(erase-buffer)
(goto-char (point-min))
(save-excursion
 - (let ((proc (start-process
 -  notmuch-search buffer
 -  notmuch-command search
 -  --format=json --format-version=1
 + (let ((proc (notmuch-start-notmuch
 +  notmuch-search buffer #'notmuch-search-process-sentinel
 +  search --format=json --format-version=1
(if oldest-first
--sort=oldest-first
  --sort=newest-first)
 @@ -951,7 +943,6 @@ Other optional parameters are used as follows:
 ;; should be called no matter how the process dies.
 (parse-buf (generate-new-buffer  *notmuch search parse*)))
 (process-put proc 'parse-buf parse-buf)
 -   (set-process-sentinel proc 'notmuch-search-process-sentinel)
 (set-process-filter proc 'notmuch-search-process-filter)
 (set-process-query-on-exit-flag proc nil
  (run-hooks 'notmuch-search-hook)))
 diff --git a/test/emacs b/test/emacs
 index f033bdf..d38ae8c 100755
 --- a/test/emacs
 +++ b/test/emacs
 @@ -853,11 +853,10 @@ test_expect_success Rendering HTML mail with images \
  'cat OUTPUT  grep -q smiley OUTPUT'
  
  
 -test_begin_subtest Search handles subprocess errors
 +test_begin_subtest Search handles subprocess error exit codes
  cat  notmuch_fail EOF
  #!/bin/sh
  echo This is output
 -echo This is an error 2
  exit 1
  EOF
  chmod a+x notmuch_fail
 @@ -874,8 +873,6 @@ sed -i -e 's/^\[.*\]$/[XXX]/' ERROR
  test_expect_equal $(cat OUTPUT; echo ---; cat MESSAGES; echo ---; cat 
 ERROR) \
  Error: Unexpected output from notmuch search:
  This is output
 -Error: Unexpected output from notmuch search:
 -This is an error
  End of search results.
  ---
  $PWD/notmuch_fail exited with status 1 (see *Notmuch errors* for more 
 details)
 @@ -885,4 +882,35 @@ $PWD/notmuch_fail exited with status 1
  command: $PWD/notmuch_fail search --format\=json --format-version\=1 
 --sort\=newest-first tag\:inbox
  exit status: 1
  
 +test_begin_subtest Search handles subprocess warnings
 +cat  notmuch_fail EOF
 +#!/bin/sh
 +echo This is output
 +echo This is a warning 2
 +echo This is another warning 2
 +exit 0
 +EOF
 +chmod a+x notmuch_fail
 +test_emacs (let ((notmuch-command \$PWD/notmuch_fail\))
 +(with-current-buffer \*Messages*\ (erase-buffer))
 +(with-current-buffer \*Notmuch errors*\ (erase-buffer))
 +(notmuch-search \tag:inbox\)
 +(notmuch-test-wait)
 +(with-current-buffer \*Messages*\
 +   (test-output \MESSAGES\))
 +(with-current-buffer \*Notmuch errors*\
 +   (test-output \ERROR\))
 +(test-output))
 +sed -i -e 's/^\[.*\]$/[XXX]/' ERROR
 +test_expect_equal $(cat OUTPUT; echo ---; cat MESSAGES; echo ---; cat 
 ERROR) \
 +Error: Unexpected output from notmuch search:
 +This is output
 

Re: [PATCH 3/5] emacs: Use async process helper for search

[David Bremner]

Mark Walters markwalters1...@gmail.com writes:

 -  (error
 -   (message %s (error-message-string err))

 This line is 
  (message %s (second err))
 in master.


This is a seperate patch sent to the list recently.

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: Solaris support - missing or incompatible functions

[David Bremner]

Vladimir Marek vladimir.ma...@oracle.com writes:

 
 There may be some prerequisites to running tests that you're missing,
 and the test setup fails to tell you about. A pastebin of the 'make
 test' output might be helpful.

 Ok, in that case please let me take a look at the tests first.


For what it's worth, I can confirm no test failures for me either on
linux (Debian wheezy).

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] emacs: Fix trimming regexp in notmuch-check-exit-status

[Tomi Ollila]

On Fri, May 17 2013, Austin Clements amdra...@mit.edu wrote:

 For such a simple regexp, this was broken in a very complicated way.

 The intent was to strip the newline (and potentially other whitespace)
 off the end of the error string so there wasn't an extra newline in
 the error signal.  However, the regexp was deeply dependent on the
 active syntax table and the subtleties of $.  We didn't notice this
 because all notmuch major modes put ?\n in the whitespace class, which
 makes this behaved as intended: the \\s  matches all newlines, but
 by matching the newline character, causes the $ *not* to match
 *except* where it matched the empty string at the very end of the
 string, which was not followed by a newline.

 However, if the syntax table declares ?\n to be non-whitespace
 (lisp-mode declares it as endcomment, and is likely to be the mode
 you're in when testing functions), then this regexp behaves completely
 differently, matching trailing spaces at the end of every line within
 the string.

 The solution is to say what we mean for whitespace *and* to switch
 from $ to \', which matches only the end of the string, rather than
 the end of each line.  Both are necessary or this will strip away
 interior newlines, which is not what we want.

LGTM.

Tomi


 ---
  emacs/notmuch-lib.el |2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

 diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
 index 790136e..59b1ce3 100644
 --- a/emacs/notmuch-lib.el
 +++ b/emacs/notmuch-lib.el
 @@ -438,7 +438,7 @@ You may need to restart Emacs or upgrade your notmuch 
 package.))
   (if err
   ;; We have an error message straight from the CLI.
   (notmuch-logged-error
 -  (replace-regexp-in-string \\s $  err) extra)
 +  (replace-regexp-in-string [ \n\r\t\f]*\\'  err) extra)
 ;; We only have combined output from the CLI; don't inundate
 ;; the user with it.  Mimic `process-lines'.
 (notmuch-logged-error (format %s exited with status %s
 -- 
 1.7.10.4

 ___
 notmuch mailing list
 notmuch@notmuchmail.org
 http://notmuchmail.org/mailman/listinfo/notmuch
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 1/1] NEWS: removal of backported support for Emacs version 22

[Tomi Ollila]

---
 NEWS | 13 +
 1 file changed, 13 insertions(+)

diff --git a/NEWS b/NEWS
index c348068..8d22e1b 100644
--- a/NEWS
+++ b/NEWS
@@ -18,6 +18,19 @@ Bash command-line completion
   `notmuch config`. The new completion support depends on the
   bash-completion package.
 
+Emacs Interface
+---
+
+Removal of backported Emacs version 23 functionality for Emacs 22
+
+  Two years ago some Emacs 23 functionality that notmuch used were
+  backported to notmuch to be conditionally used with Emacs version 22.
+  That worked for a few months until new Emacs 23 features were taken
+  into use and due to lack of interest no one continued to backport
+  those. To clean up dead code and to avoid confusion in new
+  developers these backports are now removed.
+
+
 Vim Front-End
 -
 
-- 
1.8.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 3/5] emacs: Use async process helper for search

[Austin Clements]

Quoth Mark Walters on May 18 at  8:14 am:
 
 I have only very briefly looked at this: it seems to not quite apply to
 master (one fix up see below)
 
 Also, as far as I can see condition-case-unless-debug (used in patch
 2/5) is emacs 24 only.

Indeed.  I'll just use condition-case in v2 (unless we want to
introduce a compatibility macro).

I really wish the Emacs documentation included what version functions
were introduced in.  I couldn't even *find* a copy of the non-latest
documentation; I wound up downloading a tarball of Emacs 23 and
building the documentation myself.

 Best wishes
 
 Mark
 
 On Sat, 18 May 2013, Austin Clements amdra...@mit.edu wrote:
  Previously, search started the async notmuch process directly.  Now,
  it uses `notmuch-start-notmuch'.  This simplifies the process sentinel
  a bit and means that we no longer have to worry about errors
  interleaved with the JSON output.
 
  We also update the tests of Emacs error handling, since the error
  output is now separated from the search results buffer.
  ---
   emacs/notmuch.el |   19 +--
   test/emacs   |   36 
   2 files changed, 37 insertions(+), 18 deletions(-)
 
  diff --git a/emacs/notmuch.el b/emacs/notmuch.el
  index 4c1a6ca..b8d9c44 100644
  --- a/emacs/notmuch.el
  +++ b/emacs/notmuch.el
  @@ -653,15 +653,8 @@ of the result.
  ;; For version mismatch, there's no point in
  ;; showing the search buffer
  (when (or (= exit-status 20) (= exit-status 21))
  - (kill-buffer))
  -   (condition-case err
  -   (notmuch-check-async-exit-status proc msg)
  - ;; Suppress the error signal since strange
  - ;; things happen if a sentinel signals.  Mimic
  - ;; the top-level's handling of error messages.
  - (error
  -  (message %s (error-message-string err))
 
 This line is 
  (message %s (second err))
 in master.

Sorry, I thought these patches were independent from the ones I'd sent
earlier, but obviously not.  It should apply now that David has
applied the other patch.

  -  (throw 'return nil)))
  + (kill-buffer)
  + (throw 'return nil))
  (if (and atbob
   (not (string= notmuch-search-target-thread 
  found)))
  (set 'never-found-target-thread t)
  @@ -938,10 +931,9 @@ Other optional parameters are used as follows:
 (erase-buffer)
 (goto-char (point-min))
 (save-excursion
  -   (let ((proc (start-process
  -notmuch-search buffer
  -notmuch-command search
  ---format=json --format-version=1
  +   (let ((proc (notmuch-start-notmuch
  +notmuch-search buffer #'notmuch-search-process-sentinel
  +search --format=json --format-version=1
   (if oldest-first
   --sort=oldest-first
 --sort=newest-first)
  @@ -951,7 +943,6 @@ Other optional parameters are used as follows:
;; should be called no matter how the process dies.
(parse-buf (generate-new-buffer  *notmuch search parse*)))
(process-put proc 'parse-buf parse-buf)
  - (set-process-sentinel proc 'notmuch-search-process-sentinel)
(set-process-filter proc 'notmuch-search-process-filter)
(set-process-query-on-exit-flag proc nil
   (run-hooks 'notmuch-search-hook)))
  diff --git a/test/emacs b/test/emacs
  index f033bdf..d38ae8c 100755
  --- a/test/emacs
  +++ b/test/emacs
  @@ -853,11 +853,10 @@ test_expect_success Rendering HTML mail with images 
  \
   'cat OUTPUT  grep -q smiley OUTPUT'
   
   
  -test_begin_subtest Search handles subprocess errors
  +test_begin_subtest Search handles subprocess error exit codes
   cat  notmuch_fail EOF
   #!/bin/sh
   echo This is output
  -echo This is an error 2
   exit 1
   EOF
   chmod a+x notmuch_fail
  @@ -874,8 +873,6 @@ sed -i -e 's/^\[.*\]$/[XXX]/' ERROR
   test_expect_equal $(cat OUTPUT; echo ---; cat MESSAGES; echo ---; cat 
  ERROR) \
   Error: Unexpected output from notmuch search:
   This is output
  -Error: Unexpected output from notmuch search:
  -This is an error
   End of search results.
   ---
   $PWD/notmuch_fail exited with status 1 (see *Notmuch errors* for more 
  details)
  @@ -885,4 +882,35 @@ $PWD/notmuch_fail exited with status 1
   command: $PWD/notmuch_fail search --format\=json --format-version\=1 
  --sort\=newest-first tag\:inbox
   exit status: 1
   
  +test_begin_subtest Search handles subprocess warnings
  +cat  notmuch_fail EOF
  +#!/bin/sh
  +echo This is output
  +echo This is a warning 2
  +echo This is another warning 2
  +exit 0
  +EOF
  +chmod a+x notmuch_fail
  +test_emacs (let ((notmuch-command \$PWD/notmuch_fail\))
  +  

Re: [PATCH 1/1] NEWS: removal of backported support for Emacs version 22

[David Bremner]

Tomi Ollila tomi.oll...@iki.fi writes:

 ---
  NEWS | 13 +
  
 +Emacs Interface
 +---
 +
 +Removal of backported Emacs version 23 functionality for Emacs 22
 +
 +  Two years ago some Emacs 23 functionality that notmuch used were
 +  backported to notmuch to be conditionally used with Emacs version 22.
 +  That worked for a few months until new Emacs 23 features were taken
 +  into use and due to lack of interest no one continued to backport
 +  those. To clean up dead code and to avoid confusion in new
 +  developers these backports are now removed.

Hi Tomi;

What do you think about a more user-focussed item, something like 

,
| No Emacs 22 support
| ---
| 
| We started some Emacs 22 support, but it never really worked, and the
| last remnants have been removed from this release
`

but better, of course ;).

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch