[PATCH 2/3] notmuch-mutt: support for messages that lack Message-ID headers

2015-01-24 Thread Jani Nikula
On Sat, 24 Jan 2015, "Jan N. Klug"  wrote:
> On Sat, Jan 24, 2015 at 04:21:06PM +0100, Stefano Zacchiroli wrote:
>> On Sat, Jan 24, 2015 at 04:59:16PM +0200, Jani Nikula wrote:
>> > On Sat, 24 Jan 2015, Stefano Zacchiroli  wrote:
>> > > To do the above, rewrite get_message_id() to scan the current message
>> > > line by line, incrementally computing a SHA1. As a consequence, drop
>> > > the dependency on Mail::Internet.
>> > 
>> > I am not so sure this is a good idea however, see below.
>> [...]
>> But I didn't think of header folding, and you're absolutely correct in
>> saying that might be a problem.
>
> I disagree. I do not think that header folding is a problem here. RFC
> 5322 and RFC 2822 state that FWS in the message-id are not allowed (as
> opposed to In-Reply: and other headers, where FWS may occur between, but
> not inside message-ids).

Not *inside* message-ids, but as a data point, 0.05% of my email have
FWS *before* the message-id. That's more than the amount of messages
I have without a message-id.

BR,
Jani.


[PATCH 2/3] notmuch-mutt: support for messages that lack Message-ID headers

2015-01-24 Thread Jan N. Klug
On Sat, Jan 24, 2015 at 04:21:06PM +0100, Stefano Zacchiroli wrote:
> On Sat, Jan 24, 2015 at 04:59:16PM +0200, Jani Nikula wrote:
> > On Sat, 24 Jan 2015, Stefano Zacchiroli  wrote:
> > > To do the above, rewrite get_message_id() to scan the current message
> > > line by line, incrementally computing a SHA1. As a consequence, drop
> > > the dependency on Mail::Internet.
> > 
> > I am not so sure this is a good idea however, see below.
> [...]
> But I didn't think of header folding, and you're absolutely correct in
> saying that might be a problem.

I disagree. I do not think that header folding is a problem here. RFC
5322 and RFC 2822 state that FWS in the message-id are not allowed (as
opposed to In-Reply: and other headers, where FWS may occur between, but
not inside message-ids).

Folded lines that start with "Message-Id:" might occur, the
regex-pattern used in the patch will not match those lines. 

As far as I have looked in the Mail::Internet sources, further checking of the
correctness of the message-id does not occur, so I do not see any
advantage over the proposed solution. 

> Jan: do you agree with using Mail::Header->new and fall back to
> line-by-line hasing only in case Message-ID is not found? If so, having
> an updated patch based on the one I've posted here would be awesome! If
> you cannot do that just let me know and I'll get to it, eventually :).

I can look into that, but I cannot promise to have it ready in the next
days.

Regards,

Jan
--
Jan N. Klug, Gelsenkirchen


[PATCH 2/3] notmuch-mutt: support for messages that lack Message-ID headers

2015-01-24 Thread Jani Nikula
On Sat, 24 Jan 2015, Stefano Zacchiroli  wrote:
> From: "Jan N. Klug" 
>
> For those messages, compute a synthetic Message-ID based on the SHA1
> of the whole message, in the same way that notmuch would do. See:
> http://git.notmuchmail.org/git/notmuch/blob/HEAD:/lib/sha1.c

As I said on IRC, I think this is a notmuch implementation detail, and
we don't make any promises about always generating missing message-ids
the same way. That said, I don't see any reason why we'd change this
anytime soon, so the solution is probably good enough for now.

> To do the above, rewrite get_message_id() to scan the current message
> line by line, incrementally computing a SHA1. As a consequence, drop
> the dependency on Mail::Internet.

I am not so sure this is a good idea however, see below.

> Signed-off-by: Stefano Zacchiroli 
> ---
>  contrib/notmuch-mutt/README   |  4 ++--
>  contrib/notmuch-mutt/notmuch-mutt | 23 ++-
>  2 files changed, 20 insertions(+), 7 deletions(-)
>
> diff --git a/contrib/notmuch-mutt/README b/contrib/notmuch-mutt/README
> index c661447..0013ed0 100644
> --- a/contrib/notmuch-mutt/README
> +++ b/contrib/notmuch-mutt/README
> @@ -33,10 +33,10 @@ Requirements
>  
>  To *run* notmuch-mutt you will need Perl with the following libraries:
>  
> +- Digest::SHA 
> +  (Debian package: libdigest-sha-perl)
>  - Mail::Box 
>(Debian package: libmail-box-perl)
> -- Mail::Internet 
> -  (Debian package: libmailtools-perl)
>  - String::ShellQuote 
>(Debian package: libstring-shellquote-perl)
>  - Term::ReadLine::Gnu 
> diff --git a/contrib/notmuch-mutt/notmuch-mutt 
> b/contrib/notmuch-mutt/notmuch-mutt
> index 4969e4b..4d30b0b 100755
> --- a/contrib/notmuch-mutt/notmuch-mutt
> +++ b/contrib/notmuch-mutt/notmuch-mutt
> @@ -13,11 +13,11 @@ use warnings;
>  
>  use File::Path;
>  use Getopt::Long qw(:config no_getopt_compat);
> -use Mail::Internet;
>  use Mail::Box::Maildir;
>  use Pod::Usage;
>  use String::ShellQuote;
>  use Term::ReadLine;
> +use Digest::SHA;
>  
>  
>  my $xdg_cache_dir = "$ENV{HOME}/.cache";
> @@ -75,10 +75,23 @@ sub prompt($$) {
>  }
>  
>  sub get_message_id() {
> -my $mail = Mail::Internet->new(\*STDIN);
> -my $mid = $mail->head->get("message-id") or return undef;
> -$mid =~ /^<(.*)>$/;  # get message-id value
> -return $1;
> +my $mid = undef;
> +my $sha = Digest::SHA->new(1);  # SHA1 hashing
> +
> +while() {  # scan message line by line, looking for mid
> +if ($_ =~ /^Message-ID:\s*<(.*)>$/i) {

This doesn't take into account header folding or end of headers. There's
probably a bunch of other things to consider in the relevant RFCs. I
think you really should use Mail::Internet for the common case, and only
fallback to generating the message-id when that fails.

BR,
Jani.

> + $mid = $1;
> + last;  # message-id found, abort scan
> +}
> + $sha->add($_);  # update hash
> +}
> +
> +# Generate message-id from hash if none was found, in the same way
> +# that notmuch would do.
> +# See: http://git.notmuchmail.org/git/notmuch/blob/HEAD:/lib/sha1.c
> +$mid ||= "notmuch-sha1-".$sha->hexdigest;
> +
> +return $mid;
>  }
>  
>  sub search_action($$$@) {
> -- 
> 2.1.4
>
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH v4 0/5] Index the content-type of MIME parts

2015-01-24 Thread David Bremner
Todd  writes:

> I think I've finished incorporating the feedback.  The final
> notmuch-search-terms.rst could use more details, but it should
> probably occur after the recent patch that was posted documenting the
> probablistic indexing/searching has been committed.

Pushed this version, with amended NEWS, to master.

d


[PATCH 2/3] notmuch-mutt: support for messages that lack Message-ID headers

2015-01-24 Thread Stefano Zacchiroli
First of all thanks for your feedback, Jani!
(both here, and the other day on IRC, of course)

On Sat, Jan 24, 2015 at 04:59:16PM +0200, Jani Nikula wrote:
> On Sat, 24 Jan 2015, Stefano Zacchiroli  wrote:
> >
> > For those messages, compute a synthetic Message-ID based on the SHA1
> > of the whole message, in the same way that notmuch would do. See:
> > http://git.notmuchmail.org/git/notmuch/blob/HEAD:/lib/sha1.c
> 
> As I said on IRC, I think this is a notmuch implementation detail, and
> we don't make any promises about always generating missing message-ids
> the same way. That said, I don't see any reason why we'd change this
> anytime soon, so the solution is probably good enough for now.

ACK. I've noted down the code URL in the patch for reference, and also
as a warning to keep an eye on.

> > To do the above, rewrite get_message_id() to scan the current message
> > line by line, incrementally computing a SHA1. As a consequence, drop
> > the dependency on Mail::Internet.
> 
> I am not so sure this is a good idea however, see below.

So, fun part here is that I had initially made a similar comment to Jan.
Then got convinced to do as the current patch does to avoid loading full
messages (potentially with large attachments and the like) in memory.

But I didn't think of header folding, and you're absolutely correct in
saying that might be a problem.

I've just found out Mail::Header->new which I believe solves both
problems: avoid loading the full message into memory (it will only load
the headers) and proper parsing folded Message-IDs, without having to
use more complex regexp hackery.

Jan: do you agree with using Mail::Header->new and fall back to
line-by-line hasing only in case Message-ID is not found? If so, having
an updated patch based on the one I've posted here would be awesome! If
you cannot do that just let me know and I'll get to it, eventually :).

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  zack at upsilon.cc . . . . o . . . o . o
Ma?tre de conf?rences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
? the first rule of tautology club is the first rule of tautology club ?


[PATCH v2 8/8] emacs: Support cid: references with shr renderer

2015-01-24 Thread Austin Clements
shr has really nice support for inline image rendering, but previously
we only had the hooks for w3m cid: references.
---
 emacs/notmuch-show.el | 45 +
 1 file changed, 37 insertions(+), 8 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index 34dcedd..66350d4 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -767,14 +767,43 @@ (defun 
notmuch-show-get-mime-type-of-application/octet-stream (part)
  nil

 (defun notmuch-show-insert-part-text/html (msg part content-type nth depth 
button)
-  ;; text/html handler to work around bugs in renderers and our
-  ;; invisibile parts code. In particular w3m sets up a keymap which
-  ;; "leaks" outside the invisible region and causes strange effects
-  ;; in notmuch. We set mm-inline-text-html-with-w3m-keymap to nil to
-  ;; tell w3m not to set a keymap (so the normal notmuch-show-mode-map
-  ;; remains).
-  (let ((mm-inline-text-html-with-w3m-keymap nil))
-(notmuch-show-insert-part-*/* msg part content-type nth depth button)))
+  (if (eq mm-text-html-renderer 'shr)
+  ;; It's easier to drive shr ourselves than to work around the
+  ;; goofy things `mm-shr' does (like irreversibly taking over
+  ;; content ID handling).
+  (notmuch-show--insert-part-text/html-shr msg part)
+;; Otherwise, let message-mode do the heavy lifting
+;;
+;; w3m sets up a keymap which "leaks" outside the invisible region
+;; and causes strange effects in notmuch. We set
+;; mm-inline-text-html-with-w3m-keymap to nil to tell w3m not to
+;; set a keymap (so the normal notmuch-show-mode-map remains).
+(let ((mm-inline-text-html-with-w3m-keymap nil))
+  (notmuch-show-insert-part-*/* msg part content-type nth depth button
+
+;; These functions are used by notmuch-show--insert-part-text/html-shr
+(declare-function libxml-parse-html-region "xml.c")
+(declare-function shr-insert-document "shr")
+
+(defun notmuch-show--insert-part-text/html-shr (msg part)
+  ;; Make sure shr is loaded before we start let-binding its globals
+  (require 'shr)
+  (let ((dom (let ((process-crypto notmuch-show-process-crypto))
+  (with-temp-buffer
+(insert (notmuch-get-bodypart-text msg part process-crypto))
+(libxml-parse-html-region (point-min) (point-max)
+   (shr-content-function
+(lambda (url)
+  ;; shr strips the "cid:" part of URL, but doesn't
+  ;; URL-decode it (see RFC 2392).
+  (let ((cid (url-unhex-string url)))
+(first (notmuch-show--get-cid-content cid)
+   ;; Block all external images to prevent privacy leaks and
+   ;; potential attacks.  FIXME: If we block an image, offer a
+   ;; button to load external images.
+   (shr-blocked-images "."))
+(shr-insert-document dom)
+t))

 (defun notmuch-show-insert-part-*/* (msg part content-type nth depth button)
   ;; This handler _must_ succeed - it is the handler of last resort.
-- 
2.1.3



[PATCH v2 7/8] emacs: Rewrite content ID handling

2015-01-24 Thread Austin Clements
Besides generally cleaning up the code and separating the general
content ID handling from the w3m-specific code, this fixes several
problems.

Foremost is that, previously, the code roughly assumed that referenced
parts would be in the same multipart/related as the reference.
According to RFC 2392, nothing could be further from the truth:
content IDs are supposed to be globally unique and globally
addressable.  This is nonsense, but this patch at least fixes things
so content IDs can be anywhere in the same message.

As a side-effect of the above, this handles multipart/alternate
content-IDs more in line with RFC 2046 section 5.1.2 (not that I've
ever seen this in the wild).  This also properly URL-decodes cid:
URLs, as per RFC 2392 (the previous code did not), and applies crypto
settings from the show buffer (the previous code used the global
crypto settings).
---
 emacs/notmuch-show.el | 120 +++---
 1 file changed, 74 insertions(+), 46 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index 11eac5f..34dcedd 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -525,6 +525,73 @@ (defun notmuch-show-toggle-part-invisibility ( 
button)
  (overlay-put overlay 'invisible (not show))
  t)

+;; Part content ID handling
+
+(defvar notmuch-show--cids nil
+  "Alist from raw content ID to (MSG PART).")
+(make-variable-buffer-local 'notmuch-show--cids)
+
+(defun notmuch-show--register-cids (msg part)
+  "Register content-IDs in PART and all of PART's sub-parts."
+  (let ((content-id (plist-get part :content-id)))
+(when content-id
+  ;; Note that content-IDs are globally unique, except when they
+  ;; aren't: RFC 2046 section 5.1.4 permits children of a
+  ;; multipart/alternative to have the same content-ID, in which
+  ;; case the MUA is supposed to pick the best one it can render.
+  ;; We simply add the content-ID to the beginning of our alist;
+  ;; so if this happens, we'll take the last (and "best")
+  ;; alternative (even if we can't render it).
+  (push (list content-id msg part) notmuch-show--cids)))
+  ;; Recurse on sub-parts
+  (let ((ctype (notmuch-split-content-type
+   (downcase (plist-get part :content-type)
+(cond ((equal (first ctype) "multipart")
+  (mapc (apply-partially #'notmuch-show--register-cids msg)
+(plist-get part :content)))
+ ((equal ctype '("message" "rfc822"))
+  (notmuch-show--register-cids
+   msg
+   (first (plist-get (first (plist-get part :content)) :body)))
+
+(defun notmuch-show--get-cid-content (cid)
+  "Return a list (CID-content content-type) or nil.
+
+This will only find parts from messages that have been inserted
+into the current buffer.  CID must be a raw content ID, without
+enclosing angle brackets, a cid: prefix, or URL encoding.  This
+will return nil if the CID is unknown or cannot be retrieved."
+  (let ((descriptor (cdr (assoc cid notmuch-show--cids
+(when descriptor
+  (let* ((msg (first descriptor))
+(part (second descriptor))
+;; Request caching for this content, as some messages
+;; reference the same cid: part many times (hundreds!).
+(content (notmuch-get-bodypart-binary
+  msg part notmuch-show-process-crypto 'cache))
+(content-type (plist-get part :content-type)))
+   (list content content-type)
+
+(defun notmuch-show-setup-w3m ()
+  "Instruct w3m how to retrieve content from a \"related\" part of a message."
+  (interactive)
+  (if (boundp 'w3m-cid-retrieve-function-alist)
+(unless (assq 'notmuch-show-mode w3m-cid-retrieve-function-alist)
+  (push (cons 'notmuch-show-mode #'notmuch-show--cid-w3m-retrieve)
+   w3m-cid-retrieve-function-alist)))
+  (setq mm-inline-text-html-with-images t))
+
+(defvar w3m-current-buffer) ;; From `w3m.el'.
+(defun notmuch-show--cid-w3m-retrieve (url  args)
+  ;; url includes the cid: prefix and is URL encoded (see RFC 2392).
+  (let* ((cid (url-unhex-string (substring url 4)))
+(content-and-type
+ (with-current-buffer w3m-current-buffer
+   (notmuch-show--get-cid-content cid
+(when content-and-type
+  (insert (first content-and-type))
+  (second content-and-type
+
 ;; MIME part renderers

 (defun notmuch-show-multipart/*-to-list (part)
@@ -549,56 +616,11 @@ (defun notmuch-show-insert-part-multipart/alternative 
(msg part content-type nth
   (indent-rigidly start (point) 1)))
   t)

-(defun notmuch-show-setup-w3m ()
-  "Instruct w3m how to retrieve content from a \"related\" part of a message."
-  (interactive)
-  (if (boundp 'w3m-cid-retrieve-function-alist)
-(unless (assq 'notmuch-show-mode w3m-cid-retrieve-function-alist)
-  (push (cons 'notmuch-show-mode 'notmuch-show-w3m-cid-retrieve)
-   w3m-cid-retrieve-function-alist)))
-  (setq 

[PATCH v2 6/8] emacs: Use generalized content caching in w3m CID code

2015-01-24 Thread Austin Clements
Previously this did its own caching, but this is now supported by more
generally by `notmuch-get-bodypart-binary'.
---
 emacs/notmuch-show.el | 23 ---
 1 file changed, 8 insertions(+), 15 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index f29428a..11eac5f 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -562,15 +562,14 @@ (defvar w3m-current-buffer) ;; From `w3m.el'.
 (defvar notmuch-show-w3m-cid-store nil)
 (make-variable-buffer-local 'notmuch-show-w3m-cid-store)

-(defun notmuch-show-w3m-cid-store-internal (content-id msg part content)
-  (push (list content-id msg part content)
-   notmuch-show-w3m-cid-store))
+(defun notmuch-show-w3m-cid-store-internal (content-id msg part)
+  (push (list content-id msg part) notmuch-show-w3m-cid-store))

 (defun notmuch-show-w3m-cid-store (msg part)
   (let ((content-id (plist-get part :content-id)))
 (when content-id
   (notmuch-show-w3m-cid-store-internal (concat "cid:" content-id)
-  msg part nil
+  msg part

 (defun notmuch-show-w3m-cid-retrieve (url  args)
   (let ((matching-part (with-current-buffer w3m-current-buffer
@@ -578,18 +577,12 @@ (defun notmuch-show-w3m-cid-retrieve (url  args)
 (if matching-part
(let* ((msg (nth 1 matching-part))
   (part (nth 2 matching-part))
-  (content (nth 3 matching-part))
   (content-type (plist-get part :content-type)))
- ;; If we don't already have the content, get it and cache
- ;; it, as some messages reference the same cid: part many
- ;; times (hundreds!), which results in many calls to
- ;; `notmuch part'.
- (unless content
-   (setq content (notmuch-get-bodypart-binary
-  msg part notmuch-show-process-crypto))
-   (with-current-buffer w3m-current-buffer
- (notmuch-show-w3m-cid-store-internal url msg part content)))
- (insert content)
+ ;; Request content caching, as some messages reference the
+ ;; same cid: part many times (hundreds!), which results in
+ ;; many calls to `notmuch show'.
+ (insert (notmuch-get-bodypart-binary
+  msg part notmuch-show-process-crypto 'cache))
  content-type)
   nil)))

-- 
2.1.3



[PATCH v2 5/8] emacs: Support caching in notmuch-get-bodypart-{binary, text}

2015-01-24 Thread Austin Clements
(The actual code change here is small, but requires re-indenting
existing code.)
---
 emacs/notmuch-lib.el | 64 
 1 file changed, 40 insertions(+), 24 deletions(-)

diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
index 3154725..f8e5165 100644
--- a/emacs/notmuch-lib.el
+++ b/emacs/notmuch-lib.el
@@ -529,39 +529,53 @@ (defun notmuch-parts-filter-by-type (parts type)
(lambda (part) (notmuch-match-content-type (plist-get part :content-type) 
type))
parts))

-(defun notmuch-get-bodypart-binary (msg part process-crypto)
+(defun notmuch-get-bodypart-binary (msg part process-crypto  cache)
   "Return the unprocessed content of PART in MSG as a unibyte string.

 This returns the \"raw\" content of the given part after content
 transfer decoding, but with no further processing (see the
 discussion of --format=raw in man notmuch-show).  In particular,
-this does no charset conversion."
-  (let ((args `("show" "--format=raw"
-   ,(format "--part=%d" (plist-get part :id))
-   ,@(when process-crypto '("--decrypt"))
-   ,(notmuch-id-to-query (plist-get msg :id)
-(with-temp-buffer
-  ;; Emacs internally uses a UTF-8-like multibyte string
-  ;; representation by default (regardless of the coding system,
-  ;; which only affects how it goes from outside data to this
-  ;; internal representation).  This *almost* never matters.
-  ;; Annoyingly, it does matter if we use this data in an image
-  ;; descriptor, since Emacs will use its internal data buffer
-  ;; directly and this multibyte representation corrupts binary
-  ;; image formats.  Since the caller is asking for binary data, a
-  ;; unibyte string is a more appropriate representation anyway.
-  (set-buffer-multibyte nil)
-  (let ((coding-system-for-read 'no-conversion))
-   (apply #'call-process notmuch-command nil '(t nil) nil args)
-   (buffer-string)
-
-(defun notmuch-get-bodypart-text (msg part process-crypto)
+this does no charset conversion.
+
+If CACHE is non-nil, the content of this part will be saved in
+MSG (if it isn't already)."
+  (let ((data (plist-get part :binary-content)))
+(when (not data)
+  (let ((args `("show" "--format=raw"
+   ,(format "--part=%d" (plist-get part :id))
+   ,@(when process-crypto '("--decrypt"))
+   ,(notmuch-id-to-query (plist-get msg :id)
+   (with-temp-buffer
+ ;; Emacs internally uses a UTF-8-like multibyte string
+ ;; representation by default (regardless of the coding
+ ;; system, which only affects how it goes from outside data
+ ;; to this internal representation).  This *almost* never
+ ;; matters.  Annoyingly, it does matter if we use this data
+ ;; in an image descriptor, since Emacs will use its internal
+ ;; data buffer directly and this multibyte representation
+ ;; corrupts binary image formats.  Since the caller is
+ ;; asking for binary data, a unibyte string is a more
+ ;; appropriate representation anyway.
+ (set-buffer-multibyte nil)
+ (let ((coding-system-for-read 'no-conversion))
+   (apply #'call-process notmuch-command nil '(t nil) nil args)
+   (setq data (buffer-string)
+  (when cache
+   ;; Cheat.  part is non-nil, and `plist-put' always modifies
+   ;; the list in place if it's non-nil.
+   (plist-put part :binary-content data)))
+data))
+
+(defun notmuch-get-bodypart-text (msg part process-crypto  cache)
   "Return the text content of PART in MSG.

 This returns the content of the given part as a multibyte Lisp
 string after performing content transfer decoding and any
 necessary charset decoding.  It is an error to use this for
-non-text/* parts."
+non-text/* parts.
+
+If CACHE is non-nil, the content of this part will be saved in
+MSG (if it isn't already)."
   (let ((content (plist-get part :content)))
 (when (not content)
   ;; Use show --format=sexp to fetch decoded content
@@ -572,7 +586,9 @@ (defun notmuch-get-bodypart-text (msg part process-crypto)
 (npart (apply #'notmuch-call-notmuch-sexp args)))
(setq content (plist-get npart :content))
(when (not content)
- (error "Internal error: No :content from %S" args
+ (error "Internal error: No :content from %S" args)))
+  (when cache
+   (plist-put part :content content)))
 content))

 ;; Workaround: The call to `mm-display-part' below triggers a bug in
-- 
2.1.3



[PATCH v2 4/8] emacs: Return unibyte strings for binary part data

2015-01-24 Thread Austin Clements
Unibyte strings are meant for representing binary data.  In practice,
using unibyte versus multibyte strings affects *almost* nothing.  It
does happen to matter if we use the binary data in an image descriptor
(which is, helpfully, not documented anywhere and getting it wrong
results in opaque errors like "Not a PNG image: ").
---
 emacs/notmuch-lib.el | 12 +++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
index a0a95d8..3154725 100644
--- a/emacs/notmuch-lib.el
+++ b/emacs/notmuch-lib.el
@@ -530,7 +530,7 @@ (defun notmuch-parts-filter-by-type (parts type)
parts))

 (defun notmuch-get-bodypart-binary (msg part process-crypto)
-  "Return the unprocessed content of PART in MSG.
+  "Return the unprocessed content of PART in MSG as a unibyte string.

 This returns the \"raw\" content of the given part after content
 transfer decoding, but with no further processing (see the
@@ -541,6 +541,16 @@ (defun notmuch-get-bodypart-binary (msg part 
process-crypto)
,@(when process-crypto '("--decrypt"))
,(notmuch-id-to-query (plist-get msg :id)
 (with-temp-buffer
+  ;; Emacs internally uses a UTF-8-like multibyte string
+  ;; representation by default (regardless of the coding system,
+  ;; which only affects how it goes from outside data to this
+  ;; internal representation).  This *almost* never matters.
+  ;; Annoyingly, it does matter if we use this data in an image
+  ;; descriptor, since Emacs will use its internal data buffer
+  ;; directly and this multibyte representation corrupts binary
+  ;; image formats.  Since the caller is asking for binary data, a
+  ;; unibyte string is a more appropriate representation anyway.
+  (set-buffer-multibyte nil)
   (let ((coding-system-for-read 'no-conversion))
(apply #'call-process notmuch-command nil '(t nil) nil args)
(buffer-string)
-- 
2.1.3



[PATCH v2 2/8] emacs: Create an API for fetching parts as undecoded binary

2015-01-24 Thread Austin Clements
The new function, `notmuch-get-bodypart-binary', replaces
`notmuch-get-bodypart-internal'.  Whereas the old function was really
meant for internal use in `notmuch-get-bodypart-content', it was used
in a few other places.  Since the difference between
`notmuch-get-bodypart-content' and `notmuch-get-bodypart-internal' was
unclear, these other uses were always confusing and potentially
inconsistent.  The new call clearly requests the part as undecoded
binary.

This is step 1 of 2 in separating `notmuch-get-bodypart-content' into
two APIs for retrieving either undecoded binary or decoded text.
---
 emacs/notmuch-lib.el  | 28 ++--
 emacs/notmuch-show.el | 22 +-
 2 files changed, 23 insertions(+), 27 deletions(-)

diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
index fd25f7c..d4b6684 100644
--- a/emacs/notmuch-lib.el
+++ b/emacs/notmuch-lib.el
@@ -529,25 +529,25 @@ (defun notmuch-parts-filter-by-type (parts type)
(lambda (part) (notmuch-match-content-type (plist-get part :content-type) 
type))
parts))

-;; Helper for parts which are generally not included in the default
-;; SEXP output.
-(defun notmuch-get-bodypart-internal (query part-number process-crypto)
-  (let ((args '("show" "--format=raw"))
-   (part-arg (format "--part=%s" part-number)))
-(setq args (append args (list part-arg)))
-(if process-crypto
-   (setq args (append args '("--decrypt"
-(setq args (append args (list query)))
+(defun notmuch-get-bodypart-binary (msg part process-crypto)
+  "Return the unprocessed content of PART in MSG.
+
+This returns the \"raw\" content of the given part after content
+transfer decoding, but with no further processing (see the
+discussion of --format=raw in man notmuch-show).  In particular,
+this does no charset conversion."
+  (let ((args `("show" "--format=raw"
+   ,(format "--part=%d" (plist-get part :id))
+   ,@(when process-crypto '("--decrypt"))
+   ,(notmuch-id-to-query (plist-get msg :id)
 (with-temp-buffer
   (let ((coding-system-for-read 'no-conversion))
-   (progn
- (apply 'call-process (append (list notmuch-command nil (list t nil) 
nil) args))
- (buffer-string))
+   (apply #'call-process notmuch-command nil '(t nil) nil args)
+   (buffer-string)

 (defun notmuch-get-bodypart-content (msg part process-crypto)
   (or (plist-get part :content)
-  (notmuch-get-bodypart-internal (notmuch-id-to-query (plist-get msg :id))
-(plist-get part :id) process-crypto)))
+  (notmuch-get-bodypart-binary msg part process-crypto)))

 ;; Workaround: The call to `mm-display-part' below triggers a bug in
 ;; Emacs 24 if it attempts to use the shr renderer to display an HTML
diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index df2389e..b3e339e 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -579,16 +579,14 @@ (defun notmuch-show-w3m-cid-retrieve (url  args)
(let* ((msg (nth 1 matching-part))
   (part (nth 2 matching-part))
   (content (nth 3 matching-part))
-  (message-id (plist-get msg :id))
-  (part-number (plist-get part :id))
   (content-type (plist-get part :content-type)))
  ;; If we don't already have the content, get it and cache
  ;; it, as some messages reference the same cid: part many
  ;; times (hundreds!), which results in many calls to
  ;; `notmuch part'.
  (unless content
-   (setq content (notmuch-get-bodypart-internal (notmuch-id-to-query 
message-id)
- part-number 
notmuch-show-process-crypto))
+   (setq content (notmuch-get-bodypart-binary
+  msg part notmuch-show-process-crypto))
(with-current-buffer w3m-current-buffer
  (notmuch-show-w3m-cid-store-internal url msg part content)))
  (insert content)
@@ -2162,15 +2160,14 @@ (defun notmuch-show-stash-git-send-email ( 
no-in-reply-to)

 ;; Interactive part functions and their helpers

-(defun notmuch-show-generate-part-buffer (message-id nth)
+(defun notmuch-show-generate-part-buffer (msg part)
   "Return a temporary buffer containing the specified part's content."
   (let ((buf (generate-new-buffer " *notmuch-part*"))
(process-crypto notmuch-show-process-crypto))
 (with-current-buffer buf
-  (setq notmuch-show-process-crypto process-crypto)
-  ;; Always acquires the part via `notmuch part', even if it is
-  ;; available in the SEXP output.
-  (insert (notmuch-get-bodypart-internal message-id nth 
notmuch-show-process-crypto)))
+  ;; This is always used in the content of mm handles, which
+  ;; expect undecoded, binary part content.
+  (insert (notmuch-get-bodypart-binary msg part process-crypto)))
 buf))

 (defun 

[PATCH v2 1/8] emacs: Track full message and part descriptor in w3m CID store

2015-01-24 Thread Austin Clements
This will simplify later changes.
---
 emacs/notmuch-show.el | 33 ++---
 1 file changed, 10 insertions(+), 23 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index 87b4881..df2389e 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -562,35 +562,26 @@ (defvar w3m-current-buffer) ;; From `w3m.el'.
 (defvar notmuch-show-w3m-cid-store nil)
 (make-variable-buffer-local 'notmuch-show-w3m-cid-store)

-(defun notmuch-show-w3m-cid-store-internal (content-id
-   message-id
-   part-number
-   content-type
-   content)
-  (push (list content-id
- message-id
- part-number
- content-type
- content)
+(defun notmuch-show-w3m-cid-store-internal (content-id msg part content)
+  (push (list content-id msg part content)
notmuch-show-w3m-cid-store))

 (defun notmuch-show-w3m-cid-store (msg part)
   (let ((content-id (plist-get part :content-id)))
 (when content-id
   (notmuch-show-w3m-cid-store-internal (concat "cid:" content-id)
-  (plist-get msg :id)
-  (plist-get part :id)
-  (plist-get part :content-type)
-  nil
+  msg part nil

 (defun notmuch-show-w3m-cid-retrieve (url  args)
   (let ((matching-part (with-current-buffer w3m-current-buffer
 (assoc url notmuch-show-w3m-cid-store
 (if matching-part
-   (let ((message-id (nth 1 matching-part))
- (part-number (nth 2 matching-part))
- (content-type (nth 3 matching-part))
- (content (nth 4 matching-part)))
+   (let* ((msg (nth 1 matching-part))
+  (part (nth 2 matching-part))
+  (content (nth 3 matching-part))
+  (message-id (plist-get msg :id))
+  (part-number (plist-get part :id))
+  (content-type (plist-get part :content-type)))
  ;; If we don't already have the content, get it and cache
  ;; it, as some messages reference the same cid: part many
  ;; times (hundreds!), which results in many calls to
@@ -599,11 +590,7 @@ (defun notmuch-show-w3m-cid-retrieve (url  args)
(setq content (notmuch-get-bodypart-internal (notmuch-id-to-query 
message-id)
  part-number 
notmuch-show-process-crypto))
(with-current-buffer w3m-current-buffer
- (notmuch-show-w3m-cid-store-internal url
-  message-id
-  part-number
-  content-type
-  content)))
+ (notmuch-show-w3m-cid-store-internal url msg part content)))
  (insert content)
  content-type)
   nil)))
-- 
2.1.3



[PATCH v2 0/8] Improve charset and cid: handling

2015-01-24 Thread Austin Clements
This is v2 of id:1398105468-14317-1-git-send-email-amdragon at mit.edu.
This improves some comments/documentation, fixes a bug that caused
cryptographic processing to not happen on HTML parts, and addresses
some byte compiler warnings on Emacs 23.  This version has also been
rebased against the several months of changes that happened on master
since v1 (which, remarkably, was trivial).

The diff from v1 is below.

diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
index 83cbf2f..f8e5165 100644
--- a/emacs/notmuch-lib.el
+++ b/emacs/notmuch-lib.el
@@ -535,7 +535,10 @@ (defun notmuch-get-bodypart-binary (msg part 
process-crypto  cache)
 This returns the \"raw\" content of the given part after content
 transfer decoding, but with no further processing (see the
 discussion of --format=raw in man notmuch-show).  In particular,
-this does no charset conversion."
+this does no charset conversion.
+
+If CACHE is non-nil, the content of this part will be saved in
+MSG (if it isn't already)."
   (let ((data (plist-get part :binary-content)))
 (when (not data)
   (let ((args `("show" "--format=raw"
@@ -558,6 +561,8 @@ (defun notmuch-get-bodypart-binary (msg part process-crypto 
 cache)
(apply #'call-process notmuch-command nil '(t nil) nil args)
(setq data (buffer-string)
   (when cache
+   ;; Cheat.  part is non-nil, and `plist-put' always modifies
+   ;; the list in place if it's non-nil.
(plist-put part :binary-content data)))
 data))

@@ -567,7 +572,10 @@ (defun notmuch-get-bodypart-text (msg part process-crypto 
 cache)
 This returns the content of the given part as a multibyte Lisp
 string after performing content transfer decoding and any
 necessary charset decoding.  It is an error to use this for
-non-text/* parts."
+non-text/* parts.
+
+If CACHE is non-nil, the content of this part will be saved in
+MSG (if it isn't already)."
   (let ((content (plist-get part :content)))
 (when (not content)
   ;; Use show --format=sexp to fetch decoded content
diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index d190711..66350d4 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -781,10 +781,14 @@ (defun notmuch-show-insert-part-text/html (msg part 
content-type nth depth butto
 (let ((mm-inline-text-html-with-w3m-keymap nil))
   (notmuch-show-insert-part-*/* msg part content-type nth depth button

+;; These functions are used by notmuch-show--insert-part-text/html-shr
+(declare-function libxml-parse-html-region "xml.c")
+(declare-function shr-insert-document "shr")
+
 (defun notmuch-show--insert-part-text/html-shr (msg part)
   ;; Make sure shr is loaded before we start let-binding its globals
   (require 'shr)
-  (let ((dom (let (process-crypto notmuch-show-process-crypto)
+  (let ((dom (let ((process-crypto notmuch-show-process-crypto))
   (with-temp-buffer
 (insert (notmuch-get-bodypart-text msg part process-crypto))
 (libxml-parse-html-region (point-min) (point-max)




[PATCH 08/11] emacs: Support caching in notmuch-get-bodypart-{binary, text}

2015-01-24 Thread Austin Clements
On Fri, 25 Apr 2014, Mark Walters  wrote:
> On Thu, 24 Apr 2014, Austin Clements  wrote:
>> Quoth Mark Walters on Apr 24 at 11:46 am:
>>> 
>>> On Mon, 21 Apr 2014, Austin Clements  wrote:
>>> > (The actual code change here is small, but requires re-indenting
>>> > existing code.)
>>> > ---
>>> >  emacs/notmuch-lib.el | 52 
>>> > ++--
>>> >  1 file changed, 30 insertions(+), 22 deletions(-)
>>> >
>>> > diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
>>> > index fc67b14..fee8512 100644
>>> > --- a/emacs/notmuch-lib.el
>>> > +++ b/emacs/notmuch-lib.el
>>> > @@ -503,33 +503,39 @@ (defun notmuch-parts-filter-by-type (parts type)
>>> > (lambda (part) (notmuch-match-content-type (plist-get part 
>>> > :content-type) type))
>>> > parts))
>>> >  
>>> > -(defun notmuch-get-bodypart-binary (msg part process-crypto)
>>> > +(defun notmuch-get-bodypart-binary (msg part process-crypto  
>>> > cache)
>>> >"Return the unprocessed content of PART in MSG as a unibyte string.
>>> >  
>>> >  This returns the \"raw\" content of the given part after content
>>> >  transfer decoding, but with no further processing (see the
>>> >  discussion of --format=raw in man notmuch-show).  In particular,
>>> >  this does no charset conversion."
>>> > -  (let ((args `("show" "--format=raw"
>>> > - ,(format "--part=%d" (plist-get part :id))
>>> > - ,@(when process-crypto '("--decrypt"))
>>> > - ,(notmuch-id-to-query (plist-get msg :id)
>>> > -(with-temp-buffer
>>> > -  ;; Emacs internally uses a UTF-8-like multibyte string
>>> > -  ;; representation by default (regardless of the coding system,
>>> > -  ;; which only affects how it goes from outside data to this
>>> > -  ;; internal representation).  This *almost* never matters.
>>> > -  ;; Annoyingly, it does matter if we use this data in an image
>>> > -  ;; descriptor, since Emacs will use its internal data buffer
>>> > -  ;; directly and this multibyte representation corrupts binary
>>> > -  ;; image formats.  Since the caller is asking for binary data, a
>>> > -  ;; unibyte string is a more appropriate representation anyway.
>>> > -  (set-buffer-multibyte nil)
>>> > -  (let ((coding-system-for-read 'no-conversion))
>>> > - (apply #'call-process notmuch-command nil '(t nil) nil args)
>>> > - (buffer-string)
>>> > -
>>> > -(defun notmuch-get-bodypart-text (msg part process-crypto)
>>> > +  (let ((data (plist-get part :binary-content)))
>>> > +(when (not data)
>>> > +  (let ((args `("show" "--format=raw"
>>> > + ,(format "--part=%d" (plist-get part :id))
>>> > + ,@(when process-crypto '("--decrypt"))
>>> > + ,(notmuch-id-to-query (plist-get msg :id)
>>> > + (with-temp-buffer
>>> > +   ;; Emacs internally uses a UTF-8-like multibyte string
>>> > +   ;; representation by default (regardless of the coding
>>> > +   ;; system, which only affects how it goes from outside data
>>> > +   ;; to this internal representation).  This *almost* never
>>> > +   ;; matters.  Annoyingly, it does matter if we use this data
>>> > +   ;; in an image descriptor, since Emacs will use its internal
>>> > +   ;; data buffer directly and this multibyte representation
>>> > +   ;; corrupts binary image formats.  Since the caller is
>>> > +   ;; asking for binary data, a unibyte string is a more
>>> > +   ;; appropriate representation anyway.
>>> > +   (set-buffer-multibyte nil)
>>> > +   (let ((coding-system-for-read 'no-conversion))
>>> > + (apply #'call-process notmuch-command nil '(t nil) nil args)
>>> > + (setq data (buffer-string)
>>> > +  (when cache
>>> > + (plist-put part :binary-content data)))
>>> > +data))
>>> 
>>> I am a little puzzled by this but that could be lack of familiarity with
>>> elisp. As far as I can see plist-put will sometimes modify the original
>>> plist and sometimes return a new plist. If the latter happens then I
>>> think it works out as if we hadn't cached anything as the part passed to
>>> the function is unmodified. That might not matter in this case (though I
>>> find the lack of determinism disturbing).
>>> 
>>> Or is something else going on?
>>
>> No, your familiarity with Elisp serves you well.  I'm completely
>> cheating here.  According to the specification of plist-put, it's
>> allowed to return a new list but in reality this only happens when the
>> original plist is nil.  We lean on this already all over the
>> notmuch-emacs code, but maybe that doesn't excuse me adding one more
>> cheat.
>>
>> I could add a comment here explaining what's going on, I could
>> manually do the list insertion in a way that's guaranteed to mutate it
>> in place, or I could add a nil :binary-content property when parts are
>> created (since plist-put is guaranteed to mutate existing keys in
>> place).
>
> I think a comment is fine. 

Done.

> (Incidentally what is the best way of 

[PATCH 00/11] Improve charset and cid: handling

2015-01-24 Thread Austin Clements
I added declare-functions for both of these, which should take care of
the Emacs 23 warnings and be more robust on Emacs 24.  We can't reach
the function that calls these unless shr is actually available, but the
byte compiler doesn't know that.

On Sat, 26 Apr 2014, Mark Walters  wrote:
> Aside from the minor comments I mentioned in previous emails and one
> more comment below this looks good. 
>
> The extra comment is that on emacs23 I get the following when compiling:
>
> In end of data:
> notmuch-show.el:2188:1:Warning: the following functions are not known to be
> defined: libxml-parse-html-region, shr-insert-document
>
> Finally, I have not really tested it as I mainly use emacs23
>
> Best wishes
>
> Mark
>
>
>
>
> On Mon, 21 Apr 2014, Austin Clements  wrote:
>> I set out to quickly add support for cid: links in the shr renderer
>> and wound up making our charset handling more robust and rewriting our
>> content-ID handling.  The test introduced in patch 2 passes in all but
>> one really obscure case, but only because of many unwritten and
>> potentially fragile assumptions that Emacs and the CLI make about each
>> other.
>>
>> The first three patches could reasonably go in to 0.18.  The rest of
>> this series is certainly post-0.18, but I didn't want to lose track of
>> it.
>>
>> This series comes in three stages.  Each depends on the earlier ones,
>> but each prefix makes sense on its own and could be pushed without the
>> later stages.
>>
>> Patch 1 is a simple clean up patch.
>>
>> Patches 2 through 7 robust-ify our charset handling in Emacs, mostly
>> by splitting the broken `notmuch-get-bodypart-content' API into
>> `notmuch-get-bodypart-binary' and `notmuch-get-bodypart-text' so a
>> caller can explicitly convey their requirements.
>>
>> The remaining patches improve our content-ID handling and add support
>> for cid: links for shr.
>>
>> ___
>> notmuch mailing list
>> notmuch at notmuchmail.org
>> http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 11/11] emacs: Support cid: references with shr renderer

2015-01-24 Thread Austin Clements
On Thu, 01 May 2014, David Edmondson  wrote:
> On Mon, Apr 21 2014, Austin Clements wrote:
>> +(defun notmuch-show--insert-part-text/html-shr (msg part)
>> +  ;; Make sure shr is loaded before we start let-binding its globals
>> +  (require 'shr)
>> +  (let ((dom (let (process-crypto notmuch-show-process-crypto)
>
> Missing brackets? (You let-bind both `process-crypto' and
> `notmuch-show-process-crypto' as nil.)

'Doh.  Thanks.


[PATCH 06/11] emacs: Remove broken `notmuch-get-bodypart-content' API

2015-01-24 Thread Austin Clements
On Fri, 11 Jul 2014, David Bremner  wrote:
> Austin Clements  writes:
>
>> +This returns the content of the given part as a multibyte Lisp
>
> What does "multibyte" mean here? utf8? current encoding?

Elisp has two kinds of stings: "unibyte strings" and "multibyte
strings".

  
https://www.gnu.org/software/emacs/manual/html_node/elisp/Non_002dASCII-in-Strings.html

You can think of unibyte strings as binary data; they're just vectors of
bytes without any particular encoding semantics (though when you use a
unibyte string you can endow it with encoding).  Multibyte strings,
however, are text; they're vectors of Unicode code points.

>> +string after performing content transfer decoding and any
>> +necessary charset decoding.  It is an error to use this for
>> +non-text/* parts."
>> +  (let ((content (plist-get part :content)))
>> +(when (not content)
>> +  ;; Use show --format=sexp to fetch decoded content
>> +  (let* ((args `("show" "--format=sexp" "--include-html"
>> + ,(format "--part=%s" (plist-get part :id))
>> + ,@(when process-crypto '("--decrypt"))
>> + ,(notmuch-id-to-query (plist-get msg :id
>> + (npart (apply #'notmuch-call-notmuch-sexp args)))
>> +(setq content (plist-get npart :content))
>> +(when (not content)
>> +  (error "Internal error: No :content from %S" args
>> +content))
>
> I'm a bit curious at the lack of setting "coding-system-for-read" here.
> Are we assuming the user has their environment set up correctly? Not so
> much a criticism as being nervous about everything coding-system
> related.

That is interesting.  coding-system-for-read should really go in
notmuch-call-notmuch-sexp, but I worry that, while *almost* all strings
the CLI outputs are UTF-8, not quite all of them are.  For example, we
output filenames exactly at the OS reports the bytes to us (which is
necessary, in a sense, because POSIX enforces no particular encoding on
file names, but still really unfortunate).

We could set coding-system-for-read, but a full solution needs more
cooperation from the CLI.  Possibly the right answer, at least for the
sexp format, is to do our own UTF-8 to "\u" escapes for strings that
are known to be UTF-8 and leave the raw bytes for the few that aren't.
Then we would set the coding-system-for-read to 'no-conversion and I
think everything would Just Work.

That doesn't help for JSON, which is supposed to be all UTF-8 all the
time.  I can think of solutions there, but they're all ugly and involve
things like encoding filenames as base64 when they aren't valid UTF-8.

So...  I don't think I'm going to do anything about this at this moment.

> I didn't see anything else to object to in this patch or the previous
> one.


[PATCH 04/11] emacs: Track full message and part descriptor in w3m CID store

2015-01-24 Thread Austin Clements
On Thu, 10 Jul 2014, David Bremner  wrote:
> Austin Clements  writes:
>
>> This will simplify later changes.
>
> I'd have preferred the whitespace changes as a seperate patch, but OK.

Not sure which whitespace changes you're referring to.  Everything in
this diff is actual (minor) code changes.


[LATE NIGHT PATCH] build gzipped apidoc in case have doxygen but not sphinx

2015-01-24 Thread David Bremner
David Bremner  writes:

> Tomi Ollila  writes:
>
>> In case we had doxygen but not sphinx notmuch.3 was created but
>> notmuch.3.gz not -- which means install fails!
>> This patch (with late night unpolished commit message will fix that)
>
> By install fails, you mean the man page is simply not installed? I agree
> that's a bug, but I had visions of the install failing half way through.

A related issue I noticed is that some tests fail, rather than being
skipped, if the man pages are not built.

d


[PATCH 3/3] debian packaging: drop dep notmuch-mutt -> libmailtools-perl

2015-01-24 Thread Stefano Zacchiroli
Rationale: the dependency is no longer needed after the rewrite of
get_message_id().

Note that Digest::SHA, needed by the new implementation of
get_message_id(), is shipped by the perl package itself, which is in
an implied dependency of other lib*-perl packages. So there is no need
to list perl explicitly as a dependency.
---
 debian/changelog | 8 
 debian/control   | 2 +-
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/debian/changelog b/debian/changelog
index fbd15b8..2fa0f96 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,11 @@
+notmuch (0.19-2) experimental; urgency=medium
+
+  [ Stefano Zacchiroli ]
+  * notmuch-mutt: drop dependency on libmailtools-perl, because we no
+longer need Mail::Internet
+
+ --
+
 notmuch (0.19-1) experimental; urgency=medium

   * New upstream release.
diff --git a/debian/control b/debian/control
index 4bc4cd9..84dbf32 100644
--- a/debian/control
+++ b/debian/control
@@ -144,7 +144,7 @@ Package: notmuch-mutt
 Architecture: all
 Depends:
  notmuch (>= 0.4),
- libmail-box-perl, libmailtools-perl,
+ libmail-box-perl,
  libstring-shellquote-perl, libterm-readline-gnu-perl,
  ${misc:Depends}
 Recommends: mutt
-- 
2.1.4



[PATCH 2/3] notmuch-mutt: support for messages that lack Message-ID headers

2015-01-24 Thread Stefano Zacchiroli
From: "Jan N. Klug" 

For those messages, compute a synthetic Message-ID based on the SHA1
of the whole message, in the same way that notmuch would do. See:
http://git.notmuchmail.org/git/notmuch/blob/HEAD:/lib/sha1.c

To do the above, rewrite get_message_id() to scan the current message
line by line, incrementally computing a SHA1. As a consequence, drop
the dependency on Mail::Internet.

Signed-off-by: Stefano Zacchiroli 
---
 contrib/notmuch-mutt/README   |  4 ++--
 contrib/notmuch-mutt/notmuch-mutt | 23 ++-
 2 files changed, 20 insertions(+), 7 deletions(-)

diff --git a/contrib/notmuch-mutt/README b/contrib/notmuch-mutt/README
index c661447..0013ed0 100644
--- a/contrib/notmuch-mutt/README
+++ b/contrib/notmuch-mutt/README
@@ -33,10 +33,10 @@ Requirements

 To *run* notmuch-mutt you will need Perl with the following libraries:

+- Digest::SHA 
+  (Debian package: libdigest-sha-perl)
 - Mail::Box 
   (Debian package: libmail-box-perl)
-- Mail::Internet 
-  (Debian package: libmailtools-perl)
 - String::ShellQuote 
   (Debian package: libstring-shellquote-perl)
 - Term::ReadLine::Gnu 
diff --git a/contrib/notmuch-mutt/notmuch-mutt 
b/contrib/notmuch-mutt/notmuch-mutt
index 4969e4b..4d30b0b 100755
--- a/contrib/notmuch-mutt/notmuch-mutt
+++ b/contrib/notmuch-mutt/notmuch-mutt
@@ -13,11 +13,11 @@ use warnings;

 use File::Path;
 use Getopt::Long qw(:config no_getopt_compat);
-use Mail::Internet;
 use Mail::Box::Maildir;
 use Pod::Usage;
 use String::ShellQuote;
 use Term::ReadLine;
+use Digest::SHA;


 my $xdg_cache_dir = "$ENV{HOME}/.cache";
@@ -75,10 +75,23 @@ sub prompt($$) {
 }

 sub get_message_id() {
-my $mail = Mail::Internet->new(\*STDIN);
-my $mid = $mail->head->get("message-id") or return undef;
-$mid =~ /^<(.*)>$/;# get message-id value
-return $1;
+my $mid = undef;
+my $sha = Digest::SHA->new(1);  # SHA1 hashing
+
+while() {  # scan message line by line, looking for mid
+if ($_ =~ /^Message-ID:\s*<(.*)>$/i) {
+   $mid = $1;
+   last;  # message-id found, abort scan
+}
+   $sha->add($_);  # update hash
+}
+
+# Generate message-id from hash if none was found, in the same way
+# that notmuch would do.
+# See: http://git.notmuchmail.org/git/notmuch/blob/HEAD:/lib/sha1.c
+$mid ||= "notmuch-sha1-".$sha->hexdigest;
+
+return $mid;
 }

 sub search_action($$$@) {
-- 
2.1.4



[PATCH 1/3] notmuch-mutt README: use metacpn.org/* as deps homepages

2015-01-24 Thread Stefano Zacchiroli
---
 contrib/notmuch-mutt/README | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/contrib/notmuch-mutt/README b/contrib/notmuch-mutt/README
index 382ac91..c661447 100644
--- a/contrib/notmuch-mutt/README
+++ b/contrib/notmuch-mutt/README
@@ -33,13 +33,13 @@ Requirements

 To *run* notmuch-mutt you will need Perl with the following libraries:

-- Mail::Box 
+- Mail::Box 
   (Debian package: libmail-box-perl)
-- Mail::Internet 
+- Mail::Internet 
   (Debian package: libmailtools-perl)
-- String::ShellQuote 

+- String::ShellQuote 
   (Debian package: libstring-shellquote-perl)
-- Term::ReadLine 
+- Term::ReadLine::Gnu 
   (Debian package: libterm-readline-gnu-perl)

 To *build* notmuch-mutt documentation you will need:
-- 
2.1.4



[PATCH] notmuch-mutt: support for messages that lack Message-ID

2015-01-24 Thread Stefano Zacchiroli
Please find attached a patch series that add support for messages that
lack Message-ID to notmuch-mutt, kindly provided by Jan N. Klug. I've
reviewed / adapted it, and I consider it suitable for inclusion into
the next release of notmuch(-mutt).

Please let me know if you've any question.

Cheers.



[LATE NIGHT PATCH] build gzipped apidoc in case have doxygen but not sphinx

2015-01-24 Thread David Bremner
Tomi Ollila  writes:

> In case we had doxygen but not sphinx notmuch.3 was created but
> notmuch.3.gz not -- which means install fails!
> This patch (with late night unpolished commit message will fix that)

By install fails, you mean the man page is simply not installed? I agree
that's a bug, but I had visions of the install failing half way through.

d


[PATCH] emacs: escape % in header line format

2015-01-24 Thread David Bremner
David Bremner  writes:

> Jinwoo Lee  writes:
>
>> Yup.  It works!  Thanks for the quick fix.  Is this going to be merged
>> to HEAD soon?
>>
>
> Probably in the next day or so, unless somebody complains.

This bug should be fixed in commit cc3d25d

d


Re: [PATCH 08/11] emacs: Support caching in notmuch-get-bodypart-{binary, text}

2015-01-24 Thread Austin Clements
On Fri, 25 Apr 2014, Mark Walters markwalters1...@gmail.com wrote:
 On Thu, 24 Apr 2014, Austin Clements amdra...@mit.edu wrote:
 Quoth Mark Walters on Apr 24 at 11:46 am:
 
 On Mon, 21 Apr 2014, Austin Clements amdra...@mit.edu wrote:
  (The actual code change here is small, but requires re-indenting
  existing code.)
  ---
   emacs/notmuch-lib.el | 52 
  ++--
   1 file changed, 30 insertions(+), 22 deletions(-)
 
  diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
  index fc67b14..fee8512 100644
  --- a/emacs/notmuch-lib.el
  +++ b/emacs/notmuch-lib.el
  @@ -503,33 +503,39 @@ (defun notmuch-parts-filter-by-type (parts type)
  (lambda (part) (notmuch-match-content-type (plist-get part 
  :content-type) type))
  parts))
   
  -(defun notmuch-get-bodypart-binary (msg part process-crypto)
  +(defun notmuch-get-bodypart-binary (msg part process-crypto optional 
  cache)
 Return the unprocessed content of PART in MSG as a unibyte string.
   
   This returns the \raw\ content of the given part after content
   transfer decoding, but with no further processing (see the
   discussion of --format=raw in man notmuch-show).  In particular,
   this does no charset conversion.
  -  (let ((args `(show --format=raw
  - ,(format --part=%d (plist-get part :id))
  - ,@(when process-crypto '(--decrypt))
  - ,(notmuch-id-to-query (plist-get msg :id)
  -(with-temp-buffer
  -  ;; Emacs internally uses a UTF-8-like multibyte string
  -  ;; representation by default (regardless of the coding system,
  -  ;; which only affects how it goes from outside data to this
  -  ;; internal representation).  This *almost* never matters.
  -  ;; Annoyingly, it does matter if we use this data in an image
  -  ;; descriptor, since Emacs will use its internal data buffer
  -  ;; directly and this multibyte representation corrupts binary
  -  ;; image formats.  Since the caller is asking for binary data, a
  -  ;; unibyte string is a more appropriate representation anyway.
  -  (set-buffer-multibyte nil)
  -  (let ((coding-system-for-read 'no-conversion))
  - (apply #'call-process notmuch-command nil '(t nil) nil args)
  - (buffer-string)
  -
  -(defun notmuch-get-bodypart-text (msg part process-crypto)
  +  (let ((data (plist-get part :binary-content)))
  +(when (not data)
  +  (let ((args `(show --format=raw
  + ,(format --part=%d (plist-get part :id))
  + ,@(when process-crypto '(--decrypt))
  + ,(notmuch-id-to-query (plist-get msg :id)
  + (with-temp-buffer
  +   ;; Emacs internally uses a UTF-8-like multibyte string
  +   ;; representation by default (regardless of the coding
  +   ;; system, which only affects how it goes from outside data
  +   ;; to this internal representation).  This *almost* never
  +   ;; matters.  Annoyingly, it does matter if we use this data
  +   ;; in an image descriptor, since Emacs will use its internal
  +   ;; data buffer directly and this multibyte representation
  +   ;; corrupts binary image formats.  Since the caller is
  +   ;; asking for binary data, a unibyte string is a more
  +   ;; appropriate representation anyway.
  +   (set-buffer-multibyte nil)
  +   (let ((coding-system-for-read 'no-conversion))
  + (apply #'call-process notmuch-command nil '(t nil) nil args)
  + (setq data (buffer-string)
  +  (when cache
  + (plist-put part :binary-content data)))
  +data))
 
 I am a little puzzled by this but that could be lack of familiarity with
 elisp. As far as I can see plist-put will sometimes modify the original
 plist and sometimes return a new plist. If the latter happens then I
 think it works out as if we hadn't cached anything as the part passed to
 the function is unmodified. That might not matter in this case (though I
 find the lack of determinism disturbing).
 
 Or is something else going on?

 No, your familiarity with Elisp serves you well.  I'm completely
 cheating here.  According to the specification of plist-put, it's
 allowed to return a new list but in reality this only happens when the
 original plist is nil.  We lean on this already all over the
 notmuch-emacs code, but maybe that doesn't excuse me adding one more
 cheat.

 I could add a comment here explaining what's going on, I could
 manually do the list insertion in a way that's guaranteed to mutate it
 in place, or I could add a nil :binary-content property when parts are
 created (since plist-put is guaranteed to mutate existing keys in
 place).

 I think a comment is fine. 

Done.

 (Incidentally what is the best way of telling if emacs has changed an
 object or returned a new one for other commands? Something like (setq
 oldobject object) (setq object (operation-on object)) (if (eq object
 oldobject) ... ))

If `eq' returns t, it definitely returned the original object, though it
may or may not have modified 

Re: [PATCH 04/11] emacs: Track full message and part descriptor in w3m CID store

2015-01-24 Thread Austin Clements
On Thu, 10 Jul 2014, David Bremner da...@tethera.net wrote:
 Austin Clements amdra...@mit.edu writes:

 This will simplify later changes.

 I'd have preferred the whitespace changes as a seperate patch, but OK.

Not sure which whitespace changes you're referring to.  Everything in
this diff is actual (minor) code changes.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH 2/3] notmuch-mutt: support for messages that lack Message-ID headers

2015-01-24 Thread Jan N. Klug
On Sat, Jan 24, 2015 at 04:21:06PM +0100, Stefano Zacchiroli wrote:
 On Sat, Jan 24, 2015 at 04:59:16PM +0200, Jani Nikula wrote:
  On Sat, 24 Jan 2015, Stefano Zacchiroli z...@upsilon.cc wrote:
   To do the above, rewrite get_message_id() to scan the current message
   line by line, incrementally computing a SHA1. As a consequence, drop
   the dependency on Mail::Internet.
  
  I am not so sure this is a good idea however, see below.
 [...]
 But I didn't think of header folding, and you're absolutely correct in
 saying that might be a problem.

I disagree. I do not think that header folding is a problem here. RFC
5322 and RFC 2822 state that FWS in the message-id are not allowed (as
opposed to In-Reply: and other headers, where FWS may occur between, but
not inside message-ids).

Folded lines that start with Message-Id: might occur, the
regex-pattern used in the patch will not match those lines. 

As far as I have looked in the Mail::Internet sources, further checking of the
correctness of the message-id does not occur, so I do not see any
advantage over the proposed solution. 

 Jan: do you agree with using Mail::Header-new and fall back to
 line-by-line hasing only in case Message-ID is not found? If so, having
 an updated patch based on the one I've posted here would be awesome! If
 you cannot do that just let me know and I'll get to it, eventually :).

I can look into that, but I cannot promise to have it ready in the next
days.

Regards,

Jan
--
Jan N. Klug, Gelsenkirchen
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH 11/11] emacs: Support cid: references with shr renderer

2015-01-24 Thread Austin Clements
On Thu, 01 May 2014, David Edmondson d...@dme.org wrote:
 On Mon, Apr 21 2014, Austin Clements wrote:
 +(defun notmuch-show--insert-part-text/html-shr (msg part)
 +  ;; Make sure shr is loaded before we start let-binding its globals
 +  (require 'shr)
 +  (let ((dom (let (process-crypto notmuch-show-process-crypto)

 Missing brackets? (You let-bind both `process-crypto' and
 `notmuch-show-process-crypto' as nil.)

'Doh.  Thanks.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH v2 8/8] emacs: Support cid: references with shr renderer

2015-01-24 Thread Austin Clements
shr has really nice support for inline image rendering, but previously
we only had the hooks for w3m cid: references.
---
 emacs/notmuch-show.el | 45 +
 1 file changed, 37 insertions(+), 8 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index 34dcedd..66350d4 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -767,14 +767,43 @@ (defun 
notmuch-show-get-mime-type-of-application/octet-stream (part)
  nil
 
 (defun notmuch-show-insert-part-text/html (msg part content-type nth depth 
button)
-  ;; text/html handler to work around bugs in renderers and our
-  ;; invisibile parts code. In particular w3m sets up a keymap which
-  ;; leaks outside the invisible region and causes strange effects
-  ;; in notmuch. We set mm-inline-text-html-with-w3m-keymap to nil to
-  ;; tell w3m not to set a keymap (so the normal notmuch-show-mode-map
-  ;; remains).
-  (let ((mm-inline-text-html-with-w3m-keymap nil))
-(notmuch-show-insert-part-*/* msg part content-type nth depth button)))
+  (if (eq mm-text-html-renderer 'shr)
+  ;; It's easier to drive shr ourselves than to work around the
+  ;; goofy things `mm-shr' does (like irreversibly taking over
+  ;; content ID handling).
+  (notmuch-show--insert-part-text/html-shr msg part)
+;; Otherwise, let message-mode do the heavy lifting
+;;
+;; w3m sets up a keymap which leaks outside the invisible region
+;; and causes strange effects in notmuch. We set
+;; mm-inline-text-html-with-w3m-keymap to nil to tell w3m not to
+;; set a keymap (so the normal notmuch-show-mode-map remains).
+(let ((mm-inline-text-html-with-w3m-keymap nil))
+  (notmuch-show-insert-part-*/* msg part content-type nth depth button
+
+;; These functions are used by notmuch-show--insert-part-text/html-shr
+(declare-function libxml-parse-html-region xml.c)
+(declare-function shr-insert-document shr)
+
+(defun notmuch-show--insert-part-text/html-shr (msg part)
+  ;; Make sure shr is loaded before we start let-binding its globals
+  (require 'shr)
+  (let ((dom (let ((process-crypto notmuch-show-process-crypto))
+  (with-temp-buffer
+(insert (notmuch-get-bodypart-text msg part process-crypto))
+(libxml-parse-html-region (point-min) (point-max)
+   (shr-content-function
+(lambda (url)
+  ;; shr strips the cid: part of URL, but doesn't
+  ;; URL-decode it (see RFC 2392).
+  (let ((cid (url-unhex-string url)))
+(first (notmuch-show--get-cid-content cid)
+   ;; Block all external images to prevent privacy leaks and
+   ;; potential attacks.  FIXME: If we block an image, offer a
+   ;; button to load external images.
+   (shr-blocked-images .))
+(shr-insert-document dom)
+t))
 
 (defun notmuch-show-insert-part-*/* (msg part content-type nth depth button)
   ;; This handler _must_ succeed - it is the handler of last resort.
-- 
2.1.3

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH v2 2/8] emacs: Create an API for fetching parts as undecoded binary

2015-01-24 Thread Austin Clements
The new function, `notmuch-get-bodypart-binary', replaces
`notmuch-get-bodypart-internal'.  Whereas the old function was really
meant for internal use in `notmuch-get-bodypart-content', it was used
in a few other places.  Since the difference between
`notmuch-get-bodypart-content' and `notmuch-get-bodypart-internal' was
unclear, these other uses were always confusing and potentially
inconsistent.  The new call clearly requests the part as undecoded
binary.

This is step 1 of 2 in separating `notmuch-get-bodypart-content' into
two APIs for retrieving either undecoded binary or decoded text.
---
 emacs/notmuch-lib.el  | 28 ++--
 emacs/notmuch-show.el | 22 +-
 2 files changed, 23 insertions(+), 27 deletions(-)

diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
index fd25f7c..d4b6684 100644
--- a/emacs/notmuch-lib.el
+++ b/emacs/notmuch-lib.el
@@ -529,25 +529,25 @@ (defun notmuch-parts-filter-by-type (parts type)
(lambda (part) (notmuch-match-content-type (plist-get part :content-type) 
type))
parts))
 
-;; Helper for parts which are generally not included in the default
-;; SEXP output.
-(defun notmuch-get-bodypart-internal (query part-number process-crypto)
-  (let ((args '(show --format=raw))
-   (part-arg (format --part=%s part-number)))
-(setq args (append args (list part-arg)))
-(if process-crypto
-   (setq args (append args '(--decrypt
-(setq args (append args (list query)))
+(defun notmuch-get-bodypart-binary (msg part process-crypto)
+  Return the unprocessed content of PART in MSG.
+
+This returns the \raw\ content of the given part after content
+transfer decoding, but with no further processing (see the
+discussion of --format=raw in man notmuch-show).  In particular,
+this does no charset conversion.
+  (let ((args `(show --format=raw
+   ,(format --part=%d (plist-get part :id))
+   ,@(when process-crypto '(--decrypt))
+   ,(notmuch-id-to-query (plist-get msg :id)
 (with-temp-buffer
   (let ((coding-system-for-read 'no-conversion))
-   (progn
- (apply 'call-process (append (list notmuch-command nil (list t nil) 
nil) args))
- (buffer-string))
+   (apply #'call-process notmuch-command nil '(t nil) nil args)
+   (buffer-string)
 
 (defun notmuch-get-bodypart-content (msg part process-crypto)
   (or (plist-get part :content)
-  (notmuch-get-bodypart-internal (notmuch-id-to-query (plist-get msg :id))
-(plist-get part :id) process-crypto)))
+  (notmuch-get-bodypart-binary msg part process-crypto)))
 
 ;; Workaround: The call to `mm-display-part' below triggers a bug in
 ;; Emacs 24 if it attempts to use the shr renderer to display an HTML
diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index df2389e..b3e339e 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -579,16 +579,14 @@ (defun notmuch-show-w3m-cid-retrieve (url rest args)
(let* ((msg (nth 1 matching-part))
   (part (nth 2 matching-part))
   (content (nth 3 matching-part))
-  (message-id (plist-get msg :id))
-  (part-number (plist-get part :id))
   (content-type (plist-get part :content-type)))
  ;; If we don't already have the content, get it and cache
  ;; it, as some messages reference the same cid: part many
  ;; times (hundreds!), which results in many calls to
  ;; `notmuch part'.
  (unless content
-   (setq content (notmuch-get-bodypart-internal (notmuch-id-to-query 
message-id)
- part-number 
notmuch-show-process-crypto))
+   (setq content (notmuch-get-bodypart-binary
+  msg part notmuch-show-process-crypto))
(with-current-buffer w3m-current-buffer
  (notmuch-show-w3m-cid-store-internal url msg part content)))
  (insert content)
@@ -2162,15 +2160,14 @@ (defun notmuch-show-stash-git-send-email (optional 
no-in-reply-to)
 
 ;; Interactive part functions and their helpers
 
-(defun notmuch-show-generate-part-buffer (message-id nth)
+(defun notmuch-show-generate-part-buffer (msg part)
   Return a temporary buffer containing the specified part's content.
   (let ((buf (generate-new-buffer  *notmuch-part*))
(process-crypto notmuch-show-process-crypto))
 (with-current-buffer buf
-  (setq notmuch-show-process-crypto process-crypto)
-  ;; Always acquires the part via `notmuch part', even if it is
-  ;; available in the SEXP output.
-  (insert (notmuch-get-bodypart-internal message-id nth 
notmuch-show-process-crypto)))
+  ;; This is always used in the content of mm handles, which
+  ;; expect undecoded, binary part content.
+  (insert (notmuch-get-bodypart-binary msg part process-crypto)))
 buf))
 
 (defun 

[PATCH v2 3/8] emacs: Remove broken `notmuch-get-bodypart-content' API

2015-01-24 Thread Austin Clements
`notmuch-get-bodypart-content' could do two very different things,
depending on conditions: for text/* parts other than text/html, it
would return the part content as a multibyte Lisp string *after*
charset conversion, while for other parts (including text/html), it
would return binary part content without charset conversion.

This commit completes the split of `notmuch-get-bodypart-content' into
two different and explicit APIs: `notmuch-get-bodypart-binary' and
`notmuch-get-bodypart-text'.  It updates all callers to use one or the
other depending on what's appropriate.
---
 emacs/notmuch-lib.el  | 37 -
 emacs/notmuch-show.el |  8 
 2 files changed, 32 insertions(+), 13 deletions(-)

diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
index d4b6684..a0a95d8 100644
--- a/emacs/notmuch-lib.el
+++ b/emacs/notmuch-lib.el
@@ -545,9 +545,25 @@ (defun notmuch-get-bodypart-binary (msg part 
process-crypto)
(apply #'call-process notmuch-command nil '(t nil) nil args)
(buffer-string)
 
-(defun notmuch-get-bodypart-content (msg part process-crypto)
-  (or (plist-get part :content)
-  (notmuch-get-bodypart-binary msg part process-crypto)))
+(defun notmuch-get-bodypart-text (msg part process-crypto)
+  Return the text content of PART in MSG.
+
+This returns the content of the given part as a multibyte Lisp
+string after performing content transfer decoding and any
+necessary charset decoding.  It is an error to use this for
+non-text/* parts.
+  (let ((content (plist-get part :content)))
+(when (not content)
+  ;; Use show --format=sexp to fetch decoded content
+  (let* ((args `(show --format=sexp --include-html
+,(format --part=%s (plist-get part :id))
+,@(when process-crypto '(--decrypt))
+,(notmuch-id-to-query (plist-get msg :id
+(npart (apply #'notmuch-call-notmuch-sexp args)))
+   (setq content (plist-get npart :content))
+   (when (not content)
+ (error Internal error: No :content from %S args
+content))
 
 ;; Workaround: The call to `mm-display-part' below triggers a bug in
 ;; Emacs 24 if it attempts to use the shr renderer to display an HTML
@@ -568,18 +584,21 @@ (defun notmuch-mm-display-part-inline (msg part 
content-type process-crypto)
 current buffer, if possible.
   (let ((display-buffer (current-buffer)))
 (with-temp-buffer
-  ;; In case there is :content, the content string is already converted
-  ;; into emacs internal format. `gnus-decoded' is a fake charset,
-  ;; which means no further decoding (to be done by mm- functions).
-  (let* ((charset (if (plist-member part :content)
- 'gnus-decoded
+  ;; In case we already have :content, use it and tell mm-* that
+  ;; it's already been charset-decoded by using the fake
+  ;; `gnus-decoded' charset.  Otherwise, we'll fetch the binary
+  ;; part content and let mm-* decode it.
+  (let* ((have-content (plist-member part :content))
+(charset (if have-content 'gnus-decoded
(plist-get part :content-charset)))
 (handle (mm-make-handle (current-buffer) `(,content-type (charset 
. ,charset)
;; If the user wants the part inlined, insert the content and
;; test whether we are able to inline it (which includes both
;; capability and suitability tests).
(when (mm-inlined-p handle)
- (insert (notmuch-get-bodypart-content msg part process-crypto))
+ (if have-content
+ (insert (notmuch-get-bodypart-text msg part process-crypto))
+   (insert (notmuch-get-bodypart-binary msg part process-crypto)))
  (when (mm-inlinable-p handle)
(set-buffer display-buffer)
(mm-display-part handle)
diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index b3e339e..f29428a 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -702,7 +702,7 @@ (defun notmuch-show-insert-part-text/plain (msg part 
content-type nth depth butt
   (let ((start (if button
   (button-start button)
 (point
-(insert (notmuch-get-bodypart-content msg part 
notmuch-show-process-crypto))
+(insert (notmuch-get-bodypart-text msg part notmuch-show-process-crypto))
 (save-excursion
   (save-restriction
(narrow-to-region start (point-max))
@@ -711,9 +711,9 @@ (defun notmuch-show-insert-part-text/plain (msg part 
content-type nth depth butt
 
 (defun notmuch-show-insert-part-text/calendar (msg part content-type nth depth 
button)
   (insert (with-temp-buffer
-   (insert (notmuch-get-bodypart-content msg part 
notmuch-show-process-crypto))
-   ;; notmuch-get-bodypart-content provides raw, non-converted
-   ;; data. Replace CRLF with LF before icalendar can use it.
+   (insert (notmuch-get-bodypart-text msg part 

[PATCH v2 4/8] emacs: Return unibyte strings for binary part data

2015-01-24 Thread Austin Clements
Unibyte strings are meant for representing binary data.  In practice,
using unibyte versus multibyte strings affects *almost* nothing.  It
does happen to matter if we use the binary data in an image descriptor
(which is, helpfully, not documented anywhere and getting it wrong
results in opaque errors like Not a PNG image: giant binary spew
that is, in fact, a PNG image).
---
 emacs/notmuch-lib.el | 12 +++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
index a0a95d8..3154725 100644
--- a/emacs/notmuch-lib.el
+++ b/emacs/notmuch-lib.el
@@ -530,7 +530,7 @@ (defun notmuch-parts-filter-by-type (parts type)
parts))
 
 (defun notmuch-get-bodypart-binary (msg part process-crypto)
-  Return the unprocessed content of PART in MSG.
+  Return the unprocessed content of PART in MSG as a unibyte string.
 
 This returns the \raw\ content of the given part after content
 transfer decoding, but with no further processing (see the
@@ -541,6 +541,16 @@ (defun notmuch-get-bodypart-binary (msg part 
process-crypto)
,@(when process-crypto '(--decrypt))
,(notmuch-id-to-query (plist-get msg :id)
 (with-temp-buffer
+  ;; Emacs internally uses a UTF-8-like multibyte string
+  ;; representation by default (regardless of the coding system,
+  ;; which only affects how it goes from outside data to this
+  ;; internal representation).  This *almost* never matters.
+  ;; Annoyingly, it does matter if we use this data in an image
+  ;; descriptor, since Emacs will use its internal data buffer
+  ;; directly and this multibyte representation corrupts binary
+  ;; image formats.  Since the caller is asking for binary data, a
+  ;; unibyte string is a more appropriate representation anyway.
+  (set-buffer-multibyte nil)
   (let ((coding-system-for-read 'no-conversion))
(apply #'call-process notmuch-command nil '(t nil) nil args)
(buffer-string)
-- 
2.1.3

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH v2 0/8] Improve charset and cid: handling

2015-01-24 Thread Austin Clements
This is v2 of id:1398105468-14317-1-git-send-email-amdra...@mit.edu.
This improves some comments/documentation, fixes a bug that caused
cryptographic processing to not happen on HTML parts, and addresses
some byte compiler warnings on Emacs 23.  This version has also been
rebased against the several months of changes that happened on master
since v1 (which, remarkably, was trivial).

The diff from v1 is below.

diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
index 83cbf2f..f8e5165 100644
--- a/emacs/notmuch-lib.el
+++ b/emacs/notmuch-lib.el
@@ -535,7 +535,10 @@ (defun notmuch-get-bodypart-binary (msg part 
process-crypto optional cache)
 This returns the \raw\ content of the given part after content
 transfer decoding, but with no further processing (see the
 discussion of --format=raw in man notmuch-show).  In particular,
-this does no charset conversion.
+this does no charset conversion.
+
+If CACHE is non-nil, the content of this part will be saved in
+MSG (if it isn't already).
   (let ((data (plist-get part :binary-content)))
 (when (not data)
   (let ((args `(show --format=raw
@@ -558,6 +561,8 @@ (defun notmuch-get-bodypart-binary (msg part process-crypto 
optional cache)
(apply #'call-process notmuch-command nil '(t nil) nil args)
(setq data (buffer-string)
   (when cache
+   ;; Cheat.  part is non-nil, and `plist-put' always modifies
+   ;; the list in place if it's non-nil.
(plist-put part :binary-content data)))
 data))
 
@@ -567,7 +572,10 @@ (defun notmuch-get-bodypart-text (msg part process-crypto 
optional cache)
 This returns the content of the given part as a multibyte Lisp
 string after performing content transfer decoding and any
 necessary charset decoding.  It is an error to use this for
-non-text/* parts.
+non-text/* parts.
+
+If CACHE is non-nil, the content of this part will be saved in
+MSG (if it isn't already).
   (let ((content (plist-get part :content)))
 (when (not content)
   ;; Use show --format=sexp to fetch decoded content
diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index d190711..66350d4 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -781,10 +781,14 @@ (defun notmuch-show-insert-part-text/html (msg part 
content-type nth depth butto
 (let ((mm-inline-text-html-with-w3m-keymap nil))
   (notmuch-show-insert-part-*/* msg part content-type nth depth button
 
+;; These functions are used by notmuch-show--insert-part-text/html-shr
+(declare-function libxml-parse-html-region xml.c)
+(declare-function shr-insert-document shr)
+
 (defun notmuch-show--insert-part-text/html-shr (msg part)
   ;; Make sure shr is loaded before we start let-binding its globals
   (require 'shr)
-  (let ((dom (let (process-crypto notmuch-show-process-crypto)
+  (let ((dom (let ((process-crypto notmuch-show-process-crypto))
   (with-temp-buffer
 (insert (notmuch-get-bodypart-text msg part process-crypto))
 (libxml-parse-html-region (point-min) (point-max)


___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH v2 6/8] emacs: Use generalized content caching in w3m CID code

2015-01-24 Thread Austin Clements
Previously this did its own caching, but this is now supported by more
generally by `notmuch-get-bodypart-binary'.
---
 emacs/notmuch-show.el | 23 ---
 1 file changed, 8 insertions(+), 15 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index f29428a..11eac5f 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -562,15 +562,14 @@ (defvar w3m-current-buffer) ;; From `w3m.el'.
 (defvar notmuch-show-w3m-cid-store nil)
 (make-variable-buffer-local 'notmuch-show-w3m-cid-store)
 
-(defun notmuch-show-w3m-cid-store-internal (content-id msg part content)
-  (push (list content-id msg part content)
-   notmuch-show-w3m-cid-store))
+(defun notmuch-show-w3m-cid-store-internal (content-id msg part)
+  (push (list content-id msg part) notmuch-show-w3m-cid-store))
 
 (defun notmuch-show-w3m-cid-store (msg part)
   (let ((content-id (plist-get part :content-id)))
 (when content-id
   (notmuch-show-w3m-cid-store-internal (concat cid: content-id)
-  msg part nil
+  msg part
 
 (defun notmuch-show-w3m-cid-retrieve (url rest args)
   (let ((matching-part (with-current-buffer w3m-current-buffer
@@ -578,18 +577,12 @@ (defun notmuch-show-w3m-cid-retrieve (url rest args)
 (if matching-part
(let* ((msg (nth 1 matching-part))
   (part (nth 2 matching-part))
-  (content (nth 3 matching-part))
   (content-type (plist-get part :content-type)))
- ;; If we don't already have the content, get it and cache
- ;; it, as some messages reference the same cid: part many
- ;; times (hundreds!), which results in many calls to
- ;; `notmuch part'.
- (unless content
-   (setq content (notmuch-get-bodypart-binary
-  msg part notmuch-show-process-crypto))
-   (with-current-buffer w3m-current-buffer
- (notmuch-show-w3m-cid-store-internal url msg part content)))
- (insert content)
+ ;; Request content caching, as some messages reference the
+ ;; same cid: part many times (hundreds!), which results in
+ ;; many calls to `notmuch show'.
+ (insert (notmuch-get-bodypart-binary
+  msg part notmuch-show-process-crypto 'cache))
  content-type)
   nil)))
 
-- 
2.1.3

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH v2 1/8] emacs: Track full message and part descriptor in w3m CID store

2015-01-24 Thread Austin Clements
This will simplify later changes.
---
 emacs/notmuch-show.el | 33 ++---
 1 file changed, 10 insertions(+), 23 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index 87b4881..df2389e 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -562,35 +562,26 @@ (defvar w3m-current-buffer) ;; From `w3m.el'.
 (defvar notmuch-show-w3m-cid-store nil)
 (make-variable-buffer-local 'notmuch-show-w3m-cid-store)
 
-(defun notmuch-show-w3m-cid-store-internal (content-id
-   message-id
-   part-number
-   content-type
-   content)
-  (push (list content-id
- message-id
- part-number
- content-type
- content)
+(defun notmuch-show-w3m-cid-store-internal (content-id msg part content)
+  (push (list content-id msg part content)
notmuch-show-w3m-cid-store))
 
 (defun notmuch-show-w3m-cid-store (msg part)
   (let ((content-id (plist-get part :content-id)))
 (when content-id
   (notmuch-show-w3m-cid-store-internal (concat cid: content-id)
-  (plist-get msg :id)
-  (plist-get part :id)
-  (plist-get part :content-type)
-  nil
+  msg part nil
 
 (defun notmuch-show-w3m-cid-retrieve (url rest args)
   (let ((matching-part (with-current-buffer w3m-current-buffer
 (assoc url notmuch-show-w3m-cid-store
 (if matching-part
-   (let ((message-id (nth 1 matching-part))
- (part-number (nth 2 matching-part))
- (content-type (nth 3 matching-part))
- (content (nth 4 matching-part)))
+   (let* ((msg (nth 1 matching-part))
+  (part (nth 2 matching-part))
+  (content (nth 3 matching-part))
+  (message-id (plist-get msg :id))
+  (part-number (plist-get part :id))
+  (content-type (plist-get part :content-type)))
  ;; If we don't already have the content, get it and cache
  ;; it, as some messages reference the same cid: part many
  ;; times (hundreds!), which results in many calls to
@@ -599,11 +590,7 @@ (defun notmuch-show-w3m-cid-retrieve (url rest args)
(setq content (notmuch-get-bodypart-internal (notmuch-id-to-query 
message-id)
  part-number 
notmuch-show-process-crypto))
(with-current-buffer w3m-current-buffer
- (notmuch-show-w3m-cid-store-internal url
-  message-id
-  part-number
-  content-type
-  content)))
+ (notmuch-show-w3m-cid-store-internal url msg part content)))
  (insert content)
  content-type)
   nil)))
-- 
2.1.3

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH v2 5/8] emacs: Support caching in notmuch-get-bodypart-{binary, text}

2015-01-24 Thread Austin Clements
(The actual code change here is small, but requires re-indenting
existing code.)
---
 emacs/notmuch-lib.el | 64 
 1 file changed, 40 insertions(+), 24 deletions(-)

diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
index 3154725..f8e5165 100644
--- a/emacs/notmuch-lib.el
+++ b/emacs/notmuch-lib.el
@@ -529,39 +529,53 @@ (defun notmuch-parts-filter-by-type (parts type)
(lambda (part) (notmuch-match-content-type (plist-get part :content-type) 
type))
parts))
 
-(defun notmuch-get-bodypart-binary (msg part process-crypto)
+(defun notmuch-get-bodypart-binary (msg part process-crypto optional cache)
   Return the unprocessed content of PART in MSG as a unibyte string.
 
 This returns the \raw\ content of the given part after content
 transfer decoding, but with no further processing (see the
 discussion of --format=raw in man notmuch-show).  In particular,
-this does no charset conversion.
-  (let ((args `(show --format=raw
-   ,(format --part=%d (plist-get part :id))
-   ,@(when process-crypto '(--decrypt))
-   ,(notmuch-id-to-query (plist-get msg :id)
-(with-temp-buffer
-  ;; Emacs internally uses a UTF-8-like multibyte string
-  ;; representation by default (regardless of the coding system,
-  ;; which only affects how it goes from outside data to this
-  ;; internal representation).  This *almost* never matters.
-  ;; Annoyingly, it does matter if we use this data in an image
-  ;; descriptor, since Emacs will use its internal data buffer
-  ;; directly and this multibyte representation corrupts binary
-  ;; image formats.  Since the caller is asking for binary data, a
-  ;; unibyte string is a more appropriate representation anyway.
-  (set-buffer-multibyte nil)
-  (let ((coding-system-for-read 'no-conversion))
-   (apply #'call-process notmuch-command nil '(t nil) nil args)
-   (buffer-string)
-
-(defun notmuch-get-bodypart-text (msg part process-crypto)
+this does no charset conversion.
+
+If CACHE is non-nil, the content of this part will be saved in
+MSG (if it isn't already).
+  (let ((data (plist-get part :binary-content)))
+(when (not data)
+  (let ((args `(show --format=raw
+   ,(format --part=%d (plist-get part :id))
+   ,@(when process-crypto '(--decrypt))
+   ,(notmuch-id-to-query (plist-get msg :id)
+   (with-temp-buffer
+ ;; Emacs internally uses a UTF-8-like multibyte string
+ ;; representation by default (regardless of the coding
+ ;; system, which only affects how it goes from outside data
+ ;; to this internal representation).  This *almost* never
+ ;; matters.  Annoyingly, it does matter if we use this data
+ ;; in an image descriptor, since Emacs will use its internal
+ ;; data buffer directly and this multibyte representation
+ ;; corrupts binary image formats.  Since the caller is
+ ;; asking for binary data, a unibyte string is a more
+ ;; appropriate representation anyway.
+ (set-buffer-multibyte nil)
+ (let ((coding-system-for-read 'no-conversion))
+   (apply #'call-process notmuch-command nil '(t nil) nil args)
+   (setq data (buffer-string)
+  (when cache
+   ;; Cheat.  part is non-nil, and `plist-put' always modifies
+   ;; the list in place if it's non-nil.
+   (plist-put part :binary-content data)))
+data))
+
+(defun notmuch-get-bodypart-text (msg part process-crypto optional cache)
   Return the text content of PART in MSG.
 
 This returns the content of the given part as a multibyte Lisp
 string after performing content transfer decoding and any
 necessary charset decoding.  It is an error to use this for
-non-text/* parts.
+non-text/* parts.
+
+If CACHE is non-nil, the content of this part will be saved in
+MSG (if it isn't already).
   (let ((content (plist-get part :content)))
 (when (not content)
   ;; Use show --format=sexp to fetch decoded content
@@ -572,7 +586,9 @@ (defun notmuch-get-bodypart-text (msg part process-crypto)
 (npart (apply #'notmuch-call-notmuch-sexp args)))
(setq content (plist-get npart :content))
(when (not content)
- (error Internal error: No :content from %S args
+ (error Internal error: No :content from %S args)))
+  (when cache
+   (plist-put part :content content)))
 content))
 
 ;; Workaround: The call to `mm-display-part' below triggers a bug in
-- 
2.1.3

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH 06/11] emacs: Remove broken `notmuch-get-bodypart-content' API

2015-01-24 Thread Austin Clements
On Fri, 11 Jul 2014, David Bremner da...@tethera.net wrote:
 Austin Clements amdra...@mit.edu writes:

 +This returns the content of the given part as a multibyte Lisp

 What does multibyte mean here? utf8? current encoding?

Elisp has two kinds of stings: unibyte strings and multibyte
strings.

  
https://www.gnu.org/software/emacs/manual/html_node/elisp/Non_002dASCII-in-Strings.html

You can think of unibyte strings as binary data; they're just vectors of
bytes without any particular encoding semantics (though when you use a
unibyte string you can endow it with encoding).  Multibyte strings,
however, are text; they're vectors of Unicode code points.

 +string after performing content transfer decoding and any
 +necessary charset decoding.  It is an error to use this for
 +non-text/* parts.
 +  (let ((content (plist-get part :content)))
 +(when (not content)
 +  ;; Use show --format=sexp to fetch decoded content
 +  (let* ((args `(show --format=sexp --include-html
 + ,(format --part=%s (plist-get part :id))
 + ,@(when process-crypto '(--decrypt))
 + ,(notmuch-id-to-query (plist-get msg :id
 + (npart (apply #'notmuch-call-notmuch-sexp args)))
 +(setq content (plist-get npart :content))
 +(when (not content)
 +  (error Internal error: No :content from %S args
 +content))

 I'm a bit curious at the lack of setting coding-system-for-read here.
 Are we assuming the user has their environment set up correctly? Not so
 much a criticism as being nervous about everything coding-system
 related.

That is interesting.  coding-system-for-read should really go in
notmuch-call-notmuch-sexp, but I worry that, while *almost* all strings
the CLI outputs are UTF-8, not quite all of them are.  For example, we
output filenames exactly at the OS reports the bytes to us (which is
necessary, in a sense, because POSIX enforces no particular encoding on
file names, but still really unfortunate).

We could set coding-system-for-read, but a full solution needs more
cooperation from the CLI.  Possibly the right answer, at least for the
sexp format, is to do our own UTF-8 to \u escapes for strings that
are known to be UTF-8 and leave the raw bytes for the few that aren't.
Then we would set the coding-system-for-read to 'no-conversion and I
think everything would Just Work.

That doesn't help for JSON, which is supposed to be all UTF-8 all the
time.  I can think of solutions there, but they're all ugly and involve
things like encoding filenames as base64 when they aren't valid UTF-8.

So...  I don't think I'm going to do anything about this at this moment.

 I didn't see anything else to object to in this patch or the previous
 one.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH 2/3] notmuch-mutt: support for messages that lack Message-ID headers

2015-01-24 Thread Jani Nikula
On Sat, 24 Jan 2015, Jan N. Klug jan.n.k...@rub.de wrote:
 On Sat, Jan 24, 2015 at 04:21:06PM +0100, Stefano Zacchiroli wrote:
 On Sat, Jan 24, 2015 at 04:59:16PM +0200, Jani Nikula wrote:
  On Sat, 24 Jan 2015, Stefano Zacchiroli z...@upsilon.cc wrote:
   To do the above, rewrite get_message_id() to scan the current message
   line by line, incrementally computing a SHA1. As a consequence, drop
   the dependency on Mail::Internet.
  
  I am not so sure this is a good idea however, see below.
 [...]
 But I didn't think of header folding, and you're absolutely correct in
 saying that might be a problem.

 I disagree. I do not think that header folding is a problem here. RFC
 5322 and RFC 2822 state that FWS in the message-id are not allowed (as
 opposed to In-Reply: and other headers, where FWS may occur between, but
 not inside message-ids).

Not *inside* message-ids, but as a data point, 0.05% of my email have
FWS *before* the message-id. That's more than the amount of messages
I have without a message-id.

BR,
Jani.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH 2/3] notmuch-mutt: support for messages that lack Message-ID headers

2015-01-24 Thread Stefano Zacchiroli
First of all thanks for your feedback, Jani!
(both here, and the other day on IRC, of course)

On Sat, Jan 24, 2015 at 04:59:16PM +0200, Jani Nikula wrote:
 On Sat, 24 Jan 2015, Stefano Zacchiroli z...@upsilon.cc wrote:
 
  For those messages, compute a synthetic Message-ID based on the SHA1
  of the whole message, in the same way that notmuch would do. See:
  http://git.notmuchmail.org/git/notmuch/blob/HEAD:/lib/sha1.c
 
 As I said on IRC, I think this is a notmuch implementation detail, and
 we don't make any promises about always generating missing message-ids
 the same way. That said, I don't see any reason why we'd change this
 anytime soon, so the solution is probably good enough for now.

ACK. I've noted down the code URL in the patch for reference, and also
as a warning to keep an eye on.

  To do the above, rewrite get_message_id() to scan the current message
  line by line, incrementally computing a SHA1. As a consequence, drop
  the dependency on Mail::Internet.
 
 I am not so sure this is a good idea however, see below.

So, fun part here is that I had initially made a similar comment to Jan.
Then got convinced to do as the current patch does to avoid loading full
messages (potentially with large attachments and the like) in memory.

But I didn't think of header folding, and you're absolutely correct in
saying that might be a problem.

I've just found out Mail::Header-new which I believe solves both
problems: avoid loading the full message into memory (it will only load
the headers) and proper parsing folded Message-IDs, without having to
use more complex regexp hackery.

Jan: do you agree with using Mail::Header-new and fall back to
line-by-line hasing only in case Message-ID is not found? If so, having
an updated patch based on the one I've posted here would be awesome! If
you cannot do that just let me know and I'll get to it, eventually :).

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH v4 0/5] Index the content-type of MIME parts

2015-01-24 Thread David Bremner
Todd t...@electricoding.com writes:

 I think I've finished incorporating the feedback.  The final
 notmuch-search-terms.rst could use more details, but it should
 probably occur after the recent patch that was posted documenting the
 probablistic indexing/searching has been committed.

Pushed this version, with amended NEWS, to master.

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [LATE NIGHT PATCH] build gzipped apidoc in case have doxygen but not sphinx

2015-01-24 Thread David Bremner
David Bremner da...@tethera.net writes:

 Tomi Ollila tomi.oll...@iki.fi writes:

 In case we had doxygen but not sphinx notmuch.3 was created but
 notmuch.3.gz not -- which means install fails!
 This patch (with late night unpolished commit message will fix that)

 By install fails, you mean the man page is simply not installed? I agree
 that's a bug, but I had visions of the install failing half way through.

A related issue I noticed is that some tests fail, rather than being
skipped, if the man pages are not built.

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH] emacs: escape % in header line format

2015-01-24 Thread David Bremner
David Bremner da...@tethera.net writes:

 Jinwoo Lee jinwo...@gmail.com writes:

 Yup.  It works!  Thanks for the quick fix.  Is this going to be merged
 to HEAD soon?


 Probably in the next day or so, unless somebody complains.

This bug should be fixed in commit cc3d25d

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [LATE NIGHT PATCH] build gzipped apidoc in case have doxygen but not sphinx

2015-01-24 Thread David Bremner
Tomi Ollila tomi.oll...@iki.fi writes:

 In case we had doxygen but not sphinx notmuch.3 was created but
 notmuch.3.gz not -- which means install fails!
 This patch (with late night unpolished commit message will fix that)

By install fails, you mean the man page is simply not installed? I agree
that's a bug, but I had visions of the install failing half way through.

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 3/3] debian packaging: drop dep notmuch-mutt - libmailtools-perl

2015-01-24 Thread Stefano Zacchiroli
Rationale: the dependency is no longer needed after the rewrite of
get_message_id().

Note that Digest::SHA, needed by the new implementation of
get_message_id(), is shipped by the perl package itself, which is in
an implied dependency of other lib*-perl packages. So there is no need
to list perl explicitly as a dependency.
---
 debian/changelog | 8 
 debian/control   | 2 +-
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/debian/changelog b/debian/changelog
index fbd15b8..2fa0f96 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,11 @@
+notmuch (0.19-2) experimental; urgency=medium
+
+  [ Stefano Zacchiroli ]
+  * notmuch-mutt: drop dependency on libmailtools-perl, because we no
+longer need Mail::Internet
+
+ --
+
 notmuch (0.19-1) experimental; urgency=medium
 
   * New upstream release.
diff --git a/debian/control b/debian/control
index 4bc4cd9..84dbf32 100644
--- a/debian/control
+++ b/debian/control
@@ -144,7 +144,7 @@ Package: notmuch-mutt
 Architecture: all
 Depends:
  notmuch (= 0.4),
- libmail-box-perl, libmailtools-perl,
+ libmail-box-perl,
  libstring-shellquote-perl, libterm-readline-gnu-perl,
  ${misc:Depends}
 Recommends: mutt
-- 
2.1.4

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 1/3] notmuch-mutt README: use metacpn.org/* as deps homepages

2015-01-24 Thread Stefano Zacchiroli
---
 contrib/notmuch-mutt/README | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/contrib/notmuch-mutt/README b/contrib/notmuch-mutt/README
index 382ac91..c661447 100644
--- a/contrib/notmuch-mutt/README
+++ b/contrib/notmuch-mutt/README
@@ -33,13 +33,13 @@ Requirements
 
 To *run* notmuch-mutt you will need Perl with the following libraries:
 
-- Mail::Box http://search.cpan.org/~markov/Mail-Box/
+- Mail::Box https://metacpan.org/pod/Mail::Box
   (Debian package: libmail-box-perl)
-- Mail::Internet http://search.cpan.org/~markov/MailTools/
+- Mail::Internet https://metacpan.org/pod/Mail::Internet
   (Debian package: libmailtools-perl)
-- String::ShellQuote 
http://search.cpan.org/~rosch/String-ShellQuote/ShellQuote.pm
+- String::ShellQuote https://metacpan.org/pod/String::ShellQuote
   (Debian package: libstring-shellquote-perl)
-- Term::ReadLine http://search.cpan.org/~hayashi/Term-ReadLine-Gnu/
+- Term::ReadLine::Gnu https://metacpan.org/pod/Term::ReadLine::Gnu
   (Debian package: libterm-readline-gnu-perl)
 
 To *build* notmuch-mutt documentation you will need:
-- 
2.1.4

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 2/3] notmuch-mutt: support for messages that lack Message-ID headers

2015-01-24 Thread Stefano Zacchiroli
From: Jan N. Klug jan.n.k...@rub.de

For those messages, compute a synthetic Message-ID based on the SHA1
of the whole message, in the same way that notmuch would do. See:
http://git.notmuchmail.org/git/notmuch/blob/HEAD:/lib/sha1.c

To do the above, rewrite get_message_id() to scan the current message
line by line, incrementally computing a SHA1. As a consequence, drop
the dependency on Mail::Internet.

Signed-off-by: Stefano Zacchiroli z...@upsilon.cc
---
 contrib/notmuch-mutt/README   |  4 ++--
 contrib/notmuch-mutt/notmuch-mutt | 23 ++-
 2 files changed, 20 insertions(+), 7 deletions(-)

diff --git a/contrib/notmuch-mutt/README b/contrib/notmuch-mutt/README
index c661447..0013ed0 100644
--- a/contrib/notmuch-mutt/README
+++ b/contrib/notmuch-mutt/README
@@ -33,10 +33,10 @@ Requirements
 
 To *run* notmuch-mutt you will need Perl with the following libraries:
 
+- Digest::SHA https://metacpan.org/release/Digest-SHA
+  (Debian package: libdigest-sha-perl)
 - Mail::Box https://metacpan.org/pod/Mail::Box
   (Debian package: libmail-box-perl)
-- Mail::Internet https://metacpan.org/pod/Mail::Internet
-  (Debian package: libmailtools-perl)
 - String::ShellQuote https://metacpan.org/pod/String::ShellQuote
   (Debian package: libstring-shellquote-perl)
 - Term::ReadLine::Gnu https://metacpan.org/pod/Term::ReadLine::Gnu
diff --git a/contrib/notmuch-mutt/notmuch-mutt 
b/contrib/notmuch-mutt/notmuch-mutt
index 4969e4b..4d30b0b 100755
--- a/contrib/notmuch-mutt/notmuch-mutt
+++ b/contrib/notmuch-mutt/notmuch-mutt
@@ -13,11 +13,11 @@ use warnings;
 
 use File::Path;
 use Getopt::Long qw(:config no_getopt_compat);
-use Mail::Internet;
 use Mail::Box::Maildir;
 use Pod::Usage;
 use String::ShellQuote;
 use Term::ReadLine;
+use Digest::SHA;
 
 
 my $xdg_cache_dir = $ENV{HOME}/.cache;
@@ -75,10 +75,23 @@ sub prompt($$) {
 }
 
 sub get_message_id() {
-my $mail = Mail::Internet-new(\*STDIN);
-my $mid = $mail-head-get(message-id) or return undef;
-$mid =~ /^(.*)$/;# get message-id value
-return $1;
+my $mid = undef;
+my $sha = Digest::SHA-new(1);  # SHA1 hashing
+
+while(STDIN) {  # scan message line by line, looking for mid
+if ($_ =~ /^Message-ID:\s*(.*)$/i) {
+   $mid = $1;
+   last;  # message-id found, abort scan
+}
+   $sha-add($_);  # update hash
+}
+
+# Generate message-id from hash if none was found, in the same way
+# that notmuch would do.
+# See: http://git.notmuchmail.org/git/notmuch/blob/HEAD:/lib/sha1.c
+$mid ||= notmuch-sha1-.$sha-hexdigest;
+
+return $mid;
 }
 
 sub search_action($$$@) {
-- 
2.1.4

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH] notmuch-mutt: support for messages that lack Message-ID

2015-01-24 Thread Stefano Zacchiroli
Please find attached a patch series that add support for messages that
lack Message-ID to notmuch-mutt, kindly provided by Jan N. Klug. I've
reviewed / adapted it, and I consider it suitable for inclusion into
the next release of notmuch(-mutt).

Please let me know if you've any question.

Cheers.

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH 2/3] notmuch-mutt: support for messages that lack Message-ID headers

2015-01-24 Thread Jani Nikula
On Sat, 24 Jan 2015, Stefano Zacchiroli z...@upsilon.cc wrote:
 From: Jan N. Klug jan.n.k...@rub.de

 For those messages, compute a synthetic Message-ID based on the SHA1
 of the whole message, in the same way that notmuch would do. See:
 http://git.notmuchmail.org/git/notmuch/blob/HEAD:/lib/sha1.c

As I said on IRC, I think this is a notmuch implementation detail, and
we don't make any promises about always generating missing message-ids
the same way. That said, I don't see any reason why we'd change this
anytime soon, so the solution is probably good enough for now.

 To do the above, rewrite get_message_id() to scan the current message
 line by line, incrementally computing a SHA1. As a consequence, drop
 the dependency on Mail::Internet.

I am not so sure this is a good idea however, see below.

 Signed-off-by: Stefano Zacchiroli z...@upsilon.cc
 ---
  contrib/notmuch-mutt/README   |  4 ++--
  contrib/notmuch-mutt/notmuch-mutt | 23 ++-
  2 files changed, 20 insertions(+), 7 deletions(-)

 diff --git a/contrib/notmuch-mutt/README b/contrib/notmuch-mutt/README
 index c661447..0013ed0 100644
 --- a/contrib/notmuch-mutt/README
 +++ b/contrib/notmuch-mutt/README
 @@ -33,10 +33,10 @@ Requirements
  
  To *run* notmuch-mutt you will need Perl with the following libraries:
  
 +- Digest::SHA https://metacpan.org/release/Digest-SHA
 +  (Debian package: libdigest-sha-perl)
  - Mail::Box https://metacpan.org/pod/Mail::Box
(Debian package: libmail-box-perl)
 -- Mail::Internet https://metacpan.org/pod/Mail::Internet
 -  (Debian package: libmailtools-perl)
  - String::ShellQuote https://metacpan.org/pod/String::ShellQuote
(Debian package: libstring-shellquote-perl)
  - Term::ReadLine::Gnu https://metacpan.org/pod/Term::ReadLine::Gnu
 diff --git a/contrib/notmuch-mutt/notmuch-mutt 
 b/contrib/notmuch-mutt/notmuch-mutt
 index 4969e4b..4d30b0b 100755
 --- a/contrib/notmuch-mutt/notmuch-mutt
 +++ b/contrib/notmuch-mutt/notmuch-mutt
 @@ -13,11 +13,11 @@ use warnings;
  
  use File::Path;
  use Getopt::Long qw(:config no_getopt_compat);
 -use Mail::Internet;
  use Mail::Box::Maildir;
  use Pod::Usage;
  use String::ShellQuote;
  use Term::ReadLine;
 +use Digest::SHA;
  
  
  my $xdg_cache_dir = $ENV{HOME}/.cache;
 @@ -75,10 +75,23 @@ sub prompt($$) {
  }
  
  sub get_message_id() {
 -my $mail = Mail::Internet-new(\*STDIN);
 -my $mid = $mail-head-get(message-id) or return undef;
 -$mid =~ /^(.*)$/;  # get message-id value
 -return $1;
 +my $mid = undef;
 +my $sha = Digest::SHA-new(1);  # SHA1 hashing
 +
 +while(STDIN) {  # scan message line by line, looking for mid
 +if ($_ =~ /^Message-ID:\s*(.*)$/i) {

This doesn't take into account header folding or end of headers. There's
probably a bunch of other things to consider in the relevant RFCs. I
think you really should use Mail::Internet for the common case, and only
fallback to generating the message-id when that fails.

BR,
Jani.

 + $mid = $1;
 + last;  # message-id found, abort scan
 +}
 + $sha-add($_);  # update hash
 +}
 +
 +# Generate message-id from hash if none was found, in the same way
 +# that notmuch would do.
 +# See: http://git.notmuchmail.org/git/notmuch/blob/HEAD:/lib/sha1.c
 +$mid ||= notmuch-sha1-.$sha-hexdigest;
 +
 +return $mid;
  }
  
  sub search_action($$$@) {
 -- 
 2.1.4

 ___
 notmuch mailing list
 notmuch@notmuchmail.org
 http://notmuchmail.org/mailman/listinfo/notmuch
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH 00/11] Improve charset and cid: handling

2015-01-24 Thread Austin Clements
I added declare-functions for both of these, which should take care of
the Emacs 23 warnings and be more robust on Emacs 24.  We can't reach
the function that calls these unless shr is actually available, but the
byte compiler doesn't know that.

On Sat, 26 Apr 2014, Mark Walters markwalters1...@gmail.com wrote:
 Aside from the minor comments I mentioned in previous emails and one
 more comment below this looks good. 

 The extra comment is that on emacs23 I get the following when compiling:

 In end of data:
 notmuch-show.el:2188:1:Warning: the following functions are not known to be
 defined: libxml-parse-html-region, shr-insert-document

 Finally, I have not really tested it as I mainly use emacs23

 Best wishes

 Mark




 On Mon, 21 Apr 2014, Austin Clements amdra...@mit.edu wrote:
 I set out to quickly add support for cid: links in the shr renderer
 and wound up making our charset handling more robust and rewriting our
 content-ID handling.  The test introduced in patch 2 passes in all but
 one really obscure case, but only because of many unwritten and
 potentially fragile assumptions that Emacs and the CLI make about each
 other.

 The first three patches could reasonably go in to 0.18.  The rest of
 this series is certainly post-0.18, but I didn't want to lose track of
 it.

 This series comes in three stages.  Each depends on the earlier ones,
 but each prefix makes sense on its own and could be pushed without the
 later stages.

 Patch 1 is a simple clean up patch.

 Patches 2 through 7 robust-ify our charset handling in Emacs, mostly
 by splitting the broken `notmuch-get-bodypart-content' API into
 `notmuch-get-bodypart-binary' and `notmuch-get-bodypart-text' so a
 caller can explicitly convey their requirements.

 The remaining patches improve our content-ID handling and add support
 for cid: links for shr.

 ___
 notmuch mailing list
 notmuch@notmuchmail.org
 http://notmuchmail.org/mailman/listinfo/notmuch
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch