[PATCH 2/2] NEWS: News for big endian sha1 bug fix.

[Austin Clements]

Quoth david at tethera.net on Nov 24 at  5:29 pm:
> From: David Bremner 
> 
> We could give more details about how to migrate tags, but I'm not sure
> that it's a practical problem, or just a theoretical one.
> ---
>  NEWS | 14 +-
>  1 file changed, 13 insertions(+), 1 deletion(-)
> 
> diff --git a/NEWS b/NEWS
> index 3383ecf..31c6284 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -1,6 +1,18 @@
> -Notmuch 0.17~rc1 (2013-11-20)
> +Notmuch 0.17~rc2 (2013-xx-yy)
>  =
>  
> +Incompatible change in SHA1 computation
> +---
> +
> +Previously on big endian architectures like sparc and powerpc the
> +computation of SHA1 hashes was incorrect. This meant that messages
> +with overlong or missing message-ids were given different computed
> +message-ids than on more common little endian architectures like i386
> +and amd64.  If you use notmuch on a big endian architecture, you are
> +strongly advised to make a backup of your tags using `notmuch dump`
> +before this upgrade. It should be possible to migrate the tags using a
> +script.
> +

Should this mention how to find such messages?  Something like

  notmuch dump |
  awk '/^notmuch-sha1-[0-9a-f]{40} / {system("notmuch search id:" $1)}'

[PATCH] new: Detect dirent.d_type support at configure time

[Tomi Ollila]

On Sun, Nov 24 2013, Austin Clements  wrote:

> Support for dirent.d_type is OS-specific.  Previously, we used
> _DIRENT_HAVE_D_TYPE to detect support for this, but this is apparently
> a glic-ism (FreeBSD, for example, supports d_type, but does not define
> this).  Since there's no cross-platform way to detect support for
> dirent.d_type, detect it using a test compile at configure time.
> ---

LGTM.

Tomi

>  compat/have_d_type.c | 10 ++
>  configure| 16 
>  notmuch-new.c|  2 +-
>  3 files changed, 27 insertions(+), 1 deletion(-)
>  create mode 100644 compat/have_d_type.c
>
> diff --git a/compat/have_d_type.c b/compat/have_d_type.c
> new file mode 100644
> index 000..9ca6c6e
> --- /dev/null
> +++ b/compat/have_d_type.c
> @@ -0,0 +1,10 @@
> +#include 
> +
> +int main()
> +{
> +struct dirent ent;
> +
> +(void) ent.d_type;
> +
> +return 0;
> +}
> diff --git a/configure b/configure
> index 1a8e939..d2d193c 100755
> --- a/configure
> +++ b/configure
> @@ -557,6 +557,17 @@ else
>  fi
>  rm -f compat/have_timegm
>  
> +printf "Checking for dirent.d_type... "
> +if ${CC} -o compat/have_d_type "$srcdir"/compat/have_d_type.c > /dev/null 
> 2>&1
> +then
> +printf "Yes.\n"
> +have_d_type="1"
> +else
> +printf "No (will use stat instead).\n"
> +have_d_type="0"
> +fi
> +rm -f compat/have_d_type
> +
>  printf "Checking for standard version of getpwuid_r... "
>  if ${CC} -o compat/check_getpwuid "$srcdir"/compat/check_getpwuid.c > 
> /dev/null 2>&1
>  then
> @@ -745,6 +756,9 @@ HAVE_STRCASESTR = ${have_strcasestr}
>  # build its own version)
>  HAVE_STRSEP = ${have_strsep}
>  
> +# Whether struct dirent has d_type (if not, then notmuch will use stat)
> +HAVE_D_TYPE = ${have_d_type}
> +
>  # Whether the Xapian version in use supports compaction
>  HAVE_XAPIAN_COMPACT = ${have_xapian_compact}
>  
> @@ -805,6 +819,7 @@ CONFIGURE_CFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) 
> \$(GMIME_CFLAGS)  \\
>  \$(VALGRIND_CFLAGS)   \\
>  -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR) \\
>  -DHAVE_STRSEP=\$(HAVE_STRSEP) \\
> +-DHAVE_D_TYPE=\$(HAVE_D_TYPE) \\
>  -DSTD_GETPWUID=\$(STD_GETPWUID)   \\
>  -DSTD_ASCTIME=\$(STD_ASCTIME) \\
>  -DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT)
> @@ -813,6 +828,7 @@ CONFIGURE_CXXFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) 
> \$(GMIME_CFLAGS)\\
>\$(VALGRIND_CFLAGS) \$(XAPIAN_CXXFLAGS) \\
>-DHAVE_STRCASESTR=\$(HAVE_STRCASESTR)   \\
>-DHAVE_STRSEP=\$(HAVE_STRSEP)   \\
> +  -DHAVE_D_TYPE=\$(HAVE_D_TYPE)   \\
>-DSTD_GETPWUID=\$(STD_GETPWUID) \\
>-DSTD_ASCTIME=\$(STD_ASCTIME)   \\
>-DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT)
> diff --git a/notmuch-new.c b/notmuch-new.c
> index ba05cb4..423e188 100644
> --- a/notmuch-new.c
> +++ b/notmuch-new.c
> @@ -167,7 +167,7 @@ dirent_type (const char *path, const struct dirent *entry)
>  char *abspath;
>  int err, saved_errno;
>  
> -#ifdef _DIRENT_HAVE_D_TYPE
> +#if HAVE_D_TYPE
>  /* Mapping from d_type to stat mode_t.  We omit DT_LNK so that
>   * we'll fall through to stat and get the real file type. */
>  static const mode_t modes[] = {
> -- 
> 1.8.4.rc3
>
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

notmuch sha1 implementation broken on (some) big-endian architectures

[Tomi Ollila]

On Sun, Nov 24 2013, David Bremner  wrote:

> The following code, when linked with libnotmuch.a and libutil.a does a
> passable imitation of sha1sum on amd64 (and I guess also i386) but
> computes a different digest on powerpc and probably sparc and s390x.
>
> In the long run we should maybe outsource hash computations to
> e.g. librhash, but I'd like a simpler fix for 0.17, if possible
>
> P.S. I blame Austin for adding the "missing-headers" test which found
> this bug ;).

This is interesting problem, I would have guessed that this would
fails on LITTLE_ENDIAN machines easier, if ever...

... especially as there is line

lib/libsha1.c:52:#if (PLATFORM_BYTE_ORDER == IS_LITTLE_ENDIAN)

... but...

I could not find any (other) matches for PLATFORM_BYTE_ORDER nor
IS_LITTLE_ENDIAN in source code or in /usr/include/**/*.h or
in /usr/lib/gcc/**/*.h

I did some testing and it seems that 
#if (F == BBBAAARRR)

#endif

will have  in output file in case neither of the above are
defined... :/

So, this could work:

#if   // for BYTE_ORDER && LITTLE_ENDIAN

and then

#if (BYTE_ORDER == LITTLE_ENDIAN)
...

to replace lib/libsha1.c:53 (53 now after endian.h added)


Please test on BIG_ENDIAN machine...


In case this works, then we'd need to inform users that their
long/missing Message ID:s are now coded differently in their
databases...


Tomi

[PATCH 2/2] NEWS: News for big endian sha1 bug fix.

[da...@tethera.net]

From: David Bremner 

We could give more details about how to migrate tags, but I'm not sure
that it's a practical problem, or just a theoretical one.
---
 NEWS | 14 +-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/NEWS b/NEWS
index 3383ecf..31c6284 100644
--- a/NEWS
+++ b/NEWS
@@ -1,6 +1,18 @@
-Notmuch 0.17~rc1 (2013-11-20)
+Notmuch 0.17~rc2 (2013-xx-yy)
 =

+Incompatible change in SHA1 computation
+---
+
+Previously on big endian architectures like sparc and powerpc the
+computation of SHA1 hashes was incorrect. This meant that messages
+with overlong or missing message-ids were given different computed
+message-ids than on more common little endian architectures like i386
+and amd64.  If you use notmuch on a big endian architecture, you are
+strongly advised to make a backup of your tags using `notmuch dump`
+before this upgrade. It should be possible to migrate the tags using a
+script.
+
 Command-Line Interface
 --

-- 
1.8.4.2

[PATCH 1/2] lib: fix byte order test in libsha1.c

[da...@tethera.net]

From: David Bremner 

Previously PLATFORM_BYTE_ORDER and IS_LITTLE_ENDIAN were not defined,
so the little endian code was always compiled in.

This will have the effect that the "SHA1s" on big endian architectures
will change (i.e. become actual sha1s). So someone re-indexing their
database could conceivable lose tags on messages without a message-id
header.
---
 lib/libsha1.c | 14 ++
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/lib/libsha1.c b/lib/libsha1.c
index 5d16f6a..794854b 100644
--- a/lib/libsha1.c
+++ b/lib/libsha1.c
@@ -49,11 +49,17 @@ extern "C"

 #define bswap_32(x) ((rotr32((x), 24) & 0x00ff00ff) | (rotr32((x), 8) & 
0xff00ff00))

-#if (PLATFORM_BYTE_ORDER == IS_LITTLE_ENDIAN)
-#define bsw_32(p,n) \
-{ int _i = (n); while(_i--) ((uint32_t*)p)[_i] = 
bswap_32(((uint32_t*)p)[_i]); }
+#ifdef __BYTE_ORDER__
+#  if (__BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__)
+#define bsw_32(p,n) \
+   { int _i = (n); while(_i--) ((uint32_t*)p)[_i] = 
bswap_32(((uint32_t*)p)[_i]); }
+#  elif (__BYTE_ORDER__ == __ORDER_BIG_ENDIAN__)
+#define bsw_32(p,n)
+#  else
+#error "unknown byte order"
+#  endif
 #else
-#define bsw_32(p,n)
+#error "macro __BYTE_ORDER__ is not defined"
 #endif

 #define SHA1_MASK   (SHA1_BLOCK_SIZE - 1)
-- 
1.8.4.2

[PATCH] Add NEWS entry for notmuch-compact

[David Bremner]

Ben Gamari  writes:

> Signed-off-by: Ben Gamari 
> ---
>  NEWS | 8 
>  1 file changed, 8 insertions(+)

pushed. 

d

crypto test failures on debian/kfreebsd.

[David Bremner]

0.17~rc1 is failing several crypto tests on debian/kfreebsd [1]


 FAIL   signature verification
--- crypto.2.expected   2013-11-24 04:17:13.0 +
+++ crypto.2.output 2013-11-24 04:17:13.0 +
@@ -11,7 +11,7 @@
 "id": 2
 },
 {
-"content-length": 315,
+"content-length": 318,
 "content-type": 
"application/pgp-signature",
 "id": 3
 }


According to dkg's nifty printmimestructure script, the result computed
by notmuch is actually correct. So it looks like different environments
result in different content-length headers. I'm not sure if this is a
bug that should be tracked down, or we should just sanitize the test
data more.

[1]: 
https://buildd.debian.org/status/fetch.php?pkg=notmuch=kfreebsd-i386=0.17%7Erc1-1=1385266740

[PATCH WIP] emacs: show: redesign unread/read logic

[Mark Walters]

The decisions of when to mark messages read in notmuch-show has caused
confusion/irritation (several discussions on irc and the mailing list
eg the thread starting at id:87hadi0xse.fsf at boo.workgroup). This is an
attempt to get some logic that people are happier with.

Some examples of the current problems are: notmuch marks sometimes
closed messages read, notmuch does not mark messages read if you page
down through them, and notmuch removes the unread tag too soon: when
you first see the message you do not if you have read it before.

The patch separates out two things "seeing" a message and "marking it
read".

A message is deemed seen if both the top and bottom of the message
have both been visible in the buffer's window. This is chosen so that
just seeing 1 or 2 lines of a message at the bottom of the window does
not mark it seen. A closed message is never marked seen.

The seen status is updated via a command-hook (run on every
command/key-press) so essentially any change which sees a message
should mark it as seen.

By default the unread status of seen messages is not updated until the
user quits the show buffer, and the user has the option of prefix-arg
quit to exit the show buffer without updating the unread status.

However, if the user sets the custom variable
notmuch-show-update-unread-on-seen then the unread status is updated
(ie unread tag is removed) as soon as a message is seen (in the above
sense).
---

This patch brings the unread handling roughly in line with what I
would expect, and is reasonably close to the suggestions from Austin
id:20131005162202.GJ21611 at mit.edu and Jani
id:87vc1aho64.fsf at nikula.org. It was also clear from the discussion
that different people want different things so we will need some
customisation possibilities.

It is a large patch: I am afraid I don't see a way round that.

At the moment there are two things that need fixing: first tree-view
assumes the old behaviour so its displayed tags get out of sync with
the actual tags. Secondly, a lot of tests fail for the obvious reason
that the unread tag is not removed at the same time as before.

It would be very helpful if people could test and see whether it works
as they would like, and if not say why/when it is doing the wrong
thing and what it should do in those cases.

Best wishes

Mark


 emacs/notmuch-show.el |  151 -
 emacs/notmuch-tree.el |   20 +--
 2 files changed, 151 insertions(+), 20 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index 784644c..1081eb0 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -168,6 +168,10 @@ each attachment handler is logged in buffers with names 
beginning
 \" *notmuch-part*\". This option requires emacs version at least
 24.3 to work.")

+(defvar notmuch-show-seen-plist nil)
+(make-variable-buffer-local 'notmuch-show-seen-plist)
+(put 'notmuch-show-seen-plist 'permanent-local t)
+
 (defcustom notmuch-show-stash-mlarchive-link-alist
   '(("Gmane" . "http://mid.gmane.org/;)
 ("MARC" . "http://marc.info/?i=;)
@@ -211,6 +215,15 @@ For example, if you wanted to remove an \"unread\" tag and 
add a
   :type '(repeat string)
   :group 'notmuch-show)

+(defcustom notmuch-show-update-unread-on-seen nil
+  "Update unread tags when seen rathe than when exiting show buffer.
+
+A message is seen if the top and bottom of the message have both
+been visible in the buffer. When this is nil the unread status is
+updated on exiting the show buffer. When this is t the unread
+status is updated as soon as the message is seen."
+  :type 'boolean
+  :group 'notmuch-show)

 (defmacro with-current-notmuch-show-message ( body)
   "Evaluate body with current buffer set to the text of current message"
@@ -1142,6 +1155,8 @@ function is used."
   (let ((inhibit-read-only t))

 (notmuch-show-mode)
+(add-hook 'post-command-hook #'notmuch-show-command-hook nil t)
+
 ;; Don't track undo information for this buffer
 (set 'buffer-undo-list t)

@@ -1213,6 +1228,12 @@ preferences. If invoked with a prefix argument (or 
RESET-STATE is
 non-nil) then the state of the buffer (open/closed messages) is
 reset based on the original query."
   (interactive "P")
+  ;; Do not mark seen messages read if we are resetting state. The
+  ;; idea is that resetting state is asking for the view to be reset
+  ;; to the current state of the database.
+  (unless notmuch-show-update-unread-on-seen
+(notmuch-show-mark-all-seen-read reset-state))
+
   (let ((inhibit-read-only t)
(state (unless reset-state
 (notmuch-show-capture-state
@@ -1258,6 +1279,8 @@ reset based on the original query."
 (defvar notmuch-show-mode-map
   (let ((map (make-sparse-keymap)))
(set-keymap-parent map notmuch-common-keymap)
+   ;; the following overrides the common-keymap quit
+   (define-key map [remap notmuch-kill-this-buffer] 
'notmuch-show-quit-and-mark-read)
(define-key 

notmuch sha1 implementation broken on (some) big-endian architectures

[David Bremner]

David Bremner  writes:

> The following code, when linked with libnotmuch.a and libutil.a does a
> passable imitation of sha1sum on amd64 (and I guess also i386) but
> computes a different digest on powerpc and probably sparc and s390x.
>
> In the long run we should maybe outsource hash computations to
> e.g. librhash, but I'd like a simpler fix for 0.17, if possible

Out of curiousity, I tried out a similar example with librhash, and it
works fine on powerpc.

#include 
#include "rhash.h" /* LibRHash interface */

int main(int argc, char *argv[])
{
  char digest[64];
  char output[130];

  rhash_library_init(); /* initialize static data */

  int res = rhash_file(RHASH_SHA1, argv[1], digest);
  if(res < 0) {
fprintf(stderr, "LibRHash error: %s: %s\n", argv[1], strerror(errno));
return 1;
  }

  /* convert binary digest to hexadecimal string */
  rhash_print_bytes(output, digest, rhash_get_digest_size(RHASH_SHA1),RHPR_HEX);

  printf("%s (%s) = %s\n", rhash_get_name(RHASH_SHA1), argv[1], output);
  return 0;
}

notmuch sha1 implementation broken on (some) big-endian architectures

[David Bremner]

The following code, when linked with libnotmuch.a and libutil.a does a
passable imitation of sha1sum on amd64 (and I guess also i386) but
computes a different digest on powerpc and probably sparc and s390x.

In the long run we should maybe outsource hash computations to
e.g. librhash, but I'd like a simpler fix for 0.17, if possible

P.S. I blame Austin for adding the "missing-headers" test which found
this bug ;).

/* 8<- */

#include 

#include "notmuch.h"
char * notmuch_sha1_of_file(const char* filename);

int
main (int argc, char **argv)
{

char *digest = notmuch_sha1_of_file (argv[1]);

printf("%s  %s\n",digest,argv[1]);
return 0;
}

[PATCH WIP] emacs: show: redesign unread/read logic

[Mark Walters]

The decisions of when to mark messages read in notmuch-show has caused
confusion/irritation (several discussions on irc and the mailing list
eg the thread starting at id:87hadi0xse.fsf@boo.workgroup). This is an
attempt to get some logic that people are happier with.

Some examples of the current problems are: notmuch marks sometimes
closed messages read, notmuch does not mark messages read if you page
down through them, and notmuch removes the unread tag too soon: when
you first see the message you do not if you have read it before.

The patch separates out two things seeing a message and marking it
read.

A message is deemed seen if both the top and bottom of the message
have both been visible in the buffer's window. This is chosen so that
just seeing 1 or 2 lines of a message at the bottom of the window does
not mark it seen. A closed message is never marked seen.

The seen status is updated via a command-hook (run on every
command/key-press) so essentially any change which sees a message
should mark it as seen.

By default the unread status of seen messages is not updated until the
user quits the show buffer, and the user has the option of prefix-arg
quit to exit the show buffer without updating the unread status.

However, if the user sets the custom variable
notmuch-show-update-unread-on-seen then the unread status is updated
(ie unread tag is removed) as soon as a message is seen (in the above
sense).
---

This patch brings the unread handling roughly in line with what I
would expect, and is reasonably close to the suggestions from Austin
id:20131005162202.gj21...@mit.edu and Jani
id:87vc1aho64@nikula.org. It was also clear from the discussion
that different people want different things so we will need some
customisation possibilities.

It is a large patch: I am afraid I don't see a way round that.

At the moment there are two things that need fixing: first tree-view
assumes the old behaviour so its displayed tags get out of sync with
the actual tags. Secondly, a lot of tests fail for the obvious reason
that the unread tag is not removed at the same time as before.

It would be very helpful if people could test and see whether it works
as they would like, and if not say why/when it is doing the wrong
thing and what it should do in those cases.

Best wishes

Mark


 emacs/notmuch-show.el |  151 -
 emacs/notmuch-tree.el |   20 +--
 2 files changed, 151 insertions(+), 20 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index 784644c..1081eb0 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -168,6 +168,10 @@ each attachment handler is logged in buffers with names 
beginning
 \ *notmuch-part*\. This option requires emacs version at least
 24.3 to work.)
 
+(defvar notmuch-show-seen-plist nil)
+(make-variable-buffer-local 'notmuch-show-seen-plist)
+(put 'notmuch-show-seen-plist 'permanent-local t)
+
 (defcustom notmuch-show-stash-mlarchive-link-alist
   '((Gmane . http://mid.gmane.org/;)
 (MARC . http://marc.info/?i=;)
@@ -211,6 +215,15 @@ For example, if you wanted to remove an \unread\ tag and 
add a
   :type '(repeat string)
   :group 'notmuch-show)
 
+(defcustom notmuch-show-update-unread-on-seen nil
+  Update unread tags when seen rathe than when exiting show buffer.
+
+A message is seen if the top and bottom of the message have both
+been visible in the buffer. When this is nil the unread status is
+updated on exiting the show buffer. When this is t the unread
+status is updated as soon as the message is seen.
+  :type 'boolean
+  :group 'notmuch-show)
 
 (defmacro with-current-notmuch-show-message (rest body)
   Evaluate body with current buffer set to the text of current message
@@ -1142,6 +1155,8 @@ function is used.
   (let ((inhibit-read-only t))
 
 (notmuch-show-mode)
+(add-hook 'post-command-hook #'notmuch-show-command-hook nil t)
+
 ;; Don't track undo information for this buffer
 (set 'buffer-undo-list t)
 
@@ -1213,6 +1228,12 @@ preferences. If invoked with a prefix argument (or 
RESET-STATE is
 non-nil) then the state of the buffer (open/closed messages) is
 reset based on the original query.
   (interactive P)
+  ;; Do not mark seen messages read if we are resetting state. The
+  ;; idea is that resetting state is asking for the view to be reset
+  ;; to the current state of the database.
+  (unless notmuch-show-update-unread-on-seen
+(notmuch-show-mark-all-seen-read reset-state))
+
   (let ((inhibit-read-only t)
(state (unless reset-state
 (notmuch-show-capture-state
@@ -1258,6 +1279,8 @@ reset based on the original query.
 (defvar notmuch-show-mode-map
   (let ((map (make-sparse-keymap)))
(set-keymap-parent map notmuch-common-keymap)
+   ;; the following overrides the common-keymap quit
+   (define-key map [remap notmuch-kill-this-buffer] 
'notmuch-show-quit-and-mark-read)
(define-key map Z 

notmuch sha1 implementation broken on (some) big-endian architectures

[David Bremner]

The following code, when linked with libnotmuch.a and libutil.a does a
passable imitation of sha1sum on amd64 (and I guess also i386) but
computes a different digest on powerpc and probably sparc and s390x.

In the long run we should maybe outsource hash computations to
e.g. librhash, but I'd like a simpler fix for 0.17, if possible

P.S. I blame Austin for adding the missing-headers test which found
this bug ;).

/* 8- */

#include stdio.h

#include notmuch.h
char * notmuch_sha1_of_file(const char* filename);

int
main (int argc, char **argv)
{

char *digest = notmuch_sha1_of_file (argv[1]);

printf(%s  %s\n,digest,argv[1]);
return 0;
}
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: notmuch sha1 implementation broken on (some) big-endian architectures

[David Bremner]

David Bremner da...@tethera.net writes:

 The following code, when linked with libnotmuch.a and libutil.a does a
 passable imitation of sha1sum on amd64 (and I guess also i386) but
 computes a different digest on powerpc and probably sparc and s390x.

 In the long run we should maybe outsource hash computations to
 e.g. librhash, but I'd like a simpler fix for 0.17, if possible

Out of curiousity, I tried out a similar example with librhash, and it
works fine on powerpc.

#include errno.h
#include rhash.h /* LibRHash interface */

int main(int argc, char *argv[])
{
  char digest[64];
  char output[130];

  rhash_library_init(); /* initialize static data */

  int res = rhash_file(RHASH_SHA1, argv[1], digest);
  if(res  0) {
fprintf(stderr, LibRHash error: %s: %s\n, argv[1], strerror(errno));
return 1;
  }

  /* convert binary digest to hexadecimal string */
  rhash_print_bytes(output, digest, rhash_get_digest_size(RHASH_SHA1),RHPR_HEX);

  printf(%s (%s) = %s\n, rhash_get_name(RHASH_SHA1), argv[1], output);
  return 0;
}
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

crypto test failures on debian/kfreebsd.

[David Bremner]

0.17~rc1 is failing several crypto tests on debian/kfreebsd [1]


 FAIL   signature verification
--- crypto.2.expected   2013-11-24 04:17:13.0 +
+++ crypto.2.output 2013-11-24 04:17:13.0 +
@@ -11,7 +11,7 @@
 id: 2
 },
 {
-content-length: 315,
+content-length: 318,
 content-type: 
application/pgp-signature,
 id: 3
 }


According to dkg's nifty printmimestructure script, the result computed
by notmuch is actually correct. So it looks like different environments
result in different content-length headers. I'm not sure if this is a
bug that should be tracked down, or we should just sanitize the test
data more.

[1]: 
https://buildd.debian.org/status/fetch.php?pkg=notmucharch=kfreebsd-i386ver=0.17%7Erc1-1stamp=1385266740
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] Add NEWS entry for notmuch-compact

[David Bremner]

Ben Gamari bgamari.f...@gmail.com writes:

 Signed-off-by: Ben Gamari bgamari.f...@gmail.com
 ---
  NEWS | 8 
  1 file changed, 8 insertions(+)

pushed. 

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: notmuch sha1 implementation broken on (some) big-endian architectures

[Tomi Ollila]

On Sun, Nov 24 2013, David Bremner da...@tethera.net wrote:

 The following code, when linked with libnotmuch.a and libutil.a does a
 passable imitation of sha1sum on amd64 (and I guess also i386) but
 computes a different digest on powerpc and probably sparc and s390x.

 In the long run we should maybe outsource hash computations to
 e.g. librhash, but I'd like a simpler fix for 0.17, if possible

 P.S. I blame Austin for adding the missing-headers test which found
 this bug ;).

This is interesting problem, I would have guessed that this would
fails on LITTLE_ENDIAN machines easier, if ever...

... especially as there is line

lib/libsha1.c:52:#if (PLATFORM_BYTE_ORDER == IS_LITTLE_ENDIAN)

... but...

I could not find any (other) matches for PLATFORM_BYTE_ORDER nor
IS_LITTLE_ENDIAN in source code or in /usr/include/**/*.h or
in /usr/lib/gcc/**/*.h

I did some testing and it seems that 
#if (F == BBBAAARRR)
code
#endif

will have code in output file in case neither of the above are
defined... :/

So, this could work:

#if endian.h  // for BYTE_ORDER  LITTLE_ENDIAN

and then

#if (BYTE_ORDER == LITTLE_ENDIAN)
...

to replace lib/libsha1.c:53 (53 now after endian.h added)


Please test on BIG_ENDIAN machine...


In case this works, then we'd need to inform users that their
long/missing Message ID:s are now coded differently in their
databases...


Tomi
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] new: Detect dirent.d_type support at configure time

[Tomi Ollila]

On Sun, Nov 24 2013, Austin Clements amdra...@mit.edu wrote:

 Support for dirent.d_type is OS-specific.  Previously, we used
 _DIRENT_HAVE_D_TYPE to detect support for this, but this is apparently
 a glic-ism (FreeBSD, for example, supports d_type, but does not define
 this).  Since there's no cross-platform way to detect support for
 dirent.d_type, detect it using a test compile at configure time.
 ---

LGTM.

Tomi

  compat/have_d_type.c | 10 ++
  configure| 16 
  notmuch-new.c|  2 +-
  3 files changed, 27 insertions(+), 1 deletion(-)
  create mode 100644 compat/have_d_type.c

 diff --git a/compat/have_d_type.c b/compat/have_d_type.c
 new file mode 100644
 index 000..9ca6c6e
 --- /dev/null
 +++ b/compat/have_d_type.c
 @@ -0,0 +1,10 @@
 +#include dirent.h
 +
 +int main()
 +{
 +struct dirent ent;
 +
 +(void) ent.d_type;
 +
 +return 0;
 +}
 diff --git a/configure b/configure
 index 1a8e939..d2d193c 100755
 --- a/configure
 +++ b/configure
 @@ -557,6 +557,17 @@ else
  fi
  rm -f compat/have_timegm
  
 +printf Checking for dirent.d_type... 
 +if ${CC} -o compat/have_d_type $srcdir/compat/have_d_type.c  /dev/null 
 21
 +then
 +printf Yes.\n
 +have_d_type=1
 +else
 +printf No (will use stat instead).\n
 +have_d_type=0
 +fi
 +rm -f compat/have_d_type
 +
  printf Checking for standard version of getpwuid_r... 
  if ${CC} -o compat/check_getpwuid $srcdir/compat/check_getpwuid.c  
 /dev/null 21
  then
 @@ -745,6 +756,9 @@ HAVE_STRCASESTR = ${have_strcasestr}
  # build its own version)
  HAVE_STRSEP = ${have_strsep}
  
 +# Whether struct dirent has d_type (if not, then notmuch will use stat)
 +HAVE_D_TYPE = ${have_d_type}
 +
  # Whether the Xapian version in use supports compaction
  HAVE_XAPIAN_COMPACT = ${have_xapian_compact}
  
 @@ -805,6 +819,7 @@ CONFIGURE_CFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) 
 \$(GMIME_CFLAGS)  \\
  \$(VALGRIND_CFLAGS)   \\
  -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR) \\
  -DHAVE_STRSEP=\$(HAVE_STRSEP) \\
 +-DHAVE_D_TYPE=\$(HAVE_D_TYPE) \\
  -DSTD_GETPWUID=\$(STD_GETPWUID)   \\
  -DSTD_ASCTIME=\$(STD_ASCTIME) \\
  -DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT)
 @@ -813,6 +828,7 @@ CONFIGURE_CXXFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) 
 \$(GMIME_CFLAGS)\\
\$(VALGRIND_CFLAGS) \$(XAPIAN_CXXFLAGS) \\
-DHAVE_STRCASESTR=\$(HAVE_STRCASESTR)   \\
-DHAVE_STRSEP=\$(HAVE_STRSEP)   \\
 +  -DHAVE_D_TYPE=\$(HAVE_D_TYPE)   \\
-DSTD_GETPWUID=\$(STD_GETPWUID) \\
-DSTD_ASCTIME=\$(STD_ASCTIME)   \\
-DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT)
 diff --git a/notmuch-new.c b/notmuch-new.c
 index ba05cb4..423e188 100644
 --- a/notmuch-new.c
 +++ b/notmuch-new.c
 @@ -167,7 +167,7 @@ dirent_type (const char *path, const struct dirent *entry)
  char *abspath;
  int err, saved_errno;
  
 -#ifdef _DIRENT_HAVE_D_TYPE
 +#if HAVE_D_TYPE
  /* Mapping from d_type to stat mode_t.  We omit DT_LNK so that
   * we'll fall through to stat and get the real file type. */
  static const mode_t modes[] = {
 -- 
 1.8.4.rc3

 ___
 notmuch mailing list
 notmuch@notmuchmail.org
 http://notmuchmail.org/mailman/listinfo/notmuch
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 1/2] lib: fix byte order test in libsha1.c

[david]

From: David Bremner da...@tethera.net

Previously PLATFORM_BYTE_ORDER and IS_LITTLE_ENDIAN were not defined,
so the little endian code was always compiled in.

This will have the effect that the SHA1s on big endian architectures
will change (i.e. become actual sha1s). So someone re-indexing their
database could conceivable lose tags on messages without a message-id
header.
---
 lib/libsha1.c | 14 ++
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/lib/libsha1.c b/lib/libsha1.c
index 5d16f6a..794854b 100644
--- a/lib/libsha1.c
+++ b/lib/libsha1.c
@@ -49,11 +49,17 @@ extern C
 
 #define bswap_32(x) ((rotr32((x), 24)  0x00ff00ff) | (rotr32((x), 8)  
0xff00ff00))
 
-#if (PLATFORM_BYTE_ORDER == IS_LITTLE_ENDIAN)
-#define bsw_32(p,n) \
-{ int _i = (n); while(_i--) ((uint32_t*)p)[_i] = 
bswap_32(((uint32_t*)p)[_i]); }
+#ifdef __BYTE_ORDER__
+#  if (__BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__)
+#define bsw_32(p,n) \
+   { int _i = (n); while(_i--) ((uint32_t*)p)[_i] = 
bswap_32(((uint32_t*)p)[_i]); }
+#  elif (__BYTE_ORDER__ == __ORDER_BIG_ENDIAN__)
+#define bsw_32(p,n)
+#  else
+#error unknown byte order
+#  endif
 #else
-#define bsw_32(p,n)
+#error macro __BYTE_ORDER__ is not defined
 #endif
 
 #define SHA1_MASK   (SHA1_BLOCK_SIZE - 1)
-- 
1.8.4.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 2/2] NEWS: News for big endian sha1 bug fix.

[david]

From: David Bremner da...@tethera.net

We could give more details about how to migrate tags, but I'm not sure
that it's a practical problem, or just a theoretical one.
---
 NEWS | 14 +-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/NEWS b/NEWS
index 3383ecf..31c6284 100644
--- a/NEWS
+++ b/NEWS
@@ -1,6 +1,18 @@
-Notmuch 0.17~rc1 (2013-11-20)
+Notmuch 0.17~rc2 (2013-xx-yy)
 =
 
+Incompatible change in SHA1 computation
+---
+
+Previously on big endian architectures like sparc and powerpc the
+computation of SHA1 hashes was incorrect. This meant that messages
+with overlong or missing message-ids were given different computed
+message-ids than on more common little endian architectures like i386
+and amd64.  If you use notmuch on a big endian architecture, you are
+strongly advised to make a backup of your tags using `notmuch dump`
+before this upgrade. It should be possible to migrate the tags using a
+script.
+
 Command-Line Interface
 --
 
-- 
1.8.4.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 2/2] NEWS: News for big endian sha1 bug fix.

[Austin Clements]

Quoth da...@tethera.net on Nov 24 at  5:29 pm:
 From: David Bremner da...@tethera.net
 
 We could give more details about how to migrate tags, but I'm not sure
 that it's a practical problem, or just a theoretical one.
 ---
  NEWS | 14 +-
  1 file changed, 13 insertions(+), 1 deletion(-)
 
 diff --git a/NEWS b/NEWS
 index 3383ecf..31c6284 100644
 --- a/NEWS
 +++ b/NEWS
 @@ -1,6 +1,18 @@
 -Notmuch 0.17~rc1 (2013-11-20)
 +Notmuch 0.17~rc2 (2013-xx-yy)
  =
  
 +Incompatible change in SHA1 computation
 +---
 +
 +Previously on big endian architectures like sparc and powerpc the
 +computation of SHA1 hashes was incorrect. This meant that messages
 +with overlong or missing message-ids were given different computed
 +message-ids than on more common little endian architectures like i386
 +and amd64.  If you use notmuch on a big endian architecture, you are
 +strongly advised to make a backup of your tags using `notmuch dump`
 +before this upgrade. It should be possible to migrate the tags using a
 +script.
 +

Should this mention how to find such messages?  Something like

  notmuch dump |
  awk '/^notmuch-sha1-[0-9a-f]{40} / {system(notmuch search id: $1)}'
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch