[notmuch] [PATCH] Use libgcrypt for hashing.

[Carl Worth]

On Fri, 08 Jan 2010 15:43:52 -0500, micah anderson  wrote:
> Its good that this is not a burden to maintain for the notmuch project,
> even better that Mikhail, the libsha1 maintainer, is currently active in
> this project and has volunteered to maintain the in-tree copy. 
> 
> However, the problem that has been raised is about the code-maintenance
> burden that distributions face. In fact, this is not an unique problem
> to notmuch, if it was it wouldn't be such a big deal. The reality is
> that the more projects which cargo-cult around 'convenience copies' of
> code, the more of a burden is placed on the distributors.
> 
> In some ways, the notmuch project and the role of distributors are at
> cross-purposes on this issue, each side has an argument that makes sense
> From their individual perspectives.

Well, I think it's important for notmuch to ease the burden on the
distribution as well. That's just a matter of being a good citizen.

If notmuch were including code that existed as a library package in
Debian, say. Then that would definitely be problematic, and notmuch
should be fixed to link with the library.

We could get to that point if someone wanted to package libsha1, say.

> > What might make more sense is an option to compile against an existing
> > library (if present) but not to introduce an error in the build if the
> > library is not present, (in which case just build the builtin libsha1.c
> > code).
> 
> This makes the most sense, and resolves the issue in a way that both
> sides of the issue benefit!

I'd be glad to see a patch that does that.

-Carl
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 

[notmuch] [PATCH] Use libgcrypt for hashing.

[micah anderson]

On Fri, 27 Nov 2009 22:22:01 -0800, Carl Worth  wrote:
> On Fri, 27 Nov 2009 21:28:03 -0600, "Jeffrey C. Ollie"  
> wrote:
> > Instead of including a private implementation of the SHA1 hash, use
> > libgcrypt.  This means less code of our own to maintain and it will be
> > easier to switch to a different hash function like SHA256.
> 
> I don't believe we have a significant code-maintenance burden with
> libsha1.c. And as for different hash functions, the only use of sha-1 in
> notmuch is as a fallback in the case of a message not including a
> Message-ID header.
> 
> So I don't see it as important at all to try to remove this code.

Its good that this is not a burden to maintain for the notmuch project,
even better that Mikhail, the libsha1 maintainer, is currently active in
this project and has volunteered to maintain the in-tree copy. 

However, the problem that has been raised is about the code-maintenance
burden that distributions face. In fact, this is not an unique problem
to notmuch, if it was it wouldn't be such a big deal. The reality is
that the more projects which cargo-cult around 'convenience copies' of
code, the more of a burden is placed on the distributors.

In some ways, the notmuch project and the role of distributors are at
cross-purposes on this issue, each side has an argument that makes sense

[notmuch] [PATCH] Use libgcrypt for hashing.

[Mikhail Gusarov]

Twas brillig at 21:28:03 27.11.2009 UTC-06 when jeff at ocjtech.us did gyre and 
gimble:

 JCO> Instead of including a private implementation of the SHA1 hash

xserver went this road, and now it has
--with-sha1=libc|libmd|libgcrypt|libcrypto|libsha1|CommonCrypto in
configure.

 JCO> This means less code of our own to maintain and

As libsha1 maintainer I'm volunteering to maintain in-tree copy in
notmuch :)

-- 
  http://fossarchy.blogspot.com/
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 834 bytes
Desc: not available
URL: 

[notmuch] [PATCH] Use libgcrypt for hashing.

[Ingmar Vanhassel]

Excerpts from Mikhail Gusarov's message of Sat Nov 28 04:31:15 +0100 2009:
> 
> Twas brillig at 21:28:03 27.11.2009 UTC-06 when jeff at ocjtech.us did gyre 
> and
> gimble:
> 
>  JCO> Instead of including a private implementation of the SHA1 hash
> 
> xserver went this road, and now it has
> --with-sha1=libc|libmd|libgcrypt|libcrypto|libsha1|CommonCrypto in
> configure.

>From a distribution & security point of view I'd much rather be able to
choose one hashing library & use that as widely as possible, rather than
having every application ship its own copy.

>  JCO> This means less code of our own to maintain and
> 
> As libsha1 maintainer I'm volunteering to maintain in-tree copy in
> notmuch :)

Right, but on top of that, it would still be preferable to keep the
option for packagers to use a system library instead.
Most distributions have a rather strict policy to use system libraries
over internal copies.

-- 
Exherbo KDE, X.org maintainer

[notmuch] [PATCH] Use libgcrypt for hashing.

[Jeffrey Ollie]

On Sat, Nov 28, 2009 at 12:43 AM, Carl Worth  wrote:
>
> Have you actually *looked* at the implementation of libsha1.c that we
> have in notmuch? I can't say with 100% certainty that it's free of any
> buffer overruns, but I can see that it's not doing any memory allocation
> nor network communication. So there are entire classes of security
> problems, (such as have afflicted libraries in your examples), that just
> aren't present here.

I've looked at the code, if only briefly.  But you're wrong that the
code doesn't do any "network communication" - we feed libsha1 hostile
data every time we take the hash of a message.

> And as for security compromises due to a bug in the cryptographic nature
> of this function---well, notmuch isn't even *using* SHA-1 for any secure
> purpose.

>From a distributor's point of view, it doesn't matter what you use the
code for, it only matters that it has the bug and someone has to spend
the time to track down all of the copies of the code and replace the
code with a fixed version.  If the code is confined to one shared
library it's trivial to update the shared library, if the code has
been copied to N packages, it's at least N times the work to verify
that all of those packages get updated.

> The actual functionality that we need here is *so* small that I am
> unwilling to introduce a required dependency on any library as large as
> libcrypt. I mean, look at the actual sizes we're talking about

If libgcrypt were some obscure library that wasn't already packaged up
by your favorite distribution or took up hundreds of megabytes of RAM
and/or disk you might have a point.  But the fact is that it *doesn't
matter* how big libgcrypt is because we essentially get it for free -
I'd bet that libgcrypt is already installed on most people's systems.
As a test I tried to remove libgcrypt from my laptop - if I actually
had gone though with it I would have crippled my system because things
like the X server and the package manager depend (albeit indirectly)
on it.

> Now, if somebody wanted to maintain libsha1 inside a distribution like
> Debian, say, then I'd be happy to link against that version rather than
> a locally compiled version. And like I said earlier, if people would
> rather link against a large cyptographic library for this one tiny
> function, then we could arrange that too, but I don't think that
> justifies dropping this code from notmuch and introducing a hard
> dependency.

Given Debian's reputation for packaging the kitchen sink I'm surprised
that it doesn't already.  Fedora tends not to package libraries unless
there is an application that is going to make use of it...

-- 
Jeff Ollie

[notmuch] [PATCH] Use libgcrypt for hashing.

[Jeffrey Ollie]

On Fri, Nov 27, 2009 at 9:59 PM, Ingmar Vanhassel  wrote:
>
> Most distributions have a rather strict policy to use system libraries
> over internal copies.

Fedora:

https://fedoraproject.org/wiki/Packaging:No_Bundled_Libraries

Debian:

http://www.debian.org/doc/debian-policy/ch-source.html#s-embeddedfiles

If there are other distributions out there that have similar policies
I'd be interested in learning about them.

-- 
Jeff Ollie

[notmuch] [PATCH] Use libgcrypt for hashing.

[Jeffrey Ollie]

On Fri, Nov 27, 2009 at 9:31 PM, Mikhail Gusarov
 wrote:
> Twas brillig at 21:28:03 27.11.2009 UTC-06 when jeff at ocjtech.us did gyre 
> and gimble:
>
> ?JCO> Instead of including a private implementation of the SHA1 hash
>
> xserver went this road, and now it has
> --with-sha1=libc|libmd|libgcrypt|libcrypto|libsha1|CommonCrypto in
> configure.

I doubt that we'll get to that level of insanity.

> ?JCO> This means less code of our own to maintain and
>
> As libsha1 maintainer I'm volunteering to maintain in-tree copy in
> notmuch :)

That's great that you're willing to take on the task, but as I do a
lot of work for Fedora I tend to think about these things differently.
 It's not about a project here or there making private copies of some
code, it's about tracking down *all* of the projects that have private
copies of the code when something goes wrong, especially when there
are security implications.

The most famous example is Zlib.  Based upon what Google turned up I
see that you've been around long enough that you should have heard of
the problems the computer world had when security flaws turned up in
Zlib.  There's even a project[1] that developed methods for
identifying vulnerable code in binaries the problem was so pervasive.

A more recent example is a cross-site ajax request vulnerability in
the Prototype JavaScript framework[2].  A lot of projects copied that
code into their repositories as well, and now someone has to track all
of those down and get them fixed as well.

When projects copy code into their repositories like this, it makes
things a lot more difficult for someone else down the road.  Both
Fedora[3] and Debian[4] (and Ubuntu by extension), while not expressly
forbidding the practice, *strongly* recommend against it.  I'd really
recommend reading the Fedora page that explains the policy as it has a
great summation of the problem.

[1] http://www.enyo.de/fw/security/zlib-fingerprint/
[2] http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-7220
[3] https://fedoraproject.org/wiki/Packaging:No_Bundled_Libraries
[4] http://www.debian.org/doc/debian-policy/ch-source.html#s-embeddedfiles

-- 
Jeff Ollie

[notmuch] [PATCH] Use libgcrypt for hashing.

[Carl Worth]

On Sat, 28 Nov 2009 09:31:15 +0600, Mikhail Gusarov  wrote:
>  JCO> This means less code of our own to maintain and
> 
> As libsha1 maintainer I'm volunteering to maintain in-tree copy in
> notmuch :)

And Mikhail,

I wanted to thank you publicly for your maintenance work of libsha1. It
was a pleasant surprise to see you on the notmuch list and contributing
patches after I had independently found and incorporated libsha1
previously.

-Carl

-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 

[notmuch] [PATCH] Use libgcrypt for hashing.

[Carl Worth]

On Fri, 27 Nov 2009 21:28:03 -0600, "Jeffrey C. Ollie"  
wrote:
> Instead of including a private implementation of the SHA1 hash, use
> libgcrypt.  This means less code of our own to maintain and it will be
> easier to switch to a different hash function like SHA256.

I don't believe we have a significant code-maintenance burden with
libsha1.c. And as for different hash functions, the only use of sha-1 in
notmuch is as a fallback in the case of a message not including a
Message-ID header.

So I don't see it as important at all to try to remove this code.

> libgcrypt was chosen because it has a fairly simple API, it's well
> tested (it's used in gnutls and gnupg2), and it's licensed under the
> LGPL.

What might make more sense is an option to compile against an existing
library (if present) but not to introduce an error in the build if the
library is not present, (in which case just build the builtin libsha1.c
code).

But if that wouldn't solve the problem you were trying to solve, (to
actually remove libsha1.c), then maybe we don't need to do anything for
now?

-Carl
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 

[notmuch] [PATCH] Use libgcrypt for hashing.

[Alexander Botero-Lowry]

On Fri, 27 Nov 2009 23:43:34 -0600, Jeffrey Ollie  wrote:
> On Fri, Nov 27, 2009 at 9:59 PM, Ingmar Vanhassel  
> wrote:
> >
> > Most distributions have a rather strict policy to use system libraries
> > over internal copies.
> 
> Fedora [...], Debian [...]
> 
> If there are other distributions out there that have similar policies
> I'd be interested in learning about them.
> 
FreeBSD ports are expected to not use bundled libraries as well.

Alex

[notmuch] [PATCH] Use libgcrypt for hashing.

[Jeffrey C. Ollie]

Instead of including a private implementation of the SHA1 hash, use
libgcrypt.  This means less code of our own to maintain and it will be
easier to switch to a different hash function like SHA256.

libgcrypt was chosen because it has a fairly simple API, it's well
tested (it's used in gnutls and gnupg2), and it's licensed under the
LGPL.

Signed-off-by: Jeffrey C. Ollie 
---
 Makefile   |6 +-
 configure  |   15 +++-
 lib/Makefile.local |1 -
 lib/libsha1.c  |  242 
 lib/libsha1.h  |   67 --
 lib/sha1.c |   44 +-
 6 files changed, 42 insertions(+), 333 deletions(-)
 delete mode 100644 lib/libsha1.c
 delete mode 100644 lib/libsha1.h

diff --git a/Makefile b/Makefile
index 2cd1b1b..a5ff3e7 100644
--- a/Makefile
+++ b/Makefile
@@ -10,7 +10,8 @@ gzip = gzip

 # Additional flags that we will append to whatever the user set.
 # These aren't intended for the user to manipulate.
-extra_cflags := $(shell pkg-config --cflags glib-2.0 gmime-2.4 talloc)
+extra_cflags := $(shell pkg-config --cflags glib-2.0 gmime-2.4 talloc) \
+   $(shell libgcrypt-config --cflags)
 extra_cxxflags := $(shell xapian-config --cxxflags)

 emacs_lispdir := $(shell pkg-config emacs --variable sitepkglispdir)
@@ -28,7 +29,8 @@ override CXXFLAGS += $(WARN_FLAGS) $(extra_cflags) 
$(extra_cxxflags)

 override LDFLAGS += \
$(shell pkg-config --libs glib-2.0 gmime-2.4 talloc) \
-   $(shell xapian-config --libs)
+   $(shell xapian-config --libs) \
+   $(shell libgcrypt-config --libs)

 # Include our local Makefile.local first so that its first target is default
 include Makefile.local
diff --git a/configure b/configure
index b4770ec..22f8066 100755
--- a/configure
+++ b/configure
@@ -61,6 +61,15 @@ else
 have_valgrind=
 fi

+if libgcrypt-config --version > /dev/null 2>&1; then
+echo "Checking for libgcrypt development files... Yes."
+have_libgcrypt=1
+else
+echo "Checking for libgcrypt development files... No."
+have_libgcrypt=0
+errors=$((errors + 1))
+fi
+
 if [ $errors -gt 0 ]; then
 cat > (32 - n)))
-#define rotr32(x,n)   (((x) >> n) | ((x) << (32 - n)))
-
-#define bswap_32(x) ((rotr32((x), 24) & 0x00ff00ff) | (rotr32((x), 8) & 
0xff00ff00))
-
-#if (PLATFORM_BYTE_ORDER == IS_LITTLE_ENDIAN)
-#define bsw_32(p,n) \
-{ int _i = (n); while(_i--) ((uint32_t*)p)[_i] = 
bswap_32(((uint32_t*)p)[_i]); }
-#else
-#define bsw_32(p,n)
-#endif
-
-#define SHA1_MASK   (SHA1_BLOCK_SIZE - 1)
-
-#if 0
-
-#define ch(x,y,z)   (((x) & (y)) ^ (~(x) & (z)))
-#define parity(x,y,z)   ((x) ^ (y) ^ (z))
-#define maj(x,y,z)  (((x) & (y)) ^ ((x) & (z)) ^ ((y) & (z)))
-
-#else   /* Discovered by Rich Schroeppel and Colin Plumb   */
-
-#define ch(x,y,z)   ((z) ^ ((x) & ((y) ^ (z
-#define parity(x,y,z)   ((x) ^ (y) ^ (z))
-#define maj(x,y,z)  (((x) & (y)) | ((z) & ((x) ^ (y
-
-#endif
-
-/* Compile 64 bytes of hash data into SHA1 context. Note*/
-/* that this routine assumes that the byte order in the */
-/* ctx->wbuf[] at this point is in such an order that low   */
-/* address bytes in the ORIGINAL byte stream will go in */
-/* this buffer to the high end of 32-bit words on BOTH big  */
-/* and little endian systems*/
-
-#ifdef ARRAY
-#define q(v,n)  v[n]
-#else
-#define q(v,n)  v##n
-#endif
-
-#define one_cycle(v,a,b,c,d,e,f,k,h)\
-q(v,e) += rotr32(q(v,a),27) +   \
-  f(q(v,b),q(v,c),q(v,d)) + k + h;  \
-q(v,b)  = rotr32(q(v,b), 2)
-
-#define five_cycle(v,f,k,i) \
-one_cycle(v, 0,1,2,3,4, f,k,hf(i  ));   \
-one_cycle(v, 4,0,1,2,3, f,k,hf(i+1));   \
-one_cycle(v, 3,4,0,1,2, f,k,hf(i+2));   \
-one_cycle(v, 2,3,4,0,1, f,k,hf(i+3));   \
-one_cycle(v, 1,2,3,4,0, f,k,hf(i+4))
-
-static void sha1_compile(sha1_ctx ctx[1])
-{   uint32_t*w = ctx->wbuf;
-
-#ifdef ARRAY
-uint32_tv[5];
-memcpy(v, ctx->hash, 5 * sizeof(uint32_t));
-#else
-uint32_tv0, v1, v2, v3, v4;
-v0 = ctx->hash[0]; v1 = ctx->hash[1];
-v2 = ctx->hash[2]; v3 = ctx->hash[3];
-v4 = ctx->hash[4];
-#endif
-
-#define hf(i)   w[i]
-
-five_cycle(v, ch, 0x5a827999,  0);
-five_cycle(v, ch, 0x5a827999,  5);
-five_cycle(v, ch, 0x5a827999, 10);
-one_cycle(v,0,1,2,3,4, ch, 0x5a827999, hf(15)); \
-
-#undef  hf
-#define hf(i) (w[(i) & 15] = rotl32(\
- w[((i) + 13) & 15] ^ w[((i) + 8) & 15] \
-   ^ w[((i) +  

Re: [notmuch] [PATCH] Use libgcrypt for hashing.

[Ingmar Vanhassel]

Excerpts from Mikhail Gusarov's message of Sat Nov 28 04:31:15 +0100 2009:
 
 Twas brillig at 21:28:03 27.11.2009 UTC-06 when j...@ocjtech.us did gyre and
 gimble:
 
  JCO Instead of including a private implementation of the SHA1 hash
 
 xserver went this road, and now it has
 --with-sha1=libc|libmd|libgcrypt|libcrypto|libsha1|CommonCrypto in
 configure.

From a distribution  security point of view I'd much rather be able to
choose one hashing library  use that as widely as possible, rather than
having every application ship its own copy.

  JCO This means less code of our own to maintain and
 
 As libsha1 maintainer I'm volunteering to maintain in-tree copy in
 notmuch :)

Right, but on top of that, it would still be preferable to keep the
option for packagers to use a system library instead.
Most distributions have a rather strict policy to use system libraries
over internal copies.

-- 
Exherbo KDE, X.org maintainer
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch