Re: [PATCH] build: sign tarball instead of sha256sum
On 2019-03-14 11:51 p.m., Daniel Kahn Gillmor wrote: > Sorry to only be getting to this now. I think the original mechanism, > despite being non-standard, is actually a more robust approach, so i > recommend reverting this change. > > A detached signature on object X does *not* cover the name of object X. > > So for some existing version Y of notmuch, if an attacker takes > notmuch-Y.tgz and notmuch-Y.tgz.asc and renames them both to > notmuch-Z.tgz and notmuch-Z.tgz.asc, they can make it look like a new > version (version Z) of notmuch is available! All other software I've encountered, the tarball is signed. This semantic allows for automated checks by our Open Build Server instance to make sure there is no file system corruption or other modification. If something else is signed, like the sha256, then only that small file is checked. In this case, adding explicit checks would add an extra BuildRequires in the build process to pull in gpg, which is excessive. Instead of reverting, how about distributing the .asc file and an inline signed checksum file? Then you have both signed. Just sign the .sha256 inline and sha256 will verify and at least it will not look like a detached signature. And `sha256 -c` on signed file directly also will work. sha256sum notmuch-0.28.3.tar.gz | gpg --clearsign -a - > notmuch-0.28.3.tar.gz.sha256 gpg -b -a notmuch-0.28.3.tar.gz And then you have signed sha256 for people that want to check that. And you have a detached signature, for people that want to use that. And need to have funny looking unsigned intermediaries and detached looking signatures that really aren't. - Adam ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] build: sign tarball instead of sha256sum
On 3/15/19 2:37 PM, Daniel Kahn Gillmor wrote: On Fri 2019-03-15 12:35:55 +0100, Adam Majer wrote: # osc chroot running: sudo chroot /var/tmp/build-root/openSUSE_Tumbleweed-x86_64 su - abuild # gpgv -bash: gpgv: command not found That's surprising to me, but i'm ignorant about SUSE so you shouldn't be surprised at my surprise :P How does this system cryptographically verify its software updates? or is it never updated? or updated "from the outside" or something? There is a different service that checks for signatures and keyring files that come with a package. This happens at checkin phase or at some review phase (some automated review bot would then verify signature too before allowing to accept it into more important project). Of course, one could just not have any signature then it would just be skipped. The builds don't check this as once checked in, integrity is handled by OBS and most packages are not signed :( But when you checkout a package, you can at least verify things. OBS has backend called `signer`[2] that is responsible for signing RPMs and repository files (used by zypper, which is like apt) with a project specific key (you can configure your own key per project). The nice thing about OBS is that anyone can fork any project and add/update a package, make an image, and use that. Or pick software from various projects and OBS will rebuild things if build dependencies change. It builds Debian packages too [1], Fedora, whatever, although mostly it's used for SUSE/openSUSE projects. This is actually how SUSE makes products based on other products and things remain consistent. The weakest points of all these verifications are the upstreams. Many have no signatures at all. Clearly, notmuch is not the example here :D - Adam [1] https://build.opensuse.org/package/show/home:adamm/Nudoku [2] https://build.opensuse.org/monitor ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] build: sign tarball instead of sha256sum
On 3/15/19 2:47 PM, Daniel Kahn Gillmor wrote:
Do you know of any code that actually makes use of that defense? That
is, any code that says "fetch version X of package foo and its
cryptographic signatures; verify the signature over the tarball, and
also verify that it unpacks to a directory named foo-X/ before returning
success" ? That would be great if it's out there and i'm unaware of it.
We do. For example, nodejs10,
https://build.opensuse.org/package/show/devel:languages:nodejs/nodejs10
The .spec file has (I added some comments here)
Name: nodejs10
Version:10.15.3
Source: https://nodejs.org/dist/v%{version}/node-v%{version}.tar.xz
Source1:https://nodejs.org/dist/v%{version}/SHASUMS256.txt
Source2:https://nodejs.org/dist/v%{version}/SHASUMS256.txt.sig
Source3:nodejs.keyring
The .sig is verified vs. nodejs.keyring on checkin. And for build, the
%prep phase in start of the build has,
# this checks the checksum
echo "`grep node-v%{version}.tar.xz %{S:1} | head -n1 | cut -c1-64`
%{S:0}" | sha256sum -c
# this unpacks the tarball Source0 and changes to directory
# node-v%{version}
%setup -q -n node-v%{version}
The build would break if directory is different name. The build would
break if checksum is wrong, but that has explicit verification. And
build would not even be attempted if *.sig wasn't signed by a key in the
*.keyring file.
Not only that, because the SourceX is a URL, distributions like
Tumbleweed that accept lots of submissions, have automated bot that will
download these files and compare them to what was submitted. If these
differ, it will reject. This actually caught NodeJS project adding ARM
binaries to their release after release and re-issuing the checksums.
Not malicious, just annoying (for me :)
- Adam
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] build: sign tarball instead of sha256sum
On 3/15/19 9:58 AM, Daniel Kahn Gillmor wrote: On Fri 2019-03-15 02:53:28 +0100, Adam Majer wrote: adding explicit checks would add an extra BuildRequires in the build process to pull in gpg, which is excessive. It shouldn't require gpg; it should only pull in gpgv, which is already on the base system, no? And once the "small file" is checked, it would then require sha256sum (or the equivalent) to verify the tarball itself; on any modern system, that's likely to be available anyway (e.g. coreutils' sha256sum or "openssl dgst" or whatever). # osc chroot running: sudo chroot /var/tmp/build-root/openSUSE_Tumbleweed-x86_64 su - abuild # gpgv -bash: gpgv: command not found With openSUSE, the closest thing to a base system for building would be in this log, https://build.opensuse.org/build/home:adamm:boost_test/openSUSE_Tumbleweed/x86_64/boost-defaults/_log Since this is just a dependency package, it has no BuildRequires. The base system is just what is needed to run rpm, rpmlint, etc. so 122 packages. No gpgv or gpg or python or ruby. Only gcc, perl, rpm. Instead of reverting, how about distributing the .asc file and an inline signed checksum file? The checksum file (*.sha256.asc) that is distributed by notmuch is already inline-signed (please read my proposed verification step upthread), so that part's done. (notmuch does *also* ship an unsigned *.sha256 file, which i agree doesn't serve much purpose and could be dropped) Sorry, I meant clear signed and inline. The checksum file could just be *.sha256 and be itself clear signed. Then people see as a checksum file and when they look inside, they see it as signed. There is no reason to have the checksum file encoded. The (my?) expectation is that a *.asc file is a detached signature. That's why GPG is warning when it is not a detached signature. But I can live with .sha256.asc if there is no .sha256 ;) - Adam ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] build: sign tarball instead of sha256sum
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512
On Fri 2019-03-15 15:30:56 +0100, Adam Majer wrote:
> The .spec file has (I added some comments here)
>
> Name: nodejs10
> Version:10.15.3
> Source: https://nodejs.org/dist/v%{version}/node-v%{version}.tar.xz
> Source1:https://nodejs.org/dist/v%{version}/SHASUMS256.txt
> Source2:https://nodejs.org/dist/v%{version}/SHASUMS256.txt.sig
> Source3:nodejs.keyring
interesting -- i note that the thing signed here is actually a checksum
file, and not the tarball itself. Please use this technique for notmuch
too! does this mean that you don't need the detached tarball signature?
:)
> The .sig is verified vs. nodejs.keyring on checkin. And for build, the
> %prep phase in start of the build has,
>
> # this checks the checksum
> echo "`grep node-v%{version}.tar.xz %{S:1} | head -n1 | cut -c1-64` %{S:0}"
> | sha256sum -c
(i think this was line-wrapped; i unwrapped it and i hope i got the
whitespace right)
this is interesting! I'd want to tighten up the grep a bit, and drop
the head -n1 (seems like it means "cross your fingers and hope we got
the right one", which is not great for signature verification), and it's
not clear that you need to rebuild the line itself.
If you've already verified SHASUMS256.txt.sig, Why not just:
grep -E '^[0-9a-f]{64} [ *]node-v%{version}\.tar\.xz$' %{S:1} | sha256sum -c
(this still doesn't properly escape regex metacharacters like . in
%{version}, but it's a much tighter match than the earlier line.
> # this unpacks the tarball Source0 and changes to directory
> # node-v%{version}
> %setup -q -n node-v%{version}
>
> The build would break if directory is different name.
i'm assuming that this is done starting from an empty directory by
default -- otherwise it could misbehave if there was cruft leftover from
some other unpacking.
> The build would break if checksum is wrong, but that has explicit
> verification. And build would not even be attempted if *.sig wasn't
> signed by a key in the *.keyring file.
nice, this is very good. it looks like OBS is checking:
* cryptographic signature over the tarball name (which includes package
and version)
* that the tarball contains a directory named with the package and version
that's a robust check, and it looks good.
> Not only that, because the SourceX is a URL, distributions like
> Tumbleweed that accept lots of submissions, have automated bot that will
> download these files and compare them to what was submitted. If these
> differ, it will reject. This actually caught NodeJS project adding ARM
> binaries to their release after release and re-issuing the checksums.
> Not malicious, just annoying (for me :)
It's great that you caught this! Does tumbleweed have a "wall of shame"
for "re-released" software? I'd love to know the history of what it has
caught. I'm sure most are "legitimate" mistakes. But hiding in the
noise could be some real malfeasance (which is why it'd be good to get
rid of the noise).
At this point, we're pretty far afield from notmuch -- if others are
annoyed, i'm happy to take the discussion to private mail or some other
mailing list if folks would prefer it.
--dkg
-BEGIN PGP SIGNATURE-
iHUEARYKAB0WIQTJDm02IAobkioVCed2GBllKa5f+AUCXIvXVQAKCRB2GBllKa5f
+IezAP9a6K76MbP8UQBxeLk7UTo/OVV6GZyNCa5nSL7Lm88ziQEAhr6wydCe9LBJ
wrBYRNbBLqato1XRfjBb+m8Uzz7vtwY=
=eBIA
-END PGP SIGNATURE-
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] build: sign tarball instead of sha256sum
On Fri 2019-03-15 10:50:34 -0300, David Bremner wrote: > Adam Majer writes: > >> The (my?) expectation is that a *.asc file is a detached signature. >> That's why GPG is warning when it is not a detached signature. But I can >> live with .sha256.asc if there is no .sha256 ;) > > Right, aren't detached signatures preferred in general? Or am I > misremembering some gpg folklore? Detached signatures are recommended for several reasons: * They can handle non-textual data * They are not confused by varying character sets/encodings of textual data * They clearly demarcate what is being signed (text documents with inline signatures can include arbitrary cruft before the signed text, or after the signature) * gpg (and gpgv)'s API has traditionally been ambiguous about what it means to say "gpg --verify foo.asc" depending on whether foo (without the ".asc") exists. This API "shortcut" has a nasty failure mode, because an attacker can ship you an arbitrary "foo" and an inline-signed document as "foo.asc". in that case, you might be surprised to find that while the signature verifies, it has nothing to do with "foo". I don't think these situations are relevant for the case of a signed sha256sum file verified in a sensible way: * we're dealing with textual data, guaranteed to be US-ASCII. Verifiers can explicitly make that assumption. * Modern versions of gpg (and gpgv) have an --output argument when validating a signature, making it much easier to see what specifically was signed, and avoiding any ambiguity about weird "foo.asc + unrelated foo" use cases (see my use of --output in the pipeline example upthread) Detached signatures have the additional disadvantage of having to juggle/coordinate more files when publishing or retrieving a new release. For the sake of shipping fewer files with each release, i think it makes sense to use a clearsigned *.sha256.asc. --dkg signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] build: sign tarball instead of sha256sum
On Fri 2019-03-15 10:56:58 -0300, David Bremner wrote: > Daniel Kahn Gillmor writes: > >> sure, though i'd change the .sha256.asc to be a clearsigned file instead >> of the current ASCII-armored OpenPGP message that it currently is (as >> Adam suggested elsewhere in this thread). And we can ditch the .sha256 >> itself, which doesn't seem to be doing any useful work. > > Err, wouldn't we be relying on the .sha256 file to be byte reproducible in > perpetuity then? That seems to tie us to coreutils and reduce the > options of users for verification, no? i'm not sure i understand the question. the .sha256 file is literally the same output emitted to stdout by "gpgv --output - *.sha256.asc" currently, right? we certainly expect the sha256 digest of the tarball itself to be reproducible in perpetuity. So i think you're asking about the format of the sha256sum listing… AIUI, the output/input format of sha256sum is well-known, documented [0], and stable. The only weirdness is how it handles filenames with newlines in them [1], but neither we nor the verifiers have to worry about that for the types of files we're actually signing and verifying. I'm sure several people on this list could cobble together a few lines of "openssl dgst" and awk to perform the same effect as sha256sum in the verification pipeline i sketched upthread. --dkg [0] https://www.gnu.org/software/coreutils/manual/html_node/md5sum-invocation.html#md5sum-invocation [1] https://github.com/openssl/openssl/issues/8493 signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] build: sign tarball instead of sha256sum
Daniel Kahn Gillmor writes: > > sure, though i'd change the .sha256.asc to be a clearsigned file instead > of the current ASCII-armored OpenPGP message that it currently is (as > Adam suggested elsewhere in this thread). And we can ditch the .sha256 > itself, which doesn't seem to be doing any useful work. > > --dkg Err, wouldn't we be relying on the .sha256 file to be byte reproducible in perpetuity then? That seems to tie us to coreutils and reduce the options of users for verification, no? d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] build: sign tarball instead of sha256sum
Adam Majer writes: > The (my?) expectation is that a *.asc file is a detached signature. > That's why GPG is warning when it is not a detached signature. But I can > live with .sha256.asc if there is no .sha256 ;) Right, aren't detached signatures preferred in general? Or am I misremembering some gpg folklore? d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] build: sign tarball instead of sha256sum
On Fri 2019-03-15 07:49:16 -0300, David Bremner wrote: > BTW2: In a sense everyone has other defences since the tar ball contains a > file "version" with the version in it. Right, if there was a standard/conventional way to indicate the package name and version information *within* any source tarball, that would be a great simple defense. The closest we could come to such a convention might actually be the name of the top-level directory within the tarball itself: notmuch's tarball unpacks to notmuch-0.28.3/* regardless of what the tarball's name is. But while that practice is fairly widespread in many other projects, i don't think it's nearly as universal a convention as stuffing the package name and version in the tarball name itself. (note that uscan in particular extracts the version from the tarball name, not its contents) Do you know of any code that actually makes use of that defense? That is, any code that says "fetch version X of package foo and its cryptographic signatures; verify the signature over the tarball, and also verify that it unpacks to a directory named foo-X/ before returning success" ? That would be great if it's out there and i'm unaware of it. I suppose i could test such an attack on uscan itself, but i haven't gotten the bandwidth to really trying it out. If someone wants to try it, i'd be happy to read a report! >> David, how would you feel about generating two forms of cryptographic >> signature per-tarball as an interim process? > > Yeah, that sounds fine. IIUC, the old .sha256.asc and the "new" > .tar.gz.asc? sure, though i'd change the .sha256.asc to be a clearsigned file instead of the current ASCII-armored OpenPGP message that it currently is (as Adam suggested elsewhere in this thread). And we can ditch the .sha256 itself, which doesn't seem to be doing any useful work. --dkg signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] build: sign tarball instead of sha256sum
On Fri 2019-03-15 12:35:55 +0100, Adam Majer wrote: > # osc chroot > running: sudo chroot /var/tmp/build-root/openSUSE_Tumbleweed-x86_64 su - > abuild > # gpgv > -bash: gpgv: command not found That's surprising to me, but i'm ignorant about SUSE so you shouldn't be surprised at my surprise :P How does this system cryptographically verify its software updates? or is it never updated? or updated "from the outside" or something? > Sorry, I meant clear signed and inline. The checksum file could just be > *.sha256 and be itself clear signed. Then people see as a checksum file > and when they look inside, they see it as signed. There is no reason to > have the checksum file encoded. Ah, good call. I agree that *.sha256.asc should be a clearsigned text file instead of an ASCII-armored PGP message. Thanks for catching that! --dkg signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] build: sign tarball instead of sha256sum
Daniel Kahn Gillmor writes: > On Fri 2019-03-15 02:53:28 +0100, Adam Majer wrote: >> adding explicit checks would add an extra BuildRequires in the build >> process to pull in gpg, which is excessive. > > It shouldn't require gpg; it should only pull in gpgv, which is already > on the base system, no? And once the "small file" is checked, it would > then require sha256sum (or the equivalent) to verify the tarball itself; > on any modern system, that's likely to be available anyway > (e.g. coreutils' sha256sum or "openssl dgst" or whatever). BTW gpg is needed to run the full test suite. > But you're right that we could distribute a detached signature over the > tarball in addition to the stronger mechanism. that way people who have > other defenses against rollback or version fixation attacks (or who > are willing to take the risk) can check the simpler, weaker mechanism. BTW2: In a sense everyone has other defences since the tar ball contains a file "version" with the version in it. > David, how would you feel about generating two forms of cryptographic > signature per-tarball as an interim process? Yeah, that sounds fine. IIUC, the old .sha256.asc and the "new" .tar.gz.asc? ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] build: sign tarball instead of sha256sum
On Fri 2019-03-15 02:53:28 +0100, Adam Majer wrote: > adding explicit checks would add an extra BuildRequires in the build > process to pull in gpg, which is excessive. It shouldn't require gpg; it should only pull in gpgv, which is already on the base system, no? And once the "small file" is checked, it would then require sha256sum (or the equivalent) to verify the tarball itself; on any modern system, that's likely to be available anyway (e.g. coreutils' sha256sum or "openssl dgst" or whatever). > Instead of reverting, how about distributing the .asc file and an > inline signed checksum file? The checksum file (*.sha256.asc) that is distributed by notmuch is already inline-signed (please read my proposed verification step upthread), so that part's done. (notmuch does *also* ship an unsigned *.sha256 file, which i agree doesn't serve much purpose and could be dropped) But you're right that we could distribute a detached signature over the tarball in addition to the stronger mechanism. that way people who have other defenses against rollback or version fixation attacks (or who are willing to take the risk) can check the simpler, weaker mechanism. David, how would you feel about generating two forms of cryptographic signature per-tarball as an interim process? --dkg signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] build: sign tarball instead of sha256sum
On Thu 2019-03-14 22:49:44 -0300, David Bremner wrote: > OK, so apparently this is a problem for almost every project, including > GnuPG? That's mildly terrifying... sigh, i know :( > I don't mind either way, but it does seem like there is a tradeoff, > since with the previous version I suspect many people are just not > verifying the signature (e.g. can uscan in debian handle the sha256sum > scheme?). i thought about that on my bike ride home. the right answer is "uscan needs to be able to check signatures of this form, and Someone™ should probably file a report in the BTS". So I looked in the BTS, and noticed that it's actually already filed (https://bugs.debian.org/874029) and it's not just notmuch that has something comparable. I've tagged that bug as Affects: src:notmuch, i hope that's ok. But of course the workaround for the meantime until that bug is resolved is "the debian releases are typically made by the same human who generates the signed tarballs so him checking his own signature doesn't provide much in the way of additional security" :P But I want to reduce the notmuch bus factor too, so hopefully we can get uscan improved. --dkg signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] build: sign tarball instead of sha256sum
Daniel Kahn Gillmor writes: > So for some existing version Y of notmuch, if an attacker takes > notmuch-Y.tgz and notmuch-Y.tgz.asc and renames them both to > notmuch-Z.tgz and notmuch-Z.tgz.asc, they can make it look like a new > version (version Z) of notmuch is available! The only way to detect the > attack is to store a log of timestamps of previous releases, and try to > compare timestamps (though this itself can be confusing and wrong if we > were to maintain multiple branches concurrently). This permits a > "rollback" or "version freeze" attack, which we probably don't want to > encourage. OK, so apparently this is a problem for almost every project, including GnuPG? That's mildly terrifying... I don't mind either way, but it does seem like there is a tradeoff, since with the previous version I suspect many people are just not verifying the signature (e.g. can uscan in debian handle the sha256sum scheme?). d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] build: sign tarball instead of sha256sum
On Tue 2019-03-12 07:55:19 -0300, David Bremner wrote:
> David Bremner writes:
>
>> Adam Majer pointed out in [1] the way were signing releases was
>> unusual. Neither Carl nor I could think of a good reason for
>> explicitely signing the checksum (internally of course that's what GPG
>> is going anyway).
>
> It seemed unlikely that there would be much testing for this (but feel
> free!), so I have pushed it. I can fix any glitches during the next
> release.
Sorry to only be getting to this now. I think the original mechanism,
despite being non-standard, is actually a more robust approach, so i
recommend reverting this change.
A detached signature on object X does *not* cover the name of object X.
So for some existing version Y of notmuch, if an attacker takes
notmuch-Y.tgz and notmuch-Y.tgz.asc and renames them both to
notmuch-Z.tgz and notmuch-Z.tgz.asc, they can make it look like a new
version (version Z) of notmuch is available! The only way to detect the
attack is to store a log of timestamps of previous releases, and try to
compare timestamps (though this itself can be confusing and wrong if we
were to maintain multiple branches concurrently). This permits a
"rollback" or "version freeze" attack, which we probably don't want to
encourage.
However, if the thing verified is the output of sha256sum, then the
*filename* of the tarball itself is included, then the standard
verification step will is sufficient to ensure that you've got the right
version in the filename.
In bash, verifying version 0.28.3 would look something like:
set -o pipefail
set -x
wget https://notmuchmail.org/releases/notmuch-0.28.3.tar.gz{,.sha256.asc}
gpgv --keyring ./notmuch-signers.pgp --output -
notmuch-0.28.3.tar.gz.sha256.asc | sha256sum -c -
This is resistant to any attacker who manages to place or rename files
on https://notmuchmail.org/releases/.
That's why I support reversion here. Apologies for the delay in
encountering this thread.
Regards,
--dkg
signature.asc
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] build: sign tarball instead of sha256sum
David Bremner writes: > Adam Majer pointed out in [1] the way were signing releases was > unusual. Neither Carl nor I could think of a good reason for > explicitely signing the checksum (internally of course that's what GPG > is going anyway). It seemed unlikely that there would be much testing for this (but feel free!), so I have pushed it. I can fix any glitches during the next release. d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
