Re: md5sums files... and beyond
Hi,
In case anyone wonders about the status of replacing md5sums with
something stronger _in_ the binary packages, this should be considered
to be suspended until the next development cycle. (at least, from my
PoV).
It have been pointed out that those current checksum aren't sufficient
to validate that an installed package is secure (quoting Joey Hess:
there are innumerable ways for an attacker to inject bad
behavior/backdoors onto a system without touching binaries originating
from dpkg.[1] and it's also fairly easy to modify a file in /etc to
provide a backdoor ...)
Therefore, it should be clear that there is no urgency in replacing
DEBIAN/md5sums as they are useful for corruption and local (benign)
modification checksumming. (quoting Russ Allbery[2]).
The initial proposal to replace md5sum with ${better}sum:
http://wiki.debian.org/Sha256sumsInPackages
should be enhanced with further meta-data. A very early draft is:
http://wiki.debian.org/Proposals/BinaryPackageDescriptor
Regards,
Franklin
On Thu, 2010-03-11 at 00:44 +0100, Frank Lin PIAT wrote:
On Wed, 2010-03-03 at 03:06 +0100, Wouter Verhelst wrote:
I must say I was somewhat surprised by these numbers. Out of 2483
packages installed on my laptop, 2340 install md5sums. While that
might've been useful at some point, I don't think it still is.
Hi all,
Can you think of any sensible reason for not including md5sums of
control files, especially the {pre,post}{inst,rm} scripts ?
In the shasum file, those files could be either:
1. inserted, with the patch rewritten to match their expected
location on the target system.
or
2. inserted as a *comment* in the shasum file, like:
#68b329da9893e34099c7d8ad5cb9c940 CONTROL.TAR:postinst
[1] http://lists.debian.org/msgid-search/20100308225913.ga25...@gnu.kitenet.net
[2] http://lists.debian.org/msgid-search/87wrxmbkdn@windlord.stanford.edu
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/1269982677.3574.252.ca...@solid.paris.klabs.be
Re: md5sums files
On Wed, 2010-03-03 at 03:06 +0100, Wouter Verhelst wrote:
I must say I was somewhat surprised by these numbers. Out of 2483
packages installed on my laptop, 2340 install md5sums. While that
might've been useful at some point, I don't think it still is.
Hi all,
Can you think of any sensible reason for not including md5sums of
control files, especially the {pre,post}{inst,rm} scripts ?
In the shasum file, those files could be either:
1. inserted, with the patch rewritten to match their expected
location on the target system.
or
2. inserted as a *comment* in the shasum file, like:
#68b329da9893e34099c7d8ad5cb9c940 CONTROL.TAR:postinst
Thanks,
Franklin
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/1268264672.3291.3654.ca...@solid.paris.klabs.be
Re: md5sums files
On Fri, Mar 05, 2010 at 02:07:01PM -0600, Peter Samuelson wrote: [Russ Allbery] Figuring out a better solution for why the files in /var/lib/ispell and /var/lib/aspell are excluded from the md5sums generation because they change after installation is probably needed if we're going to remove creation of those files from control of the packager. It seems odd to ship files that are expected to change, but aren't conffiles. What happens if you either reinstall or upgrade the deb in question? I mean, the local changes get overwritten - how is that not a bad thing? That file is rewritten, a script run from dictionary postinst detects that it is rewritten (really that version in it is nil or lower that current compatibility version for the spellchecking engine) and rebuilds the dictionary hash according to data provided by the new dictionary. Same happens if a new spellchecking engine (ispell or aspell) is uploaded with a higher compatibility version, ispell or aspell postinsts run the hash build utility and all affected hashes are rebuilt, transition is done in a minute. No need of massive bug reports, pressure on maintainers, ... Regards, -- Agustin -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100308094718.gb3...@agmartin.aq.upm.es
Re: md5sums files
On Fri, Mar 05, 2010 at 11:45:38AM -0800, Russ Allbery wrote:
Don Armstrong d...@debian.org writes:
On Wed, 03 Mar 2010, Wouter Verhelst wrote:
In this day and age of completely and utterly broken MD5[0], I think we
should stop providing these files, and maybe provide something else
instead. Like, I dunno, shasums? Or perhaps gpgsigs? But stop
providing md5sums.
Is there any reason why we can't just modify dpkg-deb to create
DEBIAN/md5sums and DEBIAN/sha512sums and get archive coverage relatively
quickly, automatically, as things get rebuilt?
Figuring out a better solution for why the files in /var/lib/ispell and
/var/lib/aspell are excluded from the md5sums generation because they
change after installation is probably needed if we're going to remove
creation of those files from control of the packager.
Hashes at /var/lib/{a,i}spell are recreated on each package {re,}installation,
or after an {a,i}spell with an incompatible change is upgraded. This saves a
lot of problems when such transitions appear since all dictionary packages
suffering it get their hashes automatically upgraded.
The reason to have them listed by dpkg is that is way more robust to let
dpkg control its removal than to play with maintainer scripts for that.
Those placeholders are usually created by a plain touch, with a unique
md5sum,
$ touch spanish.hash spanish2.hash
$ md5sum spanish2.hash spanish.hash
d41d8cd98f00b204e9800998ecf8427e spanish2.hash
d41d8cd98f00b204e9800998ecf8427e spanish.hash
so if all files are to use md5sums (or whatever checksum hash), a file
under /var with that md5sum can be assumed to be a placeholder, expected
to change, so no problem if changes. Keeping them out of dpkg control
(create/build/erase only from maintainer scripts) does not add any
security and makes harder to know where those files come from (dpkg -S).
Regards,
--
Agustin
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100308094044.ga3...@agmartin.aq.upm.es
Re: md5sums files
On Mon, Mar 08, 2010 at 10:47:18AM +0100, Agustin Martin wrote: On Fri, Mar 05, 2010 at 02:07:01PM -0600, Peter Samuelson wrote: [Russ Allbery] Figuring out a better solution for why the files in /var/lib/ispell and /var/lib/aspell are excluded from the md5sums generation because they change after installation is probably needed if we're going to remove creation of those files from control of the packager. It seems odd to ship files that are expected to change, but aren't conffiles. What happens if you either reinstall or upgrade the deb in question? I mean, the local changes get overwritten - how is that not a bad thing? That file is rewritten, a script run from dictionary postinst detects that it is rewritten (really that version in it is nil or lower that current compatibility version for the spellchecking engine) and rebuilds the dictionary hash according to data provided by the new dictionary. Forgot to mention that those files contain no /etc like local changes. There are two types of files there, compat (or status) files and dictionary hashes. compat file is a file that is initially empty and that after a dictionary hash is created will contain the compatibility version used for its creation. This is not to be modified my sysadmin and has nothing to do with usual local changes at /etc. That compatibility version is used to check, when ispell or aspell are upgraded, if dictionary hash needs to be rebuilt. The other type of file are dictionary hashes, created on-the-fly. They depend on the data provided by the dictionary, on system endianess and on the compatibility version of the spellchecking engine, and need to be rebuilt if this last changes (really if changes to an non compatible format). -- Agustin -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100308100321.ga9...@agmartin.aq.upm.es
Re: md5sums files
Don Armstrong d...@debian.org writes: On Sat, 06 Mar 2010, Andreas Metzler wrote: Russ Allbery r...@debian.org wrote: Figuring out a better solution for why the files in /var/lib/ispell and /var/lib/aspell are excluded from the md5sums generation because they change after installation is probably needed I think these hash files are not arch independent. Which is why the packages only ship empty files in the deb and generate the correct binary data in postinst. This way the hash files are listed in dpkg -L, and dpkg takes care of removal instead of relying on maintainerscripts. So there's a period on upgrade where the file has been overwritten with an file before the new file has been generated? That's just wrong. Why? Considering the old hash file may be invalid anyway after you've upgraded other related packages (the aspell dict ones are format-dependent on the aspell version), why would it matter that the hash file is temporarily empty? -- Captain Logic is not steering this tugboat. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87vdd6hns0@bignachos.net
Re: md5sums files
On Mon, 08 Mar 2010, Brian Nelson wrote: Don Armstrong d...@debian.org writes: So there's a period on upgrade where the file has been overwritten with an file before the new file has been generated? That's just wrong. Why? Considering the old hash file may be invalid anyway after you've upgraded other related packages (the aspell dict ones are format-dependent on the aspell version), If this is actually a problem,[1] you can remove it in your preinst in those specific cases. As it is now, any reinstall or upgrade of this package will break these files for a period of time, even if there aren't any actual changes to the format. why would it matter that the hash file is temporarily empty? Because anything that expects to be able to use it would stop being able to until the postinst is run successfully, which could be many minutes on a long install. It may not be much of a problem in this specific case, but it's certainly not the way to solve this problem in general. Don Armstrong 1: The aspell dict files have a format specifier in the front, so presumably anyting that is consuming them will check them before blindly using them, and hopefully fail with an informative error message if necessary. -- The smallest quantity of bread that can be sliced and toasted has yet to be experimentally determined. In the quantum limit we must necessarily encounter fundamental toast particles which the author will unflinchingly designate here as croutons. -- Cser, Jim. Nanotechnology and the Physical Limits of Toastability. AIR 1:3, June, 1995. http://www.donarmstrong.com http://rzlab.ucr.edu -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100308182121.gb20...@volo.donarmstrong.com
Re: md5sums files
* Anthony Towns a...@erisian.com.au [100307 07:42]: Big? It only makes a difference if: [...] c) the system memory is corrupted just enough to screw the file but not everything else For many machines installing packages is the biggest thing they ever do. So with a very low error rate, that perhaps causes a iceweasel crash once a month, it is likely that something like this actually happened. If you're still worried, perhaps about having read() return bogus data from the .deb that happens to still be valid when passed through ungzip and untar and after you've already verified the entire file by md5/sha1/sha256 when downloading, And there still is the problem of the actual .deb being corrupted somewhere. It's not like there never were packages that got corrupted when generating the .deb and neither dak nor any apt nor dpkg gave any warnings, even though the gz crc was wrong. Hochachtungsvoll, Bernhard R. Link -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100307102221.gb4...@pcpool00.mathematik.uni-freiburg.de
Re: md5sums files
On Sun, Mar 07, 2010 at 01:25:44AM +0100, Goswin von Brederlow wrote: I don't think that just because something is required, it should be necessarily part of dpkg. So far, we are talking about a policy of including md5sum in our .debs, *not* about changing the .deb format to require md5sums (at least, as far as I can tell). Yes we do. If not having a md5sum (or rather sha256sum) file in the package is a policy violation of a MUST directive then the .deb format is effectively changed. No, it is not. Debian is not only consumer of the .deb format these days. Michael -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100307105032.ga19...@nighthawk.chemicalconnection.dyndns.org
Re: md5sums files
Peter Samuelson pe...@p12n.org writes: Peter Samuelson pe...@p12n.org writes: Be that as it may, I don't think the md5sums file was ever intended to be an integrity check of the .deb itself. Fortunately, the .deb also includes checksums of control.tar.gz and data.tar.gz, thanks to use of the gzip container format. [Goswin von Brederlow] That is not about the integrity of the deb. It is about the integrity of the files on the system. And if you do have faulty memory (or any of the other problems) then calculating the checksum locally will have a high risk of calculating it from already corrupted data and miss the error. How many times do I have to say the .deb also includes checksums of control.tar.gz and data.tar.gz, thanks to use of the gzip container format before you notice? You are still missing the point. - You download and verify the deb with the checksum in Packages.gz. - You unpack and some bits toggle making files corrupt. - You generate the md5sum from corrupt data. - Some time later you notice things don't work right that work elsewehere. - You verify the file integrity and all files check out fine. - You miss the problem. At the point when you need to check the integrity you do not have a control.tar.gz or data.tar.gz file. You don't have a deb. All you have is your corrupted data. MfG Goswin -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87fx4cqpbn@frosties.localdomain
Re: md5sums files
Michael Banck mba...@debian.org writes: On Sun, Mar 07, 2010 at 01:25:44AM +0100, Goswin von Brederlow wrote: I don't think that just because something is required, it should be necessarily part of dpkg. So far, we are talking about a policy of including md5sum in our .debs, *not* about changing the .deb format to require md5sums (at least, as far as I can tell). Yes we do. If not having a md5sum (or rather sha256sum) file in the package is a policy violation of a MUST directive then the .deb format is effectively changed. No, it is not. Debian is not only consumer of the .deb format these days. Michael Then they don't have to include the changes to dpkg that generate the checksum file if they really don't want it. They will have deb files then but they won't be Debian policy conformant. Note that nobody says dpkg should refuse to install debs without md5sum files. That would be silly. You are actualy making an argument FOR changing dpkg. Because with dpkg changed 3rd party debs will automatically get checksum files as well when they update their dpkg without them having to alter their source. The possibility of them specifically excluding the change out of dpkg is way less than the likelyhood of them not caring enough to change their rules file. MfG Goswin -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87bpf0qp1v@frosties.localdomain
Re: md5sums files
Peter Samuelson pe...@p12n.org writes: How many times do I have to say the .deb also includes checksums of control.tar.gz and data.tar.gz, thanks to use of the gzip container format before you notice? [Goswin von Brederlow] - You download and verify the deb with the checksum in Packages.gz. - You unpack and some bits toggle making files corrupt. - You generate the md5sum from corrupt data. What?!? No. You generate the md5sums file from the data.tar.gz. -- Peter Samuelson | org-tld!p12n!peter | http://p12n.org/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100307203918.gn18...@p12n.org
Re: md5sums files
Russ Allbery r...@debian.org wrote: Don Armstrong d...@debian.org writes: [...] Is there any reason why we can't just modify dpkg-deb to create DEBIAN/md5sums and DEBIAN/sha512sums and get archive coverage relatively quickly, automatically, as things get rebuilt? Figuring out a better solution for why the files in /var/lib/ispell and /var/lib/aspell are excluded from the md5sums generation because they change after installation is probably needed [...] Hello, I think these hash files are not arch independent. Which is why the packages only ship empty files in the deb and generate the correct binary data in postinst. This way the hash files are listed in dpkg -L, and dpkg takes care of removal instead of relying on maintainerscripts. cu andreas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/d9g967-3i3@argenau.downhill.at.eu.org
Re: md5sums files
On 06/03/2010 09:27, Andreas Metzler wrote: Russ Allbery r...@debian.org wrote: Don Armstrong d...@debian.org writes: [...] Is there any reason why we can't just modify dpkg-deb to create DEBIAN/md5sums and DEBIAN/sha512sums and get archive coverage relatively quickly, automatically, as things get rebuilt? Figuring out a better solution for why the files in /var/lib/ispell and /var/lib/aspell are excluded from the md5sums generation because they change after installation is probably needed [...] Hello, I think these hash files are not arch independent. Which is why the packages only ship empty files in the deb and generate the correct binary data in postinst. This way the hash files are listed in dpkg -L, and dpkg takes care of removal instead of relying on maintainerscripts. Anyway, there are many more cases where dpkg -L does not list all files pertaining to a single package. Configuration files managed by postinsts, for example, instances data, etc. It's a shame there is no standard way to declare the (package-)ownership of a file (the cruft(1) format is good enough, but is under-used, to say the least). I have recently taken an interest in obsolete conffiles and it's a real mess to check whether a conffile is really obsolete (can be removed) or is now managed by postinst (or ucf). Some conffiles left behind can have side-effects (usually not important ones). -- Jean-Christophe Dubacq -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4b9223ef.6040...@free.fr
Re: md5sums files
On Fri, Mar 05, 2010 at 07:54:29PM +0100, Goswin von Brederlow wrote: Bernd Zeimetz be...@bzed.de writes: I think its about time to require to generate checksums for packages and make all packages which do not do so RC buggy. If a checksum file becomes required then it really is not the job of the package to build it. Instead dpkg should generate one and include it automatically. Following that logic, dpkg should do everything anyway (like rpm), no need for debhelper, dh, or anything. I don't think that just because something is required, it should be necessarily part of dpkg. So far, we are talking about a policy of including md5sum in our .debs, *not* about changing the .deb format to require md5sums (at least, as far as I can tell). Once the former works out fine for squeeze, we can still think about doing the latter by modifying dpkg-builddeb or something, but I think this is premature at this point (in the release cycle). Michael -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100306114901.gb13...@nighthawk.chemicalconnection.dyndns.org
Re: md5sums files
(I'm not subscribed to this list, so go ahead and Cc me)
On Thu, Mar 4, 2010 at 02:05, Peter Samuelson pe...@p12n.org wrote:
[Wouter Verhelst]
I must say I was somewhat surprised by these numbers. Out of 2483
packages installed on my laptop, 2340 install md5sums.
The surprising part, perhaps, is that dpkg itself didn't just generate
the other 143 md5sums files at installation time.
The easy (and usually correct) reason for things like that is dpkg's
source is scary.
I suggested this a long time ago and of course was met with so where's
your patch? Of course I was not willing to do the work.
See? Anyway, my patch is attached. It makes dpkg create a foo.hashes
when unpacking foo, whose contents looks like:
MD5:32b5e22f8e336b2f34e0dd87652e6dfc usr/share/doc/mawk/changelog.gz
MD5:87a34f1f55ac3f7fec2c7fc82565e8eb usr/share/doc/mawk/changelog.Debian.gz
...
Verification is a matter of something like:
$ cat /var/lib/dpkg/info/*.hashes | sed -n 's/^MD5://p' | (cd /;
md5sum -c) | grep -v ': OK$'
There's an option (--hash) that you can set to none to avoid
spending time calculating md5s if you so choose. Adding support for
sha1/sha256/whatever should be straightforward; afaik dpkg only has
code for md5 already built in though (though just invoking
/usr/bin/sha1sum etc would be an option of course).
Of course another option is just to pull the md5sums directly from the deb:
$ ar p /var/cache/apt/archives/ifupdown_0.6.9_i386.deb data.tar.gz |
tar --to-command='printf %s%s\n $(md5sum - | sed s/-$//)
${TAR_FILENAME#./}' -xzf - |
diff - /var/lib/dpkg/info/ifupdown.md5sums
1,3d0
346208729633adf45e2fa3f2bd3b19c6 etc/init.d/ifupdown
c6fffaae03271f1641920105ce68796b etc/init.d/ifupdown-clean
fab851ca87c5deb9d6f665e610184648 etc/default/ifupdown
4a2
a0f11cf1809a468c49b72e0aa0a8e26b sbin/ifup
(md5sums doesn't normally list conffiles, but does list hardlinks; the
above command does the opposite)
But
fundamentally, shipping a md5sums file is really just a tradeoff in
download size vs. installation speed, not unlike gzip vs. bzip2.
Advantages of doing in when unpacking:
- choice of checksum is the admin's decision
- we can quickly roll out support for sha1/sha256/crc/... checksums
by just changing one package
- admin has hashes of exactly what was unpacked, no matter the source
- no concerns about bugs in dh_md5sums or similar resulting in bad checksums
Advantages of doing it when uploading:
- provides some sort of double check of what's being uploaded
- saves CPU time on users' machines
For me, I'd rather have dpkg generate the hashes.
Cheers,
aj
--
Anthony Towns a...@erisian.com.au
diff -urb dpkg-1.15.5.6/debian/changelog dpkg-1.15.5.6-aj/debian/changelog
--- dpkg-1.15.5.6/debian/changelog 2010-01-09 04:02:03.0 +1000
+++ dpkg-1.15.5.6-aj/debian/changelog 2010-03-07 04:13:03.171356041 +1000
@@ -1,3 +1,11 @@
+dpkg (1.15.5.6+aj) unstable; urgency=low
+
+ * Non-maintainer upload.
+ * Automatically create /var/lib/dpkg/info/pkg.hashes containing MD5 hashes
+for unpacked files.
+
+ -- Anthony Towns a...@lazuline Sun, 07 Mar 2010 04:12:32 +1000
+
dpkg (1.15.5.6) unstable; urgency=low
* dpkg-source: with format 3.0 (quilt) ensure quilt's .pc directory is
diff -urb dpkg-1.15.5.6/configure.ac dpkg-1.15.5.6-aj/configure.ac
--- dpkg-1.15.5.6/configure.ac 2010-01-08 18:00:34.0 +1000
+++ dpkg-1.15.5.6-aj/configure.ac 2010-03-07 04:38:32.547372468 +1000
@@ -51,6 +51,16 @@
esac])
AC_SUBST(admindir)
+# Allow alternative default hash function
+hashtype=md5
+AC_ARG_WITH(hashtype,
+ AS_HELP_STRING([--with-hashtype=HASH],
+ [hash function to use for .hashes files]),
+[case $with_hashtype in
+ md5|none) hashtype=$with_hashtype ;;
+ *) AC_MSG_ERROR([invalid hashtype specified]) ;;
+esac])
+AC_SUBST(hashtype)
# Checks for programs.
AC_PROG_CC
diff -urb dpkg-1.15.5.6/src/Makefile.am dpkg-1.15.5.6-aj/src/Makefile.am
--- dpkg-1.15.5.6/src/Makefile.am 2010-01-09 03:23:06.0 +1000
+++ dpkg-1.15.5.6-aj/src/Makefile.am 2010-03-07 04:28:18.771356095 +1000
@@ -6,6 +6,7 @@
AM_CPPFLAGS = \
-DLOCALEDIR=\$(localedir)\ \
-DADMINDIR=\$(admindir)\ \
+ -DHASHTYPE=\$(hashtype)\ \
-idirafter $(top_srcdir)/lib/compat \
-I$(top_builddir) \
-I$(top_srcdir)/lib
diff -urb dpkg-1.15.5.6/src/main.c dpkg-1.15.5.6-aj/src/main.c
--- dpkg-1.15.5.6/src/main.c 2010-01-09 03:23:06.0 +1000
+++ dpkg-1.15.5.6-aj/src/main.c 2010-03-07 04:29:59.271360858 +1000
@@ -187,6 +187,7 @@
const char *admindir= ADMINDIR;
const char *instdir= ;
struct pkg_list *ignoredependss = NULL;
+const char *hashtype= HASHTYPE;
static const struct forceinfo {
const char *name;
@@ -516,6 +517,7 @@
{ admindir, 0, 1, NULL, admindir, NULL, 0 },
{ instdir, 0, 1, NULL, instdir, NULL, 0 },
{ ignore-depends,0, 1, NULL, NULL, ignoredepends, 0 },
+ { hash, 0, 1, NULL
Re: md5sums files
On Sat, 06 Mar 2010, Andreas Metzler wrote: Russ Allbery r...@debian.org wrote: Figuring out a better solution for why the files in /var/lib/ispell and /var/lib/aspell are excluded from the md5sums generation because they change after installation is probably needed I think these hash files are not arch independent. Which is why the packages only ship empty files in the deb and generate the correct binary data in postinst. This way the hash files are listed in dpkg -L, and dpkg takes care of removal instead of relying on maintainerscripts. So there's a period on upgrade where the file has been overwritten with an file before the new file has been generated? That's just wrong. This is why you don't ship files in the package that you're going to regenerate later. It's not like they can't be removed upon purge or removal fairly trivially. Don Armstrong -- I shall require that [a scientific system's] logical form shall be such that it can be singled out, by means of emperical tests, in a negative sense: it must be possible for an emperical scientific system to be refuted by experience. -- Sir Karl Popper _Logic of Scientific Discovery_ §6 http://www.donarmstrong.com http://rzlab.ucr.edu -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100306222825.gp28...@volo.donarmstrong.com
Re: md5sums files
Peter Samuelson pe...@p12n.org writes: On Wed, Mar 3, 2010 at 10:05:11 -0600, Peter Samuelson wrote: fundamentally, shipping a md5sums file is really just a tradeoff in download size vs. installation speed, not unlike gzip vs. bzip2. One Julien Cristau jcris...@debian.org writes: Only if you assume that disks never fail and thus files never get corrupted when the package gets unpacked. [Goswin von Brederlow] Or the memory, the cpu, the pci bus, the ide bus, ... have a bit toggler. There are many ways file can be corrupted between being downloaded (where apt checks them) and them being unpacked and checksumed locally. Be that as it may, I don't think the md5sums file was ever intended to be an integrity check of the .deb itself. Fortunately, the .deb also includes checksums of control.tar.gz and data.tar.gz, thanks to use of the gzip container format. That is not about the integrity of the deb. It is about the integrity of the files on the system. And if you do have faulty memory (or any of the other problems) then calculating the checksum locally will have a high risk of calculating it from already corrupted data and miss the error. MfG Goswin -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87zl2louls@frosties.localdomain
Re: md5sums files
Michael Banck mba...@debian.org writes: On Fri, Mar 05, 2010 at 07:54:29PM +0100, Goswin von Brederlow wrote: Bernd Zeimetz be...@bzed.de writes: I think its about time to require to generate checksums for packages and make all packages which do not do so RC buggy. If a checksum file becomes required then it really is not the job of the package to build it. Instead dpkg should generate one and include it automatically. Following that logic, dpkg should do everything anyway (like rpm), no need for debhelper, dh, or anything. That does not follow. Nobody says you MUST use debhelper but people in this thread want you MUST have a checksum file. I don't think that just because something is required, it should be necessarily part of dpkg. So far, we are talking about a policy of including md5sum in our .debs, *not* about changing the .deb format to require md5sums (at least, as far as I can tell). Yes we do. If not having a md5sum (or rather sha256sum) file in the package is a policy violation of a MUST directive then the .deb format is effectively changed. Once the former works out fine for squeeze, we can still think about doing the latter by modifying dpkg-builddeb or something, but I think this is premature at this point (in the release cycle). Doing the later already solves the former without any extra work for any (other) maintainer. Every single deb being build from then on would instantly and automatically have the new checksum field. No need for every maintainer for add a dh_sha256sum to debian/rules. Having every maintainer add/change the checksum generation just so they can remove it again in the next phase would be truely pointless. No package would become RC buggy due to lack of generating the checksum file. MfG Goswin -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87vdd9ou47@frosties.localdomain
Re: md5sums files
Anthony Towns a...@erisian.com.au writes:
(I'm not subscribed to this list, so go ahead and Cc me)
On Thu, Mar 4, 2010 at 02:05, Peter Samuelson pe...@p12n.org wrote:
[Wouter Verhelst]
I must say I was somewhat surprised by these numbers. Out of 2483
packages installed on my laptop, 2340 install md5sums.
The surprising part, perhaps, is that dpkg itself didn't just generate
the other 143 md5sums files at installation time.
The easy (and usually correct) reason for things like that is dpkg's
source is scary.
I suggested this a long time ago and of course was met with so where's
your patch? Of course I was not willing to do the work.
See? Anyway, my patch is attached. It makes dpkg create a foo.hashes
when unpacking foo, whose contents looks like:
MD5:32b5e22f8e336b2f34e0dd87652e6dfc usr/share/doc/mawk/changelog.gz
MD5:87a34f1f55ac3f7fec2c7fc82565e8eb usr/share/doc/mawk/changelog.Debian.gz
...
Verification is a matter of something like:
$ cat /var/lib/dpkg/info/*.hashes | sed -n 's/^MD5://p' | (cd /;
md5sum -c) | grep -v ': OK$'
There's an option (--hash) that you can set to none to avoid
spending time calculating md5s if you so choose. Adding support for
sha1/sha256/whatever should be straightforward; afaik dpkg only has
code for md5 already built in though (though just invoking
/usr/bin/sha1sum etc would be an option of course).
Of course another option is just to pull the md5sums directly from the deb:
$ ar p /var/cache/apt/archives/ifupdown_0.6.9_i386.deb data.tar.gz |
tar --to-command='printf %s%s\n $(md5sum - | sed s/-$//)
${TAR_FILENAME#./}' -xzf - |
diff - /var/lib/dpkg/info/ifupdown.md5sums
1,3d0
346208729633adf45e2fa3f2bd3b19c6 etc/init.d/ifupdown
c6fffaae03271f1641920105ce68796b etc/init.d/ifupdown-clean
fab851ca87c5deb9d6f665e610184648 etc/default/ifupdown
4a2
a0f11cf1809a468c49b72e0aa0a8e26b sbin/ifup
(md5sums doesn't normally list conffiles, but does list hardlinks; the
above command does the opposite)
But
fundamentally, shipping a md5sums file is really just a tradeoff in
download size vs. installation speed, not unlike gzip vs. bzip2.
Advantages of doing in when unpacking:
- choice of checksum is the admin's decision
- we can quickly roll out support for sha1/sha256/crc/... checksums
by just changing one package
- admin has hashes of exactly what was unpacked, no matter the source
- no concerns about bugs in dh_md5sums or similar resulting in bad checksums
Advantages of doing it when uploading:
- provides some sort of double check of what's being uploaded
- saves CPU time on users' machines
- avoids having bad checksums due to the user having bad hardware
(which is one big use case of the files)
For me, I'd rather have dpkg generate the hashes.
Cheers,
aj
--
Anthony Towns a...@erisian.com.au
MfG
Goswin
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87r5nxou09@frosties.localdomain
Re: md5sums files
Peter Samuelson pe...@p12n.org writes: Be that as it may, I don't think the md5sums file was ever intended to be an integrity check of the .deb itself. Fortunately, the .deb also includes checksums of control.tar.gz and data.tar.gz, thanks to use of the gzip container format. [Goswin von Brederlow] That is not about the integrity of the deb. It is about the integrity of the files on the system. And if you do have faulty memory (or any of the other problems) then calculating the checksum locally will have a high risk of calculating it from already corrupted data and miss the error. How many times do I have to say the .deb also includes checksums of control.tar.gz and data.tar.gz, thanks to use of the gzip container format before you notice? -- Peter Samuelson | org-tld!p12n!peter | http://p12n.org/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100307032816.gm18...@p12n.org
Re: md5sums files
On Sun, Mar 7, 2010 at 10:28, Goswin von Brederlow goswin-...@web.de wrote:
Anthony Towns a...@erisian.com.au writes:
Advantages of doing it when uploading:
- provides some sort of double check of what's being uploaded
- saves CPU time on users' machines
- avoids having bad checksums due to the user having bad hardware
(which is one big use case of the files)
Big? It only makes a difference if:
a) the corruption happens as soon as it's written, not after some time
b) the file is too big/the system is too loaded to keep the file in
the page cache
c) the system memory is corrupted just enough to screw the file but
not everything else
Compared to random make install invocations changing files in the
system and similar, that doesn't strike me as a big use case.
In any event, it's fairly easy to generate the checksum in the same
pass as generating the file, see the attached patch. (It's not as easy
to generalise to other hashes as the previous one, unfortunately)
If you're still worried, perhaps about having read() return bogus data
from the .deb that happens to still be valid when passed through
ungzip and untar and after you've already verified the entire file by
md5/sha1/sha256 when downloading, you're getting to the point of
trying to safely install on an actively malicious system, and
nothing's going to make that work.
Cheers,
aj
--
Anthony Towns a...@erisian.com.au
diff -x configure -x '*.m4' -x Makefile.in -urbN dpkg-1.15.5.6/configure.ac dpkg-1.15.5.6-aj/configure.ac
--- dpkg-1.15.5.6/configure.ac 2010-01-08 18:00:34.0 +1000
+++ dpkg-1.15.5.6-aj/configure.ac 2010-03-07 04:38:32.547372468 +1000
@@ -51,6 +51,16 @@
esac])
AC_SUBST(admindir)
+# Allow alternative default hash function
+hashtype=md5
+AC_ARG_WITH(hashtype,
+ AS_HELP_STRING([--with-hashtype=HASH],
+ [hash function to use for .hashes files]),
+[case $with_hashtype in
+ md5|none) hashtype=$with_hashtype ;;
+ *) AC_MSG_ERROR([invalid hashtype specified]) ;;
+esac])
+AC_SUBST(hashtype)
# Checks for programs.
AC_PROG_CC
diff -x configure -x '*.m4' -x Makefile.in -urbN dpkg-1.15.5.6/debian/changelog dpkg-1.15.5.6-aj/debian/changelog
--- dpkg-1.15.5.6/debian/changelog 2010-01-09 04:02:03.0 +1000
+++ dpkg-1.15.5.6-aj/debian/changelog 2010-03-07 04:13:03.171356041 +1000
@@ -1,3 +1,11 @@
+dpkg (1.15.5.6+aj) unstable; urgency=low
+
+ * Non-maintainer upload.
+ * Automatically create /var/lib/dpkg/info/pkg.hashes containing MD5 hashes
+for unpacked files.
+
+ -- Anthony Towns a...@lazuline Sun, 07 Mar 2010 04:12:32 +1000
+
dpkg (1.15.5.6) unstable; urgency=low
* dpkg-source: with format 3.0 (quilt) ensure quilt's .pc directory is
diff -x configure -x '*.m4' -x Makefile.in -urbN dpkg-1.15.5.6/lib/dpkg/buffer.c dpkg-1.15.5.6-aj/lib/dpkg/buffer.c
--- dpkg-1.15.5.6/lib/dpkg/buffer.c 2010-01-09 03:23:06.0 +1000
+++ dpkg-1.15.5.6-aj/lib/dpkg/buffer.c 2010-03-07 15:50:33.379710844 +1000
@@ -60,6 +60,13 @@
case BUFFER_WRITE_MD5:
buffer_md5_init(write_data);
break;
+ case BUFFER_WRITE_DUP:
+ {
+ struct buffer_data *bddup = write_data-arg.ptr;
+ buffer_init(NULL, bddup[0]);
+ buffer_init(NULL, bddup[1]);
+ }
+ break;
}
return 0;
}
@@ -90,6 +97,13 @@
case BUFFER_WRITE_MD5:
buffer_md5_done(write_data);
break;
+ case BUFFER_WRITE_DUP:
+ {
+ struct buffer_data *bddup = write_data-arg.ptr;
+ buffer_done(NULL, bddup[0]);
+ buffer_done(NULL, bddup[1]);
+ }
+ break;
}
return 0;
}
@@ -126,6 +140,14 @@
case BUFFER_WRITE_MD5:
MD5Updatestruct buffer_write_md5ctx *)data-arg.ptr)-ctx), buf, length);
break;
+ case BUFFER_WRITE_DUP:
+ {
+ struct buffer_data *bddup = data-arg.ptr;
+ ret = buffer_write(bddup[0], buf, length, desc);
+ if (ret != length) return ret;
+ ret = buffer_write(bddup[1], buf, length, desc);
+ }
+ break;
default:
internerr(unknown data type '%i' in buffer_write,
data-type);
diff -x configure -x '*.m4' -x Makefile.in -urbN dpkg-1.15.5.6/lib/dpkg/buffer.h dpkg-1.15.5.6-aj/lib/dpkg/buffer.h
--- dpkg-1.15.5.6/lib/dpkg/buffer.h 2010-01-08 18:00:34.0 +1000
+++ dpkg-1.15.5.6-aj/lib/dpkg/buffer.h 2010-03-07 15:53:23.319707965 +1000
@@ -36,6 +36,7 @@
#define BUFFER_WRITE_NULL 3
#define BUFFER_WRITE_STREAM 4
#define BUFFER_WRITE_MD5 5
+#define BUFFER_WRITE_DUP 6
#define BUFFER_READ_FD 0
#define BUFFER_READ_STREAM 1
@@ -52,6 +53,14 @@
buffer_hash(buf, hash, BUFFER_WRITE_MD5, limit)
#if HAVE_C99
+# define fd_fdmd5(fd1, fd2, hash, limit, ...) \
+ do { \
+ struct buffer_data fdhash[2]; \
+fdhash[0].arg.i = fd2; fdhash[0].type = BUFFER_WRITE_FD; \
+fdhash[1].arg.ptr = hash; fdhash[1].type = BUFFER_WRITE_MD5; \
+buffer_copy_IntPtr(fd1, BUFFER_READ_FD, fdhash, BUFFER_WRITE_DUP, \
+ limit, __VA_ARGS__); \
+} while(0)
# define fd_md5(fd, hash, limit, ...) \
Re: md5sums files
Bernd Zeimetz be...@bzed.de writes: Philipp Kern wrote: On 2010-03-03, Wouter Verhelst wou...@debian.org wrote: This is where I disagree. When a checksum algorithm is compromised (and MD5 *is* compromised), things only ever get worse, not better. Indeed, MD5 preimage attacks are pretty hard *today*. But switching to something more secure in preparation for the day when MD5 will be easily cracked by every script kiddo around is *not* overkill. Sure, but to be honest, not even all packages managed to generate md5sums 'till now (with some quite core, omnipresent packages missing) so it seems out of scope for squeeze. Maybe squeeze+1. I think its about time to require to generate checksums for packages and make all packages which do not do so RC buggy. If a checksum file becomes required then it really is not the job of the package to build it. Instead dpkg should generate one and include it automatically. And given the widespread generation of md5sum files that really should be automated anyway. This would go nicely with changing the checksum algorithm. Just prepare a patch for dpkg to generate a sha256sum file automatically when it builds a deb and then packages can stop generating md5sum files over time. MfG Goswin -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87aaumzjiy@frosties.localdomain
Re: md5sums files
Julien Cristau jcris...@debian.org writes: On Wed, Mar 3, 2010 at 10:05:11 -0600, Peter Samuelson wrote: fundamentally, shipping a md5sums file is really just a tradeoff in download size vs. installation speed, not unlike gzip vs. bzip2. One Only if you assume that disks never fail and thus files never get corrupted when the package gets unpacked. Cheers, Julien Or the memory, the cpu, the pci bus, the ide bus, ... have a bit toggler. There are many ways file can be corrupted between being downloaded (where apt checks them) and them being unpacked and checksumed locally. MfG Goswin -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/871vfyziay@frosties.localdomain
Re: md5sums files
On Wed, 03 Mar 2010, Wouter Verhelst wrote: In this day and age of completely and utterly broken MD5[0], I think we should stop providing these files, and maybe provide something else instead. Like, I dunno, shasums? Or perhaps gpgsigs? But stop providing md5sums. Is there any reason why we can't just modify dpkg-deb to create DEBIAN/md5sums and DEBIAN/sha512sums and get archive coverage relatively quickly, automatically, as things get rebuilt? Don Armstrong -- We cast this message into the cosmos. [...] We are trying to survive our time so we may live into yours. We hope some day, having solved the problems we face, to join a community of Galactic Civilizations. This record represents our hope and our determination and our goodwill in a vast and awesome universe. -- Jimmy Carter on the Voyager Golden Record http://www.donarmstrong.com http://rzlab.ucr.edu -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100305194048.gm28...@volo.donarmstrong.com
Re: md5sums files
Don Armstrong d...@debian.org writes: On Wed, 03 Mar 2010, Wouter Verhelst wrote: In this day and age of completely and utterly broken MD5[0], I think we should stop providing these files, and maybe provide something else instead. Like, I dunno, shasums? Or perhaps gpgsigs? But stop providing md5sums. Is there any reason why we can't just modify dpkg-deb to create DEBIAN/md5sums and DEBIAN/sha512sums and get archive coverage relatively quickly, automatically, as things get rebuilt? Figuring out a better solution for why the files in /var/lib/ispell and /var/lib/aspell are excluded from the md5sums generation because they change after installation is probably needed if we're going to remove creation of those files from control of the packager. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87fx4ek0wt@windlord.stanford.edu
Re: md5sums files
[Russ Allbery] Figuring out a better solution for why the files in /var/lib/ispell and /var/lib/aspell are excluded from the md5sums generation because they change after installation is probably needed if we're going to remove creation of those files from control of the packager. It seems odd to ship files that are expected to change, but aren't conffiles. What happens if you either reinstall or upgrade the deb in question? I mean, the local changes get overwritten - how is that not a bad thing? I've always thought if you have data shipped by a package that will be changed after the fact, it should be shipped in /usr/lib or /usr/share and then copied into /var/lib at install time, or as needed. -- Peter Samuelson | org-tld!p12n!peter | http://p12n.org/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100305200701.gk18...@p12n.org
Re: md5sums files
On Wed, Mar 3, 2010 at 10:05:11 -0600, Peter Samuelson wrote: fundamentally, shipping a md5sums file is really just a tradeoff in download size vs. installation speed, not unlike gzip vs. bzip2. One Julien Cristau jcris...@debian.org writes: Only if you assume that disks never fail and thus files never get corrupted when the package gets unpacked. [Goswin von Brederlow] Or the memory, the cpu, the pci bus, the ide bus, ... have a bit toggler. There are many ways file can be corrupted between being downloaded (where apt checks them) and them being unpacked and checksumed locally. Be that as it may, I don't think the md5sums file was ever intended to be an integrity check of the .deb itself. Fortunately, the .deb also includes checksums of control.tar.gz and data.tar.gz, thanks to use of the gzip container format. -- Peter Samuelson | org-tld!p12n!peter | http://p12n.org/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100305201042.gl18...@p12n.org
Re: md5sums files
On Wed, Mar 03, 2010 at 01:51:20PM -0800, Russ Allbery wrote:
We added a Lintian tag warning about not having md5sums in that time
frame, I think.
Gotcha, that must be it. Very impressive result.
FWIW, about the people proposing to change from md5 to something else,
it should be pretty easy to achieve too: if all of us is using
dh_md5sums, it would be enough to change its implementation to store a
different kind of checksums (the name will become pretty bogus, though).
Russ, while we are at it, would you mind a bug report on the policy to
suggest (starting at SHOULD?) to store md5sums in packages?
Thanks,
Cheers.
--
Stefano Zacchiroli -o- PhD in Computer Science \ PostDoc @ Univ. Paris 7
z...@{upsilon.cc,pps.jussieu.fr,debian.org} -- http://upsilon.cc/zack/
Dietro un grande uomo c'è ..| . |. Et ne m'en veux pas si je te tutoie
sempre uno zaino ...| ..: | Je dis tu à tous ceux que j'aime
signature.asc
Description: Digital signature
Re: md5sums files
On Thu, Mar 04, 2010 at 09:11:21AM +0100, Stefano Zacchiroli wrote: FWIW, about the people proposing to change from md5 to something else, it should be pretty easy to achieve too: if all of us is using dh_md5sums, Which is not the case. There are still people who like to package without helpers. it would be enough to change its implementation to store a different kind of checksums (the name will become pretty bogus, though). Maybe that would be a good starting point to give it a more common name although this would require somewhat more work ;) Regards, Patrick -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100304082704.ga20...@lisa
Re: md5sums files
On Thu, Mar 04, 2010 at 09:27:06AM +0100, Patrick Schoenfeld wrote:
Which is not the case. There are still people who like to package
without helpers.
Agreed, that's exactly why I've proposed also the bug report on
policy. As per our customs, the policy will then state the desired
result, the helpers will implement way(s) to achieve it.
Maybe that would be a good starting point to give it a more
common name although this would require somewhat more work ;)
In fact, if we go that way, it would probably be worth to have some
generalization as well, so that if one day we decide to change the
checksum format we don't need to change the policy. (Don't know how
simple that would be, though.)
Cheers.
--
Stefano Zacchiroli -o- PhD in Computer Science \ PostDoc @ Univ. Paris 7
z...@{upsilon.cc,pps.jussieu.fr,debian.org} -- http://upsilon.cc/zack/
Dietro un grande uomo c'è ..| . |. Et ne m'en veux pas si je te tutoie
sempre uno zaino ...| ..: | Je dis tu à tous ceux que j'aime
signature.asc
Description: Digital signature
Re: md5sums files
On Wed, Mar 03, 2010 at 10:19:21PM +0100, Stefano Zacchiroli wrote: On Wed, Mar 03, 2010 at 08:08:38PM +0100, Bernd Zeimetz wrote: I think its about time to require to generate checksums for packages and make all packages which do not do so RC buggy. Well, RC buggy is probably a tad excessive for this release, considering that we are (I hope :)) close to a freeze. Given that a) we're almost there, and b) adding a debsums (or dh_md5sums) invocation to debian/rules isn't exactly an intrusive change, I'm not sure that 'excessive' is the right word. I'd call it 'ambitious', perhaps overly so, but it should be doable. Of course, whether we decide to indeed do so would be up to the release managers. I'm inclined to agree that it's too late to add another checksum algorithm at this point in time; that should be a release goal for squeeze+1. -- The biometric identification system at the gates of the CIA headquarters works because there's a guard with a large gun making sure no one is trying to fool the system. http://www.schneier.com/blog/archives/2009/01/biometrics.html signature.asc Description: Digital signature
Re: md5sums files
Hi, On Wed, Mar 03, 2010 at 02:25:44PM -0500, Joey Hess wrote: Osamu Aoki wrote: True but debsums can address these issues by system administrator touch-ups as documented in manpage using: * /etc/apt/apt.conf.d/90debsums (debsums = 2.0.7) * debsums_init(8) (debsums = 2.0.34 @ 2007) It's not uncommon to be given an existing system and want to verify that no files were modified by its creator, and the current lack of md5sums files for a few packages prevents using debsums to do that. Wait a moment... I do not think those few packages prevents using debsums to do that. By using debsums_init, you can download (but not install) packages and create md5sums files for those few packages preventing to use debsums first. At least using debsums_init, you have easier time to do what you were thinking to do. (That is why I wrote it. It is ultra short simple shell script.) (The most common case is probably buying a Debian VM from a hosting company.) (If someone changes installed values of md5sums, debsums are helpless, as we all agree.) Osamu -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100304142850.ga4...@osamu.debian.net
Re: md5sums files
]] James Vega | On Thu, Mar 04, 2010 at 02:11:55AM +0100, Harald Braumann wrote: | I think I was finally able to decipher your message. But my other | points still hold. And while it is just a matter of programming, | simple or not, it already exists in debhelper. So doing it at build | time is SMOAOLTDR, by which I mean simply a matter of adding one | line to debian/rules. | | You make the mistake of assuming everyone use debhelper. From the graph at http://people.debian.org/~cjwatson/dhstats.png, that is not very far from the truth. Sure, we should make sure that people who aren't using debhelper can continue not doing so, but those couple of hundred packages is a very small fraction of the archive, even if some of them are important packages. -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87fx4fvrdy@qurzaw.linpro.no
Re: md5sums files
]] Frank Lin PIAT | What about a transitional dh_md5sums that would produce md5sum AND | invoke dh_sha ? Or call it dh_checksums or something so we don't have to change the tool name each time we decide to change the algorithm. (And I want the shed blue with pink spots.) -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87bpf3vrai@qurzaw.linpro.no
Re: md5sums files
Stefano Zacchiroli z...@debian.org writes: Russ, while we are at it, would you mind a bug report on the policy to suggest (starting at SHOULD?) to store md5sums in packages? Not that I've had any time to work on Policy (or Lintian) in the last month, but that does seem reasonable to me. It seems to be a widespread best practice already, and a lot of people are turning up in this thread to say that they find it useful. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87vddb23lx@windlord.stanford.edu
Re: md5sums files
This script signs each file in the package individually, but it could also concatenate them all alphabetically and create just one signature. There have been previous discussions on debian-devel about this. I believe DAK does not allow packages signed using debsigs to be uploaded. I'm not sure if that's out of objection to the entire concept, or whether there are just technical issues that need to be resolved first. (I probably would know if I had a better memory for the previous discussion, but unfortunately I appear to have recycled those brain cells.) http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=340306 dpkg-sig is only missing active maintenance before support for it could be added to the debian archive. -- bye, Joerg The Rathaus is the town hall, not where rats live (some may dispute that). (http://cioppino.blogs.com/hungrig_in_san_francisco/2006/06/englisch_oder_d.html) -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87eijzyd91@lennier.ganneff.de
Re: md5sums files
On jeu., 2010-03-04 at 22:43 +0100, Joerg Jaspert wrote: This script signs each file in the package individually, but it could also concatenate them all alphabetically and create just one signature. There have been previous discussions on debian-devel about this. I believe DAK does not allow packages signed using debsigs to be uploaded. I'm not sure if that's out of objection to the entire concept, or whether there are just technical issues that need to be resolved first. (I probably would know if I had a better memory for the previous discussion, but unfortunately I appear to have recycled those brain cells.) http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=340306 dpkg-sig is only missing active maintenance before support for it could be added to the debian archive. That's definitely something which would be nice to have, but how exactly would be handled the binary uploads (by the buildds, binNMU stuff like that)? I didn't know dpkg-sig so I'll check on that (not sure if Marc Brockschmidt is still interested, since homepage is dead and there was no upload since 2006, and it was an NMU). Cheers, -- Yves-Alexis signature.asc Description: This is a digitally signed message part
Re: md5sums files
On Wed, 3 Mar 2010 08:35:18 +0100
Mike Hommey m...@glandium.org wrote:
On Wed, Mar 03, 2010 at 06:30:34AM +, Sune Vuorela wrote:
The md5 sums isn't to be used in case of a break in, as you can't trust
anything on the system anyways, but more things like:
- did I make; sudo make install something on top of packages
- did I just quickly hack a p{erl,ython}-script on the system to do
something different and forgot
Which makes me think... wouldn't it be nice from dpkg to check the
package files haven't been modified before upgrading ?
No - if you're upgrading, you're doing it because you want to be sure
that the Debian version replaces the old version. However, if you do
use 'make; sudo make install; with a prefix of /usr instead
of /usr/local then a) you're taking a risk and b) md5sums are only a
partial protection. md5sums will NOT catch instances where the upstream
'make install' provides files that are not part of the Debian package
nor will it catch files that have been moved as part of Debian
packaging. As these files are not put into place by dpkg anyway, then
they won't get cleaned up by dpkg and cannot be detected either by
using md5sums or by using dpkg. (dpkg will complain that certain
directories are not empty if the packaged files being upgraded are in
special places but not otherwise.) There's every chance that having
extraneous and/or duplicate content in the wrong places will break the
Debian package in ways that are not easily detectable and md5sums won't
help with that.
Having md5sums around for simple checks like local admins copying
modified scripts over packaged versions is one thing but IMHO it does
not justify having md5sums in the wider case because a) local admin
changes are the responsibility of the local admin and b) md5sums only
help detect those changes where the admin changes a filename that
exactly matches the packaged name rather than adding more content /
cruft.
'sudo make install' into /usr is not something that md5sums can help us
fix and if that is the sole justification for retaining them then I
think that is a false argument.
Debian does move a lot of files around during packaging (the recent
stipulation against files in /var/www/ is just one instance) - my own
feeling is that the very packages that DO have their files moved around
for FHS reasons are the very ones that (some) local admins may want to
install from upstream tarballs ahead of the Debian package being
updated. Moving files invalidates any protection from having the
md5sum - unless the local admin retains a separate list of which files
have been installed separately, but then I thought the idea was that
such a record would be available by installing into /usr/local instead
of /usr in the first place. ;-)
Changing to SHA won't help. I'm for ditching all md5sums from packages.
It's not a lot of disc space gained but it does give a false sense of
security or 'insurance' if you want to avoid the more formal meaning of
'security'.
--
Neil Williams
=
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/
pgpGQsxssIvRd.pgp
Description: PGP signature
Re: md5sums files
On 2010-03-03, Neil Williams codeh...@debian.org wrote: Changing to SHA won't help. I'm for ditching all md5sums from packages. It's not a lot of disc space gained but it does give a false sense of security or 'insurance' if you want to avoid the more formal meaning of 'security'. Please don't. It's not about security. It's about being able to detect corruption. Also it is very helpful when recovering from ext4 root FS corruption after a sudden power loss. Sure, you cannot guarantee that the md5 store isn't corrupted too but if it isn't then debsums is helpful. Kind regards, Philipp Kern -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/slrnhos7oj.qn5.tr...@kelgar.0x539.de
Re: md5sums files
On Wed, Mar 03, 2010 at 08:29:09AM +, Neil Williams wrote:
On Wed, 3 Mar 2010 08:35:18 +0100
Mike Hommey m...@glandium.org wrote:
On Wed, Mar 03, 2010 at 06:30:34AM +, Sune Vuorela wrote:
The md5 sums isn't to be used in case of a break in, as you can't trust
anything on the system anyways, but more things like:
- did I make; sudo make install something on top of packages
- did I just quickly hack a p{erl,ython}-script on the system to do
something different and forgot
Which makes me think... wouldn't it be nice from dpkg to check the
package files haven't been modified before upgrading ?
No - if you're upgrading, you're doing it because you want to be sure
that the Debian version replaces the old version. However, if you do
use 'make; sudo make install; with a prefix of /usr instead
of /usr/local then a) you're taking a risk and b) md5sums are only a
partial protection. md5sums will NOT catch instances where the upstream
'make install' provides files that are not part of the Debian package
nor will it catch files that have been moved as part of Debian
packaging. As these files are not put into place by dpkg anyway, then
they won't get cleaned up by dpkg and cannot be detected either by
using md5sums or by using dpkg. (dpkg will complain that certain
directories are not empty if the packaged files being upgraded are in
special places but not otherwise.) There's every chance that having
extraneous and/or duplicate content in the wrong places will break the
Debian package in ways that are not easily detectable and md5sums won't
help with that.
Having md5sums around for simple checks like local admins copying
modified scripts over packaged versions is one thing but IMHO it does
not justify having md5sums in the wider case because a) local admin
changes are the responsibility of the local admin and b) md5sums only
help detect those changes where the admin changes a filename that
exactly matches the packaged name rather than adding more content /
cruft.
'sudo make install' into /usr is not something that md5sums can help us
fix and if that is the sole justification for retaining them then I
think that is a false argument.
My point is that these people doing that, or even better, people doing
an upgrade on a system where other people have been doing that would
probably *want* to have a warning about the fact that the files dpkg is
going to replace did *not* match what was supposed to be there in the
first place.
That would also possibly help spot filesystem errors, because when
upgrading, dpkg is not going to write to the same places where the
previous versions of the files were, and when it finishes upgrading, it
will just remove the previous files and the corruptions, especially if
they are due to hardware problems, will still be unnoticed and may
affect more important data much later, when the freed space is used
again.
Mike
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100303084209.ga20...@glandium.org
Re: md5sums files
Wouter Verhelst, 2010-03-03 03:06:20 +0100 : [...] In this day and age of completely and utterly broken MD5[0], I think we should stop providing these files, and maybe provide something else instead. Like, I dunno, shasums? Or perhaps gpgsigs? But stop providing md5sums. Or is it useful to be able to say if it doesn't check out, it's certainly corrupt, and if it does check out, it may be corrupt? Didn't think so. It is useful. Not too efficient against attacks, I'll grant you, but there are other use cases. One of those I run into from time to time, as maintainer of servers with webapps, is that I want to know from time to time if there have been local changes in installed files when compared to the (already custom) packages, so I can have a look and integrate the changes into the next version of the custom packages. The suggestion of having dpkg warn me on upgrade is interesting, but to be proactive I'm happy to run debsums on my own before the deployment stage. Because when you're in a deployment rush and one of the files lacks a semicolon and breaks the whole app, you just $EDITOR /usr/share/.../foo.php; some other times, your client messes with their CSS files locally, and you don't want to be grumbled at if you lose that change, even if you (and the client) know that the change *should* have been done in the packages in the first place. I'm quite okay with replacing or complementing the md5sums with something stronger if it helps with security, but removing them altogether… please no. Roland. -- Roland Mas La menace de la baffe pèse plus lourd que la baffe elle-même. -- in Sri Raoul le petit yogi (Gaudelette) -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87iq9du42q@mirexpress.internal.placard.fr.eu.org
Re: md5sums files
Erik de Castro Lopo er...@mega-nerd.com writes: Russ Allbery wrote: Wouter Verhelst wou...@debian.org writes: Or is it useful to be able to say if it doesn't check out, it's certainly corrupt, and if it does check out, it may be corrupt? Didn't think so. I don't understand why you say this. Cryptographic attacks on MD5 aren't going to happen as a result of random file corruption. The MD5 checksums are still very effective at finding file corruption or modification from what's in the Debian package unless that modification was done by a sophisticated attacker (MD5 preimage attacks are still not exactly easy). Detecting compromises is useful, but only a small part of what the MD5 checksums are useful for. If the machine has been compromised, *nothing* on the machine can be trusted, whether its gpg signed or not. However, for detecting corruptions Surely you are wrong. While you can not trust the gpg binary on the compromised system the signatures will still hold. You just need to boot from a live CD and use a garantied clean gpg and keyring to verify them. and the local sysadmin meddling Russ mentioned, md5sum is more than adequate and using something 'more secure' than md5sum is overkill. Due to the lack of a signature on the md5sum file the file can not be tusted fo security purposes at all. And for detecting unintentional changes md5sum is plenty strong enough. But imagine the file would be signed or a signature could be gotten through a safe channel. Then the file could be used for a security audit as well and something stronger would be benefitial. A simple way would be create a shasum file (instead of md5) and to include a hash of said file in Packages.gz. Erik MfG Goswin -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87zl2plmca@frosties.localdomain
Re: md5sums files
Mike Hommey m...@glandium.org writes:
On Wed, Mar 03, 2010 at 08:29:09AM +, Neil Williams wrote:
On Wed, 3 Mar 2010 08:35:18 +0100
Mike Hommey m...@glandium.org wrote:
On Wed, Mar 03, 2010 at 06:30:34AM +, Sune Vuorela wrote:
The md5 sums isn't to be used in case of a break in, as you can't trust
anything on the system anyways, but more things like:
- did I make; sudo make install something on top of packages
- did I just quickly hack a p{erl,ython}-script on the system to do
something different and forgot
Which makes me think... wouldn't it be nice from dpkg to check the
package files haven't been modified before upgrading ?
No - if you're upgrading, you're doing it because you want to be sure
that the Debian version replaces the old version. However, if you do
use 'make; sudo make install; with a prefix of /usr instead
of /usr/local then a) you're taking a risk and b) md5sums are only a
partial protection. md5sums will NOT catch instances where the upstream
'make install' provides files that are not part of the Debian package
nor will it catch files that have been moved as part of Debian
packaging. As these files are not put into place by dpkg anyway, then
they won't get cleaned up by dpkg and cannot be detected either by
using md5sums or by using dpkg. (dpkg will complain that certain
directories are not empty if the packaged files being upgraded are in
special places but not otherwise.) There's every chance that having
extraneous and/or duplicate content in the wrong places will break the
Debian package in ways that are not easily detectable and md5sums won't
help with that.
Having md5sums around for simple checks like local admins copying
modified scripts over packaged versions is one thing but IMHO it does
not justify having md5sums in the wider case because a) local admin
changes are the responsibility of the local admin and b) md5sums only
help detect those changes where the admin changes a filename that
exactly matches the packaged name rather than adding more content /
cruft.
'sudo make install' into /usr is not something that md5sums can help us
fix and if that is the sole justification for retaining them then I
think that is a false argument.
My point is that these people doing that, or even better, people doing
an upgrade on a system where other people have been doing that would
probably *want* to have a warning about the fact that the files dpkg is
going to replace did *not* match what was supposed to be there in the
first place.
That would also possibly help spot filesystem errors, because when
upgrading, dpkg is not going to write to the same places where the
previous versions of the files were, and when it finishes upgrading, it
will just remove the previous files and the corruptions, especially if
they are due to hardware problems, will still be unnoticed and may
affect more important data much later, when the freed space is used
again.
Mike
Run a nightly/weekly cron job that verifies all files. No point waiting
for a package upgrade to check this.
MfG
Goswin
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87vdddlm95@frosties.localdomain
Re: md5sums files
On Wed, Mar 03, 2010 at 03:17:52PM +1100, Erik de Castro Lopo wrote: Russ Allbery wrote: Wouter Verhelst wou...@debian.org writes: Or is it useful to be able to say if it doesn't check out, it's certainly corrupt, and if it does check out, it may be corrupt? Didn't think so. I don't understand why you say this. Cryptographic attacks on MD5 aren't going to happen as a result of random file corruption. The MD5 checksums are still very effective at finding file corruption or modification from what's in the Debian package unless that modification was done by a sophisticated attacker (MD5 preimage attacks are still not exactly easy). Detecting compromises is useful, but only a small part of what the MD5 checksums are useful for. If the machine has been compromised, *nothing* on the machine can be trusted, whether its gpg signed or not. However, for detecting corruptions and the local sysadmin meddling Russ mentioned, md5sum is more than adequate Sure, I'm not contesting that. and using something 'more secure' than md5sum is overkill. This is where I disagree. When a checksum algorithm is compromised (and MD5 *is* compromised), things only ever get worse, not better. Indeed, MD5 preimage attacks are pretty hard *today*. But switching to something more secure in preparation for the day when MD5 will be easily cracked by every script kiddo around is *not* overkill. -- The biometric identification system at the gates of the CIA headquarters works because there's a guard with a large gun making sure no one is trying to fool the system. http://www.schneier.com/blog/archives/2009/01/biometrics.html -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100303104725.ga18...@celtic.nixsys.be
Re: md5sums files
On 2010-03-03, Wouter Verhelst wou...@debian.org wrote: This is where I disagree. When a checksum algorithm is compromised (and MD5 *is* compromised), things only ever get worse, not better. Indeed, MD5 preimage attacks are pretty hard *today*. But switching to something more secure in preparation for the day when MD5 will be easily cracked by every script kiddo around is *not* overkill. Sure, but to be honest, not even all packages managed to generate md5sums 'till now (with some quite core, omnipresent packages missing) so it seems out of scope for squeeze. Maybe squeeze+1. Kind regards, Philipp Kern -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/slrnhosifd.rmi.tr...@kelgar.0x539.de
Re: md5sums files
On Wed, Mar 03, 2010 at 03:06:20AM +0100, Wouter Verhelst wrote: In this day and age of completely and utterly broken MD5[0], I think we should stop providing these files, and maybe provide something else instead. Like, I dunno, shasums? Or perhaps gpgsigs? But stop providing md5sums. Or is it useful to be able to say if it doesn't check out, it's certainly corrupt, and if it does check out, it may be corrupt? Didn't think so. As a means to check for filesystem corruptions or non-malicious changes, MD5 is good enough. So until we have something better, I guess they can stay. But it would be great if the whole chain, from beginning to end, was secured, even against a malicious and presumably very powerful attackers. That would need: * Package signatures Currently only the release file is signed, but if you have a package lying around, there is no way to check its authenticity. * Cryptographically strong hashes for all files in the package and a signature on the hash file. Then you could really check the authenticity of all files on the system. For the hash I would skip SHA-1 and move directly to SHA-256. Oh, and a good read about the lifetime of hash algorithms can be found here: [0] Cheers, harry [0] http://valerieaurora.org/hash.html -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100303133905.gb11...@nn.nn
Re: md5sums files
On Wed, Mar 03, 2010 at 02:39:05PM +0100, Harald Braumann wrote: On Wed, Mar 03, 2010 at 03:06:20AM +0100, Wouter Verhelst wrote: In this day and age of completely and utterly broken MD5[0], I think we should stop providing these files, and maybe provide something else instead. Like, I dunno, shasums? Or perhaps gpgsigs? But stop providing md5sums. Or is it useful to be able to say if it doesn't check out, it's certainly corrupt, and if it does check out, it may be corrupt? Didn't think so. As a means to check for filesystem corruptions or non-malicious changes, MD5 is good enough. So until we have something better, I guess they can stay. But it would be great if the whole chain, from beginning to end, was secured, even against a malicious and presumably very powerful attackers. That would need: * Package signatures Currently only the release file is signed, but if you have a package lying around, there is no way to check its authenticity. * Cryptographically strong hashes for all files in the package and a signature on the hash file. Then you could really check the authenticity of all files on the system. For the hash I would skip SHA-1 and move directly to SHA-256. * A way to easily create a bootable device (usb, cd, whatever) that will check everything is in order. Extra points if that is part of the rescue images on the install CDs. Mike -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100303135212.ga13...@glandium.org
Re: md5sums files
* Harald Braumann ha...@unheit.net [100303 14:49]: As a means to check for filesystem corruptions or non-malicious changes, MD5 is good enough. So until we have something better, I guess they can stay. I'd even say they should stay. The shorter the hash the more useable. And md5 is the shortest well-defined and readily available everywhere. But it would be great if the whole chain, from beginning to end, was secured, even against a malicious and presumably very powerful attackers. Checksums for files coming from packages is not really useful to defend against attackers (it's really only reliablity and not security): - an attacker can just divert any binary away and put it's own there. - an attacker can just add some additional binary where it will override another one (/sbin overriding /usr/sbin and so on). - an attacker can add things to configuration and startup files (thanks to .d directories you often not even need changing but only adding files), including search binary or library paths, so one could add binaries or behaviour changing libraries in directories not looking that suspicious. Most of those things can perhaps be fixed, but it needs much work than just replacing some hash. (And many of those tasks might also improve other areas (like http://packages.debian.org/cruft also having the problem that packages create so many files and there is no way a package can tell such programs where they are). For the hash I would skip SHA-1 and move directly to SHA-256. And not to forget to add the size. Being allowed to create arbitrary large files makes attacking interated hashes much easier... Hochachtungsvoll, Bernhard R. Link -- Never contain programs so few bugs, as when no debugging tools are available! Niklaus Wirth -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100303141608.ga21...@pcpool00.mathematik.uni-freiburg.de
Re: md5sums files
On Wed, Mar 03, 2010, Wouter Verhelst wrote: In this day and age of completely and utterly broken MD5[0], I think we should stop providing these files, and maybe provide something else instead. Like, I dunno, shasums? Or perhaps gpgsigs? But stop providing md5sums. Or is it useful to be able to say if it doesn't check out, it's certainly corrupt, and if it does check out, it may be corrupt? Didn't think so. I use them regularly after laptop crashes; I found some misinstalled packages a couple of times because a crash would happen just after an upgrade. -- Loïc Minier -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100303124010.gb7...@bee.dooz.org
Re: md5sums files
On Wed, Mar 03, 2010 at 03:16:08PM +0100, Bernhard R. Link wrote: * Harald Braumann ha...@unheit.net [100303 14:49]: But it would be great if the whole chain, from beginning to end, was secured, even against a malicious and presumably very powerful attackers. Checksums for files coming from packages is not really useful to defend against attackers (it's really only reliablity and not security): - an attacker can just divert any binary away and put it's own there. It's not about preventing an attack, but detecting it. With cryptographically strong hashes/signatures in place, you can audit the system. Of course you'd have to boot from a trusted medium. How would you do that without signatures? - an attacker can just add some additional binary where it will override another one (/sbin overriding /usr/sbin and so on). - an attacker can add things to configuration and startup files (thanks to .d directories you often not even need changing but only adding files), including search binary or library paths, so one could add binaries or behaviour changing libraries in directories not looking that suspicious. Yes, a full IDS needs additional work. It would have to check for files without hashes/signatures and would have to allow you to hash and sign files in /etc, /usr/local, /opt, whatever). Most of those things can perhaps be fixed, but it needs much work than just replacing some hash. (And many of those tasks might also improve other areas (like http://packages.debian.org/cruft also having the problem that packages create so many files and there is no way a package can tell such programs where they are). -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100303150337.gc11...@nn.nn
Re: md5sums files
[Wouter Verhelst] I must say I was somewhat surprised by these numbers. Out of 2483 packages installed on my laptop, 2340 install md5sums. The surprising part, perhaps, is that dpkg itself didn't just generate the other 143 md5sums files at installation time. I suggested this a long time ago and of course was met with so where's your patch? Of course I was not willing to do the work. But fundamentally, shipping a md5sums file is really just a tradeoff in download size vs. installation speed, not unlike gzip vs. bzip2. One can imagine in the future, on architectures where CPU is little consideration, letting those md5sums files all be generated at install time. -- Peter Samuelson | org-tld!p12n!peter | http://p12n.org/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100303160510.gf18...@p12n.org
Re: md5sums files
On Wed, Mar 03, 2010 at 11:37:17AM +, Philipp Kern wrote: On 2010-03-03, Wouter Verhelst wou...@debian.org wrote: This is where I disagree. When a checksum algorithm is compromised (and MD5 *is* compromised), things only ever get worse, not better. Indeed, MD5 preimage attacks are pretty hard *today*. But switching to something more secure in preparation for the day when MD5 will be easily cracked by every script kiddo around is *not* overkill. Sure, but to be honest, not even all packages managed to generate md5sums 'till now (with some quite core, omnipresent packages missing) so it seems out of scope for squeeze. Maybe squeeze+1. True but debsums can address these issues by system administrator touch-ups as documented in manpage using: * /etc/apt/apt.conf.d/90debsums (debsums = 2.0.7) * debsums_init(8) (debsums = 2.0.34 @ 2007) Osamu -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100303155453.ga5...@osamu.debian.net
Re: md5sums files
Hi, On Wed, Mar 03, 2010 at 03:06:20AM +0100, Wouter Verhelst wrote: Hello world, wou...@celtic:/var/lib/dpkg/info$ ls *md5sums|wc -l 2340 wou...@celtic:/var/lib/dpkg/info$ ls *sums|wc -l 2340 wou...@celtic:/var/lib/dpkg/info$ dpkg -l|sed -e'1,/=/d'|wc -l 2483 Here on my system: $ dpkg -l|sed -e'1,/=/d'|wc -l 2302 $ dpkg -l|sed -e'1,/=/d'|grep ^ii |wc -l 2301 $ dpkg -l|sed -e'1,/=/d'|grep -v ^ii rc sbcl 1:1.0.34.0-1 A Common Lisp compiler and development system So dfference can be explained. I must say I was somewhat surprised by these numbers. Out of 2483 packages installed on my laptop, 2340 install md5sums. While that might've been useful at some point, I don't think it still is. Are you sure you hava all package lines starting with ii? (I know some package may still be lacking *md5sums under some configuration. If so, I suggest to read debsums and debsums_init manpages. This issue is solved since 2007.) In this day and age of completely and utterly broken MD5[0], I think we should stop providing these files, and maybe provide something else instead. Like, I dunno, shasums? Or perhaps gpgsigs? But stop providing md5sums. gpg is slow. sha variants will be nice if there is smooth transition in place properly planned and supprted with backported package of debsums. The advantage of debsums is precalculated sum values and quick sanity check capability against random changes. Osamu -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100303161226.gb5...@osamu.debian.net
Re: md5sums files
On Thu, Mar 04, 2010 at 01:12:26AM +0900, Osamu Aoki wrote: In this day and age of completely and utterly broken MD5[0], I think we should stop providing these files, and maybe provide something else instead. Like, I dunno, shasums? Or perhaps gpgsigs? But stop providing md5sums. gpg is slow. sha variants will be nice if there is smooth transition in place properly planned and supprted with backported package of debsums. You wouldn't sign each file, just the hash sums file. harry -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100303175954.gd11...@nn.nn
Re: md5sums files
Philipp Kern wrote: On 2010-03-03, Wouter Verhelst wou...@debian.org wrote: This is where I disagree. When a checksum algorithm is compromised (and MD5 *is* compromised), things only ever get worse, not better. Indeed, MD5 preimage attacks are pretty hard *today*. But switching to something more secure in preparation for the day when MD5 will be easily cracked by every script kiddo around is *not* overkill. Sure, but to be honest, not even all packages managed to generate md5sums 'till now (with some quite core, omnipresent packages missing) so it seems out of scope for squeeze. Maybe squeeze+1. I think its about time to require to generate checksums for packages and make all packages which do not do so RC buggy. -- Bernd ZeimetzDebian GNU/Linux Developer http://bzed.dehttp://www.debian.org GPG Fingerprints: 06C8 C9A2 EAAD E37E 5B2C BE93 067A AD04 C93B FF79 ECA1 E3F2 8E11 2432 D485 DD95 EB36 171A 6FF9 435F -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4b8eb3b6.4070...@bzed.de
Re: md5sums files
On Wed, Mar 3, 2010 at 10:05:11 -0600, Peter Samuelson wrote: fundamentally, shipping a md5sums file is really just a tradeoff in download size vs. installation speed, not unlike gzip vs. bzip2. One Only if you assume that disks never fail and thus files never get corrupted when the package gets unpacked. Cheers, Julien signature.asc Description: Digital signature
Re: md5sums files
Osamu Aoki wrote: True but debsums can address these issues by system administrator touch-ups as documented in manpage using: * /etc/apt/apt.conf.d/90debsums (debsums = 2.0.7) * debsums_init(8) (debsums = 2.0.34 @ 2007) It's not uncommon to be given an existing system and want to verify that no files were modified by its creator, and the current lack of md5sums files for a few packages prevents using debsums to do that. (The most common case is probably buying a Debian VM from a hosting company.) -- see shy jo signature.asc Description: Digital signature
Re: md5sums files
[Julien Cristau] fundamentally, shipping a md5sums file is really just a tradeoff in download size vs. installation speed, not unlike gzip vs. bzip2. One Only if you assume that disks never fail and thus files never get corrupted when the package gets unpacked. Given a .deb, turning the data.tar.gz into foo.md5sums is a SMOP. This could be before, during, or after the deb is unpacked. Using the packaged foo.md5sums as an internal consistency check of data.tar.gz itself is interesting, but somewhat unwieldy. Better would be to checksum data.tar.gz in its entirety. But doesn't gzip already do that? (Yes, it's only 32 bits, but we aren't trying to detect intentional tampering, only corruption. To detect intentional tampering, you need signed debs, or at least signed Packages.bz2.) -- Peter Samuelson | org-tld!p12n!peter | http://p12n.org/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100303200201.gg18...@p12n.org
Re: md5sums files
On Wed, Mar 03, 2010 at 02:02:01PM -0600, Peter Samuelson wrote: [Julien Cristau] fundamentally, shipping a md5sums file is really just a tradeoff in download size vs. installation speed, not unlike gzip vs. bzip2. One Only if you assume that disks never fail and thus files never get corrupted when the package gets unpacked. Given a .deb, turning the data.tar.gz into foo.md5sums is a SMOP. This could be before, during, or after the deb is unpacked. If you create the hashes at unpack time, you don't catch errors that happen during unpack. Why not create them at package build time and include them? I mean we are talking bytes here for the hashes, not megabytes. I can't see a download size problem. Using the packaged foo.md5sums as an internal consistency check of data.tar.gz itself is interesting, but somewhat unwieldy. Better would be to checksum data.tar.gz in its entirety. But doesn't gzip already do that? (Yes, it's only 32 bits, but we aren't trying to detect intentional tampering, only corruption. The hash must include the whole package, not just data.tar.gz. To detect intentional tampering, you need signed debs, Yes, I think that should be the goal. or at least signed Packages.bz2.) You already have this, kind of: the Release file is signed and it contains the hash of the Packages file, wich in turn contains the hashes for the packages. The problem here is, that you can only check packages, that are currently part in some release. If you have an old version lying around or a package sent to you by someone, you're out of luck. The package signature should be part of the package. harry -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100303204038.ge11...@nn.nn
Re: md5sums files
On Tue, 2010-03-02 at 18:21 -0800, Russ Allbery wrote: Wouter Verhelst wou...@debian.org writes: Or is it useful to be able to say if it doesn't check out, it's certainly corrupt, and if it does check out, it may be corrupt? Didn't think so. I don't understand why you say this. Cryptographic attacks on MD5 aren't going to happen as a result of random file corruption. The MD5 checksums are still very effective at finding file corruption or modification from what's in the Debian package unless that modification was done by a sophisticated attacker (MD5 preimage attacks are still not exactly easy). Detecting compromises is useful, but only a small part of what the MD5 checksums are useful for. I'd more frequently use them to detect well-intentioned but misguided meddling by a local sysadmin. I certainly don't object to replacing them with SHA1 hashes, although signed deb packages would still be my preferred solution to this problem. Signed debs may introduce a fake sense of security (Only apt repository provide security updates). By signing packages, user may assume that a package is safe when it isn't. Debian is 15/20 years ahead of commercial operating system on that point. Franklin -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/1267649891.8266.233.ca...@solid.paris.klabs.be
Re: md5sums files
On Wed, 2010-03-03 at 11:37 +, Philipp Kern wrote: On 2010-03-03, Wouter Verhelst wou...@debian.org wrote: This is where I disagree. When a checksum algorithm is compromised (and MD5 *is* compromised), things only ever get worse, not better. Indeed, MD5 preimage attacks are pretty hard *today*. But switching to something more secure in preparation for the day when MD5 will be easily cracked by every script kiddo around is *not* overkill. Sure, but to be honest, not even all packages managed to generate md5sums 'till now (with some quite core, omnipresent packages missing) so it seems out of scope for squeeze. Maybe squeeze+1. What about a transitional dh_md5sums that would produce md5sum AND invoke dh_sha ? Franklin -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/1267650344.8266.262.ca...@solid.paris.klabs.be
Re: md5sums files
On Wed, 03 Mar 2010 21:58:11 +0100, Frank Lin PIAT wrote: On Tue, 2010-03-02 at 18:21 -0800, Russ Allbery wrote: Wouter Verhelst wou...@debian.org writes: Or is it useful to be able to say if it doesn't check out, it's certainly corrupt, and if it does check out, it may be corrupt? Didn't think so. I don't understand why you say this. Cryptographic attacks on MD5 aren't going to happen as a result of random file corruption. The MD5 checksums are still very effective at finding file corruption or modification from what's in the Debian package unless that modification was done by a sophisticated attacker (MD5 preimage attacks are still not exactly easy). Detecting compromises is useful, but only a small part of what the MD5 checksums are useful for. I'd more frequently use them to detect well-intentioned but misguided meddling by a local sysadmin. I certainly don't object to replacing them with SHA1 hashes, although signed deb packages would still be my preferred solution to this problem. Signed debs may introduce a fake sense of security (Only apt repository provide security updates). By signing packages, user may assume that a package is safe when it isn't. it should actually be possible to do this securely. dpkg could be made to work like apt where it only blindly trusts packages signed by keys in /etc/apt/trusted.gpg. the downfall is that there is nothing stopping the user from adding additional (potentially less than trustworthy keys), but that isn't really solvable without destroying freedom, and it isn't any different from the current state for apt. mike -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100303162036.dbf12e05.michael.s.gilb...@gmail.com
Re: md5sums files
On Wed, Mar 03, 2010 at 08:08:38PM +0100, Bernd Zeimetz wrote:
I think its about time to require to generate checksums for packages
and make all packages which do not do so RC buggy.
Well, RC buggy is probably a tad excessive for this release, considering
that we are (I hope :)) close to a freeze. But I agree with the
principle, actually I've been surprised *positively* by the numbers
reported by Wouter and by those I've found on my laptop:
z...@usha:/var/lib/dpkg/info$ dpkg -l |grep ^ii |wc -l
2629
z...@usha:/var/lib/dpkg/info$ ls *md5sums|wc -l
2603
z...@usha:/var/lib/dpkg/info$
Almost all packages (I've installed) ship md5sums, while the situation
was almost reverted a few releases ago.
I'm curious about what contributed to this positive change (dh7 and/or
CDBS invoking dh_md5sums by default?), any idea?
Cheers.
--
Stefano Zacchiroli -o- PhD in Computer Science \ PostDoc @ Univ. Paris 7
z...@{upsilon.cc,pps.jussieu.fr,debian.org} -- http://upsilon.cc/zack/
Dietro un grande uomo c'è ..| . |. Et ne m'en veux pas si je te tutoie
sempre uno zaino ...| ..: | Je dis tu à tous ceux que j'aime
signature.asc
Description: Digital signature
Re: md5sums files
On Wed, Mar 03, 2010 at 08:34:27AM +, Philipp Kern wrote: On 2010-03-03, Neil Williams codeh...@debian.org wrote: Changing to SHA won't help. I'm for ditching all md5sums from packages. It's not a lot of disc space gained but it does give a false sense of security or 'insurance' if you want to avoid the more formal meaning of 'security'. Please don't. It's not about security. It's about being able to detect corruption. Also it is very helpful when recovering from ext4 root FS corruption after a sudden power loss. Sure, you cannot guarantee that the md5 store isn't corrupted too but if it isn't then debsums is helpful. Very much agreed. Please do not remove the md5sums - even better, I'm all for requiring md5sums (the cost to do so is, I think, insignificant) because they are very helpful for the above purpose. iustin -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100303212646.gb9...@teal.hq.k1024.org
Re: md5sums files
Stefano Zacchiroli z...@debian.org writes: I'm curious about what contributed to this positive change (dh7 and/or CDBS invoking dh_md5sums by default?), any idea? We added a Lintian tag warning about not having md5sums in that time frame, I think. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87tysxt6p3@windlord.stanford.edu
Re: md5sums files
On Wed, Mar 03, 2010 at 04:20:36PM -0500, Michael Gilbert wrote:
On Wed, 03 Mar 2010 21:58:11 +0100, Frank Lin PIAT wrote:
Signed debs may introduce a fake sense of security (Only apt repository
provide security updates). By signing packages, user may assume that a
package is safe when it isn't.
it should actually be possible to do this securely. dpkg could be
made to work like apt where it only blindly trusts packages signed
by keys in /etc/apt/trusted.gpg. the downfall is that there is nothing
stopping the user from adding additional (potentially less than
trustworthy keys), but that isn't really solvable without destroying
freedom, and it isn't any different from the current state for apt.
Completely agreed. Also, because playing around is always more fun than
just talking, I've attached a script that signs/verifies binary
packages. Dpkg doesn't seem to mind the extra files.
This script signs each file in the package individually, but it could
also concatenate them all alphabetically and create just one signature.
Cheers,
harry
#!/bin/sh
usage() {
catEOF
Usage: debsign -s|-v debfile
Sign or verify Debian packages
-s sign
-v verify
EOF
}
sign() {
echo signing ${DEB}:${FILE}
ar p ${DEB} ${FILE} | gpg --detach-sign --output ${SIG} - \
ar r ${DEB} ${SIG}
}
verify() {
echo verifying signature of ${DEB}:${FILE}
ar p ${DEB} ${FILE}.sig ${SIG} \
ar p ${DEB} ${FILE} | gpg --verify ${SIG} -
}
[ $# -eq 2 ] || { usage 2; exit 1; }
DEB=$2
case $1 in
-s) OP=sign;;
-v) OP=verify;;
*) usage 2; exit 1;;
esac
[ -f ${DEB} ] || { printf %s\n ${DEB} not found 2; exit 1; }
TMPDIR=`mktemp -d --tmpdir debsign.XX`
ar t ${DEB} | while read FILE; do
[ ${FILE##*.} != sig ] || continue
SIG=${TMPDIR}/${FILE}.sig
${OP} || exit 1
done
RC=$?
rm ${TMPDIR}/* 2/dev/null
rmdir ${TMPDIR} 2/dev/null
if [ ${RC} -eq 0 ]; then
echo OK
else
echo Failed
fi
return ${RC}
Re: md5sums files
Harald Braumann ha...@unheit.net writes: Completely agreed. Also, because playing around is always more fun than just talking, I've attached a script that signs/verifies binary packages. Dpkg doesn't seem to mind the extra files. This script signs each file in the package individually, but it could also concatenate them all alphabetically and create just one signature. See debsigs. There have been previous discussions on debian-devel about this. I believe DAK does not allow packages signed using debsigs to be uploaded. I'm not sure if that's out of objection to the entire concept, or whether there are just technical issues that need to be resolved first. (I probably would know if I had a better memory for the previous discussion, but unfortunately I appear to have recycled those brain cells.) -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87tysxhubn@windlord.stanford.edu
Re: md5sums files
[Harald Braumann] Given a .deb, turning the data.tar.gz into foo.md5sums is a SMOP. This could be before, during, or after the deb is unpacked. If you create the hashes at unpack time, you don't catch errors that happen during unpack. You mean errors reading the data.tar.gz file? That is what the gzip checksum is for, as I said later in my email. Using the packaged foo.md5sums as an internal consistency check of data.tar.gz itself is interesting, but somewhat unwieldy. Better would be to checksum data.tar.gz in its entirety. But doesn't gzip already do that? (Yes, it's only 32 bits, but we aren't trying to detect intentional tampering, only corruption. The hash must include the whole package, not just data.tar.gz. Please don't hijack my little thread here. I'm talking about whether there are any disadvantages to generating foo.md5sums at unpack time, other than the obvious one (CPU usage). Signed debs and signed repositories are an entirely separate discussion. md5sums files don't even pretend to solve the same problems. They solve other problems. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100303234126.gh18...@p12n.org
Re: md5sums files
On Wed, Mar 03, 2010 at 03:14:04PM -0800, Russ Allbery wrote: Harald Braumann ha...@unheit.net writes: Completely agreed. Also, because playing around is always more fun than just talking, I've attached a script that signs/verifies binary packages. Dpkg doesn't seem to mind the extra files. This script signs each file in the package individually, but it could also concatenate them all alphabetically and create just one signature. See debsigs. Ah, thanks, good to know. There have been previous discussions on debian-devel about this. I believe DAK does not allow packages signed using debsigs to be uploaded. I'm not sure if that's out of objection to the entire concept, or whether there are just technical issues that need to be resolved first. (I probably would know if I had a better memory for the previous discussion, but unfortunately I appear to have recycled those brain cells.) Maybe this [0]? Also, it seems, that people have `discussed' it, and that there was experimental support in dpkg for it [1]. The proposal, that came out of this discussion is outlined in [2]. This was in 2000 ... I haven't found any specific reasons why it was never implemented. I guess the reason is just that it's hard to do. Not the technical side, but defining the processes. harry [0] http://lists.debian.org/debian-devel/2002/03/msg01484.html [1] http://lists.debian.org/debian-dpkg/2000/07/msg1.html [2] http://lists.debian.org/debian-dpkg/2000/07/msg00044.html -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100304000655.ga16...@nn.nn
Re: md5sums files
On Wed, Mar 03, 2010 at 05:41:26PM -0600, Peter Samuelson wrote: [Harald Braumann] Given a .deb, turning the data.tar.gz into foo.md5sums is a SMOP. This could be before, during, or after the deb is unpacked. If you create the hashes at unpack time, you don't catch errors that happen during unpack. You mean errors reading the data.tar.gz file? That is what the gzip checksum is for, as I said later in my email. Errors writing a file. If there should be support in the future for signing hash files, then creating them would have to be done at package creation time anyway. Also, I think, that it is in general better to have as much complexity as possible in the package builder and make the client tools as dumb as possible. harry -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100304002053.gb16...@nn.nn
Re: md5sums files
[Harald Braumann] On Wed, Mar 03, 2010 at 05:41:26PM -0600, Peter Samuelson wrote: [Harald Braumann] Given a .deb, turning the data.tar.gz into foo.md5sums is a SMOP. This could be before, during, or after the deb is unpacked. If you create the hashes at unpack time, you don't catch errors that happen during unpack. You mean errors reading the data.tar.gz file? That is what the gzip checksum is for, as I said later in my email. Errors writing a file. Did you read the text you quoted? Given a .deb, turning the data.tar.gz into foo.md5sums is a SMOP. (By SMOP, I meant simple matter of programming.) This has nothing to do with writing a file, except writing /var/lib/dpkg/info/foo.md5sums, which is unavoidable. Peter -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100304005028.gj18...@p12n.org
Re: md5sums files
On Wed, Mar 03, 2010 at 06:50:28PM -0600, Peter Samuelson wrote: [Harald Braumann] On Wed, Mar 03, 2010 at 05:41:26PM -0600, Peter Samuelson wrote: [Harald Braumann] Given a .deb, turning the data.tar.gz into foo.md5sums is a SMOP. This could be before, during, or after the deb is unpacked. If you create the hashes at unpack time, you don't catch errors that happen during unpack. You mean errors reading the data.tar.gz file? That is what the gzip checksum is for, as I said later in my email. Errors writing a file. Did you read the text you quoted? Given a .deb, turning the data.tar.gz into foo.md5sums is a SMOP. (By SMOP, I meant simple matter of programming.) This has nothing to do with writing a file, except writing /var/lib/dpkg/info/foo.md5sums, which is unavoidable. I think I was finally able to decipher your message. But my other points still hold. And while it is just a matter of programming, simple or not, it already exists in debhelper. So doing it at build time is SMOAOLTDR, by which I mean simply a matter of adding one line to debian/rules. harry -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100304011155.gd16...@nn.nn
Re: md5sums files
On Thu, Mar 04, 2010 at 02:11:55AM +0100, Harald Braumann wrote: I think I was finally able to decipher your message. But my other points still hold. And while it is just a matter of programming, simple or not, it already exists in debhelper. So doing it at build time is SMOAOLTDR, by which I mean simply a matter of adding one line to debian/rules. You make the mistake of assuming everyone use debhelper. -- James GPG Key: 1024D/61326D40 2003-09-02 James Vega james...@debian.org signature.asc Description: Digital signature
Re: md5sums files
James Vega james...@debian.org writes: On Thu, Mar 04, 2010 at 02:11:55AM +0100, Harald Braumann wrote: I think I was finally able to decipher your message. But my other points still hold. And while it is just a matter of programming, simple or not, it already exists in debhelper. So doing it at build time is SMOAOLTDR, by which I mean simply a matter of adding one line to debian/rules. You make the mistake of assuming everyone use debhelper. Yeah, but: @set -ex; \ cd debian/tmp; \ find . -path ./DEBIAN -prune -o -type f -printf '%P\0' \ | xargs -r0 md5sum DEBIAN/md5sums (from debian-policy's rules file). dh_md5sums isn't particularly complicated, although admittedly it deals with a few more cases for you, such as multiple packages. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/871vg0g6ru@windlord.stanford.edu
md5sums files
Hello world, wou...@celtic:/var/lib/dpkg/info$ ls *md5sums|wc -l 2340 wou...@celtic:/var/lib/dpkg/info$ ls *sums|wc -l 2340 wou...@celtic:/var/lib/dpkg/info$ dpkg -l|sed -e'1,/=/d'|wc -l 2483 I must say I was somewhat surprised by these numbers. Out of 2483 packages installed on my laptop, 2340 install md5sums. While that might've been useful at some point, I don't think it still is. In this day and age of completely and utterly broken MD5[0], I think we should stop providing these files, and maybe provide something else instead. Like, I dunno, shasums? Or perhaps gpgsigs? But stop providing md5sums. Or is it useful to be able to say if it doesn't check out, it's certainly corrupt, and if it does check out, it may be corrupt? Didn't think so. Thoughts? [0] No reference. It's all over the internet. If you didn't know about MD5 being broken yet, where have you been sleeping these past few years? -- The biometric identification system at the gates of the CIA headquarters works because there's a guard with a large gun making sure no one is trying to fool the system. http://www.schneier.com/blog/archives/2009/01/biometrics.html signature.asc Description: Digital signature
Re: md5sums files
Wouter Verhelst wou...@debian.org writes: Or is it useful to be able to say if it doesn't check out, it's certainly corrupt, and if it does check out, it may be corrupt? Didn't think so. I don't understand why you say this. Cryptographic attacks on MD5 aren't going to happen as a result of random file corruption. The MD5 checksums are still very effective at finding file corruption or modification from what's in the Debian package unless that modification was done by a sophisticated attacker (MD5 preimage attacks are still not exactly easy). Detecting compromises is useful, but only a small part of what the MD5 checksums are useful for. I'd more frequently use them to detect well-intentioned but misguided meddling by a local sysadmin. I certainly don't object to replacing them with SHA1 hashes, although signed deb packages would still be my preferred solution to this problem. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/874okyuov4@windlord.stanford.edu
Re: md5sums files
Russ Allbery wrote: Wouter Verhelst wou...@debian.org writes: Or is it useful to be able to say if it doesn't check out, it's certainly corrupt, and if it does check out, it may be corrupt? Didn't think so. I don't understand why you say this. Cryptographic attacks on MD5 aren't going to happen as a result of random file corruption. The MD5 checksums are still very effective at finding file corruption or modification from what's in the Debian package unless that modification was done by a sophisticated attacker (MD5 preimage attacks are still not exactly easy). Detecting compromises is useful, but only a small part of what the MD5 checksums are useful for. If the machine has been compromised, *nothing* on the machine can be trusted, whether its gpg signed or not. However, for detecting corruptions and the local sysadmin meddling Russ mentioned, md5sum is more than adequate and using something 'more secure' than md5sum is overkill. Erik -- -- Erik de Castro Lopo http://www.mega-nerd.com/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100303151752.835b34d3.er...@mega-nerd.com
Re: md5sums files
Erik de Castro Lopo er...@mega-nerd.com writes: Russ Allbery wrote: I don't understand why you say this. Cryptographic attacks on MD5 aren't going to happen as a result of random file corruption. The MD5 checksums are still very effective at finding file corruption or modification from what's in the Debian package unless that modification was done by a sophisticated attacker (MD5 preimage attacks are still not exactly easy). Detecting compromises is useful, but only a small part of what the MD5 checksums are useful for. If the machine has been compromised, *nothing* on the machine can be trusted, whether its gpg signed or not. However, for detecting corruptions and the local sysadmin meddling Russ mentioned, md5sum is more than adequate and using something 'more secure' than md5sum is overkill. Also, while they're no substitute for a more thorough and careful check, it's always worth remembering that most attackers are stupid or lazy. Doing something simple frequently and something more thorough but resource-intensive at a longer interval is often a good move. That said, I would never recommend debsums as a security tool. It is, however, useful when figuring out if anyone's been helpfully fixing things on a system by editing scripts or replacing binaries that came in packages originally. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87d3zmroqi@windlord.stanford.edu
Re: md5sums files
On 2010-03-03, Wouter Verhelst wou...@debian.org wrote:
wou...@celtic:/var/lib/dpkg/info$ ls *md5sums|wc -l
2340
In this day and age of completely and utterly broken MD5[0], I think we
should stop providing these files, and maybe provide something else
instead. Like, I dunno, shasums? Or perhaps gpgsigs? But stop providing
md5sums.
Or is it useful to be able to say if it doesn't check out, it's
certainly corrupt, and if it does check out, it may be corrupt? Didn't
think so.
Hi
Even crc32 or md4 would be good enough for this. Probably even counting
'1 bits' in the files would be sufficient.
The md5 sums isn't to be used in case of a break in, as you can't trust
anything on the system anyways, but more things like:
- did I make; sudo make install something on top of packages
- did I just quickly hack a p{erl,ython}-script on the system to do
something different and forgot
- after a large fsck, which system files is actually fixed
- ...
And none of this creates md5 collisions.
So md5 is a good choice for this.
/Sune
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/slrnhos0g9.nfa.nos...@sshway.ssh.pusling.com
Re: md5sums files
On Wed, Mar 03, 2010 at 06:30:34AM +, Sune Vuorela wrote:
On 2010-03-03, Wouter Verhelst wou...@debian.org wrote:
wou...@celtic:/var/lib/dpkg/info$ ls *md5sums|wc -l
2340
In this day and age of completely and utterly broken MD5[0], I think we
should stop providing these files, and maybe provide something else
instead. Like, I dunno, shasums? Or perhaps gpgsigs? But stop providing
md5sums.
Or is it useful to be able to say if it doesn't check out, it's
certainly corrupt, and if it does check out, it may be corrupt? Didn't
think so.
Hi
Even crc32 or md4 would be good enough for this. Probably even counting
'1 bits' in the files would be sufficient.
The md5 sums isn't to be used in case of a break in, as you can't trust
anything on the system anyways, but more things like:
- did I make; sudo make install something on top of packages
- did I just quickly hack a p{erl,ython}-script on the system to do
something different and forgot
Which makes me think... wouldn't it be nice from dpkg to check the
package files haven't been modified before upgrading ?
Mike
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100303073518.gb19...@glandium.org
