Bug#838779: Release.gpg not downloaded unless Release is deleted

[Michael Stapelberg]

Thanks for the detailed reply!

On Sat, Sep 24, 2016 at 11:32 PM, David Kalnischkies 
wrote:

>
> On Sat, Sep 24, 2016 at 09:04:48PM +0200, Michael Stapelberg wrote:
> > I tested this in Debian stable (with apt 1.0.9.8.3) and Debian
> > testing/experimental (with apt 1.3~rc4 and 1.3, respectively). In Debian
> > stable, this works as expected, but in Debian testing and experimental,
> > I run into the bug. See the bottom of this report for a full command
> > log.
>
> That is a bug, althrough it's about not storing the Release.gpg file if
> the Release file isn't stored at the same time – which it isn't if we
> already got the latest version in storage – not about not downloading it…
>
> Responsible seems to be apt-pkg/acquire-item.cc:1928 which checks if the
> Release file was an IMSHit and only if not stores both files. We should
> probably check here if the Release.gpg exists before skipping the
> storage. If noone beats me to it I will have a look at patch+test for
> next week… it gets a bit late to fiddle with security for today and I am
> not really around tomorrow.
>

Thanks! Let me know if there’s anything you’d like me to test.


>
>
> > When adding the http://dl.bintray.com/i3/i3-autobuild-ubuntu xenial
> > repository, the Release and main_binary-amd64_Packages.lz4 files are
> > downloaded and stored in /var/lib/apt/lists. Release.gpg is discarded
> > because the necessary public key to verify the signature is not yet
> > installed.
>
> Just for the record: lz4 files aren't downloaded. The file downloaded is
> likely xz or gz compressed and the current default is to uncompress, but
> docker images tend to be configured to store them compressed, which apt
> does by uncompressing the xz/gz and recompressing as lz4 on the fly.
> Not important for this report, but just for completeness.
>
>
> > After installing i3-autobuild-keyring, which adds the public key using
> > apt-key add, the signature can successfully be verified (apt-get update
> > no longer prints an error).
>
> Pretty please with cherry on top don't use 'apt-key add'. Just ship your
> keyring in /etc/apt/trusted.gpg.d/ avoiding the need for an installed
> gpg. That works for ages now and since recently it even shows warnings.
>

Done:
https://github.com/stapelberg/i3-autobuild-keyring/commit/d42e1af4406b23820751dd66fa2129637ac9630e


>
> And while I am suggesting changes: Try to provide an InRelease file,
> solves a bunch of problems you likely don't have – but it also "works
> around" this bug.
>

I don’t control the repository building part — that’s outsourced to
bintray.com. Are there any tangible benefits (aside from working around
this specific bug) they’d get from switching to InRelease?


>
>
> > This seems like a regression to me, and requires an extra step in the
> > instructions for our users on how to enable our repository.
>
> I haven't seen the instructions, but the outline you gave with the
> commands doesn't look too bright. I don't have to tell you that
> --allow-unauthenticated is bad and we are on an active crusade against
> it… In fact, your commands just work because you used 'apt-get' and
> even there it will stop working in buster. If you had used 'apt' it
> would have completely failed the first update already.
>
> I would suggest instructions like (= not tried):
>
> $ /usr/lib/apt/apt-helper download-file http://example.org/keyring.deb
> keyring.deb SHA256:AAA…
> # apt install ./keyring.deb
> # apt edit-sources yourcoolstuff.list
> # apt update
>

Neat! I didn’t know this was possible. Updated the instructions
accordingly:
https://github.com/i3/i3.github.io/commit/d82c5335de3e52f06a0dc6d364284d315def4d71
and
https://github.com/i3/i3.github.io/commit/28387385984f9fa52061c4cdc59e20ba1d03965f


>
> Beside that something like this will work in stretch and beyond it is
> also more secure (assuming your users read the instructions on a https
> site) as that will automatically validate at least with a hash and
> doesn't rely on a user doing a check which most don't… (and which is
> very hard to perform in the --allow-unauthenticated pattern even if you
> want to).
>
> (keep the URI http:// through as a https:// one needs the optional
> -https transport your users might or might not have installed)
>
> If you think that process could be streamlined you are probably right,
> but nobody designed and implemented it yet.
>
>
> Best regards
>
> David Kalnischkies
>



-- 
Best regards,
Michael

Bug#838779: Release.gpg not downloaded unless Release is deleted

[David Kalnischkies]

On Sat, Sep 24, 2016 at 09:04:48PM +0200, Michael Stapelberg wrote:
> I tested this in Debian stable (with apt 1.0.9.8.3) and Debian
> testing/experimental (with apt 1.3~rc4 and 1.3, respectively). In Debian
> stable, this works as expected, but in Debian testing and experimental,
> I run into the bug. See the bottom of this report for a full command
> log.

That is a bug, althrough it's about not storing the Release.gpg file if
the Release file isn't stored at the same time – which it isn't if we
already got the latest version in storage – not about not downloading it…

Responsible seems to be apt-pkg/acquire-item.cc:1928 which checks if the
Release file was an IMSHit and only if not stores both files. We should
probably check here if the Release.gpg exists before skipping the
storage. If noone beats me to it I will have a look at patch+test for
next week… it gets a bit late to fiddle with security for today and I am
not really around tomorrow.


> When adding the http://dl.bintray.com/i3/i3-autobuild-ubuntu xenial
> repository, the Release and main_binary-amd64_Packages.lz4 files are
> downloaded and stored in /var/lib/apt/lists. Release.gpg is discarded
> because the necessary public key to verify the signature is not yet
> installed.

Just for the record: lz4 files aren't downloaded. The file downloaded is
likely xz or gz compressed and the current default is to uncompress, but
docker images tend to be configured to store them compressed, which apt
does by uncompressing the xz/gz and recompressing as lz4 on the fly.
Not important for this report, but just for completeness.


> After installing i3-autobuild-keyring, which adds the public key using
> apt-key add, the signature can successfully be verified (apt-get update
> no longer prints an error).

Pretty please with cherry on top don't use 'apt-key add'. Just ship your
keyring in /etc/apt/trusted.gpg.d/ avoiding the need for an installed
gpg. That works for ages now and since recently it even shows warnings.

And while I am suggesting changes: Try to provide an InRelease file,
solves a bunch of problems you likely don't have – but it also "works
around" this bug.


> This seems like a regression to me, and requires an extra step in the
> instructions for our users on how to enable our repository.

I haven't seen the instructions, but the outline you gave with the
commands doesn't look too bright. I don't have to tell you that
--allow-unauthenticated is bad and we are on an active crusade against
it… In fact, your commands just work because you used 'apt-get' and
even there it will stop working in buster. If you had used 'apt' it
would have completely failed the first update already.

I would suggest instructions like (= not tried):

$ /usr/lib/apt/apt-helper download-file http://example.org/keyring.deb 
keyring.deb SHA256:AAA…
# apt install ./keyring.deb
# apt edit-sources yourcoolstuff.list
# apt update

Beside that something like this will work in stretch and beyond it is
also more secure (assuming your users read the instructions on a https
site) as that will automatically validate at least with a hash and
doesn't rely on a user doing a check which most don't… (and which is
very hard to perform in the --allow-unauthenticated pattern even if you
want to).

(keep the URI http:// through as a https:// one needs the optional
-https transport your users might or might not have installed)

If you think that process could be streamlined you are probably right,
but nobody designed and implemented it yet.


Best regards

David Kalnischkies


signature.asc
Description: PGP signature

Bug#838779: Release.gpg not downloaded unless Release is deleted

[Michael Stapelberg]

Package: apt
Version: 1.3
Severity: normal

I tested this in Debian stable (with apt 1.0.9.8.3) and Debian
testing/experimental (with apt 1.3~rc4 and 1.3, respectively). In Debian
stable, this works as expected, but in Debian testing and experimental,
I run into the bug. See the bottom of this report for a full command
log.

When adding the http://dl.bintray.com/i3/i3-autobuild-ubuntu xenial
repository, the Release and main_binary-amd64_Packages.lz4 files are
downloaded and stored in /var/lib/apt/lists. Release.gpg is discarded
because the necessary public key to verify the signature is not yet
installed.

After installing i3-autobuild-keyring, which adds the public key using
apt-key add, the signature can successfully be verified (apt-get update
no longer prints an error).

However, even though Release.gpg is downloaded when running apt-get
update, Release.gpg is not stored in /var/lib/apt/lists! 

Only when I delete the Release file, apt-get update will store the
corresponding Release.gpg in /var/lib/apt/lists.

This seems like a regression to me, and requires an extra step in the
instructions for our users on how to enable our repository.

Am I misunderstanding something, or is this actually a bug in apt?

Steps to reproduce:

$ docker run -t -i debian:experimental /bin/bash
root@2b9c08db7f2a:/# apt-get update
Get:1 http://mirrors.kernel.org/debian unstable InRelease [209 kB]
Get:2 http://mirrors.kernel.org/debian experimental InRelease [107 kB]
Get:3 http://mirrors.kernel.org/debian unstable/main amd64 Packages [9673 kB]
Get:4 http://mirrors.kernel.org/debian experimental/main amd64 Packages [356 kB]
Fetched 10.3 MB in 3s (2902 kB/s)   
Reading package lists... Done
root@2b9c08db7f2a:/# echo 'deb http://dl.bintray.com/i3/i3-autobuild-ubuntu 
xenial main' > /etc/apt/sources.list.d/i3-autobuild.list
root@2b9c08db7f2a:/# cd /var/lib/apt/lists
root@2b9c08db7f2a:/var/lib/apt/lists# ls -l
total 16044
-rw-r--r-- 1 root root   107378 Sep 24 15:37 
httpredir.debian.org_debian_dists_experimental_InRelease
-rw-r--r-- 1 root root   717770 Sep 24 14:41 
httpredir.debian.org_debian_dists_experimental_main_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root   209324 Sep 24 15:40 
httpredir.debian.org_debian_dists_unstable_InRelease
-rw-r--r-- 1 root root 15379248 Sep 24 14:40 
httpredir.debian.org_debian_dists_unstable_main_binary-amd64_Packages.lz4
-rw-r- 1 root root0 Sep 24 18:49 lock
drwx-- 2 _apt root 4096 Sep 24 18:49 partial
root@2b9c08db7f2a:/var/lib/apt/lists# apt-get update
Ign:1 http://dl.bintray.com/i3/i3-autobuild-ubuntu xenial InRelease
Get:3 http://dl.bintray.com/i3/i3-autobuild-ubuntu xenial Release [2664 B]
Hit:2 http://mirrors.accretive-networks.net/debian unstable InRelease   

Get:5 http://dl.bintray.com/i3/i3-autobuild-ubuntu xenial Release.gpg [821 B]   

Hit:4 http://mirrors.accretive-networks.net/debian experimental InRelease
Ign:5 http://dl.bintray.com/i3/i3-autobuild-ubuntu xenial Release.gpg
Get:6 http://dl.bintray.com/i3/i3-autobuild-ubuntu xenial/main amd64 Packages 
[1044 B]
Fetched 4529 B in 0s (6117 B/s)
Reading package lists... Done
W: GPG error: http://dl.bintray.com/i3/i3-autobuild-ubuntu xenial Release: The 
following signatures couldn't be verified because the public key is not 
available: NO_PUBKEY 070D94CCBE1DB1F1
W: The repository 'http://dl.bintray.com/i3/i3-autobuild-ubuntu xenial Release' 
is not signed.
N: Data from such a repository can't be authenticated and is therefore 
potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration 
details.
root@2b9c08db7f2a:/var/lib/apt/lists# ls -l
total 16052
-rw-r--r-- 1 root root 2664 Sep 24 18:06 
dl.bintray.com_i3_i3-autobuild-ubuntu_dists_xenial_Release
-rw-r--r-- 1 root root 1483 Sep 24 18:06 
dl.bintray.com_i3_i3-autobuild-ubuntu_dists_xenial_main_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root   107378 Sep 24 15:37 
httpredir.debian.org_debian_dists_experimental_InRelease
-rw-r--r-- 1 root root   717770 Sep 24 14:41 
httpredir.debian.org_debian_dists_experimental_main_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root   209324 Sep 24 15:40 
httpredir.debian.org_debian_dists_unstable_InRelease
-rw-r--r-- 1 root root 15379248 Sep 24 14:40 
httpredir.debian.org_debian_dists_unstable_main_binary-amd64_Packages.lz4
-rw-r- 1 root root0 Sep 24 18:49 lock
drwx-- 2 _apt root 4096 Sep 24 18:49 partial
root@2b9c08db7f2a:/var/lib/apt/lists# apt-get install --allow-unauthenticated 
i3-autobuild-keyring
Reading package lists... Done
Building dependency tree... Done
The following additional packages will be installed:
  dirmngr gnupg gnupg-agent gnupg-l10n libassuan0 libffi6 libgmp10 libgnutls30 
libhogweed4 libidn11
  libksba8 libldap-2.4-2 libnettle6 libnpth0 libp11-kit0 libreadline6 
libsasl2-2 libsasl2-modules
  libsasl2-modules-db libsqlite3-0 libssl1.0.2 libtasn1-6 pinentry-curses