Bug#838441: apt-get update fails with "Hash Sum mismatch", mixes hashes between tar.gz and tar file
Hello,
I am not familiar about the release process, so I do not know how could I
grab a fixed apt and install it to confirm this fixes my original issue.
I went without updating these VMs quite for some time (vacations!), but now
can re-test if that helps.
Thanks for checking and providing a fix for the issue.
Regards,
///Pablo
2016-11-13 20:53 GMT-03:00 Pablo Di Noto :
>
> (mhh, I wonder why I missed the initial report…)
>>
>
> Not familiar with Debian bug system, so perhaps I did not follow the
> proper procedure?
>
>
>>
>> On Sun, Nov 13, 2016 at 08:15:57PM +0100, Markus Wanner wrote:
>> > > Acquire::http::Proxy "http://10.137.255.254:8082/";;
>> >
>> > Out of curiosity and hopefully narrowing down the issue: What kind of
>> > proxy is this on your side?
>>
>> Another good point would be running apt with:
>> -o Debug::pkgAcquire::Worker=1 -o Debug::Acquire::http=1
>>
>
> Will try this during the week, once the downloaded files get expired.
>
>
>> And is "repeatedly" meant to refer to "reproducible all the time" or
>> "happens often, but no obvious pattern"?
>>
>
> The pattern I was able to detect is that:
> - It always happen on files related to dep11 and icons, which are quite
> big.
> - Happens 50% of the time
> - Happens on debian 8 and debian 9 (will collect same set of logs on the
> newer template)
>
>
>> Interesting is the failure in copy ("E: Failed to fetch copy:") as that
>> is supposed to "just" move files around without (un)compression – and we
>> have passed the stage proxies could interfere as the download itself
>> verified (or not, maybe it IMS hits in some way and copy is supposed to
>> verify it – there are various ways such a not-modified state can be
>> reached and proxies are notoriously bad with it…).
>>
>> btw: It is best to run apt with the least amount of config usually. The
>> mentioned config options are rather special case and especially the
>> "Acquire::BrokenProxy" one doesn't even exist… (I looked once, it seems
>> to have existed ~10 years ago for one year in no stable release and the
>> name was very bad for what it actually did…). Rule of thumb: If you
>> don't know what you are doing, use neither – unfortunately it seems the
>> people commonly answering questions on q&a-sites tend to be in the very
>> vocal "no idea, but I get points for posting stuff anyhow" group.
>>
>
> Now that I realize the proxy is a suspect, I will clone the template and
> perform the same update with and without the proxy. That should provide
> some clues. Same goes for the "bare minimum" apt-config file.
>
>>
> Thanks for the suggestions,
> ///Pablo
>
Bug#838441: apt-get update fails with "Hash Sum mismatch", mixes hashes between tar.gz and tar file
> (mhh, I wonder why I missed the initial report…)
>
Not familiar with Debian bug system, so perhaps I did not follow the proper
procedure?
>
> On Sun, Nov 13, 2016 at 08:15:57PM +0100, Markus Wanner wrote:
> > > Acquire::http::Proxy "http://10.137.255.254:8082/";;
> >
> > Out of curiosity and hopefully narrowing down the issue: What kind of
> > proxy is this on your side?
>
> Another good point would be running apt with:
> -o Debug::pkgAcquire::Worker=1 -o Debug::Acquire::http=1
>
Will try this during the week, once the downloaded files get expired.
> And is "repeatedly" meant to refer to "reproducible all the time" or
> "happens often, but no obvious pattern"?
>
The pattern I was able to detect is that:
- It always happen on files related to dep11 and icons, which are quite big.
- Happens 50% of the time
- Happens on debian 8 and debian 9 (will collect same set of logs on the
newer template)
> Interesting is the failure in copy ("E: Failed to fetch copy:") as that
> is supposed to "just" move files around without (un)compression – and we
> have passed the stage proxies could interfere as the download itself
> verified (or not, maybe it IMS hits in some way and copy is supposed to
> verify it – there are various ways such a not-modified state can be
> reached and proxies are notoriously bad with it…).
>
> btw: It is best to run apt with the least amount of config usually. The
> mentioned config options are rather special case and especially the
> "Acquire::BrokenProxy" one doesn't even exist… (I looked once, it seems
> to have existed ~10 years ago for one year in no stable release and the
> name was very bad for what it actually did…). Rule of thumb: If you
> don't know what you are doing, use neither – unfortunately it seems the
> people commonly answering questions on q&a-sites tend to be in the very
> vocal "no idea, but I get points for posting stuff anyhow" group.
>
Now that I realize the proxy is a suspect, I will clone the template and
perform the same update with and without the proxy. That should provide
some clues. Same goes for the "bare minimum" apt-config file.
>
Thanks for the suggestions,
///Pablo
Bug#838441: apt-get update fails with "Hash Sum mismatch", mixes hashes between tar.gz and tar file
(cc: David Kalnischkies as this may answer his question too) Hello Pablo, > > I run into the same issue when proxying my apt through apt-cacher-ng. > Not without any proxy in between. > > It seems you're using a proxy, too: > > > Acquire::http::Proxy "http://10.137.255.254:8082/";; > > Out of curiosity and hopefully narrowing down the issue: What kind of > proxy is this on your side? > You are right, there is a proxy and I failed to mention it. D'oh! This report comes from a Qubes OS "Debian 8 template", which runs on a xen-based VM, and is just a regular Debian with some additional packages that allow the VMs based on this template to play nice on a Qubes OS system (allowing copy&paste between machines, handle block device sharing, etc). They way Qubes OS works in terms of networking is that "application VMs" connect to internet through a "firewall VM", which has a WAN interface (the insecure one) connected to a "network VM". At the end, from the point of view of the VM I am reporting the issue from, it is like getting to the Internet through a Linux-based firewall and then a home router. Qubes templates have a tinyproxy-based proxy that allows the update of the machines regardless of the firewall settings. Hence, they have the Acquire::http::Proxy "http://10.137.255.254:8082/"; statement on their apt config files. This proxy config seems quite generic: User tinyproxy > Group tinyproxy > Port 8082 > Timeout 60 > DefaultErrorFile "/usr/share/tinyproxy/default.html" > #StatHost "tinyproxy.stats" > StatFile "/usr/share/tinyproxy/stats.html" > Syslog On > LogLevel Notice > PidFile "/var/run/tinyproxy-updates/tinyproxy.pid" > MaxClients 50 > MinSpareServers 2 > MaxSpareServers 10 > StartServers 2 > MaxRequestsPerChild 0 > DisableViaHeader Yes > Allow 127.0.0.1 > Allow 10.137.0.0/16 > ConnectPort 443 I will monitor syslog when performing the apt-get update and see if I can catch any special event. So far, the same issue happens on my newer template, which is Debian 9-based, and is happening almost 50% of the time when the apt-lists are no longer valid. When it happens, repeating the 'apt-get update' command succeeds. Thanks for pointing out this "small" detail. Regards, ///Pablo
Bug#838441: apt-get update fails with "Hash Sum mismatch", mixes hashes between tar.gz and tar file
(mhh, I wonder why I missed the initial report…)
On Sun, Nov 13, 2016 at 08:15:57PM +0100, Markus Wanner wrote:
> > Acquire::http::Proxy "http://10.137.255.254:8082/";;
>
> Out of curiosity and hopefully narrowing down the issue: What kind of
> proxy is this on your side?
Another good point would be running apt with:
-o Debug::pkgAcquire::Worker=1 -o Debug::Acquire::http=1
That has a bunch of output, so perhaps add: 2>&1 | tee filename.log
And is "repeatedly" meant to refer to "reproducible all the time" or
"happens often, but no obvious pattern"?
Interesting is the failure in copy ("E: Failed to fetch copy:") as that
is supposed to "just" move files around without (un)compression – and we
have passed the stage proxies could interfere as the download itself
verified (or not, maybe it IMS hits in some way and copy is supposed to
verify it – there are various ways such a not-modified state can be
reached and proxies are notoriously bad with it…).
btw: It is best to run apt with the least amount of config usually. The
mentioned config options are rather special case and especially the
"Acquire::BrokenProxy" one doesn't even exist… (I looked once, it seems
to have existed ~10 years ago for one year in no stable release and the
name was very bad for what it actually did…). Rule of thumb: If you
don't know what you are doing, use neither – unfortunately it seems the
people commonly answering questions on q&a-sites tend to be in the very
vocal "no idea, but I get points for posting stuff anyhow" group.
Best regards
David Kalnischkies
signature.asc
Description: PGP signature
Bug#838441: apt-get update fails with "Hash Sum mismatch", mixes hashes between tar.gz and tar file
Hello Pablo, I run into the same issue when proxying my apt through apt-cacher-ng. Not without any proxy in between. It seems you're using a proxy, too: > Acquire::http::Proxy "http://10.137.255.254:8082/";; Out of curiosity and hopefully narrowing down the issue: What kind of proxy is this on your side? Kind Regards Markus Wanner signature.asc Description: OpenPGP digital signature
Bug#838441: apt-get update fails with "Hash Sum mismatch", mixes hashes between tar.gz and tar file
Package: apt Version: 1.3~rc4 Severity: important Dear Maintainer, When doing a apt-get update under a Debian 9 system, I repeatedly get: Reading package lists... Done E: Failed to fetch copy:/var/lib/apt/lists/partial/http.debian .net_debian_dists_stretch_main_dep11_icons-64x64.tar.gz Hash Sum mismatch Hashes of expected file: - Checksum-FileSize:7626752 [weak] - SHA256:b7dd6cc0630334321ff822cf2fc69df1263d81548e808c46f7a77af66f54c19e - SHA1:f90289555ba000250b4f3b06c2d4fa70eae4da72 [weak] - MD5Sum:bf3fd40405388c5da311c791f93dfa9e [weak] Hashes of received file: - SHA256:34032bc756f67b5257c6a16fe9fb1ec14a9ee72890c7739b6b257fba11e90112 - SHA1:686718a66a3d2e583580a574b324ba7d03be5a0c [weak] - MD5Sum:c35c6eed0e0dfd9830b431f245eb1c5a [weak] - Checksum-FileSize:6054390 [weak] Last modification reported: Tue, 20 Sep 2016 20:56:45 + Release file created at: Wed, 21 Sep 2016 03:35:08 + E: Some index files failed to download. They have been ignored, or old ones used instead. This is happening since 10-20 days ago. Tried some common recipes for "Hash mismatch", like switching mirrors, adding Acquire::http::Pipeline-Depth 0; Acquire::http::No-Cache true; Acquire::BrokenProxytrue; to /etc/apt/apt.conf.d/99fixbadproxy with no change. Trying to debug what is happening, I pulled the file giving hash mismatch error: > user@debian-8x:~$ wget http://debian.salud.gob.sv/debian/dists/stretch/main/dep11/icons-64x64.tar.gz > --2016-09-21 04:14:58-- http://debian.salud.gob.sv/debian/dists/stretch/main/dep11/icons-64x64.tar.gz > Resolving debian.salud.gob.sv (debian.salud.gob.sv)... 190.86.223.16 > Connecting to debian.salud.gob.sv (debian.salud.gob.sv)|190.86.223.16|:80... connected. > HTTP request sent, awaiting response... 200 OK > Length: 6054390 (5.8M) [application/x-gzip] > Saving to: ‘icons-64x64.tar.gz’ > > icons-64x64.tar.gz 100%[>] 5.77M 216KB/s in 68ss > > 2016-09-21 04:16:08 (87.0 KB/s) - ‘icons-64x64.tar.gz’ saved [6054390/6054390] > > user@debian-8x:~$ ls -l > total 5956 > (...) > -rw-r--r-- 1 user user 6054390 Sep 20 17:56 icons-64x64.tar.gz > (...) > user@debian-8x:~$ md5sum icons-64x64.tar.gz > c35c6eed0e0dfd9830b431f245eb1c5a icons-64x64.tar.gz > user@debian-8x:~$ sha1sum icons-64x64.tar.gz > 686718a66a3d2e583580a574b324ba7d03be5a0c icons-64x64.tar.gz > user@debian-8x:~$ sha256sum icons-64x64.tar.gz > 34032bc756f67b5257c6a16fe9fb1ec14a9ee72890c7739b6b257fba11e90112 icons- 64x64.tar.gz ...which matches the "Hashes of received file:" part of the error. I see the file arrived ok, and gunzip does not complain when decompressing, > user@debian-8x:~$ gunzip icons-64x64.tar.gz > user@debian-8x:~$ md5sum icons-64x64.tar > bf3fd40405388c5da311c791f93dfa9e icons-64x64.tar > user@debian-8x:~$ sha1sum icons-64x64.tar > f90289555ba000250b4f3b06c2d4fa70eae4da72 icons-64x64.tar > user@debian-8x:~$ sha256sum icons-64x64.tar > b7dd6cc0630334321ff822cf2fc69df1263d81548e808c46f7a77af66f54c19e icons- 64x64.tar and even the hashes match on the uncompressed file hashes on the error report. Now, I pull http://debian.salud.gob.sv/debian/dists/stretch/Release to check if I am receiving corrupted Release by a transparent proxy or something like that. Checking its contents I see > user@debian-8x:~$ grep -n 'main/dep11/icons-64x64.tar' Release > 393: bf3fd40405388c5da311c791f93dfa9e 7626752 main/dep11/icons-64x64.tar > 394: c35c6eed0e0dfd9830b431f245eb1c5a 6054390 main/dep11/icons-64x64.tar.gz > 1278: f90289555ba000250b4f3b06c2d4fa70eae4da72 7626752 main/dep11/icons- 64x64.tar > 1279: 686718a66a3d2e583580a574b324ba7d03be5a0c 6054390 main/dep11/icons- 64x64.tar.gz > 2163: b7dd6cc0630334321ff822cf2fc69df1263d81548e808c46f7a77af66f54c19e 7626752 main/dep11/icons-64x64.tar > 2164: 34032bc756f67b5257c6a16fe9fb1ec14a9ee72890c7739b6b257fba11e90112 6054390 main/dep11/icons-64x64.tar.gz So every hash matches, both compressed and uncompressed file. Reviewing the error message from apt-get, I see it that the error is correctly listing the "Hashes of received file" (the .tar.gz), but expecting the hashes the uncompressed file (.tar): -- Package-specific info: -- apt-config dump -- APT ""; APT::Architecture "amd64"; APT::Build-Essential ""; APT::Build-Essential:: "build-essential"; APT::Install-Recommends "1"; APT::Install-Suggests "0"; APT::Sandbox ""; APT::Sandbox::User "_apt"; APT::NeverAutoRemove ""; APT::NeverAutoRemove:: "^firmware-linux.*"; APT::NeverAutoRemove:: "^linux-firmware$"; APT::NeverAutoRemove:: "^linux-image-3\.18\.17-4\.pvops\.qubes\.x86_64$"; APT::NeverAutoRemove:: "^linux-headers-3\.18\.17-4\.pvops\.qubes\.x86_64$"; APT::NeverAutoRemove:: "^linux-image-extra-3\.18\.17-4\.pvops\.qubes\.x86_64$"; APT::NeverAutoRemove:: "^linux-signed-image-3\.18\.17-4\.pvops\.qubes\.x86_64$"; APT::NeverAutoRemove:: "^kfreebsd-image-3\.18\.17-4\.pvops\.qubes\.x86_64$"; APT::NeverAutoRemove:: "^kfreebsd-headers-3\.18\.
