Bug#880170: aptitude: Read error (-5: DATA_ERROR_MAGIC)

2024-06-08 Thread Julia Longtin 
I can reproduce this on command, and have two workarounds, (one dirty, one
ugly).

I'm using FAI on Debian 9.13.

So, the problem is, that when apt writes a translations related file, it is
writing a symlink to the extracted .bz2, rather than writing a .bz2 file.

Problem reproduction:

I am using apt-cacher-ng, and am currently pointing it to archive.debian.org.
I am also using apt against a chroot, as part of using the Fully Automated
Installer. This is happening while I'm trying to set up a mirror of debian,
with only the packages I have selected.

as part of that, my apt options are as follows:
```
-o Aptitude::Log=/dev/null -o
Aptitude::CmdLine::Ignore-Trust-Violations=yes -o
APT::Get::AllowUnauthenticated=true -o DPkg::force-conflicts::=yes -o
Dir::State=/usr/fai/mirror/aptcache//var/lib/apt -o
Dir::Log=/usr/fai/mirror/aptcache//var/log/apt -o
Dir::State::extended_states=/usr/fai/mirror/aptcache//var/lib/apt/lists/extended_states
-o Dir::State::status=/usr/fai/mirror/aptcache//statefile -o
APT::Get::Force-Yes=true -o
Dir::Cache=/usr/fai/mirror/aptcache//var/cache/apt -o
Dir::State=/usr/fai/mirror/aptcache//var/cache/apt -o
Dir::Cache::Archives=/usr/fai/mirror/aptcache//var/cache/apt/archives -o
Dir::Etc=/usr/fai/mirror/aptcache//etc/apt/ -o
Dir::State::Lists=/usr/fai/mirror/aptcache//var/lib/apt/lists/
```

The error in question is easy to produce:
```
faiserver:/home/fai# apt-get $aptoptions update
Ign:1 http://127.0.0.1:3142/archive.debian.org/debian stretch InRelease
Get:2 http://127.0.0.1:3142/archive.debian.org/debian stretch Release [118
kB]
Get:3 http://127.0.0.1:3142/archive.debian.org/debian stretch Release.gpg
[3,177 B]
Ign:3 http://127.0.0.1:3142/archive.debian.org/debian stretch Release.gpg
Hit:5 http://127.0.0.1:3142/archive.debian.org/debian stretch/main
Translation-en
Ign:5 http://127.0.0.1:3142/archive.debian.org/debian stretch/main
Translation-en
Hit:4 http://127.0.0.1:3142/archive.debian.org/debian stretch/main i386
Packages
Ign:4 http://127.0.0.1:3142/archive.debian.org/debian stretch/main i386
Packages
Hit:5 http://127.0.0.1:3142/archive.debian.org/debian stretch/main
Translation-en
Ign:5 http://127.0.0.1:3142/archive.debian.org/debian stretch/main
Translation-en
Hit:4 http://127.0.0.1:3142/archive.debian.org/debian stretch/main i386
Packages
Ign:4 http://127.0.0.1:3142/archive.debian.org/debian stretch/main i386
Packages
Err:5 http://127.0.0.1:3142/archive.debian.org/debian stretch/main
Translation-en
  BZ2_bzread:
/usr/fai/mirror/aptcache/var/lib/apt/lists/partial/127.0.0.1:3142_archive.debian.org_debian_dists_stretch_main_i18n_Translation-en.bz2
Read error (-5: DATA_ERROR_MAGIC)
Get:4 http://127.0.0.1:3142/archive.debian.org/debian stretch/main i386
Packages [9,587 kB]
Fetched 121 kB in 3s (40.2 kB/s)
Reading package lists... Done
W: GPG error: http://127.0.0.1:3142/archive.debian.org/debian stretch
Release: The following signatures couldn't be verified because the public
key is not available: NO_PUBKEY 04EE7237B7D453EC NO_PUBKEY 648ACFD622F3D138
NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY EF0F382A1A7B6500
W: The repository 'http://127.0.0.1:3142/archive.debian.org/debian stretch
Release' is not signed.
N: Data from such a repository can't be authenticated and is therefore
potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration
details.
E: Failed to fetch
http://127.0.0.1:3142/archive.debian.org/debian/dists/stretch/main/i18n/Translation-en
 BZ2_bzread:
/usr/fai/mirror/aptcache/var/lib/apt/lists/partial/127.0.0.1:3142_archive.debian.org_debian_dists_stretch_main_i18n_Translation-en.bz2
Read error (-5: DATA_ERROR_MAGIC)
E: Some index files failed to download. They have been ignored, or old ones
used instead.
faiserver:/home/fai#
```

that's the error in question. A quick inspection of what the file it is
complaining looks like is as follows:
```
faiserver:/home/fai# ls -la
/usr/fai/mirror/aptcache/var/lib/apt/lists/partial/127.0.0.1:3142
_archive.debian.org_debian_dists_stretch_main_i18n_Translation-en.bz2
lrwxrwxrwx 1 root root 122 Jun  8 12:41
/usr/fai/mirror/aptcache/var/lib/apt/lists/partial/127.0.0.1:3142_archive.debian.org_debian_dists_stretch_main_i18n_Translation-en.bz2
-> /usr/fai/mirror/aptcache/var/lib/apt/lists/127.0.0.1:3142
_archive.debian.org_debian_dists_stretch_main_i18n_Translation-en
faiserver:/home/fai# ls -la
/usr/fai/mirror/aptcache/var/lib/apt/lists//127.0.0.1:3142
_archive.debian.org_debian_dists_stretch_main_i18n_Translation-en
-rw-r--r-- 1 root root 26417876 Jul 18  2020
/usr/fai/mirror/aptcache/var/lib/apt/lists//127.0.0.1:3142
_archive.debian.org_debian_dists_stretch_main_i18n_Translation-en
faiserver:/home/fai# file
/usr/fai/mirror/aptcache/var/lib/apt/lists//127.0.0.1:3142
_archive.debian.org_debian_dists_stretch_main_i18n_Translation-en
/usr/fai/mirror/aptcache/var/lib/apt/lists//127.0.0.1:3142_archive.debian.org_debian_dists_stretch_main_i18n_Translation-en:
UTF-8 Unicode text
```

so apt-get has created

Bug#880170: aptitude: Read error (-5: DATA_ERROR_MAGIC)

2020-04-07 Thread Matthijs Kooijman 
Package: apt
Followup-For: Bug #880170

Hi,

I just also ran into this problem. It's on a stretch system, so with an
older apt, but maybe my observations help regardless.

I saw this error running `aptitude update`. It happened with the stretch
security updates translation file for contrib and non-free.

I read that emptying /var/lib/apt/lists might help, so I tried a light
version: I deleted only
security.debian.org_dists_stretch_updates_non-free_i18n_Translation-en
and security.debian.org_dists_stretch_updates_contrib_i18n_Translation-en
from that directory. I had expected these files to be somehow
problematic, so I saved them to restore them later to reproduce the
problem again.

Removing both files indeed made the `aptitude update` complete
succesfully. To my surprise these two files had been recreated, with
exactly the same contents as before. Another `aptitude update` succeeds,
so apparently something else changed (but I did not make a full backup
of the lists directory, unfortunately). So I could not reproduce the
problem by restoring these files manually, as I had planned.

Looking on the server, I see that a
http://security.debian.org/dists/stretch/updates/contrib/i18n/Translation-en.bz2
does exist, so the 404-theory stated previously might not be
accurate.

I also looked for the failing file in /var/lib/apt/lists/partial, but
there were no files in there when I looked.

Sorry I don't have anything more specific, but maybe some of this
helps...

Gr.

Matthijs

-- Package-specific info:

-- apt-config dump --

APT "";
APT::Architecture "amd64";
APT::Build-Essential "";
APT::Build-Essential:: "build-essential";
APT::Install-Recommends "1";
APT::Install-Suggests "0";
APT::Sandbox "";
APT::Sandbox::User "_apt";
APT::Authentication "";
APT::Authentication::TrustCDROM "true";
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image-4\.9\.0-6-amd64$";
APT::NeverAutoRemove:: "^linux-image-4\.9\.0-8-amd64$";
APT::NeverAutoRemove:: "^linux-headers-4\.9\.0-6-amd64$";
APT::NeverAutoRemove:: "^linux-headers-4\.9\.0-8-amd64$";
APT::NeverAutoRemove:: "^linux-image-extra-4\.9\.0-6-amd64$";
APT::NeverAutoRemove:: "^linux-image-extra-4\.9\.0-8-amd64$";
APT::NeverAutoRemove:: "^linux-signed-image-4\.9\.0-6-amd64$";
APT::NeverAutoRemove:: "^linux-signed-image-4\.9\.0-8-amd64$";
APT::NeverAutoRemove:: "^kfreebsd-image-4\.9\.0-6-amd64$";
APT::NeverAutoRemove:: "^kfreebsd-image-4\.9\.0-8-amd64$";
APT::NeverAutoRemove:: "^kfreebsd-headers-4\.9\.0-6-amd64$";
APT::NeverAutoRemove:: "^kfreebsd-headers-4\.9\.0-8-amd64$";
APT::NeverAutoRemove:: "^gnumach-image-4\.9\.0-6-amd64$";
APT::NeverAutoRemove:: "^gnumach-image-4\.9\.0-8-amd64$";
APT::NeverAutoRemove:: "^.*-modules-4\.9\.0-6-amd64$";
APT::NeverAutoRemove:: "^.*-modules-4\.9\.0-8-amd64$";
APT::NeverAutoRemove:: "^.*-kernel-4\.9\.0-6-amd64$";
APT::NeverAutoRemove:: "^.*-kernel-4\.9\.0-8-amd64$";
APT::NeverAutoRemove:: "^linux-backports-modules-.*-4\.9\.0-6-amd64$";
APT::NeverAutoRemove:: "^linux-backports-modules-.*-4\.9\.0-8-amd64$";
APT::NeverAutoRemove:: "^linux-tools-4\.9\.0-6-amd64$";
APT::NeverAutoRemove:: "^linux-tools-4\.9\.0-8-amd64$";
APT::VersionedKernelPackages "";
APT::VersionedKernelPackages:: "linux-image";
APT::VersionedKernelPackages:: "linux-headers";
APT::VersionedKernelPackages:: "linux-image-extra";
APT::VersionedKernelPackages:: "linux-signed-image";
APT::VersionedKernelPackages:: "kfreebsd-image";
APT::VersionedKernelPackages:: "kfreebsd-headers";
APT::VersionedKernelPackages:: "gnumach-image";
APT::VersionedKernelPackages:: ".*-modules";
APT::VersionedKernelPackages:: ".*-kernel";
APT::VersionedKernelPackages:: "linux-backports-modules-.*";
APT::VersionedKernelPackages:: "linux-tools";
APT::Never-MarkAuto-Sections "";
APT::Never-MarkAuto-Sections:: "metapackages";
APT::Never-MarkAuto-Sections:: "contrib/metapackages";
APT::Never-MarkAuto-Sections:: "non-free/metapackages";
APT::Never-MarkAuto-Sections:: "restricted/metapackages";
APT::Never-MarkAuto-Sections:: "universe/metapackages";
APT::Never-MarkAuto-Sections:: "multiverse/metapackages";
APT::Move-Autobit-Sections "";
APT::Move-Autobit-Sections:: "oldlibs";
APT::Move-Autobit-Sections:: "contrib/oldlibs";
APT::Move-Autobit-Sections:: "non-free/oldlibs";
APT::Move-Autobit-Sections:: "restricted/oldlibs";
APT::Move-Autobit-Sections:: "universe/oldlibs";
APT::Move-Autobit-Sections:: "multiverse/oldlibs";
APT::Periodic "";
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
APT::Architectures "";
APT::Architectures:: "amd64";
APT::Compressor "";
APT::Compressor::. "";
APT::Compressor::.::Name ".";
APT::Compressor::.::Extension "";
APT::Compressor::.::Binary "";
APT::Compressor::.::Cost "0";
APT::Compressor::lz4 "";
APT::Compressor::lz4::Name "lz4";
APT::Compressor::lz4::Extension ".lz4";
APT::Compressor::lz4::Binary "false";
APT::Compressor::lz4::Cost "50";
APT::Compressor::gzip

Bug#880170: aptitude: Read error (-5: DATA_ERROR_MAGIC)

2017-10-31 Thread David Kalnischkies 
<150935636765.12708.6154635335479747081.reportbug@think>

Hi,

On Mon, Oct 30, 2017 at 10:39:27AM +0100, Mykola Nikishov wrote:
> $ sudo aptitude update 
> ...
> W: Failed to fetch 
> tor+http://security.debian.org/dists/stable/updates/contrib/i18n/Translation-en:
>  BZ2_bzread: 
> /var/lib/apt/lists/partial/security.debian.org_dists_stable_updates_contrib_i18n_Translation-en.bz2
>  Read error (-5: DATA_ERROR_MAGIC)
> E: Some index files failed to download. They have been ignored, or old ones 
> used instead.

Download issues in the update stage tend to not be the fault of
aptitude, as it shares the code with apt (via libapt). Whose fault
it is on the other hand…

Assuming you aren't doing something dangerous like disabling security
checks such an error means that the file passed hashsum checks while
downloading, but while trying to extract the file contents a bad file
was encountered where a good file should have been.

That tends to be hardware issues like a slowly failing disk – but there
is of course also the option that the archive had a bad file (less
likely through).


Would be nice if you could attach the failing file (and the Release file
which should sit there as well):
/var/lib/apt/lists/partial/security.debian.org_dists_stable_updates_Release
/var/lib/apt/lists/partial/security.debian.org_dists_stable_updates_contrib_i18n_Translation-en.bz2

So we can have a look as me trying to download what is available now
produces no such error (with apt, but same difference).


Also, if you include the output of a command, please include the entire
output: It can help figuring out what the tool did before reaching the
error as there tend to be multiple ways of arriving at the same error.
Talking back to you a triager will usually "cut down" the output to the
error messages, but that doesn't mean he hasn't read the cut lines.


> -- System Information:
> Debian Release: buster/sid
>   APT prefers stable
>   APT policy: (500, 'stable'), (70, 'unstable'), (60, 'testing'), (50, 
> 'experimental')
[…]
> Versions of packages aptitude depends on:
[…]
> ii  libapt-pkg5.0  1.5

m. That is the version from testing. Such "Frankendebian" systems
are not very advisable. I guess we have to add the possibility of
a hidden library incompatibility in the list above as a potential
contender.

btw: Yeah for using tor! You know that there are onion services for all
Debian archives available? See onion.debian.org [5nca3wxl33tzlzj5.onion]


Best regards

David Kalnischkies, apt(-transport-tor) developer


signature.asc
Description: PGP signature

Bug#880170: [Aptitude-devel] Bug#880170: aptitude: Read error (-5: DATA_ERROR_MAGIC)

2017-10-30 Thread Axel Beckert 
Control: reassign -1 apt 1.5

Hi Mykola,

thanks for the bug report.

Mykola Nikishov wrote:
> Package: aptitude
[…]
> $ sudo aptitude update 
> ...
> W: Failed to fetch 
> tor+http://security.debian.org/dists/stable/updates/contrib/i18n/Translation-en:
>  BZ2_bzread: 
> /var/lib/apt/lists/partial/security.debian.org_dists_stable_updates_contrib_i18n_Translation-en.bz2
>  Read error (-5: DATA_ERROR_MAGIC)
> E: Some index files failed to download. They have been ignored, or old ones 
> used instead.
> 
> Removing the file in question doesn't help, an error persists.

In the past, I've seen such deadlocks occassionally. But haven't had
such a case for one or two years now. Back then it usually sufficed to
also remove the according file in /var/lib/apt/lists/ or, in the worst
case, to empty /var/lib/apt/lists/ completely.

Since things like package list downloads are not done by aptitude
itself but by calling functions from libapt-*, I'm reassigning this to
apt.

Regards, Axel
-- 
 ,''`.  |  Axel Beckert , https://people.debian.org/~abe/
: :' :  |  Debian Developer, ftp.ch.debian.org Admin
`. `'   |  4096R: 2517 B724 C5F6 CA99 5329  6E61 2FF9 CD59 6126 16B5
  `-|  1024D: F067 EA27 26B9 C3FC 1486  202E C09E 1D89 9593 0EDE

Bug#880170: aptitude: Read error (-5: DATA_ERROR_MAGIC)

2017-10-30 Thread Mykola Nikishov 
Package: aptitude
Version: 0.8.9-1
Severity: normal

$ sudo aptitude update 
...
W: Failed to fetch 
tor+http://security.debian.org/dists/stable/updates/contrib/i18n/Translation-en:
 BZ2_bzread: 
/var/lib/apt/lists/partial/security.debian.org_dists_stable_updates_contrib_i18n_Translation-en.bz2
 Read error (-5: DATA_ERROR_MAGIC)
E: Some index files failed to download. They have been ignored, or old ones 
used instead.

Removing the file in question doesn't help, an error persists.

-- Package-specific info:
Terminal: screen-256color
$DISPLAY is set.
which aptitude: /usr/bin/aptitude

aptitude version information:
aptitude 0.8.9
Compiler: g++ 7.2.0
Compiled against:
  apt version 5.0.2
  NCurses version 6.0
  libsigc++ version: 2.10.0
  Gtk+ support disabled.
  Qt support disabled.

Current library versions:
  NCurses version: ncurses 6.0.20161126
  cwidget version: 0.5.17
  Apt version: 5.0.2

aptitude linkage:
linux-vdso.so.1 (0x7fffdbb25000)
libapt-pkg.so.5.0 => /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0 
(0x7f6eacfe5000)
libncursesw.so.5 => /lib/x86_64-linux-gnu/libncursesw.so.5 
(0x7f6eacdb5000)
libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 
(0x7f6eacb8b000)
libsigc-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libsigc-2.0.so.0 
(0x7f6eac984000)
libcwidget.so.3 => /usr/lib/x86_64-linux-gnu/libcwidget.so.3 
(0x7f6eac689000)
libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 
(0x7f6eac381000)
libboost_iostreams.so.1.62.0 => 
/usr/lib/x86_64-linux-gnu/libboost_iostreams.so.1.62.0 (0x7f6eac169000)
libboost_filesystem.so.1.62.0 => 
/usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.62.0 (0x7f6eabf5)
libboost_system.so.1.62.0 => 
/usr/lib/x86_64-linux-gnu/libboost_system.so.1.62.0 (0x7f6eabd4c000)
libxapian.so.30 => /usr/lib/x86_64-linux-gnu/libxapian.so.30 
(0x7f6eab941000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
(0x7f6eab724000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 
(0x7f6eab3a2000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7f6eab09e000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 
(0x7f6eaae87000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f6eaaae8000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 
(0x7f6eaa8d1000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x7f6eaa6b7000)
libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 
(0x7f6eaa4a7000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x7f6eaa281000)
liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 
(0x7f6eaa06c000)
libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x7f6ea9e4e000)
/lib64/ld-linux-x86-64.so.2 (0x7f6ead9a2000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7f6ea9c4a000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x7f6ea9a42000)
libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x7f6ea983d000)

-- System Information:
Debian Release: buster/sid
  APT prefers stable
  APT policy: (500, 'stable'), (70, 'unstable'), (60, 'testing'), (50, 
'experimental')
Architecture: amd64
 (x86_64)

Kernel: Linux 4.9.0-4-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages aptitude depends on:
ii  aptitude-common0.8.9-1
ii  libapt-pkg5.0  1.5
ii  libboost-filesystem1.62.0  1.62.0+dfsg-4+b2
ii  libboost-iostreams1.62.0   1.62.0+dfsg-4+b2
ii  libboost-system1.62.0  1.62.0+dfsg-4+b2
ii  libc6  2.24-11+deb9u1
ii  libcwidget3v5  0.5.17-6
ii  libgcc11:6.3.0-18
ii  libncursesw5   6.0+20161126-1+deb9u1
ii  libsigc++-2.0-0v5  2.10.0-1
ii  libsqlite3-0   3.20.1-2
ii  libstdc++6 6.3.0-18
ii  libtinfo5  6.0+20161126-1+deb9u1
ii  libxapian301.4.4-3

Versions of packages aptitude recommends:
pn  libparse-debianchangelog-perl  
ii  sensible-utils 0.0.10

Versions of packages aptitude suggests:
pn  apt-xapian-index
pn  aptitude-doc-en | aptitude-doc  
pn  debtags 
pn  tasksel 

-- no debconf information