Bug#1034650: debian-installer: bookworm d-i rc1: apt-get clean breaks bash-completion

[Askar Safin]

I did some experiments and wrote a patch!

Okay, so, I spend some more time thinking about this. And I think the
bug is in apt's completion scripts. Because that *.bin files are
simply cache. And lack of caches should not break program! This
follows from my interpretation of FHS 3.0 5.5 (
https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s05.html ). And
that *.bin files are located in /var/cache. And debian policy refers
to FHS 3.0 ( 
https://www.debian.org/doc/debian-policy/ch-opersys.html#file-system-hierarchy
). So, the bug is in apt's completion scripts and we should simply
remove "--no-generate" from them. And hence I attach a patch, which
does exactly that.

This will cause completion scripts to regenerate caches as needed.
This will work even if *.bin files are not present. And this will work
even if we are in unprivileged bash, I tested this.

Okay, so you can ask: what are speed implications of this? Well, I
tested this on my main system (Intel Core i7) and I feel that even
full cache regeneration takes zero time, i. e. it is absolutely
instant.

So I started qemu-system-x86_64 without -enable-kvm and tested there.

Results are this: if caches are not present, then bash-completion is
slow, it takes something like 10 seconds (in qemu in emulation
mode!!!). But if caches are present, then bash-completion feels fast.
So, it seems that apt is smart, it regenerates cache only if needed.
So, there is no need to insert special "if" in completion scripts to
check whether we should regenerate caches, apt already does this.

Also, I found that file /usr/share/bash-completion/completions/apt is
owned by apt, but file /usr/share/bash-completion/completions/apt-get
is owned by bash-completion . So, I apply patch to "apt" only. Needed
patch for "apt-get" is essentially the same.

Also, I think "apt-get" (and similar) scripts should be transferred
from "bash-completion" package to "apt".

This patch is for current apt's "main",
9da15d149f97e0f26cf5b7e32405512a5e63523c .

diff --git a/completions/bash/apt b/completions/bash/apt
index 59a8eaa94..ff3d7bd24 100644
--- a/completions/bash/apt
+++ b/completions/bash/apt
@@ -173,6 +173,7 @@ _apt()
 fi

 # specific command arguments
+# we don't pass "--no-generate" to "apt-cache" because we want
completion to work even if caches are not present (#1034650)
 if [[ -v command ]]; then
 case $command in
 remove|purge|autoremove|autopurge)
@@ -187,7 +188,7 @@ _apt()
 return 0
 ;;
 show|list|download|changelog|depends|rdepends)
-COMPREPLY=( $( apt-cache --no-generate pkgnames "$cur" \
+COMPREPLY=( $( apt-cache pkgnames "$cur" \
 2> /dev/null ) )
 return 0
 ;;
@@ -195,7 +196,7 @@ _apt()
 if [[ "$cur" == .* || "$cur" == /* || "$cur" == ~* ]]; then
 _filedir "deb"
 else
-COMPREPLY=( $( apt-cache --no-generate pkgnames "$cur" \
+COMPREPLY=( $( apt-cache pkgnames "$cur" \
 2> /dev/null ) )
 fi
 return 0
@@ -204,7 +205,7 @@ _apt()
 if [[ "$command" == build-dep && ( "$cur" == .* ||
"$cur" == /* || "$cur" == ~* ) ]]; then
 _filedir "dsc"
 else
-COMPREPLY=( $( apt-cache --no-generate pkgnames "$cur" \
+COMPREPLY=( $( apt-cache pkgnames "$cur" \
 2> /dev/null ) $( apt-cache dumpavail | \
 command grep "^Source: $cur" | sort -u | cut
-f2 -d" " ) )
 fi

Bug#1034650: debian-installer: bookworm d-i rc1: apt-get clean breaks bash-completion

[Cyril Brulebois]

Control: clone -1 -2
Control: reassign -2 debian-installer
Control: retitle -2 debian-installer: investigate setting 
APT::Keep-Downloaded-Packages=false

Hi again,

Cyril Brulebois  (2023-05-04):
> David Kalnischkies  (2023-05-04):

[…]

> Entirely fair, thanks. Putting debian-boot@ in copy since currently
> the bug is assigned to apt… and others might have some feedback about
> my proposal(s).

No feedback in either direction at the moment, I'm going ahead as the
Bookworm clock is ticking.

> > Is it perhaps just wanting to remove /var/cache/apt/archives/*.deb ?
> > If so, could you switch to: -o APT::Keep-Downloaded-Packages=false
> > (well, probably in a conf file rather than on each command line).
> 
> There are various apt-get (not clean) calls, some of them wrapped, and
> I have no idea what the consequences of tweaking them could be. In any
> case, while deplying a conf file during the installation process could
> be attempted, it's far too late to try toying with that…
> 
> We can probably keep track of the suggestion via some bug report
> against debian-installer, so that someone can investigate that
> post-Bookworm (probably removing the possible tweak discussed below).

Doing so via the control commands above.

> For Bookworm, I'm tempted to add `apt-cache gencaches` after `apt-get
> clean` in finish-install, restoring the files bash-completion relies
> on.

I'll do that in finish-install and add a pointer to this bug report; I
won't add a Closes statement though, as I'm not sure what other users
will be implementing (for Docker etc.)… I'll leave it up to you to
decide what's best regarding this bug report (the -1 one).


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#1034650: debian-installer: bookworm d-i rc1: apt-get clean breaks bash-completion

[Askar Safin]

On Fri, May 5, 2023 at 8:53 PM Tianon Gravi  wrote:
> To be clear, there was nothing _certain_ in my response, merely that
> my first impression matched Cyril's for the very similar issue on d-i.
Okay, sorry for misunderstanding.

-- 
Askar Safin

Bug#1034650: debian-installer: bookworm d-i rc1: apt-get clean breaks bash-completion

[Askar Safin]

David Kalnischkies:
> The completion script explicitly disables the recreation on-the-fly as
> creating the files takes a while robbing users for many seconds of their
> interactivity. So, we can't just "fix" the completion script as that has
> a(nother?) set of users complain as well.

Deleting /var/cache/apt/*.bin and then running "apt-cache gencaches"
takes 0.22 on my (loaded) system.

I propose regenerate caches in completion scripts only if they are not
present. This will change situation from "not working at all" (in
default d-i install and in official docker image) to "working slowly".
Also, "working slowly" is just minor annoyance. But "not working"
means that the user doesn't know how to fix. Yes, merely calling
"apt-get install apt" will regenerate caches (in d-i case, not in
docker case), but how the user should know this? The solution is not
discoverable

-- 
Askar Safin

Bug#1034650: debian-installer: bookworm d-i rc1: apt-get clean breaks bash-completion

[Tianon Gravi]

On Fri, 5 May 2023 at 10:06, Askar Safin  wrote:
> Current apt behavior also breaks official debian docker images, I
> reported this here:
> https://github.com/debuerreotype/debuerreotype/pull/153 . The images'
> developer said this is apt completion scripts' problem

To be clear, there was nothing _certain_ in my response, merely that
my first impression matched Cyril's for the very similar issue on d-i.

I'd be very happy to spend some time with APT folks to figure out a
better configuration for the Docker images (both specifically on this
issue and in general!), especially as there seems to be some tension
around our current configuration that was unintentional and I'm sorry
for. 

(Having APT maintainers say "you should do X to accomplish what you're
looking for" is a much more defensible change than anything I could
possibly make up alone! )

♥,
- Tianon
  4096R / B42F 6819 007F 00F8 8E36  4FD4 036A 9C25 BF35 7DD4

Bug#1034650: debian-installer: bookworm d-i rc1: apt-get clean breaks bash-completion

[Askar Safin]

Current apt behavior also breaks official debian docker images, I
reported this here:
https://github.com/debuerreotype/debuerreotype/pull/153 . The images'
developer said this is apt completion scripts' problem

-- 
Askar Safin

Bug#1034650: debian-installer: bookworm d-i rc1: apt-get clean breaks bash-completion

[Cyril Brulebois]

Hi David,

David Kalnischkies  (2023-05-04):
> before I am playing bug ping-pong (as I don't think we can do much
> about this on the apt side), let me ask a d-i question first:

Entirely fair, thanks. Putting debian-boot@ in copy since currently the
bug is assigned to apt… and others might have some feedback about my
proposal(s).

> Why is d-i calling 'apt-get clean' to begin with?

Checking occurrences of `apt-get clean` (I didn't check for variations)
in our packages, I see two callers.

One is pkgsel (which deals with tasksel and pulls many/most packages),
and one later on in finish-install (which as you can guess runs at the
end).

> Is it perhaps just wanting to remove /var/cache/apt/archives/*.deb ?
> If so, could you switch to: -o APT::Keep-Downloaded-Packages=false
> (well, probably in a conf file rather than on each command line).

There are various apt-get (not clean) calls, some of them wrapped, and I
have no idea what the consequences of tweaking them could be. In any
case, while deplying a conf file during the installation process could
be attempted, it's far too late to try toying with that…

We can probably keep track of the suggestion via some bug report against
debian-installer, so that someone can investigate that post-Bookworm
(probably removing the possible tweak discussed below).

> This instructs apt(-get) to remove the deb file right after it has
> installed them with dpkg, which might even be beneficial for space
> constraint installation targets. At the very least it is a much used
> & tested option as it is the default with apt (but not apt-get).

OK, thanks.

> d-i could also "just" run 'apt-cache gencaches' to create the caches;
> but perhaps it is intended that the caches are gone (docker people e.g.
> like to do that I am told, but they nuke other things as well…).

The finish-install call can be traced back to:
  https://bugs.debian.org/586434

> > > - Fix "bash-completion". Make it work even if /var/cache/apt/*.bin are
> > >   not present (for example, by recreating them on-the-fly)
> 
> The completion script explicitly disables the recreation on-the-fly as
> creating the files takes a while robbing users for many seconds of their
> interactivity. So, we can't just "fix" the completion script as that has
> a(nother?) set of users complain as well.

With all the backstory, the interactions/dependencies between
bash-completion and apt's internal files became clear, thanks.

For Bookworm, I'm tempted to add `apt-cache gencaches` after `apt-get
clean` in finish-install, restoring the files bash-completion relies on.

That'd waste some little time, rebuilding some of the files that were
just deleted… but at least we wouldn't risk anything in the rest of the
installation process.

(Maybe making the extra call conditional to bash-completion's being
installed? Not sure about the “small optimization” vs. “different apt
files depending on the installed packages” balance.)


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#1034650: debian-installer: bookworm d-i rc1: apt-get clean breaks bash-completion

[David Kalnischkies]

Hi,

before I am playing bug ping-pong (as I don't think we can do much about
this on the apt side), let me ask a d-i question first:

On Thu, Apr 20, 2023 at 10:33:08PM +0200, Cyril Brulebois wrote:
> Askar Safin  (2023-04-20):
> > I don't know what should really be done. I can list possible solutions:
> > 
> > - Fix d-i. Remove /usr/lib/finish-install.d/60cleanup or replace it with
> >   something else
> 
> d-i is merely exposing a bug in bash-completion, so I don't think we
> should change anything there.
> 
> > - Fix "apt-get clean". Make it not remove /var/cache/apt/*.bin

Why is d-i calling 'apt-get clean' to begin with?

Is it perhaps just wanting to remove /var/cache/apt/archives/*.deb ?
If so, could you switch to: -o APT::Keep-Downloaded-Packages=false
(well, probably in a conf file rather than on each command line).

This instructs apt(-get) to remove the deb file right after it has
installed them with dpkg, which might even be beneficial for space
constraint installation targets. At the very least it is a much used
& tested option as it is the default with apt (but not apt-get).


d-i could also "just" run 'apt-cache gencaches' to create the caches;
but perhaps it is intended that the caches are gone (docker people e.g.
like to do that I am told, but they nuke other things as well…).


> > - Fix "bash-completion". Make it work even if /var/cache/apt/*.bin are
> >   not present (for example, by recreating them on-the-fly)

The completion script explicitly disables the recreation on-the-fly as
creating the files takes a while robbing users for many seconds of their
interactivity. So, we can't just "fix" the completion script as that has
a(nother?) set of users complain as well.


Best regards

David Kalnischkies


signature.asc
Description: PGP signature

Bug#1034650: debian-installer: bookworm d-i rc1: apt-get clean breaks bash-completion

[Askar Safin]

Четверг, 20 апреля 2023, 23:33 +03:00 от "Cyril Brulebois" :
> Clearly one of these.
It seems apt completion is located in 
/usr/share/bash-completion/completions/apt, which belongs to apt package.
So, anyway this belongs to apt package, so I reassigned


==
Askar Safin

Bug#1034650: debian-installer: bookworm d-i rc1: apt-get clean breaks bash-completion

[Cyril Brulebois]

Hi Askar,

and thanks for the report.

Askar Safin  (2023-04-20):
> I don't know what should really be done. I can list possible solutions:
> 
> - Fix d-i. Remove /usr/lib/finish-install.d/60cleanup or replace it with
>   something else

d-i is merely exposing a bug in bash-completion, so I don't think we
should change anything there.

> - Fix "apt-get clean". Make it not remove /var/cache/apt/*.bin
> - Fix "bash-completion". Make it work even if /var/cache/apt/*.bin are
>   not present (for example, by recreating them on-the-fly)

Clearly one of these. While I'm no apt maintainer and can't speak for
the semantics behind “apt-get clean”, it seems to me bash-completion's
relying on files that might not be there is the crux of the problem.


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#1034650: debian-installer: bookworm d-i rc1: apt-get clean breaks bash-completion

[Askar Safin]

Package: debian-installer
Severity: normal
Tags: d-i

Steps to reproduce:
- Install debian using recently published debian-bookworm-DI-
rc1-amd64-netinst.iso
- At first boot as a very first thing to do type "apt-get install a" (as
root, in normal root login shell)
- You will see that bash-completion doesn't work
- If you type "apt-get install apt", and try "apt-get install a"
again, you will see that bash-completion works

This situation is very bad. Imagine you installed Debian. Of course, the first
thing you may try to do is install some package to it. So it is quite possible
you will type "apt-get install ..." as a first thing. And you will see that
bash-completion doesn't work!

(Don't confuse "apt-get install a" with "apt-get insta")

I think I know why this happens. d-i has file /usr/lib/finish-
install.d/60cleanup. This file calls "apt-get clean". "apt-get clean" removes
"/var/cache/apt/*.bin". And "/var/cache/apt/*.bin" are needed for bash-
completion to work with apt.

If I remove /usr/lib/finish-install.d/60cleanup, then this bug disappears.

So, if you want to quickly fix this bug, simply delete /usr/lib/finish-
install.d/60cleanup.

I don't know what should really be done. I can list possible solutions:

- Fix d-i. Remove /usr/lib/finish-install.d/60cleanup or replace it with
something else
- Fix "apt-get clean". Make it not remove /var/cache/apt/*.bin
- Fix "bash-completion". Make it work even if /var/cache/apt/*.bin are not
present (for example, by recreating them on-the-fly)

(Disregard "System information" below)



-- System Information:
Debian Release: 9.13
  APT prefers oldoldstable
  APT policy: (500, 'oldoldstable')
Architecture: amd64 (x86_64)

Kernel: Linux 5.10.0-0.deb9.21-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), 
LANGUAGE=en_US:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled