Bug#1036952: rootskel: text installs on aarch64 lack glyphs for many languages

[Samuel Thibault]

Hello,

Cyril Brulebois, le jeu. 01 juin 2023 21:12:18 +0200, a ecrit:
> Do we have other ttys than just tty1 that people might want to switch
> to, and that might benefit from a similar adjustment?

This script is actually not used for the other consoles, so it has never
had any effect on them on any arch anyway.

Samuel

Bug#1036952: rootskel: text installs on aarch64 lack glyphs for many languages

[Samuel Thibault]

Emanuele Rocca, le jeu. 01 juin 2023 15:11:53 +0200, a ecrit:
> On 2023-05-31 05:46, Samuel Thibault wrote:
> > I'd rather see a patch like
> > 
> > if [ "$TERM" = vt102 -a `tty` = /dev/tty1 ] ; then
> > # Busybox's init uses a global TERM across all consoles.
> > # If the serial console is the default such as on arm64, that
> > # will force vt102 on the Linux VT. Fix this back so we get
> > # colors, utf-8, etc.
> > TERM=linux
> > fi
> > 
> > (to be tested etc.)
> 
> The following patch works. I've built a netboot image with patched rootskel,
> see attached screenshots for before and after the cure.

Thanks for testing!

I have uploaded it
(with fixed indent)

Samuel

> diff -Nur rootskel-1.135/src/lib/debian-installer.d/S40term-linux 
> /home/ema/debian/rootskel-1.135+nmu1/src/lib/debian-installer.d/S40term-linux
> --- rootskel-1.135/src/lib/debian-installer.d/S40term-linux 2011-01-20 
> 01:05:16.0 +0100
> +++ 
> /home/ema/debian/rootskel-1.135+nmu1/src/lib/debian-installer.d/S40term-linux 
>   2023-06-01 15:05:32.514361854 +0200
> @@ -1,5 +1,13 @@
>  export LANG=C
> 
> +if [ "$TERM" = vt102 -a `tty` = /dev/tty1 ] ; then
> +   # Busybox's init uses a global TERM across all consoles.
> +# If the serial console is the default such as on arm64, that
> +# will force vt102 on the Linux VT. Fix this back so we get
> +   # colors, utf-8, etc.
> +   TERM=linux
> +fi
> +
>  if [ "$TERM" = linux ] ; then
> if [ "$TERM_TYPE" = virtual ]; then
> echo -ne "\033[9;0]" # Turn off console blanking.

Bug#1036952: rootskel: text installs on aarch64 lack glyphs for many languages

[Cyril Brulebois]

Hi everyone,

Thanks for tracking down why the strange vt102 thing happens, and why it
mostly happens/was mostly reported on arm64 hardware… The patch below
(or further variations should there be any) looks mergeable to me.

Emanuele Rocca  (2023-06-01):
> On 2023-05-31 05:46, Samuel Thibault wrote:
> > I'd rather see a patch like
> > 
> > if [ "$TERM" = vt102 -a `tty` = /dev/tty1 ] ; then
> > # Busybox's init uses a global TERM across all consoles.
> > # If the serial console is the default such as on arm64, that
> > # will force vt102 on the Linux VT. Fix this back so we get
> > # colors, utf-8, etc.
> > TERM=linux
> > fi
> > 
> > (to be tested etc.)
> 
> The following patch works. I've built a netboot image with patched
> rootskel, see attached screenshots for before and after the cure.
> 
> diff -Nur rootskel-1.135/src/lib/debian-installer.d/S40term-linux 
> /home/ema/debian/rootskel-1.135+nmu1/src/lib/debian-installer.d/S40term-linux
> --- rootskel-1.135/src/lib/debian-installer.d/S40term-linux 2011-01-20 
> 01:05:16.0 +0100
> +++ 
> /home/ema/debian/rootskel-1.135+nmu1/src/lib/debian-installer.d/S40term-linux 
>   2023-06-01 15:05:32.514361854 +0200
> @@ -1,5 +1,13 @@
>  export LANG=C
> 
> +if [ "$TERM" = vt102 -a `tty` = /dev/tty1 ] ; then
> +   # Busybox's init uses a global TERM across all consoles.
> +# If the serial console is the default such as on arm64, that
> +# will force vt102 on the Linux VT. Fix this back so we get

Indentation looks sketchy :)

> +   # colors, utf-8, etc.
> +   TERM=linux
> +fi
> +

Do we have other ttys than just tty1 that people might want to switch
to, and that might benefit from a similar adjustment? In any case, I'd
assume people can just use screen's windows to get more consoles, so it
might not matter…


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#1036952: rootskel: text installs on aarch64 lack glyphs for many languages

[Alper Nebi Yasak]

On 01/06/2023 16:27, Emanuele Rocca wrote:
> Hi again,
> 
> On 2023-05-31 05:46, Samuel Thibault wrote:
>> The problem is that both are frown-prone. I guess there is a reason why
>> on arm the default console is set to the serial port, e.g. for simpler
>> debugging or something like that.
> 
> Also worth mentioning: there is no bug on real hardware (eg: RPi 3B+).
> 
> So far I think we've only heard about people getting this issue with
> qemu.

I've had this problem on my chromebook (rk3399-gru-kevin) since forever.
I don't know what's best here, but wanted to provide more context in
case it helps. Thank you all for working on it.


There's actually two different things that can make Linux set a serial
port as the preferred console on arm64 systems:

- An SPCR table on devices using ACPI
- A /chosen stdout-path property on devices using device-tree

I had once tried to change it on the kernel side, going through that
email thread [1] might make the how-and-why clearer. (I've silently
given up on that series)

Furthermore, the same thing happens in the installed system as well,
which will ask for your encryption password over a serial console that
you have no access to [2] unless you have ended up installing plymouth
transitively through a desktop environment.

Adding console=tty0 would persist into the installed system and fix
that, so I'm used to doing that as a workaround on my own systems. But
that disables the serial console on device-tree systems, so doing it by
default is a problem for SBCs etc. where people might genuinely be
trying to install over serial. (I'd still like it for "Graphical
Install" entries [3], but that's orthogonal to this).


[1] Prefer working VT console over SPCR and device-tree stdout-path
https://lore.kernel.org/lkml/20200513143755.GM17734@linux-b0ei/

[2] initramfs-tools: prefers serial console over framebuffer console
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=952452

[3] arm64 graphical installer fixes merge request
https://salsa.debian.org/installer-team/debian-installer/-/merge_requests/17

Bug#1036952: rootskel: text installs on aarch64 lack glyphs for many languages

[Emanuele Rocca]

Hi again,

On 2023-05-31 05:46, Samuel Thibault wrote:
> The problem is that both are frown-prone. I guess there is a reason why
> on arm the default console is set to the serial port, e.g. for simpler
> debugging or something like that.

Also worth mentioning: there is no bug on real hardware (eg: RPi 3B+).

So far I think we've only heard about people getting this issue with
qemu.

ciao,
  ema

Bug#1036952: rootskel: text installs on aarch64 lack glyphs for many languages

[Emanuele Rocca]

Hi,

On 2023-05-31 05:46, Samuel Thibault wrote:
> I'd rather see a patch like
> 
> if [ "$TERM" = vt102 -a `tty` = /dev/tty1 ] ; then
>   # Busybox's init uses a global TERM across all consoles.
> # If the serial console is the default such as on arm64, that
> # will force vt102 on the Linux VT. Fix this back so we get
>   # colors, utf-8, etc.
>   TERM=linux
> fi
> 
> (to be tested etc.)

The following patch works. I've built a netboot image with patched rootskel,
see attached screenshots for before and after the cure.

diff -Nur rootskel-1.135/src/lib/debian-installer.d/S40term-linux 
/home/ema/debian/rootskel-1.135+nmu1/src/lib/debian-installer.d/S40term-linux
--- rootskel-1.135/src/lib/debian-installer.d/S40term-linux 2011-01-20 
01:05:16.0 +0100
+++ 
/home/ema/debian/rootskel-1.135+nmu1/src/lib/debian-installer.d/S40term-linux   
2023-06-01 15:05:32.514361854 +0200
@@ -1,5 +1,13 @@
 export LANG=C

+if [ "$TERM" = vt102 -a `tty` = /dev/tty1 ] ; then
+   # Busybox's init uses a global TERM across all consoles.
+# If the serial console is the default such as on arm64, that
+# will force vt102 on the Linux VT. Fix this back so we get
+   # colors, utf-8, etc.
+   TERM=linux
+fi
+
 if [ "$TERM" = linux ] ; then
if [ "$TERM_TYPE" = virtual ]; then
echo -ne "\033[9;0]" # Turn off console blanking.

Bug#1036952: rootskel: text installs on aarch64 lack glyphs for many languages

[Samuel Thibault]

Emanuele Rocca, le mer. 31 mai 2023 17:29:31 +0200, a ecrit:
> >   1. Why is aarch64 special here?
> >   2. Where does that difference come from?
> 
> According to Jessica Clarke this is due to busybox using vt102:
> https://society.oftrolls.com/@jrtc27@mastodon.social/110459684352427882

Is it not possible to fix TERM after busybox dumbly set it?

> >   3. Which other architectures might be impacted if we were to change
> >  that?
> 
> I'm not sure, and I haven't tested the S40term-linux patch yet. However I can
> report that booting the installer by passing console=tty0 to the kernel fixes
> the problem (thanks alpernebbi!).
> 
> Which of the two changes (console=tty0 vs S40term-linux patch) is less risky?

The problem is that both are frown-prone. I guess there is a reason why
on arm the default console is set to the serial port, e.g. for simpler
debugging or something like that. And considering vt102 as "ok it's a
Linux console" is meaningless.

I'd rather see a patch like

if [ "$TERM" = vt102 -a `tty` = /dev/tty1 ] ; then
# Busybox's init uses a global TERM across all consoles.
# If the serial console is the default such as on arm64, that
# will force vt102 on the Linux VT. Fix this back so we get
# colors, utf-8, etc.
TERM=linux
fi

(to be tested etc.)

Samuel

Bug#1036952: rootskel: text installs on aarch64 lack glyphs for many languages

[Emanuele Rocca]

Hi,

On Tue, May 30, 2023 at 09:08:45PM +0200, Cyril Brulebois wrote:
> Philip Hands  (2023-05-30):
> > Apparently, this MR fixes the problem:
> > 
> >   https://salsa.debian.org/installer-team/rootskel/-/merge_requests/8
> > 
> > Although this does prompt the question of why aarch64 has TERM set to
> > 'vt102' at this point, rather than 'linux'.
> 
> Glancing at the merge request earlier, my first (intertwined) questions
> were:
> 
>   1. Why is aarch64 special here?
>   2. Where does that difference come from?

According to Jessica Clarke this is due to busybox using vt102:
https://society.oftrolls.com/@jrtc27@mastodon.social/110459684352427882

>   3. Which other architectures might be impacted if we were to change
>  that?

I'm not sure, and I haven't tested the S40term-linux patch yet. However I can
report that booting the installer by passing console=tty0 to the kernel fixes
the problem (thanks alpernebbi!).

Which of the two changes (console=tty0 vs S40term-linux patch) is less risky?

Bug#1036952: rootskel: text installs on aarch64 lack glyphs for many languages

[Samuel Thibault]

Samuel Thibault, le mar. 30 mai 2023 21:21:51 +0200, a ecrit:
> Samuel Thibault, le mar. 30 mai 2023 21:16:46 +0200, a ecrit:
> > I'm wondering what kind of console aarch64 is using: is that the Linux
> > virtual Terminal on an fbdev, or a ttyS console? Something else? The
> > kernel boot logs could be useful to determine that.
> 
> Ah, https://openqa.debian.net/tests/151286/logfile?filename=serial0.txt
> provides it.
> 
> May 30 06:37:27 init: starting pid 246, tty '/dev/ttyAMA0': 
> '/sbin/debian-installer'
> May 30 06:37:27 init: starting pid 247, tty '/dev/tty1': 
> '/sbin/debian-installer'
> 
> So it might be "something else" :) Is the shown output from ttyAMA0, or
> tty1?

Also, to be noted:

May 30 06:37:27 kernel: [0.531174] Run /init as init process
May 30 06:37:27 kernel: [0.531176]   with arguments:
May 30 06:37:27 kernel: [0.531177] /init
May 30 06:37:27 kernel: [0.531178] ---
May 30 06:37:27 kernel: [0.531179]   with environment:
May 30 06:37:27 kernel: [0.531180] HOME=/
May 30 06:37:27 kernel: [0.531181] TERM=linux
May 30 06:37:27 kernel: [0.531181] BOOT_IMAGE=/install.a64/vmlinuz

So init does get started with TERM=linux, not vt102. So it's something
in-between that seems to be setting that.

Samuel

Bug#1036952: rootskel: text installs on aarch64 lack glyphs for many languages

[Samuel Thibault]

Samuel Thibault, le mar. 30 mai 2023 21:16:46 +0200, a ecrit:
> I'm wondering what kind of console aarch64 is using: is that the Linux
> virtual Terminal on an fbdev, or a ttyS console? Something else? The
> kernel boot logs could be useful to determine that.

Ah, https://openqa.debian.net/tests/151286/logfile?filename=serial0.txt
provides it.

May 30 06:37:27 init: starting pid 246, tty '/dev/ttyAMA0': 
'/sbin/debian-installer'
May 30 06:37:27 init: starting pid 247, tty '/dev/tty1': 
'/sbin/debian-installer'

So it might be "something else" :) Is the shown output from ttyAMA0, or
tty1?

Samuel

Bug#1036952: rootskel: text installs on aarch64 lack glyphs for many languages

[Samuel Thibault]

Hello,

Cyril Brulebois, le mar. 30 mai 2023 21:08:45 +0200, a ecrit:
> Philip Hands  (2023-05-30):
> > Apparently, this MR fixes the problem:
> > 
> >   https://salsa.debian.org/installer-team/rootskel/-/merge_requests/8
> > 
> > Although this does prompt the question of why aarch64 has TERM set to
> > 'vt102' at this point, rather than 'linux'.
> 
> Glancing at the merge request earlier, my first (intertwined) questions
> were:
> 
>   1. Why is aarch64 special here?
>   2. Where does that difference come from?

I'm wondering what kind of console aarch64 is using: is that the Linux
virtual Terminal on an fbdev, or a ttyS console? Something else? The
kernel boot logs could be useful to determine that.

Samuel

>   3. Which other architectures might be impacted if we were to change
>  that?
> 
> > Also, I wonder whether there could be other circumstances when $TERM
> > is set to 'vt102' where this change could be problematic (I'm guessing
> > that the $TERM_TYPE checks deal with that, but I'm not sure).
> 
> Yeah, that's the same kind of questions one would get by combined my
> second and third questions…

Bug#1036952: rootskel: text installs on aarch64 lack glyphs for many languages

[Cyril Brulebois]

Hi,

Philip Hands  (2023-05-30):
> Apparently, this MR fixes the problem:
> 
>   https://salsa.debian.org/installer-team/rootskel/-/merge_requests/8
> 
> Although this does prompt the question of why aarch64 has TERM set to
> 'vt102' at this point, rather than 'linux'.

Glancing at the merge request earlier, my first (intertwined) questions
were:

  1. Why is aarch64 special here?
  2. Where does that difference come from?
  3. Which other architectures might be impacted if we were to change
 that?

> Also, I wonder whether there could be other circumstances when $TERM
> is set to 'vt102' where this change could be problematic (I'm guessing
> that the $TERM_TYPE checks deal with that, but I'm not sure).

Yeah, that's the same kind of questions one would get by combined my
second and third questions…



Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#1036952: rootskel: text installs on aarch64 lack glyphs for many languages

[Philip Hands]

Package: rootskel
Version: 1.135
Severity: normal
Tags: patch

Hi,

One can see the symptoms by looking here:

  https://openqa.debian.net/tests/151286

the orange "Soft Failed" boxes highlight some of the failing screens, where the
failure can be seen in the screenshot immediately preceeding the "Soft Failed"

I'll attach an example to save people from having to follow that link.

Apparently, this MR fixes the problem:

  https://salsa.debian.org/installer-team/rootskel/-/merge_requests/8

Although this does prompt the question of why aarch64 has TERM set to 'vt102' at
this point, rather than 'linux'.

Also, I wonder whether there could be other circumstances when $TERM is set to
'vt102' where this change could be problematic (I'm guessing that the $TERM_TYPE
checks deal with that, but I'm not sure).

BTW This seems to have some overlap with #263137.

Cheers, Phil.